Disgruntled Engineer Hijacks San Francisco's Computer System 1082
ceswiedler writes "A disgruntled software engineer has hijacked San Francisco's new multimillion-dollar municipal computer system. When the Department of Technology tried to fire him, he disabled all administrative passwords other than his own. He was taken into custody but has so far refused to provide the password, and the department has yet to regain admin access on their own. They're worried that he or an associate might be able to destroy hundreds of thousands of sensitive documents, including emails, payroll information, and law enforcement documents."
Frankly (Score:3, Informative)
If he met the same kind of problems I did when I worked in public sector and tried to push changes that would prevent this kind of idiocy in the first place only to be ignored because policy and process changes for better security and general good practice improvements meant management actually having to do some work then well, good on him for having the balls to do it.
Certainly in the UK in public sector those who work hard get shit on because those who refuse to do any work or could care less about a good job own the IT departments due to nothing more than hanging around for the job long enough that everyone higher than them dies/retires. As there's no accountability in local government and most other public sector these people can't be sacked or disposed of in other ways so they just hang around until they are 65.
RTFA (Score:4, Informative)
He was arrested AFTER he disabled everyone else's account.
What do you recommend they do next time, use a crystal ball or ouija board to predict who's going to pull such a stunt?
Read the Article - He wasn't fired. (Score:5, Informative)
Mmmm... (Score:2, Informative)
Noodleboarding...
on any Linux system you can: (Score:5, Informative)
in
root:$2$3bJ7DS4R$rV45lDlqNsfDRntfO1NCk0:14069:0:::::
look exactly like this:
root::14069:0:::::
this and you can log in to root without any password
maybe other *nixes are close enough to do the same (BSD or solaris)
on ubuntu the root shadow is a little differrent since it is disabled with an asterisk:
root:*:14069:0:::::
just remove the asterisk
Enough with the speculation! (Score:2, Informative)
Or maybe he hasn't. He might have done a lot of other things as well. A few of them are worth investigating but speculating publicly makes them sound a little hysterical.
Re:Tried to fire him? (Score:1, Informative)
I worked in local government for nearly 8 years. In a state that is "right to work," the local government was an island apart. "Tried to fire him" makes perfect sense to me.
If a manager wanted to take disciplinary action for performance issues, s/he had to keep a 6 month detailed log of everything that went on, including "coaching efforts." If within 6 months the employee showed *any* improvement, no matter how slight, the clock got reset.
If actual discipline ever happened, then the employee could appeal to a "jury of peers" regarding the discipline -- up to and including getting a job back if the employee was fired. In some recent cases, the reasons for termination were so obvious it boggles the mind how the employee got rehired, but there you go.
The idea behind the jury and the process was to keep personal issues out of the workplace. Another "good intention" on the road to hell.
I've seen enough informal geek-hostage-taking in government IT that I'm surprised we see so little of it in the news.
Which OS? (Score:1, Informative)
If it's Windows, why don't they just get that USB stick from MS that pulls admin user/pass and other sensitive information straight out of the system and provides it in a handy to read format?
Re:Backups? (Score:5, Informative)
Pretty much all Unix systems are hackable with local access.
I'm guessing either the entire file system is encrypted, or the problem is getting into an application that's running under the OS. Most times the OS isn't the final gakekeeper in high security; the application itself may run everything encrypted, and may very well have no easy way to restore access if a password is lost.
Re:Tried to fire him? (Score:2, Informative)
Did you miss the part where he has a government job? It can be extremely difficult to fire anyone for even the most egregious conduct. My father works for the USDA, and he has had people at his office caught sleeping more than once, and they did not get fired because it takes so much effort.
Re:This is why... (Score:5, Informative)
Private as in privacy, no. But private as in private property? Yes. If they don't allow someone to gather their things before they leave they could be looking at serious legal troubles.
No, it's pretty common practice. They can directly escort you out of the building without your personal property and they have a reasonable amount of time to gather up your stuff and get it back to you.
Things like car keys, wallet, jacket, briefcase, etc. yes. They'll escort you to your desk to pick those up. But gathering your pictures, books, etc. Nope. They'll do it for you or have you come back at a later date.
Re:I had a dream... (Score:5, Informative)
Re:Frankly (Score:3, Informative)
Why the hate? Because the organizations are often run by posers who want to "keep taxes low" and don't care whether the job gets done?
I worked for a major metro U.S. city's MIS director in the 80s who had the balls to tell the City Council that the job would take this many dollars and these many personnel or it wouldn't get done. They restructured his ass out of there, put in a yes-man and paid for a zillion bucks of private contractors (because they get paid in "different" money I guess). The project was your classic buggy debacle that ran many years over schedule and enraged citizens with the likes of $5000 water bills at rollout.
Re:Tried to fire him? (Score:3, Informative)
If you go back and look at when Yoda was first introduced as a character he didn't do that cutesy backwards sentence construction.
Your nostalgia is showing...
"Not far Yoda is, not far."
"Help you find him, I will"
That was from empire, right after Luke first meets Yoda. He's always done it, it's just gotten more pronounced as the movies progressed.
True but special case (Score:3, Informative)
Can you get into a system that has intentionally been locked off? And can you do so in a way that you're sure won't set off any little surprises that will, say, overwrite all backups with Star Trek Furry fan fiction, change who owns which files, e-mail/post confidential medical/legal data all over the interwebs, change data in a harmful way (switch names on booking records, for instance) and/or destroy all the relevant data?
Their safest bet short term might be to not try to access the system at all really. Pull the storage media and hope its not encrypted.
Now, we have no information indicating he's good enough to pull off some kind of massive lockdown in less than three weeks (even if he had planned something like this). But while you can always get access to a system if you have physical access, that doesn't mean you can get access with zero damage to the system.
Re:Frankly (Score:4, Informative)
That's because the anglo-saxon culture has a visceral hatred of everything that comes from the State. Anglo-saxons find that the State is the embodiment of evil, that it cannot do anything good and they will always try to gut it to it's simplest expression.
This dates back to 1215 when weak king John (Jean d'Angleterre [wikipedia.org]) got bullied by his barons and signed the magna carta [wikipedia.org] which essentially robbed him of most of his power. Thus started the notion that people other than the king could earn more power to the point of rivalling the State.
At the beginning, this was restricted to nobility, but when the industrial revolution saw the bourgeois rise to unprecedented wealth, to the point of even eclipsing the State's, the bourgeois managed to totally subvert the State and effectively gut it to an almost nonexistent value during victorian times.
Such distrust of the State is not found amongst other cultures. For example, the French have no problem with an overbearing State that nitpickingly regulates every aspect of their life, but since their culture will not demean the State, working for the State is not viewed as something bad, and the State will have no problem in recruiting competent people which will insure that whatever action the State takes, it will be done competently. Witness, for example, the network of high-speed trains ran by the French State Railroad, all developped by the State-Owned railroad. The same comment can be made about the extensive network of french nuclear power plant, all operated by a State agency, and exporting power to the rest of Europe.
Re:Just hack *his* hack (Score:2, Informative)
I would be very leery of booting the system to net or cdrom. What ELSE did he setup?
It's trivial to add a crontab or other mechanism that would F the whole system from many sources (S99xxxx, K99xxxx, altered shutdown, altered scripts, etc...). Even if he does provide the right password, how can you trust the system? Because he's being a dick, and holding out on passwords, this should be treated like a confirmed intrusion. They are going to be rebuilding the entire fleet of servers, if they have any sense.
Echoing other comments, yes, I'm sure we've all thought about ways to do this, or something like, and being a very creative bunch, I'd bet nothing short of a full system and source code audit would ever find some of things we could do.
They are going to be rebuilding the entire fleet of servers, if they have any sense.
----
For rent: 1 slightly used evil overlord/bofh/sysadmin
Technical background (Score:5, Informative)
For those who wonder what kind of working environment DTIS has:
PeopleSofts HRMS 8.x application software.
PeopleTools 8.4x, PeopleCode, SQL, SQR, COBOL, Application Engine, Oracle and HP/UNIX.
IBM hosts and DB2
Microsoft SQL Server 2000
Just look for open positions and you know what they are running.
Re:I smell a rat (Score:4, Informative)
FTFA:
"At a news conference announcing Childs' arrest, District Attorney Kamala Harris was tightlipped about what his motive may have been."
I think there's more going on here than we're being told.
You have to understand the nepotism and corruption that runs SF. The DA is purportedly Willie Brown's ex-girlfriend. She probably hasn't been told what to say yet because her handlers have been locked out of their computers. They have to cover up the corruption that contributed to this (or was merely exposed) first, then they'll decide what he did and throw the book at him.
Re:Tried to fire him? (Score:3, Informative)
Re:Got to love damage assessments (Score:3, Informative)
Re:Tried to fire him? (Score:3, Informative)
To expand on this, it's important to understand 'deep pockets'. Large organizations have deep pockets, which makes them juicier targets for 'unfair dismissal' lawsuits. Government is the worst for it, because you cannot (essentially) sue them out of business. For example, the Sheriff's Office must continue operating; so, if the lawsuit is lost, tax money is used to cover it. If we run out of tax money, we raise taxes.
Back on point: HR and Legal must assume that the dismissal is going to result in an unfair dismissal lawsuit. So they always push management to make sure the dismissal is 100% airtight. Anything less than that, and the government could lose huge sums of cash (far more than it takes to ride the bad employee for a couple years).
Re:$150K salary+bonus!!?? (Score:3, Informative)
Beautiful area of the country, but 150K doesn't go far out there.
Integrity (Score:5, Informative)
If you need a recognized code of ethics to tell you that sabotaging your ex-employer's system isn't right, then no code of ethics can help you.
Integrity and reputation is typically more profitable than malice and destruction.
I've been in the business a few years, and as you get older, you acquire positions of trust. You have too, you can't be "starting out" your whole career. This sort of behavior is a deal breaker. No one will hire him.
When laid off or fired. Collect your stuff, shake hands with your boss, tell them what is left to be completed, politely and with insight, try to be constructive with any discussions on the exit interview. Even a complete moron will leave a better impression than the greatest genius.
Once out, have a beer or two. Calm down. If you'r any good at all, when they are picking up the pieces of the layoff, they'll remember you attitude and professionalism and probably pay you contractor wages to do stuff while you collect unemployment and look for a new job.
Re:Tried to fire him? (Score:1, Informative)
Terminating employment in a government job is very different than in the private sector. People can not just be fired. The employee must have violated written policy and the supervisor must have it documented through HR. Then, very specific disciplinary action must occur before termination is even an option. If a step is missed, or if any documentation is missing, or if the employee disputes the claims, then it becomes very difficult, or even impossible, to have them fired.
Re:This is why... (Score:2, Informative)
I would not care. You can escort me if you desire but if you try to steal my stuff and hear glass breaking don't worry it is just me creating an alternate door.
I actually saw a terrible company that I was at escort someone out a week after they gave two weeks notice (WTF ???) It was real tacky and made the customer whose guards were used really mad.
Not so easy for sysadmins (Score:5, Informative)
So, routing rules from home. Public SSH keys on various border-servers with my USB-drive having the private keys, etc. They're all used for doing my job, and if I'm fired (not sure why I would be though) I'll just move on to the next one without tainting my career and doing something stupid to burn bridges. However, I could see a *bad* sysadmin using these same tools and more to entrench himself so deeply that you'd almost have to rebuild the entire infrastructure from scratch to find all the back-doors.
If this guy was a real dick (but a clever+smart one), knew it, knew he was going to be canned, and prepared for it... then how are you going to know that your authentication methods, your binaries, or even your kernels haven't been messed with in some way? MD5 sums only go so far when you have hundreds of systems tied together.
Re:Right to remain silent... (Score:2, Informative)
According to the Untied States Supreme Court in Doe v. United States, 487 U.S. 201 (1988), a defendant's right to remain silent means that a defendant cannot be compelled to provide a combination to a safe, even though he could be compelled to turn over a copy of a key to a lock box. I wonder if the court in California will follow this or will he be compelled to divulge the password.
Slight difference here is that the password is a work product not owned by him. It is not the combination of his own safe, or his own password to his system. Don't know if it will play out that way.
Laugh or cry? (Score:3, Informative)
I don't know whether to laugh or cry. What kind of incompetence does it take on the part of the rest of the system administrators there to be unable to regain access to a system to which they have physical access?
Re:Backups? (Score:5, Informative)
Unless you are inept, which, given that this is a government system, could be a plausible explanation here.
Re:Backups? (Score:3, Informative)
What's more, if he choose a REALLY good password, brute force decrypt might take a *long* time...
Most of the password circumventions I've seen for windows don't actually crack the password but let you overwrite it with a new one. It's not so great for undetected access but it's just fine for taking control of a system that has been locked up by a disgruntled employee.
I worked at a school district once where an art teacher got canned due to budget cuts. Before she left, she changed the passwords to a bunch of computers she managed to get for the school district on a special grant. I have to admit, I felt bad as I removed the passwords using a linux password breaking floppy, but the computers did belong to the school.
Big iron is another story - but that's where your maintenance contracts should allow for the vendors to come in and undo the damage. There must be a way to connect its OS drive to another computer and make changes to whatever password/shadow files are there to allow for access once it's booted up again.
Re:Backups? (Score:3, Informative)
Re:This is why... (Score:1, Informative)
Re:This is why... (Score:3, Informative)
Wikipedia has an article on Structural Abuse [wikipedia.org], you might want to consider it.
Longtime harassment and abuse will lead people to irrational behavior.
Just because someone had a bunch of attempts to be fired doesn't mean that the manager was unjustified in his position.
Plus, no evidence of his tampering was available until implemented... as a result you cannot punish someone for acts that they "might" do.
Re:This is why... (Score:3, Informative)
They should have forced him to release all admin passwords a long time ago, and performed regular system audits. Any non-compliance on his part would have just helped them fire him faster.
It sounds like they have a management problem in general. Any sizeable company or government body needs to have regular audit and appraisels of their security. The network admin should NOT be in charge of security, there needs to be at least one other person involved.
Duh.
Here's the problem with that "simple" notion. Once you start treating him differently from everyone else, it sets up a pattern of abuse or harassment. Especially, if you start restricting their work so that they cannot perform the duties required by their position. When their performance review rolls around, BOOM, they're hit with a brick wall of "you didn't accomplish anything that was expected of you." I'm not entirely sure about California, but I think at that point, someone can make a case for being fired without reason... and as part of a Union, that answer is usually insufficient to fire someone.
Re:ha (Score:4, Informative)
IMO this guy had a personal disagreement with his manager, and was fired because that guy was working full time trying to find a way to fire someone he disliked.. considering he earned an extra 30k as a trouble shooter and was able to pull off a time bomb, i'm sure he knew what he was doing with technology...
And this incident proves the manager was completely in the right to get rid of him. No amount of tech skills are worth the damage that an unprofessional employee can cause.
Very keen observation (Score:3, Informative)
Having good knowledge of events, concepts and other things published by mainstream media at a later date, you've really hit the nail on the head on what I've observed, with regards to reporters. Reporters most of the time don't have the technical acumen to realize the nuances of what they're reporting, the context and when to spot bad data. Eg. While reporting an accident, they blindly take the "speed was a factor" quote from police. Since parked cars rarely spontaneously collide, it's a falsehood. When GPS was new in the early 90's I read multiple articles that stated, "GPS tracks your movements". Ummmmm....no. The satellites have no clue how many users are out there and where they may be.
In a few weeks this will blow over, they'll get their access back, the guy will be in jail and none of what was reported in the FA will come to pass. A few managers may get fired for being incompetent though, which you saw no mention of in the article.
Re:Backups? (Score:3, Informative)
Binary patching is not by any means preferred, but if the source was MIA it might have been the only option available.
Re:Backups? (Score:4, Informative)
Re:Resignation Letter (Score:3, Informative)
Also, I'm unclear as to the legal theories by which a bad recommendation could be illegal.
That's because giving a negative reference is perfectly legal (1st Amendment and all), but it is generally not good business practice. The reason is, as you note, the potential for legal action accusing company of libel/slander/defamation.
I am a landlord, and this applies in my business for renters. Landlords get sued all the time for negative references, so some won't give references at all. Those of us who do simply remove any subjectivity from the conversation (who can really say if a renter was "clean", "a good neighbor", etc.?).
The way I handle it is to get permission from an applicant to seek references from former landlords, fax that permission letter to the former landlord, along with the following questions:
The answers to those questions tell me everything I need to know from that reference. We never get into if he paid rent on time, or often he was late, or if he damaged the unit, etc. These are all things that could be debated in a court of law (is rent on time if it's paid after the first but before late fees kick in?, was that damage or was it really normal wear and tear?) On the other hand, the answer to question #3 could be "no" for any number of reasons, but it is not debatable, because it asks the landlord specifically for his opinion, and only he knows his own opinion. No one can question whether or not that is his opinion.
Incidentally, I don't ask the former landlord if applicant was evicted or caused damages. This information is public record, so I already know.
Re:Backups? (Score:2, Informative)
Re:Backups? (Score:3, Informative)
If the government abolished public schools and the taxes that go along with them, I would have about $200 more per month
I'm calling BS on that. If you are paying $400 a month in taxes, it seems reasonable that you don't have the money to send your child to a decent private school or tutor anyway. (By all means send them off to be indoctrinated at your local church school, though. See how that works out for them.)
We are spending so much more on defense than education. Get your facts straight (assuming you live in the US).
public school ... faced with a teacher trying to dumb down the coursework for the biggest retard in the class
Not this fallacy again :( It sure doesn't help that we are taking away funding from the worst schools, either.
english, religious history, and the 'home economics' ... geek stuff like programming, and basic electronics, along with political science.
What about history, geography, chemistry, physics, physical exercise, creative arts, music, social skills, etc?
we both hate higher-level math
You hate it? That certainly speaks well for both of you and your ideas about education.
Dumping more money into our schools won't solve the problem.
Yeah. And taking money away from them won't either.