The Inside Story On the San Francisco Network Hijacking 471
snydeq writes "A source with direct knowledge of San Francisco's IT infrastructure has tipped off Paul Venezia to the real story behind Terry Childs' lockout of San Francisco's network, providing a detailed account of the city's FiberWAN, interdepartmental politics, and Terry Childs himself. Childs pleaded not guilty to charges of tampering yesterday and is being held on $5 million bail. According to the source, Childs' purview was limited to the city's FiberWAN — a network he himself built and, believing no one competent enough to touch the network but himself, guarded religiously, sharing details with no one, including routing configuration and log-in information. Childs was so concerned about the network's security that he refused even to write router and switch configurations to flash. But what may prove difficult for the prosecution in its case against Childs is that his restricted access to the network was widely known and accepted among managers and the city's other network engineers. Venezia, who has been suspicious of the official story from the start, suspects that the Childs case may be that 'of an overprotective admin who believed he was protecting the network — and by extension, the city — from other administrators whom he considered inferior, and perhaps even dangerous.' Further evidence is that fact that the network, from what Venezia understands, has been running smoothly since Childs' arrest."
and in stargate news..... (Score:2, Interesting)
Re:and in stargate news..... (Score:5, Insightful)
Mods on crack (Score:5, Insightful)
This analogy is spot on, and whoever modded it off-topic obviously is incapable of understanding the topic and shouldn't have had the keys to the mod-car in the first place.
Re:Mods on crack (Score:5, Funny)
It's my considered opinion that many people with keys to the mod-car shouldn't have been allowed off the short bus.
Re:Mods on crack (Score:5, Insightful)
Except that Childs is not the cars owner, he's the mechanic hired to fix and maintain it for the owners (the city government of SF).
Re:and in stargate news..... (Score:4, Insightful)
It seems that both his co-workers and his management recognized his considerable expertise. So, if he thought that he was working with incompetent freeloaders, then he probably was right. This has nothing to do with primadonna status. The problem here is workload and responsibility.
I support several hundred servers and two of my backups are telecommuting from the west coast. Their Unix expertise is limited to creating user accounts, pushing power buttons, and sending me emails with stupid questions. I don't go so far as to lock them out of "my" servers, but I do my best to keep these two characters busy in their sandbox.
This has nothing to do with my (admittedly very considerable) ego, but has everything to do with me being able to enjoy weekends without being called to fix various problems. When something breaks, the ops calls me and not the two clowns in California. Any problem - big or small - they will find me, wake me up, drag me into a telecon, where I would have to fix the problem while simultaneously explaining to them how I did it and answering "are we there yet" questions from various random managers who couldn't sleep at night.
I would love to have a colleague whom I can trust to do upgrades and architectural changes, so I can spend more time fishing. The way things are right now, I am forced to keep other sysadmins at an arm's length just so I don't have to work even more hours (for which I am not being paid) to clean up their messes.
Re:and in stargate news..... (Score:5, Insightful)
Consider mentoring. The God complex management style rarely works out well in the end.
Re: (Score:3, Insightful)
Re: (Score:3, Insightful)
I see it the other way around. "I'm the best, and I always will be" attitude is often used by good technical, but socially-immature, admins who have no people skills.
In the long run, the company would be better with an admin who both has and is good at both technical skills and people skills, and you know it. Are you worth more to your employer if you can both admin machines and also get along with co-workers? Hell yes. You'll see this very clearly if you're ever asked to manage a large group of people.
Re: (Score:3, Insightful)
You're a twit. People learn by making mistakes. As senior guy it's your job to create learning experiences (situations in which your backups can make mistakes without doing serious damage) to teach them the concepts of the care you take when doing your work.
If you're not training them and you're actively denying them the ability to make the same mistakes you did once, you're doing yourself, the company and them a disservice and an unprofessional job.
Is this really the case? (Score:4, Insightful)
Re:Is this really the case? (Score:5, Insightful)
I find that easy to believe. Even easier to believe that they didn't know this was the case, or knew but did not understand.
Re:Is this really the case? (Score:4, Insightful)
This doesn't sound reasonable. If management behaved like this they would have been fired before this guy was -- the management problems would be greater than the technical ones.
Re:Is this really the case? (Score:5, Insightful)
Never worked for the government, have you? ;)
Management is where people who are too incompetent for technical work go. No one gets fired, they get moved to different departments. As a last resort, they get assigned to 'special projects' for about a year in the hopes that everyone will forget what an imbecile they are, and will be safe to move back into the management structure.
Re:Is this really the case? (Score:5, Insightful)
Re: (Score:3, Funny)
You just described Harris County, Texas to a degree that's downright frightening. Recently the payroll information for the entire county was put online by the Houston Chronicle newspaper and a quick look around has confirmed, at least to my mind that the straightest path to a high salary and zero responsibility here is to excel at being a nincompoop.
I'm getting right on that by the way. Look for me in upper-middle management in about three years time.
Exit stage left (Score:5, Insightful)
.
modded +3, Informative.
but this attitude sets off alarms.
exposing a geek who despises his supervisors and is used to thinking of the server rooms as his personal playground.
Re:Is this really the case? (Score:4, Interesting)
Re:Is this really the case? (Score:5, Insightful)
It's nice to believe that but, to abuse an oft-quoted phrase, quis sacko ipsos pointyhaires?
Before you can fire someone for being a complete idiot, you have to not be totally out to lunch yourself. More importantly you have to possess evidence to back up your decision which is at least strong enough to outweigh the political costs of making it.
If you think this all sounds like a load of crap, then consider yourself lucky that you have never been in the middle of it.
Re: (Score:3, Insightful)
This doesn't sound reasonable. If management behaved like this they would have been fired ...
Hah! You clearly have never worked for the government. It may not sound reasonable, but bureaucrats are almost always some combination of ignorant and oblivious. I mean, part of the reason they put this guy in charge is that he's probably the only person who knew how to do anything. And you have to ask yourself, who's going to fire these marginally competent managers? Their marginally competent bosses? People who know what they're doing are unfortunately the exception in government. Most competent folks fin
Re: (Score:3, Interesting)
It's not actually the case of people not knowing the passwords or such, from what I've read
Re:Is this really the case? (Score:5, Interesting)
I post AC because of my position, which is basically a guy who was hired as the second network tech to help manage the network for a sizeable city (large enough that we host several professional sports teams). I had no real qualifications other than knowing how to google my way out of most basic computer situations. My supervisor managed all City-owned Cisco equipment and it has only been 2 of us for 2 years. We manage over 300 Cisco devices at over 100 sites and I can honestly say that after reading a few more details on this story, I can easily understand how this can happen in a local government. I believe that the problem is in management. We have similar problems in our City regarding the lack of passing of knowledge and lack of staffing, but we have a good security team that knows more about Cisco networks than the 2 of us that regularly work on the Cisco equipment in our City. They are not normally watching our backs (that we know of) but they would certainly do so if they got a bad vibe about us. We have to share passwords with them and they have as much access to our equipment as we do. It is simply a requirement in a publicly owned system that knowledge is shared. Taxpayers have payed for the equipment and expect that there are not single points of failure. There are many reasons that more people than work on one thing on a regular basis have knowledge of and access to the most basic systems. If there was no redundancy, then it is a fundamental failure of management.....I'm not saying the guy should have set one password and not passed it on.....but I understand.
Comment removed (Score:5, Interesting)
Re: (Score:3, Funny)
Re:Is this really the case? (Score:5, Insightful)
It seems pretty idiotic to me. I still think they should throw this guy in the clink, but at the same time, I think some of his superiors should be told to collect their belongings and then have security escort them through the front door, because there was a colossal breakdown of management here if a single guy was permitted to basically hold the entire network's architecture in his head.
Re: (Score:3, Insightful)
Re:Is this really the case? (Score:4, Informative)
My best guess from my understanding of SFGOV is that his boss answers directly to the mayor.
Most of San Francisco government answers directly to the mayor. San Francisco is a city and county so it has no city council, or city manager only county supervisors, a controller and a mayor, along with many other oddities that are only in San Francisco.
Fortunately/unfortunately there is a civil service commission and fairly strong employees unions that cover all but the political appointees, somewhat muting the mayors power. (oh and the board of supervisors and the voters can override the mayor when ever they feel like it, but the mayor still has ultimate control, unless he has been over ridden. )
Re:Is this really the case? (Score:5, Insightful)
Au contraire--it's the rule, not the exception... (Score:5, Insightful)
In my experience, it's a rare company indeed whose managers can fathom the implications of a situation like this. In general, I'm unable to get management to even understand Rule Zero of system administration. Which is: Do everything you need to do to be drop dead certain that you always have a reasonable backup of your important systems. This doesn't sound too difficult, but in practice it's difficult to convince managers that an event that could happen with probability == 0.01 could ever happen...
Re:Au contraire--it's the rule, not the exception. (Score:4, Insightful)
"Sure, the odds are 1000:1 against that I'll be hit by a bus, but there are a lot of ways disaster can strike, and they add up. You willing to ignore 5:1 odds? How about 10:1, or 15:1?"
This is why technical people need to strive to learn to have relationships with supervisors of a non-technical bent. From reading the article, it seems that Childs' demeanor meant that he could easily be dismissed as the brilliant-but-whacked-out-network-curmudgeon. Fair or not, that means that all of his concerns could be waved off as paranoia (for instance, him trying to get an information security policy in place). Unfortunately, the wisdom of our caution only becomes evident when a disaster occurs or is narrowly averted (e.g. "Thank God we backed that data up!").
On the other hand, non-technical managers should learn to not instantly dismiss the concerns of technical people as unlikely or unrealistic.
Re: (Score:3, Informative)
You've obviously never worked for local government.
Re:Is this really the case? (Score:4, Informative)
This all depends on who accepts it when. If when Childs started his lockdown, he was under the watch of a manager that either didn't care, or agreed with him, and so he did as he pleased.
A lot of IT organizations have a single person that is the core, the one person that has comprehensive knowledge of all systems and fully understands how they interact. These are the people that are brought in on any major problem or decision, and whose input counts more than double. (and often simply hold "veto power") Now I'm not saying this is a good thing, I'm just saying it happens from time to time and you have to accept that. Some systems just evolve in this direction, and once they get past a certain point, it becomes very hard to change them.
There IS one easy way to solve these problems, but it involves the managers taking a walk out on a shaky limb and take some heat. One example is a week of paid leave. On Monday Joe's manager announces "Joe is on paid leave effective immediately. (no warning to Joe OR the staff in advance of this) Go home Joe, see you in a week and enjoy your paid time off, courtesy of the company." Then, "OK for the next week you are on your own. NO ONE is to call, page, IM, email, or otherwise contact Joe for ANY REASON. Joe got hit by a bus this morning on the way to work, that's how you will behave. You are to keep written track of every problem you run into this week that you would normally rely on Joe to help with. Do not simply shelve problems for next week - treat them like Joe is never coming back." Any critical questions you bring to ME, and I will call Joe if it's really necessary, but I will not be happy about it, and be prepared to justify to me that you've already tried everything else possible. If I find someone is hiding problems for next week there will be serious disciplinary action taken.
Needless to say, when Joe gets back on Monday, the next 2-4 weeks will probably be planned out, documenting things and teaching people how to do stuff. You could also make this a two week leave depending on your situation. If you're a big organization, the longer the better, but at the worst three weeks will shake out most of the bugs. This also gives the managers a very clear picture of how well distributed knowledge is within the department. You've probably heard someone say "but what if you got hit by a bus tomorrow?" when discussing something you are the only one that knows how to do. Now you get answers. We call this the "hit by a bus test". Any decently sized IT department with one central person should conduct this test periodically, say every two years. The first one should be a gimme. If on the second test, things have not improved over the first, time to take disciplinary action. Letting one of your staff continue to hold the keys to the kingdom is unacceptable and is everybody's fault to some degree.
Re:Is this really the case? (Score:4, Insightful)
As a sysadmin who is part of a small team that "holds the keys", I find your comment interesting. In most organizations, you don't have to specifically plan for a "hit by a bus test", because it happens all on its own. Don't your employee's take vacations? Don't they ever call in sick? If your employee's have to call the guy on vacation, that is a HUGE HUGE HUGE indicator that there is a problem.
He's still not justified... (Score:5, Interesting)
You can try and defend him and glorify him all you want... but as a professional system administrator he should have known that his singular access and pathological behavior was more dangerous than helpful.
What if, instead of being fired he was the victim of an accident or crime? What if he had a health problem? What if a serious, life threatening issue came up (say, you know, an earthquake) that caused the system to be unstable and, at the same time, prevented him from getting there to fix things?
He's still a criminal. But, he's not alone in his behaviour. Whoever his managers are sound to be guilty of criminal negligence. This never should have been possible in a city government the size of San Francisco. Especially when it comes to critical infrastructure. If I were a citizen of San Fran I'd be asking why heads aren't rolling at the highest levels. Why was this allowed to happen? In San Francisco, where you think they'd have no problem finding competent replacements.
Absolutely mind boggling.
Re:He's still not justified... (Score:5, Interesting)
If this was a case of "He was the only one with the passwords and knowledge, we stupidly fired him without getting that info, and now we realized we're screwed" then he isn't a criminal. His boss maybe, but not him.
Hell, even if the situation was "tell us the info so we can replace you - no - you're fired", he still isn't a criminal. Other than maybe stretching a denial of service crime to fit, other than he hasn't really denied them a service if it is still running.
Re:He's still not justified... (Score:5, Insightful)
We still don't know all the details. Perhaps all the accusations are trumped. But, if when his performance became a question he started hiding backups, monitoring his managers' email exchanges and is now not cooperating, he's definitely a criminal.
How can you possibly argue otherwise? Sure, he's the network admin, but does that authorize him to read people's email without authorization?
Sure, he's the admin, but does that give him the right to create a situation that basically takes the city's IT infrastructure hostage?
I'm not questioning that his superiors should share the larger part of the blame here. But I can't see how he's not at all at fault.
Re:He's still not justified... (Score:5, Insightful)
>
How can you possibly argue otherwise? Sure, he's the network admin, but does that authorize him to read people's email without authorization?
Not at all. But then charge him with that, not some pseudo-terrorist computer tampering charge.
Re:He's still not justified... (Score:5, Insightful)
>>How can you possibly argue otherwise? Sure, he's the network admin, but does that authorize him to read people's email without authorization?
>Not at all. But then charge him with that, not some pseudo-terrorist computer tampering charge.
The Electronic Communication Privacy Act of 1986 protects administrators if "in the performance of their duty" they read email. Please note the date. If you are unfamiliar with it, you should be even if you're "just a user", no excuses.
He's an administrator. He's shielded.
Y'all should know that by now.
You should also know that if you store your email on company servers/isp servers, they get /less/ protected as time goes on, with most protection going to those "in flight" and least to those being stored for over a year.
If you have anything confidential, encrypt it and remove it from your provider's machines and store elsewhere. If you don't ever want the admin to see the email in flight, then end-to-end encryption. These days it's easier than the mid 1980's.
OB On Topic: I can see where he's coming from. A network administrator, if he's doing his job, gains a bit of paranoia. Sometimes that can become unhealthy, and it appears that he's crossed the line into "unhealthy". Criminal? I don't think so. It appears that he's been severely mismanaged by those who never understood "Mack Truck Syndrome". One guy for an entire city? I'm not sure who's crazier, the management or him.
--
BMO
Re:He's still not justified... (Score:4, Informative)
I think you're full of brown smelly stuff.
The ECPA only seems to apply to common carriers and public information services. I don't see any evidence it provides any liability for the sysadmins of internal networks.
If you're not IANAL, here it is:
http://www.usdoj.gov/criminal/cybercrime/wiretap2510_2522.htm [usdoj.gov]
And even if so, you're being really retarded if you think that reading his bosses' email falls under the "system monitoring" provision of the law.
Re:He's still not justified... (Score:5, Informative)
"The burden of proof is on you to back up your bullshit, and I'm a calling you on it. Quote some laws here, if you can."
I'll do you one better:
I'll point you at a book on the matter:
http://www.amazon.com/Netlaw-Your-Rights-Online-World/dp/0078820774 [amazon.com]
And I'll quote from here:
http://www.rbs2.com/email.htm [rbs2.com]
The executive summary of what I've been talking about and what you've been talking out your ass about:
"Reading e-mail that is stored on a computer is not an "interception" under 18 U.S.C. 2510, et seq., because an interception must be contemporaneous with the transmission of the message between different locations. Steve Jackson Games v. U.S. Secret Service, 816 F.Supp. 432, 442 (W.D.Tex. 1993), aff'd, 36 F.3d 457, 460 (5thCir. 1994). This holding has been accepted in several subsequent cases, including Wesley College v. Pitts, 974 F.Supp. 375, 384-390 (D.Del. 1997); U.S. v. Moriarty, 962 F.Supp. 217, 221 (D.Mass. 1997); Bohach v. City of Reno, 932 F.Supp. 1232, 1235-36 (D.Nev. 1996)."
--
BMO - Not a lawyer, but dammit I can read for myself.
in the performance of his duty (Score:4, Insightful)
That law is there to make it possible for administrators to do their work. If you are working with emails, and you happen to see a few, you don't go to jail for it.
But monitoring his bosses' email so you can tell what they are saying specifically about him is highly unlikely to be in his job description, and thus he is not protected when he does that. Nor should he be.
Re:He's still not justified... (Score:5, Informative)
"There is a big difference between "in the performance of their duty" and "because are able to do so, they felt like doing so and so they went ahead and did so.""
The thing is you have to prove it that an admin did it for BOFH style "shits and giggles" or some other motivation other than official use - beyond a reasonable doubt. That's a pretty big hurdle for a prosecution. Some would call it an impossible hurdle.
That's for when the email is in-flight. Once it hits storage, an admin basically has free reign. As email gets older and older, it gets less protected. Beyond 180 days it's unprotected - the gubmint can even do a search without a warrant.
Email isn't as protected as paper documents, as the last time this came up before the 6'th circuit, it was refused review on procedural grounds.
Don't ask me, go read the law yourself. ECPA of 1986.
If you think that the legal privacy of email is pretty weak because of the ECPA, this was an *improvement* on privacy back in 1986 because prior to that, email was basically equivalent to shouting out the window (and sometimes still is). Once the ECPA passed, BBS operators like myself became paranoid so we decided to put up disclaimers announcing that users should not expect privacy. Such disclaimers during login and registration notified the users and thus shielded the admin from privacy lawsuits and such. Some people think that this gets rid of plausible deniability, because once you say your users have no privacy, the guys in the FBI PartyVan parked in your driveway might suspect that you know what your users are doing, or so the theory goes. But a section of the CDA of 1996 supposedly shields the admins from the actions of a service's users. It gets really complicated if you research even a little bit of this stuff.
--
BMO
Re:He's still not justified... (Score:4, Insightful)
As a system administrator - oh I assure you, they ~can~.
Now whether they can use anything they found in there to assist police in their prosecution of someone, or whether they can even publicly or privately even admit that I did it is another story - but admins can look, and some do (I don't, out of personal conviction.)
Trust me on that. And they remember what they see.
Re:He's still not justified... (Score:4, Insightful)
If he really did explicitly "hold the network hostage", actually said "I'll trash it if I don't get what I want", then he commited a crime. But what it sounds like so far is "Do your job the way we want, not your way" and he said no and was fired for it, which is generally not a criminal act.
I've known half a dozen people who "knew things" that would ruin their company if they were hit by a bus. None of them would get charged with a crime if they refused to give up that information *after* being fired(although their company might get sued by the shareholders). But none of them are in IT.
As for the email, from the correspondance provided, it doesn't say if he had access to the city's mail servers, but then he isn't being charged with breaking in to them either. Seeing as he ran the network, it'd probably be easy to sniff and read the email "on the wire" without breaking into a computer, since I doubt anyone in the city government used encryption.
Ok, now I'm being a bit nitpicky, sorry:), but how often do we compare email to sending postcards? Other than cellular communications, where else is it illegal to detect something broadcast in the clear?
Re:He's still not justified... (Score:4, Insightful)
how often do we compare email to sending postcards?
On the Cypherpunks mailing list, all the time. On Slashdot, I don't think I've ever seen anyone bring it up. Email is just that - a postcard. If you care about the privacy of your mail, encrypt it.
Re: (Score:3, Interesting)
I am not gonna use my normal login on this and you will understand in a second.
I have been working in, on, and around computers (hardware and software) since before most of you were born. Some of you, your parents. And at almost all of the companies that I have worked for, there was something that "everybody knew" that would have gotten the company heavily penalized if not shut down if the word got to the appropriate regulatory agency.
Example: did you know that the sales tax you paid on your CompUSA purch
Re: (Score:3, Informative)
Putting a network card into promiscuous mode is not the same as using root access to copy your boss's email store, nor is it running the password file through jack the ripper.
This sort of logic is where nerd myopia falls right on its four-eyed face. If he was reading in on personnel-related email, it really doesn't matter what measures he used, he still fucked up. Especially so if he acted on them.
The argument that his bosses were l00s3rz because they were conducting normal business through email without any special encryption doesn't fly anywhere sorry. Professional job, professional rules, l335ness does not apply.
Re: (Score:3, Funny)
> Sure, he's the admin, but does that give him the right to create a situation that basically takes the city's IT infrastructure hostage?
He's just following the most basic security rule that has been told here thousands of times:
Never give anybody your password! Ever!
Re: (Score:3, Informative)
I have been following this story in the San Francisco Chronicle.
According to their reporting, he was asked by management for the passwords. He said no.
Then he was asked by police for the passwords. He still said no.
Then they had him arrested.
The reporting by various news organizations has been marred by confusion
on the part of reporters and average people between controlling network hardware and controlling
data on various servers. They often seem to describe it as data being
stored on the network he contr
Re:He's still not justified... (Score:5, Interesting)
Yah, I agree it he probably is a huge jerk and should've given up any passwords or other info when he was canned, just out of professionalism(and maybe a little "here's the knife, cut your own wrists"). But I think the management is probably blowing this out of proportion to cover their own asses.
A company I shared a parking lot with during the dot bomb laid off their entire programming department a few months after they hit release and hired an outside company to "sanitise" the computers in the building. After the contractors wiped the CVS server, management threatened to sue/charge several of the programmers for "mislabeling" the CVS server deliberately so that would happen(it was labeled "Walgreens", bad pun).
That fell flat eventually, the guy who proposed the 100% layoff got the axe for it, and I heard the story from a couple of the programmers that were contracted back to get things back up to snuff(ie, they "failed" to destroy "illegal" backups and were able to save the company's bacon).
Re: (Score:3, Insightful)
Yah, I agree it he probably is a huge jerk and should've given up any passwords or other info when he was canned, just out of professionalism(and maybe a little "here's the knife, cut your own wrists").
Granted this is all speculation but I could see him feeling he was in a no-win situation. If he gave them the passwords and documentation and they fucked up the network they'd come after him claiming he sabotaged it. If he didn't, they'd claim he was holding it hostage. If you were in that situation (or just felt like you were), which option would you choose?
Are you sure he's a criminal? (Score:5, Interesting)
In America, they have to prove that first. Looking at the statute, it seems it all comes down to the issue of "without permission." The main point the article makes is that he might have had at least understood or standing permission to do most or all of what he did. Just like when you take your parents' car somewhere as a teenager, it isn't theft if it's understood that you are allowed to use it.
The article is one-sided, and his alleged refusal to give up the passwords looks bad (perhaps he is remaining silent until he speaks with counsel), but proving he didn't have permission might be hard. Ergo, no criminal.
Re:Are you sure he's a criminal? (Score:5, Insightful)
He was in their employ. Once they asked for access and/or recinded his 'permission' and he refused to cooperate he became a criminal. Let's not rationalize or glorify him just because he's a geek...shades of the apologists for Reiser come to mind now, though this crime isn't as bad as murder.
Re:Are you sure he's a criminal? (Score:4, Funny)
Well, now that you've invoked Reiser, it'll probably be true. It'll be a new rule: "If somebody mentions Reiser, the accused geek is probably guilty."
Re:Are you sure he's a criminal? (Score:5, Insightful)
We're getting the same sort of wagon-circling that we saw when Hans Reiser was charged. No one seems willing to admit that some of us "geeks" are self-important prima donas who border on pathologically criminal behavior. This guy is clearly a criminal. Of course, proper management would have recognized this behavior much earlier, and wouldn't have given him the keys to the kingdom, so it's a combination of a very bad guy and some very incompetent guys. There's no worse a combination.
It's guys like this that bring our IT occupations into ill-repute, by furthering their stereotype of Coke-swilling social retards on power trips. I hope they throw the book at him, and I hope that while he's sitting in prison he has time to ponder the fact that he isn't a god, but merely an employee.
Re: (Score:3, Insightful)
If the article is right, the guy was on perpetual on call duty. Quite frankly, some of the things that are expected of certain IT people (and basically nobody else except the occasional doctor or military personnel) go beyond the realm of "merely" being an employee (and those other vocations are pitched as lifestyles rather than careers, as well). For folks in those positions, if you don't go
Re:Are you sure he's a criminal? (Score:4, Insightful)
You seem willing to.
I'm waiting to hear the whole story.
Re:Are you sure he's a criminal? (Score:5, Insightful)
It's guys like this that bring our IT occupations into ill-repute, by furthering their stereotype of Coke-swilling social retards on power trips.
On the other hand, the more people like this there are, the more employment I get. I may not be as technically capable as folks like Child seems to be, but I am able to work with large groups of people AND the work gets done and documented. I can turn a pretty penny because of how "Customer Service Oriented" I am and how well I document my work.
Re:Are you sure he's a criminal? (Score:4, Insightful)
Yes, he may very well have bordered on "pathological criminal behaviour", but that is what is expected of employees. It is necessary to exhibit exactly that behaviour if you wish to successfully rise through the ranks.
I dislike that, and believe it is one incredibly unhealthy attitude - tied utterly to America's Puritan "Work Ethic". However, that is neither here nor there. The guy is a product of such attitudes. If he is a monster, then Herr Frankenstein bears the greater responsibility.
That does not make him blameless. It means that he is culpable only to some degree below 100%. Punish him for that percentage he is culpable. Fine. But it means there is also a non-zero component of responsibility elsewhere, which should not go unpunished, and that there is a non-zero element of illness the guy has developed as a result, for which he should be treated.
In any such system, blaming one person is extremely easy but utterly futile. It doesn't fix the underlying problems which made the failure possible, and the ultimate problem is invariably the mental illness prevalent in modern management methods.
Re:Are you sure he's a criminal? (Score:5, Interesting)
Now, if the CIO had called me up and asked me to assist with something, by your statements, I'd be a criminal to tell him to fuck himself. I somehow have some duty to a company that was firing me. I disagree, and I had no requirement to assist them in making anything work better, and if there was a password I had neglected to pass along, I have no legal requirement to share that with them. I've worked with the protective guys, and I hate it, but I've never seen any of them as criminal and think that's an unfair characterization. If he's a criminal, then it's a conspiracy and his boss should be in jail beside him. His boss knew what he was doing, allowed it, and even paid him to do it. If you pay someone to commit a criminal act, knowing it is a criminal act, you are complicit.
So yes, I can see how people can say it is "wrong" to do what he did. I agree. But the issue is the law. Murdering someone is a thing I think we can all agree is illegal. But not telling someone a work password when they demand it after you have already been fired? There is no law I know against that. We aren't circling tthe wagons because we think the guy is a saint. We are circling the wagons because we don't want a court ruling that could result in 10 years of jailtime for forgetting a password (and believe me, a cop demanding an answer from you takes "I don't remember" to be the same as "I know the answer and I won't tell you, fuck you pig").
Geek apoloist? Uh, no. (Score:3, Insightful)
With that said, government organizations tend
Re:Are you sure he's a criminal? (Score:5, Insightful)
He was in their employ. Once they asked for access and/or recinded his 'permission' and he refused to cooperate he became a criminal.
I'll be the first to admit that I don't know the entire story here, but since when is disagreeing with your boss a criminal offense?
What he did is inappropriate, but once they asked for access and/or rescinded his 'permission' and he refused to cooperate, he became a candidate for termination and perhaps civil liability. Whether or not he committed any criminal acts is up for debate. I think it's very dangerous to suppose that resisting your employer - even, no, especially if your employer is the government - is illegal.
Re:Are you sure he's a criminal? (Score:4, Interesting)
What? How is refusing to cooperate a criminal act?
Look, I've worked in government. In fact, I've met with the SF staff before (many moons ago). It sounds to me like he got caught up in a political battle. His saying no has probably been what has kept the network up, and this time was likely no different, except that the guy he said no to decided to make a martyr out of him.
Is what he did silly? Perhaps - but perhaps not. We can say that "They can just bring in Cisco", but I also used to work for MSFT - and I know that not everyone who comes in from the field is going to be able to transition vital systems without a hiccup.
There's a lot more to the story, and given what history I know of the SF departments from back in the day, I'm sure that this guy is guilty of nothing other than wanting to protect a critical network, and being a little misguided.
Re: (Score:3, Interesting)
I think everyone agrees that the guy is being a jerk. And from what I originally read, I thought he was probably a criminal too. However, this makes it sounds like he might not be.
Refusing to do what your boss says to do isn't a crime. Getting fired for it and then refusing to do what your former boss said to do isn't a crime. Since the situation before that was considered satisfactory by his former employers (because they were so naÃve to trust one guy with the keys to
Re:Are you sure he's a criminal? (Score:4, Interesting)
You're only hearing one side of the story, and that's not his. Consider this scenario. After Childs was fired by someone higher up in the poop-rolling-downhill order, said manager (Let's just refer to him 'Bonehead' for now) realized that the network policy he had written himself four years ago ("Leave all network devices at their default passwords, put a hard copy of all config files on the bulletin board in the lobby") was no longer valid. Bonehead then tried to break into the systems with a very clumsy brute force attack and got himself locked out. Rather than admit that he really had no clue of what he should be doing and should never have been allowed into the same room as anything more complicated than two cans and some string, Bonehead bravely announced that he was the victim of some kind of sabotage or terrorism and that the inconvenient former administrator should be arrested immediately.
When contacted about the problem, Childs replied "What? I already told you all the passwords you need. Here they are again. Read your email, you twit.", but of course those accounts had already been locked out thanks to Bonehead's bumbling attempt at cracking his own network. Things have now gone from bad to worse and poor, misunderstood Terry is now torn being asked to cough up passwords to accounts that he has already provided and is torn between just keeping his mouth shut to avoid being drawn further into this whole stupidity and quoting Ray Patterson by saying "You know, I'm not much on speeches, but it's so gratifying to leave you wallowing in the mess you've made. You're screwed, thank you, bye."
But what about those million-dollar-an-hour Cisco engineers who are desperately trying to fix the network but can't make any headway against Terry Childs' evil hackerish plots? It's hard to fix anything when Bonehead keeps getting in the way. "No no no... For security reasons nobody else can touch the network. Why don't you just tell me what commands you need me to run and then I will do them? No I already tried that! The admin password is 'password'. I already told you that. Stop wasting your time with this."
I'm not saying that this has to be what happened, but remember that you're only hearing one side of the story here. It's just as possible that Childs is just taking the blame for someone else's screwup and is just pissed off enough about being arrested over it that he's not cheerfully volunteering to help clean it all up again as it is that he really is the James Bond style evil mastermind who is trying to hold an entire city for ransom.
Re:He's still not justified... (Score:5, Informative)
>In San Francisco, where you think they'd have no
>problem finding competent replacements.
I guess then that you've never been to San Francisco? San Francisco can't balance their budget and had a hiring freeze since 2007 [sanfranciscosentinel.com] and laid off a lot of people, and only had a skeleton crew running things like IT departments. So things like a network freeze were just bound to happen sooner or later.
George W. Bush isn't the only political leader in the USA who can't balance a budget and is also incompetent and has an incompetent staff. Just look at many state and local governments in places like New York and California. They all want Federal hand-outs to help balance their budgets.
Re:He's still not justified... (Score:5, Interesting)
Why was this allowed to happen? In San Francisco, where you think they'd have no problem finding competent replacements.
This man was living in Pittsburg. They could not find *anyone* in SF to do the job.
I knew there was more to the story when we got the first article. The fact that he built the network, management allowed him to be the sole caretaker of the configuration *and* that the system is still running smoothly unattended makes it hard to accuse him of sabotage or "hijacking". The time to beg a system administrator to document his work is certainly not after you have him arrested.
Heads should be rolling in the city government.
Oh, I think I know this guy. (Score:4, Funny)
Simon Travaglia? [wikipedia.org] Is that you?
configs are not written to flash, eh? (Score:5, Interesting)
so the network is NOT locked up, it's just unrestoreble after "password recovery."
sounds like what they need to do is get some qualified engineers to redesign it, and when it's on paper, pull the plug on everything, and reconfigure from scratch.
because if it isn't saved in flash, it's going away as soon as the power light goes out.
which makes our jailed genius a little less than blazing fast. in fact, about half fast. parts of the system ARE going to go down. it's the nature of the beast. no records, no writes... the first time the janitor plugs in a 18-amp vacuum in a rack, it's gone.
they'll come along and take his Cisco cert away for not saving the configs, if for nothing else.
Re: (Score:3, Interesting)
Re: (Score:3, Interesting)
Yeah, but there's no evidence! What are they going to do, bring the router to the courtroom?
Bail (Score:5, Insightful)
Re:Bail (Score:4, Insightful)
I agree, however... high profile case, prosecutor (arguably much more powerful than a judge) wants to win with glory, so keep the suspect incarcerated to make him look guilty, makes an exaggerated case for flight risk, and pulls from his tool bag his only tool, his personal fly-swatter (which is actually an over-sized sledgehammer), and with absolutely zero finesse, smashed that fly with an absurd display of force. This is normal operating proceedure.
Re: (Score:3, Informative)
If you post the standard 10% for release, he could possibly come up with the $500,000. By mortgaging any property he owns, he just might be able to get that.
The bigger deal is that I guess they think he's a flight risk.
FiberWAN should not have been deployed then (Score:5, Insightful)
That's my first reaction to the news. Critical infrastructure should have redundancy everywhere, including the support staff.
To give a stupid but obvious example what if Childs was run over by a car? OK, he wouldn't care but all the rest of SF would.
So they should never have put the network online until the information was in several places (the brains of several people if formal electronic/paper records were too inflexible).
Stll, this sounds like political infighting more than ever. Given the situation why were they trying to fire a critical person like Childs? Sounds like some bureaucrat with an ego as big as Childs would be involved to cause this, rather than Childs "going rogue". And he (the bureaucrat) was more skilled in the political game. Of course this person would be covering his tracks, and not be obvious in any way. So Childs and the whole of SF lost. His firing does not make sense otherwise, given his critical position.
Ah, the fun of weaving conspiracy theories :-)
Re: (Score:3, Interesting)
More details here [informationweek.com]
Childs, who works in the city Department of Technology, allegedly created a password that gave him exclusive access to the city's new FiberWAN (wide area network), authorities told the newspaper. He has refused to divulge the password, leaving other system administrators locked out.
Undoing Childs' alleged tampering could cost millions of dollars, city officials said. In the meantime, the system is operating, even though administrators have limited or no access.
Childs, who has worked for th
Like This is Shocking (Score:5, Interesting)
Every software company I have worked for... if one or two people were hit by a bus... the company would be out-of-business. Management knew this... fellow developers knew it. Its a commonplace thing. Engineers take the work so *personally*. "No one can touch that code but me... " blah... blah. Ånd the stupid management goes along w/ these primadonna's. Of course... if they demanded more money... they'd be gone in a NY minute.
Re:Like This is Shocking (Score:4, Insightful)
Engineers take the work so *personally*. "No one can touch that code but me... " blah... blah.
I dunno. There's a fundamental difference between someone being naturally protective over their work and someone who volunteers to be on call 24/7 because he doesn't trust his coworkers with the passwords.
I've been in both positions. Shitty political situations where I hand over documentation and walk out the door with my head held up. And as the guy who comes in to inherit the mess when the "indispensable guru" quit.
Neither situation is really all that life-threatening. Nobody is really indispensable.
I don't believe for a second that the guy was irreplaceable except for the passwords that he intentionally withheld. The city could easily make a call and have an even bigger Cisco genius on site within a week. (After all the Bay Area is where Cisco is HQed.) A legend in his own mind.
Accidents happen, too. (Score:5, Interesting)
Every time I see a situation like this, I have to wonder what would happen if an "indispensable" person got hit by a bus. It strikes me that Childs was using his absolute control of the network as a way to put the fear of god in others within the department while attaining more prestige and autonomy than he deserved. The fact that Childs locked everyone out of the system after apparently receiving a poor job assessment backs that up. Sooner or later, the IT department had to take action to strip his stranglehold of the network, especially if he was on the verge of burnout or increasingly difficult to deal with.
I suspect that no one had the interpersonal wherewithal to figure out how to approach him in a non-confrontational manner. The best approach would have been to find someone who Childs respected who could share the load and provide backup and support while the organization attempted to deal with an overly possessive employee who is behaving irrationally.
Hit by a bus (Score:5, Funny)
Maybe they could suggest "crushed in an orgy" or "broke lightspeed and turned to photons". Getting hit by a bus is such a boring way to go.
Re: (Score:3, Funny)
Every time someone describes "hit by a bus" scenario, I think that we really should get rid of buses altogether.
Complete bunk... (Score:5, Interesting)
I know someone who worked on the cisco side with this guy. This had been going on for a while. The dude was threatening co-workers doing all kinds of odd stuff. The idea that he was somehow just a little protective is an off the charts miss-representation.
citations and evidence please ... (Score:4, Insightful)
"I know someone who worked on the cisco side with this guy .. The dude was threatening co-workers"
What was the name of this someone, who did Terry Childs threaten, what was the nature of these threats?
A tale of 3 losers (Score:3, Funny)
What was Childs' job? (Score:3, Insightful)
Let's leave out the legal ramifications here, and let's not go to the hysteria of "he's being thrown to the wolves to protect management" or "he's an evil hacker who shut down the city government networks."
When it comes down to it, one has to ask what Childs' job was. He was supposed to manage the network for the San Francisco city government.
As a result, he was supposed to implement policy as communicated to him by his bosses... but he also had the latitude to take actions to support the spirit of those policies where the right action was unclear. And yes, this is a Pollyanna-esque (is that a word?) view of the situation, but it leaves out the concept of malice as the driving force for either side - because it didn't start out as a plan to shut down the city.
Somehow it morphed into him becoming the sole support for the network routers, be it through arrogance ("I can't believe anyone else would do this right!") or being the only one available ("There's nobody else who works here who even understands the need!"), and at that point this became an incident waiting to happen.
So, either he refused to do his job (at which point he would have deserved to be fired), or his job was such that he was prevented from doing it (at which point professional ethics would have suggested his resignation - or at least, that's what engineering associations would have recommended in similar scenarios).
Instead, he stayed on and we have the current state of affairs.
Simple test (Score:3, Informative)
Power cycle the network equipment. If it comes back up, pay him for the rest of the year as severance and let him go his own way. If it doesn't come back up, put him away for 10-15 years for public endangerment, and fine him whatever the cost is to the city to recreate the network and for any loss of productivity in the meantime. Either way he is a terrible admin - no one single person should be a single point of failure. What if he got hit by Muni at lunch one day?
don't wag the dog here. (Score:4, Insightful)
all networks once configured properly, run smoothly until they don't.
when they don't, there's one man who can fix it.
I can fully understand setting up a complex system and getting it working perfectly and then some other admin or consultant coming in and fucking it up.
when they fuck it up, you have to fix it. And you don't get bonus pay for that.
not only that, but network/system administrators have to worry a lot about whether management wants to can them simply because things are running so smoothly that they have nothing to do. Which is bullshit because half of the job is keeping up with current tech trends, learning new technologies, and protecting your network on a daily basis. I don't blame the man for guarding his creation jealously. When you start handing over the keys, you are no longer necessary. You get paid too much and this kid who just quit his job of six months from bimblebomble.com seems to know how to do what you do. And we can pay him a lot less and potentially cut out benefits.
The City has a big liability issue (Score:4, Insightful)
The city has a huge issue here.
This guy will have a hard time getting a job in the future, and a guy with his credentials commands a lot more than he is making right now.
If it turns out that the facts of this case are far from the original story, and nobody from the city is stepping in to correct it, then SF is in the same situation as the US when Ashcroft pointed the finger at the Anthrax guy (who recently won a big chunk of change for the false accusation).
Something tells me that the wheels of government turn slowly enough that even if they wanted to correct themselves at this point, they won't until well after the publicity is over.
Echos from my own past (Score:5, Interesting)
Childs supposedly had a crimminal background (Score:3, Interesting)
http://www.csoonline.com.au/index.php/id;1895501252;fp;2;fpid;1 [csoonline.com.au]
Yet the city gave hime full admin access to a critical, and sensitive system. The city also didn't bother to insure that the system was safe from being locked out in that manner.
IMO: if Childs goes to prison, the city's IT managers should go with him.
Re: (Score:3, Interesting)
You're wrong. Your comparison with Diebold does not even merit cursory contemplation.
My Point was. (Score:5, Insightful)
Re: (Score:3, Insightful)
This makes no sense. A properly secure network should be in complete control of those creating it, simply through password and other authentication. Sure, good documentation is helpful in a worst case scenario, but you really need a hit-by-a-bus contingency team.
Re:Open Source (Score:5, Insightful)
Open source does not equal open data.
short version (Score:5, Funny)
Re:short version (Score:5, Funny)
wait (Score:4, Funny)
i'm not sure if i am being trolled
are you lampooning how a paranoid schizophrenic thinks or are you actually also a paranoid schizophrenic?
Re: (Score:3, Insightful)
Re: (Score:3, Informative)
He wouldn't write configs to flash?
It means they can't power cycle or reboot anything, or the network is screwed.
No device stays up forever.
It also means they just have to power cycle a switch to gain access to it, and then do what they can to figure out how it was configured.
IOW: They have to break it to fix it.
Re:Running smoothly because no one can touch it (Score:4, Funny)
Let's hope his mailbox doesn't flow over and crash the servers with congrat mails for sysadmin appreciation day next week.