Slashdot Log In
The Inside Story On the San Francisco Network Hijacking
Posted by
Soulskill
on Fri Jul 18, 2008 08:55 PM
from the connection-reset-by-lack-of-peers dept.
from the connection-reset-by-lack-of-peers dept.
snydeq writes "A source with direct knowledge of San Francisco's IT infrastructure has tipped off Paul Venezia to the real story behind Terry Childs' lockout of San Francisco's network, providing a detailed account of the city's FiberWAN, interdepartmental politics, and Terry Childs himself. Childs pleaded not guilty to charges of tampering yesterday and is being held on $5 million bail. According to the source, Childs' purview was limited to the city's FiberWAN — a network he himself built and, believing no one competent enough to touch the network but himself, guarded religiously, sharing details with no one, including routing configuration and log-in information. Childs was so concerned about the network's security that he refused even to write router and switch configurations to flash. But what may prove difficult for the prosecution in its case against Childs is that his restricted access to the network was widely known and accepted among managers and the city's other network engineers. Venezia, who has been suspicious of the official story from the start, suspects that the Childs case may be that 'of an overprotective admin who believed he was protecting the network — and by extension, the city — from other administrators whom he considered inferior, and perhaps even dangerous.' Further evidence is that fact that the network, from what Venezia understands, has been running smoothly since Childs' arrest."
Related Stories
[+]
Disgruntled Engineer Hijacks San Francisco's Computer System 1082 comments
ceswiedler writes "A disgruntled software engineer has hijacked San Francisco's new multimillion-dollar municipal computer system. When the Department of Technology tried to fire him, he disabled all administrative passwords other than his own. He was taken into custody but has so far refused to provide the password, and the department has yet to regain admin access on their own. They're worried that he or an associate might be able to destroy hundreds of thousands of sensitive documents, including emails, payroll information, and law enforcement documents."
[+]
IT: SF Admin Gives Up Keys To Hijacked City Network 581 comments
snydeq writes "Jailed IT admin Terry Childs relinquished his hold over San Francisco's multimillion-dollar FiberWAN, handing his administrative passwords over to San Francisco Mayor Gavin Newsom, who was 'the only person he felt he could trust.' Childs is still being held on $5 million bail for his lockout of the city's FiberWAN, a case that has been called into question since an insider came forward with details about both the network and Childs himself. The case hinges on No Service Password Recovery commands Childs allegedly configured onto several Cisco devices, as well as dial-up and DSL modems the SFPD has discovered that would allow unauthorized connections to the FiberWAN. Childs intends to 'expose the utter mismanagement, negligence, and corruption at DTIS, which if left unchecked, will in fact place the City of San Francisco in danger,' according to his motion. The Department of Telecom and IS has cut 200 of its 350 IT positions since 2000 — pressure that may have contributed to Childs' actions, according to interviews with current and former DTIS staffers. Newsom secured the passwords without first telling the DTIS that he was meeting with Childs."
[+]
Entertainment: San Francisco DA Discloses City's Passwords 333 comments
snydeq writes "The office of San Francisco District Attorney Kamala Harris has made public close to 150 usernames and passwords used by various departments to connect to the city's VPN. The passwords were filed this week as Exhibit A in a court document arguing against a reduction in $5 million bail in the case against Terry Childs. Though they placed the passwords in the public record, city prosecutors do seem to think that they are sensitive. InfoWorld's Paul Venezia, who has been following the case closely, provides further analysis of the technical details in the city's case. 'By themselves, [the passwords] would not be enough to allow anyone to access the network via VPN,' Venezia writes, 'but the fact that the city entered them into evidence is quite shocking. At the very least, they'll have to shut down their VPN access for awhile until they've changed them all and modified the configurations of some large number of VPN clients.'"
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Is this really the case? (Score:4, Insightful)
Re:Is this really the case? (Score:5, Insightful)
I find that easy to believe. Even easier to believe that they didn't know this was the case, or knew but did not understand.
Parent
Re:Is this really the case? (Score:4, Insightful)
This doesn't sound reasonable. If management behaved like this they would have been fired before this guy was -- the management problems would be greater than the technical ones.
Parent
Re:Is this really the case? (Score:5, Insightful)
Never worked for the government, have you? ;)
Management is where people who are too incompetent for technical work go. No one gets fired, they get moved to different departments. As a last resort, they get assigned to 'special projects' for about a year in the hopes that everyone will forget what an imbecile they are, and will be safe to move back into the management structure.
Parent
Re:Is this really the case? (Score:5, Insightful)
Parent
Exit stage left (Score:5, Insightful)
.
modded +3, Informative.
but this attitude sets off alarms.
exposing a geek who despises his supervisors and is used to thinking of the server rooms as his personal playground.
Parent
Re:Is this really the case? (Score:5, Insightful)
It's nice to believe that but, to abuse an oft-quoted phrase, quis sacko ipsos pointyhaires?
Before you can fire someone for being a complete idiot, you have to not be totally out to lunch yourself. More importantly you have to possess evidence to back up your decision which is at least strong enough to outweigh the political costs of making it.
If you think this all sounds like a load of crap, then consider yourself lucky that you have never been in the middle of it.
Parent
Re:Is this really the case? (Score:5, Interesting)
I post AC because of my position, which is basically a guy who was hired as the second network tech to help manage the network for a sizeable city (large enough that we host several professional sports teams). I had no real qualifications other than knowing how to google my way out of most basic computer situations. My supervisor managed all City-owned Cisco equipment and it has only been 2 of us for 2 years. We manage over 300 Cisco devices at over 100 sites and I can honestly say that after reading a few more details on this story, I can easily understand how this can happen in a local government. I believe that the problem is in management. We have similar problems in our City regarding the lack of passing of knowledge and lack of staffing, but we have a good security team that knows more about Cisco networks than the 2 of us that regularly work on the Cisco equipment in our City. They are not normally watching our backs (that we know of) but they would certainly do so if they got a bad vibe about us. We have to share passwords with them and they have as much access to our equipment as we do. It is simply a requirement in a publicly owned system that knowledge is shared. Taxpayers have payed for the equipment and expect that there are not single points of failure. There are many reasons that more people than work on one thing on a regular basis have knowledge of and access to the most basic systems. If there was no redundancy, then it is a fundamental failure of management.....I'm not saying the guy should have set one password and not passed it on.....but I understand.
Parent
Re:Is this really the case? (Score:5, Interesting)
Parent
Re:Is this really the case? (Score:5, Insightful)
It seems pretty idiotic to me. I still think they should throw this guy in the clink, but at the same time, I think some of his superiors should be told to collect their belongings and then have security escort them through the front door, because there was a colossal breakdown of management here if a single guy was permitted to basically hold the entire network's architecture in his head.
Parent
Re:Is this really the case? (Score:5, Insightful)
Parent
Au contraire--it's the rule, not the exception... (Score:5, Insightful)
In my experience, it's a rare company indeed whose managers can fathom the implications of a situation like this. In general, I'm unable to get management to even understand Rule Zero of system administration. Which is: Do everything you need to do to be drop dead certain that you always have a reasonable backup of your important systems. This doesn't sound too difficult, but in practice it's difficult to convince managers that an event that could happen with probability == 0.01 could ever happen...
Parent
He's still not justified... (Score:5, Interesting)
You can try and defend him and glorify him all you want... but as a professional system administrator he should have known that his singular access and pathological behavior was more dangerous than helpful.
What if, instead of being fired he was the victim of an accident or crime? What if he had a health problem? What if a serious, life threatening issue came up (say, you know, an earthquake) that caused the system to be unstable and, at the same time, prevented him from getting there to fix things?
He's still a criminal. But, he's not alone in his behaviour. Whoever his managers are sound to be guilty of criminal negligence. This never should have been possible in a city government the size of San Francisco. Especially when it comes to critical infrastructure. If I were a citizen of San Fran I'd be asking why heads aren't rolling at the highest levels. Why was this allowed to happen? In San Francisco, where you think they'd have no problem finding competent replacements.
Absolutely mind boggling.
Re:He's still not justified... (Score:5, Interesting)
If this was a case of "He was the only one with the passwords and knowledge, we stupidly fired him without getting that info, and now we realized we're screwed" then he isn't a criminal. His boss maybe, but not him.
Hell, even if the situation was "tell us the info so we can replace you - no - you're fired", he still isn't a criminal. Other than maybe stretching a denial of service crime to fit, other than he hasn't really denied them a service if it is still running.
Parent
Re:He's still not justified... (Score:5, Insightful)
We still don't know all the details. Perhaps all the accusations are trumped. But, if when his performance became a question he started hiding backups, monitoring his managers' email exchanges and is now not cooperating, he's definitely a criminal.
How can you possibly argue otherwise? Sure, he's the network admin, but does that authorize him to read people's email without authorization?
Sure, he's the admin, but does that give him the right to create a situation that basically takes the city's IT infrastructure hostage?
I'm not questioning that his superiors should share the larger part of the blame here. But I can't see how he's not at all at fault.
Parent
Re:He's still not justified... (Score:5, Insightful)
>
How can you possibly argue otherwise? Sure, he's the network admin, but does that authorize him to read people's email without authorization?
Not at all. But then charge him with that, not some pseudo-terrorist computer tampering charge.
Parent
Re:He's still not justified... (Score:5, Insightful)
>>How can you possibly argue otherwise? Sure, he's the network admin, but does that authorize him to read people's email without authorization?
>Not at all. But then charge him with that, not some pseudo-terrorist computer tampering charge.
The Electronic Communication Privacy Act of 1986 protects administrators if "in the performance of their duty" they read email. Please note the date. If you are unfamiliar with it, you should be even if you're "just a user", no excuses.
He's an administrator. He's shielded.
Y'all should know that by now.
You should also know that if you store your email on company servers/isp servers, they get /less/ protected as time goes on, with most protection going to those "in flight" and least to those being stored for over a year.
If you have anything confidential, encrypt it and remove it from your provider's machines and store elsewhere. If you don't ever want the admin to see the email in flight, then end-to-end encryption. These days it's easier than the mid 1980's.
OB On Topic: I can see where he's coming from. A network administrator, if he's doing his job, gains a bit of paranoia. Sometimes that can become unhealthy, and it appears that he's crossed the line into "unhealthy". Criminal? I don't think so. It appears that he's been severely mismanaged by those who never understood "Mack Truck Syndrome". One guy for an entire city? I'm not sure who's crazier, the management or him.
--
BMO
Parent
Re:He's still not justified... (Score:5, Informative)
"There is a big difference between "in the performance of their duty" and "because are able to do so, they felt like doing so and so they went ahead and did so.""
The thing is you have to prove it that an admin did it for BOFH style "shits and giggles" or some other motivation other than official use - beyond a reasonable doubt. That's a pretty big hurdle for a prosecution. Some would call it an impossible hurdle.
That's for when the email is in-flight. Once it hits storage, an admin basically has free reign. As email gets older and older, it gets less protected. Beyond 180 days it's unprotected - the gubmint can even do a search without a warrant.
Email isn't as protected as paper documents, as the last time this came up before the 6'th circuit, it was refused review on procedural grounds.
Don't ask me, go read the law yourself. ECPA of 1986.
If you think that the legal privacy of email is pretty weak because of the ECPA, this was an *improvement* on privacy back in 1986 because prior to that, email was basically equivalent to shouting out the window (and sometimes still is). Once the ECPA passed, BBS operators like myself became paranoid so we decided to put up disclaimers announcing that users should not expect privacy. Such disclaimers during login and registration notified the users and thus shielded the admin from privacy lawsuits and such. Some people think that this gets rid of plausible deniability, because once you say your users have no privacy, the guys in the FBI PartyVan parked in your driveway might suspect that you know what your users are doing, or so the theory goes. But a section of the CDA of 1996 supposedly shields the admins from the actions of a service's users. It gets really complicated if you research even a little bit of this stuff.
--
BMO
Parent
Re:He's still not justified... (Score:5, Informative)
"The burden of proof is on you to back up your bullshit, and I'm a calling you on it. Quote some laws here, if you can."
I'll do you one better:
I'll point you at a book on the matter:
http://www.amazon.com/Netlaw-Your-Rights-Online-World/dp/0078820774 [amazon.com]
And I'll quote from here:
http://www.rbs2.com/email.htm [rbs2.com]
The executive summary of what I've been talking about and what you've been talking out your ass about:
"Reading e-mail that is stored on a computer is not an "interception" under 18 U.S.C. 2510, et seq., because an interception must be contemporaneous with the transmission of the message between different locations. Steve Jackson Games v. U.S. Secret Service, 816 F.Supp. 432, 442 (W.D.Tex. 1993), aff'd, 36 F.3d 457, 460 (5thCir. 1994). This holding has been accepted in several subsequent cases, including Wesley College v. Pitts, 974 F.Supp. 375, 384-390 (D.Del. 1997); U.S. v. Moriarty, 962 F.Supp. 217, 221 (D.Mass. 1997); Bohach v. City of Reno, 932 F.Supp. 1232, 1235-36 (D.Nev. 1996)."
--
BMO - Not a lawyer, but dammit I can read for myself.
Parent
Re:He's still not justified... (Score:4, Insightful)
If he really did explicitly "hold the network hostage", actually said "I'll trash it if I don't get what I want", then he commited a crime. But what it sounds like so far is "Do your job the way we want, not your way" and he said no and was fired for it, which is generally not a criminal act.
I've known half a dozen people who "knew things" that would ruin their company if they were hit by a bus. None of them would get charged with a crime if they refused to give up that information *after* being fired(although their company might get sued by the shareholders). But none of them are in IT.
As for the email, from the correspondance provided, it doesn't say if he had access to the city's mail servers, but then he isn't being charged with breaking in to them either. Seeing as he ran the network, it'd probably be easy to sniff and read the email "on the wire" without breaking into a computer, since I doubt anyone in the city government used encryption.
Ok, now I'm being a bit nitpicky, sorry:), but how often do we compare email to sending postcards? Other than cellular communications, where else is it illegal to detect something broadcast in the clear?
Parent
Re:He's still not justified... (Score:5, Interesting)
Yah, I agree it he probably is a huge jerk and should've given up any passwords or other info when he was canned, just out of professionalism(and maybe a little "here's the knife, cut your own wrists"). But I think the management is probably blowing this out of proportion to cover their own asses.
A company I shared a parking lot with during the dot bomb laid off their entire programming department a few months after they hit release and hired an outside company to "sanitise" the computers in the building. After the contractors wiped the CVS server, management threatened to sue/charge several of the programmers for "mislabeling" the CVS server deliberately so that would happen(it was labeled "Walgreens", bad pun).
That fell flat eventually, the guy who proposed the 100% layoff got the axe for it, and I heard the story from a couple of the programmers that were contracted back to get things back up to snuff(ie, they "failed" to destroy "illegal" backups and were able to save the company's bacon).
Parent
Are you sure he's a criminal? (Score:5, Interesting)
In America, they have to prove that first. Looking at the statute, it seems it all comes down to the issue of "without permission." The main point the article makes is that he might have had at least understood or standing permission to do most or all of what he did. Just like when you take your parents' car somewhere as a teenager, it isn't theft if it's understood that you are allowed to use it.
The article is one-sided, and his alleged refusal to give up the passwords looks bad (perhaps he is remaining silent until he speaks with counsel), but proving he didn't have permission might be hard. Ergo, no criminal.
Parent
Re:Are you sure he's a criminal? (Score:5, Insightful)
He was in their employ. Once they asked for access and/or recinded his 'permission' and he refused to cooperate he became a criminal. Let's not rationalize or glorify him just because he's a geek...shades of the apologists for Reiser come to mind now, though this crime isn't as bad as murder.
Parent
Re:Are you sure he's a criminal? (Score:4, Funny)
Well, now that you've invoked Reiser, it'll probably be true. It'll be a new rule: "If somebody mentions Reiser, the accused geek is probably guilty."
Parent
Re:Are you sure he's a criminal? (Score:5, Insightful)
We're getting the same sort of wagon-circling that we saw when Hans Reiser was charged. No one seems willing to admit that some of us "geeks" are self-important prima donas who border on pathologically criminal behavior. This guy is clearly a criminal. Of course, proper management would have recognized this behavior much earlier, and wouldn't have given him the keys to the kingdom, so it's a combination of a very bad guy and some very incompetent guys. There's no worse a combination.
It's guys like this that bring our IT occupations into ill-repute, by furthering their stereotype of Coke-swilling social retards on power trips. I hope they throw the book at him, and I hope that while he's sitting in prison he has time to ponder the fact that he isn't a god, but merely an employee.
Parent
Re:Are you sure he's a criminal? (Score:5, Insightful)
It's guys like this that bring our IT occupations into ill-repute, by furthering their stereotype of Coke-swilling social retards on power trips.
On the other hand, the more people like this there are, the more employment I get. I may not be as technically capable as folks like Child seems to be, but I am able to work with large groups of people AND the work gets done and documented. I can turn a pretty penny because of how "Customer Service Oriented" I am and how well I document my work.
Parent
Re:Are you sure he's a criminal? (Score:5, Interesting)
Now, if the CIO had called me up and asked me to assist with something, by your statements, I'd be a criminal to tell him to fuck himself. I somehow have some duty to a company that was firing me. I disagree, and I had no requirement to assist them in making anything work better, and if there was a password I had neglected to pass along, I have no legal requirement to share that with them. I've worked with the protective guys, and I hate it, but I've never seen any of them as criminal and think that's an unfair characterization. If he's a criminal, then it's a conspiracy and his boss should be in jail beside him. His boss knew what he was doing, allowed it, and even paid him to do it. If you pay someone to commit a criminal act, knowing it is a criminal act, you are complicit.
So yes, I can see how people can say it is "wrong" to do what he did. I agree. But the issue is the law. Murdering someone is a thing I think we can all agree is illegal. But not telling someone a work password when they demand it after you have already been fired? There is no law I know against that. We aren't circling tthe wagons because we think the guy is a saint. We are circling the wagons because we don't want a court ruling that could result in 10 years of jailtime for forgetting a password (and believe me, a cop demanding an answer from you takes "I don't remember" to be the same as "I know the answer and I won't tell you, fuck you pig").
Parent
Re:Are you sure he's a criminal? (Score:5, Insightful)
He was in their employ. Once they asked for access and/or recinded his 'permission' and he refused to cooperate he became a criminal.
I'll be the first to admit that I don't know the entire story here, but since when is disagreeing with your boss a criminal offense?
What he did is inappropriate, but once they asked for access and/or rescinded his 'permission' and he refused to cooperate, he became a candidate for termination and perhaps civil liability. Whether or not he committed any criminal acts is up for debate. I think it's very dangerous to suppose that resisting your employer - even, no, especially if your employer is the government - is illegal.
Parent
Re:He's still not justified... (Score:5, Informative)
>In San Francisco, where you think they'd have no
>problem finding competent replacements.
I guess then that you've never been to San Francisco? San Francisco can't balance their budget and had a hiring freeze since 2007 [sanfranciscosentinel.com] and laid off a lot of people, and only had a skeleton crew running things like IT departments. So things like a network freeze were just bound to happen sooner or later.
George W. Bush isn't the only political leader in the USA who can't balance a budget and is also incompetent and has an incompetent staff. Just look at many state and local governments in places like New York and California. They all want Federal hand-outs to help balance their budgets.
Parent
Re:He's still not justified... (Score:5, Interesting)
Why was this allowed to happen? In San Francisco, where you think they'd have no problem finding competent replacements.
This man was living in Pittsburg. They could not find *anyone* in SF to do the job.
I knew there was more to the story when we got the first article. The fact that he built the network, management allowed him to be the sole caretaker of the configuration *and* that the system is still running smoothly unattended makes it hard to accuse him of sabotage or "hijacking". The time to beg a system administrator to document his work is certainly not after you have him arrested.
Heads should be rolling in the city government.
Parent
Oh, I think I know this guy. (Score:4, Funny)
Simon Travaglia? [wikipedia.org] Is that you?
configs are not written to flash, eh? (Score:5, Interesting)
so the network is NOT locked up, it's just unrestoreble after "password recovery."
sounds like what they need to do is get some qualified engineers to redesign it, and when it's on paper, pull the plug on everything, and reconfigure from scratch.
because if it isn't saved in flash, it's going away as soon as the power light goes out.
which makes our jailed genius a little less than blazing fast. in fact, about half fast. parts of the system ARE going to go down. it's the nature of the beast. no records, no writes... the first time the janitor plugs in a 18-amp vacuum in a rack, it's gone.
they'll come along and take his Cisco cert away for not saving the configs, if for nothing else.
Bail (Score:5, Insightful)
Re:Bail (Score:4, Insightful)
I agree, however... high profile case, prosecutor (arguably much more powerful than a judge) wants to win with glory, so keep the suspect incarcerated to make him look guilty, makes an exaggerated case for flight risk, and pulls from his tool bag his only tool, his personal fly-swatter (which is actually an over-sized sledgehammer), and with absolutely zero finesse, smashed that fly with an absurd display of force. This is normal operating proceedure.
Parent
FiberWAN should not have been deployed then (Score:5, Insightful)
That's my first reaction to the news. Critical infrastructure should have redundancy everywhere, including the support staff.
To give a stupid but obvious example what if Childs was run over by a car? OK, he wouldn't care but all the rest of SF would.
So they should never have put the network online until the information was in several places (the brains of several people if formal electronic/paper records were too inflexible).
Stll, this sounds like political infighting more than ever. Given the situation why were they trying to fire a critical person like Childs? Sounds like some bureaucrat with an ego as big as Childs would be involved to cause this, rather than Childs "going rogue". And he (the bureaucrat) was more skilled in the political game. Of course this person would be covering his tracks, and not be obvious in any way. So Childs and the whole of SF lost. His firing does not make sense otherwise, given his critical position.
Ah, the fun of weaving conspiracy theories :-)
Like This is Shocking (Score:5, Interesting)
Every software company I have worked for... if one or two people were hit by a bus... the company would be out-of-business. Management knew this... fellow developers knew it. Its a commonplace thing. Engineers take the work so *personally*. "No one can touch that code but me... " blah... blah. Ånd the stupid management goes along w/ these primadonna's. Of course... if they demanded more money... they'd be gone in a NY minute.
Accidents happen, too. (Score:5, Interesting)
Every time I see a situation like this, I have to wonder what would happen if an "indispensable" person got hit by a bus. It strikes me that Childs was using his absolute control of the network as a way to put the fear of god in others within the department while attaining more prestige and autonomy than he deserved. The fact that Childs locked everyone out of the system after apparently receiving a poor job assessment backs that up. Sooner or later, the IT department had to take action to strip his stranglehold of the network, especially if he was on the verge of burnout or increasingly difficult to deal with.
I suspect that no one had the interpersonal wherewithal to figure out how to approach him in a non-confrontational manner. The best approach would have been to find someone who Childs respected who could share the load and provide backup and support while the organization attempted to deal with an overly possessive employee who is behaving irrationally.
Hit by a bus (Score:5, Funny)
Maybe they could suggest "crushed in an orgy" or "broke lightspeed and turned to photons". Getting hit by a bus is such a boring way to go.
Parent
Complete bunk... (Score:5, Interesting)
I know someone who worked on the cisco side with this guy. This had been going on for a while. The dude was threatening co-workers doing all kinds of odd stuff. The idea that he was somehow just a little protective is an off the charts miss-representation.
Echos from my own past (Score:5, Interesting)
Re:and in stargate news..... (Score:5, Insightful)
Parent
Mods on crack (Score:5, Insightful)
This analogy is spot on, and whoever modded it off-topic obviously is incapable of understanding the topic and shouldn't have had the keys to the mod-car in the first place.
Parent
Re:Mods on crack (Score:5, Funny)
It's my considered opinion that many people with keys to the mod-car shouldn't have been allowed off the short bus.
Parent
Re:Mods on crack (Score:5, Insightful)
Except that Childs is not the cars owner, he's the mechanic hired to fix and maintain it for the owners (the city government of SF).
Parent
Re:and in stargate news..... (Score:5, Insightful)
Consider mentoring. The God complex management style rarely works out well in the end.
Parent
short version (Score:5, Funny)
Parent
Re:short version (Score:5, Funny)
Parent
My Point was. (Score:5, Insightful)
Parent
Re:Open Source (Score:5, Insightful)
Open source does not equal open data.
Parent
wait (Score:4, Funny)
i'm not sure if i am being trolled
are you lampooning how a paranoid schizophrenic thinks or are you actually also a paranoid schizophrenic?
Parent