Post Mortem of GunnAllen IT Meltdown 192
CowboyRobot writes "The story begins when GunnAllen, a financial company, outsourced all of its IT to The Revere Group. Before long, it was discovered that 'A senior network engineer had disabled the company's WatchGuard firewalls and routed all of the broker-dealer's IP traffic--including trades and VoIP calls--through his home cable modem.' In addition to the obvious security concerns of sending information such as bank routing information and driver's license numbers, the act violated SEC rules because the routed information was not being logged. Regardless of whether the cause was negligence, incompetence, or sabotage, the matter was swept under the rug for a time until unpaid SQL Server licenses meant threatening calls from Microsoft as well. The rest of the story is one of greed, mismanagement, and neglect, and ends with the SEC's first-ever fine for failure to protect customer data."
Outsourced (Score:5, Interesting)
Yeah keep outsourcing the responsibility of something so crucial that IT people hold the keys to the kingdom.
This is nothing new in the world of IT. Save a dime to lose a million dollars.
I am in a comany right now where they hired IT consultants for well over 3 years and come to find out so called "Experts" where just patching the system but never really fixing the real issues. It's amazing to see what these contractors were selling to a company who had the money to buy great gear only to discover pure incompetence at implementing it. I am no expert by any means but I can smeel bullshit when I see a network in need of a lot of TLC.
Sigh... (Score:5, Interesting)
A financial company outsourcing its IT ought to be considered criminal negligence.
(Though an own employee could do the same thing, in this case.)
Re:HAHA (Score:5, Interesting)
It's not mentioned in the summary, but the first sign of the rerouting was, as you'd expect, their network slowing to a crawl. That earned the IT guy responsible for it a reprimand. A reprimand, for routing an entire company's trading data through his home modem for a week!
There's other gold in there too, like the time the guy pulled the cable on a production rack in order to create a catastrophe so he wouldn't have to travel to a business meeting, or his habit of remoting into IT infrastructure (Blackberry and Exchange servers were mentioned) on the weekends to fuck up their configuration, just so he could "magically" fix it on Monday morning.
He was, apparently, eventually fired.
Re:Wait a minute... (Score:4, Interesting)
Negligence, Incompetence, or Sabotage? (Score:5, Interesting)
Re:Outsourced (Score:3, Interesting)
With the revolving door nature of CEO and other top level jobs these days, you could argue that upper management is already outsourced away from the actual company. Just that they compete on paying the most instead of the least.
Re:Wait a minute... (Score:4, Interesting)
And it didn't help that a Republican-controlled Congress cut their funding to the point where the DoJ was damned near useless.
Even with funding, the DoJ would be pretty useless. I'll just trot out the current Republican talking points about Fast and Furious since they'll illustrate a good reason why the Republicans wouldn't be inclined to fund the Department of Justice.
Here, you have a pretty much cut and dry case. ATF agents allowed roughly two thousand fairly high quality guns to pass to Mexican drug cartels with no attempt made to track those weapons. Since those weapons have turned up at many crime scenes, including the murder of a US border agent (which is what finally shut down Fast and Furious). Further, the ATF agents involved knew for a few months before that final murder that these weapons were turning up at crime scenes, including murders. So a prosecutor has a pretty good case that someone committed a bunch of acts of accessory to murder (with reckless disregard for human life) and other crimes, plus the murder of a federal law enforcement officer. So what is the Department of Justice doing with this case? Hiding the agents involved in Washington DC. When will they investigate this?
This is why the "more funding" argument doesn't work. If the Department of Justice isn't going to do its job, then it doesn't really matter how much they're paid so might as well make it a little rather than a lot. The SEC is particularly notorious for providing the illusion of security for novice investors, or in other words, helping keep the marks from getting scared off before they can be fleeced.
Re:Hard time reading train wreck stories (Score:5, Interesting)
There seems to be a lot of that attitude with the cloud outsourcing. I put an example up here earlier of 25k email accounts inaccessible for a week due to a DNS typo and a long job queue to do the two second fix, but people seemed to think it was OK to have that so long as there was someone else to blame. In that case it was Microsoft doing the hosting so good luck in getting anywhere with blaming them, a customer with twenty-five thousand email accounts is ignorable small fry and legal action is pointless.
Re:jeez, exchange is still used? (Score:5, Interesting)
Yeah yeah we know it does work, mostly, and is probably written in VBscript or cobol.
But damn, you can afford a EX licence, but cannot afford a high end intel 512G SSD x 2.
Restore in 5mins.
Hardrives, puhhhh.... so 90s, like C64 tapes. Get with the future dude.
Sure. So you restore in minutes but that's when you realize that the information store is - by definition - backed up dirty because it's in use. A moment later you discover that Exchange insists on you running some nice ISINTEG routines to mark the database as clean before it can be mounted. Those routines joyfully take a minor eternity, even on SSD if you have a huge database. Like... 450G. When you're done with ISINTEG, if you're really lucky you can have a bonus round of ESEUTIL followed by ISINTEG again if it turns out there was any minor database structural issues you didn't know about.
High I/O absolutely helps, but don't write this off as if massive database restores are trivial just because someone follows your advice. For businesses that are big enough to accrue huge amounts of data but not big enough to afford redundant servers, TIME is the cost they pay.