How To Lose $172,222 a Second For 45 Minutes 327
An anonymous reader writes "Investment firm Knight Capital made headlines in 2012 for losing over $400 million on the New York Stock Exchange because of problems with their algorithmic trading software. Now, the owner of a Python programming blog noticed the release of a detailed SEC report into exactly what went wrong (PDF). It shows how a botched update rollout combined with useless or nonexistent process guidelines cost the company over $172,000 a second for over 45 minutes. From the report: 'When Knight used the Power Peg code previously, as child orders were executed, a cumulative quantity function counted the number of shares of the parent order that had been executed. This feature instructed the code to stop routing child orders after the parent order had been filled completely. In 2003, Knight ceased using the Power Peg functionality. In 2005, Knight moved the tracking of cumulative shares function in the Power Peg code to an earlier point in the SMARS code sequence. Knight did not retest the Power Peg code after moving the cumulative quantity function to determine whether Power Peg would still function correctly if called. ... During the deployment of the new code, however, one of Knight's technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Knight had no written procedures that required such a review.'"
Garden Variety Upgrade SNAFU (Score:4, Insightful)
I'm sure someone will chime in and claim to be the em-effing Change MASTER! but this seems like an ordinary error, the sort I've seen a hundred times before, where one server is a tiny bit wonky, and during the change, something doesn't happen as expected. Normally it's an "inexpensive" error, where some of your VPN users get randomly disconnected... ...and sometimes it's the sort of error where you lose half a billion dollars an hour by HFT trading... ...badly.
There should be a mandatory one second delay. (Score:5, Insightful)
This level of trading does not do the market any good, and puts individual investors at a severe disadvantage against firms like this.
It can be stopped. And it should be stopped. And the only reason it is not being stopped is because too many rich and powerful people make too much money on it.
Re:The efficiency of capitalism (Score:1, Insightful)
This corporation can't send people with guns to your house to get more money. Obama Can Do It! ...won't borrow 40 cents for ever dollar it spends, from China, leaving the mess for your kids to pay for. ...won't forward this email to 10 friends!
This corporation can't print more money, watering down the money you yourself have, Obama Can Do It!
This corporation won't get the smug approval of 4 out of the 5 major news outlets, Obama Can Do It!
This corporation won't blame its losses on Bush.
Failure of best practices (Score:5, Insightful)
No proper change management, no peer review, no proper lab testing. Dev should always reflect production to the greatest reasonable level. No proper maintenance windows. You should never be surprised by a change in production. This is a case study in incompetence and the failure to execute industry best practices. I'm guessing the guy or gal who raised the best practices flag was ignored as being inconvenient or too expensive.
If I'd done this kind of thing when I was working with the exchanges I would have been fired in a heartbeat. Whoever failed to utilize best practices, or whoever failed to allow the utilization of best practices had damn well better have been fired. This is incompetence of the highest level and a perfect example of why ITIL based best practices were born.
Re:The efficiency of capitalism (Score:3, Insightful)
The corporations (and other moneyed interests) don't have to do any of that because it pays the government du jour to do its bidding. The whole idea of representative democracy is to prevent and minimize that by introducing checks on power by the public.
Re:There should be a mandatory one second delay. (Score:5, Insightful)
You are confused. "Algorithmic trading" and "High Frequency Trading" are two different things, used for two different purposes. This was caused by AT, and you are complaining about HFS. That makes no sense.
IT Debt (Score:5, Insightful)
They had some code that processed orders in a special way. There was a flag on the order they could set that would trigger that code. We will call this Power Peg. They later moved away from that functionality but it still existed in the system. It sat there for years untested and unused. 9 years later they added new functionality and decided to reuse that same flag. The new code also disabled Power Peg.
When they pushed the new code into production, they missed a server. That missed server still had Power Peg looking for that flag. Orders started setting that flag and it was processed correctly on all but one server. But that last server was placing orders incorrectly. The logic that Power Peg used was not valid anymore. In a panic they rolled back the code on the servers. Not knowing that Power Peg was the issue, they now had all the servers running Power Peg again.
Re:In other news.. (Score:3, Insightful)
So...where did the money go?
The money went to the people on the other side of the trades. Knight lost $400M. Many other people, accumulatively, gained $400M.
Re:Failure of best practices (Score:4, Insightful)
Dude, Wall Street traders demand software engineers get things done in minutes and hours!
There is no process as it all had to be done yesterday if you ask anyone who has worked in that environment.
They pay top dollars and change core algorithms on the fly as each millisecond costs money to a competitor who has a more efficient trading algorithm that steals from their own HTC network.
So when when skims and manipulates the prices in milliseconds they steal from those who have process engineered designs who are too slow to react.
If it messes up they get bailed out by the SEC anyway and they can just fire the programmer.
Re:The efficiency of capitalism (Score:5, Insightful)
This is what I like best about /. (Score:5, Insightful)
Re:The efficiency of capitalism (Score:5, Insightful)
Re:This is what I like best about /. (Score:5, Insightful)
Capitalism produces large corporations that are very efficient at doing dumb things.
The have your punk kid nephew do it mentality (Score:4, Insightful)
1. Python. I thought all the quants liked C, assembler, and even VHDL for their high frequency stuff. No matter
2. "2nd technician to review". If this were flight hardware or a bridge or skyscraper, there would be a second "technician" to review and at least one "engineer" to personally sign off that what was built/deployed is a) done right and b) is what you want
3. "no written procedures". There are a very small number of things in life about which it is absolutely imperative to keep a rod firmly up one's ass: a. moving machinery, b. formal mathematics, and c: hundreds of millions of dollars of your clients and shareholders' money.
Re:This is what I like best about /. (Score:3, Insightful)
I'm too busy thinking about how many children with cancer could be saved for 400 million US dollars.
Re:Garden Variety Upgrade SNAFU (Score:5, Insightful)
There were a number of errors made here.
There were a lot of places that you could put in a control to prevent or limit the effect of these kinds of errors, and that's the lesson people need to learn. Yes, mistakes happen! But try to make it hard to make a mistake, easy to recover from a mistake, and really easy to NOTICE a mistake.
Re:This is what I like best about /. (Score:5, Insightful)
Re:The efficiency of capitalism (Score:3, Insightful)
It is usually just wasted.
By your own previous logic, money can never be "wasted", just redistributed.
Every dollar that the government collects in taxes or creates out of thin air is redistributed back into the private sector.
Re:$400 billion / year is "essentially zero"? (Score:3, Insightful)
No. You're basically wrong on every point you bring up.
"Looking at what has happened in other countries, the "you're fucked" point, the point at which you can't escape the death spiral, is about 100% of GDP - when a country owes as much as it generates."
No, there is no magic debt/GDP limit: http://krugman.blogs.nytimes.com/2013/05/26/reinhart-and-rogoff-are-not-happy/ [nytimes.com]
"That tells us we are about six to eight years from becoming Greece."
No, Greece doesn't have its own currency, so we are in no way the same as Greece: http://krugman.blogs.nytimes.com/2013/10/18/the-china-debt-syndrome/?_r=0 [nytimes.com]
Re:private dumb: $20K. Govt dumb: $400 billion (Score:5, Insightful)
And the reason your gov't is making an agency to do a contract to get something done isn't inefficiency, it's socialism. Seriously. It's make work projects to try and spread some money around because otherwise the natural tendency if for money to be hoarded at the top by a few wealthy oligarchs (we call 'em the 1% these days).
The US Postal Service has God like efficiency. Just ask Netflix. Or just mail a letter and watch it travel across the breath of the United States in less than a week for 33 1/3 cents. The Space programs were (and are) amazing, and did things private industry couldn't hope to accomplish and that we're still reaping the benefits from today. The Gov't can be plenty efficient when it wants to be. It can also be very inefficient when it wants to. There are times when that's a good thing.
Re:This is what I like best about /. (Score:5, Insightful)
The public and private sector do different things better. While the private sector is more efficient in terms of service or widgets per hour, there's also heavy pressure to trick, mislead, and rip off consumers and customers.
Such "trick" pressure is noticeably lower in the public sector. There is relatively little pressure for a line worker and middle manager to rip off or trick customers because they don't have to compete with other cut-throat competition and are not sweating over narrow margins. They generally don't want the drama of customers sending or phoning complaints and so do an adequate job, even if a bit slow at it.
Certain kinds of services are better under one versus the other. If it's a service where it's easy to fool customers, especially over longer-term features, then it's probably best done by or heavily regulated by the gov't. If it's something that's relatively easy for a consumer to verify the quality of, then it's probably best done by the private sector.
Re:Combining nerdism with capitalism (Score:5, Insightful)
>> Capitalism is very effective in what it does
Pumping the money from the poor to the rich.
Re:private dumb: $20K. Govt dumb: $400 billion (Score:1, Insightful)
>> I WORK for that agency
So, you are a repuglican saboteur troll that takes public money and then badmouths the hand that feeds them.
The government needs to fire repuglicans that don't believe in the system, and hire more democrats who cares and gets things done.
The problem with the government isn't intrinsic, it's the scum that saboteurs the system on a daily basis and then walks around screaming how "it's not working". Get rid of the scum, hire decent people that believe in public good, and things work marvelously.
Re:Garden Variety Upgrade SNAFU (Score:4, Insightful)
Except they built in enough safeguards that they were able to debug it and get it fully operational. Having done so, it then accomplished it's mission and considerably more.
The appropriate measure varies based on the type of risk.
Re:The efficiency of capitalism (Score:3, Insightful)
Indeed, I always forget that I have the FREEDOM to be homeless, live under the stars with my wife and children, enjoy the nature, you know. Republican dream for working people.
Re:private dumb: $20K. Govt dumb: $400 billion (Score:3, Insightful)
You may be right about the "make work" projects but that doesn't make it not exceedingly stupid. How about they pay their own worker to do that work and use the other money to get other things. If they do that they are providing more value for everyone while distributing money.
Re:The efficiency of capitalism (Score:4, Insightful)