US Switches Air Traffic Control To New Computer System 160
coondoggie writes: The Federal Aviation Administration this week said it had completed the momentous replacement of the 40-year-old main computer systems that control air traffic in the US. Known as En Route Automation Modernization (ERAM), the system is expected to increase air traffic flow, improve automated navigation and strengthen aircraft conflict detection services, with the end result being increased safety and less flight congestion. The FAA said the Lockheed Martin-developed ERAM systems “uses nearly two million lines of computer code to process critical data for controllers, including aircraft identity, altitude, speed, and flight path. The system almost doubles the number of flights that can be tracked and displayed to controllers.”
Prepare (Score:1)
Re:Prepare (Score:5, Informative)
Oh, my! Re: Glitches (Score:2)
The article you're pointing to was about how one of the ERAM systems crashed trying to cope with a bizarre flight plan for a U-2 spy plane.
When I was working on AAS in the late 80s, one thing I was mildly concerned about was that the planned "upgrade" our project was trying to design wouldn't really be able to cope with super-sonic aircraft over the continental US. The requirements for how much area had to show on a controller's screen and how fast the radar sweeps were meant that anything at Concorde spee
I worked on the 1980s version (Score:2)
Back in the 1980s, the FAA's shiny new Advanced Automation System project (AAS) was being designed to replace the 1960s-vintage En-Route system, which used IBM 360/90 and 360/50 computers that were getting to be old, unmaintainable, and unreplaceable. (It was getting hard to even get cable connectors for components - imagine coming up with new SCSI-1 terminators these days.)
As with many military aircraft system contracts, they ran a design competition, which had funneled down from 4 bidders to two by the t
Uh, only doubled? (Score:5, Insightful)
So how does a 40 year old computer system get replaced and only doubles the number of flights capable of being tracked?
Re:Uh, only doubled? (Score:5, Funny)
So how does a 40 year old computer system get replaced and only doubles the number of flights capable of being tracked?
Very very slowly and at great expense.
Re:Uh, only doubled? (Score:5, Insightful)
So how does a 40 year old computer system get replaced and only doubles the number of flights capable of being tracked?
How about this concept: Maybe that is all that they set it up for. The rate limiting step of the Airway Traffic Control system just might be somewhere else so there would be no need to do anything else.
I do find it concerning that the system comprises of 'two million lines of code'. Last time I heard that metric was "Jurassic Park". And we know how well that turned out.
Re:Uh, only doubled? (Score:4, Interesting)
Just off the top of my head, major limiting factors are runways to get the flights into and out of the air, passenger demand, and the number of air traffic controllers. And like most projects, the cost and effort to scale rises dramatically with the amount of scale you target. Besides, if the system is anything like the air traffic management system I worked on, then it should scale much better than the system it replaced.
The software on the plane has more lines of code than that and some of that code actually controls the plane, auto-negotiate collision avoidance, etc. I'd be more worried about that - if ERAM goes down for a brief period, controllers wouldn't be able to see flights, but those aircraft would be able to maintain control of their aircraft until ERAM came back up. If the flight's control system went, then the traffic controller would only be able to watch the flight as it hurtled out of control.
Re: (Score:2)
Re: (Score:3)
Re: (Score:2)
It was supposed to be a funny, guys. Jurassic Park. Dinosaurs.
Randall [xkcd.com] would have figured it out.
Re: (Score:2)
Tell them it is written in ADA _then_ they will think that it is a joke
Re: (Score:2)
Re: (Score:2)
Especially considering that ADA is a DOD language. So you have to code in triplicate.
Re: (Score:2)
Re:Uh, only doubled? (Score:5, Insightful)
One of my most productive days was throwing away 1,000 lines of code.
- Ken Thompson
Re: (Score:3)
I do find it concerning that the system comprises of 'two million lines of code'. Last time I heard that metric was "Jurassic Park". And we know how well that turned out.
Marketing wank. They added up all the lines from everything, including the firmware in the mouse and the windows.h header file that is 99.9% irrelevant to their project, included all the comments, treated every "\r\n" as two lines, and threw in the Linux kernel for good measure because their office wifi router runs that.
I really doubt that the actual ATC system is 2 million lines, not least because it would be extremely difficult to audit.
Re: (Score:2)
You did get the bit about how this system was decades behind schedule and tens or hundreds of billions over budget, with a couple of major iterations thrown away in the process? 2MLOC sounds nice, clean, compact, and surprisingly low.
Re: (Score:2)
How many lines of code to track airplanes using the FAA's satellite network?
How many lines of code to generate flight courses?
How many lines of code to generate flight progress strips as airplanes approach a control area?
etc.
Re:Uh, only doubled? (Score:5, Funny)
So how does a 40 year old computer system get replaced and only doubles the number of flights capable of being tracked?
They switched from 7-bit ASCII to 8-bit ASCII...
Re:Uh, only doubled? (Score:4, Interesting)
Tracking double the number of flights likely requires about 4x the about of computing power. A naive comparison grows at a rate of (n)(n-1)/2. You might be able to reduce that by not comparing aircraft that aren't going to be anywhere near each other (e.g. a plane in Washington D.C. cannot readily crash into a plane in Los Angeles, CA until they get close to halfway across the country), but still....
Re: (Score:1)
It's amazing computing power has increased by as much as four times since 1970!
Re: (Score:3)
It's more likely a limitation of the hardware they use to track and communicate with aircraft. There are only so many radio channels, so many radar installations, so much bandwidth available. Many of the comms protocols used are ancient and can't easily be replaced by more efficient ones.
Re: (Score:2)
"So how does a 40 year old computer system get replaced and only doubles the number of flights capable of being tracked?"
VMS simulators are not that fast, after all only planes and trains and a few factories use it.
Re: (Score:2)
So how does a 40 year old computer system get replaced and only doubles the number of flights capable of being tracked?
I believe a clue can be found in the following choice quote:
nearly two million lines of computer code
So what's the new system? (Score:1)
/me waits to hear that it's Windows-based...
Ada on AIX (Score:4, Informative)
It's mostly Ada running on AIX. See http://www.iaeng.org/publication/IMECS2009/IMECS2009_pp1095-1099.pdf.
"Display System (DS), User Requested Evaluation Tool (URET) and ERAM and have been developed mainly in the Ada programming language. " Page 2.
"Product supportability advantages led to the selection of the IBM P series processors, the AIX operating system, and CISCO switches." Page 3.
Re: (Score:1)
It's mostly Ada running on AIX.
The *backend* is mostly Ada running on AIX. The front end definitely is not. In the demo video they're running Internet Explorer to do conflict checking. Unless they're running it in Wine. :)
Re: (Score:2)
Just watched the video from the linked story
There may have been a couple of applications running on windows to view the data, but the bulk of the screens that the controllers were staring at looked distinctly like x-windows
I have to wonder how much they pay for those big square flat screens
Two million lines of code (Score:4, Funny)
what could possibly go wrong?
Re: (Score:3, Insightful)
Two million lines of code actually isn't that impressive, either for economy of code, or for scale of code, the two goals that you may publish such a statistic to support.
Windows 8? 40 million lines.
Quake 3 engine? 30 million lines.
The government has just come out and told us that the scale of complexity in a system that "doubled" capacity and that they paid who knows how much for... has about the complexity of the average enterprise class iPhone application.
Re: (Score:2)
The average enterprise class iPhone application isn't trusted lives with. Also, not inside an industry where an accident means deaths of hundreds of people at once. Nobody brings the average car accident in the news, for example when somebody kills themselves at the highway. But when a plane crashes, it comes in the news, so politicians and representatives of the airlines promise they do something, and tighten regulations. Meanwhile, car security is still shit as hell.
I guess its all formally proven. Is the
Re: (Score:2)
When it's connected to an implanted insulin pump, it's controlling lives pretty directly:
http://www.washingtonpost.com/... [washingtonpost.com]
Re: (Score:1)
Re: (Score:1)
Mainly to aid in compile time detection of errors... I've never programmed in ADA but a little VHDL in school and it looks very familiar.
And let me tell you... VHDL has the potential to be extremely verbose (behavioral models help as do other new features.. but thats off topic realy).
Re: (Score:2)
PL/SQL is a descendant of Ada. As a result I was involved in code review of a bunch of orbital mechanics code for y2k
Sometimes its pascal-iness makes it seem like you are reading pseudo code
Re: (Score:1)
I've never programmed in ADA but a little VHDL in school and it looks very familiar.
Ada's no more verbose than C++ or Java (Score:2)
It's designed for object-oriented use, with lots of type specification and such upfront, to push decisions into upfront design time rather than coding time, and it's not as terse as C or APL, but it's nowhere near as verbose as COBOL. I wouldn't use it today (mostly because its main uses are for military stuff I won't do, and for antique maintenance, and it doesn't have all the friendly libraries that I'm used to and probably doesn't easily link to non-Ada systems), but it's a fairly cromulent language.
Re: Two million lines of code (Score:2)
Lines of code = complexity?
Lemme guess. You're a programmer.
Ignat.
Re: (Score:3)
The average enterprise class iPhone application is nowhere near 2M LOC. You're off by at least one order of magnitude.
Re: (Score:2)
I think if you want to account for all sorts of things like weather, fuel of the planes cycling in the sky, collision pathing avoidance, and so on, it might be very complex. You factor in some functionality you can automate to make air traffic controller's lives less stres
Re: (Score:2)
Re: (Score:1)
I have no faith in the system. (Score:1, Troll)
Oh... and you will need another entirely new system to accommodate drones.
Only doubles?! (Score:1)
Wait, you write a new application from the ground up to operate on new hardware, in an era of grid computing, ridiculous amounts of possible ram and multi-core compute nodes, with modern programming structures that can hold obscene amounts of data in a single variable.... and you only managed to "double" the number of flights which can be tracked and analyzed?
Re: (Score:2)
Well, yes, but keep in mind they started on this project 20 years ago. It's about time now to start on this new system's replacement, which is scheduled to go operational in 2035.
Re: (Score:1)
In some ways, I hope that you are joking about this. 20 years to deploy an application which tracks flight paths? Lets go crazy conservative. A year to write the app and 3 years of testing accross airports using parallel PoCs for integration UAT. Anything more than 5 or 6 is absurd @ 2 million lines of code, even if you credit a year or two for government scale requirements gathering.
Re:Only doubles?! (Score:5, Informative)
Re: (Score:1)
Re:Only doubles?! (Score:5, Insightful)
Re:Only doubles?! (Score:5, Insightful)
Re: (Score:2)
I have had projects take as much as a decade.
If you have had projects that have taken as much as a decade then name a couple. If they were that big I bet that are not confidential.
Re: (Score:2)
Re: (Score:2)
I didn't say "open" I said "public". Her are some possible projects; IRS tax system, Stock Exchange system, Telephone switching network, etc.
I am basically calling into question the poster's assertion that he has worked on systems that have taken a decade to implement. Maybe a few years to implement and many more years to add features and debug but probably not a decade before being deployed. There are very few systems as big as the US air traffic control system And very few of those have been recently repl
LOTSa Naivete was involved. (Score:2)
Most of it on the part of the people who started the original project, who thought it would be done in 3-4 years, made way too many incorrect decisions for the wrong reasons, specified lots of requirements without understanding how impossible they were to meet, picked multiple sets of pie from multiple sets of skies, and didn't start with the ability to get kinds of budget they would have needed to do the job right (if they'd picked a definition of "right" that could have been implemented in the 1980s, when
Re: (Score:2)
Re: (Score:2)
First, it has to get the data -- which covers everything from radar skin-paints if the aircraft transponder isn't operating, to unpacking the data that that transponder is sending (which could include anything from a simple 4-digit number to altitude, airspeed, heading, etc, etc.). Oh, and it has to raise appropriate alerts if that 4-digit number happens to be one of several special codes (indicating anything from voice-radio outage to a hijacking). There are plenty
Re: (Score:2)
Over and above all that, there are plenty of other components which relate to Air Traffic Control system, such as various navaids (VORs and such, although they're slowly losing favor to GPS), ATIS and D-ATIS info updates, ACARS messaging, METAR info, etc. Again, these may not be under the control of the current new system, but they should certainly be considered in any design for the future.
Re: Only doubles?! (Score:2)
Sounds like a video game.
That's not a knock. Publish a buggy game and watch players complain. Then watch your stock plummet. Then watch the sheriff padlock the doors. And that's just a game.
Software is important.
Redundancy is really hard. (Score:2)
That's not even counting the huge amount of code that's designed to make sure all the other parts of the code are working, and to do something appropriate if they're not, and the code that's designed to make sure that code is also working. That stuff's a lot harder than the basic code, and getting it right is the difference between a system with double- or triple-redundant hardware that gets you the 8 9s of reliability the FAA naively thought was possible with 1980s hardware and a air-traffic control syste
Re: (Score:2)
Okay, shoot, I feel sort of bad now. I thought twenty years was pretty obvious as a joke. Honestly, I have no idea how long this project took.
I've worked on a five year project that easily topped half a million lines of code, maybe more, with well over a hundred developers working on it. And oddly enough, it actually was a videogame (as mentioned later in this thread) - an MMO, which actually shares some characteristics with such a system, I suppose. No one died if the game crashed or calculated somethi
Re: (Score:2)
9/11 from home... (Score:1)
According to researchers with MITRE and other experts, this hybrid system is the FAA’s first challenge as a system made up of both IP-connected and point-to-point subsystems increases the potential for the point-to-point systems to be compromised because of the increased connectivity to the system as a whole provided by the IP-connected systems, the GAO stated.
“The older systems are difficult to access remotely because few of them connect from FAA to external entities such as through the Internet. They also have limited lines of direct connection within FAA. Conversely, the new information systems for NextGen programs are designed to interoperate with other systems and use IP networking to communicate within FAA. According to experts, if one system connected to an IP network is compromised, damage can potentially spread to other systems on the network, continually expanding the parts of the system at risk,” the GAO stated.
Re: (Score:2)
It's about time that someone democratizes terrorism!
I could go all day on this... (Score:4, Informative)
It was a shortage of computer memory in the $2.4 billion air traffic control system while a U-2 spy plane flew over southwestern US that caused LAX computers to crash and hundreds of flights to be delayed on April 30. “In theory, the same vulnerability could have been used by an attacker in a deliberate shut-down,” security experts told Reuters. Now that the “very basic limitation of the system” is known, experts expressed concerns about aviation cyberattacks.
$2 billion air traffic control system failure blamed on shortage of computer memory
Lockheed Martin, which created the En Route Automation Modernization (ERAM) air traffic control system, claims it conducts "robust testing" on all its systems, yet the lack of altitude information in the U-2’s flight plan caused the automated system to cycle off and on trying to fix the error.
http://www.computerworld.com/a... [computerworld.com]
Re: (Score:3)
Due to a bug in the code, the data size became an order of magnitude larger than usual. This was a bug that sufficient memory would have obscured.
Re: (Score:2)
I'm surprised they were using dynamic memory allocation at all. When you want to create a robust, reliable system like this you normally statically allocate all RAM and don't allow the system to process things outside those limits. That way you don't run the risk of bugs like this happening, or memory leaks, or any number of other issues. It's standard practice for high reliability systems.
Re: (Score:3)
Obligatory Dilbert. (Score:2)
Requirement for very high reliability (Score:1)
Re:Requirement for very high reliability (Score:4, Insightful)
Written in Ada can make things better, but written by Lockheed Martin, so it balances itself out.
Re:Requirement for very high reliability (Score:5, Interesting)
Run, Don't Walk, From Software (Score:3, Interesting)
My car, TV, and entire life are now filled with much more software than ever. Now that they can "do" more, they are also slower, flakier, and more complicated. And as a computer programmer, I know why: even the simplest program is amazingly complex. Every keystroke is a pitfall.
Two million lines? I think I'll drive --- no, just walk.
Re: (Score:2)
Re: (Score:3)
And if you go way way back to get a car with no code, you end up with one of these:
http://themetapicture.com/cras... [themetapicture.com]
Re: (Score:2)
Re: (Score:2)
Crumple zones, collapsing steering wheels, airbags...they make a really huge difference.
Re: (Score:2)
If they had used a Saratoga (or any of it's sisters) it would have continued on in a straight line, leaving plastic parts in it's wake.
They were banned from demolition derby, because the metal they were made of was only technically sheet metal (should have been called plate).
Also the BelAir was an empty shell. No motor, no trans. Agenda driven testing.
2 million lines of code?? (Score:1)
can't they do it in one line of perl?
Flying (Score:1)
Re: (Score:3)
This isn't just landing approaches. It's following planes as they fly all over the country.
What are you suggesting? Thousands of spotters with binoculars and CB radios? So commercial flights are to be restricted to a time slot between 10 AM and 3 PM in the summer only?
Goodluckwiththat.
Re: (Score:2)
Re: (Score:2)
Radar is so twentieth century. ADS-B broadcasts GPS position, heading and some air data. Every other aircraft in the area is free to recieve and display nearby planes and tracks.
Re: (Score:2)
Heck, if IFR (I Follow Roads) is good enough for me, it should be good enough for anyone, right?
(One thing that struck me about several of the old Soviet Aeroflot planes I saw -- and flew on -- in Russia was the bomber-like downward looking windows in the cockpit. I don't know if that reflected the aircraft's original bomber roots or the fact that sometimes they did follow roads. My flight to Krasnoyarsk was diverted because of fog, for example. What, no autoland?)
In case anyone is curious (Score:5, Informative)
This is what one the old systems looked like [stanford.edu]
Went to Smithsonian Air/Space Museum for research (Score:2)
Back in the late 80s, when I was working on that decade's failed project to replace the 360/90-based systems, my coworker and I were in DC for a meeting on some phase of the project (or one of the related projects), and we had half a day spare, so we went to the Smithsonian Air&Space Museum to do "research". They didn't have examples of the system we were working on, but they did have some other air traffic control systems (Tracon, I think), and other cool stuff like astronaut ice cream. After that we
Nearly 2 Million Lines of Code (Score:2)
uses nearly two million lines of computer code .... The system almost doubles the number of flights that can be tracked and displayed to controllers
Nearly two million lines, and almost double the capacity... If they bumped it up to an even two million I wonder if they could've completely doubled the number of the flights that could be tracked.
And what if they expanded it to four million lines of code, could they have quadrupled the number of flights that could be tracked?
And what if they made the code self-replicating? Could they have support an infinite number of flights?
The real question is (Score:2)
Questions which are not sexy... (Score:2)
Were all developers of the system required to complete training and pass a knowledge check prior to beginning work?
Has the application had manual/dynamic penetration testing performed against it?
Are there any critical/high/medium findings?
What is the timeline to address pen test findings?
How is access authenticated?
Is the application segmented housed in a dedicated DMZ?
Is there firewalling within the application stack?
Are Web Application Firewalls used?
What intrusion detection systems are in place?
What logs
Re: (Score:1)
Moore's law times government equals...
Re:40 years & merely "almost doubles" performa (Score:5, Insightful)
Hmmm. People are still the same size, fuel is still the same, turbines still use the same theories, the planet hasn't gotten bigger, the atmosphere is still the same, our materials are still the same..
Could it be, and this might be a shocker, could it be that the limits on materials have nothing to do with information processing?
For example, you might want to sit down for this and read it a few times, could it be that just because processors got a thousand times faster it doesn't mean that we can somehow actually put a thousand times more airplanes in the air?
I'm just wondering out loud here.
Re:40 years & merely "almost doubles" performa (Score:4, Insightful)
You are largley right here, the gains in thoughput in the system will be made by reducing seperation between aircraft, so you can have twice as many aircraft on the same airways. Those reductions in seperation can only go so far, as you have to have a system that can still fail back to stone age (100% down) and still be reasonably safe. At that point controllers fall back to using primary radar, radio and bits of paper in stacks, i.e. how it used to be done before computers.
The improved processing and tracking allows some saftey margins to be compressed, but not many, and not by much.
Re: (Score:1)
HA! (Score:3)
Stopped reading right there.
Re: (Score:1)
And I can also spoof radar, and I'm absolutely sure that the FAA is ignorant of the possibility of spoofing and has done nothing at all to detect it, ever.
And, given that these things are deliberately for sale, and have to accept input from GPS, how the fuck are you going to prevent them from being given bad data? Physical access to the hardware will render it compromised anyway.
Re: (Score:2)
If they're ignorant, how the hell are they supposed to detect it, ever?
Re: (Score:1)
Satellites better at tracking "drones", I think not.
Re: (Score:2)
Yes... because a system that requires the aircraft to transmit a signal containing GPS data is going to make it easier to track drones without the necessary equipment?
Re: (Score:1)
Probably not, and the system shat it's self when a single U2 spy plane flew into the air space at 60K feet.
Apparently it tried to check all aircraft/altitudes for collision courses and then took a giant shit and crashed.
http://arstechnica.com/informa... [arstechnica.com]
Re: (Score:2)
That would be a kind of 'false positive'. A report of an aircraft in a position where there is none. Soon, false negatives (no ADS-B where surveillance radar shows one) will be investigated by the dispatch of armed fighter aircraft. False positives can be handled in a similar manner by tracking the source of spoof signals and dispatching the appropriate countermeasures.