Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
United States IT Technology

FAA Needs Until 2030 To Fix Safety System That Failed Last Month (bloomberg.com) 86

Posted by msmash from the how-about-that dept.
US aviation authorities are years behind on updating the critical-alert system that failed spectacularly last month, causing thousands of flight disruptions. Critics say the delay is a threat to passenger safety. From a report: House lawmakers are scheduled to hold a hearing Tuesday on aviation safety at which they're likely to raise questions about the Jan. 11 meltdown of the Federal Aviation Administration's Notice to Air Missions system, or Notam. While the FAA has taken steps to ensure that the platform won't fail in the same way again, its problems go far deeper after years of neglect, including issues that contributed to one of the worst near-disasters in US aviation history six years ago.

Notam produces bulletins for pilots flying in the US about any safety issues along a route. They could include anything from broken airport lights to an emergency closing of airspace, such as when the FAA temporarily suspended flights along the US East Coast on Feb. 4 during the military mission to destroy a Chinese surveillance balloon. Pilots are required to check them before departing. But according to government records, industry groups and dozens of pilot reports, the system is packed with unnecessary information that's difficult to sort, and its antiquated language makes the bulletins hard to comprehend. The FAA acknowledges the shortcomings and plans improvements, but acting Administrator Billy Nolen notified House lawmakers Jan. 27 that fixes wouldn't be fully completed until 2030. Congress first ordered the agency to begin upgrading the Notam system in 2012.
This discussion has been archived. No new comments can be posted.

FAA Needs Until 2030 To Fix Safety System That Failed Last Month More

FAA Needs Until 2030 To Fix Safety System That Failed Last Month

Comments Filter:

  • No change to NOTAM since 2012 (Score:1)

    by Anonymous Coward

    Congress ordered the FAA to start upgrading the system in 2012... but that system hasn't changed at all since that time (though the FAA is probably spending millions a year on "upgrades").

    The FAA is probably the slowest agency in the entire Federal Government when it comes to making upgrades and changes. They still use machines with 5.25" 360K floppy disks for mission-critical systems for Christ's sake.

    • They still use machines with 5.25" 360K floppy disks for mission-critical systems for Christ's sake.

      Because they work. Let's redo the whole thing with handful of Raspberry Pi's and python 2.7. What could possibly go wrong?

      • > [keep old systems] Because they work.

        Or mostly work. The problem is that a complete rewrite of a system of this magnitude so heavily tied to safety has a very high probability of turning into a boondoggle such that incremental improvements may be the safer route even if clunky. Humans have not figured out how to consistently build large reliable software systems, especially if they are one-of-a-kind. If I were the the CEO of a big consulting firm, I'd never bid on such a project (unless I plan on reti

        • Does it have to be that bad? Surely the code is a reflection of written policy documents. Start fresh, translate the rules and regs to the code in a modern manner. Run the systems side by side for a couple years with the newer one obviously not calling the shots.

          • Re:Not with a ten-foot pole (Score:4, Interesting)

            by Tablizer ( 95088 ) on Tuesday February 07, 2023 @11:58AM (#63272603) Journal

            > Surely the code is a reflection of written policy documents.

            I'm sure such documents are full of ambiguities and gaps. Whoever wrote the code probably made a best-guess interpretation, and in many cases it was changed to a different interpretation when practical problems were discovered in operation. Such decisions are either not tracked, or get lost in the shuffle over time.

            BeenThereDoneThat.

            > Run the systems side by side for a couple years with the newer one obviously not calling the shots.

            I'm skeptical that's practical. There will probably be notable differences because as the TFA pointed out, some of the output of the existing system is confusing because its based on fudging of outmoded processes. It would be expensive for operations staff to pause to study the differences and decide whether such is expected or a mistake. If they are not going to clean up the historical cruft in the rewrite, might as well keep patching the old system and write emulators for obsolete hardware.

            • alright, no doubt it sounds brutal and a fresh start might be the only way to move forward. At the end of the day the job is not to crash the damn plane either into the mountain or each other. surely that singular feature can be modeled, coded, tested and QA'd to hell and back. We are talking about big moving objects in the real world. There are finite permutations of where they can be and what they can be doing.

          • Here's the problem that you're not seeing. Administrations change over the course of a project this large (the money movers). Point of Contacts change (the policy driver). The fed handler will most likely not be same throughout the majority of the project and neither will the contractor (following the POCs changes). The scope of work will take years to capture. The technology will be obsolete by the time you deploy to production. If you're lucky, there will be a board of review that might actually hav

        • Re: (Score:1)

          by Anonymous Coward

          Why is this an one-off? Or really, anybody have a decent description of the NOTAM internals, beyond the woke waffle?

          The trick of reliably sending out text messages has quite a bit of precedent. The naval boys do it. FidoNet Echomail is pretty solid. Netnews can at least provide a solid basis to start from. Email lists have been around for ages and work Just Fine in controlled environments. (The problem with mailinglists and email in general is idiot^Wuneducated users. Write a proper "how to use" procedure,

          • Re: (Score:2)

            by jd ( 1658 )

            The trick of reliably sending out text messages has quite a bit of precedent. The naval boys do it.

            I know. I helped write some of the software used for that purpose by the Navy. It's not the greatest software ever written, I found quite a few bugs and the static checkers I tried generally had mental breakdowns, but it worked when I was there and presumably still works today unless it has been replaced - which TBH is quite likely. But whether any replacement is any better is an open question given the qualit

        • Re: (Score:1)

          by Anonymous Coward

          Exactly. Banks don't like to lose money so they run 50 year old COBOL code. It's been vetted for decades. Nuclear plants are still running reactors from PDP-11 minicomputers because well they don't like having "oopsies" either. Yeah you could probably replace it with something newer, but are you willing to sign off on it?

    • Re: (Score:3)

      by fermion ( 181285 )
      This is not a situation for agile development. We need a process, and validation

      What is sad here is these highly paid professionals are using high school excuses. It was too hard to read. It was too complicated.

    • Others here are offering up strawmen and false dichotomies. You're right that the FAA is painfully slow. I mean, they have reason to be conservative given that a huge proportion of their regulations are written in blood, but they take it to extreme measures. This has led the the slow death of private aircraft because they haven't allowed any real innovation that could improve performance and safety. Fifty years ago, you could buy a plane for about twice the cost of an expensive car. A brand-new Cessna 172L

  • Typical bullshit do-nothing money-wasting process at the federal government level. Brought to you by the same people who blessed us with $20k toilet seats.

    There is zero accountability for lack of project completion or any noticeable progress.

    People should lose their jobs over this.

    • Re:Typical (Score:5, Informative)

      by david.emery ( 127135 ) on Tuesday February 07, 2023 @11:23AM (#63272481)

      Just out of curiosity, how much experience do you have working on/managing large government contracts? "Typical" implies sufficient knowledge to establish a pattern, but $20k toilet seats is just a single anecdotal data point, and one not from a large scale automated system. And do you have any way to fix this other than firing people?

      (I worked most of my career on such contracts. There's LOTS that goes wrong and MUCH need for improvement. Some of the blame is on rules set by Congress that agencies must obey. Some of it on government contracting practices that emphasize 'treating everyone fairly' or similar priorities other than 'most efficient execution.' Some of it is a lack of knowledge on the part of the government workers and their contractors. And some of it is just the problems they're trying to solve, such as air traffic control which complex interfaces to other national/international systems and standards, are not easy in their own right. Oh, yes, I HAVE seen managers get fired for non-delivery. Usually but not always for good cause.)

      • Re: (Score:2, Troll)

        by DarkOx ( 621550 )

        Some of it on government contracting practices that emphasize 'treating everyone fairly' or similar priorities other than 'most efficient execution.'

        This is a big problem right here. We should have some spot check audit + whales audit scheme to make sure agencies and people issuing contracts are doing so in a FACIALLY neutral way.

        However dwelling on everything being functionally neutral is a LUXURY for nations that don't have ballooning deficits.

      • Sadly, I have quite a bit of experience working in that environment. Not most of my career, but a number of years on 9 and 10 figure projects. It's interesting that you'd question my ability to comment about the process at all.

        Failure to be a good steward of taxpayer money and priorities should be grounds for dismissal. What we get instead is leadership by committee and zero incentive to be efficient since the chance of being actually held accountable for failure is very small.

        Best,

      • The FAA Nextgen upgrade is going to turn into another NASA SLS. Just a neverending string of government handouts^h^h^h^h^h^hcontracts for certain contractors who never finish the work.

    • Re: (Score:3, Interesting)

      by Tablizer ( 95088 )

      The average political career is often shorter than certain important projects. Thus, there's little incentive to fund and support them properly: the problem is dumped onto the next generation of politicians. It's why gov't pensions are often under-funded: hand out favors now and let the next batch figure out how to pay for it. This is one reason I'm against term limits in Congress: let them have some skin in the future game.

      Fix gerrymandering instead (such as limiting the number of lines that can be used to

    • Re:Typical (Score:5, Interesting)

      by RedMage ( 136286 ) on Tuesday February 07, 2023 @12:37PM (#63272737) Homepage

      I love the $20k toilet seat story - it's a great example of how things get simplified to the point of uselessness.
      Yes, the effect of the problem was that a toilet seat had a total acquisition cost of around $20k. But consider really that the actual seat cost no more than a typical seat. The real cost was in designing and creating the molds for that seat, and materials testing for the specific application. Obviously the application was unique (space shuttle), and required that it function properly (or it would have been a literal shit-show.) And because those up-front costs were high, and the actual number of units low, the aggregate cost of a seat was very expensive, per unit.
      Where there was "government project overhead" (i.e. waste) was in the administration and management of the project itself, but not so much in the physical object. I'm sure there was plenty of paperwork and tracking and special shipping involved that added significantly.

    • Re: (Score:3)

      by quetwo ( 1203948 )

      So..... Those $20k toilet seats. Digging into that one a bit further, you'll find out that was mostly because the military had to purchase some secret tech, but couldn't publicly disclose the details. They re-allocate the dollars to other things that aren't secret.

      There was a NPR reports many, many years ago about people being pissed about a $100k oak desk that was purchased for some government office in Indiana. When they dug into it, it was actually for the SCIF room they built, but didn't list it out

    • There was never a $20,000 toilet seat. The supposedly scandalous price was $640, and it turned out that the item was a fiberglass wall panel for a helicopter that had the toilet seat built in. The $7000 coffee pot was also a non story. It was a custom unit for a C-5A aircraft. Aircraft coffee makers have to withstand sudden decompression. The C-5B was redesigned to use commercial coffee makers which only cost $4000, in back then dollars.

  • This is another example of an over-scoped project. This is a notification system, ever hear about The EAS/Amber Alert System? [fcc.gov] The FAA needs to talk to the FCC and find out what they did and reuse it. Better yet, let the FCC manage the system since they seem to be doing a better job anyway. Just a simple opt-in mode straight to the cell phone.

    • EAS does not need to work with non USA systems but NOTAM does

    • This is another example of an over-scoped project. This is a notification system, ever hear about The EAS/Amber Alert System? [fcc.gov] The FAA needs to talk to the FCC and find out what they did and reuse it. Better yet, let the FCC manage the system since they seem to be doing a better job anyway. Just a simple opt-in mode straight to the cell phone.

      My first reaction is meta-snark: "Do you want 5G? Because this is how you get 5G."

      But in all seriousness, the NOTAM system has nothing to do with cell phones. It is not that kind of "notification" system.

      NOTAMs are compiled into a section of the pre-flight briefing documents. The most common ones are fairly small: runway or taxiway out of service, radio navigation devices (of assorted kinds, both enroute and takeoff/landing) broken, all kinds of airport lights broken, airport construction, nearby non-airpor

      • Re: (Score:2)

        by cstacy ( 534252 )

        quote>
        the NOTAM system (to initiate backsup or something) each day.

        Nasty backsups, we hates them forever!

      • The FCC knows something about communication, it's their business and we as taxpayers have invested a lot of money in the systems they've implemented. It's time for the executive branch to stop thinking within strict vertical silos and start leveraging cross-platform synergies. That means an entire IT organization for the federal gov't that supports cross-functional domains. Granted, that's a bigger problem to solve but it would at least take care of things like life cycles, upgrades, and the endless need fo

        • Re: (Score:2)

          by cstacy ( 534252 )

          The FCC knows something about communication, it's their business

          The FCC does know something about radios, I'll grant you that. What makes you think they are experts in "communication" in general? Amber Alerts are nothing like NOTAMS. And by the way, the FCC is most decidedly not a "business". It's a very political and very bureaucratic regulatory agency.

          It's time for the executive branch to stop thinking within strict vertical silos and start leveraging cross-platform synergies.

          Oh! I see you're some kind of bureaucratic "business" manager yourself! Well, let's all get Agile and leverage those cross-platform synergies! Hopefully we can use the blockchain.

          We'd like to think that NOTAMs is some sort of particular dinosaur, but it is just that, a dinosaur that needs to be extinct in its current fashion. Nothing it produces is a national security concern, it's just a stupid, obsolete broadcast mechanism.

          I don't think you appreciate how difficul

    • Re: (Score:2)

      by quetwo ( 1203948 )

      EAS is really old tech that still works, but is mostly dismantled at this point. It's been replaced by IPAWS -- which has been an utter disaster. IPAWS has been the primary system for nearly 10 years, and out of those 10 years, the system failed 8 times during the yearly tests/audits. It relies on this weird XML based data flow over IP connected systems that only do a connection every few months, except for an emergency. There are different versions of the protocol and a set of non-technologists that ar

  • You could write a completely new system from scratch in a quarter of this time. You just need to cut the BS crap and stop trying to resurrect or patch the old system and move on to the better future.

  • The 737Max debacle proved the FAA is in over its head.

    https://www.floridatechonline.... [floridatechonline.com]

    The Case for Privatization

    Most of the airlines are for it: Airlines for America (A4A), the primary lobbying group for airlines, has been pushing for privatization for many years. American Airlines CEO Doug Parker said “We need to get ATC reform done” [thestreet.com] at a recent industry gathering. At the same event, Southwest CEO Gary Kelly said privatizing air traffic control is the industry’s top issue. (Note: Delta

    • What? The 737Max was Boeing building a shitty product because they are no longer run by former engineers. Today the company is run by MBA suits that give zero fucks about anything besides immediate growth. Boeing employees even said they wouldn't let their families fly on it. https://www.theguardian.com/bu... [theguardian.com]

      • You're both right, actually. FAA (and their associated engineering base, including FFRDC) lost technical knowledge and managerial expertise. But more importantly, they gave away oversight in "reforms" in the early 2000s. Those reforms gave manufacturers (particularly Boeing) more authority and less oversight.

        Boeing, on its part, changed significantly with the McDonell-Douglas merger, becoming much more financial focused and less engineering driven. Peter Robison's book, "Flying Blind" describes this in

      • Re: (Score:2)

        by schwit1 ( 797399 )

        No disagreement the 737Max was designed and built by Boeing. The FAA's incompetence allowed it to be certified.

    • Re: (Score:2)

      by tlhIngan ( 30335 )

      Private jet owners and operators are not on board: The National Business Aviation Association says that airlines will have an oversized influence on any new, privatized organization, which may limit private jetsâ(TM) access to airspace and airports.

      General Aviation is not on board. Privatizing the FAA helps airlines a lot to the detriment of a large segment of aviation that flies mostly under the radar (all meanings). Airlines have such an outsized influence that General Aviation has to lobby hard to m

      • I like GA far more than commercial so I am quite supportive.

        So we need to figure out a way to make sure they have fair access to working system, and that can't be managed by the FAA.

        We've tried that experiment and it failed. Monopolies have no incentives.

        It's got to be more like the Internet with coalitions of aligned interests, including public interest advocates.

        Anybody can still pay $3.50 a month and get an IPv4 so it's working.

    • The 737Max debacle proved the FAA is in over its head.

      The FAA didn't build the 737Max, a private company did. Why do you think a private company would do any better? You do realize that not implementing federal regulations (because it's cheaper to not) is why the texas grid keeps failing, right? How do you not understand that corporations will only do the bare minimum?

    • How on earth will privatization help? If you don't like the lane you were assigned, you take your business to a different air traffic organisation?

      Privatisation just means some company gets to run the monopoly, but cream off profit with less accountability.

  • ...is high speed rail. There are many city pairs that airlines don't want to fly because they aren't cost-effective [sfexaminer.com] but there's currently no practical alternative.

    When your airport is snowed or fogged out, it would be useful to be able to fly in or out of a different airport. To that end, airports should have HSR terminals so airlines can seamlessly reroute passengers and their baggage when flights are grounded.

    The nation needs this kind of resiliency in our long distance travel network.

    • They would never be able to secure the land for rail. They can't even get land for transmission lines.

    • Re: (Score:2)

      by ebh ( 116526 )

      As long as any new rail system would be expected to turn a profit, that will never happen. :(

      • Re: (Score:2)

        by Ichijo ( 607641 )

        As long as any new rail system would be expected to turn a profit, that will never happen. :(

        Or we could expect air travel to make a profit, including having airlines pay for the overhaul of the FAA's systems and making airports pay property taxes.

    • Aaah, I remember the age of idealism with fond memories! Not to be too snarky... I really do remember when it seemed like an obvious solution to so many problems.

      Yes, it would be great if the US had a real high-speed rail network that connected every city with at least a million people plus cities over half a million people that are more than ~200 miles from an airport with primary air service.

      An optimistic cost for such an endeavour is $15 trillion. A pessimistic cost is 5x that. And, that is without ev

      • Re: (Score:2)

        by Ichijo ( 607641 )

        You could make a few more optimistic assumptions and maybe get the cost per passenger mile travelled over 100 years down to $1,000.

        Inflation's a pain, isn't it?

    • No, frak those. T(ele/rans)porter/Portal!

    • If an airline can't justify the expense of operating a route between two cities, how can we can justify building high speed rail infrastructure at a cost of ~$30million per mile, making the very bold assumption we can get the price down to as low as China has managed?

      I don't challenge that there are scenarios where high speed rail would be more efficient than air travel (small/medium distance, high passenger volume), but I would expect those routes to be cost effective for airlines to operate today.

      • Re: (Score:2)

        by Ichijo ( 607641 )

        Airliners typically take 100-150 flight miles to get to cruising altitude, and that uses up a lot of fuel. This makes shorter flights more expensive per passenger mile than longer flights.

  • They have so much red tape it's stupid! A minor change in their system takes an act of God to carry out. Why is this? Because every damn person in charge of everything is afraid of taking responsibilities. If one thing goes wrong, they don't want their job gone. What I want is that if people sit together and plan the upgrades, with the condition that if something fails, they fix it and not retaliate against anybody. That's the only way to move forward.

    On the other hand, our politicians are full of BS too

    • Re: (Score:2)

      by bobby ( 109046 )

      I absolutely agree with you, and I'll add: I think the way the system has devolved is a detriment to good people wanting to get involved, so we end up with the upper management / govt. dept workers being some of the least qualified, and motivated by some of the worst goals.

      I think an additional component is that (far too many) people tend to like power and control. People who desire and attain positions of power and decision-making love to hold their power, telling you and me to "move along".

      Any oversight

    • They have so much red tape it's stupid! A minor change in their system takes an act of God to carry out. Why is this? Because every damn person in charge of everything is afraid of taking responsibilities. If one thing goes wrong, they don't want their job gone. What I want is that if people sit together and plan the upgrades, with the condition that if something fails, they fix it and not retaliate against anybody. That's the only way to move forward.

      If something fails, PEOPLE CAN DIE.

      So, you know, the stakes are pretty high. It's not just about your bonus %.

      • People die because government agencies do nothing, too. The NTSB has provided hundreds of recommendations over the years to improve ground, water, and air transport safety that have been utterly ignored by various agencies. Sometimes recommendations are technically infeasible, but others are just blown off or blame is shifted. After USAir Flight 1549 ("Miracle on the Hudson"), the NTSB recommended a few things like moving the aft rafts forward to make them more accessible, as they ended up underwater after

  • Critics say the delay is a threat to passenger safety.

    Well, of course critics are going to say that. They're critics, so they're going to criticize. What else does anybody expect?
  • The best argument against more public ownership of infrastructure, which I support, is the incompetence of organisations like the FAA

  • Better to say 2030 and mean it than to say 2025 and know it won't be done until 2030 anyways.

    And to those who say it can/should be done sooner:
    You have a point: With more resources it might be doable sooner, but only up to a point. Fredrick Brooks Jr.'s Mythical Man-Month applies.

  • Any technical upgrade which lasts 18 years is bound to fail, because too many technologies will change over that long of a period. Idiocy to push the completion of fixes back 7 additional years. Fire the CIO.

  • Clone Mastodon, add better offline capabilities, lock down access, clone some clients, customize the document formats. Done and done. Really. We shouldn't be reinventing the messaging wheel.

    • I worked at a place that had an aging shared journaling system, it was mission-critical and desperately needed updating. We suggested a Slack Channel. People took to it immediately and just loved it, esp. the ability to search. They had to standardize a bit on vocabulary, but that's it.

  • 1. The FAA puts together a team of analysts and developers and produces a better version of NOTAM in-house.

    Most likely outcome: Commercial development companies will complain to their congressional representatives over this incursion of government into private enterprise. Congress will de-fund the FAA to the point that they can't afford to hire competent IT people who know enough not to erase a critical database, let alone re-engineer a more fault tolerant application.

    2. The FAA seeks bids for a new NOTAM

  • The government needs 7 years to do something that the private sector could probably pull off in less that 4 years, and cost at least 1/3 cheaper.

  • What I see from government procurement of computer systems, it's going to go like this:

    * Specifications will be issued 2024
    * Appropriation in 2025
    * Design in 2026
    * Design approval in 2027
    * Development starts 2028
    * Releases 2031

    And in 2031 when it releases, it will (optimistically) be based on 5 year old technology because the design was specified in 2026. The wheels of government grind slowly in the United States. Meanwhile, the rest of the world will have moved on. Why do you think it took so long to kill

  • Does anyone on /. actually know what OS is used for NOTAM system? I can't seem to find that information. There are a lot of bold statements made about fixing the system. But, what is the starting point?

Slashdot Top Deals

You are always doing something marginal when the boss drops by your desk.

Close