Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
×
The Almighty Buck Microsoft

Delta Seeks Damages From CrowdStrike, Microsoft After Outage (cnbc.com) 201

An anonymous reader quotes a report from CNBC: Delta Air Lines has hired prominent attorney David Boies to seek damages from CrowdStrike and Microsoft following an outage this month that caused millions of computers to crash, leading to thousands of flight cancellations. CrowdStrike shares fell as much as 5% in extended trading on Monday after CNBC's Phil Lebeau reported on Delta's hiring of Boies, chairman of Boies Schiller Flexner. Microsoft was little changed. [...] While no suit has been filed, Delta plans to seek compensation from Microsoft and CrowdStrike, Lebeau reported. The outages cost Delta an estimated $350 million to $500 million. Delta is dealing with over 176,000 refund or reimbursement requests after almost 7,000 flights were canceled.

Boies is known for representing the U.S. government in its landmark antitrust case against Microsoft and for helping win a decision that overturned California's ban on gay marriage. He also worked with Harvey Weinstein, the imprisoned former Hollywood mogul, and Theranos founder Elizabeth Holmes, who is currently serving a prison sentence for defrauding investors. Insurance startup Parametrix estimated that the CrowdStrike incident resulted in a total loss of $5.4 billion for Fortune 500 companies, not including Microsoft.

This discussion has been archived. No new comments can be posted.

Delta Seeks Damages From CrowdStrike, Microsoft After Outage

Comments Filter:
  • Liability (Score:5, Interesting)

    by aaarrrgggh ( 9205 ) on Tuesday July 30, 2024 @05:25AM (#64666172)

    While CrowdStrike (and maybe Microsoft) really screwed up, Delta will likely only be able to recover the direct IT costs of the BSOD restoration effort. The magnitude of business losses are squarely on Delta for having grossly inadequate business continuity plans and systems.

    • by Luckyo ( 1726890 )

      That depends. Ranging from the specifics of contract they have to if court finds CroudStrike's negligence to be gross or not.

      • Re:Liability (Score:5, Insightful)

        by gweihir ( 88907 ) on Tuesday July 30, 2024 @06:04AM (#64666230)

        As an engineer, gross negligence is very clear hear. Invalid pointer references can be found by automated tools. The situation was "high damage" should a crash happen. Competent risk management would have been aware of that.

        Obviously, what a court will decide is a different matter.

        • by ljw1004 ( 764174 )

          As an engineer, gross negligence is very clear here. Invalid pointer references can be found by automated tools.

          Do you reckon gross negligence by Microsoft is also a plausible case?

          • Re:Liability (Score:5, Informative)

            by Luckyo ( 1726890 ) on Tuesday July 30, 2024 @06:18AM (#64666246)

            I have no idea, and frankly even if I did, it wouldn't be relevant. Courts look at evidence presented to them. Not at what merits of the case are in general.

            If I were arguing the case for Delta, I would most certainly be pushing for gross negligence because of how fucking idiotic the CrowdStrike error was and how grossly negligent it was to push those updates out the way they did is.

            Microsoft would be harder to argue because they diffuse responsibility onto third party vendors, and that's actually one of the main benefits of windows and why it's such a useful OS. Because anyone can make hardware or software for it, and anyone can contract anyone else to make software for it. It's completely open in this regard.

            So Microsoft's legal team would have a very strong argument in my view that they're not negligent at all. They offer a system that by design is open to developers, and you buy it in large part because it's open to some many developers. They can even reference Apple as the opposite kind of an offering, a very locked down system. The negligence is on CrowdStrike for offering a faulty service. Not on Microsoft for providing operating system. I'd likely use legal analogies like road makers being innocent of responsibility for grossly negligent truck maintenance that resulted in a severe accident on said road.

            That said, while I dabble in reading the law, IANAL.

            • by gweihir ( 88907 )

              There is some fault on the side of Microsoft: For code updates in this use-cases, they require that the user can defer or turn off automatic updates. For configuration changes, they did not require that. And the Cloudstrike disaster was a configuration update, automatically pushed to all users with no way to disable or defer it. It could be argued that Microsoft put protections in place, hence clearly saw they were needed, but then did not implement them in a way that was effective. And they failed to do so

              • by Luckyo ( 1726890 )

                The easy defence here is that this is by design. The choice is offered so that 3rd party supplier of software and his client figure out the best way to do it. Microsoft is completely agnostic on the issue, and requirement of competence for third party cannot be on Microsoft, as Microsoft is not a party to transaction between software vendor and client.

                It merely offers OS that third party software runs on that client uses.

                • by gweihir ( 88907 )

                  The question is whether MS takes responsibility for some quality aspects for in-kernel modules. They so far have pretended to do that. Now, they claim they do not and maybe will escape liability with that. But they will also declare their oversight of in-kernel modules as worthless. Or the can take responsibility and say the care about this aspect of Windows security and reliability but then they may become liable.

                  • They have claimed to take responsibility for some quality aspects of third-party code? Do you have a citation for this?
                    • Yes I know Microsoft places conditions and I'm familiar with them. All of those conditions are around ensuring that the authors can be correctly identified. I've never seen one instance where a virtual driver has been required to meet any quality gates for signing. If you want to distribute *hardware* with certain logos, you have to run (and pass) the WHQL tests. However, that's not applicable here. So, please, citation needed. A forceful assertion is not a citation.
                  • by sjames ( 1099 )

                    This is an important point. They either need to actually assure all kernel drivers or they need to document the whole thing and be hands off for 3rd party drivers.

            • So Microsoft's legal team would have a very strong argument in my view that they're not negligent at all.

              They signed Crowdstrikes kernel drivers.

          • by gweihir ( 88907 )

            For Windows as a whole? Clearly.

            That said, MS did mandate that automatic code updates could be turned off or deferred in the given context. They did not mandate that for configuration updates and the Cloudstrike crash was caused by an automatic configuration update that effectively contained code or at least jump-vectors and that could not be turned off or deferred. MS certainly half-assed things here. Looks more like simple negligence to me though.

            • by Bert64 ( 520050 )

              Well MS mandated deferral ability for code updates, and then crowdstrike subverted this system by pushing code as a configuration update. MS may be guilty of a flawed process which has such a loophole, but this is no different to any buggy software.

              There is also no such entity as cloudstrike.

              • by gweihir ( 88907 )

                I beg to differ. A fundamentally flawed process like this one is on a completely different level than a software bug.

          • by AmiMoJo ( 196126 )

            I don't think you could make a good case of Microsoft being negligent. They created the APIs but they came with instructions about the developer needing to be extremely careful with them, because they can brick the system.

            The argument would have to be that merely providing the APIs, which many other anti-virus vendors use without bricking anything, was negligent. That seems far fetched.

            It would open up all sorts of other liability if it was, e.g. car manufacturers that build cars capable of dangerous speeds

            • by Zocalo ( 252965 )

              It would open up all sorts of other liability if it was, e.g. car manufacturers that build cars capable of dangerous speeds.

              And that's going to give the current SCOTUS (which is where this may well end up) pause, why, exactly? Considering secondary implications before making laws or passing legal rulings doesn't exactly seem like it's a forte of the US system at the moment so much as "as long as it pleases *my* base, it's all good!"

              • Re:Liability (Score:4, Interesting)

                by cayenne8 ( 626475 ) on Tuesday July 30, 2024 @10:59AM (#64666890) Homepage Journal

                And that's going to give the current SCOTUS (which is where this may well end up) pause, why, exactly? Considering secondary implications before making laws or passing legal rulings doesn't exactly seem like it's a forte of the US system at the moment so much as "as long as it pleases *my* base, it's all good!"

                The SCOTUS is not supposed to rule based on "secondary implications" if a law is constitutional or not...it is only supposed to be if the law is constitutionally valid.

                A bad law, may have some positive results....but if at its base it is unconstitutional, it needs to be revoked. If you want the positive things the bad law provided..then make new legislation to instantiate it.

                This is nothing to do with "the base"...the SCOTUS rulings have been more in line with strict constitutionalism...and that is what they're supposed to do. They aren't there to determine if a law is "good" or beneficial...only if it is Constitutional.

                I'm guessing you're going off about the overturning of Roe vs Wade. I hated it got overturned too, BUT....the grounds for it being law in the first place was on VERY shaky ground...even Ruth Bader Ginsberg, one of the most liberal ever of SCOTUS judges, said the same thing and that it likely could and should be overturned on a purely legal basis.

                The results of that being overturned do suck, BUT, the legal justifications we sound.

                Most things belong with the states and that's where abortion is now.

                If you live somewhere where you don't like the laws, get active, vote...make your voice heard.

                IF the majority of people in your state disagree with you, the great thing about the US is, you can move and live in a state where more people feel like and want to live like you do.

                One size almost always does not fit in a country as large and diverse (in people and land) as the US, and therefore most things should NOT be determined on a national basis.

                • > you can move and live in a state where more people
                  > feel like and want to live like you do.

                  That's fine for things like zoning rules, last-call for bars and nightclubs, and which parks should be off-leash or not for your dog. But it's not okay for civil rights, non-discrimination, and equal access and treatment under the law. Those are meant to be universal and inalienable.

                  • thank you for this cogent and reasonable rebuttal. I would add to your list, pollution control as the air knows no state as it's only state and the water, in many cases, tends to flow through multiple states. Things which are more than just in one state, need federal oversight. In addition, I would claim that states making travel to other states illegal should also be regulated by the feds. I mean, don't we have the right to interstate commerce enshrined in the bill of rights?

                    A quick read of: https://www. [cornell.edu]

                  • hat's fine for things like zoning rules, last-call for bars and nightclubs, and which parks should be off-leash or not for your dog. But it's not okay for civil rights, non-discrimination, and equal access and treatment under the law.

                    And...so far, I do not see various states abusing or denying any of what you just stated....?

            • by sjames ( 1099 )

              Agreed. MS isn't the nanny. We don't want MS to be the nanny. Part of not having a nanny is that you need to make your own efforts to not do harmful things. This debacle falls squarely on CrowdStrike's shoulders.

          • Microsoft doesn't allow kernel access without a signed certificate from them saying the code is safe to execute. Crowdstrike's software had obvious portions that allowed them to point to a reference file to gather code to inject into the kernel without having to go through the extensive testing to get it approved before the update. Why Crowdstrike would do that is obvious; it allows them to respond to zero-day attacks in hours instead of weeks. Why Microsoft would allow that, in fact certify code that did t

        • Re:Liability (Score:5, Interesting)

          by Luckyo ( 1726890 ) on Tuesday July 30, 2024 @07:00AM (#64666322)

          The problem is that what is clear from perspective of an engineer is not from the point of law. Especially when intermixed with contract law and contracts.

          Legal representation of Delta is one of the lawyers that like to go to court, rather than settle judging by his history of high profile cases that were decided in court. So we may actually get to see and hear a lot of details on this one, rather than everything getting kept out of public's eye because it got settled.

          Time will tell how this goes. Should be an interesting case to follow, as it may set some precedent for things like what is reasonable level of expected support and responsibility for software failures that bring larger systems down.

        • by 50000BTU_barbecue ( 588132 ) on Tuesday July 30, 2024 @07:34AM (#64666418) Journal

          "As an engineer, gross negligence is very clear hear."
          I sea what you did they're. Yup, your a engineer.

        • Invalid pointer references can (and are) found by automated tools on a regular basis. Although I work for a company that produces one of the most popular of the tools (as an expert in the inner workings of the product), I'm speaking on behalf of myself here and certainly not making any representation on behalf of my employer.

          There is no tool that can find *all* invalid pointer references and I'm not aware of any that even make such a claim. We certainly wouldn't even allude to such a thing. If somebo

    • by gweihir ( 88907 )

      Typical BCM/DR does not require you to be able to replace an OS or a major component (like Cloudstrike) at this time. The assumption is that you can essentially recover the former state to make it work again. With ransomware, that already works only with the condition that you find and remove the attack vector. Things are probably about to change more and requirements like being able to move cloud or out of the cloud, change OS and change major security and other components may become the standard.

      • Re:Liability (Score:5, Insightful)

        by aaarrrgggh ( 9205 ) on Tuesday July 30, 2024 @06:14AM (#64666242)

        Specific to the airline industry, they need to be able to get planes and crews to the right place within about 12 hours of an incident. The systems that do that should be completely independent of primary systems and dependencies. There are a few real challenges with the aircraft records system, but the other parts are easy.

        I know it is impossible today to revert to a paper system, but 20 years ago large banks had full (and tested) plans to deal with complete systems outages. This included having all software needed and plans for re-imaging desktops to a known state. Today we are doing things more efficiently... at a tremendous cost of reliability and recovery.

    • Re:Liability (Score:5, Informative)

      by AmiMoJo ( 196126 ) on Tuesday July 30, 2024 @06:02AM (#64666228) Homepage Journal

      This lawyer seems to think that Crowdstrike's negligence opens it up to a lot of liability: https://www.youtube.com/watch?... [youtube.com]

      IANAL but it seems like Crowdstrike would have a hard time arguing that Delta's lack of business continuity planning offered much mitigation. Consider a case where someone is injured in a car crash that was the other driver's fault due to negligence. It would be hard to argue that the victim would have sustained less severe injuries if only they have bought a safer car, or been driving substantially below the speed limit. It would require negligence on the part of the victim, such as not wearing a seatbelt, for it to be mitigating.

      In the video Mr. French explains how negligence works, and the limits of it. Passengers might be too remote from the negligence to claim, but it's an open question. For Delta, the question would be if Crowdstrike's actions amounted to negligence by being so out of the ordinary, which it seems pretty clear that they were. This sort of thing doesn't happen often, and seems to have been the kind of thing that proper procedure and testing would have caught.

      And again, it will be difficult to argue that Delta didn't have adequate protections in place for something this unusual, this negligent.

      • From my experience (not as a lawyer), even gross negligence is going to have a limit in terms of multiple of direct recovery cost or a multiple of fees paid. I doubt CrowdStrike will have a liability total of over $200 million for the mess they created.

        • If you are guilty of gross negligence you pay for the damage. Independent of the contract. Even for plain negligence, you pay for the cost. If you drive your car and hit me, and itâ(TM)s your fault through negligence, you pay my damages (usually indirectly through your insurance). The difference with gross negligence is that the insurance company might want its money back from you, and you might get prosecuted.
          • or they could simply argue - Delta failed to have even the basic good practice industry systems in place to test patch compatability. After all if you opush a patch out to all your 'production' machines without testing simply to cut costs. Well thats a decision that falls squarely on the business.
            • Delta (and everyone else) didn't have control over the CrowdStrike update that broke everything. CrowdStrike can push the configuration updates at will. It's CrowdStrike that pushed the update to everyone's production machines without testing it.
            • CrowdStrike is on control of when updates go out. They do not have any facility for holding such changes back. So, no, Delta was in no position to perform a managed rollout of this patch; only CrowdStrike was in that position.

            • Delta didn't have a choice in the matter. People who opted out of immediate updates got the update anyway.

          • Hmm, so if I can 'prove' Delta wouldn't have had to cancel my flight if they'd had more backup resources, I wouldn't have missed my business meeting that afternoon and would have closed a big deal. So Delta owes me for all the business losses I incurred from them cancelling my flight.
      • by gweihir ( 88907 )

        There is a second question: In order to be adequately prepared for what happened, Delta would have have to be able to replace Cloudstrike on short notice. Yes, there was an emergency procedure that did not require that, but it was in no way assured that procedure would become available so their preparation would have to assume it might not be.

        So, the question really is, does Delta have to be prepared to replace OS, cloud and major components (like Cloudstrike) on short notice (a few hours) in order to not b

    • Absolutely not. Unless crowdstrike has a contract where delta agreed to have a business contingency plan that survives total shutdown of their computers. Which I think is unlikely. If Delta owned say 10,000 Dell computers, itâ(TM)s reasonable that they need to survive 100 of them going down and Dell only pays for fixing the computers. But not if all 10,000 go down.
    • I agree that disaster recovery and backup systems should be an integral part of any business. However, apparently in this case the auto-updates were pretty much out of the control of the individual businesses. Since it was "just" a data file that caused the problem, it was not subject to the usual scrutiny.

      IMHO, the fault lies 60% with CrowdStrike, for not having software that checks data validity before using it, 30% with Microsoft, for not moving this API out of the kernel, and only maybe 10% with the i

      • Re: Liability (Score:5, Insightful)

        by reanjr ( 588767 ) on Tuesday July 30, 2024 @07:26AM (#64666398) Homepage

        This is 90% CrowdStrike's fault for allowing a configuration file to lead to a null dereference in a kernel driver.

        And 10% Delta's fault for choosing CrowdStrike knowing they install a kernel driver and don't allow you to manage rollouts.

        MS can't fix stupid.

        • MS can't fix stupid.

          MS intentionally designed this stupid in. They allowed 3rd party kernel drivers into the system. 3rd party drivers, according to Microsoft's own metrics, are responsible for the vast majority of Windows crashes.

          • by LostOne ( 51301 )

            They were also *forbidden* to do a better solution by EU regulators.

            • They were also *forbidden* to do a better solution by EU regulators.

              No, they weren't. That a convenient lie MS likes to tell. MS was forbidden from using secret APIs that gave their product an unfair advantage. Rather than put the work into creating the necessary APIs to allow for user-level code to hook into the kernel, they simply opened the kernel.

              Between Windows, Linux and macOS, macOS is by far the most locked down of any kernel, even to the point of running on its own sealed, read-only partition. Yet the CrowdStrike system runs just fine on macOS, even as a user-level

    • While CrowdStrike (and maybe Microsoft) really screwed up, Delta will likely only be able to recover the direct IT costs of the BSOD restoration effort. The magnitude of business losses are squarely on Delta for having grossly inadequate business continuity plans and systems.

      Business continuity discussions are going to get very interesting, especially when they realize their options on the desktop are between Microsoft, Microsoft, and Microsoft. (It’s a bitch when all of your OS eggs are carried around in a basket by a short ugly guy mumbling about The Precious.)

      Oh, and good luck with the civil suits after prying MS Excel from the lifeless hands of thousands of dead accountant junkies. There is no alternative for that spreadsheet heroin.

  • by thegarbz ( 1787294 ) on Tuesday July 30, 2024 @05:26AM (#64666174)

    If they prove negligence (which will be quite a battle, but necessary in order to invalidate the liability limit in the EULA) then they can claim for reasonable damages. Unfortunately for Delta there is literally every other airline to look at as evidence that the huge number of cancellations and the incredible cost to Delta was of their own incompetence. Most airlines were up and running by the afternoon and even major airlines limited the damages to double digit millions - a cost of which you can expect to be eaten up in lawyer fees.

    This won't work out the way Delta thinks.

    • by bradley13 ( 1118935 ) on Tuesday July 30, 2024 @07:14AM (#64666348) Homepage

      If Delta were the only company affected, you would be right. However, most other airlines (and many other businesses) were, in fact, severly impacted.That points the finger pretty squarely and CrowdStrike, and to a lesser extent to Microsoft.

      The initial reactions on /. were that this could be a business-ending event. I submit that it should be a business-ending event, and one that should also see top management subject to personal liability.

      Most blame goes to CrowdStrike for having code that doesn't bother to validate data before using it. However, I am exceptionally irritated with Microsoft for trying to blame the EU. The EU only ever said that they had to provide the same access to all anti-malware programs (no secret API access for their own product). The EU never said that they had to allow API access within the kernel. That was their own stupid decision, and they should carry some of the blame, and liability, as a result.

      • So, MS was forced to either accept CrowdStrike or gut their own Windows Defender. That means MS did the least worse thing, by allowing companies to shoot themselves in the foot by choosing a third party kernel driver instead of only trusting that sort of thing to MS.

        As an IT manager, I would immediately balk at installing ANY third party kernel driver. I would require a full vetting, which would have informed me that CrowdStrike uses config updates to change the code at the binary level and that they do not

    • If they prove negligence (which will be quite a battle

      LOL, no. Negligence is easy to prove in this scenario. Negligence from Crowdstrike AND Microsoft is absurdly easy to prove; however, Delta is the victim of the EXACT same negligence as Microsoft, so it is unlikely that Delta will get a penny from Microsoft.

      Crowdstrike will likely not be found liable either, since that would erase the company in its entirety... and there are too many people getting money from that to allow it to die off in legal proceedings.

      So, ultimately, it would appear that we predict the

  • by Canberra1 ( 3475749 ) on Tuesday July 30, 2024 @05:41AM (#64666194)
    No liability! . Practically all software comes with a blanket disclaimer of no liability and not for any consequential damages either. Often there is arbitration snuck in there - but is comes down to all care no responsibility. In thirty years, I don't think anyone has won any real money, except for those wishing to contrive a tax free transfer. In addition Delta and their executives should have had in place, a DR plan. Looks like they were the worst of the lot for DR negligence - where was the warm or hot site to cutover too? When did the evaluate the reliability of any of their third party software suppliers? Pretty sure Airlines have National Security considerations to uphold, and looks like they failed here too. Was not some FTP software at the root problem, no so long ago? The skinny is other Airlines managed, and probably quite a few banks too. I hope everyone who missed their flight gets a bonus $10 coffee voucher too.
    • Damages depend on the agreement and negligence. Product liability helps if someone was injured but since software for an administration PC seems unlikely angle to win a claim. Lawyers will fight and maybe extort a modest settlement to save time and effort. But risk opening floodgates to others making similar claims overrunning CS with legal challenges. Wake up call on resilience system investment. Expensive but that is trade off otherwise deal with downtime. Cellular outages occur periodically but the telc
    • Just because you put it into a contract doesn't mean it holds up in court. Especially for gross negligence.
    • Practically all software comes with a blanket disclaimer of no liability and not for any consequential damages either.
      That is complete nonsense.

  • by ISayWeOnlyToBePolite ( 721679 ) on Tuesday July 30, 2024 @05:46AM (#64666198)

    For longtime slashdot readers David Boies is probably most familiar for representing SCO in the infamous SCO vs IBM lawsuit https://en.wikipedia.org/wiki/... [wikipedia.org]

    • For longtime slashdot readers David Boies is probably most familiar for representing SCO in the infamous SCO vs IBM lawsuit https://en.wikipedia.org/wiki/... [wikipedia.org]

      So the guy who had his arse handed to him on several platters, then? Got it. Best of luck, Delta!

    • For longtime slashdot readers David Boies is probably most familiar for representing SCO in the infamous SCO vs IBM lawsuit

      Actually, he is also well known for representing the Justice Department against Microsoft (which he won).

  • Gift card (Score:5, Funny)

    by Roadmaster ( 96317 ) on Tuesday July 30, 2024 @06:14AM (#64666244) Homepage Journal

    What, do you mean the generous ten-dollar Uber eats gift card did not cover their damages and losses? Unthinkable.

  • I'm sorry to break it to them, but in no way Microsoft can be held responsible for this SNAFU.
    • by Zocalo ( 252965 )
      They'd already got their statement in on that point: "We're required to give low level kernel access to third party developers because of the EU" or words to that effect. It's not a huge leap to go from there to "which prevents us from providing and mandating use of an API that allows managed access to the Kernel without allowing companies like Crowdstrike to brick the OS."

      That was put out while investigations were still ongoing too - it's almost as if they had a lawyer on speed dial who told them what
  • Wow, 5% stock price drop? That's almost as high as the 34% that it ACTUALLY dropped.
  • This seems like the ideal application of a class-action lawsuit? There are just SO MANY litigants, and the courts would prefer to just hold one case rather than thousands.

  • by SoSueMe ( 263478 ) on Tuesday July 30, 2024 @07:30AM (#64666410)

    I find it amusing an airline would even consider the possibility of reimbursement for a "cancellation" that wasn't their fault.
    I hope they get nothing.

  • Stop all this forced updating - here you see why it can be a bad thing.

    Give ME the control over my goddamned system. When I need an update, I'll do it myself.

    • The "forced updating" is not the culprit here. I'm actually in favor of "forced updates" if they are tested and rolled out properly. Without forced updates, many machines, probably a majority, would gradually "rot" becoming more and more insecure over time.

      The issue here, is that there was no progressive (green blue) deployment, mitigating the risk of a large scale "all at once" rollout.

  • by laughingskeptic ( 1004414 ) on Tuesday July 30, 2024 @09:21AM (#64666680)
    Delta used Crowdstrike's default configuration which takes the latest software updates for their production servers. I know of other Crowdstrike customers that had this set at N-1 and only applied the latest to "Test" servers like one would any other production software release. Those customers did not experience any outages other than on their test deploys. This fundamentally comes down to a risk analysis failure on Delta's IT department. They had everything, including fallback systems configured to take the latest update and so everything failed at once.

    Maybe Crowdstrike's defaults are not the best for production deployment, but an organization as large as Delta should have at least one person who understands what they are deploying and what the configuration options mean.
    • by Thud457 ( 234763 )
      The triggering event was a botched content update, not a FUBAR'ed code update.
      The release tiering only applies to code updates.

      We're 10 days into this, how can you have your facts so wrong?
      • I am not a CrowdStrike customer, but the IT guy I talked to at a kids birthday party on Saturday told me that they avoided this issue using an N-1 policy. So that must be the content updates too. If CrowdStrike does not support this, then some customers must be controlling access to updates some other way, such as by limiting network access to the update server until the test server is given a clean bill.
    • there is a reason to want the latest updates on systems to protect versus the just discovered 0-days, but the N-1 on production is same thing we do for production, N-0 on test to ensure no issues and then a rollout to production after about 3 weeks which should be enough time to shake out any issues... but to have a homogenous config across your production and failover systems is ignoring the threat of exactly this- either a single sec vulnerability will bring down the entire system when exploited, or a bro

  • As someone who works in IT and provides consulting to corporations with in-house IT staff, the overwhelming number of IT people are incompetent and I believe "fake it till you make it" is how so many people get to where they are. With that being said their IT staff should have resolved this problem a lot sooner and if they were competent, wouldn't have experienced the issue in the first place. There's a reason why our clients who use Crowdstrike were not affected. Also, this has nothing to do with Microsoft
  • how about switching all systems to Linux, and dump Crowdstrike, and learn to set up iptables and anything else you add on keep it out of kernel space and only allow it in userspace so when and if it crashes it goes down without taking the entire operating system down,
  • Short outage costs them $350-500 Million? yet they won't invest in building redundancy and resiliency into their systems that are apparently this critical to their operations. Their entire board needs to be replaced with one that knows how to operate a business in 2024. Delta had an opportunity to be the only carrier operating when everyone else collapsed and it would have required a little thought into architecting their IT system to be rolled back to pre-update state. Basic business continuity prep, te

  • Isn't Boies like 100 years old these days?

  • I hope that this case goes to court and that the evidence presented will be available to us to review and for the rest of the public to analyze what the hell exactly happened .

    We know the technical details of the problem, but we don't know the communications and the internal emails and messages that were exchanged as part of this outage and the planning parts of their build system that allowed a completely empty updates to be pushed out to so many devices. I would love to see the emails that get subpoenaed

  • Delta Air Lines has hired prominent attorney David Boies to seek damages from CrowdStrike and Microsoft following an outage this month that caused millions of computers to crash
  • Who did not see this coming?
    I think we will see more of these lawsuits.

"It is easier to fight for principles than to live up to them." -- Alfred Adler

Working...