Slashdot Log In
Amazon Explains Why S3 Went Down
Posted by
timothy
on Sat Jul 26, 2008 04:33 PM
from the not-mere-sluttiness dept.
from the not-mere-sluttiness dept.
Angostura writes "Amazon has provided a decent write-up of the problems that caused its S3 storage service to fail for around 8 hours last Sunday. It providers a timeline of events, the immediate action take to fix it (they pulled the big red switch) and what the company is doing to prevent re-occurrence.
In summary: A random bit got flipped in one of the server state messages that the S3 machines continuously pass back and forth. There was no checksum on these messages, and the erroneous information was propagated across the cloud, causing so much inter-server chatter that no customer work got done."
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
for want of a nail ... (Score:5, Interesting)
Re:for want of a nail ... (Score:5, Funny)
It was the evil bit...
Parent
haha (Score:5, Informative)
Parent
Re: (Score:3, Informative)
Re: (Score:2)
It's like a self-replicating virus that arose from the result of a random mutation.
"Ever since the first computers,
there have always been
ghosts in the machine.
Random segments of code that
have grouped together to
form unexpected protocols."
Re: (Score:3, Funny)
1 million code monkeys typing out Aleister Crowley?
Re:for want of a nail ... (Score:5, Informative)
Parent
Re:for want of a nail ... (Score:4, Interesting)
Parent
ECC memory, anyone? (Score:4, Interesting)
I hafta wonder if the bit flipped due to a bad RAM stick?
Nothing specific about *what* caused the bit to flip.
This comes to mind only because bad RAM on a new server at work caused installation of a stock Perl module to throw excessive errors during the XS compile phase - the same package installed without error on an identical machine 20 minutes earlier. Took over an hour before we realized it was probably hardware. Memtest86 [memtest86.com] quickly turned up the problem.
Would hashes and the like protect against RAM suddenly going south? Wouldn't any piece of data that passes through main memory be vulnerable to corruption? Makes me wonder why ECC memory [wikipedia.org] isn't being used much anymore... we have various flavors of RAID to protect slow memory from corruption, but not many machines I see have ECC anymore.
Parent
Re:ECC memory, anyone? (Score:4, Informative)
Parent
Re:ECC memory, anyone? (Score:5, Informative)
No, that's just one kind of memory system. There are a number of designs, and recovery also depends on the kind of error. IIRC, one design is somewhat similar to the CD Red Book spec, in that the bits for a given byte are distributed around - a physical byte is composed of bits all from different memory locations. If part or all of one byte goes bad, the rest of the bits and the parity code are unchanged, and the affected bytes can be reconstructed.
Also like Red Book CDs are multiply redundant memory systems, with -just what it sounds like- multiple copies of each byte, and the memory controller arbitrates differences. CDs effectively contain three copies of the data, striped and parity encoded. That's how scratched CDs can still operate error-free (sometimes.) The space shuttle's computer systems are relatively fault-tolerant - multiple redundant computers all running the same programs and data, with a fourth computer evaluating the output of the other computers, looking for failures.
Where there's a will, there's a way, but the will in the mainstream x86 server industry to build truly fault-tolerant computers is slim. It's a specialty, and that makes it very expensive. Stratus [stratus.com], for example, makes a line of fault-tolerant servers [stratus.com], with some of the fail-over in hardware, so they make their 99.999% uptime claim (about 5 minutes downtime per year.)
"Five nines [wikipedia.org]" is a claim I've heard from most top-dollar *nix hosting companies, but have *never* experienced - it's generally been hours of downtime per year. Not even their network infrastructure gets close to 99.999% uptime! Cadillac prices, but downtime contingency planning is all up to the client, even with "managed hosting." They all suck.
Parent
Re: (Score:3, Informative)
Red Book audio CDs, Sony MiniDisks and DATs all use a form of Cross-Interleaved Reed-Solomon [wikipedia.org] coding, which is has the nice characteristic of being able to use the fact that a piece of information i
Re: (Score:3, Informative)
Re:for want of a nail ... (Score:5, Funny)
"On Sunday, we saw a large number of servers that were spending almost all of their time gossiping and a disproportionate amount of servers that had failed while gossiping. With a large number of servers gossiping and failing while gossiping, Amazon S3 wasn't able to successfully process many customer requests."
sounds like a restaurant, gossiping servers were failing to process customer requests
Parent
Re: (Score:2)
A flag can be stored as a bit, but not all bits are used as flags..
Simple (Score:3, Insightful)
S3 is a total slut.
Re: (Score:2)
Aw. And it appears that this has been posted from the not-mere-sluttiness dept. It appears that slashdot editors already stooped to the lowest from of humor in posting the headline, and I've just fallen into their redundancy trap. Tricky, tricky!
...make lemonade. (Score:4, Funny)
Cosmic Rays perhaps? I guess they could line the room with lead, or simply re-market S3 as a Neutrino detector [wikipedia.org]. :-)
Re: (Score:3, Insightful)
Or they could checksum their UDP packets. The entire packet, not just the customer payload. Duh.
Re: (Score:3, Interesting)
There's probably information that changes as the packets move around, and they probably wanted to avoid the overhead. I'm guessing it was a deliberate design decision, but it turned out to be the wrong one. It's easy to see that after a failure, but it's hard to design large distributed systems and foresee every possible way things can break, and where the computation overhead is worth it. The number of interactions between servers here makes any small design flaw a big thing.
Re:...make lemonade. (Score:5, Insightful)
No, I mean favoring speed and computational simplicity over error detection.
It is often a valid trade off. For example, most filesystems do not validate the stored data at all for size and computational reasons. As hard drives and arrays get bigger, that trade of no longer makes much sense, and most all new filesystems being designed have hash based error detection built in at some level.
Good design takes experience. There aren't that many systems like S3 that have been built in the past, and there are many tricky decisions to be made. No system gets it all correct out of the gate.
Parent
Re: (Score:2)
It is quite simple to prove mathematically, that it is in general impossible to devise a protocol that guarantees 100 percent correctness of a signal on an independable line.
(You simply do it by recursive reduction: If the protocol works even if the last bit is missing, then you could send the message without the last bit anyway. If the second last bit can go missing without damaging the integrity of the message, you can leave out that one also... etc.pp. until you don't need to send any bit at all to deliv
Re: (Score:3, Insightful)
True, but not particularly informative. The point is to detect errors. Error correction is nice, but error detection is enough if the sender can then retransmit. Oh, and your 'proof' is flawed, since you are completely ignoring the fact that any correction scheme contains redundant information, so the while n bits might work instead of n+1 bits, n-1 might not.
If the last bit is missing, then your receiver knows that there is an error. If the last bit is flipped, then it knows that there is an error. A
Re:...make lemonade. (Score:5, Interesting)
I see you completely miss the point of the proof. I know that you can minimize the impact of a bit error by checksums, and that you can improve reliability by adding redundance. But what is the consequence of error detection? Normally the protocol then asks for resending the message. But how do you (as the sender) know that the message finally arrived correctly? You wait for an aknowledgement. But what if the aknowledgement gets lost or scrambled? You add redundancy in the handshaking. But how does redundancy help reliability? You can ask for a resend due to detected errors etc.pp.
Your protocol never finds an end because it has to secure the correctness of the security of the correctness of the security...
Parent
Lost time? (Score:2)
By 11:05am PDT, ..., the system's state cleared.
At 2:57pm PDT, Amazon S3's EU location began successfully completing customer requests.
So WTF happened during the four hours between 11:05AM and 2:57PM?
And... have they learned nothing from the TIBCO fiasco?
Re:Lost time? (Score:5, Informative)
Read "the system's state cleared" as "we turned everything off" and they proceeded to turn every server on one by one until around 3PM when the EU location was complete and not showing any symptoms.
Parent
Re: (Score:3, Informative)
sun enterprise and most other enterprise servers takes 30-45 minutes to pass the prom tests after you start them up. thats per server, before you even boot the os. my guess is they brought up servers in large batches of 25% at a time and took about an hour per batch.
Other companies could learn from this... (Score:5, Insightful)
Other large businesses could learn a lot from Amazon's example.
How often do you have the problem really explained to you, an apology, and a reasonable set of changes to stop it occurring again?
Most businesses would never explain the root of any problem. They simply list "hardware issues." And they NEVER say sorry anymore - supposedly it opens them up to more liability or something.
If I was an Amazon customer I would be happy with their explanation and apology even if obviously the downtime is still an issue.
Re: (Score:2, Insightful)
Looking back, I feel that the one thing all our technological progress has given us more than anything else, is more and better means of communication.
You can talk to people on the other side of the world, heck, on the other side of the solar system if you don't mind the delay. Video feeds, planes that'll actually get you there in less than a day, ...
And yet, with all of this, it seems that we're not actually doing it. A company explaining what went wrong is the exception. Internet forums without flamers an
Re:Other companies could learn from this... (Score:5, Funny)
Other companies could learn something from this, unfortunately they won't be able to do anything similar as Amazon has patented the process of explaining technological problems to customers.
Parent
It's quite an old story - see RFC789 (Score:5, Interesting)
Vulnerabilities of Network Control Protocols: An Example [ietf.org], published in January 1981.
What do they say about those who ignore history?
Parent
Re:It's quite an old story - see RFC789 (Score:5, Funny)
[...]
What do they say about those who ignore history?
I think it was, they're doomed to reimplement it... poorly. Or was that Unix? ;)
Parent
Re: (Score:3, Insightful)
Well, technically speaking, there isn't an apology there:
Finally, we want you to know that we are passionate about providing the best storage service at the best price so that you can spend more time thinking about your business rather than having to focus on building scalable, reliable infrastructure. Though we're proud of our operational performance in operating Amazon S3 for almost 2.5 years, we know that any downtime is unacceptable and we won't be satisfied until performance is statistically indistinguishable from perfect.
Allow me to translate:
We screwed up. We'll do better next time.
Nowhere in the document do the words "I'm sorry" appear. That's entirely implied.
Re:Other companies could learn from this... (Score:5, Insightful)
The words may not be there, but there is a pretty clear message there for me to see that they are not happy or smug about this event, and are agreeing with the consumer that this shouldn't have happened, and won't happen again if they can help it. That's enough for me.
And I'm actually one of their consumers, compared to some of the dilletantes here. We use S3 and EC2 to manage training and demo instances of our software, and are pretty pleased so far.
Parent
Re: (Score:2)
Um, what does package delivery speed have to do with their hosting service?
Re: (Score:2)
It was a design defect (Score:5, Informative)
It has been generally well-known for a number of years now that any time you have a large cluster you cannot count on hardware checksums to catch every bit flip that may occur during copies and transmission, particularly with consumer hardware which has many internal paths with no checksums at all. Google learned this the hard way, like the supercomputing people before them, and now like Amazon after Google. And some of the better database engines also do their own internal software checksums as well to catch uncaught errors introduced as the data gets copied across the silicon, disks, and network -- it is one way they get their very high uptime and low failure rate.
It does not reflect well on the software community that most people *still* do not know to do this for very large scale system designs. The performance cost of doing a software CRC on your data every time it is moved around is low enough that it is generally worth it these days. If your system is large enough, the probability of getting bitten by this approaches unity. Very fast implementations of Adler-32 and other high-performance checksum algorithms are widely available online.
Re:It was a design defect (Score:5, Interesting)
Adler-32 wouldn't be a great choice. It's fast but it's weak for short messages [ietf.org] and I've seen it fooled by multi-bit errors on large messages too.
See Koopman's paper 32-bit cyclic redundancy codes for Internet applications [ieee.org] for some better ideas.
Parent
Re: (Score:2)
For these purposes, a CRC that is fast and weak is generally superior to one that is slow and strong because the CPU load of the implementation does have a measurable impact on system performance. Remember, the software CRC is supposed to catch failures in the other layers of CRC and error detection, so it does not have to be perfect. If it reduces the probability of an uncaught problem from a few times a year to a few times per millenium, that may be sufficient if using a stronger CRC means burning 10% of
Re:It was a design defect (Score:5, Interesting)
I've seen Adler-32 fail twice, so your assumption of "a few times per millennium" doesn't seem to work for me (I'm under 40). In fact in those cases it was even stacked on top of at least one other checksum mechanism, too.
One of the problems with it is poor spreading of the input bits into s2. There are other algorithms which don't have that weakness but don't (IIRC) cost any more to compute.
Parent
Re: (Score:2)
It does not surprise me at all that there are better algorithms out there or that Alder-32 failed, but the failure probability is compound. It only has to work a few times a year in many cases, and Adler-32 *is* used in some respectable systems as the error trap of last resort. Out of curiosity, not being a topical expert, what would be the best CRC algorithm to use on modern silicon? It is also worth noting that some places are starting to use cryptographic hashes instead of CRC for software checksum pu
Always happens on a Sunday (Score:2, Interesting)
Re:Always happens on a Sunday (Score:4, Informative)
After working in a production environment as a developer, I can assure you that the correct interpretation is "Anybody else noticed that these major problems always occur when no one is around to catch them (before they get out of hand)?"
InnerWeb
Parent
Byzantine failure (Score:3, Interesting)
So the whole cloud is in trouble if one node starts spewing nonsense? So much for redundancy. Amazon developers would be well advised to read up on the "Byzantine Generals" problem.
Re:They need more Erlang. (Score:5, Insightful)
They need to start using Erlang more. It's designed specifically for building highly-distributed, concurrent systems that must scale to millions of transactions per minute. So it's a natural fit between Erlang and what Amazon is trying to offer with their S3 service.
I think Erlang's cool and all, but it's not the magical bullet that will solve this. It's still possible to have information corrupted during message passing between processes in Erlang (say, as the result of an intermittently failing network switch) as it is in any language.
Parent
Re:They need more Erlang. (Score:5, Insightful)
Erlang or any other functional language will not account for lack of good design. If you have a good design with the right concerns you can implement in Java, C, Fortran, ASM and if done right, it will work.
I'm sick of hear "Erlang is THE solution". It is not. Good design and implementation practices are.
Parent
Re: (Score:2)
In 99.9% of cases it isn't the protocol that causes a server crash and instead it is the way that the protocol is implemented.
This story is another example of that. Although they're fixing it by changing the protocol spec' that is really just a much cheaper way of resolving the core issue (e.g. That any input shouldn't ever cause a crash - corrupt or otherwise).
Programmers need to learn something but I think the real lesson here is simply that input over the network cannot ever be trusted. You should assume
Re: (Score:2)
Programmers need to learn something but I think the real lesson here is simply that input over the network cannot ever be trusted. You should assume that it is corrupt, untrusted, or wrong.
Well, I have to say I'm guilty. I just assume that all the checksums at the various levels of TCP/IP make sure the data that comes over a socket is the same as what went in.
Of course, if I was working on a critical/must-never-go-down system, I'd maybe be a little more paranoid.
Re:Programmers never learn... (Score:4, Interesting)
I think the real lesson here is simply that input over the network cannot ever be trusted.
It is their network, and S3 is built on tech which is explicitly designed to not have any kind of security built-in. The security is applied at the API level, but any misbehaving machine within the S3 cluster could cause some serious damage.
I actually agree with this philosophy, to an extent. After all, this is essentially a large number of computers acting as a hard disk. How would you approach talking to a hard disk in your own machine? Do you assume that everything is corrupt, untrusted, or wrong?
Parent
Re: (Score:2)
No. Even if the end user is not malicious or stupid, the network itself will not always transmit accurately. Interference or signal degradation on the lines, or a dodgy switch can screw things up.