Amazon Says Hardware, Not Hackers, Caused Outage 128
ChiefMonkeyGrinder writes "Amazon has denied reports that its European sites were brought down by a DoS attack by a hacker group sympathetic to Wikileaks cause. The retailer was a planned target for the attackers, called Anonymous. But Amazon said it was a hardware failure in its European data centre network that caused the half hour outage in Europe."
Lakes vs an Ocean (Score:1)
I doubt a bunch of piddly home connections can take on a backbone anyways. .
I don't know (Score:3)
Re:I don't know (Score:4, Informative)
And Anonymous themselves said they weren't going to attack Amazon.
Re:I don't know (Score:5, Funny)
Anonymous themselves said they weren't going to... (Score:2)
Is the group really that organized that you if someone says "we're not going to ', that (nearly) everyone will follow?
Re: (Score:1)
Try again, this time in something resembling English. ;^) I really can't figure out what you're trying to say.
Re: (Score:3)
Re: (Score:1)
So do I, from their Black Friday debacle in the UK which was essentially a lottery rather than a sale because you got an HTTP 503 Service Unavailable response to attempts to purchase the offers 90% of the time as they clearly couldn't handle the load, through to the fact that it's currently taking them around 5 days to deliver orders which they're telling you on ordering are GUARANTEED to be delivered in 1 day via Amazon Prime I'd have to agree that Amazon have become quite excellent at fucking up their own
Re: (Score:2)
I can't comment on the 503s except to say that I didn't see one, but Amazon UK have yet to hit any of their estimated delivery dates for our household's orders over the past month or so, and the delay is now measured in weeks in some cases. They did have a big banner up on their web site during the snow saying that some deliveries might be delayed by a couple of days, but they are way past that with us.
Re: (Score:2)
You wont have seen the 503s because they were being returned in response to the AJAX request issued by clicking the add to basket button on the Black Friday offers. If you tried to by any of the items and found yourself stuck indefinitely in a queue, or at least for a few minutes or more then it was because Amazon's Javascript wasn't setup to handle 503 responses from the server, and so just hung there with the queue animation, even though the reality is your request was never even processed.
I looked into i
Re: (Score:2)
I can't comment on the 503s except to say that I didn't see one, but Amazon UK have yet to hit any of their estimated delivery dates for our household's orders over the past month or so
Not sure what's going on about deliveries, but there's been enough people in the office complaining about having not received deliveries from Amazon, and it's not like we've had any snow here either since the week before last, it was all melted off by last Monday so if they are overdue there doesn't really seem to be an excuse now.
Define "here". Parts of Scotland got a lot more snow in the middle of last week (whereas my part of Scotland- which got a lot the previous week- was unaffected despite being not that far away, so don't assume that your local area reflects the whole of the UK). It's possible that any Amazon operations based in Scotland may be affected. (I don't know if they store the same item at multiple warehouses and choose the most efficient one to dispatch from according to the customer's location- but if so I'd guess t
Re: (Score:2)
In my case, "here" means Cambridge, UK. There was a bit of snow here and rather more in places like Essex and Kent, but it was all cleared several days ago, and unlike those of you further north, we really weren't disrupted for that long. I can understand the various retailers and delivery services who are tightening up on what commitments they will give in Scotland, but I don't see any excuse for having a week or more of backlog in delivery services in this region.
Re: (Score:2)
I don't see any excuse for having a week or more of backlog in delivery services in this region.
Well, I was trying to say that if something is being dispatched from a snowed-in warehouse based in Scotland, your delivery is still going to be delayed, regardless of whether the home counties are swanning around in a heatwave, knocking back Pimms and G&Ts or not :-)
As I said, I don't know where Amazon orders are generally dispatched from- I strongly suspect that popular items will be held at all their UK warehouses, but am less sure about the obscure stuff.
Re: (Score:2)
Well, I was trying to say that if something is being dispatched from a snowed-in warehouse based in Scotland, your delivery is still going to be delayed
Sure, I understood what you meant. Given that Amazon UK tend to sell you things like DVDs using their "preferred Jersey merchant" these days (for tax reasons, presumably), I wouldn't expect a warehouse much further north to factor into the equation, but I suppose with business logistics these days you can't assume too much!
Re: (Score:2)
Sure, I understood what you meant. Given that Amazon UK tend to sell you things like DVDs using their "preferred Jersey merchant" these days (for tax reasons, presumably), I wouldn't expect a warehouse much further north to factor into the equation, but I suppose with business logistics these days you can't assume too much!
They definitely do have at least one warehouse in Scotland- Glenrothes in Fife- but as you imply, whether that would be used for your delivery is unclear.
:-/
I guess deliveries from the "preferred Jersey merchant" are less likely to be affected by snow. But more likely to be affected by Nazi occupation of the territory...
FWIW, the mysteriously-named "Jersey merchant" is actually Indigo Starfish, which I've heard some not entirely complimentary things about (mainly due to bad packaging on DVDs and the li
Re: (Score:2)
West and North Yorkshire in my office. I asked around this morning where deliveries where the tracking data on tracked parcels people hadn't received were from and a few people said Kettering, Northants. and one said Milton Keynes, others didn't have any tracking details so it doesn't seem in at least some of these cases that there really is much of an excuse. We did have bad snow around here for a few days, but it was maybe 3 days out of a week the week before last.
I can certainly see why people who went f
Re: (Score:3)
Amazon is currently hosting wikileaks [wlcentral.org] for a Danish media group, just not for wikileaks themselves.
I've a quick question for all the network gurus out there. The JS LOIC [calgarc.com] stops sending packets once the currently open connections without responses reaches a preset limit, which happens to be 1000.
Is that really how one should approach a DDoS? Would doing otherwise be a SYN flood which kernels now block fairly effectively? Or are they perhaps doing this to make it more like a virtual sit-in, i.e. all the compu
possible cause of failure (Score:5, Insightful)
Re: (Score:2)
Re: (Score:1)
Self-esteem meme for hackers?
My packets were big enough, my packet count was enough, it fried the hardware.
Re: (Score:2)
Load from the DDoS traffic fried the switch. So you see, it's a hardware failure.
Is that even possible? +5 Insightful??
Re: (Score:3)
It is why we air condition data centers.
Re: (Score:2)
Heat.
It is why we air condition data centers.
Uh huh. And these digital routers just keep revving up more and more as data comes through until they catch fire?
Re: (Score:2)
Heat.
It is why we air condition data centers.
Uh huh. And these digital routers just keep revving up more and more as data comes through until they catch fire?
I was just wondering why my processor never overheats, and now I know it's because it's digital.
Re: (Score:2)
I was just wondering why my processor never overheats, and now I know it's because it's digital.
Good. Now go read more about why things like web-servers don't actually catch fire when they are slashdotted.
Re: (Score:2)
nothing's going to catch fire but it only takes one bad transistor to take down a machine during abnormal load. i had a sup720 go belly-up on me a while back as soon as we started load testing. it happens.
Re: (Score:2)
Okay, so the cause was a defective bit of hardware and not a DoS attack, right?
Re: (Score:2)
i'm not saying it wasn't a DDoS. i'm saying that it could have been defective hardware, and that the fault could have been caused by a DDoS.
Re: (Score:2)
i'm not saying it wasn't a DDoS. i'm saying that it could have been defective hardware, and that the fault could have been caused by a DDoS.
Right, I get what you're saying, but you're wrong about what the 'cause' is. If a transistor blows under a high load, then the transistor is defective. The only reason a DDoS would be to blame for it is if part of the process was that it changed how the hardware behaved (i.e. bypassing a safety mechanism) to cause a burn-out.
You said yourself, you caught it during load testing. You didn't break the hardware.
Re: (Score:2)
yeah, we broke the hardware. fried, blue smoke, nasty smell.
Re: (Score:2)
I mean that you didn't break it, it was defective.
Re: (Score:2, Insightful)
You quite overestimate the hardware. Switches do fail, overheat, etc, it happens all the "F." time.
There's always some which are less tolerant than the others. Especially when it's really big, complicated hardware.
Welcome to the real world.
Re: (Score:1)
You quite overestimate the hardware. Switches do fail, overheat, etc, it happens all the "F." time.
Yes, you are right, but you've lost track of the discussion. Nobody's saying the hardware can't fail (as you say, it happens all the time), but that has nothing to do with a DDoS. If this DDoS came in right after a cooling fan failed (for example), that's nothing more than coincidental timing.
Re: (Score:1)
In the late 80s, Bulgaria tried to produce color monitors. Those could be easily destroyed by making the video card cycle through refresh rates. And no, it is not a rumor, I fried two myself - the first one by accident while trying to figure how a game got more than 4 colors on a CGA adapter, and the second to fine tune the attack.
Then I attached a few hundreds bytes to a program I knew would be running on the PCs of a guy who had swindled a bunch of people. Yes, it worked, but my revenge palled compared
Re:possible cause of failure (Score:4, Funny)
You don't watch many action movies, do you? From them I learned that you can even make computers explode by things like entering the wrong password or just hitting the wrong key. And they also throw you halfway across the room when they blow up in sparkles.
I know one thing... (Score:5, Insightful)
Re: (Score:2)
Wisdom from my old man: Don't believe anything you hear or read, and only half of what you see.
Re: (Score:2, Funny)
I don't believe that. ;-)
Re: (Score:1)
I don't believe that. ;-)
I don't believe that you don't believe it then.
Re: (Score:2)
Wisdom from my old man: Don't believe anything you hear or read, and only half of what you see.
Instead, I heard it through the grapevine.
Yes but... (Score:2)
Re: (Score:1)
Re:Yes but... (Score:4, Insightful)
You post on slashdot and you pay attention to the news. I seriously doubt you fit into the category of "average customer".
I believe you'd find that the "average customer" doesn't share the "hate" that you suddenly gained, but are rather quite apathetic about the whole thing so long as they can get a cheap Playstation 3.
Re: (Score:2)
Apathetic in general or doesn't care that a(n insult to) weasel(s) found out he had to play by the rules. Or they do care and want to reward Amazon for supporting freedom of association.
Re: (Score:2, Flamebait)
You're not taking into account the customers they retain or attract as people learn more about the actual facts. Amazon is under no obligation to let oily weasles like Assange operate outside of their TOS, and they have a strong incentive to been seen actually enforcing their stated policies. Likewise, people who see that Amazon can easily weather a moderately aggressive DDoS like that juvenile tantrum thrown by Ano
Re:Yes but... (Score:5, Informative)
I agree, back in 1776 a bunch of evil weasels took over the colonies and look at what happened. a lot of innocent British soldiers sent over to protect the citizens died!
One man's TERRORIST is another mans Freedom Fighter.
Re: (Score:1)
Moral relavatism.....
Re: (Score:3)
Moral relativism exists, like it or not. Morality != (something objectively verifiable)
Re: (Score:2)
Which is what reality has been for the last 90,000,000,003 years. Before then we had Moral Absolutism... but Cthulhu put a stop to that.
Re: (Score:3, Interesting)
I agree, back in 1776 a bunch of evil weasels took over the colonies and look at what happened. a lot of innocent British soldiers sent over to protect the citizens died!
On their first terrorist attack, they cowardly disguised themselves to put the blame on innocent Native Americans. If they had succeeded, they could have caused an Indian massacre by the British.
Re: (Score:2)
If they had succeeded, they could have caused an Indian massacre by the British.
They didn't fail completely though, the British got their revenge on the other Indians.
Re: (Score:1)
On a similar note, one man's traitor is another man's informant.
Re: (Score:2)
Sure, but only if you're an irrational moral relativist.
Re: (Score:2)
Likewise, people who see that Amazon can easily weather a moderately aggressive DDoS like that juvenile tantrum thrown by Anonymous now have that much more of a reason to trust Amazon while buying goods or considering where to host cloud-ish stuff.
Juvenile tantrum? They took down Paypal, Mastercard, and Visa. That's not a juvenile tantrum, that is "hulk smash".
Re: (Score:2)
No offense, but that's fucking stupid. Why should Amazon risk compromising their servers just because one of their hosted sites has become a target? I fully support Wikileaks and their efforts, but you can't really put the blame on Amazon for dropping Wikileaks...Amazon is a business, and risking that business because of a single client would be a horrible idea.
Re: (Score:2)
Why should Amazon risk compromising their servers just because one of their hosted sites has become a target?
Hmmm, look like they did exactly that :-)
Amazon is a business, and risking that business because of a single client would be a horrible idea.
So why did they do it, if it was such a horrible idea?
Re: (Score:1)
Re: (Score:2)
That's what I was getting at, more or less...between pressure from the government and pressure from all the faceless people targeting wikileaks, it just wasn't a sound business decision.
I very rarely take the side of big business, but Amazon couldn't afford those kinds of problems. I have no idea why they agreed to host the servers in the first place (maybe they didn't think it would affect their end as much as it did?) but I completely understand why they decided to stop hosting them.
As I said in my OP, I
Re: (Score:2)
I have no idea why they agreed to host the servers in the first place (maybe they didn't think it would affect their end as much as it did?) but I completely understand why they decided to stop hosting them.
My understanding was that they didn't "Agree to host the servers" as a concious act. Wikileaks just set up an account and provisioned them automatically...
I agree with you, FWIW. While I have issues with the payment services cutting off donations, I have trouble objecting to what amazon did.
Re: (Score:1)
... and if I was trying to host an illegal file sharing site on one of their clouds they would shut it down for exactly the same reasons as they did WikiLeaks. It is stuff that is illegal to have. Now if Wikileaks had moved it to China :) or one of them countries that doesn't believe in copyright then they would be fine.
Re: (Score:2)
"... and if I was trying to host an illegal file sharing site on one of their clouds they would shut it down for exactly the same reasons as they did WikiLeaks."
Except for the "illegal" part.
"It is stuff that is illegal to have."
Don't think so. And provided that no USA fiscal office has started procedures against Wikileaks for the published contents, so do think USA legal body. It is stuff that it may be illegal to share if you are managing it. Perfectly legal to have and publish if you have managed to g
Re: (Score:2)
You and those like you amount to 0.01% of amazons customers.
Most people watch dancing with the stars+Glee and buy every shiney they see.
"Ohh this is pretty and made with the hearts of baby seals that were clubbed to death and assembled in china by slave children that get whipped! I'll buy 2!"
And yes, that is accurate. Otherwise Nike would be out of business as everyone knows their factories are horrible crimes against humanity.
Re: (Score:2)
The Teabaggers would have probably boycotted Amazon for supporting "terrorists" if they did host Wikileaks. Either way they lose customers.
Managing Perceptions (Score:5, Insightful)
Re:Managing Perceptions (Score:5, Insightful)
It would also be bad for business to say "We weren't attacked, we just suck at doing our jobs." This is precisely what hardware-related outages mean for an outfit like Amazon. Why would you trust your business to Amazon hosted services when they are incompetent?
Re: (Score:2)
Yeah, I wouldn't have expected Amazon to have a single point of failure anywhere, it's pretty strange.
Re: (Score:2)
Yeah, I'd expect them to have MANY points of failure, all in serial. So that's multiple, um, single-points-of-failure. Er, nevermind. Carry on.
Re: (Score:1)
Most would prefer a "Oh someone just forgot to do their job" over "an internet mob took down our site, and it could happen again at any moment"
Re: (Score:2)
One of them would be the result of a very small chance of failure at any given time. The other would imply a high chance of failure at the whim of whoever wants to attack them.
A rare screw-up looks bad, but folding whenever someone attacks you is worse.
Re: (Score:2)
How many outages for Amazon over the past 12 months, and what aggregate duration?
How many outages for Google (GMail, News, etc.) over the past 12 months, and what aggregate duration?
If you want to take it further, figure out what the % uptime comes out to. Seriously, this is a technical web site. I'd expect people here to understand the difference between 100% uptime and reality.
Re: (Score:1)
I wonder how much an hour of downtime would cost Amazon in lost sales?
Re: (Score:2)
Re: (Score:2)
Beginning of the End of DDoS? (Score:4, Interesting)
Re:Beginning of the End of DDoS? (Score:5, Interesting)
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
I imagine they'd like to see if a herd of ED-209s would help prevent/discourage it.
wait (Score:5, Insightful)
A hardware failure bringing down the site of a corporation who also is a cloud provider it's pretty bad PR, there should be no single point of failure in a proper cloudy system :)
Re: (Score:2)
I think it's better to have a system that might suffer from hardware failures than a system that can be taken down by a group of loosely-organized script kiddies.
Take a small PR hit to mitigate a larger one? Sounds like a plan to me.
Re: (Score:2)
Take a small PR hit to mitigate a larger one? Sounds like a plan to me.
On one hand you have incompetence. On the other hand you have incompetence. But on one hand you have someone to blame, and on the other you have only yourself. If a big piece of cloud can be taken down because one switch failed, then it's not a cloud, is it? It's just another centrally-managed cluster designed by idiots.
Re: (Score:1)
One hand, incompetence that might cause a fault in service once in a blue moon that can be repaired in less than half an hour. On the other, you've got incompetence that allows a system to be dropped by, again, a group of loosely organized script kiddies.
I'm not arguing for or against Amazon's service- I'm just stating that if I was in charge of damage control at Amazon, this is how I'd do it.
blame the hackers (Score:2)
It's one reason why threats of vandalism can be a bad idea. No matter what happens, there's now a group who has volunteered to be the whipping boys. As if it wasn't already too easy to blame it all on hackers to cover up incompetence.
Re: (Score:2)
Re: (Score:2)
Should be redundant, like switches, external connections, power, and everything else.
Opportunistic Single Point of Failure (Score:1)
If there is no demand/traffic, wouldn't that be the single point of failure that every cloudy system is designed to take advantage of?
Could still be related (Score:2)
What's to say a disgruntled Amazon employee who is also an anonymous follower didn't "accidentally" unplug the A/C. Presto, hardware failure!
Thanks (Score:1)
by a hacker group sympathetic to Wikileaks cause.
Thank you.
Damn you Amazon. (Score:2)
Re: (Score:2)
Re: (Score:2)
You mean it's a DDoS of the postal service?
Re: (Score:2)
The posties say "If it fits, it ships." [prioritymail.com] Apparently it just doesn't fit.
Just Amazon Europe? (Score:2)
Amazon.com was screwed yesterday afternoon too. Or was that just all the Christmas shopping?
"The cloud you can depend on" (Score:2)
Quoting from their site: "Reliable and redundant – Our datacenter is built using large amounts of commodity hardware. When one node fails, millions of other nodes pick up the slack. As a result, you gain all the benefits of an always-on and self-healing infrastructure, without ever having to configure or replace hardware."
So how did "a" hardware failure brought their cloud down? Anyway, a hardware failure still sounds better than a DDoS though.
Re: (Score:3)
Re: (Score:2)
So the likes of amazon.co.uk and amazon.fr are not showcases for their flagship technology then?
Re: (Score:2)
Harry Crumb would say (Score:2)
Sure Sure ....
There it is again..."sympathetic" (Score:2)
Amazon is correct (Score:2)
It was a hardware problem.
the hardware wasn't enough to keep up with all the requests coming in.