Recurly's Backup Mess Takes Days to Clean Up

A cascading hardware outage struck subscription payment provider Recurly last week, and that started a long example in how not to manage critical infrastructure. From the article: "Last Monday, the payment provider suffered an intermittent hardware failure, which prevented the company from processing either payments or refunds. The company says it serves over 1,000 customers, including Adobe, BrightCove, and Fox News Radio, processing recurring payments for subscriptions. By Friday, the company still hadn’t completely straightened out the mess, providing updates to customers using payment gateways such as Authorize.net and LinkPoint/First Data."

Reminds me of Authorize.net

[Mr. Kinky]

This case reminds me of our payment processor Authorize.net in 2009, when a fire took down the whole network and infrastructure for many days. It was only solved when one of the guys over at Authorize.net literally

Re:Reminds me of Authorize.net

[MetalliQaZ]

He would have finished the story but he had a cascading hardware failure that took out his network...

Re:

[maxwell demon]

Yeah, that never could happen with me, because I

Re:

[carlos92]

Literally what? The suspense is killing me!

Re:

[tstrunk]

I know that technician! His name was Candlejack, right?
When he came to

Re:

[arglebargle_xiv]

Pretty common practice to half-ass everything, they don't care about supporting the customers just getting their percentage off your transactions..

A friend of mine runs a networking services company who got called into a medium-sized payment processor a few months back to upgrade a server, about an afternoon's work. After several months of 10-12 hour days he's now got them up to the level where they're about quarter-arsed. With another few months' work they'll be at the level of half-arsed. When he described the original setup he found I thought he was making it up, it was just fail layered upon fail layered upon fail, like something a bunch of dru

Re:

[cusco]

Makes me glad that I pay cash for everything possible.

I would've been leery of...

[Anonymous Coward]

...a service provider named Recurly in the first place.

Same goes for any provider named Relarry, Remoe or Reshemp either for that matter.

Re:

[Alien Being]

I'm Honest Moe, that's Honest Shemp, and that's... that's Larry.

No backups

[Anonymous Coward]

This is a perfect example of redundancy not being the same as backups. They had redundant encryption devices, but the failure of one rolled over into the other. They had no backups (that's right, none at all) that they could restore from. From what they've told us, they intend to resolve this issue by adding more redundancy.

Yes, really.

Re:

[Anonymous Coward]

They should have used RAID.

Re:No backups

[tlhIngan]

This is a perfect example of redundancy not being the same as backups. They had redundant encryption devices, but the failure of one rolled over into the other. They had no backups (that's right, none at all) that they could restore from. From what they've told us, they intend to resolve this issue by adding more redundancy.

Correction, they have no backups of the keys that the encryption accellerators used. End result is now they have a bunch of encrypted data, with little in the way of being able to recover it because the keys used are lost or corrupted.

Sounds like they need to be hacked and their information "liberated" so they can recover it :).