Forgot your password?
typodupeerror
Businesses Bug The Almighty Buck

Recurly's Backup Mess Takes Days to Clean Up 21

Posted by samzenpus
from the best-practices dept.
A cascading hardware outage struck subscription payment provider Recurly last week, and that started a long example in how not to manage critical infrastructure. From the article: "Last Monday, the payment provider suffered an intermittent hardware failure, which prevented the company from processing either payments or refunds. The company says it serves over 1,000 customers, including Adobe, BrightCove, and Fox News Radio, processing recurring payments for subscriptions. By Friday, the company still hadn’t completely straightened out the mess, providing updates to customers using payment gateways such as Authorize.net and LinkPoint/First Data."
This discussion has been archived. No new comments can be posted.

Recurly's Backup Mess Takes Days to Clean Up

Comments Filter:
  • by Mr. Kinky (2726685) on Monday September 10, 2012 @02:58PM (#41291297)
    This case reminds me of our payment processor Authorize.net in 2009, when a fire took down the whole network and infrastructure for many days. It was only solved when one of the guys over at Authorize.net literally
    • by MetalliQaZ (539913) on Monday September 10, 2012 @03:00PM (#41291331)

      He would have finished the story but he had a cascading hardware failure that took out his network...

    • by carlos92 (682924)
      Literally what? The suspense is killing me!
    • by tstrunk (2562139)

      I know that technician! His name was Candlejack, right?
      When he came to

  • by Anonymous Coward on Monday September 10, 2012 @03:23PM (#41291625)

    ...a service provider named Recurly in the first place.

    Same goes for any provider named Relarry, Remoe or Reshemp either for that matter.

  • No backups (Score:3, Interesting)

    by Anonymous Coward on Monday September 10, 2012 @03:51PM (#41292099)

    This is a perfect example of redundancy not being the same as backups. They had redundant encryption devices, but the failure of one rolled over into the other. They had no backups (that's right, none at all) that they could restore from. From what they've told us, they intend to resolve this issue by adding more redundancy.

    Yes, really.

    • Re: (Score:3, Funny)

      by Anonymous Coward

      They should have used RAID.

    • Re:No backups (Score:4, Informative)

      by tlhIngan (30335) <slashdot@wor f . n et> on Monday September 10, 2012 @05:18PM (#41293193)

      This is a perfect example of redundancy not being the same as backups. They had redundant encryption devices, but the failure of one rolled over into the other. They had no backups (that's right, none at all) that they could restore from. From what they've told us, they intend to resolve this issue by adding more redundancy.

      Correction, they have no backups of the keys that the encryption accellerators used. End result is now they have a bunch of encrypted data, with little in the way of being able to recover it because the keys used are lost or corrupted.

      Sounds like they need to be hacked and their information "liberated" so they can recover it :).

This place just isn't big enough for all of us. We've got to find a way off this planet.

Working...