Forgot your password?

Close
typodupeerror
Businesses Bug The Almighty Buck

Recurly's Backup Mess Takes Days to Clean Up 21

Posted by samzenpus
from the best-practices dept.
A cascading hardware outage struck subscription payment provider Recurly last week, and that started a long example in how not to manage critical infrastructure. From the article: "Last Monday, the payment provider suffered an intermittent hardware failure, which prevented the company from processing either payments or refunds. The company says it serves over 1,000 customers, including Adobe, BrightCove, and Fox News Radio, processing recurring payments for subscriptions. By Friday, the company still hadn’t completely straightened out the mess, providing updates to customers using payment gateways such as Authorize.net and LinkPoint/First Data."
This discussion has been archived. No new comments can be posted.

Recurly's Backup Mess Takes Days to Clean Up More

Recurly's Backup Mess Takes Days to Clean Up

Comments Filter:

  • Re:No backups (Score:4, Informative)

    by tlhIngan (30335) <slashdot&worf,net> on Monday September 10, 2012 @05:18PM (#41293193)

    This is a perfect example of redundancy not being the same as backups. They had redundant encryption devices, but the failure of one rolled over into the other. They had no backups (that's right, none at all) that they could restore from. From what they've told us, they intend to resolve this issue by adding more redundancy.

    Correction, they have no backups of the keys that the encryption accellerators used. End result is now they have a bunch of encrypted data, with little in the way of being able to recover it because the keys used are lost or corrupted.

    Sounds like they need to be hacked and their information "liberated" so they can recover it :).

In any world menu, Canada must be considered the vichyssoise of nations -- it's cold, half-French, and difficult to stir. -- Stuart Keate

Close