Basecamp Details 'Obscene' $3.2 Million Bill That Prompted It To Quit the Cloud (theregister.com) 95
An anonymous reader shares a report: David Heinemeier Hansson, CTO of 37Signals -- which operates project management platform Basecamp and other products -- has detailed the colossal cloud bills that saw the outfit quit the cloud in October 2022. The CTO and creator of Ruby On Rails did all the sums and came up with an eye-watering cloud bill for $3,201,564 in 2022 -- or $266,797 each month. Plenty of that spend -- $759,983 -- went on compute, in the form of Amazon Web Services' EC2 and EKS services.
On Twitter, Hansson contrasted that cost with the spend needed to acquire servers packing 288 vCPUs and plenty more besides over three years. Hansson was at pains to point out that even that bill was the result of a concerted effort to keep it low. "Getting this massive spend down to just $3.2 million has taken a ton of work. The ops team runs a vigilant cost-inspection program, with monthly reporting and tracking, and we've entered into long-term agreements on Reserved Instances and committed usage, as part of a Private Pricing Agreement," he wrote. "This is a highly optimized budget."
On Twitter, Hansson contrasted that cost with the spend needed to acquire servers packing 288 vCPUs and plenty more besides over three years. Hansson was at pains to point out that even that bill was the result of a concerted effort to keep it low. "Getting this massive spend down to just $3.2 million has taken a ton of work. The ops team runs a vigilant cost-inspection program, with monthly reporting and tracking, and we've entered into long-term agreements on Reserved Instances and committed usage, as part of a Private Pricing Agreement," he wrote. "This is a highly optimized budget."
It is cheaper to buy yourself servers (Score:2)
Re: It is cheaper to buy yourself servers (Score:2)
A few cabinets in a data center with around the clock or on call staff would cost you about half that in fees and payroll in the US or EU. The equipment couls be amortized over a few years but would be a large initial capital investment.
Most likely the best choice is the most complex. Build your own to cover the baseline demand in your most important regions. Dynamiccally add more through one or more cloud providers as needed. Harder to design something general purpose that can be split this way but kinda w
Re: (Score:3, Insightful)
It's not that complex. Car analogy? You wouldn't run your fleet on rent-a-cars. Rentals are great to fill in gaps, but for day-to-day operations, you lease or you buy. Why lease? Because the beancounters (well, "stakeholders" like activi^Wasshole investors) like opex over capex. So another company will happily make it its business to invest in capital and rent it to you at a premium. Also convenient one-stop pricing rather than depreciating, paying for maintenance, and all the extra crap that comes with own
Re: It is cheaper to buy yourself servers (Score:1)
Leasing is renting...
Re: (Score:2)
it's in the ballpark of what I'd charge to drive to Seattle.
Re:It is cheaper to buy yourself servers (Score:5, Insightful)
That is the thing with "the cloud", it is similar to canned ERP systems. They sell you on how you will *not* need to hire "expensive" IT people and just anyone can manage this. Then hoping you do not look too deep, your bill raises and raises. By then you are locked in and it is way too expensive to change.
Re: (Score:3)
The cloud can be extremely useful to a business that is growing rapidly. However, you must test your code in a development environment with limited resources, or it will bankrupt you quickly.
Re: (Score:2)
That is the thing with "the cloud", it is similar to canned ERP systems. They sell you on how you will *not* need to hire "expensive" IT people and just anyone can manage this. Then hoping you do not look too deep, your bill raises and raises. By then you are locked in and it is way too expensive to change.
I've always compared "cloud" providers like Azure and AWS to budget airlines. They get you in with cheap prices but then have extra costs for everything else. Much the same as flying Ryanair (or Spirit for the Americans playing along at home), sure there is that £12 seat... but it's in the middle of the night flying from "London" Bradford Airport to somewhere within 300 miles of it's intended destination. Want to fly at a reasonable time and actually to the place you want to go, that's a £50 far
Re: (Score:2)
If Azure and AWS are the budget airlines, then what are providers like Akamai's Linode, which costs a fraction as much?
Cloud isn't always cheaper. (Score:2, Interesting)
It's certainly easier for some things, but you're paying for the flexibility and convenience of someone else doing IT for you. The big cloud providers rob you on egress bandwidth costs.
When you're big enough to need a data centre, you might as well just own the hardware yourself, like when dropbox abandoned AWS.
Re: (Score:1)
Re: (Score:2)
At any significant scale, public cloud is almost never cheaper. What you're paying for is convenience and speed to market. For some companies, that seems like a fair trade. For many, who simply followed the "cool cats" into public cloud, they've gotten bitten firmly in the ass by trying to run legacy applications with legacy mindsets on someone else's infrastructure. The costs very quickly balloon.
The next big thing is the wave of Fortune 500 firms who jumped into AWS/Azure/GCP because everyone else was
Re: (Score:2)
Yes. I worked for a company that went full "we're moving to the cloud". It was a Windows shop, with "classic" .NET desktop apps, and SQL Server.
Someone thought it was a great idea to go cloud. Their approach? "remote desktop into a VM, and install SQL Server in the cloud server". over time the realized that this is not really "Cloud" as they still have to do IT and ransomware still affects you.
they now have a proper "cloud" infrastructure with a team that quoted $250/mo for a static website for internal use
Re: (Score:2)
The cloud isn't always cheaper - actually, it probably never is - but it gives you opportunities for operational flexibility, and sometimes, cost savings (in ops).
AWS is almost never cheaper than anything though - I have no idea what their negotiated pricing is, but there are many other smaller cloud providers that will happily blow some cold air over some servers for you for a fraction of what the equivalent at AWS would cost you. You do lose some of the operational flexibility and cost savings you'd have
Basecamp? (Score:2)
Canâ(TM)t tell if they actually realize sales. Seems to have a large staff for relatively little income. Skimping on the necessary part of your product, servers, seems counter indicated.
Re: (Score:2)
Perhaps they can save with their own servers, but (Score:2)
The figures they give are simply to buy some of the hardware.
The figures don't include: physical hosting, connectivity, manpower, backups and, most importantly, they don't include all the storage, which is the single biggest line item on their AWS bill.
It would be interesting to see a proper comparison.
Re: (Score:2)
Re: (Score:3)
Actually, specific to EC2 it sounds like they can significantly reduce costs. I don't know what they need to manage the servers in terms of manpower and automation tools, but they are unlikely to need to spend more than $5k/month to host the machines identified, and server management should be under $20k/month. Capital cost is under $2k/month. There is a big delta there between ~$30k and $266k/month... even if you need to duplicate hardware and labor.
None of this should be a surprise though; of course yo
Re: (Score:2)
Doh, missed that it was just $60k/month for EC2... so it is about what I would have expected in premium.
Re: (Score:3)
Figures that sound high to most people, like $3 million per year as used in this article, without context are meaningless. The same article says they spent over $900k per year to store 8 petabytes of data, and gave a server spec which would cost almost $25 million to buy (533 servers with 3 terabytes each). Obviously you can put more than 3 terabytes of data in a single server, but it goes to show that the article and CTO's Twitter posts don't even try to give a good comparison. No mention of cost of their
Re: (Score:2)
Servers and spinning media to store 8 PB would easily come in under $3M and you get to amortize that over 5 years or so.
You'll spend considerably less money than cloud if you put that hardware into colo racks somewhere.
Re: (Score:3)
I'd love to see a full, apples to apples, break
Colo (Score:2)
There are colocation companies that specialize in data redundancy across multiple sites. My friend works for one. You ship them three or four similarly configured servers and they can clone and deploy them however you need them to. Also, they run nearly everything bare-metal. Very thin Linux distributions running docker and/or kubernetes - no VM anything. Depending on your resource usage, it can end up being a *lot* cheaper than AWS. The only downside is you need to use their approved server vendors as they
Re: (Score:2)
"no VM anything"
The whole Docker running a VM on a VM has always bugged me. Docker is still a VM, but at least on dedicated hardware it's only one abstraction. Everything comes in cycles in IT, I imagine we'll see some form of reversion to hosted hardware, possibly paired with VMs being replaced with innovation in process sandboxing.
Re: Colo (Score:3)
Docker is most definitely NOT a VM on Linux. It's containers. Completely different architecture. It's got more in common with a chroot.
Re: (Score:3)
I've been hearing this argument for as long as cloud has been around. What I found is cloud is at a minimum 10 times as expensive as doing it yourself. Cloud advertises the Rolls but provides the Escort. Yes cloud can have redundant data centers but if you don't pay for them then you don't have them either. Same goes for backups. You pay for the maintenance contracts or you are not provided the maintenance. It's not free on the cloud and it's not included either. You pay for what you get and you pay a lot m
Re: (Score:2)
What I found is cloud is at a minimum 10 times as expensive as doing it yourself.
There is almost no chance you do that kind of analysis for a living for anything larger than a SMB if you think cloud is 10x the cost of using your own servers for most use cases. Perhaps it is true for some cases, but probably with a horrible cloud architecture and/or very specialized hardware.
Re: (Score:1)
I calculated our costs once a year for 7-8 years. It was consistently 8x to be in cloud long term. You can't just look at the first year startup cost in a colo and compare that to your cloud. You have to amortize over the life of the hardware you're buying. I used 4 years as my hardware life cycle instead of the usual 3. Most places use their stuff after 3 anyway so I was just trying to be honest about it.
We didn't have a lot of outbound network traffic but did use a lot of storage and cpu and ridiculo
Re: (Score:2)
Actually, that's where cloud does make sense. Run production on your own servers and use the cloud for backup.
You're paying for a lot (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Another, likely more effective strategy is to improve the efficiency of your systems. When the components of your system are heavy runtime memory pigs your costs will be high.
Re: (Score:3)
Re: You're paying for a lot (Score:2)
That might have something to do with it. Ruby is shockingly slow. Jruby is practically as fast as ruby though. I guess they are not using it.
Re: (Score:2)
> maintenance staff, programmers and admins,
And those personnel costs will be higher than you realize. IT/OPS people who can properly spec out, plan, buy, and build out a datacenter deployment have become pretty scarce, and command a nice salary premium, these last several years since pretty much everyone have been going "cloud" these days. In fact, the deciding factor that landed me a job a few years back was that I had DC experience before I learned my way around AWS. Plus, you need a squad of PFYs,
Re: (Score:2)
"magic cloud" technology.
includes a considerable markup for profits.
there are lots of other costs than servers... (Score:4, Interesting)
Re: (Score:1)
Ah magic. Can you list a bit more jargon please. Peering is a cost? Don't you need a data center to run a tech company? Won't you have anything local? Don't you need knowledgeable people to run your cloud services? Don't they need to understand networking, security, SAN or NAS and all the people to understand and support all that?
It's not magic and it gets more complicated than running it yourself because now it's proprietary and out of your control. SLA's are only as good as your legal team and most of the
Re: there are lots of other costs than servers... (Score:2, Insightful)
Re: (Score:3)
I read a similar post about leaving the cloud recently which said that they found the cost of the people they needed with cloud expertise to run on the cloud was about the same as the cost of the people they needed with IT expertise to run things on-prem. For that business, at least, it turned out to be about replacing one set of expensive people with another set of similarly expensive people, and none of the cut-your-headcount promises actually worked out.
Those tradeoffs will probably change business-by
Re: (Score:2)
The benefit isn't cutting headcount, the benefit is not having to pay for another corporation's profit in addition to paying for the headcount. Plus, you get those people in your organization, where you can fire them if they aren't doing their job, instead of hoping the cloud provider will handle such issues.
Re: (Score:2)
> Data centers, networking, peering, probably SAN or NAS, and all the PEOPLE that need to support all of that.
If you go with dedicated servers, bare metal or whatever they call them nowdays, you rent it from a DC, you don't run data centers, networking, peering, etc. by yourself, you just rent servers and the DC staff are taking care of everything else. And he's right. The price of "cloud" is ridiculously expensive. $50 a month you get 16 core amd ryzen or equivalent intel with 64 gb ram, 1 tb of ssd dri
When you're the 800lb Gorilla (Score:2)
Amazon has lured a lot of businesses into the cloud and some of their services are dirt cheap, storage for example while others like EC2 are costly. Spot instances are one way of controlling the latter and for ad-hoc instances, you can't ignore it. From an Economics standpoint, there should have been a cost-benefit analysis of using cloud-based resources vs. say co-lo or on-prem deployment. Just "going to the cloud" isn't an answer sometimes if you're looking to save money in one area. All of that service
Looks like they are near the transition cost. (Score:5, Interesting)
It is good to review the numbers, and it does seem like they are around the transition point.
"The Cloud" means site redundancy, multiple backups, high bandwidth, true. It means staff on site 24/7 rather than on call. It means constantly monitoring all the data center sites, replacing running redundant parts before they fail without visible downtime. Continuous backups and the ability to roll back or otherwise recover from the inevitable attacks that occur.
It is not just the hardware, and the people running the hardware, but the software and the services, too. All the programs for the cloud infrastructure must be kept up to date, monitoring for threats, and kept patched. It needs to be kept safe from ransomware attacks, government spying, corporate espionage, or at least, be no worse at this than the big companies are.
At $3M/year they really should be considering the value for the money, but realistically they are probably looking for a similar cost across the company to replicate the services. They will need a big expansion for the staff, from the executive level down for an entire branch of the company dealing with online services.
It is good that they say they will post more. The follow-up will be interesting.
My hunch is a big up front transition cost, and ultimately a pair of data centers with somewhat fewer services, more tailored to their needs, with a similar comprehensive bottom line cost.
Re: (Score:2)
You make it sound as if cloud providers aren't in it for the money. The items you list must be paid for. Cloud doesn't give you redundancy unless you pay for it. You pay for the 24/7 cloud monitoring but you need to pay extra to monitor your services. If your business runs 24/7 cloud doesn't monitor that your services are running. They monitor that their services are running. If you pay cloud to monitor your services then that's an extra charge.
Re: (Score:3)
Re: (Score:2)
Cloud computing is predicated on the idea of underutilization leaving spare resources. But as someone uses more resources, they're going to want to optimize more and more, and they will eventually get it to the point where they're not leaving anything on the table. At that point, there are no free lunches to be had anywhere, and the cloud services provided is just a middleman who has to add cost in order to profit. Consequently not only do only small customers benefit, but you have to have those small custo
Re: (Score:2)
Re: (Score:2)
Cloud computing is predicated on the idea of underutilization leaving spare resources.
Some services are, but big computing uses aren't. For those needing "big iron", heavy utilization is fine. For sparsely used resources, all those web hosts that might get ten visitors a month, shared resources are great and can be incredibly efficient.
Repeating the theme, it's all about scale.
When done at scale, it is cost efficient to have people whose only job is to replace mirrored hard drives across a data center before they fail. At scale, it is cost efficient to have skilled workers monitoring the
Re: (Score:3)
TEN years ago - a supermarket presentation (Score:2)
Re: (Score:2)
> "The Cloud" means site redundancy, multiple backups, high bandwidth, true. It means staff on site 24/7 rather than on call. It means constantly monitoring all the data center sites, replacing running redundant parts before they fail without visible downtime. Continuous backups and the ability to roll back or otherwise recover from the inevitable attacks that occur.
For smaller organisations, they wouldn't have the staff to handle application or service issues 24/7 anyway. So having infrastructure that c
So now we know... (Score:2)
AWS Net Margin as indicator of cost equivalents (Score:1)
AWS net margin on services provided is 35%. Thus on 3.2 mill spend, it cost AWS about 2.1mill (all inclusive) to deliver those services. So the question really is, can you as a private entity recreate all the tangible and intangible benefits of AWS for 2.1 mil.
My gut instinct is that only some percentage of Basecamp's workload really needs all the benefits of AWS and thus with a reduced spend on Colo (and all of it's direct and indirect costs), a good portion could move off cloud with the remainder of 'AWS-
Re: (Score:1)
He might pay a million or two for hardware to duplicate his current services. Once.
AWS he pays every year, forever.
Re: (Score:1)
Maintenance costs were included in my 4 year contract upon purchase. Overall, that's a pretty small part of DC costs.
4 years of cloud vs 4vyears of all in full cost of DC was very consistently 8x.
After 4 years it got worse for cloud because a ton of hardware is still good to go and just needs a new contract at worst. Other things like my racks are 1 time cost forever. No one replaces the actual 4 post rack every 4 years. My network cabling was professionally installed. 1 time per rack, forever, done. A
Re: (Score:1)
Btw, my sites were not trivial. The last real company I worked at had a 5.5m/month aws bill. That's 66m a year, yes.
I would've never done that with the type of load we had but C level was building his whole career on being cloud guy so I was stuck for it until I left.
At place prior to that I used aws only for back ups and site redundancy. Just that much cost 300k a month with out running prod there.
This isn't mom n pop smb stuff.
Re: (Score:1)
I'm aware of storage costs and how storage works. It's likely a bunch of your files are stored on the net right now on a system I designed./architected.
The cost answer to storing 8PB or any random number can not be determined by simply saying "oh we need a SAN and those are expensive and ..."
You need to know what is being stored. What are the performance requirements for read and write. How much downtime is allowed, how critical is the data (meaning how many copies do you need). What is the structure of
Re: (Score:2)
It does seem like 8PB of data in S3 is an indication that something is wrong.
what part of the cycle are we in? (Score:4, Insightful)
Emerging theme of the 2020s... (Score:1)
Re: (Score:1)
For Lego, it isn't even about having a normal business model or loss leaders or anything.
They sell you a few thousand bits of hard plastic in a box and some instructions for a few hundred bucks that costs them about 6 bucks to produce.
It always amazes me when I see the Lego store at the mall how crowded they are.
I feel robbed buying stuff off Ali express for a fraction of the cost. Why do folks pay full retail? I don't get how the market can carry that full price.
Re: (Score:1)
Re: (Score:1)
Maybe the hipsters over at 37 Handjobs should look in the mirror to find the real problem.
Re: (Score:1)
What if it is not SaaS? (Score:1)
EC2 = vanilla iron (Score:1)
More proof that RoR is a huge pig (Score:2)
Most web-type ventures' cloud hosting costs should be weighted toward bandwidth--of the CDN variety if they're doing it right. CPU should only dominate if they're working on AI training or somesuch, and storage might be a big cost for some other types of usage patterns.
I messed with RoR when it first came out about 20 years ago and dismissed it as ju
Re: (Score:2)
Unlikely. They have 8PB of data in S3, which is their biggest cloud expense. I don't know why they have that much data in S3, but it has nothing to do with Ruby.
Cloud Caching (Score:2)
Someone mining bitcoin (Score:2)
Sure, blame the cloud. (Score:1)
Every dollar is a testament to incompetence of the CTO and technical stack chosen.
If you can't trim down costs, you are the problem.
Everything costs money, use a balanced approach (Score:4, Insightful)
I would suggest that everyone company have a load balancer and a small server farm. When you hit capacity you load balance to the cloud and use AWS and then when the rush is over you drop back to local servers. The thing I fear even more that the bill is getting dropped by my cloud provider, As politics and the culture war enter the tech space the last thing I need is the CEO saying something stupid then having AWS suspend, slow down, or drop services e.g. Parler comes to mind.
I would encourage ever company to maintain an in-house production version of their Internet services because you never know what will happen in these crazy times. Additionally, this allows you to switch cloud providers or have multiple cloud providers and compares your services and bills. The cloud just keeps getting more and more expensive every year.
Of course you "save money" cutting insurance (Score:2)
Links from TFA explain that a lot of the cost is due to the cloud service being able to handle unusual loads, such as 10X more users than they expected signing up for a new service. If you can guarantee that you won't "go viral" for any reason, and if you have methods in place to prevent DoS attacks, then of course you'll save money going off the cloud; but are they really going to be safe, or do they just feel like rolling the dice now because they aren't doing so hot and are looking for some place to swi
Re: (Score:2)
Baseload local plus flex in the cloud is the solution there though.
Most companies don't cost compare (Score:3)
My primary job is developing software. I spend way less than 5% of my time maintaining the servers. There really is nothing to maintain. OVH maintains hardware. OS security updates are applied automatically. Remote monitoring software keeps tabs on every running service and disk, memory and CPU utilization. We do not have a dedicated system admin. Once the systems are setup, and we have an automated pipeline to test and deploy changes, I am not sure we have a use for a dedicated system admin. Organizations with hundreds of servers might need them, but small teams with a handful of servers do not.
Is RoR part of the problem? (Score:1)
Ill be honest, I have not been involved with any RoR development in a long time. Im sure both ruby and the rails framework have come a long way since the last I touched it.
The most cost savings and benefit one will see in the cloud is moving to "cloud native" architectures. Lambdas/functions and event streams along with other cloud native features. Ruby is not a first class citizen in many of these, which makes me wonder -- are they just hosting everything in containers and trying to manage it all as a RoR
Someone finally noticed this? (Score:2)
Before the cloud, servers and hardware were cheap. If I wanted one, I would just ask IT what they had to spare and I was assigned one of those.
Now, management will make a developer spend days or weeks doing some sort of tap dance so they don't have to add what is now, an expensive resource (i.e. hardware).
The only people who love the cloud are accountants who hear the words "operation cost vs capital costs," after which their tiny brains turn off.
little dog big dog (Score:2)
If you farm out anything to someone a lot bigger than you, don't expect them to pay any attention to what you want, such as fixing a bug or adding a feature, or even competent help using their service.
Mainframe (Score:2)
Who knew that renting a mainframe would cost so much!
Cloud is Not Cheaper (Score:1)
CLOUD COMPUTING IS A SCAM (Score:2)
Get out while you can.
Seriously, it's STUPID expensive, and one wrong keystroke, and you've got a bill increase of $45k for the month.
Sure, it's great if you don't currently have the infrastructure, but long term, it's a money pit. GTFO.
It's just not worth it.
Still cheaper than having incompetent server staff (Score:2)