'Why You Should Not Use Google Cloud' (medium.com) 508
A user on Medium named "Punch a Server" says you should not use Google Cloud due to the "'no-warnings-given, abrupt way' they pull the plug on your entire system if they (or the machines) believe something is wrong." The user has a project running in production on Google Cloud (GCP) that is used to monitor hundreds of wind turbines and scores of solar plants scattered across 8 countries. When their project goes down, money is lost. An anonymous Slashdot reader shares the report: Early today morning (June 28, 2018) I receive an alert from Uptime Robot telling me my entire site is down. I receive a barrage of emails from Google saying there is some "potential suspicious activity" and all my systems have been turned off. EVERYTHING IS OFF. THE MACHINE HAS PULLED THE PLUG WITH NO WARNING. The site is down, app engine, databases are unreachable, multiple Firebases say I've been downgraded and therefore exceeded limits.
Customer service chat is off. There's no phone to call. I have an email asking me to fill in a form and upload a picture of the credit card and a government issued photo id of the card holder. Great, let's wake up the CFO who happens to be the card holder. What if the card holder is on leave and is unreachable for three days? We would have lost everything -- years of work -- millions of dollars in lost revenue. I fill in the form with the details and thankfully within 20 minutes all the services started coming alive. The first time this happened, we were down for a few hours. In all we lost everything for about an hour. An automated email arrives apologizing for "inconvenience" caused. Unfortunately The Machine has no understanding of the "quantum of inconvenience" caused.
Customer service chat is off. There's no phone to call. I have an email asking me to fill in a form and upload a picture of the credit card and a government issued photo id of the card holder. Great, let's wake up the CFO who happens to be the card holder. What if the card holder is on leave and is unreachable for three days? We would have lost everything -- years of work -- millions of dollars in lost revenue. I fill in the form with the details and thankfully within 20 minutes all the services started coming alive. The first time this happened, we were down for a few hours. In all we lost everything for about an hour. An automated email arrives apologizing for "inconvenience" caused. Unfortunately The Machine has no understanding of the "quantum of inconvenience" caused.
Sorry, but... (Score:5, Insightful)
Re:Sorry, but... (Score:5, Interesting)
Re:Sorry, but... (Score:5, Insightful)
As if servers doing down can't happen if you host it yourself.
But then you're in control, instead of having to rely on some amorphous, anonymous monster that only allows communication via automated email.
Cheap service, cheap results (Score:5, Insightful)
The company thought they could get away with paying less for server infrastructure. They can. But they get less. This is one of the "less" things they get.
If you value your data, host it yourself, preferably in multiple locations. If you want to go cheap, then you can expect to lose things.
Like your data, or access to it, or availability of it.
It's not such a smart thing to cheap out on the important stuff.
Of course, convincing the bean counters of future risk inherent in what appears to them to be current savings... good luck with that.
Well, best to get rid of your bean counters. :)
Here's a maxim of mine I like to drop on the table during discussions like these:
Re:Cheap service, cheap results (Score:5, Interesting)
This.
I'm retired now, but my firm had a plan to replace me with the cloud.
We were a law firm.
During my last two weeks, an elderly couple drove in from about 70 miles away to sign some family law papers and they were waiting in the conference room when a partner got hold of me and told me, "The cloud's down again .
I called the support number and they said they were aware of the problem and that they were working on it.
After a lot of pressure, I called again and told them to fail-safe over to the mirror that they had bragged about.
They said the outage got the mirror, as well.
I informed the partner and she started screaming at me. She yelled, "WHAT IS PLAN B?"
I said, "Ma'am, plan B is plan A."
It was quite a shit storm.
I had argued against the cloud, and I documented their rejection of my recommendations and they signed off on it.
They spent a a butt load of money bringing all that shit home, and I worked there another 5 years.
I had full hands-on control of the shit I built. All of it. I was totally responsible and could either fix, or get fixed, anything running in my house.
Re:Cheap service, cheap results (Score:5, Interesting)
I'll happily go back to them when they ask me to clean up the shitstorm their current CTO doesn't see coming. As CTO and for double what I was getting as a contractor, of course.
Under my lead, they went from everything running on a single unstable server and going down every other day to everything running on a distributed cluster of servers and not a single outage in sight. This, in the matter of under a year, on a site that is their entire business and sees over 500k uniques and serves over 20 million pages per month.
Their new solution costs them more per month and limits them to 1.5 million pages served per month on the best package they actually list a price for, with no uptime SLA. In short, they'll be paying 15x as much for infrastructure for the capacity the current system has, which more than covers what they're "saving" by not paying me.
It is what it is, but they thought I was only looking out for my own ass when I pointed all of this out. Well, I was looking out for me but, as they were providing me steady full-time work, a huge part of that was looking out for them in-kind.
Re:Cheap service, cheap results (Score:5, Funny)
I have (neigh, had) a client making the same choice right now.
+1 for having the horse sense to move on from that client, after you told them to hold their horses and they didn't listen. I guess you can lead a horse to water, but you can't make him drink. If they're going to get on their high horse and be as stubborn as a mule, then they're going to end up trying to close the barn door after the horse has bolted. When comparing in-house vs cloud, in-house really is a horse of a different colour. Trying to run a tech business on the cloud is just horsing around.
Re: (Score:3)
Stop being such an ass.
Re:Cheap service, cheap results (Score:4, Funny)
Let's rein in the puns, ok? Feels like we're flogging a dead horse.
Exactly!! Ding, Ding, Ding! (Score:5, Insightful)
I may be one of the "old timers" who I'm told is thinking about things in an "old school" way when I say this. But I've *always* warned people that "The Cloud" just means you're giving somebody else the responsibility of handling your data and the systems it runs on.
That makes sense sometimes. I'm not "anti cloud". But for anything really critically important to a business, I feel you should have it running locally and THEN consider cloud options as hot-failover sites, backup sites, etc. With cloud hosting, the whole thing is off limits to you as soon as your Internet circuit goes down, for one thing. With it running locally, you can still use it just fine anywhere on your LAN.
But additionally, if the provider hosting your stuff goes bankrupt or merges with someone else, or just plain decides it's not profitable enough without some pricing changes -- where does that leave you? Technically, they can just disappear with your whole software and data configuration overnight. Or they can put trained apes in charge of maintaining things so it suddenly has huge security holes. Who knows?
When you run things yourself, YOU are where the buck stops if things go wrong. If you're good at what you do, that should be more of a comforting thing than a scary thing. I've seen too many shops trying to cut corners on the I.T. hiring budget by bringing in less experienced people who really can't properly run the systems they're supposed to be caring for. The cloud for them is a crutch ... a way to get things done that are beyond their abilities. But that's not an ideal situation for a business to put itself in.
Re: Exactly!! Ding, Ding, Ding! (Score:3)
Here's the thing about hosting on AWS. When your shit goes down, everyone's shit goes down. Github is down, Slack is down, Pinterest, Reddit, etc.
In the end, none of our customers blame us because we can simply point to all the other well known sites that are down at the same time.
It is what it is.
Re: (Score:3)
Guess what, having someone else manage the hardware for you is only a teeny tiny portion of the workload involved in managing IT infrastructure. Everything you build on top of said hardware still falls on your plate to maintain, cloud or not.
Re: Cheap service, cheap results (Score:4, Interesting)
Yeah, he is right about that "single point of failure".
There are a huge number of companies that fall into that hole between "too small to have our own IT department" and "big enough to have a large IT department with formal procedures and personnel redundancies".
I was that "one guy who does everything" for a long time. It is easily the most efficient way to run a company. Even when we were a small department with a half-dozen really sharp guys, it was still a high-wire act, dependent on having A players who knew the business and systems well.
It was only after we hit maybe 15 or so IT employees that we really had full redundancy in personnel and reasonably well documented procedures. And even that was pretty small... we didn't have dedicated QA employees, for example.
It all depends on what your needs are and what your risk tolerance is. For most growing small companies, the reward of having the "one really smart guy" as a key point of failure is worth the risk. Later, when it comes time to sell out to an investor or go public, that risk/reward equation flips and you have to figure out how to eliminate that red flag on the audit.
As many here have said, going "cloud" isn't a panacea to fix that problem.
Re: (Score:3)
Cloud hosting is seldom cheaper than renting space in a DC and throwing your own servers in. The most common exception is if you're really small scale (ie, you would only use one or two physical servers). Most of the time, cloud hosting is quite a bit more expensive.
Re: Cheap service, cheap results (Score:4, Insightful)
Then take the contract you have with them to court and get your effing money back. What? No contract? You deserve what you get.
Re: (Score:3, Informative)
Whoosh. The GP's point was the Google never contractually obligates itself to customers, because it can get away with that, flouting the norms of commerce. They can get away with this because their customers are stupid.
Re:Cheap service, cheap results (Score:5, Insightful)
You get what you pay for. They thought they could get much more than they paid for. 100% their fault. At the very least, they failed to evaluate what they actually got. And, since this is apparently the second time this happens, they also seem to be unable to learn.
Re:Cheap service, cheap results (Score:5, Insightful)
Re: (Score:3)
Simple: What you actually bought is a straw-hut and what you were primarily using it for is excessive open-fire cooking...
Re:Sorry, but... (Score:5, Insightful)
As if servers doing down can't happen if you host it yourself.
But then you're in control, instead of having to rely on some amorphous, anonymous monster that only allows communication via automated email.
Exactly right. If your servers go down, YOU can fix them immediately and you don't have the problem of trying to get in touch with some support person who may or may not respond quickly, or may not respond at all if you can't even figure out how to get in touch with them.
That said, this was a case of a company trying to be cheap, and instead of an enterprise account they tried to run critical applications on what is supposed to be a consumer-level account, which is why they started getting e-mails from Google about "suspicious activity".
Re:Sorry, but... (Score:5, Informative)
Spoken like someone who has never had to fix a server immediately :-)
One of the strong points of cloud computing is the infrastructure to shift load to accommodate failing hardware. To reproduce that capability with your own hardware & infrastructure requires a tremendous amount of planning and capital investment: in power, servers, and network. It's almost never a simple matter of "fix the server immediately".
Re: (Score:3)
If you're running your own, then you should have hardware in 3 separate locations. If you're running in the cloud, then you better have 3 separate providers.
Re:Sorry, but... (Score:4, Interesting)
Of course you need to pay for the required infrastructure. That is what "host yourself" means. Either you are paying the higher up front cost of building it out for yourself or you are paying the much higher costs to have google/amazon/digitalocean/etc do it for you. As for shifting load to accomodate failing hardware, it doesn't work as well as pitched (which is why these services go down all the time and have such high latency) but it works better and faster on a private cloud implementation. Which is where you've built a cloud stack on your own gear. Just be sure you actually have redundant controllers across the whole thing and didn't just build redundant controllers on a virtualized layer sitting on top of a not redundant system.
Re: (Score:3)
As if servers doing down can't happen if you host it yourself.
But then you're in control, instead of having to rely on some amorphous, anonymous monster that only allows communication via automated email.
If you only have a single site, then you are still at the mercy of things like a cooling systems or power failure. If you have multiple sites, then to have full redundancy of operation, you need sufficient resources at each site to take up the entire slack from a single site, and sufficient network resources to ensure consistency of databases. This is all possible, but it takes sufficient effort and planning. This can get complex if your business is also expanding rapidly, as the time taken to obtain extra
Re: (Score:3)
Yeah, if you're a small shop and your web presence isn't your whole business, go for it. If being taken offline means you're out of business until services are restored, perhaps you should take it a little more seriously.
I currently host with a single provider; but, then, I'm just hosting my personal shit, so it's no big deal if it's down for a minute here and the
Re:Sorry, but... (Score:5, Insightful)
In the case of TFA, it sounds like a consumer-grade solution was applied to an enterprise-level problem. Maybe that's Google's fault, maybe that's the customer's fault -- sounds like there's plenty of stupid to go around.
Re:Sorry, but... (Score:4, Insightful)
Google cloud isn't supposed to be enterprise grade? I bet that's news to google.
If you read the fucking summary it doesn't say the servers went down, it says that some robot shut them down when it detected "suspicious activity". No review was done, nobody at google called the customer, nothing.
This is clearly 100% Google's fault.
Re: (Score:2)
Maybe it is news to Google...
https://cloud.google.com/solut... [google.com]
Why cloud storage makes sense for small and medium businesses...
Re: (Score:2, Insightful)
Google cloud isn't supposed to be enterprise grade? I bet that's news to google.
... it says that some robot shut them down when it detected "suspicious activity". No review was done, nobody at google called the customer, nothing.
Those two statements seem, at least to me, clearly contradictory. I can understand summarily shutting down a customer on a residential Internet connection, or a small business shared web hosting provider. However when providing an "enterprise grade" service, you should be prepared to give your customers the benefit of the doubt. About the only instances I can think of for an enterprise service to shut down a customer is if they are greatly exceeding their allocated resources and/or the activity
Re:Sorry, but... (Score:5, Informative)
I can understand summarily shutting down a customer on a residential Internet connection, or a small business shared web hosting provider. However when providing an "enterprise grade" service, you should be prepared to give your customers the benefit of the doubt. About the only instances I can think of for an enterprise service to shut down a customer is if they are greatly exceeding their allocated resources and/or the activity associated with the customer is actively in the process of harming other customers.
Cheapfuck customer was running enterprise-level applications on what was supposed to be a consumer-level account. That's why Google Bots detected "suspicious" actitivy and shut them down. If they hadn't tried to be cheapfucks and were using an enterprise account this most likely wouldn't have happened.
Re:Sorry, but... (Score:5, Insightful)
Google's cloud services are enterprise grade if you pay enterprise prices for them.
If you pay for it on a credit card assigned to the CFO then you are not an enterprise and you are not paying enterprise prices.
They chose a cheap, no-SLA no-support service, probably because it was cheap. Then they get upset that they aren't receiving the support they didn't pay for.
Re:Sorry, but... (Score:5, Interesting)
Ah, yes. And that is at the core of the problem. But since this apparently happened before and they did not learn from that, my guess would be they have far larger problems than the known unreliability of cheap cloud services.
Re:Sorry, but... (Score:5, Informative)
Google's cloud services are enterprise grade if you pay enterprise prices for them.
If you pay for it on a credit card assigned to the CFO then you are not an enterprise and you are not paying enterprise prices.
They chose a cheap, no-SLA no-support service, probably because it was cheap. Then they get upset that they aren't receiving the support they didn't pay for.
Exactly this. I don't use Google Cloud, but my 'enterprise' AWS service comes with a dedicated account manager and architect who I can call at anytime for help.
Sounds like this guy cheaped out then is bitching because he received cheap service.
Re:Sorry, but... (Score:5, Interesting)
I signed a previous client up for Google Cloud services. I did so using the company credit card. I was in touch with Google pre-sales people throughout, and spoke to some account management as well. In other words, I did it correctly, with their oversight throughout.
What I'm saying is... you *can* pay for 'enterprise' services from Google with a credit card. Why anyone would want to is another matter, but to get invoice billing does take a few additional steps, which my client actually failed because of a change of business address at about the same time as we were doing the application. It all got resolved, as my client was in generally good standing, but I guess if your company is a bit flaky you might not pass such checks.
By the looks of things though, this particular company was using a consumer account, which is of course the source of the problem. If it also happened more than once, you'd think they'd have learned by now.
Re:Sorry, but... (Score:5, Insightful)
If you read the fucking summary it doesn't say the servers went down, it says that some robot shut them down when it detected "suspicious activity". No review was done, nobody at google called the customer, nothing.
This is clearly 100% Google's fault.
Yes, Google was a little too quick to pull the trigger. They should have sent out a couple of e-mails first before shutting things down.
But
The "suspicious activity" was due to the cheapfuck customer running heavy, 24/7, critical applications on what is supposed to be a consumer-level account.
You get what you pay for.
Re:Sorry, but... (Score:5, Informative)
In the rare cases where enterprise services are paid for with a credit card, any and all verification for the $10k+/mo in charges is done when the contract is signed in person.
Re:Sorry, but... (Score:5, Informative)
I had a $2million/18 month defense contract paid by credit card via a random accounting person 2000 miles away. I would say you are not quite up with the times... $10k/month can be peanuts.
Re: (Score:3)
Re: (Score:3)
It is funny to read the "puny $6k /mo" denigration. When you are not exposed to these enterprise level vendors, you can really get taken aback by the scale of the thing.
I had my first experience with this in negotiating package shipping. I was dealing with UPS and FedEx, trying to get contracts in place and looking for better pricing. In Telecommunications I had really good success, with lots of players competing for my business. We were spending $50k per year (and growing) with FedEx, so I expected a
Re: (Score:2)
one colo? (Score:2)
Who do you work for? I'm divesting. Hell, as a 6 man startup in the 90's, we knew better to have only one server farm in one colo. Granted, our failover was to the developmental farm on a T-1 in our office, but it was at least *some* failover. h, and the colo texted when there was a problem, real or imaginary.
That was 6 drunk amateurs 2 decades ago.
Re:Sorry, but... (Score:5, Insightful)
Re: (Score:2)
If you do a second location, you may be inclined to go to another "availability zone" at the *same* provider, and then watch all your redundancy go poof.
Some may say "oh that's why you should have *mulptiple* cloud vendors too", which greatly increases cost and complexity when even *one* vendor is more expensive than owning your own equipment.
Re: (Score:3)
Re: (Score:2)
That's why you also configure a failover at a secondary location.
Isn't that exactly what the Google cloud is sold as - a huge 'cloud' of servers for massive redundancy?
Here, read the sales blurb: https://cloud.google.com/solut... [google.com]
Re: (Score:2)
The brochure covers up the fact that a massive "cloud" at one provider doesn't help with the single point of failure that is one provider's automated policy enforcement.
Re: (Score:2)
The failure modes in a hosting service are more insidious, when they don't want your services to be up, they take them down, all at once. No amount of being good about your availability zones and such will stand up to the reality that your vendor can shut down *everything*.
Exacerbated by the reality they have so *much* activity, they can't have humans police it so instead they have automation systems that are prone to false positives.
When you host yourself, the fundamental operational technology failure mo
Re: (Score:2)
Your vendor can't shut down everything if they don't *have* everything. That's why any cloud service redundancy worthy of the name will have backups sourced to other vendors.
Re:Sorry, but... (Score:5, Insightful)
That Amazon message is very different. It's basically telling you (days or weeks in advance!) that there's a serious hardware failure and the underlying hardware needs to be pulled from service.
It's literally as simple as a reboot to move to new hardware. You can even catch the notification easily with a CloudWatch Alarm and trigger the purpose-built auto-recovery action to do it for you the moment the instance goes into that state. Or use an AutoScale group and it'll just cycle the hardware out for you w/o any downtime or manual action. TMTOWTDI
If you aren't living by the motto, "Everything Fails, All The Time", then you're simply doing Cloud wrong. To be fair, even if you're entirely on your own physical hardware in a datacenter...you're still Doing It Wrong if you aren't counting on your hardware to always be failing all the time.
Re:Sorry, but... (Score:5, Informative)
I came here to say essentially the same thing. Three letters come to mind: S....L....A....
If you do not have an agreement with the provider that indicates the measures they will take to restore service in the event of an outage, and the escalating penalties on the provider if they fail to restore service according to the agreement, then you really have no business running a revenue-generating production service with that provider. Or, you don't have a business at all, just a hobby.
Re: (Score:2)
Moreover... build your solution with redundancy if you are going cloud! There are times it is very hard and/or expensive, but you need an order of magnitude better reliability/availability options if you don't have an SLA.
Re:Sorry, but... (Score:5, Insightful)
Ah, but SLAs cost money, and part of the cloud movement has been "see, these vendors can make hosting cheaper than it used to be", precisely because all they have to do when they screw you over is say "whoops, sorry", and as such don't need to invest in *too* much resiliancy and don't have to worry too much about liability.
Third parties (Score:2)
Never trust your data to a third party when millions are at stake -- let alone critical infrastructure reliability.
While that is reasonable advice, sometimes that isn't an option. Sometimes the only reasonable way to do things is through a third party. Furthermore sometimes the third parties can do a better job than I could do myself, even accounting for their flaws.
Re: (Score:3)
As other people have pointed out, the magic letters here are "SLA". You must have a contract stating what the vendor's responsibilities are and be able to enforce that contract. Otherwise, you don't have a business, you just have a hobby.
Contracts (Score:3)
As other people have pointed out, the magic letters here are "SLA". You must have a contract stating what the vendor's responsibilities are and be able to enforce that contract.
A contract is only as valuable as your ability to ensure it is enforceable. When you are dealing with a company the size of Google they can hire some flesh eating lawyers and have the bank account to keep you busy until you die and so if you plan to bring a lawsuit you'd better be prepared for shock and awe. Just having a contract isn't enough by itself.
You are right that a service level agreement is a very good idea but it isn't going to matter if it is cheaper for them to screw you anyway.
Otherwise, you don't have a business, you just have a hobby.
That's a nice
Re: (Score:2)
Re: Sorry, but... (Score:5, Informative)
Why is it "getting harder" to host your own solutions?
Many applications are not even available for purchase of a copy to run on your server. Instead, they are available exclusively on service as a software substitute (SaaSS) [gnu.org] terms, namely that the application runs on an application service provider's server. One example of these is the server for any major MMO game.
Re: (Score:2)
Agreed, the title should be "Why You Should Not Use The Cloud."
Re: (Score:3, Insightful)
Apparently Netflix and friends (who use these services) don't agree. Neither do I.
Running your own system doesn't fix anything. Yes, you won't go down because someone forgot to renew the credit card, but you will go down because of a faulty RAM chip, or an air-conditioner going out, or a ISP getting a route wrong, or a backhoe going through a cable. None of those failures are likely to take out Google as they will just move you
Re: (Score:2)
Good thing I don't work for Amazon. Cause I just found some synergy. Amazon video would be about the get much better, at zero cost to them.
Re: (Score:3)
One, note that just because they are a big name, it does not mean all their decisions are bullet proof guaranteed the best. Dropbox has the exact opposite story to tell.
For another, Netflix has a rather special position. They are *the* go-to reference customer for AWS. Amazon with almost every other breath references just how *awesome* Netflix is doing with AWS. As such, they assuredly have special status, Amazon is not going to just screw with Netflix because the second Netflix so much as whispers a ne
Re: (Score:3)
s3 storage is *massively* expensive at scale, compared to in house. Even among cloud providers, there are competitors that are 75% less.
Wait (Score:5, Insightful)
Why was there a second time?
Yeah, big warning signs on the user side here (Score:5, Insightful)
Why was there a second time?
So many of the problems here (ex. paying with a credit card and one that has only a single person's name on it? Having no fallback that can be spun up elsewhere?) are foolish if this has never happened before, and utterly, mind-bogglingly idiotic if this in fact has already happened before. It's one thing to be blind of something you should know could be a problem, it's quite another to be blind and wholly unprepared for a problem you've personally experienced! Something seems fundamentally wrong at this company.
Also, if your entire business can die because it takes an unexpected few days off, then perhaps your business is running a bit too raggedly and doesn't have enough meat on the bones . . .
Re:Yeah, big warning signs on the user side here (Score:5, Insightful)
There probably is just one problem here: Utterly incompetent and greedy management that made this demented decision.
Donâ(TM)t trust an ad company. (Score:3, Insightful)
Over 90 percent of Google income is adverts. You would be absolutely insane to trust them with your business or educational institution data.
Iâ(TM)m not saying MS or Amazon is great but at least their revenue model is not based exclusively or largely on data mining of users.
No mission critical activities on the internet (Score:3)
You need to design the systems such that they have a fall-back and can continue to operate without an internet connection.
Really ...
What are you going to do the next time a major blackout occurs, and the grid wants you to restart your turbines?
Amazon's cloud s no better (Score:5, Interesting)
Our company tried to use Amazon a few years ago and ran into the same issues. Although google and amazon allow you to
spin up a single instance, they are really designed for companies that have hundred if not thousands of servers. Amazon
assumes that you have dozens of fault tolerant servers and if one goes down you just replace it with another one. This works
great for companies like Netflix but Amazon is a disaster for a company that isn't fully fault tolerant and has critical servers
that can't go down. Liquidweb, Rackspace, Linode, and even Digitalocean are more reliable when it comes to wanting to
keep a single server up and running with minimal downtime. Now if you need to keep thousands of servers up and don't care
if any one server goes down then Amazon works fine.
Re:Amazon's cloud s no better (Score:5, Insightful)
... but Amazon is a disaster for a company that isn't fully fault tolerant and has critical servers that can't go down.
If your company has "critical servers that can't go down" (wherever they are) and you're not fully fault-tolerant, you're the disaster, not Amazon.
Not here to pick a fight, just sayin'. (one finger pointing at someone else is also three fingers pointing at yourself)
Re: Amazon's cloud s no better (Score:2)
What do you mean "even Digitalocean"?
Digitalocean is usually considered a low cost provider. I'm not sure they even have 24/7 phone number you can call. They are also completely self service as far as I know as is google/amazon unless you have a multi thousand dollar contract. Liquidweb and rackspace on the other hand are considered full service and you can always reach a technician if you have a problem.
Disaster Recovery (Score:5, Informative)
If an extended system outage can cause "millions of dollars in lost revenue" then you should have a DR plan. Don't put all your eggs in one basket. Have copies of everything at another site (EC2, Azure, Colo, etc) that you can turn on and switch to in this event. If millions of dollars are on the line, then it shouldn't be unreasonable to have such a plan and infrastructure established.
Re:Disaster Recovery (Score:4, Insightful)
They did not have a DR plan _after_ this happened before. They did apparently not even buy enterprise grade cloud services, you know, those with an SLA. The problem is not the cloud here, but the utter morons that decided to use it in that way.
Re:Disaster Recovery (Score:4, Insightful)
No.
Relying on a single company no matter what their internal redundancy is, is not having a good DR plan. Especially when the amount of revenue involved his so high.
As others have pointed out, the other cloud providers do the same thing. You can have your stuff spread all across the country/world with a single provider and if a glitch in their system says you shouldn't have service, it will all be turned off (as is presented in the TFA and from others suggesting at least Amazon has done the same thing).
Re: (Score:3)
Then of course when something goes wrong with that it isn't enough people will be like ah ha! You actually should have used multiple providers
Exactly; one of the core selling points of "cloud" services is that they're supposed to offer increased fault tolerance (e.g. a virtual server can be restored in a different data center if one data center explodes) ... so now these knee-jerk Google defenders are telling us we need 'multiple cloud providers - let's call this new service a "cloudcloud" - so when your cl
Comment removed (Score:3, Interesting)
Re: (Score:2, Insightful)
This is a good time to remind people that XKCD wrote that comic to justify the forceful expulsion from the entire Internet of ordinary people who disapproved of the corruption of a gang of liars who happened to be connected to multiple foreign intelligence agencies, including state sponsors of al-Qaeda. And it turns out they own all of the press too, given how nobody has told this story correctly.
All of that "Gamergate harassment?" There was one tweet. ONE. It was sent by Slad
Re: (Score:3)
Let me get this straight. You're asserting that Donald J Trump and XKCD are sponsoring al-Qaeda?
Re:It makes sense why Google is like this (Score:5, Insightful)
This is a good time to remind people that XKCD wrote that comic to justify the forceful expulsion from the entire Internet of ordinary people
No it's a good time to remind people you've gone way off the deep end, mate.
It does however prove that just aobut anything in favour of gamergate no matter how batshit insane will get modded up here. Like this bit:
of al-Qaeda.
Aside: isn't that sort of old news even for the crazies? Aren't ISIS responsible for chemtrails now or are they merely a false flag perpertrated by the deep state t ostop us knowing the truth about how a chemtrail spraying plane actually did 9/11?
All of that "Gamergate harassment?" There was one tweet.
Gamergate was one tweet: +3 Insightful. I think that might be a new low for slashdot moderation.
Re:It makes sense why Google is like this (Score:4, Informative)
A simpler and more likely explanation is that you paid $0 for the service and got your money's worth.
There are also lots of obvious examples that disprove your conspiracy theory. If it was politically motivated then what kind of politics wants to silence both the far right and progressives and non-political trans make-up videos and help the copyright industry abuse people?
s/Google// (Score:3)
Seriously. When someone else owns and operates your infrastructure, things like this are going to happen. When that someone earns their revenue from something other than the bill from them you pay every month, it's going to happen a lot more often because they'll be acting based on what's good for their business, not what's good for yours. This is life on any cloud platform. This was life with mainframe service bureaus back when they were the cloud platform of choice.
You have to make a call based on what the trade-offs are. Make sure you know what those trade-offs are going to be, bearing in mind that any contract you have is probably going to say the provider's only responsible for refunding your month's payment no matter what the cost to you of their mistake was. It's that that you're balancing against the cost of running your own hardware, not the monthly bill.
What should they have done? (Score:2)
Turned off their automated systems, and who ever caused the flag to be raised with your Google Payments Account gets in and takes over your entire system, maxes out your CFO's credit card?
Points and laughs, HA HA! (Score:2)
Put all your infrastructure under the physical control of some other entity well beyond your reach and then discover they can summarily turn it off and refuse to respond - Duh!
The Cloud Stikes Back!
Redundancy (Score:3)
You should not have ANY one single point of failure.
Only 1 card holder? Single point of failure.
More importantly: Only 1 cloud provider? Single point of failure.
If you're running that level of cash, and still insist on outsourcing infrastructure, then fucking distribute it. Mirror the infrastructure between AWS, GCloud, and Azure. Even these companies themselves know this. Look up Amazon's DNS providers. Hint, its not JUST AWS, but they outside their own shit too *JUST IN CASE* their servers go offline.
Your failure to plan... (Score:5, Insightful)
I have an email asking me to fill in a form and upload a picture of the credit card and a government issued photo id of the card holder. Great, let's wake up the CFO who happens to be the card holder. What if the card holder is on leave and is unreachable for three days?
Uh, I don't know - take a picture of each and save them on your phone, in case you need them?
You report everything was back up within 20 minutes once you submitted the requested information - that seems pretty good to me.
Now, about your decision to only run one instance of your mission critical application suite on exactly one cloud service...
The story here is you consider it someone else's fault for your failure to plan/prepare for an outage.
Not a Google Cloud problem. (Score:2)
This is not a problem with Google Cloud, this is a problem with all "cloud" platforms. It's really simple, they can be held liable so they put acquit ass-covering in the contract so that they can shut you down on a whim. If this doesn't work for you then you should not any "cloud" platform.
This is just an example of reality catching up to all the idiots who said "put it in the cloud!" while ignoring all the risks. Play with fire and you'll eventually get burned.
Re: Not a Google Cloud problem. (Score:3)
This is not a problem with Google Cloud, this is a problem with all "cloud" platforms. It's really simple, they can be held liable so they put acquit ass-covering in the contract so that they can shut you down on a whim. If this doesn't work for you then you should not any "cloud" platform.
Not true. There are plenty of full service providers like liquidweb and rackspace that won't pull the plug on your server. I've had servers act up and I immediately get a phone call. In severe cases they might even disconnect the network until they can contact you and resolve the problem but they aren't going to destroy your data or even disable your computer without first contacting you. Even in cases of spam they will work with you and try to fix the problem and unless you really are a spammer they wo
What exactly happened? (Score:3)
Maybe I just didn't read enough, but it seems like he doesn't say anywhere exactly what happened. He implies it was a billing issue. That's all. Without knowing exactly what went on, it's very hard to care. I imagine it's something like "well the credit card details changed, oh, and we were 107 days overdue."
Also, millions of dollars are on the line for short downtime and you're billing to a credit card?
New scam incoming! (Score:5, Insightful)
I have an email asking me to fill in a form and upload a picture of the credit card and a government issued photo id of the card holder. Great, let's wake up the CFO who happens to be the card holder. What if the card holder is on leave and is unreachable for three days? We would have lost everything -- years of work -- millions of dollars in lost revenue.
Somewhere in Russia, India and Nigeria, several callcenters full of scammers came all at once.
Read your contract and SLA (Score:2)
I hate to blame the messenger, but either you didn't buy the right service level agreement or Google broke the contract.
If it's the first case, blame yourself and learn a lesson. You get what yo pay for. If Google doesn't offer the level of service you need, go elsewhere. If they do, either pay up or go elsewhere.
In the second case, you are rightfully upset but you should be talking to lawyers before talking to Slashdot.
Welcome to the age of digital serfdom. (Score:2)
365 vs. Apps Story (Score:2, Informative)
As a consultant a typical war I get into with customers is to pick the cloud setup email, drive etc... I always recommend Microsoft instead of Google products, and I have to always remind people that, altough the strong brand name, Google is an advertising company not an Enterprise partner, and I have countless stories of google pulling the plug on services because of "reasons" whatever, also, they have no respect for the customer when they drop a product, they just send an email with a month notice, and th
Also... (Score:3)
1) Don't trust another company with your critical IT infrastructure!
2) Have redundant facilities with different ISPs. 3) Have tested backup/standby power systems.
Yes, it is expensive, but - how much would it cost you to be down a week? A month? There is no free ride.
In summary (Score:2)
This company sucks at running a business.
If millions are on the line (Score:2)
You should think about divesting from just-google. The cloud is costing you more already, get yourself a number of real servers with real hosting providers dispersed geographically. Running something solely on Google or Amazon clouds is technically identical to hosting everything on a single server.
Comment removed (Score:3)
Don't use Google Cloud's... (Score:3, Insightful)
... cheapest service they offer, the one that doesn't include 24/7 phone support - let alone a guaranteed SLA, to host your multi-million dollar wind/solar plant, where any service outage will cost you millions in service penalties.
If you look at the first comment in TFA (Score:5, Informative)
This sounds like asking fro trouble to me!
not a failure (Score:3)
Frack that. never. not 1 dollar.
Re: (Score:2)
Are you saying he should have used Google Wind [youtube.com] instead?
Re: (Score:2)
They can't knock out the connection between a local monitoring server and the local equipment.
In this case, they might have lost *a* power plant remotely, but could still reach out to the local personnel and *they* would still see monitoring.
Also, the same ISP can knock out your ability to manage your cloud hosted infrastructure, so the cloud adds a party to screw you over without removing any.