Faulty Patch Freezes Millions of UK Bank Accounts 155
frisket writes with news from The Register about ongoing problems for some UK banks: "'RBS and Natwest have failed to register inbound payments for up to three days, customers have reported, leaving people unable to pay for bills, travel and even food. The banks — both owned by RBS Group — have confirmed that technical glitches have left bank accounts displaying the wrong balances and certain services unavailable. There is no fix date available.' Customers of NatWest subsidiary Ulster Bank in Ireland have also been left without banking services. RTE reports that 'the problem had arisen within the systems of parent bank RBOS when an incorrect patch was applied.'"
Comment removed (Score:5, Funny)
Re: (Score:2)
Made in the US. Only for the US. The rest of the world can get its own miracle cure.
Re: (Score:1)
Made in the US. Only for the US. The rest of the world can get its own miracle cure.
And now we can't even get our money to buy it. (This also affects a number of smaller building societies and companies that use RBS to process payments)
Re: (Score:2)
That's why you should always use RBN [wikipedia.org] not RBS [wikipedia.org] for your "Transactions"
</wikitard>
Re: (Score:2, Informative)
STOP IT. Do not turn that in to a meme. You are rewarding them for spamming.
They outsourced their IT dept... (Score:4, Insightful)
... getting rid of all the expensive people with experience in the mainframe backend system...
Re: (Score:1)
http://www.theregister.co.uk/2012/06/22/rbs_natwest_outage_fourth_day/
Re: (Score:2)
Re:They outsourced their IT dept... (Score:5, Informative)
Re: (Score:2)
A later commenter did: http://www.computerweekly.com/news/1280093677/Royal-Bank-of-Scotland-cuts-1000-IT-jobs [computerweekly.com]
The Unite union says RBS plans to offshore upwards of 500 technology roles to its existing IT operations in the US, India and the Far East.
Re: (Score:1)
This is only interesting if it's true. Could you cite your source?
http://www.computerweekly.com/news/1280093677/Royal-Bank-of-Scotland-cuts-1000-IT-jobs
-A different AC
Re:They outsourced their IT dept... (Score:5, Informative)
I had the same question, and then saw this post [slashdot.org] further down, referencing this story [computerweekly.com] from 2010 (which in turn points to another story [computerweekly.com] discussing this as an ongoing strategy). Seems to have validity.
Re: (Score:2)
I read the GP as saying that the story had validity given he's provided 3 links as sources. Maybe hold off on the fucking retard comments until you are sure you aren't yourself a fucking retard?
Re: (Score:2)
Re:They outsourced their IT dept... (Score:5, Insightful)
And how much money do you think they're saving with their outsourced people today?
Since when biterness .... (Score:2)
... is just expressing the truth.
The fact is that many (all?) banks outsource whole departments to "reputable" companies where the attrition rate is high, so what happens is that your systems end supported by a guy that had a week of certification on a given technology but no experience whatsoever with highly sensitive business systems.
If explaining this reality is biternees, so be it.
Aargh (Score:4, Funny)
I always mess up some mundane detail.
Re:Aargh (Score:5, Funny)
This is not a mundane detail, Michael!
it's obvious (Score:2, Insightful)
They're doing tests on locking down funds transfer & electronic payments systems. This is probably much harder than you'd think, because they're designed to, well, just work. A few weeks ago (5th June, to be precise) a similar thing happened in Belgium. Caused chaos on the railways.
If anyone thinks it isn't a rehearsal for when Greece drops out of the Euro then I've got a nice bridge for sale.
Re: (Score:1)
That looks interesting. Would you mind providing links on which we can read more about on that case?
Re:it's obvious (Score:4, Funny)
A few weeks ago (5th June, to be precise) a similar thing happened in Belgium. Caused chaos on the railways.
What, SNCB suddenly had trains leave on time?
Re: (Score:2)
The trains, yes. The passengers, no. They were all in line trying to pay with cash at the single manual sales point, because if you board without a ticket they fine you.
Re:it's obvious (Score:4, Funny)
I've got a nice bridge for sale.
ahh, so you're the guy who bought it! Big of you to own up.
Re:it's obvious (Score:5, Funny)
If anyone thinks it isn't a rehearsal for when Greece drops out of the Euro then I've got a nice bridge for sale.
I'd take you up on that offer except my bank account seems to be locked. However, I have a good friend in a Central African state that has access to many large bank's internal systems. He has generously offered to transfer the funds through his contacts. All he needs is your banking account number and routing data and he will gladly oblige us both.
Simply reply to this communicationl with the appropriate information.
Thanking you in advance.
Re:it's obvious (Score:4, Insightful)
I honestly don't think you're right. I've worked in enough "brack & mortar" companies that have slowly over time transferred all of their operations and value to IT so that nobody in the company outside of IT clearly knows how the system is supposed to be working.
This would all be well and good if only the management - the very people that ordered the transfer - were aware of it. But no, they still see the value in all the little people doing nothing in their offices. After all, IT is a cost and doesn't bring money in ! Need to cut costs? Lay off those geeks. They serve no purpose.
And all of a sudden, you've lost the only people that keep the entire company afloat.
Because what these people need to realise is that the ONLY people needed in the company are the IT. You can layoff ALL OF THE REST OF THEM and the company will keep on running. Badly, not as efficient, maybe at 10% of its capacity. But the IT people can take over as a cashier, delivery boy, salesmen, etc. Those jobs are complex, if you want to be efficient at them. But anybody can do them badly.
Lay off ALL OF IT. You company dies right there. End of the story. Because nobody else in the company can understand the first thing about IT. They wouldn't even know how to log in the production servers.
They outsourcedtheir IT? Equivalent of selling off the company. They outsourced to some random dudes in some place they cannot even reach? They killed themselves.
obviously you have not spent much time (Score:2)
as a cashier, salesman, delivery boy, etc.
RBS was a too-big-to-fail bank (Score:3)
lest we forget, a huge number of the subprime mortgage securites (CDOs) had the "RBS" name on them. RBS only exists because the taxpayers kept it afloat. it was 'too big to fail'.
the fact that it would fail again in some catastrophic way is not surprising. it is like watching a hippopotamous shit all over the floor and then scratching your head and asking "why did the hippo shit all over the floor?". because thats what hippos do. they shit on the floor. if you want to stop having to wipe hippo shit out of
Re: (Score:2)
You think I cannot be any of those roles in any kind of structure? You would be wrong.
Use me as a CEO of any company. I'd do a bad job at it no doubt, but I'm pretty sure I won't run the company into the ground in a year.
Use me as a salesman for anything. I'll sell some.
Use me as a cashier. I'll do it.
Legal I may be very wrong.
Remember the context: I need to be in this role at a company for which I've built the IT.
The bottom line is: IT is a very very very technical job. Normal people cannot start to unders
have you ever actually done any of those jobs (Score:2)
have you ever actually been a cashier?
have you ever actually been a file clerk?
have you ever actually been a mail clerk?
have you ever actually been a salesman?
if not, then i suggest you go try it. you will learn a lot.
Re: (Score:2)
Yes I've been. And you apparently are under the impression that I am discarding those jobs as being of no value while that's not at all what I'm saying.
Anyone can be a bad salesman by replacing a salesman in case of an emergency. Anyone can be a bad file clerk. Anyone can be a bad cashier.
That's easy. Now being a good salesman/cashier/clerk is not easy at all. And there's value to that.
But take a cashier and tell him to fix your critical application that just failed. You'll see the difference btw taking a c
anyone can be a bad computer scientist (Score:2)
the system can fail regardless of how 'genius' the person you stick on the job.
witness the financial crash of 2008 - hundreds of PHDs were involved in creating CDOs, Synthetic CDOs, RMBS, and giving AAA+ ratings to migrant farm laborers with $300,000 home loans. all those geniuses didn't help the system one single bit, not at all, not one iota.
Re: (Score:2)
IT was up and running the whole time. The computers faithfully reported the nosediving financial indicators. The networks all stayed up. That crash took place in the business side where networking means gladhanding and golfing. Where people tell everyone else that only they can keep things running because they're more powerful than God and that's why they should have the GDP of a small country as a salary even though they're never in before 10 and never stay past 3.
Had it been turned over to IT people the c
Re: (Score:3)
I'd bet on something far more pragmatic. I work for one of their rivals, so I'm sure our systems are broadly comparable.
They've outsourced their mainframe support to an Indian company. They're probably running a similar mainframe set-up to what we are. From the snippets of non-technical fluff that their spokesperson was spouting on Radio 4 yesterday, it sounded like it was their mainframe's daily update programme (which applies changes to customer accounts that have been received in the queue from the mainf
Re: (Score:2)
I've never held much faith in Hanlon's razor.
I'd say that any sufficiently advanced conspiracy is indistinguishable from a cock-up.
What idiotic nonsense. (Score:2)
The only thing you need to do to lock down funds is to shut down the website down in purpose, no need of conspiracy theories.
food! (Score:4, Funny)
Re:food! (Score:5, Insightful)
You know, typo's aside, this is tragic. Many people live paycheck to paycheck. I used to, up till fairly recently, and I'd still be hosed if this happened because my rainy day money is not in a bank fund.
I would be calling for people to be put against the wall for this.
-nB
Re: (Score:2)
You know, typo's aside, this is tragic. Many people live paycheck to paycheck. I used to, up till fairly recently, and I'd still be hosed if this happened because my rainy day money is not in a bank fund.
I would be calling for people to be put against the wall for this.
If you can show you should have been paid (e.g. payslip, last month's statement) they are giving you £300 if they can't access your account. The bank branches stayed open extra late yesterday, opened at 8am today, and will be open for long hours all weekend.
Re:food! (Score:5, Insightful)
This is not people's poor choices causing them hardship, this is their bank fscking the system hard.
Like I said, even I, with a 12 month rainy day fund, would be in a world of hurt, because I would not be able to put the money into the account to pay bills with, even if I could get it out from where it was.
Someone living wholly beyond their means is one thing. A bank not posting deposits is entirely different. Different, and unforgivable.
-nB
Re: (Score:2)
So, do you keep all of your assets in gold under the mattress?
How do you pay bills?
Do you live in the woods in a cabin with no services (ala Ted Kazinski)?
Re: (Score:2)
There is only one person who can be put to the wall for failure to control your own finances.
Most people in the UK are paid by an electronic transfer of the money into their bank account. If your bank is NatWest you can get £300 from them by proving you should have been paid, but I'm not sure what you can do if your employer's bank is Natwest.
you are saying this to a taxpayer funded bank (Score:2)
that literally exists only because taxpayers funded it through government enforced bailouts.
methinks your anger is misplaced. please find an arena where the people who run RBS, Goldman, Morgan, Deutsche, and all of the other 'too big to fail' banks were propped up by taxpayer funding, and tell them that they should be "put to the wall".
That's one way ... (Score:5, Funny)
To prevent another run on UK banks.
Re: (Score:3)
Or cause one, once things come back up.
Re: (Score:2)
Actually you over estimate the intelligence of the GDP, generally dumb public. .
That explains why they say that the USA has a higher GPT per capita than Europe.
Re: (Score:1)
Actually you over estimate the intelligence of the GDP, generally dumb public. .
That explains why they say that the USA has a higher GPT per capita than Europe.
Damn keyboadr!
I keep getting headhunted by RBS/Citizens (Score:2)
Re: (Score:1)
And I've always wanted nothing to do with them. Banking in general is very distasteful to me.
So they were looking for competent help and you refused them. Thus... we can conclude this is all your fault! FOR SHAME!
Re: (Score:2)
Actually, it is the fault of management for creating a lousy work environment.
test labs (Score:2)
Re:test labs (Score:4, Interesting)
For our critical stuff we hav 4 layers to get to production:
1) I do Foo and test it on my dev machine on an unsigned system
2) I submit my Foo to the build system, it builds it for unsigned systems and it is tested by our QRE/Validation department
3) Once things look good, it is signed, then deployed to our Validation dept to run on signed systems.
4) If it is still looking good, then it is deployed.
And we're not even a banking related operation...
-nB
Re:test labs (Score:5, Insightful)
Re: (Score:2)
But that costs MONEY! If they keep doing things that cost MONEY, how will they pay out the huige executive bonuses?
Wow. (Score:2)
Well, they've always looked like a bunch of clowns.
They were always a bit useless. Then they kind of bought each other up to increase the lavel of failure when they went bust.
It's government owned now, but you can't blame the government for this computer foul up (there are many other even more spectacular ones), since it was all done by them before the bail out anyway.
The UK retail banking industry is a poster child for why private industry is every bit as crap as a government, and in fact often worse. The
Re: (Score:2)
[roman_mir]It's all the government's fault because they wouldn't let the banks go bust and people lose all their money. Vote Ron Paul![/roman_mir]
Annoyed customer (Score:3, Interesting)
Thankfully I don't live week to week off my wage like some people do, but if I did I'd be having major problems as evidenced by some of the BBC [bbc.co.uk] stories.
Re: (Score:3, Insightful)
Even though you were not impacted, you should still move your money to another bank ASAP.
Re: (Score:1)
Don't take the description at face value.
I think it is more probably a premature application of some anti-bank-run version. There are financial earthquakes coming soon to Europe, one way or the other, and that always means bank runs and naked cheques.
I think it is more plausible that they just flipped the "do not trust other counterparty until really verified" switch too early.
Re:Annoyed customer (Score:5, Insightful)
Natwest are not alone, another british bank (Barclays) has often been reported as having Monday-morning outages, which sounds a lot like a weekend update that went wrong.
As it is, having a single account is like having a single credit-card, no spare car key or only one kidney. You can get by until something goes wrong, but in an ideal world you'd have at least one spare.
Re: (Score:2)
customers are the problem (Score:2)
look, all of the 'disgruntled customers' like yourself may be upset, but are you upset to take your money out and go to a smaller institution? one that charges slightly higher fees and has less profitable interest rates, but that actually takes it's job seriously?
this is the situation in the US with people who 'hate big banks'. if they 'hate' them so much, there are literally tens of thousands of small banks and credit unions - and they are insured up to $250,000 if you are worried about losing your money.
t
Re: (Score:2)
Guess this was inevitable.... (Score:5, Insightful)
When you remove a 1000 members of IT staff [many of which were probably your best] and replace them with 500 offshore workers combined with the need to support *legacy* systems, you are asking for trouble.
http://www.computerweekly.com/news/1280093677/Royal-Bank-of-Scotland-cuts-1000-IT-jobs [computerweekly.com]
Regardless of the technical problems, the root cause of this seems to be management......
Re:Guess this was inevitable.... (Score:5, Insightful)
Regardless of the technical problems, the root cause of this seems to be management......
Management always gets the credit when things go well. They should always get the blame when things go wrong.
Re: (Score:3)
Regardless of the technical problems, the root cause is pretty much ALWAYS management......
FTFY.
Fix has already been made (Score:5, Informative)
The summary says there's no fix date available, and I know that's what it says in the Register article, but the second article, linking to the bank's site, has this to say:
The bank says the issue has now been fixed but it will take the weekend to clear the backlog which amounts to millions of euro in transactions.
Re: (Score:1)
First thing to do when the fix happens ... (Score:1)
... move your money to another bank. I hope in UK they have smaller local banks like we have here in the former colonies. If not, maybe find some favorite small towns in the USA.
Re: (Score:1)
Why? What does that achieve, exactly?
We used to have building societies but most of them converted into banks in the 90's, then they all bought each other, then they merged into the big banks or went bust. So no, not really.
Re: (Score:1)
Co-operative Bank would be the best bet.
And you would move your account to tell the bank that incidents like this aren't acceptable for a bank. That cutting experienced IT staff and outsourcing is not a good idea in the long run.
Re: (Score:2)
Actually, since this has happened and been fixed at this bank, it's now the place to keep your money to avoid the problem. At other banks, it just hasn't happened yet. :/
Cockup or conspiracy (Score:1)
Cockup or conspiracy? Probably cockup - but quite likely they'll discover they've made more by hanging on to cash for a few extra days than this will cost them in customers.
I switched off NW/RBOS when I (banked with them since 1978) joined their online banking and saw .asp in their URL.
Re: (Score:2)
Cockup or conspiracy? Probably cockup - but quite likely they'll discover they've made more by hanging on to cash for a few extra days than this will cost them in customers.
Not with £300 per customer compensation as their first offer.
I switched off NW/RBOS when I (banked with them since 1978) joined their online banking and saw .asp in their URL.
ASPX, not ASP. And I personally don't see an awful lot wrong with ASP.NET MVC. I'm assuming they didn't use vanilla ASP.NET, because their pages aren't using the witless state tracking mechanism that ordinary ASP.NET pages use.
Problem and fix (Score:5, Informative)
It would appear that an update to CA-7 resulted in the actual schedule for these being corrupted or deleted. Therefore they do not know how much any customer actually has in their account, since accounts were not updated with transactions from the previous day.
The problem now appears to be fixed (read: update backed out and control datasets restored), but they still have to run through three days of unprocessed transactions, so people are not getting money paid in during these three days into their accounts as expected, resulting in misery.
This is something which should have been detected and fixed in a competent mainframe site very quickly indeed, so I imagine that the wisdom of outsourcing any "back-office" function of this nature is shortly going to be a matter of very close scrutiny.
Hope this helps.
Re: (Score:2)
Re: (Score:2)
Re: (Score:1)
That makes no sense at all. I'm sure Gmail processes more emails (transactions) and saves them on many distributed places so as to never lose any data, in real time, without processing batch transactions overnight.
Where is the complexity of processing transactions? it's one addition per transaction, my desktop pc can do millions of additions per second. Surely a very safe database will do less with all the disk access but still.. overnight? for 4 million customers? 10 - 20 million transactions tops? there m
Re: (Score:1)
Re: (Score:2)
OK. Here we go. The SWIFT message arrives. The bank initiates the payment. The SWIFT message is parsed - it's a complex and flexible layout. The sender may have made mistakes. It might need an auto repair, it might need a manual repair. Now we know what it is. Next, check it against the many, many regulatory issues. Does it look like fraud? Does it look like terrorist funding? Does it match the account's usual patterns? A
Re: (Score:2)
The wisdom of outsourcing will not be questioned because they can just blame and sue the outsourcing company. No blame will be attached to the fools responsible for outsourcing (if they're even still there).
consequences (Score:5, Interesting)
Well, I don't know if it's still the case, but when I worked in banking IT in the late 80s here in the US there was a standing rule: if you don't process checks for more than 24 hours, you can be taken over by the Federal Reserve--where that takeover implies the possibility of being shut down and your assets distributed to other banks.
That really kept the fear of god in management with regard to keeping core IT running, backups, disaster recovery, etc. Daily offsite backups, periodically loading the backups at a backup facility and running test loads...
There should still be such a rule, and it should apply to electronic transactions as well as checks (not much difference anymore anyway), and the UK ought to adopt it. If a bank takes down its main system with a fucked-up patch, and can't get its disaster recovery plan working in 24 hours, shut it down.
Re:consequences (Score:4, Interesting)
I'd go a step further and limit the size of banks to something the government can reasonably deal with should things go south.
If the biggest banks had tens or hundreds of millions in deposits it wouldn't be such a big deal when one of them has an issue like this. Just have the government step in and take care of the people affected, and then fine the living daylights out of whoever messed up once it is all sorted out.
The problem is that these banks are getting so large that you can fit those responsible for 80% of the world's wealth into a single room. That means that they talk to each other, adopt similar policies, and so on. We have far more systemic risk, and even when you get a random failure like this the impact is enormous.
Re: (Score:2)
I'd go a step further and limit the size of banks to something the government can reasonably deal with should things go south.
I'd call that a different subject rather than a step further ;-) I believe that history will eventually come to see the single greatest failure of the Bush administration was not forcing a break up of the large failing banks when they had the chance.
and Obama did what exactly? (Score:2)
Fannie and Freddie - still going strong. Executives still making millions.
JP Morgan - still dealing in Credit Default Swaps, with absolutely zero consequences
Morgan Stanley - still churning out shitty IPOs like Facebook that rip people for millions, with absolutely zero consequences
Goldman - still being Goldman
the hedge funds and investment banks can now give unlimited anonymous money to political campaigns. including banks that are essentially 'international' with no allegiance to the US or its constitutio
Re: (Score:2)
The point, which you seem to be willfully ignoring, is that the Bush administration had a unique opportunity to force change and squandered it. The Obama administration has had no such opportunity.
Re: (Score:2)
I'd argue that Obama had a chance when he first took office, but there was no way he'd make any traction on that and healthcare. Politically not tackling healthcare would have been suicide - not that many are happy with the results.
The problems in the US are far bigger than who is in the White House. Heck - there was an avalanche of calls from constituents opposing the bailouts and it was basically ignored. And only a huge emergency seems to motivate any action at all on these issues. I'm not convinced
Re: (Score:2)
I'd argue that Obama had a chance when he first took office...
And that argument would be complete bullshit. By the time Obama took office 2/3 of TARP funds had already been disbursed; the "too big to fail" banks had already been purchased by other big banks, creating even bigger than too-big-to-fail banks. The unique opportunity was when the big banks were on the verge of failure and required government money to survive. Bush had the chance then to use that leverage to force acceptance of change; Obama has had no opportunity that is even remotely comparable.
no, it's not still the case (Score:2)
" I worked in banking IT in the late 80s"
if you want to know how things have changed, id start with any of the recent crisis books - "The Big Short" by Lewis, "Fool's Gold" by Tett, "Colossal Failure of Common Sense", by McDonald, "On the Brink", Paulson, "In Fed We Trust", Wessel, etc etc etc.
The stories about Citigroup and it's incestuous relationship with the high offices of government are of particular interest.
The Federal Reserve is kind of a joke as far as regulation goes. It does a lot of mole whack
Re: (Score:2)
It does a lot of mole whacking, but you have to understand, we gave 2 trillion dollars to shitty banks that in essence, could not process transactions.
Bullshit. They were perfectly able to process transactions, and what you've said has no bearing whatsoever on the regulation I was discussing.
I agree the bailout was botched. But your "in essence" is way too far of a reach.
Re: (Score:2)
Well, I don't know if it's still the case, but when I worked in banking IT in the late 80s here in the US there was a standing rule: if you don't process checks for more than 24 hours, you can be taken over by the Federal Reserve--where that takeover implies the possibility of being shut down and your assets distributed to other banks.
RBS Group has already been taken over by the government; they can't take them over any more than they already have.
And the reason they were taken over is because they were "too big to fail"- so no real possibility of chopping them up and distributing their assets. If they could have done that, they could have just let them collapse and be picked over by their rivals in the first place.
Re: (Score:2)
that's not a bug.. (Score:2)
Can I Write a Cheque? (Score:1)
Can I write a cheque? You do take cheques don't you?
Bitcoin (Score:1)
Suddenly Bitcoin becomes a real solution to avoid this kind of problem...
Re: (Score:2)
Japan (Score:1)
has some brilliant banking laws. A bank is required to be able to give a customer up to every cent of his money almost 24 hours per day immediately. This has resulted in faster money flow, better competition in banking, and a generally smoother economy. There are other perks like ATMs that cash out in denominations down to a single yen. Finally banks are unable to pull shit like creating branch withdrawal limits to screw over customers.
In Japan if I had a million dollars in an account and went to withdraw i
Re: (Score:2)
Re: (Score:3)
Sometimes there's a limit to what you can test. Simulating a multilevel network with very broad trees, as a big bank probably uses, probably makes do with a significantly simplified environment. All it takes is some yutz somewhere to try to freeze an account that's already frozen (or do something else obtuse that's ver
Re: (Score:2)
Re: (Score:2)