Forgot your password?
typodupeerror
Bug The Almighty Buck United Kingdom Upgrades IT News

Faulty Patch Freezes Millions of UK Bank Accounts 155

Posted by timothy
from the cuba-will-send-doctors dept.
frisket writes with news from The Register about ongoing problems for some UK banks: "'RBS and Natwest have failed to register inbound payments for up to three days, customers have reported, leaving people unable to pay for bills, travel and even food. The banks — both owned by RBS Group — have confirmed that technical glitches have left bank accounts displaying the wrong balances and certain services unavailable. There is no fix date available.' Customers of NatWest subsidiary Ulster Bank in Ireland have also been left without banking services. RTE reports that 'the problem had arisen within the systems of parent bank RBOS when an incorrect patch was applied.'"
This discussion has been archived. No new comments can be posted.

Faulty Patch Freezes Millions of UK Bank Accounts

Comments Filter:
  • Solution: (Score:5, Funny)

    by Cornwallis (1188489) on Friday June 22, 2012 @10:34AM (#40412171)

    Where are those cleanmypc.com ads when you need them?

    • Made in the US. Only for the US. The rest of the world can get its own miracle cure.

      • by Chrisq (894406)

        Made in the US. Only for the US. The rest of the world can get its own miracle cure.

        And now we can't even get our money to buy it. (This also affects a number of smaller building societies and companies that use RBS to process payments)

    • Re: (Score:2, Informative)

      by Anonymous Coward

      STOP IT. Do not turn that in to a meme. You are rewarding them for spamming.

  • by Anonymous Coward on Friday June 22, 2012 @10:39AM (#40412233)

    ... getting rid of all the expensive people with experience in the mainframe backend system...

  • Aargh (Score:4, Funny)

    by Anonymous Coward on Friday June 22, 2012 @10:39AM (#40412241)

    I always mess up some mundane detail.

  • it's obvious (Score:2, Insightful)

    by Hognoxious (631665)

    They're doing tests on locking down funds transfer & electronic payments systems. This is probably much harder than you'd think, because they're designed to, well, just work. A few weeks ago (5th June, to be precise) a similar thing happened in Belgium. Caused chaos on the railways.

    If anyone thinks it isn't a rehearsal for when Greece drops out of the Euro then I've got a nice bridge for sale.

    • by Anonymous Coward

      A few weeks ago (5th June, to be precise) a similar thing happened in Belgium. Caused chaos on the railways.

      That looks interesting. Would you mind providing links on which we can read more about on that case?

    • by GNious (953874) on Friday June 22, 2012 @11:26AM (#40412941)

      A few weeks ago (5th June, to be precise) a similar thing happened in Belgium. Caused chaos on the railways.

      What, SNCB suddenly had trains leave on time?

      • The trains, yes. The passengers, no. They were all in line trying to pay with cash at the single manual sales point, because if you board without a ticket they fine you.

    • by petes_PoV (912422) on Friday June 22, 2012 @12:00PM (#40413423)

      I've got a nice bridge for sale.

      ahh, so you're the guy who bought it! Big of you to own up.

    • by ColdWetDog (752185) on Friday June 22, 2012 @12:42PM (#40413961) Homepage

      If anyone thinks it isn't a rehearsal for when Greece drops out of the Euro then I've got a nice bridge for sale.

      I'd take you up on that offer except my bank account seems to be locked. However, I have a good friend in a Central African state that has access to many large bank's internal systems. He has generously offered to transfer the funds through his contacts. All he needs is your banking account number and routing data and he will gladly oblige us both.

      Simply reply to this communicationl with the appropriate information.

      Thanking you in advance.

    • Re:it's obvious (Score:4, Insightful)

      by Pieroxy (222434) on Friday June 22, 2012 @03:29PM (#40416195) Homepage

      I honestly don't think you're right. I've worked in enough "brack & mortar" companies that have slowly over time transferred all of their operations and value to IT so that nobody in the company outside of IT clearly knows how the system is supposed to be working.

      This would all be well and good if only the management - the very people that ordered the transfer - were aware of it. But no, they still see the value in all the little people doing nothing in their offices. After all, IT is a cost and doesn't bring money in ! Need to cut costs? Lay off those geeks. They serve no purpose.

      And all of a sudden, you've lost the only people that keep the entire company afloat.

      Because what these people need to realise is that the ONLY people needed in the company are the IT. You can layoff ALL OF THE REST OF THEM and the company will keep on running. Badly, not as efficient, maybe at 10% of its capacity. But the IT people can take over as a cashier, delivery boy, salesmen, etc. Those jobs are complex, if you want to be efficient at them. But anybody can do them badly.

      Lay off ALL OF IT. You company dies right there. End of the story. Because nobody else in the company can understand the first thing about IT. They wouldn't even know how to log in the production servers.

      They outsourcedtheir IT? Equivalent of selling off the company. They outsourced to some random dudes in some place they cannot even reach? They killed themselves.

      • as a cashier, salesman, delivery boy, etc.

        • lest we forget, a huge number of the subprime mortgage securites (CDOs) had the "RBS" name on them. RBS only exists because the taxpayers kept it afloat. it was 'too big to fail'.

          the fact that it would fail again in some catastrophic way is not surprising. it is like watching a hippopotamous shit all over the floor and then scratching your head and asking "why did the hippo shit all over the floor?". because thats what hippos do. they shit on the floor. if you want to stop having to wipe hippo shit out of

        • by Pieroxy (222434)

          You think I cannot be any of those roles in any kind of structure? You would be wrong.

          Use me as a CEO of any company. I'd do a bad job at it no doubt, but I'm pretty sure I won't run the company into the ground in a year.
          Use me as a salesman for anything. I'll sell some.
          Use me as a cashier. I'll do it.
          Legal I may be very wrong.

          Remember the context: I need to be in this role at a company for which I've built the IT.

          The bottom line is: IT is a very very very technical job. Normal people cannot start to unders

          • have you ever actually been a cashier?

            have you ever actually been a file clerk?

            have you ever actually been a mail clerk?

            have you ever actually been a salesman?

            if not, then i suggest you go try it. you will learn a lot.

            • by Pieroxy (222434)

              Yes I've been. And you apparently are under the impression that I am discarding those jobs as being of no value while that's not at all what I'm saying.

              Anyone can be a bad salesman by replacing a salesman in case of an emergency. Anyone can be a bad file clerk. Anyone can be a bad cashier.

              That's easy. Now being a good salesman/cashier/clerk is not easy at all. And there's value to that.

              But take a cashier and tell him to fix your critical application that just failed. You'll see the difference btw taking a c

              • the system can fail regardless of how 'genius' the person you stick on the job.

                witness the financial crash of 2008 - hundreds of PHDs were involved in creating CDOs, Synthetic CDOs, RMBS, and giving AAA+ ratings to migrant farm laborers with $300,000 home loans. all those geniuses didn't help the system one single bit, not at all, not one iota.

                • by sjames (1099)

                  IT was up and running the whole time. The computers faithfully reported the nosediving financial indicators. The networks all stayed up. That crash took place in the business side where networking means gladhanding and golfing. Where people tell everyone else that only they can keep things running because they're more powerful than God and that's why they should have the GDP of a small country as a salary even though they're never in before 10 and never stay past 3.

                  Had it been turned over to IT people the c

    • by Patch86 (1465427)

      I'd bet on something far more pragmatic. I work for one of their rivals, so I'm sure our systems are broadly comparable.

      They've outsourced their mainframe support to an Indian company. They're probably running a similar mainframe set-up to what we are. From the snippets of non-technical fluff that their spokesperson was spouting on Radio 4 yesterday, it sounded like it was their mainframe's daily update programme (which applies changes to customer accounts that have been received in the queue from the mainf

      • I've never held much faith in Hanlon's razor.

        I'd say that any sufficiently advanced conspiracy is indistinguishable from a cock-up.

    • The only thing you need to do to lock down funds is to shut down the website down in purpose, no need of conspiracy theories.

  • food! (Score:4, Funny)

    by AltGrendel (175092) <ag-slashdot @ e x i t0.us> on Friday June 22, 2012 @10:44AM (#40412311) Homepage
    I hate it when I can't food. We all need to food sometime.
    • Re:food! (Score:5, Insightful)

      by networkBoy (774728) on Friday June 22, 2012 @11:08AM (#40412659) Homepage Journal

      You know, typo's aside, this is tragic. Many people live paycheck to paycheck. I used to, up till fairly recently, and I'd still be hosed if this happened because my rainy day money is not in a bank fund.

      I would be calling for people to be put against the wall for this.
      -nB

      • by xaxa (988988)

        You know, typo's aside, this is tragic. Many people live paycheck to paycheck. I used to, up till fairly recently, and I'd still be hosed if this happened because my rainy day money is not in a bank fund.

        I would be calling for people to be put against the wall for this.

        If you can show you should have been paid (e.g. payslip, last month's statement) they are giving you £300 if they can't access your account. The bank branches stayed open extra late yesterday, opened at 8am today, and will be open for long hours all weekend.

  • by ackthpt (218170) on Friday June 22, 2012 @10:45AM (#40412325) Homepage Journal

    To prevent another run on UK banks.

    • by samkass (174571)

      Or cause one, once things come back up.

  • And I've always wanted nothing to do with them. Banking in general is very distasteful to me.
    • by Anonymous Coward

      And I've always wanted nothing to do with them. Banking in general is very distasteful to me.

      So they were looking for competent help and you refused them. Thus... we can conclude this is all your fault! FOR SHAME!

  • Always have one. Test it in your testing environment and if it works correctly, deploy on live servers. server idiots guide 101 available in all stores right now
    • Re:test labs (Score:4, Interesting)

      by networkBoy (774728) on Friday June 22, 2012 @10:59AM (#40412551) Homepage Journal

      For our critical stuff we hav 4 layers to get to production:
      1) I do Foo and test it on my dev machine on an unsigned system
      2) I submit my Foo to the build system, it builds it for unsigned systems and it is tested by our QRE/Validation department
      3) Once things look good, it is signed, then deployed to our Validation dept to run on signed systems.
      4) If it is still looking good, then it is deployed.

      And we're not even a banking related operation...
      -nB

    • Re:test labs (Score:5, Insightful)

      by cusco (717999) <<moc.liamg> <ta> <ybxib.nairb>> on Friday June 22, 2012 @02:26PM (#40415411)
      Ah, but they've outsource their IT staff to save money, so they probably got rid of the test lab for the same reason. Never underestimate the stupidity of a group of executives looking for a short-term cost saving.
    • by sjames (1099)

      But that costs MONEY! If they keep doing things that cost MONEY, how will they pay out the huige executive bonuses?

  • Well, they've always looked like a bunch of clowns.

    They were always a bit useless. Then they kind of bought each other up to increase the lavel of failure when they went bust.

    It's government owned now, but you can't blame the government for this computer foul up (there are many other even more spectacular ones), since it was all done by them before the bail out anyway.

    The UK retail banking industry is a poster child for why private industry is every bit as crap as a government, and in fact often worse. The

    • [roman_mir]It's all the government's fault because they wouldn't let the banks go bust and people lose all their money. Vote Ron Paul![/roman_mir]

  • Annoyed customer (Score:3, Interesting)

    by ralphius (1466709) on Friday June 22, 2012 @10:53AM (#40412455)
    As a customer, I'm annoyed that a) A major high street bank doesn't have enough failsafes/testing to prevent this and b) That there is so little communication as to the cause and expected time to fix the problem.
    Thankfully I don't live week to week off my wage like some people do, but if I did I'd be having major problems as evidenced by some of the BBC [bbc.co.uk] stories.
    • Re: (Score:3, Insightful)

      by Skaperen (1481527)

      Even though you were not impacted, you should still move your money to another bank ASAP.

    • Don't take the description at face value.

      I think it is more probably a premature application of some anti-bank-run version. There are financial earthquakes coming soon to Europe, one way or the other, and that always means bank runs and naked cheques.

      I think it is more plausible that they just flipped the "do not trust other counterparty until really verified" switch too early.

    • by petes_PoV (912422) on Friday June 22, 2012 @12:07PM (#40413533)
      The bank won't ever say what the problem is/was. The reasons are that they think their customers are stupid so wouldn't understand - but they also probably don't actually know, themselves. That is also the reason they wouldn't issue a time-to-fix estimate. Their only IT people are outsourced, cheap staff in a foreign country and it's not likely that they have enough understanding of the systems to do much more than try undoing the last thing that happened before it all went wrong.

      Natwest are not alone, another british bank (Barclays) has often been reported as having Monday-morning outages, which sounds a lot like a weekend update that went wrong.

      As it is, having a single account is like having a single credit-card, no spare car key or only one kidney. You can get by until something goes wrong, but in an ideal world you'd have at least one spare.

      • Why would you need more than one credit card? Honestly don't see why I would need two credit cards. I only have one bank card as well. If I lose my card I will have another within 3 days. I always have enough food in my house to last me a week and I cycle everywhere so I don't need to pay for transport. Worst case scenario and go into my bank and take out money directly from my account.
    • look, all of the 'disgruntled customers' like yourself may be upset, but are you upset to take your money out and go to a smaller institution? one that charges slightly higher fees and has less profitable interest rates, but that actually takes it's job seriously?

      this is the situation in the US with people who 'hate big banks'. if they 'hate' them so much, there are literally tens of thousands of small banks and credit unions - and they are insured up to $250,000 if you are worried about losing your money.

      t

    • by Ash-Fox (726320)

      As a customer, I'm annoyed that a) A major high street bank doesn't have enough failsafes/testing to prevent this

      When was the last time there was a massive failure like this from a major high street bank? I really cannot remember.

  • by trancemission (823050) on Friday June 22, 2012 @10:57AM (#40412527)

    When you remove a 1000 members of IT staff [many of which were probably your best] and replace them with 500 offshore workers combined with the need to support *legacy* systems, you are asking for trouble.

    http://www.computerweekly.com/news/1280093677/Royal-Bank-of-Scotland-cuts-1000-IT-jobs [computerweekly.com]

    Regardless of the technical problems, the root cause of this seems to be management......

  • by kiwimate (458274) on Friday June 22, 2012 @11:14AM (#40412737) Journal

    The summary says there's no fix date available, and I know that's what it says in the Register article, but the second article, linking to the bank's site, has this to say:

    The bank says the issue has now been fixed but it will take the weekend to clear the backlog which amounts to millions of euro in transactions.

  • ... move your money to another bank. I hope in UK they have smaller local banks like we have here in the former colonies. If not, maybe find some favorite small towns in the USA.

    • by Vanders (110092)

      move your money to another bank.

      Why? What does that achieve, exactly?

      I hope in UK they have smaller local banks like we have here in the former colonies.

      We used to have building societies but most of them converted into banks in the 90's, then they all bought each other, then they merged into the big banks or went bust. So no, not really.

      • by Anonymous Coward

        Co-operative Bank would be the best bet.

        And you would move your account to tell the bank that incidents like this aren't acceptable for a bank. That cutting experienced IT staff and outsourcing is not a good idea in the long run.

    • by magarity (164372)

      Actually, since this has happened and been fixed at this bank, it's now the place to keep your money to avoid the problem. At other banks, it just hasn't happened yet. :/

  • Cockup or conspiracy? Probably cockup - but quite likely they'll discover they've made more by hanging on to cash for a few extra days than this will cost them in customers.

    I switched off NW/RBOS when I (banked with them since 1978) joined their online banking and saw .asp in their URL.

    • by julesh (229690)

      Cockup or conspiracy? Probably cockup - but quite likely they'll discover they've made more by hanging on to cash for a few extra days than this will cost them in customers.

      Not with £300 per customer compensation as their first offer.

      I switched off NW/RBOS when I (banked with them since 1978) joined their online banking and saw .asp in their URL.

      ASPX, not ASP. And I personally don't see an awful lot wrong with ASP.NET MVC. I'm assuming they didn't use vanilla ASP.NET, because their pages aren't using the witless state tracking mechanism that ordinary ASP.NET pages use.

  • Problem and fix (Score:5, Informative)

    by evilgraham (1020325) on Friday June 22, 2012 @11:29AM (#40412983) Homepage
    Ok, RBS group (which includes NatWest) updates customer accounts via a vast number of batch jobs on a (very big) mainframe overnight. They use CA-7 (a job scheduler, originally written by Uccel) to manage the release, interdependencies and status of these jobs.

    It would appear that an update to CA-7 resulted in the actual schedule for these being corrupted or deleted. Therefore they do not know how much any customer actually has in their account, since accounts were not updated with transactions from the previous day.

    The problem now appears to be fixed (read: update backed out and control datasets restored), but they still have to run through three days of unprocessed transactions, so people are not getting money paid in during these three days into their accounts as expected, resulting in misery.

    This is something which should have been detected and fixed in a competent mainframe site very quickly indeed, so I imagine that the wisdom of outsourcing any "back-office" function of this nature is shortly going to be a matter of very close scrutiny.

    Hope this helps.

    • Guy on the radio just said his house purchase didn't go through because NatWest failed to release the funds. That's got to suck.
    • by Anonymous Coward

      That makes no sense at all. I'm sure Gmail processes more emails (transactions) and saves them on many distributed places so as to never lose any data, in real time, without processing batch transactions overnight.
      Where is the complexity of processing transactions? it's one addition per transaction, my desktop pc can do millions of additions per second. Surely a very safe database will do less with all the disk access but still.. overnight? for 4 million customers? 10 - 20 million transactions tops? there m

      • Erm, they are not using Gmail to process your bank transactions, you tool.
      • Where is the complexity of processing transactions? it's one addition per transaction,

        OK. Here we go. The SWIFT message arrives. The bank initiates the payment. The SWIFT message is parsed - it's a complex and flexible layout. The sender may have made mistakes. It might need an auto repair, it might need a manual repair. Now we know what it is. Next, check it against the many, many regulatory issues. Does it look like fraud? Does it look like terrorist funding? Does it match the account's usual patterns? A

    • The wisdom of outsourcing will not be questioned because they can just blame and sue the outsourcing company. No blame will be attached to the fools responsible for outsourcing (if they're even still there).

  • consequences (Score:5, Interesting)

    by sribe (304414) on Friday June 22, 2012 @12:30PM (#40413835)

    Well, I don't know if it's still the case, but when I worked in banking IT in the late 80s here in the US there was a standing rule: if you don't process checks for more than 24 hours, you can be taken over by the Federal Reserve--where that takeover implies the possibility of being shut down and your assets distributed to other banks.

    That really kept the fear of god in management with regard to keeping core IT running, backups, disaster recovery, etc. Daily offsite backups, periodically loading the backups at a backup facility and running test loads...

    There should still be such a rule, and it should apply to electronic transactions as well as checks (not much difference anymore anyway), and the UK ought to adopt it. If a bank takes down its main system with a fucked-up patch, and can't get its disaster recovery plan working in 24 hours, shut it down.

    • Re:consequences (Score:4, Interesting)

      by Rich0 (548339) on Friday June 22, 2012 @04:47PM (#40416945) Homepage

      I'd go a step further and limit the size of banks to something the government can reasonably deal with should things go south.

      If the biggest banks had tens or hundreds of millions in deposits it wouldn't be such a big deal when one of them has an issue like this. Just have the government step in and take care of the people affected, and then fine the living daylights out of whoever messed up once it is all sorted out.

      The problem is that these banks are getting so large that you can fit those responsible for 80% of the world's wealth into a single room. That means that they talk to each other, adopt similar policies, and so on. We have far more systemic risk, and even when you get a random failure like this the impact is enormous.

      • by sribe (304414)

        I'd go a step further and limit the size of banks to something the government can reasonably deal with should things go south.

        I'd call that a different subject rather than a step further ;-) I believe that history will eventually come to see the single greatest failure of the Bush administration was not forcing a break up of the large failing banks when they had the chance.

        • Fannie and Freddie - still going strong. Executives still making millions.

          JP Morgan - still dealing in Credit Default Swaps, with absolutely zero consequences

          Morgan Stanley - still churning out shitty IPOs like Facebook that rip people for millions, with absolutely zero consequences

          Goldman - still being Goldman

          the hedge funds and investment banks can now give unlimited anonymous money to political campaigns. including banks that are essentially 'international' with no allegiance to the US or its constitutio

          • by sribe (304414)

            The point, which you seem to be willfully ignoring, is that the Bush administration had a unique opportunity to force change and squandered it. The Obama administration has had no such opportunity.

            • by Rich0 (548339)

              I'd argue that Obama had a chance when he first took office, but there was no way he'd make any traction on that and healthcare. Politically not tackling healthcare would have been suicide - not that many are happy with the results.

              The problems in the US are far bigger than who is in the White House. Heck - there was an avalanche of calls from constituents opposing the bailouts and it was basically ignored. And only a huge emergency seems to motivate any action at all on these issues. I'm not convinced

              • by sribe (304414)

                I'd argue that Obama had a chance when he first took office...

                And that argument would be complete bullshit. By the time Obama took office 2/3 of TARP funds had already been disbursed; the "too big to fail" banks had already been purchased by other big banks, creating even bigger than too-big-to-fail banks. The unique opportunity was when the big banks were on the verge of failure and required government money to survive. Bush had the chance then to use that leverage to force acceptance of change; Obama has had no opportunity that is even remotely comparable.

    • " I worked in banking IT in the late 80s"

      if you want to know how things have changed, id start with any of the recent crisis books - "The Big Short" by Lewis, "Fool's Gold" by Tett, "Colossal Failure of Common Sense", by McDonald, "On the Brink", Paulson, "In Fed We Trust", Wessel, etc etc etc.

      The stories about Citigroup and it's incestuous relationship with the high offices of government are of particular interest.

      The Federal Reserve is kind of a joke as far as regulation goes. It does a lot of mole whack

      • by sribe (304414)

        It does a lot of mole whacking, but you have to understand, we gave 2 trillion dollars to shitty banks that in essence, could not process transactions.

        Bullshit. They were perfectly able to process transactions, and what you've said has no bearing whatsoever on the regulation I was discussing.

        I agree the bailout was botched. But your "in essence" is way too far of a reach.

    • by Patch86 (1465427)

      Well, I don't know if it's still the case, but when I worked in banking IT in the late 80s here in the US there was a standing rule: if you don't process checks for more than 24 hours, you can be taken over by the Federal Reserve--where that takeover implies the possibility of being shut down and your assets distributed to other banks.

      RBS Group has already been taken over by the government; they can't take them over any more than they already have.

      And the reason they were taken over is because they were "too big to fail"- so no real possibility of chopping them up and distributing their assets. If they could have done that, they could have just let them collapse and be picked over by their rivals in the first place.

      • by rb12345 (1170423)
        RBS weren't fully taken over (Wiki says 84%); I suppose the government could claim the other 16% too but it's unlikely to happen.
  • That's not a bug for the bank, that's a feature!
  • by Anonymous Coward

    Can I write a cheque? You do take cheques don't you?

  • by Anonymous Coward

    Suddenly Bitcoin becomes a real solution to avoid this kind of problem...

    • by Ash-Fox (726320)

      Suddenly Bitcoin becomes a real solution to avoid this kind of problem...

      Not really, if your 'bank' on Bitcoin like flexcoin [flexcoin.com] had issues like this, I fail to see how Bitcoin would make it work.

  • by Anonymous Coward

    has some brilliant banking laws. A bank is required to be able to give a customer up to every cent of his money almost 24 hours per day immediately. This has resulted in faster money flow, better competition in banking, and a generally smoother economy. There are other perks like ATMs that cash out in denominations down to a single yen. Finally banks are unable to pull shit like creating branch withdrawal limits to screw over customers.

    In Japan if I had a million dollars in an account and went to withdraw i

As in certain cults it is possible to kill a process if you know its true name. -- Ken Thompson and Dennis M. Ritchie

Working...