World's Biggest 'Agile' Software Project Close To Failure 349
00_NOP writes "'Universal Credit' — the plan to consolidate all Britain's welfare payments into one — is the world's biggest 'agile' software development project. It is now close to collapse, the British government admitted yesterday. The failure, if and when it comes, could cost billions and have dire social consequences. 'Some steps have been taken to try to rescue the project. The back end – the benefits calculation – has reportedly been shifted to a "waterfall" development process – which offers some assurances that the government at least takes its fiduciary duties seriously as it should mean no code will be deployed that has not been finished. The front end – the bit used by humans – is still meant to be “agile” – which makes some sense, but where is the testing? Agile is supposed to be about openness between developer and client and we – the taxpayers – are the clients: why can’t we see what our money is paying for?'"
Agile doesn't mean that the project won't fail (Score:5, Informative)
But it might make it clear that it will fail much earlier and then at a lower cost.
Re: (Score:2, Informative)
Re:Agile doesn't mean that the project won't fail (Score:5, Insightful)
...since there is daily accountability and you're working on smaller pieces.
So that means it's impossible to do anything big or that requires extended planning? Sometimes a developer needs to be left alone for a week to come up with something good. Regimenting the process into days and forcing a daily bullshit update is just abusive to the creative process.
Re:Agile doesn't mean that the project won't fail (Score:4, Insightful)
...since there is daily accountability and you're working on smaller pieces.
So that means it's impossible to do anything big or that requires extended planning? Sometimes a developer needs to be left alone for a week to come up with something good. Regimenting the process into days and forcing a daily bullshit update is just abusive to the creative process.
You've already failed at project of that size if you're letting a developer be "alone for a week to come up with something good". A developer's job, in a project like that, isn't to come up with something good. Their job is to implement a specific piece of functionality in the specific way defined by the people whose job is to have the broader view of the project.
Some developers may not like working in that environment, and they shouldn't be working on a project that size. Cowboy programming is for small companies. (And, for what its worth, I've worked directly with many hundreds of engineers in the last couple of decades, and the happiest engineers are the ones who recognize which kind of project they like to work on, and avoids working on the other.)
Re:Agile doesn't mean that the project won't fail (Score:5, Interesting)
The short(er) and honest version is that projects like that are inclined to fail, spectacularly and at great expense, whilst everyone on the job gets paid either way.
People are either accountable or not. When you have teams of "project managers" that don't know shit about shit, shoveling ridiculous and disparate feature changes into the (often offshore) dev shops inbox, while their programmers simply crank out code to match, you're begging for the whole thing to go right in the toilet.
Large consulting firms like Accenture build their entire business around this kind of approach. The result is always a failure that costs millions (or in this case billions) of company/government dollars on a product that doesn't do what it was supposed to.
Re:Agile doesn't mean that the project won't fail (Score:4, Insightful)
Agreed. Agile is a WHOLE lot easier to get wrong than it is to get right. The benefits it promises come only from the synthesis of ALL its components. Project managers who think they understand it, but don't quite get it, wind up tweaking it just a bit (in a way that seems to make perfect sense) and that tweak completely undoes the approach. The effect often isn't instantaneous...it is felt over the life of the project.
Be that as it may, for some types of projects Agile is flat out wrong.
Agile is awesome when you can't afford to frontload your development costs, don't really know which features will get a lot of use and which ones will just collect dust, and when your project manager actually understands agile. But there is a very severe drawback when you are building a huge feature-rich juggernaut of a system that needs a very sophisticated and precise back-end: refactoring costs go through the roof.
Agile proponents often get angry when one says anything bad about it, and yell about how writing maintainable code is one of those elements of agile that you can't sacrifice. And I agree. And the refactoring cost gets more evenly spread that way, but remains astronomically high for complicated systems. Here is exactly why:
Every feature you add to a back end system, no matter how well coded, increases the complexity of the system.
The more complicated the system, the more expensive it is to add a new feature to it.
Waterfall can give you some savings in this specific department because the initial system design already includes the final (or near-final) level of complexity, so you don't pay the extra cost of adding a new feature to an already existing system over and over again. Waterfall still has costs in that some of those features may never be used, and hence are wasted effort, and also misestimation and requirement changes are more expensive and put a project past deadline. But for huge systems, these costs are preferable to the project-destroying costs of agile refactoring.
There isn't one approach that is right for every problem, and agile is not the right approach for this one.
Re: (Score:3)
Of course waterfall and other strict-spec methodologies have a 72 year history [1] of failing spectacularly too, and from about 1980 have had the added bonus of being too slow for business velocity.
sPh
[1] counting from about 1940, although projects using the mechanical accounting systems of the 1920s may have had similar problems
Re: (Score:3)
I'd want to see a cite for that one. Waterfall has been accused of being slow, of being expensive, and making it difficult and expensive to make changes after the first phase. Those are all legit "cons" of waterfall development.
But the form that is actually used and called waterfall in practice is also known to work very well. Maybe not on a cost/benefit analysis for small business. But on large projects it not only works, it has been used to build most of the stuff that has, you know, been built. It is a c
Re:Agile doesn't mean that the project won't fail (Score:4, Informative)
I'd want to see a cite for that one.
This is not an area where it is possible to give "a cite" since there are whole genres of literature covering this topic alone. If you haven't read ."The Mythical Man Month" [wikipedia.org] (please note; the book has a Wikipedia page; this is not an Amazon link) then that is where you should start. Not because it is complete, not because it is up to date, but because it will make you realise that the problems of today's IT were already fully described in the '70s and that our advances in the last decades have been incremental and mostly small.
Next time you drive over a bridge, be glad they used a waterfall-like development paradigm.
Bridges do not work the same as software development. Whilst each individual bridge has some differences in environment and location, in general you are just repeating a structure which has already been build long ago. In sofware the equivalent of building a new bridge is the "cp -ar" command. Agile is mostly designed to address development of new features on pre-existing software which is completely different.
Re:Agile doesn't mean that the project won't fail (Score:4, Insightful)
I'd want a cite on that claim that there is a difference in engineering there. Sounds a lot like, "... on a computer!" to me.
Re:Agile doesn't mean that the project won't fail (Score:5, Insightful)
The benefits it promises come only from the synthesis of ALL its components.
Yes, yes, we've heard it a thousand times, if an agile project fails, it must not have been truly agile. Probably isn't a true Scotsman, either.
Re:Agile doesn't mean that the project won't fail (Score:5, Funny)
Repeat after me, "The King has no clothes." It just doesn't get old.
Re:Agile doesn't mean that the project won't fail (Score:5, Insightful)
The unstated assumption in that theory is that there is a group of human beings who are unto demigods, capable of defining detailed specifications that cover all eventualities, see far into the future, and can be implement mechanistically by simple code grinders. I have run into two or three such people in my entire career, out of the thousands of system architects and business analysts I have worked with in some capacity. And even those three people (a) often got caught by surprise by unintended side-effects (b) were often part of a process that took so long to come to fruition the system so designed was obsolete by the time it was deployed.
The only industry I am aware of where strict, spec-controlled waterfall is probably appropriate and generally works is aerospace controls systems. Of course that industry is notorious for huge schedule and cost overruns (the A400 software fiasco is particularly instructive in this regard), and even their software is not free from bugs and unintended consequences. Ref the numerous changes Boeing has made to the electrical system control software on the B-787 since it went into service - why didn't they just "get the spec right the first time"? Enrico Fermi's comment on how to manage the unknown is very instructive.
sPh
Re:Agile doesn't mean that the project won't fail (Score:4, Insightful)
...since there is daily accountability and you're working on smaller pieces.
So that means it's impossible to do anything big or that requires extended planning? Sometimes a developer needs to be left alone for a week to come up with something good. Regimenting the process into days and forcing a daily bullshit update is just abusive to the creative process.
You've already failed at project of that size if you're letting a developer be "alone for a week to come up with something good". A developer's job, in a project like that, isn't to come up with something good. Their job is to implement a specific piece of functionality in the specific way defined by the people whose job is to have the broader view of the project.
Parent poster here. I see your point, however, my experience has been that separating the designer/architect role from the developer role is fraught with pitfalls. The people writing the code should be the ones designing it, otherwise you end up with a skyscraper put together with superglue instead of bolts and welds. The response to that is to have precise specs, but the time spent writing specs that are precise enough to be flawless is more productively spent doing the actual coding.
The bottom line to all management fads and whether a development process is effective and ineffective are fundamental human limitations:
1. Quadratically growing communication overhead as teams get larger [wikipedia.org]
2. Finite human memory capacity, limiting how much information about a project each developer can be aware of (and each developer must be aware of the whole project, to avoid causing havoc or reinventing the wheel, or they must be sequestered in their own subcomponent)
A good team size appears to be five, and an organization should not have more than five teams that need to intercommunicate. The challenge is to break up a piece of software to fit this kind of arrangement. Anything else will lead to company-wide split-brain, NIH, and communication bottlenecks. This is true regardless of the development process selected. Fundamental human limits are like the speed of light, you can get closer to them, but you cannot exceed them. So assemble good teams, architect your project to fit those teams (or vice-versa), and remember that everyone is human.
Re: (Score:3)
You've already failed at project of that size if you're letting a developer be "alone for a week to come up with something good". A developer's job, in a project like that, isn't to come up with something good. Their job is to implement a specific piece of functionality in the specific way defined by the people whose job is to have the broader view of the project.
You're going to have a hard time hiring and retaining decent developers with that kind of approach.
Re: (Score:3)
You've already failed at project of that size if you're letting a developer be "alone for a week to come up with something good". A developer's job, in a project like that, isn't to come up with something good. Their job is to implement a specific piece of functionality in the specific way defined by the people whose job is to have the broader view of the project.
You're going to have a hard time hiring and retaining decent developers with that kind of approach.
If I'm running a billion dollar engineering project, the engineers who can't work with the team or think -- with their extremely limited view of the project -- that they know what needs to be done aren't people I'd want on the project anyway. I might have a hard time hiring and retaining, but I'd have an easy time firing. Been there, done that. Cowboys are toxic to large scale engineering.
Just as you don't want a steel worker deciding a beam should be attached differently when you're building a skyscraper b
Re: (Score:3)
Having bashed waterfall quite a bit upthread, I will say that I have no desire to have Facebook's developers program my bank account or inventory control system. Sometimes consistent, validated transactions that don't randomly disappear really are important.
sPh
Re:Agile doesn't mean that the project won't fail (Score:4, Interesting)
Small companies like Facebook?
You're citing a company whose motto is "Move fast and break things", a company that is indeed infamous for breaking its software all over the place and introducing changes its users don't want, as evidence that cowboy programming works at scale?
Facebook have been shrewd about their marketing, and they've developed probably the most effective lock-in/network effect strategy in software history, and they've also been lucky at a few times when it counted, and they're successful as a result. Please don't confuse any of that with technical merit, which really has very little to do with their success or failure at this point.
Re: (Score:3)
Remember, the customer isn't the users that don't pay anything to use facebook. The customer is facebook itself and the companies that use it for advertising. From that perspective, it seems to work incredibly well.
Actually, it hasn't. Their stock is in the toilet. Their ad revenue is stagnant because the introduction of ads has done nothing but drive people away.
Facebook is a textbook example of the fundamental downside of rapid-paced, planning-free development.
Re:Agile doesn't mean that the project won't fail (Score:5, Insightful)
I agree but the daily accountability is still something that a lot of hard core developers don't buy into. The "leave me alone" mentality still prevails in big shops. There's also a lot of gaps in Agile and IMO while Stories are great they are not a substitute for fully defined requirements analysis. Not that they can't go hand in hand but I've watched lots of Agile projects fail because of incessant changes in vision and invalidation of stories subsequent to their definition by other stakeholders. The key stakeholders either don't pay attention or louder voices who have really no relative bearing to the project somehow get suddenly important. These are often folks with something to gain by holding things up or creating confusion. That is always the problem in all projects but it seems more acute in Agile because "hey, we have a process that can allow for these changes." Ultimately the team gets into a tail chasing situation and nothing of value (to the key stakeholders) gets built and the project gets cancelled. On the successful projects that I've worked with in Agile, there's strong stakeholders, good architecture keeping the vision in place and project management that keeps things well orchestrated. Without those in the mix, it'll fail just like all software projects.
Re: (Score:3, Insightful)
I hate to break it to you but "fully defined requirements analysis" is a pipe dream filled with rainbows and unicorns. I have never, not once, seen a requirements document that accurately captures exactly what the system will do. Even if it did it would be true for all of 5 minutes before the product owner/user changed their mind and redefined what they want. The whole poi
Re: (Score:3)
It sounds to me like the direction they are taking the project (waterfall model for the back-end and Agile for th
Re:Agile doesn't mean that the project won't fail (Score:5, Insightful)
Re:Agile doesn't mean that the project won't fail (Score:4, Informative)
That's right, it's all just programming, motherfucker [programmin...fucker.com]
Re:Agile doesn't mean that the project won't fail (Score:5, Interesting)
I hate to break it to you but "fully defined requirements analysis" is a pipe dream filled with rainbows and unicorns. I have never, not once, seen a requirements document that accurately captures exactly what the system will do.
Well, I've written them, and I've never had a project fail.
One example involved interviewing
i) the owners of the company
ii) an executive from each department
iii) a "regular joe" representative from each department
This became a 40+ page project specification, which was signed off by all stakeholders and became the contract.
Then this document was fed into a series of code generation engines, which created hundreds of thousands of lines of code. This was all done with an eye towards allowing various professionals to go away and do what they do best without getting held up waiting on each other or tripping over each other, filling in the missing functionality in the generated code.
That system is still in operation close to a decade later, organizing the working lives of thousands and serving the needs of millions.
Now I work in Agile. I hate it. I'm always having to check with other people constantly to move forward, I never get in the zone, there's a lack of clarity and vision, and I feel like I'm getting stupider each day and I'm not producing my best work.
Re: (Score:3)
I just want to add...
Try writing a project that meets ISO specifications for medical use using the waterfall method.
People will die.
Re: (Score:3)
Can you honestly say you, alone or with a team, can write a 3000-page specifications or requirements document and still be sure there are no contradictions, omissions, or any defect?
Yes. I'm entirely capable of that. That is indeed what I'm saying.
Re: (Score:3)
I hate to break it to you but "fully defined requirements analysis" is a pipe dream filled with rainbows and unicorns. I have never, not once, seen a requirements document that accurately captures exactly what the system will do.
Well take a look at a bridge or a tall building. "Waterfall" design process is just "traditional engineering process." You've never seen a fully engineered system, well, sure. I believe you. That doesn't mean it is doesn't exist.
Re: (Score:3)
Deloitte tried classical waterfall for our SAP implementation.
The specs delivered by the first wave (who got bonuses, parties, and 24 months of 45 hour weeks) were not even FUNCTIONAL in many cases and did not capture essential business rules. Noone at the start really knew what was right or wrong so they just wrote up crap and it got signed off on as complete specs.
Then the rest of us got 36 months of 60 to 80 hour weeks (multiple deaths, non fatal heart attacks, divorces) doing the work in a waterfall mo
Remember what this project is for (Score:3)
I have never, not once, seen a requirements document that accurately captures exactly what the system will do. [...] The whole point of stories is to do things in small, end-to-end slices to produce functionality quickly, let the product owner see it and play with it and then get a better idea of what they really want.
Well, in this case, "what they really want" tends to be defined literally by Acts of Parliament and/or policies set by the highest legal authorities in the government. I know it's popular to mock politicians in Europe for having no idea what they're doing economically, and it seems there's some truth to that in light of recent events, but you don't implement software to automate a major component of your national tax and benefits system in incremental changes with one guy designated as the project owner wh
Re: (Score:3)
I hate to break it to you but "fully defined requirements analysis" is a pipe dream filled with rainbows and unicorns. I have never, not once, seen a requirements document that accurately captures exactly what the system will do. Even if it did it would be true for all of 5 minutes before the product owner/user changed their mind and redefined what they want. The whole point of stories is to do things in small, end-to-end slices to produce functionality quickly, let the product owner see it and play with it and then get a better idea of what they really want. I know what you're going to say next: "If they keep changing their minds then how does it ever get done?" Simple, you make it extremely clear that continuously changing their mind directly equates to more time and money spent and prevent other functionality from being implemented.
LOL, you haven't done much with big clients in big corporations and the government have you? Requirements analysis can be done and yes, it's like anything else if you pay attention and have committed stakeholders who are willing to validate what's put down on paper then you'll be successful. Stories are great, don't get me wrong but invariably they don't get into enough detail. I realize it's about breaking things down into smaller pieces to show progress but progress to what? If's it's something small
Re:Agile doesn't mean that the project won't fail (Score:5, Interesting)
Pretty much this exactly. Also, it's tough to get programmers and managers who have never worked in an Agile environment to buy into it. My company started using it 4 years ago and we still have a few holdouts despite the obvious benefits in both productivity, cost and simply a better work environment for everyone. Hell, I think the best part about the Agile process is those one or two guys on a piece of a project that never seem to do anything and could end up causing drama simply doesn't happen in a proper Agile setup since there is daily accountability and you're working on smaller pieces.
"Waterfall" -- i.e., the "old fashioned" way of doing things -- does one thing very well that Agile loses. And that thing is something that was understood for a century of large project management planning. Waterfall ensures quality with a team of varying abilities, or large teams. Agile ensures predictable delivery, but quality is very dependent on the individual abilities of the team members.
Anyone who has done large projects would know immediately that you don't do a billion dollar project with a pure Agile methodology. You simply can't get enough people who are strong enough to deliver a quality output without a very high amount of formal planning and progress gates.
The most successful large (multi-hundred engineer) projects I've seen in the last five years tend to use waterfall for the overall project, but encourage teams to run their parts of the work in an agile manner. You get the visibility into progress that way, but the formality of process to ensure you're really building the cohesive system right.
Re:Agile doesn't mean that the project won't fail (Score:5, Interesting)
Actually Agile would probably handle mediocrity quite well, at least making it apparent.
What agile really sucks at is handling the political aspect of the project, because simply agile requires complete honesty and honesty does not scale very well above a small team. In a very large project involving loads of teams, management is a lot closer to a poker game and you don't win at poker by showing your cards to all the players.
Re:Agile doesn't mean that the project won't fail (Score:4, Insightful)
Its more than that. Managers prefer mediocre programmers because then you are not dependent on them.
Managers have this dream that programmers are a kind of generic glorp that can be poured into any job slot and be easily replaced with another generic glorp programmer.
One manager had this dream of putting all programmers for a multi billion dollar company into a single oncall rotation. Whoever was up that night would cover accounting, inventory, plant, invoicing, database, pc problems, etc..
Re:Agile doesn't mean that the project won't fail (Score:4, Interesting)
Upper management prefers this. I don't think this is anywhere near universal at the lower levels- they realize that the top 10% of developers save their asses. Then again, this is why I work for small companies and startups and not anything outside the tech industry- management at small places is smarter than that or fails quickly.
Re: (Score:3)
Re: (Score:3)
Nah, agile is a tool that works well when dealing with small projects with loosely defined processes that might change rapidly that are usually defined by the end user. Well defined processes, or large projects are usually better served by waterfall at least for the first release that includes the backend and middleware. The front end can then go through many iterative revisions based on user feed back.
Going agile for a well defined large government project was a bad decision whether it ultimately failed
Re:Agile doesn't mean that the project won't fail (Score:5, Insightful)
But apparently it didn't succeed in that, either. They're discovering massive failure only after three years and billions of pounds.
Re: Agile doesn't mean that the project won't fail (Score:4, Insightful)
Re: (Score:2)
But apparently it didn't succeed in that, either. They're discovering massive failure only after three years and billions of pounds.
The people doing this project are human they make mistakes. Besides we don't know how Agile they are I've worked on a lot of Agile projects and a lot of Waterfall projects and I've never seen one yet that was wholly Agile. I'd be more interested in how much of it was outsourced but then I didn't RTFA
Projects don't fail... (Score:2)
...because of the development model.
They fail because there was not enough though put in up front and the requirements are vague.
Addendum: (Score:3)
Being called 'Agile' doesn't mean that it is in the spirit or letter of 'Agile'.
The reality is that 'Agile' is in practice more of a brand than anything else. Project Managers love to apply the terminology to their projects. This does not mean they actually meaningfully follow a consistent set of behaviors, just that they use similar sounds words.
'Agile' is like 'Cloud' and 'Web 2.0'. While each phrase may have coined with a particular specific concept in mind, they became more hype than anything usefull
Re: (Score:3)
He says he is a Scotsman, but until we know if his kilt length is 1650 or 1750 we can't be sure.
That said, you should actually look up "agile development" because it is NOT like "cloud " or "web 2.0" at all, it is a very specific set of things that comes from actual documents and books that established it.
What's that saying about agile? (Score:5, Insightful)
Re:What's that saying about agile? (Score:5, Insightful)
Agile is just the latest management fad. In a year or two something else will come along, and the lemmings will follow.
Re: (Score:3, Insightful)
Year or two? Agile and it's variants have come into being because the old methods of software engineering were thought deficient and slow. Agile does work when teams are focused and stakeholders engaged. Agile does work just like the other methodologies but it's not the ultimate solution and companies or governments in this case who take on a development project with it need to know the pitfalls and how to avoid them. Otherwise you'll still have a steaming pile of dog crap at the end.
Re:What's that saying about agile? (Score:4, Insightful)
Its mostly people and talent, but the question of what you're doing is important too. Agile is great for things that are easily visible to customers- UI development or web pages, for example. It fails where you have other systems depending on you- a back end service can't change that quickly or have stability issues. Of course, you can do different methodologies for different parts of a complex system and use each where they make sense.
Re:What's that saying about agile? (Score:5, Insightful)
"DWP IT chief and government chief information officer Joe Harley said in May 2011 that agile would ensure his department delivered Universal Credit on time in October 2013." [computerweekly.com]
So, a two year iteration and a guaranteed delivery date? Yeah, that really sounds like Agile.
The article goes on: "Attention must turn to Accenture and IBM, who are on track to earn £1bn between them as lead developers of the system. They may have played the most significant part in agile's failure at DWP, or DWP's failure at agile. Accenture and IBM may have found agile commercially inconvenient. Neither has yet been able to speak about it."
Ass-Center and IBM? Yes, two companies who are well known for their love of Agile [rolls eyes].
Re: (Score:2)
Re: (Score:3, Funny)
Re: (Score:2)
Do you actually work as a project manager? I do. Try executing your idea in an average company and see how far you get. Good luck getting only "smart, talented, dedicated" individuals for your project! Even those employees will have character flaws, personality conflicts or other reasons that prevent the project from succeeding.
Re: (Score:2)
Agile assumes you have smart, talented, dedicated individuals doing the work.
And, I would wager, reasonable, patient, even handed clients (which will be the government, and not the ridiculous assertion as stated in the article that it's the taxpayers). I challenge anyone to look at the Victorian dinosaurs in power in the UK and assume any of those qualities.
That was just a cheap shot at the government, I admit it.
I imagine whatever will run UC on a technical level will just be another government collossus. Their minds are incapable of conceiving or implementing anything else.
BTW should I mention my pet name for Agile (Score:5, Interesting)
To quote Bruce Lee... (Score:2, Insightful)
"Before I learned the art, a punch was just a punch, and a kick, just a kick.
After I learned the art, a punch was no longer a punch, a kick, no longer a kick.
Now that I understand the art, a punch is just a punch and a kick is just a kick."
I've worked with a few different methodologies in my time, but now that I'm older I realize all you have to do is follow common sense. It's really not that difficult.
Extremely accurate. (Score:4, Insightful)
Whether it was 'waterfall', 'agile'', or whatever, every project that I've worked with that seemed to put more effort into using the most hyped phrasing to describe their process than into actually developing the project has been doomed.
I liken it to religion. The spirit of most holy texts is quickly lost in the actions of adherents as they focus on the specific content rather than the message. For Christianity, specific belief in the divinity of Jesus seems to often be more important than adhering to his teachings. Similarly, in Agile, managing to map words like 'scrum', 'sprint', 'epic', 'user stories', and so on to what you do is more important than internalizing the original intent behind those words.
Projects that don't make a lot of effort to 'conform' to any specific renowned fad tend to do well. They also tend to do the sorts of stuff Agile advocates without using the words.
Re:Extremely accurate. (Score:4)
For Christianity, specific belief in the divinity of Jesus seems to often be more important than adhering to his teachings.
Why would someone listen to the teachings of an individual but deny his most central message? Why would you say, "This guy is a complete liar about being God! But I'll follow everything else he says."
To quote C.S. Lewis:
A man who was merely a man and said the sort of things Jesus said would not be a great moral teacher. He would either be a lunatic — on the level with the man who says he is a poached egg — or else he would be the Devil of Hell. You must make your choice. Either this man was, and is, the Son of God, or else a madman or something worse.
Re: (Score:3)
Indeed. The problem is that not many people actually have the common sense required and that complex software projects are routinely attempted with cheaper, not very competent developers. That can never work. Complex projects need masters of the art, no matter what the art is. And the masters of the art need to be put in charge, not managed by some "managers" that do not have a clue. If you do not have a "chief engineer" in charge that could do most parts with his/her own hands, then you are doing it wrong.
Re:Simple formula (Score:4, Informative)
Government department + software project = total failure. .
I would love to know how cheaply this same project could be done. Probably by one person. Probably a $10,000 project with the final project size 100 times smaller, run 100 times faster, 100 times more accurate. [That is what I achieved after a payroll application they tried to force on our dept. was discarded and we rolled our own.]
Not a real biggie, just a replacement for Jobseeker’s Allowance, Income-related Employment and Support Allowance, Income Support, Child Tax Credits, Working Tax Credits, Housing Benefit for the whole of the UK population. I mean how hard can it be? Given your obvious talents I am sure you could knock something together using a few Excel macros by next Tuesday.
One of the things about this is that it is being driven by an ideologue who doesn't give a toss about evidence, not quite the person who thinks all government sponsored software development but pretty close.
Where's the testing? (Score:3)
Re:Where's the testing? (Score:5, Funny)
Agile, unlike everything else in the world, is the perfect silver bullet. There can't be anything wrong with Agile. The seminar even said so.
Open Source It (Score:3, Interesting)
Agile Needs Testing? Open Source It.
It's perfect for that. Open an issue tracker, let folks dump in the requirements. Give us the backend API, and I'll work on it for free an hour or two here and there, just for grins.
Oh, it's closed source? Well, I hope it's a massive disaster and you learn your lesson. You can't pay for the brightest minds. They wouldn't be caught dead slaving away at those software houses. But let them do it "for the good of mankind", they'll be brow bashing each other for a chance to get their beautiful bit of brilliance in the code base.
Re: (Score:2)
Agile Needs Testing? Open Source It.
Open Source a project to consolidate all national welfare transfer payments in a country of 63 million. Meeting all accounting and legal requirements. Now tell me how you recruit and manage OS "volunteers" with the depth and experience needed to do that.
Re: (Score:3)
Open source developers will not flock to work on a bureaucracy-ridden welfare allocation and tracking platform, of all things.
I will if I get to work on the 'open source developers benefits module'.
The public is not the client (Score:2, Insightful)
The government is the client. There is no reason for the public to be involved in an unfinished project.
Re: (Score:2)
Really? Why shouldn't the public get "full real time insight" into just about everything government does? What is there to be gained by having government be secretive?
Re:The public is not the client (Score:4, Insightful)
Just because the people delegate authority to representatives doesn't mean that those representatives should operate in secret.
world's biggest? (Score:5, Interesting)
Re: (Score:2)
World's most agile Zeppelin?
Re:world's biggest? (Score:5, Insightful)
The most important thing is tight iterations. If a 2 week sprint fails, then it is not that big of a deal. If a 2 year death march fails? Someone's getting fired, since its the equivalent in agile-land of failing 52 sprints straight.
But is it two weeks sprint down a dead end? For a project this size, agile is like trying to build a skyscraper first as a one story building, then two story building, then three story building and so on. Apparently you're making great progress the first sprint and you have a shack up, that's 1/100 floors done already. Except it doesn't work like that, so sometime around the 20th floor you've got people all over the first 19 trying to build in extra support columns and stronger walls and propping up the foundation. Things grind to a halt and you're not making any real progress. Then the orders come to get moving and you start going upwards again more and more rickety until eventually you find the straw that broke the mule's back and it all comes crumbling down.
Agile is nice if you're close enough you can start delivering actual features that would belong in the end product at the end. In practice it often means you build the first iteration with string and duct tape planning to replace it with something more solid on the back end in time, but I think everyone knows how that goes - the string and duct tape has a tendency to stay because that part is "done". Of course hindsight is always much easier but agile I feel lacks foresight, we do this now to meet our sprint goals and then if we need to change something to meet our next sprint goals, we'll deal with that then. In practice, there's not time to go back and rework things every time you figure out this should have been done differently.
Re:world's biggest? (Score:5, Insightful)
You can build a perfectly suitable "one room shack" by slapping some canvas & boards as temporary walls around the first story of a steel-beam skyscraper. Sure, it'll be one of the most over-engineered one room shacks in history, but that shack is architected for future expansion, which is why those giant steel beams are being used as support members, rather than scrap 2-by-4's.
If you find that you've reached the 20th floor only to realize that all your steel beams are only able to carry 20% of their expected load, then that is a failure of your architects, who failed to provide a design for a building which will be 100 stories tall, but which must get built just a couple stories at a time.
I think what the grandparent was trying to say is that a lot of Agile projects tail to give people proper time to actually do some initial architecture of the project before people jump into the sprints. Realistically, you need to combine waterfall with Agile methodologies since large projects need to have a solid foundation to work off of that might take a couple months to put in place.
To go back to the building analogy, if you are building a skyscraper you are digging a basement and putting footings and everything else in before you even build the first floor.
And who were the contractors? (Score:2, Informative)
It's the usual crowd screwing money out of the government
HP/IBM/Accenture/BT
https://www.whatdotheyknow.com/request/130677/response/322518/attach/html/3/FOI%203648%20Response%20181012.pdf.html
Re:And who were the contractors? (Score:5, Insightful)
Yeah... (Score:5, Insightful)
Agile is (usually) BS (Score:5, Insightful)
All Agile methodologies really are are different ways of implementing the Spiral Model of development. If used correctly, any of them can work fine. Unfortunately, that's only in theory. In practice, Agile generally becomes an excuse to use Code and Fix, which is the worst methodology and the most prone to failure. Beware anyone who claims that Agile is the solution to anything.
The total misunderstanding of Agile. (Score:3, Interesting)
Someone still has to helm this and there has to be a rather obvious set of goals and objectives to attain.
The problem I've seen with Agile, is that often, the software is being developed as it goes and to me, that's poor use of the "Agile" experience.
To me, Agile would mean that the software architect(s) know what they are doing, and have a bloody good idea of the road map to take to get there.
The "Agile" part is that for each of these goals to achieve, there is a separation of tasks and an agreement on implementation specs.
Any single team has full knowledge of the objectives of other teams and thus, this team can concentrate of their specific task and at the same time, mockup whatever is required to support their work based on any of the dependencies which depends on other teams work.
Unit Testing, Functional Testing, etc, all of that must come into play.
The Agile process should NOT be 'on-the-fly', which is what I've often seen, for that is the reason why we get delays and development costs going over budgets.
Re: (Score:3)
But you also have to have good tests. And automated tests. I have had "agile" projects where the "testing" I was required to do consisted of basically making sure that when you SELECT * FROM EMPLOYEES, Oracle DB did, in fact, return the contents of the Employees table. Not that the records are in any way accurate, or entered correctly, or used properly, or make sense in the context of the application. Just that the database server is a database server. TESTED. APPROVED. I felt like this guy: http://weknowme [weknowmemes.com]
What happen to CCTA? (Score:3, Insightful)
The UK Government used to have its own internal computer consultancy. This was called the Central Computer Agency - the Central Computer and Telecommunications Agency after it took over running the government phone network.
CCTA was staffed by a mixture of experienced civil servants and expert contractors, and provided support to all UK government department's computer projects. It had procurement experts, sizing experts, code, architecture, the lot. When these experts were not working for Departments they worked on UK and international standards and methodologies. Some of you may remember the OSI, ITIL, PRINCE, PROMPT, SSADM, CRAMM, BS7799/ISO 27001/2, BS5750, etc...
In those days UK Government projects ran to time and came in on budget, with CCTA project managers. But CCTA was constantly under attack from the computer industry, who saw CCTA as an expert gatekeeper, stopping them from making major profits out of government projects. They lobbied for its closure constantly, and in 2000 they got their wish. CCTA's major functions were closed down and the rump moved to OGC
Interestingly the CCTA Security and Privacy Group (the only UK Government computer security organisation at the time) had been closed down earlier at the request of GCHQ and MI5, who wanted to take its budget and responsibilities. The SPG arguably ran the first CERT (though not called by that name) in 1984, and was influential in developing, inter alia, early AV company liaison and security accreditation with initiatives like Common Criteria.
The UK government departments, encouraged by a variety of groups with ulterior motives, cut off its nose to spite its face. They are now a byword for incompetence and overspend. The collection of experts which existed in Riverwalk House during the 1980s and 1990s will never be replicated again...
Brogrammers (Score:5, Interesting)
A lot of the 'agile' models reward talkers and people who take immediate action and can rattle off buzzwords, at the expense of more introverted engineers who like to investigate and plan before they act. In a continuously moving environment such as a social networking site, that reward system might be appropriate. In a financial back end doing mission critical work, that sounds like a disaster. So, no surprises here. One size does not fit all companies.
Agile/Extreme is analogous to alternative medicine (Score:4, Interesting)
Agile/Extreme programming is the alternative medicine of software development. It's a collection of mostly unrelated and sometimes contradictory concepts, the only common thing between them being lack of widespread adoption. Like alternative medicine, it has components that are useful in some circumstances. These give unearned credibility to Agile, even though they were there well before it. The problem is also similar, these components are taken to the extreme and are claimed to be universal solutions to every situation. For example, acupuncture is useful to ease pain, but no matter what the charlatans say it doesn't help against cancer. Similarly, frequent testing during development can be useful, but test-driven development is taking it to the extreme, and it doesn't work at all in for example web development. And like alternative medicine has homeopathy, Agile/Extreme has their own set of ideas that are total bullshit like pair programming.
Development and manufacturing. (Score:3)
That is the worst article I ever red (Score:5, Informative)
This should be suppose an article about "agile" and the Universal Credit. After reading the article there is no information what-so-ever, except that the Universal Credit project has been admitted to be failing.
So why is Universal Credit an "agile" project?
Why it is failing?
What is Universal Credit anyway?
Maybe that is why Twitter is so successful, the whole article is just a Twitter message: "Universal Credit, suppose to be biggest Agile Software Project, is failing".
Here is some more information:
http://www.guardian.co.uk/society/2013/apr/29/universal-credit-pilot-scheme [guardian.co.uk]
http://www.guardian.co.uk/commentisfree/2013/apr/30/universal-credit-iain-duncan-smith [guardian.co.uk]
Is it called "agile" because it's a "step-by-step approach" ?
Been there, still there (Score:5, Interesting)
Over my career, i've worked in the UK Benefits Agency processing claims, i've worked in their IT departments, i've worked for the outsourced departments later supporting them, and i've worked for a software company which loves agile (but will do waterfall if pushed).
The problem here isn't waterfall/agile. The problem here isn't .Net/Linux.
The problem here is the parties involved. On one side you have a government agency where people obtain seniority largely through age, not skills, and the main skill that is relevant is passing the buck when things go wrong and taking the credit when things go right (really, this is government agencies through and through - not to mention, most people with real skills/brains get out as soon as they can). On the other side you have the dinosaurs of development (not necessarily age, but sizewise).
Somebody earlier in the thread stated this whole project could have been delivered with much lower cost, with just a few devs, in a much shorter time. I'm 100% in agreement with them. The only real complexity of most government systems is the labyrinthine workflows, but they are documents and strictly followed in their paper variants, its just a matter of getting an understanding of this and turning it into software.
My recommended development approach for this project would have been as follows:
1) Hire some decent devs. They don't need to be hotshots, what is being developed is fairly simple from a technical standpoint. Mainly guys who can follow a spec.
2) Take a bunch of people who actually do the work for real, the paper pushers. Take them down the pub and get them rat-arsed. Listen to them bitch and whine about all the idiotic things they have to do in day-to-day operations.
3) Take notes of their bitching! It may help if you are drunk!
4) Any requirements given to you through the official channels are probably worthless. Dump them. They will simply mislead you from what is actually required.
5) Build the system based on what you learned from the drunk employees.
6) Demo it to the stakeholders and hand over.
7) Contract fulfilled.
Oh, and of course.... 4) Profit... erm...
I found RUP to be better than Agile. (Score:3)
Two principles were key and could be used in any methodology.
1) Address risky (new technology, undefined specs, etc.) first
2) Regular time boxed functional builds.
If you can't address the risks successfully, then at least you can cancel the project early.
As George Shultz would say... (Score:4, Interesting)
There are three things I have learned never to discuss with people... Religion, Politics, and Agile.
Re:Because (Score:5, Funny)
You wouldn't understand it.
You would exploit it.
You wouldn't do anything to make it better.
You would waste time complaining about everything.
Burma Shave!
Re: (Score:2)
Re: (Score:3)
You wouldn't steal a keyboard? Would you steal a handbag? A DVD?
Oops. Sorry. Wrong thread.
Re:It is based on Linux.... (Score:5, Interesting)
Just because the failure of one project is caused by .NET does not mean that a project not based on .NET is going to succeed. In fact, traditionally 80% of software projects fail.
This project is clearly failing for the second from last reason. It is also failing because it is not an "agile project". An "agile" project cannot fail and cost Billions because it must always deliver runnable software with a maximum of a few weeks delay if you use some "semi agile" process like scrum or immediately any point if you use some true agile process.
Once you deviate from "Working software" for more than a couple of sprints (everybody can make a mistake) then you are no longer doing agile. I have seen so many "agile" projects which seem to define "Working" as meaning something like "a prototype which would never work at full scale" and so they have never addressed the major problems of their class of system.
If they are "billions" of pounds down whilst doing agile, then they should have already delivered plenty of working systems and have hundreds of happy users. In this case they are a "success" even if they were a bit slower and more expensive than some other projects. If, however, they really haven't delivered anything then what they were doing was an unplanned disaster using "agile" as an excuse for not having a proper plan.
Whilst I know that the "waterfall" method of development is famed for it's failures. Whilst I know that those failures are spectacular and huge. I really don't see how you deliver, for example, 5% of a working mobile phone network. You just have to have a big interlocked plan with a working phone, transmitter, backend, management and interconnection all planned together. I don't believe that such a thing can be done in a true "agile" way and pretending that you are doing it in an agile way is a dangerous fantasy. Only once you have a working network can you start to improve it in Agile increments.
Mandatory requirements and Agile fallacies (Score:5, Insightful)
An "agile" project cannot fail and cost Billions because it must always deliver runnable software with a maximum of a few weeks delay if you use some "semi agile" process like scrum or immediately any point if you use some true agile process.
The trouble is, you can have software that runs and passes some tests, yet still does not meet all of the mandatory requirements for the project and therefore may have no value at all in the real world. You don't get any credit for meeting 90% of the mandatory requirements on a job like this. The idea that having software that runs and maybe passes some tests has some sort of inherent value might be the biggest fallacy perpetuated by the whole Agile movement. It's just not true, and therefore neither is the claim that agile projects can't fail as a result.
Re: (Score:3)
The trouble is, you can have software that runs and passes some tests, yet still does not meet all of the mandatory requirements for the project and therefore may have no value at all in the real world.
The agile word they say is "Working software". "Runs and passes some tests" does not match my meaning of working software.
It's just not true, and therefore neither is the claim that agile projects can't fail as a result.
I see this a bit differently. I take the "working software" bit as it is. I think this means that agile projects are mostly impossible for most things other than incremental software development of changes to pre-existing useful systems and that 90% of projects claiming to be "agile" actually aren't. However, more or less it seems to me we agree. The idea of delivering something t
Re: (Score:3)
Sorry, but you are completely missing my point. My comment wasn't really about tests, it was about the fact that these huge projects are practically all-or-nothing in their success.
For that kind of project, you can't know whether you will ultimately succeed just because you adopt some sort of incremental development strategy. You could spend years of development time and a small fortune in expenses making exemplary progress and getting 90% of your system working fine, but if the last 10% turns out to be imp
Re:Mandatory requirements and Agile fallacies (Score:5, Insightful)
Sorry, but you are completely missing my point. My comment wasn't really about tests, it was about the fact that these huge projects are practically all-or-nothing in their success.
Exactly. You cannot use Agile to build a 100-mile canal, as the whole thing would be useless even if you completed 99 miles.
If the system cannot be useful until a large set of functions are in place and working, then it is not suitable for Agile, period.
Re: (Score:3)
Stop being dumb and bigoted.
I think you failed to notice the shill posting that I was responding to. I didn't bring up .NET to bash it, rather to respond to someone who was trying to compare it to RedHat which is a system that .NET clearly doesn't approach in any of maturity, flexibility or stability.
There are tens of thousands of .NET projects if not more that have succeeded.
That's hardly a recommendation. There are hundreds of thousands of projects that have been based on PHP. There are probably millions of such projects which are based on excels. The fact is that almost every problem system has lots of
Re: (Score:3)
They are choosing a system which has a record of a number of serious disasters and lack of performance at higher system demand levels.
What serious disasters? Do you have any references that of any, especially ones that say it was because of .NET? Those sound like typical things that people who only read Slashdot for news seem to believe. I remember the London Stock Exchange fiasco, but that was because of incompetent people rather than the technology itself. In the last quarter, Microsoft's Server and Tools division recorded an increase in revenue of 11% despite completely free alternatives available.
if for example your Java supplier has a bad security record, to migrate to a different one which is more responsive
Like who?
Re: (Score:3)
I could have told you in advance, just from that list, that the project was going to fail.
Fail, that is, from the perspective of the agency and its taxpayers. From the perspective of the consulting companies, it worked just fine. They wanted big fees and got them.
Re: (Score:3)
Its been done before :D
Re: (Score:2)
Project failures are hardly a programming problem.
Project management, software engineering processes (specially requirement engineering) and design are the most risky. Hiring programmers which are fit for the project is still a risk factor though.
Re: (Score:3)
Having seen both the waterfall model and the agile model I would say that both models needs to be applied correctly to work. The agile is working for incremental development in small steps where you can adjust and adapt as you go, the waterfall works for long iterations but it requires that each step is thoroughly analyzed for problems. It works fine for slow projects but not for quick to market and cheap projects.
Changing process method in the middle of a project is a sure sign that it's time to abandon sh
Re: (Score:3)
Agile can be as foolish a project management method as any other but it seems to have a religious component to it.
Like everything else about software development, from methodology to choice of language to how you name variables and format code.
I find it hard to imagine that people in other fields are as opinionated as we are, with so little evidence to back up their opinions.
(But they're people too, so they probably are.)
Re: (Score:3)
Indeed. No MBA will ever be on the level of insight that a good developer has. As the MBAs know this they consistently hire bad developers, going for the fallacy that more bad developers are better than less good developers and not wanting to hire people that are smarter than they are. While it is true that the number of subordinates you have determine your importance in a bureaucracy, bad developers cannot do complex projects, no matter how many you have. Something these MBAs cretins are not equipped to un