DoD Ditches Open Source Medical Records System In $4.3B Contract 186
dmr001 writes: The US Department of Defense opted not to use the Department of Veterans Affairs' open source VistA electronic health record system in its project to overhaul its legacy systems, instead opting for a consortium of Cerner, Leidos and Accenture. The initial $4.3 billion implementation is expected to be the first part of a $9 billion dollar project. The Under Secretary for Acquisition stated they wanted a system with minimum modifications and interoperability with private sector systems, though much of what passes for inter-vendor operability in the marketplace is more aspirational than operable. The DoD aims to start implementation at 8 sites in the Pacific Northwest by the end of 2016, noting that "legacy systems are eating us alive in terms of support and maintenance," consuming 95% of the Military Health Systems IT budget.
A consortium of Cerner, Leidos and Accenture (Score:5, Insightful)
Jesus Christ what a waste of money and to the worst possible people.
Re:A consortium of Cerner, Leidos and Accenture (Score:4, Interesting)
We are deeply grateful to Source Forge for providing us with our place in cyberspace for this web site.
http://worldvista.org/AboutVis... [worldvista.org]
Agreed, but this sounds kinda weird too
Re: (Score:1, Insightful)
Their only noncommercial option was the VA's VistA, requiring constant redevelopment to meet federal HITECH and ONC mandates and built entirely around a 1960s-era "key-value" database language called MUMPS. If you're a migraine enthusiast, enjoy night terrors, or otherwise are in the market for a drinking problem I recommend learning MUMPS.
Here's a high-level summary of MUMPS:
There is no database. Everything is stored in a multidimensional array. There are no ints or floats. Everything is stored and ret
Re: (Score:2)
There are no tables, tablespaces, or schemas. Everything in the global array is persistent and is saved directly to a disk.
So the bloody thing is NoSQL and ready for the biggest revolution in computing soon hitting the market - persistent memory such as Memristor and 3D XPoint.
Re: (Score:3)
Re: (Score:3, Informative)
You obviously don't know anything about MUMPS.
I'll just discuss these specific points:
- It *is* a database.
- The "multidimensional array" is some sort of b-tree, the same kind of approach that is used by an RDBMS.
- The language is dynamically typed ("no ints or floats"), which in practice means it's like working in Python or Perl. Internally, numerical data is stored in a numerical format for efficiency.
- There's no need for reserved names, that's correct. Nonetheless, coding practices dictate that you don'
Re: (Score:2)
You obviously don't know anything about MUMPS.
- There's no need for reserved names, that's correct. Nonetheless, coding practices dictate that you don't name a variable "if" or something silly like that.
A variable of NULL would have no problems? As time goes on people are not aware.
I've nothing against MUMPS and your write up informative but that one line caught me.
Re: (Score:2)
Re: (Score:1)
It gave me spots all over, and I still lived!
Re:A consortium of Cerner, Leidos and Accenture (Score:5, Insightful)
Re: (Score:2)
Re: (Score:2)
Re: (Score:1)
It guarantees that the DoD managers of this project are guaranteed private sector jobs for life after they leave governmental "service"... Ain't the revolving door nice?
Re: (Score:2)
Jesus Christ what a waste of money and to the worst possible people.
Think of the money you can earn, working to fix the consortium's system and then a contract to convert their systemto the open source version.
Wow, A cast of hundreds with salaries to three times the norm.
300 million for the hardware/software/salaries (Score:1)
4 billion leftover for all the people in charge.
follow the money (Score:5, Interesting)
follow the money and the answer is in front of you.
Trading one for the other (Score:5, Insightful)
Re: (Score:1)
I know people working on the veterans affairs stuff... if they made this choice, they didn't make it lightly...
In particular their legacy system was a crap ton of post-it notes stuck to computer screens.
The Obama administration has gotten a lot of help from Google's SRE groups moving in to help fix the government with everything they learned at Google. These are not bureaucratic choices being made, they're Google SRE choices...
Re: (Score:3)
I have no knowledge of the particulars in this case, but lobbying isn't even really necessary. It's often just the revolving door: The procurement people on the government side now have very lucrative careers in the private sector to look forward to, and that is something you can never get by going with the open source solution. But who knows, maybe this time they did make the call purely on its technical merits.
Re: (Score:3)
Re: (Score:2, Insightful)
The open source solution was not a good one in this case. You can have religious wars all you want about which language is best, but when striving for worst, MUMPS is a real contender. Please don't copy the database structure of VISTA. It's utterly useless for data integrity -- imagine a database where every field is a string. There are no numeric fields...and what happens? Oh.
The biggest problem is that the DoD lacks the organizational and technical skills at higher levels to use anything other than a
Re: (Score:2)
And if DoD decided to do it inhouse, the libertards would be hanging from the lampposts crying the blues about big government.
Re: (Score:2)
It's utterly useless for data integrity -- imagine a database where every field is a string. There are no numeric fields...and what happens? Oh.
Somehow I don't think a computer system should have a problem with this. Ultimately, in every computer system, every field in a database is an 8-bit string. On some kind of disk. ;) Actual lack of checks is what would give you problems.
Re:Trading one for the other (Score:5, Insightful)
Re: (Score:2)
Regardless of whether they are starting with open source software, or closed source software........if I ever paid $4.3 billion for some software, I guarantee I would be getting the source for it. If the government pays that much for a system, one of the requirements should be that it ends up open source.
At least.
Re: (Score:2)
Imagine if the government contracted Gmail to provide unclassified email for the entire DoD for 10 years for 4.3 billion. (2 million active and civilian accounts x 10 years = $215 a year per account.) Why would DoD want the source code for Gmail just because they are paying for the service? DoD isn't going to turn it into open source and they certainly aren't going to try to maintain a separate Gmail system. Google isn't going to bid $215
Re: (Score:2)
Also, as to your gmail example, if the DoD plans on storing medical records in someone's crappy 'cloud,' they are giving themselves even more problems. You want to be able to control that data.
Re: (Score:2)
When things go right, you don't need the source code, but when things go wrong, you'll wish you had it.
Again, Why?
Do you think DoD is going to tell the company how to rewrite the code to fix it? When you write the contract you put in performance specifications (remember five 9s?) and penalties for non-compliance. You don't spend months reviewing the vendor's software code to find edge cases that cause problems. DoD is paying them for their expertise, manpower, and infrastructure that DoD doesn't have to maintain.
--
Re: (Score:2)
Also, programmers write better code when they know people will be looking at it. The open source method for motivating better programming.
Re: (Score:2)
Also, programmers write better code when they know people will be looking at it. The open source method for motivating better programming.
citation required
Re: (Score:2)
Re: (Score:2)
The first rule of SIPRNet is you don't talk about SIPRNet. -_^
--
Re: (Score:2)
In this fictitious example, DoD is paying Gmail for a proven email system and the personnel to keep it running, up to date, and secure.
The other huge difference in your example is that DoD would merely be paying for something already built and functional.
TFA is about three companies building a system from scratch. Even if they use something that mostly already exists, it will be highly customized for DoD, to the point that it will be different from anything they have already done, and nowhere near "proven" or "tested". Since the history of those three companies shows they want nothing more than keeping their clients locked in as long as
$9 billion dollar project? (Score:1)
Try paper, it still works.
Re: (Score:3)
Paper doesn't scale to the level required. Trust me, I've attended presentations from ex-Googlers on the topic.
Especially, when the paper weighs so much that it started deforming a building...
Re: (Score:2)
Re: (Score:2)
Maybe it's in the basement and menacing the soil's integrity ; earth itself can collapse under the weight of a few years of DoD paperwork.
$4.3 billion == guaranteed failure. (Score:5, Insightful)
As far as I can tell, any IT project costing a billion or more is 100% guaranteed to fail.
Also, it sounds like they decided to source IT from Lufier, Mephistopheles and Satan, which incidentally also guarantees it to fail.
Re:$4.3 billion == guaranteed failure. (Score:5, Insightful)
Tom DeMarco talks about the air traffic control software project in one of his books. The description of the hopeless situation in that case supports your idea.
I think when you have a lot of people's butts on the line and so failure is not an option but stagnation IS, what we would perceive as failure is almost certainly coming. You can retire without any fallout so long as you make sure nothing happens for 15 years. It's easy to do: Just make the specs vague, self-contradictory, and long. Very, very, long.
The project won't fail, but it won't succeed either. And you're safe, which is all that matters.
They would do much better to set up a few small teams and have them compete to build something with enough in common so one can be replaced by the other. And starting with the open source base would make sense there.
Re: (Score:2)
Just make the specs vague, self-contradictory, and long. Very, very, long.
When you get to software on this scale just writing the specs is an insanely difficult problem. Most big systems fail due to poor requirements, and I think that is because good requirements are a LOT harder to get right than people appreciate, especially for waterfall-style projects that are always the favorites of big RFP projects like these.
Re:$4.3 billion == guaranteed failure. (Score:5, Insightful)
As far as I can tell, any IT project costing a billion or more is 100% guaranteed to fail.
No kidding.
If you pay everybody $200 000 per year, that equates to 21 500 man-years (!) of work. I don't know what kind of problems in record keeping they're going to solve, but for that kind of money it'd better involve employees doing that in gold plated jets flown by an artificially engineered unicorn that continually snorts prime-grade cocaine.
Re: (Score:2)
Re: (Score:2)
Screw your implicit apologistic bullshit.
You could hire 100 lawyers and pay them $1 000 000 yearly for 10 years and still have 3.3 billion dollars left.
This is the industry suckering the government out of their cash and you know it.
Re: (Score:2)
Re: (Score:2)
You are veritably ridiculous.
You are arguing that there are lawyers that could ask 1 billion dollars yearly. 'Because they'd still be worth it'.
Re: (Score:2)
Re: (Score:2)
That is not how that works. At all.
You have at best a primary school level understanding of capitalism and economics in general.
Re: (Score:2)
Re: (Score:2)
Accenture? (Score:3, Insightful)
Project already failed.
Re: (Score:3)
OK, years late, most promised feature not done/working, Budget 3x+ over and you will move to a proprietary commercial product with 100% lock in. ;)
Sweet, yes would be fun to see who's back accounts/family members got hired to land this future failure in motion
You left out the part where no senior executive service (SES) people get fired but middle-level people who actually tried to make it work get the blame. Meanwhile the Beltway Bandits make off with millions and hire the SES people for fat salaries as a reward for sending all the money their way.
UK NHS (Score:4, Informative)
Sounds alot like the disasterous Nhs epr systems. After 20 years weve had progress but the number suppliers is down 1 and a major cash sinkhole
Re: (Score:3)
I thought about the NHS program when posting this, described as "the biggest IT failure ever seen" [theguardian.com]. After £10 billion+ was spent, Her Majesty's government largely abandoned the effort, though the linked article notes Computer Sciences Corporation declaring victory as 3 of 220 NHS trusts managed to use portions of the system. I first heard this story a couple of years ago on a shuttle bus to the headquarters of a large privately held EMR vendor in Wisconsin, when I noticed the accents around me weren't
Re: (Score:3)
We do have interoperability standards : I used to work for the department that specifies them.
They just don't follow them. Case in point : the meta-standard (HL7 v3) that we used for our messaging had a mechanism for not just sending NULL values, but also sending a reason why they were null. (e.g. - the value wasn't measured, etc). The vendor had no truck with that, and was using magic numbers instead (e.g. baby weights of 9999g which is outside the realms of sanity for a newborn). I was tasked with revampi
Re: (Score:2)
Sometimes updating legacy systems is a real pain. (Score:1)
Re: (Score:2)
"Space is hard" and this is harder, because it's about ourselves.
This has no chance until someone like Pres. Kennedy chooses to do it "because it is hard" to do.
I think this, updating legacy system (something I do), is close to the second problem, which is that no one person or team has a grasp on how it works now, and what ever is put in place will be in the exact the same situation: no person knows it all. The only way I've seen this addressed is that after the Big Problem is cut into the Smaller Problem
As an Accenture stock holder (Score:1)
As an Accenture stock holder I think this is definitely a step in the right direction.
$4.3 billion (Score:5, Insightful)
wow fuck. imagine how much advancement in software libre could be had for $4.3 billion if the contract had been awared. hell, even 1% of that would make a big fucking difference. someone - such as the gnumed developers to take even one random example - could, with help, have developed a medical records system for ohhh i dunno... the U.S. Dept of Defense, with that kind of money. just to take a random example, y'ken.
gnu project? really? (Score:4, Insightful)
And who, in the gnu community, is going to take on the responsibility for all the enterprise scale stuff that needs to be done. I can see lots of folks wanting to scratch their particular itch by coding up some piece, but who's going to do the architecture design, ride herd on the developers, etc.; make sure that the documentation gets done and is usable and readable (because, ya know, all those packages out on github and sourceforge are ever so well documented)..
I mean responsibility as in "be willing to stand up in front of Congress and explain your progress or lack thereof". I don't see a Linus or Theo or Eric or, gods forbid, Richard, filling that role.
Re: (Score:3)
You have no idea of scale.
Re: (Score:2)
Really? Put that much money into open source and you'll have one fork for each $20 put in. Then the project is going to be complete in the mythical Year of the Linux Desktop (TM)
Speakings of the OSS in question (Score:2)
They use MUMPS. I know its all supported by some people and I know the flames are coming, but really? MUMPS. I'd say integration could very well be considered a pain point in the language. In my cursory investigation, the recommended integration for other languages / technology chains is a Node.js based web services adapter... oh well.
Re: (Score:2)
Tyson [thedailywtf.com], is that you??
Part of the problem (Score:4, Insightful)
That is part of the problem .. trying to design a drop in replacement that replicates the current functionality and interoperability with other systems.
With government especially, you have lists of exceptions and custom one-off code to get something working, that it becomes impracticable to replace it without an equal or additional number of exceptions.
It's the kind of system that benefits from a "flush it all away" mentality of defining new standards and sticking to them.
Eating them alive (Score:2)
Well, it'll solve the problem of the system eating them alive in terms of maintenance and support. Now it'll be eating them alive in terms of development costs instead.
No, wait, they'll need to keep the legacy systems running until the new ones are running, so it'll still be eating them alive in terms of maintenance and support too.
Accenture? (Score:4, Informative)
Accenture? Better double that initial estimate to $18 billion, and count on it rising further. :(
not enough bribes by OSS (Score:3)
US Government purchasing "works" by the payment of bribes. Usually, these are not simply cash payments, but the opportunity for lucrative "consulting contracts" at the providing companies for senior Penagon and Civil Service officials after leaving government "service". I've seen it enough to know that saving money, at equal or better performance, will not get a government contract. Maybe, if enough congresscritters and/or senaturds are bribed with campaign contributions and/or honorariums, they'll push a deal one way or the other, but that rarely has anything to do with saving purchasing costs.
Re: (Score:2)
Exactly !
In fact, people signing large contracts always expect large kickbacks in return, at least 15% of it.
This will probably finance the next presidential campaign.
Why? (Score:1)
We all know the US department of defense demands a fixed-cost project then signs an open-cost contract. The usual excuse being the contractor must build the software from scratch. Since this project copies the contractor's existing system, the government can easily demand a fixed-cost contract. Like that's going to happen!
So they're paying $9 billion for something that's already been built. It sounds overpriced already.
The US DOD is always saying they need a do-everything system that prevents data fiefd
VistA is a nightmare (Score:5, Interesting)
i was interested in VistA and what all the fuss was about, so i decided to check it out. turns out the backend is nightmare code that would would swear was machine generated. after some investigation i found out it's MUMPS (Massachusetts General Hospital Utility Multi-Programming System) [wikipedia.org] code. a lot of useful stuff started way back in the 1970s... but MUMPS is a 1960s nightmare come to life.
think i'm exagerating? here's a module from VistA's code [github.com] which was apparently updated in 1989.
DENTA1 ;ISC2/SAW,HAG-DENTAL TREATMENT DATA SERVICE REPORTS ; 1/10/89 11:08 AM ; ;;1.2;DENTAL;**24**;JAN 26, 1989
D:'$D(DT) DT^DICRW S %O="OPT",U="^",S=";",O=$T(@(%O)),DENTV=$$VERSION^XPDUTL("DENT") I $D(^DOPT($P(O,S,5),"VERSION")),(DENTV=^DOPT($P(O,S,5),"VERSION")) G IN
K ^DOPT($P(O,S,5))
F I=1:1 Q:$T(@(%O)+I)="" S ^DOPT($P(O,S,5),I,0)=$P($T(@(%O)+I),S,3),^DOPT($P(O,S,5),"B",$P($P($T(@(%O)+I),S,3),"^",1),I)=""
S K=I-1,^DOPT($P(O,S,5),0)=$P(O,S,4)_U_1_U_K_U_K K I,K,X S ^DOPT($P(O,S,5),"VERSION")=DENTV
IN I $P(O,S,6)'="" D @($P(O,S,6))
PR S O=$T(@(%O)),S=";" S IOP=$I D ^%ZIS W:IOST'["PK-" @IOF K IOP
I $P(O,S,7)'="" D @($P(O,S,7))
E W !!,$P(O,S,3),":",!,$$VERSION^XPDUTL("DENT")," ",$P($T(+1),S,1),!!,$P(O,S,4),"S:",!
F J=1:1 Q:'$D(^DOPT($P(O,S,5),J,0)) S K=$S(J0 S Z2=Z1
G:Z3=0 W I Z3>1 S DIC="^DENT(225,",DIC(0)="AEMNQ",DIC("A")="Select STATION.DIVISION: " S:$D(DENTSTA) DIC("B")=$S(DENTSTA[" ":+DENTSTA,1:DENTSTA) D ^DIC Q:Y
Re: (Score:3)
Wow... I mean... wow. This is the definition of a "write-only" language. I'm pretty sure you'd need external documentation just describing what these routines do, because it sure as hell looks like you're not going to derive it from this encryption disguised as source code. I thought perhaps you had chosen some particularly horrible section, like maybe it was a data definition of some sort. Nope, after sifting through a bunch of code, it pretty much all looked like that. My brain hurts just trying to p
Re: (Score:2)
If you expand the crazy one-letter commands into their full length and color them, maybe it's a bit less terrible.
Re: (Score:2)
Perhaps, but that's not what they're working with, since this is the original source. This language and it's horrible style was obviously invented in a day when the size of your variables and source files was actually a real consideration, and machine efficiency was prioritized over programmer efficiency. It looks like assembly language for distributed databases.
That's precisely the opposite of what most large-scale enterprise systems need nowadays. Computers are incredibly powerful and cheap and program
Re: VistA is a nightmare (Score:1)
Anyone able to use a touring complete language in a productive way can learn basic M in a day or two.
Expand the commands from their one-letter form and a programmer most likely can read the code within the time it takes to read up on the language: what the 26 commands do, how to apply conditionals to individual statements and the format to chain commands (use of syntactic whitespace) together inside a routine.
Using it to the fullest will take longer because one needs to grasp the elegance of the design to r
Re: (Score:2)
Anyone able to use a touring complete language in a productive way can learn basic M in a day or two.
Yeah, the hospital I used to work at would take tech school kids and teach them M in six months, and then have a competent programmer. It's not work I wanted to do, but it's a real job. Jesus, hasn't anybody here done assembly?
Using it to the fullest will take longer because one needs to grasp the elegance of the design to realize why constructs commonly seen in other languages seem to be missing in M.
The
Re: (Score:2)
OK, so build a new system, a superset of MUMPS that adds the needed functionality but continues to support the legacy code as well as some newer, better syntax.
That way you can transfer in all the old crap and replace it bit by bit over some time. I'm sure I've even seen technology that analyzes executables and re-factors them - perhaps similar techniques could be applied to the old MUMPS source. Sure it's hard, but with $9B and several years I'm sure a handful of CS guys at a university department could ac
Re: (Score:2)
Re: (Score:2)
The one interesting thing here is that if you rewrite the execution environment, without creating a new language, you should be able to use the old code as tests for the functionality of your new code. You need some way to set up and tear down state for the snippets you're going to be executing, so the old execution environment that ran the system in production probably won't be suitable. Your best hope would probably be starting with a new "implementation" that can parse the code and either interpret it, o
Re: (Score:2)
That code is written like every byte costed $500. Don't let this reflect badly on M. The problem is they used one character variables and no formatting. You can wrote worse code in C. M is perfectly readable if you use good variable names and formatting.
Re: (Score:2)
Re: (Score:2)
You can write MUMPS like that, sure, but then you can either write a good kernel in C or win the IOCCC. If one sticks to modern conventions such as using full length keywords, not refedining keywords and actually indenting (pretty much all things you can also do in C, even if redefining keywords is technically illegal, I've never seen it not work), then the code as far as I can tell looks fine.
But even if all the code looks like that, for $9,000,000,000, you could reformat then refactor the entire lot, then
Re: (Score:2)
E W !!
Couldn't have put it better myself.
Re: (Score:2)
Of course, if you look at the individual page of code for a while, you'll decipher it easily enough. Whether it's manageable or not depends on whether the overall structure is reasonable or not.
Re: (Score:2)
It would have been best to hire some in house programmers , certainly could have been done with some portion of the few billion, and have them write a new system, which could then be made public like VISTA. This way the government owns it, instead, there will be an opaque system and the government will be held captive to extortion from providers.
Re: (Score:1)
Cerner does not use M for anything. Cerner developed an in-house language called CCL (Cerner Command Language) which is standard SQL with additional functions built in, mostly for writing, saving and running scripts. All custom functions written for Cerner are typically done in CCL. The backend server code is almost entirely C++ on the backend, with a few pieces done in Java. Frontend code is roughly the same mix.
As for databases, it's Oracle RDBMS and MySQL.
Source: Former Cerner software engineer.
Queensland balls up (Score:1)
Of course it's costing 95% for support and maintenance, there are no licensing or purchase costs. Would that last 5% be hardware costs, management costs, biscuits for morning tea ?
When is a commercial EHR system not proprietary and when is an open source electronic health record system proprietary ?
Governments all over the world are busting to give OUR money away to big corporations !!!
Projections of $11B over 18 years, who's a betting man ? will it double or triple over the 18 years ???
The health departmen
Has anybody checked out the VistA help doc !!! (Score:1)
Not all Open source is good. (Score:2)
Re: (Score:2)
VistA is a shit legacy system,
From Stroustrup's FAQ:
What is "legacy code"?
"Legacy code" is a term often used derogatorily to characterize code that is written in a language or style that (1) the speaker/writer consider outdated and/or (2) is competing with something sold/promoted by the speaker/writer. "Legacy code" often differs from its suggested alternative by actually working and scaling.
The VA system probably works. 50 bucks says the $9,000,000,000 system will never work.
Re: (Score:2)
Guess which large-scale EMR physicians prefer above all others? That would be VistA [hitconsultant.net]. I've heard the same from colleagues, and found it reasonably sensible back when I rotated through the local VA as a family medicine resident. It was fast and fairly benign on the infuriation scale. Of course, the VA is apparently working with Accenture [healthcareitnews.com] to update VistA, and are eventually looking to replace it with a commercial system. I have a feeling many VA docs will offer this to be prized from their cold, dead hands.
And
Staggering Amount of Money (Score:2)
As an outsider, who writes software for a living (proper, highly available transactional systems [finance industry but I do know some general stuff]). This amount of money is simply staggering. Even if we assume the published number (4.3B), 3% inflation, a relatively aggressive annualised ROI and 10 years over which to apply the costs, that turns into between 80 and 160 [20% ROI to 10% _annual_ ROI] million dollars per year in costs. IN COSTS. Even if you margin those costs at 33% (profit is already accoun
Cheaper than licences tomake OS S/W interoperate ? (Score:2)
Seriously, why not have paid for licences (if required) and make the OS software interoperable with commercial systems?
With the added benefit that the result would mean the DOD and VA systems would be compatible !
95% tells us nothing (Score:2)
If they were only planning to support legacy systems, of course that money is designated to support legacy systems. The only way to drop it from 95% is to increase the budget and add other line items, such as new systems.
Re: (Score:2)
Iit is the only way to guarantee it can be fixed.
I offer MS Word in evidence.
Re: (Score:2)
I offer MS Word in evidence.
You'd be better off choosing as an example something other than a piece of software that a significant percentage of businesses on the planet pay for and use with no significant problems whatsoever. It could be argued that MS Word may be one of the most successful pieces of software in history. There are plenty of examples of terrible closed-source, proprietary software. Why not use one of those?
Re: (Score:2)