Computers May Thwart 2010 Census

smooth wombat writes "With the Constitutionally mandated census of 2010 just around the corner, it appears the Commerce Department's attempt to use handheld computers to gather census information may not come to fruition. Originally, the contract was awarded at a cost of $596 million to Harris Corporation. However, the GAO has now estimated the revised contract, now costing $647 million, could grow to $2 billion and the equipment may still not work properly. There is consideration that the paper and pencil method might have to be employed to complete the census."

Any history buffs out there?

Recall that Herman Hollerith came up with punched cards for the 1890 census. He founded the company that became IBM. Here's some linky goodness [wikipedia.org].

Re:Any history buffs out there?

Yeah punched cards seem to have worked well for Florida ...

Re:Any history buffs out there?

Yo Chad, whats hangin?

Anyone have any idea...

...what accounts for the differences in the estimate and the cost? What cost(s) were underestimated?

Re:Anyone have any idea...

Government departments often have regulations that require them to put contracts out to tender and give them to the lowest bidder to prevent people handing out government contracts to thier friends.

The problem is if they put them out for bidding as fixed price contracts they probablly wouldn't get any bids and if they did those bids would be very high. So the bids are only estimates. Of course this makes the bidding a farce as everyone tries to put in the lowest estimate they can and sponge more money later once the governement department is locked in.

Re:Anyone have any idea...

So the bids are only estimates.

I've done work on construction estimates that were of similar (~$250M) size that were submitted as "GMP" - Guaranteed Max Price. Yeah, Change Orders will pad that somewhat, and there is some contingency, but together they won't be more than 15%.

Of course this makes the bidding a farce as everyone tries to put in the lowest estimate

Which is why I've always felt that the process should pick the second lowest bid. It's trivial to shoot for the bottom ... it's impossible to shoot for second from the bottom.

Re:Anyone have any idea...

Yay for non-technical decision makers making technical decisions!

My favorite procurement/supply story is from a guy I worked with in the Navy; we needed a switch for some system that required replacement parts to be some super-special reliability grade, so he got the part number and called up the company from home. "Hi this is Billy Joe Ray Bob's electronics supply in Podunk, Lousiana; I need a BR-549 limit switch, how much are they?" The answer was something like $12.

He calls back a week later as Petty Officer so-and-so from the USS Neveryoumind, and then the BR-549 limit switch costs $349. Apparently the super-special reliability grade sticker they put on it after they take it out of the bin costs $337.

Open source

The bloat is occurring because the project is not open-source, it could be done for pennies, but would take 20 years to complete.

1% error

He also said the computers actually are easy to use, with a failure rate of less than 1 percent when tested in the field.

One percent of three hundred million is three million.

Re:1% error

Ya, people are really bad when it comes to big numbers. I was a part of a research study as an undergraduate with the following premise:

You're on a jury for a murder case with the scenario that a tan/brown man seen running away from a murder scene on a college campus. There was not enough of the attacker's DNA at the scene, but they were able to extract a DNA derivative that has matched that of a tan man in custody. Given that this derivative has a 99.9% successful rate, do you feel comfortable convicting the man in custody.

I was the only one in my group of 12 to say "No, I will not convict based on this evidence." No one else understood that .1% = 1/1000. Nor did they realize that our university alone had 20,000+ people on the campus at any time let alone that it was in the middle of a city of 200,000+.

Most people know what "fifty" is. Many know what "one hundred" is. Few understand what "one thousand" is. Too few understand the effects of millions, billions, and trillions.
There's no way I'd convict with a .1% error, there's no way I'd accept a 1% error in the business of millions.

How to get big numbers across

Many people don't understand big numbers... My favorite way to get across just how big ONE BILLION (pinky-to-mouth) is, is as follows:

A rich man wanted his wife to stop bugging him for money. So he gave her $1Million, and told her to spend $1000 a day.

Three years later, she came back, and said she was out of money. So he gave her $1Billion.

She didn't bug him again for 3000 years.

Re:How to get big numbers across

Uhhh, $1 billion is 100 times $1 million

Google [google.com] doesn't seem to think so....

People in stone houses shouldn't throw bricks moron.

Indeed.

wow... ideal role for the XO

of course do a more businesslike version with a larger keyboard... but the XO with custom census gathering application saving the data off onto flash drives would have been perfect... pity the timescale is a bit short now...

Are you serious?

Originally, the contract was awarded at a cost of $596 million to Harris Corporation. However, the GAO has now estimated the revised contract, now costing $647 million, could grow to $2 billion and the equipment may still not work properly.

1.4 billion is one hell of an overrun...and after all that, the equipment may still not work properly?

Is the Harris Corporation currently hiring? I'd like to get me some of that boondoggle.

Re:Are you serious?

The more pressing question on my mind is why they haven't been sued into oblivion. How can you seriously get away with that?

Re:Are you serious?

Corruption.

Is Diebold making the computers?

Lets hope not. The population of the West Coast and the North East combined would come out at zero!

Foreign census experience

I was living in a Gulf country a few years ago when the government there conducted the census. They just sent out an army of 20-somthings with PDAs to do the surveys. I believe most of the survey was multiple-choice, but there were some numeric entries (how much I earned per month and what y rent cost, for example). You could do those with multiple-choice too, obviously, with a selection of ranges.

The whole census survey took about 15 minutes. They collected a lot of data - I'd say there were between 60 and 80 questions. Since I'm a geeky sort of person, I asked the kid how it worked and he showed me - the PDA (a Compac Pocket PC) just ran a macro in MS Office which dumped each survey as a file into a folder. That folder synced via wireless/mobile-phone link to where the main data center was.

The country has a population of about 4 million, and he said there were 200 people doing the survey for several months. Seemed pretty straightforward, and I can't imagine it cost that much - certainly labor and not the PDAs was the primary expense.

Major IT failures seem so common

Bet if you did a study, a serious one, you'd find there's an irrefutable inverse relationship between the amount of money bid for a project and the success of that project. I know it sounds like a flippant witticism, but I'm sure of it, do the research and the figures will prove a direct *causal link* between the amount of money put in and project failure.

I mean, what is it with these large scale IT projects? They take a simple problem and turn it into a money pit. Here in the UK we've had several high profile massive budget IT failures in the last 10 years, air traffic control, national health patient record databases, in fact the more critical it is the more of a spectacular unqualified fuck-up it becomes.

Now, if you got a couple of average hacker nerds and gave then the same specs, but didn't tell them it was for a large scale project, or for whom, they would give you a faultless solution using commodity hardware, stock methods and free software in a few months at one *millionth* the cost we're looking at here. Every one of you here knows it to be true. So, my question is, what goes wrong? How can it possibly go so wrong? Are the people involved complete idiots? Or corrupt?

What are the factors that turn a simple software project into an impossible task? Is it the stress of high budgets? Too many crooks spoiling the broth? And more to the point, when is some bright person going to break from this pattern of failure and realise that to award a major government IT contract to *more than one* complete no-name outsiders bidding a fraction of the cost makes more sense than giving billions of dollars to one contractor and putting all your eggs in one basket?

Re:Major IT failures seem so common

Bet if you did a study, a serious one, you'd find there's an irrefutable inverse relationship between the amount of money bid for a project and the success of that project.

I can refute it without even breaking a sweat. The Manhattan Project. The Apollo Project. The creation of the Polaris, Atlas, and Titan missiles... The creation of nuclear powered ships... Etc... Etc... Big ticket projects all - unqualified successes all.
 
 

I mean, what is it with these large scale IT projects? They take a simple problem and turn it into a money pit. Here in the UK we've had several high profile massive budget IT failures in the last 10 years, air traffic control, national health patient record databases, in fact the more critical it is the more of a spectacular unqualified fuck-up it becomes.

Mostly because we really don't have all that much experience building huge monolithic IT projects from scratch and to spec. The vast majority of the [truly tremendously] big IT projects to date (the telephone system, the networks big banks use, etc...) have been built piecemeal and grown from small beginnings.
 
 

Now, if you got a couple of average hacker nerds and gave then the same specs, but didn't tell them it was for a large scale project, or for whom, they would give you a faultless solution using commodity hardware, stock methods and free software in a few months at one *millionth* the cost we're looking at here. Every one of you here knows it to be true.

I know it's a common conceit of IT workers to believe so. I don't believe for a single second that it's true. 'Average Hacker Nerds' have essentially zero experience in building large systems, triply so for distributed ones.
 
 

So, my question is, what goes wrong? How can it possibly go so wrong? Are the people involved complete idiots? Or corrupt?

Or, just maybe, the projects are Really Hard in extremely specialized project domains.
 
 

What are the factors that turn a simple software project into an impossible task?

The persistent belief that these projects are 'simply software' and thus easy to do. Especially among people with essentially zero knowledge of the problem domain(s) and the issues involved.

Canada had census forms online in 2006

Our last census had the option of filling out the census forms online. I didn't find out how many actually did it, but they were originally estimating 20% usage. Instead of getting the full booklet to fill out, you got an access code.

http://www.cbc.ca/story/canada/national/2006/04/27/online-census060427.html [www.cbc.ca]

While searching for a reference article, i found that there were some issues with Linux users, although they attempted to correct it.

http://www.linux.com/articles/54366 [linux.com]

It is done elsewhere

This is not new and pioneering technology. There are companies [acnielsen.com] that take similar surveys for market research purposes. Have you ever been asked to take a survey at a mall? Have you ever been at a bar when a beautiful woman with a tablet computer asks you to take a survey about cigarettes? I have. The Government is wasting billions of dollars to develop technology that has existed for years.

Why doesn't the government just outsource the whole census to a market research company and be done with it?

Good old government waste.

They could have purchased hundreds or thousands of off-the-shelf PDAs and had a company develop a basic piece of census-tracking software for a mere fraction of what this project costs. Instead they'll argue they need some elaborate, over-priced piece of hardware under the pretense that only something so fancy can reliably handle the government's needs. The best part is that the devices might not even work properly. What in the hell are these companies doing that even with this much money thrown at them they can't do anything right?

Still, that doesn't excuse the government's stupidity. It's like that stimulus package. As if enough money hasn't already been dumped into that some halfwit decided they needed to send out letters informing recipients that they were going to be receiving these checks. In many cases these notices will be arriving barely a month before the check arrives. Sending these letters out has cost the government over $40 million.

It's time the government's budget were capped at the rate of inflation making allowances only for population growth. It's time they learned how to manage their expenses like the rest of us have to.

Re:Another waste of money

BTW, I'm assuming here that a census should be just counting heads; that all of the other questions that the census people ask, such as level of education, are none of their business. The constitution requires that people be counted. The goal was to ensure proportional representation. It does not require all of the intrusive questions that they ask now.

You are not required to answer any other question on the census, either.

You can really just say "nine people live here, go away" and they will.

All that information IS necessary for the government to provide all the services they provide today in a reasonable and efficient manner.

Unfortunately, it would also require that those in charge be interested in reason or efficiency. All they want to do is separate you from money.

So, I agree, but only in that the government should get their nose out of places it doesn't belong in a more general sense. Unfortunately, we could probably argue about what those things are all day.

Re:Another waste of money

....the lowest income quintile keeps getting poorer. If the Federal government is going to do anything to help these people along, it needs to know a little more about them.

Why are we implicitly assuming that the feds should do anything in particular to help these people? The last thing they need is more government 'help'. The 'war on poverty' has been going on for 40+ years now, and - as noted - the poorest group is getting poorer, not wealthier.

If you think Congress makes dumb decisions, then complain about the dumb decisions. But don't cut off their access to information that could help them make better decisions.

Parent - and congress - seem to assume that if you just knew enough about people, you could make good decisions for them. The latest atack on poverty didn't work? Well we just need more data.

But the fact is that you can never know enough about a person's needs and circumstances as the person himself. Leave him alone, quit meddling in his life, lower his taxes ( by not wasting his money on cencus boondogles ) and he will probably get rich on his own.