Bush Administration's E-Mail Deluge May Overload Archive System

Lucas123 writes "The Clinton administration generated 32 million e-mails. Bush's administration has generated 50 times as much data — 140TB, 20TB of which is email — which soon will have to be archived through a new government-built records management system. The new system may not be up to the task because the technology behind it may not be able to handle the sheer volume of data along with the fact that the Bush administration has been slow in providing the National Archives and Records Administration (NARA) with needed information about the records, according to a Computerworld story. Questions have also been raised about millions of missing e-mails from between March 2003 and October 2006. 'It wasn't until this summer that an intensive effort began to share information,' said Ken Thibodeau, director of NARA's Electronic Records Archives."

It is Clinton's porn stash!

The other 120 TB was probably just Clinton's porn stash that the Bush administration found while purging off records.

Re:It is Clinton's porn stash!

Probably 120 TB because of the larger image sizes needed to accommodate fat chicks.

Number of emails generated.

"The Clinton administration generated 32 million e-mails. Bush's administration has generated 50 times as much data -- 140TB, 20TB of which is email -- which soon will have to be archived through a new government-built records management system.

Well, to be fair, email wasn't quite as popular during Clinton's administration as it is now. Then again, the 400GB of e-mails that the Clinton administration must have generated (if it is 50 times less than 20TB) must have been rather hard to store when he left office.

Re:

Well, most of that 400GB from Clinton's administration was dirty pictures of interns. In all seriousness, though, I don't think the problem will be finding a way to store all that data. The real kicker will be finding information you need in it. Seems to me like the best way to hide relevant and/or damaging e-mails would be to have them stored right alongside truckloads of chain letters.

Re:

It isn't storage and it isn't finding it, the problem is preserving it long enough to look through and index it. I'm sure that Google and companies that do similar work have the technology to do it. I'm also quite sure that for the right price the Federal government could obtain software to do most of the heavy lifting.

The problem is that the Bush administration deliberately migrated only partially to a new system leaving it in a state of constant risk for bit rot and corruption. It's hard to say how much o

First, not enough emails...

...Now too many many emails.

Whining is Washington's most favorite thing to do.

Text only, no html

Start by mandating text only mail.

No more fancy signatures and html crap will cause a 60-80% drop in volume if not more.
Mandate the Usenet way with replies after the original, (it will) teach people to cut irrelevant repeats.
Stop the addition of stupid and ineffective disclaimers.

Teach the use of (ftp) servers for sharing large documents, no more Microsoft sized attachments, send a link.

Re:Text only, no html

I would rather buy another hard disk than waste precious time editing the mail I'm replying too, in most cases it simply isn't necessary. For Usenet it's a different story as many people read it, so it's worth the effort.

Re:Text only, no html

Longer email threads seem to end up forwarded and brought to the attention of many people you never expected at the outset.
Judicious editing of the emails to include only the relevant sections for the replies, giving the context of the emerging thread of conversation means that someone being brought up to speed with that segment of the conversation doesn't need to trawl through masses of irrelevant junk to get at the meat of the issue.
I tend to do it as an efficiency gain, rather than taking storage space into account. All comes back to that quote you hear people come out with after sitting through a bad movie "Well, that's an hour of my life I'll never get back". It may only be a few minutes at a time, but they mount up over time. Plus, crafting things to cut to the heart of the matter puts things into sharp perspective, and means people are far less likely to digress, saving even more wasted time.

Re:

When not on Slashdot we're expected to read the message we reply to.

Deleting the bit that's already answered, not relevant or whatever can hardly be called 'editing', it has more to do with comprehension.

One of the worst things for the latter is a typical corporate Outlook mail exchange (I know that word...) with at the bottom text that hasn't been read for the last ten replies.

Re:Text only, no html

If it anything like our corporate mail server I would bet you the number one space filler is people making minor changes on documents then reattaching them and forwarding them back to the same 50 people who just got the previous version of the document, repeated over 100 iterations as the email soon becomes a 2GB mess.

Re:

Some database driven mail servers like Citadel, Exchange, Zimbra and probably Domino support only storing the message and attachments once no matter how many people it was sent to.

It goes a long way in preventing the attachment * user mess.

Re:Text only, no html

Did you stop paying attention to email in 1995?

No more fancy signatures and html crap will cause a 60-80% drop in volume if not more.

I know you hate it when your mom or the boss' secretary at work sends out a cutesy formatted email but some people can actually use HTML email effectively in lieu of sending a document or a link

Mandate the Usenet way with replies after the original, (it will) teach people to cut irrelevant repeats.

Irrelevant repeats for you may be important context for someone else.

Stop the addition of stupid and ineffective disclaimers.

Often times, those disclaimers are required by law. Most people don't add them for fun or to make themselves feel important.

Teach the use of (ftp) servers for sharing large documents, no more Microsoft sized attachments, send a link.

FTP? Are you serious? Sending documents by carrier pigeon is more secure and reliable than FTP

Re:Text only, no html

Sheesh. I call this phenomenon "technological puritanism". All tech must be ugly! 80 columns should be enough for anyone! Fixed-width fonts were good enough for my granddaddy, they were good enough for me, and they should be good enough for everyone! Words are worth a thousand pictures! Get off my damn lawn!

Nothing personal, but if people like you were in charge of the world, we'd all be living in gray, cast concrete cubes. Think of the efficiency! No more wasted paint. You can just make a bigger house by stacking the blocks and adding a ladder.

Most of us *like* color, pictures, paragraphs, and most of all, convenience. Use FTP when I can just add an attachment that goes directly to the source? Give me a frickin' break. No one gives you respect points when you prove how miserably you can live.

Let's put this in perspective... that 120 terabytes costs 12,000 dollars in hard drives. Retail at Fry's. The entire output of the Bush Administration costs less than what they probably spend on coffee in a month.

P.S. And, yes, this is from someone who used a teletype in high school, and was ecstatic when we got a 300 baud modem (whoa! It's almost 3 times faster than the ol' 110!) and a Televideo terminal. Those days were not better.

What's up?

It hasn't helped that the Bush administration has been slow in providing NARA with needed information about the types and volume of data that will need to be archived. It wasn't until this summer that an intensive effort began to share information, Thibodeau says.

I can understand the reasoning that for national security, some information needs to be kept secret. The thing is, the more I hear of this administration's obfuscation of their communications and dealings, I can't help but wonder what in the World they are hiding.

Shadowy Government

Whenever I receive news that information that we're supposed to have access to from the Bush administration has gone missing, it makes me queasy. There's so much secrecy surrounding random little things that it's started to make me paranoid. Maybe it's just me wanting to blame the last eight years on a scapegoat, but I feel like someone at the top is trying to hide something really big and succeeding.

Re:Shadowy Government

I have understood this outgoing administration to be more than secretive. they're positively paranoid and the only administration in memory that was similar was Nixon. All internal memos have been classified first. Declassification only happens when there is a strong and abiding reason why the memo should be declassified. Contrast that with Clinton, where all internal memos are not classified, unless there was a strong and abiding reason why the memo(s) should be classified.

When Bush announced that his administration would immediately prepare for a transition (before the 4th of November, which was election day in the US), I assumed that the first course of action was that this Bush administration would do what the last Bush administration did: [Rip] the hard drives out of their computers and tried to erase "sensitive" computer files in the White House and West Wing. [consortiumnews.com]

To say that the Clinton Administration started with a "clean slate" was an understatement. Later, Clinton lawyers ignored the dangers of historical archive deletion when faced with Republican destruction of historical records. Presumably, they wanted a "pass" from future Republican administrations.

Republican administrations tend to be very secretive. Democratic administrations tend to not. I shall expect the Obama administration shall have to purchase all new computers -- or at least hard drives -- in order to simply start up in their first week. This is a horrid waste of taxpayers' money all in the name of whitewashing one's past deeds (for good or ill).

Due to record-keeping, we now know that Nixon did know about the Watergate break-in. And we do know that he was very interested in its coverup. Nobody can be prosecuted at this time for that (those who were found guilty have all ready served their time). I would be very interested to know if Reagan's CIA planted the stacks of AK-47s used as evidence by his administration that the attack on Grenada was justified. And we still do not know everything about the Iran-Contra affair. These historical records are worth keeping because, well after the Statute of Limitations, America gets another look at how an administration dealt with the world.

It is a shame that any Administration is that interested in "rewriting history" in order to unfairly burnish a legacy, which in the case of "W" is hardly salvageable.

Re:Shadowy Government

No; you are partisan when you think an accusation against one side can be answered by an accusation against the other side. They are both bad (they are US politicians; corruption is so endemic that it's legal and called lobbying), but Clinton's presidency ended about eight years ago and isn't something worth discussing now.

The questions are; how to make sure Bush follows the law for what he still does? How to make sure Obama doesn't start off like Bush?

Not the same thing.

Perhaps you're too young to remember, but Clinton's administration had a problem with missing emails during investigations too (Lewinsky, why hundreds of FBI records on their political enemies ended up in the White House, illegal campaign donations from China, etc).

Yes, but there is a magnitude of difference in importance between lost emails about blow jobs and a little dirty money, and emails about the loss of privacy and civil liberties of US citizens, torture of POWs, and the various other nastiness that GWB et al are suspected of. Much different.

Re:

Well, Clinton never tried to insist that his VP wasn't part of the executive branch,

It's called "Unitary Executive." That is, there's only one guy in the executive branch that gets to make the decisions. It's entirely up to the President how much of a role he gives the Vice President. Under George Washington, John Adams lamented that the only thing he could do was preside over the Senate and then, he had no say on anything unless there was a tie. It drove him nuts.

If you read the Constitution, Article II groups the Vice President in with the executive branch, but the ONLY place it provi

How much is spam?

How much of that is spam? I can imagine they are not allowed to delete spam. Spam has increased, so this would mean that all of it is still there.

The rest can mean a lot of different things. I am forced to work (otherwise no food) with 150MB excel files that I would love to put in a database and would take up at least 10 times less space. And I am not even talking about speed increase and ease of use, because somebody else has the file open, so I can not change the content.

Or perhaps Clinton did not keep everything. Or ...

If they hadn't gone to exchange....

The Bush administration moved the White House from a Notes/Domino based system to a Microsoft Exchange based system.

Before moving, they'd had no downtime -- even when congress was taken out for 2 days by the code red word (they were on Exchange).

In moving, they mysteriously 'lost' all their backups for a period of time that was suspicious as hell, and now they can't scale to handle the capacity issues they face.

In a Notes/Domino world, this kind of archiving problem wouldn't be all that hard to deal with. You'd just need enough storage for it, and create archives per week/month/year (or an archive per individual's mailbox, or whatever) to put on as much hardware as was required. I single checkbox would be all that was needed to have it encrypted as well.

Oh well. I guess if conveniently "loosing" mail when you don't want it found is one of your design goals, than you probably want to migrate to something less reliable.

Dear staff

It has come to my attention that as I prepare to leave office my previous instructions to make all email and other documentation available to the shredder was incorrect. The correct policy is to make everything available to the archiver. If you have any concerns please feel free to pick up a copy of the standard presidential pardon boilerplate from my secretary's desk. Thank you, W

Re:What the hell does the summary say?

Besides, only 140TB (or 20 TB)? That's child's play for any competent DB admin, never mind only about $2k worth of hardware to hold it.

Assuming that none of it's been put into the archival system yet, that means they're dumping 140TB on it in one go.

You index 140TB on $2k worth of hardware and come back to me when you're done. Hopefully I won't have died by then.

Re:

Maybe not $2k worth of hardware but $200k will do. Which is still peanuts in government terms. They probably spend that amount on paperclips and toilet paper in the pentagon alone.
Honestly, storing and indexing 140TB of e-mail is a trivial task when you can apply a six digit budget to it.

If their "archival system" blinks at the sight of 140TB of mostly text then it doesn't even deserve the name.