US Government Upgrades RAM

Deep Throat writes "Techworld has the scoop on a new super-sized RAM disk that the US government has just bought for a few million dollars in order to speed up searching through huge databases. It's 2.5TB! The VP of the company that made it says it is for Washington DC and searching databases but won't say who. Techworld explains why it reckons it's the Department of Homeland Security searching in the NSA and Pentagon databases for terrorists. And apparently the government is 'very happy' with the purchase and thinking about getting more."

Don't be paranoid

[afra242]

Before many users start discussing the privacy laws and what not, it should be noted that the data being stored is probably not new. It's the medium on which it is stored on, which is.

Even without this, the old database could have been searched for some terrorists. Nothing has really changed.

Re:A trivial expense

[Anonymous Coward]

In and of itself, $4.7 million is not that much money. It's when the government spends $4.7 million of taxpayer money frivilously on this and 10,000 other similar projects that it adds up.

Re:Google?

[ThogScully]

While a good point, perhaps, I'm sure the government has no interest in even considering that unless they have the necessary government classification clearances, which I'm guessing would be pretty high given the assumed use of a database of this magnitude.
-N

Re:Not for the DHS

[Mysticalfruit]

More likely, the NSA's already got a dozen of these things and loves them...

OMG my rights online

[stratjakt]

You have no idea what it's for. The list of known terrorists and their acquaitances is relatively short, I cant imagine more than a few gigabytes being needed.

Perhaps it's to store tax returns so the government can mail you your refund check faster. (Job required, sorry).

Maybe INS (or USCIS or whatever they're called) want to track the tidal wave of benifits being handed to Mexican illegals.

I'm a little tired of all this Big Brother speculation. Get over it.

SMASH THE GIANT COMPUTER!!!

[Qrlx]

Sure, it could be used for good. But you know and I know that it will eventually be used for evil.

Re:Required line

[maxwell demon]

Sorry, but the best I can imagine is a RAID of those things.

Re:Lets see

[RoundSparrow]

You can put a lot of personal data in 8KB.

You assume the government is efficent :)

For those talking about restoring data on it...

[Vthornheart]

... it sounds to me, based on the article, that the theoretical use for it would be more like a giant, freakin' enormous Cache system than an outright storage system. In otherwords, on the beginning of a query, all pertinent info is copied to the Solid State drive for analysis. There it is analyzed, and wanted data extracted at astoundingly fast speeds.

Perhaps we should begin to consider the implications of this step in analytical ability. Every techie knows that the Government (in cooperation with major Corporations) has had the methods in place to track individuals... RFID tags, GPS locating (in automobiles as well as hand held units. You can tell your position, but the position is also transmitted back... Five Star anyone?), Cell Phone triangulating, and thanks to the Patriot Act, the Government now has legal access to the records of pretty much any transaction we make with bookstores, libraries, etc (and probably more places as well... and this isn't even taking into account information that they might be recieving in ways that we do not know of).

The thing that (we shall assume) they didn't have before was the ability to instantaneously cross check this information. (I assume this because... well... why would they have bought the drive otherwise?) Now that they can check such information so quickly, will we be brought into an era of "Total Information Awareness" as the government spoke of not too long ago? Does this smell the same as the Thought Police to anyone else?

To me, this presents at least the intent by the government to achieve total information awareness, if it doesn't actually achieve it. And the intent is bad enough. Perhaps they're not reading our minds, but the ability to monitor our actions in such vast varities of levels comes pretty close to doing so.

Re:Very interesting because...

[Lordrashmi]

Can I get some sources to document this?

Re:Google?

[Anonymous Coward]

Actually no, Google crawls a vast amount of information but a very small percentage of that information is useful (specifically the links and keywords). Google also remebers common queries and plays other tricks to speed up searches. When your looking at a 2.5 TB database it is very different. First of all history doesn't matter as much since it is unlikly that they will be running multiple searches on the same thing. Second the 2.5TB is all meta data, so its all relivant, so it all must be searched. Third a query is likly to be much more complex then a normal google query. Google has one index that can be clearly defined in an alphabetic way. The govenment has many data bases that are indexed multiple different ways. This makes searching for connections between databases very difficult. In short Google is highly optimised for a specific type of search and probably will not work in the much fuzzier realm of inteligence.

Re:Don't be paranoid

[ArsonSmith]

Ohh, good I'm glad it's all old privacy violating data rather than something new. That sure clears my worries.

Re:Google?

[dj245]

Customs and Border protection (specifically, the old "Immigration" branch) could benifit from distributed computing for their database. They scan a car licsence plate and run a search while they ask the usual questions- where have you been, where are you going, got any meat? When the search completes they look at it and send them on their way. Quick searching could allow the CBP agent to notice that the car was stolen and that the drug dealer always carries a gun, so that they could wave them through and avoid conflict.

Re:Google?

[RickHunter]

True, but as I said... (Possibly in another post) Google has some very good people working for them. Like, a sizable number of the major contributions to graph theory research over the last ten years major. I'm betting the USG could also deliver a (deliberately fuzzy) list of requirements to them and get back something that'd do what they wanted.

Re:Very interesting because...

[Anonymous Coward]

None credible. It's a conspiracy theory. It has been denied repeatedly by the United States government. That does not mean it is not true, no, but on the same token, NASA denies that they didn't fake the moon landings.

I pay scarce attention to this guy's little fantasy, frankly, and walk away from people who start spouting this kind of stuff to me in real life. No "bye," no "you're a conspiracy theorist"; best to just walk away from the nut. Trust me. :P

Re:Lets see

[prisoner-of-enigma]

Correction: you can put a lot of textual data in 8KB. If you start putting other things like fingerprint scans, voiceprints, DNA profiles, mug shots, and other things that aren't character-based, 8KB immediately becomes ridiculously tiny.

Re:Google?

[micromoog]

Putting the right indexes on a table with 100 million records to satisfy a very small set of well-documented expected queries is not exactly rocket science. And I think you underestimate what Google does.

Re:Google?

[diablobynight]

I think your very wrong. Everything that goes into the government databases is designed by them, tables and queries, all designed by them, all the data is in the same form, and language. Now take google. searching sites made by everyone, with only meta tags as guides, which sometimes aren't there, and there is a constant influx of new data. Plus I would be willing to bet, with no doubt whatsoever, the internet is bigger than the government database.

This is probably just a bunch of govs sitting around going, well, we got this new budget, how are we going to spend it. And one guy said, I bet it would be really cool to have a 2.5 TB RAM

Re:SMASH THE GIANT COMPUTER!!!

[Nynaeve]

That is why the "good" reason of "tracking terrorists" had to be used. Like you need a 100+ TB database to track terrorists! The real reason would cause too much of a fuss.
The sheer number of naive and/or apathetic citizens that can not or will not ponder the ramifications of the construction of such a large people-tracking infrastructure -- regardless of its purpose -- is depressing.

One Word:

[Anonymous Coward]

Echelon [echelonwatch.org]

Has happened in the past

[bobobobo]

Look at Chile for a past example. Venezuela today too, although hard to say for certain.

Re:A Single Disk Hit Kills Responsiveness

[Bamafan77]

That is why Google has multiple copies of the entire web in memory.

Interesting link, though I wonder if everything it claims is true. Specifically I'm referring to the business about every page Google has indexed necessarily being in memory simultaneously. Possible, but I'd have to hear it from a Google programmer familiar with the area to start to believe it.

And even if it were true, the statement that Google has "multiple copies of the entire web in memory" is certainly false because there are still many webpages that are not indexed by Google.

Re:A Single Disk Hit Kills Responsiveness

[Anonymous Coward]

Thats a freaking BLOG!

How can you possibly believe that? True disk accesses are slow, but it doesnt mean that they have to be completely avoided to still get good response time.

Caching everything in RAM definitely helps, but is not feasable. There are many other concepts out there that Google must use as well (dedicated RAID disk subsystems, metafiles, heuristic-based searches, etc...)

Re:Lets see

[afidel]

Fingerprints can be (and are) encoded as a couple of vectors, voiceprints are similar. DNA profiles only need to record the magnitude of around a dozen markers to accurate to the person, mugshots are the only thing on your list which would require a lot of storage, and you don't need those for the searching part of the DB, just have them stored in a blob with a key from the main record pointing to it. Don't bring up facial recognition, it doesn't work, false positives are through the roof and false negatives are too high to trust.