Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
×
Government Databases Programming Software News Your Rights Online IT

Terror Watchlist "Crippled By Technical Flaws" 324

I Don't Believe in Imaginary Property writes "The database used by the government to generate lists like the No-Fly List is 'crippled by technical flaws,' according to the chairman of a House technology oversight subcommittee. And the upgrade may be worse than the original. Rep. Brad Miller (D-NC) says that 'if actually deployed, [the upgrade] will leave our country more vulnerable than the existing yet flawed system in operation today.' It seems that the current database doesn't have any easy way to do plain-text matching, forcing users to enter SQL queries. That might not sound so bad until you learn that the database contains 463 poorly indexed tables. How long until there's a terrorist named Robert'); DROP DATABASE; —?"
This discussion has been archived. No new comments can be posted.

Terror Watchlist "Crippled By Technical Flaws"

Comments Filter:
  • by ericspinder ( 146776 ) on Tuesday August 26, 2008 @08:12AM (#24749783) Journal
    That's what happens when your interview questions are a political loyalty test.
    • by QX-Mat ( 460729 )

      I understand that's what happened in the justice department - is that true of other departments?

      • by gEvil (beta) ( 945888 ) on Tuesday August 26, 2008 @08:51AM (#24750015)
        I understand that's what happened in the justice department - is that true of other departments?

        Considering the modus operandi of this administration, I'd be very surprised if this weren't a widespread practice.
        • by Anonymous Coward on Tuesday August 26, 2008 @08:57AM (#24750055)
          I heard they douse you in Holy Water. If it tastes like burning, you're hired on the spot.
      • Re: (Score:2, Insightful)

        by wisty ( 1335733 )
        It happens in any place where IT is an essential part, and an optional extra. If IT is essential you need meetings and accountability, and no feature gets a cost-benefit analysis by anyone with a clue because all of them are "essential".
      • by smilindog2000 ( 907665 ) <bill@billrocks.org> on Tuesday August 26, 2008 @09:09AM (#24750157) Homepage

        Err... yes. Just FEMA, the CIA, and nearly every other major department. Bush's loyalty test brought us the Katrina aftermath fiasco, and mass resignations at the CIA. He even tried to appoint his personal lawyer to the Supreme Court. As they say, "sh-t flows down-hill." When the man in charge is a complete moron, the entire government suffers.

        Sorry, you were probably making a joke. A lot of us on this forum don't get sarcasm as easily as we should.

        • by QX-Mat ( 460729 ) on Tuesday August 26, 2008 @10:53AM (#24751385)

          I was intrigued! You gave me the info I wanted to google with - Mass CIA resignations lead me to this [washingtonpost.com]

          I had no idea how bad it was. Retrospectively, the bashing the CIA got seems stupid considering the impossibility of what they have to accomplish... not just now, but after pissing off most of the world in the last 8 years.

        • Re: (Score:3, Insightful)

          by bickerdyke ( 670000 )

          As they say, "sh-t flows down-hill."

          Nah. it much too often floats at the top.

    • by fastest fascist ( 1086001 ) on Tuesday August 26, 2008 @09:08AM (#24750145)
      Don't knock it. This is proof a poor process CAN lead to good results. Those responsible for this should be generously rewarded.
      • by SpicyLemon ( 803639 ) on Tuesday August 26, 2008 @10:53AM (#24751389)

        Yeah! I fail to see the problem here. So, due to design flaws the terrorist watch list is difficult to do searches on. Maybe they can just get the California Cobol [slashdot.org] programmers to fix it.

        I fail to see how the terrorist watch list is ANY different from the communist black list of the 60s. All it takes to get put on there is a neighbor that doesn't like you. In order to get taken off, an agent has to be assigned to your case and you have to be investigated so that they are sure you're not a terrorist. With the current size of the list, good luck with that.

        CNN has had several articles in the last few weeks dealing with the terrorist watch list. My favorite was about three people named "James Robinson" [cnn.com]. The article mentions that one of the Jameses would just get tickets using the first name "Jim" and he wouldn't be hassled. Another would just run his first and middle name together and it wouldn't get flagged. Of note from that article, "The TSA has said the problem lies with the airlines and threatened to fine airlines that tell passengers they are on the watch list." Yeah. Wow. They're trying to make it illegal to tell someone why they're being held and discomforted. If you don't want the information to get out, don't share it. Keep it to yourself.

        Article V says, "[you can't be] deprived of life, liberty, or property, without due process of law..." This list deprives liberty (and sometimes property) and is missing a key element.

        Article XI says, "The enumeration in the Constitution, of certain rights, shall not be construed to deny or disparage others by the people." Isn't that EXACTLY what's happening?

  • Oh yes. Little Bobby Datas, we call him.

    xkcd. [xkcd.com] Always relevant.
  • by daveime ( 1253762 ) on Tuesday August 26, 2008 @08:14AM (#24749813)

    The amount of people they want to include on their "t3rr0rz l1zt" it'll only be a matter of time before we have

    Osama Bin CREATE INDEX;

    and

    Saddam OPTIMIZE TABLE;

    Then everything will be hunk dory again.

  • Number of tables (Score:5, Interesting)

    by suso ( 153703 ) * on Tuesday August 26, 2008 @08:15AM (#24749821) Journal

    That might not sound so bad until you learn that the database contains 463 poorly indexed tables.

    This is not a good measure of how good or bad a database is. Its good to have a table for every type of data and every data type. Read about normalization. You can go overboard, but as long as your database is designed well, having 463 tables might be just fine.

    I say this because once I heard consultant say something like "This web application shouldn't need more than 40 tables, when in fact they didn't know much about the details of the web app, which were quite sophisticated and the real application had more than 100 tables."

    • by ericspinder ( 146776 ) on Tuesday August 26, 2008 @08:34AM (#24749959) Journal
      The problem is not the number of tables, but the fact that they are apparently 'poorly indexed'. Table indexes are important, both for the speed of queries, and data integrity.
      • Data integrity? (Score:3, Interesting)

        by SuperKendall ( 25149 )

        Please do explain how data INTEGRETY is affected by the way you define indexes, as opposed to the ways in which you have denormalized tables for performance.

        From the article, it would be good to see an explanation of just what they mean by "poorly indexed". That seems much more likely to refer to the need for more indexes for faster search results, rather than indexes done badly...

    • Re: (Score:3, Informative)

      by jedidiah ( 1196 )

      > This is not a good measure of how good or bad a database is.

      Oh yes it is.

      In order for a database to be USEFUL, you need to query it. If you can't
      query the database because of the way that it's laid out or indexed then
      it is infact broken.

      400+ tables for an identity resolution database is absurd. There are MUCH
      better ways to account for all sorts of bizzarre types of identifying
      information.

      This database probably isn't properly normalized either. With 400+ tables
      and a simple problem you would think that a

  • by curmudgeon99 ( 1040054 ) on Tuesday August 26, 2008 @08:17AM (#24749829)
    The same US government that screws everything else up should be expected to screw up the terror DB. It was probably written by a junior developer who had never heard of a SQL injection. Isn't making a search form about the easiest project there is to build? I hate to say it, but I'm glad our government is so full of screw ups: pity the list exists at all...
    • by polar red ( 215081 ) on Tuesday August 26, 2008 @08:22AM (#24749867)

      One could wonder whether the project was set up to adress terrorism OR it was setup to generate media-attention ?

      • Both (Score:5, Interesting)

        by BitterOldGUy ( 1330491 ) on Tuesday August 26, 2008 @09:21AM (#24750291)

        One could wonder whether the project was set up to adress terrorism OR it was setup to generate media-attention ?

        It was both and then some.

        I'm trying to find the link of the guy who started this BS. It was a private citizen who, IIRC, was the one who was involved with Choicepoint. He wrote some code and his algorithm pulled up most of the 9/11 hijackers and then some. He had some false positives even then, but it was the Government's wet dream and it solved some of their problems (such as that pesky little Constitutional problem of spying on Americans. It's OK if a private company does it -Choicepoint.) and it makes great security theater and it creates some big fat Governemtn contracts for some big fat cats with Government connections.

        Need more caffeine and I'm getting tons of false hits from Google trying to find the cite - it is over 7 years old, ya know.

      • by MindStalker ( 22827 ) <mindstalker@@@gmail...com> on Tuesday August 26, 2008 @09:31AM (#24750433) Journal

        C) Keep the terror level level artificially high.
        http://www.dhs.gov/xinfoshare/programs/Copy_of_press_release_0046.shtm [dhs.gov]

        The United States government's national threat level is Elevated, or Yellow.

        The U.S. threat level is High, or Orange, for all domestic and international flights.

        So for the rest of you its only Yellow, but if your flying, its Orange!

        • Re: (Score:3, Funny)

          by Jay L ( 74152 ) *

          So for the rest of you its only Yellow, but if your flying, its Orange!

          Well, duh. The sky is blue, and when you're flying, you're closer to the sky. yellow + blue = orange.

          [For you armchair optical physicists: I'm talking about color mixing at high altitude. Your classroom yellow + blue = brown doesn't work up there, because of sunspots.]

      • by DesScorp ( 410532 ) on Tuesday August 26, 2008 @11:02AM (#24751481) Journal

        One could wonder whether the project was set up to adress terrorism OR it was setup to generate media-attention ?

        I work at an airport, in administration, and trust me when I say this has very little to do with dark political conspiracies, and a lot to do with the government's haste to show they were "doing something" after 9/11. This project was quickly rushed into service, and has been widely reviled by airports and airport police departments across the country. And other similar measures... the current background check process for giving access to secured areas, and the very creation of TSA itself, were all measures to reassure the public that something was getting done. The problem is that government enterprises like these tend to become bipartisan boondoggles, with every state and major city wanting a piece of the political and funding action these things entail. Federal agencies tend to become monsters that need to justify their own existence by constant growth. TSA in particular is quickly becoming a large federal law enforcement agency, not just a baggage security team. When they were first set up, several of their nascent teams moved and basically tried to take control of several airports... I know of one major southern airport where they simply showed up one day, declared that a series of offices now belonged to them, and when the airport director came down to see what was going on, they tried to have him arrested by his own police force for "violating federal facilities". Anyone that works with AAAE members (airport execs group) knows what incident I'm talking about.

        Did you know that TSA will now be issued police-like blue uniforms, with metal badges, just like cops? Airport police and the metropolitan police departments that supplement them just looooove that, and there's the inevitable talk of actually giving said TSA agents firearms. Unlike some other police departments, TSA agents are being encouraged to wear their uniforms and badges in their spare time, in order to enhance the agency's "visibility" to the public. There are already jokes that TSA SWAT teams are inevitable at airports. The problem is, the laughter doesn't last very long when we realize that the way things are going, that might not be a joke so much as a prediction of the future.

        • Re: (Score:3, Insightful)

          by Chris Burke ( 6130 )

          I work at an airport, in administration, and trust me when I say this has very little to do with dark political conspiracies, and a lot to do with the government's haste to show they were "doing something" after 9/11.

          I didn't need an airport administrator to tell me that this is all just Security Theater, but thanks for the confirmation just the same.

          Did you know that TSA will now be issued police-like blue uniforms, with metal badges, just like cops? Airport police and the metropolitan police departments t

    • by Dan667 ( 564390 ) on Tuesday August 26, 2008 @08:35AM (#24749969)
      Only problem is that it actually affects people try to travel. If the US Gov want to be idiots, fine. But if they want to do it in my name like I somehow want this, there is a problem. If they want to treat me like a criminal in my own country for trying to travel in it, I have a problem. If they want to seize my laptop for no reason because I am trying to travel, I have a problem.

      I like the idea of having a fly at your own risk airline where you can just "risk it" and not have all these so called "protections". I bet it would put the airlines with the TSA out of business in a week.
      • Re: (Score:2, Insightful)

        by mrbluze ( 1034940 )

        I like the idea of having a fly at your own risk airline where you can just "risk it" and not have all these so called "protections". I bet it would put the airlines with the TSA out of business in a week.

        Thinking of joining the air force?

      • by HungryHobo ( 1314109 ) on Tuesday August 26, 2008 @08:58AM (#24750067)

        That could work.
        Risk it airlines, where there are no security checks to get on board and the only security measures are to detect when a plane has been hijacked and once confirmed a killswitch is activated to simply blow it out of the sky. Might have to pay the pilots more but I'd travel on one of those.

    • by Tridus ( 79566 ) on Tuesday August 26, 2008 @08:58AM (#24750061) Homepage

      It was outsourced. Near the bottom of TFA it says that some of the money was used to renovate a building owned by Boeing.

      Its amazing just how many "government screwups" are actually caused by politicians outsourcing to their buddies in private industry (with little to no penalties for failing to deliver what was promised), and have nothing to do with the abilities of actual government employees.

      There's actually quite a few smart IT folks in government, but they're not the ones who make decisions on who to outsource this stuff to. In fact, most of them would probably rather build a team and do it In-House, since that way you build up the knowledge internally and can more easily support it later.

      So please don't blame government employees for something that Boeing screwed up.

    • by elrous0 ( 869638 ) *
      No-bid contract FTW!
    • Re: (Score:3, Informative)

      by suso ( 153703 ) *

      Believe it or not but there are some good applications out there in the government. I worked on a Naval base for a year as a contractor and was fortunate enough to work on a really kick ass PHP application. I can't tell you what it was, but to this day it was the best web application I've seen as far as security, design, functionality and sophistication goes. I think it was over 130,000 lines of code and was written by 2 guys over 3 years. I learned a lot from working on that application. So there are

  • Comment removed (Score:3, Interesting)

    by account_deleted ( 4530225 ) on Tuesday August 26, 2008 @08:17AM (#24749835)
    Comment removed based on user account deletion
    • Re: (Score:3, Interesting)

      by Tridus ( 79566 )

      "Miller also alleged that some of the $500 million spent on Railhead already had been improperly used to renovate a facility owned by contractor Boeing."

      Its easy to waste a lot of money when a department that has a virtually unlimited budget outsources with little to no oversight.

      We had similar problems in Canada with the Long Gun Registry, which was a dumb idea to begin with. Then they outsourced it. All told it cost more then $1 billion to set up, and didn't work properly at first. (It does work now, thou

  • by account_deleted ( 4530225 ) on Tuesday August 26, 2008 @08:18AM (#24749837)
    Comment removed based on user account deletion
  • by Ihlosi ( 895663 ) on Tuesday August 26, 2008 @08:19AM (#24749849)

    It's crippled by being a moronic concept in the first place ("You've got the wrong name and _maybe_ the wrong date of birth, and you're not flying.") and an absolutely arbitrary process of putting names on the list, and no way of ever getting a name off the list.

    Fix those points first, and _then_ worry about technical details.

    • by Hektor_Troy ( 262592 ) on Tuesday August 26, 2008 @08:49AM (#24750007)

      "You've got the wrong name and _maybe_ the wrong date of birth, and you're not flying."

      Oh, come on! We all know to be terrified of letting 5-year-olds onto the plane [king5.com] (video). If they share a name, they're bound to share ideologies. And what better place to hide explosives than in a shitty diaper?

      And that kid was only wanted by the INS! I can just imagine the hillarity ensuing when they clear an airport because another kid "made a stink bomb" in his diaper - we all know how much the TSA loves words like those.

    • by hellwig ( 1325869 ) on Tuesday August 26, 2008 @08:57AM (#24750053)
      Exactly. The No Fly List [wikipedia.org] is useless because it contains an estimated 1,000,000+ names (really, 1 million terrorists we can't track down?). It's useless because it contains generic entries such as T. Kennedy, which doesn't refer to a person but an alias once used in a crime (Tater Salad might be in there too). It's useless because even once they bomb a terrorist into tiny pieces his name is still on the list (sry, can't rememer who). Not only that, but the list is used for political dissidents too, not just terrorists or dangerous criminals. Apparently Nelson Mandela was on the list, until the fact was embarrasingly publicized and he was finally removed.
      • by Anonymous Coward on Tuesday August 26, 2008 @09:23AM (#24750303)

        Technically, the Terrorist Watch List Database contains about 400,000 unique persons, of which the remainder represents known aliases. This is the so-called "green light" list, with no restrictions on them whatsoever. The "yellow light" list is much smaller, about 10,000 unique persons, and only subjects these people to desk check-ins and special searches. The *actual* No Fly list (the "red light" list) is itself a small fraction of that, perhaps 1,000 people at the most.

        Add that to the fact that Congress is starting to mandate some sanity checks and ways to be removed from the list, I could see this someday being useful... just not today.

      • by clickety6 ( 141178 ) on Tuesday August 26, 2008 @09:31AM (#24750417)

        Apparently Nelson Mandela was on the list, until the fact was embarrasingly publicized and he was finally removed.

        So, easy solution - if you don't want to be bothered by the no fly list then change your name to Nelson Mandela...

      • Re: (Score:3, Insightful)

        I think we should start putting everyone on the watch list. Then we don't have to worry about exceptions, and all of our wait times become the same.

        If employees of the TSA can't sort out when someone, like a 6 year old boy, is not the right person named on the terror list, then WHAT THE FUCK ARE THEY GOING TO DO WITH A REAL TERRORIST?
      • Re: (Score:3, Interesting)

        by Chris Burke ( 6130 )

        Not only that, but the list is used for political dissidents too, not just terrorists or dangerous criminals.

        ZOMG NO! The promised that they'd only use DHS/TSA authority to combat terrorism! They pinky swore!

        If you can't trust a government pinky-swear, what can you trust? Man, I'm so disillusioned.

    • by samweber ( 71605 ) on Tuesday August 26, 2008 @09:12AM (#24750187)

      But hey, it's not that bad! After all, since all terrorists use their real names when flying, it is sure to catch them all.

      Ever wonder why no suicide bomber has been able to strike twice? It's because of the no-fly-list, I tell you!

      • by Anonymous Brave Guy ( 457657 ) on Tuesday August 26, 2008 @11:30AM (#24751811)

        After all, since all terrorists use their real names when flying, it is sure to catch them all.

        The irony of your post is that most of the perpetrators of recent terrorist attacks in the West had valid ID and were, in many cases, citizens of the country they attacked. Even with all the intrusive surveillance, vast databases and draconian security measures, they still got through, just by keeping a low profile until they were ready to attack. Which tells you exactly how much measures like the list we're talking about are actually worth in practice...

  • Size Comparison (Score:5, Informative)

    by Wellington Grey ( 942717 ) on Tuesday August 26, 2008 @09:01AM (#24750095) Homepage Journal
    For those interested: the size of the terror watchlist compared to US cities and States [wellingtongrey.net].
  • Foolish, poorly designed, wasteful, corrupt Federal Database application fails to deliver on promises,

    Film at eleven !
  • *gasp!* (Score:2, Informative)

    by Luke_22 ( 1296823 )
    how are we going to recognise all those terrorists [wellingtongrey.net] now?
    it may have been a good idea, but the implementation was horrible, come on....
  • "Technical Flaws" (Score:3, Insightful)

    by T.E.D. ( 34228 ) on Tuesday August 26, 2008 @09:28AM (#24750375)

    I'm not sure you can call having names on the list matching 1/3 of the population of the earth a "technical flaw".

    What they really need to do to make it useful is get it down to perhaps a couple thousand real concerns.

  • Ethics vs. results? (Score:3, Interesting)

    by Shoten ( 260439 ) on Tuesday August 26, 2008 @09:31AM (#24750425)

    So, the question that comes to mind for me is this: what if I were a database architecture guru who had been asked to build this system (or its replacement)? At first, my thought is that I'd refuse on grounds of my opposition to the whole thing...but now I'm suddenly wondering if some of the better options did just that, and then it got designed and built by the knob who would take the job. Unlikely, sure, but it's something that I've never thought about before. Is the ethical cost of not doing something like this (that's going to get done anyways one way or another) when you're the right guy for the job potentially higher than the ethical cost of doing it?

  • by clickety6 ( 141178 ) on Tuesday August 26, 2008 @09:35AM (#24750493)

    http://video.aol.com/video-detail/snl-funny-terrorist-names/4040669571 [aol.com]

    "M'balz es-Hari"
    "Haid D'Salaami"
    "Mustaf Herod Apyur Poupr"
    "Usuqa M'diq"
    "Hous bin Phartin"
    "I'zheet m'drawrz"

  • The lists. (Score:4, Interesting)

    by Anonymous Coward on Tuesday August 26, 2008 @09:43AM (#24750585)

    A friend of mine is the security manager for a fairly large company. They have offices all over the world and business in many countries. He tells me that there are at least three "terrorist" lists. The EU list, the UN list and the US list. They are listed from poor to really shitty.
    If a person or a company is on either of these lists then they aren't allowed to do business with them as they are suspected terrorists r terrorist backers.
    The US list can contain things like "Muhammad, Saudi Arabia", or "Iqbal, Pakistan".

    The lists are of no use to them and impossible to follow, but they are required to do so or risk sanctions from EU or the US.

    Happy times!

  • 463 tables? (Score:3, Interesting)

    by pluther ( 647209 ) <pluther@@@usa...net> on Tuesday August 26, 2008 @12:46PM (#24752885) Homepage

    Having designed a couple of poorly-designed databases myself, I can understand how this can happen.

    What I don't understand is why the hell there are 463 tables in this thing?

    I mean, what all information do they need in there? Names, maybe a list of known addresses, social security numbers, phone numbers, other identifying information?

    Perhaps a reason why they're on the watchlist at all? List of evidence putting them there? Political activities they've been involved in, letters to congress they've written? Types of books they've checked out of the library?

    Maybe a list of all flights they've taken, and notes on how much trouble they've given to the TSA people when going through the checkpoints?

    OK, that's three tables. What on earth are the other 460 for??

  • Who built it? (Score:3, Interesting)

    by PPH ( 736903 ) on Tuesday August 26, 2008 @01:20PM (#24753331)

    Was this built in-house or by a contractor?

    I ask, because I've been involved with government contracting work, specifically for the FAA. One aspect of the relationship I've repeatedly seen is private business' efforts to cripple the in-house engineering and software expertise of government agencies they do business with. We'd hire their key people away and call the legislators we owned to get funding for in-house projects killed just to drive the work out to us. Once the agency fell on its face a few times, political pressure would grow to quit wasting money and contract it out. To us. For big money.

    Back when I was still in that biz, the Australian government's equivalent of the FAA, CASA, had undertaken a project to build some advanced air traffic control systems in-house. The attitude of our management was rage. "If this had been the United States, we'd have had them shut down."

    If you need work done fast, you need people who can do it on the inside. Even if it goes out for contract, you've got to get the requirements written down correctly.

  • by John Hasler ( 414242 ) on Tuesday August 26, 2008 @02:29PM (#24754325) Homepage

    ...and yet despite it's failure to protect us, we have not been attacked.

    Perhaps, just perhaps, this is evidence that it is not necessary?

"If it ain't broke, don't fix it." - Bert Lantz

Working...