Forgot your password?
typodupeerror
Security The Internet News

Web App Scanners Miss Half of Vulnerabilities 68

Posted by kdawson
from the hey-we-caught-half-too dept.
seek3r sends news of a recent test of six web application security scanning products, in which the scanners missed an average of 49% of the vulnerabilities known to be on the test sites. Here is a PDF of the report. The irony is that the test pitted each scanner against the public test files of all the scanners. This reader adds, "Is it any wonder that being PCI compliant is meaningless from a security point of view? You can perform a Web app scan, check the box on your PCI audit, and still have the security posture of Swiss cheese on your Web app!" "NTOSpider found over twice as many vulnerabilities as the average competitor having a 94% accuracy rating, with Hailstorm having the second best rating of 62%, but only after extensive training by an expert. Appscan had the second best 'Point and Shoot' rating of 55% and the rest averaged 39%."
This discussion has been archived. No new comments can be posted.

Web App Scanners Miss Half of Vulnerabilities

Comments Filter:
  • by ls671 (1122017) * on Saturday February 06, 2010 @04:57PM (#31047860) Homepage

    > Web App Scanners Miss Half of Vulnerabilities

    Well this is no surprise to me. Designing/testing secure systems is much more than scanning for vulnerabilities.

    Scanning is only one of the tool to use to accomplish the goal.

    • by Anonymous Coward

      The web was clearly never designed to do even a fraction of what it is expected to do today. Now, neither were computers. But at least when it comes to hardware, we're willing to throw everything away and start from scratch. We don't seem able to do that with the web.

      Basically everything about the web today is just one dirty hack upon another bunch of dirty hacks. SSL and TLS are a good example. JavaScript is another. Everything built on top of JavaScript, such as AJAX, is a huge hack. So it's no wonder tha

      • by ls671 (1122017) *

        Slow down man ! ;-) One step at the time, I suggest that you start with this:

        http://news.slashdot.org/comments.pl?sid=1540138&cid=31047988 [slashdot.org]

      • Re: (Score:3, Informative)

        Basically everything about the web today is just one dirty hack upon another bunch of dirty hacks. SSL and TLS are a good example. JavaScript is another. Everything built on top of JavaScript, such as AJAX, is a huge hack. So it's no wonder that it's so damn easy to write insecure web apps.

        <Sarcasm>
        Basically everything about the Internet today is just one dirty hack upon another bunch of dirty hacks. Ethernet and IP are good examples. TCP is another. Everything that does not limit itself inside a single OSI layer, such as PPPoE, all kinds of VPN and NAT, are huge hacks. So it's no wonder it's so damn easy to exploit remote machines over the Internet.

        ...

        We need to throw it all away. Everyone routinely do this with their plastic bags. We now need to extend that practice to all Intern

      • by RzUpAnmsCwrds (262647) on Sunday February 07, 2010 @01:17AM (#31050552)

        But at least when it comes to hardware, we're willing to throw everything away and start from scratch.

        Is that why I'm using a pipelined, out-of-order implementation of a 64-bit extension to a 32-bit extension of a 16-bit ISA?

        I mean, shit, my Core 2 Duo supports everything from 128-bit vector instructions to segmented addressing. I have USB and PCI Express busses on my ThinkPad, but also CardBus/PCMCIA and a modem. I have Gigabit Ethernet but it is still compatible with 10Base-T. I have a DVI port (through the dock) but also a VGA port. My DVD-RW will read CDs which are 30 years old.

      • There has been some approaches to deal with web programming, redone. I'm thinking about OPA at http://mlstate.com/ [mlstate.com] in particular.
  • Take buffer overflows for example. You can build a generic tool to create buffer overflows by feeding in long messages but there is no generic way to exploit the overflow, because every system arranges its data differently.

    BTW there is a typo in the summary pitted eah scanner

    • Re: (Score:3, Funny)

      by truthsearch (249536)

      When I first quickly ran through the summary I read that as "I pity the scanner". After re-reading the summary it seems appropriate.

    • by gandhi_2 (1108023)

      So we should probably be thankful that "web app scanners catch half of vulnerabilities".

    • by ls671 (1122017) *

      > buffer overflows for example

      What do you mean ? The platform we use checks for array/buffer bounds on any access to them. We also use a persistence tool that is pretty good at preventing SQL injections.

      It sure beats scanning on the efficiency level ;-))

      • SQL injection is only one half of web security, though. The other big threat Cross Site Scripting/Request Forgery.

        Output filtering is a lot harder to keep track of than database queries (even with a good templating system) since data must be filtered in different ways: Some stuff may contain any markup including document-level elements, script/stylesheet inclusion, other data may only contain local formatting markup (emphasis, etc.) and still other stuff has to be escaped entirely...

  • by nuckfuts (690967) on Saturday February 06, 2010 @05:23PM (#31047996)

    No vulnerability scanner will ever detect 100% of the vulnerabilities possible. They're still very useful, however, because no website is going to have 100% of all the vulnerabilities possible.

    Think of it another way. If your website has only 1 vulnerability and the scanner detects it, then it's 100% effective.

    If your website has only 1 vulnerability and no scanner detects, score 1 for the bad guys. The cat and mouse game continues.

    • by Dumnezeu (1673634)

      If your website has only 1 vulnerability and no scanner detects, then it's 100% ineffective. The cat and mouse game continues.

      FTFY

      • OK. I'll bite.

        Let's say 50 years later RSA is found to be completely insecure. So mathematically anything based on RSA, depending on RSA or containing RSA has at least 1 vulnerability. Yet today it's impossible for any security scanner to tell whether somebody's SSL has that particular vulnerability or not. But as it stands, as long as that vulnerability is not known by anyone, nobody can exploit it. So you can't say the security scanner is n% ineffective because it doesn't detect a vulnerability.

        Beside
    • Re: (Score:3, Insightful)

      by ls671 (1122017) *

      > If your website has only 1 vulnerability and no scanner detects, score 1 for the bad guys.

      except that the "bad guys" mostly use scanners to discover holes ;-))

      So interestingly enough, holes detectable with scanners are more exploited.

    • by JWSmythe (446288) <jwsmythe@jwsmyt h e . com> on Saturday February 06, 2010 @05:52PM (#31048166) Homepage Journal

          From what I recall doing this for sites that handled credit card processing (me being in the tested side), those tests are pretty much worthless.

          If you had 1 vulnerability, you'd get pages of false positives or irrelevant information. I recall a particular 10 page report we got back that we were advised to fix or we'd fail on. The only item to fix was the version of the web server was just one behind current. The changelog indicated that it was to fix a vulnerability on a different platform, so it was completely unrelated to us. We'd frequently have points marked off because we couldn't be pinged or portscanned. I'd have to open the firewall up to them, just to be scanned. Our security would identify an attempted port scan as a hostile action, and react by dropping all traffic from them. Sorry my security stopped your scanning, but that's the intention of it. {sigh}

          After opening the firewall to them, and changing the version number on the web server (there were reasons we couldn't do the trivial upgrade), we passed with flying colors.

          For them, they were interested in the version numbers handed off by the server, not what they actually were. For example, if it was Apache, we could have it report Apache version 9.9.9, and that would have made us pass on that part without fail for years.

      • by merreborn (853723)

        The only item to fix was the version of the web server was just one behind current. The changelog indicated that it was to fix a vulnerability on a different platform, so it was completely unrelated to us...

        After opening the firewall to them, and changing the version number on the web server (there were reasons we couldn't do the trivial upgrade), we passed with flying colors.

        For them, they were interested in the version numbers handed off by the server, not what they actually w

        • by JWSmythe (446288)

          You're very correct on that.

          For a while, I was doing that with a few things, including Apache and the Linux kernel. There were pieces I needed that didn't progress, so I handled my own backporting of various things. That was a long time ago, and those problems were resolved in more current versions, so it hasn't been necessary for years.

          But, if you're using say mod_ssl to handle your SSL on Apache, and you're still in the 1.3.x tree, you'd now be scored down. A

    • by ircmaxell (1117387) on Saturday February 06, 2010 @06:14PM (#31048302) Homepage
      To tell you the truth, the percentage of actual vulnerabilities that it finds mean nothing to me. What matters to me is the rate of false positives. Even better would be the number of actual vulnerabilities found divided by the number of false issues found.

      I had a chance to see the outputs of a few of these scanners run against a particular open source content management system. Not one of them found an actual, confirmable vulnerability. But one found over 9,000 false positives. All found a fair number of false positives. Even if could find real vulnerabilities, digging though all those false positives to find a real one is a really daunting task.

      What I find works better than these scanners is hand audits by someone who knows what they are looking for. It's most definitely an intensive task, but let me ask you. What's more a better use of time, an expert doing a hand audit who may find vulnerabilities that the scanner didn't), or the expert digging through all 9000 of those "results" trying to figure which, if any, are real? I assert that the best use is going to be a combination of the two. Just don't put your faith in either one...
      • by nuckfuts (690967)

        Any scanner that returned over 9,000 false positives is clearly a joke. So you ignore it and run a better scanner.

        I've dealt with scans for credit card purposes and for the most part I thought they were pretty good. Yes, there was the occasional false positive. So you provide some explanatory information and then you get passed. False positives are preferable to false negatives.

        It's simply unrealistic to expect these scans to yield perfect results. Despite that, I believe they're useful.

  • I noticed Whitehat Security Declined to participate. I wonder why that is? We just purchased there service, I like there concept, especially as they sold it, we haven't gotten into full use of the product yet, but I can tell you some of the execution of there service could be improved. There seems to be a little bit of a disconnect between the sales force and the operations team. I would have been very interested to see how they fare in a test like this.
  • "Is it any wonder that being PCI compliant is meaningless from a security point of view?"

    Where's that quote from? I can't find it on either the page or in the PDF...

    • Re: (Score:3, Informative)

      by julesh (229690)

      Where's that quote from? I can't find it on either the page or in the PDF...

      It's the submitter's opinion. And it's quite accurate: no such standardized set of requirements can guarantee security, because security is much more complicated than the simple kinds of rules that you can include in them. PCI compliance gives the illusion of security where it may well not actually exist at all.

      • And it's quite accurate: nothing can guarantee security.

        FTFY. There is no perfect security. I don't know anyone that says PCI compliance guarantees you are secure. But it is an indication of the controls you have in place protecting cardholder data.

        For instance, hiring a licensed, bonded plumber doesn't guarantee they won't screw something up. But your chances of a good outcome are a lot better.

    • by maxume (22995)

      Try a little harder, the attribution is just before it (apparently that is the submitters opinion).

  • PCI Still Important (Score:4, Interesting)

    by savanik (1090193) on Saturday February 06, 2010 @05:34PM (#31048060)

    The key message here is that simply testing your web site with a vulnerability scanner doesn't make it secure. Well, duh.

    PCI is still important because before the guidelines, most people weren't scanning their web sites at all. Even when they knew how - they couldn't convince management it was worth the trouble, time, dollars, and so on. And without scans, the number of discovered web vulnerabilities approaches 0%.

    PCI isn't just about scanning your website, either. There's hundreds of things you have to do to secure everything from the physical layer up to the application layer. And having PCI be required to process credit cards makes everything much more secure. I'm talking about small businesses so cheap they don't want to put LOCKS on the doors between the outside world and the servers holding your plain-text, unencrypted credit card numbers, and who don't have the expertise to set up a web camera on their own building.

    You might not like PCI, it might be inconvenient, but it's necessary to protect the general public.

    Disclaimer: I am an information security professional.

    • by trentblase (717954) on Saturday February 06, 2010 @06:06PM (#31048236)

      You might not like PCI

      The only thing I don't like about PCI is the acronym they chose.

    • Re: (Score:1, Interesting)

      by Anonymous Coward

      Scanning vendors are quite good at discovering non-issues such as the availability of "weak" SSL ciphers and known problems in technology stacks. They are unfortunately useless when it comes to discovery of application level security problems.

      Its all just a big scam where people pay lots of money to companies who mis-represent actual PCI compliance requirements, hit their servers with Nessus and print out a security audit pass certificate the company can hang on their wall. Its about as useful and seedy a

  • A vendor will sell you, or often give you a free trial of, their vulnerability scanning tool. They will then turn right around and sell you a tool that is supposed to fix those problems. Does anyone else see a problem with that? One reason I prefer the FOSS tools going back to Nmap and SATAN is that they do what real intruders try to do, not what some marketing department wants them to do as a way to scare you into buying stuff.

  • At least when it comes to security. By the time any standard is published and a test suit is assembled, the whole threat scenario has changed by 180 degrees. We're dealing here with an industry that has a half-life period of its knowledge of about 3 months. Not the usual 2-3 years anywhere else in IT.

    Don't be compliant. Either get up to speed with curent security problems or hire someone who does. Standards are worth jack, at least from a security point of view (they're still quite valuable to get contracts

    • by bbn (172659)

      Don't be compliant.

      How stupid is this? PCI is a set of minimum requirements. It is all stuff that any competent admin would have done even without the standard. If you are as cool, as you apparently think you are, you will be compliant with zero work.

      • Ok, maybe I should have written "don't try to be compliant, try to be secure". I thought I'd get a reply like that one right the moment I hit submit...

        Yeah, yeah, I'll use preview more now, promised.

        What I wanted to express is that managers usually don't care for security. The ONLY reason my last company finally implemented bare minimum security was that they needed a certificate to get a fat contract. There was literally no other motivation for them. And, again, they only did the bare minimum of what was e

  • This guy is trying to hype his own findings a bit too much. Removing half of the vulnerabilities is actually really good! If you happen to remove the vulnerability that some mass-defacement takes advantage of, you really did ad a lot of value by using the imperfect scanning tool.

    One of the most common and least helpful fallacies about security is that something is either secure or it is not. Nothing is 100% secure. Removing half of the vulnerabilities is a huge improvement over removing none.

  • Scanners exist because people want scanners, and so people can sell a product labelled "security scanner". And get a feel-good (false) sense that everything is secure when the scanner reports no issues.

    This idea started with the general idea of vulnerability scanner, tools designed to scan hosts for open ports, check software versions, and try exploits against known issues.

    The problem with all of them is they can only detect anticipated vulnerabilities.

    Unknown vulnerabilities are not properly detect

    • by perlchild (582235)

      A quicker fix would be to get the vendors to share their corpus of vulnerabilities, probably through the Owasp group. That way at least, they'd all get a complete picture of vulnerabilities to scan for, and they might improve. Right now, it seems the resources applied to getting the vulnerabilities in are so minute, compared to the amounts of effort performed by two groups:
      a) hackers trying to find vulnerabilities that had been unknown before
      b) the people paying for their service's expectations(I used to

  • by noidentity (188756) on Saturday February 06, 2010 @06:06PM (#31048234)
    If these scanners report only half the vulnerabilities, they just need to double the reported number. Simple fix, really.
  • I read TFA because summary does not make sense only to find out that TFA does not make sense.

  • by Hero Zzyzzx (525153) <dan@g[ ]uprising.com ['eek' in gap]> on Saturday February 06, 2010 @07:20PM (#31048722) Homepage

    My favorite from a past employer - one of these PCI scanning companies asked us to take down our iptables rules for a set time period while they scanned us. That's right, they wanted us to be less secure while they checked how secure we were.

    We were eventually able to get an ip range from them, but not until we fought them a bit. They *would not* do the scan unless we took down our firewall. I wanted to just REJECT everything but 80 and 443 and not tell them, but the higher-ups told me to play along.

    Anyway - the whole idea felt really ... wrong. And they didn't point out anything useful, either.

    • by DavidTC (10147)

      Well, I can sorta see their point in saying 'You have to give us the permissions of the least restrictive IP you have'.

      But it's actually still dumb. The only IPs in my firewalls (Besides the mail server which has temp blocks for spammers) are the IPs of my other servers, so I can restrict specific things to them.

      The only two that I can think of are access to the mysql port (So other servers can use a database), and access to a special mail submission port without any other security on it. (So my web serve

    • > one of these PCI scanning companies asked us to take down our iptables rules for a set time period while they scanned us

      Can you gave examples of companies that scan companies in the manner you describe. My understanding is that to achieve PCI compliance, you fill in a bunch of forms. I mean Heartland Payment Systems were PCI compliance, and look what happened to them.

  • Don't forget these results supposed to be 100% because their own test application has been scanned. It means an actual results will be much lower against a real application.
  • > being PCI compliant is meaningless from a security point of view? You can perform a Web app scan, check the box on your PCI audit, and still have the security posture of Swiss cheese on your Web app!"

    Print this out and stick it on the wall, for the next time your PHB starts waffling on about compliance .. :)

The clearest way into the Universe is through a forest wilderness. -- John Muir

Working...