Slashdot is powered by your submissions, so send in your scoop

 


Forgot your password?
Close
typodupeerror
×
Education Security Software IT

Researchers Add Software Bugs To Reduce the Number of Software Bugs (networkworld.com) 73

Posted by msmash from the innovative-techniques dept.
Reader alphadogg writes: Researchers are adding bugs to experimental software code in order to ultimately wind up with programs that have fewer vulnerabilities. The idea is to insert a known quantity of vulnerabilities into code, then see how many of them are discovered by bug-finding tools. By analyzing the reasons bugs escape detection, developers can create more effective bug-finders, according to researchers at New York University in collaboration with others from MIT's Lincoln Laboratory and Northeastern University. They created large-scale automated vulnerability addition (LAVA), which is a low-cost technique that adds the vulnerabilities."The only way to evaluate a bug finder is to control the number of bugs in a program, which is exactly what we do with LAVA," says Brendan Dolan-Gavitt, a computer science and engineering professor at NYU's Tandon School of Engineering.
This discussion has been archived. No new comments can be posted.

Researchers Add Software Bugs To Reduce the Number of Software Bugs More

Researchers Add Software Bugs To Reduce the Number of Software Bugs

Comments Filter:

  • This is NOT a new thing. (Score:5, Interesting)

    by sconeu ( 64226 ) on Friday July 08, 2016 @11:47AM (#52471847) Homepage Journal

    The practice is known as "bug farming", and has been around since at least the mid-80s.

    I learned it when I was in college in '84.

    • Re: (Score:2, Redundant)

      by The-Ixian ( 168184 )

      Actually, is there anything that can be considered "new"?

      Everything is just a rehash of stuff we have done before.

      You might say that technology is new... but is it really? Technology is just a different way of doing the same old things like communicating with each other and building communities.

      All "new" ideas are really just reboots of old ideas.

      • You might say that technology is new... but is it really? Technology is just a different way of doing the same old things like communicating with each other and building communities.

        If you're going to get ridiculously reductive, then nothing new has shown up since the time very shortly after the big bang when the different forces separated and the laws of physics as we know them took hold (assuming those theories are even close to correct). Everything since then has just been different ways of combining the same old subatomic particles.

      • All "new" ideas are really just reboots of old ideas.

        You are just repeating one of Herodotus' tweets.

    • Re: (Score:2)

      by porges ( 58715 )

      I learned as "bug seeding", but yes, and I'm of about the same vintage as you.

    • > The idea is to insert a known quantity of vulnerabilities into code, then see how many of them are discovered by bug-finding tools.

      I heard that called "bebugging" in the 70s. Still seems like a good idea.

  • Knowns
    Known Unknowns
    Unknown Unknowns

    You can't train your test for Unknown Unknowns. At best you can heuristically catch Known Unknowns. And Knowns are already Known, you just need to make sure your tools catch them.

    • I always hated that that quote ignores the unknown knowns. (Things we know, but don't know we know) They are often the most important things - and things we don't write down until after someone asks us "how did you do that?"

      • I always hated that that quote ignores the unknown knowns. (Things we know, but don't know we know)

        It also ignores the known unknowns (Things we think we know, but actually don't). Don Rumsfeld famously used this quote, and he "knew" that Iraq had WMDs. He also "knew" that Iraqis would greet us as liberators, throw flowers at our feet, and would be grateful as we spread peace and prosperity across their land. Those things were not in doubt.

    • Dick Cheney taught us all about the unknown unknowns [youtube.com].

    • Well what constitutes a bug?
      I am guessing this catches those compile errors or buffer overflow issues. However the most bugs I experience is due to miscommunication of the requirement.

      • My view of bugs are that the requirements are known, but are changed along the way, so how you deal with the original requirement isn't necessarily how you deal with the new, improved requirement, and trying to not have to rewrite the entire codebase simply to adjust for the whims of a project manager. It is this "miscommunication" that causes the greatest bugs in my experience.

        A radio button is not a check box. Designing for one, and having to switch to another, without having to rewrite the surrounding co

        • It's not a lame example, It's a perfect example.

          Using a radio button means you want a max of 1 choice, yet 3 months down the line they'll change their mind and say they want to allow for multiple choices. You then have to decide on the default state for each checkbox like you decided on what the default radio button was. Then you have to change your database to store a flag for each checkbox instead of a value that has a relationship to a primary key table. And you have to decide on whether or not NULL i

    • You're looking for this picture ... "Ignorance Map"
      https://lh3.googleusercontent.... [googleusercontent.com]

      * Known Unknowns -- All the things you you know you don't know
      * Unknown Unknowns -- All the things you don't know you don't know
      * Errors -- All the things you think you know but don't
      * Unknown Knowns -- All the things you don't know you know
      * Taboos -- Dangerous, polluting or forbidden knowledge
      * Denials -- All the things too painful to know, so you don't

      Ironically, it is missing the most important part:

      * Known Knowns -- Eve

  • Unit Testing (Score:5, Insightful)

    by darkain ( 749283 ) on Friday July 08, 2016 @11:59AM (#52471961) Homepage

    So wait, what you're telling me is that someone just finally discovered that it is a good idea to unit test the unit testing tools!?

    • You are right that they are unit tests. You are wrong when you classify the software under test as unit testing tools. What is just slightly different here is that the unit tests are not developed by the developers of the tools. Somebody is doing 3rd party unit tests of code analysis tools.
  • So this is what Computer Science research has come to? How would you test bug finding software without introducing known bugs to the programs under test? Magic?

    • Re: (Score:3)

      by godrik ( 1287354 )

      Please do not judge CS researcher by a single paper. BTW, this paper was not even the best paper at IEEE SP. (complete list with best papers at [1].)

      From a quick read of the lava paper. It seems that the novelty aspect of LAVA is that the software can inject bugs automatically in complex codebase. So you no longer have a grad student writing obviously faulty code on a toy program or inserting a few bugs manually to test one or two software. LAVA allows you to insert a myriad of bug in a myriad of software t

  • ...that leverages the actual practice of research. Read all about it.

  • Artificially introduced bugs are too predictable and contrived. "I'll introduce a buffer overrun error here." Fine. But unless the algorithm can also find real-world buffer overruns, it's not worth much. In real world bugs, the bug is often not obvious to a casual observer, as it would be with an artificially introduced bug.

    • Re: (Score:2)

      by mysidia ( 191772 )

      You need a parallel experiment, where you have a team of HUMANS who are given the task to introduce sneaky security bugs into the software that allow arbitrary code execution, encryption key exposure, Or accidentally leak other data over the network but are hard to detect and won't show up in unit tests.

      You announce that there will be monetary compensation for the authors of the 10 bugs in each category that survive the largest number of code reviews.

      • You announce that there will be monetary compensation for the authors of the 10 bugs in each category that survive the largest number of code reviews.

        So you have categorized bugs already, well done! That way you can prevent claims based on submissions that you don't consider bugs.
        —
        Of course it's a bug, otherwise it would have been documented!

    • "Artificially introduced bugs are too predictable and contrived."

      Actually, it is the predictable and contrived nature that makes them useful. How else do you propose to prove that a tool can catch a bug of a particular type than actually designing such a bug of said type? You seem to think they are saying: "Hey, once we've done this, that's everything! Our unit tests cover every possible scenario!" You are like they guy who argues: "Testing is useless because it won't catch every possible problem, so wh

      • How else do you propose to prove that a tool can catch a bug of a particular type

        I'd analyze commits to actual living systems, looking for fixes made to correct bugs that were found "in the wild." I'd then test the system against the previous version of the code in which the bugs weren't yet fixed, and see if the system could catch them.

        By "casual observer," I was assuming that the observer would be proficient in programming.

  • what about having real qa & not end users and your own coders be the testes?

    And the QA needs to have the rights / means to change settings and set up the environment in different ways to test out different things. Helps to have people with a QA / testing mind set there.

    For some things changes how things work from an standard / the way it has been for a long time to a new default can seem like an bug / lead to bad stuff happening. Like the Gear shift confusion issue that can kill people where was QA on

  • Title is really misleading...
  • My first day as a video game tester, I was told find bugs in a newly released video game for the PC. I expected to find none, as I naively assumed no one would knowingly ship a product with bugs. By the end of the day, I presented the bugs found. I found every bug that the game shipped with, including a hard-to-reproduce crashed bug. When I later became a lead video game, I fought battles with the developers to ship bug free products and got overruled every time. Their bonuses were dependent on the game bei
    • I would overrule you too. There is no such thing as a bug free product of any complexity.

      • I would overrule you too.

        A fellow lead tester got overruled on a last minute crash bug for code release. Developers were unable to reproduce the crash, and, since their bonus was on the line, got management to approve for release. Turned out that the executable file got infected with a virus when the developer zipped up the files for the ftp server, which caused the video game to crash after being burned to a CD. A quarter-million CDs got trashed when management learned that the CD master still had the virus. Not something you want

    • " I found every bug that the game shipped with"

      No you didn't. The fact that you think you did is frigging hilarious!

      • No you didn't. The fact that you think you did is frigging hilarious!

        My boss was the lead tester for that video game. He confirmed that I found every open bug that the video game shipped with. He was very impressed that I found them all.

        • " He confirmed that I found every open bug that the video game shipped with. "

          "I found every bug that the game shipped with"

          Show me the word open in your original statement.

          • Show me the word open in your original statement.

            Why should I entertain your nitpicking? You obviously don't believe what I wrote. Not the first time, not the second time, and probably not the third time.

            • "Why should I entertain your nitpicking?"

              That is rich. A supposed highly qualified software tester who thinks that identification of the omission of an element that completely changes the meaning and context is "nitpicking". Yeah. You're great at your job buddy. Off you go now ...

              • A supposed highly qualified software tester who thinks that identification of the omission of an element that completely changes the meaning and context is "nitpicking".

                If you think that being a video game tester is equivalent to being a "highly qualified software tester," you don't know shit about software testing.

                You're great at your job buddy.

                I was a video game tester for six years. I spent the last 14+ years in IT support.

                • "If you think that being a video game tester is equivalent to being a "highly qualified software tester,""

                  Wow. You really are a complete fucking idiot :-)

                  " I spent the last 14+ years in IT support."

                  Never mind. That is so much better. Details are certainly not important in IT support! ROTFLMAO

  • I thought this was already called mutation testing [wikipedia.org] in Computer Science. At least in that you create variants of the original program with subtle changes so you can see which mutants are "killed" (aka detected) by your test cases. The more mutants you kill the better the test suite. It seems strange the paper doesn't reference that research at all...

  • They're features.
    And by adding more features, we're getting less - if it's Microsoft.

  • Article summary sucks (Score:3)

    by CrashNBrn ( 1143981 ) on Friday July 08, 2016 @03:50PM (#52474245)

    The research showed that the bug-finding tools they tested had a dismal aggregate detection rates -- 2%. Not only that, they often found bugs that weren’t even there, creating unnecessary work for quality assurance teams trying to clean up software before it’s released.
    ...

    The research team is planning a competition for this summer in which developers of bug-finding software receive a score based on how many vulnerabilities their tools detect in a piece of software made vulnerable by LAVA. The idea is to help the developers produce better products.

Slashdot Top Deals

Every nonzero finite dimensional inner product space has an orthonormal basis. It makes sense, when you don't think about it.

Close