In October 2007, Larry Suto, an application security consultant, stirred the web application security industry by publishing a whitepaper "Analyzing the Effectiveness and Coverage of Web Application Security Scanners". The paper compared results from three leading commercial web application security scanners: HP WebInspect, IBM AppScan and NTObjective NTOSpider. The study was carried out to analyze the effectiveness of these scanners in the following four areas:
- Web site links crawled by the scanners.
- Coverage of application code (using Fortify Tracer).
- Number of verified vulnerabilities found.
- Number of false positives.
The results of the study were quite surprising as the lesser known NTOSpider came on top in every aspect of the study by covering more code base and finding more vulnerabilities while also finding less false positives, leaving behind both AppScan and WebInspect. This also meant that both AppScan and WebInspect missed a large number of vulnerabilities (i.e. false negatives). Larry's original paper can be found here.
The study caused a stir amongst the web application scanning vendors who were forced to conduct a similar study in their labs to verify the results of the original study. IBM responded by publishing a paper "Better Untaught Than Ill-Taught" and HP responded by publishing a similar analysis here. Both the vendors have criticized the results of original study by giving their own analysis of the results.
Even if we do not go by the numbers of this analysis, the real question here would be that how much do we trust these scanners for web application security testing? Jeremiah Grossman had conducted a Web Application Security Professional Survey some time ago, in which one of the questions was that during a web application security testing, how much of the testing methodology do the commercial security scanners complete? Only 57% of the respondents said that the commercial scanners complete only half or lesser part of their testing methodology, while other 43% did not use these scanners at all. Some of the respondents said that they do not use these scanners because of the false positives generated. But I would be more concerned about the false negatives, because if we rely too much on these scanning tools, we may be missing a good number of vulnerabilities within the tested applications and thus having a false sense of security.
The study caused a stir amongst the web application scanning vendors who were forced to conduct a similar study in their labs to verify the results of the original study. IBM responded by publishing a paper "Better Untaught Than Ill-Taught" and HP responded by publishing a similar analysis here. Both the vendors have criticized the results of original study by giving their own analysis of the results.
Even if we do not go by the numbers of this analysis, the real question here would be that how much do we trust these scanners for web application security testing? Jeremiah Grossman had conducted a Web Application Security Professional Survey some time ago, in which one of the questions was that during a web application security testing, how much of the testing methodology do the commercial security scanners complete? Only 57% of the respondents said that the commercial scanners complete only half or lesser part of their testing methodology, while other 43% did not use these scanners at all. Some of the respondents said that they do not use these scanners because of the false positives generated. But I would be more concerned about the false negatives, because if we rely too much on these scanning tools, we may be missing a good number of vulnerabilities within the tested applications and thus having a false sense of security.
Posts
0 comments:
Post a Comment