Role of Web Application Security Scanners in Pentesting

Pentesters use web application security scanners during web application security testing, not because these scanners can do things that a pentester cannot do manually, but because these scanners can speed up the whole testing process and thus reduce the overall time scale for testing. The scanners may be capable of conducting some of the tests in a matter of few hours that a tester may otherwise take a few days to do manually and thus help improving overall efficiency.

It is very essential to understand these scanners and the role these scanners can play during a pentest before making them part of your testing methodology. These scanners may be capable of finding technical vulnerabilities such as cross-site scripting, injection attacks (SQL injection, command injection etc.), directory traversal, default files and directories and configuration related issues but may not be capable of finding other logical issues such as business processes, shopping cart related issues, session management, privilege escalation, user enumeration through error messages and cryptographic issues. This can be illustrated as follows:

A scanner may be able to find all variants of cross-site scripting vulnerabilities within all input fields encountered during crawling, but may not be able to change the price of an item in a shopping cart or escalate the privileges of a user by modifying request parameters.

Thus these scanners should only be considered as an aid to your manual efforts and cannot be thought to replace a human brain completely. The best approach would be to use these scanners as a first pass followed by significant manual investigation that includes not just to validate the findings from these tools but to conduct more in depth testing using other methods.

Stir Caused by Analysis of Web Application Security Scanners

In October 2007, Larry Suto, an application security consultant, stirred the web application security industry by publishing a whitepaper "Analyzing the Effectiveness and Coverage of Web Application Security Scanners". The paper compared results from three leading commercial web application security scanners: HP WebInspect, IBM AppScan and NTObjective NTOSpider. The study was carried out to analyze the effectiveness of these scanners in the following four areas:

1. Web site links crawled by the scanners.
2. Coverage of application code (using Fortify Tracer).
3. Number of verified vulnerabilities found.
4. Number of false positives.
   

The results of the study were quite surprising as the lesser known NTOSpider came on top in every aspect of the study by covering more code base and finding more vulnerabilities while also finding less false positives, leaving behind both AppScan and WebInspect. This also meant that both AppScan and WebInspect missed a large number of vulnerabilities (i.e. false negatives). Larry's original paper can be found here.

The study caused a stir amongst the web application scanning vendors who were forced to conduct a similar study in their labs to verify the results of the original study. IBM responded by publishing a paper "Better Untaught Than Ill-Taught" and HP responded by publishing a similar analysis here. Both the vendors have criticized the results of original study by giving their own analysis of the results.

Even if we do not go by the numbers of this analysis, the real question here would be that how much do we trust these scanners for web application security testing? Jeremiah Grossman had conducted a Web Application Security Professional Survey some time ago, in which one of the questions was that during a web application security testing, how much of the testing methodology do the commercial security scanners complete? Only 57% of the respondents said that the commercial scanners complete only half or lesser part of their testing methodology, while other 43% did not use these scanners at all. Some of the respondents said that they do not use these scanners because of the false positives generated. But I would be more concerned about the false negatives, because if we rely too much on these scanning tools, we may be missing a good number of vulnerabilities within the tested applications and thus having a false sense of security.

FireCAT - Using Firefox as a Penetration Testing Tool

How about the idea of using a web browser as a h4cking tool?

Yes, with its library of security oriented extensions, Firefox is now evolving as an ethical h4cking tool. The framework that maps the security oriented collection of Firefox extensions is called FireCAT, which means Firefox Catalog of Auding exTensions.

FireCAT 1.3 is the latest version of the framework that was released on 27th November 2007 and include extensions from basic information gathering to advanced application security audits including cross-site scripting and SQL injection. In the current version of the framework, the number of these extensions has already exceeded sixty. The further details of the extensions in the framework are given in the paper 'Turning Firefox to an Ethical Hacking Platform'.

I have used some of these extensions and find them really useful during testing. But I would like to use more of these and see how powerful can a web browser become as a security testing tool?