Larry Suto published a report comparing the various commercial web application security scanners. As you'd expect the vendors are likely to respond about how inaccurate the report is, however in this case both HP and Acunetix argued valid points.
- From Acunetix "They were not found because Larry didn’t authenticated our scanner
(didn’t provided any credentials). No wonder that Acunetix didn’t found
the vulnerabilities." – Acunetix - From HP "We also identified several vulnerabilities that we believe to be false
positives or not consistent representations with the author’s
vulnerability results. Beyond the "reported" vulnerabilities, we have
also found that WebInspect identifies several vulnerabilities in other
vendors websites not mentioned in the report. Each of these findings
and methodologies used by Suto raise serious doubts about the validity
of the conclusions reached within his report for WebInspect as well as
the other vendor’s scanners." – HP
For those of you who have been reading CGISecurity for a long time, I used to work at an application security scanning vendor (SPI Dynamics). These types of tools are very complicated, require expert customization on a per site basis for the best/most accurate results, and untrained point and shoot is a TERRIBLE comparison methodology. Every vendor makes their own demo site and will ensure they score well against, and if you hear a sales guy spinning scan speed as a sales point you want to run away (quick scans = less being checked for). Additionally false positives need to be tweaked away on a per site basis (you'll always get false positives). I would ensure that you do your own due diligence/testing against your own applications if you want to run a tool such as these against your own site. I haven't gotten around to doing a deep dive on the report, expect an update once I get some time.
In 2006 I posted a lengthy entry on challenges on automated scanning, I suggest checking it out if you are considering using a product such as this in your organization.
