« R.I.P. Apache 1.x: Apache 1.3.42 marks of end life | Main | 2010 SANS Top 25 Most Dangerous Programming Errors Released »

Larry Suto Web Application Security Scanner Comparison Report Inaccurate Vendors Say

Larry Suto published a report comparing the various commercial web application security scanners. As you'd expect the vendors are likely to respond about how inaccurate the report is, however in this case both HP and Acunetix argued valid points.

  • From Acunetix "They were not found because Larry didn’t authenticated our scanner (didn’t provided any credentials). No wonder that Acunetix didn’t found the vulnerabilities." - Acunetix

  • From HP "We also identified several vulnerabilities that we believe to be false positives or not consistent representations with the author’s vulnerability results. Beyond the "reported" vulnerabilities, we have also found that WebInspect identifies several vulnerabilities in other vendors websites not mentioned in the report. Each of these findings and methodologies used by Suto raise serious doubts about the validity of the conclusions reached within his report for WebInspect as well as the other vendor’s scanners." - HP

For those of you who have been reading CGISecurity for a long time, I used to work at an application security scanning vendor (SPI Dynamics). These types of tools are very complicated, require expert customization on a per site basis for the best/most accurate results, and untrained point and shoot is a TERRIBLE comparison methodology. Every vendor makes their own demo site and will ensure they score well against, and if you hear a sales guy spinning scan speed as a sales point you want to run away (quick scans = less being checked for). Additionally false positives need to be tweaked away on a per site basis (you'll always get false positives). I would ensure that you do your own due diligence/testing against your own applications if you want to run a tool such as these against your own site. I haven't gotten around to doing a deep dive on the report, expect an update once I get some time.

In 2006 I posted a lengthy entry on challenges on automated scanning, I suggest checking it out if you are considering using a product such as this in your organization.


Feed You can follow this conversation by subscribing to the comment feed for this post.

All Comments are Moderated and will be delayed!

I would like to make a few comments on this post. My basic conclusion in this report was that many of the scanners required extensive configuration by an expert to obtain satisfactory results. When I mentioned to some of the vendors that this was the case, they felt that comments like this would mar the easy to use perception with customers.

Also many of the vendors made constructive comments and sent me direct emails about where they thought I went wrong. HP made no such attempts, published no real findings. I question their results and commentary.

Interesting that so many people want to have their cake and eat it too.

Strange he included Qualys which is still in it's generation 1 as a app scanner and not Rapid7.

Yeah they absolutely do. I use a scanner as part of my job duty (as a customer this time instead of as a vendor) and you need to know what you're doing for the best results.

I wasn't saying 'your review of point and shoot' was a bad evaluation, I was saying 'point and shoot' in general is bad and not an accurate way to judge tools. Vendors in particular are bad about this from a marketing perspective (as you point out).

Overall, the discussion has been useful in educating the consumer public that Web app scanning overall is problematic.

How would you classify an alternative technology that protected Web apps without the need for scanning?

"quick scans = less being checked for"

Right. Absolutely nothing else could account for quicker scans - better multithreading, faster backend engine, better network support: none of THOSE things would affect the scan speed whatsoever.

With the advent of widespread mobile banking apps, these security issues can multiply as more and more people start to use them. As the post mentioned, the lure of faster connection speeds and simpler functions might lead sites to cut corners to provide a more attractive service for the public than their competition. And, is not the biggest security risk for applications the end user anyway? Even in my own online browsing, I find myself constantly using the same password over and over just because of the volume of sites I visit.


I agree with you, you will gain better performance doing these things and some vendors do handle general app performance better than others.

In my experience previously working for these types of vendors that isn't how sales is spinning it to the end consumer. Sales is spinning scan times as a huge selling point to uneducated users who don't know any better. In some cases I've seen this 'optimization' based on checking for less (but still covering the major 4-5 issues like xss). I probably should have better articulated this in the post.

"In 2006 I posted a lengthy entry on challenges on automated scanning"

I read this report which was very informative but I was wondering whether there is a new version to cover someof the modifications made in the interim period. I will have a check around your website, because there is good learning here.

@Security Clearance

Surprisingly the issues are relatively the same still. I guess the only thing I could add is that user's still do not understand the need of flow based testing. Flow based testing is where you are required to visit a sequence of urls in a particular order in order to perform a site function. Many tools support recording macro's that allow for testing of flow based testing, however the community as a whole hasn't educated people about the need to perform this type of scanning which is a huge shame. My general rule of thumb is, if you see a website and it has a multipage flow look for vulns in the flow because chances are no blackbox tool is finding them.

Hi everyone,

i know this arcticle is a little bit old, but it would be very nice, if someone could confirm our results.


Usually online scanner won’t be compared with software scanners, but the result looks interesting.