tssci has a VERY long post about crawling in relation to vuln assessments.
"This post isn’t intended to be a retort to Jeremiah Grossman’s post
last month on Why crawling matters, but more of a follow-up post to my
latest blog entry on Why pen-testing doesn’t matter. Hint: both
pen-testing and crawling are still important/matter, but my CPSL
process described in my last post leans towards a new, idealistic
approach on the vulnerability problem.
I’ve covered web crawling before, in Scraping the web for fun
and profit as well as when I mentioned papers written by Shreeraj Shah
in my longest-yet post on 2007 Security Testing tools in review. Ajax
crawling was only partially covered, so we’ll continue to expand on
that topic (as well as other topics) in this blog entry. Another
motivator for posting information about crawling is that I’m seeing
some repeated information over at blogs such as
securitytechscience.com/blog — especially in their latest on Screen
Scraping."
Article Link: http://www.tssci-security.com/archives/2007/12/02/why-crawling-doesnt-matter/
