Parsero is a free script written in Python which helps you to automatically audit the Robots.txt file of a web server. In just a few seconds, you are able to get a lot of valuable information which is needed  when you are auditing a website.

TUTO : http://www.behindthefirewalls.com/2014/02/parsero-v06-is-out.html
(Permalink)