Have an account?
It looks like you're new here. If you want to get involved, click one of these buttons!
Apply for Membership
Who's Online (0)
Looking to introduce yourself? Look no further, and click here! We also have IRC! [irc.evilzone.org #iexploit]
General Security Discussions
robot and how to find it
what is robot and how it can be configured to hide some parts of a website from search engine for example admin 's login page?
robots.txt is found in the sites root directory if it exists.
use a crawler and scan the site.. you will find it something like robots.txt
and regarding your second question,you answered the question by your-self :)
when I'm using crawler I have to use Proxy or vpn?
If you want to stay hidden and you'll be conducting further activities on the same host then using a vpn/proxy may be worthwhile but either-way crawling a site is quite noisy and will soon be filling up the server's access logs.
Add a Comment