recon_00 (/robots.txt)
View the exercise here: PentesterLab: Recon 00
OBJECTIVE
For this challenge, your goal is to retrieve the robots.txt
from the main website for hackycorp.com
.
THE ROBOTS.TXT
FILE
ROBOTS.TXT
FILEThe robots.txt
file is used to tell web spiders how to crawl a website. To avoid having confidential information indexed and searchable, webmasters often use this file to tell spiders to avoid specific pages. This is done using the keyword Disallow
. You can find more about the robots.txt
file by reading Robots exclusion standard
SOLUTION
Checking the /robots.txt
file reveals paths that are restricted from search engine indexing. These paths may contain sensitive or hidden information that attackers can exploit, making it essential to review during reconnaissance.
We'll find the flag for this challenge in hackycorp.com/robots.txt
.

Last updated
Was this helpful?