Automated crawling and finding hidden spots

In the preceding section, I showed you the manual method for finding interesting directories, and you can do the same for finding pages, as well. If you have no time and you want to use an automated method, Burp offers you an easy way to scan your projects:

1. Go to the Target tab, and, in the site map, right-click on your Target project and select Spider this branch from the menu. At this point, the Spider tab will blink, telling you that it's in progress; depending on the application depth, this action should not take too long to execute.
2. If you want to check for the progress of the crawling, go to the Spider tab and select the Control sub-tab, and you should get an idea of what is going on there:

1. An even more powerful and time-consuming tool that can find hidden files and directories also exists in Burp. Be careful with this one, because it is aggressive, and it can sometimes cause the site to malfunction. If you use it, ask the development team to back up the database, in case things go in the wrong direction. 

1. To find hidden contents, right-click on the Application directory in the Target/Site map section. From the menu, select Engagement tools; then, click on the Discover content menu item. After this action, a pop-up menu will appear. To run it, all you have to do is click on the Session is not running button, and the advanced crawling will start executing: