####TODO
- Problem in Proxy Authentication and provide proxy authentication services as per user choice.
- Code organize properly
This is a website which lets you crawl any website and return a text file of all the urls and keywords in that domain.Just enter the name of website and it will crawl.
#USE
To use it download the zip file and extract it in the htdocs folder of your XAMPP Server.Run the Xampp Server and also the server.py file which creates a CGI Server for running the CGI script.
To start the server.py file from the terminal run the command ./server.py or run chmod +x server.py and then ./server.py
Run this file from the browser and you can crawl whichever site you want.