A little configurable spider to generate traffic.
|
2 years ago | |
---|---|---|
Dockerfile | 2 years ago | |
Readme.md | 2 years ago | |
build.sh | 2 years ago | |
requests_spider.py | 2 years ago | |
run.sh | 2 years ago | |
urls.json | 2 years ago |
Requests Spider is a simple Python script that crawls the links found in a list of URLs and returns the response times for each link. It uses the Requests library to fetch the HTML for each URL and BeautifulSoup to parse the HTML and extract the links.
To run the Requests Spider script, you will need:
git clone https://github.com/your-username/requests_spider.git
pip install -r requirements.txt
The Requests Spider script can be run from the command line using the following syntax:
python requests_spider.py [-h] [-f FORKS] [-d DELAY] [-v] url_file
where:
url_file
is the path to the JSON file containing the URLs to crawl-h
shows the help message and exits-f FORKS
sets the number of forks to use (default is 1)-d DELAY
sets the delay between each loop in seconds (default is 10)-v
enables verbose mode, which displays additional outputTo run the Requests Spider script with 2 forks and a delay of 5 seconds between loops, use the following command:
python requests_spider.py -f 2 -d 5 urls.json
This project is licensed under the MIT License - see the LICENSE.md file for details.