requests_spider

Requests Spider

Requests Spider is a simple Python script that crawls the links found in a list of URLs and returns the response times for each link. It uses the Requests library to fetch the HTML for each URL and BeautifulSoup to parse the HTML and extract the links.

Getting Started

Prerequisites

To run the Requests Spider script, you will need:

• Python 3.x
• Requests library
• BeautifulSoup library

Installation

1. Clone the repository: git clone https://github.com/your-username/requests_spider.git
2. Install the required libraries: pip install -r requirements.txt

Usage

The Requests Spider script can be run from the command line using the following syntax:

python requests_spider.py [-h] [-f FORKS] [-d DELAY] [-v] url_file

where:

• url_file is the path to the JSON file containing the URLs to crawl
• -h shows the help message and exits
• -f FORKS sets the number of forks to use (default is 1)
• -d DELAY sets the delay between each loop in seconds (default is 10)
• -v enables verbose mode, which displays additional output

Example

To run the Requests Spider script with 2 forks and a delay of 5 seconds between loops, use the following command:

python requests_spider.py -f 2 -d 5 urls.json

License

This project is licensed under the MIT License - see the LICENSE.md file for details.

Acknowledgments

• Requests library: https://requests.readthedocs.io/
• BeautifulSoup library: https://www.crummy.com/software/BeautifulSoup/bs4/doc/