# Add URLs to Scrape

**API Endpoint**

The following endpoint/URL should be used in order to access this API:

```
https://app.stammer.ai/en/chatbot/api/website/crawl
```

***

**Request Headers**

The API must have the following request headers:

```
Content-Type: application/json
Authorization: Token <Your-API-Token>
```

***

**Request Body**

The API requires the following parameters in the request body:

* `website_urls` (Type: List, Required) - List of website URLs to scrape.
* `max_urls_to_scrape` (Type: Integer, Required) - Maximum number of URLs to scrape.
* `follow_links` (Type: Boolean, Optional) - Whether to follow links found on the scraped pages.
* `auto_scraping` (Type: Boolean, Optional) - Whether to enable automatic scraping of websites.
* `use_readability` (Type: Boolean, Optional) - If enabled, readability is applied to URLs and only text content is extracted.
* `use_advance_scraper` (Type: Boolean, Optional) - If enabled, the chatbot will use the Advanced Scraper for scraping.
* `scrape_images` (Type: Boolean, Optional) - If enabled, images will also be scraped from the websites.
* `scroll_to_end` (Type: Boolean, Optional) - Whether to scroll to the end of the page during scraping.
* `wait_for` (Type: Integer, Optional) - Time in seconds to wait for page loading before scraping.
* `chatbot_uuid` (Type: UUID, Required) - None
* `initiate_training` (Type: Boolean, Optional) - Start training after scraping the urls.

<figure><img src="https://1359281993-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkLIQnOFYHkQtdWxUzzFE%2Fuploads%2FcbfBZXRFiqRmrMhmgYCV%2FScreenshot%202024-07-03%20at%202.54.23%E2%80%AFPM.jpg?alt=media&#x26;token=4dd4f9a7-022f-4318-a4ea-495300ebc77d" alt=""><figcaption></figcaption></figure>

**Response Example**

<figure><img src="https://1359281993-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkLIQnOFYHkQtdWxUzzFE%2Fuploads%2FivsnCfmJOiaseymoXiEp%2FScreenshot%202025-03-21%20at%201.32.05%E2%80%AFPM.jpg?alt=media&#x26;token=339ba8f4-0506-46ef-a6f4-2a5464e3663f" alt=""><figcaption></figcaption></figure>

**Request Example**

Here's an example of the request:

```python
import requests

# Define the API endpoint
url = "https://app.stammer.ai/en/chatbot/api/website/crawl"

# Set up authentication and headers
headers = {
    'Authorization': 'Token <YOUR-API-TOKEN>',
    'Content-Type': 'application/json',
    'Accept': 'application/json'
}



# Data is passed in the request body as JSON
data = {

    "website_urls": ['Item 1', 'Item 2', 'Item 3'],

    "max_urls_to_scrape": 610,

    "follow_links": True,

    "auto_scraping": True,

    "use_readability": True,

    "use_advance_scraper": True,

    "scrape_images": True,

    "scroll_to_end": True,

    "wait_for": 610,

    "chatbot_uuid": "12345678-1234-5678-1234-567812345678",

    "initiate_training": True

}
response = requests.post(url, headers=headers, json=data)



# Process the response
if response.status_code in (200, 201, 202):
    result = response.json()
    print("Response data:", result)
else:
    try:
        error_data = response.json()
        error_message = error_data.get('message') or error_data.get('error', 'Unknown error')
        print(f"Error: {error_message}")
    except ValueError:
        print(f"Error: Status code {response.status_code}")

```

Learn More - <https://app.stammer.ai/en/api-docs/chatbot/QA/>