# Add URLs to Scrape

**API Endpoint**

The following endpoint/URL should be used in order to access this API:

```
https://app.stammer.ai/en/chatbot/api/website/crawl
```

***

**Request Headers**

The API must have the following request headers:

```
Content-Type: application/json
Authorization: Token <Your-API-Token>
```

***

**Request Body**

The API requires the following parameters in the request body:

* `website_urls` (Type: List, Required) - List of website URLs to scrape.
* `max_urls_to_scrape` (Type: Integer, Required) - Maximum number of URLs to scrape.
* `follow_links` (Type: Boolean, Optional) - Whether to follow links found on the scraped pages.
* `auto_scraping` (Type: Boolean, Optional) - Whether to enable automatic scraping of websites.
* `use_readability` (Type: Boolean, Optional) - If enabled, readability is applied to URLs and only text content is extracted.
* `use_advance_scraper` (Type: Boolean, Optional) - If enabled, the chatbot will use the Advanced Scraper for scraping.
* `scrape_images` (Type: Boolean, Optional) - If enabled, images will also be scraped from the websites.
* `scroll_to_end` (Type: Boolean, Optional) - Whether to scroll to the end of the page during scraping.
* `wait_for` (Type: Integer, Optional) - Time in seconds to wait for page loading before scraping.
* `chatbot_uuid` (Type: UUID, Required) - None
* `initiate_training` (Type: Boolean, Optional) - Start training after scraping the urls.

<figure><img src="/files/MmfPfJw0jEQjI1DHiJa6" alt=""><figcaption></figcaption></figure>

**Response Example**

<figure><img src="/files/TGgW84wZwMJEA1Be1I3E" alt=""><figcaption></figcaption></figure>

**Request Example**

Here's an example of the request:

```python
import requests

# Define the API endpoint
url = "https://app.stammer.ai/en/chatbot/api/website/crawl"

# Set up authentication and headers
headers = {
    'Authorization': 'Token <YOUR-API-TOKEN>',
    'Content-Type': 'application/json',
    'Accept': 'application/json'
}



# Data is passed in the request body as JSON
data = {

    "website_urls": ['Item 1', 'Item 2', 'Item 3'],

    "max_urls_to_scrape": 610,

    "follow_links": True,

    "auto_scraping": True,

    "use_readability": True,

    "use_advance_scraper": True,

    "scrape_images": True,

    "scroll_to_end": True,

    "wait_for": 610,

    "chatbot_uuid": "12345678-1234-5678-1234-567812345678",

    "initiate_training": True

}
response = requests.post(url, headers=headers, json=data)



# Process the response
if response.status_code in (200, 201, 202):
    result = response.json()
    print("Response data:", result)
else:
    try:
        error_data = response.json()
        error_message = error_data.get('message') or error_data.get('error', 'Unknown error')
        print(f"Error: {error_message}")
    except ValueError:
        print(f"Error: Status code {response.status_code}")

```

Learn More - <https://app.stammer.ai/en/api-docs/chatbot/QA/>


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.stammer.ai/stammer.ai-docs/white-label/white-label-api/knowledge-base-api/add-urls-to-scrape.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.