Add URLs to Scrape

You can interact with this API by sending a POST request. This API allows you to crawl and train URLS.

API Endpoint

The following endpoint/URL should be used in order to access this API:

https://app.stammer.ai/en/chatbot/api/website/crawl

---

Request Headers

The API must have the following request headers:

Content-Type: application/json
Authorization: Token <Your-API-Token>

---

Request Body

The API requires the following parameters in the request body:

• website_urls (Type: List, Required) - List of website URLs to scrape.

• max_urls_to_scrape (Type: Integer, Required) - Maximum number of URLs to scrape.

• follow_links (Type: Boolean, Optional) - Whether to follow links found on the scraped pages.

• auto_scraping (Type: Boolean, Optional) - Whether to enable automatic scraping of websites.

• use_readability (Type: Boolean, Optional) - If enabled, readability is applied to URLs and only text content is extracted.

• use_advance_scraper (Type: Boolean, Optional) - If enabled, the chatbot will use the Advanced Scraper for scraping.

• scrape_images (Type: Boolean, Optional) - If enabled, images will also be scraped from the websites.

• scroll_to_end (Type: Boolean, Optional) - Whether to scroll to the end of the page during scraping.

• wait_for (Type: Integer, Optional) - Time in seconds to wait for page loading before scraping.

• chatbot_uuid (Type: UUID, Required) - None

• initiate_training (Type: Boolean, Optional) - Start training after scraping the urls.

Response Example

Request Example

Here's an example of the request:

import requests

# Define the API endpoint
url = "https://app.stammer.ai/en/chatbot/api/website/crawl"

# Set up authentication and headers
headers = {
    'Authorization': 'Token <YOUR-API-TOKEN>',
    'Content-Type': 'application/json',
    'Accept': 'application/json'
}



# Data is passed in the request body as JSON
data = {

    "website_urls": ['Item 1', 'Item 2', 'Item 3'],

    "max_urls_to_scrape": 610,

    "follow_links": True,

    "auto_scraping": True,

    "use_readability": True,

    "use_advance_scraper": True,

    "scrape_images": True,

    "scroll_to_end": True,

    "wait_for": 610,

    "chatbot_uuid": "12345678-1234-5678-1234-567812345678",

    "initiate_training": True

}
response = requests.post(url, headers=headers, json=data)



# Process the response
if response.status_code in (200, 201, 202):
    result = response.json()
    print("Response data:", result)
else:
    try:
        error_data = response.json()
        error_message = error_data.get('message') or error_data.get('error', 'Unknown error')
        print(f"Error: {error_message}")
    except ValueError:
        print(f"Error: Status code {response.status_code}")

Learn More - https://app.stammer.ai/en/api-docs/chatbot/QA/