> For the complete documentation index, see [llms.txt](https://docs.stammer.ai/stammer.ai-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.stammer.ai/stammer.ai-docs/chat-ai-agents/knowledge-base-explained/scrape-websites/24-hour-auto-scraping.md).

# 24-hour Auto-Scraping

This is a powerful option to enable via the knowledge base that will tell the AI agent to rescrape the selected website pages every 24 hours. 

All new or changed information will be updated within the knowledge base. 

When you enter in a URL to scrape there will be an option to select the auto-scrape option.

<figure><img src="https://1359281993-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkLIQnOFYHkQtdWxUzzFE%2Fuploads%2FeauCxG6QQ8z9L0uYPSaf%2FScreenshot%202024-05-02%20at%208.51.20%E2%80%AFAM.jpg?alt=media&amp;token=a014c935-2ec0-4381-9a37-3d4ba0a6859f" alt=""><figcaption></figcaption></figure>