> Markdown for /glossary/robots-txt. Visit the full page for interactive content.

[Back to glossary](/glossary)

# robots.txt

A text file at a website's root that instructs web crawlers which pages or directories to crawl or skip. While traditionally used for search engine bots, rob...

A text file at a website's root that instructs web crawlers which pages or directories to crawl or skip. While traditionally used for search engine bots, robots.txt now also governs AI crawler access. Blocking key AI bots here directly impacts GEO visibility.

**Example:** A robots.txt that includes 'User-agent: GPTBot\\nDisallow: /private/\\nAllow: /' permits GPTBot to crawl public pages while blocking private areas — balancing AI visibility with access control.


---
*Content served via ServeMD for LLM consumption*