> Markdown for /docs/llms-txt-crawlability. Visit the full page for interactive content.

[Back to docs](/docs)

🔧 Technical Fixes4 min read

# llms.txt & AI Crawlability

On this page

## What Is llms.txt?

llms.txt is a recommended file that tells AI crawlers which pages to prioritise when indexing your site. Similar to robots.txt for search engines, llms.txt helps AI systems like ChatGPT, Claude, and Perplexity discover your most important content efficiently. Topicker checks for its presence and generates a template if it's missing.

## AI Crawler Access

Topicker inspects your robots.txt rules for known AI crawlers: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Bytespider (ByteDance), and CCBot (Common Crawl). For each crawler, it reports whether access is Permitted, Blocked, or Missing from your robots.txt configuration.

## Fixing Blocked Crawlers

If AI crawlers are blocked in your robots.txt, Topicker provides specific directives to fix the issue. For example, to allow GPTBot, add 'User-agent: GPTBot\\nAllow: /' to your robots.txt. Be intentional about which crawlers you allow — blocking all AI crawlers will prevent your content from appearing in AI-generated answers.

## Generating llms.txt

If your llms.txt is missing, Topicker generates a template based on your site's structure. The template includes your site description, primary content categories, and recommended priority pages. Customise the template to highlight your most important content, then upload it to your site's root directory.


---
*Content served via ServeMD for LLM consumption*