llms.txt: guide file for AI crawlers

What is a llms.txt?

Definition: llms.txt is a human and machine-readable document in the website root directory example.com/llms.txt. It contains contact details, usage rules and optional information for LLMs.

Objective:

• Transparency towards AI systems
• Correct attribution
• Clarification of usage purposes (e.g. “allowed for Q&A”)

Delimitation:

• robots.txt regulates crawling/indexing for web crawlers.
• llms.txt describes usage and citation rules for LLMs.

Structure & fields (examples)

Typical keys (simple plain text, each on its own line):

• contact: email or contact URL
• website: main URL of the organization
• policy: “allow”/“disallow” (note for LLM usage)
• purpose: short description of permitted use
• comment: Free text, e.g. attribution notes
• Optional sections: Brief profile, services, unique selling points, important sources/ “sameAs” links

Sample snippet

# llms.txt for Beispiel GmbH
# Guidelines for large language model systems
# Standard: https://llmstxt.org/

contact: info@beispiel-gmbh.de
website: https://www.beispiel-gmbh.de/
policy: allow
purpose: Inhalte dürfen für Q&A genutzt werden, bitte mit Quellenangabe.
comment: Quelle: Beispiel GmbH, https://www.beispiel-gmbh.de

# ------------------------------
# Unternehmensbeschreibung
Die Beispiel GmbH ist ein mittelständisches Industrieunternehmen mit Sitz in Deutschland. Wir liefern Komponenten und technische Lösungen für den Maschinen- und Anlagenbau.

# Kernleistungen
- Dichtungstechnik
- Verbindungselemente
- Sonderanfertigungen nach Zeichnung
- Logistiklösungen (Just-in-Time, Kanban)

# sameAs
https://www.linkedin.com/company/beispiel-gmbh
https://www.verzeichnis.de/beispiel-gmbh

Best Practices

• Location: always at https://deine-domain.de/llms.txt (root level).
• Clarity before creativity: short, unambiguous phrases; a language.
• Contact & attribution: E-Mail + Please quote the source
• Consistency: Statements should match imprint/data protection.
• Versioning: Date major changes (e.g. # Updated: 2025-09-24).
• Addition, not substitute: robots.txt, sitemaps, and meta tags remain relevant.

Common mistakes

• Marketing texts only: Too vague, without specific policies.
• Hiding sensitive content: llms.txt is public; no secrets.
• Unclear rights: “allow” ≠ blank permission — name the purpose and attribution.
• Forgotten root path: Crawlers often ignore filing in subfolders.

Step-By-Step: Create llms.txt (How-To)

1. Set a plan: Purpose (“What can LLMs use content for? “) + attribution.
2. Create file: Text file llms.txt in UTF-8 format.
3. Write core entries: Contact, website, policy, purpose, comment.
4. Add optional information: Brief profile, services, sameAS links.
5. Save & deploy: Load a file to the domain root.
6. Test: Check https://deine-domain.de/llms.txt in the browser.
7. Maintain: Date changes, document them when policy changes.

TL; DR

llms.txt is a simple policy file for AI crawlers. Place them in the root, keep yourself brief and specific (contact, policy, purpose, attribution). It complements robots.txt, but it doesn't replace it.