How to Create an llms.txt File: Manual and Automated Guide

You understand what llms.txt is and why it matters. Now you want to actually create one. The good news is that building an llms.txt file takes less time than writing this sentence takes to read. The bad news is that most guides on the internet skip the details that determine whether your file is genuinely useful or just a placeholder nobody will read.

In this guide, we walk through every step of creating an llms.txt file, from opening your text editor to verifying it works correctly. We also cover automated options for WordPress users, common mistakes that ruin otherwise good files, and a maintenance schedule that keeps your llms.txt accurate as your site evolves.

Whether you are a developer running a documentation site, a content manager for an LMS platform, or a marketer exploring AI visibility tactics, this guide gives you everything you need to create a professional llms.txt file today.

What an llms.txt File Contains

An llms.txt file is a Markdown document. It uses plain text formatting with H2 headers to organize sections and Markdown links to point to resources. There is no XML, no JSON, no special syntax to memorize. The structure is intentionally simple so that non-technical team members can maintain it without developer involvement.

The typical sections are: an overview describing your site, organization, or project; a documentation section linking to technical resources; a policies section with legal and usage terms; a blog or content section listing recent or important posts; a products or courses section for e-commerce or LMS platforms; and a data section for open datasets or downloadable resources. The exact sections depend on your content type, but these six cover the vast majority of use cases.

Real-World Example Structure

Here is a representative structure for a SaaS company that sells a developer tool with documentation, a changelog, and public datasets:

Each section uses an H2 header and a short description followed by a list of Markdown links. Notice how the link descriptions give context: the changelog link is described as a “monthly release log,” not just labeled “Changelog.” This extra context helps AI systems understand the nature of each linked resource.

What Belongs and What Does Not

Not every URL on your site belongs in llms.txt. The file is a curated selection, not a comprehensive sitemap. Include your most authoritative, stable, and context-rich pages. Exclude login pages, user-specific dashboards, temporary promotional pages, and anything behind a paywall that you do not want AI systems summarizing.

The filtering step is the most important part of creating a good llms.txt file. A list of 200 links organized into six vague sections is less useful to AI crawlers than 30 precisely selected links with clear descriptions. Quality over quantity is not just a content strategy cliche. It is the defining principle of effective llms.txt curation.

Building Your llms.txt File: Step by Step

Now let us build an actual file. The process has five steps: inventory your content, define sections, write descriptions, add links, and publish. Each step takes minutes, but together they produce a file that genuinely helps AI systems understand your site.

Step 1: Inventory Your Content

Before opening your text editor, list the content you want AI systems to be able to find. Go through your site and identify: core documentation pages, API references, product or service pages, blog posts you are proud of, downloadable resources, legal and policy pages, and structured datasets. Do not try to capture everything. Focus on the 20 to 50 resources that best represent what your site offers.

For each resource, write down its URL, a one-sentence description of what it contains, and which section it belongs in. This inventory becomes your working draft. Having it in a spreadsheet or notes app before you start writing the file keeps the process organized and prevents you from forgetting key resources halfway through.

Step 2: Define Your Sections

Group your inventory into sections using H2 headers. Each section should contain related resources. For a developer platform, you might separate API docs, SDK guides, and changelogs. For an educational site, you might separate courses, tutorials, and research papers. The section names should be descriptive enough that an AI system reading them understands what each section covers.

Avoid generic section names like “Resources” or “Important Pages.” Use specific names like “API Reference,” “Integration Guides,” or “Open Datasets.” The more specific you are, the more useful your llms.txt file becomes. An AI system scanning your file should be able to get a quick mental model of your content architecture without reading every link.

Step 3: Write One-Line Descriptions

Every link in your llms.txt file should be preceded by a brief description. Not a paragraph, not a sentence fragment, but a complete thought that tells the reader what the linked resource contains and why it matters. One line is enough. Two lines is acceptable. A link with no description is almost useless to an AI system trying to decide whether to fetch and cite that page.

Write descriptions from the perspective of an AI system encountering your site for the first time. What context would you want? Mention the content type, the topic it covers, and whether it is updated regularly. For example: “Monthly changelog documenting all product updates, bug fixes, and new feature releases since 2024.” That description tells the AI system what the page is, how it is organized, and how current it is.

Step 4: Assemble and Format

Now write the actual file. Open your text editor, create a new file called llms.txt, and assemble your content using Markdown formatting. Start with a title, add your H2 sections with their descriptions and links, and review the formatting for consistency. Use the same style for all internal links: either all inline descriptions or all list items, but not a random mix of both.

Save the file and upload it to the root directory of your web server. The final URL should be https://yourdomain.com/llms.txt. Test this URL in a browser to verify it loads correctly and displays the raw Markdown text. If you see a 404, a redirect, or a styled page instead of raw text, fix the server configuration before proceeding.

Step 5: Verify and Maintain

After publishing, verify that AI crawlers can access the file. Use curl from a terminal to confirm the raw content is delivered correctly without authentication or redirects. Check that all links in the file resolve to existing pages with 200 status codes. Any broken links inside llms.txt will waste AI crawler resources and reduce trust in your signal.

Set up a quarterly review process. Every three months, check your llms.txt file against your current content. Remove links to deleted pages. Add links to new resources. Update descriptions if the nature of a resource has changed. If you use version control, commit these updates alongside other content changes so the history of your llms.txt is tracked.

Automated llms.txt Generation with Yoast SEO

If your site runs on WordPress, you do not need to manually write and maintain llms.txt from scratch. Yoast SEO now includes a built-in llms.txt generator that can automatically detect structured content and create the file for you. This is particularly useful for LMS websites, membership platforms, and any WordPress site with organized course or content structures.

How Yoast’s Feature Works

Yoast scans your WordPress content structure, identifies pages that follow learning-oriented patterns such as courses, lessons, modules, and quizzes, and uses that structure to populate an llms.txt file automatically. The result is a properly formatted Markdown file that reflects your actual content organization without requiring manual curation.

The generator offers two modes. Automatic mode lets Yoast decide which pages to include based on its internal pattern detection. Manual mode gives you full control over every section and every link, which is useful if your content structure is unique or if you want to exclude specific pages. For most LMS sites, automatic mode produces solid results. For complex or mixed-use sites, manual mode gives you the precision you need.

Setting It Up in Yoast SEO

Navigate to your WordPress admin, go to SEO, then Advanced, and look for the llms.txt toggle. Enable the feature, choose your preferred mode, and save. Yoast will generate the file at https://yourdomain.com/llms.txt automatically. If you are using manual mode, you will see a list of detected learning content that you can select, deselect, and organize before saving.

After enabling, visit the file URL directly in a browser to confirm it is loading correctly. If you have caching plugins active, you may need to clear your cache for the new file to appear. Yoast also shows a preview of the generated file within the plugin settings, which is useful for reviewing before publishing.

Best Practices for Yoast llms.txt

Even with automated generation, you should review the output. Yoast may include pages you intended to keep private, such as draft courses or preview-only content. It may miss custom post types that your theme registers but Yoast does not recognize as learning content. A five-minute review after generating the file catches these issues before AI crawlers encounter an inaccurate signal.

If you want a comprehensive walkthrough of setting up llms.txt specifically for LMS and learning management websites, including scenario-based examples and troubleshooting tips, our dedicated guide on llms.txt for learning websites covers every detail.

Common Mistakes to Avoid

Most llms.txt issues stem from a handful of avoidable mistakes. Recognizing them early saves you from sending misleading signals to AI crawlers.

Using llms.txt as a Sitemap

The most common error is including every page on your site in llms.txt. This is not what the file is for. XML sitemaps are for comprehensive page discovery. llms.txt is for curated guidance. If your llms.txt contains hundreds of links in a flat list, AI systems will not be able to extract meaningful structure from it, and the file loses its purpose.

Mixing Up Syntax with robots.txt

Do not copy robots.txt directives like Disallow or User-agent into llms.txt. The syntax is completely different because the purpose is completely different. robots.txt controls access. llms.txt describes content. A Disallow rule has no meaning inside a Markdown file and will simply appear as confusing text to any AI crawler that reads it.

Leaving Broken Links

A link in llms.txt that returns a 404 or redirect wastes an AI crawler’s resources and reduces the trustworthiness of your overall signal. Before publishing, click through every link in your llms.txt file. If you manage a large site, use a link checker tool to automate this process. Broken links in technical files are more damaging than broken links in content pages because the audience is automated and unforgiving.

Forgetting to Update After Site Changes

Content moves. Products get discontinued. Documentation gets reorganized. Your llms.txt file should evolve with your site. The easiest way to ensure this happens is to tie llms.txt updates to your regular content review process. If your team reviews the site architecture every quarter, add llms.txt to that checklist. It takes five minutes and prevents stale signals from accumulating.

Testing Your llms.txt File

After creating or updating your llms.txt file, test it thoroughly before assuming it works correctly. The testing process is simple but covers a few specific areas.

First, verify file accessibility. Visit https://yourdomain.com/llms.txt directly and confirm you see raw Markdown text, not a styled page or error message. If your server serves HTML for unknown file types, configure it to serve llms.txt as text or plain. AI crawlers expect raw text, and serving HTML can break their parsing logic.

Second, check all links. Use a browser or command-line tool to confirm each URL returns a 200 status. Broken links are the most common reason otherwise good llms.txt files fail to deliver value.

Third, test from multiple environments. If you use a CDN, clear your cache and test again. If your site has both HTTP and HTTPS versions, test both. If you have regional variants, test each one. Consistency across environments is important because AI crawlers may access any of these entry points.

Beyond the File: What Actually Makes AI Crawlers Cite Your Content

Creating an llms.txt file is a smart forward-looking move. But if you want your content cited by AI systems today, llms.txt alone will not get you there. There are specific, proven tactics that move the needle right now.

Structured content with clear FAQ sections performs better in AI-generated answers. Proper schema markup helps AI systems understand your content context. Consistent publishing signals freshness and authority. Backlinks from reputable sites build the trust signals that AI systems use to decide what to cite. These are the tactics that work today, and they will continue working regardless of whether llms.txt becomes a standard.

For a complete breakdown of what actually influences AI citations versus what sounds good but delivers nothing, check out our guide on AI search visibility versus llms.txt.