Build a Custom AI Internal Linking Pipeline with Python

No-code tools are fantastic until they are not. Maybe your site runs on a custom CMS that no tool supports. Maybe you want full control over how links are scored and inserted. Or maybe you simply enjoy building things yourself. Whatever the reason, building a custom AI internal linking pipeline with Python is more approachable than it sounds.

In this guide, we will build a working pipeline from scratch. You will learn how to load your sitemap, generate embeddings for your content, run semantic similarity searches, and use an LLM to insert links naturally into articles. We will cover every file, every library choice, and the specific prompt that prevents link stuffing.

This guide is part of our series on how to automate internal links with AI. Start there for the broader context if you have not read the parent article yet.

What You Will Build

By the end of this guide, you will have a Python pipeline that takes a target article URL and a sitemap URL, then returns a revised version of that article with relevant internal links inserted at natural anchor points.

The pipeline has four stages:

1. Ingestion: Extract content from your sitemap and target article
2. Embedding: Convert all content into vector embeddings for semantic search
3. Retrieval: Find the top-N most semantically similar pages to your target
4. Generation: Use an LLM to insert links into the target article naturally

Each stage runs independently, which means you can modify, replace, or skip stages as needed. The embedding model can be swapped. The LLM can be changed. The vector store can be replaced with a different backend.

Prerequisites

Before writing code, you need a few things in place. You will need Python 3.9 or later, an OpenAI API key (or equivalent from another provider), access to your website’s sitemap XML, and basic comfort with running Python scripts from the command line.

The total cost per run is minimal. Generating embeddings for 500 pages costs around fifty cents. Each article rewrite costs a few cents in LLM tokens. For a blog with 100 articles, you are looking at total costs in the low single-digit dollar range.

Setting Up the Environment

Start by installing the required packages. You will need openai for embeddings and the LLM, beautifulsoup4 for content extraction, requests for fetching URLs, and faiss-cpu (or chromadb) for vector storage. The full requirements breakdown is in our Python AI internal linking pipeline guide.

pip install openai beautifulsoup4 requests faiss-cpu tiktoken

Create a project folder and set up your environment variables. You need your OpenAI API key at minimum. If you are using a different provider for embeddings or the LLM, add those keys too.

Stage 1: Content Ingestion

The first stage reads your sitemap and extracts clean body text from each page. You want the actual article content, not navigation, footers, or sidebar text.

Extracting the Sitemap

Start by fetching your sitemap XML. Most WordPress sites use sitemap_index.xml at the root. Parse it to get all page URLs, then fetch each one and extract the article body. For WordPress sites, the article content typically lives inside a <article> tag or a div with a class like entry-content.

Fetching the Target Article

The target article is the one you want to optimize with internal links. Fetch it the same way and extract clean text. You also want to preserve the original HTML structure because the LLM will need to insert links into that structure later.

Stage 2: Generating Embeddings

This is where semantic search becomes possible. You convert each page’s content into a vector embedding, which is a long list of numbers representing the semantic meaning of the text.

Understanding Embeddings

Think of embeddings as a map of meaning. Two pieces of text about the same topic will land close together on this map, even if they use different words. “How to brew pour-over coffee” and “The science of water temperature for coffee extraction” might share no keywords but will be neighbors in embedding space because they are about the same concept.

That is what makes semantic search powerful for internal linking.

Building the Vector Index

For a blog with under 1000 articles, FAISS (Facebook AI Similarity Search) running on a CPU is fast enough. Store your embeddings in a FAISS index alongside a list of URLs so you can match search results back to actual pages.

The workflow:
1. Chunk each article into 500-1000 word segments (longer articles become multiple vectors)
2. Generate an embedding for each chunk using your chosen model
3. Store all embeddings in the FAISS index
4. Save the URL-to-chunk mapping for result lookup

Choosing an Embedding Model

OpenAI’s text-embedding-3-small is fast, affordable, and produces strong results for English content. For multilingual sites, consider a multilingual model like multilingual-e5-large.

For the technical details on how embedding models work and how to evaluate them for your specific use case, our deep dive on semantic search for internal links and embeddings covers the theory and practical implementation options.

Stage 3: Semantic Retrieval

With your vector index built, finding relevant pages is straightforward. Take your target article, generate an embedding for it, and search the index for the top-N nearest neighbors. Those nearest neighbors are your candidate pages for linking.

Filtering Candidates

Not every high-similarity page deserves a link. Filter out pages that are already linked from the target article. Filter by a minimum similarity threshold to avoid stretching relevance. And consider excluding very short pages like tag archives or author pages that would not make useful link targets.

Preparing the LLM Prompt

The retrieval stage gives you a ranked list of candidate pages. Each candidate includes a URL, title, and a short content excerpt. You pass these to the LLM along with the full target article and ask it to insert links naturally.

The prompt is everything here. A poorly written prompt produces forced links everywhere. A well-written one produces links that blend seamlessly into your existing prose.

Stage 4: Content Rewriting with the LLM

This is where the magic happens. The LLM reads your target article end to end, considers the candidate pages, and rewrites the article with links inserted only where they genuinely improve the content.

The Prompt That Prevents Link Stuffing

Your prompt needs to accomplish several things at once. It must tell the LLM what the target article is about, provide the candidate pages with their URLs and titles, instruct it to insert links only where contextually appropriate, and specify that links should feel natural, not forced.

Key instructions to include:
– Maximum of 2-3 links per section
– Do not add links if no relevant connection exists
– Anchor text must be descriptive and varied
– Do not link to pages already linked in the article
– Output only the revised HTML, no explanations

For the full prompt template and an explanation of why each instruction matters, see our guide to LLM prompt engineering for natural link insertion.

Handling the Output

After the LLM returns the revised HTML, you need a few validation checks before pushing it live. Verify that link URLs are valid and reachable. Count the number of new links added. Check that anchor text varies and avoids exact-match over-optimization. Spot-check a few sections to make sure the linking feels natural to a human reader.

Putting It All Together: The Full Pipeline Script

Here is how the four stages connect into a single pipeline. The script reads configuration from a YAML file, fetches your sitemap, builds or loads a cached vector index, retrieves candidates for a target article, calls the LLM for rewriting, and saves the result.

The core execution flow looks like this:

• Load config: API keys, sitemap URL, target article URL, number of candidates
• Check cache: If the vector index is recent enough, skip re-embedding
• Ingest content: Fetch sitemap pages and target article, extract text
• Embed: Generate embeddings for all cached pages
• Retrieve: Find top-N candidates for the target article
• Rewrite: Call LLM with candidates and target article
• Validate: Check link count, URL validity, anchor text diversity
• Output: Save revised HTML to file

We provide the complete, commented script in our Python AI internal linking pipeline guide. It is structured for readability and easy customization, not just copy-paste.

Extending the Pipeline

Once you have the basic pipeline working, there are natural extensions that make it more powerful.

Scheduled Runs

Instead of running the pipeline manually for each new article, schedule it to run automatically whenever a new post is published. A cron job or a webhook trigger from your CMS can call the pipeline scripts directly.

CMS Integration

The revised HTML output can be pushed directly to your CMS via API. WordPress has a REST API endpoint for updating posts. Webflow has its own CMS API. For headless CMSs like Contentful or Sanity, the process is similar. Add a small API call after the validation stage and your pipeline becomes fully automated end-to-end.

Link Health Monitoring

Over time, internal links break when pages are deleted or URLs change. Add a periodic health check that verifies all internal links on your site are returning 200 status codes. Our guide to auditing and fixing broken internal links with AI covers the full process.

And Here Is the Thing

Building your own pipeline is not just about cost savings. It is about understanding your content at a level no off-the-shelf tool can match. You can tune the similarity thresholds, customize the linking criteria, and set rules that match your specific site structure. A tool built for e-commerce will not understand that your “beginner Python tutorials” should link to your “intermediate Python tutorials” but not to your “product reviews.” Your pipeline can encode that logic.

The development pattern we cover here mirrors the approach outlined in our guide to building Claude Code plugins, where custom automation extends your existing tools with domain-specific intelligence.

Start with the basic four-stage pipeline. Get it running on a few articles. Review the output manually. Adjust the prompt, tune the similarity thresholds, and iterate. Within a few runs, you will have a system that inserts the same quality of links you would add manually, but at scale.

For the complete step-by-step tutorial with full code, configuration examples, and troubleshooting tips, see our complete guide to building a custom AI internal linking pipeline with Python.