Supercharging LLM understanding of Cerbos documentation

Large Language Models (LLMs) are becoming an integral part of how we interact with information and the web. As they evolve, their ability to understand and process vast amounts of online content is improving in near real-time. However, this process isn't without its challenges.

Modern websites are often cluttered with content that, while useful for human interaction, can be confusing for an LLM. Ads, navigation menus, pop-up banners, and scripts can obscure the core information on a page. This means that when you ask an LLM a question, it might miss key details buried within a website's complex structure.

To address this, a new standard is emerging: the llms.txt file. We at Cerbos have embraced this to ensure our documentation is as clear and accessible to these models as it is to our users.

What is llms.txt?

At its core, llms.txt is a simple yet powerful idea. It's a Markdown file, located at the root of a website (e.g., https://docs.cerbos.dev/llms.txt), that provides a clean, concise, and LLM-friendly version of the site's content. It acts as a set of notes, giving LLMs the essential information they need to understand what a site or a specific page is about, free from the noise of modern web design.

The primary benefits of this approach are:

• Efficiency: By providing a distilled version of the content, LLMs can generate better and more accurate responses, even if their training data didn't include the specific content.
• Improved context: LLMs have a limited "context window," meaning they can only process so much information at once. Feeding them only the most relevant content maximizes their ability to understand the subject matter.
• Better user experience: When LLMs have a clearer understanding of our documentation, they can more accurately match a user's query to the solutions Cerbos provides, leading to a better experience for everyone.

The proposed standard also includes an extended version of this file, llm-full.txt. This file, which we have also implemented, can contain more detailed information, such as specific URLs and additional sections, for a more comprehensive overview.

Why we added it to the Cerbos documentation

Just as with traditional SEO, our goal is to ensure that the information people find about Cerbos is accurate, helpful, and reflects our messaging. We want to guide LLMs to the most important features and documentation, especially as our product evolves and documentation is updated.

By providing llms.txt and llms-full.txt files for our documentation, we can help these models navigate the content, distinguish between different versions, and surface the most relevant information for our users' queries.

Open sourcing our solution for the Antora community

The Cerbos documentation is built using Antora, a popular open-source documentation site generator. To implement llm.txt, we developed our own Antora extension to automatically generate these files from our source content.

In the spirit of open source, we are delighted to share this tool with the wider community. We have open-sourced the Cerbos Antora LLM Generator, making it easy for any project using Antora to add llm.txt and llm-full.txt files to their documentation sites.

We invite you to take a look at the generated files on our documentation site:

• https://docs.cerbos.dev/llms.txt
• https://docs.cerbos.dev/llms-full.txt

As the way we find and consume information continues to evolve, we must adapt. By embracing new standards like llm.txt and contributing to the open-source tools that power them, we are helping to build a more accessible and intelligent web for everyone.