RFC: LLM-optimized documentation for Zig

[liby]

Summary

This RFC proposes adding AI-optimized documentation to the Zig ecosystem, primarily through a standardized /llms.txt file and corresponding Markdown files, designed specifically to help LLMs provide accurate and up-to-date information about Zig.

Motivation

• The Zig community frequently reports documentation challenges
• Pre-1.0 languages with frequent changes need documentation that's easily kept current
• LLMs increasingly serve as programming assistants but often struggle with Zig due to limited or outdated training data
• Without official AI-optimized resources, LLMs may provide incorrect or outdated information about Zig

Guide-level explanation

This proposal introduces:

1. A /llms.txt file at the root of ziglang.org that:

   • Acts as an entry point for LLMs seeking information about Zig
   • Contains links to Markdown-formatted documentation resources
   • Follows the emerging /llms.txt standard established by other projects
   • Is officially maintained to ensure accuracy and freshness

2. Corresponding .md files for each linked resource that:

   • Are optimized for LLM parsing and understanding
   • Include version information to indicate recency
   • Follow consistent formatting patterns

The goal is not to replace existing documentation or create dual-purpose docs, but rather to supplement with AI-specific resources.

Reference-level explanation

The implementation would include:

1. A /llms.txt file structured similarly to Stripe's implementation:

   # Zig Programming Language
   
   > Zig is a general-purpose programming language and toolchain for maintaining robust, optimal, and reusable software.
   
   ## Standard Library
   - [std.mem](https://ziglang.org/documentation/std/mem.md): Memory management and manipulation functions.
   - [std.fs](https://ziglang.org/documentation/std/fs.md): File system operations.
   ...
   
   ## Language Concepts
   - [Error handling](https://ziglang.org/documentation/concepts/errors.md): Zig's approach to error handling.
   - [Comptime](https://ziglang.org/documentation/concepts/comptime.md): Compile-time execution and metaprogramming.
   ...

2. Extensions to docgen.zig and doctest.zig to generate these AI-optimized formats

3. A standardized Markdown format for linked resources that includes:

   • Clear version information (e.g., "Valid for Zig 0.11.0 and later")
   • Syntax highlighting hints
   • Semantic sectioning
   • Explicit delineation of breaking changes

Drawbacks

• Additional maintenance burden
• Potential confusion between different documentation sources
• Limited direct benefit to human users, however, it is beneficial for users who use AI to learn Zig.

Rationale and alternatives

This approach:

• Focuses exclusively on AI needs rather than trying to serve dual purposes
• Leverages existing documentation generation infrastructure
• Follows the emerging /llms.txt standard used by companies like Stripe
• Creates minimal maintenance overhead while providing significant benefit to LLM users

Prior art

• Stripe's /llms.txt implementation: https://docs.stripe.com/llms.txt
• The /llms.txt standard: https://llmstxt.org/

Unresolved questions

• What's the minimal viable content set to begin with?
• How to balance maintenance effort with comprehensiveness?
• What criteria determine which topics get AI-optimized documentation?

Future possibilities

• API-specific optimizations for more accurate code completion
• Auto-generation tools to create and update the AI-optimized documentation
• Integration with language servers for contextual AI assistance

[Maarrk]

This proposal is a very temporary solution for multiple reasons:

1. The stated problem may solve itself by waiting:

• improving the documentation for humans, also improves it for language models; currently a lot of standard library features need to have their functionality guessed from name or understood from reading the source code, I feel like at the moment we're lacking the content to put there
• LLMs are supposedly getting better and better
• there is more and more Zig code to be stolen from public repositories to train on

2. The Zig language is not finished:

• before 1.0, the focus should be on finishing the language (and as far as I can tell it is, with language server being developed by the community, and all tooling only getting core team work as much as it's needed for the compiler)
• after 1.0 there is little need for all the infrastructure for versioning the documentation

3. The llms.txt file is itself not standardised, and another early proposal:

• it's not obvious that it provides all the stated benefits now
• what if that "standard" changes as well?

---

In the spirit of transparency, I will disclose that I am a human, so may exhibit a bias towards human needs.