llm-docs-builder

What does this tool actually do?

This tool takes your existing documentation and makes it smaller and cleaner so AI models can understand it better. It removes unnecessary parts like navigation bars and footers, and converts HTML to markdown. This saves up to 95% of the tokens (words) that AI models need to process, making your docs cheaper and faster to use with LLMs.

Deep dive

llm-docs-builder is a Ruby-based CLI tool that transforms and optimizes markdown documentation for Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems. It addresses the problem that LLMs waste 70-90% of their context window on human-oriented HTML content (navigation, footers, JavaScript, CSS). The tool normalizes links, removes unnecessary content, optimizes documents for LLM context windows, and enhances documents for RAG retrieval with hierarchical heading context and metadata. It can automatically convert HTML to markdown, measure token savings by comparing human vs AI versions, and generate llms.txt files for standardized documentation indexing. Real-world tests show an average reduction of 83% fewer tokens. The tool supports Docker and RubyGems installation, and can process local files or remote URLs.

Additional use cases

• Prepare documentation for RAG systems to improve retrieval accuracy.
• Compare token efficiency of human vs AI-optimized documentation pages.

Security and Trust Notes

• Token reduction may remove content that is important for human readers; review optimized output.