feat(website): add llms.txt support for LLM-friendly content by cscheid · Pull Request #13932 · quarto-dev/quarto-cli

cscheid · 2026-01-21T22:05:06Z

Add support for generating llms.txt and .llms.md files for Quarto websites, providing LLM-friendly markdown versions of HTML pages.

Features:

New llms-txt: true option in website config
Generates .llms.md companion files alongside HTML output
Creates llms.txt index file linking to all markdown pages
Converts HTML to clean markdown using Pandoc with Lua filter
Handles callouts (blockquotes with bold type markers)
Converts images to markdown syntax
Converts internal links from .html to .llms.md
Respects draft settings (excludes drafts from output)
Cleans listing pages (removes empty links, category badges)
Matches sitemap behavior for incremental builds

New files:

src/project/types/website/website-llms.ts
src/resources/filters/llms/llms.lua

Test coverage:

Basic file generation
Content conversion (callouts, code, tables, links)
Draft handling
Listing page cleanup

posit-snyk-bot · 2026-01-21T22:05:26Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Add support for generating llms.txt and .llms.md files for Quarto websites, providing LLM-friendly markdown versions of HTML pages. Features: - New `llms-txt: true` option in website config - Generates .llms.md companion files alongside HTML output - Creates llms.txt index file linking to all markdown pages - Converts HTML to clean markdown using Pandoc with Lua filter - Handles callouts (blockquotes with bold type markers) - Converts images to markdown syntax - Converts internal links from .html to .llms.md - Respects draft settings (excludes drafts from output) - Cleans listing pages (removes empty links, category badges) - Matches sitemap behavior for incremental builds New files: - src/project/types/website/website-llms.ts - src/resources/filters/llms/llms.lua Test coverage: - Basic file generation - Content conversion (callouts, code, tables, links) - Draft handling - Listing page cleanup Co-Authored-By: Claude Opus 4.5 <[email protected]>

…tests - Add **/*.llms.md to projectHiddenIgnoreGlob() to prevent cascading renders of llms.txt companion files - Fix ensureLlmsTxt* test functions to use dirname(htmlFile) instead of treating file path as directory - Update llms-txt test files to use correct two-element array format for regex matches [matches, no_matches] - Add render-project: true where needed for llms.txt generation Co-Authored-By: Claude Opus 4.5 <[email protected]>

…tibility Use pathWithForwardSlashes() to ensure paths in llms.txt use forward slashes on all platforms. Also adds changelog entry for the llms-txt feature. Co-Authored-By: Claude Opus 4.5 <[email protected]>

cderv · 2026-01-26T21:27:51Z

Overall, it looks good! I think this is a great way to do it.

So I had a look at pkgdown implementation to compare:

They use llm-docs: false as fields to opt-out.
It also generates a llms.txt by mixing the content of README.md for the description part, and then an index of all the pages. For pkgdown, usually README.md is the index.html content. So it is included as such. Not sure it makes sense to add index.html content. 🤔
All pages '.htmlare just.md- notllm.md. I don't know if there is a convention. Website like Claude doc all use .md` (example with https://code.claude.com/docs/en/settings.md)
Links are absolute paths in pkgdown - they contain the site url. Same for claude docs website: https://code.claude.com/docs/llms.txt

After links there is a description based on source file header, the : <description> optional part. It would be good. I wonder if we should add a title and description field under llm-docs to customize that ? Or if we can find a way to use a default if provided (like open-graph.description ? ). Otherwise there is a description-meta available in HTML if set already

quarto-cli/src/resources/formats/html/pandoc/html.template

Lines 16 to 18 in 6c8a9b1

    
           $if(description-meta)$ 
        
             <meta name="description" content="$description-meta$" /> 
        
           $endif$

They do some special processing of some content, probably based on what a pkgdown can have. We will probably add more as users use this feature if we missed some. Just one example below.

For example Definition list are converted to bullet list. Probably because GFM does not support them

❯ pandoc -f html -t gfm
<dl>
<dt>Term 1</dt>
<dd>
<p>Definition 1</p>
</dd>
</dl>
<p>Term 2 with <em>inline markup</em></p>
^Z
Term 1
Definition 1

Term 2 with *inline markup*

unless activated but is this really GFM syntax ?

❯ pandoc -f html -t gfm+definition_lists
<dl>
<dt>Term 1</dt>
<dd>
<p>Definition 1</p>
</dd>
</dl>
<p>Term 2 with <em>inline markup</em></p>
^Z
Term 1

:   Definition 1

Term 2 with *inline markup*

An example of their output: https://pkgdown.r-lib.org/llms.txt

I am thinking among good ideas:

Using absolute url for links if site-url is set? A LLM would not have to reconstruct the url from base + relative link that way. 🤔
using same argument llm-docs ? but llm-txt seems more clear regarding the specification website and created file.
allow to customize [title](url): description or just use <meta name="description" content="$description-meta$" /> content if available ?

Just some ideas - I am sure we'll have more feedback when this will be tested.

cderv · 2026-01-26T21:43:35Z

tests/verify.ts

+  return {
+    name: `File ${llmsFile} exists`,
+    verify: (_output: ExecuteOutput[]) => {
+      verifyPath(llmsFile);
+      return Promise.resolve();
+    },


We have fileExists() if we want to refactor and avoid duplication

quarto-cli/tests/verify.ts

Lines 228 to 236 in 6c8a9b1

export const fileExists = (file: string): Verify => {

return {

name: `File ${file} exists`,

verify: (_output: ExecuteOutput[]) => {

verifyPath(file);

return Promise.resolve();

},

};

};

cderv · 2026-01-26T21:44:52Z

tests/verify.ts

+  return {
+    name: `File ${llmsFile} does not exist`,
+    verify: (_output: ExecuteOutput[]) => {
+      verifyNoPath(llmsFile);
+      return Promise.resolve();
+    },


We have pathDoNotExists() if we want to reuse and avoid code duplication

quarto-cli/tests/verify.ts

Lines 238 to 246 in 6c8a9b1

export const pathDoNotExists = (path: string): Verify => {

return {

name: `path ${path} do not exists`,

verify: (_output: ExecuteOutput[]) => {

verifyNoPath(path);

return Promise.resolve();

},

};

};

cderv · 2026-01-26T21:49:42Z

tests/verify.ts

+// Verify the llms.txt index file in a website output directory.
+// Takes the HTML file path and looks for llms.txt in the same directory.
+export const ensureLlmsTxtRegexMatches = (
+  htmlFile: string,
+  matchesUntyped: (string | RegExp)[],
+  noMatchesUntyped?: (string | RegExp)[],
+): Verify => {
+  const llmsTxtPath = join(dirname(htmlFile), "llms.txt");
+  return verifyFileRegexMatches(regexChecker, `Inspecting ${llmsTxtPath} for Regex matches`)(llmsTxtPath, matchesUntyped, noMatchesUntyped);
+};


This verify helper is to be used only for index.qmd or another .qmd test that will be at the root of the output dir right ?

If we need to have verify function that works on output-dir as input, it is just a matter of adding the function as special handling in smoke-all.test.ts and you could have

export const ensureLlmsTxtRegexMatches = ( outputDir: string, matchesUntyped: (string | RegExp)[], noMatchesUntyped?: (string | RegExp)[], ): Verify => { const llmsTxtPath = join(outputDir, "llms.txt"); return verifyFileRegexMatches(regexChecker, `Inspecting ${llmsTxtPath} for Regex matches`)(llmsTxtPath, matchesUntyped, noMatchesUntyped); };

But I guess this is just a matter of being sure to use ensureLlmsTxtRegexMatches() only in compatible source document.

Just a thought while reviewing the new functions in verify.ts

cderv · 2026-01-26T21:50:05Z

tests/verify.ts

+  return {
+    name: `File ${llmsTxtPath} exists`,
+    verify: (_output: ExecuteOutput[]) => {
+      verifyPath(llmsTxtPath);
+      return Promise.resolve();
+    },
+  };


Same - could be fileExists()

cderv · 2026-01-26T21:52:39Z

tests/verify.ts

+  return {
+    name: `File ${llmsTxtPath} does not exist`,
+    verify: (_output: ExecuteOutput[]) => {
+      verifyNoPath(llmsTxtPath);
+      return Promise.resolve();
+    },
+  };


And same could be pathDoNotExists

cscheid force-pushed the feature/llms-txt branch from 20d8156 to b5a829e Compare January 21, 2026 22:14

cscheid requested a review from cderv January 21, 2026 22:53

fix(website): normalize path separators in llms.txt for Windows compa…

1a824a6

…tibility Use pathWithForwardSlashes() to ensure paths in llms.txt use forward slashes on all platforms. Also adds changelog entry for the llms-txt feature. Co-Authored-By: Claude Opus 4.5 <[email protected]>

cderv self-assigned this Jan 23, 2026

cderv reviewed Jan 26, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(website): add llms.txt support for LLM-friendly content#13932

feat(website): add llms.txt support for LLM-friendly content#13932
cscheid wants to merge 3 commits intomainfrom
feature/llms-txt

cscheid commented Jan 21, 2026

Uh oh!

posit-snyk-bot commented Jan 21, 2026 •

edited

Loading

Uh oh!

cderv commented Jan 26, 2026

Uh oh!

cderv Jan 26, 2026

Uh oh!

cderv Jan 26, 2026

Uh oh!

cderv Jan 26, 2026

Uh oh!

cderv Jan 26, 2026

Uh oh!

cderv Jan 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	export const fileExists = (file: string): Verify => {
	return {
	name: `File ${file} exists`,
	verify: (_output: ExecuteOutput[]) => {
	verifyPath(file);
	return Promise.resolve();
	},
	};
	};

	export const pathDoNotExists = (path: string): Verify => {
	return {
	name: `path ${path} do not exists`,
	verify: (_output: ExecuteOutput[]) => {
	verifyNoPath(path);
	return Promise.resolve();
	},
	};
	};

Conversation

cscheid commented Jan 21, 2026

Uh oh!

posit-snyk-bot commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

cderv commented Jan 26, 2026

Uh oh!

cderv Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

cderv Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

cderv Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

cderv Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

cderv Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

posit-snyk-bot commented Jan 21, 2026 •

edited

Loading