WIP: Fragments

[simonw]

Refs:

• -f/--fragment option for passing in longer context fragments, stored in a deduped table #617

TODO:

• Make aliases such as llm prompt -f alias work
• Document how to use -f option
• llm fragments list needs non-JSON output (that truncates) - this should work similar to the new llm logs --short mode.
• llm fragments -q one -q two
• Make -f fragment-hash work in addition to -f fragment-alias
• Documentation for the llm fragments family of commands
• Make sure -f $URL follows up to three redirects
• Fix the problem where prompt_json still duplicates the prompt - I can use https://pypi.org/project/condense-json/ for that
• Update llm logs to correctly display prompts using this new mechanism
  • Consider hiding fragments in a <details><summary> by default in log output
  • ... or maybe just show fragment IDs but not fragments? Would make llm logs output easier to visually scan. Could have a -e/--expand option to expand those fragments.
  • Tempted to add magic that says if the source ended in .py or .js then it gets wrapped in a syntax highlight block with the correct language tag - bit of a weird feature, maybe have this turned on by llm logs --syntax or similar? Might be easier to ignore programming languages but still wrap in triple backticks if it looks like code - if most of the lines are less than 120 chars for example.
• llm fragments show hash-or-alias - to show the full fragment based on the hash from the previous command
• Comprehensive tests

Stretch goals

• Add this to llm chat - so you can do llm chat -f docs
• Does this need documentation in the Python docs?
• Consider if this should be added to templates (maybe not?)
• Consider how this design might be used to implement prompt caching in Claude

[simonw]

Here's a but of a weird problem. We're storing fragments in the logs.db database - but that means this logic here isn't quite right:

llm/llm/cli.py

Lines 476 to 482 in 9b2d4d6

	# Log to the database
	if (logs_on() or log) and not no_log and not async_:
	log_path = logs_db_path()
	(log_path.parent).mkdir(parents=True, exist_ok=True)
	db = sqlite_utils.Database(log_path)
	migrate(db)
	response.log_to_db(db)

Here we are only even creating that logs.db database at the very end of prompt() if the user has logging enabled.

But now we need that database at the start of that CLI command in case we need to resolve any fragment aliases using it.

It's a bit weird that it's called logs.db now that it's supporting an additional feature that's not related to if logs are turned on or not. I think I can live with that though.

[simonw]

Lots more fragments available now - I added Datasette, sqlite-utils, s3-credentials and shot-scraper docs to https://github.com/simonw/llm-docs

[simonw]

Docs: https://github.com/simonw/llm/blob/30331e9a4b24d2f871d71e27df214a4b8233308d/docs/usage.md#fragments

[simonw]

Claude idea: since fragments are visible on the Prompt class the Anthropic plugin could add a -o cache 1 option which adds a cache marker directly after the fragments but before the next part of the prompt.

This is the simplest thing that could possibly work, but I think it would be worthwhile.