Estimating the tokens (cost) of household LLM queries

[MaxGhenis]

We need to estimate the tokens of household LLM queries for understanding our costs of this service.

@anth-volk shared this sample. request for the household AI explanation, in this case for SNAP. This includes three levels of the computation tree.

I ran this through https://platform.openai.com/tokenizer, which counted 7,801 tokens.

I then ran this through various LLMs and again through the OpenAI tokenizer to count output tokens:

• Claude 3.5 Sonnet: 265
• GPT-4o: 564 tokens
• Deepseek R1: 414
• Gemini 2.0 Advanced (1.5 provided an unacceptably poor response): 974

To be conservative, let's price a prompt with 10,000 input tokens and 1,000 output tokens. Based on https://docsbot.ai/tools/gpt-openai-api-pricing-calculator (which looks accurate from my checks), here's how it'd cost by model in cents:

• Claude 3.5 Sonnet: 4.2
• GPT-4o: 3.3
• Deepseek R1: 0.73
• Gemini 1.5 Pro (2.0 Pro not available by API): 1.65