cachellm

Prompt caching makes a repeated prompt prefix far cheaper, but every provider does it differently. Anthropic wants explicit cache breakpoints in the request. OpenAI caches a prefix automatically, as long as the stable part comes first. Gemini wants a cached content object created ahead of time and referenced later.

Doing this by hand means provider specific code at every call site, plus a guess about which parts of a prompt are stable enough to be worth caching.

For Anthropic and OpenAI you pass in the client you already have and get the same client back. For Gemini you pass the client and its cache manager and get back a model wrapper. Calls go through unchanged, except that the prompt is prepared for caching on the way out. There is a TypeScript package on npm and a Python port on PyPI.

The library has no runtime dependencies and runs on Node 18 or Python 3.9 and later. Token counts come from a characters per token heuristic by default, which is the main approximation in the design; you can pass your own counter for prompt text when you need exact numbers.

The token counts behind the savings it reports are not estimates: they come from the usage figures each provider returns. The dollar figure applies a built in price table to those counts.