Keeping a language model honest after launch.

The product moves even when you do not ship.

Conventional software is stable between releases. If nobody deploys, nothing changes. A feature built on a language model does not work that way. The provider updates a checkpoint, the safety layer is retuned, a rate limit is revised, and the answers your users see change without a line of your code moving.

The failure this produces is not a crash. It is a slope. Output quality declines gradually as the distribution of real prompts drifts away from the ones the feature was designed against, or as an underlying checkpoint shifts beneath it. Industry guidance on LLM observability describes exactly this: quality degradation arrives as a gradient rather than an outage, and without a baseline to compare against, teams do not notice they are on the slope until the drop is large enough for someone to complain.

Everything below exists to turn that slope into something with a number attached, so it can be seen before a client sees it.

Pinning, so that change is a decision.

Every model reference in a HaveStack system names an explicit version. Not the provider alias that quietly follows the newest checkpoint, the version. An upgrade is then something the practice chooses and tests, on a date, rather than something that arrives on a Tuesday.

Pinning the weights alone is not enough, because the model is only one of the artefacts that decides an answer. Current LLMOps practice treats prompt templates, retrieval indexes and guardrail configuration as versioned artefacts in their own right, each tied to the release that shipped it. A response that was wrong last week can then be traced to the exact combination that produced it.

• The model version is written down, never inherited from an alias
• Prompt templates are versioned with the code that sends them
• Retrieval index builds carry a build identifier
• Guardrail and safety configuration is versioned alongside
• An upgrade is tested against the standing evaluation set first
• The previous version stays reachable until the new one is accepted

Evaluation is a standing test, not a launch gate.

A fixed evaluation set is a body of representative inputs with known good outputs, held constant so that results are comparable over time. It is run before any change to a model, a prompt or a retrieval source, and the result is compared against the last accepted run rather than against an abstract idea of quality.

The distinction that matters in practice is between evaluation and monitoring. Evaluation validates a candidate before it reaches users, on data you control. Monitoring catches what evaluation could not anticipate, on live traffic. A system with only the first has no idea what real users are experiencing; a system with only the second learns about every problem from the people it affects.

HaveStack runs both, and keeps the evaluation set under version control with the system, so that a client can read it.

Cost is an operational limit, not a surprise.

Token spend is the one part of a language feature that can fail while everything else works perfectly. Usage grows, a prompt gets longer, a retrieval step starts returning more context, and the monthly figure moves without any single change looking significant.

Every system HaveStack maintains logs input and output token counts per call, attributes them to a feature, and carries a monthly ceiling agreed in the service agreement. Crossing an agreed fraction of that ceiling raises an alert to a named contact rather than an invoice at the end of the month.

What happens when the provider is having a bad day.

Hosted models degrade in ways that are not outages. Latency triples, a region rate limits, a safety filter starts refusing a category of legitimate request. A system with a single hard coded provider passes all of that straight through to the user.

Fallback routing means the system has a defined second path and a rule for taking it: a latency threshold, an error rate, a refusal pattern. The fallback is exercised on a schedule, not assumed, because a route nobody has ever taken is a guess rather than a control.

Retrieval that is current, or honest about not being.

Where a feature answers from an organisation’s own documents, the answer is only as good as the last index build. A retrieval corpus that has not been rebuilt since a policy changed will answer confidently from the old policy, and nothing in the output will indicate that it is out of date.

Re indexing runs on a schedule written into the agreement, the build carries a timestamp, and the system can report when it last ran. Where currency genuinely matters, that timestamp is surfaced to the user rather than kept in a log.

Written down, or it does not happen.

Maintenance that is not written down is a favour, and a favour stops the moment somebody is busy. These are the clauses that make the practice above enforceable.

Terms used here.

Evaluation set

A fixed body of representative inputs with known good outputs, held constant so results are comparable between runs.

Drift

Movement in the distribution of real inputs, or in the model beneath them, that changes output quality without any code change.

Token ceiling

An agreed monthly limit on spend, with an alert threshold below it, so cost is bounded rather than observed.

Fallback routing

A defined second provider and the measured conditions under which traffic moves to it.

Re indexing

Rebuilding a retrieval corpus so that answers reflect current source documents rather than the state at launch.

References.

Published guidance this page draws on. HaveStack does not claim authorship of the practice described; it claims to follow it.

1. 01
   LLM observability: a complete guide to monitoring AI applications. Openlayer.
2. 02
   The complete guide to LLM observability. Portkey.
3. 03
   LLMOps guide: building fast, cost effective LLM applications. Redis.
4. 04
   The roadmap for mastering LLMOps. Machine Learning Mastery.

The rest of what stays under management.