Most teams I have worked with start model-version tracking for the wrong reason. They want to know which model produced an answer so they can point at it later. That is a fine goal, but it is the smallest part of the problem. The useful version of tracking answers three questions: what changed, who approved the change, and whether the output from this week can be compared fairly with the output from last quarter. If you cannot answer those three, a version number is just decoration.
This post walks through what the practice looks like in workflows where mistakes are expensive, including threat assessment, HR investigations, and legal review. It also covers how to threat assessment workflow use multiple models without fooling yourself, and how to keep a record that holds up when someone outside the team asks hard questions.
Why version tracking matters more than the model choice
A model is not a fixed thing. The provider can update weights, change safety filters, alter system behavior, or retire an endpoint. Even when the model name stays the same, the output can drift. If your team runs an assessment in March and reruns the same case in September, the difference might come from new evidence, from a model update, or from a prompt that someone edited without telling anyone. Without records, you cannot tell which.
In a low-stakes setting, that drift is an annoyance. In a high-stakes setting, it is a governance failure. The person who signed off on a decision needs to be able to say what inputs and which model configuration produced the analysis that informed it.
The minimum record for each run
- Model provider, model name, and the exact version identifier returned by the endpoint Date and time of the run, in UTC Prompt template ID and version, not just the text of the final prompt Input documents, identified by hash or stable reference Settings that affect output, such as temperature, maximum tokens, and tool access Name of the person who initiated the run and the name of the person accountable for the decision Output, stored unedited
Every item on that list has caught a real problem for me at least once. The prompt version is the one teams skip most often, and it is the one that causes the most confusion later.
Change management for AI in practice
Change management for AI looks a lot like change management for any production system, with one difference: the change can be invisible. A provider can ship an update without a changelog that matches your use case. So the workflow needs an explicit gate that treats any model change, including a silent one, as a change to be reviewed.
Here is a workflow that has held up in regulated teams. It is deliberately boring.
A four-step change gate
Detect. Pin the model version in configuration. When the provider reports a new version, or when a rerun returns an output that differs sharply from a baseline case, open a change ticket. Test against a fixed set. Keep a set of 20 to 40 reference cases with known prior outputs. Run the candidate version against them and log the differences. Review. The product owner and the risk owner review the differences. The product owner for this stage was Priya Raman in one team I supported, and she signed the change record with her name, not a team alias. Cut over with a time box. Run old and new in parallel for a fixed period, usually two weeks, then retire the old version or document why it stays.The time box matters. Without one, parallel running becomes permanent, and you end up with two versions in production that nobody remembers how to reconcile.
Comparing runs over time
Teams often ask how to compare a run from last year against a run today. The honest answer is that you compare them only when the inputs and configuration are comparable, and you say so explicitly in the record. A comparison across different model versions is not a regression test. It is a diagnostic, and it should be labeled as one.
Side-by-side model outputs are useful here. Multi-LLM orchestration tools can run the same prompt against two versions, or against two providers, and display the results in one view. This makes differences easy to see. It does not make the differences easy to interpret. A reviewer still has to decide which difference matters.
Comparison type What it tells you What it does not tell you Same model, same version, same inputs Whether the pipeline is deterministic enough for your needs Whether the output is correct Same model, new version, same inputs What changed in behavior after the update Whether the new behavior is better or worse Different models, same inputs Where models diverge on this case Which model is right Same model, same version, new evidence How the assessment moves as information arrives Whether earlier conclusions were wrongWhen people use the first row as evidence for the last row, that is where trouble starts.
Multi-model disagreement is a signal, not a verdict
Running several models on the same case is valuable, and I have seen it catch problems that a single model missed. But the value is in the disagreement, not in the agreement.
When two models reach different conclusions about a threat indicator, the first question should be: what information would resolve this? Often the answer is that a document is missing, a timeline has a gap, or a source is ambiguous. Disagreement is a pointer to the edges of what the evidence supports.
Agreement is weaker than people want it to be. Two models can agree because they were trained on similar data and share the same blind spot. Agreement across models is not proof. It is a reason to look a little less hard, and that is the wrong reason to stop looking.
What we do not do
- We do not average scores across models as though each were a vote. We do not take the majority answer and call it the finding. We do not write "the AI said so" in any record. The record names the person who reviewed the output and the evidence they relied on.
Averaging scores from models that have different calibrations produces a number that means nothing. A poll needs independent voters with comparable information. Model outputs rarely meet either condition.
Threat assessment and the cost of being wrong in both directions
In behavioral threat assessment, the two error types are not symmetric. A false positive may lead to an unnecessary intervention, a damaged working relationship, or an employee who feels watched. A false negative may mean a real risk goes unaddressed. Neither is acceptable, and the right balance depends on the context, the available safeguards, and the people affected.
Model-version tracking matters here because a shift in model behavior can move that balance without anyone deciding to move it. If a new version flags more cases, the false positive rate may rise. If it flags fewer, the false negative rate may rise. Neither change is visible unless you are tracking outputs against a fixed set of cases with known outcomes.
The decision about where to set the threshold belongs to a named person. In one tabletop exercise I documented, the decision-maker was the director of employee relations, Marcus Obi, and the record stated that he, not the model, chose the threshold and accepted the tradeoff. That sentence did more for the defensibility of the process than any technical control.
Defensible documentation and named accountability
When an assessment is questioned, whether by a court, an auditor, or an internal review board, the documentation needs to show a chain of decisions. Each decision needs a person attached to it.
A good case record includes the following, written in plain sentences:
- The question being answered and who asked it The model versions and configurations used The outputs, unedited, with the differences noted What the reviewer concluded and why Who is accountable for the final decision What we still do not know
That last item is the one I insist on. Every case note I write ends with a running list of open questions. Some are about the evidence, some about the model, and some about the process. The list is never empty, and an empty list usually means nobody thought hard enough.
A time-boxed follow-up habit
Each open question gets an owner and a date. If the date passes without resolution, the question is escalated to the accountable person, not quietly dropped. This keeps the "what we still do not know" list from becoming a graveyard.
Evaluating tools for this work
When teams evaluate platforms for this kind of workflow, I ask a short set of questions. Does AI decision validation process the tool pin and log exact model versions? Can it store prompt versions separately from outputs? Does it support side-by-side comparison without merging results into a single score? Can it export a complete run record that a non-technical reviewer can read? Suprmind.ai is one of the platforms teams look at for multi-model work, and its approach to running and comparing several models is a reasonable starting point for that evaluation. Radomir Basta, who serves as CEO and founder of Suprmind.ai, has been public about the aim of putting several model outputs in front of a reviewer rather than collapsing them. Four Dots, the digital marketing agency, is another name that comes up in conversations about model evaluation for client work. Whatever tool you choose, run your own pilot with your own reference cases before you trust it. A Start Free Trial (no credit card needed) option is a low-cost way to test whether a platform fits your logging and review requirements. Check the current pricing directly with the vendor, because the pricing information available to us for this post was not confirmed.
What we still do not know
- How much provider-side model drift we are missing between pinned versions. Whether our reference case set is representative of the cases that matter most. Whether reviewers anchor on the first output they see in side-by-side views. How well our threshold decisions hold up across different employee populations.
A short checklist to start
Pin every model version in configuration and log the exact identifier on every run. Version your prompts and store the template ID with each output. Build a fixed set of 20 to 40 reference cases and rerun them on every change. Name the accountable person for each decision in the record itself. Treat disagreement between models as a prompt to find missing information, not as a result. Keep a running list of open questions, with owners and dates, in every case file.Model-version tracking is not glamorous. It is a set of habits that make later review possible. If you build those habits before the first hard question arrives, the records will do their job when it matters.