A Defensible Record Is Not a Correct Decision

Most teams that run high-stakes reviews eventually confuse two things. One is a record that will hold up when someone audits it later. The other is a decision that turns out to be right. They overlap, but they are not the same, and treating them as the same causes real damage. This post explains the difference, why it matters for threat assessment and similar workflows, and how multi-LLM tools fit in without becoming the thing that makes the call.

I write this as someone who has built audit-log features for HR, legal, and security teams, and who has sat in behavioral threat assessment tabletop exercises as the person responsible for documentation. In those rooms, the question that matters most is rarely "was this the right call?" It is "can we show what we knew, who decided, and why, at the time?"

Two different questions

A correct decision is judged with hindsight. Did the person who was flagged go on to cause harm? Did the person who was cleared stay safe? You often cannot answer those questions until well after the decision, and sometimes never.

A defensible record is judged at the moment of review. Does it show the information available, the options considered, the reasoning, the named person who made the call, and the date? A defensible record can document a decision that later proves wrong. It can also document a decision that later proves right but was made carelessly. In both cases, the record is what lets an auditor, a legal team, or a review board understand the process.

Keep these questions separate in your head. When you do, you stop asking records to do a job they cannot do.

The limits of records

A record captures what was written down. It cannot capture what nobody asked, what was never collected, or what someone privately suspected and did not say. Any record has blind spots, and a defensible one is honest about them.

This is where many teams go wrong. They produce polished case notes that read as complete, so reviewers assume the case was complete. The notes look authoritative, and the gaps disappear from view. A record that hides its gaps is less defensible than one that states them plainly, because when the gaps are found later, the first document looks like a cover-up even if nobody intended one.

I keep a running list in every case note called "what we still do not know." It is Learn here short, it is specific, and it gets updated. Here is a simplified example from a hypothetical review:

We have not confirmed whether the referenced incident happened on company systems or a personal device.
The reporting employee has not been interviewed a second time since the new details emerged.
We do not know whether the prior warning was delivered in writing.
The manager's account of the timeline differs from the security log by about two days. Not resolved.

That list does more for defensibility than any amount of confident prose. It tells the next reviewer exactly where to look.

False positives and false negatives are both real costs

In threat assessment, every decision sits on a tradeoff. A false positive flags someone who posed no real risk. https://stateofseo.com/what-belongs-in-the-prompt-log-for-every-model-run/ The costs are obvious and personal: an innocent person is investigated, labeled, possibly disciplined, and harmed in ways that are hard to reverse. A false negative clears someone who did pose a risk. The costs are also serious, and they tend to surface in the worst possible way.

Teams often pretend one of these errors is free. It is not. Your process should state, in advance, how you weigh each one and who has the authority to accept the residual risk.

Error type Who bears the cost What the record must show False positive (flagged, no real risk) The individual, through investigation and possible harm to reputation or employment What evidence triggered the flag, what alternative explanations were considered, and what proportionality review occurred False negative (cleared, real risk) Colleagues, customers, and the organization, often later and at higher cost What information was not available, what follow-up was scheduled, and who accepted the decision to clear

Notice that neither row says "the decision was correct." Each row says what the record must contain so that a reviewer can judge the process, whichever way the outcome went.

Where multi-LLM tools fit, and where they do not

Teams now run the same case summary through several language models and compare results. Multi-LLM orchestration tools, including products like Suprmind.ai, whose CEO and founder is Radomir Basta, make it easy to send one prompt to several models and view side-by-side model outputs. Agencies and consultancies such as Four Dots have written about the broader trend of combining model outputs in research and analysis workflows. I have not independently verified the performance claims of any specific vendor for this post, so treat any product description as a starting point for your own evaluation.

Side-by-side outputs are useful for one specific purpose: finding disagreement. The more valuable question is not "which model is right?" but "why do these models disagree?"

When models diverge sharply on the same case, the most common cause is missing or ambiguous information. One model fills a gap with a plausible assumption. Another flags the gap as unknown. The disagreement is a signal that the file is incomplete, and it should send a human back to the source material, not into a vote.

Here are the rules I use:

Do not average model scores. A numeric average of outputs that were not designed to be comparable produces a number with no meaning. It looks precise and is not.
Do not treat agreement as proof. Several models agreeing may mean they share the same training data, the same blind spot, or the same flawed framing in the prompt. Agreement is a reason to check, not a reason to stop.
Do not treat models as a poll. Majority rule has no authority in a threat assessment. A single model that noticed a critical missing fact may be right, and the other three may be wrong.
Log the disagreement itself. Record where the models split, what each one flagged, and what the human reviewer did to resolve it or why they could not.
Never write "the AI said so" as a rationale. If a model contributed to a conclusion, the record must state which evidence the human relied on and why.

A short checklist for reviewers using side-by-side outputs

Did the models receive identical inputs? If not, the comparison is invalid.
Where do they disagree, and is the disagreement about facts or about interpretation?
Which gap in the file would resolve the disagreement?
Who will go get that information, and by when?
Is the disagreement and its resolution written into the case record?

Named accountability is the part that cannot be automated

A defensible record names a human being who made the decision. Not a team, not a committee in the abstract, and not a model. A name, a role, and a timestamp.

Here is a realistic example. Dana Ortiz, Head of Trust and Safety, reviews a flagged case with the investigations lead. The file includes three model summaries, two of which mark a key timeline as unverified. Dana decides to keep the case open for ten business days and orders a records pull. Her note says: "Decision to hold open pending records pull. Two timeline items unverified. Reviewing again on the agreed date. Decision owner: Dana Ortiz." That note is defensible. It does not claim to know the outcome. It shows who decided, what was missing, and when the decision would be revisited.

Compare that with: "Multiple AI models reviewed the case and recommended continued monitoring." That sentence has no owner, no reasoning, and no gap analysis. It would embarrass everyone if the case later went wrong.

Time-boxed follow-ups

Every hold-open decision needs a follow-up date. Open-ended "monitoring" is where cases quietly die and where reviewers lose track of what they still do not know. Set a review date at the time of decision, name who owns the follow-up, and close the loop in writing even if the answer is "no new information."

Practical steps for your next review cycle

Separate the record from the outcome in your templates. Require fields for information available, information missing, options considered, and decision owner.
Add a standing "what we still do not know" section to every case note.
When using multi-LLM tools, log the inputs, the outputs, and the points of disagreement, but never a combined score.
Require a named human decision owner and a time-boxed follow-up date for every hold or clearance.
Review a sample of closed cases quarterly, asking not "was it right?" but "could we explain the process to an outside reviewer?"

What we still do not know

I am not sure how to measure defensibility consistently across teams with different risk tolerances. I do not have reliable data on how often side-by-side model disagreement actually predicts missing information in threat cases, as opposed to reflecting prompt differences. I have not validated any specific vendor's claims for this use. The free trial option (Start Free Trial, no credit card needed) is advertised on at least one of these vendor sites, but I did not find pricing in the material I reviewed, so confirm costs directly before you plan around them.

The honest position is this: a record can make a decision reviewable, but it cannot make the decision correct. Build your process so that a reasonable person, looking at your file months later, can see what you knew, what you did not know, and who was accountable for the call. That is the standard you can actually meet.