Evidence Over Signaling: A Rule for AI-Assisted Technical Work
When an AI agent suggests adding something 'for credibility,' ask one question: does it demonstrate the property, or does it signal it? The two are not the same. Most AI-generated technical artifacts fail this test silently.
The Rule
When an AI agent recommends adding something to a technical artifact — a diagram, an equation, a chart, an architectural overlay — ask one question before accepting it:
What does this actually do?
If the answer is "signals rigor," "implies the analysis was done," or "suggests credibility" — reject it and find a replacement that demonstrates the underlying property directly.
This is the difference between evidence (a measurement) and signaling (a decoration of measurement). One earns trust. The other borrows it.
How Signaling Sneaks Into AI Output
Modern AI agents are trained on the surface forms of expert work. They have absorbed which diagrams appear in physics presentations, which charts appear in ML papers, which boxes appear in architecture documents. When asked to "make this more rigorous," they will suggest adding those surface forms — because surface forms are what their training data tagged as "rigorous output."
This is well-meaning. It's also wrong.
A Penrose diagram in a presentation that didn't perform causal-structure analysis is a rigor signal, not rigor itself. A confusion matrix added to a classifier report where accuracy was the actual claim is a rigor signal. A 30-box architecture diagram where 5 boxes explain the system is a rigor signal.
The agent isn't lying. It's doing what training data taught it: adding the appearance of rigor when asked to make something rigorous-looking. The category error is in the request, not the response.
The fix is the question: what does this actually do?
The Tell: "Even If the Audience Can't Read It..."
The most reliable signal that an addition is decoration, not evidence, is the phrase that justifies it:
"Even if the audience can't read it, its presence implies the analysis was done."
That sentence shape is the tell. It admits that the artifact functions as a signal, not as proof. Real evidence does not need to be read to do its work — but it also doesn't substitute for the work. A measurement that conveys information through its visual presence alone is acting as a credibility prop.
When you see that justification — from an AI agent, a colleague, a slide template — pause. The artifact is performing signaling, and there is almost always a better replacement that does the actual job.
A Better Replacement Exists Almost Always
For every signaling artifact, there is usually a corresponding demonstrating artifact that uses data you already have:
| Signaling Addition | Replaced By Evidence Artifact |
|---|---|
| Static diagram implying causal analysis was done | Live numerical readout of the causal quantity, updated in real time |
| Confusion matrix decorating an accuracy claim | The disaggregated per-class accuracies, with confidence intervals |
| 30-box architecture diagram | A 5-box diagram + a live system-health dashboard |
| Stack of equations no one will read | One equation + the measured value it predicts vs the value observed |
| "Verified by senior reviewer" badge | A linked diff showing the reviewer's exact line-by-line approvals |
In every case, the second column does what the first column claims to do. The second column also takes less space, requires the original analysis to exist, and rewards close inspection rather than discouraging it.
Why This Matters More When AI Is in the Pipeline
If a human spends an hour adding a Penrose diagram to their slide deck, they have at least thought about whether it belongs there. They might still be wrong, but the artifact has passed one filter.
When an AI agent inserts an equivalent addition, it has passed zero filters. The agent's training data said "expert decks have Penrose diagrams," and so a Penrose diagram appeared. The decision to add it was statistical, not analytical.
This is fine when you have a process to catch it. The process is the question: what does this actually do?
Without that filter, AI-assisted work converges to plausible-looking output. With it, AI-assisted work converges to actually demonstrated output. The difference compounds across every artifact you ship.
A Concrete Example: The Live Measurement Pattern
A common challenge in technical demonstrations is communicating that a numerical method is correct. The signaling approach: add a static plot showing that the method converged at some point. The demonstrating approach: render a live HUD next to the running system, showing the conservation quantity in real time, with the actual drift value updating as the system evolves.
The signal version says: we measured this once.
The evidence version says: watch us measure it right now, while you're looking.
For an audience that includes domain experts, the second version is overwhelmingly more convincing. For an audience of non-experts, the second version is also more convincing — because watching a number stay stable while a system moves is intuitively meaningful. The bar that doesn't shift is the universal language of "conservation."
This pattern generalizes. Anywhere you would insert a "we did the analysis" artifact, ask whether a live measurement using the analysis data would land better. It almost always does.
The Discipline Compounds
The first time you apply the rule, you remove one piece of decoration from one artifact. Modest.
The tenth time you apply it, your default has shifted. You start writing demonstrating artifacts in the first draft, because asking the question early is faster than removing the signal later.
The hundredth time, your AI agents adapt to you. The patterns you accept and the patterns you reject become part of how they generate output in your context. The agent that suggested a Penrose diagram in iteration 1 suggests a live conservation readout in iteration 100, because that is the pattern your previous corrections have taught it.
This is the slow compounding of rigor as a working practice. It is not glamorous. It produces no flashy diagrams. The output reads as "obvious" because every claim is backed by something inspectable. That obviousness is the goal.
Summary
| Stage | Question | Outcome |
|---|---|---|
| Before accepting an addition | What does this actually do? | Filter applied |
| If answer is "signals rigor" | What measurement could replace it? | Replacement found |
| If no replacement exists | Was the analysis actually done? | Honest acknowledgment |
| Across many iterations | Default shifts to evidence-first | Discipline compounds |
The rule fits on one line: evidence over signaling, always.
The discipline is asking the question before the artifact ships.
This post is part of an ongoing series on engineering discipline for AI-assisted work. Other posts cover geometric routing, phi-scored memory persistence, and why I maintain products at 55 to 95 percent.