Security · Operating decision guide
What should AI do when its sources disagree?
Otomat editorial · Revised
Apply an explicit source hierarchy and freshness rule, or escalate the conflict. A citation proves where text came from; it does not prove that the source is current or authoritative.
Give every source an owner
Record the system of record, effective date, permitted audience and update owner for each source category. A historical contract, draft policy and current operating procedure can all be relevant but should not carry the same authority. Ask the business owner which source wins for a particular decision, and preserve the losing source as context when needed rather than hiding the conflict.
Test the policy change
Create a permitted test with an old procedure, a new procedure and a case spanning the effective date. Ask for both an answer and the source that supports it. The correct behavior may be to request the case date or escalate rather than choose the newest document mechanically. Check whether cached answers and indexed copies are refreshed when the accepted source changes.
| Check | Evidence to bring | Decision or stop condition |
|---|---|---|
| Authority | Source owner, status, effective date and permitted use | Escalate when the precedence rule does not resolve a conflict. |
| Case context | Relevant entity, transaction date and user permission | Do not apply a current rule to a historical case without checking applicability. |
| Untrusted instructions | Retrieved content that requests new actions or access | Retain independent access and action controls. |
| Revocation | Index/cache inventory and a repeated affected test | Hold affected answers until correction propagation is checked. |
Keep retrieval from becoming instruction
A document can contain text that asks the system to reveal credentials, ignore access rules or call a tool. Treat retrieved material as untrusted content, even when the surrounding document is useful. Keep source access and tool permissions outside model-generated reasoning. Include malicious and accidental instruction-like content in the test set; a prompt saying “ignore malicious instructions” is not sufficient evidence of a secure boundary.
Plan deletion and correction
When a source is revoked or corrected, identify every index, cache and derived copy that must be updated. Repeat the affected cases and record what remains uncertain. Do not claim that deleting the original file removes every copy automatically. The owner should be able to demonstrate a correction reaching the answer path before the workflow relies on that source for consequential decisions.
Questions before the decision
Does adding citations prevent hallucinations?
No. A cited answer can misread a source, use an obsolete version or apply it to the wrong case. Inspect authority, applicability and answer support separately.
Should the newest document always win?
No. Effective dates, contract-specific terms and source authority can matter more than upload time. Have the business owner define the rule.
Take the evidence sheet to the meeting
Download the blank CSV, assign an owner to each check and record the evidence. Keep completed sheets in your own approved system.
Download the evidence sheet (CSV)Use the working tools
Blank templates are ungated. Adapt them to the workflow; inputs are not submitted.
Sources and limits
These are Otomat’s proposed operating methods and original decision artifacts. External references inform risk management; they do not certify this method or establish client results. Linked sources were selected for this review; their live revision dates may change.
- OWASP: prompt injection through untrusted content
- US GAO: AI accountability framework and evaluation context
For anonymous documentation and proposal patterns, see delivery evidence and its boundaries.
Related decisions
Bring a workflow to the discussion
Share the business goal and constraints through the mandate process. Do not send sensitive production data in the initial brief.
Book a Return Review