Security · Operating decision guide
When should a company approve an AI workflow change?
Otomat editorial · Revised
Requalify the business task when a model, prompt, source, tool or permission changes. Compare the candidate with the accepted version and keep a tested way to restore it.
Treat the workflow as the release unit
A provider’s model upgrade can alter answers even when your application code stays the same. A new retrieval source can introduce old instructions; a connector update can change what an agent can do. Record the candidate model, instructions, source snapshot, tools and permissions together. A release note that lists only the model name cannot explain which operating behavior changed.
Use the same cases for both versions
Run the current version and the candidate on a permitted, frozen set with ordinary work, exceptions and denied actions. Have the workflow owner define which failures block release. Compare accepted outputs, review time, latency and full task cost. An improvement in the average score cannot compensate for a new ability to disclose a restricted record or approve an unauthorized action.
| Check | Evidence to bring | Decision or stop condition |
|---|---|---|
| Changed component | Model, prompt, source, tool and permission versions | Re-test the affected task and security boundaries. |
| Release blockers | Named unacceptable failures and frozen comparison cases | Reject a candidate that fails a blocking case despite a better average. |
| Restoration | Runnable accepted version and irreversible-action list | Keep sensitive actions disabled until recovery is understood. |
| Post-release watch | Review sample, exception owner and stop threshold | Investigate a changed workload before declaring model deterioration. |
Practice the reversal before exposure
Confirm that the previous application and configuration remain available, and identify which external writes cannot be reversed automatically. Start with a bounded cohort or read-only comparison. Give an operator a specific stop signal and a restoration procedure. A version tag is not a rollback if the source data, credentials or schema needed by that version no longer exist.
Watch real cases after approval
Keep a small reviewed sample of production results and an exception queue. Compare it with the qualification set without treating them as identical populations. If the case mix changes, expand the evaluation before increasing reliance. Log the decision and named approvers; do not turn a successful test into permission for the next unreviewed change.
Questions before the decision
Can a vendor’s benchmark replace our acceptance test?
No. It may inform a shortlist, but it does not reproduce your sources, permissions, users, exceptions or cost of failure.
What if the provider retires the accepted model?
Track that dependency early. Qualify a replacement and a reduced-capability fallback before the retirement date; do not assume the old version will remain available.
Take the evidence sheet to the meeting
Download the blank CSV, assign an owner to each check and record the evidence. Keep completed sheets in your own approved system.
Download the evidence sheet (CSV)Use the working tools
Blank templates are ungated. Adapt them to the workflow; inputs are not submitted.
Sources and limits
These are Otomat’s proposed operating methods and original decision artifacts. External references inform risk management; they do not certify this method or establish client results. Linked sources were selected for this review; their live revision dates may change.
- US GAO: AI accountability framework and evaluation context
- OWASP: excessive agency and action permissions
For anonymous documentation and proposal patterns, see delivery evidence and its boundaries.
Related decisions
Bring a workflow to the discussion
Share the business goal and constraints through the mandate process. Do not send sensitive production data in the initial brief.
Book a Return Review