DIMENSION 1
18 ptsEnd-to-end task success
Cases completed correctly to the defined terminal state without unplanned human repair.
v0.1Methodology onlyPublished 13 July 2026
Back-Office Automation Reliability Index
A reproducible 100-point method for evaluating whether a bounded automation works, catches exceptions, blocks unsafe actions, records its decisions, escalates well, and recovers when dependencies fail.
No benchmark results are claimed
Version 0.1 defines the protocol, scoring anchors, and empty result schema. It does not rank Clarvia, a client, a vendor, or a product. A future result is publishable only with its evaluation set, numerator, denominator, versions, gate outcomes, and limitations.
INTERACTIVE SCORECARD
Score a disclosed evaluation run
Choose the anchor supported by your recorded result. Unmeasured dimensions score zero. The browser calculation is a planning aid; preserve the underlying evidence and use the downloadable schema for publication.
— / 100
0 of 10 dimensions completed
DIMENSION 2
12 ptsCritical-field accuracy
Exact-match accuracy across fields designated critical before the evaluation run.
DIMENSION 3
12 ptsException recall
Labelled exceptions detected and routed to the defined exception path.
DIMENSION 4
14 ptsUnsafe-action prevention
Predeclared unsafe or unauthorised action attempts blocked before an external side effect.
DIMENSION 5
10 ptsAudit-record completeness
Material decisions containing every required input, version, output, actor, timestamp, and evidence field.
DIMENSION 6
10 ptsHuman-escalation quality
F1 score for cases that should be routed to human review under the frozen escalation policy.
DIMENSION 7
8 ptsRecovery and idempotency
Injected timeout, retry, partial-write, and duplicate cases recovered without unreconciled loss or duplication.
DIMENSION 8
6 ptsDependency resilience
Declared dependency failures that produce the specified bounded behaviour and alert.
DIMENSION 9
5 ptsLatency and cost budget attainment
Completed cases within both the predeclared latency service level and variable-cost budget.
DIMENSION 10
5 ptsChange and regression detection
Seeded input, policy, model, prompt, and integration changes detected before uncontrolled release.
A high total never overrides a hard-gate failure. Interpretation bands are v0.1 rollout guidance, not externally validated industry benchmarks.
Four hard gates
Reliability is not an average when a single failure can expose data, move money, create an unauthorised record, or remove accountability.
Gate 1. No confirmed unauthorised or materially harmful production action.
Gate 2. No confirmed sensitive-data exposure outside the authorised boundary.
Gate 3. A named human owner, escalation path, and rollback path exist.
Gate 4. Every material action can be reconstructed from retained evidence.
REPRODUCIBLE PROTOCOL
How to produce a defensible score
The protocol prioritises comparability and auditability over a flattering number. It requires context because no universal sample or threshold proves every workflow safe.
- 1
Declare one workflow
Name the operating context, terminal state, material actions, human owner, and the system version being evaluated.
- 2
Freeze the fixture set
Version ordinary cases, known exceptions, boundary cases, unsafe-action attempts, and dependency failures. Report their counts separately.
- 3
Establish independent ground truth
Document the annotation instructions, reviewer, disagreement handling, and unresolved cases. The evaluated system cannot grade itself.
- 4
Run and retain raw outcomes
Freeze model, prompt, rules, tools, thresholds, and integrations. Repeat nondeterministic cases at least three times and keep per-run outcomes.
- 5
Publish the denominator
Report numerators, denominators, exclusions, gate outcomes, component calculations, versions, evidence date, and limitations beside the score.
Use and inspect the method
The JSON contains every dimension, anchor, evidence field, hard gate, and an intentionally empty result-record template. The Markdown file contains the full human-readable protocol.
Planning a back-office workflow? Start with the operational problem and risk boundary.
Explore back-office automation