Agentic AI underwriting systems, where one AI step pulls data, a second scores risk, and a third drafts or issues a quote with a human reviewing only the final output, fail in ways a single-model validation program was never built to catch. Traditional validation tests whether one model’s output is accurate and fair at a point in time. An agentic workflow can pass every one of those individual tests and still produce a biased or wrong final decision, because the error enters at one step and compounds as later steps act on it without re-checking it. The NAIC AI Model Bulletin requires testing for errors, bias, and unfair discrimination, but it does not spell out how to test a workflow instead of a model. This guide gives you a four-layer framework for closing that gap.
This is for the Chief Risk Officer, Chief Compliance Officer, or VP of Underwriting at a carrier piloting, or already running, an AI system that performs more than one step of underwriting or quoting with limited human review between steps. It assumes you already have, or are building, an AI Systems Program under the NAIC bulletin; the gap here is the part of that program designed for a single model that an agentic workflow outgrows.
Want to put this into practice? Use our AI incident scenario builder.
Why a single-model validation program misses an agentic workflow
A conventional underwriting model takes a defined set of inputs and produces one output: a risk score, a price, a decline. Validating it means holdout testing against historical outcomes, checking discrimination metrics across protected classes at deployment, and re-running those checks on a schedule to catch drift. That program works because the thing being validated has one input boundary and one output boundary, both stable enough to test repeatedly against the same baseline.
An agentic underwriting workflow has neither property. It might pull applicant data from three external sources, hand that data to a scoring step, pass the score and supporting documents to a drafting step that writes the quote language, and route only edge cases to a human underwriter. Each step can be a different model, a different vendor, or a different version updated on its own schedule. The workflow’s output depends on the interaction between steps, not any single step’s accuracy. Validating each step in isolation tells you each piece works. It tells you nothing about whether the sequence does.
What the NAIC bulletin actually requires, and what it leaves open
The bulletin directs insurers to test AI systems for accuracy, reliability, and unfair discrimination, proportionate to the system’s potential for consumer harm, and to document that testing as part of the AI Systems Program. That language does not distinguish between a single model and a workflow of several models acting in sequence, and it was drafted before agentic, multi-step underwriting tools were common in the market. A carrier that validates each step of an agentic workflow the way it would validate a standalone model has technically followed the letter of the bulletin while leaving the workflow-level failure mode, where individually sound steps combine into an unsound outcome, untested.
That gap does not reduce the carrier’s exposure. The unfair trade practices and unfair discrimination statutes the bulletin interprets apply to the final underwriting or pricing decision a policyholder receives, regardless of how many AI steps produced it. A market-conduct examiner asking why a protected class saw worse outcomes from an agentic workflow will not accept “each step passed its own test” as a complete answer if the workflow as a whole produced the disparity. Closing the gap is a matter of designing a testing program thorough enough to find what the bulletin’s text does not explicitly require, not waiting for a revision that spells it out.
Four failure modes unique to agentic underwriting
These are the ways an agentic workflow fails that step-by-step validation does not catch on its own.
- Compounding error. A data-retrieval step returns a slightly wrong value, a scoring step treats it as correct and produces a defensible score from bad input, and a drafting step issues a quote based on that score. No single step is wrong in isolation; the error entered upstream and nothing downstream was designed to catch it.
- Proxy drift across steps. A scoring model passes its own bias testing on the features it directly uses, but an upstream retrieval step feeds it a derived field, such as a risk indicator built from location or occupation data, that recreates a protected-class correlation the scoring model’s own testing never saw because that correlation lived in a different step.
- Autonomy creep. A workflow is validated and deployed with a human reviewing every output. Over months, the review threshold quietly narrows, either through a configuration change or because high case volume pushes more decisions into auto-approval, until the workflow is operating with less oversight than the validation that approved it assumed.
- Vendor-side step changes. When one step in the workflow is a vendor-supplied model or API, the vendor can update that step on its own release schedule. The carrier’s workflow-level validation goes stale the moment any one step changes underneath it, and nothing in a single-model validation program is designed to notice.
Traditional validation versus agentic workflow validation
| Validation dimension | Traditional single-model approach | What an agentic workflow needs instead |
|---|---|---|
| Accuracy testing | Holdout testing against historical outcomes for one model | Holdout testing per step, plus end-to-end testing of the full workflow’s final output against the same holdout set |
| Bias and discrimination testing | Discrimination metrics computed on one model’s output at deployment | Metrics computed at each step and again on the workflow’s combined output, since a disparity can appear only after steps interact |
| Stability monitoring | Periodic re-validation of one model on a fixed schedule | Re-validation triggered by any step changing, not only by a calendar date, since a vendor-side update to one step invalidates the whole chain |
| Human oversight | A defined review threshold documented at deployment | A logged, auditable record of the actual review rate over time, checked against the rate the validation assumed |
| Error propagation | Not applicable; one step, nowhere for an error to propagate to | Deliberate fault-injection testing: feed a known-flawed input to an early step and confirm a later step catches it rather than amplifying it |
A four-layer validation framework for agentic underwriting
- Per-step validation. Validate each individual model or agent the way the bulletin already expects: holdout testing, discrimination metrics, and conceptual soundness, scoped to that step’s own inputs and outputs. This layer does not change from traditional practice; it is the foundation the other three layers sit on, and skipping it is not an option just because workflow-level testing exists.
- End-to-end outcome testing. Separately from per-step testing, run the full workflow against a holdout set and measure the final decision, the quote, the decline, the price, the same way you would measure a single model’s output. Compare discrimination metrics on the workflow’s final output to the metrics on each individual step; a disparity that appears only at the end is the signature of a compounding or proxy-drift failure a per-step program would never surface.
- Fault-injection testing. Deliberately feed a known-flawed or boundary-case input into an early step, a slightly stale data value, a malformed document, an edge-case applicant profile, and confirm that a later step either corrects it or routes it to human review rather than passing it through unchanged. This is the layer that tests whether the workflow is resilient to its own errors, not just accurate when every input is clean.
- Workflow-level continuous monitoring. Monitor the workflow as a unit, not just its component models: the actual human-review rate against the rate the validation assumed, a change log of every step’s version so a vendor-side update triggers a re-validation rather than going unnoticed, and periodic re-measurement of the end-to-end discrimination metrics from layer two, not only the per-step metrics from layer one.
A carrier that can show all four layers, documented and current, for its highest-harm agentic systems has built a testing program that answers the question the bulletin’s text leaves implicit: not just whether each piece works, but whether the system does.
Where this intersects vendor oversight
Most agentic underwriting tools on the market today are vendor-supplied in whole or in part, so the bulletin’s third-party oversight expectation and this validation framework have to connect. A carrier cannot validate layers two through four of a workflow it has no visibility into, so the contract terms matter as much as the testing plan: audit rights that extend to workflow behavior, not only the vendor’s security posture; a notice-of-material-change clause covering any step the vendor controls; and enough transparency into the architecture to know how many steps exist and where human review actually sits. A vendor’s own bias-testing certification describes one step. It does not describe the workflow the carrier deployed it into.
Common mistakes
- Treating workflow validation as a documentation exercise after the fact. The four layers above work only if layers two through four are designed before deployment, not reconstructed to explain a result an examiner has already questioned.
- Assuming per-step sign-off is sufficient because the bulletin does not explicitly say otherwise. The bulletin’s silence on multi-step architecture is not a safe harbor; the unfair discrimination statutes it interprets apply to the outcome regardless of how it was produced.
- Letting the human review rate drift without a monitoring trigger. A validation built around a stated review threshold is only as good as the evidence that the threshold is still the one actually operating in production.
- Losing workflow visibility when a step is vendor-controlled. If the contract does not guarantee notice of a step-level change, the carrier’s workflow-level validation has an expiration date it cannot see coming.
Questions buyers ask
Does the NAIC AI Model Bulletin specifically require validating multi-step or agentic AI workflows?
The bulletin requires testing AI systems for errors, bias, and unfair discrimination, proportionate to potential consumer harm, but its text does not specifically address a workflow made up of several AI steps acting in sequence. That silence does not reduce a carrier’s exposure: the unfair trade practices and unfair discrimination statutes the bulletin interprets apply to the final decision a policyholder receives, however many steps produced it.
If every step in our underwriting workflow passed its own bias testing, are we covered?
Not necessarily. Per-step testing catches bias that lives inside a single model’s own features. It does not catch a disparity that only appears once steps interact, such as an upstream data step feeding a scoring step a proxy for a protected class that the scoring model never tested for directly. End-to-end testing on the workflow’s final output is what catches that failure mode.
How often should an agentic underwriting workflow be re-validated?
On a schedule proportionate to the system’s harm tier, the same principle the bulletin applies generally, but also any time a single step changes, not only on a calendar date. If a vendor updates one model in the chain, the workflow-level validation that covered the prior version no longer describes the system in production, even if the other steps are untouched.
Our agentic underwriting tool is entirely vendor-supplied. Does that change our testing obligation?
No. The bulletin is explicit that an insurer remains responsible for a vendor’s AI use, and that responsibility extends to a vendor-supplied workflow the same way it applies to a vendor-supplied single model. It does change what the contract needs to guarantee: audit rights and change-notice terms broad enough to cover every step in the chain, not just the vendor’s overall security posture.
Can DSE certify that our agentic underwriting validation program satisfies the NAIC bulletin?
No. DSE prepares programs for market-conduct examination and audit by building the four-layer validation framework, the fault-injection test plan, and the evidence an examiner expects to see. The result is audit-ready and supervisory-framework-aligned. DSE does not certify compliance or guarantee any examination outcome; that determination rests with your state regulators and your own compliance function.
The Bottom Line
An agentic underwriting workflow can pass every individual step’s validation and still produce a biased or erroneous outcome, because the failure modes that matter, compounding error, proxy drift across steps, autonomy creep, and vendor-side step changes, live in the interaction between steps, not inside any one of them. The NAIC bulletin’s testing language was not written with that architecture in mind, but the unfair discrimination statutes it interprets do not care how many AI steps produced the outcome a policyholder received. Closing the gap means running per-step validation, end-to-end outcome testing, deliberate fault-injection testing, and workflow-level continuous monitoring together, not substituting the first for the other three.
Building that four-layer framework against your own agentic underwriting systems is exactly the work our AI governance for insurers engagement runs for carriers. Start with the AI governance checklist to see where your current validation program stands, or review finserv compliance for how this fits alongside your other regulatory obligations. For the bulletin’s baseline requirements before layering on agentic-specific testing, see our NAIC AI Model Bulletin requirements guide.
Key facts
- The NAIC Model Bulletin on the Use of Artificial Intelligence Systems by Insurers, adopted December 4, 2023, requires testing AI systems for errors, bias, and unfair discrimination, but its testing language was written around a single AI system producing a point-in-time decision, not a multi-step workflow where several models or agents act in sequence (DSE, 2026).
- Holdout testing, discrimination metrics at deployment, and periodic stability checks each validate one model's output; none of them test whether an agentic underwriting workflow's intermediate steps compound into a biased or erroneous final decision even when every individual step passes validation on its own (DSE, 2026).