A Controlled LLM Workflow for Evidence-Grounded Diagnosis of Risk Score Differences

Introduction

Understanding why two similar transactions receive different risk scores is a common challenge in finance, insurance, and fraud detection. Traditional large language models (LLMs) can generate plausible narratives, but without a disciplined process they often produce vague or untraceable explanations. The paper A Controlled LLM Workflow for Evidence‑Grounded Diagnosis of Risk Score Differences proposes a systematic pipeline that couples external evidence retrieval, contrastive SHAP analysis, and guided LLM reasoning to produce diagnoses that are both interpretable and auditable.

1. Core Components of the Workflow

a. Reference Retrieval

The pipeline begins by pulling a set of historical transactions that closely resemble the pair under investigation. These references serve as concrete anchors, ensuring that any subsequent reasoning is tied to real, comparable cases rather than abstract speculation.

b. Contrastive SHAP Evidence

For each transaction, SHAP values are computed to quantify the contribution of individual features to the model’s risk score. By contrasting the SHAP profiles of the two target transactions, the system highlights the specific features that drive the score divergence. This contrastive evidence is concise, numerically grounded, and directly interpretable.

c. Reference‑Based LLM Reasoning

The LLM receives a prompt that includes:

• A brief description of the two target transactions.
• Summaries of the retrieved reference cases.
• The contrastive SHAP table.

The model is instructed to weave these inputs into a coherent narrative, explicitly citing the evidence and avoiding hallucinations. The prompt also enforces a deterministic response format, making downstream validation straightforward.

d. Deterministic Validation

After the LLM generates its diagnosis, a rule‑based validator checks that every claim is supported by either a reference case or a SHAP entry. Any unsupported statement triggers a regeneration loop, ensuring that the final output remains fully traceable.

2. Benefits Demonstrated in Experiments

The authors evaluated the workflow on a curated test set of transaction pairs with known score differences. Compared to a baseline LLM that received only the raw transaction data, the controlled pipeline achieved:

• Higher consistency: Repeated runs produced identical explanations, thanks to the deterministic validation step.
• Improved evidence grounding: Over 90% of factual claims could be directly linked to a reference case or a SHAP entry.
• Better traceability: Auditors could follow a clear chain from raw data → evidence → narrative, facilitating regulatory compliance.

These results suggest that structuring LLM output with external evidence and validation can bridge the gap between powerful language generation and the rigor required in high‑stakes domains.

3. Practical Takeaways for Practitioners

• Start with solid retrieval: The quality of reference cases heavily influences the final explanation. Use similarity metrics that capture both feature space proximity and domain relevance.
• Leverage contrastive explanations: SHAP is not just a diagnostic tool for single predictions; contrasting two SHAP vectors isolates the decisive factors for score differences.
• Design prompts that enforce citation: Explicitly ask the LLM to reference specific rows of the SHAP table or particular reference IDs. This reduces hallucination risk.
• Implement a validation layer: Simple rule checks (e.g., every cited feature must appear in the SHAP table) can catch unsupported statements before they reach end users.

Conclusion

By integrating retrieval, contrastive SHAP, guided LLM reasoning, and deterministic validation, the proposed workflow delivers explanations that are both human‑readable and evidence‑backed. This approach offers a practical blueprint for organizations that need transparent, auditable insights into why risk scores diverge, paving the way for more trustworthy AI‑driven decision systems.