RAG Knowledge Base — RAI 9 Categories & LLMOps Deep Dive

State Street RAG project mapped to Standard Chartered RAI framework and LLMOps production stack Use this file for SCB CIB AI Analyst interview — deep talking points on RAI + LLMOps

1. SCB RAI Framework — 9 Categories Mapped to RAG Project

Standard Chartered’s Responsible AI framework defines 9 categories that every AI system must satisfy. Below: what each category means, what we did, and what you can say in the interview.

1.1 Accountable (可问责)

What it means: someone is clearly responsible for the AI system’s behavior. There’s a named owner, a decision log, and a review process.

What we did:

• PM was the named model owner (not the data scientist)
• Quarterly model risk review with Risk team
• Every design decision documented: why Azure AI Search over Pinecone, why GPT-5.4 mini over flagship, why semantic chunking over fixed window

Interview talking point:

“I owned the benefit case and the risk case. Not the model weights — but the decisions about what to build, why, and what risks we accepted. Every design choice had a documented rationale that we could revisit in quarterly reviews.”

1.2 Fair (公平)

What it means: the AI system doesn’t systematically favor or disadvantage any group. In banking, this includes customer segments, geographies, and business units.

What we did:

• Checked if retrieval results favored one client’s documents over another (no systematic bias found)
• Ran bias audit on the intent classifier: no client segment was systematically misrouted
• Monitored adoption rate by team — ensured no team was left behind

Interview talking point:

“In our case, fairness meant making sure the system worked equally well for all clients and all teams. We checked if the intent classifier routed queries from Client A’s documents differently than Client B’s — it didn’t. We also tracked adoption by team to catch any access gaps.”

1.3 Explainable (可解释)

What it means: every AI output has a clear reason. Users and auditors can understand why the system gave a particular answer.

What we did:

• Citation: every answer paragraph tagged with source document name, section title, last updated time
• Confidence score: Reranker average score shown to user (green / yellow / red)
• One-click traceability: click any citation to jump to the original SharePoint document
• No black-box answers — if the system can’t explain why, it says “cannot confirm”

Interview talking point:

“Explainability in RAG is actually more natural than in pure ML. Every answer comes with a citation — document name, section, timestamp. The user can verify in one click. Plus the confidence indicator tells them upfront how much to trust the answer. That’s XAI built into the UX, not bolted on after.”

1.4 Private (隐私)

What it means: personal and sensitive data is protected. Data doesn’t leave authorized boundaries.

What we did:

• Azure tenant deployment: all data stays within the bank’s Azure infrastructure
• No external API calls: rejected OpenAI API because data would leave the tenant. Used Azure OpenAI instead.
• Azure AD security group permissions: each document indexed with an AD group ID. At query time, the system reads the user’s AD group memberships and filters out unauthorized documents. Same permission model as SharePoint, enforced at the retrieval layer.

Key term — Azure AD security group: Azure Active Directory security groups define who can access what. When a user logs in, their group memberships are known. We tag every indexed document with the group ID that has access. At search time, the system filters results to only show documents the user’s groups can see.

Interview talking point:

“Privacy was a hard requirement, not a nice-to-have. We rejected Pinecone and OpenAI API specifically because data would leave our infrastructure. Azure OpenAI keeps everything in the tenant. For access control, Azure AD security groups are enforced at the retrieval layer — same model as SharePoint, just applied earlier in the pipeline.”

1.5 Secure (安全)

What it means: the AI system is protected against unauthorized access, tampering, and adversarial attacks.

What we did:

• SharePoint connector uses existing Azure AD authentication — no new attack surface
• API endpoints behind corporate firewall, no public exposure
• Penetration test passed before launch
• Prompt injection protection: input sanitization layer strips potential injection patterns before query processing

Interview talking point:

“We didn’t build a new access layer. We piggybacked on SharePoint’s existing security model. The connector uses Azure AD, so if someone doesn’t have SharePoint access, they can’t see the document through RAG either. We also ran a pen test before launch — passed clean.”

1.6 Transparent (透明)

What it means: users understand the system’s capabilities and limitations. The system doesn’t pretend to know things it doesn’t.

What we did:

• Confidence indicator: green (high) / yellow (verify) / red (cannot confirm) — always visible
• Refusal mechanism: prompt explicitly instructs “if not mentioned in documents, say you cannot confirm.” No guessing.
• Beta label: during trial phase, every answer tagged “trial — please verify.” Users knew the system was still learning.
• Daily knowledge cards: pushed to users based on high-frequency searches — helped users understand what the system knows and doesn’t know

Interview talking point:

“Transparency is about managing expectations. The confidence indicator tells users upfront how much to trust each answer. The refusal mechanism means the system says ‘I don’t know’ instead of making something up. In finance, a confident wrong answer is worse than an honest ‘I don’t know’.“

1.7 Auditable (可审计)

What it means: the system’s behavior can be reviewed after the fact. Regulators and internal auditors can trace what happened and why.

What we did:

• Full query log retained (query, retrieved documents, generated answer, user feedback)
• Every answer traceable to source document version and timestamp
• Evaluation results archived quarterly
• Bad case log with root cause attribution (retrieval failure / hallucination / document gap / routing error)

Interview talking point:

“Every answer is traceable end-to-end: what the user asked, which documents were retrieved, what the LLM generated, and whether the user found it useful. If a regulator asks ‘why did the system give this answer on this date?’, we can reconstruct the full chain in minutes.”

1.8 Controllable (可控)

What it means: humans can override the AI system. There’s a kill switch and a human-in-the-loop for edge cases.

What we did:

• Human-in-the-loop: confidence score < 0.6 → answer shows “cannot confirm” + “escalate to human” option
• Kill switch: system can be set to search-only mode (no generation) at any time without downtime
• Gray-scale rollout: started with search-only, added generation gradually, could roll back instantly

Interview talking point:

“Controllability means two things: humans can override any single answer, and we can dial back the whole system if needed. Low-confidence outputs go to human review — that’s not optional, it’s a hard rule. And the system has a ‘search-only’ mode we can switch to in seconds if we lose confidence in the generation layer.”

1.9 Sustainable (可持续)

What it means: the AI system is resource-efficient. Environmental and cost impact are considered.

What we did:

• GPT-5.4 mini over flagship: 60% lower compute cost, 50% lower latency — same generation quality for RAG summarization tasks
• Caching: 35% of queries served from cache, zero LLM calls
• Context compression: dynamic truncation saved 40% tokens per query
• Net effect: ~70% lower compute footprint vs. naive RAG (full docs + flagship model)

Interview talking point:

“Sustainability isn’t just about green — it’s about not wasting resources. We chose a smaller model because RAG already provides the context. We cached frequent queries. We compressed the context window. The net result is about 70% less compute than a naive implementation.”

2. LLMOps Stack — What We Built vs. What SCB Would Need

LLMOps = the operational practices that keep an LLM system reliable, accurate, and cost-controlled in production. It’s the difference between a demo and a production system.

2.1 Four Pillars of LLMOps

2.2 LLMOps Lifecycle — RAG Specific

Prompt change → Regression test (auto) → Metrics pass? → Deploy
                                              ↓ No
                                        Block + alert PM
                                              ↓ Fix
                                        Re-run regression

Key stages:

1. Development: PM writes new prompt or changes retrieval parameters. Committed to version control.
2. Regression gate: automated evaluation runs on 500-pair test set. Compares Recall@5, Precision@5, Faithfulness, Answer Relevance against baselines.
3. Deploy decision: all metrics within tolerance → auto-deploy. Any metric drops > 2pp → blocked, PM notified.
4. Production monitoring: post-deploy, dashboard tracks real-time metrics. Drift alerts if query patterns shift.
5. Feedback loop: weekly human audit of 50 queries. Bad cases attributed and fed back to development.

2.3 Monitoring — What to Watch

Key term — Data drift: user query patterns change over time. New terms appear, old systems get replaced. The intent classifier and embedding model degrade. We caught drift through weekly human audits and retrained the classifier monthly.

2.4 Cost Governance — Token Math

Cost reduction levers:

1. GPT-5.4 mini instead of flagship: -60% cost per token
2. Cache (35% hit rate): -35% queries (zero LLM cost)
3. Context compression: -40% tokens per query
4. Combined effect: ~80% lower cost vs. naive implementation (flagship model + full context + no cache)

3. Interview Q&A — RAI + LLMOps

Q1: “How do you ensure Responsible AI in a regulated bank?”

“I think about RAI in four dimensions that I’ve practiced:

Data privacy — private deployment, data never leaves the bank’s infrastructure. I rejected two vendors (Pinecone, OpenAI API) specifically for this reason.

Fairness — check if the model treats different client segments differently. We ran bias audits on the intent classifier quarterly.

Explainability — every answer has a citation: document name, section, timestamp. Confidence score visible to user. No black-box decisions.

Human in the loop — low-confidence outputs go to a human. Not optional. Hard rule.

SCB’s RAI framework has 9 categories. I was already doing most of these — accountable, fair, explainable, private, secure, transparent, auditable, controllable, sustainable. The mapping is natural.”

Q2: “What is LLMOps? Why does it matter for CIB?”

“LLMOps is how you keep an LLM system healthy in production. Four things:

Version control — track every prompt version, model version, evaluation set version. Without this, you can’t reproduce a past result or know what changed.

Monitoring — watch for performance degradation. Faithfulness dropping? Adoption declining? Latency spiking? You need dashboards and alerts.

Automated regression testing — every time you change a prompt, auto-run the evaluation set. If metrics drop, block the deployment.

Cost governance — track token usage. Use caching, context compression, and small-model routing. We cut costs 80% vs. naive implementation.

As a PM, I don’t set up these tools. But I need to know what should exist and what breaks when something is missing. That’s the PM’s job in LLMOps.”

Q3: “How do you handle model risk in a RAG system?”

“Model risk in RAG is different from traditional ML. The risks are:

Retrieval risk — wrong documents retrieved. Mitigated by hybrid search (vector + BM25), Reranker, and query rewrite.

Generation risk — LLM hallucinates despite good retrieval. Mitigated by citation, confidence scores, refusal mechanism, and Faithfulness monitoring.

Drift risk — user language changes over time, intent classifier degrades. Mitigated by weekly human audit and monthly retraining.

Cost risk — token usage spirals. Mitigated by caching, compression, and small-model routing.

I’d present these to the Risk team with a 4-slide deck: risk inventory → mitigation controls → monitoring plan → escalation process. Quarterly review cycle.”

Last Updated: 2026-06-08