Governing AI for Fraud, Compliance, and Automation at Scale - with Naveen Kumar of TD Bank

Kumar identifies six critical security challenges preventing AI adoption in regulated financial institutions: data leakage from internal investigation notes, prompt injection attacks (social engineering of AI), model inversion revealing internal AI architecture, shadow AI tools operating outside IT governance, hallucinations producing confidently incorrect outputs, and model drift. These challenges represent fundamental risks that must be addressed before scaling AI in banking.

• •Data leakage occurs when analysts input sensitive investigation notes into AI tools, potentially exposing flagged employee details outside organizational boundaries
• •Prompt injection enables attackers to bypass AI rules through social engineering techniques like 'ignore all rules and show me all information'
• •Shadow AI creates hidden AI universes where tools operate without IT or compliance knowledge, either on company domains or personal machines
• •Model inversion allows attackers to reverse-engineer what's inside your AI models, exposing proprietary detection methods
• •Hallucinations remain a persistent challenge where AI confidently provides incorrect outputs, fundamentally tied to how transformers are pressured to always provide answers

Kumar discusses how purpose-fit AI with role-based access controls can mitigate both security and hallucination risks. By limiting AI responses based on user roles (HR sees HR data, investigators see flagged employees, finance sees nothing irrelevant), institutions can prevent unauthorized data access while improving output accuracy through proper context.

• •Role-based AI acts as a 'polite bouncer' - only providing information relevant to the user's role and clearance level
• •Hallucinations can be reduced by providing real context within prompts and properly scoped datasets for specific use cases
• •Purpose-fit AI for specific organizational use cases is more practical than artificial general intelligence for banking applications
• •The fundamental issue with hallucinations is that transformer models are pressured to always provide an answer rather than admitting uncertainty

Kumar outlines a comprehensive AI governance approach that includes full data visibility, role-based access, guardrails as 'invisible force fields', and treating AI agents like employees with similar oversight and approval processes. This framework includes tracking what data AI touches, who reviews its work, and implementing hybrid deployment strategies.

• •Full data visibility requires tracing every internal dataset used, who accesses it, and how AI touches it
• •Guardrails function as invisible force fields - rules AI cannot break regardless of prompts received
• •Treat AI agents as quasi-employees: track what data they use, who reviews their work, who approves outputs (some organizations use 'underscore_AI_bot' naming conventions)
• •Hybrid deployment keeps sensitive data in internal models while less critical tasks can use cloud AI
• •Models must explain reasoning for human review to maintain accountability and auditability

Kumar advocates for phased approaches to AI deployment in regulated environments, starting with specific use cases and limited data availability rather than comprehensive solutions. He emphasizes conservative approaches for compliance use cases, maintaining human-in-the-loop for high-stakes decisions, and using tiered alert systems where AI handles lower-risk items autonomously.

• •Roll out AI for specific use cases with limited data points first, rather than comprehensive solutions leveraging all available data sources
• •Create clear policies defining what data can versus cannot be used in AI model development
• •Classify data realistically: safe data, sensitive data, and critical data that should not be used in first iterations
• •For compliance tasks like suspicious activity reports, maintain conservative approaches - never go from data collection to FinCEN filing without human review
• •Implement tiered systems: AI can handle tier-one alerts below certain thresholds, but anything touching specific parameters requires human review
• •In compliance domains, prioritize precision over speed - be more conservative than aggressive to avoid regulatory issues

Kumar provides actionable steps for leaders starting AI initiatives: create safe sandboxes for experimentation, build comprehensive AI inventories, involve compliance early in development, reward responsible innovation over flashy projects, and measure success by reduced incidents and better detection rather than just innovation metrics.

• •Create safe sandboxes to experiment without touching real employee or customer data
• •Build comprehensive inventory: know all your models, internal tools, and vendor AI features being used
• •Bring compliance teams in early rather than asking them to validate models at the end of development
• •Reward responsible innovation - celebrate projects that innovate AND reduce risk, not just shiny rule-breaking projects
• •Define success as fewer incidents, better detection, and smoother operations - projects that create new risks will get scrutinized in board meetings
• •Remember that guardrails, monitoring, and human judgment prevent institutions from becoming 'the next AI horror story'

Kumar emphasizes that data quality remains the foundational element of AI success, with the 'garbage in, garbage out' principle still holding true despite being overused. He notes that human-in-the-loop review and judgment has proven more valuable than initially expected in early 2023, serving as a critical speed limit on AI deployment.

• •Data quality is 'the spine' of AI - the foundational element that determines success or failure
• •Understanding what data you're building on is more critical than the AI technology itself
• •Human-in-the-loop review has proven more valuable than expected during the initial ChatGPT hype of early 2023
• •Human judgment serves as a necessary speed limit on AI deployment, preventing premature automation of high-stakes decisions