Note: The job is a remote job and is open to candidates in USA. The client is a growing technology consultancy helping banks build and deploy secure, production-grade AI solutions. They are seeking a Lead AI/ML Engineer to build and operate production LLM applications, RAG and retrieval systems, model-serving infrastructure, tool orchestration, evaluation frameworks, and security guardrails for regulated banking environments.
Responsibilities
- Design and build production-grade RAG and retrieval systems that provide verifiable answers with supporting queries, sources, and citations
- Develop retrieval and verification workflows across structured and unstructured banking data
- Deploy and operate self-hosted open-weight LLMs and embedding models, including within individual bank environments
- Build and evolve the organization's standardized RAG architecture and patterns
- Design and implement an MCP/tooling layer that allows AI systems to securely interact with governed data, metrics, documents, and customer information
- Build tool/function-calling workflows with strong auditing, access controls, and clear boundaries around what AI systems can access
- Develop and own an AI evaluation framework covering golden-question regression testing, retrieval quality, groundedness, answer accuracy, and appropriate "I don't know" behavior
- Make evaluation a formal release gate for production AI systems
- Implement LLM guardrails including PII redaction, prompt-injection defenses, context controls, and query/row limits
- Partner with data engineering to ensure AI systems use governed semantic-layer metrics rather than generating unreliable ad hoc SQL
- Work with platform and security engineering to deploy reliable AI infrastructure in controlled environments
- Monitor and optimize latency, inference cost, retrieval quality, and model performance across different models and deployment configurations
- Document model architecture, evaluation methodology, security controls, and guardrails for customer security reviews and audits
- Provide technical leadership while remaining deeply hands-on with implementation and production troubleshooting
Skills
- • 6+ years of software/ML engineering experience, with at least 2–3 years building and shipping production LLM, RAG, NLP, or generative AI systems
- • Proven experience taking AI systems from prototype to production and real users
- • Strong Python skills and solid software engineering fundamentals
- • Hands-on experience building production RAG/retrieval pipelines
- • Strong understanding of LLM evaluation, including groundedness, retrieval quality, regression testing, and hallucination reduction
- • Experience implementing AI guardrails and security controls
- • Experience deploying or operating self-hosted/open-weight models
- • Ability to reason about the trade-offs between model quality, latency, infrastructure, and cost
- • Strong understanding of prompt engineering, tool calling, structured outputs, and LLM application architecture
- • A strong engineering mindset around accuracy and verifiability, rather than simply building compelling AI demos
- • Comfortable working cross-functionally with data, platform, security, and product teams
- • Experience building AI systems in banking, financial services, healthcare, cybersecurity, or another high-stakes/regulated environment
- • Experience deploying AI systems in on-premises or highly controlled environments
- • Experience with fine-tuning, adapters, or retrieval-quality optimization
- • Familiarity with banking, financial services, or financial data
- • Experience with AI security, governance, or compliance frameworks
- • Experience working with GPUs, CUDA, and production inference infrastructure
Benefits
- Your work will be deployed into production and used by banking professionals.
- You'll have meaningful influence over RAG, model serving, evaluation, tooling, and guardrails.
- Build and operate LLM infrastructure where security, privacy, and deployment constraints make traditional cloud-only approaches impractical.
- Focus on making AI systems accurate, grounded, observable, and resistant to failure.
- Work at the intersection of LLM engineering, structured data, semantic layers, and intelligent retrieval.
- Establish engineering standards and best practices for production AI systems while remaining hands-on.
Company Overview