Note: The job is a remote job and is open to candidates in USA. BMO is building enterprise AI platform capabilities that are safe, governed, and scalable across regulated banking environments. The Principal Engineer will design, implement, test, ship, and operate core AI platform infrastructure, including gateways, policy engines, identity fabric, registries, guardrails, and observability systems, while mentoring engineers and ensuring runtime evidence for regulatory review.
Responsibilities
- Own capabilities end to end: design, implement, test, ship, and operate production infrastructure
- Engineer for operability and defensibility from day one: instrumentation, SLOs, latency budgets, failure modes, and runtime evidence built in, not bolted on
- Build the APIs, SD'able interfaces, and integrations through which domains, DevOps pipelines, and enterprise systems consume platform capabilities
- Implement policy enforcement, guardrails, identity attestation, and audit as first-class engineering concerns: correct, performant, and provable
- Ensure every capability produces runtime evidence connecting AI activity to policy enforcement, identity, and lineage for model-risk and regulatory review (OSFI E-23, OCC)
- Assess emerging AI infrastructure, foundation-model access patterns, and standards; make deliberate, cost-aware engineering choices
- Mentor and raise the bar: set engineering standards, review designs and code, and grow depth across the team
- Partner closely with AI Developer Experience, AI Security and AI SDLC, and the Senior AI Architect in your platform build and operations
Skills
- Bachelor's degree in Computer Science, Software Engineering, or a related technical discipline
- 8+ years of software/platform engineering experience (Principal), or 5+ years (Senior), with substantial time **building and operating shared platform services** at enterprise scale
- Demonstrated experience operating **production infrastructure with real SLOs and on-call** ownership
- Strong distributed-systems and platform-engineering fundamentals: latency-sensitive request paths, resilience patterns (circuit breakers, failover), multi-tenancy, and high availability
- Cloud-native architecture across **AWS and Azure**: containers/Kubernetes, service mesh, and Infrastructure as Code (CDK, Terraform, CloudFormation/ARM)
- Robust CI/CD, GitOps, and DevSecOps practice; Git-based workflows (Bitbucket/GitHub), Jira, Confluence
- **Policy/Authorization:** Cedar, OPA/Rego, policy compilation and distribution, risk-tiered approval workflows
- **Identity/Zero-Trust:** SPIFFE/SPIRE, mTLS, token exchange, OAuth/OIDC, federated and cross-cloud identity, Entra ID/Agent ID
- **Observability/Audit:** OpenTelemetry (incl. GenAI conventions), distributed tracing, Dynatrace/Splunk or equivalents, tamper-evident/immutable audit stores, data lineage
- **Gateway/Guardrails:** API gateway internals, inline enforcement, LLM routing/abstraction, prompt-injection and PII defenses, hallucination detection, AI evaluation
- **Registry/Portal:** service catalogues, asset registries, lifecycle workflows, federation with external registries
- Working knowledge of GenAI platform patterns: LLM/AI gateways, RAG and agentic patterns, foundation models, embeddings, and guardrails — sufficient to build the infrastructure they depend on
- Familiarity with AI/ML platforms (Bedrock, Azure OpenAI, SageMaker, MLflow) and orchestration frameworks (LangChain, LlamaIndex)
- Grounding in Responsible AI, AI/data governance, privacy, cloud security, and IAM as applied to AI workloads
- Strong communication and collaboration across engineering, security, architecture, and domain teams
- A critical thinker with strong analytical and problem-solving skills
- Self-directed, comfortable with ambiguity and a fast-evolving mandate
- Able to deliver complex work under tight timelines; participates in on-call rotation for owned services
- (Master's preferred)
- Demonstrated experience operating **production infrastructure with real SLOs and on-call** ownership, ideally in a regulated industry (financial services strongly preferred)
- Strong programming skills (Python and/or Go preferred; TypeScript/Java an asset) for building performant services, APIs, and integrations
- AWS Certified Solutions Architect (Associate/Professional) / ML – Specialty
- Microsoft Certified: Azure Solutions Architect Expert / Azure AI Engineer Associate
- Security/identity certifications (relevant to Identity Fabric roles)
- Databricks / Google Cloud ML credentials; relevant GenAI/LLM credentials
Benefits
- May include a commission structure
- Performance-based incentives
- Discretionary bonuses
- Health insurance
- Tuition reimbursement
- Accident and life insurance
- Retirement savings plans
- Tools and resources to help you reach new milestones
- In-depth training and coaching
- Manager support
- Network-building opportunities
Company Overview
Company H1B Sponsorship