Experience: 6+ years in software engineering, including 2+ year building LLM or agent solutions
The role
- You build, evaluate and deploy agentic solutions from an approved spec, using coding agents for most code, tests and documentation.
- You direct the agents, check their output and own the result.
- You deliver working software that is instrumented, evaluated, and ready for the business to accept.
What you will do
- Size stories with the product owner, flag effort and risks, and confirm data and access early.
- Test the spec for buildability, prepare the repo and agent context, and record open questions.
- Slice stories into tasks, estimate in story points and commit the sprint with the POD.
- Build with coding agents: code, unit and integration tests, and documentation.
- Implement agents on Copilot Studio, Azure AI Foundry Agents or Microsoft Agent Framework, including prompts, tools, connectors, workflows and integrations.
- Build the data and retrieval layer: Document extraction pipelines with Azure AI Document Intelligence, including validation and confidence handling.
- Chunking, embedding, indexing and hybrid search in a vector store such as Azure AI Search.
- Calls to decision and rules APIs, with the LLM handling interpretation and drafting only.
- Apply context engineering: write and version prompts and instructions, manage context windows, memory and tool results, and keep token use within budget.
- Handle LLM calls reliably: structured outputs, retries, fallbacks, caching and rate limits.
- Build evaluation sets and run evals on every release: automated metrics, LLM-as-judge and human review samples.
- Fix findings and resubmit.
- Instrument every agent with OpenTelemetry.
- Build traces, dashboards and alerts for latency, token use, cost, errors and quality.
- Run A/B tests as the architect designs them: variant routing, metric capture and result readouts.
- Deploy through CI/CD, compile release evidence for go/no-go, and support the business through validation and handover.
Must have
- Strong full-stack engineering in Python and/or C#/.NET, plus API and integration development.
- Hands-on delivery of LLM or agent applications: prompt design, function and tool calling, RAG and workflow orchestration.
- Practical experience with at least one of Azure AI Foundry, Copilot Studio or Microsoft Agent Framework.
- Experience with a document intelligence service (Azure AI Document Intelligence or similar) on real, messy documents.
- Experience with a vector database or search index: embeddings, chunking, hybrid search and relevance tuning.
- Working knowledge of context engineering and token handling.
- Evaluation practice: building golden datasets, running evals in CI, and interpreting results for non-deterministic output.
- OpenTelemetry instrumentation and use of Azure Monitor, Application Insights or an equivalent observability tool to diagnose agent failures.
- Familiarity with A/B testing, feature flags and traffic splitting.
- Daily use of AI coding agents (GitHub Copilot, Claude Code, Cursor or similar).
- You write clear specs and context files, review agent code critically and catch its mistakes.
- CI/CD, Git workflows, and Azure fundamentals (identity, secrets, networking, monitoring).
- Clear communication with business users and daily Agile collaboration.
- Microsoft certifications (Azure AI Engineer Associate / Azure AI Apps and Agents Developer Associate)
Nice to have
- Natural-language-to-data solutions and data pipelines (Azure Data Factory, Databricks or Fabric, SQL).
- Rules engines or workflow automation (Power Automate, Logic Apps).
- Healthcare or payer exposure, including PHI handling.