Note: The job is a remote job and is open to candidates in USA. Hercules is hiring a Staff Technical Product Manager to own evaluations for its AI coding agent. The role focuses on improving the agent’s cost, speed, and accuracy through offline and online evaluations, using product data while protecting customer privacy.
Responsibilities
- Your #1 and only goal is to make Hercules' AI coding agent cheaper, faster, and more accurate (our pareto frontier)
- You will own Hercules Bench, our offline eval set
- You will own Hercules online evals
- You will leverage our 500k+ users, 1M+ code repos, and millions of chat threads to guide your eval creation
- You will anonymize data to protect our customer's privacy
Skills
- Senior or staff level product manager
- Software fundamentals: You are building an AI coding agent. We expect you to be technical and able to code
- We expect you to have a decent understanding of system design, architecture, data modeling, and API design
- You have explicitly created and/or managed technical evals before
- You should be able to work across a variety of different eval methodologies and deeply understand optimal reward modeling
- Product sense: strong intuition for the goals customers are trying to achieve and novel solutions to help them achieve them
- Design sense: Track record of building simple, delightful UX/UI that makes it easy for customers to achieve their goals
- AI context and harness engineering: strong experience with the most up to date context and harness engineering best practices like summarization/compression, prompting, prompt caching, subagents, memory, context management, tools, async tools, deferred tools, skills, etc
- + Yes, we know this industry is evolving rapidly and the "best practices" change every week, but yes, we are still actively looking for someone who has wrestled with these challenges at scale and has strong opinions about what works and what doesn't
- Our core stack: TypeScript, React, (some sort of agent framework like AI SDK), (some sort of eval platform like LangSmith)
- AI: able to truly 10x your workflow by being on frontier of AI coding
- We ship. We ship fast, iterate, and correct mistakes quickly. Our bias for action speeds up our learning and delights our customers
- We strive for excellence. Doing things well is in our DNA. We have good taste. We get the details right. Our outputs are exceptionally simple and self-serve. We're proud of our work
- We are hungry. We are willing to work hard and break through walls to achieve the outcomes we want. We are willing to operate at any level to get things done
Benefits
- Equity (variable based on seniority, equity, and location)
- Unlimited AI token budget
- Take what you need vacation policy
- Healthcare, vision, and dental
- Daily lunches, dinners and snacks in the office
- Remote work for exceptional candidates who are willing to work in US time zones and travel to the office regularly
Company Overview