Note: The job is a remote job and is open to candidates in USA. G2 is a software marketplace and data foundation that provides trusted software insights to help businesses make better buying decisions. The Principal Data Scientist will shape experimentation practices and use data science to inform product decisions across G2's product offerings, while building models, designing auctions, shipping production systems, mentoring team members, and promoting data science best practices.
Responsibilities
- Own the experimentation standard for G2 and G2 Digital Markets. Consolidate the practices currently spread across properties into a single, documented, defensible approach covering experiment design, randomization, metric definition, guardrails, sample size and duration, and how results get read and acted on
- Influence engineering and product to adopt it. Standards that live in a doc do not change outcomes. You will work directly with engineering teams on instrumentation, assignment, and platform behavior, and make the correct approach the low-friction one — through tooling, templates, review, and consistent partnership rather than mandate
- Define the requirements for holistic experimentation at G2. Determine what our experimentation capability needs to look like end to end — platform requirements, telemetry, metric layers, analysis tooling, review process, and the organizational habits around them — and make the case for it to product and engineering leadership
- Design methods that fit our actual traffic and iteration constraints. Build a portfolio of approaches suited to different volume regimes and decision speeds (e.g., high-throughput testing where traffic is abundant, variance reduction and sensitivity techniques where it is not, sound alternatives where a randomized test is not feasible at all)
- Drive the exploration of modern experimentation methods. Evaluate emerging approaches and operationalize the ones that earn their place, with a considered point of view on when each is and is not the right tool (e.g., Bayesian decision frameworks, always-valid inference and sequential testing, multi-armed bandits, variance reduction methods, interference-aware designs for marketplace settings)
- Evangelize data science and make its impact legible. Teach the practice across product, engineering, and business stakeholders; raise the quality of how experiment results are interpreted and communicated; and hold the organization to an honest account of what our tests actually demonstrate
Skills
- 8+ years of relevant experience in data science, applied statistics, or a closely related field, with a substantial portion focused on experimentation; or a PhD in a quantitative discipline plus 6+ years
- Deep expertise in experimental design and causal inference, including a working knowledge of the failure modes of online controlled experiments (e.g., power and sensitivity analysis, variance reduction, multiple testing correction, heterogeneous treatment effects)
- Working command of both frequentist and Bayesian approaches, and the judgment to choose between them for a given decision rather than by default
- Demonstrated experience designing experimentation methodology for a real product — not only analyzing tests, but deciding how they should be run
- A track record of setting technical standards that other teams actually adopted, and of influencing engineering partners without authority over them
- Strong applied experience with time series and web/behavioral data at scale
- Fluency in Python and SQL, and comfort in modern data ecosystems (e.g., Snowflake, Spark, Airflow, dbt)
- Exceptional written and verbal communication, including the ability to explain a methodological trade-off to a non-technical stakeholder and defend it to a skeptical technical one
- 10+ years of relevant experience, including ownership of experimentation practice or platform at a company running experiments across multiple products or properties
- Hands-on experience taking modern inference methods into production (e.g., always-valid p-values, sequential testing, bandit-based allocation)
- Experience with low-traffic or low-power experimentation problems, and with quasi-experimental methods where randomization is constrained (e.g., geo tests, synthetic control, difference-in-differences, switchback designs)
- Experience with interference, network effects, or two-sided marketplace experimentation
- Experience serving as a technical lead — setting direction for a group, mentoring senior data scientists, reviewing others' designs and analyses
- Experience with evaluation of ML or LLM systems (e.g., offline/online evaluation design, metric development)
- Contributions to the experimentation community through publications, open-source work, or conference participation
- PhD in a quantitative discipline or MS in STEM
Company Overview
Company H1B Sponsorship