Note: The job is a remote job and is open to candidates in USA. Alpaca is a global provider of agent-first brokerage infrastructure and financial services APIs. The company is seeking a Senior Data Engineer to design, build, and scale its data platform infrastructure, including ingestion, lakehouse, BI, monitoring, and data governance capabilities.
Responsibilities
- Design, build, and evolve the core data platform infrastructure e.g., distributed query engines, orchestration, warehousing, cataloging, and more
- Own our lakehouse infrastructure as code, managing deployments through Terraform and Ansible on Kubernetes
- Build and maintain low-latency streaming and CDC ingestion pipelines, as well as batch ingestion paths landing in Iceberg
- Develop and scale our BI landscape so downstream teams and agents get performant, self-serve access to lakehouse data
- Enforce platform reliability best practices, including monitoring and alerting, on-call rotations, incident response, maintenance windows, runbooks, and SLAs
- Partner with DevOps, Analytics Engineering, and other stakeholders to close infrastructure gaps and support new data requirements
Skills
- 5+ years of experience in Data Engineering, including 2+ years building and operating scalable, low-latency data platforms handling > 100M events/day
- Strong hands-on experience running data infrastructure on Kubernetes, with cloud-native tooling like Docker and Helm
- Production experience with IaC: Terraform, Ansible, and ArgoCD (or equivalents)
- Deep knowledge of distributed systems (storage, transactions, and query processing) with hands-on experience operating open-source query engines like Trino or Presto
- Strong experience with object storage and open table formats, specifically Apache Iceberg
- Experience with streaming and CDC systems: Kafka, Redpanda, and Debezium
- Hands-on experience with orchestration frameworks (Airflow) and ELT tools (Airbyte)
- Strong working knowledge of Python and SQL for building pipelines and platform tooling
- Experience with Google Cloud Platform and its data services (GCS, Cloud Build, Cloud SQL, Dataproc, etc); or related experience with other cloud services
- Ability to thrive in a fast-paced startup environment and adapt infrastructure to rapidly changing needs
- Experience with semantic/metrics layers (Cube, dbt, Looker)
- Familiarity with transformation frameworks (dbt)
- Familiarity with reverse ETL tooling (Hightouch)
- Familiarity with data catalog and lineage tooling (OpenMetadata, Datahub)
- Experience with data access control and governance frameworks (Apache Ranger)
Benefits
- Stock options
- Health benefits
- New Hire Home-Office Setup: One-time USD $500
- Monthly Stipend: USD $150 per month via a Brex Card
- 100% distributed and remote work
Company Overview
Company H1B Sponsorship