Note: The job is a remote job and is open to candidates in USA. System One is seeking an AWS Data Engineer for a remote contract-to-hire role. The engineer will build and operate AWS-based data pipelines and lakehouse infrastructure, supporting secure, reliable, governed data for analytics and machine learning.
Responsibilities
- Build and operate data pipelines (batch and streaming) from various sources including APIs, relational databases, file drops, event streams, and external partners
- Design, implement, and optimize ETL/ELT pipelines using Python and PySpark to produce analytics-ready datasets for reporting, visualization, and machine learning
- Implement incremental processing, change data capture (CDC), data contracts, schema validation, and reusable transformation frameworks
- Improve pipeline reliability through automated testing, orchestration, monitoring, retry handling, and operational runbooks
- Build and manage a scalable lakehouse on Amazon S3 using Apache Iceberg and open columnar formats such as Apache Parquet
- Implement SQL-like table reliability features including ACID transactions, schema evolution, partition evolution, snapshot isolation, time travel, and rollback capabilities
- Enable fast, interactive querying of lakehouse data using AWS-native query and compute services such as Amazon Athena, Amazon EMR, AWS Glue, and Amazon Redshift
- Optimize performance and cost efficiency through partitioning, compaction, file sizing, caching, lifecycle policies, and efficient compute/storage separation
- Establish standardized development, testing, and production environments with consistent configuration and controlled promotion across stages
- Implement data governance and fine-grained access control utilizing AWS-native services like AWS Lake Formation, AWS Glue Data Catalog, IAM, KMS, and related security tools
- Create a managed metadata repository for dataset cataloging, ownership, tagging, classification, and discoverability
- Support end-to-end data lineage for source, transformation, and consumption to facilitate auditability and impact analysis
- Apply security policies such as least privilege access, data classification, encryption, retention, and secure data handling
- Build operational data quality checks for metrics such as freshness, completeness, validity, and anomaly detection, along with publishing SLAs/SLOs
- Implement automated AWS provisioning through Infrastructure as Code (IaC) to ensure consistent, secure environments
- Enhance CI/CD pipelines for data workflows and lakehouse components, including automated testing, security validation, packaging, deployment, promotion, and rollback
- Maintain observability with centralized metrics, logs, traces, alerts, dashboards, runbooks, and incident response procedures
- Continually evaluate platform performance, scalability, reliability, security, and cost, and implement measurable improvements
- Collaborate closely with data, application, analytics, AI/ML, security, networking, and cloud platform teams to support mission-critical requirements
- Maintain high-quality documentation including architecture diagrams, SOPs, data models, interface specs, and operational runbooks
- Present technical findings, trade-offs, risks, and recommendations clearly to stakeholders
Skills
- US Citizen with the ability to obtain a public trust
- Bachelor's degree in Engineering, Information Technology, Computer Science, Data Engineering, or a related field, or four (4) years of equivalent practical experience
- Six (6) years of relevant hands-on experience in data engineering
- Extensive experience designing, implementing, and operating AWS-native data lake or lakehouse architectures with Amazon S3, AWS Glue, Amazon Athena, Amazon EMR, AWS Lake Formation, and Amazon Redshift
- Proven ability developing production ETL/ELT pipelines using Python and PySpark, including data modeling, transformation, tuning, and error handling
- Hands-on experience with Apache Iceberg covering ACID transactions, snapshots, schema evolution, and query optimization
- Advanced SQL skills supporting analytical queries, reporting, and data visualization workloads
- Demonstrated experience with data governance, cataloging, lineage, ownership, classification, and access controls
- Knowledge of AWS security fundamentals including IAM, encryption (KMS), secrets management, and network security
- Experience provisioning resources via Infrastructure as Code (IaC) and managing multi-environment platforms
- Skilled in building and maintaining CI/CD pipelines for data workflows with automation, testing, and rollback strategies
- Strong troubleshooting skills for distributed data workloads with focus on performance, reliability, and cost management
- Preference is given to those who are located in the Washington, DC Metro in case the team needs to meet in person
Benefits
- Remote – offsite; preference is given to those located in the Washington, DC Metro in case the team needs to meet in person.
- Eligible employees: health and welfare benefits coverage options including medical, dental, vision, spending accounts, life insurance, and voluntary plans.
- Eligible employees: participation in a 401(k) plan.
Company Overview