Role Overview
We are seeking an experienced Senior Data Engineer to design, develop, deploy and maintain scalable, reliable and secure data pipelines and reusable data products using AWS and Databricks.
The successful candidate will provide technical leadership across data ingestion, transformation, modelling, quality, orchestration, monitoring and performance optimisation to support analytics, reporting and AI/ML workloads.
This is a hands-on technical role requiring strong expertise in cloud-based data engineering, lakehouse architecture, production pipeline operations and engineering best practices.
Key Responsibilities
- Lead the design, development and implementation of scalable batch and streaming data pipelines using AWS and Databricks.
- Design and maintain lakehouse architectures using Bronze, Silver and Gold data layers, Delta Lake and reusable data models.
- Develop reliable data ingestion processes from databases, APIs, files and event streams, including incremental loading and Change Data Capture (CDC).
- Implement automated data quality checks, schema validation, reconciliation and anomaly detection to ensure data accuracy and consistency.
- Build and manage workflow orchestration using Databricks Workflows, Lakeflow and relevant AWS services.
- Implement CI/CD pipelines, automated testing and Infrastructure as Code (IaC) to support efficient development and deployment.
- Establish data governance practices, including metadata management, data lineage, access controls and data classification.
- Monitor pipeline performance, reliability and data freshness through logging, dashboards, alerts and operational metrics.
- Troubleshoot production issues, perform root-cause analysis and implement preventive measures to improve system stability.
- Optimise Apache Spark and SQL workloads, storage configurations and processing performance to improve efficiency and manage cloud costs.
- Implement cloud security best practices, including IAM, encryption, secrets management, audit logging and secure data sharing.
- Review technical designs and code, establish engineering standards and mentor junior engineers.
- Collaborate with data scientists, analysts, architects, product teams and business stakeholders to translate requirements into technical solutions.
- Maintain technical documentation, data models, architecture decisions and operational procedures.
Requirements
Education & Experience
- Bachelor's degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related discipline.
- 6–10 years of relevant experience in data engineering or software engineering.
- Minimum 2 years of hands-on experience with cloud-based data platforms.
- Proven experience designing, developing and maintaining production-grade data pipelines.
- Demonstrated technical leadership in solution architecture, code reviews, engineering standards or mentoring.
Technical Requirements (Must-Have)
- Strong proficiency in Python, PySpark and SQL, including complex data transformations, distributed processing, query optimisation and debugging.
- Hands-on experience with Databricks, Apache Spark and Delta Lake in production environments.
- Strong AWS data engineering experience with services such as S3, Glue, Lake Formation, IAM, KMS, Lambda, Step Functions, EventBridge, CloudWatch, Redshift and Kinesis/MSK.
- Experience designing and implementing data lake or lakehouse architectures, including Bronze, Silver and Gold data layers.
- Strong understanding of data modelling, dimensional modelling and reusable curated datasets.
- Experience with incremental data ingestion, CDC, backfills, replay processing, idempotency and recovery mechanisms.
- Experience implementing automated data quality validation, schema enforcement, reconciliation, metadata management and data lineage.
- Hands-on experience with workflow orchestration, Git, CI/CD and Infrastructure as Code tools such as Terraform.
- Experience implementing monitoring, logging, alerting and incident management for production data pipelines.
- Good understanding of cloud security, IAM, encryption, secrets management and data governance.
- Ability to conduct technical design reviews, code reviews and provide engineering guidance.
Preferred Qualifications (Good-to-Have)
- Experience with Databricks Unity Catalog, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, Delta Sharing and Databricks Asset Bundles.
- Familiarity with Apache Airflow, dbt, Kafka, AWS MSK/Kinesis and AWS DMS.
- Experience with data products, data contracts, semantic layers, master data management or data mesh architecture.
- Experience supporting machine learning datasets, feature engineering, feature stores or MLOps pipelines.
- Experience working on Singapore public-sector projects, including IM8 compliance, data classification, VAPT and security assurance.
Relevant certifications such as:
- Databricks Certified Data Engineer Professional
- AWS Certified Data Engineer – Associate
- AWS Certified Solutions Architect – Associate or Professional
Soft Skills
- Strong analytical thinking, problem-solving and troubleshooting abilities.
- Excellent communication skills with the ability to explain technical concepts to both technical and non-technical stakeholders.
- Ability to work collaboratively across engineering, analytics, data science, cybersecurity and business teams.
- Strong ownership mindset with a focus on system reliability, maintainability and operational excellence.
- Ability to translate business requirements into technical specifications and practical implementation plans.
- Experience mentoring engineers and promoting engineering best practices.
Originally posted on