Responsibilities
- Design and implement large-scale data processing pipelines capable of handling millions of records across a cloud-native data platform.
- Establish standardized and reusable architecture patterns for low-latency data analysis and querying at petabyte scale.
- Design and deliver REST/gRPC APIs, web applications, and natural-language/AI-driven interfaces that enable users to interact with large datasets efficiently.
- Own the reliability, scalability, performance, and cost efficiency of the platform, including workload orchestration, monitoring, debugging, and optimization.
- Design systems capable of supporting millions of concurrent workloads, with strong fault tolerance, resiliency, reproducibility, and operational visibility.
- Partner with cloud infrastructure, engineering, product, and other technical teams to translate complex business requirements into scalable product architectures and realistic delivery plans.
- Drive the transition of innovative technical concepts and prototypes into commercial-grade, production-ready software solutions.
- Set technical direction and raise engineering standards through architecture reviews, design reviews, mentoring, hands-on development, and rapid prototyping.
- Collaborate effectively with globally distributed teams and maintain a strong customer-focused approach to product and technology development.
Requirements
- Degree in Computer Science, Computer Engineering, Mathematics, or a related technical discipline.
- Proven experience designing and delivering large-scale data-intensive production software in the cloud using AWS, Azure, or GCP.
- Strong hands-on experience with Docker, Kubernetes, infrastructure automation, and CI/CD.
- Deep knowledge of modern data lake and lakehouse architectures, including Apache Iceberg or Delta Lake, catalog services, snapshot management, partitioning, data pruning, compaction, and schema evolution.
- Strong understanding of columnar data formats such as Parquet and distributed query engines such as Spark, Trino, or Databricks.
- Strong algorithms and systems engineering foundation, with proven ability to develop performance-critical software in C/C++, including concurrency, memory management, vectorization, and SIMD.
- Experience working with columnar and vectorized processing technologies such as Apache Arrow, DuckDB, or Velox.
- Proven track record of profiling and optimizing large-scale systems for runtime, memory, I/O, scalability, and cloud infrastructure costs.
- Strong proficiency in Python for rapid prototyping, data processing, automation, and production tooling.
- Experience designing and operating distributed batch processing and workflow systems handling very large numbers of concurrent jobs.
- Strong understanding of production observability, including metrics, tracing, structured logging, retries, checkpointing, idempotency, and failure recovery.
- Experience with workflow orchestration and execution environments such as Kubernetes or cloud-based batch processing platforms.
- Experience designing and delivering API-first services, including REST/gRPC, API versioning, backward compatibility, authentication, authorization, and multi-tenant architectures.
- Sufficient full-stack experience to collaborate effectively on modern web applications and data-intensive user interfaces.
- Experience developing highly reliable and reproducible software, including automated testing, benchmarking, validation, and performance testing.
- Proven ability to provide technical leadership and influence at senior level, with excellent communication and stakeholder management skills.
- Demonstrated experience using AI-powered development tools to accelerate and improve the software development lifecycle, engineering productivity, and software quality.
Experience / Education
- Typically requires a minimum of 12 years of relevant experience with a Bachelor's degree, or 8 years with a Master's degree, or a PhD with 5 years of relevant experience; equivalent experience may also be considered.
- Demonstrated track record of leading complex technical initiatives from concept and prototyping through production deployment and commercialisation.
If this opportunity aligns with your skills and career goals, we encourage you to apply.
EA License: 21C0783
EAP Registration No: R24123529
Originally posted on