Note: The job is a remote job and is open to candidates in USA. CodeX Tech-IT LLC is seeking a Data Engineer to help lead the evolution of their core data platform. In this role, you will design fault-tolerant data pipelines, scale the cloud lakehouse storage layer, and build automated data quality and observability frameworks while collaborating with various teams.
Responsibilities
- Design, build, and maintain high-throughput batch and real-time streaming pipelines using Python, SQL, and Apache Spark / PySpark or Kafka
- Model modular, scalable data schemas in open formats (e.g., Apache Iceberg, Delta Lake) using modern cloud platforms (Snowflake, Databricks, or BigQuery)
- Implement automated test suites, schema enforcement policies, and proactive alerting (e.g., using dbt, Great Expectations, or Monte Carlo) to maintain high SLA uptime
- Audit distributed jobs, refactor slow-running SQL queries, and optimize cloud warehouse compute/storage configurations to minimize cost without sacrificing speed
- Manage data infrastructure using Terraform, containerize services via Docker, and enforce robust CI/CD deployment pipelines using GitLab/GitHub Actions
- Partner with Product and AI teams to prepare feature stores, support RAG/LLM data feeds, and maintain strict data governance (RBAC, PII masking, SOC 2/GDPR compliance)
Skills
- 2+ years of dedicated experience in data engineering, data platform engineering, or backend software development with heavy data focus
- Bachelor's degree in Computer Science, Software Engineering, Information Systems, or equivalent practical experience
- Advanced proficiency in Python (or Scala/Java) with strong exposure to OOP/functional paradigms, unit testing, Git workflows, and CI/CD pipelines
- Deep expertise in advanced SQL (window functions, CTEs, query plan tuning) on platforms like Snowflake, Databricks, BigQuery, or AWS Redshift
- Hands-on experience building and managing complex workflow DAGs using Apache Airflow, Dagster, or Prefect
- Working knowledge of cloud native architecture (AWS, GCP, or Azure)
- Hands-on experience with streaming architectures (Apache Kafka, AWS Kinesis, or Flink)
- Experience using dbt (data build tool) for version-controlled, software-managed data transformation layers
- Familiarity with container orchestration using Kubernetes
- Professional certifications (AWS Certified Data Engineer, Databricks Certified Data Engineer, or SnowPro Advanced)
Company Overview