Note: The job is a remote job and is open to candidates in USA. 24-MAG LLC connects experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams. The Data Engineer will design, build, and operate scalable data pipelines and cloud-native infrastructure using Python, SQL, Apache Spark, and AWS to support analytics, experimentation, AI/ML model development, and large-scale data processing.
Responsibilities
- Design, build, and maintain large-scale data pipelines
- Ingest and transform data from multiple structured and unstructured sources
- Develop reliable workflows for high-volume data processing
- Improve pipeline scalability, maintainability, and operational performance
- Support downstream analytics, experimentation, and model-development requirements
- Develop distributed processing workflows using Apache Spark
- Optimise large-scale data transformations and computational workloads
- Design partitioning strategies appropriate for high-volume datasets
- Identify and resolve performance bottlenecks across distributed environments
- Apply cloud-native technologies to scalable data-processing workflows
- Design scalable architectures across SQL and NoSQL systems
- Build storage layers that support reliability, performance, and growth
- Evaluate database and storage technologies for different workload requirements
- Improve data accessibility while maintaining system integrity
- Apply sound architecture principles to large and complex datasets
- Design and implement cloud-native data architectures on AWS
- Support high-volume ingestion, processing, storage, and distribution
- Build infrastructure that can scale with increasing data workloads
- Improve reliability and operational efficiency across cloud data systems
- Apply AWS services appropriately across different data-processing scenarios
- Write efficient Python code for data ingestion, transformation, validation, and analysis
- Develop performant SQL queries and data-processing logic
- Improve existing pipelines and transformation workflows
- Apply strong engineering standards to production data systems
- Maintain clear and reusable data-processing code
- Ensure data quality and integrity throughout pipelines and storage layers
- Implement monitoring for data workflows and infrastructure
- Identify failures, anomalies, and processing issues
- Improve operational reliability through automation and validation
- Establish processes that support consistent and trustworthy data delivery
- Collaborate with AI researchers, data scientists, and engineering teams
- Support data-intensive applications and experimentation
- Provide infrastructure for model-training and evaluation workflows where relevant
- Translate research and product requirements into scalable data systems
- Adapt infrastructure as AI and data requirements evolve
Skills
- * Strong professional experience in data engineering or distributed data systems
- * Advanced proficiency in Python and SQL
- * Hands-on experience with Apache Spark or comparable distributed-processing frameworks
- * Strong experience with AWS data services and cloud-native architecture
- * Experience working with both SQL and NoSQL databases
- * Demonstrated experience processing and managing large-scale datasets
- * Strong understanding of data partitioning and performance optimisation
- * Experience designing scalable and reliable data architectures
- * Familiarity with automation, orchestration, and data-pipeline monitoring
- * Strong understanding of data quality and operational reliability
- * Full-time engagement
- * Fully remote
- * Exposure to AI/ML workflows or research environments is advantageous
- * Familiarity with LLM-related training, evaluation, or prompt-experimentation datasets is beneficial
- * Experience with data-visualisation tools such as Matplotlib, Seaborn, or Plotly is a plus
Benefits
- Full-time engagement
- Fully remote
Company Overview