Note: The job is a remote job and is open to candidates in USA. Aditi Consulting is seeking a Data Engineer to collaborate with cross-functional teams and develop scalable, efficient, and secure data engineering practices across Azure and GCP. The role focuses on building data pipelines, implementing machine learning feature engineering workflows, automating testing, optimizing data workflows, and documenting data solutions.
Responsibilities
- Build and Maintain Data Pipelines: Design, build, and maintain scalable, efficient, and reliable data pipelines to support data ingestion, transformation, and integration across diverse sources and destinations, using tools such as Kafka, Databricks, and similar toolsets
- Drive Digital Innovation: Leverage innovative technologies and approaches to modernize and extend core data assets, including SQL-based, NoSQL-based, cloud-based, and real-time streaming data platforms
- Implement Feature Engineering: Develop and manage feature engineering pipelines for machine learning workflows, utilizing tools like Vertex AI, BigQuery ML, and custom Python libraries
- Implement Automated Testing: Design and implement automated unit, integration, and performance testing frameworks to ensure data quality, reliability, and compliance with organizational standards
- Optimize Data Workflows: Optimize data workflows for performance, cost efficiency, and scalability across large datasets and complex environments
- Draft and Review Documentation: Draft and review architectural diagrams, interface specifications, and other design documents to ensure clear communication of data solutions and technical requirements
Skills
- Bachelor's degree typically in Computer Science, Management Information Systems, Mathematics, Business Analytics or another STEM degree
- 4+ years of professional Data Development experience
- 4+ years of experience with SQL and NoSQL technologies
- 3+ years of experience building and maintaining data pipelines and workflows
- 2+ years of experience developing with Python
- Experience in feature engineering for machine learning pipelines
- Experience with CI/CD pipelines and processes
- Experience with automated unit, integration, and performance testing
- Experience with version control software such as Git
- Full understanding of ETL and Data Warehousing concepts
- Strong understanding of Agile principles (Scrum)
- Knowledge of Structured Streaming (Spark, Kafka, EventHub, or similar technologies)
- Experience with GCP services (or Databricks equivalent) such as BigQuery, Vertex AI Platform, Cloud Storage, AutoMLOps, and Dataflow
- Experience with GitHub SaaS/GitHub Actions
- Experience understanding Databricks concepts
- Experience with PySpark and Spark development
- Experience with Service Oriented Architecture
Benefits
- Remote work arrangement
- Non-exempt positions are eligible for overtime at a rate of 1.5 times the base hourly rate for all hours worked in excess of 40 in a work week, or as required by state or local law.
- Full-time employees are eligible to select from different benefits packages.
- Medical benefits
- Dental benefits
- Vision benefits
- Health savings accounts with qualified medical plan enrollment
- 10 paid days off
- 3 days paid bereavement leave
- 401(k) plan participation with employer match
- Life and disability insurance
- Commuter benefits
- Dependent care flexible spending account
- Accident insurance
- Critical illness insurance
- Hospital indemnity insurance
- Accommodations and reimbursement for work travel
- Discretionary performance or recognition bonus
- Sick leave based on state or local law
- Mobile phone reimbursement based on state or local law
Company Overview
Company H1B Sponsorship