Note: The job is a remote job and is open to candidates in USA. SoftStandard Solutions has deep expertise in Data Engineering, AI, and Machine Learning and is currently seeking a skilled Data Engineer with strong AI and ML experience for immediate placement across their active client engagements. The role involves designing and maintaining data pipelines, collaborating with data scientists, and implementing data governance practices.
Responsibilities
- Design, build, and maintain scalable batch and real-time data pipelines across cloud environment
- Architect and manage data lake, data warehouse, and lakehouse solutions using Snowflake, Databricks, and Delta Lake
- Develop ETL and ELT workflows to ingest, transform, and deliver reliable data to ML models, dashboards, and business applications
- Build and maintain feature engineering pipelines and feature stores to support ML model training and inference
- Collaborate with data scientists and ML engineers to design data infrastructure supporting model development and production deployment
- Implement real-time streaming pipelines using Apache Kafka, Spark Streaming, or Apache Flink
- Build vector data pipelines and embedding workflows to support LLM-powered applications and RAG system
- Implement data quality frameworks, validation checks, and observability tooling across all data assets
- Enforce data governance practices including lineage tracking, metadata management, and data privacy compliance
- Optimize data models and query performance for large-scale analytical and ML workloads
- Implement CI/CD pipelines for data infrastructure using infrastructure-as-code and DevOps best practices
Skills
- 4–6 years of experience in data engineering with strong focus on AI and ML data infrastructure
- 2+ years of experience integrating AI and ML workflows into data pipelines and platforms
- Strong proficiency in Python and SQL for pipeline development and data transformation
- Hands-on experience with Apache Spark and PySpark for large-scale distributed data processing
- Proven expertise with Snowflake, Databricks, BigQuery, or Redshift for data warehousing and lakehouse architecture
- Experience with data orchestration tools including Apache Airflow, Prefect, or Dagster
- Solid knowledge of real-time streaming using Apache Kafka, Spark Streaming, or AWS Kinesis
- Familiarity with ML concepts including feature engineering, model training pipelines, and data preparation for AI use cases
- Cloud proficiency on AWS, GCP, or Azure including managed services for storage, compute, and orchestration
- Experience with containerization and orchestration using Docker and Kubernetes
- Knowledge of data governance, data lineage, and data quality best practices
- Degree in Computer Science, Engineering, Mathematics, or equivalent practical experience
Company Overview
Company H1B Sponsorship