Job Description:
Role: Senior Data Engineer
Location: Remote
Duration: Full Time
Job Description
Seeking a Senior Data Engineer with strong hands-on experience in Databricks, Spark, and AWS to build scalable, secure data pipelines and lakehouse solutions supporting analytics, AI/ML, and healthcare data.
Key Responsibilities
Design and build data pipelines using Databricks, Delta Lake, Spark, Unity Catalog, and Workflows.
Develop batch and real-time pipelines using AWS S3, Glue, Redshift, Athena, Lambda, Step Functions, and Kinesis.
Build and optimize lakehouse architecture, data models, and ETL/ELT processes.
Support ML/AI initiatives including feature stores, LLM/RAG data pipelines, and data integration.
Implement data security, governance, and HIPAA-compliant data handling, including access controls and auditing.
Partner with Data Science, Engineering, Product, and business teams.
Review code, establish engineering best practices, and mentor junior engineers.
Requirements
8+ years of Data Engineering experience.
4+ years working with cloud-based data lake/lakehouse architectures.
Deep hands-on experience with Databricks, Apache Spark/PySpark, Delta Lake, Unity Catalog, and Workflows.
Strong AWS experience with S3 and data/analytics services.
Strong Python and SQL skills.
Experience with Airflow/dbt, Kafka/Kinesis, CI/CD, Docker/Kubernetes, and data quality practices.
Healthcare or regulated-data experience with HIPAA required.
Experience supporting ML/AI data pipelines and feature stores.
Strong communication and technical leadership skills.
Thanks,
Suresh- Sr. Technical Recruiter