Job Title: Data Engineer (Databricks, Azure, Kafka)
Location: Remote
Job Type: W2 / C2C / 1099
Job Summary:
We are looking for an experienced Data Engineer with 9+ years of experience to design, build, and maintain data pipelines using Databricks, Spark, and Azure cloud services. The ideal candidate has hands-on experience with modern data engineering tools, cloud automation, and event-driven architectures.
Job Responsibilities:
Design, develop, and optimize scalable data pipelines using Databricks and Spark (PySpark)
Build and maintain ETL/ELT workflows integrating with Snowflake and other data platforms
Develop and manage Azure-based data solutions using ADF, ADLS, Azure Functions, and Key Vault
Implement CI/CD pipelines and infrastructure automation using GitHub Actions, Terraform, and DevOps best practices
Build and maintain event-driven architectures using Kafka, including CDC and streaming integrations
Work with AI/GenAI initiatives involving LLM fine-tuning, RAG, and prompt engineering
Monitor, troubleshoot, and optimize data pipeline performance and reliability
Collaborate with data architects, analysts, and business stakeholders to translate requirements into technical solutions
Ensure data quality, security, and governance standards are met across pipelines
Orchestrate and schedule workflows using Airflow
Required Skills:
8+ years of experience in Data Engineering
Data Engineering: Databricks, Spark (PySpark), Python/Scala, SQL, Snowflake
Azure: Azure Cloud Services, ADF, ADLS, Azure Functions, Key Vault, DevOps
AI/GenAI: LLM Fine-Tuning, RAG, Prompt Engineering
Automation & DevOps: Airflow, GitHub, GitHub Actions, CI/CD, Terraform, Infrastructure Automation
Kafka, Event-Driven Architecture, CDC/Streaming Integrations
Preferred Qualifications:
Bachelor's or Master's degree in Computer Science, Engineering, or related field
Azure certifications (e.g., Azure Data Engineer Associate, Azure Solutions Architect) is a plus
Experience working in large-scale enterprise or regulated industry environments (finance, healthcare, insurance)
Strong communication skills with ability to work directly with client stakeholders