Note: The job is a remote job and is open to candidates in USA. TECHEAD is seeking a Senior Data Engineer to transform core data capabilities using Databricks. The role focuses on building enterprise data frameworks, optimizing pipeline infrastructure, implementing medallion architecture, integrating AI-driven document processing, and managing end-to-end SDLC activities.
Responsibilities
- Framework & Capability Building: Enhance out-of-the-box Databricks features by building scalable frameworks, dashboards, and Delta table infrastructure (e.g., SLA data freshness tracking, data observability, pipeline monitoring)
- Data Architecture & Engineering: Design and implement medallion architecture (Bronze, Silver, Gold layers) using ER and dimensional modeling to transform unstructured and legacy data into high-value assets
- AI & Document Processing: Leverage features like Intelligent Document Processing and AI extraction tools to parse images, PDFs, and unstructured records into structured analytical models
- Full SDLC Ownership: Collaborate with business users, architects, and technical leads to gather requirements, estimate work efforts, write code, run unit/SIT testing, and support UAT
- Enablement & Best Practices: Create clear documentation and training materials to help cross-functional and vendor teams (onshore/offshore) adopt your custom frameworks and Databricks best practices
- Governance & Security: Implement and maintain robust data security policies, Unity Catalog governance, and industry-standard compliance controls
Skills
- 3+ years of full-time experience using Databricks (in Azure or AWS) leveraging medallion architecture, batch/streaming orchestration, and compute/storage optimization
- Expert-level proficiency in SQL and intermediate-to-advanced proficiency in PySpark
- Demonstrated experience creating standardized dimension/fact tables, handling SCD Type 2 time tracking, and configuring ER models
- Proven track record building reusable coding frameworks for data standardization (e.g., missing values, date alignment, reference data management, data repair/imputation)
- Strong background with Git/GitLab CI/CD, Lakeflow, data quality rules, Spark UI, and job scheduling
- Comfortable operating in an Agile environment (defining technical tasks, writing user stories, and providing effort estimates)
Company Overview