Note: The job is a remote job and is open to candidates in USA. NTT DATA is seeking a hands-on Senior Data / AI Native Engineer with expertise in Databricks, AWS, PySpark, and Lakehouse architecture. The role leads and delivers enterprise data engineering initiatives, builds scalable data platforms and pipelines, and applies AI-assisted engineering practices to data workflows.
Responsibilities
- Design, develop, and optimize high-volume enterprise data pipelines using PySpark, Spark, and Databricks
- Build scalable Lakehouse solutions using Delta Lake and Medallion Architecture (Bronze/Silver/Gold)
- Implement data governance, access controls, and cataloging using Databricks Unity Catalog
- Design and develop cloud-native data solutions using AWS S3, EMR, Glue, Lambda, and Redshift
- Lead complex Data Engineering initiatives and provide technical guidance to engineering teams
- Develop batch and real-time data processing pipelines for large-scale enterprise datasets
- Use GitHub Copilot and/or Claude Code CLI daily to accelerate coding, pipeline development, testing, troubleshooting, and documentation
- Rapidly develop POCs and MVPs using AI-assisted engineering practices
- Apply AI within data pipelines for use cases such as schema inference, automated data-quality rule generation, anomaly detection, and PySpark transformation generation
- Build AI-ready data platforms supporting RAG, embeddings, vector stores, and AI/ML applications
- Develop streaming pipelines using Structured Streaming, Kafka, Kinesis, or Databricks Auto Loader
- Implement modern data engineering patterns including CDC, SCD Type 2, schema evolution, idempotent processing, and data contracts
- Implement data quality, validation, monitoring, lineage, and observability across data pipelines
- Conduct code/design reviews and establish reusable Data Engineering patterns and standards
- Collaborate with Data Architects, AI/ML Engineers, Software Engineers, and business stakeholders to deliver enterprise data products
Skills
- 7+ years of experience in Data Engineering and Software Engineering, building production-grade enterprise data solutions
- 4+ years of hands-on experience with Databricks, Delta Lake, Medallion Architecture, and PySpark/Spark
- 4+ years of experience with the AWS data ecosystem, including S3, EMR, Glue, Lambda, and/or Redshift
- Proven experience leading complex Data Engineering initiatives or providing technical leadership to Data Engineering teams
- Strong hands-on experience processing high-volume datasets using PySpark, rather than Scala-only Spark development
- Demonstrated daily use of GitHub Copilot and/or Claude Code CLI, with the ability to explain specific examples of how these tools improve Data Engineering productivity
- Proven experience rapidly developing POCs and MVPs using AI-assisted development practices
- Databricks, Delta Lake, Unity Catalog
- PySpark, Apache Spark
- AWS – S3, EMR, Glue, Lambda, Redshift
- Lakehouse, Medallion Architecture, Data Lakes
- Spark Structured Streaming, Kafka, Kinesis, Auto Loader
Benefits
- Medical, dental, and vision insurance
- Flexible spending or health savings account
- Life and AD&D insurance
- Short- and long-term disability coverage
- Paid time off
- Employee assistance
- Participation in a 401k program with company match
- Additional voluntary or legally required benefits
Company Overview
Company H1B Sponsorship