Note: The job is a remote job and is open to candidates in USA. Visionary Innovative Technology Solutions LLC is seeking an experienced Data Engineer to develop, maintain, and optimize scalable data pipelines using Databricks, PySpark, SQL, and Delta Lake. The role involves delivering data engineering solutions from development through deployment, ensuring data quality, reliability, and performance.
Responsibilities
- Develop and maintain ETL/ELT data pipelines using PySpark, Python, and SQL within Databricks
- Design, implement, and optimize Delta Lake tables for efficient data processing
- Configure and manage Databricks Workflows to orchestrate and automate data pipelines
- Write unit tests and perform data validation to ensure code quality and reliability
- Investigate and resolve data quality issues and maintain data integrity
- Use Git for version control and follow CI/CD practices for deployment
- Prepare and maintain technical documentation for pipelines and data solutions
- Participate in peer code reviews and follow established coding standards
- Troubleshoot pipeline failures and optimize job performance
Skills
- 5+ years of experience in Data Engineering, including 3+ years of hands-on Databricks experience
- Strong proficiency in PySpark, Python, and SQL
- Working knowledge of Delta Lake and Databricks Workflows
- Experience with at least one major cloud platform: Azure, AWS, or GCP
- Familiarity with Git, CI/CD pipelines, and software development best practices
- Experience with ETL/ELT processes, data transformation, and data quality checks
- Ability to independently develop, test, and deploy technical solutions
- Strong troubleshooting, analytical, and problem-solving skills
- Databricks Data Engineer Associate Certification
- Experience with data platform standards and engineering best practices
- Knowledge of automated testing, monitoring, and performance optimization
- Strong communication and collaboration skills
Company Overview