Description
About the Role:
We are seeking an experienced Data Quality Engineer to support ATLAS, the VA’s open-architected, multi-cloud enterprise data management and analytics platform. In this role, you will design and automate validation, testing, profiling, and monitoring frameworks that ensure the accuracy, consistency, completeness, and reliability of multi-source data as it is ingested, conditioned, curated, and delivered as consumable enterprise data products. Working with Azure Synapse, Databricks, Collibra Data Quality, Profisee, SQL, Python, and related tools, you will detect anomalies, enforce governance standards, and improve trust in data across interoperating ATLAS platforms. You will collaborate with data engineers, analysts, data scientists, and product teams to provide consistent, secure access to high-quality data that enables evidence-based decisions and improves outcomes for Veterans and their families.
Key Responsibilities:
- Design and implement data validation and quality assurance processes for cloud-based environments.
- Automate data quality checks and develop monitoring solutions to detect data anomalies.
- Collaborate with data engineers and analysts to ensure data governance and compliance.
- Analyze and profile structured and unstructured datasets for accuracy and consistency.
- Build and maintain automated testing frameworks to proactively catch data issues.
- Utilize Azure data engineering and ELT tools, including Azure Synapse and Databricks.
- Create and execute comprehensive test plans using tools such as JIRA’s X-RAY.
- Implement and execute test automation using CoTS tooling like Collibra's Data Quality, Microsoft’s Profisee, or custom scripts in Python or Scala.
TAG:
TAG: INDMJC
Requirements
Minimum Qualifications:
- 3+ years of experience working in a cloud-based environment.
- Experience in analyzing and profiling data.
- Proficiency in querying and working with data using SQL and Python.
- Experience in building automated testing frameworks for data quality assurance.
- Hands-on experience with Azure tools for data engineering and ELT, including Azure Synapse and Databricks.
- Strong knowledge of best testing practices and patterns in cloud environments.
- Ability to work independently within a mid-sized team and rapidly develop test plans using JIRA's X-RAY.
- Proficiency in implementing automated test plans using Collibra's Data Quality, Microsoft Profisee, or script-based testing with Python or Scala.
- Ability to obtain and maintain a Public Trust or Suitability/Fitness determination based on client requirements.
- Bachelor’s degree or 4+ years of relevant experience in lieu of a degree.
- Experience with Synapse Pipelines, Azure Data Factory, Starburst, Azure Data Lake Service, Azure ML/AI services, and Pyramid Analytics/Power BI for visualization.
Preferred Qualifications:
- Familiarity with JIRA and Confluence for project tracking and documentation.
- Ability to integrate testing protocols with monitoring applications such as Dynatrace, Splunk, or Log Analytics.
- Bachelor’s degree in Computer Science or a related field.
- Azure Data Engineer Certification.
Benefits
Comprehensive healthcare coverage, HSA Plan, Life and Disability Insurance, and Retirement Savings Plan.
Originally posted on