Note: The job is a remote job and is open to candidates in USA. Index Analytics LLC supports government clients with data and analytics solutions. The Senior Data Engineer will modernize data ecosystems by designing scalable cloud-based solutions, databases, integrations, and automated data pipelines using AWS, Databricks, Python, and SQL. The role also focuses on data quality, analytics enablement, DevOps, documentation, and Agile delivery.
Responsibilities
- Collaborate closely with stakeholders, cross‑functional and internal technical teams to gather requirements, document business rules and develop a thorough understanding of the business context and objectives
- Configure and manage connections from analytic tools to back end data sources, repositories, and platforms including Databricks and various APIs
- Oversee Databricks unity catalog population and administration
- Formulate and document technical and coding standards
- Collaborate to design secure, scalable, and cost‑optimized data solutions
- Design, build, and maintain scalable, reliable ETL/ELT data pipelines using AWS and Databricks
- Develop and optimize data models, both conceptual and physical, to support analytics, reporting, and operational consumption
- Implement data quality, validation, and monitoring frameworks to ensure accuracy and reliability
- Ensure data workflows are modular, testable, and properly version‑controlled
- Operationalize pipelines with monitoring, alerting, and automated recovery mechanisms
- Conduct advanced data analysis using languages such as Python and SQL
- Develop documentation to include data models, data dictionaries, and data usage guides
- Improve end-to-end performance of data workflows
- Build and maintain CI/CD pipelines using GitHub to support automated testing, deployments, and continuous integration
- Meet schedule deadlines and commitments with a high-level of quality of deliverables
- Collaborate with a team of cross-functional resources in an Agile delivery environment to deliver iterative value
Skills
- * U.S. Citizen or authorized to work in the United States and have resided in the U.S. for at least three of the past five years
- * Bachelor's degree and a minimum of 10 years of professional experience, including at least 4 years designing, developing, and implementing Business Intelligence (BI), data management, analytics, or similar enterprise IT solutions. An equivalent combination of education and experience may be considered. Four years of specialized experience may substitute for a bachelor's degree
- * Demonstrated expertise in data engineering, data integration, and enterprise data platform development, with hands-on experience designing and implementing scalable data pipelines, data warehouses, data lakes, and cloud-based data solutions
- * Advanced proficiency in SQL and Python, including experience developing, optimizing, testing, and maintaining data transformation processes, data pipelines, and analytical workflows
- * Strong experience with AWS cloud services and Databricks, including the design, implementation, and support of modern data architectures. Experience with artificial intelligence, machine learning, or generative AI technologies is highly desirable
- * Proven experience using GitHub for source control management, branching strategies, code reviews, versioning, and collaborative software development practices
- * Strong understanding of modern data architecture principles, including ETL/ELT frameworks, data modeling, data governance, data integration patterns, relational and non-relational databases, and enterprise data warehousing concepts
- * Experience applying software engineering best practices, including object-oriented programming, design patterns, code quality, testing, documentation, and maintainable solution development
- * Familiarity with DevOps methodologies and CI/CD pipelines, including experience with automated build, deployment, testing, and monitoring tools
- * Strong analytical, problem-solving, and communication skills, with the ability to translate complex technical concepts into actionable insights and effectively present findings and recommendations to technical and non-technical stakeholders
- * Excellent written and verbal communication skills, including the ability to collaborate across multidisciplinary teams, facilitate discussions, develop technical documentation, and provide executive-level reporting
- * Working knowledge of Agile frameworks and Scrum methodologies, including experience using tools that support Agile delivery, sprint planning, backlog management, and team collaboration
- * Experience working with CMS enterprise repositories, including the Integrated Data Repository (IDR), is preferred
- * Prior experience supporting federal government agencies, healthcare programs, or large-scale public sector data environments is preferred
Benefits
- Fully remote work arrangement
Company Overview