Note: The job is a remote job and is open to candidates in USA. ICF is a mission-driven global advisory and technology services provider focused on solving complex challenges and improving people's lives. The Senior Data Engineer will design and maintain scalable data pipelines, develop cloud-based data processing workflows and APIs, and create data visualizations. The role also involves optimizing data infrastructure, ensuring data quality and security, writing tests, and collaborating with DevOps engineers.
Responsibilities
- Design, develop, and maintain scalable data pipelines using Spark, Hive, and Airflow
- Develop and deploy data processing workflows on the Databricks platform
- Develop API services to facilitate data access and integration
- Create interactive data visualizations and reports using AWS QuickSight
- Builds required infrastructure for optimal extraction, transformation and loading of data from various data sources using AWS and SQL technologies
- Monitor and optimize the performance of data infrastructure and processes
- Develop data quality and validation jobs
- Assembles large, complex sets of data that meet non-functional and functional business requirements
- Write unit and integration tests for all data processing code
- Work with DevOps engineers on CI, CD, and IaC
- Read specs and translate them into code and design documents
- Perform code reviews and develop processes for improving code quality
- Improve data availability and timeliness by implementing more frequent refreshes, tiered data storage, and optimizations of existing datasets
- Maintain security and privacy for data at rest and while in transit
- Other duties as assigned
Skills
- Bachelor's degree
- 7+ years of hands-on software development experience
- 4+ years of experience building data pipelines using Python, Java, and cloud technologies, with hands-on experience leveraging Spark and Hive for large-scale data processing
- Candidate must be able to obtain and maintain a Public Trust clearance
- Candidate must reside in the US, be authorized to work in the US, and work must be performed in the US
- Must have lived in the US 3 full years out of the last 5 years
- Experience building job workflows with the Databricks platform
- Strong understanding of AWS products including S3, Redshift, RDS, EMR, AWS Glue, AWS Glue DataBrew, Jupyter Notebooks, Athena, QuickSight, EMR, and Amazon SNS
- Familiar with work to build processes that support data transformation, workload management, data structures, dependency and metadata
- Experienced in data governance process to ingest (batch, stream), curate, and share data with upstream and downstream data users
- Experienced in data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up
- Demonstrated understanding using software and tools including relational NoSQL and SQL databases including Cassandra and Postgres; workflow management and pipeline tools such as Airflow, Luigi and Azkaban; stream-processing systems like Spark-Streaming and Storm; and object function/object-oriented scripting languages including Scala, C++, Java and Python
- Familiar with DevOps methodologies, including CI/CD pipelines (Github Actions) and IaC (Terraform)
- Ability to obtain and maintain a Public Trust; residing in the United States
- Experience with Agile methodology, using test-driven development
Benefits
- Remote work arrangement within the United States
Company Overview