Note: The job is a remote job and is open to candidates in USA. Allstate is seeking a Data Engineer for its Risk Data Team. The role builds scalable pipelines and governed, analytics-ready risk data products to improve data reliability and support reporting, decision-making, and future AI capabilities.
Responsibilities
- Design, build, and maintain trusted risk data products and scalable batch and streaming data pipelines using cloud-native technologies
- Integrate data from risk platforms, control systems, vendor tools, operational applications, and external sources into governed, analytics-ready datasets
- Develop and optimize ETL/ELT workflows that automate data ingestion, transformation, validation, reconciliation, and delivery
- Build and manage data processing workloads within modern data lake and lakehouse environments, including Microsoft Fabric and OneLake
- Implement data quality, monitoring, lineage, and reconciliation processes to ensure data reliability, consistency, and accuracy
- Create curated datasets and analytical outputs that support operational reporting, leadership dashboards, trend analysis, risk identification, and decision-making
- Optimize data architectures, schemas, and processing patterns for scalability, performance, resilience, and cost efficiency
- Develop reusable frameworks, engineering standards, CI/CD pipelines, automated testing, and operational monitoring to improve productivity and maintainability
- Partner with analytics, product, risk, governance, security, and compliance stakeholders to establish data definitions, metrics, quality standards, and secure data access
- Participate in Agile delivery activities including sprint planning, backlog refinement, design reviews, and continuous improvement initiatives
Skills
- 4+ years of experience as a Data Engineer or in a similar role building and supporting production-grade data pipelines and data products
- Hands-on experience with Apache Spark for large-scale data processing and transformation
- Strong proficiency in Python, SQL, and modern data engineering best practices
- Experience developing ETL/ELT solutions within cloud-based data lake, lakehouse, or analytics platforms
- Experience designing and optimizing analytical data models that support reporting, dashboards, operational insights, and advanced analytics
- Strong understanding of data quality, validation, monitoring, reconciliation, and production support processes
- Experience with CI/CD pipelines, version control, automated testing, and infrastructure-as-code concepts
- Demonstrated ability to troubleshoot complex data quality, performance, and integration issues across multiple data sources
- Strong verbal and written communication skills with the ability to explain technical concepts to both technical and non-technical audiences
- Experience with Microsoft Fabric, OneLake, or similar modern analytics platforms
- Experience working with risk, controls, compliance, audit, governance, or regulatory data domains
- Experience building real-time, near real-time, or event-driven data processing solutions
- Familiarity with data governance, metadata management, data lineage, access controls, and enterprise data quality frameworks
- Experience with orchestration and workflow management tools for data pipelines
- Experience supporting operational reporting, executive dashboards, trend analysis, advanced analytics, or machine learning initiatives
- Interest in building governed, scalable datasets that support AI and data-driven products
- Apache Spark, Data Engineering, Data ETL, Python (Programming Language), Software Development, Structured Query Language (SQL)
- Experience with Microsoft Fabric, OneLake, or similar modern analytics platforms preferred
Benefits
- Fully remote work
- Comprehensive technology setup, including a laptop, monitors, headset, keyboard, and mouse
- Employees eligible to work from home also receive a monthly connectivity reimbursement to help offset internet costs
Company Overview