Note: The job is a remote job and is open to candidates in USA. WorkTrust Solutions is seeking an experienced Geospatial Data Engineer to support large-scale wildfire modeling and grid safety initiatives. The role focuses on building and optimizing scalable data pipelines, Palantir Foundry workflows, and distributed geospatial processing solutions for utility, asset, outage, and location-based datasets used in wildfire risk analysis and operational decision-making.
Responsibilities
- Design, build, optimize, and support high-performance automated data pipelines in AWS and cloud environments
- Utilize PySpark and Python to ingest, transform, and process large-scale utility asset, outage, and geospatial datasets
- Build scalable ETL/data engineering solutions capable of handling large historical and location-based datasets
- Optimize pipelines for performance, reliability, scalability, and maintainability
- Develop and support automated end-to-end data workflows within Palantir Foundry
- Build and maintain Foundry data transformations, pipelines, and code repositories
- Work within the Foundry ecosystem to prepare and deliver datasets supporting wildfire models and downstream analytical applications
- Troubleshoot and optimize existing Foundry pipelines and data processing workflows
- Implement advanced distributed geospatial data processing using Apache Sedona
- Utilize spatial libraries and frameworks such as GeoPandas, Shapely, and Apache Sedona/GeoSpark
- Process, manipulate, and analyze large volumes of ArcGIS and other location-based data
- Develop efficient approaches for partitioning and processing extremely large spatial datasets
- Perform and optimize spatial joins, indexing, geometry operations, and other distributed geospatial workloads
- Support the underlying data engineering capabilities and datasets used by PG&E's existing wildfire models
- Ensure geospatial and utility data is appropriately processed, transformed, validated, and delivered to support wildfire-related analysis and operational decision-making
- Interface with systems and datasets supporting:
- Remote Inspections
- Public Safety Power Shutoffs (PSPS)
- LiDAR-driven Vegetation Management
- Asset Risk Modeling
- Work with large-scale utility asset and outage data used in grid safety and wildfire risk initiatives
- Develop efficient strategies for partitioning very large geospatial datasets
- Optimize distributed data processing to improve performance and scalability
- Work with complex database structures, network topology, and spatial analytics frameworks
- Support large historical datasets and enterprise-scale data models
- Troubleshoot data quality, pipeline, geometry, performance, and scalability issues
- Participate actively in Agile/Scrum ceremonies
- Apply strong software engineering principles including:
- Unit testing
- CI/CD
- Source/version control
- Code reviews
- Reusable and maintainable development practices
- Collaborate with Data Scientists, Engineers, GIS specialists, modeling teams, and other stakeholders supporting PG&E's wildfire and grid safety initiatives
Skills
- • Bachelor's degree in Computer Science, Engineering, GIS , or another related quantitative/technical discipline
- • 5+ years of experience working within Data Engineering, ETL, or large-scale data processing ecosystems
- • Strong hands-on proficiency with:
- • PySpark
- • Python
- • SQL
- • Apache Sedona
- • Demonstrated experience building scalable data pipelines for large and complex datasets
- • Strong understanding of distributed data processing and performance optimization
- • Strong hands-on experience working with large-scale geospatial and spatial datasets
- • Experience with geospatial frameworks/libraries including:
- • Apache Sedona / GeoSpark
- • GeoPandas
- • Shapely
- • Experience working with ArcGIS-related or comparable enterprise geospatial datasets
- Strong understanding of Coordinate Reference Systems (CRS) , including:
- • EPSG codes
- • NAD83
- • WGS84
- • Coordinate transformations and projections
- Candidates should understand how differences between coordinate systems impact spatial processing, transformations, joins, distance calculations, and analytical results
- Hands-on knowledge of common vector and spatial data formats, including:
- • Shapefile
- • GeoJSON
- • GeoParquet
- • GeoPackage
- • KML
- Experience working with raster formats including:
- • GeoTIFF
- • Cloud Optimized GeoTIFF (COG)
- Strong understanding of distributed spatial indexing and partitioning concepts, including:
- • Spatial joins
- • R-trees
- • Grid indexing
- • Quadtree indexing
- • Broadcast joins vs. partitioned joins
- • Apache Sedona partitioning and optimization techniques
- Candidates should understand how to determine the appropriate processing and partitioning strategy for very large geospatial datasets
- Experience performing and optimizing large-scale geometry operations including:
- • Buffers
- • Intersections
- • Nearest-neighbor calculations
- • Topology validation
- • Geometry simplification
- • Handling invalid geometries
- • Handling self-intersecting geometries
- The consultant should understand both the functional and performance implications of executing these operations across large distributed datasets
- Experience with workflow orchestration technologies such as:
- • Airflow
- • Dagster
- • Palantir Foundry-native orchestration/workflow capabilities
- • Comparable enterprise orchestration platforms
- Strong understanding of enterprise Data Engineering concepts including:
- • Dimensional data modeling
- • Slowly Changing Dimensions (SCD)
- • Historical data management
- • Large-scale data warehousing
- • ETL/ELT patterns
- • Data quality and validation
- Candidates with comparable experience on large-scale cloud or enterprise big-data platforms may also be considered
- Experience developing automated data transformations, pipelines, workflows, and code repositories within an enterprise data platform
- • Experience dealing with:
- • Complex database structures
- • Network topology
- • Spatial analytics frameworks
- • Large-scale geometric datasets
- • Ability to use technologies such as Apache Sedona/GeoSpark to efficiently resolve and process complex spatial datasets
- • Hands-on experience working within Palantir Foundry is strongly preferred
- • Previous Palantir Foundry Data Engineering experience
- • Experience supporting utility, energy, infrastructure, or other asset-intensive organizations
- • Experience working with wildfire modeling or wildfire risk data
- • Experience with electric utility asset and outage datasets
- • Experience with Public Safety Power Shutoffs (PSPS)
- • Experience with LiDAR and vegetation management datasets
- • Experience supporting asset risk modeling
- • Experience with ArcGIS and enterprise GIS environments
- • Experience supporting Data Science, predictive modeling, or risk modeling teams
Company Overview