Note: The job is a remote job and is open to candidates in USA. Scicom Infrastructure Services is seeking a Data Engineer / Business Analyst to support a large-scale Databricks implementation. This hybrid role will bridge business, contracting, data-governance, and engineering teams to define business requirements, construct data contracts, map source data, and develop reliable Databricks data pipelines.
Responsibilities
- Lead requirements-gathering sessions with procurement, contracting, program, analytics, governance, and technical stakeholders
- Define, construct, document, and maintain data contracts between data producers and consumers
- Establish data-contract requirements covering: Dataset purpose and ownership, Source and target systems, Schemas, fields, and data types, Required and optional attributes, Business definitions and transformation rules, Data-quality expectations, Validation and reconciliation rules, Refresh frequency and delivery schedules, Versioning and schema-evolution requirements, Security classifications and access controls, Service-level expectations, Issue ownership and change-management procedures
- Translate business requirements into user stories, acceptance criteria, process flows, data mappings, interface specifications, and technical requirements
- Conduct source-system analysis, data profiling, gap assessments, and source-to-target mapping
- Identify differences between existing procurement data and required OCDS structures
- Facilitate agreement among data owners, producers, consumers, architects, and governance teams
- Maintain traceability from business requirements through data models, engineering implementation, testing, and acceptance
- Evaluate requested changes for downstream effects on data products, reports, integrations, and analytical use cases
- Support backlog refinement, sprint planning, demonstrations, testing, and stakeholder acceptance
- Apply the Open Contracting Data Standard to procurement and public-contracting datasets
- Map source-system data to appropriate OCDS fields and structures
- Work with data across the contracting lifecycle, including: Planning, Tender, Award, Contract, Implementation
- Develop and maintain mappings for OCDS releases, records, release packages, record packages, identifiers, organizations, parties, items, milestones, documents, transactions, amendments, and related contracting elements
- Interpret and apply OCDS schemas, codelists, validation rules, and implementation guidance
- Determine whether standard OCDS fields meet project requirements or whether documented extensions are necessary
- Support the construction of complete contracting records from multiple transactional releases
- Establish rules for handling amendments, updates, cancellations, corrections, and historical changes
- Validate transformed data against applicable OCDS JSON schemas
- Identify missing, invalid, inconsistent, or nonconforming procurement data and work with stakeholders to resolve deficiencies
- Document assumptions, business rules, mappings, extensions, and exceptions
- Support the publication, exchange, analysis, or internal use of standardized contracting data
- Design, develop, test, and maintain data pipelines using Databricks, Apache Spark, PySpark, Python, and SQL
- Build ingestion and transformation pipelines for structured and semi-structured procurement data
- Process JSON, CSV, XML, Parquet, relational database, API, and file-based data sources
- Implement Bronze, Silver, and Gold data layers using medallion architecture
- Build normalized, dimensional, analytical, and OCDS-aligned data models
- Use Delta Lake capabilities for schema enforcement, schema evolution, versioning, auditability, and reliable processing
- Develop reusable frameworks for mapping source procurement data into OCDS-compatible outputs
- Implement batch and incremental ingestion patterns
- Use Databricks Workflows, notebooks, jobs, Auto Loader, Delta Live Tables or Lakeflow capabilities, as appropriate
- Develop REST API integrations for source ingestion and downstream data delivery
- Implement automated data-quality, reconciliation, completeness, and conformity checks
- Support Unity Catalog implementation for metadata, ownership, lineage, access control, and data discovery
- Optimize Spark jobs, SQL queries, clusters, partitioning, file sizes, and data layouts
- Participate in code reviews, automated testing, CI/CD, deployment, monitoring, and production support
- Investigate pipeline failures, data discrepancies, and performance issues
- Define measurable quality rules for accuracy, completeness, validity, timeliness, consistency, and uniqueness
- Develop validation controls for required OCDS fields, identifiers, dates, amounts, currencies, organizations, classifications, and contracting relationships
- Create dashboards or reports that show data-contract compliance and data-quality results
- Establish processes for detecting and managing schema drift
- Document data lineage from original procurement systems through Databricks transformations and downstream products
- Work with governance teams to assign data owners, stewards, classifications, retention requirements, and access policies
- Ensure sensitive procurement and supplier information is handled according to security and privacy requirements
- Support auditability through documented rules, version-controlled mappings, validation results, and change histories
Skills
- Bachelor's degree in computer science, information systems, data analytics, business analysis, engineering, public administration, supply-chain management, or a related field
- At least five years of combined data-engineering, data-analysis, business-analysis, or data-integration experience
- Hands-on experience implementing or working with the Open Contracting Data Standard
- Demonstrated experience constructing, documenting, negotiating, or maintaining data contracts
- Experience mapping procurement or contracting data to OCDS schemas
- Strong knowledge of OCDS releases, records, schemas, codelists, identifiers, contracting stages, and validation practices
- At least three years of hands-on experience with Databricks
- Strong experience with: Apache Spark, PySpark, Python, SQL, Delta Lake, ETL and ELT pipelines, Data modeling, JSON and JSON Schema, REST APIs, Data-quality validation, Source-to-target mapping
- Experience gathering requirements and translating them into implementable engineering specifications
- Ability to communicate effectively with technical and nontechnical stakeholders
- Experience writing user stories, acceptance criteria, business rules, data dictionaries, interface specifications, and process documentation
- Strong analytical, troubleshooting, facilitation, and documentation skills
- Experience working within Agile delivery teams
- Experience with government procurement, public-sector contracting, grants, acquisition, supplier, or financial data
- Experience implementing OCDS extensions or tailoring OCDS for specific organizational requirements
- Familiarity with procurement classifications, organizational identifiers, tender processes, awards, amendments, milestones, transactions, and contract implementation
- Databricks Certified Data Engineer Associate or Professional certification
- Experience with Unity Catalog, Delta Live Tables, Lakeflow, Databricks Workflows, or Structured Streaming
- Experience with Microsoft Azure, AWS, or Google Cloud
- Experience with Azure Data Factory, ADLS Gen2, AWS Glue, Amazon S3, Snowflake, dbt, Kafka, Airflow, or Power BI
- Experience using JSON Schema validation tools and automated data-quality frameworks
- Knowledge of metadata management, master-data management, data lineage, and reference-data governance
- Experience with CI/CD, Git, Azure DevOps, GitHub Actions, Jenkins, or Terraform
- Experience supporting large federal, state, local-government, or regulated-enterprise data programs
- Familiarity with federal acquisition, procurement, reporting, transparency, or open-data requirements
Company Overview