Note: The job is a remote job and is open to candidates in USA. TechTrend is seeking a Senior Cloud Data Engineer / Data Architect to design, build, modernize, and support enterprise-scale cloud data platforms, primarily in the Microsoft Azure ecosystem. The role combines hands-on data engineering and architecture, including building production data solutions, establishing secure and governed data platforms, and enabling analytics and AI/ML use cases.
Responsibilities
- Architect and implement modern cloud data platforms and Lakehouse architectures using Azure and Databricks
- Design scalable data lakes, Lakehouse, data warehouse, and analytical architectures supporting structured, semi-structured, and unstructured data
- Build and optimize batch and streaming data pipelines using technologies such as Azure Databricks, Spark/PySpark, Azure Data Factory, Synapse pipelines, Event Hubs, and related services
- Design and implement solutions using ADLS Gen2, Delta Lake, Databricks SQL, Unity Catalog, Azure Synapse Analytics, and Azure SQL
- Design ingestion and integration patterns for data originating from databases, APIs, files, applications, cloud platforms, and legacy/on-premises systems
- Define data modeling strategies including dimensional models, curated data products, transformation layers, and analytical datasets
- Establish data governance, lineage, cataloging, access-control, data-quality, and security patterns, including Unity Catalog and Microsoft Purview where appropriate
- Design secure data architectures using Microsoft Entra ID, managed identities, RBAC, Key Vault, private endpoints, and least-privilege principles
- Work with AI/ML teams to prepare governed enterprise data for machine learning, Generative AI, RAG, analytics, and other AI-enabled use cases
- Support integration between Databricks and Azure AI/ML capabilities, APIs, applications, reporting platforms, and enterprise systems
- Evaluate performance, scalability, availability, security, and cost characteristics of proposed architectures
- Develop reusable engineering patterns, reference architectures, notebooks, pipelines, IaC, technical standards, and deployment documentation
- Troubleshoot complex data pipeline, integration, Spark, SQL, performance, security, and connectivity issues
- Collaborate directly with customers, architects, application developers, cybersecurity teams, cloud engineers, and business stakeholders
- Provide technical leadership and mentoring to data engineers while remaining actively involved in design and implementation
Skills
- The ideal candidate will combine strong data architecture expertise with the engineering depth necessary to build and troubleshoot production data solutions using Azure Databricks, Azure Data Lake Storage (ADLS Gen2), Azure Synapse Analytics, Azure Data Factory, SQL, Python/PySpark, and related Azure Data & AI services
- The candidate should be comfortable working across architecture, data engineering, analytics, governance, security, and AI/ML integration and translating business and mission requirements into scalable production solutions
- 7+ years of professional experience in data engineering, data architecture, analytics engineering, cloud engineering, or related disciplines
- 4+ years of hands-on experience designing or implementing cloud-based data platforms
- Strong hands-on expertise with Microsoft Azure data services
- Demonstrated experience with Azure Databricks and Apache Spark/PySpark
- Strong understanding of Lakehouse and Data Lake architecture, including Delta Lake concepts
- Experience with ADLS Gen2, Azure Synapse Analytics, Azure Data Factory, Azure SQL, or comparable Azure data services
- Strong SQL and Python/PySpark engineering skills
- Experience designing enterprise data ingestion, ETL/ELT, transformation, orchestration, and integration pipelines
- Experience with data modeling, schema design, performance optimization, partitioning, and large-scale dataset processing
- Understanding of Databricks Unity Catalog, data governance, lineage, RBAC, and access-control concepts
- Experience implementing secure cloud architectures for sensitive or regulated data
- Experience integrating cloud data platforms with downstream analytics, applications, APIs, and AI/ML workloads
- Strong analytical and troubleshooting skills
- Ability to translate business requirements into scalable technical architectures
- Excellent written and verbal communication skills, including the ability to explain architectural decisions and technical trade-offs
- Experience with Azure Government and/or federal government environments
- Previous experience supporting USDA or other federal civilian agencies
- Experience with Microsoft Fabric and Power BI
- Experience with Azure AI services, Azure Machine Learning, MLflow, Mosaic AI, Generative AI, RAG, vector search, or AI/ML pipelines
- Experience with real-time/streaming architectures using Kafka, Event Hubs, or Spark Structured Streaming
- Experience with Terraform, Bicep, GitHub, Azure DevOps, and CI/CD for data-platform deployment and automation
- Experience with Microsoft Purview or other enterprise data governance/catalog platforms
- Experience integrating Databricks with enterprise systems and APIs
- Familiarity with federal security and governance frameworks including NIST 800-53, FedRAMP, and FISMA
- Experience designing hybrid or cross-cloud data architectures
- Databricks Certified Data Engineer Professional
- Databricks Certified Data Engineer Associate
- Microsoft Azure data/AI certifications
- Azure architecture certifications
Benefits
- Medical, Dental & Vision coverage
- Life Insurance
- Short, Long Term Disability Insurance
- PTO & Federal Holidays Off
- 401(k) Plan (matching component included)
- Primarily remote, with occasional onsite attendance at our Reston Town Center office, up to one day per week.
Company Overview