Note: The job is a remote job and is open to candidates in USA. Brillio is hiring a Senior Data Engineer to build and scale its next-generation data and AI platform. The role focuses on developing scalable data pipelines and trusted, AI-ready data foundations while enabling analytics, knowledge management, and AI use cases with governance, performance, and compliance in mind.
Responsibilities
- Design, develop, and maintain highly scalable data pipelines on Databricks leveraging Delta Lake, Unity Catalog, and Databricks Workflows
- Ingest, integrate, and process large-scale datasets from research, commercial, content, operational, and third-party sources
- Implement batch and real-time data processing solutions to support enterprise analytics and AI initiatives
- Ensure reliable, secure, and high-performance data movement across cloud platforms and business systems
- Build the core data infrastructure supporting enterprise ontology, semantic layer, and knowledge graph initiatives
- Design and implement robust data models that enable efficient data discovery, contextual understanding, and relationship mapping
- Develop entity resolution frameworks to improve data consistency and create trusted views of business-critical information
- Create curated, reusable, and AI-ready data products that can be leveraged across multiple business domains
- Prepare and deliver high-quality datasets for Azure AI Foundry and advanced AI/ML applications
- Build and optimize chunking, embedding, vectorization, and retrieval pipelines for Generative AI and RAG-based solutions
- Develop feature engineering pipelines that support machine learning and predictive analytics initiatives
- Enable seamless data access for AI applications through real-time and batch processing frameworks
- Implement enterprise data governance standards using Unity Catalog
- Establish robust data quality controls, lineage tracking, metadata management, and auditing capabilities
- Define and maintain role-based access controls and security policies to ensure data privacy and regulatory compliance
- Collaborate with governance and compliance teams to support enterprise data management standards
- Monitor and optimize platform performance, scalability, and operational efficiency
- Implement cost optimization strategies across data storage, processing, and compute layers
- Develop reusable frameworks, templates, and engineering standards that accelerate onboarding of new data and AI use cases
- Champion engineering best practices including automation, CI/CD, testing, monitoring, and observability
Skills
- Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related field
- 7+ years of experience in Data Engineering, Data Platform Development, or Big Data environments
- Hands-on expertise with Databricks, including Delta Lake, Unity Catalog, Workflows, Spark, and Lakehouse architecture
- Strong experience building large-scale ETL/ELT pipelines and modern cloud data platforms
- Experience working with Azure Data Services and enterprise cloud environments
- Strong proficiency in Python, PySpark, and SQL
- Deep understanding of data modeling, metadata management, master data management, and data governance principles
- Experience designing and implementing enterprise-scale data architectures
Company Overview
Company H1B Sponsorship