Note: The job is a remote job and is open to candidates in USA. URUS Group is a holding company serving the dairy and beef industry through genetics, reproductive services, and dairy management solutions. The Senior Data Engineer will design, develop, and maintain Databricks-based data integration, warehousing, reporting, and analytics solutions supporting reporting and AI/ML workloads.
Responsibilities
- Design, develop, and maintain robust and efficient ETL pipelines and processes on Databricks
- Troubleshoot and resolve Databricks pipeline errors and performance issues
- Troubleshoot and resolve SSIS package errors and performance issues
- Optimize data flow performance and minimize data latency
- Implement data quality checks and validations within ETL processes
- Maintain legacy SSIS packages for ETL processes
- Develop and maintain Databricks pipelines and datasets using Python, Spark and SQL
- Drive the design and implementation of batch and streaming data lake architectures supporting both analytics and AI/ML workloads
- Optimize Databricks jobs for performance and cost-effectiveness
- Integrate Databricks with other data sources and systems
- Migrate legacy systems (e.g. SSIS packages, SQL Server procs) to Databricks pipelines
- Participate in the design and implementation of data warehousing
- Support data quality initiatives and implement data cleansing procedures
- Collaborate with business users to understand data requirements for department driven reporting needs
- Maintain existing library of complex SSRS reports, dashboards, and visualizations
- Troubleshoot and resolve SSRS report issues, including performance bottlenecks and data inconsistencies
- Comfortable in entrepreneurial, self-starting, and fast-paced environment, working both independently and with our highly skilled teams
- Collaborate effectively with business users, data analysts, and other IT teams
- Communicate technical information clearly and concisely, both verbally and in writing
- Document all development work and procedures thoroughly
- Keep abreast of the latest advancements in data integration, reporting, data engineering
- Actively contribute to company’s AI-assisted tooling, strategies, and best practices
- Continuously improve skills and knowledge through training and self-learning
Skills
- Bachelor's Degree in Computer Science, Information Systems, or a related field
- 7+ years of experience in data integration and reporting
- Extensive experience with Databricks, including Python, Spark, and Delta Lake
- Experience with data warehousing concepts and best practices
- Strong proficiency with AI-assisted development tools
- Strong proficiency in SQL Server, including T-SQL, stored procedures, and functions
- Experience with SSIS (SQL Server Integration Services) development and maintenance
- Experience with SSRS (SQL Server Reporting Services) report design and development
- Strong analytical and problem-solving skills
- Excellent communication and interpersonal skills
- Ability to work independently and as part of a team
- Experience with Agile methodologies
- Experience with Microsoft Azure cloud platform and Microsoft Fabric desirable
Company Overview
Company H1B Sponsorship