Note: The job is a remote job and is open to candidates in USA. Lumen is a telecommunications and connectivity company providing secure, high-performance fiber network infrastructure for enterprise, government, and community customers. The Senior Software Engineer will modernize critical data platforms using PySpark and Databricks, while developing scalable data engineering solutions, supporting cloud migration and AI enablement, and contributing to agile delivery and engineering quality practices.
Responsibilities
- Partner in an Agile Scrum team environment to solution engineer, design, develop, code, test, debug, and support production-quality enterprise-grade systems
- Support cloud migration and data modernization initiatives, including migration of data warehouse capabilities to PySpark and Databricks
- Develop scalable, high-performance solutions that support large data volume processing and meet agreed SLAs for reliability, scalability, and performance
- Work closely with business partners and technical staff to create solutions that meet business, technical, compliance, and governance requirements
- Support AI enablement and automation initiatives, including work involving Databricks AI Genie where applicable
- Build and maintain data engineering solutions using Databricks, PySpark, Python, SQL, Azure Data Factory, Java, Confluent Kafka, and Bash scripting
- Collaborate with product developers and other engineering team members to define and implement effective automated testing and quality practices
- Document work clearly to promote transparency, supportability, and reuse
- Actively participate in code reviews and contribute to engineering best practices
- Prioritize and manage multiple concurrent projects, operational support needs, and delivery commitments
- Support regulated or critical infrastructure environments where compliance, security, and continuity of operations are required
Skills
- 6+ years of experience with a bachelor's degree (or equivalent) in Computer Science or a related field, or 4+ years with a master's degree. Strong experience with Databricks, PySpark, Apache Spark, Python, SQL, Azure Data Factory, ETL concepts, and distributed data processing. Proficiency in PostgreSQL (queries, migrations, performance tuning), Java, Spring Boot, REST APIs, and building AI-ready data pipelines with metadata governance and analytics enablement
- Experience with Kafka/event-driven architectures (Kafka Connect, JDBC connectors), Azure cloud technologies including AKS (Kubernetes), cloud networking, SSL/certificates, and deployments. Knowledge of CI/CD, GitHub, DevOps practices, Jira-based Agile delivery, software development methodologies, coding standards, and engineering best practices. Experience creating reconciliation, monitoring, and audit frameworks, as well as scalable, high-performance systems handling large data volumes
- Strong collaboration, communication, documentation, problem-solving, and time-management skills
- Highly motivated, organized, reliable, detail-oriented, and quick to learn new technologies
- Experience with Bash/Unix/Linux scripting; Informatica, Oracle RDBMS/PL/SQL, compliance-sensitive systems, and telecommunications industry experience are preferred
- Required Technical Skills: Databricks (including PySpark and AI Genie), Python, SQL, Azure Data Factory, Java, Confluent Kafka, Bash/Unix/Linux scripting, and Oracle RDBMS/PL/SQL (preferred)
Benefits
- Work from home position within the US
- A comprehensive package featuring a broad range of Health, Life, Voluntary Lifestyle benefits and other perks that enhance your physical, mental, emotional and financial wellbeing
Company Overview
Company H1B Sponsorship