Note: The job is a remote job and is open to candidates in USA. Lumen Technologies is a trusted network provider for the AI-powered world, connecting people, data, and applications. The Senior Software Engineer will modernize critical data platforms using PySpark and Databricks, working in an agile environment to accelerate AI enablement and business innovation.
Responsibilities
- Partner in an Agile Scrum team environment to solution engineer, design, develop, code, test, debug, and support production-quality enterprise-grade systems
- Support cloud migration and data modernization initiatives, including migration of data warehouse capabilities to PySpark and Databricks
- Develop scalable, high-performance solutions that support large data volume processing and meet agreed SLAs for reliability, scalability, and performance
- Work closely with business partners and technical staff to create solutions that meet business, technical, compliance, and governance requirements
- Support AI enablement and automation initiatives, including work involving Databricks AI Genie where applicable
- Build and maintain data engineering solutions using Databricks, PySpark, Python, SQL, Azure Data Factory, Java, Confluent Kafka, and Bash scripting
- Collaborate with product developers and other engineering team members to define and implement effective automated testing and quality practices
- Document work clearly to promote transparency, supportability, and reuse
- Actively participate in code reviews and contribute to engineering best practices
- Prioritize and manage multiple concurrent projects, operational support needs, and delivery commitments
- Support regulated or critical infrastructure environments where compliance, security, and continuity of operations are required
Skills
- 6+ years of experience with a bachelor's degree (or equivalent) in Computer Science or a related field, or 4+ years with a master's degree
- Strong experience with Databricks, PySpark, Apache Spark, Python, SQL, Azure Data Factory, ETL concepts, and distributed data processing
- Proficiency in PostgreSQL (queries, migrations, performance tuning)
- Java, Spring Boot, REST APIs, and building AI-ready data pipelines with metadata governance and analytics enablement
- Experience with Kafka/event-driven architectures (Kafka Connect, JDBC connectors)
- Azure cloud technologies including AKS (Kubernetes), cloud networking, SSL/certificates, and deployments
- Knowledge of CI/CD, GitHub, DevOps practices, Jira-based Agile delivery, software development methodologies, coding standards, and engineering best practices
- Experience creating reconciliation, monitoring, and audit frameworks, as well as scalable, high-performance systems handling large data volumes
- Strong collaboration, communication, documentation, problem-solving, and time-management skills
- Required Technical Skills: Databricks (including PySpark and AI Genie), Python, SQL, Azure Data Factory, Java, Confluent Kafka, Bash/Unix/Linux scripting, and Oracle RDBMS/PL/SQL (preferred)
- Experience with Bash/Unix/Linux scripting
- Informatica
- Oracle RDBMS/PL/SQL
- Compliance-sensitive systems
- Telecommunications industry experience
Benefits
- Lumen offers a comprehensive package featuring a broad range of Health, Life, Voluntary Lifestyle benefits and other perks that enhance your physical, mental, emotional and financial wellbeing.
Company Overview
Company H1B Sponsorship