Intermediate Data Engineer
Minimum Requirements:
- Matric (Grade 12)
- Masters in Software Engineering, Data Engineering, Computer Science or related field
- 2-3 years of relevant work experience
- Strong Scala or Java background
- Knowledge of AWS, GCP, Azure, or other cloud platform
- Understanding of data modeling principles
- Ability to work with complex data models
- Experience with relational and NoSQL databases (e.g. Postgres, ElasticSearch/OpenSearch, graph databases such as Neptune or neo4j)
- Experience with technologies that power analytics (Spark, Hadoop, Kafka, Docker, Kubernetes) or other distributed computing systems
- Knowledge of API development and machine learning deployment
Responsibilities:
- Build and scale data infrastructure that powers real-time data processing of billions of records in a streaming architecture
- Build scalable data ingestion and machine learning inference pipelines
- Build general-purpose APIs to deliver data science outputs to multiple business units
- Scale up production systems to handle increased demand from new products, features, and users
- Provide visibility into the health of our data platform (comprehensive view of data flow, resources usage, data lineage, etc) and optimize cloud costs
- Automate and handle the life-cycle of the systems and platforms that process our data
Originally posted on