Note: The job is a remote job and is open to candidates in USA. Zignal Labs is a company that provides real-time intelligence technology to help organizations protect their people and assets. They are seeking a Data Engineer to build and operate data pipelines that process unstructured data and support their AI systems, ensuring reliable and efficient data flow for mission-critical decision making.
Responsibilities
- Build and maintain pipelines. Develop and operate batch and streaming pipelines that ingest and enrich high-volume unstructured data. Own components end to end, from implementation through production monitoring
- Support our AI systems. Build and extend the data pathways that feed downstream NLP, LLM, and retrieval services — including data preparation, embedding generation, and indexing workflows
- Work with search and storage layers. Integrate with and tune our search and vector stores to support semantic search, clustering, and real-time retrieval
- Build services and APIs. Implement and improve the microservices and APIs that deliver analytics to enterprise customers
- Operate what you build. Write clean, tested, maintainable code. Participate in code review, CI/CD, and infrastructure-as-code practices. Debug production issues and improve reliability over time
- Collaborate across teams. Work with Data Science, ML, Product, and Security to take ideas from prototype into production
Skills
- 3+ years building and operating data pipelines in production
- Strong programming skills in a JVM language — Scala, Java, or Kotlin. Our core pipeline code is Scala. If you're strong in Java or Kotlin and want to learn Scala, we'll support that; we care more about your fundamentals than your current syntax
- Working proficiency in Python for data and scripting work
- Hands-on experience with a distributed processing framework, most likely Apache Spark
- Hands-on experience with a streaming platform, most likely Kafka — including a real understanding of consumer groups, offsets, partitioning, and what happens when things fall behind
- Practical AWS experience and comfort with Docker. You should be able to work in a Kubernetes environment; you don't need to administer one
- Experience with a workflow orchestrator such as Airflow, Prefect, or Dagster
- Solid SQL and experience with at least one NoSQL or caching layer (Redis, MongoDB, DynamoDB, or similar)
- Sound CS fundamentals — data structures, algorithms, and the judgment to reason about performance and correctness in a distributed setting
- Strong written communication and the ability to work asynchronously across U.S. time zones. We're fully remote; writing clearly is part of the job
- Databricks or Delta Lake specifically
- Flink, or other stream-processing frameworks beyond Kafka
- Vector databases (Pinecone, Qdrant, Milvus, pgvector) and hands-on RAG or embedding pipeline work
- Elasticsearch or OpenSearch
- Experience parsing messy, unstructured, or multilingual text at scale
- Deep database performance tuning, or experience with distributed consensus systems
- Bachelor's degree in Computer Science, Engineering, or a related field
Company Overview
Company H1B Sponsorship