Note: The job is a remote job and is open to candidates in USA. Bitdeer is a technology company focused on AI and Bitcoin mining infrastructure. The Staff AI Observability & Telemetry Engineer will architect and scale the telemetry and observability infrastructure for Bitdeer's AI-native NeoCloud platform, enabling real-time monitoring, diagnosis, remediation, and billing across GPU and network resources.
Responsibilities
- Architect and scale a high-cardinality telemetry infrastructure using highly available time-series databases (e.g., VictoriaMetrics, Thanos, or Mimir) capable of handling massive ingestion rates
- Integrate complex hardware-level exporters (NVIDIA DCGM, network switch telemetry, IPMI/Redfish) directly into the Kubernetes observability stack to provide a unified view of the cluster
- Build eBPF-based diagnostic tools to trace network congestion, kernel-level I/O latency, and distributed training bottlenecks across the cluster
- Develop automated dashboards and alerting pipelines that trigger proactive cordoning of degraded hardware before it impacts customer training jobs
- Design the metric pipelines required for accurate, multi-tenant consumption billing based on real-time GPU and network utilization metrics
- Collaborate with the GPU Systems and Scheduling teams to create observability standards for "AI-native" workloads, ensuring deep insight into job efficiency and resource utilization
- Lead technical design reviews for observability architecture, mentoring team members on best practices for high-performance telemetry collection and analysis
Skills
- Bachelor's or Master's degree in Computer Science, Electrical Engineering, or a related field
- 6+ years of software or site reliability engineering, with deep, hands-on expertise in the Prometheus/OpenTelemetry ecosystem
- Advanced proficiency in Go and extensive experience writing custom Kubernetes metric exporters and operators
- Hands-on experience with kernel-level tracing tools (eBPF, BCC) and deep performance tuning of Linux systems
- Strong familiarity with AI hardware metrics (GPU power states, SM utilization, memory bandwidth) and high-performance network telemetry
- Proven track record of operating, debugging, and scaling large-scale telemetry stacks in high-performance computing or cloud environments
- Strong technical leadership skills; ability to influence architectural decisions and align cross-functional teams around observability standards
- Excellent communication skills, with the ability to translate complex system requirements into manageable engineering milestones
- Experience working in high-velocity, high-growth engineering environments is strongly preferred
Benefits
- Full-Time employment
- Remote (within locations)
Company Overview