The Observability Engineer is responsible for designing, building, implementing, and administering the tools and pipelines that give the organization full-stack visibility into its infrastructure, applications, and business services. This role owns the day-to-day operation of the observability platform(s) and the NOC/TOC (Network/Technical Operations Center) function to ensure stability, reliability, availability, and performance of the services the business depends on.
Where traditional monitoring answered, “is it up or down,” this role is expected to build observability that answers “why is it behaving this way” — correlating metrics, logs, traces, and events into actionable insight, and reducing time-to-detect and time-to-resolve across the environment.
Duties and Responsibilities
Provides expertise and develops observability solutions for:
Network technology, connections, and cloud networking (SD-WAN, VPC/VNet constructs)
Directory services (Active Directory, LDAP, Entra ID/Azure AD)
Messaging and collaboration platforms (Exchange Online, Microsoft 365, Teams)
eCommerce and customer-facing digital environments
Remote access and VPN infrastructure
Wireless, mobile, and endpoint device fleets
Operating system environments (Windows, Linux)
Data backup, storage, and disaster recovery systems
VOIP and unified communications systems
Virtualized and containerized environments (VMware)
Public and hybrid cloud infrastructure (AWS, Azure, GCP)
Critical business services and transaction flows
Design, deploy, and maintain observability platforms across metrics, logs, traces, synthetic monitoring, and real user monitoring
Own and operate the NOC/TOC function end-to-end: alert intake, triage, escalation, incident coordination, and post-incident review
Develop and maintain monitoring/observability standards, naming conventions, and instrumentation guidelines across infrastructure and application teams
Build and tune alerting logic to reduce noise and alert fatigue; drive toward actionable, high-signal alerting
Define, track, and report on SLIs, SLOs, error budgets, and other key observability/operational KPIs
Develop dashboards and visualizations that give both technical teams and business stakeholders a clear picture of service health
Mentor technical staff and cross-functional teams on observability concepts, instrumentation practices, and tooling
Develop, document, and continuously improve monitoring and incident response processes and runbooks
Troubleshoot and resolve issues within the monitoring/observability stack itself (data gaps, false positives, agent/collector health)
Evaluate, pilot, and recommend new observability tools and platforms
Partner with Help Desk, Infrastructure, DevOps/SRE, and Application teams to implement monitoring, alerting, and response processes
Manage the call notification, paging, and escalation system for alerts
Lead capacity planning, performance trending, and fault prediction using historical observability data
Integrate observability data with automation/AIOps workflows to enable auto-remediation where appropriate
Support security operations by feeding relevant telemetry into SIEM/SOC tooling and collaborating on anomaly detection
After-hours work, shift work, and on-call rotation required
Scope
Staff supervision and development: No
Decision making:
Review data and provide recommendations on alert actions, monitoring and responses.
Travel: Up to 5%
Flex Designation: Anywhere
The anticipated salary range for this position is $87,600– $131,500 depending on location, knowledge, skills, education and experience. This position is also eligible for an annual discretionary bonus. In addition, we offer comprehensive and competitive benefits to Associates (and their families) such as medical, dental, vision, life insurance, short-term and long-term disability. Eligible Associates are able to enroll in our company’s 401k plan. Associates will accrue paid time off up to 236 hours per year (inclusive of PTO, floating holidays, and paid holidays). Paid sick time up to 80 hours per year unless otherwise required by law.
Requirements
Education and Experience
Education Requirements
Bachelors Degree in technical discipline, Computer Science or related field, or equivalent experience. Relevant Certifications are a bonus (Vendor-specific observability certifications, CCNA, AWS/Azure Certifications).
Years of Experience
2-4 years experience with network/systems management and observability tools.
Skills
Hands-on experience with modern observability/monitoring platforms such as Dynatrace, SolarWinds, Zabbix, Datadog, New Relic, Splunk, PRTG, or Nagios
Experience with log aggregation and analysis platforms i.e. Splunk
Familiarity with distributed tracing and APM tooling (Dynatrace, Jaeger, Honeycomb, AppDynamics)
Deep understanding of monitoring protocols and data collection methods: SNMP (v1/v2c/v3), MIB structure, OID notation, syslog, NetFlow/sFlow/IPFIX, WMI, JMX, and RESTful API-based telemetry
Solid understanding of networking fundamentals: routing, switching, TCP/IP, DNS, and network troubleshooting
Strong grasp of the “three pillars of observability” (metrics, logs, traces) and how they correlate to reduce MTTD/MTTR
Strong experience with Windows and Linux systems architecture and administration
Experience monitoring cloud-native and hybrid environments across AWS, Azure, and/or GCP
Proficiency in scripting/automation languages (Python, Bash, PowerShell)
Experience integrating monitoring data with automation, ticketing, and AIOps/anomaly-detection workflows
Demonstrated experience owning or operating within a NOC/TOC environment across its full lifecycle — detection, triage, escalation, resolution, and postmortem
Ability to develop quantitative, prioritized action plans to address operational and reliability issues
Strong incident management skills; familiarity with ITIL or similar frameworks a plus
Excellent written and oral communication and documentation skills
Ability to work after hours, shifts, and rotating on call as required
Physical Requirements
Corporate - Remote - General office environment requiring ability to:
Stand, walk, sit for extended periods of time .
Speak and listen to others in person and over the phone and video conferencing.
Use keyboard and read from computer screen and reports.
The ability to lift up to 15 lbs.
Safety
Must be able to perform this job safely in accordance with standard operating procedures and good manufacturing practices, without endangering the health or safety of self or others.
About Harbor Freight Tools
Founded in 1977, we are a $10 billion national tool retailer with the energy, enthusiasm, and growth mindset of a Championship Team. With more than 1,600 stores across 48 states, we're continuing to grow and open new locations every month.
We offer our customers more than 6,000 tools and accessories, from hand tools and generators to air and power tools, shop equipment, and automotive tools. By delivering quality and value to our customers, we've built a culture where Associates can learn, grow, and do meaningful work every day.
Observability Engineer - Remote | Harbor Freight Tools