Monitoring the health and status of nodes and servers is a critical part of effective infrastructure monitoring.
How to monitor node reboots?
One of the most critical tasks of monitoring an infrastructure is to check the health of its servers/nodes. In most cases, this results in setting up a “Hardware manager” from the hardware vendor delivering these servers or setting up an SNMP (or similar) agent to continuously monitor the availability of the server and report when there is a reboot / failure.
As important as these steps are, they are heavily infrastructure dependent - Bare metal/Cloud hosted servers, Linux/Windows OS, Hosted applications etc. And each of these come with their own monitoring software/tool:
- Hardware Manager - Delivered and maintained by the hardware vendor.
- Cloud Monitoring tools - Delivered and maintained by the Cloud provider and sometimes requires additional licenses to gain access to these tools.
- Software embedded tools - Delivered and maintained by the product house creating the product (integration with Zabbix, Prometheus or in-house software)
In a complex infrastructure setup, there is a necessity to have a single pane view for monitoring all the components of your infrastructure: hardware, OS, Cloud infrastructure, applications, etc. With Netdata, we strive to achieve this!
Uptime for monitoring node reboots
Uptime is the duration of time that a system/node has been up and running.
Netdata provides a system.uptime metric which continuously monitors the uptime of all the nodes in your infrastructure. The uptime for any node that reboots will instantly roll back to 0. The default setting of the chart shows the average time for which all the nodes in your infrastructure have been up. But, this is not really too helpful when you have hundreds of nodes.
If you change the group by option by node, you can see a nice stacked graph showing you the uptimes of all your nodes.
If any of the nodes reboot, you will instantly see a change in the uptime metric specific to the node. For example in the example below, the node “satya-vm” was up and running for 34 days and a bit before it was rebooted and you can instantly see the metric dropping.
This is all good if you are going to monitor your system 24/7 but in reality you want to be notified when a node reboots and you can write a simple custom alert by monitoring the system.uptime metric.
alarm: system_uptime
on: system.uptime
os: linux
lookup: min -1s
hosts: *
units: seconds
every: 1m
crit: $this < 300
delay: down 15m multiplier 1.5 max 1h
info: system uptime (time from last system reboot)
The sample alert above monitors the “system.uptime” context, looks up the minimum value in the last 1 second, and raises a critical alert when the time since last reboot is less than 5 minutes. And the alert will remain active for 20 minutes (5 + the 15 minute down in the delay hysteresis) unless it is rebooted within this interval.
In case of ephemeral environments that spin up and terminate hosts constantly, it can be challenging to distinguish new hosts from rebooted hosts. You can use placeholder alerts in your alert definition to only alert when the uptime of an existing host goes down.
alarm: last_uptime_val
on: system.uptime
lookup: max -1s at -300s unaligned
calc: $this
every: 15s
alarm: system_uptime
on: system.uptime
os: linux
lookup: min -1s
hosts: *
units: seconds
every: 1m
crit: $this < $last_uptime_val AND $last_uptime_val != nan
delay: down 15m multiplier 1.5 max 1h
info: system uptime (time from last system reboot)
Additionally, you can also monitor the reachability notifications which indicate a connection lost between the Netdata agent and the cloud and in some cases indicate a node being rebooted / down - Read more.
Let us hear from you
If you haven’t already, sign up now for a free Netdata account!
We’d love to hear from you – if you have any questions, complaints or feedback please reach out to us on Discord or Github.
Happy Troubleshooting!
