For a tool like Netdata, monitoring crashes and abnormal events extends far beyond bug fixing—it’s essential for identifying edge cases, preventing regressions, and delivering the most dependable observability experience possible. With millions of daily downloads, each event provides a vital signal for maintaining the integrity of our systems.

The Challenge with Traditional Solutions

Over the years, we’ve evaluated many monitoring tools, each with significant limitations:

We consistently encountered these core challenges:

• Customization Difficulties: Off-the-shelf tools treat annotations as special cases, often missing the flexibility required for deep analysis.
• High Costs: A per-event pricing model can become prohibitively expensive at scale.
• Effortful Reporting: Generating nuanced, detailed reports required a lot of manual intervention and often didn’t capture the full complexity of our system’s behavior.

Identifying the Requirements

For Netdata, we required a solution offering:

• Comprehensive Ingestion: A system capturing every event without sampling, preserving all critical details.
• Complete Customization: A flexible data structure that supports multidimensional analysis across all fields and parameters.
• Economic Scalability: A framework that expands without triggering prohibitive operational expenses.
• Superior Performance: Capacity to handle tens of thousands of events per second while maintaining optimal processing speed.

Our Solution: Simple, Powerful, and Efficient

After reviewing traditional options, we discovered an innovative approach that leverages existing infrastructure. Our solution transforms event monitoring through systemd’s journal:

Core Mechanism: Agent Status Tracking

The implementation required minimal development effort because we realized we already had all the necessary parts!

Each Netdata Agent records its operational status to disk, documenting whether it exited normally or crashed—and if crashed, capturing detailed diagnostic information. Upon restart, the Agent evaluates this status file to determine reporting requirements. When anonymous telemetry is enabled or a crash is detected, the Agent transmits this status to our agent-events backend. An intelligent deduplication system prevents redundant reporting, ensuring each unique event is logged only once per day per Agent.

Backend Processing Pipeline

The agent-events backend performs the crucial transformation:

First, a lightweight Go-based web server receives JSON payloads via HTTP POST and sends them to stdout. We selected this implementation for its exceptional performance characteristics after evaluating several options (including Python and Node.js alternatives).

{
  "message": "Netdata was last crashed while starting, because of a fatal error",
  "cause": "fatal on start",
  "@timestamp": "datetime",
  "version": 8,
  "version_saved": 8,
  "agent": {
    "id": "UUID",
    "ephemeral_id": "UUID",
    "version": "v2.2.6",
    "uptime": 20,
    "ND_node_id": null,
    "ND_claim_id": null,
    "ND_restarts": 55,
    "ND_profile": [
      "parent"
    ],
    "ND_status": "initializing",
    "ND_exit_reason": [
      "fatal"
    ],
    "ND_install_type": "custom"
  },
  "host": {
    "architecture": "x86_64",
    "virtualization": "none",
    "container": "none",
    "uptime": 1227119,
    "boot": {
      "id": "04077a68-8296-4abf-bd77-f20527bb5cde"
    },
    "memory": {
      "total": 134722347008,
      "free": 51673772032
    },
    "disk": {
      "db": {
        "total": 3936551493632,
        "free": 1999957258240,
        "inodes_total": 244170752,
        "inodes_free": 239307909,
        "read_only": false,
        "name": "Manjaro Linux",
        "version": "25.0.0",
        "family": "manjaro",
        "platform": "arch"
      },
      "fatal": {
        "line": 0,
        "filename": "",
        "function": "",
        "message": "Cannot create unique machine id file '/var/lib/netdata/registry/netdata.public.unique.id'",
        "errno": "13, Permission denied",
        "thread": "",
        "stack_trace": "stack_trace_formatter+0x196\nlog_field_strdupz+0x237\nnd_logger_log_fields.constprop.0+0xc4\nnd_logger.constprop.0+0x597\nnetdata_logger_fatal+0x18d\nregistry_get_this_machine_guid.part.0+0x283\nregistry_get_this_machine_guid.constprop.0+0x20\nnetdata_main+0x264c\nmain+0x2d __libc_init_first+0x8a\n__libc_start_main+0x85 _start+0x21"
      },
      "dedup": [
        {
          "@timestamp": "2025-03-01T12:23:43.61Z",
          "hash": 15037880939850199034
        }
      ]
    }
  }
}

Our processing pipeline uses this simple sequence:

web-server | log2journal json | systemd-cat-native

Both log2journal and systemd-cat-native are included with Netdata, making this solution immediately available to all users.

Data Transformation and Storage

The pipeline transforms complex nested JSON structures into flattened journal entries. Regardless of field count or nesting depth, log2journal expands every data point into a discrete journal field, preserving complete information while enabling powerful querying. To keep system logs clean, we isolate this process in a dedicated systemd unit with its own log namespace, providing operational separation and preserving analytical power.

Implementation and Results

The entire implementation process took only a few days, though we encountered several technical obstacles:

• Robust Stack Trace Capture: Obtaining reliable stack traces across various crash scenarios required multiple iterations of our signal handling mechanisms to ensure complete diagnostic information.
• Schema Optimization: Identifying the precise fields necessary for comprehensive root cause analysis involved careful refinement of the JSON schema sent to the agent-events backend.

Once these challenges were resolved, this approach produced remarkable results. The Netdata Logs UI transformed into a powerful analytical platform. It became an exceptional analytical environment, allowing us to filter, correlate, and investigate any combination of events through powerful journald queries. We gained unprecedented visibility into our codebase’s behavior across diverse environments and configurations.

By implementing this system, we identified and resolved dozens of critical issues, dramatically improving Netdata’s reliability. The solution proved highly scalable—each backend instance processes approximately 20,000 events per second, with the ability to deploy additional instances as needed. From a resource perspective, the system requires only modest computing resources and storage capacity to maintain appropriate retention periods.

Conclusion

Our monitoring journey taught us that effective systems must balance customization, cost, and performance.

By combining a lightweight HTTP server with log2journal and systemd’s journald, we created a solution that captures every event without sampling or compromise.

This straightforward approach provides powerful insights while eliminating per-event costs, proving that native tools and thoughtful engineering can outperform expensive specialized solutions.

The system has already helped us identify and resolve dozens of critical issues, significantly improving Netdata’s reliability for our users worldwide.

As we continue to grow, this foundation will evolve with us, providing the visibility we need to maintain the highest standards of software quality.

FAQ

Q1: Do you collect IPs?

No, we don’t need IPs, and we don’t collect any.

Q2: Why do you collect normal restarts?

We only collect normal restarts when anonymous telemetry is enabled (enabled by default, users can opt-out). And we only collect at most one normal restart per day per agent (actually, we collect only one of all kinds of restarts/crashes per agent per day).

We collect normal restarts to understand the spread of an issue. We provide packages for a lot of Linux distributions, Linux kernel versions, CPU architectures, library versions, etc. For many combinations there are just a handful of users.

When we have just a few events reported, it is important to know if all of them are crashing, a few are crashing, or just a tiny percentage is crashing. It completely changes the context of the investigation needed.

So, the normal restarts act like a baseline for us to understand the severity of issues.

Q3: Why do you need the unique UUID of the agent?

UUIDs allow us to correlate events over time. How many events do we have for this agent? Is it always crashing with the same event? Was it working a few days ago? Did we fix the issue for this particular agent after troubleshooting?

Using these UUIDs, we can find regressions and confirm our patches actually fix the problems for our users.

Keep in mind that we don’t have the technical ability to correlate UUIDs to anonymous open-source users. So, these UUIDs are just random numbers for us.

Q4: Do you collect crash reports when anonymous telemetry is disabled?

No. We only collect crash reports when anonymous telemetry is enabled.

Q5: Does telemetry affect my system’s performance?

No. Telemetry is kicking in instantly when the agent starts, stops or crashes. Other than that, the agent is not affected.

Q6: Is any of my telemetry data shared with third parties or used for purposes beyond crash analysis?

All telemetry data is used exclusively for internal analysis to diagnose issues, track regressions, and improve Netdata’s reliability. We don’t share raw data or any personally identifiable information with third parties.

In some cases, these reports can be used to steer our focus. A great example is the breakdown per operating system that allows us to focus on providing binary packages for the distributions that are used more than others.

Q7: How long do you store the telemetry data?

We keep these data for 3 months, to identify how the different Netdata releases affect the stability and reliability of the agent over time.

Q8: Can I opt out of telemetry entirely, and what are the implications of doing so?

Of course. If you choose to disable it, you won’t contribute data that helps identify and resolve Netdata issues. Opting out might also mean you miss out on receiving some benefits derived from our improved diagnostics and performance enhancements.