AWS monitoring with Netdata

What is AWS?

AWS, or Amazon Web Services, is an expansive suite of cloud-based computing services that offer businesses of all sizes the ability to build, store, and manage data in the cloud. It is one of the largest and most successful cloud-computing services in the world. With its range of features and capabilities, AWS enables businesses to optimize their data storage and management, enabling them to scale as their needs grow. You can use AWS to store and access your data from anywhere, as well as to quickly deploy applications or databases without having to manage physical hardware.

Install Netdata on AWS

Netdata is fully compatible with Amazon Web Services (AWS). You can install Netdata on cloud instances to monitor the apps/services running there, or use multiple instances in a parent-child streaming configuration.

1. Sign up to https://app.netdata.cloud/
2. You will be presented with an empty space, and a prompt to “Connect Nodes” with the install command for each platform
3. Select the platform you want to install Netdata to, copy and paste the script into your node’s terminal, and run it

You can add one or more VMs or the entire Elastic Kubernetes Service (AWS EKS) Cluster on Netdata Cloud and monitor 100s of 1000s of system level and application metrics with zero configuration. All the AWS EC2 VMs (or the EKS Cluster) will show up on the Nodes tab.

What AWS metrics are important to monitor - and why?

Like with any cloud solution, it is absolutely essential to monitor the AWS system level metrics to keep a tab on the Cloud resources being utilised in addition to Operational and Performance metrics from the applications running on the Cloud.

1. Infrastructure monitoring: Monitor the underlying infrastructure of the cloud environment, such as the hardware, software, and networks that make up the cloud.
2. Availability Monitoring: Ensure that applications and services are available and accessible to users.
3. Performance monitoring: Ensure that applications are running smoothly by tracking resource usage over time such as CPU utilization and memory consumption and help optimize the cloud environment for better performance while reducing costs.
4. Security monitoring: Cloud security is crucial and thus monitoring for potential threats is a must. Security monitoring tools can help detect malicious activity such as data breaches, unauthorized access attempts, DDoS attacks, and more.
5. Capacity monitoring: Ensure that there is enough capacity to handle the workload without any performance issues and handle changes in workloads.
6. Usage Monitoring: Usage monitoring helps businesses understand how their cloud resources are being used and by whom to make decisions about usage policies, cost optimization strategies, and more.

AWS monitoring with Netdata using Cloudwatch

The Amazon Cloudwatch, is a powerful monitoring and logging service provided by Amazon Web Services (AWS). In order to holistically monitor AWS and all its hosted services, it is extremely important to gather and visualise metrics from Cloudwatch.

Netdata is capable of scraping metrics in the OpenMetrics format. There are a number of Prometheus collectors available to collect metrics from Amazon Cloudwatch and the following are the most famous ones:

1. Cloudwatch exporter - This is an AWS Java SDK based collector.
2. Yet another Cloudwatch exporter - YACE - This is written in Go and uses the official AWS SDK.

The setup and configuration details for these collectors are available in the links above. Once these are setup, you can immediately visualise all the needed AWS metrics with ready to use dashboards without the need to create them manually. You can configure the Netdata Prometheus endpoint collector to define labels, filter out metrics etc.

Cloudwatch is a pay-as-you-go service, and the pricing structure can be complex, with different features and metrics having different costs. The users must evaluate their use-case and the costs associated with monitoring their AWS infrastructure and the managed services. And this brings in the need for organizations to look at an alternative monitoring solution without fetching metrics from Cloudwatch.

AWS monitoring with Netdata

Netdata is a comprehensive monitoring solution and you can monitor and troubleshoot various aspects of your infrastructure including servers, VMs, Network, Disks, K8s, wide variety of applications (databases, gateways, web servers etc) with zero configuration, out of the box.

• Automatically gather system level metrics from operating system, containers, network, storage, processes, including the system calls these processes (with eBPF) do and it will automatically organize and correlate all the information in ready to use dashboards. AWS system level metrics from the EC2 VMs show up in the System Overview section on the Overview tab.
• Automatically gather operational and other performance metrics from almost every packaged application available including popular web servers such as Apache, Nginx, and HAProxy, databases like Redis, MongoDB, PostgreSQL, and MySQL/MariaDB etc and these metrics show up on the corresponding application sections on the Overview tab.
• Automatically gather metrics from your Kubernetes EKS Cluster and visualise all the Kubernetes Containers, States, Kubelet and Kubeproxy related metrics on the Kubernetes tab.

Troubleshooting with Netdata

Netdata offers powerful tools to optimize your troubleshooting and solve your response time issues faster than ever. Finding the proverbial needle in the haystack is a lot easier with Netdata.

• Alerts, the simplest way of troubleshooting your infrastructute. Netdata comes with 100s of pre-defined Alerts out of the box and allows you to create Custom Alerts based on your requirements and these can be notified through multiple notification mechanisms.

• Metrics Correlations, a tool that scans all metrics to find how they correlate in a given time-frame. This allows you to highlight an area with a spike or a dive on a chart and let Netdata find which other metrics have changed similarly at the same time.

• Anomaly Advisor, a tool that scans all metrics for anomalies during a given time-frame. This allows you to highlight an area with a spike or a dive on a chart and let Netdata find which anomalies were detected during that time-frame, across your infrastructure.

These tools can be of great help identifying and revealing anomalies in your infrastructure and to understand interdependencies among infrastructure components. For example, you can see how CPU utilization affects response time, how disk throughput affects database queries, how network bandwidth affects web requests etc.

You can also see how different applications interact with each other on the same server or across different servers. By correlating application metrics and server metrics you can identify root causes, troubleshoot problems, optimize performance, improve availability, enhance security etc on your infrastructure.