How to setup monitoring on AKS Cluster using Prometheus and Grafana | Setup monitoring on AKS Cluster using Prometheus and Grafana | Monitor Azure Kubernetes Service(AKS) with Prometheus and Grafana

How to setup monitoring on AKS Cluster using Prometheus and Grafana?

What is Azure Log Analytics Workspace?

Azure Log Analytics workspace is a central place where you can monitor and manage all the Azure Monitor logs collected from one or more Azure resources such as Azure Virtual machine or Azure Kubernetes Service. You can think of the workspace as a folder where all your monitoring data is stored.

What is Prometheus?

• Prometheus is an open source monitoring and alerting tool.
• Provides out-of-the-box monitoring capabilities for the Kubernetes container orchestration platform. It can monitor servers and databases as well.
• Collects and stores metrics as time-series data, recording information with a timestamp 
• It is based on pull and collects metrics from targets by scraping metrics HTTP endpoints.

What is Grafana?

• Grafana is an open source visualization and analytics tool. 
• It allows you to query, visualize, alert on, and explore your metrics no matter where they are stored.

Installation Method:

The are are many ways you can setup Prometheus and Grafana. You can install in following ways:

1. Create all configuration files of both Prometheus and Grafana and execute them in right order.

2. Prometheus Operator - to simplify and automate the configuration and management of the Prometheus monitoring stack running on a Kubernetes cluster

3. Helm chart (Recommended) - Using helm to install Prometheus Operator including Grafana

Why to use Helm?

Helm is a package manager for Kubernetes. Helm simplifies the installation of all components in one command. Install using Helm is recommended as you will not be missing any configuration steps and very efficient. 

Pre-requisites:

• Azure subscription, click here if you don't have one.
• Install Azure CLI on your local machine
• Install Helm3

Create AKS Cluster

Make sure you are login to Azure portal first.

az login

enter your Microsoft credentials.

Create a resource group first

az group create --name myResourceGroup --location southcentralus

Create Log Analytics workspace

az monitor log-analytics workspace create --resource-group myResourceGroup
                                           --workspace-name my-loganalytics-workspace \
                                           --query id \
                                           -o tsv)

Output of the above command will display log analytics Id which is needed for next command while creating AKS cluster

Create AKS cluster with 2 worker nodes with Monitoring Enabled

az aks create --resource-group myResourceGroup --name myAKSCluster --node-count 2 --enable-addons monitoring --workspace-resource-id /subscriptions/XXXXX/resourceGroups/myResourceGroup/providers/Microsoft.OperationalInsights/workspaces/aks-loganalytics-workspace

Verify all the resources are created in Azure Portal

Click on resource group name, you will see AKS cluster, log analytics workspace

Display Details of Cluster

az aks show --name myAKSCluster --resource-group myResourceGroup

The above command will display Cluster details.

Connect to the cluster

az aks get-credentials --resource-group myResourceGroup --name myAKSCluster --overwrite-existing

To verify the connection to your cluster, use the kubectl get command to return a list of the cluster nodes.

kubectl get nodes

 

 

Implementation steps

We need to add the Helm Stable Charts for your local client. Execute the below command:

helm repo add stable https://charts.helm.sh/stable

# Add prometheus Helm repo

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# Search for newly installed repositories

helm repo list

helm search repo prometheus-community

Prometheus and grafana helm chart moved to kube prometheus stack

Create Prometheus namespace
kubectl create namespace prometheus

Install kube-prometheus-stack

Below is helm command to install kube-prometheus-stack. The helm repo kube-stack-prometheus (formerly prometheus-operator) comes with a grafana deployment embedded.

helm install stable prometheus-community/kube-prometheus-stack -n prometheus

Lets check if prometheus and grafana pods are running already

kubectl get pods -n prometheus

kubectl get svc -n prometheus

This confirms that prometheus and grafana services have been created successfully using Helm. To access them, you will need to run the following commands:

Login to Grafana and Prometheus

Expose Prometheus

# Port forward the Prometheus service

kubectl port-forward -n prometheus prometheus-stable-kube-prometheus-sta-prometheus-0 9090

Now Open localhost:9090 in browser to access Prometheus

Expose Grafana

# Port forward the Grafana service

kubectl port-forward  -n prometheus stable-grafana-XXX 3000

Now Open localhost:3000 in browser to access Grafana

UserName: admin

Password: prom-operator

Create a Service Principal and Assign Role

We need to create Service principal(SPN) and assign Monitoring Reader Role on the AKS Cluster ResourceGroup. Execute below command to 

First Let's get Resource Group ID 

az group show --name  myResourceGroup --query id --output tsv

output of the above command will be resource group ID. Create service principal now:

az ad sp create-for-rbac --role="Log Analytics Reader" --scopes="/subscriptions/xxxxxx-xxxx-xxxx-xxxxx/resourceGroups/myResourceGroup"

Creating 'Log Analytics Reader' role assignment under scope '/subscriptions/XXX/resourceGroups/myResourceGroup'

{
  "appId": "xxx",
  "displayName": "azure-cli-2023-04-12-22-06-28",
  "password": "xxx",
  "tenant": "xxx"
}

Note all the information above and save it.

Create Data Source in Grafana

Go to Grafana, configuration and click Data Sources. Click on Add Data Source and search for Azure Monitor.

Enter tenant id, app id and secret information. click on Load subscriptions. Click on Save and Test.

Create Dashboard in Grafana

In Grafana, we can create various kinds of dashboards as per our needs.

How to Create Azure Monitor For Containers Dashboard?

Click '+' button on left panel and select ‘Import’.

Enter 10956 dashboard id under Grafana.com Dashboard.

Click ‘Load’.

Click ‘Import’.

This will show monitoring dashboard for all cluster nodes

Clean up Resources

Let's see how to clean up the resources that were created. We can use the az group delete command to remove the resource group, AKS cluster, and all related resources. 

az group delete --name myResourceGroup --yes --no-wait