Continuous Monitoring using Prometheus and Grafana

Till now I have shown “How to make webserver automatic” “How to integrate webserver on Kubernetes” . But what about monitoring. How will we monitor or keep an eye.

Monitoring is a necessity in today’s world.

In this article I will try to show you how to monitor using Prometheus and Grafana.

I hope you have some knowledge about K8s and promql.

So, let’s proceed

Creating images ( Using Dockerfile )

Prometheus

For reference: https://prometheus.io/docs/prometheus/latest/storage/

FROM    centos:latestRUN    yum install wget -y
RUN wget https://github.com/prometheus/prometheus/releases/download/v2.19.0/prometheus-2.19.0.linux-amd64.tar.gz
RUN tar -xzf prometheus-2.19.0.linux-amd64.tar.gz
RUN mkdir -p /metrics
CMD [ "./prometheus-2.19.0.linux-amd64/prometheus", \
"--config.file=/prometheus-2.19.0.linux-amd64/prometheus.yml", \
"--storage.tsdb.path=/metrics" ]

We will create prometheus image on top of centos. We will create a new directory metrics for storing metrics generated by Prometheus.

We will upload the image to DockerHub, so, you can directly download the image from there.

I have used jenkins for building and publishing image because we also want automation.

Grafana

FROM centos:latestRUN yum install wget -y
RUN wget https://dl.grafana.com/oss/release/grafana-7.0.3-1.x86_64.rpm
RUN yum install grafana-7.0.3-1.x86_64.rpm -y
WORKDIR /usr/share/grafana
CMD /usr/sbin/grafana-server start && /usr/sbin/grafana-server enable && /bin/bash

We will create grafanaimage on top of centos.

We will upload the image to DockerHub, so, you can directly download the image from there.

Create a PersistentVolume

apiVersion: v1kind: PersistentVolumemetadata:
name: prom-pv-vol
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/sdr/data"

The configuration file specifies that the volume is at /mnt/sdr/data on the cluster's Node. The configuration also specifies a size of 5 gibibytes and an access mode of ReadWriteOnce, which means the volume can be mounted as read-write by a single Node. It defines the StorageClass name manual for the PersistentVolume, which will be used to bind PersistentVolumeClaim requests to this PersistentVolume.

#Create the PersistentVolumekubectl create -f prom-pv-vol.yml

Create a PersistentVolumeClaim

The next step is to create a PersistentVolumeClaim. Pods use PersistentVolumeClaims to request physical storage. In this exercise, you create a PersistentVolumeClaim that requests a volume of at least three gibibytes that can provide read-write access for at least one Node.

apiVersion: v1kind: PersistentVolumeClaimmetadata:
name: prom-pv-volclaim
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi

Create the PersistentVolumeClaim:

kubectl create -f prom-pv-vol-claim.yml

Create Deployment

Prometheus

apiVersion: apps/v1kind: Deploymentmetadata:
name: prom-deploy
labels:
app: prometheus
spec:
replicas: 1
selector:
matchLabels:
app: prometheus
template:
metadata:
name: prom-deploy
labels:
app: prometheus
spec:
volumes:
- name: prom-pv-vol
persistentVolumeClaim:
claimName: prom-pv-volclaim

containers:
- name: prometheus
image: rajnishthegreat/prometheus:v2
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9090
volumeMounts:
- mountPath: "/metrics"
name: prom-pv-vol

Start Deployment:

kubectl create -f prom-deploy.yml#For exposing
kubectl expose deployment prom-deploy --port=3000 --type=NodePort

Grafana

apiVersion: apps/v1kind: Deploymentmetadata:
name: grafana
labels:
app: grafana
spec:
selector:
matchLabels:
app: grafana
template:
metadata:
name: grafana
labels:
app: grafana

spec:
containers:
- name: grafana
image: rajnishthegreat/grafana:v1
imagePullPolicy: IfNotPresent
restartPolicy: Always

Start Deployment:

kubectl create -f grafana.yml#For exposing to outside world
kubectl expose deployment grafana --port=3000 --type=NodePort

Grafana and Prometheus are to integrate and monitor

Monitoring

Prometheus is a free software application used for event monitoring and alerting. It records real-time metrics in a time series database built using a HTTP pull model, with flexible queries and real-time alerting.

But monitoring by keeping an eye on metrics isn’t a good idea. So, we integrate it with grafana

Grafana is an open source visualization tool that can be used on top of a variety of different data stores but is most commonly used together with Graphite, InfluxDB, Prometheus, Elasticsearch and Logz.io.

Configuring Grafana

  • Default Username: admin
  • Default Password: admin
  • You will get a screen to change password
  • We have to goto Data Source and select Prometheus and provide prometheus URL
  • Now we are ready to Monitor

Example

You will have to learn PromQL for quering.

We can can now view the real-time Metrics information visuals through Grafana using PromQl. We also have created Persistent storage so that even if the Pods crash due to some reason, Prometheus will automatically retrieve the data back.

I hope I was able to explain the project to you

For any query DM or comment below.

Any suggestion are welcomed