Instructions to install prometheus
Requirements
Helm charts installed
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install my-kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 45.29.0
Make sure the pods are created. Check the status by running the command from post-installation guide
kubectl --namespace default get pods -l "release=my-kube-prometheus-stack"
The Helm chart for the Kube-prometheus stack will also create Kubernetes services, which you can
check using the command below:
kubectl get services -n default | grep kube-prometheus
Create services port-forward as below. Run all the port forwarding on background, by adding & at the
end of port-forward command:
Prometheus (ClusterIP port 9090) to host port 9090
kubectl port-forward service/my-kube-prometheus-stack-prometheus 9090:9090 &
Grafana (ClusterIP port 80) to host port 8080 (port 80 on host is already occupied by Traefik Ingress
Controller)
kubectl port-forward service/my-kube-prometheus-stack-grafana 8080:80 &
Alert Manager (ClusterIP port 9093) to host port 9093
kubectl port-forward service/my-kube-prometheus-stack-alertmanager 9093:9093 &
Testing with curl command
curl -i http://localhost:9090
Working with helm charts
helm list
helm list -n data-store
You can check the password value by using the following command:
kubectl get secret --namespace data-store solution-one-redis -o jsonpath="{.data.redis-password}" |
base64 -d
To check that the Redis FLUSHALL command is enabled, create a Redis client.
First, export the Redis password to environment variable:
export REDIS_PASSWORD=$(kubectl get secret --namespace data-store solution-one-redis -o
jsonpath="{.data.redis-password}" | base64 -d)
Run a Redis client on the namespace data-store
kubectl run --namespace data-store redis-client --restart='Never' --env
REDIS_PASSWORD=$REDIS_PASSWORD --image docker.io/bitnami/redis:7.0.11-debian-11-r7 --
command -- sleep infinity
Connect to running Redis client terminal:
kubectl exec --tty -i redis-client --namespace data-store -- bash
From Redis client, connect to Redis server (master node):
REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h solution-one-redis-master.data-store
Set one data with key: my-key and value: my-dummy-value which will expire in 5 minutes
SET my-key my-other-dummy-value EX 300
Make sure that the key exists
KEYS my-key
Try to flush all keys using the Redis command. This time, command FLUSHALL should work:
FLUSHALL
Make sure that the key was removed (this command should return an empty array):
KEYS my-other-key
Installing Prometheus Stack with personalized configuration
The configuration is quite complex, but basically you will need to create a yaml file on the server
containing the items below:
prometheus:
ingress:
enabled: true
annotations:
ingress.kubernetes.io/rewrite-target: /
paths:
- /prometheus
prometheusSpec:
routePrefix: /prometheus
grafana:
adminPassword: changeme
ingress:
enabled: true
annotations:
ingress.kubernetes.io/rewrite-target: /
path: /grafana
grafana.ini:
server:
root_url: "%(protocol)s://%(domain)s:%(http_port)s/grafana"
serve_from_sub_path: true
alertmanager:
ingress:
enabled: true
annotations:
ingress.kubernetes.io/rewrite-target: /
paths:
- /alertmanager
alertmanagerSpec:
routePrefix: /alertmanager
These configurations define the settings for the Ingress resources, including path mappings,
annotations, and specific configurations for Prometheus, Grafana, and Alertmanager within the kube-
prometheus stack installation.
Let's break down the contents:
a. prometheus section:
ingress: create the Ingress resource for Prometheus.
annotations: Specifies annotations for the Ingress resource, including ingress.kubernetes.io/rewrite-
target which rewrites the target URL path to "/".
paths: Defines the path(s) for the Ingress resource, in this case, "/prometheus".
prometheusSpec: Allows configuration of specific settings for Prometheus, such
as routePrefix which sets the URL route prefix to "/prometheus".
b. grafana section:
ingress: create the Ingress resource for Grafana.
annotations: Specifies annotations for the Ingress resource, including ingress.kubernetes.io/rewrite-
target which rewrites the target URL path to "/".
path: Defines the path for the Ingress resource, which is set to "/grafana".
grafana.ini: Configures settings for Grafana, such as root_url which specifies the base URL for
Grafana and serve_from_sub_path to serve Grafana from a subpath.
c. alertmanager section:
ingress: create the Ingress resource for Alert manager.
annotations: Specifies annotations for the Ingress resource, including ingress.kubernetes.io/rewrite-
target which rewrites the target URL path to "/".
paths: Defines the path(s) for the Ingress resource, in this case, "/alertmanager".
alertmanagerSpec: Allows configuration of specific settings for Alertmanager, such
as routePrefix which sets the URL route prefix to "/alertmanager".
Go to the terminal, and create the file values-monitoring.yml using the following command:
nano values-monitoring.yml
Install Kube-Prometheus stack using custom configuration. Use this command:
helm install solution-one-kube-prometheus --repo https://prometheus-community.github.io/helm-charts
kube-prometheus-stack --namespace monitoring --create-namespace --values values-monitoring.yml
Check that Kubernetes pods, services, and ingress, were created on the monitoring namespace
kubectl get pod -n monitoring | grep kube-prometheus
kubectl get service -n monitoring
kubectl get ingress -n monitoring
Try checking that the applications are accessible through ingress. Use curl to show only the
HTTP response status code, as the response body will be long (HTML). The response status code
should be 200 (meaning "OK")
curl -L -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1/prometheus/metrics
curl -L -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1/grafana
curl -L -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1/alertmanager
kubectl port-forward -n monitoring service/solution-one-kube-promethe-alertmanager 9093:9093 &
