Install Prometheus and Grafana using Daemonset

About Prometheus

Prometheus is an opensource monitoring and alerting toolkit. Prometheus consists of several components some of which are listed below:

The Prometheus server which scrapes(collects) and stores time series data based on a pull mechanism.
A rules engine which allows generation of Alerts based on the scraped metrics.
An alertmanager for handling alerts.
Multiple integrations for graphing and dashboarding.

This document walks you through the steps required to monitor your Portworx cluster with Prometheus and Grafana. Portworx natively supports Prometheus since it exposes the metrics on a REST endpoint. Then, Prometheus can scrape this endpoint.

The following instructions allows you to monitor Portworx via Prometheus and allow the Alertmanager to provide alerts based on configured rules.

The Prometheus Operator creates, configures and manages a prometheus cluster.

The Prometheus operator manages 3 customer resource definitions namely:

Prometheus: The Prometheus CRD defines a Prometheus setup to be run on a Kubernetes cluster. The Operator creates a Statefulset for each definition of the Prometheus resource.
ServiceMonitor: The ServiceMonitor CRD allows the definition of how Kubernetes services could be monitored based on label selectors. The Service abstraction allows Prometheus to in turn monitor underlying Pods.
Alertmanager: The Alertmanager CRD allows the definition of an Alertmanager instance within the Kubernetes cluster. The alertmanager expects a valid configuration in the form of a secret called alertmanager-name.

About Grafana

Grafana is a dashboarding and visualization tool with integrations to several timeseries datasources. It is used to create dashboards for the monitoring data with customizable visualizations. We would use Prometheus as the source of data to view Portworx monitoring metrics.

Prerequisites

A running Portworx cluster.

Installation

If you installed Portworx using the Operator, skip to Installing Grafana.

Install the Prometheus Operator

Enter the following commands to download and apply the Prometheus Operator YAML for your Kubernetes version:

VER=$(kubectl version --short | awk -Fv '/Server Version: / {print $3}')
kubectl apply -f  "http://install.portworx.com/2.11/?comp=prometheus-operator&kbver=$VER"

Install the Service Monitor

Create a file named service-monitor.yaml with the below contents:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  namespace: kube-system
  name: portworx-prometheus-sm
  labels:
    name: portworx-prometheus-sm
spec:
  selector:
    matchLabels:
      name: portworx
  namespaceSelector:
    any: true
  endpoints:
  - port: px-api
    targetPort: 9001
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: autopilot-prometheus-sm
  namespace: kube-system
  labels:
    name: portworx-prometheus-sm
spec:
  selector:
    matchLabels:
      name: autopilot-service
  namespaceSelector:
    any: true
  endpoints:
    - port: autopilot

Next, apply the spec:

kubectl apply -f <service-monitor.yaml>

Optional: Specify alerting rules for Prometheus Alertmanager

Specify your alerting rules. Create a file named alertmanager.yaml, specifying your configuration options for the following:

email_configs:
- to: with the address of the recipient
- from: with the address of the sender
- smarthost: with the address of your SMTP server
- auth_username: with your STMP username
- auth_identity: with the address of the sender
- auth_password: with your SMTP password.
- text: with the text the notification

slack_configs:

api_url: with your Slack API URL. To retrieve your Slack API URL, you must follow the steps in the Sending messages using Incoming Webhooks page of the Slack documentation.
channel: with the Slack channel you want to send notifications to.

text: with the text of the notification

global:
  # Global variables
route:
  group_by: [Alertname]
  receiver: email_and_slack
receivers:
- name: email_and_slack
  email_configs:
  - to:
    from:
    smarthost:
    auth_username:
    auth_identity:
    auth_password:
    text: |-
      {{ range .Alerts }}
        *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
        *Description:* {{ .Annotations.description }}
        *Details:*
        {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
        {{ end }}
      {{ end }}
  slack_configs:
  - api_url:
    channel:
    text: |-
      {{ range .Alerts }}
        *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
        *Description:* {{ .Annotations.description }}
        *Details:*
        {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
        {{ end }}
      {{ end }}

For a description of the properties in this schema, see the Configuration file section of the Prometheus documentation.

Create a secret from the alertmanager.yaml file:

kubectl create secret generic alertmanager-portworx --from-file=alertmanager.yaml -n kube-system

secret/alertmanager-portworx created

Install and configure Prometheus Alertmanager

Create a file named alertmanager-cluster.yaml, and copy in the following spec:

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  name: portworx #This name is important since the Alertmanager pods wont start unless a secret named alertmanager-${ALERTMANAGER_NAME} is created. in this case if would expect alertmanager-portworx secret in the kube-system namespace
  namespace: kube-system
  labels:
    alertmanager: portworx
spec:
  replicas: 3

Apply the spec by entering the following command:

kubectl apply -f alertmanager-cluster.yaml

alertmanager.monitoring.coreos.com/portworx created

Create a file named alertmanager-service.yaml with the following content:

apiVersion: v1
kind: Service
metadata:
  name: alertmanager-portworx
  namespace: kube-system
spec:
  type: NodePort
  ports:
  - name: web
    port: 9093
    protocol: TCP
    targetPort: web
  selector:
    alertmanager: portworx

Apply the spec by entering the following command:

kubectl apply -f alertmanager-service.yaml

service/alertmanager-portworx created