Pragmatic Observability [3]: Simple Start to export Springboot Application Metrics with Grafana and Prometheus

This guide is meant to give you a simple start to export the default Prometheus metrics and build a Grafana dashboard.

Prequisite

• Docker Desktop or
• Prometheus and Grafana binaries

Springboot application with metrics

You can create a Springboot application from Spring Initializr and configure the following files

build.gradle

implementation("io.micrometer:micrometer-registry-prometheus")

application.yml

management:
  endpoints:
    web:
      exposure:
        include: prometheus #restrict the endpoint exposure
  endpoint: 
    prometheus:
      enabled: true

This would enable the PrometheusScrapeEndpoint

Deploying Prometheus to Kubernetes

I use Docker Desktop’s Kubernetes. Alternatively you may search from docker-compose example online. Both are straighforward to begin with. Note that you should pay attention to the storage path to make sure Prometheus’s data is persisted

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  namespace: observability
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
    scrape_configs:
      - job_name: 'prometheus-trace-sample'
        static_configs: # require service name and target port to access from within the cluster
          - targets: ['observability-app-service:28081'] 
        metrics_path: /actuator/prometheus # for springboot
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-data
  namespace: observability
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Mi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-server
  namespace: observability
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-server
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: config-volume
              mountPath: /etc/prometheus
            - name: prometheus-data
              mountPath: /prometheus
      volumes:
        - name: config-volume
          configMap:
            name: prometheus-server-conf
            defaultMode: 420
        - name: prometheus-data
          persistentVolumeClaim:
            claimName: prometheus-data
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus-service
  namespace: observability
spec:
  selector:
    app: prometheus-server
  ports:
    - protocol: TCP
      port: 9090
      targetPort: 9090
      nodePort: 30002
  type: NodePort # set to expose to windows host

Deploying Grafana to Kubernetes

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
  namespace: observability
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: observability
spec:
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
        fsGroup: 472
        supplementalGroups:
          - 0
      containers:
        - name: grafana
          image: grafana/grafana:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http-grafana
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /robots.txt
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 2
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3000
            timeoutSeconds: 1
          resources:
            requests:
              cpu: 250m
              memory: 750Mi
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-pv
      volumes:
        - name: grafana-pv
          persistentVolumeClaim:
            claimName: grafana-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
spec:
  ports:
    - port: 3000
      protocol: TCP
      targetPort: 3000
      nodePort: 30003
  selector:
    app: grafana
  sessionAffinity: None
  type: NodePort

Getting a Grafana Dashboard up and running

I started with hand-crafting the dashboard and soon realize Prometheus and Grafana have a steep learning curve.

After a while, I found the community already provide templates. Dashboards | Grafana Labs

I used SpringBoot APM Dashboard | Grafana Labs, a simple import into Grafana gave me a very comprehensive dashboard and many examples to follow.