Document describes how kafka should be monitored, what are components involved, what are metrics to be monitored, what are ones to configure to receive alarms and what are thresholds.

We might need to use prometheus exporter, but not sure yet, we should come up with metrics to monitor and then see which method provides right amount.

https://github.com/danielqsj/kafka_exporter

https://artifacthub.io/packages/helm/prometheus-community/prometheus-kafka-exporter

https://docs.digitalocean.com/products/databases/kafka/how-to/monitor-clusters/

• AWS MSK

  • Grafana queries CloudWatch directly (or via the Prometheus CloudWatch Exporter) against the AWS/Kafka and AWS/KafkaTopic namespaces

  • IAM role with cloudwatch:GetMetricData permissions

• DigitalOcean Managed Kafka

  • Prometheus scrapes the cluster’s built-in HTTPS /metrics endpoint using basic-auth and DO’s root-CA

  • Static scrape_config in prometheus.yml (30–60 s interval)

  • Optional JMX exporter for consumer-lag metrics

• On-Prem Kafka

  • Prometheus JMX Exporter deployed alongside each broker (embedded or sidecar)

  • Scrape via a dedicated job_name: kafka_onprem in prometheus.yml

  • TLS and auth per your internal security policies

That covers the agentless managed-service scrapes plus the JMX exporter for any self-hosted clusters.

Use the same monitoring sub-module in each of your terraform-<provider>-kafka repositories. It consumes outputs from your local Kafka module and provisions ServiceMonitors (or CloudWatch alarms), dashboards, and alerts.

Notes:

• All three provider repos use the same monitoring sub-module signature.

• The AWS MSK variant can either hook into CloudWatch alarms or automatically deploy a Prometheus exporter.

• Application-specific Kafka metrics (e.g., queue depths for App A/B) should live in those apps’ own dashboards alongside these shared Kafka vitals.

Below is how you can fold the “Common Kafka Failure Scenarios” table into your runbooks and escalation procedures. First, present the table as a quick reference. Then for each critical or high-severity issue, provide a templated runbook entry.

For each Critical or High severity issue, create a runbook entry following this pattern:

• Precondition / Alert Trigger

  • Reference the alert name (e.g. Under-Replicated Partitions > 0 for 5m).

• Initial Diagnostics

  1. Check alert details in Alertmanager or CloudWatch (depending on platform).

  2. Identify affected broker(s) via cluster_name and instance labels.

  3. Review recent metrics (ISR count, GC pause, disk usage, consumer lag) in Grafana.

• Remediation Steps

  1. Broker Outage

     • SSH into the node or kubectl describe pod / kubectl logs.

     • If OOM, review memory limits and JVM heap usage; consider restarting the pod or freeing OS memory.

  2. Under-Replicated Partitions

     • Verify slow follower metrics (kafka_server_fetchstats_...), GC pauses, disk I/O.

     • Restart lagging broker or rebalance partitions with kafka-reassign-partitions.sh.

  3. Offline Partitions

     • Check controller logs (kafka.controller.KafkaController).

     • Force a leader election:

  4. Excessive GC Pauses

     • Inspect GC logs for pause causes; adjust -Xms/-Xmx or GC algorithm.

     • Roll brokers one by one after tuning.

• Verification

  • Confirm ISR returns to full set (UnderReplicatedPartitions == 0).

  • Ensure no further alerts for this issue in the next 10 minutes.

• Escalation

  • Immediate paging for Critical alerts.

  • If unresolved after 10 minutes, escalate to SRE lead.

  • After 30 minutes without resolution, involve senior Kafka engineer or DBA team.

1. Critical

   • Page on-call immediately.

   • Triage expectation: acknowledge within 5 min, resolve or escalate within 15 min.

2. High

   • Notify on-call via email/Slack.

   • Acknowledge within 15 min, resolve or escalate within 1 hour.

3. Medium

   • Log to ticketing system; review in next business cycle unless recurring.

Define these alerts centrally (Alertmanager or CloudWatch), scoped by a service label so each team sees only their own issues.

Add these panel groups to both your main application-stack dashboard and each service-specific dashboard.

Use a $service template variable to render one block per service:

1. App Throughput (msg/s)
   rate(app_messages_processed_total{service="$service"}[1m])

2. App 99th-Pct Latency
   histogram_quantile(0.99, app_processing_duration_seconds_bucket{service="$service"}[5m])

3. App Error Rate (%)
   increase(app_message_errors_total{service="$service"}[5m]) / increase(app_messages_processed_total{service="$service"}[5m])

4. App Queue Depth
   app_queue_depth{service="$service"}

5. DLQ Depth
   app_dlq_depth{service="$service"}

6. Consumer Lag
   max by(partition)(kafka_consumergroup_lag{group=~"$service-.*"})

7. Under-Replicated Partitions
   sum(kafka_controller_replicamanager_underreplicatedpartitions{topic=~"$service-.*"})

8. Produce Errors
   increase(kafka_network_requestmetrics_requests_total{request="Produce", error!=""}[5m])

Arrange panels in two rows—Application Metrics on top, Kafka-Infra Metrics below—and keep a consistent grid layout for each service block.

For each service’s own dashboard (no selector needed), include the same eight panels, pre-filtered to that service:

• Throughput

• 99th-Pct Latency

• Error Rate

• Queue Depth

• DLQ Depth

• Consumer Lag

• Under-Replicated Partitions

• Produce Errors

Group panels into two sections—application KPIs first, then Kafka infra—so service owners get both business and platform insights at a glance!