Prometheus

Follow the steps below to configure a service so as to extract its related alert data from Prometheus.

Squadcast will then process this information to create incidents for this service as per your preferences.

Using Prometheus as an Alert Source

(1) From the navigation bar on the left, select Services. Pick the applicable Team from the Team-picker on the top. Next, click on Alert Sources for the applicable Service

(2) Search for Prometheus from the Alert Source drop-down and copy the Webhook URL

Create a Squadcast Webhook in Prometheus

Now open your Prometheus Alertmanager’s Configuration file. (You need Prometheus Alertmanager to handle alerts generated by Prometheus. Click here for more information )

Add a new webhook receiver and enter the URL obtained earlier as the webhook url.

Now you can use this receiver in any of your alert routes (You can also make it the default receiver, as per your requirements).

Also please make sure that your alerts has the severity under labels, and annotations has the summary & description fields configured in the alert rules file as shown below in the example rules config.

serverFiles:
  alerts:
    groups:
    - name: gt.k8s.kafka.rules
      rules:
      - alert: GtK8sKafkaConsumerLag1Warning
        expr: kafka_consumergroup_lag > 500
        for: 120m
        labels:
          severity: warning
        annotations:
          description: "Kafka consumer { { $labels.consumergroup } } lags with { { $value } } messages in topic { { $labels.topic } }"
          summary: "Kafka affected in { { $labels.instance } }"

Important: We use the summary field along with Severity & Alert group name as incident message and the description field along with generator url as incident description. Hence, severity (labels), summary & description (annotations) are mandatory for integration with Squadcast.

Note: Any alert which gets resolved from Prometheus will automatically be resolved inside Squadcast as well, unless you have explicitly set send_resolved to false in the config file.