To ensure that Prometheus is always available, enable the High Availability mode of Backyards. That way the monitoring stack uses multiple Prometheus instances and Thanos Query to ensure metric consistency and high availability. Complete the following steps.
Enable high availability. Backyards is controlled by a
ControlPlanecustom resource found in the Backyard’s namespace (default:
backyards-system) named backyards. The following commands change the
true, indicating that you want to have a Highly Available deployment.
Run the following commands:
$ cat > enable-ha.yaml <<EOF spec: backyards: highAvailability: enabled: true EOF $ kubectl patch controlplane --type=merge --patch "$(cat enable-ha.yaml)" backyards
If you are using Backyards in operator mode, skip this step.
Otherwise, execute a reconciliation so Backyards updates your Kubernetes cluster to the desired state described by the ControlPlane Custom Resource. Run the following command:
$ backyards operator reconcile
Verify that High Availability is enabled. To verify that Prometheus is running in Highly Available mode, check the pods running in the
backyards-systemnamespace for multiple Prometheus instances. Run the following command.
$ kubectl get pods -l app=prometheus -n backyards-system NAME READY STATUS RESTARTS AGE prometheus-backyards-prometheus-0 5/5 Running 1 21m prometheus-backyards-prometheus-1 5/5 Running 1 21m
If two running pods are present then the High Availability setup has been completed.