Custom COS dashboards and alert rules

The 12-factor charms ship with default observability assets used by the Grafana, Loki, and Prometheus integrations (dashboards and alert rules).

You can add your own assets by placing a cos_custom/ directory at the root of your charm source tree.

The default assets are merged with custom assets on first charm start. The merged result is stored in a persistent merged directory and reused on subsequent hook executions.

Directory layout

Create the following directory structure in your charm project:

cos_custom/
  grafana_dashboards/
  loki_alert_rules/
  prometheus_alert_rules/

Note

Only the three subdirectories above are allowed. Any other subdirectory name is considered invalid.

Add dashboards

Custom dashboards are loaded from cos_custom/grafana_dashboards/.

Save each dashboard as a .json file, for example:

  • cos_custom/grafana_dashboards/app-overview.json

When creating a dashboard, use variables for your data sources and name them prometheusds and lokids. You can make use of the Juju topology variables ($juju_model, $juju_model_uuid, $juju_application, $juju_unit).

For dashboard authoring guidance and JSON structure reference, see:

Add alert rules

Custom alert rules are loaded from:

  • cos_custom/prometheus_alert_rules/ for Prometheus rules

  • cos_custom/loki_alert_rules/ for Loki rules

The alert rules use these relation interfaces:

  • metrics-endpoint (Prometheus)

  • logging (Loki)

A minimal alert rule file looks like:

alert: AppTargetMissing
expr: up == 0
for: 1m
labels:
  severity: critical
annotations:
  summary: Target missing (instance {{ $labels.instance }})
  description: Prometheus target disappeared.

Prometheus charm libraries document these formats in the prometheus_scrape reference. Loki alert rules must include the %%juju_topology%% placeholder (as described in the loki_push_api library).

Save the rule as a .rule / .rules / .yml / .yaml file, for example:

  • cos_custom/prometheus_alert_rules/app.rule

  • cos_custom/loki_alert_rules/app.rule

On first charm start, custom assets are merged with the default assets and the merged result is reused afterwards.

For rule syntax and advanced examples, see the official documentation: