monitoring

Monitoring stack gathering metrics and logs with a small resource footprint.

Experimental

This service is experimental and will change in the future.

Usage🔗

inventory.instances = {
  monitoring = {
    module = {
      name = "monitoring";
      input = "clan-core";
    };

    roles = {
      client = {
        # Enable monitoring for all machines in the clan.
        tags = [ "all" ];
        # Decide whether or not your server is reachable via https.
        settings.useSSL = true;
        settings.loki.journal.relabelRules.beforeNormalization = [
          ''
            // Create labels from raw journal fields.
            rule {
              source_labels = ["__journal_com_docker_swarm_service_name"]
              regex = "^.*_(.*)$"
              target_label = "oci_platform_service_name"
            }
          ''
        ];
        settings.loki.journal.relabelRules.afterNormalization = [
          ''
            // Drop debug-level logs after `level` is created.
            rule {
              action = "drop"
              source_labels = ["level"]
              regex = "debug"
            }
          ''
        ];
      };

      # Select one machine as the central monitoring server.
      # Hint: This is currently limited to exactly one server.
      server.machines.<machine>.settings = {
        # Optionally enable grafana for dashboards and alerts.
        grafana.enable = true;
      };
    };
  };
};

Architecture Overview🔗

Roles🔗

Client🔗

Clients are machines that create metrics and logs. Those are sent to the central monitoring server for storage and visualization.

Journal relabeling can be customized in two phases:

• settings.loki.journal.relabelRules.beforeNormalization for raw journal labels such as __journal__*
• settings.loki.journal.relabelRules.afterNormalization for normalized labels such as instance, service_name, and level

The generated monitoring collector config is installed as /etc/alloy/config.alloy. Additional local collector fragments can be added with environment.etc."alloy/<name>.alloy".

Server🔗

Servers store metrics and logs. They also provide optional dashboards for visualization and an alerting system.

By default the server sets up nginx as a reverse proxy on the machine's FQDN, exposing mimir under /mimir/, loki under /loki/ (both guarded by basic auth) and grafana under /grafana/.

Using an external reverse proxy🔗

If the server machine already runs another reverse proxy (e.g. caddy or traefik) on ports 80/443, disable the built-in nginx:

server.machines.<machine>.settings = {
  grafana.enable = true;
  host = "monitoring.example.com";
  proxy.enable = false;
  # Set if your proxy serves the monitoring endpoints via https.
  proxy.useSSL = true;
};

The external proxy must replicate these routes:

The htpasswd files are available on the server machine at config.clan.core.vars.generators.mimir-auth.files.htpasswd.path and config.clan.core.vars.generators.loki-auth.files.htpasswd.path.

Roles🔗

The monitoring service has the following roles:

• client
• server

Options for the client role🔗

loki.journal.relabelRules.afterNormalization🔗

Additional Alloy rule blocks inserted into loki.relabel "journal" after the built-in label normalization rules.

Use this for rules that depend on normalized labels such as instance, service_name, or level.

Type: list of string

Default:

[ ]

[
  ''
    rule {
      action = "drop"
      source_labels = ["level"]
      regex = "debug"
    }
  ''
]

Declared in: clanServices/monitoring/default.nix

loki.journal.relabelRules.beforeNormalization🔗

Additional Alloy rule blocks inserted into loki.relabel "journal" before the built-in label normalization rules.

Use this for rules that need raw journal labels such as __journal__*.

Type: list of string

Default:

[ ]

[
  ''
    rule {
      source_labels = ["__journal_com_docker_swarm_service_name"]
      regex = "^.*_(.*)$"
      target_label = "oci_platform_service_name"
    }
  ''
]

Declared in: clanServices/monitoring/default.nix

monitoredSystemdServices🔗

List of systemd services which are shown in the clan infrastructure grafana dashboard. Logs sent to the monitoring server are filtered using this list.

Options: "all" - all systemd services "nixos" (default) - services that have been explicitly enabled through nixos config listOf str - custom list of systemd services

Type: one of "all", "nixos" or list of string

Default:

"nixos"

[
  "alloy.service"
  "grafana.service"
  "loki.service"
  "mimir.service"
  "nginx.service"
]

Declared in: clanServices/monitoring/default.nix

useSSL🔗

Whether to send metrics data via http or https. Enable this if your monitoring server is addressable using https.

Type: boolean

Default:

false

true

Declared in: clanServices/monitoring/default.nix

Options for the server role🔗

grafana.enable🔗

Whether to enable grafana.

Type: boolean

Default:

false

true

Declared in: clanServices/monitoring/default.nix

host🔗

Hostname or address of the monitoring server (e.g. "qube.email"). The protocol (http/https) is controlled by the client's useSSL option. If null, derived automatically from the server machine name and meta.domain.

Type: null or string

Default:

null

"monitoring.example.com"

Declared in: clanServices/monitoring/default.nix

proxy.enable🔗

Whether to set up nginx as a reverse proxy exposing mimir, loki and grafana under /mimir, /loki and /grafana on the machine's FQDN, with basic auth for /mimir and /loki.

Disable this if the machine already runs another reverse proxy (e.g. caddy or traefik). The external proxy must replicate these routes and guard /mimir and /loki with the htpasswd files from the mimir-auth and loki-auth vars generators.

Type: boolean

Default:

true

Declared in: clanServices/monitoring/default.nix

proxy.useSSL🔗

Only used when proxy.enable is false: whether the external reverse proxy serves the monitoring endpoints via https. Controls grafana's root_url scheme and secure cookies. With the built-in nginx proxy this is detected automatically from the nginx virtual host.

Type: boolean

Default:

false

Declared in: clanServices/monitoring/default.nix