monitoring

Monitoring stack gathering metrics and logs with a small resource footprint.

Experimental

This service is experimental and will change in the future.

Usage🔗

inventory.instances = {
  monitoring = {
    module = {
      name = "monitoring";
      input = "clan-core";
    };

    roles = {
      client = {
        # Enable monitoring for all machines in the clan.
        tags = [ "all" ];
        # Decide whether or not your server is reachable via https.
        settings.useSSL = true;
        settings.loki.journal.relabelRules.beforeNormalization = [
          ''
            // Create labels from raw journal fields.
            rule {
              source_labels = ["__journal_com_docker_swarm_service_name"]
              regex = "^.*_(.*)$"
              target_label = "oci_platform_service_name"
            }
          ''
        ];
        settings.loki.journal.relabelRules.afterNormalization = [
          ''
            // Drop debug-level logs after `level` is created.
            rule {
              action = "drop"
              source_labels = ["level"]
              regex = "debug"
            }
          ''
        ];
      };

      # Select one machine as the central monitoring server.
      # Hint: This is currently limited to exactly one server.
      server.machines.<machine>.settings = {
        # Optionally enable grafana for dashboards and alerts.
        grafana.enable = true;
      };
    };
  };
};

Architecture Overview🔗

Roles🔗

Client🔗

Clients are machines that create metrics and logs. Those are sent to the central monitoring server for storage and visualization.

Journal relabeling can be customized in two phases:

• settings.loki.journal.relabelRules.beforeNormalization for raw journal labels such as __journal__*
• settings.loki.journal.relabelRules.afterNormalization for normalized labels such as instance, service_name, and level

The generated monitoring collector config is installed as /etc/alloy/config.alloy. Additional local collector fragments can be added with environment.etc."alloy/<name>.alloy".

Server🔗

Servers store metrics and logs. They also provide optional dashboards for visualization and an alerting system.

Roles🔗

The monitoring service has the following roles:

• client
• server

Options for the client role🔗

loki.journal.relabelRules.afterNormalization🔗

Additional Alloy rule blocks inserted into loki.relabel "journal" after the built-in label normalization rules.

Use this for rules that depend on normalized labels such as instance, service_name, or level.

Type: list of string

Default:

[ ]

[
  ''
    rule {
      action = "drop"
      source_labels = ["level"]
      regex = "debug"
    }
  ''
]

Declared in: clanServices/monitoring/default.nix

loki.journal.relabelRules.beforeNormalization🔗

Additional Alloy rule blocks inserted into loki.relabel "journal" before the built-in label normalization rules.

Use this for rules that need raw journal labels such as __journal__*.

Type: list of string

Default:

[ ]

[
  ''
    rule {
      source_labels = ["__journal_com_docker_swarm_service_name"]
      regex = "^.*_(.*)$"
      target_label = "oci_platform_service_name"
    }
  ''
]

Declared in: clanServices/monitoring/default.nix

monitoredSystemdServices🔗

List of systemd services which are shown in the clan infrastructure grafana dashboard. Logs sent to the monitoring server are filtered using this list.

Options: "all" - all systemd services "nixos" (default) - services that have been explicitly enabled through nixos config listOf str - custom list of systemd services

Type: one of "all", "nixos" or list of string

Default:

"nixos"

[
  "alloy.service"
  "grafana.service"
  "loki.service"
  "mimir.service"
  "nginx.service"
]

Declared in: clanServices/monitoring/default.nix

useSSL🔗

Whether to send metrics data via http or https. Enable this if your monitoring server is addressable using https.

Type: boolean

Default:

false

true

Declared in: clanServices/monitoring/default.nix

Options for the server role🔗

grafana.enable🔗

Whether to enable grafana.

Type: boolean

Default:

false

true

Declared in: clanServices/monitoring/default.nix

host🔗

Hostname or address of the monitoring server (e.g. "qube.email"). The protocol (http/https) is controlled by the client's useSSL option. If null, derived automatically from the server machine name and meta.domain.

Type: null or string

Default:

null

"monitoring.example.com"

Declared in: clanServices/monitoring/default.nix