data-mesher

Set up data-mesher

Experimental

This service is experimental and will change in the future.

This service will set up data-mesher, a file synchronization daemon that uses a gossip protocol to replicate files across a cluster of nodes.

Architecture🔗

Data-mesher v2 uses a file-based approach where:

• Files are defined with a list of authorized public keys (ED25519)
• Only files listed in the configuration can be uploaded or synced
• Files must be signed by one of the configured public keys

Roles🔗

• default: A node that can sign files, act as a bootstrap node and distribute files to other nodes

Usage🔗

inventory.instances = {
  data-mesher = {
    module = {
      name = "data-mesher";
      input = "clan-core";
    };   
    roles.default.settings = {                  
      files = {
        "config_app" = [
          "azwT+VhTxA+BF73Hwq0uqdXHG8XvHU2BknoVXgmEjww="
        ];
        "shared_data" = [
          "azwT+VhTxA+BF73Hwq0uqdXHG8XvHU2BknoVXgmEjww="
          "Mdtz9s2DEyEk0DL8ZzW7WqwAehoQ97PFHVbJJdskkGo="
        ];
      };
    };
  };
}

Configuration Options🔗

• logLevel: Log level (default: info)
• port: Port for cluster communication (default: 7946)
• certificateValidity: Validity of the generated node identity certificate as a Go duration (default: 2160h, 90 days). Regenerate the data-mesher-node-identity vars before it expires.
• extraBootstrapPeers: List of extra bootstrap peers to connect to when joining the cluster.
• interfaces: The network interface(s) for cluster communication - defaults to all interfaces
• files: Map of file names to lists of authorized ED25519 public keys

Uploading Files🔗

Once the cluster is running, you can upload files using the CLI:

# Create a file and upload it
echo "my content" > /tmp/myfile
data-mesher file update /tmp/myfile --url http://localhost:7331 --key-path /path/to/signing.key

Files will automatically sync to all nodes in the cluster that have the same file definition in their configuration.

Roles🔗

The data-mesher service has the following roles:

• admin
• bootstrap
• default
• peer
• signer

The admin role has no configuration🔗

The bootstrap role has no configuration🔗

Options for the default role🔗

certificateValidity🔗

Validity period of the node identity certificate, as a Go duration string (largest unit is h; e.g. 2160h for 90 days).

Type: string matching the pattern ([0-9]+(\.[0-9]+)?(ns|us|µs|ms|s|m|h))+

Default:

"2160h"

"87600h"

Declared in: clanServices/data-mesher/default.nix

extraBootstrapPeers🔗

Additional bootstrap peers that act as an initial gateway when joining the cluster. These are merged with machines from the 'bootstrap' role.

They must be libp2p multi addresses: https://libp2p.io/guides/addressing

Type: list of string

Default:

[ ]

[
  "/ip4/192.168.1.1/tcp/7946/p2p/12D3KooWHs5NXo2CPFvFT6cdsu8eSkfggNtfqTtKBVojHY7En86i"
  "/ip4/192.168.1.2/tcp/7946/p2p/12D3KooWDSrQ6xLqFmS44bZKXDujBmTDNu7BQS9HmDz6FRS52CyN"
]

Declared in: clanServices/data-mesher/default.nix

interfaces🔗

We will bind to each each interface listed, listening for connections on cluster.port. If no interfaces are provided, we will bind to all available.

Type: list of string

Default:

[ ]

[
  "eth1"
  "tailscale0"
]

Declared in: clanServices/data-mesher/default.nix

logLevel🔗

Log level

Type: one of "fatal", "error", "warn", "info", "debug"

Default:

"info"

Declared in: clanServices/data-mesher/default.nix

network.files🔗

A mapping of file names to lists of base64-encoded ED25519 public keys. Only files listed here can be uploaded or imported from other nodes, and they must be signed by one of the configured public keys.

Type: attribute set of list of string

Default:

{ }

{
  "config:app" = [
    "ZasdhiAVJTa5b2qG8ynWvdHqALUxC6Eg8pdn6RVXuQE="
    "1ru2QQ1eWV7yDlyfTTDEml3xTiacASYn0KprzknN8Pc="
  ];
  "dns:sol" = [
    "P6AE0lukf9/qmVglYrGPNYo5ZnpFrnqLeAzlCZF0lTk="
  ];
}

Declared in: clanServices/data-mesher/default.nix

network.namespaces🔗

List of namespace names for certificate-based file authorization. When configured, files named {namespace}/{signer_public_key_url_encoded} are permitted from any peer with a valid certificate signed by this network. These are merged with namespaces exported by other clan services.

Type: list of string

Default:

[ ]

[
  "dns"
  "web"
]

Declared in: clanServices/data-mesher/default.nix

port🔗

Port to listen on for cluster communication.

Type: 16 bit unsigned integer; between 0 and 65535 (both inclusive)

Default:

7946

Declared in: clanServices/data-mesher/default.nix

The peer role has no configuration🔗

The signer role has no configuration🔗