Consul Integration

Consul is a tool for discovering and configuring services in your infrastructure. Consul's key features include service discovery, health checking, a KV store, and robust support for multi-datacenter deployments. Nomad's integration with Consul enables automatic clustering, built-in service registration, and dynamic rendering of configuration files and environment variables. The sections below describe the integration in more detail.

Configuration

In order to use Consul with Nomad, you will need to configure and install Consul on your nodes alongside Nomad, or schedule it as a system job. Nomad does not run Consul for you.

To enable Consul integration, please refer to the Nomad agent Consul configuration documentation.

Automatic Clustering with Consul

Nomad servers and clients will be automatically informed of each other's existence when a running Consul cluster already exists and the Consul agent is installed and configured on each host. Please refer to the Automatic Clustering with Consul guide for more information.

Service Discovery

Nomad schedules workloads of various types across a cluster of generic hosts. Because of this, placement is not known in advance and you will need to use service discovery to connect tasks to other services deployed across your cluster. Nomad integrates with Consul to provide service discovery and monitoring.

To configure a job to register with service discovery, please refer to the service job specification documentation.

Dynamic Configuration

Nomad's job specification includes a template block that uses a Consul ecosystem tool called Consul Template. This mechanism creates a convenient way to ship configuration files that are populated from environment variables, Consul data, Vault secrets, or just general configurations within a Nomad task.

For more information on Nomad's template block and how it leverages Consul Template, please see the template job specification documentation.

Using Nomad Workload Identity with Consul

Starting in Nomad 1.7, Nomad clients will use a task or service's Workload Identity to authenticate to Consul and obtain a Consul token specific to the service or task. When using Workload Identity you will no longer need to pass in a Consul token to submit a job.

Configuring Consul Authentication

Create a configuration file for a Consul JWT Auth Method. The JWKSURL field should point to all the Nomad servers; you should use a proxy or DNS A Record to ensure that requests can go to any server in the cluster. The remaining fields are required to match those shown here.

{
  "JWKSURL": "https://nomad.example.com:4646/.well-known/jwks.json",
  "JWTSupportedAlgs": ["RS256"],
  "BoundAudiences": "consul.io",
  "ClaimMappings": {
    "nomad_namespace": "nomad_namespace",
    "nomad_job_id": "nomad_job_id",
    "nomad_task": "nomad_task",
    "nomad_service": "nomad_service"
  }
}

Note

If you're using an https endpoint on Nomad server itself for JWKS, your server must be configured with tls.verify_https_client = false, because currently there is no way to provide Nomad certificates to Consul (or Vault) clients. If you want to use mTLS, which is the recommended practice, you need to terminate TLS before the requests reach Nomad JWKS endpoint.

Using that configuration file, you'll create a Consul Auth Method named nomad-workloads, which controls:

Authentication for Service Identity tokens used to register services and configure Consul Connect.
Access to reading service data and Consul KV for your jobs' template blocks.

The auth method will have two binding rules attached: one mapping Nomad service blocks to Consul services, and one mapping Nomad namespaces or your job IDs to Consul Roles.

Note

Nomad also supports having separate auth methods for services and tasks, for additional flexibility. Consult service_auth_method and task_auth_method configuration parameters documentation.

Auth Method and Binding Rules for Services and Templates

Using the configuration file shown above, create a Consul Auth Method to support Nomad workloads. Note that you should not set the -max-token-ttl flag for Consul Auth Methods used for Nomad.

In Consul Community Edition, you'll always bind to the default Consul namespace.

$ consul acl auth-method create \
         -name 'nomad-workloads' \
         -type jwt \
         -description 'login method for Nomad workloads' \
         -format json \
         -namespace-rule-selector='' \
         -namespace-rule-bind-namespace='default' \
         -config "@consul-auth-method-config.json"

In Consul Enterprise, you'll map a Nomad namespace to a Consul namespace.

$ consul acl auth-method create \
         -name 'nomad-workloads' \
         -type jwt \
         -description 'login method for Nomad workloads' \
         -format json \
         -namespace-rule-selector='' \
         -namespace-rule-bind-namespace='${value.nomad_namespace}' \
         -config "@consul-auth-method-config.json"

Create a Consul Binding Rule that maps the Nomad Workload Identity to the Consul Service Identity.

$ consul acl binding-rule create \
         -method 'nomad-workloads' \
         -description 'binding rule for Nomad workload identities (WI)' \
         -bind-type service \
         -bind-name '${value.nomad_namespace}-${value.nomad_service}' \
         -selector '"nomad_service" in value`

Next, create a Consul Binding Rule that maps Nomad namespaces to Consul Roles:

$ consul acl binding-rule create \
         -method 'nomad-workloads' \
         -description 'binding rule for Nomad template workload identities (WI)' \
         -bind-type role \
         -bind-name 'nomad-${value.nomad_namespace}-tasks' \
         -selector '"nomad_service" not in value`

Role for Templates

For each Nomad namespace that you want to grant access to Consul, create a Consul role with a name like nomad-$nomadNamespace. For example, for the Nomad namespace named prod you'll create the following Consul role.

$ consul acl role create \
       -name "nomad-prod" \
       -description "role for Nomad production workloads with templates" \
       -policy-name "example-policy"

The policy you assign to the role should have sufficient service:read and kv:read permissions. An example policy might look like the following.

key_prefix "" {
  policy = "read"
}

service_prefix "" {
  policy = "read"
}

The binding rule you created above for Nomad namespaces will map workloads to the role with this policy.

Authenticating Without Workload Identity (Legacy)

If Consul ACLs are enabled, the allow_unauthenticated configuration parameter will control whether a Consul token will be required when submitting a job with Consul namespace configured. The provided Consul token must belong to the correct namespace, and must be backed by a Consul ACL Policy with sufficient service:write and kv:read permissions. An example policy might look like the following.

key_prefix "" {
  policy = "read"
}

service_prefix "" {
  policy = "write"
}

Note

This legacy workflow will be removed in Nomad 1.9. Before upgrading to Nomad 1.9 you will need to have configured authentication with Consul as described in Configuring Consul Authentication.

Migrating to Using Workload Identity with Consul

Migrating from the legacy (pre-1.7) workflow where workload use the agent's Consul token requires configuation on your Consul cluster and your Nomad server agents. It does not require updating your running Nomad jobs. To migrate:

Create the Consul auth method and binding rules on your Consul cluster.
Enable consul.service_identity blocks in your Nomad server agent configurations.
Enable consul.task_identity blocks in your Nomad server agent configurations.
(Optionally) add identity blocks to your jobs if you want to use a different identity because of how your auth method and binding rules are configured.

Note that when using Workload Identity you will no longer need to pass in a Consul token to submit a job.

Consul Namespaces

Nomad provides integration with Consul Namespaces for service registrations specified in service blocks and Consul KV reads in template blocks.

By default, Nomad will not specify a Consul namespace on service registrations or KV store reads, which Consul then implicitly resolves to the "default" namespace. This default namespace behavior can be modified by setting the namespace field in the Nomad agent Consul configuration block.

For more control over Consul namespaces, Nomad Enterprise supports configuring the Consul namespace at the group or task level in the Nomad job spec as well as the -consul-namespace command line argument for job run.

The Consul namespace used for a set of group or task service registrations within a group, as well as template KV store access is determined from the following hierarchy from lowest to highest precedence:

Consul default: If no Consul namespace options are configured, Consul will automatically make use of the "default" namespace.
agent configuration: If the namespace Nomad agent Consul configuration parameter is set, this namespace will be used instead of the Consul default.
job run command: Enterprise If the -consul-namespace command line argument is specified on job submission, this namespace will take precedence over the namespace set in Nomad agent configuration.
group and task configuration: Enterprise If the group level or task level Consul namespace field is configured, this namespace will take precedence over all other options.

Assumptions

Each Nomad client should have a local Consul agent running on the same host, reachable by Nomad. Nomad clients should never share a Consul agent or talk directly to the Consul servers. Nomad is not compatible with Consul Data Plane.
The service discovery feature in Nomad depends on operators making sure that the Nomad client can reach the Consul agent.
Tasks running inside Nomad also need to reach out to the Consul agent if they want to use any of the Consul APIs. Ex: A task running inside a docker container in the bridge mode won't be able to talk to a Consul Agent running on the loopback interface of the host since the container in the bridge mode has its own network interface and doesn't see interfaces on the global network namespace of the host. There are a couple of ways to solve this, one way is to run the container in the host networking mode, or make the Consul agent listen on an interface in the network namespace of the container.
The consul binary must be present in Nomad's $PATH to run the Envoy proxy sidecar on client nodes.
Consul service mesh using network namespaces is only supported on Linux.

Compatibility

Most supported versions of Nomad are compatible with most recent versions of Consul, with some exceptions.

Nomad versions 1.6.0+, 1.5.6+, and 1.4.11+ are compatible with any currently supported version of Consul.
Nomad versions 1.4.4 to 1.4.11 and 1.5.0 to 1.5.6 are compatible with any currently supported version of Consul except 1.13.8.
Nomad versions 1.4.0 through 1.4.3 are compatible with Consul versions 1.13.0 through 1.13.7, and 1.13.9. Changes to Consul service mesh in version 1.14 are incompatible with Nomad 1.4.3 and earlier.
Nomad is not compatible with Consul Data Plane.

	Consul 1.13.0 - 1.13.7	Consul 1.13.8	Consul 1.13.9	Consul 1.14.0+
Nomad 1.6.0+	✅	✅	✅	✅
Nomad 1.5.6+	✅	✅	✅	✅
Nomad 1.5.0-1.5.5	✅	❌	✅	✅
Nomad 1.4.11-1.4.13	✅	✅	✅	✅
Nomad 1.4.4-1.4.10	✅	❌	✅	✅
Nomad 1.4.0-1.4.3	✅	❌	✅	❌