Dynamic configuration (control plane)

This example walks through configuring Envoy using the Go Control Plane reference implementation.

It demonstrates how configuration provided to Envoy persists, even when the control plane is not available, and provides a trivial example of how to update Envoy’s configuration dynamically.

Step 1: Start the proxy container

Change directory to examples/dynamic-config-cp in the Envoy repository.

First build the containers and start the proxy container.

This should also start two upstream HTTP echo servers, service1 and service2.

The control plane has not yet been started.

$ pwd
envoy/examples/dynamic-config-cp
$ docker-compose build --pull
$ docker-compose up -d proxy
$ docker-compose ps

       Name                            Command                State                     Ports
------------------------------------------------------------------------------------------------------------------------
dynamic-config-cp_proxy_1      /docker-entrypoint.sh /usr ... Up      0.0.0.0:10000->10000/tcp, 0.0.0.0:19000->19000/tcp
dynamic-config-cp_service1_1   /bin/echo-server               Up      8080/tcp
dynamic-config-cp_service2_1   /bin/echo-server               Up      8080/tcp

Step 2: Check initial config and web response

As we have not yet started the control plane, nothing should be responding on port 10000.

$ curl http://localhost:10000
curl: (56) Recv failure: Connection reset by peer

Dump the proxy’s static_clusters configuration and you should see the cluster named xds_cluster configured for the control plane:

$ curl -s http://localhost:19000/config_dump  | jq '.configs[1].static_clusters'
[
  {
    "cluster": {
      "@type": "type.googleapis.com/envoy.api.v2.Cluster",
      "name": "xds_cluster",
      "type": "STRICT_DNS",
      "connect_timeout": "1s",
      "http2_protocol_options": {},
      "load_assignment": {
        "cluster_name": "xds_cluster",
        "endpoints": [
          {
            "lb_endpoints": [
              {
                "endpoint": {
                  "address": {
                    "socket_address": {
                      "address": "go-control-plane",
                      "port_value": 18000
                    }
                  }
                }
              }
            ]
          }
        ]
      }
    },
    "last_updated": "2020-10-25T20:20:54.897Z"
  }
]

No dynamic_active_clusters have been configured yet:

$ curl -s http://localhost:19000/config_dump  | jq '.configs[1].dynamic_active_clusters'
null

Step 3: Start the control plane

Start up the go-control-plane service.

You may need to wait a moment or two for it to become healthy.

$ docker-compose up --build -d go-control-plane
$ docker-compose ps

        Name                                Command                  State                    Ports
-------------------------------------------------------------------------------------------------------------------------------------
dynamic-config-cp_go-control-plane_1  bin/example -debug             Up (healthy)
dynamic-config-cp_proxy_1             /docker-entrypoint.sh /usr ... Up            0.0.0.0:10000->10000/tcp, 0.0.0.0:19000->19000/tcp
dynamic-config-cp_service1_1          /bin/echo-server               Up            8080/tcp
dynamic-config-cp_service2_1          /bin/echo-server               Up            8080/tcp

Step 4: Query the proxy

Once the control plane has started and is healthy, you should be able to make a request to port 10000, which will be served by service1.

$ curl http://localhost:10000
Request served by service1

HTTP/1.1 GET /

Host: localhost:10000
Accept: */*
X-Forwarded-Proto: http
X-Request-Id: 1d93050e-f39c-4602-90f8-a124d6e78d26
X-Envoy-Expected-Rq-Timeout-Ms: 15000
Content-Length: 0
User-Agent: curl/7.72.0

Step 5: Dump Envoy’s dynamic_active_clusters config

If you now dump the proxy’s dynamic_active_clusters configuration, you should see it is configured with the example_proxy_cluster pointing to service1, and a version of 1.

$ curl -s http://localhost:19000/config_dump  | jq '.configs[1].dynamic_active_clusters'
[
  {
    "version_info": "1",
    "cluster": {
      "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
      "name": "example_proxy_cluster",
      "type": "LOGICAL_DNS",
      "connect_timeout": "5s",
      "dns_lookup_family": "V4_ONLY",
      "load_assignment": {
        "cluster_name": "example_proxy_cluster",
        "endpoints": [
          {
            "lb_endpoints": [
              {
                "endpoint": {
                  "address": {
                    "socket_address": {
                      "address": "service1",
                      "port_value": 8080
                    }
                  }
                }
              }
            ]
          }
        ]
      }
    },
    "last_updated": "2020-10-25T20:37:05.838Z"
  }
]

Step 6: Stop the control plane

Stop the go-control-plane service:

$ docker-compose stop go-control-plane

The Envoy proxy should continue proxying responses from service1.

$ curl http://localhost:10000 | grep "served by"
Request served by service1

Step 7: Edit go file and restart the control plane

The example setup starts the go-control-plane service with a custom resource.go file which specifies the configuration provided to Envoy.

Update this to have Envoy proxy instead to service2.

Edit resource.go in the dynamic configuration example folder and change the UpstreamHost from service1 to service2:

33const (
34	ClusterName  = "example_proxy_cluster"
35	RouteName    = "local_route"
36	ListenerName = "listener_0"
37	ListenerPort = 10000
38	UpstreamHost = "service1"
39	UpstreamPort = 8080
40)

Further down in this file you must also change the configuration snapshot version number from "1" to "2" to ensure Envoy sees the configuration as newer:

164func GenerateSnapshot() cache.Snapshot {
165	return cache.NewSnapshot(
166		"1",
167		[]types.Resource{}, // endpoints
168		[]types.Resource{makeCluster(ClusterName)},
169		[]types.Resource{makeRoute(RouteName, ClusterName)},
170		[]types.Resource{makeHTTPListener(ListenerName, RouteName)},
171		[]types.Resource{}, // runtimes
172		[]types.Resource{}, // secrets
173		[]types.Resource{}, // extensions configs
174	)

Now rebuild and restart the control plane:

$ docker-compose up --build -d go-control-plane

You may need to wait a moment or two for the go-control-plane service to become healthy again.

Step 8: Check Envoy uses the updated configuration

Now when you make a request to the proxy it should be served by the service2 upstream service.

$ curl http://localhost:10000 | grep "served by"
Request served by service2

Dumping the dynamic_active_clusters you should see the cluster configuration now has a version of 2, and example_proxy_cluster is configured to proxy to service2:

$ curl -s http://localhost:19000/config_dump  | jq '.configs[1].dynamic_active_clusters'
[
  {
    "version_info": "2",
    "cluster": {
      "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
      "name": "example_proxy_cluster",
      "type": "LOGICAL_DNS",
      "connect_timeout": "5s",
      "dns_lookup_family": "V4_ONLY",
      "load_assignment": {
        "cluster_name": "example_proxy_cluster",
        "endpoints": [
          {
            "lb_endpoints": [
              {
                "endpoint": {
                  "address": {
                    "socket_address": {
                      "address": "service2",
                      "port_value": 8080
                    }
                  }
                }
              }
            ]
          }
        ]
      }
    },
    "last_updated": "2020-10-26T14:35:17.360Z"
  }
]

See also

Dynamic configuration (control plane) quick start guide

Quick start guide to dynamic configuration of Envoy with a control plane.

Envoy admin quick start guide

Quick start guide to the Envoy admin interface.

Dynamic configuration (filesystem) sandbox

Configure Envoy using filesystem-based dynamic configuration.

Go control plane

Reference implementation of Envoy control plane written in go.