Runtime

The runtime configuration specifies a virtual file system tree that contains re-loadable configuration elements. This virtual file system can be realized via a series of local file system, static bootstrap configuration and admin console derived overlays.

Virtual file system

Layering

The runtime can be viewed as virtual file system consisting of multiple layers:

  1. Static bootstrap configuration
  2. Local disk file system
  3. Local disk file system *override_subdirectory*
  4. Admin console overrides

with values in higher layers overriding corresponding values in lower layers.

File system layout

Various sections of the configuration guide describe the runtime settings that are available. For example, here are the runtime settings for upstream clusters.

Each ‘.’ in a runtime key indicates a new directory in the hierarchy, The terminal portion of a path is the file. The contents of the file constitute the runtime value. When reading numeric values from a file, spaces and new lines will be ignored.

numerator or denominator are reserved keywords and may not appear in any directory.

Static bootstrap

A static base runtime may be specified in the bootstrap configuration via a protobuf JSON representation.

Local disk file system

When the runtime virtual file system is realized on a local disk, it is rooted at symlink_root + subdirectory. For example, the health_check.min_interval key would have the following full file system path (using the symbolic link):

/srv/runtime/current/envoy/health_check/min_interval

Assume that the folder /srv/runtime/v1 points to the actual file system path where global runtime configurations are stored. The following would be a typical configuration setting for runtime:

  • symlink_root: /srv/runtime/current
  • subdirectory: envoy
  • override_subdirectory: envoy_override

Where /srv/runtime/current is a symbolic link to /srv/runtime/v1.

The override_subdirectory is used along with the --service-cluster CLI option. Assume that --service-cluster has been set to my-cluster. Envoy will first look for the health_check.min_interval key in the following full file system path:

/srv/runtime/current/envoy_override/my-cluster/health_check/min_interval

If found, the value will override any value found in the primary lookup path. This allows the user to customize the runtime values for individual clusters on top of global defaults.

Admin console

Values can be viewed at the /runtime admin endpoint. Values can be modified and added at the /runtime_modify admin endpoint. If runtime is not configured, an empty provider is used which has the effect of using all defaults built into the code, except for any values added via /runtime_modify.

Attention

Use the /runtime_modify endpoint with care. Changes are effectively immediately. It is critical that the admin interface is properly secured.

Protobuf and JSON representation

The runtime file system can be represented inside a proto3 message as a google.protobuf.Struct modeling a JSON object with the following rules:

An example representation of a setting for the health_check.min_interval key in YAML is:

health_check:
  min_interval: 5

Comments

Lines starting with # as the first character are treated as comments.

Comments can be used to provide context on an existing value. Comments are also useful in an otherwise empty file to keep a placeholder for deployment in a time of need.

Using runtime overrides for deprecated features

The Envoy runtime is also a part of the Envoy feature deprecation process.

As described in the Envoy breaking change policy, feature deprecation in Envoy is in 3 phases: warn-by-default, fail-by-default, and code removal.

In the first phase, Envoy logs a warning to the warning log that the feature is deprecated and increments the deprecated_feature_use runtime stat. Users are encouraged to go to deprecated to see how to migrate to the new code path and make sure it is suitable for their use case.

In the second phase the message and filename will be added to runtime_features.cc and use of that configuration field will cause the config to be rejected by default. This fail-by-default mode can be overridden in runtime configuration by setting envoy.deprecated_features.filename.proto:fieldname to true. For example, for a deprecated field Foo.Bar.Eep in baz.proto set envoy.deprecated_features.baz.proto:Eep to true. Use of this override is strongly discouraged. Fatal-by-default configuration indicates that the removal of the old code paths is imminent. It is far better for both Envoy users and for Envoy contributors if any bugs or feature gaps with the new code paths are flushed out ahead of time, rather than after the code is removed!

Statistics

The file system runtime provider emits some statistics in the runtime. namespace.

Name Type Description
load_error Counter Total number of load attempts that resulted in an error
override_dir_not_exists Counter Total number of loads that did not use an override directory
override_dir_exists Counter Total number of loads that did use an override directory
load_success Counter Total number of load attempts that were successful
deprecated_feature_use Counter Total number of times deprecated features were used.
num_keys Gauge Number of keys currently loaded