Runtime configuration

Envoy supports “runtime” configuration (also known as “feature flags”). Runtime configuration can be used to modify various server settings without restarting Envoy. The runtime settings that are available depend on how the server is configured. Runtime guards which are not expected to be transient are documented in the relevant sections of the configuration guide.

Runtime guards are also used as a mechanism to disable new behavior or risky changes not otherwise guarded by configuration. Such changes will tend to introduce a temporary runtime guard that can be used to disable the new behavior/code path. The names of these runtime guards will be included in the release notes alongside an explanation of the change that warranted the runtime guard.

Due to this usage of runtime guards, some deployments might find it useful to set up dynamic (filesystem, RTDS, etc.) runtime configuration as a safety measure to be able to quickly disable the new behavior without having to revert to an older version of Envoy or redeploy it with a new set of static runtime flags.

See Runtime configuration as well as the contributing guide for more information.

Introducing a breaking change

“Breaking change” is a code change that may have observable impact on existing production traffic. When introducing a breaking change in Envoy, it desirable to surface in observability potential impact of enabling the change. Operators can assess the impact via access logs before older behaviors are modified or removed.

Observability for breaking changes is disabled by default. It can be enabled by setting the enable_breaking_changes_observability flag in Bootstrap. Envoy emits observability data for all breaking changes encountered in the traffic. Envoy stops emitting observability data when a breaking change flag is decommissioned.

Breaking changes are subject to the deprecation policy and are removed after deprecation window ends.

Adding a breaking change flag introduces Envoy reloadable flag with the envoy.reloadable_features.<flag_name> name and puts tracking state in the BreakingChangesTracker class. Breaking changes are enabled by setting the envoy.reloadable_features.<flag_name> flag in Envoy Runtime.

Presently breaking changes at the request level are tracked.

To introduce and track a breaking change:

  1. Add a new flag to source/common/runtime/breaking_changes_flags.h. Define the flag inside the ALL_BREAKING_CHANGES macro list using either the ENABLED or DISABLED macro:

    #define ALL_BREAKING_CHANGES(ENABLED, DISABLED) \
      ENABLED(my_breaking_change)
    

    Use the ENABLED macro for flags that are enabled by default and DISABLED macro for flags that are disabled by default.

  2. At the point of breaking change introduction, add the OBSERVED_REQUEST_BREAKING_CHANGE macro:

    OBSERVED_REQUEST_BREAKING_CHANGE(my_breaking_change, stream_info.filterState());
    

    This records the occurrence in the envoy.breaking_changes_tracker filter state object. The tracked breaking changes can then be logged in access logs using %FILTER_STATE(envoy.breaking_changes_tracker:PLAIN)%.

  3. To check in the code whether the breaking change is enabled or disabled, use the BREAKING_CHANGE_ENABLED macro:

    if (BREAKING_CHANGE_ENABLED(my_breaking_change)) {
      // Code path when the breaking change is enabled.
    } else {
      // Code path when the breaking change is disabled.
    }
    

Enabling observability for breaking changes

Operators can enable observability for breaking changes to monitor their occurrence in production traffic. Observability for breaking changes is disabled by default.

To enable observability:

  1. Enable the enable_breaking_changes_observability flag in the Bootstrap configuration:

    enable_breaking_changes_observability: true
    
  2. Configure access logging to include the envoy.breaking_changes_tracker filter state using %FILTER_STATE(envoy.breaking_changes_tracker:PLAIN)%:

    access_log:
      - name: envoy.access_loggers.file
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
          path: /dev/stdout
          log_format:
            text_format_source:
              inline_string: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% breaking_changes=\"%FILTER_STATE(envoy.breaking_changes_tracker:PLAIN)%\"\n"
    

When the enable_breaking_changes_observability flag in Bootstrap is enabled, breaking changes encountered during request processing will be recorded in the envoy.breaking_changes_tracker filter state and included in the access log. If the flag is disabled, no breaking changes are recorded in filter state.