1.28.0 (October 18, 2023)
Incompatible behavior changes
Changes that are expected to cause an incompatibility if applicable; deployment changes are likely required
UHV: Introduced runtime flag
envoy.reloadable_features.enable_universal_header_validatorfor toggling Universal Header Validator (UHV) on and off. The default value is off. This option is currently functional only when the
ENVOY_ENABLE_UHVbuild flag is enabled. See https://github.com/envoyproxy/envoy/issues/10646 for more information about UHV.
eds: Introduced caching of EDS assignments when used with ADS. Prior to this change, Envoy required that EDS assignments were sent after an EDS cluster was updated. If no EDS assignment was received for the cluster, it ended up with an empty assignment. Following this change, after a cluster update, Envoy waits for an EDS assignment until initial_fetch_timeout times out, and will then apply the cached assignment and finish updating the warmed cluster. This change is disabled by default, and can be enabled by setting the runtime flag
http: Add runtime flag
http.max_requests_per_io_cyclefor setting the limit on the number of HTTP requests processed from a single connection in a single I/O cycle. Requests over this limit are processed in subsequent I/O cycles. This mitigates CPU starvation by connections that simultaneously send high number of requests by allowing requests from other connections to make progress. This runtime value can be set to 1 in the presence of abusive HTTP/2 or HTTP/3 connections. By default this limit is disabled.
http: Introduced a new runtime flag
envoy.reloadable_features.no_downgrade_to_canonical_nameto disable the name downgrading in the per filter config searching. See doc Http filter route specific config or issue https://github.com/envoyproxy/envoy/issues/29461 for more specific detail and examples.
http: Switch from http_parser to BalsaParser for handling HTTP/1.1 traffic. See https://github.com/envoyproxy/envoy/issues/21245 for details. This behavioral change can be reverted by setting runtime flag
jwt: Changed behavior of the JWT extraction, passing entire token for validation, instead cut him in the non-Base64 character. This change can be reverted temporarily by setting the runtime guard
udp_proxy: When the UDP proxy has session filters, choosing the upstream host and creating a socket only happens after iterating all
onNewSession()calls for all the filters in the chain. Upstream host health check for each downstream datagram does not apply when there are session filters, and per-packet load balancing can’t be used when there are session filters.
zone-aware routing: Zone-aware routing is now enabled even when the originating and upstream cluster have different numbers of zones. Previously, zone-aware routing was disabled in that case and the
lb_zone_number_differsstat on the cluster was incremented. This behavioral change can be reverted by setting runtime guard
false. Additionally, zone-aware routing now works correctly even when the originating and upstream cluster have different zone sets. Previously, zone-aware routing would not route fairly in this case. To revert the entire change, set the runtime flag
falseto get the old behavior and well-tested codepaths, undoing both changes.
Minor behavior changes
Changes that may cause incompatibilities for some users, but should not for most
access_log: When emitting grpc logs, only downstream filter state was used. Now, both downstream and upstream filter states will be tried to find the keys configured in filter_state_objects_to_log.
alternate_protocols_cache_filter: Changed the alternate protocols cache filter to get the cache from cluster config rather than filter config. This allows one downstream filter to be used with multiple clusters with different caches. This change can be reverted by setting runtime guard
ext_authz: Don’t append the local address to
x-forwarded-forheader when sending an http (not gRPC) auth request. This behavior can be reverted by setting runtime flag
ext_authz: Removing any query parameter in the presence of repeated query parameter keys no longer drops the repeats.
ext_proc: Envoy will only take mode_override when waiting for the header responses. It will be ignored if it is in other processing states.
http: Change the proxy status for
HttpResponseTimeout. It can be disabled by the runtime guard
http oauth2 filter: Change HMAC cookie encoding to base64-encoded only. This change can be reverted temporarily by setting the runtime guard
local_rate_limit: Added new configuration field always_consume_default_token_bucket to allow for setting if default token bucket should be always consumed or only be consumed when there is no matching descriptor.
outlier detection: A node must stay in healthy state for at least one period of check interval before ejection time multiplier is decremented.
outlier detection: Outlier detection will always respect
max_ejection_percentnow. This behavioral change can be reverted by setting runtime guard
quic: Enable QUICHE request and response headers validation. This behavior can be reverted by setting runtime flag
redis: The redis network filter connection_rate_limit_per_sec must be greater than 0. A config that sets this value to 0 will be rejected.
router: Enable copying response_code from the upstream stream_info onto the downstream stream_info. This behavior can be reverted by setting runtime guard
router: Enable environment_variable in router direct response.
tls: changed ssl failure reason format in ssl socket for a better handling. It can be disabled by the runtime guard
tls_inspector: Updated the security posture of the TLS inspector listener filter to robust against untrusted downstreams and upstreams.
upstream: Deprecate code path of legacy upstream load balancer. Ideally, this is implementation detail changes and should not affect users. However, if there is any user who encounters issues, this behavior can be reverted by setting runtime flag
xds: Set the lower bound of fill_rate to once per year. Values lower than once per year will automatically be set to that value.
Changes expected to improve the state of the world and are unlikely to have negative effects
aws signer: Fixed a bug where expiration timestamp on task roles failed to validate. This causes failure of credential caching which results in constant hits to the task role metadata URL.
connection limit: Fixed a use-after-free bug in the connection limit filter.
dns: Fixed a bug where dns response was not always conforming RFC 2181 for TTL values. Previously a malicious user could add a TTL greater than 2^31 - 1, and with c-ares library using 32 bit signed int data type would overflow and send a negative TTL.
dns: Fixed a bug where when
respect_dns_ttlwas set to
true, c-ares dns resolver only considered address record for TTL calculation while ignoring CNAME records TTL. Now when
respect_dns_ttlis set to
trueminimum of all TTL records is considered.
docker/publishing: Update base images to resolve various glibc vulnerabilities.
extension_discovery_service: Fixed a bug causing crash if ECDS is used with upstream HTTP filters.
healthcheck: The default behavior of unejecting outlier-detection-ejected host on successful active health checking can be disabled by setting outlier_detection.successful_active_health_check_uneject_host to
false. This new configuration flag is a substitute for the removed runtime option
http: Close HTTP/2 and HTTP/3 connections that prematurely reset streams. The runtime key
overload.premature_reset_min_stream_lifetime_secondsdetermines the interval where received stream reset is considered premature (with 1 second default). The runtime key
overload.premature_reset_total_stream_count, with the default value of 500, determines the number of requests received from a connection before the check for premature resets is applied. The connection is disconnected if more than 50% of resets are premature, or if the number of suspect streams is already large enough to guarantee that more than 50% of the streams will be suspect upon reaching the total stream threshold (even if all the remaining streams are considered benign). Setting the runtime key
falsecompletely disables this check.
http: Fixed a bug that could cause metadata to be decoded after a local reply has been triggered. Can be disabled by setting
http1: Fixed a bug where HTTP/1.1 requests with
Connection: closeheader is handled differently if the requested is internally redirected. Without internal redirect, the response will also have a
Connection: closeheader and the connection will be closed after finishing that request. Requests with internal redirect should be handled in the same way. This behavior can be reverted by setting runtime
redis: Fixed a bug where redis key formatter is using the closed stream because of life time issues.
redis: Fixed a bug where redis key with % in the key is failing with a validation error.
router check tool: Fixed a bug where the route coverage is not correctly calculated when a route has weighted clusters.
subset load balancer: Fixed a bug where overprovisioning_factor and weighted_priority_health values were not respected when subset load balacing was enabled. The default values of
falsewere always used.
tls: Fixed a bug where handshake may fail when both private key provider and cert validation are set.
tracing: Fixed a bug in the Datadog tracer where Datadog’s “operation name” field would contain what should be in the “resource name” field.
unix domain sockets: Fixed a crash on some versions of macOS when using a listener on a unix-domain socket.
xds: Fix a bug where the nonce was not reset after reconnecting to the xDS server, when using State-of-the-World.
Removed config or runtime
Normally occurs at the end of the deprecation period
envoy.restart_features.use_apple_api_for_dns_lookupsand legacy code paths.
envoy.reloadable_features.http_ext_auth_failure_mode_allow_header_addruntime option and substituted it with failure_mode_allow_header_add configuration flag.
envoy.reloadable_features_successful_active_health_check_uneject_hostruntime option and substituted it with outlier_detection.successful_active_health_check_uneject_host outlier detection configuration flag.
http filters: Removed
envoy_reloadable_features_http_filter_avoid_reentrant_local_replyruntime flag and legacy code paths.
envoy.reloadable_features.enable_update_listener_socket_optionsruntime flag and legacy code paths.
envoy.reloadable_features.reject_require_client_certificate_with_quicand legacy code paths.
envoy.restart_features.remove_runtime_singletonand legacy code paths.
envoy_reloadable_features_append_query_parameters_path_rewriterand legacy code paths.
tcp: Removed runtime key
envoy_reloadable_features_finish_reading_on_decode_trailersruntime flag and legacy code paths.
envoy.restart_features.explicit_wildcard_resourceand legacy code paths.
%RESPONSE_FLAGS_LONG%substitution string, that will output a pascal case string representing the response flags. The output response flags will correspond with
%RESPONSE_FLAGS%, only with a long textual string representation.
access_log: Added a field lookup to
%FILTER_STATE%for objects that have reflection enabled.
admin: Added a new
skip_exitquery parameter to
/drain_listenersto skip exiting after the drain period.
admin_logging: added support for glob control of fine-grain loggers in admin /logging interface.
compression: Added remove_accept_encoding_header for per-route configuration of this value.
config: Added the capability to defer broadcasting of certain cluster (CDS, EDS) to worker threads from the main thread. This optimization can save significant amount of memory in cases where there are (1) a large number of workers and (2) a large amount of config, most of which is unused. This capability is guarded by enable_deferred_cluster_creation.
extension_discovery_service: Added ECDS support for downstream network filters.
extension_discovery_service: Added ECDS support for upstream network filters.
extension_discovery_service: Added metric
listener.listener_stat.network_extension_config_missingto track closed connections due to missing config.
geoip: Added support for Maxmind geolocation provider.
http: Added Json-To-Metadata filter.
http: Added disabled flag to disable a filter in the filter chain by default. The filter can be enabled explicitly by setting valid per filter config in the route configuration.
listener: Added possibility to track global downstream connection limit via downstream connections monitor in overload manager.
lua: Added downstreamRemoteAddress() method to the Stream info object API.
oauth2: Added use_refresh_token to support updating an access token via a refresh token if that is provided by authorization server.
original_dst: added support for the internal listener address recovery using the original destination listener filter.
otlp_stats_sink: Added stats prefix option to OTLP stats sink that enables adding a static prefix to all stats flushed by this sink.
quic: Added support for QUIC listener filters with ECDS support reusing the same config API listener_filters as TCP does.
redis: Added new configuration field read_command_policy to specify Envoy should route read commands to another cluster.
redis: Added support for
redis: Added support for
timecommand (returns a local response).
tap: Added custom_sink type to enable writing tap data out to a custom sink extension.
tap: Added record_downstream_connection to control writing downstream connection address info in trace output.
tap: Added record_headers_received_time to control writing request and response headers received time in trace output.
tcp: Added the support to detect and send TCP RST for raw buffer socket based connections. This is currently supported on Linux only. It can be disabled by the runtime guard
tls: Added disable_stateful_session_resumption config option to disable stateful TLS session resumption.
tls: Added fallback fallback to support private key provider to fallback to boringssl TLS handshake. If the private key provider isn’t available (eg. the required hardware capability doesn’t existed), Envoy will fallback to the BoringSSL default implementation when the fallback is
true. The default value is
tracing: Added spawn_upstream_span to control whether to create separate upstream span for upstream request.
tracing: Added support for exporting spans via HTTP on the OpenTelemetry tracer.
udp_proxy: Added http_capsule UDP session filter that can be used to encapsule or decapsulate UDP datagrams in HTTP, when used for UDP tunneling.
udp_proxy: Added session_filters config to support optional filters that will run for each upstream UDP session. More information can be found in the UDP proxy documentation.
injectDatagramToFilterChain()callback to UDP session filters that allows session filters to inject datagrams downstream or upstream the filter chain during a filter chain iteration. This can be used, for example, by session filters that are required to buffer datagrams due to an asynchronous call.
udp_proxy: added dynamic_forward_proxy UDP session filter that can be used to have dynamic forward proxy UDP flows, when used in conjunction with another session filter that sets required filter state values.
upstream: Added allow_redundant_keys to suppport redundant keys in request metadata for subset load balancing.
upstream: Added the ability to specify a custom upstream local address selector using local_address_selector.
zookeeper: Added support for emitting per opcode request bytes metrics via enable_per_opcode_request_bytes_metrics. added support for emitting per opcode response bytes metrics via enable_per_opcode_response_bytes_metrics.
listener: deprecated runtime key
overload.global_downstream_max_connectionsin favor of downstream connections monitor.
tracing: OpenTracing is deprecated and will be removed at version 1.30, since the upstream project has been abandoned.
tracing: Opencensus is deprecated and will be removed at version 1.30, since the upstream project has been abandoned.