.. _version_history_1.28.0: 1.28.0 (October 18, 2023) ========================== Incompatible behavior changes ----------------------------- *Changes that are expected to cause an incompatibility if applicable; deployment changes are likely required* * **UHV**: Introduced runtime flag ``envoy.reloadable_features.enable_universal_header_validator`` for toggling Universal Header Validator (UHV) on and off. The default value is off. This option is currently functional only when the ``ENVOY_ENABLE_UHV`` build flag is enabled. See https://github.com/envoyproxy/envoy/issues/10646 for more information about UHV. * **eds**: Introduced caching of EDS assignments when used with ADS. Prior to this change, Envoy required that EDS assignments were sent after an EDS cluster was updated. If no EDS assignment was received for the cluster, it ended up with an empty assignment. Following this change, after a cluster update, Envoy waits for an EDS assignment until :ref:`initial_fetch_timeout ` times out, and will then apply the cached assignment and finish updating the warmed cluster. This change is disabled by default, and can be enabled by setting the runtime flag ``envoy.restart_features.use_eds_cache_for_ads`` to ``true``. * **http**: Add runtime flag ``http.max_requests_per_io_cycle`` for setting the limit on the number of HTTP requests processed from a single connection in a single I/O cycle. Requests over this limit are processed in subsequent I/O cycles. This mitigates CPU starvation by connections that simultaneously send high number of requests by allowing requests from other connections to make progress. This runtime value can be set to 1 in the presence of abusive HTTP/2 or HTTP/3 connections. By default this limit is disabled. * **http**: Introduced a new runtime flag ``envoy.reloadable_features.no_downgrade_to_canonical_name`` to disable the name downgrading in the per filter config searching. See doc :ref:`Http filter route specific config ` or issue https://github.com/envoyproxy/envoy/issues/29461 for more specific detail and examples. * **http**: Switch from http_parser to BalsaParser for handling HTTP/1.1 traffic. See https://github.com/envoyproxy/envoy/issues/21245 for details. This behavioral change can be reverted by setting runtime flag ``envoy.reloadable_features.http1_use_balsa_parser`` to ``false``. * **jwt**: Changed behavior of the JWT extraction, passing entire token for validation, instead cut him in the non-Base64 character. This change can be reverted temporarily by setting the runtime guard ``envoy.reloadable_features.token_passed_entirely`` to ``false``. * **udp_proxy**: When the UDP proxy has session filters, choosing the upstream host and creating a socket only happens after iterating all ``onNewSession()`` calls for all the filters in the chain. Upstream host health check for each downstream datagram does not apply when there are session filters, and per-packet load balancing can't be used when there are session filters. * **zone-aware routing**: Zone-aware routing is now enabled even when the originating and upstream cluster have different numbers of zones. Previously, zone-aware routing was disabled in that case and the ``lb_zone_number_differs`` stat on the cluster was incremented. This behavioral change can be reverted by setting runtime guard ``envoy.reloadable_features.enable_zone_routing_different_zone_counts`` to ``false``. Additionally, zone-aware routing now works correctly even when the originating and upstream cluster have different zone sets. Previously, zone-aware routing would not route fairly in this case. To revert the entire change, set the runtime flag ``envoy.reloadable_features.locality_routing_use_new_routing_logic`` to ``false`` to get the old behavior and well-tested codepaths, undoing both changes. Minor behavior changes ---------------------- *Changes that may cause incompatibilities for some users, but should not for most* * **access_log**: When emitting grpc logs, only downstream filter state was used. Now, both downstream and upstream filter states will be tried to find the keys configured in filter_state_objects_to_log. * **alternate_protocols_cache_filter**: Changed the alternate protocols cache filter to get the cache from cluster config rather than filter config. This allows one downstream filter to be used with multiple clusters with different caches. This change can be reverted by setting runtime guard ``envoy.reloadable_features.use_cluster_cache_for_alt_protocols_filter`` to ``false``. * **ext_authz**: Don't append the local address to ``x-forwarded-for`` header when sending an http (not gRPC) auth request. This behavior can be reverted by setting runtime flag ``envoy.reloadable_features.ext_authz_http_send_original_xff`` to ``false``. * **ext_authz**: Removing any query parameter in the presence of repeated query parameter keys no longer drops the repeats. * **ext_proc**: Envoy will only take :ref:`mode_override ` when waiting for the header responses. It will be ignored if it is in other processing states. * **http**: Change the proxy status for ``UpstreamRequestTimeout`` to ``HttpResponseTimeout``. It can be disabled by the runtime guard ``envoy.reloadable_features.proxy_status_upstream_request_timeout``. * **http oauth2 filter**: Change HMAC cookie encoding to base64-encoded only. This change can be reverted temporarily by setting the runtime guard ``envoy.reloadable_features.hmac_base64_encoding_only`` to ``false``. * **local_rate_limit**: Added new configuration field :ref:`always_consume_default_token_bucket ` to allow for setting if default token bucket should be always consumed or only be consumed when there is no matching descriptor. * **outlier detection**: A node must stay in healthy state for at least one period of :ref:`check interval ` before ejection time multiplier is decremented. * **outlier detection**: Outlier detection will always respect ``max_ejection_percent`` now. This behavioral change can be reverted by setting runtime guard ``envoy.reloadable_features.check_mep_on_first_eject`` to ``false``. * **quic**: Enable QUICHE request and response headers validation. This behavior can be reverted by setting runtime flag ``envoy.reloadable_features.FLAGS_envoy_quic_reloadable_flag_quic_act_upon_invalid_header`` to ``false``. * **redis**: The redis network filter :ref:`connection_rate_limit_per_sec ` must be greater than 0. A config that sets this value to 0 will be rejected. * **router**: Enable copying response_code from the upstream stream_info onto the downstream stream_info. This behavior can be reverted by setting runtime guard ``envoy.reloadable_features.copy_response_code_to_downstream_stream_info`` to ``false``. * **router**: Enable environment_variable in router direct response. * **tls**: changed ssl failure reason format in ssl socket for a better handling. It can be disabled by the runtime guard ``envoy.reloadable_features.ssl_transport_failure_reason_format``. * **tls_inspector**: Updated the security posture of the :ref:`TLS inspector listener filter ` to robust against untrusted downstreams and upstreams. * **upstream**: Deprecate code path of legacy upstream load balancer. Ideally, this is implementation detail changes and should not affect users. However, if there is any user who encounters issues, this behavior can be reverted by setting runtime flag ``envoy_reloadable_features_convert_legacy_lb_config`` to false. * **xds**: Set the lower bound of :ref:`fill_rate ` to once per year. Values lower than once per year will automatically be set to that value. Bug fixes --------- *Changes expected to improve the state of the world and are unlikely to have negative effects* * **aws signer**: Fixed a bug where expiration timestamp on task roles failed to validate. This causes failure of credential caching which results in constant hits to the task role metadata URL. * **connection limit**: Fixed a use-after-free bug in the connection limit filter. * **dns**: Fixed a bug where dns response was not always conforming `RFC 2181 `_ for TTL values. Previously a malicious user could add a TTL greater than 2^31 - 1, and with c-ares library using 32 bit signed int data type would overflow and send a negative TTL. * **dns**: Fixed a bug where when ``respect_dns_ttl`` was set to ``true``, c-ares dns resolver only considered address record for TTL calculation while ignoring CNAME records TTL. Now when ``respect_dns_ttl`` is set to ``true`` minimum of all TTL records is considered. * **docker/publishing**: Update base images to resolve various glibc vulnerabilities. * **extension_discovery_service**: Fixed a bug causing crash if ECDS is used with upstream HTTP filters. * **healthcheck**: The default behavior of unejecting outlier-detection-ejected host on successful active health checking can be disabled by setting :ref:`outlier_detection.successful_active_health_check_uneject_host ` to ``false``. This new configuration flag is a substitute for the removed runtime option ``envoy.reloadable_features_successful_active_health_check_uneject_host``. * **http**: Close HTTP/2 and HTTP/3 connections that prematurely reset streams. The runtime key ``overload.premature_reset_min_stream_lifetime_seconds`` determines the interval where received stream reset is considered premature (with 1 second default). The runtime key ``overload.premature_reset_total_stream_count``, with the default value of 500, determines the number of requests received from a connection before the check for premature resets is applied. The connection is disconnected if more than 50% of resets are premature, or if the number of suspect streams is already large enough to guarantee that more than 50% of the streams will be suspect upon reaching the total stream threshold (even if all the remaining streams are considered benign). Setting the runtime key ``envoy.restart_features.send_goaway_for_premature_rst_streams`` to ``false`` completely disables this check. * **http**: Fixed a bug that could cause metadata to be decoded after a local reply has been triggered. Can be disabled by setting ``envoy.reloadable_features.stop_decode_metadata_on_local_reply`` to false. * **http1**: Fixed a bug where HTTP/1.1 requests with ``Connection: close`` header is handled differently if the requested is internally redirected. Without internal redirect, the response will also have a ``Connection: close`` header and the connection will be closed after finishing that request. Requests with internal redirect should be handled in the same way. This behavior can be reverted by setting runtime ``envoy.reloadable_features.http1_connection_close_header_in_redirect`` to ``false``. * **redis**: Fixed a bug where redis key formatter is using the closed stream because of life time issues. * **redis**: Fixed a bug where redis key with % in the key is failing with a validation error. * **router check tool**: Fixed a bug where the route coverage is not correctly calculated when a route has weighted clusters. * **subset load balancer**: Fixed a bug where :ref:`overprovisioning_factor` and :ref:`weighted_priority_health ` values were not respected when subset load balacing was enabled. The default values of ``140`` and ``false`` were always used. * **tls**: Fixed a bug where handshake may fail when both private key provider and cert validation are set. * **tracing**: Fixed a bug in the Datadog tracer where Datadog's "operation name" field would contain what should be in the "resource name" field. * **unix domain sockets**: Fixed a crash on some versions of macOS when using a listener on a unix-domain socket. * **xds**: Fix a bug where the nonce was not reset after reconnecting to the xDS server, when using State-of-the-World. Removed config or runtime ------------------------- *Normally occurs at the end of the* :ref:`deprecation period ` * **dns**: Removed ``envoy.restart_features.use_apple_api_for_dns_lookups`` and legacy code paths. * **ext_authz**: Removed ``envoy.reloadable_features.http_ext_auth_failure_mode_allow_header_add`` runtime option and substituted it with :ref:`failure_mode_allow_header_add ` configuration flag. * **healthcheck**: Removed ``envoy.reloadable_features_successful_active_health_check_uneject_host`` runtime option and substituted it with :ref:`outlier_detection.successful_active_health_check_uneject_host ` outlier detection configuration flag. * **http filters**: Removed ``envoy_reloadable_features_http_filter_avoid_reentrant_local_reply`` runtime flag and legacy code paths. * **listener**: Removed ``envoy.reloadable_features.enable_update_listener_socket_options`` runtime flag and legacy code paths. * **quic**: Removed ``envoy.reloadable_features.reject_require_client_certificate_with_quic`` and legacy code paths. * **runtime**: Removed ``envoy.restart_features.remove_runtime_singleton`` and legacy code paths. * **runtime**: Removed ``envoy_reloadable_features_append_query_parameters_path_rewriter`` and legacy code paths. * **tcp**: Removed runtime key ``envoy.reloadable_features.tcp_pool_idle_timeout``. * **tcp_proxy**: Removed ``envoy_reloadable_features_finish_reading_on_decode_trailers`` runtime flag and legacy code paths. * **xDS**: Removed ``envoy.restart_features.explicit_wildcard_resource`` and legacy code paths. New features ------------ * **access_log**: Added ``%RESPONSE_FLAGS_LONG%`` substitution string, that will output a pascal case string representing the response flags. The output response flags will correspond with ``%RESPONSE_FLAGS%``, only with a long textual string representation. * **access_log**: Added a field lookup to ``%FILTER_STATE%`` for objects that have reflection enabled. * **access_logs**: Added :ref:`json_format_options ` config option to support JSON output formatting and the :ref:`sort_properties ` option to print the JSON output with sorted properties. * **admin**: Added a new ``skip_exit`` query parameter to ``/drain_listeners`` to skip exiting after the drain period. * **admin_logging**: added support for glob control of fine-grain loggers in admin /logging interface. * **compression**: Added :ref:`remove_accept_encoding_header ` for per-route configuration of this value. * **config**: Added the capability to defer broadcasting of certain cluster (CDS, EDS) to worker threads from the main thread. This optimization can save significant amount of memory in cases where there are (1) a large number of workers and (2) a large amount of config, most of which is unused. This capability is guarded by :ref:`enable_deferred_cluster_creation `. * **ext_proc**: Added :ref:`disable_immediate_response ` config API to ignore the :ref:`immediate_response ` message from the external processing server. * **extension_discovery_service**: Added ECDS support for :ref:`downstream network filters `. * **extension_discovery_service**: Added ECDS support for :ref:`upstream network filters `. * **extension_discovery_service**: Added metric ``listener.listener_stat.network_extension_config_missing`` to track closed connections due to missing config. * **filters**: Added filters to update the filter state for :ref:`the HTTP requests ` and :ref:`the TCP connections `. * **geoip**: Added support for :ref:`Maxmind geolocation provider `. * **http**: Added :ref:`Json-To-Metadata filter `. * **http**: Added :ref:`disabled ` flag to disable a filter in the filter chain by default. The filter can be enabled explicitly by setting valid per filter config in the route configuration. * **listener**: Added possibility to track global downstream connection limit via :ref:`downstream connections monitor ` in overload manager. * **lua**: Added :ref:`downstreamRemoteAddress() ` method to the Stream info object API. * **oauth2**: Added :ref:`use_refresh_token ` to support updating an access token via a refresh token if that is provided by authorization server. * **original_dst**: added support for the internal listener address recovery using the original destination listener filter. * **otlp_stats_sink**: Added :ref:`stats prefix option ` to OTLP stats sink that enables adding a static prefix to all stats flushed by this sink. * **quic**: Added support for QUIC listener filters with ECDS support reusing the same config API :ref:`listener_filters ` as TCP does. * **redis**: Added new configuration field :ref:`read_command_policy ` to specify Envoy should route read commands to another cluster. * **redis**: Added support for ``lmove`` command. * **redis**: Added support for ``time`` command (returns a local response). * **router**: Added ``metadata`` support for :ref:`virtual host ` and :ref:`route configuration `. * **tap**: Added :ref:`custom_sink ` type to enable writing tap data out to a custom sink extension. * **tap**: Added :ref:`record_downstream_connection ` to control writing downstream connection address info in trace output. * **tap**: Added :ref:`record_headers_received_time ` to control writing request and response headers received time in trace output. * **tcp**: Added the support to detect and send TCP RST for raw buffer socket based connections. This is currently supported on Linux only. It can be disabled by the runtime guard ``envoy_reloadable_features_detect_and_raise_rst_tcp_connection``. * **tls**: Added :ref:`disable_stateful_session_resumption ` config option to disable stateful TLS session resumption. * **tls**: Added fallback :ref:`fallback ` to support private key provider to fallback to boringssl TLS handshake. If the private key provider isn't available (eg. the required hardware capability doesn't existed), Envoy will fallback to the BoringSSL default implementation when the fallback is ``true``. The default value is ``false``. * **tracing**: Added :ref:`spawn_upstream_span ` to control whether to create separate upstream span for upstream request. * **tracing**: Added support for exporting spans via HTTP on the OpenTelemetry tracer. * **udp_proxy**: Added :ref:`http_capsule ` UDP session filter that can be used to encapsule or decapsulate UDP datagrams in HTTP, when used for UDP tunneling. * **udp_proxy**: Added :ref:`session_filters ` config to support optional filters that will run for each upstream UDP session. More information can be found in the UDP proxy documentation. * **udp_proxy**: Added ``injectDatagramToFilterChain()`` callback to UDP session filters that allows session filters to inject datagrams downstream or upstream the filter chain during a filter chain iteration. This can be used, for example, by session filters that are required to buffer datagrams due to an asynchronous call. * **udp_proxy**: added :ref:`dynamic_forward_proxy ` UDP session filter that can be used to have dynamic forward proxy UDP flows, when used in conjunction with another session filter that sets required filter state values. * **upstream**: Added :ref:`allow_redundant_keys ` to suppport redundant keys in request metadata for subset load balancing. * **upstream**: Added the ability to specify a custom upstream local address selector using :ref:`local_address_selector `. * **zookeeper**: Added support for emitting per opcode request bytes metrics via :ref:`enable_per_opcode_request_bytes_metrics `. added support for emitting per opcode response bytes metrics via :ref:`enable_per_opcode_response_bytes_metrics `. Deprecated ---------- * **listener**: deprecated runtime key ``overload.global_downstream_max_connections`` in favor of :ref:`downstream connections monitor `. * **tracing**: :ref:`start_child_span ` is deprecated by :ref:`spawn_upstream_span `. Please use the new field to control whether to create separate upstream span for upstream request. * **tracing**: OpenTracing is deprecated and will be removed at version 1.30, since the upstream project has been abandoned. * **tracing**: Opencensus is deprecated and will be removed at version 1.30, since the upstream project has been abandoned.