.. _version_history_1.27.0: 1.27.0 (July 26, 2023) ======================= Incompatible behavior changes ----------------------------- *Changes that are expected to cause an incompatibility if applicable; deployment changes are likely required* * **active health check**: Preserve the active-health check status of a host after a cluster/assignment update. This is now preserved in cases where the assignment updates a host's locality. This behavioral change can be temporarily reverted by setting the runtime flag ``envoy.reloadable_features.keep_endpoint_active_hc_status_on_locality_update`` to ``false``. * **build**: Moved the subset, ring_hash, and maglev LB code into extensions. If you use these load balancers and override :repo:`bazel/extensions_build_config.bzl` you will need to include them explicitly. * **build**: Moved xDS code extensions. If you use the xDS and override :repo:`bazel/extensions_build_config.bzl` you will need to include the new config_subscriptions explicitly. * **ext_proc**: Apply header mutation rules from the ext_proc config to the ``ImmediateResponse``. This behavior change can be temporarily reverted by setting the runtime guard ``envoy_reloadable_features_immediate_response_use_filter_mutation_rule`` to false. * **ext_proc**: The proto field :ref:`value ` type is string. This make it unable to support enconding non-utf8 characters in the ``HeaderValue`` message. To support sending header value with non-utf8 characters, a new proto field is added in the HeaderValue message: :ref:`raw_value `. The header values are now encoded in this ``raw_value`` field when Envoy ext_proc filter sending and receiving messages from the ext_proc server. This behavioral change can be temporarily reverted by setting the runtime guard ``envoy_reloadable_features_send_header_raw_value`` to ``false``. * **http**: When ``append_x_forwarded_host`` is enabled for a given route action it is now only appended iff it is different from the last value in the list. This resolves issues where a retry caused the same value to be appended multiple times. This behavioral change can be temporarily reverted by setting runtime guard ``envoy_reloadable_features_append_xfh_idempotent`` to ``false``. * **quic**: Add a default false runtime flag ``envoy.reloadable_features.quic_reject_all`` to disable QUIC listener if needed. * **stats tls**: Fixed metric tag extraction so that TLS parameters are properly extracted from the stats, both for listeners and clusters. This changes the Prometheus names from ``envoy_listener_ssl_ciphers_ECDHE_RSA_AES128_GCM_SHA256{envoy_listener_address="0.0.0.0_10000"}`` to ``envoy_listener_ssl_ciphers{envoy_listener_address="0.0.0.0_10000", envoy_ssl_cipher="ECDHE_RSA_AES128_GCM_SHA256"}``, and similar for ``envoy_listener_ssl_versions_TLSv1_2``, ``envoy_cluster_ssl_versions_TLSv1_2``, ``envoy_listener_ssl_curves_P_256``, ``envoy_cluster_ssl_curves_P_256``, ``envoy_listener_ssl_sigalgs_rsa_pss_rsae_sha256``. Minor behavior changes ---------------------- *Changes that may cause incompatibilities for some users, but should not for most* * **aws**: Added support for fetching credentials from the AWS credentials file, which only happens if credentials cannot be fetched from environment variables. This behavioral change can be reverted by setting runtime guard ``envoy.reloadable_features.enable_aws_credentials_file`` to ``false``. * **connection pool**: Increase granularity mapping connection pool failures to specific stream failure reasons to make it more transparent why the stream is reset when a connection pool's connection fails. * **custom response**: The filter now traverses matchers from most specific to least specific per filter config till a match is found for the response. * **dns**: Changing the DNS cache to use ``host:port`` as the cache key rather than ``host``. This allows a downstream DFP filter to serve both secure and insecure clusters. This behavioral change can be reverted by setting runtime flag ``envoy.reloadable_features.dfp_mixed_scheme`` to ``false``. * **eds**: Added the ability to specify mulitple addresses for a host in an EDS cluster. Connections to the host with more than one address will be established using the Happy Eyeballs algorithm. * **ext_authz**: Forward :ref:`typed_filter_metadata ` selected by ``typed_metadata_context_namespaces`` and :ref:`filter_metadata ` selected by :ref:`metadata_context_namespaces ` from connection metadata to external auth service. This is addition to the current behavior of forwarding request metadata. In the event of both connection and request metadata containing the requested metadata the request value will be provided. * **ext_proc**: Filter metadata containing ext proc stats has been moved from ``ext-proc-logging-info`` to a namespace corresponding to the name of the ext_proc filter. * **ext_proc**: When :ref:`clear_route_cache ` is set, ext_proc will check for header mutations beforce clearing the route cache. Failures due to this check will be counted under the ``clear_route_cache_ignored`` stat. * **http**: Envoy will now lower case scheme values by default. This behaviorial change can be temporarily reverted by setting runtime guard ``envoy.reloadable_features.lowercase_scheme`` to ``false``. * **http**: Round trip time will not be refreshed for every request by default. And if this is necessary, it can be enabled by setting runtime guard ``envoy.reloadable_features.refresh_rtt_after_request`` to ``true``. * **http cookies**: Changed internal format of http cookie to protobuf and added expiry timestamp. Processing expired cookie results in selection of a new upstream host and sending a new cookie to the client. Previous format of the cookie is still accepted, but is planned to be obsoleted in the future. This behavior change can be reverted by setting ``envoy.reloadable_features.stateful_session_encode_ttl_in_cookie`` to ``false``. * **http1**: Allowing mixed case schemes in absolute urls (e.g. HtTp://www.google.com). Mixed case schemes will be normalized to the lower cased equivalents before being forwarded upstream. This behavior can be reverted by setting runtime flag ``envoy.reloadable_features.allow_absolute_url_with_mixed_scheme`` to false. * **http1**: The HTTP1 server-side codec no longer considers encoding 1xx headers as starting the response. This allows the codec to raise protocol errors, sending detailed local replies instead of just closing the connection. This behavior can be reverted by setting runtime flag ``envoy.reloadable_features.http1_allow_codec_error_response_after_1xx_headers`` to ``false``. * **overload manager**: Changed behavior of the overload manager to error on unknown overload manager actions. Prior it would silently fail. This change can be reverted temporarily by setting the runtime guard ``envoy.reloadable_features.overload_manager_error_unknown_action`` to false. * **resource_monitors**: Changed behavior of the fixed heap monitor to count unused mapped pages as free memory. This change can be reverted temporarily by setting the runtime guard ``envoy.reloadable_features.count_unused_mapped_pages_as_free`` to ``false``. * **router**: Added check for existing metadata before setting metadata due to ``auto_sni``, ``auto_san_validation``, or ``override_auto_sni_header`` to prevent triggering ``ENVOY_BUG`` when an earlier filter has set the metadata. * **stats**: Added new type of gauge with type hidden. These stats are hidden from admin/stats-sinks but can shown with a query-parameter of ``/stats?hidden=include`` or ``/stats?hidden=showonly``. * **uhv**: Allow malformed URL encoded triplets in the default header validator. This behavior can be reverted by setting runtime flag ``envoy.reloadable_features.uhv_allow_malformed_url_encoding`` to ``false``, in which case requests with malformed URL encoded triplets in path are rejected. This setting is only applicable when the Unversal Header Validator is enabled and has no effect otherwise. * **uhv**: Preserve case of %-encoded triplets in the default header validator. This behavior can be reverted by setting runtime flag ``envoy.reloadable_features.uhv_preserve_url_encoded_case`` to ``false``, in which case %-encoded triplets are normalized to uppercase characters. This setting is only applicable when the Unversal Header Validator is enabled and has no effect otherwise. * **upstream**: Changed behavior of the unpausing connect with 2xx status codes. This change can be reverted temporarily by setting the runtime guard ``envoy.reloadable_features.upstream_allow_connect_with_2xx`` to ``false``. Bug fixes --------- *Changes expected to improve the state of the world and are unlikely to have negative effects* * **boringssl**: Fixed the crash that occurs when contrib is compiled with ``boringssl=fips`` defined. * **cors**: Fix a use-after-free bug that occurs in the CORS filter if the ``origin`` header is removed between request header decoding and response header encoding. Fix `CVE-2023-35943 `_. * **dependency**: Update C-ares -> 1.91.1 to resolve: - `CVE-2023-31130 `_. - `CVE-2023-31147 `_. - `CVE-2023-31124 `_. - `CVE-2023-32067 `_. * **dependency**: Update Wasmtime and related deps -> 9.0.3 to resolve `CVE-2023-30624 `_. * **ext_authz**: Fix a bug where the ext_authz filter will ignore the request body when the :ref:`pack_as_bytes ` is set to ``true`` and HTTP authorization service is configured. * **ext_authz**: Fix a bug where the ext_authz filter will remove non UTF-8 characters from the body of a request when configured to use :ref:`http_service `, if configured to send the body. * **http**: Switched Envoy internal scheme checks from case sensitive to case insensitive. This behaviorial change can be temporarily reverted by setting runtime guard ``envoy.reloadable_features.handle_uppercase_scheme`` to ``false``. Fix `CVE-2023-35944 `_. * **http**: The :ref:`is_optional ` field of HTTP filter can only be used for configuration loading of :ref:`HTTP filter ` and will be ignored for loading of route or virtual host level filter config. This behavioral change can be temporarily reverted by setting runtime guard ``envoy.reloadable_features.ignore_optional_option_from_hcm_for_route_config`` to ``false``. You can also use :ref:`route/virtual host optional flag ` as a replacement of the feature. * **http2**: Fix memory leak in nghttp2 when scheduled requests are cancelled due to the ``GOAWAY`` frame being received from the upstream service. * **logging**: Do not display GRPC_STATUS_NUMBER for non gRPC requests. This behavioral change can be temporarily reverted by setting runtime guard ``envoy.reloadable_features.validate_grpc_header_before_log_grpc_status`` to ``false``. * **maglev loadbalancer**: Fixes maglev stability problem. Previously, maglev returns slightly different backend assignment from the same backends and keys. * **oauth2**: Fixed a cookie validator bug that meant the HMAC calculation could be the same for different payloads. This prevents malicious clients from constructing credentials with permanent validity in some specific scenarios. * **oauth2**: The Max-Age attribute of Set-Cookie HTTP response header was being assigned a value representing Seconds Since the Epoch, causing cookies to expire in ~53 years. This was fixed an now it is being assigned a value representing the number of seconds until the cookie expires. This behavioral change can be temporarily reverted by setting runtime guard ``envoy.reloadable_features.oauth_use_standard_max_age_value`` to ``false``. * **oauth2**: The ``httpOnly`` attribute for ``Set-Cookie`` for tokens in HTTP response header was missing, causing tokens to be accessible from the JavaScript making the apps vulnerable. This was fixed now by marking the cookie as ``httpOnly``. This behavioral change can be temporarily reverted by setting runtime guard ``envoy.reloadable_features.oauth_make_token_cookie_httponly`` to ``false``. Fix `CVE-2023-35941 `_. * **opentelemetry/grpc/access log**: Fixed a bug in the open telemetry access logger. This logger now uses the server scope for stats instead of the listener's global scope. This fixes a use-after-free that can occur if the listener is drained but the cached gRPC access logger uses the listener's global scope for stats. Fix `CVE-2023-35942 `_. * **original_dst**: Fixes an issue with the ``ORIGINAL_DST`` cluster cleanup timer lifetime, which can occur if the cluster is removed while the timer is armed. * **postgres**: Enable parsing when using upstream SSL. * **redis**: Fixes a bug where redis transactions do not work properly when redis traffic is mirrored. * **redis_proxy**: Fixes a bug where route properties such as ``key_formatter``, ``prefix`` and ``remove_prefix`` do not take effect when configured for :ref:`catch_all_route `. * **router**: Fixed the bug that updating :ref:`scope_key_builder ` of SRDS config doesn't work and multiple HCM share the same ``scope_key_builder``. * **tcp_proxy**: Fixed assert crash when multiple ``readDisable`` are called for TCP tunneling scenarios, by allowing multiple calls. This will also cause stats that indicate disable or enable of downstream read to be flushed only once per actual disabling or enabling. * **tls**: Fix build FIPS compliance when using both FIPS mode and Wasm extensions (``--define boringssl=fips`` and ``--define wasm=v8``). * **upstream**: Fixes a bug where the ``healthStatus()`` method of host return incorrect health status when the host status is updated by the EDS. * **upstream**: Fixes a bug where the ``healthStatus()`` method of host return unmatched health status with the ``coarseHealth()`` method. Removed config or runtime ------------------------- *Normally occurs at the end of the* :ref:`deprecation period ` * **config**: Removed runtime key ``envoy.reloadable_features.delta_xds_subscription_state_tracking_fix`` and legacy code paths. * **grpc_stats**: Removed runtime key ``envoy.deprecated_features.grpc_stats_filter_enable_stats_for_all_methods_by_default`` and legacy code paths. * **header_formatters**: Removed runtime key ``envoy.reloadable_features.unified_header_formatter`` and legacy code paths. * **http**: Removed runtime key ``envoy.reloadable_features.allow_upstream_filters`` and legacy code paths. * **http**: Removed runtime key ``envoy.reloadable_features.closer_shadow_behavior`` and legacy code paths. * **http**: Removed runtime key ``envoy.reloadable_features.http_response_half_close`` and legacy code paths. * **http**: Removed runtime key ``envoy.reloadable_features.http_strip_fragment_from_path_unsafe_if_disabled`` and legacy code paths. * **logging**: Removed runtime key ``envoy.reloadable_features.correct_remote_address`` and legacy code paths. * **quic**: Removed runtime key ``envoy.reloadable_features.quic_defer_send_in_response_to_packet`` and legacy code paths. * **tls**: Remove runtime key ``envoy.reloadable_features.tls_async_cert_validation`` and legacy code paths. * **udp**: Removed runtime key ``envoy.reloadable_features.udp_proxy_connect`` and legacy code paths. * **upstream**: Removed runtime key ``envoy.reloadable_features.fix_hash_key`` and legacy code paths. New features ------------ * **access_log**: (QUIC only) Added support for %BYTES_RETRANSMITTED% and %PACKETS_RETRANSMITTED%. * **access_log**: Added :ref:`CEL ` access log formatter to print CEL expression. * **access_log**: Added :ref:`DisableBuiltinLables ` to disable envoy builtin resource labels. * **access_log**: Added ``%ACCESS_LOG_TYPE%`` substitution string, to help distinguishing between access log records and when they are being recorded. Please refer to the access log configuration documentation for more information. * **access_log**: Added access log filter :ref:`log_type_filter ` to filter access log records based on the type of the record. * **access_log**: Added additional HCM access log option :ref:`flush_log_on_tunnel_successfully_established `. Enabling this option will write a log to all access loggers when HTTP tunnels (e.g. Websocket and ``CONNECT``) are successfully established. * **access_log**: Added bytes snapshotting for upstream and downstream logging that will be reset after every periodic log. Downstream periodic loggers should read ``BytesMeter::bytesAtLastDownstreamPeriodicLog()``, and upstream periodic loggers should read ``BytesMeter::bytesAtLastUpstreamPeriodicLog()``. * **admin**: Adds a new admin stats html bucket-mode ``detailed`` to generate all recorded buckets and summary percentiles. * **application_logs**: Added bootstrap option :ref:`application_log_format ` to enable setting application log format as JSON structure. * **application_logs**: Added bootstrap option :ref:`application_log_format ` to enable setting application log text format from config. * **config**: added a statistic :ref:`warming_state ` to indicate the current warming state of a cluster. * **dynamic_forward_proxy**: Added :ref:`sub_clusters_config ` to enable independent sub cluster for each host:port, with STRICT_DNS cluster type. * **ext_proc**: :ref:`forward_rules ` to only allow headers matching the rules to be forwarded to the external processing server. * **ext_proc**: Added new configuration field :ref:`allow_mode_override ` If set to true, the filter config :ref:`processing_mode ` can be overridden by the :ref:`mode_override ` in the response message from the external processing server. If not set, the ``mode_override`` API in the response message will be ignored. * **ext_proc**: Added new configuration field :ref:`disable_clear_route_cache ` to force the ext_proc filter from clearing the route cache. Failures to clear from setting this field will be counted under the ``clear_route_cache_disabled`` stat. * **ext_proc**: Added new field ``filter_metadata `. * **health_check**: Added host related information :ref:`metadata ` and :ref:`locality ` to the :ref:`health check event ` definition. * **http**: Add support to the route/virtual host level :ref:`is_optional ` field. A route/virtual host level per filter config can be marked as optional, which means that if the filter fails to load, the configuration will no be rejected. * **http**: Added :ref:`outbound_control_frames_active ` and :ref:`outbound_frames_active ` statistic. * **http**: Added ``OVERWRITE_IF_EXISTS`` header manipulation keyword to overwrite a header only when it exists before manipulation. * **http**: Added runtime feature ``envoy.reloadable_features.max_request_headers_size_kb`` to override the default value of :ref:`max request headers size `. * **http**: Added support for CONNECT-UDP (RFC 9298). Can be disabled by setting runtime feature ``envoy.reloadable_features.enable_connect_udp_support`` to false. * **http**: Added support for configuring additional :ref:`cookie attributes `. * **http**: Added support for the route/virtual host level :ref:`disabled ` field. A route/virtual host level per filter config can be marked as disabled, which means that the filter will be disabled in a specific route/virtual host. * **lds**: Pause SRDS when LDS is updated. * **listeners**: Added :ref:`max_connections_to_accept_per_socket_event ` that sets the maximum number of new connections to be accepted per socket event on a listener. If there are more connections to be accepted beyond the maximum, the remaining connections would be processed in later dispatcher loop iterations. Added listener histogram ``connections_accepted_per_socket_event`` to allow users to empirically determine an appropriate configuration for their deployment. * **load shed point**: Added load shed point ``envoy.load_shed_points.http1_server_abort_dispatch`` that rejects HTTP1 server processing of requests. * **load shed point**: Added load shed point ``envoy.load_shed_points.http2_server_go_away_on_dispatch`` that sends ``GOAWAY`` for HTTP2 server processing of requests. When a ``GOAWAY`` frame is submitted by this the counter ``http2.goaway_sent`` will be incremented. * **load shed point**: Added load shed point ``envoy.load_shed_points.http_connection_manager_decode_headers`` that rejects new http streams by sending a local reply. * **load_balancing**: Added new option :ref:`weighted_priority_health ` to compute the health of a :ref:`priority level ` by using :ref:`load balancing weight ` instead of the count of healthy hosts. * **match_delegate**: Added :ref:`per route configuration ` to the :ref:`ExtensionWithMatcher ` filter. Which allows the associated matcher to be defined on a per route basis. * **match_delegate**: If no matcher is set the :ref:`ExtensionWithMatcher ` filter is now set to skip rather than erroring out. * **matchers**: Added :ref:`RuntimeFraction ` input matcher. It allows matching hash of the input on a runtime key. * **matching**: Added CEL(Common Expression Language) matcher support :ref:`CEL data input ` and :ref:`CEL input matcher `. * **original_dst**: Filter state is pulled from request context first (if available), then falls back to connection context. Added ability to pick host from dynamic metadata using :ref:`metadata_key `. Same behavior - looks in request context first (if available), falls back to connection context. * **ratelimit**: Added new configuration field :ref:`domain ` to allow for setting rate limit domains on a per-route basis. * **redis_proxy**: Added new configuration field :ref:`key_formatter ` to format redis key. The field supports using %KEY% as a formatter command for substituting the redis key as part of the substitution formatter expression. * **redis_proxy**: Added new field :ref:`connection_rate_limit ` to limit reconnection rate to redis server to avoid reconnection storm. * **stat_sinks**: Added ``envoy.stat_sinks.open_telemetry`` stats_sink, that supports flushing metrics by the OTLP protocol, for supported Open Telemetry collectors. * **stats**: Added config :ref:`enable_deferred_creation_stats `. When set to ``true``, enables deferred instantiation on supported stats structures. * **tls**: Added FIPS compliant build for arm64. * **tls**: Added support for hot-reloading CRL file when the file changes on disk. This works with dynamic secrets when :ref:`CertificateValidationContext ` is delivered via SDS. * **tls**: Added support to configure the new config option :ref:`enforce_rsa_key_usage `. This can be used to override its configuration in BoringSSL. It is currently default to false but expected to be changed to true by default in a future release. ``ssl.was_key_usage_invalid`` is added to :ref:`listener metrics ` and will be incremented for certificate configurations that would fail if this option were set to true. * **tls_inspector**: Added histogram ``bytes_processed`` which records the number of bytes of the tls_inspector processed while analyzing for tls usage. In cases where the connection uses tls this records the tls client hello size. In cases where the connection doesn't use tls this records the amount of bytes the tls_inspector processed until it realized the connection was not using tls. * **tls_inspector**: Added new configuration field :ref:`initial_read_buffer_size ` to allow users to tune the buffer size requested by the filter. If configured, and the filter needs additional bytes, the filter will double the number of bytes requested up to the default 64KiB maximum. * **upstream**: Added :ref:`cluster provided extension ` to suppport the :ref:`load balancer policy `. * **zookeeper**: Added the ``addWatch`` opcode support to the ZooKeeper proxy filter. Deprecated ---------- * **access_log**: Added ``%DOWNSTREAM_LOCAL_DNS_SAN%``, ``%DOWNSTREAM_PEER_DNS_SAN%``, ``%DOWNSTREAM_LOCAL_IP_SAN%`` and ``%DOWNSTREAM_PEER_IP_SAN%`` substitution formatters. * **access_log**: Deprecated (1.25.0) :ref:`intermediate_log_entry ` in favour of :ref:`access_log_type `. * **health_check**: deprecated the :ref:`HealthCheck event_log_path ` in favor of :ref:`HealthCheck event_logger extension `. * **stats**: Added :ref:`enable_deferred_creation_stats `. support for ``ClusterTrafficStats``.