Incompatible Behavior Changes¶
Changes that are expected to cause an incompatibility if applicable; deployment changes are likely required
grpc_stats: the default value for stats_for_all_methods is switched from true to false, in order to avoid possible memory exhaustion due to an untrusted downstream sending a large number of unique method names. The previous default value was deprecated in version 1.14.0. This only changes the behavior when the value is not set. The previous behavior can be used by setting the value to true. This behavior change by be overridden by setting runtime feature envoy.deprecated_features.grpc_stats_filter_enable_stats_for_all_methods_by_default.
http: resolving inconsistencies between :scheme and X-Forwarded-Proto. :scheme will now be set for all HTTP/1.1 requests. This changes the behavior of the gRPC access logger, Wasm filters, CSRF filter and oath2 filter for HTTP/1 traffic, where :scheme was previously not set. This change also validates that for front-line Envoys (Envoys configured with xff_num_trusted_hops set to 0 and use_remote_address set to true) that HTTP/1.1 https schemed requests can not be sent over non-TLS connections. All behavioral changes listed here can be temporarily reverted by setting envoy.reloadable_features.add_and_validate_scheme_header to false.
Minor Behavior Changes¶
Changes that may cause incompatibilities for some users, but should not for most
dns: both the strict DNS and logical DNS cluster types now honor the hostname field if not empty. Previously resolved hosts would have their hostname set to the configured DNS address for use with logging, auto_host_rewrite, etc. Setting the hostname manually allows overriding the internal hostname used for such features while still allowing the original DNS resolution name to be used.
hds: support custom health check port via health_check_config.
healthcheck: the health check filter now sends the x-envoy-immediate-health-check-fail header for all responses when Envoy is in the health check failed state. Additionally, receiving the x-envoy-immediate-health-check-fail header (either in response to normal traffic or in response to an HTTP active health check) will cause Envoy to immediately exclude the host from load balancing calculations. This has the useful property that such hosts, which are being explicitly told to disable traffic, will not be counted for panic routing calculations. See the excluded documentation for more information. This behavior can be temporarily reverted by setting the envoy.reloadable_features.health_check.immediate_failure_exclude_from_cluster feature flag to false. Note that the runtime flag covers both the health check filter responding with x-envoy-immediate-health-check-fail in all cases (versus just non-HC requests) as well as whether receiving x-envoy-immediate-health-check-fail will cause exclusion or not. Thus, depending on the Envoy deployment, the feature flag may need to be flipped on both downstream and upstream instances, depending on the reason.
http: allow to use path canonicalizer from googleurl instead of //source/common/chromium_url. The new path canonicalizer is enabled by default. To revert to the legacy path canonicalizer, enable the runtime flag envoy.reloadable_features.remove_forked_chromium_url.
http: increase the maximum allowed number of initial connection WINDOW_UPDATE frames sent by the peer from 1 to 5.
http: upstream flood and abuse checks increment the count of opened HTTP/2 streams when Envoy sends initial HEADERS frame for the new stream. Before the counter was incrementred when Envoy received response HEADERS frame with the END_HEADERS flag set from upstream server.
oauth filter: added the optional parameter auth_scopes with default value of ‘user’ if not provided. Enables this value to be overridden in the Authorization request to the OAuth provider.
perf: allow reading more bytes per operation from raw sockets to improve performance.
router: extended custom date formatting to DOWNSTREAM_PEER_CERT_V_START and DOWNSTREAM_PEER_CERT_V_END when using custom request/response header formats.
upstream: host weight changes now cause a full load balancer rebuild as opposed to happening atomically inline. This change has been made to support load balancer pre-computation of data structures based on host weight, but may have performance implications if host weight changes are very frequent. This change can be disabled by setting the envoy.reloadable_features.upstream_host_weight_change_causes_rebuild feature flag to false. If setting this flag to false is required in a deployment please open an issue against the project.
Changes expected to improve the state of the world and are unlikely to have negative effects
active http health checks: properly handles HTTP/2 GOAWAY frames from the upstream. Previously a GOAWAY frame due to a graceful listener drain could cause improper failed health checks due to streams being refused by the upstream on a connection that is going away. To revert to old GOAWAY handling behavior, set the runtime feature envoy.reloadable_features.health_check.graceful_goaway_handling to false.
buffer: tighten network connection read and write buffer high watermarks in preparation to more careful enforcement of read limits. Buffer high-watermark is now set to the exact configured value; previously it was set to value + 1.
fault injection: stop counting as active fault after delay elapsed. Previously fault injection filter continues to count the injected delay as an active fault even after it has elapsed. This produces incorrect output statistics and impacts the max number of consecutive faults allowed (e.g., for long-lived streams). This change decreases the active fault count when the delay fault is the only active and has gone finished.
filter_chain: fix filter chain matching with the server name as the case-insensitive way.
grpc-web: fix local reply and non-proto-encoded gRPC response handling for small response bodies. This fix can be temporarily reverted by setting envoy.reloadable_features.grpc_web_fix_non_proto_encoded_response_handling to false.
grpc_http_bridge: the downstream HTTP status is now correctly set for trailers-only responses from the upstream.
http: disallowing “host:” in request_headers_to_add for behavioral consistency with rejecting :authority header. This behavior can be temporarily reverted by setting envoy.reloadable_features.treat_host_like_authority to false.
http: reverting a behavioral change where upstream connect timeouts were temporarily treated differently from other connection failures. The change back to the original behavior can be temporarily reverted by setting envoy.reloadable_features.treat_upstream_connect_timeout_as_connect_failure to false.
jwt_authn: reject requests with a proper error if JWT has the wrong issuer when allow_missing is used. Before this change, the requests are accepted.
listener: prevent crashing when an unknown listener config proto is received and debug logging is enabled.
overload: fix a bug that can cause use-after-free when one scaled timer disables another one with the same duration.
sni: as the server name in sni should be case-insensitive, envoy will convert the server name as lower case first before any other process inside envoy.
tls: fix the subject alternative name of the presented certificate matches the specified matchers as the case-insensitive way when it uses DNS name.
upstream: fix handling of moving endpoints between priorities when active health checks are enabled. Previously moving to a higher numbered priority was a NOOP, and moving to a lower numbered priority caused an abort.
Removed Config or Runtime¶
Normally occurs at the end of the deprecation period
access_logs: removed legacy unbounded access logs and runtime guard envoy.reloadable_features.disallow_unbounded_access_logs.
dns: removed legacy buggy wildcard matching path and runtime guard envoy.reloadable_features.fix_wildcard_matching.
dynamic_forward_proxy: removed envoy.reloadable_features.enable_dns_cache_circuit_breakers and legacy code path.
http: removed legacy connection close behavior and runtime guard envoy.reloadable_features.fixed_connection_close.
http: removed legacy HTTP/1.1 error reporting path and runtime guard envoy.reloadable_features.early_errors_via_hcm.
http: removed legacy sanitization path for upgrade response headers and runtime guard envoy.reloadable_features.fix_upgrade_response.
http: removed legacy date header overwriting logic and runtime guard envoy.reloadable_features.preserve_upstream_date deprecation.
listener: removed legacy runtime guard envoy.reloadable_features.listener_in_place_filterchain_update.
router: removed envoy.reloadable_features.consume_all_retry_headers and legacy code path.
access log: added the formatters extension point for custom formatters (command operators).
access log: support command operator: %FILTER_CHAIN_NAME% for the downstream tcp and http request.
access log: support command operator: %REQUEST_HEADERS_BYTES%, %RESPONSE_HEADERS_BYTES%, and %RESPONSE_TRAILERS_BYTES%.
config: add envoy.features.fail_on_any_deprecated_feature runtime key, which matches the behaviour of compile-time flag ENVOY_DISABLE_DEPRECATED_FEATURES, i.e. use of deprecated fields will cause a crash.
Nodedynamic context parameters are populated in discovery requests when set on the server instance.
dispatcher: supports a stack of Envoy::ScopeTrackedObject instead of a single tracked object. This will allow Envoy to dump more debug information on crash.
ext_authz: added response_headers_to_add to support sending response headers to downstream clients on OK authorization checks via gRPC.
ext_authz: added allowed_client_headers_on_success to support sending response headers to downstream clients on OK external authorization checks via HTTP.
grpc_json_transcoder: added request_validation_options to reject invalid requests early.
grpc_json_transcoder: filter can now be configured on per-route/per-vhost level as well. Leaving empty list of services in the filter configuration disables transcoding on the specific route.
http: added support for Envoy::ScopeTrackedObject for HTTP/1 and HTTP/2 dispatching. Crashes while inside the dispatching loop should dump debug information.
http: added an option to not add content-length: 0 for requests which should not have bodies. This behavior can be enabled by setting envoy.reloadable_features.dont_add_content_length_for_bodiless_requests true.
http: added support for preconnecting. Preconnecting is off by default, but recommended for clusters serving latency-sensitive traffic, especially if using HTTP/1.1.
http: change frame flood and abuse checks to the upstream HTTP/2 codec to ON by default. It can be disabled by setting the envoy.reloadable_features.upstream_http2_flood_checks runtime key to false.
json: introduced new JSON parser (https://github.com/nlohmann/json) to replace RapidJSON. The new parser is disabled by default. To test the new RapidJSON parser, enable the runtime feature envoy.reloadable_features.remove_legacy_json.
original_dst: added support for Original Destination on Windows. This enables the use of Envoy as a sidecar proxy on Windows.
overload: add support for scaling transport connection timeouts. This can be used to reduce the TLS handshake timeout in response to overload.
postgres: added ability to terminate SSL.
route config: added allow_post field for allowing POST payload as raw TCP.
server: added fips_mode to server compilation settings related statistic.
tcp_proxy: add support for converting raw TCP streams into HTTP/1.1 CONNECT requests. See upgrade documentation for details.
tcp_proxy: added a use_post field for using HTTP POST to proxy TCP streams.
tcp_proxy: added a headers_to_add field for setting additional headers to the HTTP requests for TCP proxing.