gRPC Field Extraction (proto)

This extension has the qualified name envoy.filters.http.grpc_field_extraction

Note

This extension is functional but has not had substantial production burn time, use only with this caveat.

This extension has an unknown security posture and should only be used in deployments where both the downstream and upstream are trusted.

Tip

This extension extends and can be used with the following extension category:

envoy.filters.http

This extension must be configured with one of the following type URLs:

type.googleapis.com/envoy.extensions.filters.http.grpc_field_extraction.v3.GrpcFieldExtractionConfig

Overview

This filter supports extracting the fields from the first gRPC request message no matter if it is unary or streaming and writing the result to the destination, for which currently only the static Envoy dynamic metadata envoy.filters.http.grpc_field_extraction is supported.

Assumptions

This filter assumes

this filter is only applicable for gRPC with Protobuf as payload.
for bi-directional and client-side gRPC streaming, the initial message from the client should not depend on receiving the server initial metadata.

Process Flow

When a request reaches the filter, it will check

if the request is the gRPC request configured for extraction, the filter tries to:

block the incoming data before decoding the first complete gRPC message

look up the target field from the buffered gRPC message

write the extraction result into the dynamic metadata and resume the request propagation.

otherwise, pass through the request.

If the request is a malformed one found during 1.a or 1.b, the filter will reject the request.

Config Requirements

Here are config requirements

the target field should be among the following primitive types: string, uint32, uint64, int32, int64, sint32, sint64, fixed32, fixed64, sfixed32, sfixed64, float, double.
the target field could be repeated.
the intermediate type could also be repeated.

Output Format

the extracted field names/values will be wrapped in be field<StringValue> -> values<ListValue of StringValue>, which will be added in the dynamic metadata<google.protobuf.Struct>.
if the field value is empty, a empty <ListValue> will be set.

Performance

This filter should be performant as it

converts between the gRPC message from EnvoyBuffer without data copy.
parse the gRPC message binary directly without deserialization.

though buffering the first message introduces some latency.

Example

We have the following request definition for the gRPC method pkg.svc.Method.

message MethodRequest {
  string foo = 1;
  Nested nested = 2;
  uint32 baz = 3;
  ...
}

message Nested {
 repeated string bar = 1;
}

This is the filter config in JSON.

{
  "descriptor_set":{},
  "extractions_by_method":{
     "pkg.svc.Method":{
        "request_field_extractions":{
           "foo":{
           },
           "nested.bar":{
           }
           "baz":{
           }
        }
     }
  }
}

During runtime, the filter receives the following MethodRequest message in JSON.

{
  "foo": "val_foo",
  "nested": { "bar": ["val_bar1", "val_bar2"]}
}

The filter will write the following dynamic metadata(envoy.filters.http.grpc_field_extraction) in JSON.

{
  "foo":[
     "val_foo"
  ],
  "nested.bar":[
     "val_bar1", "val_bar2"
  ]
  "baz":[
  ]
}

extensions.filters.http.grpc_field_extraction.v3.GrpcFieldExtractionConfig

[extensions.filters.http.grpc_field_extraction.v3.GrpcFieldExtractionConfig proto]

{
  "descriptor_set": {...},
  "extractions_by_method": {...}
}

descriptor_set

(config.core.v3.DataSource, REQUIRED) The proto descriptor set binary for the gRPC services.

It could be passed by a local file through Datasource.filename or embedded in the Datasource.inline_bytes.

extractions_by_method

(repeated map<string, extensions.filters.http.grpc_field_extraction.v3.FieldExtractions>) Specify the extraction info. The key is the fully qualified gRPC method name. ${package}.${Service}.${Method}, like endpoints.examples.bookstore.BookStore.GetShelf

The value is the field extractions for individual gRPC method.

extensions.filters.http.grpc_field_extraction.v3.FieldExtractions

[extensions.filters.http.grpc_field_extraction.v3.FieldExtractions proto]

This message can be used to support per route config approach later even though the Istio doesn’t support that so far.

{
  "request_field_extractions": {...}
}

request_field_extractions

(repeated map<string, extensions.filters.http.grpc_field_extraction.v3.RequestFieldValueDisposition>) The field extractions for requests. The key is the field path within the grpc request. For example, we can define foo.bar.name if we want to extract Request.foo.bar.name.

message Request {
  Foo foo = 1;
}

message Foo {
  Bar bar = 1;
}

message Bar {
  string name = 1;
}

extensions.filters.http.grpc_field_extraction.v3.RequestFieldValueDisposition

[extensions.filters.http.grpc_field_extraction.v3.RequestFieldValueDisposition proto]

{
  "dynamic_metadata": ...
}

dynamic_metadata

(string) The dynamic metadata namespace. If empty, “envoy.filters.http.grpc_field_extraction” will be used by default.

Unimplemented. Uses “envoy.filters.http.grpc_field_extraction” for now.