PUBLIC

Exploration: Gateway API for E/W Traffic

Authors: Keith Mattix (Microsoft), John Howard (Google), The GAMMA Initiative
See version history & comments for collaborators

Overview
The Gateway API describes a way of managing N/S (ingress) L4 and L7 traffic in a Kubernetes
cluster. Members of the Kubernetes, Istio and Service Mesh Interface (SMI) communities are
coming together to standardize an approach for using the Gateway API for E/W (mesh) traffic.
This document looks at the traffic routing APIs in a variety of service mesh implementations and
proposes a number of possible designs for a standard. When agreed upon by the community,
this work will form the basis of one or more Gateway API enhancement proposal.

Motivation
The Gateway API project has seen great success as a general specification for north/south L4
and L7 traffic routing. The features present in the spec are by no means specific to ingress
traffic; in fact, most service meshes provide near identical functionality to their users (e.g. traffic
splitting, timeouts, etc). Thus, it is clear that the greater cloud-native community would
greatly benefit from Gateway API being a universal set of resources to describe all
Kubernetes traffic, north/south and east/west.

Goal
This document aims to establish a common standard, API, and spec for using Gateway API
Routes (HTTPRoute, TCPRoute, etc) for service meshes.

While the existing route API takes care of a variety of behaviors (upstream backend selection,
header manipulation and matching, etc), there are a few areas that are ambiguous for Mesh
currently:
● How to associate a route to "the mesh"
● How to match traffic (while routes have matching behavior, it is incomplete; with ingress
there is matching logic split between Gateway and Route)
● Permission model on consumer and producer services/route

Terminology
Before discussing design, it's helpful to have a shared vocabulary on the various relevant
concepts. Many of these terms are highly overloaded or have many different meanings in
different communities; the goal here is not to make any new standards or definitions, but rather
to ensure readers are on the same page with important terms.

Service Mesh
This is both core to the design and highly overloaded. Many folks have wildly different views of
what "service mesh" means, and emotional responses to the word. For the purposes of this
document, a "mesh" just means some component in a cluster that is capable of adding
functionality to users network requests. The gateway-api for Ingress traffic has been described
as something that translates from things without context (outside of the cluster) to things with
context (inside the cluster). Similarly, a service mesh could be described as something that uses
that context to modify cluster network requests.

This is intentionally broad - while most see a service mesh as "userspace sidecar proxies using
iptables to redirect traffic", this document doesn't depend on this narrow definition, and includes
architectures such as Kubernetes itself, "proxyless" architectures, and more.

Policy
Policy is a broad term for any resource that can attach to another resource to configure it. This
is defined in the Policy Attachment GEP. Examples of (hypothetical) policies could be resources
like `TimeoutPolicy` or `AuthorizationPolicy`.

Routes
Routes are a collective term for a number of types in gateway-api: `HTTPRoute`, `TLSRoute`,
`TCPRoute`, `UDPRoute`, etc. In the future there may be more defined, in core or extensions
(such as `SQLRoute`).This document focuses on routes, rather than "Policy", as these are the
primary existing core resources in gateway, and help form a foundation for policy attachment.
Future work from GAMMA will likely look into policies, such as authorization policies.

Service
"service" is a key component to meshes. In this document, "service" refers to a broader concept
than just a Kubernetes Service resource, and attempts to differentiate between these by code
font when referring to the resource. A service can be a Service, but it can also be something
like a ServiceImport (from the MCS spec), or a custom type (a made up example would be a
CloudService abstracting cloud endpoints).

While a Service in Kubernetes is a single resource, logically it handles two different roles - a
"service frontend" and "service backend".
 Resource level view of Service Decomposed view of Service

The "service frontend" refers to how we call a service. In a Service, this is an automatically
allocated DNS name (name.namespace.svc.cluster.local) and IP address
(ClusterIP). Not all services have frontends - for example, "headless" Services.

The "service backend" refers to the target of a service. In Service, this is Endpoints (or
EndpointSlice).

In the Gateway API today, Service is used as a "service backend". For example, in a route we
would refer to Service as the (aptly named) backendRef:

A user connecting to a Gateway, which has a Route forwarding to "Service Backend"

However, there are other ways that services could be used in the API, which are explored in the
design - by utilizing the "service frontend".

Below shows a hypothetical way to model splitting calls to the a service frontend to backends
composed of a-v1 and a-v2 (see real world example). In this case, service is used as both a
frontend and a backend. The route attaches to "Service a frontend", and any calls to that will be
directed to the a-v1 and a-v2 backends (as determined by the logic within the route.
 A user connecting to a service (frontend), which has a Route forwarding to two different services (backends)

Producer and Consumer

A service Producer and service Consumer describe the two roles in a mesh.

A service producer is someone that authors and operates a service. Some example actions
they may take are deploy a Service and Deployment, setup authorization policies to limit
access to their workloads, and setup routing rules to canary their workloads.

A service consumer is someone that utilizes a service by communicating with it. For example,
they may want to take actions like curl hello-world in their application - we could call this
"consuming the hello-world service".

Consumer and producer are personas, not concrete workloads or resources. One important
aspect of these personas is that they may live in different trust boundaries (typically
namespace).

Design
Fundamentally, we need to be able to give each implementation enough information to find out,
for a given request/connection, what routes should apply.

To do this, we need to solve two core problems:

Mesh Representation describes how we actually refer to our "mesh" so that we can bind
routes, or other policies, to it. This ensures an implementation knows a route should apply to the
mesh, and not something else (another mesh, an ingress Gateway, etc).

Additional matching information, in the form of a Service binding, describes how we bind a
route to a specific service. Unlike for ingress, we need additional matching logic due to the
nature of mesh routing. For example, we may want to write a rule that matches TCP requests to
the foo Service; this cannot be represented by standard ingress-oriented route resources,
which rely on acting as reverse proxies rather than transparent proxies. Note that we may not
want any Service binding in some instances (for example, "match any request with Host:
example.com").

Below, we describe some options for primitives, and what various combinations of them would
look like.

Mesh Representation
Tracked via #1291
GEP: GEP-1291: Mesh Representation

New standalone MeshClass resource

A new MeshClass resource mirroring GatewayClass would be introduced to represent a
mesh.

Example:
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: MeshClass
metadata:
name: cool-mesh
spec:
controllerName: example.com/mesh-controller
description: "my super cool mesh"
parametersRef: { ... }

MeshClass mirrors GatewayClass almost 1:1 and is a cluster scoped resource. Mesh
implementations would register a MeshClass, which would allow resources to bind to this
mesh.

Because the resource is cluster scoped, this requires that policy/route attachment handles
namespaces within those attachments.

Reuse GatewayClass resource

This is the same as the MeshClass proposal, but simply reusing GatewayClass to be used
directly to avoid additional resources. The info that it represents a "mesh" could be either implicit
or through a new field.
New Mesh and MeshClass resource
Another option is to expand on MeshClass to also add a new Mesh resource. Alternatively,
Mesh could be associated with a GatewayClass directly to avoid additional resources. mesh is
with a new resource, such as Mesh and associated MeshClass, mirroring Gateway and
GatewayClass (or Mesh with GatewayClass).

The benefit of this approach would be that the Mesh resource becomes a namespaced
instantiation of the mesh. This could take a few different forms.

The Mesh could simply represent a configuration scoping, or it could actually enroll workloads
into a mesh. For example, deploying a Mesh in the foo namespace could trigger sidecars to be
injected into the foo namespace (for sidecar based Meshes - the pattern is generic).

Reuse Gateway resource

This is similar to above with Mesh, but re-using Gateway instead.

In this simplest case, this could be represented as a Gateway with no listeners (validation would
be loosened):
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
name: mesh
spec:
gatewayClassName: my-mesh

For advanced use cases, explicit listeners could be described, allowing more advanced
configuration (this could be dropped from the proposal)
Example:
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
name: gateway
spec:
gatewayClassName: my-mesh
listeners:
- name: terminate
hostname: "echo"
port: 443
protocol: HTTPS
tls:
mode: Terminate
certificateRefs:
 - name: my-cert-http
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: echo
spec:
parentRefs:
- name: gateway
sectionName: terminate
hostname: [echo.default.svc.cluster.local]
rules:
- backendRefs:
- name: echo
port: 80

This would allow accepting traffic on port 443, which is terminating TLS, for requests to the
"echo" service and forwarding to port 80. Note: this would be for application TLS. mTLS or other
encryption used as the transport protocol for the mesh is not related.

Service Binding
Tracked via #1294
GEP doc: GEP-1294: Mesh Service Binding

Attach directly to service resources

Binding to services (which could be Service, ServiceImport, custom types, etc) would be done
through parentRefs.

In this option, we are not explicitly declaring which mesh this applies to; only that it applies to
"mesh" in general. In the majority of cases, this is expected to not be an issue. Almost all users
have a single mesh, and even if they do use multiple meshes they likely have the same
configuration between them. While not specified explicitly, any mesh controller applying this
route could set a corresponding RouteParentStatus so it would be clear at runtime if and where
it has been applied.

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: my-route
namespace: default
spec:
parentRefs:
 - kind: Service
name: httpbin
sectionName: port-http # optional, specify port
hostnames: [ … ]
rules:
- backendRefs:
- name: httpbin-v1
port: 80
- name: httpbin-v2
port: 80

This option doesn't easily allow a way to bind to the mesh but not to a Service, unless we also
allow binding directly to the Mesh Representation for these cases.

Attach to a service and mesh inline

Similar to above, we could attach to both a service resource and a mesh resource, by adding an
additional field in the spec.

Example:
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: my-route
namespace: default
spec:
parentRefs:
- kind: Mesh
name: cool-mesh
sectionName: subset-of-mesh # optional
services:
- name: my-frontend
# namespace: could be allowed, see further discussion on namespaces
kind: Service # the default, but this could be something else
port: 80 # optional; if not specified, applies to all ports
hostnames: [ … ]
rules:
- backendRefs:
- name: my-backend
port: 80

This example uses Mesh, but could use other resources decided above to represent the mesh.

One nice aspect of this is that services is optional. This would allow binding to the mesh
without binding to a service at all, allowing things like "Match all traffic on port 3306 to mysql-
svc" and "Match all traffic with Host: example.com".
Attach to a binding resource
We could bind to an indirection of Service. This would allow us to put additional information into
Service/Route without modifying those types.

Example:
kind: ServiceMeshBinding
metadata:
name: httpbin-binding
spec:
service: # what service we are configuring
name: httpbin
kind: Service
meshes: # define which Meshes this service is configured for
- kind: Mesh
name: my-cool-mesh

Then we would declare this as our parent:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: httpbin-route
spec:
parentRefs:
- kind: ServiceMeshBinding
name: httpbin-binding

This has the benefit of not requiring any changes to the API. However, it does introduce a
substantial boilerplate. Another benefit is that it would also allow us to augment the Service with
any other metadata we may need (protocols, TLS settings, etc), if desired.

This option doesn't easily allow a way to bind to the mesh but not to a Service, unless we also
allow binding directly to the Mesh Representation for these cases.

Runtime behavior (WIP, Recommended to solve only after above

are resolved)
The Mesh Representation and Service Binding sections describe the schema for the API to
configure the mesh. However, the schema leaves some ambiguity to how implementations
should actually behave. Below describes some options here. Note that with all options, we have
a choice in how prescriptive we are in enforcing/recommending certain behaviors or in allowing
implementation-specific choices.
Route selection
For a given request, there may be potentially many routes relevant to the request. For example,
consider requests from a ingress-oriented Gateway and from another client in the mesh:

Here, both requests have two routes potentially associated with requests to backend - the one
defined by the service producer (Route (producer)) and the ones defined by the service
consumers (Gateway or mesh client).

At a high level, there are two approaches to this problem: either pick one route to use, or apply
both.

Detour: mesh implementation types

In discussion here, it helps to categorize mesh implementations in terms of where the logic is
actually implemented (here depicted as "proxy").

Namespaced proxy
 A namespace scoped proxy
A common implementation is with a sidecar proxy alongside each workload. In this diagram, we
have a proxy alongside both the consumer and producer, but in mixed states we could have
either one or the other. Note: for this discussion, it doesn't matter if the proxy is 1:1 with a
workload, just that it acts on behalf of the namespace (i.e. can handle traffic to/from that
namespace and view configurations in the namespace)

1:1 proxies
The "namespaced" aspect describes scoping of the proxy and permission/ownership models.
Within this, there is another class of proxies - the most common in mesh implementations -
where the proxy is 1:1 with a workload ("sidecar"). In this topology, it is hard/impossible to
implement routing at the server-side proxy; when a proxy receives a request it is expected to
either reject or forward the request to the same workload. While it may be mutated along the
way, it is atypical for it to be forwarded to a completely different workload.

Cross namespace proxy

A cross-namespace proxy

Another possible implementation is a cross-namespace proxy, such as a single centralized

proxy or per-node proxy; the important thing is that it handles requests and configuration from
many namespaces.
Option: Pick one
This is the simplest option in terms of implementation - we simply pick one route and ignore the
other.

For Gateway cases, we almost certainly would need to pick the Gateway's route, as that is the
only thing that actually directs traffic to the backend -- the Route (producer) route only has
configuration for when requests are already going to the backend.

For mesh cases, I think the same option - using the consumer's route, if there is one, makes the
most sense here. This provides consistency with Gateway (consumer has precedence over
producer) and allows some user cases where we want per-client settings.

The major downside of this option is the producer of a service no longer has autonomy over
how requests to their service behave. This may lead to unexpected results. For example the
producer may have a rule that should send only 1% of traffic to the canary version, but
consumer overrides may accidentally ignore this rule.

Another concern is that for 1:1 proxies, all routing rules effectively need to apply to the
consumer namespace's proxy. This leads to problematic ownership models - configuration
created by one namespace impacts workloads running in another namespace. This can lead to
security concerns, and cost concerns (some configurations are computationally expensive).

Option: Pick both

Alternatively, both routes could be applied when there are multiple requests along the request
path. This most clearly expresses the users intent and delinitates RBAC of routes; routes
created by the consumer are implemented by a proxy responsible for that consumer and vise-
versa.

This introduces some constraints on implementations, however. For example, consider a

traditional sidecar proxy approach:

In some cases, we could simply apply the two routes at the two proxies. For example, if the
consumer route defined some rules to configure different paths to different backends, while the
producer route modified headers, most proxies could implement this. However, if we end up
doing "routing" in the producer rule (for example, 1% traffic to canary), most implementations
would struggle with this -- the proxy is tightly coupled to the workload, so once a request arrives
at the workload it is assumed it will always be forwarded only to the workload, not to some
external location.

If we look at a more centralized proxy implementation, we have different concerns:

Here, we may need to apply two routes to the same requests on the same connection. Routes
cannot be statically merged in all cases (see the Route Delegation GEP for more discussion),
which would limit most implementations to doing two 'cycles' of processing in some way, which
may be complex or impossible.

Traffic matching
While the Gateway API defines some concrete matching logic for routes, there are some
ambiguities introduced in mesh use cases. While the core should remain unchanged (such as
header match semantics), we do still need to figure out which route should apply for a given
request.

With standard Gateways, this is done by explicitly defining listeners that listen on some port,
and optionally some subset of hosts (HTTP Host header or SNI). With mesh, we don't have this
extra matching metadata. Additionally, mesh use cases often differ - we often want to match
"requests to the foobar Service". This is not something Gateway can handle today; while it
could have a match for hostname: foobar.namespace.svc.cluster.local, this is not
quite the same, and certainly will not work for TCPRoute.

This is where the Service Binding concept is important, by binding routes directly to Services.

Consider the following example (using one form of Service Binding, but applies equally to any
option):
 apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: my-route
namespace: default
spec:
parentRefs:
- kind: Mesh
name: cool-mesh
services:
- name: my-frontend
kind: Service
port: 80 #
hostnames: ["frontend.example.com"]
rules:
- backendRefs:
- name: my-backend
port: 80

In this case, an implementation would handle requests on the ClusterIP of my-frontend on

port 80 and apply the rules defined in this route. In this example, there is an additional
hostnames match defined. This would only match requests like curl my-frontend:80 -H
"Host: frontend.example.com". These types of requests are atypical, and it is expected
that generally mesh routes would not have hostnames set at all since the match is done based
on IP. However, for consistency with the spec, the hostnames field should be respected if set.

Implicit routing
While matching traffic for things explicitly defined in Routes is fairly straightforward, most users'
applications will make requests to things not defined in Routes. There are many options on how
these requests should be handled. For example, we could allow them or reject them. We can
treat them as opaque TCP streams or handle them as HTTP requests (based on automatic
detection, appProtocol, etc).

One challenge with this is that this behavior all happens before any API, aside from mesh
deployment/enablement, is used at all. This means that for existing mesh implementations, if
the spec chooses an option that didn't align with the original implementation, migration may be
challenging.

As a result, I recommend we currently treat this as implementation specific behavior, at least

initially.
Examples
My goal is to take my httpbin service, and deploy a new httpbin-v2 service which receives
only 1% of traffic.

Below describes the end to end user experience with a few of the options above. Note: the
combination of Service Binding and Mesh Representation are largely picked at random here.

Mesh, MeshClass, and direct attachment

First, I deploy the mesh implementation (via some implementation specific mechanism, such as
helm install cool-mesh).

This would typically instantiate a MeshClass resource describing the mesh (user doesn't need
to do this):

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: MeshClass
metadata:
name: cool-mesh
spec:
controllerName: example.com/mesh-controller
description: "my super cool mesh"

Now I need to create a Mesh resource to enable the mesh in my namespace:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Mesh
metadata:
name: cool-mesh
namespace: httpbin
spec:
meshClassName: cool-mesh

At this point, the httpbin namespace now has the service mesh enabled. While this may have
some implementation specific default behaviors (such as traffic encryption, telemetry collection,
etc), there isn't yet any Gateway API driven behavior.

Next I will deploy the v2 service:

$ kubectl apply httpbin-v2.yaml

deployment.apps/httpbin-v2 created
 service/httpbin-v2 created

Since no one is calling httpbin-v2 directly, this should not yet receive traffic yet.

Finally, we create a route to actually implement the traffic split:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: httpbin-canarying
namespace: httpbin
spec:
# Match requests to httpbin
parentRefs:
- kind: Service
name: httpbin
# ...and send 1% to the v2 version
rules:
- backendRefs:
- name: httpbin
port: 80
weight: 99
- name: httpbin-v2
port: 80
weight: 1

MeshClass and ServiceMeshBinding

First, I deploy the mesh implementation (via some implementation specific mechanism, such as
helm install cool-mesh).

This would typically instantiate a MeshClass resource describing the mesh (user doesn't need
to do this):

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: MeshClass
metadata:
name: cool-mesh
spec:
controllerName: example.com/mesh-controller
description: "my super cool mesh"

Now I would need to use some implementation specific mechanism to enable the mesh in my
namespace(s).
At this point, the httpbin namespace now has the service mesh enabled. While this may have
some implementation specific default behaviors (such as traffic encryption, telemetry collection,
etc), there isn't yet any Gateway API driven behavior.

Next I will deploy the v2 service:

/
'
$ kubectl apply httpbin-v2.yaml
deployment.apps/httpbin-v2 created
service/httpbin-v2 created

Since no one is calling httpbin-v2 directly, this should not yet receive traffic yet.

Next, we create our ServiceMeshBinding so we can reference the httpbin Service in

routes:

kind: ServiceMeshBinding
metadata:
name: httpbin-binding
spec:
service: # what service we are configuring
name: httpbin
kind: Service
meshes: # define which Meshes this service is configured for
- kind: MeshClass
name: my-cool-mesh

Finally, we create a route to actually implement the traffic split:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: httpbin-canarying
namespace: httpbin
spec:
# Match requests to httpbin
parentRefs:
- kind: ServiceMeshBinding
name: httpbin-binding
# ...and send 1% to the v2 version
rules:
- backendRefs:
- name: httpbin
 port: 80
weight: 99
- name: httpbin-v2
port: 80
weight: 1

Gateway and Service+Mesh Inline

First, I deploy the mesh implementation (via some implementation specific mechanism, such as
helm install cool-mesh).

This would typically instantiate a GatewayClass resource describing the mesh (user doesn't
need to do this):

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: GatewayClass
metadata:
name: cool-mesh
spec:
controllerName: example.com/mesh-controller
description: "my super cool mesh"

Now I need to create a Gateway resource to enable the mesh in my namespace:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
name: cool-mesh
namespace: httpbin
spec:
gatewayClassName: cool-mesh

At this point, the httpbin namespace now has the service mesh enabled. While this may have
some implementation specific default behaviors (such as traffic encryption, telemetry collection,
etc), there isn't yet any Gateway API driven behavior.

Next I will deploy the v2 service:

$ kubectl apply httpbin-v2.yaml

deployment.apps/httpbin-v2 created
service/httpbin-v2 created
Since no one is calling httpbin-v2 directly, this should not yet receive traffic yet.

Finally, we create a route to actually implement the traffic split:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: httpbin-canarying
namespace: httpbin
spec:
# For the 'cool-mesh'...
parentRefs:
- kind: Gateway
name: cool-mesh
# ...match requests to httpbin...
services:
- kind: Service
name: httpbin
# ...and send 1% to the v2 version
rules:
- backendRefs:
- name: httpbin
port: 80
weight: 99
- name: httpbin-v2
port: 80
weight: 1
Appendix
Use Cases
Below describes general use cases for Service Meshes. Not all of these will be addressed in
this design, but knowing the set of different use cases is helpful to drive decisions.

1. As a service producer…
a. I want to deploy a canary version of my application that splits traffic based on
HTTP properties.
b. I want to change the behavior (such as timeouts, retries, header manipulation) of
my application through configuration, rather than modifying my application.
c. I want to apply authorization policies, using client identities and/or HTTP
properties.
d. I want to collect HTTP metrics for my application without modifying my
application.
e. I want to run a mix of protocols (HTTP, TCP, TLS, gRPC, …) within my
application.
f. I want to be able to gradually opt-in to a mesh (no mesh, L4 only, L7 enabled) so
I can choose the right fit for my application's performance and compatibility goals.
2. As a service consumer…
a. I want to change the behavior (such as timeouts, retries, header manipulation)
when my application connects to services through configuration, rather than
modifying my application.
b. I want to collect HTTP metrics for services I connect to.
c. I want to be able to connect to Kubernetes Services and external services.
d. I want to override the destination of my traffic, for example, to send requests to
external services to an internal replica, or to send all requests to an egress proxy.
e. I want to be able to gradually opt-in to a mesh (no mesh, L4 only, L7 enabled) so
I can choose the right fit for my application's performance and compatibility goals.
3. As a mesh administrator…
a. I want to enforce that all traffic within my cluster is encrypted.

Prior Art
It's helpful to look at what existing APIs are doing to find common ground.

Note: I am not an expert on all implementations, feel free to correct any details.
Service (Kubernetes)
While not quite a traditional mesh, this serves as the basis for most APIs. Service groups
endpoints and gives a single unique VIP (ClusterIP) to access these backends with very basic
load balancing (L4 only). Additionally, a DNS entry maps to the VIP.

All matching of traffic is done based on the Service VIP.

Services are namespaced and can only select endpoints by label in the same namespace.
Advanced applications can manually fill in the endpoints for a Service; this is used by
applications such as Knative to “catch” inbound requests when scaling a serverless service from
zero replicas.

Knative is also experimenting with mapping containerPorts to Services in different namespaces

to enable the use of networkPolicy with HTTP reverse proxies. (e.g. this PR and this document)

TrafficSplit (SMI/OSM)
Example:
kind: TrafficSplit
metadata:
name: ab-test
spec:
service: website
matches:
- kind: HTTPRouteGroup
name: ab-test
backends:
- service: website-v1
weight: 0
- service: website-v2
weight: 100
---
kind: HTTPRouteGroup
metadata:
name: ab-test
matches:
- name: firefox-users
headers:
user-agent: ".*Firefox.*"

Traffic is matched first by service. This is matching the Service IP. Further refinements to the
match (such as HTTP attributes) can be done through the matches section.

The Service binding can only apply to services in the same namespace.
VirtualService (Istio)
Example:
spec:
hosts:
- reviews.prod.svc.cluster.local # could also be just "reviews"
http:
- match:
- uri:
prefix: "/frontpage"
route:
- destination:
host: frontpage.prod.svc.cluster.local

The matching logic here is a bit tricky, as hosts is overloaded. First, the host is matched
against known service's hostnames.

If it matches a known service:

● HTTP: On ports the service is exposed on, HTTP will match the Host header, using
various forms (reviews, reviews.prod, etc). IP address is never used.
● TCP: On ports the service is exposed on, match the Service IP
if it is not a known service:
● HTTP: just match the Host header as-is
● TCP: never match

Rules can apply to hostnames in other namespaces; cross-namespace rules only apply to
clients in that namespace, while same-namespace rules apply to all clients. "service" may be a
Service or any service known to the registry (typically from the ServiceEntry custom type).

Note: because hosts can reference arbitrary domains, and a single VirtualService can attach to
a Gateway and the mesh, a common pattern is to define a single VirtualService for E-W and N-
S traffic.

TrafficRoute (Kuma)
Example:
spec:
sources:
- match:
kuma.io/service: backend_default_svc_80
destinations:
- match:
kuma.io/service: redis_default_svc_6379
conf:
http:
 - match: …

I believe this is matching the IP of the destination service.

CiliumEnvoyConfig (Cilium)
Example:
spec:
services:
- name: httpbin
namespace: default
listener: envoy-lb-listener
backendServices:
- name: echo
namespace: default
resources: …

This matches the IP of the destination Service.

There is a similar cluster-scoped resource called CiliumClusterwideEnvoyConfig and applies to

all traffic going to the destination Service from any source.

ServiceRouter (Consul)
Example:
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceRouter
metadata:
name: web
spec:
routes:
- match:
http:
pathPrefix: /admin
destination:
service: admin

This matches the service web. I believe this is based on the Service IP and can only apply to
services in the same namespace.

ServiceProfile (Linkerd)
Example:
 apiVersion: linkerd.io/v1alpha2
kind: ServiceProfile
metadata:
name: webapp.my-service-namespace.svc.cluster.local
spec:
routes: …

ServiceProfile is a bit different from the rest, as it is not defining how traffic should route, but
rather sub-dividing a service into various endpoints (/admin vs /store, for example) which
allows telemetry generation and configuring policies for these (timeout, retry, etc).

Matching is done based on the name, which will match the Host header of traffic. ServiceProfile
only applies to HTTP traffic. IP address is not used.

Services (Traffic Director Service Mesh)

Matching is done for Services based on the HTTP header properties, grpc traffic properties.
Current does not support matching purely on service, but on service ip.

Common Trends
● Most implementations are matching traffic by Service IP. Exceptions:
○ Istio; but Istio is interested in moving to an IP based matching
○ ServiceProfile (Linkerd); but it is a somewhat different use case.
● Most implementations only allow configuring traffic for the same namespace.
Exceptions:
○ Istio; allows consumer overrides
○ Cilium has per-namespace and cluster-wide resources
○ Consul allows configuring traffic across namespaces as long as authorization
policy ("intentions") has been configured by the destination to allow it
● Some implementations allow only matching traffic to Service. Exceptions:
○ Istio; allows arbitrary additional services (via `ServiceEntry)
○ Traffic Director; doesn't match Service at all, instead matches arbitrary Host
headers and IP address
○ Kuma; allows non-Service types (I think, not quite sure the details)
○ Consul; allows arbitrary non-Kubernetes services registered with a Consul agent,
and services from Nomad
● No implementations support configuring traffic to Pods (beyond raw IP/Host header
support)
Old design
Below contains a copy of the initial iteration of the "Design" section of the doc. To avoid
confusion, this was rewritten but copied below to retain history and comments. See Design for
up to date discussion.

An example barebones route for reference:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: my-route
namespace: default
spec:
parentRefs:
- kind: …
name: …
hostnames: [ … ]
rules:
- backendRefs:
- name: my-backend
port: 80

On route today, we have two main "attachment" points - parentRefs and hostnames (not
present on all types).

Mesh attachment
For a given implementation, when a request is received we need to be able to find out what
routes (if any) should apply to the request.

To do this, we need at the very least to know that a route applies to our mesh (rather than some
other mesh implementation, or to some Gateway, etc). We may also need to know what part of
the mesh it applies to. For example, does the route impact all namespaces or only some? Does
the route apply to traffic going into or coming out of this part of the mesh? Beyond namespace,
some implementations may also divide the mesh further if there are multiple installations of the
same mesh in the cluster, or a single installation with some partitioning concept (example).

Note that this doesn't necessarily mean we need to explicitly specify a particular Mesh
implementation in the route; we could declare a route that applies to "mesh". Each mesh
implementation (very typically a single one) can use that route and write their own status for it
by specifying controllerName in RouteParentStatus.
Once we know the route should apply to a given mesh, the next phase is matching actual
requests. The overwhelming pattern here is to attach routes to Services, and match these
requests based on the Service IP. I recommend we continue this pattern.

Finally, we may also need some additional matching information, as Gateway provides today. I
think the only missing piece is the port number.

So in summary, we need a way to:

● Declare this applies to mesh (optional: what mesh(es) this route applies to)
● Reference what Service this route impacts
● (Optionally) what port's this route impacts

Option 1: new service field in route

A new Mesh type would be introduced, mirroring GatewayClass. This primarily acts as a way to
describe which meshes are available to the cluster and provide an attachment point.

Example:
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Mesh
metadata:
name: cool-mesh
spec:
controllerName: example.com/mesh-controller
description: "my super cool mesh"
parametersRef: { ... }

Within a Route, a Mesh can be selected as the parentRef. Vendor-specific sectionNames

can be used to identify a mesh partition, if needed.
Additionally, a new services section can be added, which declares which services this route
applies to.
The rest of the spec remains the same.
Example:
 apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: my-route
namespace: default
spec:
parentRefs:
- kind: Mesh # New type in the gateway-api spec
name: cool-mesh
sectionName: subset-of-mesh # optional
services:
- name: my-frontend
# namespace: could be allowed, see further discussion on namespaces
kind: Service # the default, but this could be something else
port: 80 # optional; if not specified applies to all ports
hostnames: [ … ]
rules:
- backendRefs:
- name: my-backend
port: 80

This reads: "For traffic in 'cool-mesh' to the 'my-frontend' service on port 80, apply this route
which sends requests to 'my-backend'"

Note: in Gateway, hostname serves a dual purpose of matching the Host header and binding
to specific Gateway Listeners. In this model, hostname is purely a Host header match.
Because we are already matching on the service IP, adding a Host header match is not typical,
but can be used for some services which handle multiple vhosts.

Possible extensions:
● Vendor specific kinds that have equivalent semantics as Service (i.e. we can identify
that a request was intended for that service.
● Cross namespace services references. These would provide "consumer" policies,
while same namespace references provide "producer" policies.
○ Based on implementation details, some meshes may be able to support both of
these simultaneously (consumer and producer route for the same service).
Others may only be able to apply one or the other.
● Empty services could be extended to mean that it is not bound to any particular
service; instead, any traffic that doesn't match a more explicit route (one with a service
specified) should attempt to match against this route (using the standard matching rules
and precedence logic).

Option 2: new MeshServiceBinding

The main problems being solved in Option 1 is that neither Route nor Service have sufficient
metadata to communicate all information we need to represent a mesh route. Option 1 takes the
approach of expanding Route to hold this information; an alternative is to split this out into a
separate resource entirely.

Example:
kind: ServiceMeshBinding
metadata:
name: httpbin
namespace: consumer
spec:
name: httpbin
kind: Service
meshes: # define which Meshes this service is configured for
- kind: Mesh
name: my-cool-mesh

This would declare this as our parent:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: httpbin-route
spec:
parentRefs:
- kind: ServiceMeshBinding
name: httpbin

This has the benefit of not requiring any changes to the API. However, it does introduce a
substantial boilerplate.

Alternative: sectionName hacks

Rather than having a split resource, we could reference the same by abusing sectionName.
This results in the minimal boilerplate of Option 1, while keeping no API changes like Option 2.
However, it is not really in the spirit of the API and just sticking fields in places that they
shouldn't be. It also limits some flexibility (Mesh subsections, etc):

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: httpbin-route
spec:
 parentRefs:
- kind: Mesh # This is for a mesh
name: httpbin # Attach to the httpbin service
namespace: my-ns # Namespace of httpbin service
sectionName: my-mesh # Attach to the "my-mesh" mesh

Option 3: direct Service reference

In all of these examples, we are attaching the Route to a Service, rather than referencing a
Route from the Service. One reason for this is that we cannot change Service. However, we can
use annotations/labels:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: httpbin # match by name
spec:
rules: { … }
---
apiVersion: v1
kind: Service
metadata:
name: httpbin
labels:
gateway.networking.k8s.io/mesh: my-mesh
spec:
ports:
- name: http
port: 80

This would attach the same-named route to the Service, and pick the mesh by annotation. This
is pretty simple, but does have a few problems:
● Any more customization (mesh section name, etc) require annotation proliferation which
is one of the anti-goals of gateway-api.
● Hard to bind multiple routes to a single Service (useful for canaries, for example, as well
as multiple ports)
● Uses annotation instead of explicit API
● Cross namespace references would require creating dummy Services that only exist to
point to another Service

Option 4: Service parentRef

Similar to Option 1, we can bind to a Service. However, by eliding the Mesh type we can use
Service as a parentRef directly.
In this option, we are not explicitly declaring which mesh this applies to; only that it applies to
"mesh" in general. In the majority of cases, this is expected to not be an issue. Almost all users
have a single mesh, and even if they do use multiple meshes they likely have the same
configuration between them. The main issue would be if users have "partitioned" meshes and
need to select the partition.

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: my-route
namespace: default
spec:
parentRefs:
- kind: Service
name: httpbin
sectionName: port-http # optional, specify port
hostnames: [ … ]
rules:
- backendRefs:
- name: httpbin-v1
port: 80
- name: httpbin-v2
port: 80

If we want to support configuring traffic to arbitrary hostnames (e.g. match google.com and
send to google-internal service) we would need something else to bind to since we don't
have a Service. Some options:

● Bind to a new "Mesh" type, which has a similar definition as GatewayClass (see
examples above). This would allow matching purely on existing route properties. It would
not help if we want to bind to things that are not Service but do provide additional routing
information (IPs, etc).
● Bind to an actual Gateway which represents the mesh. This would allow for more
advanced options like utilizing the TLS settings in Gateway (see option 5 for more
details).
● Create a new type (or possibly abuse ExternalName) and reference that instead

Option 5: Bind to Gateway

We could choose to represent the mesh with Gateways. For example:

apiVersion: gateway.networking.k8s.io/v1alpha2
 kind: Gateway
metadata:
name: mesh
spec:
gatewayClassName: my-mesh
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: echo
spec:
parentRefs:
- name: mesh
rules:
- backendRefs:
- name: echo
port: 80

Meaning:
● We have a Gateway that uses a mesh class; this means in our namespace we should
enable the mesh. This may mean deploying sidecars, configuring some proxy
somewhere, etc. Simply having the Gateway without any routes has meaning, as it
enables the mesh.
● Routes bind to the gateway explicitly. You can only bind to a gateway in your own
namespace.

The listeners field can be empty to use defaults, or allow entries for advanced configuration.
For example:
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
name: gateway
spec:
gatewayClassName: my-mesh
listeners:
- name: terminate
hostname: "echo"
port: 443
protocol: HTTPS
tls:
mode: Terminate
certificateRefs:
- name: my-cert-http
---
apiVersion: gateway.networking.k8s.io/v1alpha2
 kind: HTTPRoute
metadata:
name: echo
spec:
parentRefs:
- name: gateway
sectionName: terminate
rules:
- backendRefs:
- name: echo
port: 80

This would allow accepting traffic on port 443, which is terminated, then forwarded to port 80.

Note: this would be for application TLS. mTLS or other encryption used as the transport protocol
for the mesh is not related.

Pros:
● Most advanced configuration
● Also handles the "enable mesh for these workloads" problem
● Minimal changes to API (just allow listeners to be empty)
Cons:
● Doesn't solve how to bind a route to a specific service
● A bit complex, especially with listeners
● Can only scope mesh to a full namespace

Service binding
This option would be paired with one of the others to address the gaps.

Paired with Option 1:

We would just have a parentRef of Gateway, and fill in the new services field
 apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: my-route
namespace: default
spec:
parentRefs:
- kind: Gateway
name: cool-mesh
sectionName: subset-of-mesh # optional
services:
- name: my-frontend
kind: Service # the default, but this could be something else
port: 80 # optional; if not specified applies to all ports
hostnames: [ … ]
rules: …

Paired with Option 2:

Setup a binding of Service <-> Gateway
kind: ServiceMeshBinding
metadata:
name: httpbin
namespace: consumer
spec:
# Define the Service this is for
name: httpbin
kind: Service
meshes: # define which Gateway this service is configured for
- name: gateway

Then associate our route with this

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: httpbin-route
spec:
parentRefs:
- kind: ServiceMeshBinding
name: httpbin

Option 4:

Option 4 could be used by implicitly associating with the Gateway. For example, we simply
create a route for a Service:
apiVersion: gateway.networking.k8s.io/v1alpha2
 kind: HTTPRoute
metadata:
name: my-route
namespace: default
spec:
parentRefs:
- kind: Service
name: httpbin
sectionName: port-http # optional, specify port
hostnames: [ … ]
rules: …

For types that are handled by our mesh-type Gateways that are running in that namespace,
they will automatically bind. For example, in the above example I have a Gateway named
gateway of class my-mesh. When the my-mesh implementation sees this route, it would