– Mark 的滿紙方糖言

k8s headless svc

k8s headless and dns

https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
A/AAAA records

“Normal” (not headless) Services are assigned a DNS A or AAAA record, depending on the IP family of the service, for a name of the form my-svc.my-namespace.svc.cluster-domain.example. This resolves to the cluster IP of the Service.

“Headless” (without a cluster IP) Services are also assigned a DNS A or AAAA record, depending on the IP family of the service, for a name of the form my-svc.my-namespace.svc.cluster-domain.example. Unlike normal Services, this resolves to the set of IPs of the pods selected by the Service. Clients are expected to consume the set or else use standard round-robin selection from the set.

headless svc design for stateful service

https://medium.com/swlh/discovering-running-pods-by-using-dns-and-headless-services-in-kubernetes-7002a50747f4

headless svc

让人头大的『无头服务』-下
经过测试验证，Istio 1.6 及之后的版本中，Envoy 在 Upstream 链接断开后会主动断开和 Downstream 的长链接，建议尽快升级到 1.6 版本，以彻底解决本问题。也可以直接采用腾讯云上的云原生 Service Mesh 服务 TCM（Tencent Cloud Mesh），为微服务应用快速引入 Service Mesh 的流量管理和服务治理能力，而无需再关注 Service Mesh 基础设施自身的安装、维护、升级等事项。

https://istio.io/latest/docs/ops/configuration/traffic-management/traffic-routing/#headless-services
A headless Service is a Service that does not have a ClusterIP assigned. Instead, the DNS response will contain the IP addresses of each endpoint (i.e. the Pod IP) that is a part of the Service.

In general, Istio does not configure listeners for each Pod IP, as it works at the Service level. However, to support headless services, listeners are set up for each IP:Port pair in the headless service.

Without Istio, the ports field of a headless service is not strictly required because requests go directly to pod IPs, which can accept traffic on all ports. However, with Istio the port must be declared in the Service, or it will not be matched.

`PILOT_ENABLE_EDS_FOR_HEADLESS_SERVICES` pilot-discovery env

https://istio.io/latest/docs/reference/commands/pilot-discovery/
PILOT_ENABLE_EDS_FOR_HEADLESS_SERVICES pilot-discovery env:
If enabled, for headless service in Kubernetes, pilot will send endpoints over EDS, allowing the sidecar to load balance among pods in the headless service. This feature should be enabled if applications access all services explicitly via a HTTP proxy port in the sidecar.

headless service 问题汇总

负载均衡策略不生效

由于 istio 默认对 headless service 进行 passthrougth，使用 ORIGINAL_DST 转发，即直接转发到原始的目的 IP，不做任何的负载均衡，所以 Destinationrule 中配置的 trafficPolicy.loadBalancer 都不会生效，影响的功能包括:

会话保持 (consistentHash)
地域感知 (localityLbSetting)

解决方案: 单独再创建一个 service (非 headless)

envoy ORIGINAL_DST cluster

https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/cluster.proto

config.cluster.v3.Cluster

1
2
3
4
5


{
  "type": "...",
    "cleanup_interval": "{...}",
    "original_dst_lb_config": "{...}",
}

type

(config.cluster.v3.Cluster.DiscoveryType) The service discovery type to use for resolving the cluster.
  Enum config.cluster.v3.Cluster.DiscoveryType¶

  [config.cluster.v3.Cluster.DiscoveryType proto]

  Refer to [service discovery type](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/service_discovery#arch-overview-service-discovery-types) for an explanation on each type.

  STATIC

      (DEFAULT) ⁣Refer to the static discovery type for an explanation.

  STRICT_DNS

      ⁣Refer to the strict DNS discovery type for an explanation.

  LOGICAL_DNS

      ⁣Refer to the logical DNS discovery type for an explanation.

  EDS

      ⁣Refer to the service discovery type for an explanation.

  ORIGINAL_DST

      ⁣Refer to the original destination discovery type for an explanation.
        Original destination¶

        Original destination cluster can be used when incoming connections are redirected to Envoy either via an iptables REDIRECT or TPROXY target or with Proxy Protocol. In these cases requests routed to an original destination cluster are forwarded to upstream hosts as addressed by the redirection metadata, without any explicit host configuration or upstream host discovery. Connections to upstream hosts are pooled and unused hosts are flushed out when they have been idle longer than cleanup_interval, which defaults to 5000ms. If the original destination address is not available, no upstream connection is opened. Envoy can also pickup the original destination from a HTTP header. Original destination service discovery must be used with the original destination load balancer.

      > https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/original_dst
      Original destination¶

      This is a special purpose load balancer that can only be used with an original destination cluster. Upstream host is selected based on the downstream connection metadata, i.e., connections are opened to the same address as the destination address of the incoming connection was before the connection was redirected to Envoy. New destinations are added to the cluster by the load balancer on-demand, and the cluster periodically cleans out unused hosts from the cluster. No other load balancing policy can be used with original destination clusters.
      Original destination host request header¶

      Envoy can also pick up the original destination from a HTTP header called x-envoy-original-dst-host. Please note that fully resolved IP address should be passed in this header. For example if a request has to be routed to a host with IP address 10.195.16.237 at port 8888, the request header value should be set as 10.195.16.237:8888.

cleanup_interval

(Duration) The interval for removing stale hosts from a cluster type ORIGINAL_DST. Hosts are considered stale if they have not been used as upstream destinations during this interval. New hosts are added to original destination clusters on demand as new connections are redirected to Envoy, causing the number of hosts in the cluster to grow over time. Hosts that are not stale (they are actively used as destinations) are kept in the cluster, which allows connections to them remain open, saving the latency that would otherwise be spent on opening new connections. If this setting is not specified, the value defaults to 5000ms. For cluster types other than ORIGINAL_DST this setting is ignored.

headed no selector svc

https://medium.com/swlh/kubernetes-endpoints-2d77eca96a7b

Ref

https://zhaohuabing.com/post/2020-09-19-headless-mtls/
https://zhaohuabing.com/post/2019-03-29-how-to-choose-ingress-for-service-mesh/
https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/cluster.proto
https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/cluster.proto#envoy-v3-api-enum-config-cluster-v3-cluster-discoverytype
https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/service_discovery#arch-overview-service-discovery-types