Please enable Javascript to view the contents

逆向工程与云原生现场分析 Part4 —— eBPF 跟踪 Istio/Envoy 之 upstream/downstream 事件驱动协作下的 HTTP 反向代理流程

 ·  ☕ 13 分钟

logo

注，原文来自 https://blog.mygraphql.com/zh/posts/low-tec/trace/trace-istio/trace-istio-part4/ 。如你看到的转载图片不清，请回到原文。

为何要了解 upstream/downstream 事件驱动协作下的 HTTP 反向代理流程

想不到还真坚持写到 Part4 了。不管你信不信，每一 part 的“为何”一节，是我觉得最难写的 :) 。如果你是第一次读这个系列，不用担心，每一节都是比较独立的。

不出意外的话，你之前看到的 Envoy 简介是这样描述 Envoy 的特性的：

  • C++ 编写，原生底层，没有 GC stop the world，所以性能优异
  • 异步事件驱动，多路复用，完美解决 C10k problem
  • 由于单线程负责多连接，在大连接量时，减少了大量线程的内存开销，和 CPU 上下文切换开销

这些描述，当然有其合理性。但很多远看很美丽的事物，在微距放大后，可能有很多有意思，有价值的东西。我相信只要分析得足够细致，总可能针对我们实际运行环境和流量特性对作一些有意义的优化。可能只是修改一个 Envoy / Kernel 的配置，也可能是修改一行 Envoy 的代码。或你的应用的流程相关行为，如每次写 socket 的 buffer 数据包的大小。

而这些，都需要建立在了解实现细节的基础上。除非觉得运气或经验值特别好，可以瞎猜。

逆向工程与云原生现场分析 系列介绍

开始前先做个预告，逆向工程与云原生现场分析 系列（将）包括：

在系列中，我将演示如何让 bpftrace “读” 懂运行期的由 C++ 11 编写成的 envoy 进程中的对象数据。
为免吓跑人，还是老套路，多图少代码，不过有的图有点点复杂。程序员大叔开始讲故事了。🚜

HTTP 反向代理的总流程

整体看，Socket 事件驱动的 HTTP 反向代理总流程如下：
图：Socket 事件驱动的 HTTP 反向代理总流程

图中看出，有4种事件驱动了整个流程。后面几节会逐个分析。

为免一下子进入各个步骤细节而让人迷途，现在先看看所有步骤的总流程：

图：Istio/Envoy 模块协作总图
图：Istio/Envoy 模块协作总图

Downstream Read Request 模块协作

图：Downstream Read-Ready 模块协作
图：Downstream Read-Ready 模块协作

大概说明一下流程：

  1. downstream socket 可读回调
  2. Http::ConnectionManagerImpl 读取 socket，增量放入 Http1::ConnectionImpl
  3. Http1::ConnectionImpl 调用 nghttp2 增量解释 HTTP 请求
  4. 如果 nghttp2 认为已经 完整读取了 HTTP Request 请求，则调用 Http::ServerConnection::onMessageCompleteBase()
  5. Http::ServerConnection::onMessageCompleteBase() 首先 停止 downstream ReadReady 监听
  6. Http::ServerConnection::onMessageCompleteBase() 调用 Http::FilterManager ，发起 http filter chain 的 decodeHeaders 迭代流程
  7. 一般，http filter chain 的最后一个 http filter 是 Router::Filter ，Router::Filter::decodeHeaders() 被调用
  8. Router::Filter::decodeHeaders() 的逻辑就见下图了。

图：Downstream Request Router 模块协作
图：Downstream Request Router 模块协作

大概说明一下流程：

  1. Router::Filter ，Router::Filter::decodeHeaders() 被调用
  2. 根据配置的 Router 规则，匹配到 Cluster
  3. 如 Cluster 连接池对象不存在，则新建
  4. 新建 Envoy::Router::UpstreamRequest 对象。
  5. 调用 Envoy::Router::UpstreamRequest::encodeHeaders(bool end_stream) ， encode HTTP header
  6. 经过一系列的负载均衡算法，匹配到 upstream 的 host(endpoint)
  7. 发现到选定的 upstream host 的连接不足，则：
    1. 打开一新的 socket fd(未连接)
    2. 注册 upstream socket FD 的 WriteReady / Connected 事件。 准备在事件回调时写 upstream request
    3. 用 socket fd 发起到 upstream host 的异步连接请求
  8. 关联 downstream 与 upstream fd

Upstream Write Request 模块协作

图：upstream connect & write 模块协作
大概说明一下流程：

  1. upstream socket write ready 回调
  2. 发现是连接成功回调，关联 upstream socket 到 ConnectionPool::ActiveClient
  3. upstream socket write ready 回调
  4. 发现是连接可写，写入 upstream HTTP request

Upstream Read Response 模块协作

图：Upstream Read-Response 模块协作

Downstream Write Response 模块协作

图：Downstream Write Response 模块协作

BPF 脚本输出

以上图片，不是只读源码就得出的。是需要通过 bpf 埋点观察：

  1. 记录 downstream FD(socket 的 file descriptor)，可以认为是进程中的 socket id
  2. 埋点进程级和 kernel 级的相关函数入口和出口，观察其出入参与 stack
  3. 关联起 downstream FD 与 upstream FD

其中当然有很多细节，不过我不打算一个个讲了，实在有需要用，想深入了解的同学可以联系我探讨。下一节的 bpf 代码是最好的细节。

https://github.com/labilezhu/pub-diy/blob/main/low-tec/trace/trace-istio/trace-istio-part4/trace-envoy-filter-router.3.log

#### Downstream Read Request: accept downstream socket， 可见 downstream socket 的 FD=41

OS handshaked TCP:
01:22:11 4215   wrk:worker_0   172.30.207.129                          58708 172.21.206.232                          15006 0/4096
sys_exit_accept4 fd=41

        accept4+96
        Envoy::Network::IoSocketHandleImpl::accept(sockaddr*, unsigned int*)+82
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+216
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217
 

***** elapsed=1621361178: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZNSt3__110__function6__funcIZN5Envoy10Extensions15ListenerFilters12TlsInspector25TlsInspectorConfigFactory36createListenerFilterFactoryFromProtoERKN6google8protobuf7MessageERKNS_10shared_ptrINS2_7Network21ListenerFilterMatcherEEERNS2_6Server13Configuration22ListenerFactoryContextEEUlRNSD_21ListenerFilterManagerEE_NS_9allocatorISO_EEFvSN_EEclESN_

***** elapsed=1621785490: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN5Envoy10Extensions15ListenerFilters12TlsInspector25TlsInspectorConfigFactory36createListenerFilterFactoryFromProtoERKN6google8protobuf7MessageERKNS_10shared_ptrINS3_7Network21ListenerFilterMatcherEEERNS3_6Server13Configuration22ListenerFactoryContextEEUlRNSE_21ListenerFilterManagerEE_SO_EEEvDpOT_

***** elapsed=1630898092: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterC1ENSt3__110shared_ptrINS2_6ConfigEEE

***** elapsed=1630908037: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterC2ENSt3__110shared_ptrINS2_6ConfigEEE

***** elapsed=1631036930: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter8onAcceptERNS_7Network23ListenerFilterCallbacksE

***** elapsed=1631045047: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter6onReadEv

***** elapsed=1631070582: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_recvfrom, fd=41, ret=89

        recv+108


***** elapsed=1631085065: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter16parseClientHelloEPKvm

#### Downstream Read Request: 选择 network filter chain
***** elapsed=1631145371: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterD0Ev
comm:wrk:worker_0,tid:5327: Got setFilterChainName=0.0.0.0_8080, lenght=12

        Envoy::StreamInfo::StreamInfoImpl::setFilterChainName(absl::string_view)+0
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107

#### Downstream Read Request: 监听 upstream FD 事件
***** elapsed=1631199672: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80000005
EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger

        epoll_ctl+14
        epoll_nochangelist_add+54
        evmap_io_add_+421
        event_add_nolock_+603
        event_add+54
        Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+362
        Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+284
        Envoy::Network::IoSocketHandleImpl::initializeFileEvent(Envoy::Event::Dispatcher&, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+126
        Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+1026
        Envoy::Network::ServerConnectionImpl::ServerConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+107
        Envoy::Event::DispatcherImpl::createServerConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&)+70
        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+307
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0    : setsockopt: level=6, fd=41, optname=1, optval=1, optlen=4. 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl9addFilterENSt3__110shared_ptrINS0_6FilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41 

        Envoy::Network::FilterManagerImpl::addFilter(std::__1::shared_ptr<Envoy::Network::Filter>)+0
        std::__1::__function::__func<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0, std::__1::allocator<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+93
        Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50
        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41 

        Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0
        virtual thunk to Envoy::Network::ConnectionImpl::addFilter(std::__1::shared_ptr<Envoy::Network::Filter>)+62
        std::__1::__function::__func<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0, std::__1::allocator<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+93
        Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50
        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41 

        Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0
        std::__1::__function::__func<Envoy::Extensions::NetworkFilters::HttpConnectionManager::HttpConnectionManagerFilterConfigFactory::createFilterFactoryFromProtoTyped(envoy::extensions::filters::network::http_connection_manager::v3::HttpConnectionManager const&, Envoy::Server::Configuration::FactoryContext&)::$_7, std::__1::allocator<Envoy::Extensions::NetworkFilters::HttpConnectionManager::HttpConnectionManagerFilterConfigFactory::createFilterFactoryFromProtoTyped(envoy::extensions::filters::network::http_connection_manager::v3::HttpConnectionManager const&, Envoy::Server::Configuration::FactoryContext&)::$_7>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+353
        Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50
        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 

        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+0
        Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
        Envoy::Server::ActiveTcpSocket::newConnection()+377
        Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
        Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
        Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 

***** elapsed=1631443267: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1631456738: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=31393726, tid2epollNrFdReady=1
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=100 
***************************

#### Downstream Read Request: 读 HTTP Request
***** elapsed=1631475016: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98e26ee0, fd=41, events=0x26
libevent: EV_READ
libevent: EV_WRITE
libevent: EV_ET


***** elapsed=1631513821: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=89

        readv+77
        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
        Envoy::Network::ConnectionImpl::onReadReady()+753
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


***** elapsed=1631527775: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=-11

        readv+77
        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
        Envoy::Network::ConnectionImpl::onReadReady()+753
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl6onReadEv,FilterManagerImpl.this=0x559f98d38758,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 

        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+0
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98c07200,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98c07200,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
***** elapsed=1631671079: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80000005

#### Downstream Read Request: downstream 的 HTTP Reuest 已经完全读取（onMessageComplete()）。停止监听 downstream 的 ReadReady 事件
***** elapsed=1631678652: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80002004
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger

        epoll_ctl+14
        epoll_nochangelist_add+54
        evmap_io_add_+421
        event_add_nolock_+603
        event_add+54
        Envoy::Network::ConnectionImpl::readDisable(bool)+1077
        Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+86
        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
        http_parser_execute+7959
        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
        virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
        Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
        Envoy::Network::ConnectionImpl::onReadReady()+1622
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 

#### Downstream Read Request: 监听新建立的 upstream socket 的 FD=44 ， 这里 socket 还处于未连接状态。现在监听 upstream socket 的 所有事件。以备后面异步连接成功时收到回调
***** elapsed=1632388633: tid=5327,comm=wrk:worker_0: register upstream event trigger:sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=44, events=0x80000005
EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger

        epoll_ctl+14
        epoll_nochangelist_add+54
        evmap_io_add_+421
        event_add_nolock_+603
        event_add+54
        Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+362
        Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+284
        Envoy::Network::IoSocketHandleImpl::initializeFileEvent(Envoy::Event::Dispatcher&, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+126
        Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+1026
        Envoy::Network::ClientConnectionImpl::ClientConnectionImpl(Envoy::Event::Dispatcher&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&)+283
        Envoy::Event::DispatcherImpl::createClientConnection(std::__1::shared_ptr<Envoy::Network::Address::Instance const>, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&)+75
        Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterInfo const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, Envoy::Network::TransportSocketFactory&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>)+561
        Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>) const+126
        non-virtual thunk to Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>) const+21
        Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+112
        Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
        std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
        Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
        Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
        Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
        non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
        Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
        Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
        Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
        Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139
        Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
        http_parser_execute+7959
        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
        virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
        Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
        Envoy::Network::ConnectionImpl::onReadReady()+1622
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98f08478,fd=41 

        Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0
        Envoy::Http::CodecClient::CodecClient(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&)+357
        Envoy::Http::CodecClientProd::CodecClientProd(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&)+70
        std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1>, std::__1::unique_ptr<Envoy::Http::CodecClient, std::__1::default_delete<Envoy::Http::CodecClient> > (Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*&&)+94
        Envoy::Http::FixedHttpConnPoolImpl::createCodecClient(Envoy::Upstream::Host::CreateConnectionData&)+35
        Envoy::Http::ActiveClient::initialize(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase&)+113
        Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+171
        Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
        std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
        Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
        Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
        Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
        non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
        Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
        Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
        Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
        Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139
        Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
        http_parser_execute+7959
        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
        virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
        Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
        Envoy::Network::ConnectionImpl::onReadReady()+1622
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0    : setsockopt: level=6, fd=44, optname=1, optval=1, optlen=4. 

#### Downstream Read Request: 发起异步 upstream 连接
connecting to 8080:
01:23:36 4215     wrk:worker_0     127.0.0.6                               36383  172.21.206.232                          8080  

        connect+75
        Envoy::Network::IoSocketHandleImpl::connect(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+104
        Envoy::Network::SocketImpl::connect(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+64
        Envoy::Network::ClientConnectionImpl::connect()+650
        Envoy::Http::CodecClient::connect()+543
        Envoy::Http::CodecClientProd::CodecClientProd(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&)+694
        std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1>, std::__1::unique_ptr<Envoy::Http::CodecClient, std::__1::default_delete<Envoy::Http::CodecClient> > (Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*&&)+94
        Envoy::Http::FixedHttpConnPoolImpl::createCodecClient(Envoy::Upstream::Host::CreateConnectionData&)+35
        Envoy::Http::ActiveClient::initialize(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase&)+113
        Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+171
        Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
        std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
        Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
        Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
        Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
        non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
        Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
        Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
        Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
        Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139
        Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
        http_parser_execute+7959
        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
        virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
        Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
        Envoy::Network::ConnectionImpl::onReadReady()+1622
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41 

***** elapsed=1632608064: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1632618881: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=1145219, tid2epollNrFdReady=1
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=92 
***************************


***** elapsed=1632635526: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98e26ee0, fd=41, events=0x24
libevent: EV_WRITE
libevent: EV_ET


***** elapsed=1632654517: tid=5327,comm=wrk:worker_0: END:EventFired

#### Upstream Write Request: 异步 upstream 连接成功，回调。
***** elapsed=1632659321: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98eb6540, fd=44, events=0x24
libevent: EV_WRITE
libevent: EV_ET

comm:wrk:worker_0,tid:5327: ConnPoolImplBase*attachStreamToClient: uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy14ConnectionPool16ConnPoolImplBase20attachStreamToClientERNS0_12ActiveClientERNS0_13AttachContextE 

        Envoy::ConnectionPool::ConnPoolImplBase::attachStreamToClient(Envoy::ConnectionPool::ActiveClient&, Envoy::ConnectionPool::AttachContext&)+0
        Envoy::ConnectionPool::ConnPoolImplBase::onConnectionEvent(Envoy::ConnectionPool::ActiveClient&, absl::string_view, Envoy::Network::ConnectionEvent)+497
        Envoy::Network::ConnectionImplBase::raiseConnectionEvent(Envoy::Network::ConnectionEvent)+59
        Envoy::Network::ConnectionImpl::raiseEvent(Envoy::Network::ConnectionEvent)+511
        Envoy::Network::ConnectionImpl::onWriteReady()+1648
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

#### Upstream Write Request: 向 upstream socket 写 HTTP Request
***** elapsed=1632741863: tid=5327,comm=wrk:worker_0: socket_write, probe=tracepoint:syscalls:sys_exit_writev, fd=44, ret=262

        writev+77
        Envoy::Network::IoSocketHandleImpl::writev(Envoy::Buffer::RawSlice const*, unsigned long)+263
        Envoy::Network::IoSocketHandleImpl::write(Envoy::Buffer::Instance&)+107
        Envoy::Network::RawBufferSocket::doWrite(Envoy::Buffer::Instance&, bool)+121
        Envoy::Network::ConnectionImpl::onWriteReady()+1876
        Envoy::Network::ConnectionImpl::raiseEvent(Envoy::Network::ConnectionEvent)+529
        Envoy::Network::ConnectionImpl::onWriteReady()+1648
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


***** elapsed=1632756069: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1632764448: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=129648, tid2epollNrFdReady=2
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=60 
***************************

#### Upstream Read Response: upstream socket ReadReady 回调
***** elapsed=1637821708: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98eb6540, fd=44, events=0x26
libevent: EV_READ
libevent: EV_WRITE
libevent: EV_ET

#### Upstream Read Response: 读 upstream HTTP Response
***** elapsed=1637898855: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=44, ret=4511

        readv+77
        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
        Envoy::Network::ConnectionImpl::onReadReady()+753
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


***** elapsed=1637915998: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=44, ret=-11

        readv+77
        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
        Envoy::Network::ConnectionImpl::onReadReady()+753
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

***** elapsed=1638220184: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80002004

#### Upstream Read Response: 已经完整读取 upstream HTTP Response，开始监听 downstream 的 WriteReady 事件，准备写 downstream HTTP Response
***** elapsed=1638231319: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80000005
EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger

        epoll_ctl+14
        epoll_nochangelist_add+54
        evmap_io_add_+421
        event_add_nolock_+603
        event_add+54
        Envoy::Network::ConnectionImpl::readDisable(bool)+938
        Envoy::Http::Http1::StreamEncoderImpl::~StreamEncoderImpl()+112
        non-virtual thunk to Envoy::Http::Http1::ServerConnectionImpl::onEncodeComplete()+54
        Envoy::Http::Http1::StreamEncoderImpl::endEncode()+166
        Envoy::Http::Http1::StreamEncoderImpl::encodeData(Envoy::Buffer::Instance&, bool)+340
        Envoy::Http::ConnectionManagerImpl::ActiveStream::encodeData(Envoy::Buffer::Instance&, bool)+679
        Envoy::Http::FilterManager::encodeData(Envoy::Http::ActiveStreamEncoderFilter*, Envoy::Buffer::Instance&, bool, Envoy::Http::FilterManager::FilterIterationStartState)+2138
        Envoy::Router::UpstreamRequest::decodeData(Envoy::Buffer::Instance&, bool)+230
        Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool)+59
        Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool)+59
        Envoy::Http::Http1::ClientConnectionImpl::onMessageCompleteBase()+619
        Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
        Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
        http_parser_execute+7705
        Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
        Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
        Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
        Envoy::Http::Http1::ClientConnectionImpl::dispatch(Envoy::Buffer::Instance&)+29
        Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&)+48
        Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&, bool)+21
        Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
        Envoy::Network::ConnectionImpl::onReadReady()+1622
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92


***** elapsed=1638898463: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1638934709: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=1123611, tid2epollNrFdReady=1
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=56 
***************************


***** elapsed=1638955421: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98eb6540, fd=44, events=0x24
libevent: EV_WRITE
libevent: EV_ET


***** elapsed=1638975366: tid=5327,comm=wrk:worker_0: END:EventFired

#### Downstream Write Response: downstream 的 WriteReady 事件回调。
***** elapsed=1638980295: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98e26ee0, fd=41, events=0x24
libevent: EV_WRITE
libevent: EV_ET

#### Downstream Write Response: 写 downstream HTTP Response
***** elapsed=1639253752: tid=5327,comm=wrk:worker_0: socket_write, probe=tracepoint:syscalls:sys_exit_writev, fd=41, ret=4631

        writev+77
        Envoy::Network::IoSocketHandleImpl::writev(Envoy::Buffer::RawSlice const*, unsigned long)+263
        Envoy::Network::IoSocketHandleImpl::write(Envoy::Buffer::Instance&)+107
        Envoy::Network::RawBufferSocket::doWrite(Envoy::Buffer::Instance&, bool)+121
        Envoy::Network::ConnectionImpl::onWriteReady()+1876
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217


***** elapsed=1639293010: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1639308346: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=354486, tid2epollNrFdReady=2
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=56 
***************************


***** elapsed=1643785802: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98e26ee0, fd=41, events=0x26
libevent: EV_READ
libevent: EV_WRITE
libevent: EV_ET


***** elapsed=1643855427: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=0

        readv+77
        Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
        Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
        Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
        Envoy::Network::ConnectionImpl::onReadReady()+753
        Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
        std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
        Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
        0x7fffffffe000
        event_base_loop+1953
        Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
        Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
        start_thread+217

***** elapsed=1643880789: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80000005
sys_enter_close fd=41
delete fd2filterManagerImpl, fd=41
delete fd2filterManagerImpl=0x559f98d38758
delete fd2filterManagerImpl, fd=41
delete fd2filterManagerImpl=0x559f98996380

***** elapsed=1644006858: tid=5327,comm=wrk:worker_0: END:EventFired

******* WAKE-ROUND:END Summary *******
***** elapsed=1644056452: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=289169, tid2epollNrFdReady=1
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=52 
***************************

^C


@epoll_ctl_op[1]: EPOLL_CTL_ADD
@epoll_ctl_op[2]: EPOLL_CTL_DEL
@epoll_ctl_op[3]: EPOLL_CTL_MOD


@fd2sockopt[44, 6, 1, 1]: 1
@fd2sockopt[41, 6, 1, 1]: 1


@filterManagerImpl2fd[94143954060408]: 41
@filterManagerImpl2fd[94143950909952]: 41

BPF 脚本

https://github.com/labilezhu/pub-diy/blob/main/low-tec/trace/trace-istio/trace-istio-part4/trace-envoy-filter-router.bt

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654

#!usr/bin/bpftrace
/*
IMPORT-ENV: $ENVOY_PID
args: $1=ENVOY_PID
*/

/*

#export PODID=`sudo crictl  pods | grep fortio-server | egrep '\bReady' | awk '{print $1}'`
#export CONTAINERID=`sudo crictl ps | grep $PODID | grep istio-proxy |  awk '{print $1}'`

kubectl -n idm-mark exec -it fortio-server -c  istio-proxy  -- bash -c 'ls -l /proc/`pgrep envoy`/ns/pid'
lrwxrwxrwx 1 istio-proxy istio-proxy 0 Mar 31 09:18 /proc/15/ns/pid -> 'pid:[4026532799]'
pgrep envoy | tee /dev/fd/2 | xargs -L 1 -I '{}' sudo ls -l /proc/'{}'/ns 

export PID=4215
export SCRIPT_HOME=`pwd`
export bpftrace_image=cndt-bcc-ub

docker run -it --rm --init  --privileged --name bpftrace -h bpftrace \
    --pid host \
    --net host \
    -e SCRIPT_HOME=$SCRIPT_HOME \
    -e PID=$PID \
    -e ENVOY_PID=$PID \
    -e BT=trace-envoy-filter-router.bt \
    -v /etc/localtime:/etc/localtime:ro \
    -v /sys:/sys:rw \
    -v /usr/src:/usr/src:rw \
    -v /lib/modules:/lib/modules:ro \
    -v ${SCRIPT_HOME}:${SCRIPT_HOME}:rw \
    $bpftrace_image \
    ${SCRIPT_HOME}/warp-bt.sh $PID
*/


/*
1. BPF Map 说明:

1.1. range tid map
- @watchedWakeRound[tid]=tid
       - sys_exit_accept4
       - FileEventImpl*assignEvents*
- @fdFired[tid]=$fd;
       - uprobe: *FileEventImpl*assignEvents*
       - uretprobe: *FileEventImpl*assignEvents*

1.2. fd map
- @fd2sockpair[$fd]=@sockpair[tid]
       - sys_exit_accept4
       - sys_enter_close
- @fd2sockopt[$fd, $level, $optname, $optval_int] = 1
       - sys_enter_setsockopt

- @filterManagerImpl2fd[arg0] = @fdFired[tid] // filterManagerImpl to downstream fd
- @fd2filterManagerImpl[$fd] = arg0 
- @fd2filterManagerImpl2[$fd] = arg0
- @downFD2upFD[$downFD] = $fd; //downstream fd to upstream fd
       - sys_enter_epoll_ctl
- @upFD[$fd] = $fd; //upstream fd list
       - sys_enter_epoll_ctl

1.3. temp tid map
- @sockpair[tid]=($sk->__sk_common.skc_daddr, $dport, $sk->__sk_common.skc_rcv_saddr, $lport);
- @sockpair_exist[tid]=1
- @watchedIo[tid]=fd

*/

#include <linux/in.h>
#include <linux/in6.h>
#include <linux/socket.h>
#include <net/sock.h>
// #include <sys/epoll.h>

// The event argument describes the object linked to the file descriptor fd.  The struct epoll_event is defined as:
// struct epoll_data_t {
//     int          fd;
// };

// struct epoll_event {
//     //a bit mask composed by ORing together zero or more of the following available event types
//     uint32_t     events;      /* Epoll events */

//     struct epoll_data_t data;        /* User data variable */
// };



BEGIN
{
	printf("Tracing Envoy. Hit Ctrl-C to end.\n");
       printf("#define EPOLL_CTL_ADD 1\n #define EPOLL_CTL_DEL 2\n #define EPOLL_CTL_MOD 3\n");

       @epoll_ctl_op[1]="EPOLL_CTL_ADD";
       @epoll_ctl_op[2]="EPOLL_CTL_DEL";
       @epoll_ctl_op[3]="EPOLL_CTL_MOD";


}

/*
获取 accept 连接时的本地和对端地址。并标记本轮 wakeup 需要监控
 */
kretprobe:inet_csk_accept
/pid==$1 /
{
	$sk = (struct sock *)retval;
	$inet_family = $sk->__sk_common.skc_family;

	if ($inet_family == AF_INET || $inet_family == AF_INET6) {
		// initialize variable type:
		$daddr = ntop(0);
		$saddr = ntop(0);
		if ($inet_family == AF_INET) {
			$daddr = ntop($sk->__sk_common.skc_daddr);
			$saddr = ntop($sk->__sk_common.skc_rcv_saddr);
		} else {
			printf("not support IPv6.\n");
			return;
		}
		
		$lport = $sk->__sk_common.skc_num;

              // printf("accept(), port=%d\n", $lport);

              //only watch listen port 15006
		if( 15006 != $lport ) { 
			return;
		}

		$dport = $sk->__sk_common.skc_dport;
		$qlen  = $sk->sk_ack_backlog;
		$qmax  = $sk->sk_max_ack_backlog;

		// Destination port is big endian, it must be flipped
		$dport = ($dport >> 8) | (($dport << 8) & 0x00FF00);

              printf("OS handshaked TCP:\n");
		time("%H:%M:%S ");
		printf("%-6d %-14s ", pid, comm);
		printf("%-39s %-5d %-39s %-5d ", $daddr, $dport, $saddr,
		    $lport);
		printf("%d/%d\n", $qlen, $qmax);

		@sockpair[tid]=($sk->__sk_common.skc_daddr, $dport, $sk->__sk_common.skc_rcv_saddr, $lport);
		@sockpair_exist[tid]=1;

              @watchedWakeRound[tid]=tid;
	}
}

/**
 * 获取新连接的 FD 。并标记本轮 libevent 回调需要监控
 */
tracepoint:syscalls:sys_exit_accept4
/pid==$1 && @sockpair_exist[tid] /
{
	$fd = args->ret;
	if( $fd < 0 ) {
		return;
	}
	printf("sys_exit_accept4 fd=%d\n", $fd);
	@fd2sockpair[$fd]=@sockpair[tid];

       @fdFired[tid]=$fd;

	delete(@sockpair[tid]);
	delete(@sockpair_exist[tid]);
       printf("%s \n", ustack());
}

/**
 * 记录 FD 的 sockopt
 */
tracepoint:syscalls:sys_enter_setsockopt
/pid==$1/
{
       // socket opts: https://elixir.bootlin.com/linux/v5.16.3/source/include/uapi/linux/tcp.h#L92     
       $level = args->level;
       $fd = args->fd;

       if( @fd2sockpair[$fd].0 || @upFD[$fd] ) {
              $optname = args->optname;
              $optval = args->optval;
              $optval_int = *$optval;
              $optlen = args->optlen;
              // printf("\n########## setsockopt() ##########\n");
              printf("comm:%-16s: setsockopt: level=%d, fd=%d, optname=%d, optval=%d, optlen=%d. \n", comm, $level, $fd, $optname, $optval_int, $optlen);
              @fd2sockopt[$fd, $level, $optname, $optval_int] = 1;
       }
}



/*
记录 epoll 监听的 FD

cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_epoll_ctl/format
 */
tracepoint:syscalls:sys_enter_epoll_ctl
/pid==$1/
{
       $fd=args->fd;
       $epollEvents = (uint32) (args->event->events);
       $op = args->op;
       if( @fd2sockpair[$fd].0 ) {

              printf("***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_ctl, epfd=%d, op=%s, fd=%d, events=0x%x\n", 
                     elapsed, tid, comm, args->epfd, @epoll_ctl_op[$op], $fd, $epollEvents);


              if( $op == 1/*EPOLL_CTL_ADD*/ || $op == 3/*EPOLL_CTL_MOD*/ ) { //add watch epoll event
                     if( $epollEvents & (uint32)0x001 /*EPOLLIN*/ ) {
                            printf("EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)\n");
                     }
                     if( $epollEvents & (uint32)0x004 /*EPOLLOUT*/ ) {
                            printf("EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)\n");
                     }
                     if( $epollEvents & (1u << 31) /*EPOLLET*/ ) {
                            printf("EPOLL_CTL_ADD/MOD EdgeTrigger\n");
                     }
                     printf("%s\n", ustack(30));
              }
              
       } else {
              $downFD = @fdFired[tid];
              if( $downFD ) {//in downstream fd event callback, register upstream event trigger
                     @upFD[$fd] = $fd;
                     @downFD2upFD[$downFD] = $fd;

                     printf("***** elapsed=%d: tid=%d,comm=%s: register upstream event trigger:sys_enter_epoll_ctl, epfd=%d, op=%s, fd=%d, events=0x%x\n", 
                            elapsed, tid, comm, args->epfd, @epoll_ctl_op[$op], $fd, $epollEvents);


                     if( $op == 1/*EPOLL_CTL_ADD*/ || $op == 3/*EPOLL_CTL_MOD*/ ) { //add watch epoll event
                            if( $epollEvents & (uint32)0x001 /*EPOLLIN*/ ) {
                                   printf("EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)\n");
                            }
                            if( $epollEvents & (uint32)0x004 /*EPOLLOUT*/ ) {
                                   printf("EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)\n");
                            }
                            if( $epollEvents & (1u << 31) /*EPOLLET*/ ) {
                                   printf("EPOLL_CTL_ADD/MOD EdgeTrigger\n");
                            }
                            printf("%s\n", ustack(60));
                     }                     
              }
       }
}

/*
结束本轮的 wakeRound/runnableRound，并等待下一轮
cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_epoll_wait/format
 */
tracepoint:syscalls:sys_enter_epoll_wait
/pid==$1/
{

       if( @watchedWakeRound[tid] ) {
              $runnableStartTime=@tid2Waketime[tid];
              if( $runnableStartTime ) {
                     $runableDuaration = elapsed - $runnableStartTime;
                     printf("\n******* WAKE-ROUND:END Summary *******\n");
                     printf("***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_wait, runableDuaration=%d, tid2epollNrFdReady=%d\n", 
                            elapsed, tid, comm, $runableDuaration, @tid2epollNrFdReady[tid]);
                     $tid_last_epoll_wait_args = @last_epoll_wait_args[tid];
                     if( $tid_last_epoll_wait_args.0  ) {
                            printf("*** last_epoll_wait_args: epfd=%d, events=%d, maxevents=%d, timeout=%d \n", 
                                   $tid_last_epoll_wait_args.0, $tid_last_epoll_wait_args.1, $tid_last_epoll_wait_args.2, $tid_last_epoll_wait_args.3);
                     }
                     printf("***************************\n\n");
              }
       }

       delete(@tid2Waketime[tid]);
       delete(@watchedWakeRound[tid]);
       delete(@tid2epollNrFdReady[tid]);

       //read in next sys_enter_epoll_wait
       @last_epoll_wait_args[tid]=(args->epfd, args->events, args->maxevents, args->timeout);
}

/*
开始本轮 wakeRound/runnableRound
cat /sys/kernel/debug/tracing/events/syscalls/sys_exit_epoll_wait/format
 */
tracepoint:syscalls:sys_exit_epoll_wait
/pid==$1/
{
       // printf("\n***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_wait\n", elapsed, tid, comm);

       // printf("epfd: 0x%08lx, events: 0x%08lx, maxevents: 0x%08lx, timeout: 0x%08lx \n", 
       //        ((args->epfd)), ((args->events)), ((args->maxevents)), ((args->timeout)) );

       @tid2Waketime[tid]=elapsed;
       @tid2epollNrFdReady[tid]=args->ret;
}

/*
记录本轮 epoll 事件触发的 FD 级回调开始

Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int fd, short events, void* fileEventImplThis)  !!!NOT!!: FileEventImpl::assignEvents()
C++11 Lambda expressions:
event_assign( &raw_event_, base, fd_, xxx, [](evutil_socket_t, short what, void* arg)-> void {}, this );

event_assign(): https://libevent.org/doc/event_8h.html#a3e49a8172e00ae82959dfe64684eda11
       event_assign	(	struct event * 	ev,
              struct event_base * 	base,
              evutil_socket_t 	fd,
              short 	events,
              event_callback_fn 	callback,
              void * 	callback_arg 
       )	

       https://libevent.org/doc/event_8h.html#aed2307f3d9b38e07cc10c2607322d758
       typedef void(* event_callback_fn) (evutil_socket_t, short, void *)
              fd	An fd or signal
              events	One or more EV_* flags
              arg	A user-supplied argument.
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FileEventImpl*assignEvents*
/pid == $1/ 
{
       $fd = arg0;
       if( @fd2sockpair[$fd].0 || @upFD[$fd] ) {
              $libevent_events = arg1;

              printf("\n***** elapsed=%d: tid=%d,comm=%s: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()\n", elapsed, tid, comm);
              printf("FileEventImpl*=%p, fd=%d, events=0x%x\n",arg2, $fd, $libevent_events);

              if( $libevent_events & (uint16)0x01 /*EV_TIMEOUT*/ ) {
                     printf("libevent: EV_TIMEOUT\n");
              }
              if( $libevent_events & (uint16)0x02 /*EV_TIMEOUT*/ ) {
                     printf("libevent: EV_READ\n");
              }
              if( $libevent_events & (uint16)0x04 /*EV_TIMEOUT*/ ) {
                     printf("libevent: EV_WRITE\n");
              }
              if( $libevent_events & (uint16)0x20 /*EV_TIMEOUT*/ ) {
                     printf("libevent: EV_ET\n");
              }
              if( $libevent_events & (uint16)0x80 /*EV_TIMEOUT*/ ) {
                     printf("libevent: EV_CLOSED\n");
              }
              
              printf("%s\n", kstack);
              @fdFired[tid]=$fd;
              @watchedWakeRound[tid]=tid;
       }
}

/*
记录本轮 epoll 事件触发的 FD 级回调结束
*/
uretprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FileEventImpl*assignEvents*
/pid == $1 && @fdFired[tid] / 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: END:EventFired\n", elapsed, tid, comm);
       delete(@fdFired[tid]);
}

/*
埋点 socket 读
*/ 
tracepoint:syscalls:sys_enter_read,tracepoint:syscalls:sys_enter_readv,tracepoint:syscalls:sys_enter_recv*
/pid == $1 && (@fd2sockpair[args->fd].0 || @upFD[args->fd] ) / 
{
       @watchedIo[tid]=args->fd;
}

tracepoint:syscalls:sys_exit_read,tracepoint:syscalls:sys_exit_readv,tracepoint:syscalls:sys_exit_recv*
/pid == $1 && @watchedIo[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: socket_read, probe=%s, fd=%d, ret=%d\n", elapsed, tid, comm, probe, @watchedIo[tid],args->ret);
       delete(@watchedIo[tid]);
       printf("%s\n", ustack(20));
}

/*
埋点 socket 写
*/
tracepoint:syscalls:sys_enter_write,tracepoint:syscalls:sys_enter_writev
/pid == $1 && ( @fd2sockpair[args->fd].0 || @upFD[args->fd] ) / 
{
       @watchedIo[tid]=args->fd;
}

tracepoint:syscalls:sys_exit_write,tracepoint:syscalls:sys_exit_writev
/pid == $1 && @watchedIo[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: socket_write, probe=%s, fd=%d, ret=%d\n", elapsed, tid, comm, probe, @watchedIo[tid], args->ret);
       delete(@watchedIo[tid]);
       printf("%s\n", ustack(20));
}


/*
打印 TlsInspector 的所有函数调用
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*TlsInspector*
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: TlsInspector*, probe=%s\n", elapsed, tid, comm, probe);
}

/*
打印连接确认协议后的函数调用
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*ConnectionSocketImpl*setRequestedApplicationProtocols*
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: ConnectionSocketImpl::setRequestedApplicationProtocols\n", elapsed, tid, comm);
       // printf("%s", ustack);
}

//SslSocket::SslSocket
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocketC2ENSt3__110shared_ptrINS_3Ssl7ContextEEENS2_12InitialStateERKNS5_IKNS_7Network22TransportSocketOptionsEEENS4_8functionIFNS5_INS6_10HandshakerEEENS4_10unique_ptrI6ssl_stN4bssl8internal7DeleterISK_EEEEiPNS6_18HandshakeCallbacksEEEE
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}

// setTransportSocketCallbacks
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket27setTransportSocketCallbacksERNS_7Network24TransportSocketCallbacksE
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}


// SslSocket::doRead
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket6doReadERNS_6Buffer8InstanceE
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}


// SslSocket::doWrite
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket7doWriteERNS_6Buffer8InstanceEb
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}

// SslSocket::onSuccess
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_N5Envoy10Extensions16TransportSockets3Tls9SslSocket9onSuccessEP6ssl_st
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}

// SslSocket::connection
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_NK5Envoy10Extensions16TransportSockets3Tls9SslSocket10connectionEv
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}

// SslSocket::connection
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_NK5Envoy10Extensions16TransportSockets3Tls9SslSocket10connectionEv
/pid == $1 && @fdFired[tid]/ 
{
       printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
       printf("%s\n", ustack(10));
}

/*
打印匹配到的 Network Fitler Chain 名字
void setFilterChainName(absl::string_view filter_chain_name)
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10StreamInfo14StreamInfoImpl18setFilterChainNameEN4absl11string_viewE 
/pid == $1 && @fdFired[tid]/ 
{ 
/**
[Assembly 2: Calling convention](https://cs61.seas.harvard.edu/site/2018/Asm2/)

1. A structure argument that fits in a single machine word (64 bits/8 bytes) is passed in a single register.
   
    Example: `struct small { char a1, a2; }`

2. A structure that fits in two to four machine words (16–32 bytes) is passed in sequential registers, as if it were multiple arguments.
   
    Example: `struct medium { long a1, a2; }`
*/
    $filterName = str(reg("si"));
    $filterNameLength = reg("dx");

    // printf("tid:%d: Got setFilterChainName=%s, lenght=%d\n %s \n", tid, $filterName, $filterNameLength, ustack ); 
    printf("comm:%s,tid:%d: Got setFilterChainName=%s, lenght=%d\n", comm, tid, $filterName, $filterNameLength ); 

    if( $filterNameLength > 0 ) {
        printf("%s\n", ustack(3));
    }
}

/*
埋点 FilterManagerImpl::add*Filter()
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*add*Filter*,uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*FilterManagerImpl*
/pid == $1 && @fdFired[tid]/ 
{ 
       if( @upFD[@fdFired[tid]] ) {
              return;
       }

       $fd = @fdFired[tid];
       @filterManagerImpl2fd[arg0] = $fd;
       if( @fd2filterManagerImpl[$fd] ) {
              @fd2filterManagerImpl2[$fd] = arg0;
       } else {
              @fd2filterManagerImpl[$fd] = arg0;
       }
       printf("comm:%s,tid:%d: FilterManagerImpl.add*Filter|FilterManagerImpl::%s,FilterManagerImpl.this=%p,fd=%d \n", comm, tid, probe, arg0, $fd);
       printf("%s\n", ustack(40));
}

/*
埋点 FilterManagerImpl::on*()
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*on*
/pid == $1 && @fdFired[tid]/ 
{ 
       if( @upFD[@fdFired[tid]] ) {
              return;
       }

       $fd = @fdFired[tid];
       @filterManagerImpl2fd[arg0] = $fd;
       if( @fd2filterManagerImpl[$fd] ) {
              @fd2filterManagerImpl2[$fd] = arg0;
       } else {
              @fd2filterManagerImpl[$fd] = arg0;
       }
       printf("comm:%s,tid:%d: FilterManagerImpl.on*::%s,FilterManagerImpl.this=%p,fd=%d \n", comm, tid, probe, arg0, $fd); 
}

/*
埋点 ConnPoolImplBase::attachStreamToClient()
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*ConnPoolImplBase*attachStreamToClient*
/pid == $1 && @fdFired[tid]/ 
{ 
       printf("comm:%s,tid:%d: ConnPoolImplBase*attachStreamToClient: %s \n", comm, tid, probe); 
       printf("%s\n", ustack(40));
}

/*
埋点 socket connect upstream
*/
kprobe:tcp_connect
/pid == $1/
{
       $sk = ((struct sock *)arg0);
       $inet_family = $sk->__sk_common.skc_family;

       if ($inet_family == AF_INET || $inet_family == AF_INET6)
       {
              if ($inet_family == AF_INET)
              {
                     $daddr = ntop($sk->__sk_common.skc_daddr);
                     $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
              }
              else
              {
                     $daddr = ntop($sk->__sk_common.skc_v6_daddr.in6_u.u6_addr8);
                     $saddr = ntop($sk->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr8);
              }
              $lport = $sk->__sk_common.skc_num;
              $dport = $sk->__sk_common.skc_dport;

              // Destination port is big endian, it must be flipped
              $dport = ($dport >> 8) | (($dport << 8) & 0x00FF00);

              // only watch listen port 8080
              if (8080 != $dport)
              {
                     return;
              }

              printf("connecting to 8080:\n");
              time("%H:%M:%S ");
              printf("%-8d %-16s ", pid, comm);
              printf("%-39s %-6d %-39s %-6d\n", $saddr, $lport, $daddr, $dport);
              printf("%s\n", ustack(40));
       }
}


/*
清理关闭的 FD 相关的 Map
cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_close/format
*/
tracepoint:syscalls:sys_enter_close
/pid==$1/
{
	$fd = args->fd;
	if( $fd < 0 ) {
		return;
	}

       delete(@upFD[$fd]);

       if( @fd2sockpair[$fd].0 ) {
              printf("sys_enter_close fd=%d\n", $fd);
       }
       else {
              return;
       }
	delete(@fd2sockpair[$fd]);

       /////

       $filterManagerImpl = @fd2filterManagerImpl[$fd];
       if( $filterManagerImpl ) {
              printf("delete fd2filterManagerImpl, fd=%d\n", $fd );
              delete( @fd2filterManagerImpl[$fd] );
       }

       if( @filterManagerImpl2fd[$filterManagerImpl] ) {
              printf("delete fd2filterManagerImpl=%p\n", $filterManagerImpl );
              delete(@filterManagerImpl2fd[$filterManagerImpl]);
       }


       $filterManagerImpl = @fd2filterManagerImpl2[$fd];
       if( $filterManagerImpl ) {
              printf("delete fd2filterManagerImpl, fd=%d\n", $fd );
              delete( @fd2filterManagerImpl2[$fd] );
       }

       if( @filterManagerImpl2fd[$filterManagerImpl] ) {
              printf("delete fd2filterManagerImpl=%p\n", $filterManagerImpl );
              delete(@filterManagerImpl2fd[$filterManagerImpl]);
       }

       delete(@upFD[@downFD2upFD[$fd]]);
       delete(@downFD2upFD[$fd]);
}

END
{
       clear(@last_epoll_wait_args);
       clear(@tid2Waketime);
       clear(@tid2epollNrFdReady);
}

结尾

这 part，从 socket 事件驱动的角度，了解了 envoy 作为反向代理的主要流程。我自己觉得是学习到东西的，你呢？
写博客的初憧是记录自己。想不到也能通过博客认识到很多志同道合，对技术认真之人（不能叫同志？）。谢谢你们，我才能坚持写下去，也认为这样的写作有一定价值！

ending
这是几年前拍的老照片了，共勉。

分享
Mark Zhu
作者
Mark Zhu
An old developer

相关内容