 |
After a while of searching around, turns out, that an old school approach like connection lib has shown its greatly effective in this extremely low latency problem.
This architecture was famous and implemented within 3 open-source libraries: Uber Hyperbahn [13], Java Maven library Netflix Ribbon [14], and Meta ServiceRouter Lib (SRLib) [15].
The Lib connection approach mainly serves the need for extremely low latency connection and self-managed data flow. It creates direct connections between peers, on the layer 4 network connection and manages each inside the internal client load balancing (ILB) module embedded in each application.
This approach required us to care more about engineering aspects of a connection like lag, misconfigured, and incidents, ... however, also brought us the raw latency of the low-level network.
The ILB can be used independently or it has interfaces to interact with the centralized control plane. ILB gives us a platform to observe and control peer connections. It gives us the ability to imply full-time protection for the system and customize utilities for our use cases.
The cloud infrastructure in terms of provisioning, scale-up, scale down to adapt to user demand, delivers very late results in comparison to system demand changes. Meanwhile, some instances may overload, leak to death, and lead to a chain of catastrophe while waiting for resources to be provisioned.
the principle of ILB is simple, try to deliver the best decision with as much local information as possible. It controls the flow per-hop by per-flow on a p2p basis and acts immediately on each request it sends.
ILB solved the latency problem by removing hops of external proxy and sidecar, bringing the ability to be quick, continuous, and self-decidable in complex misinformation cases, and response to local factors without adding network delay.
One of the minor problems of ILB is that it increases computing overhead for flow control. But with an instance allocated with more than 2 vCPUs and less than 30kTPS, this is not a big problem.
The problem of lib connection from point of view of ILB connection lib is that all messages are equal, which means the lib can not use some dimensions to make decisions, like 1) latency and timing, Timeout is defined by business, 2) message Size, 3) API Destination or path
|
IEEE/ICACT20240417 Slide.07
[Big Slide]