In day 2, I focused on Equal Cost Multi-Path (ECMP) of Fortigate but let me dig deep into the design – architecture aspect.
first, this post is just my personal view and unlocking further about the ECMP.
ECMP job is pretty simple. Its job is to move packets from a source to a destination using multiple equal-cost paths.
It doesn’t care about sessions.
It doesn’t care about users.
It doesn’t care about applications.
Routers don’t remember flows because remembering flows would slow them down. (Router POV: look at a packet and make a forwarding decision and send it out.) Core routing is stateless by design.
Instead, they hash packet headers to ensure that packets belonging to the same flow usually follow the same path. This is not a feature for applications, it’s a feature for packet ordering.
The router does not remember the flow, but it performs a deterministic calculation on every packet.
What gets hashed?
Typically a combination of:
Source IP
Destination IP
Source port
Destination port
Protocol (TCP/UDP/ICMP)
This is often called a:
5-tuple hash (Some platforms allow 2-tuple, 3-tuple, or custom fields.)
Another interesting insight after digging through several technical article & discussions online.
The Moment a Path Becomes a Server (refer to anycast deployment)
The destination is no longer a stable endpoint.
It is now a pool of servers that scale, reboot, deploy, and fail.
When one path fails, the traffic get re-shuffled over the available number of paths.
Packets that once arrived at a server that knew the connection now arrive at one that has never seen it before. From the servers perspective, these packets are invalid.
The only safe response is to reset the connection.
If ECMP uses hashing, shouldn’t it be stable when things change? The answer is no.
ECMP uses deterministic hashing.
Given the same inputs and the same number of next hops, the result is stable.
Change the number of next hops, and the entire mapping shifts.
Consistent hashing exists specifically to solve this problem.
It accepts that backends change and designs to minimize disruption when they do.
That’s why load balancers implement it.
That’s why reverse proxies track state.
That’s why large-scale systems separate packet movement from session ownership.
Routers optimize for speed.
Load balancers optimize for continuity.
Once this is understood, the architecture becomes obvious.
ECMP moves packets.
Load balancers own connections.
Proxies understand sessions.
Each layer solves a different problem.
Each layer has different assumptions.
Most production outages don’t happen because a system failed.
They happen because we asked a system to do something it never promised to do.
ECMP doesn’t need to be fixed.
It needs to be used correctly.
And the real skill in infrastructure engineering isn’t knowing every protocol detail.
It’s knowing where one responsibility ends and where another must begin.
reference(s):
https://www.imperva.com/learn/performance/anycast/
https://blog.ipspace.net/2021/05/tcp-anycast-hard/
https://community.cisco.com/t5/service-providers-knowledge-base/xr-ncs5500-asr9000-persistent-loadbalancing-or-quot-sticky-ecmp/ta-p/3361883
https://www.reddit.com/r/networking/comments/x8gpsg/unstable_nature_of_flowbased_ecmp_routing/
https://www.rfc-editor.org/rfc/rfc2992