* Re: pasta: UDP flow sockets use the default receive buffer, dropping datagrams of a WireGuard client
[not found] <E2C65470-86B4-4194-B233-5E4571B0DBDC@wiesing.me>
@ 2026-11-19 5:36 ` Stefano Brivio
0 siblings, 0 replies; only message in thread
From: Stefano Brivio @ 2026-11-19 5:36 UTC (permalink / raw)
To: Tobias Wiesing; +Cc: passt-user
Hi Tobias,
On Wed, 18 Nov 2026 20:04:15 +0100
Tobias Wiesing <tobias@wiesing.me> wrote:
> Setup: passt 2026_10_02.cba3570, Podman 6.1.4, kernel 7.3.4 (openSUSE
> MicroOS), rootless; a WireGuard client (gluetun, kernel WireGuard) in a
> container on a Podman bridge network, so its UDP leaves through pasta.
> Host: net.core.rmem_default = 212992, rmem_max = 4194304.
>
> Observed: downloads through the tunnel reach 4-8 MB/s on a line that
> serves ~30 MB/s without it. During a 15 s download the host counts
> 300-15000 UdpRcvbufErrors (nstat). Counting the tunnel's datagrams at the
> host NIC, pasta's tap in the rootless netns, the container's eth0 and
> the WireGuard interface shows no loss after pasta once the buffer is
> large enough.
>
> With net.core.rmem_default = 4194304 (nothing else changed, container
> restarted so pasta opens new sockets): 0 UdpRcvbufErrors, 24-27 MB/s.
> The same tunnel under rootful Docker on another host reaches line rate
> with the kernel defaults, as no socket is involved there.
>
> Source: udp.c / udp_flow.c / sock_l4() set no SO_RCVBUF on UDP sockets;
> SO_RCVBUF/SO_SNDBUF are set only on the tap fd (tap.c), after
> sock_probe_features(). Unlike TCP there is no autotuning for UDP, so the
> socket keeps rmem_default.
>
> Question: could pasta set SO_RCVBUF on its UDP sockets (e.g. up to
> rmem_max when sock_probe_features() allows large buffers, as for the tap
> fd), or offer an option for it? Raising rmem_default host-wide works but
> affects every socket on the host.
Thanks for the report and for the suggestion. I think it needs a bit
of research.
For context, for TCP, we used to set large values, and then we could
finally drop those settings (and let auto-tuning do its course) after a
number of optimisations and fixes:
https://passt.top/passt/commit/?id=71249ef3f9bcf1dbb2d6c13cdbc41ba88c794f06
and the topic is tracked at https://bugs.passt.top/show_bug.cgi?id=138
in general.
For UDP, we never really looked into it. I had no idea it mattered
until your report, actually.
For maximum throughput, of course, we could simply try to set SO_RCVBUF
to rmem_max on all sockets, and the buffer allocation in the kernel
should only happen as that memory is actually used, so we don't
_necessarily_ waste memory by doing that, but I suppose there might be
a risk (especially in case the whole buffer is used at some point, and
then buffer usage decreases).
I see three possibilities, broadly speaking:
1. implement some kind of auto-tuning ourselves. For example
udp_sock_recv() (or callers) could check how many bytes are dequeued
at once, and periodically adjust the buffer size for a specific
socket with some hysteresis (say, expand if we use more than 50 % of
the buffer, shrink if we use less than 20 %)
2. add a command line option, or two (what about SO_SNDBUF?), that
would apply to all the UDP sockets we use. Safe, but not really
convenient as most users wouldn't know about it, and by default we
might impose a performance penalty on them
3. just go with the maximum size, maybe it's fine and we don't actually
waste any memory (but it needs to be checked)
One further thought before trying to evaluate one of these, though: this
is UDP, where latency might matter more than throughput, especially if
you carry TCP on top of it, and just making buffers larger might by
itself contribute to bufferbloat.
Could your issue with Wireguard come from the fact that we dequeue "too
many" frames at once? What happens if you decrease UDP_MAX_FRAMES in
udp.c to, say, 8? Should we try to (dynamically?) tune that value
instead?
All in all, I think any of those alternatives would be viable, so feel
free to send changes, but they need to be supported by some evidence
and testing.
--
Stefano
^ permalink raw reply [flat|nested] only message in thread