From: Stefano Brivio <sbrivio@redhat.com>
To: Tobias Wiesing <tobias@wiesing.me>
Cc: passt-user@passt.top
Subject: Re: pasta: UDP flow sockets use the default receive buffer, dropping datagrams of a WireGuard client
Date: Thu, 19 Nov 2026 06:36:03 +0100 (CET) [thread overview]
Message-ID: <20261119063602.7095a07c@elisabeth> (raw)
In-Reply-To: <E2C65470-86B4-4194-B233-5E4571B0DBDC@wiesing.me>
Hi Tobias,
On Wed, 18 Nov 2026 20:04:15 +0100
Tobias Wiesing <tobias@wiesing.me> wrote:
> Setup: passt 2026_10_02.cba3570, Podman 6.1.4, kernel 7.3.4 (openSUSE
> MicroOS), rootless; a WireGuard client (gluetun, kernel WireGuard) in a
> container on a Podman bridge network, so its UDP leaves through pasta.
> Host: net.core.rmem_default = 212992, rmem_max = 4194304.
>
> Observed: downloads through the tunnel reach 4-8 MB/s on a line that
> serves ~30 MB/s without it. During a 15 s download the host counts
> 300-15000 UdpRcvbufErrors (nstat). Counting the tunnel's datagrams at the
> host NIC, pasta's tap in the rootless netns, the container's eth0 and
> the WireGuard interface shows no loss after pasta once the buffer is
> large enough.
>
> With net.core.rmem_default = 4194304 (nothing else changed, container
> restarted so pasta opens new sockets): 0 UdpRcvbufErrors, 24-27 MB/s.
> The same tunnel under rootful Docker on another host reaches line rate
> with the kernel defaults, as no socket is involved there.
>
> Source: udp.c / udp_flow.c / sock_l4() set no SO_RCVBUF on UDP sockets;
> SO_RCVBUF/SO_SNDBUF are set only on the tap fd (tap.c), after
> sock_probe_features(). Unlike TCP there is no autotuning for UDP, so the
> socket keeps rmem_default.
>
> Question: could pasta set SO_RCVBUF on its UDP sockets (e.g. up to
> rmem_max when sock_probe_features() allows large buffers, as for the tap
> fd), or offer an option for it? Raising rmem_default host-wide works but
> affects every socket on the host.
Thanks for the report and for the suggestion. I think it needs a bit
of research.
For context, for TCP, we used to set large values, and then we could
finally drop those settings (and let auto-tuning do its course) after a
number of optimisations and fixes:
https://passt.top/passt/commit/?id=71249ef3f9bcf1dbb2d6c13cdbc41ba88c794f06
and the topic is tracked at https://bugs.passt.top/show_bug.cgi?id=138
in general.
For UDP, we never really looked into it. I had no idea it mattered
until your report, actually.
For maximum throughput, of course, we could simply try to set SO_RCVBUF
to rmem_max on all sockets, and the buffer allocation in the kernel
should only happen as that memory is actually used, so we don't
_necessarily_ waste memory by doing that, but I suppose there might be
a risk (especially in case the whole buffer is used at some point, and
then buffer usage decreases).
I see three possibilities, broadly speaking:
1. implement some kind of auto-tuning ourselves. For example
udp_sock_recv() (or callers) could check how many bytes are dequeued
at once, and periodically adjust the buffer size for a specific
socket with some hysteresis (say, expand if we use more than 50 % of
the buffer, shrink if we use less than 20 %)
2. add a command line option, or two (what about SO_SNDBUF?), that
would apply to all the UDP sockets we use. Safe, but not really
convenient as most users wouldn't know about it, and by default we
might impose a performance penalty on them
3. just go with the maximum size, maybe it's fine and we don't actually
waste any memory (but it needs to be checked)
One further thought before trying to evaluate one of these, though: this
is UDP, where latency might matter more than throughput, especially if
you carry TCP on top of it, and just making buffers larger might by
itself contribute to bufferbloat.
Could your issue with Wireguard come from the fact that we dequeue "too
many" frames at once? What happens if you decrease UDP_MAX_FRAMES in
udp.c to, say, 8? Should we try to (dynamically?) tune that value
instead?
All in all, I think any of those alternatives would be viable, so feel
free to send changes, but they need to be supported by some evidence
and testing.
--
Stefano
parent reply other threads:[~2026-11-19 5:36 UTC|newest]
Thread overview: expand[flat|nested] mbox.gz Atom feed
[parent not found: <E2C65470-86B4-4194-B233-5E4571B0DBDC@wiesing.me>]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261119063602.7095a07c@elisabeth \
--to=sbrivio@redhat.com \
--cc=passt-user@passt.top \
--cc=tobias@wiesing.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for IMAP folder(s).