Issues when using pasta with bubblewrap

Issues when using pasta with bubblewrap

[Lisa Gnedt]

Hi,

I am working on integrating pasta into NixPak [1], which boils down to 
using pasta together with bubblewrap [2].
It basically works, but in a specific edge-case I am running into problems.
The edge-case is when bubblewrap creates two layers of user namespaces.

What I am basically doing is let bubblewrap create a new network 
namespace and then start pasta to create the interfaces accordingly.
In the NixPak support, I am doing this in coordination with bubblewrap 
to start pasta before the actual application is launched.
However, here are a few minimum examples for re-producing the problem.
All testcases were run using pasta 2025_06_11.0293c6f on Linux 
6.12.34-hardened1.


Testcase A: Single layer of user namespaces -> Works
----------------------------------------------------

First, I start the bwrap sandbox with a new network namespace:

$ bwrap --unshare-all --ro-bind / / /bin/sh
sh-5.2$ ip -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128

In another terminal window, I start pasta with the pid of the bubblewrap 
child which runs inside the new Linux namespaces:

$ pasta --config-net --no-dhcp --no-dhcpv6 --no-ndp --no-ra --no-map-gw 
--tcp-ns none --udp-ns none --tcp-ports none --udp-ports none 
--ns-ifname eth0 --address 192.168.1.100 --netmask 255.255.255.0 
--gateway 192.168.1.1 --mac-addr 52:54:00:12:34:56 --dns-forward 
192.168.1.1 --search none 389267
No interfaces with usable IPv6 routes
Template interface: eno2 (IPv4)
Namespace interface: eth0
MAC:
     host: 52:54:00:12:34:56
DNS:
     192.168.1.1

Then, I go back to the bwrap sandbox and check that the network 
namespace is now fully set up:

sh-5.2$ ip -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0             UNKNOWN        192.168.1.100/24 
fe80::bceb:d3ff:fe9c:d037/64


Testcase B: Two layers of user namespaces -> Fails directly, but works 
with nsenter
-----------------------------------------------------------------------------------

First, I start again the bwrap sandbox with a new network namespace and 
the option --dev which is one case where bubblewrap creates two layers 
of user namespaces:

$ bwrap --unshare-all --ro-bind / / --dev /dev /bin/sh
sh-5.2$ ip -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128

In another terminal window, I try to start pasta with the pid of the 
bubblewrap child which runs inside the new Linux namespaces:

$ pasta --config-net --no-dhcp --no-dhcpv6 --no-ndp --no-ra --no-map-gw 
--tcp-ns none --udp-ns none --tcp-ports none --udp-ports none 
--ns-ifname eth0 --address 192.168.1.100 --netmask 255.255.255.0 
--gateway 192.168.1.1 --mac-addr 52:54:00:12:34:56 --dns-forward 
192.168.1.1 --search none 390352
No interfaces with usable IPv6 routes
Couldn't switch to pasta namespaces: Operation not permitted

This does not work, since pasta joined the second layer user namespace 
which does not own the network namespace.
What works although is if I join the first layer user namespace with 
nsenter and then let pasta run:

$ nsenter -t 390352 -U --preserve-credentials --user-parent -- pasta 
--config-net --no-dhcp --no-dhcpv6 --no-ndp --no-ra --no-map-gw --tcp-ns 
none --udp-ns none --tcp-ports none --udp-ports none --ns-ifname eth0 
--address 192.168.1.100 --netmask 255.255.255.0 --gateway 192.168.1.1 
--mac-addr 52:54:00:12:34:56 --dns-forward 192.168.1.1 --search none 
--netns /proc/390352/ns/net
No interfaces with usable IPv6 routes
Template interface: eno2 (IPv4)
Namespace interface: eth0
MAC:
     host: 52:54:00:12:34:56
DNS:
     192.168.1.1

Then, I go back to the bwrap sandbox and check that the network 
namespace is now fully set up:

sh-5.2$ ip -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0             UNKNOWN        192.168.1.100/24 fe80::54fa:e1ff:fe87:79e/64


Ideas for Solutions
-------------------

I am trying to find a solution that works with both testcases (single 
and two layers of user namespaces).
My idea would be to always join the owning user namespace of the network 
namespace.
I tried to simulate this with nsenter, but for some reason I am not 
getting pasta working for the single layer user namespace (testcase A):

$ bwrap --unshare-all --ro-bind / / /bin/sh
sh-5.2$ ip -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128

$ nsenter -t 390424 -U --preserve-credentials -- pasta --config-net 
--no-dhcp --no-dhcpv6 --no-ndp --no-ra --no-map-gw --tcp-ns none 
--udp-ns none --tcp-ports none --udp-ports none --ns-ifname eth0 
--address 192.168.1.100 --netmask 255.255.255.0 --gateway 192.168.1.1 
--mac-addr 52:54:00:12:34:56 --dns-forward 192.168.1.1 --search none 
--netns /proc/390424/ns/net
No interfaces with usable IPv6 routes
Couldn't switch to pasta namespaces: Operation not permitted


While experimenting with this, I wondered if it is a scenario that pasta 
would like to support out of the box.
It might be easier to get it correct when directly controlling all 
syscalls involved and not have to mix and match multiple tools.
Since Linux 4.9 it seems to be possible to get the owning user namespace 
of a network namespace with the ioctl NS_GET_USERNS [3].

Do you consider looking into this or would you accept a patch for adding 
support for this?


Best regards,
Lisa Gnedt

[1] https://github.com/nixpak/nixpak
[2] https://github.com/containers/bubblewrap
[3] https://man7.org/linux/man-pages/man2/ns_get_userns.2const.html

Re: Issues when using pasta with bubblewrap

[Lisa Gnedt]

Hi,

On 2025-07-06 17:15, Lisa Gnedt wrote:
> It might be easier to get it correct when directly controlling all
> syscalls involved and not have to mix and match multiple tools.
> Since Linux 4.9 it seems to be possible to get the owning user namespace
> of a network namespace with the ioctl NS_GET_USERNS [3].

I just wrote a hacky patch as proof-of-concept of this idea.
It is working for me fine in both testcases.
However, in its current form it breaks the --userns parameter. But it should not be too hard to address this issue.

I am not sure, what kernel version compatibility you are targeting, since the ioctl is only available since Linux 4.9.
Would it be an option for you to make it the default behavior when a PID is specified?
From my perspective this should be the expected behavior and should not break any previously working use case.

Best regards,
Lisa Gnedt


diff --git a/conf.c b/conf.c
index 36845e2..cd67e7a 100644
--- a/conf.c
+++ b/conf.c
@@ -642,7 +642,7 @@ static void conf_pasta_ns(int *netns_only, char *userns, char *netns,
 
 			if (!*userns) {
 				if (snprintf_check(userns, PATH_MAX,
-						   "/proc/%ld/ns/user", pidval))
+						   "/proc/%ld/ns/net", pidval))
 					die_perror("Can't build userns path");
 			}
 		}
diff --git a/isolation.c b/isolation.c
index bbcd23b..cbfe0f0 100644
--- a/isolation.c
+++ b/isolation.c
@@ -81,6 +81,7 @@
 #include <linux/audit.h>
 #include <linux/capability.h>
 #include <linux/filter.h>
+#include <linux/nsfs.h>
 #include <linux/seccomp.h>
 
 #include "util.h"
@@ -254,6 +255,14 @@ void isolate_user(uid_t uid, gid_t gid, bool use_userns, const char *userns,
 		if (ufd < 0)
 			die_perror("Couldn't open user namespace %s", userns);
 
+		int real_ufd;
+		real_ufd = ioctl(ufd, NS_GET_USERNS);
+		if (real_ufd < 0)
+			die_perror("Couldn't get user namespace from network namespace %s", userns);
+
+		close(ufd);
+		ufd = real_ufd;
+
 		if (setns(ufd, CLONE_NEWUSER) != 0)
 			die_perror("Couldn't enter user namespace %s", userns);
 

Re: Issues when using pasta with bubblewrap

[Stefano Brivio]

Hi Lisa,

On Sun, 6 Jul 2025 19:08:46 +0200
Lisa Gnedt via user <passt-user@passt.top> wrote:

> Hi,
> 
> On 2025-07-06 17:15, Lisa Gnedt wrote:
> > It might be easier to get it correct when directly controlling all
> > syscalls involved and not have to mix and match multiple tools.
> > Since Linux 4.9 it seems to be possible to get the owning user namespace
> > of a network namespace with the ioctl NS_GET_USERNS [3].  
> 
> I just wrote a hacky patch as proof-of-concept of this idea.
> It is working for me fine in both testcases.
> However, in its current form it breaks the --userns parameter. But it should not be too hard to address this issue.
> 
> I am not sure, what kernel version compatibility you are targeting, since the ioctl is only available since Linux 4.9.

Thanks for reporting this and for the draft.

I didn't look into your issue and patch yet, I plan to get to it later
today, but just as a quick answer to this point: the earlier the
better, not everything is reasonable, but 4.9 should be.

And yes, patches for compatibility are always warmly welcome.

-- 
Stefano

Re: Issues when using pasta with bubblewrap

[Stefano Brivio]

[Cc'ing Paul as Podman maintainer, thread at:
 https://archives.passt.top/passt-user/671252c8-88f6-45b7-b719-b82786e84bb7@gnedt.at/]

On Sun, 6 Jul 2025 19:08:46 +0200
Lisa Gnedt via user <passt-user@passt.top> wrote:

> Hi,
> 
> On 2025-07-06 17:15, Lisa Gnedt wrote:
> > It might be easier to get it correct when directly controlling all
> > syscalls involved and not have to mix and match multiple tools.
> > Since Linux 4.9 it seems to be possible to get the owning user namespace
> > of a network namespace with the ioctl NS_GET_USERNS [3].  
> 
> I just wrote a hacky patch as proof-of-concept of this idea.
> It is working for me fine in both testcases.
> However, in its current form it breaks the --userns parameter. But it should not be too hard to address this issue.
> 
> I am not sure, what kernel version compatibility you are targeting, since the ioctl is only available since Linux 4.9.
> Would it be an option for you to make it the default behavior when a PID is specified?
> >From my perspective this should be the expected behavior and should not break any previously working use case.  

I finally had a second look, a bit quicker than I wanted but I think I
grasped the issue.

For context, "this" is: always join the user namespace owning a network
namespace.

It looks reasonable (and desirable) to me, but I'm not sure how / why
it breaks the --userns parameter.

We should probably never do this when --netns-only is given (that's
Podman's case, for example).

It would be good to have a way to "cleanly" exclude this new behaviour,
but, once we add the NS_GET_USERNS trick, --netns-only doesn't exactly
get us back to the previous behaviour. What about --userns-from-pid or
something like that? That name isn't great though.

Now, 4.9 feels "old" enough, but pasta used to run on a 3.13 kernel a
while ago, then a few things were (inadvertently) broken. But it
"almost" does. Couldn't we just add a fallback for the case where
NS_GET_USERNS fails? You're already handling the error. You could just
print a warning and continue instead of calling die_perror()...

> diff --git a/conf.c b/conf.c
> index 36845e2..cd67e7a 100644
> --- a/conf.c
> +++ b/conf.c
> @@ -642,7 +642,7 @@ static void conf_pasta_ns(int *netns_only, char *userns, char *netns,
>  
>  			if (!*userns) {
>  				if (snprintf_check(userns, PATH_MAX,
> -						   "/proc/%ld/ns/user", pidval))
> +						   "/proc/%ld/ns/net", pidval))
>  					die_perror("Can't build userns path");
>  			}
>  		}
> diff --git a/isolation.c b/isolation.c
> index bbcd23b..cbfe0f0 100644
> --- a/isolation.c
> +++ b/isolation.c
> @@ -81,6 +81,7 @@
>  #include <linux/audit.h>
>  #include <linux/capability.h>
>  #include <linux/filter.h>
> +#include <linux/nsfs.h>
>  #include <linux/seccomp.h>
>  
>  #include "util.h"
> @@ -254,6 +255,14 @@ void isolate_user(uid_t uid, gid_t gid, bool use_userns, const char *userns,
>  		if (ufd < 0)
>  			die_perror("Couldn't open user namespace %s", userns);
>  
> +		int real_ufd;
> +		real_ufd = ioctl(ufd, NS_GET_USERNS);
> +		if (real_ufd < 0)
> +			die_perror("Couldn't get user namespace from network namespace %s", userns);
> +
> +		close(ufd);
> +		ufd = real_ufd;
> +
>  		if (setns(ufd, CLONE_NEWUSER) != 0)
>  			die_perror("Couldn't enter user namespace %s", userns);
>  

-- 
Stefano