Re: Apparmor (and other) Issues

Re: Apparmor (and other) Issues

[Stefano Brivio]

Hi,

On Wed, 29 Jan 2025 09:14:12 +0000
Prafulla Giri <prafulla.giri@protonmail.com> wrote:

> Esteemed maintainer,
> 
> First and foremost, thank you very much for your hard work: passt is awesome and allows one to run more useful user-space VM-s.
> 
> I have encountered 2 particular issues with the usage of passt with Debian, and wanted to bring them to your attention as I think you are probably the best person to deal with this. I do plan on sending a report to the Debian team afterwards.
> 
> For reference, I tested these on Debian Testing Daily Image dated 28 January 2025, with updates, and the version of passt available with it is passt 0.0~git20250121.4f2c8e7-1
> 
> - Passt's default Apparmor config needs to allow writes to $XDG_RUNTIME_DIR (which is at /run/user/$UID). Currently it doesn't. Virt-manager, at least, tries to create the necessary sockets in the directory but apparmor prevents that from happening (and the error message Virt-Manager gives isn't helpful either: the first time around I falsely believed it was a segfault or similar issue). I managed to get passt​ working past this flaw (pun intended) by manually disabling apparmor for the binary. Passt works just fine in Fedora 41 as it doesn't use Apparmor but uses SELinux, and thus the configs don't affect it.

Thanks for reporting this! I'm the maintainer of the Debian package, by
the way. Cc'ing Andrea, who is a maintainer of the libvirt package for
Debian and surely more knowledgeable about this.

Note that virt-manager uses passt through libvirt (I think that's only
possibility) and this should actually be allowed in libvirt's AppArmor
policy, in the sub-profile for passt:

  https://gitlab.com/libvirt/libvirt/-/blob/0264a7704ada52f686cafe8f6402d5b60f9f0fc4/src/security/apparmor/libvirt-qemu.in#L204

the rationale is that passt itself doesn't know which directory libvirt
will pick for its socket and PID files, so libvirt's policy has to
specify that.

So I think you should file an issue for the libvirt package in this
case, unless Andrea has some pointers.

> - This second issue is perhaps a bit more Debian-specific, but I am going to mention it so that you might drop some hints for the Debian maintainers to debug this: Once Apparmor is disabled and a VM is configured to work with passt, DNS resolution doesn't work in the VM (IP Addresses work just fine) i.e. ping fsf.org​ doesn't work but `ping 209.51.188.174` does. The hypervisor details follow:
> $ virsh version # on Debian Testing a.k.a. 'Trixie'
> Compiled against library: libvirt 11.0.0
> Using library: libvirt 11.0.0
> Using API: QEMU 11.0.0Running hypervisor: QEMU 9.2.0
> This, again, isn't an issue with Fedora 41, where everything just works. The hypervisor details for Fedora 41 are:
> $ virsh version # on Fedora 41
> Compiled against library: libvirt 10.6.0
> Using library: libvirt 10.6.0
> Using API: QEMU 10.6.0
> Running hypervisor: QEMU 9.1.2

Oops. Can you share the command line of passt as run by libvirt
(say, 'ps aux|grep passt') for this case? passt has some basic
DNS forwarding capabilities, which are configured depending on
the host's resolver configuration.

> Again, I will be making a report to the Debian maintainers, should they wish to chime in regarding Apparmor configs or the DNS resolution issue.

Please file a separate issue, in case. This one would be for
passt.

> Thank you once again for this awesome tool.

And thanks again for trying it out and reporting issues!

-- 
Stefano

Re: Apparmor (and other) Issues

[Prafulla Giri]

Hello,

On Wednesday, January 29th, 2025 at 3:26 PM, Stefano Brivio <sbrivio@redhat.com> wrote:

> Hi,
> 
> On Wed, 29 Jan 2025 09:14:12 +0000
> Prafulla Giri prafulla.giri@protonmail.com wrote:
> 
> > Esteemed maintainer,
> > 
> > First and foremost, thank you very much for your hard work: passt is awesome and allows one to run more useful user-space VM-s.
> > 
> > I have encountered 2 particular issues with the usage of passt with Debian, and wanted to bring them to your attention as I think you are probably the best person to deal with this. I do plan on sending a report to the Debian team afterwards.
> > 
> > For reference, I tested these on Debian Testing Daily Image dated 28 January 2025, with updates, and the version of passt available with it is passt 0.0~git20250121.4f2c8e7-1
> > 
> > - Passt's default Apparmor config needs to allow writes to $XDG_RUNTIME_DIR (which is at /run/user/$UID). Currently it doesn't. Virt-manager, at least, tries to create the necessary sockets in the directory but apparmor prevents that from happening (and the error message Virt-Manager gives isn't helpful either: the first time around I falsely believed it was a segfault or similar issue). I managed to get passt working past this flaw (pun intended) by manually disabling apparmor for the binary. Passt works just fine in Fedora 41 as it doesn't use Apparmor but uses SELinux, and thus the configs don't affect it.
> 
> 
> Thanks for reporting this! I'm the maintainer of the Debian package, by
> the way. Cc'ing Andrea, who is a maintainer of the libvirt package for
> Debian and surely more knowledgeable about this.
> 

I'm glad to have bumped into you. Because of the email domain, I thought you weren't the Debian maintainer. Silly me.

> Note that virt-manager uses passt through libvirt (I think that's only
> possibility) and this should actually be allowed in libvirt's AppArmor
> policy, in the sub-profile for passt:
> 
> https://gitlab.com/libvirt/libvirt/-/blob/0264a7704ada52f686cafe8f6402d5b60f9f0fc4/src/security/apparmor/libvirt-qemu.in#L204
> 
> the rationale is that passt itself doesn't know which directory libvirt
> will pick for its socket and PID files, so libvirt's policy has to
> specify that.
> 
> So I think you should file an issue for the libvirt package in this
> case, unless Andrea has some pointers.

I will wait for the maintainers input on this one.

> 
> > - This second issue is perhaps a bit more Debian-specific, but I am going to mention it so that you might drop some hints for the Debian maintainers to debug this: Once Apparmor is disabled and a VM is configured to work with passt, DNS resolution doesn't work in the VM (IP Addresses work just fine) i.e. ping fsf.org doesn't work but `ping 209.51.188.174` does. The hypervisor details follow:
> > $ virsh version # on Debian Testing a.k.a. 'Trixie'
> > Compiled against library: libvirt 11.0.0
> > Using library: libvirt 11.0.0
> > Using API: QEMU 11.0.0Running hypervisor: QEMU 9.2.0
> > This, again, isn't an issue with Fedora 41, where everything just works. The hypervisor details for Fedora 41 are:
> > $ virsh version # on Fedora 41
> > Compiled against library: libvirt 10.6.0
> > Using library: libvirt 10.6.0
> > Using API: QEMU 10.6.0
> > Running hypervisor: QEMU 9.1.2
> 
> 
> Oops. Can you share the command line of passt as run by libvirt
> (say, 'ps aux|grep passt') for this case? passt has some basic
> DNS forwarding capabilities, which are configured depending on
> the host's resolver configuration.
> 

Certainly! I'm sorry I didn't do this earlier. I'd checked on this: there is no difference between the command that runs passt on Fedora 41 or Debian Trixie.

This is the command on Fedora 41:
passt --one-off --socket /run/user/1000/libvirt/qemu/run/passt/4-dragora-net0.socket --pid /run/user/1000/libvirt/qemu/run/passt/4-dragora-net0-passt.pid

and this is the command on Debian Trixie:
passt --one-off --socket /run/user/1000/libvirt/qemu/run/passt/1-vm1-net0.socket --pid /run/user/1000/libvirt/qemu/run/passt/1-vm1-net0-passt.pid

Just for the record, I'm also putting in the QEMU commands for Fedora 41 and Debian Trixie, as well:

Fedora 41:
/usr/bin/qemu-system-x86_64 -name guest=debian-trixie,debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/home/larryboy/.config/libvirt/qemu/lib/domain-5-debian-trixie/master-key.aes"} -machine pc-q35-9.1,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram,hpet=off,acpi=on -accel kvm -cpu host,migratable=on -m size=8388608k -object {"qom-type":"memory-backend-ram","id":"pc.ram","size":8589934592} -overcommit mem-lock=off -smp 4,sockets=4,cores=1,threads=1 -uuid d00ad47e-3cfe-4a1a-af01-b23417aad670 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=31,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-shutdown -global ICH9-LPC.disable_s3=1 -global ICH9-LPC.disable_s4=1 -boot strict=on -device {"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"} -device {"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"} -device {"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"} -device {"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"} -device {"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"} -device {"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"} -device {"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"} -device {"driver":"pcie-root-port","port":23,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x7"} -device {"driver":"pcie-root-port","port":24,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x3"} -device {"driver":"pcie-root-port","port":25,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x3.0x1"} -device {"driver":"pcie-root-port","port":26,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x3.0x2"} -device {"driver":"pcie-root-port","port":27,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x3.0x3"} -device {"driver":"pcie-root-port","port":28,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x3.0x4"} -device {"driver":"pcie-root-port","port":29,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x3.0x5"} -device {"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.2","addr":"0x0"} -device {"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.3","addr":"0x0"} -blockdev {"driver":"file","filename":"/home/larryboy/.local/share/libvirt/images/debian-13-nocloud-amd64-daily.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":true,"discard":"unmap","driver":"qcow2","file":"libvirt-2-storage","backing":null} -blockdev {"driver":"file","filename":"/home/larryboy/.local/share/libvirt/images/debian-trixie.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"discard":"unmap","driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"} -device {"driver":"virtio-blk-pci","bus":"pci.4","addr":"0x0","drive":"libvirt-1-format","id":"virtio-disk0","bootindex":1} -netdev {"type":"stream","addr":{"type":"unix","path":"/run/user/1000/libvirt/qemu/run/passt/5-debian-trixie-net0.socket"},"server":false,"reconnect":5,"id":"hostnet0"} -device {"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:8f:e7:c3","bus":"pci.1","addr":"0x0"} -chardev pty,id=charserial0 -device {"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0} -chardev socket,id=charchannel0,fd=30,server=on,wait=off -device {"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"} -chardev spicevmc,id=charchannel1,name=vdagent -device {"driver":"virtserialport","bus":"virtio-serial0.0","nr":2,"chardev":"charchannel1","id":"channel1","name":"com.redhat.spice.0"} -device {"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"} -audiodev {"id":"audio1","driver":"spice"} -spice port=5901,addr=127.0.0.1,disable-ticketing=on,image-compression=off,seamless-migration=on -display egl-headless,rendernode=/dev/dri/renderD128 -device {"driver":"virtio-vga-gl","id":"video0","max_outputs":1,"bus":"pcie.0","addr":"0x1"} -device {"driver":"ich9-intel-hda","id":"sound0","bus":"pcie.0","addr":"0x1b"} -device {"driver":"hda-duplex","id":"sound0-codec0","bus":"sound0.0","cad":0,"audiodev":"audio1"} -global ICH9-LPC.noreboot=off -watchdog-action reset -chardev spicevmc,id=charredir0,name=usbredir -device {"driver":"usb-redir","chardev":"charredir0","id":"redir0","bus":"usb.0","port":"2"} -chardev spicevmc,id=charredir1,name=usbredir -device {"driver":"usb-redir","chardev":"charredir1","id":"redir1","bus":"usb.0","port":"3"} -device {"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.5","addr":"0x0"} -object {"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"} -device {"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.6","addr":"0x0"} -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

Debian Trixie:
/usr/bin/qemu-system-x86_64 -name guest=vm1,debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/home/larryboy/.config/libvirt/qemu/lib/domain-1-vm1/master-key.aes"} -machine pc-i440fx-9.2,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram,hpet=off,acpi=on -accel kvm -cpu Denverton,vmx=on,fma=on,pdcm=on,pcid=on,avx=on,f16c=on,hypervisor=on,ss=on,tsc-adjust=on,bmi1=on,avx2=on,bmi2=on,invpcid=on,avx512f=on,avx512dq=on,adx=on,avx512ifma=on,clwb=on,avx512cd=on,avx512bw=on,avx512vl=on,avx512vbmi=on,umip=on,pku=on,avx512vbmi2=on,gfni=on,vaes=on,vpclmulqdq=on,avx512vnni=on,avx512bitalg=on,avx512-vpopcntdq=on,rdpid=on,movdiri=on,movdir64b=on,fsrm=on,avx512-vp2intersect=on,md-clear=on,stibp=on,flush-l1d=on,xsaves=on,abm=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,ibrs-all=on,mds-no=on,pschange-mc-no=on,fbsdp-no=on,gds-no=on,rfds-no=on,vmx-activity-wait-sipi=on,vmx-xsaves=on,vmx-tsc-scaling=on,vmx-invvpid=on,mpx=off -m size=1048576k -object {"qom-type":"memory-backend-ram","id":"pc.ram","size":1073741824} -overcommit mem-lock=off -smp 1,sockets=1,cores=1,threads=1 -uuid aa332a62-1b7f-4a3c-b2c5-908e5e339b72 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=27,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=on,strict=on -device {"driver":"ich9-usb-ehci1","id":"usb","bus":"pci.0","addr":"0x5.0x7"} -device {"driver":"ich9-usb-uhci1","masterbus":"usb.0","firstport":0,"bus":"pci.0","multifunction":true,"addr":"0x5"} -device {"driver":"ich9-usb-uhci2","masterbus":"usb.0","firstport":2,"bus":"pci.0","addr":"0x5.0x1"} -device {"driver":"ich9-usb-uhci3","masterbus":"usb.0","firstport":4,"bus":"pci.0","addr":"0x5.0x2"} -device {"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.0","addr":"0x6"} -blockdev {"driver":"file","filename":"/home/larryboy/.local/share/libvirt/images/vm1.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null} -device {"driver":"ide-hd","bus":"ide.0","unit":0,"drive":"libvirt-2-format","id":"ide0-0-0","bootindex":2} -blockdev {"driver":"file","filename":"/home/larryboy/.local/share/libvirt/images/dsl-2024.rc7.iso","node-name":"libvirt-1-storage","read-only":true} -device {"driver":"ide-cd","bus":"ide.0","unit":1,"drive":"libvirt-1-storage","id":"ide0-0-1","bootindex":1} -netdev {"type":"stream","addr":{"type":"unix","path":"/run/user/1000/libvirt/qemu/run/passt/1-vm1-net0.socket"},"server":false,"reconnect-ms":5000,"id":"hostnet0"} -device {"driver":"rtl8139","netdev":"hostnet0","id":"net0","mac":"52:54:00:a0:e1:7c","bus":"pci.0","addr":"0x3"} -chardev pty,id=charserial0 -device {"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0} -chardev spicevmc,id=charchannel0,name=vdagent -device {"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"com.redhat.spice.0"} -device {"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"} -audiodev {"id":"audio1","driver":"spice"} -spice port=5900,addr=127.0.0.1,disable-ticketing=on,seamless-migration=on -display egl-headless,rendernode=/dev/dri/renderD128 -device {"driver":"virtio-vga-gl","id":"video0","max_outputs":1,"bus":"pci.0","addr":"0x2"} -device {"driver":"AC97","id":"sound0","audiodev":"audio1","bus":"pci.0","addr":"0x4"} -chardev spicevmc,id=charredir0,name=usbredir -device {"driver":"usb-redir","chardev":"charredir0","id":"redir0","bus":"usb.0","port":"2"} -chardev spicevmc,id=charredir1,name=usbredir -device {"driver":"usb-redir","chardev":"charredir1","id":"redir1","bus":"usb.0","port":"3"} -device {"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.0","addr":"0x7"} -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

> > Again, I will be making a report to the Debian maintainers, should they wish to chime in regarding Apparmor configs or the DNS resolution issue.
> 
> 
> Please file a separate issue, in case. This one would be for
> passt.
> 

I think I no longer have to, since I have the Debian maintainer right here. (:

> > Thank you once again for this awesome tool.
> 
> 
> And thanks again for trying it out and reporting issues!
> 
I'm happy to be of some assistance (:
> --
> Stefano

Re: Apparmor (and other) Issues

[Stefano Brivio]

On Wed, 29 Jan 2025 18:10:36 +0000
Prafulla Giri <prafulla.giri@protonmail.com> wrote:

> Hello,
> 
> On Wednesday, January 29th, 2025 at 3:26 PM, Stefano Brivio <sbrivio@redhat.com> wrote:
> 
> > Hi,
> > 
> > On Wed, 29 Jan 2025 09:14:12 +0000
> > Prafulla Giri prafulla.giri@protonmail.com wrote:
> >   
> > > Esteemed maintainer,
> > > 
> > > First and foremost, thank you very much for your hard work: passt is awesome and allows one to run more useful user-space VM-s.
> > > 
> > > I have encountered 2 particular issues with the usage of passt with Debian, and wanted to bring them to your attention as I think you are probably the best person to deal with this. I do plan on sending a report to the Debian team afterwards.
> > > 
> > > For reference, I tested these on Debian Testing Daily Image dated 28 January 2025, with updates, and the version of passt available with it is passt 0.0~git20250121.4f2c8e7-1
> > > 
> > > - Passt's default Apparmor config needs to allow writes to $XDG_RUNTIME_DIR (which is at /run/user/$UID). Currently it doesn't. Virt-manager, at least, tries to create the necessary sockets in the directory but apparmor prevents that from happening (and the error message Virt-Manager gives isn't helpful either: the first time around I falsely believed it was a segfault or similar issue). I managed to get passt working past this flaw (pun intended) by manually disabling apparmor for the binary. Passt works just fine in Fedora 41 as it doesn't use Apparmor but uses SELinux, and thus the configs don't affect it.  
> > 
> > 
> > Thanks for reporting this! I'm the maintainer of the Debian package, by
> > the way. Cc'ing Andrea, who is a maintainer of the libvirt package for
> > Debian and surely more knowledgeable about this.
> 
> I'm glad to have bumped into you. Because of the email domain, I thought you weren't the Debian maintainer. Silly me.

:)

> > Note that virt-manager uses passt through libvirt (I think that's only
> > possibility) and this should actually be allowed in libvirt's AppArmor
> > policy, in the sub-profile for passt:
> > 
> > https://gitlab.com/libvirt/libvirt/-/blob/0264a7704ada52f686cafe8f6402d5b60f9f0fc4/src/security/apparmor/libvirt-qemu.in#L204
> > 
> > the rationale is that passt itself doesn't know which directory libvirt
> > will pick for its socket and PID files, so libvirt's policy has to
> > specify that.
> > 
> > So I think you should file an issue for the libvirt package in this
> > case, unless Andrea has some pointers.  
> 
> I will wait for the maintainers input on this one.

One thing that might help meanwhile is if you have a look at
/var/log/audit/audit.log after the failure occurs. Look for 'passt'
there. There should be a message logging a denied access to some
file: what does it say?

> > > - This second issue is perhaps a bit more Debian-specific, but I am going to mention it so that you might drop some hints for the Debian maintainers to debug this: Once Apparmor is disabled and a VM is configured to work with passt, DNS resolution doesn't work in the VM (IP Addresses work just fine) i.e. ping fsf.org doesn't work but `ping 209.51.188.174` does. The hypervisor details follow:
> > > $ virsh version # on Debian Testing a.k.a. 'Trixie'
> > > Compiled against library: libvirt 11.0.0
> > > Using library: libvirt 11.0.0
> > > Using API: QEMU 11.0.0Running hypervisor: QEMU 9.2.0
> > > This, again, isn't an issue with Fedora 41, where everything just works. The hypervisor details for Fedora 41 are:
> > > $ virsh version # on Fedora 41
> > > Compiled against library: libvirt 10.6.0
> > > Using library: libvirt 10.6.0
> > > Using API: QEMU 10.6.0
> > > Running hypervisor: QEMU 9.1.2  
> > 
> > 
> > Oops. Can you share the command line of passt as run by libvirt
> > (say, 'ps aux|grep passt') for this case? passt has some basic
> > DNS forwarding capabilities, which are configured depending on
> > the host's resolver configuration.
> >   
> 
> Certainly! I'm sorry I didn't do this earlier. I'd checked on this: there is no difference between the command that runs passt on Fedora 41 or Debian Trixie.
> 
> This is the command on Fedora 41:
> passt --one-off --socket /run/user/1000/libvirt/qemu/run/passt/4-dragora-net0.socket --pid /run/user/1000/libvirt/qemu/run/passt/4-dragora-net0-passt.pid
> 
> and this is the command on Debian Trixie:
> passt --one-off --socket /run/user/1000/libvirt/qemu/run/passt/1-vm1-net0.socket --pid /run/user/1000/libvirt/qemu/run/passt/1-vm1-net0-passt.pid

Okay, nothing unexpected so far. Could you also please compare the
output of 'passt -f -d' between the two cases? Just terminate it with
^C once you have the output.

How are resolvers configured on the two hosts? What does
/etc/resolv.conf say?

If nothing is visible from there, next check: 'virsh edit vm1' on
Debian and add a log file in the XML, that is, replace this line:

  <backend type='passt'/>

with:

  <backend type='passt' logFile='/tmp/passt.log'/>

and then share the log.

-- 
Stefano

Re: Apparmor (and other) Issues

[Prafulla Giri]

On Thursday, January 30th, 2025 at 12:33 AM, Stefano Brivio <sbrivio@redhat.com> wrote:

> On Wed, 29 Jan 2025 18:10:36 +0000
> Prafulla Giri prafulla.giri@protonmail.com wrote:
> 
> > Hello,
> > 
> > On Wednesday, January 29th, 2025 at 3:26 PM, Stefano Brivio sbrivio@redhat.com wrote:
> > 
> > > Hi,
> > > 
> > > On Wed, 29 Jan 2025 09:14:12 +0000
> > > Prafulla Giri prafulla.giri@protonmail.com wrote:
> > > 
> > > > Esteemed maintainer,
> > > > 
> > > > First and foremost, thank you very much for your hard work: passt is awesome and allows one to run more useful user-space VM-s.
> > > > 
> > > > I have encountered 2 particular issues with the usage of passt with Debian, and wanted to bring them to your attention as I think you are probably the best person to deal with this. I do plan on sending a report to the Debian team afterwards.
> > > > 
> > > > For reference, I tested these on Debian Testing Daily Image dated 28 January 2025, with updates, and the version of passt available with it is passt 0.0~git20250121.4f2c8e7-1
> > > > 
> > > > - Passt's default Apparmor config needs to allow writes to $XDG_RUNTIME_DIR (which is at /run/user/$UID). Currently it doesn't. Virt-manager, at least, tries to create the necessary sockets in the directory but apparmor prevents that from happening (and the error message Virt-Manager gives isn't helpful either: the first time around I falsely believed it was a segfault or similar issue). I managed to get passt working past this flaw (pun intended) by manually disabling apparmor for the binary. Passt works just fine in Fedora 41 as it doesn't use Apparmor but uses SELinux, and thus the configs don't affect it.
> > > 
> > > Thanks for reporting this! I'm the maintainer of the Debian package, by
> > > the way. Cc'ing Andrea, who is a maintainer of the libvirt package for
> > > Debian and surely more knowledgeable about this.
> > 
> > I'm glad to have bumped into you. Because of the email domain, I thought you weren't the Debian maintainer. Silly me.
> 
> 
> :)
> 
> > > Note that virt-manager uses passt through libvirt (I think that's only
> > > possibility) and this should actually be allowed in libvirt's AppArmor
> > > policy, in the sub-profile for passt:
> > > 
> > > https://gitlab.com/libvirt/libvirt/-/blob/0264a7704ada52f686cafe8f6402d5b60f9f0fc4/src/security/apparmor/libvirt-qemu.in#L204
> > > 
> > > the rationale is that passt itself doesn't know which directory libvirt
> > > will pick for its socket and PID files, so libvirt's policy has to
> > > specify that.
> > > 
> > > So I think you should file an issue for the libvirt package in this
> > > case, unless Andrea has some pointers.
> > 
> > I will wait for the maintainers input on this one.
> 
> 
> One thing that might help meanwhile is if you have a look at
> /var/log/audit/audit.log after the failure occurs. Look for 'passt'
> there. There should be a message logging a denied access to some
> file: what does it say?
> 
I didn't have auditd installed on Debian and installed it, and running everything with the default auditd config (with my Apparmor disabled for passt, as mentioned previously) does not result in anything. Do I have to configure auditd manually? Any pointers on that, please?

On Fedora 41, which seems to have auditd preconfigured, there aren't any significant reports about passt.

> > > > - This second issue is perhaps a bit more Debian-specific, but I am going to mention it so that you might drop some hints for the Debian maintainers to debug this: Once Apparmor is disabled and a VM is configured to work with passt, DNS resolution doesn't work in the VM (IP Addresses work just fine) i.e. ping fsf.org doesn't work but `ping 209.51.188.174` does. The hypervisor details follow:
> > > > $ virsh version # on Debian Testing a.k.a. 'Trixie'
> > > > Compiled against library: libvirt 11.0.0
> > > > Using library: libvirt 11.0.0
> > > > Using API: QEMU 11.0.0Running hypervisor: QEMU 9.2.0
> > > > This, again, isn't an issue with Fedora 41, where everything just works. The hypervisor details for Fedora 41 are:
> > > > $ virsh version # on Fedora 41
> > > > Compiled against library: libvirt 10.6.0
> > > > Using library: libvirt 10.6.0
> > > > Using API: QEMU 10.6.0
> > > > Running hypervisor: QEMU 9.1.2
> > > 
> > > Oops. Can you share the command line of passt as run by libvirt
> > > (say, 'ps aux|grep passt') for this case? passt has some basic
> > > DNS forwarding capabilities, which are configured depending on
> > > the host's resolver configuration.
> > 
> > Certainly! I'm sorry I didn't do this earlier. I'd checked on this: there is no difference between the command that runs passt on Fedora 41 or Debian Trixie.
> > 
> > This is the command on Fedora 41:
> > passt --one-off --socket /run/user/1000/libvirt/qemu/run/passt/4-dragora-net0.socket --pid /run/user/1000/libvirt/qemu/run/passt/4-dragora-net0-passt.pid
> > 
> > and this is the command on Debian Trixie:
> > passt --one-off --socket /run/user/1000/libvirt/qemu/run/passt/1-vm1-net0.socket --pid /run/user/1000/libvirt/qemu/run/passt/1-vm1-net0-passt.pid
> 
> 
> Okay, nothing unexpected so far. Could you also please compare the
> output of 'passt -f -d' between the two cases? Just terminate it with
> ^C once you have the output.
Here are the outputs:
$ passt -f -d # on Debian Testing/Trixie
0.0016: No interfaces with usable IPv6 routes
0.0017: Failed to detect external interface for IPv6
0.0028: UNIX domain socket bound at /tmp/passt_1.socket
0.0029: Template interface: enp1s0 (IPv4)
0.0029: MAC:
0.0029:     host: 9a:55:9a:55:9a:55
0.0029:     NAT to host 127.0.0.1: 192.168.100.1
0.0029: DHCP:
0.0029:     assign: 192.168.100.157
0.0029:     mask: 255.255.255.0
0.0029:     router: 192.168.100.1
0.0029: DNS:
0.0029:     192.168.100.1
0.0029: DNS search list:
0.0029:     .
0.0056: 
You can now start qemu (>= 7.2, with commit 13c6be96618c):
0.0056:     kvm ... -device virtio-net-pci,netdev=s -netdev stream,id=s,server=off,addr.type=unix,addr.path=/tmp/passt_1.socket
0.0057: or qrap, for earlier qemu versions:
0.0057:     ./qrap 5 kvm ... -net socket,fd=5 -net nic,model=virtio
0.0067: SO_PEEK_OFF supported
0.0067: TCP_INFO tcpi_snd_wnd field  supported
0.0067: TCP_INFO tcpi_bytes_acked field  supported
0.0067: TCP_INFO tcpi_min_rtt field  supported

$ passt -f -d
0.0022: UNIX domain socket bound at /tmp/passt_1.socket
0.0022: Template interface: wlp0s20f3 (IPv4)
0.0022: MAC:
0.0022:     host: 9a:55:9a:55:9a:55
0.0022:     NAT to host 127.0.0.1: 192.168.100.1
0.0023: DHCP:
0.0023:     assign: 192.168.100.157
0.0023:     mask: 255.255.255.0
0.0023:     router: 192.168.100.1
0.0023: DNS:
0.0023:     192.168.100.1
0.0023: DNS search list:
0.0023:     .
0.0047: 
You can now start qemu (>= 7.2, with commit 13c6be96618c):
0.0047:     kvm ... -device virtio-net-pci,netdev=s -netdev stream,id=s,server=off,addr.type=unix,addr.path=/tmp/passt_1.socket
0.0047: or qrap, for earlier qemu versions:
0.0047:     ./qrap 5 kvm ... -net socket,fd=5 -net nic,model=virtio
0.0055: SO_PEEK_OFF supported
0.0055: TCP_INFO tcpi_snd_wnd field  supported
0.0055: TCP_INFO tcpi_bytes_acked field  supported
0.0055: TCP_INFO tcpi_min_rtt field  supported

> 
> How are resolvers configured on the two hosts? What does
> /etc/resolv.conf say?
$ cat /etc/resolv.conf # On Fedora 41
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
[...]
nameserver 127.0.0.53
options edns0 trust-ad
search .
$ cat /etc/resolv.conf # On Debian Trixie
# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
[...]
nameserver 192.168.100.1
search .
$ cat /etc/resolv.conf # On a Debian 11 OS
# Generated by NetworkManager
nameserver 192.168.100.1

Also the output of `resolvectl status` for good measure:
# On Fedora 41
Global
         Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (wlp0s20f3)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.100.1
       DNS Servers: 192.168.100.1

# On Debian Trixie
Global
         Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: uplink

Link 2 (enp1s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
       DNS Servers: 192.168.100.1
     Default Route: yes

> 
> If nothing is visible from there, next check: 'virsh edit vm1' on
> Debian and add a log file in the XML, that is, replace this line:
> 
> <backend type='passt'/>
> 
> 
> with:
> 
> <backend type='passt' logFile='/tmp/passt.log'/>
> 
> 
> and then share the log.
> 
The log from Debian Trixie host for VM1:
passt 0.0~git20250121.4f2c8e7-1: /usr/bin/passt.avx2 (6428)
0.0017: info:    No interfaces with usable IPv6 routes
0.0029: info:    UNIX domain socket bound at /run/user/1000/libvirt/qemu/run/passt/2-vm1-net0.socket
0.0030: info:    Template interface: enp1s0 (IPv4)
0.0030: info:    MAC:
0.0030: info:        host: 9a:55:9a:55:9a:55
0.0030: info:        NAT to host 127.0.0.1: 192.168.100.1
0.0030: info:    DHCP:
0.0031: info:        assign: 192.168.100.157
0.0031: info:        mask: 255.255.255.0
0.0031: info:        router: 192.168.100.1
0.0031: info:    DNS:
0.0031: info:        192.168.100.1
0.0031: info:    DNS search list:
0.0031: info:        .
0.0066: info:    
You can now start qemu (>= 7.2, with commit 13c6be96618c):
0.0066: info:        kvm ... -device virtio-net-pci,netdev=s -netdev stream,id=s,server=off,addr.type=unix,addr.path=/run/user/1000/libvirt/qemu/run/passt/2-vm1-net0.socket
0.0066: info:    or qrap, for earlier qemu versions:
0.0066: info:        ./qrap 5 kvm ... -net socket,fd=5 -net nic,model=virtio
0.0617: info:    accepted connection from PID 0
38.6257: info:    DHCP: offer to discover
38.6257: info:        from 52:54:00:a0:e1:7c
38.6471: info:    DHCP: ack to request
38.6471: info:        from 52:54:00:a0:e1:7c
451.4989: info:    Client connection closed, exiting

The log from Fedora 41:
passt 0^20250121.g4f2c8e7-2.fc41.x86_64: /usr/bin/passt.avx2 (3138)
0.0017: info:    UNIX domain socket bound at /run/user/1000/libvirt/qemu/run/passt/3-debian-trixie-net0.socket
0.0018: info:    Template interface: wlp0s20f3 (IPv4)
0.0018: info:    MAC:
0.0018: info:        host: 9a:55:9a:55:9a:55
0.0018: info:        NAT to host 127.0.0.1: 192.168.100.1
0.0018: info:    DHCP:
0.0018: info:        assign: 192.168.100.157
0.0018: info:        mask: 255.255.255.0
0.0018: info:        router: 192.168.100.1
0.0018: info:    DNS:
0.0018: info:        192.168.100.1
0.0018: info:    DNS search list:
0.0018: info:        .
0.0043: info:    
You can now start qemu (>= 7.2, with commit 13c6be96618c):
0.0043: info:        kvm ... -device virtio-net-pci,netdev=s -netdev stream,id=s,server=off,addr.type=unix,addr.path=/run/user/1000/libvirt/qemu/run/passt/3-debian-trixie-net0.socket
0.0043: info:    or qrap, for earlier qemu versions:
0.0043: info:        ./qrap 5 kvm ... -net socket,fd=5 -net nic,model=virtio
0.0591: info:    accepted connection from PID 0
10.7894: info:    DHCP: ack to discover (Rapid Commit)
10.7894: info:        from 52:54:00:8f:e7:c3
99.6704: info:    Client connection closed, exiting


> --
> Stefano

Re: Apparmor (and other) Issues

[Stefano Brivio]

On Thu, 30 Jan 2025 10:05:14 +0000
Prafulla Giri <prafulla.giri@protonmail.com> wrote:

> I didn't have auditd installed on Debian and installed it, and
> running everything with the default auditd config (with my Apparmor
> disabled for passt, as mentioned previously) does not result in
> anything. Do I have to configure auditd manually? Any pointers on
> that, please?

No, the default configuration is just fine. But you should re-enable
AppArmor for passt so that we can see meaningful messages in the logs.

> $ passt -f -d # on Debian Testing/Trixie
> 0.0016: No interfaces with usable IPv6 routes
> 0.0017: Failed to detect external interface for IPv6
> 0.0028: UNIX domain socket bound at /tmp/passt_1.socket
> 0.0029: Template interface: enp1s0 (IPv4)
> 0.0029: MAC:
> 0.0029:     host: 9a:55:9a:55:9a:55
> 0.0029:     NAT to host 127.0.0.1: 192.168.100.1
> 0.0029: DHCP:
> 0.0029:     assign: 192.168.100.157
> 0.0029:     mask: 255.255.255.0
> 0.0029:     router: 192.168.100.1
> 0.0029: DNS:
> 0.0029:     192.168.100.1

So, judging from this configuration, it looks like we advertise to
the guest (via DHCP) 192.168.100.1 as resolver (copied from the host),
and when we receive packets from the guest for 192.168.100.1, we'll
re-map them to the host.

Nothing strange so far, systemd-resolved is running on the host, it
should get our queries and reply to them.

> $ cat /etc/resolv.conf # On Debian Trixie
> # This is /run/systemd/resolve/resolv.conf managed by
> man:systemd-resolved(8). [...]
> nameserver 192.168.100.1
> search .
> $ cat /etc/resolv.conf # On a Debian 11 OS
> # Generated by NetworkManager
> nameserver 192.168.100.1
> 
> Also the output of `resolvectl status` for good measure:
> # On Fedora 41
> Global
>          Protocols: LLMNR=resolve -mDNS -DNSOverTLS
> DNSSEC=no/unsupported resolv.conf mode: stub
> 
> Link 2 (wlp0s20f3)
>     Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
>          Protocols: +DefaultRoute LLMNR=resolve -mDNS -DNSOverTLS
> DNSSEC=no/unsupported Current DNS Server: 192.168.100.1
>        DNS Servers: 192.168.100.1
> 
> # On Debian Trixie
> Global
>          Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
>   resolv.conf mode: uplink
> 
> Link 2 (enp1s0)
>     Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
>          Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS
> DNSSEC=no/unsupported DNS Servers: 192.168.100.1
>      Default Route: yes

Everything as expected here, I don't see any obvious reason why
systemd-resolved should discard our queries.

> The log from Debian Trixie host for VM1:
> passt 0.0~git20250121.4f2c8e7-1: /usr/bin/passt.avx2 (6428)
> 0.0017: info:    No interfaces with usable IPv6 routes
> 0.0029: info:    UNIX domain socket bound at
> /run/user/1000/libvirt/qemu/run/passt/2-vm1-net0.socket 0.0030: info:
>    Template interface: enp1s0 (IPv4) 0.0030: info:    MAC:
> 0.0030: info:        host: 9a:55:9a:55:9a:55
> 0.0030: info:        NAT to host 127.0.0.1: 192.168.100.1
> 0.0030: info:    DHCP:
> 0.0031: info:        assign: 192.168.100.157
> 0.0031: info:        mask: 255.255.255.0
> 0.0031: info:        router: 192.168.100.1
> 0.0031: info:    DNS:
> 0.0031: info:        192.168.100.1
> 0.0031: info:    DNS search list:
> 0.0031: info:        .
> 0.0066: info:    
> You can now start qemu (>= 7.2, with commit 13c6be96618c):
> 0.0066: info:        kvm ... -device virtio-net-pci,netdev=s -netdev
> stream,id=s,server=off,addr.type=unix,addr.path=/run/user/1000/libvirt/qemu/run/passt/2-vm1-net0.socket
> 0.0066: info:    or qrap, for earlier qemu versions: 0.0066: info:
>     ./qrap 5 kvm ... -net socket,fd=5 -net nic,model=virtio 0.0617:
> info:    accepted connection from PID 0 38.6257: info:    DHCP: offer
> to discover 38.6257: info:        from 52:54:00:a0:e1:7c
> 38.6471: info:    DHCP: ack to request
> 38.6471: info:        from 52:54:00:a0:e1:7c
> 451.4989: info:    Client connection closed, exiting

Unfortunately libvirt doesn't let us enable more verbose logging. I
hoped to see DNS queries there, but without --debug given to passt,
that won't work.

Another idea: pasta(1) does the same job as passt(1) (it's the same
code and same binary) and it's intended for containers, but it has a
stand-alone mode that can probably help us to debug this, because it's
a network namespace that will look like your guest, and it can also
take packet captures.

What happens if you run:

  pasta --config-net --trace --pcap /tmp/dns.pcap -- nslookup fsf.org

?

If the query fails (that's what I would expect), can you please attach
the output and the resulting packet capture?

Note that the packet capture might have sensitive data, please have a
look at it with e.g. Wireshark/tshark before sharing it.

By the way, if you run this without specifying any command, for example:

  pasta --config-net --pcap /tmp/dns.pcap

you will get a shell where you can play around in a separate network
namespace and with a network stack that looks pretty much the same as
passt for your guest.

-- 
Stefano

Re: Apparmor (and other) Issues

[Prafulla Giri]

On Sunday, February 2nd, 2025 at 8:06 PM, Prafulla Giri <prafulla.giri@protonmail.com> wrote:

> I wonder if these 2 issues are mutually exclusive: If I disable apparmor and let passt run, it doesn't get DNS configs right; however, if I don't make any changes to apparmor configs, while passt won't be able to create the socket at /run/*, it will get DNS configs right (?)

Nope. I just disabled app-armor and tested
$ pasta --config-net
and could ping fsf.org but could not nslookup fsf.org

Re: Apparmor (and other) Issues

[Stefano Brivio]

On Sun, 02 Feb 2025 14:21:49 +0000
Prafulla Giri <prafulla.giri@protonmail.com> wrote:

> On Saturday, February 1st, 2025 at 2:05 AM, Stefano Brivio <sbrivio@redhat.com> wrote:
> 
> > On Thu, 30 Jan 2025 10:05:14 +0000
> > Prafulla Giri prafulla.giri@protonmail.com wrote:
> >   
> > > I didn't have auditd installed on Debian and installed it, and
> > > running everything with the default auditd config (with my Apparmor
> > > disabled for passt, as mentioned previously) does not result in
> > > anything. Do I have to configure auditd manually? Any pointers on
> > > that, please?  
> > 
> > 
> > No, the default configuration is just fine. But you should re-enable
> > AppArmor for passt so that we can see meaningful messages in the logs.
> >   
> The following is the output of grep-ping 'passt' after re-enforcing the apparmor config and trying to start a VM:
> $ grep passt /var/log/audit/audit.log # Debian Trixie
> type=AVC msg=audit(1738501259.829:124): apparmor="STATUS" operation="profile_load" profile="unconfined" name="passt" pid=1935 comm="apparmor_parser"
> type=AVC msg=audit(1738501309.118:135): apparmor="DENIED" operation="file_mmap" class="file" profile="passt" name="/usr/bin/passt" pid=2030 comm="passt" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0\x1dFSUID="larryboy" OUID="root"
> type=SYSCALL msg=audit(1738501309.118:135): arch=c000003e syscall=59 success=no exit=-13 a0=7faf24035fc0 a1=7faf24035210 a2=7ffc063280d0 a3=0 items=0 ppid=1964 pid=2030 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=1 comm="passt" exe="/usr/bin/passt" subj=passt key=(null)\x1dARCH=x86_64 SYSCALL=execve AUID="larryboy" UID="larryboy" GID="larryboy" EUID="larryboy" SUID="larryboy" FSUID="larryboy" EGID="larryboy" SGID="larryboy" FSGID="larryboy"
> type=ANOM_ABEND msg=audit(1738501309.118:136): auid=1000 uid=1000 gid=1000 ses=1 subj=passt pid=2030 comm="passt" exe="/usr/bin/passt" sig=11 res=1\x1dAUID="larryboy" UID="larryboy" GID="larryboy"

System call number 59 (this seems to be x86_64) is "execve":

  $ ausyscall 59
  execve

and this seems to be libvirt failing to execute passt itself, but:

  https://gitlab.com/libvirt/libvirt/-/blob/0264a7704ada52f686cafe8f6402d5b60f9f0fc4/src/security/apparmor/libvirt-qemu.in#L195

Do you see the libvirtd profile loaded if you run 'aa-status'?

Do you have this line:

  /usr/bin/passt Cx -> passt,

in /etc/apparmor.d/abstractions/libvirt-qemu? I wonder if something
bad happened during installation.

Can you perhaps grep a bit before and after those messages (say, grep
-A5 -B5) to see if we spot something else related to libvirt?

> type=AVC msg=audit(1738501923.727:148): apparmor="DENIED" operation="file_mmap" class="file" profile="passt" name="/usr/bin/passt" pid=2088 comm="passt" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0\x1dFSUID="larryboy" OUID="root"
> type=SYSCALL msg=audit(1738501923.727:148): arch=c000003e syscall=59 success=no exit=-13 a0=7ff564035d40 a1=7ff564039d00 a2=7fffe9aa1de0 a3=0 items=0 ppid=2060 pid=2088 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=1 comm="passt" exe="/usr/bin/passt" subj=passt key=(null)\x1dARCH=x86_64 SYSCALL=execve AUID="larryboy" UID="larryboy" GID="larryboy" EUID="larryboy" SUID="larryboy" FSUID="larryboy" EGID="larryboy" SGID="larryboy" FSGID="larryboy"
> type=ANOM_ABEND msg=audit(1738501923.727:149): auid=1000 uid=1000 gid=1000 ses=1 subj=passt pid=2088 comm="passt" exe="/usr/bin/passt" sig=11 res=1\x1dAUID="larryboy" UID="larryboy" GID="larryboy"
> type=AVC msg=audit(1738502301.651:174): apparmor="DENIED" operation="file_mmap" class="file" profile="passt" name="/usr/bin/passt" pid=2145 comm="passt" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0\x1dFSUID="larryboy" OUID="root"
> type=SYSCALL msg=audit(1738502301.651:174): arch=c000003e syscall=59 success=no exit=-13 a0=7fe208034ce0 a1=7fe208034350 a2=7fffd2e60120 a3=0 items=0 ppid=2117 pid=2145 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=1 comm="passt" exe="/usr/bin/passt" subj=passt key=(null)\x1dARCH=x86_64 SYSCALL=execve AUID="larryboy" UID="larryboy" GID="larryboy" EUID="larryboy" SUID="larryboy" FSUID="larryboy" EGID="larryboy" SGID="larryboy" FSGID="larryboy"
> type=ANOM_ABEND msg=audit(1738502301.651:175): auid=1000 uid=1000 gid=1000 ses=1 subj=passt pid=2145 comm="passt" exe="/usr/bin/passt" sig=11 res=1\x1dAUID="larryboy" UID="larryboy" GID="larryboy"
> 
> > > $ passt -f -d # on Debian Testing/Trixie
> > > 0.0016: No interfaces with usable IPv6 routes
> > > 0.0017: Failed to detect external interface for IPv6
> > > 0.0028: UNIX domain socket bound at /tmp/passt_1.socket
> > > 0.0029: Template interface: enp1s0 (IPv4)
> > > 0.0029: MAC:
> > > 0.0029: host: 9a:55:9a:55:9a:55
> > > 0.0029: NAT to host 127.0.0.1: 192.168.100.1
> > > 0.0029: DHCP:
> > > 0.0029: assign: 192.168.100.157
> > > 0.0029: mask: 255.255.255.0
> > > 0.0029: router: 192.168.100.1
> > > 0.0029: DNS:
> > > 0.0029: 192.168.100.1  
> > 
> > 
> > So, judging from this configuration, it looks like we advertise to
> > the guest (via DHCP) 192.168.100.1 as resolver (copied from the host),
> > and when we receive packets from the guest for 192.168.100.1, we'll
> > re-map them to the host.
> > 
> > Nothing strange so far, systemd-resolved is running on the host, it
> > should get our queries and reply to them.
> >   
> > > $ cat /etc/resolv.conf # On Debian Trixie
> > > # This is /run/systemd/resolve/resolv.conf managed by
> > > man:systemd-resolved(8). [...]
> > > nameserver 192.168.100.1
> > > search .
> > > $ cat /etc/resolv.conf # On a Debian 11 OS
> > > # Generated by NetworkManager
> > > nameserver 192.168.100.1
> > > 
> > > Also the output of `resolvectl status` for good measure:
> > > # On Fedora 41
> > > Global
> > > Protocols: LLMNR=resolve -mDNS -DNSOverTLS
> > > DNSSEC=no/unsupported resolv.conf mode: stub
> > > 
> > > Link 2 (wlp0s20f3)
> > > Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
> > > Protocols: +DefaultRoute LLMNR=resolve -mDNS -DNSOverTLS
> > > DNSSEC=no/unsupported Current DNS Server: 192.168.100.1
> > > DNS Servers: 192.168.100.1
> > > 
> > > # On Debian Trixie
> > > Global
> > > Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
> > > resolv.conf mode: uplink
> > > 
> > > Link 2 (enp1s0)
> > > Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
> > > Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS
> > > DNSSEC=no/unsupported DNS Servers: 192.168.100.1
> > > Default Route: yes  
> > 
> > 
> > Everything as expected here, I don't see any obvious reason why
> > systemd-resolved should discard our queries.
> >   
> > > The log from Debian Trixie host for VM1:
> > > passt 0.0~git20250121.4f2c8e7-1: /usr/bin/passt.avx2 (6428)
> > > 0.0017: info: No interfaces with usable IPv6 routes
> > > 0.0029: info: UNIX domain socket bound at
> > > /run/user/1000/libvirt/qemu/run/passt/2-vm1-net0.socket 0.0030: info:
> > > Template interface: enp1s0 (IPv4) 0.0030: info: MAC:
> > > 0.0030: info: host: 9a:55:9a:55:9a:55
> > > 0.0030: info: NAT to host 127.0.0.1: 192.168.100.1
> > > 0.0030: info: DHCP:
> > > 0.0031: info: assign: 192.168.100.157
> > > 0.0031: info: mask: 255.255.255.0
> > > 0.0031: info: router: 192.168.100.1
> > > 0.0031: info: DNS:
> > > 0.0031: info: 192.168.100.1
> > > 0.0031: info: DNS search list:
> > > 0.0031: info: .
> > > 0.0066: info:
> > > You can now start qemu (>= 7.2, with commit 13c6be96618c):
> > > 0.0066: info: kvm ... -device virtio-net-pci,netdev=s -netdev
> > > stream,id=s,server=off,addr.type=unix,addr.path=/run/user/1000/libvirt/qemu/run/passt/2-vm1-net0.socket
> > > 0.0066: info: or qrap, for earlier qemu versions: 0.0066: info:
> > > ./qrap 5 kvm ... -net socket,fd=5 -net nic,model=virtio 0.0617:
> > > info: accepted connection from PID 0 38.6257: info: DHCP: offer
> > > to discover 38.6257: info: from 52:54:00:a0:e1:7c
> > > 38.6471: info: DHCP: ack to request
> > > 38.6471: info: from 52:54:00:a0:e1:7c
> > > 451.4989: info: Client connection closed, exiting  
> > 
> > 
> > Unfortunately libvirt doesn't let us enable more verbose logging. I
> > hoped to see DNS queries there, but without --debug given to passt,
> > that won't work.
> > 
> > Another idea: pasta(1) does the same job as passt(1) (it's the same
> > code and same binary) and it's intended for containers, but it has a
> > stand-alone mode that can probably help us to debug this, because it's
> > a network namespace that will look like your guest, and it can also
> > take packet captures.
> > 
> > What happens if you run:
> > 
> > pasta --config-net --trace --pcap /tmp/dns.pcap -- nslookup fsf.org
> > 
> > ?  
> This one errors out. dns.pcap is attached.
> $ pasta --config-net --trace --pcap /tmp/dns.pcap -- nslookup fsf.org # Debian Trixie/Testing
> 0.0015: No interfaces with usable IPv6 routes
> 0.0015: Failed to detect external interface for IPv6
> 0.0073: Template interface: enp1s0 (IPv4)
> 0.0073: Namespace interface: enp1s0
> 0.0074: MAC:
> 0.0074:     host: 9a:55:9a:55:9a:55
> 0.0074:     NAT to host 127.0.0.1: 192.168.100.1
> 0.0074: DHCP:
> 0.0074:     assign: 192.168.100.157
> 0.0074:     mask: 255.255.255.0
> 0.0075:     router: 192.168.100.1
> 0.0075: DNS:
> 0.0075:     192.168.100.1
> 0.0076: DNS search list:
> 0.0076:     .
> 0.0146: SO_PEEK_OFF supported
> 0.0146: TCP_INFO tcpi_snd_wnd field  supported
> 0.0146: TCP_INFO tcpi_bytes_acked field  supported
> 0.0146: TCP_INFO tcpi_min_rtt field  supported
> 0.0147: Saving packet capture to /tmp/dns.pcap
> 0.0197: pasta: epoll event on /dev/net/tun device 16 (events: 0x00000001)
> 0.0371: pasta: epoll event on /dev/net/tun device 16 (events: 0x00000001)
> 0.0372: pasta: epoll event on /dev/net/tun device 16 (events: 0x00000001)
> 0.0372: tap: protocol 17, 192.168.100.157:41892 -> 192.168.100.1:53 (1 packet)
> 0.0372: Flow 0 (NEW): FREE -> NEW
> 0.0372: Flow 0 (INI): NEW -> INI
> 0.0372: Flow 0 (INI): TAP [192.168.100.157]:41892 -> [192.168.100.1]:53 => ?
> 0.0372: Flow 0 (TGT): INI -> TGT
> 0.0373: Flow 0 (TGT): TAP [192.168.100.157]:41892 -> [192.168.100.1]:53 => HOST [0.0.0.0]:41892 -> [127.0.0.1]:53
> 0.0373: Flow 0 (UDP flow): TGT -> TYPED
> 0.0373: Flow 0 (UDP flow): TAP [192.168.100.157]:41892 -> [192.168.100.1]:53 => HOST [0.0.0.0]:41892 -> [127.0.0.1]:53
> 0.0373: Flow 0 (UDP flow): Side 0 hash table insert: bucket: 31049
> 0.0374: Flow 0 (UDP flow): TYPED -> ACTIVE
> 0.0374: Flow 0 (UDP flow): TAP [192.168.100.157]:41892 -> [192.168.100.1]:53 => HOST [0.0.0.0]:41892 -> [127.0.0.1]:53
> 0.0374: pasta: epoll event on UDP reply socket 95 (events: 0x00000008)
> 0.0374: ICMP error on UDP socket 95: Connection refused

Ouch. I just sent a patch for this, you can test it by checking out
passt locally:

  git clone git://passt.top/passt; cd passt

applying it (you might need to install 'b4'):

  b4 shazam
  https://archives.passt.top/passt-dev/20250203082210.2114348-1-sbrivio@redhat.com/

then:

  make

and ./pasta --config-net --trace --pcap /tmp/dns.pcap -- nslookup
fsf.org

You can also install it to /usr/local with 'make install', it's just a
couple of files and will uninstall cleanly with 'make uninstall' if
needed. Note that AppArmor profiles don't apply to binaries under
/usr/local/bin.

> [...]
>
> This is rather strange: from the shell I successfully managed to `ping fsf.org` and `ping gnu.org`.

That's because they were already cached on your host at this point (I
suppose by systemd-resolved) and ping didn't need to resolve anything,
just call getaddrinfo() and get the addresses back. There's no UDP at
all in the output below:

> I went back and ran
> $ pasta --config-net --trace --pcap /tmp/ping.pcap -- ping -c 5 fsf.org # and the following output followed (ping.pcap is also attached)
> 
> 0.0013: No interfaces with usable IPv6 routes
> 0.0013: Failed to detect external interface for IPv6
> 0.0091: Template interface: enp1s0 (IPv4)
> 0.0091: Namespace interface: enp1s0
> 0.0092: MAC:
> 0.0092:     host: 9a:55:9a:55:9a:55
> 0.0092:     NAT to host 127.0.0.1: 192.168.100.1
> 0.0092: DHCP:
> 0.0092:     assign: 192.168.100.157
> 0.0093:     mask: 255.255.255.0
> 0.0093:     router: 192.168.100.1
> 0.0093: DNS:
> 0.0093:     192.168.100.1
> 0.0093: DNS search list:
> 0.0094:     .
> 0.0161: SO_PEEK_OFF supported
> 0.0161: TCP_INFO tcpi_snd_wnd field  supported
> 0.0161: TCP_INFO tcpi_bytes_acked field  supported
> 0.0161: TCP_INFO tcpi_min_rtt field  supported
> 0.0162: Saving packet capture to /tmp/ping.pcap
> PING fsf.org (209.51.188.174) 56(84) bytes of data.
> 0.0181: pasta: epoll event on /dev/net/tun device 16 (events: 0x00000001)
> 0.0181: pasta: epoll event on /dev/net/tun device 16 (events: 0x00000001)
> 0.0182: tap: protocol 1, 192.168.100.157 -> 209.51.188.174 (1 packet)

-- 
Stefano

Re: Apparmor (and other) Issues

[Stefano Brivio]

On Tue, 04 Feb 2025 08:21:53 +0000
Prafulla Giri <prafulla.giri@protonmail.com> wrote:

> type=SERVICE_START msg=audit(1738501309.082:134): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=polkit comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
> type=AVC msg=audit(1738501309.118:135): apparmor="DENIED" operation="file_mmap" class="file" profile="passt" name="/usr/bin/passt" pid=2030 comm="passt" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0FSUID="larryboy" OUID="root"
> type=SYSCALL msg=audit(1738501309.118:135): arch=c000003e syscall=59 success=no exit=-13 a0=7faf24035fc0 a1=7faf24035210 a2=7ffc063280d0 a3=0 items=0 ppid=1964 pid=2030 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=1 comm="passt" exe="/usr/bin/passt" subj=passt key=(null)ARCH=x86_64 SYSCALL=execve AUID="larryboy" UID="larryboy" GID="larryboy" EUID="larryboy" SUID="larryboy" FSUID="larryboy" EGID="larryboy" SGID="larryboy" FSGID="larryboy"
> type=PROCTITLE msg=audit(1738501309.118:135): proctitle="(null)"
> type=ANOM_ABEND msg=audit(1738501309.118:136): auid=1000 uid=1000 gid=1000 ses=1 subj=passt pid=2030 comm="passt" exe="/usr/bin/passt" sig=11 res=1AUID="larryboy" UID="larryboy" GID="larryboy"

So, it looks like passt is running as its own profile. This shouldn't
happen because the libvirt profile has an own subprofile and we should
see that in "profile" on the type=AVC line but... I just reproduced
this! Clean Debian sid install, fresh install of libvirtd:

error: internal error: Child process (passt --one-off --socket /run/user/1000/libvirt/qemu/run/passt/1-alpine-net0.socket --pid /run/user/1000/libvirt/qemu/run/passt/1-alpine-net0-passt.pid --tcp-ports 40922:22) unexpected fatal signal 11

I'll keep you posted.

> > https://archives.passt.top/passt-dev/20250203082210.2114348-1-sbrivio@redhat.com/
> > 
> > then:
> > 
> > make
> > 
> > and ./pasta --config-net --trace --pcap /tmp/dns.pcap -- nslookup
> > fsf.org
> >   
> $ ./pasta --config-net --trace --pcap /tmp/dns.pcap -- nslookup fsf.org # On Debian Trixie, dns.pcap attached
> 0.0002: No interfaces with usable IPv6 routes
> 0.0002: Failed to detect external interface for IPv6
> 0.0035: Template interface: enp1s0 (IPv4)
> 0.0035: Namespace interface: enp1s0
> 0.0035: MAC:
> 0.0035:     host: 9a:55:9a:55:9a:55
> 0.0035:     NAT to host 127.0.0.1: 192.168.100.1
> 0.0036: DHCP:
> 0.0036:     assign: 192.168.100.157
> 0.0036:     mask: 255.255.255.0
> 0.0036:     router: 192.168.100.1
> 0.0036: DNS:
> 0.0036:     192.168.100.1
> 0.0036: DNS search list:
> 0.0036:     .
> 0.0204: SO_PEEK_OFF supported
> 0.0204: TCP_INFO tcpi_snd_wnd field  supported
> 0.0205: TCP_INFO tcpi_bytes_acked field  supported
> 0.0205: TCP_INFO tcpi_min_rtt field  supported
> 0.0205: Saving packet capture to /tmp/dns.pcap
> 0.0281: pasta: epoll event on /dev/net/tun device 16 (events: 0x00000001)
> 0.0413: pasta: epoll event on /dev/net/tun device 16 (events: 0x00000001)
> 0.0414: pasta: epoll event on /dev/net/tun device 16 (events: 0x00000001)
> 0.0414: tap: protocol 17, 192.168.100.157:56205 -> 192.168.100.1:53 (1 packet)
> 0.0415: Flow 0 (NEW): FREE -> NEW
> 0.0415: Flow 0 (INI): NEW -> INI
> 0.0415: Flow 0 (INI): TAP [192.168.100.157]:56205 -> [192.168.100.1]:53 => ?
> 0.0416: Flow 0 (TGT): INI -> TGT
> 0.0416: Flow 0 (TGT): TAP [192.168.100.157]:56205 -> [192.168.100.1]:53 => HOST [0.0.0.0]:56205 -> [192.168.100.1]:53
> 0.0416: Flow 0 (UDP flow): TGT -> TYPED
> 0.0416: Flow 0 (UDP flow): TAP [192.168.100.157]:56205 -> [192.168.100.1]:53 => HOST [0.0.0.0]:56205 -> [192.168.100.1]:53
> 0.0417: Flow 0 (UDP flow): Side 0 hash table insert: bucket: 121236
> 0.0417: Flow 0 (UDP flow): TYPED -> ACTIVE
> 0.0417: Flow 0 (UDP flow): TAP [192.168.100.157]:56205 -> [192.168.100.1]:53 => HOST [0.0.0.0]:56205 -> [192.168.100.1]:53
> 0.3059: pasta: epoll event on UDP reply socket 96 (events: 0x00000001)
> 0.3059: Flow 0 (UDP flow): Received 1 datagrams on reply socket
> Server:		192.168.100.1
> Address:	192.168.100.1#53
> 
> Non-authoritative answer:
> Name:	fsf.org
> Address: 209.51.188.174

Okay, at least the DNS issue is fixed. I'll apply the fix in a moment,
it will be available in an updated package in a while.

-- 
Stefano

Re: Apparmor (and other) Issues

[Andrea Bolognani]

On Tue, Feb 04, 2025 at 09:50:00AM +0100, Stefano Brivio wrote:
> On Tue, 04 Feb 2025 08:21:53 +0000 Prafulla Giri <prafulla.giri@protonmail.com> wrote:
> > type=SERVICE_START msg=audit(1738501309.082:134): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=polkit comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
> > type=AVC msg=audit(1738501309.118:135): apparmor="DENIED" operation="file_mmap" class="file" profile="passt" name="/usr/bin/passt" pid=2030 comm="passt" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0FSUID="larryboy" OUID="root"
> > type=SYSCALL msg=audit(1738501309.118:135): arch=c000003e syscall=59 success=no exit=-13 a0=7faf24035fc0 a1=7faf24035210 a2=7ffc063280d0 a3=0 items=0 ppid=1964 pid=2030 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=1 comm="passt" exe="/usr/bin/passt" subj=passt key=(null)ARCH=x86_64 SYSCALL=execve AUID="larryboy" UID="larryboy" GID="larryboy" EUID="larryboy" SUID="larryboy" FSUID="larryboy" EGID="larryboy" SGID="larryboy" FSGID="larryboy"
> > type=PROCTITLE msg=audit(1738501309.118:135): proctitle="(null)"
> > type=ANOM_ABEND msg=audit(1738501309.118:136): auid=1000 uid=1000 gid=1000 ses=1 subj=passt pid=2030 comm="passt" exe="/usr/bin/passt" sig=11 res=1AUID="larryboy" UID="larryboy" GID="larryboy"
>
> So, it looks like passt is running as its own profile. This shouldn't
> happen because the libvirt profile has an own subprofile and we should
> see that in "profile" on the type=AVC line but... I just reproduced
> this! Clean Debian sid install, fresh install of libvirtd:
>
> error: internal error: Child process (passt --one-off --socket /run/user/1000/libvirt/qemu/run/passt/1-alpine-net0.socket --pid /run/user/1000/libvirt/qemu/run/passt/1-alpine-net0-passt.pid --tcp-ports 40922:22) unexpected fatal signal 11
>
> I'll keep you posted.

I've skimmed the conversation trying to understand whether there's
anything that I need do from the libvirt side, but AFAICT no explicit
action has been called for so far.

It looks like you're making good progress in figuring out what's
going on. Being able to reproduce the issue yourself is certainly
going to help. I'm happy to leave all the debugging to you, since as
you know I'm not very good at the AppArmor stuff and I'm really,
really bad at the networking stuff ;)

Once a clearer picture emerges, if it turns out that changes are
needed in either libvirt or its Debian packaging, I can definitely
look into making that happen.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: Apparmor (and other) Issues

[Stefano Brivio]

On Tue, 4 Feb 2025 09:50:40 +0000
Andrea Bolognani <abologna@redhat.com> wrote:

> On Tue, Feb 04, 2025 at 09:50:00AM +0100, Stefano Brivio wrote:
> > On Tue, 04 Feb 2025 08:21:53 +0000 Prafulla Giri <prafulla.giri@protonmail.com> wrote:  
> > > type=SERVICE_START msg=audit(1738501309.082:134): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=polkit comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
> > > type=AVC msg=audit(1738501309.118:135): apparmor="DENIED" operation="file_mmap" class="file" profile="passt" name="/usr/bin/passt" pid=2030 comm="passt" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0FSUID="larryboy" OUID="root"
> > > type=SYSCALL msg=audit(1738501309.118:135): arch=c000003e syscall=59 success=no exit=-13 a0=7faf24035fc0 a1=7faf24035210 a2=7ffc063280d0 a3=0 items=0 ppid=1964 pid=2030 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=1 comm="passt" exe="/usr/bin/passt" subj=passt key=(null)ARCH=x86_64 SYSCALL=execve AUID="larryboy" UID="larryboy" GID="larryboy" EUID="larryboy" SUID="larryboy" FSUID="larryboy" EGID="larryboy" SGID="larryboy" FSGID="larryboy"
> > > type=PROCTITLE msg=audit(1738501309.118:135): proctitle="(null)"
> > > type=ANOM_ABEND msg=audit(1738501309.118:136): auid=1000 uid=1000 gid=1000 ses=1 subj=passt pid=2030 comm="passt" exe="/usr/bin/passt" sig=11 res=1AUID="larryboy" UID="larryboy" GID="larryboy"  
> >
> > So, it looks like passt is running as its own profile. This shouldn't
> > happen because the libvirt profile has an own subprofile and we should
> > see that in "profile" on the type=AVC line but... I just reproduced
> > this! Clean Debian sid install, fresh install of libvirtd:
> >
> > error: internal error: Child process (passt --one-off --socket /run/user/1000/libvirt/qemu/run/passt/1-alpine-net0.socket --pid /run/user/1000/libvirt/qemu/run/passt/1-alpine-net0-passt.pid --tcp-ports 40922:22) unexpected fatal signal 11
> >
> > I'll keep you posted.  
> 
> I've skimmed the conversation trying to understand whether there's
> anything that I need do from the libvirt side, but AFAICT no explicit
> action has been called for so far.

Not yet, because I was hoping to figure out what's going on, but I'm
actually (almost?) stuck now. I don't think this is the same as
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1094583 by the way,
because:

$ find /etc/apparmor.d/ -ls | grep -i virt
   148926      4 drwxr-xr-x   2 root     root         4096 Feb  4 08:44 /etc/apparmor.d/libvirt
   148927      4 -rw-r--r--   1 root     root          192 Jan 15 08:06 /etc/apparmor.d/libvirt/TEMPLATE.qemu
   149882      4 -rw-r--r--   1 root     root          342 Jan 15 08:06 /etc/apparmor.d/libvirt/TEMPLATE.lxc
     6098      8 -rw-r--r--   1 root     root         4780 Jan 30 22:47 /etc/apparmor.d/usr.sbin.libvirtd
    20741      0 -rw-r--r--   1 root     root            0 Feb  4 08:44 /etc/apparmor.d/local/usr.sbin.libvirtd
    20572      0 -rw-r--r--   1 root     root            0 Feb  4 08:44 /etc/apparmor.d/local/usr.lib.libvirt.virt-aa-helper
     6826     12 -rw-r--r--   1 root     root         9258 Jan 30 22:47 /etc/apparmor.d/abstractions/libvirt-qemu
    20662      8 -rw-r--r--   1 root     root         4610 Jan 30 22:47 /etc/apparmor.d/abstractions/libvirt-lxc
     6099      4 -rw-r--r--   1 root     root         1898 Jan 30 22:47 /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper
$ find /etc/libvirt/    -ls | grep -i conf
find: '/etc/libvirt/secrets': Permission denied
   268888     20 -rw-r--r--   1 root     root        17826 Jan 30 22:47 /etc/libvirt/libvirtd.conf
   269240      4 -rw-r--r--   1 root     root         2169 Jan 30 22:47 /etc/libvirt/libxl-lockd.conf
   324760      4 -rw-r--r--   1 root     root          547 Nov  1 09:13 /etc/libvirt/libvirt.conf
   269243      4 -rw-r--r--   1 root     root         3058 Jan 15 08:06 /etc/libvirt/virtlockd.conf
   263460      4 -rw-r--r--   1 root     root         2465 Jan 30 22:47 /etc/libvirt/qemu-sanlock.conf
   263459      4 -rw-r--r--   1 root     root         2169 Jan 30 22:47 /etc/libvirt/qemu-lockd.conf
   269238      4 -rw-r--r--   1 root     root         1175 Jan 15 08:06 /etc/libvirt/lxc.conf
   324759      4 -rw-r--r--   1 root     root          450 Nov  1 09:13 /etc/libvirt/libvirt-admin.conf
   269241      4 -rw-r--r--   1 root     root         2465 Jan 30 22:47 /etc/libvirt/libxl-sanlock.conf
   269242      4 -rw-r--r--   1 root     root         2268 Jan 15 08:06 /etc/libvirt/libxl.conf
   263461     40 -rw-------   1 root     root        39106 Jan 30 22:47 /etc/libvirt/qemu.conf
   262245      4 -rw-r--r--   1 root     root         4095 Jan 15 08:06 /etc/libvirt/virtlogd.conf
   268889      4 -rw-r--r--   1 root     root         1041 Jan 30 22:47 /etc/libvirt/network.conf

> It looks like you're making good progress in figuring out what's
> going on. Being able to reproduce the issue yourself is certainly
> going to help. I'm happy to leave all the debugging to you, since as
> you know I'm not very good at the AppArmor stuff and I'm really,
> really bad at the networking stuff ;)

...no, wait, I'm still failing to understand the bigger picture of what
happens AppArmor-wise when I do 'virsh start something'. :)

This is really pretty simple: fresh Debian sid image, all packages
updated to today. Then:

  virt-install -d --name alpine --memory 1024 --noreboot --osinfo alpinelinux3.20 --network backend.type=passt,portForward0.proto=tcp,portForward0.range0.start=40922,portForward0.range0.to=22 --import --disk nocloud_alpine-3.21.2-x86_64-bios-tiny-r0.qcow2

this works. But:

  $ virsh start alpine
  error: Failed to start domain 'alpine'
  error: internal error: Child process (passt --one-off --socket /run/user/1000/libvirt/qemu/run/passt/1-alpine-net0.socket --pid /run/user/1000/libvirt/qemu/run/passt/1-alpine-net0-passt.pid --tcp-ports 40922:2) unexpected fatal signal 11

execve() of passt is denied by AppArmor. Starting passt on its own
(passt -f) works, instead.

At this point, which libvirtd (?) process should associate with which
libvirtd profile? Once that's clear to me, I can probably debug further.

I can also give you access to the machine if needed.

> Once a clearer picture emerges, if it turns out that changes are
> needed in either libvirt or its Debian packaging, I can definitely
> look into making that happen.

I'm fairly sure it's libvirt, because I didn't change anything
substantial in passt, and it's anyway the AppArmor profile for libvirtd
that seems to be... missing? And yet it's there.

-- 
Stefano

Re: Apparmor (and other) Issues

[Andrea Bolognani]

On Tue, Feb 04, 2025 at 11:17:24AM +0100, Stefano Brivio wrote:
> On Tue, 4 Feb 2025 09:50:40 +0000 Andrea Bolognani <abologna@redhat.com> wrote:
> > I've skimmed the conversation trying to understand whether there's
> > anything that I need do from the libvirt side, but AFAICT no explicit
> > action has been called for so far.
>
> Not yet, because I was hoping to figure out what's going on, but I'm
> actually (almost?) stuck now. I don't think this is the same as
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1094583 by the way,

That issue can only manifest itself when upgrading from bookworm, so
if you have performed a fresh install of sid you don't need to worry
about it.

> This is really pretty simple: fresh Debian sid image, all packages
> updated to today. Then:
>
>   virt-install -d --name alpine --memory 1024 --noreboot --osinfo alpinelinux3.20 --network backend.type=passt,portForward0.proto=tcp,portForward0.range0.start=40922,portForward0.range0.to=22 --import --disk nocloud_alpine-3.21.2-x86_64-bios-tiny-r0.qcow2
>
> this works. But:
>
>   $ virsh start alpine
>   error: Failed to start domain 'alpine'
>   error: internal error: Child process (passt --one-off --socket /run/user/1000/libvirt/qemu/run/passt/1-alpine-net0.socket --pid /run/user/1000/libvirt/qemu/run/passt/1-alpine-net0-passt.pid --tcp-ports 40922:2) unexpected fatal signal 11
>
> execve() of passt is denied by AppArmor. Starting passt on its own
> (passt -f) works, instead.

I was initially very surprised to read this, but it makes sense after
spending some more time looking into it.

Normally, virt-install will perform guest creation in two phases. The
first one is the installation itself, which needs the install media
to be connected; once that's done, the OS installer will usually
initiate a reboot, and when the VM comes up again the installl media
will no longer be connected. Other changes to the configuration might
happen as well, I'm not 100% sure.

Anyway, since you've passed --noreboot above, virt-install will only
go through the first phase of installing the guest; additionally,
since you've passed --import, the VM will only be defined and the
installation will be considered done without having to start it once.

As a counter-example, see how the behavior changes when performing a
"regular" install from CDROM instead:

  $ virt-install --name alpine --memory 1024 --osinfo alpinelinux3.20 \
    --network backend.type=passt --graphics none --disk none \
    --cdrom ./alpine-virt-3.21.2-x86_64.iso
  Starting install...
  ERROR    internal error: Child process (passt --one-off
    --socket /run/user/1000/libvirt/qemu/run/passt/3-alpine-net0.socket
    --pid /run/user/1000/libvirt/qemu/run/passt/3-alpine-net0-passt.pid)
    unexpected fatal signal 11

So there's no magic to that first VM start that apparently worked
even when subsequent ones wouldn't. The VM start simply didn't happen
in the first place.

> At this point, which libvirtd (?) process should associate with which
> libvirtd profile? Once that's clear to me, I can probably debug further.
>
[...]
>
> I'm fairly sure it's libvirt, because I didn't change anything
> substantial in passt, and it's anyway the AppArmor profile for libvirtd
> that seems to be... missing? And yet it's there.

We're going back to the question of how AppArmor works for
unprivileged VMs... The short answer is that I'm not convinced it
does.

In order to make a direct comparison, I've create the very same
Alpine VM, with no network interface, both on qemu:///system and
qemu:///user.

When I start the former, the (filtered) output for aa-status goes
from

  22 profiles are in enforce mode.
     libvirtd
     libvirtd//qemu_bridge_helper
     virt-aa-helper
  25 profiles are in complain mode.
     dnsmasq//libvirt_leaseshelper
  2 processes are in enforce mode.
     /usr/sbin/libvirtd (862) libvirtd

to

  24 profiles are in enforce mode.
     libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4
     libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4//passt
     libvirtd
     libvirtd//qemu_bridge_helper
     virt-aa-helper
  25 profiles are in complain mode.
     dnsmasq//libvirt_leaseshelper
  3 processes are in enforce mode.
     /usr/bin/qemu-system-x86_64 (1310)
libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4
     /usr/sbin/libvirtd (862) libvirtd

The additional profiles (libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4
and its passt subprofile) are generated dynamically when the VM is
started and the QEMU process gets the correct one assigned to it. So
far so good.

After launching the qemu:///session VM, however, the situation is a
lot different:

  22 profiles are in enforce mode.
     libvirtd
     libvirtd//qemu_bridge_helper
     virt-aa-helper
  25 profiles are in complain mode.
     dnsmasq//libvirt_leaseshelper
  4 processes are in enforce mode.
     /usr/sbin/libvirtd (1589) libvirtd
     /usr/sbin/virtlogd (1632) libvirtd

As you can see, the per-VM profile hasn't been created, and so
obviously it couldn't be applied to the QEMU process either.

This is probably undesirable, but at the same time I'm not really
sure how it could be fixed. Creating a new profile requires dropping
a file in /etc/apparmor.d/libvirt, and the unprivileged libvirtd
intentionally doesn't have the necessary permissions to do that...

Do you have any ideas?

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: Apparmor (and other) Issues

[Stefano Brivio]

On Tue, 4 Feb 2025 15:50:43 +0000
Andrea Bolognani <abologna@redhat.com> wrote:

> On Tue, Feb 04, 2025 at 11:17:24AM +0100, Stefano Brivio wrote:
> > On Tue, 4 Feb 2025 09:50:40 +0000 Andrea Bolognani <abologna@redhat.com> wrote:  
> > > I've skimmed the conversation trying to understand whether there's
> > > anything that I need do from the libvirt side, but AFAICT no explicit
> > > action has been called for so far.  
> >
> > Not yet, because I was hoping to figure out what's going on, but I'm
> > actually (almost?) stuck now. I don't think this is the same as
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1094583 by the way,  
> 
> That issue can only manifest itself when upgrading from bookworm, so
> if you have performed a fresh install of sid you don't need to worry
> about it.
> 
> > This is really pretty simple: fresh Debian sid image, all packages
> > updated to today. Then:
> >
> >   virt-install -d --name alpine --memory 1024 --noreboot --osinfo alpinelinux3.20 --network backend.type=passt,portForward0.proto=tcp,portForward0.range0.start=40922,portForward0.range0.to=22 --import --disk nocloud_alpine-3.21.2-x86_64-bios-tiny-r0.qcow2
> >
> > this works. But:
> >
> >   $ virsh start alpine
> >   error: Failed to start domain 'alpine'
> >   error: internal error: Child process (passt --one-off --socket /run/user/1000/libvirt/qemu/run/passt/1-alpine-net0.socket --pid /run/user/1000/libvirt/qemu/run/passt/1-alpine-net0-passt.pid --tcp-ports 40922:2) unexpected fatal signal 11
> >
> > execve() of passt is denied by AppArmor. Starting passt on its own
> > (passt -f) works, instead.  
> 
> I was initially very surprised to read this, but it makes sense after
> spending some more time looking into it.

Ah, sure, sorry! I didn't mean to waste your time with that, it was
clear to me that passt doesn't start there... I just wanted to show the
setup.

> [...]
>
> So there's no magic to that first VM start that apparently worked
> even when subsequent ones wouldn't. The VM start simply didn't happen
> in the first place.
> 
> > At this point, which libvirtd (?) process should associate with which
> > libvirtd profile? Once that's clear to me, I can probably debug further.
> >  
> [...]
> >
> > I'm fairly sure it's libvirt, because I didn't change anything
> > substantial in passt, and it's anyway the AppArmor profile for libvirtd
> > that seems to be... missing? And yet it's there.  
> 
> We're going back to the question of how AppArmor works for
> unprivileged VMs... The short answer is that I'm not convinced it
> does.
> 
> In order to make a direct comparison, I've create the very same
> Alpine VM, with no network interface, both on qemu:///system and
> qemu:///user.
> 
> When I start the former, the (filtered) output for aa-status goes
> from
> 
>   22 profiles are in enforce mode.
>      libvirtd
>      libvirtd//qemu_bridge_helper
>      virt-aa-helper
>   25 profiles are in complain mode.
>      dnsmasq//libvirt_leaseshelper
>   2 processes are in enforce mode.
>      /usr/sbin/libvirtd (862) libvirtd
> 
> to
> 
>   24 profiles are in enforce mode.
>      libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4
>      libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4//passt
>      libvirtd
>      libvirtd//qemu_bridge_helper
>      virt-aa-helper
>   25 profiles are in complain mode.
>      dnsmasq//libvirt_leaseshelper
>   3 processes are in enforce mode.
>      /usr/bin/qemu-system-x86_64 (1310)
> libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4
>      /usr/sbin/libvirtd (862) libvirtd
> 
> The additional profiles (libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4
> and its passt subprofile) are generated dynamically when the VM is
> started and the QEMU process gets the correct one assigned to it. So
> far so good.

Why do we need additional profiles, I wonder? Are we trying to
replicate the MCS (Multi-Category Security) stuff from SELinux?

> After launching the qemu:///session VM, however, the situation is a
> lot different:
> 
>   22 profiles are in enforce mode.
>      libvirtd
>      libvirtd//qemu_bridge_helper
>      virt-aa-helper
>   25 profiles are in complain mode.
>      dnsmasq//libvirt_leaseshelper
>   4 processes are in enforce mode.
>      /usr/sbin/libvirtd (1589) libvirtd
>      /usr/sbin/virtlogd (1632) libvirtd
> 
> As you can see, the per-VM profile hasn't been created, and so
> obviously it couldn't be applied to the QEMU process either.

Oh, now I understand the whole "template" thing finally! And now that
you mention virt-aa-helper (I wasn't aware of that) I also found:

  https://gitlab.com/apparmor/apparmor/-/wikis/Libvirt

Yes, of course, that won't work unless you're root.

> This is probably undesirable, but at the same time I'm not really
> sure how it could be fixed.

If I understood the purpose correctly: implementing equivalent MCS
functionality in AppArmor would probably be the only "complete" fix. But
for the moment (or more realistically):

> Creating a new profile requires dropping
> a file in /etc/apparmor.d/libvirt, and the unprivileged libvirtd
> intentionally doesn't have the necessary permissions to do that...

Profiles could reside in other places too, but they won't be associated
"just like that". A privileged helper could... help. But if users can
edit their own profiles in their homes, that defies a bit the point of
AppArmor profiles altogether.

> Do you have any ideas?

Probably you should ask somebody more familiar (openSUSE/Ubuntu
maintainers of libvirt, or the authors of the virt-aa-helper thing?),
but a couple of quick ideas:

1. user-specific subprofiles could be added at adduser time (is there a
   hook?) and libvirtd could use aa_change_hat(2) to select the
   appropriate one based on UID

2. (most reasonable I think) don't use per-VM profiles for the rootless
   case. Define a single "libvirt-user" (or "libvirt-session") profile
   and use that. We could copy it from the existing ones I suppose.

I think that MCS-equivalent functionality is anyway much less critical
if guests are started as unprivileged users because you have a strong
separation given by the fact that guests are started as different users.

If all the guests are started as root, and multiple users have access
to that facility, user separation is much more seriously endangered,
hence a stronger need for per-VM profiles.

And, stating the obvious, it's counterproductive to force users to run
guests as root because we can't have that kind of functionality.

By the way, I recently happened to introduce an AppArmor profile for
guestfs-tools:

  https://salsa.debian.org/libvirt-team/guestfs-tools/-/merge_requests/1

there are no separate profiles, there.

I can try to work on this but I'm really a bit too busy right now. If
you can, great, otherwise, let me know. On the other hand this feels
pretty urgent. :(

-- 
Stefano

Re: Apparmor (and other) Issues

[Andrea Bolognani]

On Tue, Feb 04, 2025 at 05:22:42PM +0100, Stefano Brivio wrote:
> On Tue, 4 Feb 2025 15:50:43 +0000 Andrea Bolognani <abologna@redhat.com> wrote:
> > The additional profiles (libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4
> > and its passt subprofile) are generated dynamically when the VM is
> > started and the QEMU process gets the correct one assigned to it. So
> > far so good.
>
> Why do we need additional profiles, I wonder? Are we trying to
> replicate the MCS (Multi-Category Security) stuff from SELinux?

I'm not well-versed enough in SELinux to be able to answer the latter
question, but I can answer the former. The per-VM profile is needed
to ensure that each VM is only granted access to its own resources.

  $ sudo tail -4
/etc/apparmor.d/libvirt/libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4
  profile libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4
flags=(attach_disconnected) {
    #include <abstractions/libvirt-qemu>
    #include if exists
<libvirt/libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4.files>
  }

  $ sudo cat /etc/apparmor.d/libvirt/libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4.files
    "/var/log/libvirt/**/alpine.log" w,
    "/var/lib/libvirt/qemu/domain-alpine/monitor.sock" rw,
    "/var/lib/libvirt/qemu/domain-1-alpine/*" rw,
    "/run/libvirt/**/alpine.pid" rwk,
    "/run/libvirt/**/*.tunnelmigrate.dest.alpine" rw,
    "/var/lib/libvirt/images/alpine.qcow2" rwk,
    "/var/lib/libvirt/qemu/domain-1-alpine/{,**}" rwk,
    "/run/libvirt/qemu/channel/1-alpine/{,**}" rwk,
    "/var/lib/libvirt/qemu/domain-1-alpine/master-key.aes" rwk,
    "/dev/userfaultfd" rwk,

The stuff in the libvirt-qemu abstraction is generic, all VMs get
access to it. The stuff in .files is all specific to the VM.

> > Do you have any ideas?
>
> Probably you should ask somebody more familiar (openSUSE/Ubuntu
> maintainers of libvirt, or the authors of the virt-aa-helper thing?),
> but a couple of quick ideas:
>
> 1. user-specific subprofiles could be added at adduser time (is there a
>    hook?) and libvirtd could use aa_change_hat(2) to select the
>    appropriate one based on UID
>
> 2. (most reasonable I think) don't use per-VM profiles for the rootless
>    case. Define a single "libvirt-user" (or "libvirt-session") profile
>    and use that. We could copy it from the existing ones I suppose.

Sounds to me like this would require granting the QEMU process access
to roughly the entire filesystem? The disk image could live anywhere
after all, and if we can't dynamically add a rule for the exact path
the only way out is a free-for-all approach.

> I think that MCS-equivalent functionality is anyway much less critical
> if guests are started as unprivileged users because you have a strong
> separation given by the fact that guests are started as different users.
>
> If all the guests are started as root, and multiple users have access
> to that facility, user separation is much more seriously endangered,
> hence a stronger need for per-VM profiles.

Ideally you'd still want to isolate unprivileged VMs from each other.
If you don't, what's the point of using AppArmor in the first place?

> I can try to work on this but I'm really a bit too busy right now. If
> you can, great, otherwise, let me know. On the other hand this feels
> pretty urgent. :(

I'm afraid I have neither the time nor the expertise to work on this
myself.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: Apparmor (and other) Issues

[Stefano Brivio]

On Tue, 4 Feb 2025 18:46:37 +0000
Andrea Bolognani <abologna@redhat.com> wrote:

> On Tue, Feb 04, 2025 at 05:22:42PM +0100, Stefano Brivio wrote:
> > On Tue, 4 Feb 2025 15:50:43 +0000 Andrea Bolognani <abologna@redhat.com> wrote:  
> > > The additional profiles (libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4
> > > and its passt subprofile) are generated dynamically when the VM is
> > > started and the QEMU process gets the correct one assigned to it. So
> > > far so good.  
> >
> > Why do we need additional profiles, I wonder? Are we trying to
> > replicate the MCS (Multi-Category Security) stuff from SELinux?  
> 
> I'm not well-versed enough in SELinux to be able to answer the latter
> question, but I can answer the former. The per-VM profile is needed
> to ensure that each VM is only granted access to its own resources.
> 
>   $ sudo tail -4
> /etc/apparmor.d/libvirt/libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4
>   profile libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4
> flags=(attach_disconnected) {
>     #include <abstractions/libvirt-qemu>
>     #include if exists
> <libvirt/libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4.files>
>   }
> 
>   $ sudo cat /etc/apparmor.d/libvirt/libvirt-1cdfdfd3-a020-45d9-8f88-8e0ce31745c4.files
>     "/var/log/libvirt/**/alpine.log" w,
>     "/var/lib/libvirt/qemu/domain-alpine/monitor.sock" rw,
>     "/var/lib/libvirt/qemu/domain-1-alpine/*" rw,
>     "/run/libvirt/**/alpine.pid" rwk,
>     "/run/libvirt/**/*.tunnelmigrate.dest.alpine" rw,
>     "/var/lib/libvirt/images/alpine.qcow2" rwk,
>     "/var/lib/libvirt/qemu/domain-1-alpine/{,**}" rwk,
>     "/run/libvirt/qemu/channel/1-alpine/{,**}" rwk,
>     "/var/lib/libvirt/qemu/domain-1-alpine/master-key.aes" rwk,
>     "/dev/userfaultfd" rwk,
> 
> The stuff in the libvirt-qemu abstraction is generic, all VMs get
> access to it. The stuff in .files is all specific to the VM.

Ah, thanks, it never occurred to me to look into that. So, yes,
essentially the same thing as it's done with MCS and SELinux.

> > > Do you have any ideas?  
> >
> > Probably you should ask somebody more familiar (openSUSE/Ubuntu
> > maintainers of libvirt, or the authors of the virt-aa-helper thing?),
> > but a couple of quick ideas:
> >
> > 1. user-specific subprofiles could be added at adduser time (is there a
> >    hook?) and libvirtd could use aa_change_hat(2) to select the
> >    appropriate one based on UID
> >
> > 2. (most reasonable I think) don't use per-VM profiles for the rootless
> >    case. Define a single "libvirt-user" (or "libvirt-session") profile
> >    and use that. We could copy it from the existing ones I suppose.  
> 
> Sounds to me like this would require granting the QEMU process access
> to roughly the entire filesystem? The disk image could live anywhere
> after all, and if we can't dynamically add a rule for the exact path
> the only way out is a free-for-all approach.

Right. That's what we did for libguestfs as well, the image can be
*almost* anywhere. But it's not free-for-all: you're just granting
*limited* filesystem access (not to sysfs, not to /etc, and so on).

And I had to build a *very* loose profile for libguestfs because that
applies to root as well, but for rootless libvirtd, it might even make
sense to restrict access to just @{HOME}/** and /tmp/** (that's what I
did for stand-alone passt, for example).

> > I think that MCS-equivalent functionality is anyway much less critical
> > if guests are started as unprivileged users because you have a strong
> > separation given by the fact that guests are started as different users.
> >
> > If all the guests are started as root, and multiple users have access
> > to that facility, user separation is much more seriously endangered,
> > hence a stronger need for per-VM profiles.  
> 
> Ideally you'd still want to isolate unprivileged VMs from each other.
> If you don't, what's the point of using AppArmor in the first place?

Well, you are still restricting them significantly, for example they
can't run random binaries, send arbitrary signals to other processes
or to each other, and access a ton of places on the filesystem.

Perhaps that's even more important than curbing filesystem access,
because that should already be significantly restricted.

> > I can try to work on this but I'm really a bit too busy right now. If
> > you can, great, otherwise, let me know. On the other hand this feels
> > pretty urgent. :(  
> 
> I'm afraid I have neither the time nor the expertise to work on this
> myself.

I'll try to submit a pull request at least for Debian in a couple of
days. I guess it would be nice if you could ask other maintainers
meanwhile.

-- 
Stefano

Re: Apparmor (and other) Issues

[Andrea Bolognani]

On Tue, Feb 04, 2025 at 08:14:48PM +0100, Stefano Brivio wrote:
> On Tue, 4 Feb 2025 18:46:37 +0000 Andrea Bolognani <abologna@redhat.com> wrote:
> > > 2. (most reasonable I think) don't use per-VM profiles for the rootless
> > >    case. Define a single "libvirt-user" (or "libvirt-session") profile
> > >    and use that. We could copy it from the existing ones I suppose.
> >
> > Sounds to me like this would require granting the QEMU process access
> > to roughly the entire filesystem? The disk image could live anywhere
> > after all, and if we can't dynamically add a rule for the exact path
> > the only way out is a free-for-all approach.
>
> Right. That's what we did for libguestfs as well, the image can be
> *almost* anywhere. But it's not free-for-all: you're just granting
> *limited* filesystem access (not to sysfs, not to /etc, and so on).
>
> And I had to build a *very* loose profile for libguestfs because that
> applies to root as well, but for rootless libvirtd, it might even make
> sense to restrict access to just @{HOME}/** and /tmp/** (that's what I
> did for stand-alone passt, for example).

That could work, I suppose. Needs to be discussed upstream, making
sure to involve those who are more experienced with AppArmor than I
am, especially since it's not just a matter of updating the policy
but fundamentally changing how the driver works when operating in
unprivileged mode.

> I'll try to submit a pull request at least for Debian in a couple of
> days.

Be aware that I will emphatically refuse to introduce changes to the
Debian package unless they have been merged upstream first. AppArmor
support lives in the upstream repository, and all fixes and
improvements have to go through it.

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: Apparmor (and other) Issues

[Stefano Brivio]

On Tue, 4 Feb 2025 14:19:34 -0800
Andrea Bolognani <abologna@redhat.com> wrote:

> On Tue, Feb 04, 2025 at 08:14:48PM +0100, Stefano Brivio wrote:
> > On Tue, 4 Feb 2025 18:46:37 +0000 Andrea Bolognani <abologna@redhat.com> wrote:  
> > > > 2. (most reasonable I think) don't use per-VM profiles for the rootless
> > > >    case. Define a single "libvirt-user" (or "libvirt-session") profile
> > > >    and use that. We could copy it from the existing ones I suppose.  
> > >
> > > Sounds to me like this would require granting the QEMU process access
> > > to roughly the entire filesystem? The disk image could live anywhere
> > > after all, and if we can't dynamically add a rule for the exact path
> > > the only way out is a free-for-all approach.  
> >
> > Right. That's what we did for libguestfs as well, the image can be
> > *almost* anywhere. But it's not free-for-all: you're just granting
> > *limited* filesystem access (not to sysfs, not to /etc, and so on).
> >
> > And I had to build a *very* loose profile for libguestfs because that
> > applies to root as well, but for rootless libvirtd, it might even make
> > sense to restrict access to just @{HOME}/** and /tmp/** (that's what I
> > did for stand-alone passt, for example).  
> 
> That could work, I suppose. Needs to be discussed upstream, making
> sure to involve those who are more experienced with AppArmor than I
> am

Sure, that makes sense of course.

> especially since it's not just a matter of updating the policy
> but fundamentally changing how the driver works when operating in
> unprivileged mode.

Well, right now it runs unconfined, and (at least with passt) networking
doesn't work. Pretty much any change would be good. :)

I've been disabling AppArmor when I start guests for quite a while now,
thinking that I just broke something on my setup while developing
stuff (I reported that to you but I wasn't sure how the whole thing
worked...). Oops.

> > I'll try to submit a pull request at least for Debian in a couple of
> > days.  
> 
> Be aware that I will emphatically refuse to introduce changes to the
> Debian package unless they have been merged upstream first. AppArmor
> support lives in the upstream repository, and all fixes and
> improvements have to go through it.

Then never mind. I can help filing a Debian issue if needed, let me
know. Or let me know how I can help otherwise.

I would consider a workaround for passt for the moment, say, the whole
block:

    /usr/bin/passt r,

    signal (receive) set=("term") peer=/usr/sbin/libvirtd,
    signal (receive) set=("term") peer=libvirtd,
    signal (receive) set=("term") peer=virtqemud,

    owner @{run}/user/[0-9]*/libvirt/qemu/run/passt/* rw,
    owner @{run}/libvirt/qemu/passt/* rw,

as part of the profile for usr.bin.passt. If you don't see issues with
it, I'll go ahead with that. I still need to check openSUSE though (I
haven't tried for a while there).

-- 
Stefano

Re: Apparmor (and other) Issues

[Prafulla Giri]

I am glad such capable maintainers are handling this.

If I may ask, however: could this simply not be dealt with by allowing passt binary access to $XDG_RUNTIME_DIR of the user in the apparmor profile? Forgive me, I am just a novice. But from my lack-of-understanding this issue looks like an issue of passt process not being able to create a socket inside a libvirt-maintained directory inside /run/user/$UID and that is why disabling the apparmor profile for passt seems to work-around this (?) Are there security concerns with this? Only asking out of curiosity.

Re: Apparmor (and other) Issues

[Stefano Brivio]

On Wed, 05 Feb 2025 07:40:34 +0000
Prafulla Giri <prafulla.giri@protonmail.com> wrote:

> If I may ask, however: could this simply not be dealt with by
> allowing passt binary access to $XDG_RUNTIME_DIR of the user in the
> apparmor profile? Forgive me, I am just a novice.

Yes, that's a workaround. But the rationale for the current mechanism
based on a passt 'abstraction' is that if libvirtd starts passt, then
those /var/run/... paths are needed for the socket, and otherwise not.

Only the libvirt profile "knows" about that. That's why it's in the
libvirt profile. But the libvirt profile is not associated to the
process, oops.

> But from my
> lack-of-understanding this issue looks like an issue of passt process
> not being able to create a socket inside a libvirt-maintained
> directory inside /run/user/$UID and that is why disabling the
> apparmor profile for passt seems to work-around this (?) Are there
> security concerns with this? Only asking out of curiosity.

Your understanding is correct. We're just trying to make things as
strict as possible, and depending on specific paths.

We'll probably need to make them a bit looser for the moment being and
perhaps just allow passt, no matter who starts it, to write to
/var/run/**.

-- 
Stefano

Re: Apparmor (and other) Issues

[Prafulla Giri]

On Wednesday, February 5th, 2025 at 4:01 PM, Stefano Brivio <sbrivio@redhat.com> wrote:

> But the libvirt profile is not associated to the
> process, oops.

Oh, so this is what is being worked upon: that Apparmor is not making the association, whereas SELinux is doing it's thing as it's supposed to.

> We're just trying to make things as
> strict as possible, and depending on specific paths.

I see. I'm glad this approach of as-strict-as-possible is being taken.

> We'll probably need to make them a bit looser for the moment being and
> perhaps just allow passt, no matter who starts it, to write to
> /var/run/**.

I believe user-mode virtual machines only need access to /run/user/$USER and not /var/run. Not even /run/*, but only /run/user/$USER. So if that work-around is to be implemented, that would be the strictest version of it: each user-started passt process gets access to $XDG_RUNTIME_DIR of it's owner (and not outside of it).

It also seems that more and more of us use $XDG_RUNTIME_DIR in lieu of /tmp in our personal shell scripts, because it kinda' feels like a more private /tmp.

Also, the `passt` update fixing DNS issue hasn't yet made it to Debian Trixie, yet. I figure it's going to take some time (?) Perhaps I should venture to Debian Sid, myself.

Re: Apparmor (and other) Issues

[Stefano Brivio]

On Fri, 07 Feb 2025 06:49:45 +0000
Prafulla Giri <prafulla.giri@protonmail.com> wrote:

> On Wednesday, February 5th, 2025 at 4:01 PM, Stefano Brivio
> <sbrivio@redhat.com> wrote:
> 
> > But the libvirt profile is not associated to the
> > process, oops.  
> 
> Oh, so this is what is being worked upon: that Apparmor is not making
> the association

That, I'm not sure, but at least Andrea asked openSUSE and Ubuntu
people for comments. I just prepared (and merged) a workaround for the
moment. You are Cc'ed on the patch. If you want to test it, you should
add this:

  # Workaround: libvirt's profile comes with a passt subprofile which includes,
  # in turn, <abstractions/passt>, and adds libvirt-specific rules on top, to
  # allow passt (when started by libvirtd) to write socket and PID files in the
  # location requested by libvirtd itself, and to execute passt itself.
  #
  # However, when libvirt runs as unprivileged user, the mechanism based on
  # virt-aa-helper, designed to build per-VM profiles as guests are started,
  # doesn't work. The helper needs to create and load profiles on the fly, which
  # can't be done by unprivileged users, of course.
  #
  # As a result, libvirtd runs unconfined if guests are started by unprivileged
  # users, starting passt unconfined as well, which means that passt runs under
  # its own stand-alone profile (this one), which implies in turn that execve()
  # of /usr/bin/passt is not allowed, and socket and PID files can't be written.
  #
  # Duplicate libvirt-specific rules here as long as this is not solved in
  # libvirt's profile itself.
  /usr/bin/passt r,
  owner @{run}/user/[0-9]*/libvirt/qemu/run/passt/* rw,
  owner @{run}/libvirt/qemu/passt/* rw,

to your /etc/apparmor.d/usr.bin.passt. Note that changes to AppArmor
policy files are retained as configuration, so, if you edit it, package
upgrades won't override things automatically. You will need to:

  apt-get -o Dpkg::Options::="--force-confmiss" install --reinstall passt

> whereas SELinux is doing it's thing as it's supposed to.

Right, that's because SELinux can do this:

  https://selinuxproject.org/page/MultiCategorySecurity

with AppArmor, it needs to be "emulated", somehow.

> > We're just trying to make things as
> > strict as possible, and depending on specific paths.  
> 
> I see. I'm glad this approach of as-strict-as-possible is being taken.
> 
> > We'll probably need to make them a bit looser for the moment being
> > and perhaps just allow passt, no matter who starts it, to write to
> > /var/run/**.  
> 
> I believe user-mode virtual machines only need access to
> /run/user/$USER and not /var/run. Not even /run/*, but only
> /run/user/$USER. So if that work-around is to be implemented, that
> would be the strictest version of it: each user-started passt process
> gets access to $XDG_RUNTIME_DIR of it's owner (and not outside of it).

It depends, because if you start passt as root, socket and PID go to
@{run}/libvirt/qemu/passt/*, and if you don't, the location becomes
@{run}/user/[0-9]*/libvirt/qemu/run/passt/*.

@{run} is an AppArmor handle for /var/run and /run, I think (aren't
they generally linked anyway?). Note that Ubuntu and openSUSE might
use slightly different paths.

> It also seems that more and more of us use $XDG_RUNTIME_DIR in lieu
> of /tmp in our personal shell scripts, because it kinda' feels like a
> more private /tmp.

Yeah, it is.

> Also, the `passt` update fixing DNS issue hasn't yet made it to
> Debian Trixie, yet.

I didn't release the fix yet. I merged it (upstream), but actually I
was expecting you would give it a quick try. If I'm more confident
about the change, I can do things faster.

> I figure it's going to take some time (?) Perhaps
> I should venture to Debian Sid, myself.

It's not in Debian Sid either because I didn't make a new release of
passt, yet.

It probably makes sense to make one next week (we release quite often,
especially if there's one or more fixes that might be important for
somebody, such as this one). As Debian maintainer I also update Debian
packages within a couple of hours.

Sid to testing is usually five days of difference, look:

  https://tracker.debian.org/pkg/passt/news/

-- 
Stefano

Re: Apparmor (and other) Issues

[Prafulla Giri]

On Friday, February 7th, 2025 at 3:01 PM, Stefano Brivio <sbrivio@redhat.com> wrote:

> On Fri, 07 Feb 2025 06:49:45 +0000
> Prafulla Giri prafulla.giri@protonmail.com wrote:
> 
> > On Wednesday, February 5th, 2025 at 4:01 PM, Stefano Brivio
> > sbrivio@redhat.com wrote:
> > 
> > > But the libvirt profile is not associated to the
> > > process, oops.
> > 
> > Oh, so this is what is being worked upon: that Apparmor is not making
> > the association
> 
> 
> That, I'm not sure, but at least Andrea asked openSUSE and Ubuntu
> people for comments. I just prepared (and merged) a workaround for the
> moment. You are Cc'ed on the patch. If you want to test it, you should
> add this:
> 
> # Workaround: libvirt's profile comes with a passt subprofile which includes,
> # in turn, <abstractions/passt>, and adds libvirt-specific rules on top, to
> 
> # allow passt (when started by libvirtd) to write socket and PID files in the
> # location requested by libvirtd itself, and to execute passt itself.
> #
> # However, when libvirt runs as unprivileged user, the mechanism based on
> # virt-aa-helper, designed to build per-VM profiles as guests are started,
> # doesn't work. The helper needs to create and load profiles on the fly, which
> # can't be done by unprivileged users, of course.
> #
> # As a result, libvirtd runs unconfined if guests are started by unprivileged
> # users, starting passt unconfined as well, which means that passt runs under
> # its own stand-alone profile (this one), which implies in turn that execve()
> # of /usr/bin/passt is not allowed, and socket and PID files can't be written.
> #
> # Duplicate libvirt-specific rules here as long as this is not solved in
> # libvirt's profile itself.
> /usr/bin/passt r,
> owner @{run}/user/[0-9]/libvirt/qemu/run/passt/ rw,
> owner @{run}/libvirt/qemu/passt/* rw,
> 
> to your /etc/apparmor.d/usr.bin.passt. Note that changes to AppArmor
> policy files are retained as configuration, so, if you edit it, package
> upgrades won't override things automatically. You will need to:
> 

I seem to have botched things up really good, or we're getting into more and more trouble here:

1. I have manually `make install`-ed passt (and friends).
$ passt version # I don't know what's causing the non-AVX2 thing issue
Can't run AVX2 build, using non-AVX2 version: Permission denied
passt 2025_01_21.4f2c8e7-27-gfe8b6a7
Copyright Red Hat
GNU General Public License, version 2 or later
  <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

This is my apparmor profile (for /usr/local/bin/passt):
$ cat /etc/apparmor.d/usr.bin.passt
# SPDX-License-Identifier: GPL-2.0-or-later
#
# PASST - Plug A Simple Socket Transport
#  for qemu/UNIX domain socket mode
#
# PASTA - Pack A Subtle Tap Abstraction
#  for network namespace/tap device mode
#
# contrib/apparmor/usr.bin.passt - AppArmor profile for passt(1)
#
# Copyright (c) 2022 Red Hat GmbH
# Author: Stefano Brivio <sbrivio@redhat.com>

abi <abi/3.0>,

include <tunables/global>

profile passt /usr/local/bin/passt{,.avx2} {
  include <abstractions/passt>

  # Alternatively: include <abstractions/user-tmp>
  owner /tmp/**				w,	# tap_sock_unix_open(),
						# tap_sock_unix_init(), pcap(),
						# pidfile_open(),
						# pidfile_write(),
						# logfile_init()

  owner @{HOME}/**			w,	# pcap(), pidfile_open(),
						# pidfile_write()

  # Workaround: libvirt's profile comes with a passt subprofile which includes,
  # in turn, <abstractions/passt>, and adds libvirt-specific rules on top, to
  # allow passt (when started by libvirtd) to write socket and PID files in the
  # location requested by libvirtd itself, and to execute passt itself.
  #
  # However, when libvirt runs as unprivileged user, the mechanism based on
  # virt-aa-helper, designed to build per-VM profiles as guests are started,
  # doesn't work. The helper needs to create and load profiles on the fly, which
  # can't be done by unprivileged users, of course.
  #
  # As a result, libvirtd runs unconfined if guests are started by unprivileged
  # users, starting passt unconfined as well, which means that passt runs under
  # its own stand-alone profile (this one), which implies in turn that execve()
  # of /usr/bin/passt is not allowed, and socket and PID files can't be written.
  #
  # Duplicate libvirt-specific rules here as long as this is not solved in
  # libvirt's profile itself.
  /usr/local/bin/passt r,
  owner @{run}/user/[0-9]*/libvirt/qemu/run/passt/* rw,
  owner @{run}/libvirt/qemu/passt/* rw,
}

I have restarted the machine after making these changes, for good measure.

Then:
$ virsh start --domain vm1 # throws the following error:
error: Failed to start domain 'vm1'
error: internal error: Child process (passt --one-off --socket /run/user/1000/libvirt/qemu/run/passt/1-vm1-net0.socket --pid /run/user/1000/libvirt/qemu/run/passt/1-vm1-net0-passt.pid) unexpected exit status 126: libvirt:  error : cannot execute binary passt: Permission denied

However,
$ passt --one-off --socket /run/user/1000/libvirt/qemu/run/passt/1-vm1-net0.socket --pid /run/user/1000/libvirt/qemu/run/passt/1-vm1-net0-passt.pid # this command on it's own works
Can't run AVX2 build, using non-AVX2 version: Permission denied
No interfaces with usable IPv6 routes
UNIX domain socket bound at /run/user/1000/libvirt/qemu/run/passt/1-vm1-net0.socket
Template interface: enp1s0 (IPv4)
MAC:
    host: 9a:55:9a:55:9a:55
    NAT to host 127.0.0.1: 192.168.100.1
DHCP:
    assign: 192.168.100.157
    mask: 255.255.255.0
    router: 192.168.100.1
DNS:
    192.168.100.1
DNS search list:
    .

You can now start qemu (>= 7.2, with commit 13c6be96618c):
    kvm ... -device virtio-net-pci,netdev=s -netdev stream,id=s,server=off,addr.type=unix,addr.path=/run/user/1000/libvirt/qemu/run/passt/1-vm1-net0.socket
or qrap, for earlier qemu versions:
    ./qrap 5 kvm ... -net socket,fd=5 -net nic,model=virtio

If, however, I uninstall the manually made version (with the apparmor profile back to pointing to /usr/bin/passt):

$ passt --version
passt 0.0~git20250121.4f2c8e7-1
Copyright Red Hat
GNU General Public License, version 2 or later
  <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

$ virsh start --domain vm1 
Domain 'vm1' started

So I figure it's okay after all.

I am surprised, the upstream tree does not seem to contain the recent DNS related patch that I shazamed and tested just a couple of days earlier ( b4 shazam https://archives.passt.top/passt-dev/20250203082210.2114348-1-sbrivio@redhat.com/ ). I have just emailed a reply at the patch's own thread, too: https://archives.passt.top/passt-dev/pXatirFM4Zm6ZSKXYsfeyZnHlL8JlQDKRUNtyLqbzrUOFcItX06upuk1cPYaufL1HzqHygkWvySVhawtV8zVi8Ou9qmhz_EvuQxr46UOi3Q=@protonmail.com/

Please let me know if I'm still missing (or failing to report) something. I'd hate to unknowingly be a bottle-neck by failing to test/report something.

> apt-get -o Dpkg::Options::="--force-confmiss" install --reinstall passt
> 
> > whereas SELinux is doing it's thing as it's supposed to.
> 
> 
> Right, that's because SELinux can do this:
> 
> https://selinuxproject.org/page/MultiCategorySecurity
> 
> with AppArmor, it needs to be "emulated", somehow.
> 
> > > We're just trying to make things as
> > > strict as possible, and depending on specific paths.
> > 
> > I see. I'm glad this approach of as-strict-as-possible is being taken.
> > 
> > > We'll probably need to make them a bit looser for the moment being
> > > and perhaps just allow passt, no matter who starts it, to write to
> > > /var/run/**.
> > 
> > I believe user-mode virtual machines only need access to
> > /run/user/$USER and not /var/run. Not even /run/*, but only
> > /run/user/$USER. So if that work-around is to be implemented, that
> > would be the strictest version of it: each user-started passt process
> > gets access to $XDG_RUNTIME_DIR of it's owner (and not outside of it).
> 
> 
> It depends, because if you start passt as root, socket and PID go to
> @{run}/libvirt/qemu/passt/, and if you don't, the location becomes
> @{run}/user/[0-9]/libvirt/qemu/run/passt/*.
> 
> @{run} is an AppArmor handle for /var/run and /run, I think (aren't
> they generally linked anyway?). Note that Ubuntu and openSUSE might
> use slightly different paths.
> 

/var/run deems to be a symbolic link to /run which is a tmpfs mount on my Debian Trixie machine.

> > It also seems that more and more of us use $XDG_RUNTIME_DIR in lieu
> > of /tmp in our personal shell scripts, because it kinda' feels like a
> > more private /tmp.
> 
> 
> Yeah, it is.
> 
> > Also, the `passt` update fixing DNS issue hasn't yet made it to
> > Debian Trixie, yet.
> 
> 
> I didn't release the fix yet. I merged it (upstream), but actually I
> was expecting you would give it a quick try. If I'm more confident
> about the change, I can do things faster.
> 
I thought I'd already reported it working, but, as I mentioned earlier, the patch from isn't available on the master branch yet: https://archives.passt.top/passt-dev/pXatirFM4Zm6ZSKXYsfeyZnHlL8JlQDKRUNtyLqbzrUOFcItX06upuk1cPYaufL1HzqHygkWvySVhawtV8zVi8Ou9qmhz_EvuQxr46UOi3Q=@protonmail.com/

Perhaps I'm missing something.
> > I should venture to Debian Sid, myself.
> 
> 
> It's not in Debian Sid either because I didn't make a new release of
> passt, yet.
> 
> It probably makes sense to make one next week (we release quite often,
> especially if there's one or more fixes that might be important for
> somebody, such as this one). As Debian maintainer I also update Debian
> packages within a couple of hours.
> 
> Sid to testing is usually five days of difference, look:
> 
> https://tracker.debian.org/pkg/passt/news/
> 
> --
> Stefano

Re: Apparmor (and other) Issues

[Stefano Brivio]

On Sat, 08 Feb 2025 17:19:59 +0000
Prafulla Giri <prafulla.giri@protonmail.com> wrote:

> On Friday, February 7th, 2025 at 3:01 PM, Stefano Brivio <sbrivio@redhat.com> wrote:
> 
> > On Fri, 07 Feb 2025 06:49:45 +0000
> > Prafulla Giri prafulla.giri@protonmail.com wrote:
> >   
> > > On Wednesday, February 5th, 2025 at 4:01 PM, Stefano Brivio
> > > sbrivio@redhat.com wrote:
> > >   
> > > > But the libvirt profile is not associated to the
> > > > process, oops.  
> > > 
> > > Oh, so this is what is being worked upon: that Apparmor is not making
> > > the association  
> > 
> > 
> > That, I'm not sure, but at least Andrea asked openSUSE and Ubuntu
> > people for comments. I just prepared (and merged) a workaround for the
> > moment. You are Cc'ed on the patch. If you want to test it, you should
> > add this:
> > 
> > # Workaround: libvirt's profile comes with a passt subprofile which includes,
> > # in turn, <abstractions/passt>, and adds libvirt-specific rules on top, to
> > 
> > # allow passt (when started by libvirtd) to write socket and PID files in the
> > # location requested by libvirtd itself, and to execute passt itself.
> > #
> > # However, when libvirt runs as unprivileged user, the mechanism based on
> > # virt-aa-helper, designed to build per-VM profiles as guests are started,
> > # doesn't work. The helper needs to create and load profiles on the fly, which
> > # can't be done by unprivileged users, of course.
> > #
> > # As a result, libvirtd runs unconfined if guests are started by unprivileged
> > # users, starting passt unconfined as well, which means that passt runs under
> > # its own stand-alone profile (this one), which implies in turn that execve()
> > # of /usr/bin/passt is not allowed, and socket and PID files can't be written.
> > #
> > # Duplicate libvirt-specific rules here as long as this is not solved in
> > # libvirt's profile itself.
> > /usr/bin/passt r,
> > owner @{run}/user/[0-9]/libvirt/qemu/run/passt/ rw,
> > owner @{run}/libvirt/qemu/passt/* rw,
> > 
> > to your /etc/apparmor.d/usr.bin.passt. Note that changes to AppArmor
> > policy files are retained as configuration, so, if you edit it, package
> > upgrades won't override things automatically. You will need to:
> 
> I seem to have botched things up really good, or we're getting into more and more trouble here:

Worry not, I can explain:

> 1. I have manually `make install`-ed passt (and friends).
> $ passt version # I don't know what's causing the non-AVX2 thing issue

If you install it to /usr/local/bin, other than adding a profile for
/usr/local/bin/passt{,.avx2} as you correctly did, you also need to add
the /usr/local/bin path to the "abstraction" that's included by the
profile. In /etc/apparmor.d/abstractions/passt we have:

  /usr/bin/passt.avx2                   ix,     # arch_avx2_exec(), arch.c

and if you want to do experiments with a local version that needs to be:

  /usr/bin/passt.avx2                   ix,     # arch_avx2_exec(), arch.c
  /usr/bin/local/passt.avx2             ix,

> [...]
>
> If, however, I uninstall the manually made version (with the apparmor profile back to pointing to /usr/bin/passt):
> 
> $ passt --version
> passt 0.0~git20250121.4f2c8e7-1
> Copyright Red Hat
> GNU General Public License, version 2 or later
>   <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> 
> $ virsh start --domain vm1 
> Domain 'vm1' started
> 
> So I figure it's okay after all.

Good, thanks for checking! That patch is already applied by the way. As
I mentioned, I'll make a new release (and Debian packages) in a few
days.

> I am surprised, the upstream tree does not seem to contain the recent DNS related patch that I shazamed and tested just a couple of days earlier ( b4 shazam https://archives.passt.top/passt-dev/20250203082210.2114348-1-sbrivio@redhat.com/ ). I have just emailed a reply at the patch's own thread, too: https://archives.passt.top/passt-dev/pXatirFM4Zm6ZSKXYsfeyZnHlL8JlQDKRUNtyLqbzrUOFcItX06upuk1cPYaufL1HzqHygkWvySVhawtV8zVi8Ou9qmhz_EvuQxr46UOi3Q=@protonmail.com/

Right, I thought I applied it, but now I remember: I wasn't entirely
sure it fixed your issue, and I was waiting for reviews as well, then I
forgot about it. It's merged now.

> Please let me know if I'm still missing (or failing to report) something. I'd hate to unknowingly be a bottle-neck by failing to test/report something.

No worries, you're definitely not a bottleneck, my mind sometimes is.
Thanks for the all the tests!

> > > I believe user-mode virtual machines only need access to
> > > /run/user/$USER and not /var/run. Not even /run/*, but only
> > > /run/user/$USER. So if that work-around is to be implemented, that
> > > would be the strictest version of it: each user-started passt process
> > > gets access to $XDG_RUNTIME_DIR of it's owner (and not outside of it).  
> > 
> > It depends, because if you start passt as root, socket and PID go to
> > @{run}/libvirt/qemu/passt/, and if you don't, the location becomes
> > @{run}/user/[0-9]/libvirt/qemu/run/passt/*.
> > 
> > @{run} is an AppArmor handle for /var/run and /run, I think (aren't
> > they generally linked anyway?). Note that Ubuntu and openSUSE might
> > use slightly different paths.
> 
> /var/run deems to be a symbolic link to /run which is a tmpfs mount on my Debian Trixie machine.

Right, yes, hence that @{run} alias that covers whatever you have on
different distributions or different versions.

-- 
Stefano

Re: Apparmor (and other) Issues

[Prafulla Giri]

Hello there,

Please forgive me for being MIA.

On Sunday, February 9th, 2025 at 2:53 PM, Stefano Brivio <sbrivio@redhat.com> wrote:

> On Sat, 08 Feb 2025 17:19:59 +0000
> Prafulla Giri prafulla.giri@protonmail.com wrote:
> 
> > On Friday, February 7th, 2025 at 3:01 PM, Stefano Brivio sbrivio@redhat.com wrote:
> > 
> > > On Fri, 07 Feb 2025 06:49:45 +0000
> > > Prafulla Giri prafulla.giri@protonmail.com wrote:
> > > 
> > > > On Wednesday, February 5th, 2025 at 4:01 PM, Stefano Brivio
> > > > sbrivio@redhat.com wrote:
> > > > 
> > > > > But the libvirt profile is not associated to the
> > > > > process, oops.
> > > > 
> > > > Oh, so this is what is being worked upon: that Apparmor is not making
> > > > the association
> > > 
> > > That, I'm not sure, but at least Andrea asked openSUSE and Ubuntu
> > > people for comments. I just prepared (and merged) a workaround for the
> > > moment. You are Cc'ed on the patch. If you want to test it, you should
> > > add this:
> > > 
> > > # Workaround: libvirt's profile comes with a passt subprofile which includes,
> > > # in turn, <abstractions/passt>, and adds libvirt-specific rules on top, to
> > > 
> > > # allow passt (when started by libvirtd) to write socket and PID files in the
> > > # location requested by libvirtd itself, and to execute passt itself.
> > > #
> > > # However, when libvirt runs as unprivileged user, the mechanism based on
> > > # virt-aa-helper, designed to build per-VM profiles as guests are started,
> > > # doesn't work. The helper needs to create and load profiles on the fly, which
> > > # can't be done by unprivileged users, of course.
> > > #
> > > # As a result, libvirtd runs unconfined if guests are started by unprivileged
> > > # users, starting passt unconfined as well, which means that passt runs under
> > > # its own stand-alone profile (this one), which implies in turn that execve()
> > > # of /usr/bin/passt is not allowed, and socket and PID files can't be written.
> > > #
> > > # Duplicate libvirt-specific rules here as long as this is not solved in
> > > # libvirt's profile itself.
> > > /usr/bin/passt r,
> > > owner @{run}/user/[0-9]/libvirt/qemu/run/passt/ rw,
> > > owner @{run}/libvirt/qemu/passt/* rw,
> > > 
> > > to your /etc/apparmor.d/usr.bin.passt. Note that changes to AppArmor
> > > policy files are retained as configuration, so, if you edit it, package
> > > upgrades won't override things automatically. You will need to:
> > 
> > I seem to have botched things up really good, or we're getting into more and more trouble here:
> 
> 
> Worry not, I can explain:

You are a very good teacher, I must say.
> 
> > 1. I have manually `make install`-ed passt (and friends).
> > $ passt version # I don't know what's causing the non-AVX2 thing issue
> 
> 
> If you install it to /usr/local/bin, other than adding a profile for
> /usr/local/bin/passt{,.avx2} as you correctly did, you also need to add
> the /usr/local/bin path to the "abstraction" that's included by the
> profile. In /etc/apparmor.d/abstractions/passt we have:
> 
> /usr/bin/passt.avx2 ix, # arch_avx2_exec(), arch.c
> 
> and if you want to do experiments with a local version that needs to be:
> 
> /usr/bin/passt.avx2 ix, # arch_avx2_exec(), arch.c
> /usr/bin/local/passt.avx2 ix,
> 

I did just that and
$ pasta --config-net --trace --pcap /tmp/dns.pcap -- nslookup fsf.org # `which pasta` -> /usr/local/bin/pasta
works as expected, thank you.

However, for some reason libvirt still can't run pasta.

$ virsh start --domain vm1
rror: Failed to start domain 'vm1'
error: internal error: Child process (passt --one-off --socket /run/user/1000/libvirt/qemu/run/passt/2-vm1-net0.socket --pid /run/user/1000/libvirt/qemu/run/passt/2-vm1-net0-passt.pid) unexpected exit status 126: libvirt:  error : cannot execute binary passt: Permission denied

It seems adding merely '/usr/bin/local/passt.avx2 ix,' in the abstractions file isn't quite enough. Perhaps there's something missing? Strangely enough, the libvirt-qemu abstraction file does import this abstraction file.

> > [...]
> > 
> > If, however, I uninstall the manually made version (with the apparmor profile back to pointing to /usr/bin/passt):
> > 
> > $ passt --version
> > passt 0.0~git20250121.4f2c8e7-1
> > Copyright Red Hat
> > GNU General Public License, version 2 or later
> > https://www.gnu.org/licenses/old-licenses/gpl-2.0.html
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law.
> > 
> > $ virsh start --domain vm1
> > Domain 'vm1' started
> > 
> > So I figure it's okay after all.
> 
> 
> Good, thanks for checking! That patch is already applied by the way. As
> I mentioned, I'll make a new release (and Debian packages) in a few
> days.
> 
> > I am surprised, the upstream tree does not seem to contain the recent DNS related patch that I shazamed and tested just a couple of days earlier ( b4 shazam https://archives.passt.top/passt-dev/20250203082210.2114348-1-sbrivio@redhat.com/ ). I have just emailed a reply at the patch's own thread, too: https://archives.passt.top/passt-dev/pXatirFM4Zm6ZSKXYsfeyZnHlL8JlQDKRUNtyLqbzrUOFcItX06upuk1cPYaufL1HzqHygkWvySVhawtV8zVi8Ou9qmhz_EvuQxr46UOi3Q=@protonmail.com/
> 
> 
> Right, I thought I applied it, but now I remember: I wasn't entirely
> sure it fixed your issue, and I was waiting for reviews as well, then I
> forgot about it. It's merged now.
> 
> > Please let me know if I'm still missing (or failing to report) something. I'd hate to unknowingly be a bottle-neck by failing to test/report something.
> 
> 
> No worries, you're definitely not a bottleneck, my mind sometimes is.
> Thanks for the all the tests!
> 
> > > > I believe user-mode virtual machines only need access to
> > > > /run/user/$USER and not /var/run. Not even /run/*, but only
> > > > /run/user/$USER. So if that work-around is to be implemented, that
> > > > would be the strictest version of it: each user-started passt process
> > > > gets access to $XDG_RUNTIME_DIR of it's owner (and not outside of it).
> > > 
> > > It depends, because if you start passt as root, socket and PID go to
> > > @{run}/libvirt/qemu/passt/, and if you don't, the location becomes
> > > @{run}/user/[0-9]/libvirt/qemu/run/passt/*.
> > > 
> > > @{run} is an AppArmor handle for /var/run and /run, I think (aren't
> > > they generally linked anyway?). Note that Ubuntu and openSUSE might
> > > use slightly different paths.
> > 
> > /var/run deems to be a symbolic link to /run which is a tmpfs mount on my Debian Trixie machine.
> 
> 
> Right, yes, hence that @{run} alias that covers whatever you have on
> different distributions or different versions.
> 
> --
> Stefano

Re: Apparmor (and other) Issues

[Stefano Brivio]

On Mon, 17 Feb 2025 06:37:18 +0000
Prafulla Giri <prafulla.giri@protonmail.com> wrote:

> Hello there,
> 
> Please forgive me for being MIA.
> 
> On Sunday, February 9th, 2025 at 2:53 PM, Stefano Brivio <sbrivio@redhat.com> wrote:
> 
> > On Sat, 08 Feb 2025 17:19:59 +0000
> > Prafulla Giri prafulla.giri@protonmail.com wrote:
> >   
> > > On Friday, February 7th, 2025 at 3:01 PM, Stefano Brivio sbrivio@redhat.com wrote:
> > >   
> > > > On Fri, 07 Feb 2025 06:49:45 +0000
> > > > Prafulla Giri prafulla.giri@protonmail.com wrote:
> > > >   
> > > > > On Wednesday, February 5th, 2025 at 4:01 PM, Stefano Brivio
> > > > > sbrivio@redhat.com wrote:
> > > > >   
> > > > > > But the libvirt profile is not associated to the
> > > > > > process, oops.  
> > > > > 
> > > > > Oh, so this is what is being worked upon: that Apparmor is not making
> > > > > the association  
> > > > 
> > > > That, I'm not sure, but at least Andrea asked openSUSE and Ubuntu
> > > > people for comments. I just prepared (and merged) a workaround for the
> > > > moment. You are Cc'ed on the patch. If you want to test it, you should
> > > > add this:
> > > > 
> > > > # Workaround: libvirt's profile comes with a passt subprofile which includes,
> > > > # in turn, <abstractions/passt>, and adds libvirt-specific rules on top, to
> > > > 
> > > > # allow passt (when started by libvirtd) to write socket and PID files in the
> > > > # location requested by libvirtd itself, and to execute passt itself.
> > > > #
> > > > # However, when libvirt runs as unprivileged user, the mechanism based on
> > > > # virt-aa-helper, designed to build per-VM profiles as guests are started,
> > > > # doesn't work. The helper needs to create and load profiles on the fly, which
> > > > # can't be done by unprivileged users, of course.
> > > > #
> > > > # As a result, libvirtd runs unconfined if guests are started by unprivileged
> > > > # users, starting passt unconfined as well, which means that passt runs under
> > > > # its own stand-alone profile (this one), which implies in turn that execve()
> > > > # of /usr/bin/passt is not allowed, and socket and PID files can't be written.
> > > > #
> > > > # Duplicate libvirt-specific rules here as long as this is not solved in
> > > > # libvirt's profile itself.
> > > > /usr/bin/passt r,
> > > > owner @{run}/user/[0-9]/libvirt/qemu/run/passt/ rw,
> > > > owner @{run}/libvirt/qemu/passt/* rw,
> > > > 
> > > > to your /etc/apparmor.d/usr.bin.passt. Note that changes to AppArmor
> > > > policy files are retained as configuration, so, if you edit it, package
> > > > upgrades won't override things automatically. You will need to:  
> > > 
> > > I seem to have botched things up really good, or we're getting into more and more trouble here:  
> > 
> > 
> > Worry not, I can explain:  
> 
> You are a very good teacher, I must say.
> >   
> > > 1. I have manually `make install`-ed passt (and friends).
> > > $ passt version # I don't know what's causing the non-AVX2 thing issue  
> > 
> > 
> > If you install it to /usr/local/bin, other than adding a profile for
> > /usr/local/bin/passt{,.avx2} as you correctly did, you also need to add
> > the /usr/local/bin path to the "abstraction" that's included by the
> > profile. In /etc/apparmor.d/abstractions/passt we have:
> > 
> > /usr/bin/passt.avx2 ix, # arch_avx2_exec(), arch.c
> > 
> > and if you want to do experiments with a local version that needs to be:
> > 
> > /usr/bin/passt.avx2 ix, # arch_avx2_exec(), arch.c
> > /usr/bin/local/passt.avx2 ix,
> 
> I did just that and
> $ pasta --config-net --trace --pcap /tmp/dns.pcap -- nslookup fsf.org # `which pasta` -> /usr/local/bin/pasta
> works as expected, thank you.
> 
> However, for some reason libvirt still can't run pasta.

You mean 'passt', and:

> $ virsh start --domain vm1
> rror: Failed to start domain 'vm1'
> error: internal error: Child process (passt --one-off --socket /run/user/1000/libvirt/qemu/run/passt/2-vm1-net0.socket --pid /run/user/1000/libvirt/qemu/run/passt/2-vm1-net0-passt.pid) unexpected exit status 126: libvirt:  error : cannot execute binary passt: Permission denied
> 
> It seems adding merely '/usr/bin/local/passt.avx2 ix,' in the abstractions file isn't quite enough. Perhaps there's something missing? Strangely enough, the libvirt-qemu abstraction file does import this abstraction file.

https://salsa.debian.org/sbrivio/passt/-/commit/5bb812e79143670a57440cd8aa7f2979583c5a0a

might explain it. You need to create hard links ('make install' doesn't
do that) to associate different AppArmor profiles.

In any case, I'm releasing (and packaging) a new version with the
AppArmor workaround today.

-- 
Stefano

Re: Apparmor (and other) Issues

[Prafulla Giri]

On Monday, February 17th, 2025 at 1:22 PM, Stefano Brivio <sbrivio@redhat.com> wrote:

> 
> You mean 'passt', and:
> 
Strangely enough, I did mean pasta: that's the one that gives a shell. passt only creates namespace thingy. I thought pasta used passt underneath and that is why changes to passt was visible by testing pasta. Am I doing something wrong?

> 
> https://salsa.debian.org/sbrivio/passt/-/commit/5bb812e79143670a57440cd8aa7f2979583c5a0a
> 
> might explain it. You need to create hard links ('make install' doesn't
> do that) to associate different AppArmor profiles.
> 
> In any case, I'm releasing (and packaging) a new version with the
> AppArmor workaround today.
> 
I just checked on Debian Sid and I can confirm that everything is working as expected. Thank you very much for your hard work. As I understand it, a better Apparmor fix is being discussed with other maintainers in the meantime. But as things stand right now, Debian users ought to be able to use passt with libvirt, as expected.

In the meantime, I have noticed another error and want to ask where I ought to report it: If a VM isn't able to run, passt configs aren't cleared. I just had a VM not start because of permission errors (I resolved it), but trying to restart the VM threw a passt error saying the port being forwarded was already in use (a remnant of previous run that failed). I'll have to come up with a way to make this reproducible, of course. But it seems that on unclean exits, passt isn't being allowed to clean things up (destroy the created namespace, stop the port-forwards, etc.) Perhaps I ought to report this on libvirt side? I encountered this error through virt-manager. Perhaps I should test with virsh as well.

Re: Apparmor (and other) Issues

[Stefano Brivio]

On Wed, 19 Feb 2025 06:31:49 +0000
Prafulla Giri <prafulla.giri@protonmail.com> wrote:

> On Monday, February 17th, 2025 at 1:22 PM, Stefano Brivio <sbrivio@redhat.com> wrote:
> 
> > You mean 'passt', and:
> >   
> Strangely enough, I did mean pasta: that's the one that gives a shell. passt only creates namespace thingy. I thought pasta used passt underneath and that is why changes to passt was visible by testing pasta. Am I doing something wrong?

They are the same binary. Simplistically: one (pasta) gives you a shell
but you can also use it with Podman (or Docker), the other one (passt)
gives you a UNIX domain socket and you can use it with QEMU (or
libkrun/muvm).

They just need to be invoked as different commands, so a symlink would
normally be enough, except that AppArmor profiles can't be (separately)
associated to symlinks, so the Debian and openSUSE packages install a
hard link (and Fedora packages a copy).

> > https://salsa.debian.org/sbrivio/passt/-/commit/5bb812e79143670a57440cd8aa7f2979583c5a0a
> > 
> > might explain it. You need to create hard links ('make install' doesn't
> > do that) to associate different AppArmor profiles.
> > 
> > In any case, I'm releasing (and packaging) a new version with the
> > AppArmor workaround today.
> >   
> I just checked on Debian Sid and I can confirm that everything is working as expected. Thank you very much for your hard work. As I understand it, a better Apparmor fix is being discussed with other maintainers in the meantime. But as things stand right now, Debian users ought to be able to use passt with libvirt, as expected.
> 
> In the meantime, I have noticed another error and want to ask where I ought to report it: If a VM isn't able to run, passt configs aren't cleared. I just had a VM not start because of permission errors (I resolved it), but trying to restart the VM threw a passt error saying the port being forwarded was already in use (a remnant of previous run that failed). I'll have to come up with a way to make this reproducible, of course. But it seems that on unclean exits, passt isn't being allowed to clean things up (destroy the created namespace, stop the port-forwards, etc.) Perhaps I ought to report this on libvirt side? I encountered this error through virt-manager. Perhaps I should test with virsh as well.

There's no persistent configuration stuff left around: if you run
stand-alone pasta, the detached namespace will go away on its own.

The sockets passt(1) creates remain until libvirt cleans them up (passt
can't do that because it remounts its root filesystem to an empty
filesystem as it starts).

Bound ports, you might need to wait up to two minutes after sockets are
closed, because they will be in TIME_WAIT state for that time. That
comes from the definition of MSL (Maixmum Segment Lifetime, RFC 9293
section 4.). The kernel wants to make sure that in-flight TCP segments
don't reach another process by mistake.

-- 
Stefano

Re: Apparmor (and other) Issues

[Prafulla Giri]

On Wednesday, February 19th, 2025 at 4:32 PM, Stefano Brivio <sbrivio@redhat.com> wrote:

> 
> There's no persistent configuration stuff left around: if you run
> stand-alone pasta, the detached namespace will go away on its own.
> 
> The sockets passt(1) creates remain until libvirt cleans them up (passt
> can't do that because it remounts its root filesystem to an empty
> filesystem as it starts).
> 
> Bound ports, you might need to wait up to two minutes after sockets are
> closed, because they will be in TIME_WAIT state for that time. That
> comes from the definition of MSL (Maixmum Segment Lifetime, RFC 9293
> section 4.). The kernel wants to make sure that in-flight TCP segments
> don't reach another process by mistake.
>

I see. That makes sense. So it must have been just a case of me not waiting long enough.

Thank you for all of this. And please don't hesitate to CC me in any issue where you need things to be tested on Debian. I'll be happy to help out!

P. S: Is there a way for me to only receive emails that I'm CC'ed into in the mailing list? I don't want to unsubscribe, but only receive a small volume of mails. I couldn't find anything about that with a quick internet search.