Re: qemu couldn't connect the unix domain socket

[Li Feng <fengli@smartx.com>]

[-- Attachment #1: Type: text/plain, Size: 5791 bytes --]

On Fri, Oct 29, 2021 at 3:44 PM Stefano Brivio <sbrivio(a)redhat.com> wrote:
>
> Hi Feng Li,
>
> On Fri, 29 Oct 2021 13:27:44 +0800
> Li Feng <fengli(a)smartx.com> wrote:
>
> > Hi Stefano,
> >
> > I got the coredump file, it reports the `fork` syscall is bad:
> >
> > Program terminated with signal SIGSYS, Bad system call.
> > #0  __GI__Fork () at ../sysdeps/nptl/_Fork.c:50
> > 50   return pid;
> > (gdb) bt
> > #0  __GI__Fork () at ../sysdeps/nptl/_Fork.c:50
> > #1  0x00007f8c04fdc02a in __libc_fork () at fork.c:73
> > #2  0x00007f8c05009f8b in daemon (nochdir=0, noclose=0) at daemon.c:48
> > #3  0x000000000040c1e9 in main (argc=1, argv=0x7ffd10b5cd78) at passt.c:368
> > quit)
>
> That's not necessarily because of fork() -- fork() is already in the
> list of allowed syscalls. The signal is asynchronous, it might be
> received a bit before or after passt is executing what you see in gdb.
>
> This is probably another syscall triggered by daemon() in the specific
> glibc version (2.34-7.fc35) on your system -- I haven't tested Fedora
> 35 yet. An easy way to find out which one is the syscall causing this
> is using strace.
>
> For example, suppose I forgot to add listen() to the list of allowed
> syscalls:
>
> diff --git a/passt.c b/passt.c
> index 6436a45..43249cf 100644
> --- a/passt.c
> +++ b/passt.c
> @@ -277,3 +277,3 @@ static void pid_file(struct ctx *c) {
>   * #syscalls read write open close fork dup2 exit chdir ioctl writev syslog
> - * #syscalls prlimit64 epoll_ctl epoll_create1 epoll_wait accept4 accept listen
> + * #syscalls prlimit64 epoll_ctl epoll_create1 epoll_wait accept4 accept
>   * #syscalls socket bind connect getsockopt setsockopt recvfrom sendto shutdown
>
> Then:
>
> $ strace ./passt
> [...]
> setsockopt(6, SOL_SOCKET, SO_SNDBUF, [1073741823], 4) = 0
> getsockopt(6, SOL_SOCKET, SO_SNDBUF, [268435456], [4]) = 0
> setsockopt(6, SOL_SOCKET, SO_RCVBUF, [1073741823], 4) = 0
> getsockopt(6, SOL_SOCKET, SO_RCVBUF, [268435456], [4]) = 0
> close(6)                                = 0
> socket(AF_UNIX, SOCK_STREAM, 0)         = 6
> socket(AF_UNIX, SOCK_STREAM|SOCK_NONBLOCK, 0) = 7
> connect(7, {sa_family=AF_UNIX, sun_path="/tmp/passt_1.socket"}, 110) = -1 ENOENT (No such file or directory)
> close(7)                                = 0
> unlink("/tmp/passt_1.socket")           = -1 ENOENT (No such file or directory)
> bind(6, {sa_family=AF_UNIX, sun_path="/tmp/passt_1.socket"}, 110) = 0
> write(2, "UNIX domain socket bound at /tmp"..., 48UNIX domain socket bound at /tmp/passt_1.socket
> ) = 48
> write(2, "\n", 1
> )                       = 1
> listen(6, 0)                            = ?
> +++ killed by SIGSYS +++
> Bad system call
>
> you would see that listen() is the first syscall not returning here
> (strace can't see a return from there).
>
> It's around daemon(), and the process might have forked already, so you
> should run strace with the -f option, which also traces child processes:
>
>         strace -f ./passt
>
> the missing syscall should now be obvious from the output.
Thanks for the detailed explanation.
I finally found out that the `qrap` was the root cause.
I patched the qemu, and it works well.

In VM，I got the ip 192.168.64.217, which is the same to host.

This is in VM:
root(a)192.168.64.217 08:42:35 /tmp $ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel
state UP group default qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
    altname enp0s2
    inet 192.168.64.217/20 brd 192.168.79.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::9fa9:7232:8d10:96be/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

I have tested the ping and curl, it works, amazing!

>
> > Looks like the seccomp is still badly configured.
> > I have little knowledge about the seccomp.
>
> Short summary: this is seccomp in filter mode (seccomp-bpf), it's a
> mechanism to block the system call (terminating the process, here) in
> case it's a syscall we didn't expect to be executed.
>
> It's a security feature: that's to avoid that an attacker, who already
> gained some control on the process execution, is able to potentially
> exploit a further vulnerability (e.g. in the kernel) by executing a
> particular syscall. This is a relatively famous example of it:
>
>         https://reverse.put.as/2017/11/07/exploiting-cve-2017-5123/
>
> passt implements this as a list of syscalls in code comments, those are
> translated by seccomp.sh into a BPF program, which is then loaded by
> seccomp() in passt.c. If a syscall not included in the resulting list
> is triggered, the kernel will terminate the process with a SYGSYS
> signal.
>
> However, different C libraries (on different architectures) might issue
> different syscalls to implement the same function (daemon(), here), and
> the list I made was just tested on the systems I use and the reports of
> a few other users, so some are surely missing right now.
>
> While adding tests for OpenSUSE and Debian, I already found a few
> alternative syscalls for some functions (I'll prepare a patch soon) --
> I haven't started with Fedora 35 tests yet.
>
This background knowledge helped me a lot.
The seccomp works well with Fedora 35 without the `qrap`.
Thanks again for your great work.

> --
> Stefano
>