[PATCH] arp: only send ARP replies for --gateway address

[PATCH] arp: only send ARP replies for --gateway address

[Nikolay Edigaryev]

Problem: when passt/pasta are working in a broadcast domain with more
than one host machine, it will answer for all of these machines,
except for the one having --address. This is akin to ARP spoofing
and breaks connection with these machines if passt/pasta ARP reply
arrives before the original one.

Solution: only be responsible and send ARP replies
for the --gateway's address.
---
 arp.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arp.c b/arp.c
index a35c1b6..f873491 100644
--- a/arp.c
+++ b/arp.c
@@ -67,8 +67,8 @@ int arp(const struct ctx *c, const struct pool *p)
 	    !memcmp(am->sip, am->tip, sizeof(am->sip)))
 		return 1;
 
-	/* Don't resolve our own address, either. */
-	if (!memcmp(am->tip, &c->ip4.addr, sizeof(am->tip)))
+	/* Don't resolve anything but gateway address. */
+	if (memcmp(am->tip, &c->ip4.gw, sizeof(am->tip)) != 0)
 		return 1;
 
 	ah->ar_op = htons(ARPOP_REPLY);
-- 
@@ -67,8 +67,8 @@ int arp(const struct ctx *c, const struct pool *p)
 	    !memcmp(am->sip, am->tip, sizeof(am->sip)))
 		return 1;
 
-	/* Don't resolve our own address, either. */
-	if (!memcmp(am->tip, &c->ip4.addr, sizeof(am->tip)))
+	/* Don't resolve anything but gateway address. */
+	if (memcmp(am->tip, &c->ip4.gw, sizeof(am->tip)) != 0)
 		return 1;
 
 	ah->ar_op = htons(ARPOP_REPLY);
-- 
2.39.2 (Apple Git-144)

Re: [PATCH] arp: only send ARP replies for --gateway address

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1608 bytes --]

On Fri, Sep 15, 2023 at 06:20:45PM +0400, Nikolay Edigaryev wrote:
> Problem: when passt/pasta are working in a broadcast domain with more
> than one host machine,

Oof.  So, at present, passt/pasta is really not designed to have more
than a single machine on the "tap" side.  Changing the ARP behaviour
is likely to be the least of the problems with that setup.

I do have plans to change that so we can handle multiple logical guest
side machines, but accomplishing that is a ways off.

> it will answer for all of these machines,
> except for the one having --address. This is akin to ARP spoofing
> and breaks connection with these machines if passt/pasta ARP reply
> arrives before the original one.
> 
> Solution: only be responsible and send ARP replies
> for the --gateway's address.
> ---
>  arp.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/arp.c b/arp.c
> index a35c1b6..f873491 100644
> --- a/arp.c
> +++ b/arp.c
> @@ -67,8 +67,8 @@ int arp(const struct ctx *c, const struct pool *p)
>  	    !memcmp(am->sip, am->tip, sizeof(am->sip)))
>  		return 1;
>  
> -	/* Don't resolve our own address, either. */
> -	if (!memcmp(am->tip, &c->ip4.addr, sizeof(am->tip)))
> +	/* Don't resolve anything but gateway address. */
> +	if (memcmp(am->tip, &c->ip4.gw, sizeof(am->tip)) != 0)
>  		return 1;
>  
>  	ah->ar_op = htons(ARPOP_REPLY);

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] arp: only send ARP replies for --gateway address

[Stefano Brivio]

On Mon, 18 Sep 2023 12:26:03 +1000
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Fri, Sep 15, 2023 at 06:20:45PM +0400, Nikolay Edigaryev wrote:
> > Problem: when passt/pasta are working in a broadcast domain with more
> > than one host machine,  
> 
> Oof.  So, at present, passt/pasta is really not designed to have more
> than a single machine on the "tap" side.  Changing the ARP behaviour
> is likely to be the least of the problems with that setup.

Now I'm confused on which "side" this happens. :) Nikolay, can you
articulate the issue a bit better? Do you really have multiple *host*
machines? Does the passt process... move between them?

By the way, the only concern I have with this change is that the guest
might ignore the gateway address it's being assigned, for whatever
reason, and by just resolving "almost everything" we guarantee the
traffic goes out anyway.

If there's no other way to solve the issue you're facing, I would
rather propose to have this as an option, and perhaps have it off by
default.

-- 
Stefano

Re: [PATCH] arp: only send ARP replies for --gateway address

[Nikolay Edigaryev]

Hello Stefano, I will try to clarify:

I have a single host machine, a dedicated amd64 server, capable of
running multiple Cloud Hypervisor virtual machines backed by /dev/kvm.

I also have a daemon-less CLI software that can provision as many VM
instances as the user wants, e.g. by running "mycli create --kernel
... --disk ... ubuntu".

To run a VM, the user types "mycli run ubuntu", which results in the
creation of two TAP interfaces: one is for passt, one is for Cloud
Hypervisor

"mycli run" then creates a bridge(8) interface, assigns a free IP from
/29 network to it (for example, 10.0.0.3/29), and adds both the TAP
interfaces to that bridge forming up a virtual switch, which allows
passt <-> VM and host <-> communication.

"mycli run ubuntu" also invokes the passt with the following arguments:

>passt --foreground --address 10.0.0.2 --netmask 255.255.255.248 --gateway 10.0.0.1 --mac-addr 52:f1:18:34:28:0b -4 --mtu 1500 --tap-fd 3

Now to the issue: if the user wants to access the VM, for provisioning
purposes, e.g. by running "ssh 10.0.0.2", there's a race between the
real ARP reply from that VM and an ARP reply from passt due to the
code fixed in the patch above.

And even if we add a static ARP entry for that VM on the host, there's
still exist a race on the VM's side.

Here the VM looks up the host's ethernet address and receives one
reply from host (ba:46:4e:27:8b:93) and another from passt
(52:f1:18:34:28:0b):

17:26:42.685718 5a:b7:e3:dc:bb:9f > ba:46:4e:27:8b:93, ethertype ARP
(0x0806), length 42: Request who-has 10.0.0.3 tell 10.0.0.2, length 28
17:26:42.685744 ba:46:4e:27:8b:93 > 5a:b7:e3:dc:bb:9f, ethertype ARP
(0x0806), length 42: Reply 10.0.0.3 is-at ba:46:4e:27:8b:93, length 28
17:26:42.685908 52:f1:18:34:28:0b > 5a:b7:e3:dc:bb:9f, ethertype ARP
(0x0806), length 42: Reply 10.0.0.3 is-at 52:f1:18:34:28:0b, length 28

On Mon, Sep 18, 2023 at 6:01 PM Stefano Brivio <sbrivio@redhat.com> wrote:
>
> On Mon, 18 Sep 2023 12:26:03 +1000
> David Gibson <david@gibson.dropbear.id.au> wrote:
>
> > On Fri, Sep 15, 2023 at 06:20:45PM +0400, Nikolay Edigaryev wrote:
> > > Problem: when passt/pasta are working in a broadcast domain with more
> > > than one host machine,
> >
> > Oof.  So, at present, passt/pasta is really not designed to have more
> > than a single machine on the "tap" side.  Changing the ARP behaviour
> > is likely to be the least of the problems with that setup.
>
> Now I'm confused on which "side" this happens. :) Nikolay, can you
> articulate the issue a bit better? Do you really have multiple *host*
> machines? Does the passt process... move between them?
>
> By the way, the only concern I have with this change is that the guest
> might ignore the gateway address it's being assigned, for whatever
> reason, and by just resolving "almost everything" we guarantee the
> traffic goes out anyway.
>
> If there's no other way to solve the issue you're facing, I would
> rather propose to have this as an option, and perhaps have it off by
> default.
>
> --
> Stefano
>

Re: [PATCH] arp: only send ARP replies for --gateway address

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 5941 bytes --]

On Mon, Sep 18, 2023 at 07:52:23PM +0400, Nikolay Edigaryev wrote:
> Hello Stefano, I will try to clarify:
> 
> I have a single host machine, a dedicated amd64 server, capable of
> running multiple Cloud Hypervisor virtual machines backed by /dev/kvm.
> 
> I also have a daemon-less CLI software that can provision as many VM
> instances as the user wants, e.g. by running "mycli create --kernel
> ... --disk ... ubuntu".
> 
> To run a VM, the user types "mycli run ubuntu", which results in the
> creation of two TAP interfaces: one is for passt, one is for Cloud
> Hypervisor
> 
> "mycli run" then creates a bridge(8) interface, assigns a free IP from
> /29 network to it (for example, 10.0.0.3/29), and adds both the TAP
> interfaces to that bridge forming up a virtual switch, which allows
> passt <-> VM and host <-> communication.

Ok.  So, to check my understanding: the VM only has a single virtual
NIC, which connects to this bridge, then you're connecting the bridge
to the outside world using passt.  Is that correct?

> "mycli run ubuntu" also invokes the passt with the following arguments:
> 
> >passt --foreground --address 10.0.0.2 --netmask 255.255.255.248 --gateway 10.0.0.1 --mac-addr 52:f1:18:34:28:0b -4 --mtu 1500 --tap-fd 3

What owns the address 10.0.0.1 here?  I'm assuming that's an address
of the host, but is it on an external interface, or on this special
bridge?  Or somewhere else?

[Btw, clamping the passt mtu to 1500 is probably going to be pretty
bad for TCP throughput]

> Now to the issue: if the user wants to access the VM, for provisioning
> purposes, e.g. by running "ssh 10.0.0.2", there's a race between the
> real ARP reply from that VM and an ARP reply from passt due to the
> code fixed in the patch above.
> 
> And even if we add a static ARP entry for that VM on the host, there's
> still exist a race on the VM's side.
> 
> Here the VM looks up the host's ethernet address and receives one
> reply from host (ba:46:4e:27:8b:93) and another from passt
> (52:f1:18:34:28:0b):
> 
> 17:26:42.685718 5a:b7:e3:dc:bb:9f > ba:46:4e:27:8b:93, ethertype ARP
> (0x0806), length 42: Request who-has 10.0.0.3 tell 10.0.0.2, length 28
> 17:26:42.685744 ba:46:4e:27:8b:93 > 5a:b7:e3:dc:bb:9f, ethertype ARP
> (0x0806), length 42: Reply 10.0.0.3 is-at ba:46:4e:27:8b:93, length 28
> 17:26:42.685908 52:f1:18:34:28:0b > 5a:b7:e3:dc:bb:9f, ethertype ARP
> (0x0806), length 42: Reply 10.0.0.3 is-at 52:f1:18:34:28:0b, length 28

Right.

Ok, so Stefano mentioned that this change will break the case of a
guest not using the gateway it's supposed to.  That's true, but
there's certainly a pretty strong case that no-one has any right to
expect that case to work anyway, so we need not consider it.

I believe there's some other rare but legitimate cases it can also
break though.  For now I think these can only occur with pasta, not
passt, but they'd still be affected:

 * Although it's not common, it's possible to have a default route
   with an interface, but no gateway (this can occur if the host has
   connectivity over a point to point link like a VPN).  With pasta
   --config-net we'll copy that gateway-less default route to the
   namespace, and it will then ARP for *everything*.  That will work
   now, because we'll answer all those arps, but would not if we only
   arp the gateway address.

 * A lesser version of the same same thing: even if we have a normal
   default gateway, we may also have specific subnet routes on the
   host which override it.  With pasta --config-net again we will copy
   those routes to the namespace, and so packets routed that way will
   induce ARPs for something other than the default gateway (either
   for the destination address or for the route specific gateway).


Apart from the ARP issues, I think there's at least one other
fragility in the setup you've described.  This is what I was thinking
about when I mentioned elsewhere that I don't think ARP will be the
only issue with having a non-trivial broadcast domain on the guest
side of passt:

If from the host you to send packets on the bridge addressed to
passt's address, rather than the host, I believe that would cause
passt to update its 'addr_seen' to that of the host.  That could then
cause packets which should be going to the guest to be sent to the
host instead.  That could have a variety of effects from just a brief
interruption to essentially breaking connectivity.

> On Mon, Sep 18, 2023 at 6:01 PM Stefano Brivio <sbrivio@redhat.com> wrote:
> >
> > On Mon, 18 Sep 2023 12:26:03 +1000
> > David Gibson <david@gibson.dropbear.id.au> wrote:
> >
> > > On Fri, Sep 15, 2023 at 06:20:45PM +0400, Nikolay Edigaryev wrote:
> > > > Problem: when passt/pasta are working in a broadcast domain with more
> > > > than one host machine,
> > >
> > > Oof.  So, at present, passt/pasta is really not designed to have more
> > > than a single machine on the "tap" side.  Changing the ARP behaviour
> > > is likely to be the least of the problems with that setup.
> >
> > Now I'm confused on which "side" this happens. :) Nikolay, can you
> > articulate the issue a bit better? Do you really have multiple *host*
> > machines? Does the passt process... move between them?
> >
> > By the way, the only concern I have with this change is that the guest
> > might ignore the gateway address it's being assigned, for whatever
> > reason, and by just resolving "almost everything" we guarantee the
> > traffic goes out anyway.
> >
> > If there's no other way to solve the issue you're facing, I would
> > rather propose to have this as an option, and perhaps have it off by
> > default.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] arp: only send ARP replies for --gateway address

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 6178 bytes --]

On Tue, Sep 19, 2023 at 12:09:01PM +1000, David Gibson wrote:
> On Mon, Sep 18, 2023 at 07:52:23PM +0400, Nikolay Edigaryev wrote:
> > Hello Stefano, I will try to clarify:
> > 
> > I have a single host machine, a dedicated amd64 server, capable of
> > running multiple Cloud Hypervisor virtual machines backed by /dev/kvm.
> > 
> > I also have a daemon-less CLI software that can provision as many VM
> > instances as the user wants, e.g. by running "mycli create --kernel
> > ... --disk ... ubuntu".
> > 
> > To run a VM, the user types "mycli run ubuntu", which results in the
> > creation of two TAP interfaces: one is for passt, one is for Cloud
> > Hypervisor
> > 
> > "mycli run" then creates a bridge(8) interface, assigns a free IP from
> > /29 network to it (for example, 10.0.0.3/29), and adds both the TAP
> > interfaces to that bridge forming up a virtual switch, which allows
> > passt <-> VM and host <-> communication.
> 
> Ok.  So, to check my understanding: the VM only has a single virtual
> NIC, which connects to this bridge, then you're connecting the bridge
> to the outside world using passt.  Is that correct?
> 
> > "mycli run ubuntu" also invokes the passt with the following arguments:
> > 
> > >passt --foreground --address 10.0.0.2 --netmask 255.255.255.248 --gateway 10.0.0.1 --mac-addr 52:f1:18:34:28:0b -4 --mtu 1500 --tap-fd 3
> 
> What owns the address 10.0.0.1 here?  I'm assuming that's an address
> of the host, but is it on an external interface, or on this special
> bridge?  Or somewhere else?
> 
> [Btw, clamping the passt mtu to 1500 is probably going to be pretty
> bad for TCP throughput]
> 
> > Now to the issue: if the user wants to access the VM, for provisioning
> > purposes, e.g. by running "ssh 10.0.0.2", there's a race between the
> > real ARP reply from that VM and an ARP reply from passt due to the
> > code fixed in the patch above.
> > 
> > And even if we add a static ARP entry for that VM on the host, there's
> > still exist a race on the VM's side.
> > 
> > Here the VM looks up the host's ethernet address and receives one
> > reply from host (ba:46:4e:27:8b:93) and another from passt
> > (52:f1:18:34:28:0b):
> > 
> > 17:26:42.685718 5a:b7:e3:dc:bb:9f > ba:46:4e:27:8b:93, ethertype ARP
> > (0x0806), length 42: Request who-has 10.0.0.3 tell 10.0.0.2, length 28
> > 17:26:42.685744 ba:46:4e:27:8b:93 > 5a:b7:e3:dc:bb:9f, ethertype ARP
> > (0x0806), length 42: Reply 10.0.0.3 is-at ba:46:4e:27:8b:93, length 28
> > 17:26:42.685908 52:f1:18:34:28:0b > 5a:b7:e3:dc:bb:9f, ethertype ARP
> > (0x0806), length 42: Reply 10.0.0.3 is-at 52:f1:18:34:28:0b, length 28
> 
> Right.
> 
> Ok, so Stefano mentioned that this change will break the case of a
> guest not using the gateway it's supposed to.  That's true, but
> there's certainly a pretty strong case that no-one has any right to
> expect that case to work anyway, so we need not consider it.
> 
> I believe there's some other rare but legitimate cases it can also
> break though.  For now I think these can only occur with pasta, not
> passt, but they'd still be affected:
> 
>  * Although it's not common, it's possible to have a default route
>    with an interface, but no gateway (this can occur if the host has
>    connectivity over a point to point link like a VPN).  With pasta
>    --config-net we'll copy that gateway-less default route to the
>    namespace, and it will then ARP for *everything*.  That will work
>    now, because we'll answer all those arps, but would not if we only
>    arp the gateway address.
> 
>  * A lesser version of the same same thing: even if we have a normal
>    default gateway, we may also have specific subnet routes on the
>    host which override it.  With pasta --config-net again we will copy
>    those routes to the namespace, and so packets routed that way will
>    induce ARPs for something other than the default gateway (either
>    for the destination address or for the route specific gateway).
> 
> 
> Apart from the ARP issues, I think there's at least one other
> fragility in the setup you've described.  This is what I was thinking
> about when I mentioned elsewhere that I don't think ARP will be the
> only issue with having a non-trivial broadcast domain on the guest
> side of passt:
> 
> If from the host you to send packets on the bridge addressed to
> passt's address, rather than the host, I believe that would cause
> passt to update its 'addr_seen' to that of the host.  That could then
> cause packets which should be going to the guest to be sent to the
> host instead.  That could have a variety of effects from just a brief
> interruption to essentially breaking connectivity.


To summarise where we're at with this issue (including some points
Stefano and I discussed elsewhere):

Because of the case described above (default route with no gateway),
we're not going to apply this patch for the time being

Of course, we'd like to support cloud-hypervisor and, eventually more
general broadcast domains behind passt.  However, while the change
here might be sufficient for the specific case, it's extremely fragile:

At present, passt/pasta expects only a single addressable machine to
be on its guest side, and not just in the handling of ARP.  For
example if anything in the broadcast domain other than the expected
guest contacted passt for any reason, the 'ip[46]_seen' variables
would be update causing further traffic from passt to be misdirected.

Fixing this robustly requires substantial changes to how we keep track
of what addresses exist on the guest side.  We're working on that, for
this reason amongst others, but it's going to be a while before it's
ready.

In the shorter term, I think the most likely way forward for clh is to
only use passt in configurations where the guest side broadcast domain
has nothing on it, except for a single VM and passt.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]