Re: Pasta-networked rootless Podman container gets Connection Refused with the host's public IP

Re: Pasta-networked rootless Podman container gets Connection Refused with the host's public IP

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3910 bytes --]

On Wed, Jun 28, 2023 at 11:02:55AM +0300, jklaiho@iki.fi wrote:
> Hi; I previously asked this on the Podman mailing list, but I'm not
> sure if the issue in question is a feature of Podman or Passt (or
> both), and I got no replies from the Podman list, so I figured I'd
> try here as well.

This behaviour is a property of passt.  It's a consequence of a
tradeoff that we make differently from slirp or kernel masquerading
approaches.

Pasta (usually) avoids NAT, which can avoid a number of problems, but
the way it does this is by giving the container the host's IP address
(or one of them, if the host has multiple).  The tradeoff is that that
implies the container can't contact the host by that IP address.

> We're running some rootless Podman containers set up to use Pasta
> 2023_03_29.b10b983 for networking. One of the containers needs to
> access the host machine port 443 with its public IP address, but
> this causes a Connection Refused error. Any other public IP is
> accessible normally.

Right, this is expected.  Because the container itself also has that
IP, it will route that traffic to itself.  It will never even reach
pasta, let alone the host.  I'm assuming there's no server on that
port in the container, hence, connection refused.

> This is specific to the containers; the host has no problem
> accessing itself with the public IP.
> 
> The containers are set up with systemd generators (quadlet), with networking configured very simply:
> 
> "Network=pasta:-t,auto,-T,auto"
> 
> Podman has a --map-gw option useable with Pasta that seemed like it
> might help, but it didn't.
> 
> "Network=pasta:--map-gw,-t,auto,-T,auto" fails like this at container startup:
> 
> Error: failed to start pasta:
> Port forwarding mode 'none' conflicts with previous mode
> 
> "Network=pasta:-t,auto,-T,auto,--map-gw" started the container fine,
> but did not fix the Connection Refused error. Apparently --map-gw
> just isn't the right option here.

It's odd that those last two approaches gave different results, AFAIK
just changing the order of options shouldn't make a difference here.

But in any case, --map-gw (which from pasta's point of view is
removing the --no-map-gw option) will not do what you want for two
reasons.
   1. map-gw does provide a means for the container to access the
      host, but it's not via the host's normal public IP (that's
      impossible if the guest has it).  Instead it repurposes the IP
      of the default gateway to refer to the host when used from the
      container.  So to use this you'd need to change the address that
      your clients in the container use, which I gather isn't
      possible.

   2. With map-gw, traffic forwaded appears on the host side to both
      come from and go to the loopback interface.  That is from the
      servier's point of view the connection will go to 127.0.0.1, not
      the host's public IP.  AIUI that won't work for your situation.

> I don't know if the inability to contact the public IP is a feature
> of Podman or Pasta, but I'm hoping you're able to at least narrow it
> down for me.
> 
> Is there a workaround on the Pasta side?

Maybe.  You can add the -a <address> option to the pasta command line
which will tell it to assign the given address to the guest instead of
the host's address.  This should make the host contactable using it's
public IP.  However, it may cause other issues, since the container's
IP as it sees itself will no longer be the same as the container's IP
as things outside see it.  You'll obviously also have to pick an
address which won't conflict with anything else the guest needs to
contact.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Pasta-networked rootless Podman container gets Connection Refused with the host's public IP

[Stefano Brivio]

Hi JK,

On Wed, 28 Jun 2023 11:02:55 +0300
"jklaiho@iki.fi" <jklaiho@iki.fi> wrote:

> Hi; I previously asked this on the Podman mailing list, but I'm not
> sure if the issue in question is a feature of Podman or Passt (or
> both), and I got no replies from the Podman list, so I figured I'd
> try here as well.

I actually saw an answer to that, did you miss this perhaps?

  https://lists.podman.io/archives/list/podman@lists.podman.io/message/AAL6OMIB2CD4RHEXHDAAWZM4PYJNZXB2/

Anyway, in more detail:

> We're running some rootless Podman containers set up to use Pasta
> 2023_03_29.b10b983 for networking. One of the containers needs to
> access the host machine port 443 with its public IP address, but this
> causes a Connection Refused error. Any other public IP is accessible
> normally.
> 
> This is specific to the containers; the host has no problem accessing
> itself with the public IP.
> 
> The containers are set up with systemd generators (quadlet), with
> networking configured very simply:
> 
> "Network=pasta:-t,auto,-T,auto"
> 
> Podman has a --map-gw option useable with Pasta that seemed like it
> might help, but it didn't.

That would help you (minus the issue you hit) if you wanted to connect
to the parent network namespace ("host") using another address, namely
the address of the default gateway (only option at the moment, we plan
to make it configurable) -- but not the public address.

> "Network=pasta:--map-gw,-t,auto,-T,auto" fails like this at container
> startup:
> 
> Error: failed to start pasta:
> Port forwarding mode 'none' conflicts with previous mode

The problem here comes from the fact that the Podman integration
already passes "-t none" by default, and the override here doesn't
quite work, because pasta doesn't accept overriding options on its
command line. Paul (Cc'ed) plans to send a patchset to fix this in
pasta itself.

> "Network=pasta:-t,auto,-T,auto,--map-gw" started the container fine,
> but did not fix the Connection Refused error. Apparently --map-gw
> just isn't the right option here.

"-T auto" is yet a different thing: that would map ports from your
container to the "host" via loopback addresses (127.0.0.1 or ::1).

> I don't know if the inability to contact the public IP is a feature
> of Podman or Pasta, but I'm hoping you're able to at least narrow it
> down for me.
> 
> Is there a workaround on the Pasta side?

I think the issue you're facing is that, by default, your container
gets the same set of addresses as _one interface_ (the first one with
a default route) on the host. Then you use one of those addresses to
connect... but you already have that address in the container, so the
container will try to connect to itself (hence the connection refused).

There are two options I see:

- you could change the address you use to connect to the HTTPS service
  on the host -- if you're in the container, you can use the address of
  the default gateway as reported by ip -4 route show / ip -6 route show
  (with --map-gw), or 127.0.0.1/::1 (with -T auto, or -T 443)

- you could change the address of your container. You can specify
  address and default gateway with -a / -g (currently, only once for
  IPv4, and once for IPv6). Or you can select another interface on the
  host which pasta uses to source addresses and routes.

  For that, on the version of the passt package you're using, you're
  limited to "-i", which simply specifies the interface from which
  address and default gateways are sourced. Later versions implement a
  distinction between that "template" interface and outbound interfaces
  (--outbound-if4, --outbound-if6).

I hope this helps. If you're facing issues with this, it would help if
you could share addressing and routing information from your host
and container (you can obfuscate by translating addresses consistently
to TEST-NET subnets for IPv4, such as 192.0.2.0/24 and 198.51.100.0/24,
or 2001:db8::/32 for IPv6).

By the way of package versions, I requested just yesterday the first
upload of an updated version after the Debian 12 release (I'm a
maintainer without upload rights):
  https://qa.debian.org/cgi-bin/vcswatch?package=passt

which should be synchronised to Ubuntu in a while (I think it's a
couple of weeks after that).

-- 
Stefano