public inbox for passt-dev@passt.top
 help / color / mirror / code / Atom feed
From: David Gibson <david@gibson.dropbear.id.au>
To: passt-dev@passt.top, Stefano Brivio <sbrivio@redhat.com>
Cc: David Gibson <david@gibson.dropbear.id.au>
Subject: [PATCH 00/32] Use dual stack sockets to listen for inbound TCP connections
Date: Wed, 16 Nov 2022 15:41:40 +1100	[thread overview]
Message-ID: <20221116044212.3876516-1-david@gibson.dropbear.id.au> (raw)

When forwarding many ports, passt can consume a lot of kernel memory
because of the many listening sockets it opens.  There are not a lot
of ways we can reduce that, but here's one.

Currently we create separate listening sockets for each port for both
IPv4 and IPv6.  However in Linux (and probably other platforms), it's
possible to listen for both IPv4 and IPv6 connections on an IPv6
socket.  This series uses such dual stack sockets to halve the number
of listening sockets needed for TCP.  When forwarding all TCP and UDP
ports, this reduces the kernel memory used from around 677 MiB to
around 487 MiB (kernel 6.0.8 on an x86_64 Fedora 37 machine).

This should also be possible for UDP, but that will require a mostly
separate implementation.

David Gibson (32):
  clang-tidy: Suppress warning about assignments in if statements
  style: Minor corrections to function comments
  tcp_splice: #include tcp_splice.h in tcp_splice.c
  tcp: Remove unused TCP_MAX_SOCKS constant
  tcp: Better helpers for converting between connection pointer and
    index
  tcp_splice: Helpers for converting from index to/from tcp_splice_conn
  tcp: Move connection state structures into a shared header
  tcp: Add connection union type
  tcp: Improved helpers to update connections after moving
  tcp: Unify spliced and non-spliced connection tables
  tcp: Unify tcp_defer_handler and tcp_splice_defer_handler()
  tcp: Partially unify tcp_timer() and tcp_splice_timer()
  tcp: Unify the IN_EPOLL flag
  tcp: Separate helpers to create ns listening sockets
  tcp: Unify part of spliced and non-spliced conn_from_sock path
  tcp: Use the same sockets to listen for spliced and non-spliced
    connections
  tcp: Remove splice from tcp_epoll_ref
  tcp: Don't store hash bucket in connection structures
  inany: Helper functions for handling addresses which could be IPv4 or
    IPv6
  tcp: Hash IPv4 and IPv4-mapped-IPv6 addresses the same
  tcp: Take tcp_hash_insert() address from struct tcp_conn
  tcp: Simplify tcp_hash_match() to take an inany_addr
  tcp: Unify initial sequence number calculation for IPv4 and IPv6
  tcp: Have tcp_seq_init() take its parameters from struct tcp_conn
  tcp: Fix small errors in tcp_seq_init() time handling
  tcp: Remove v6 flag from tcp_epoll_ref
  tcp: NAT IPv4-mapped IPv6 addresses like IPv4 addresses
  tcp_splice: Allow splicing of connections from IPv4-mapped loopback
  tcp: Consolidate tcp_sock_init[46]
  util: Allow sock_l4() to open dual stack sockets
  util: Always return -1 on error in sock_l4()
  tcp: Use dual stack sockets for port forwarding when possible

 Makefile     |  11 +-
 conf.c       |  12 +-
 inany.h      |  94 +++++
 siphash.c    |   2 +
 tap.c        |   6 +-
 tcp.c        | 978 ++++++++++++++++++++++-----------------------------
 tcp.h        |  11 +-
 tcp_conn.h   | 192 ++++++++++
 tcp_splice.c | 337 ++++++++----------
 tcp_splice.h |  12 +-
 util.c       |  19 +-
 11 files changed, 887 insertions(+), 787 deletions(-)
 create mode 100644 inany.h
 create mode 100644 tcp_conn.h

-- 
2.38.1


             reply	other threads:[~2022-11-16  4:42 UTC|newest]

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-11-16  4:41 David Gibson [this message]
2022-11-16  4:41 ` [PATCH 01/32] clang-tidy: Suppress warning about assignments in if statements David Gibson
2022-11-16 23:10   ` Stefano Brivio
2022-11-17  1:20     ` David Gibson
2022-11-16  4:41 ` [PATCH 02/32] style: Minor corrections to function comments David Gibson
2022-11-16 23:11   ` Stefano Brivio
2022-11-17  1:21     ` David Gibson
2022-11-16  4:41 ` [PATCH 03/32] tcp_splice: #include tcp_splice.h in tcp_splice.c David Gibson
2022-11-16  4:41 ` [PATCH 04/32] tcp: Remove unused TCP_MAX_SOCKS constant David Gibson
2022-11-16  4:41 ` [PATCH 05/32] tcp: Better helpers for converting between connection pointer and index David Gibson
2022-11-16 23:11   ` Stefano Brivio
2022-11-17  1:24     ` David Gibson
2022-11-16  4:41 ` [PATCH 06/32] tcp_splice: Helpers for converting from index to/from tcp_splice_conn David Gibson
2022-11-16  4:41 ` [PATCH 07/32] tcp: Move connection state structures into a shared header David Gibson
2022-11-16  4:41 ` [PATCH 08/32] tcp: Add connection union type David Gibson
2022-11-16  4:41 ` [PATCH 09/32] tcp: Improved helpers to update connections after moving David Gibson
2022-11-16  4:41 ` [PATCH 10/32] tcp: Unify spliced and non-spliced connection tables David Gibson
2022-11-16  4:41 ` [PATCH 11/32] tcp: Unify tcp_defer_handler and tcp_splice_defer_handler() David Gibson
2022-11-16  4:41 ` [PATCH 12/32] tcp: Partially unify tcp_timer() and tcp_splice_timer() David Gibson
2022-11-16  4:41 ` [PATCH 13/32] tcp: Unify the IN_EPOLL flag David Gibson
2022-11-16  4:41 ` [PATCH 14/32] tcp: Separate helpers to create ns listening sockets David Gibson
2022-11-16 23:51   ` Stefano Brivio
2022-11-17  1:32     ` David Gibson
2022-11-16  4:41 ` [PATCH 15/32] tcp: Unify part of spliced and non-spliced conn_from_sock path David Gibson
2022-11-16 23:53   ` Stefano Brivio
2022-11-17  1:37     ` David Gibson
2022-11-17  7:30       ` Stefano Brivio
2022-11-17  8:58         ` David Gibson
2022-11-16  4:41 ` [PATCH 16/32] tcp: Use the same sockets to listen for spliced and non-spliced connections David Gibson
2022-11-16 23:54   ` Stefano Brivio
2022-11-17  1:43     ` David Gibson
2022-11-16  4:41 ` [PATCH 17/32] tcp: Remove splice from tcp_epoll_ref David Gibson
2022-11-16  4:41 ` [PATCH 18/32] tcp: Don't store hash bucket in connection structures David Gibson
2022-11-16  4:41 ` [PATCH 19/32] inany: Helper functions for handling addresses which could be IPv4 or IPv6 David Gibson
2022-11-16 23:54   ` Stefano Brivio
2022-11-17  1:48     ` David Gibson
2022-11-16  4:42 ` [PATCH 20/32] tcp: Hash IPv4 and IPv4-mapped-IPv6 addresses the same David Gibson
2022-11-16  4:42 ` [PATCH 21/32] tcp: Take tcp_hash_insert() address from struct tcp_conn David Gibson
2022-11-16  4:42 ` [PATCH 22/32] tcp: Simplify tcp_hash_match() to take an inany_addr David Gibson
2022-11-16  4:42 ` [PATCH 23/32] tcp: Unify initial sequence number calculation for IPv4 and IPv6 David Gibson
2022-11-16  4:42 ` [PATCH 24/32] tcp: Have tcp_seq_init() take its parameters from struct tcp_conn David Gibson
2022-11-16  4:42 ` [PATCH 25/32] tcp: Fix small errors in tcp_seq_init() time handling David Gibson
2022-11-16  4:42 ` [PATCH 26/32] tcp: Remove v6 flag from tcp_epoll_ref David Gibson
2022-11-17  0:15   ` Stefano Brivio
2022-11-17  1:50     ` David Gibson
2022-11-16  4:42 ` [PATCH 27/32] tcp: NAT IPv4-mapped IPv6 addresses like IPv4 addresses David Gibson
2022-11-17  0:15   ` Stefano Brivio
2022-11-17  2:00     ` David Gibson
2022-11-16  4:42 ` [PATCH 28/32] tcp_splice: Allow splicing of connections from IPv4-mapped loopback David Gibson
2022-11-17  0:15   ` Stefano Brivio
2022-11-17  2:05     ` David Gibson
2022-11-16  4:42 ` [PATCH 29/32] tcp: Consolidate tcp_sock_init[46] David Gibson
2022-11-16  4:42 ` [PATCH 30/32] util: Allow sock_l4() to open dual stack sockets David Gibson
2022-11-16  4:42 ` [PATCH 31/32] util: Always return -1 on error in sock_l4() David Gibson
2022-11-16  4:42 ` [PATCH 32/32] tcp: Use dual stack sockets for port forwarding when possible David Gibson
2022-11-17  0:15   ` Stefano Brivio
2022-11-17  2:08     ` David Gibson

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20221116044212.3876516-1-david@gibson.dropbear.id.au \
    --to=david@gibson.dropbear.id.au \
    --cc=passt-dev@passt.top \
    --cc=sbrivio@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://passt.top/passt

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for IMAP folder(s).