[PATCH v6 00/26] RFC: Unified flow table

[David Gibson <david@gibson.dropbear.id.au>]

This is a sixth draft of an implementation of more general
"connection" tracking, as described at:
    https://pad.passt.top/p/NewForwardingModel

This series changes the TCP connection table and hash table into a
more general flow table that can track other protocols as well.  Each
flow uniformly keeps track of all the relevant addresses and ports,
which will allow for more robust control of NAT and port forwarding.

ICMP and UDP are converted to use the new flow table.

This is based on the most recent series of flowtable preliminaries.

NOTE: this fails the pasta UDP perf tests.  This is one aspect of a
curly issue I'm not sure how to deail with yet.  So, this isn't ready
for merge yet, but I'm posting because there have been a *lot* of
changes since v5, and it could do with another round of review,

Other caveats:
 * We roughly double the size of a connection/flow entry
 * UDP still has a number of forwarding bugs because we don't consider
   the local address when looking for a socket to use.  There are at
   least some FIXMEs noted for this.

Changes since v5:
 * flowside_from_af() is now static
 * Small fixes to state verification
 * Pass protocol specific types into deferred/timer callbacks
 * No longer require complete forwarding address info for the hash
   table (we won't have it for UDP)
 * Fix bugs with logging of flow addresses
 * Make sure to initialise sin_zero field sockaddr_from_inany
 * Added patch better typing parameters to flow type specific callbacks
 * Terminology change "forwarded side" to "target side"
 * Assorted wording and style tweaks based on Stefano's review
 * Fold introduction of struct flowside and populating the initiating
   side together
 * Manage outbound addresses via the flow table as well
 * Support for UDP
 * Correct type of 'b' in flowside_lookup() (was a signed int)

Changes since v4:
 * flowside_from_af() no longer fills in unspecified addresses when
   passed NULL
 * Split and rename flow hash lookup function
 * Clarified flow state transitions, and enforced where practical
 * Made side 0 always the initiating side of a flow, rather than
   letting the protocol specific code decide
 * Separated pifs from flowside addresses to allow better structure
   packing

Changes since v3:
 * Complex rebase on top of the many things that have happened
   upstream since v2.
 * Assorted other changes.
 * Replace TAPFSIDE() and SOCKFSIDE() macros with local variables.

Changes since v2:
 * Cosmetic fixes based on review
 * Extra doc comments for enum flow_type
 * Rename flowside to flowaddrs which turns out to make more sense in
   light of future changes
 * Fix bug where the socket flowaddrs for tap initiated connections
   wasn't initialised to match the socket address we were using in the
   case of map-gw NAT
 * New flowaddrs_from_sock() helper used in most cases which is cleaner
   and should avoid bugs like the above
 * Using newer centralised workarounds for clang-tidy issue 58992
 * Remove duplicate definition of FLOW_MAX as maximum flow type and
   maximum number of tracked flows
 * Rebased on newer versions of preliminary work (ICMP, flow based
   dispatch and allocation, bind/address cleanups)
 * Unified hash table as well as base flow table
 * Integrated ICMP

Changes since v1:
 * Terminology changes
   - "Endpoint" address/port instead of "correspondent" address/port
   - "flowside" instead of "demiflow"
 * Actually move the connection table to a new flow table structure in
   new files
 * Significant rearrangement of earlier patchs on top of that new
   table, to reduce churn

David Gibson (26):
  flow: Common address information for initiating side
  flow: Common address information for target side
  tcp, flow: Remove redundant information, repack connection structures
  tcp: Obtain guest address from flowside
  tcp: Manage outbound address via flow table
  tcp: Simplify endpoint validation using flowside information
  tcp_splice: Eliminate SPLICE_V6 flag
  tcp, flow: Replace TCP specific hash function with general flow hash
  flow, tcp: Generalise TCP hash table to general flow hash table
  tcp: Re-use flow hash for initial sequence number generation
  icmp: Remove redundant id field from flow table entry
  icmp: Obtain destination addresses from the flowsides
  icmp: Look up ping flows using flow hash
  icmp: Eliminate icmp_id_map
  icmp: Manage outbound socket address via flow table
  flow, tcp: Flow based NAT and port forwarding for TCP
  flow, icmp: Use general flow forwarding rules for ICMP
  fwd: Update flow forwarding logic for UDP
  udp: Create flow table entries for UDP
  udp: Direct traffic from tap according to flow table
  udp: Direct traffic from host to guest tap according to flow table
  udp: Direct spliced traffic according to flow table
  udp: Remove 'splicesrc' tracking
  udp: Remove tap port flags field
  udp: Remove rdelta port forwarding maps
  udp: Eliminate 'splice' flag from epoll reference

 Makefile       |   2 +-
 conf.c         |  14 +-
 flow.c         | 402 +++++++++++++++++++++++++-
 flow.h         |  45 +++
 flow_table.h   |  45 ++-
 fwd.c          | 177 +++++++++++-
 fwd.h          |   9 +
 icmp.c         | 105 +++----
 icmp_flow.h    |   2 -
 inany.h        |  31 +-
 passt.h        |   3 +
 tap.c          |  11 -
 tap.h          |   1 -
 tcp.c          | 524 +++++++++-------------------------
 tcp_buf.c      |   6 +-
 tcp_conn.h     |  51 ++--
 tcp_internal.h |  10 +-
 tcp_splice.c   |  98 +------
 tcp_splice.h   |   5 +-
 udp.c          | 757 +++++++++++++++++++++++--------------------------
 udp.h          |  22 +-
 udp_flow.h     |  25 ++
 util.c         |   6 +-
 util.h         |   3 +
 24 files changed, 1328 insertions(+), 1026 deletions(-)
 create mode 100644 udp_flow.h

-- 
2.45.2