[PATCH v5 0/7] Refactor epoll handling in preparation for multithreading

[Laurent Vivier <lvivier@redhat.com>]

This series refactors how epoll file descriptors are managed throughout
the codebase in preparation for introducing multithreading support.
Currently, passt uses a single global epollfd accessed through the
context structure. With multithreading, each thread will need its own
epollfd managing its subset of flows.

The key changes are:

1. Centralize epoll management by extracting helper functions into a new
   epoll_ctl.c/h module and moving union epoll_ref from passt.h to its
   more logical location in epoll_ctl.h.

2. Simplify epoll_del() to take the epollfd directly rather than
   extracting it from the context structure, reducing coupling between
   epoll operations and the global context.

3. Move epoll registration out of sock_l4_sa() into protocol-specific
   code, giving callers explicit control over which epoll instance
   manages each socket.

4. Replace the boolean in_epoll flag in TCP connections with an epollid
   (epoll identifier) field in flow_common. This serves dual purposes:
   tracking registration status (EPOLLFD_ID_INVALID = not registered)
   and identifying which epoll instance manages the flow. An epoll ID to
   epoll fd mapping allows retrieving the actual epoll file descriptor.
   The epollid field is 8 bits, limiting values to 0-254 (255 =
   EPOLLFD_ID_INVALID).

5. Apply this pattern consistently across all protocol handlers (TCP,
   ICMP, UDP), storing the managing epoll ID in each flow's common
   structure.

6. Extract the event loop processing logic into a separate passt_worker()
   function, preparing the structure for future threading where this will
   become a worker thread callback.

These changes make epoll instance ownership explicit in the flow tracking
system, enabling flows to be managed by different epoll instances, a
prerequisite for per-thread epollfd design in upcoming multithreading
work.

Changes since v1:
- New patch: "epoll_ctl: Extract epoll operations" - centralizes epoll
  helpers into a dedicated module and relocates union epoll_ref from
  passt.h to epoll_ctl.h
- Changed epollfd type in flow_common from int to 8-bit bitfield to avoid
  exceeding cacheline size threshold
- Added flow_epollfd_valid() helper to check epoll registration status
- Added flow_set_epollfd() helper to set the epoll instance for a flow

Changes since v2:
- Renamed field in flow_common from epollfd to threadnb (thread number)
  to better reflect that flows are now associated with threads rather
  than directly with epoll file descriptors
- Introduced thread-to-epollfd mapping via threadnb_to_epollfd[] array
  in flow.c, providing an indirection layer between threads and their
  epoll instances
- Renamed/refactored helper functions:
  - flow_set_epollfd() -> flow_epollfd_set() - now takes thread number
    and epollfd parameters
  - Added flow_epollfd_get() - retrieves the epoll fd for a flow's thread
  - flow_epollfd_valid() - now checks threadnb instead of epollfd
- Updated constants: replaced EPOLLFD_* with FLOW_THREADNB_* equivalents
  (e.g., EPOLLFD_INVALID -> FLOW_THREADNB_INVALID)
- Applied thread-based pattern consistently across TCP, ICMP, and UDP

Changes since v3:
- Change warn() by err() in epoll_add()
- Renamed flow_epollfd_valid() to flow_in_epoll(), flow_epollfd_get() to
  flow_epollfd().
- Added flow_thread_register() to register an epollfd to a thread number,
  and called it from main() to register c->epollfd for thread 0 (main loop)
- Replaced flow_epollfd_set() by flow_thread_set() to set the thread
  number of a flow.
- Added new patch:
  "passt: Move main event loop processing into passt_worker()"

Changes since v4:
- Renamed field in flow_common from threadnb to epollid (epoll identifier)
  to better reflect the abstraction - flows are associated with an epoll
  instance identifier rather than specifically a thread number
- Renamed mapping array from threadnb_to_epollfd[] to epoll_id_to_fd[]
- Renamed/refactored helper functions:
  - flow_thread_set() -> flow_epollid_set() - sets the epoll ID
  - flow_thread_register() -> flow_epollid_register() - registers the
    epollfd for a given epoll ID
  - Added flow_epollid_clear() - explicitly clears the epoll ID
- Updated constants: replaced FLOW_THREADNB_* with EPOLLFD_ID_* equivalents
  (e.g., FLOW_THREADNB_INVALID -> EPOLLFD_ID_INVALID, FLOW_THREADNB_MAX ->
  EPOLLFD_ID_MAX)
- Added EPOLLFD_ID_DEFAULT constant for the default (main loop) epoll instance
- Clarified that EPOLLFD_ID_BITS limits the number of epoll instances, not
  threads (though initially there will be one epoll instance per thread)

Laurent Vivier (7):
  util: Simplify epoll_del() interface to take epollfd directly
  epoll_ctl: Extract epoll operations
  util: Move epoll registration out of sock_l4_sa()
  tcp, flow: Replace per-connection in_epoll flag with an epollid in
    flow_common
  icmp: Use epoll instance management for ICMP flows
  udp: Use epoll instance management for UDP flows
  passt: Move main event loop processing into passt_worker()

 Makefile     |  22 +++----
 epoll_ctl.c  |  45 ++++++++++++++
 epoll_ctl.h  |  51 ++++++++++++++++
 flow.c       |  71 +++++++++++++++++++---
 flow.h       |  15 ++++-
 icmp.c       |  23 +++++---
 passt.c      | 163 ++++++++++++++++++++++++++++-----------------------
 passt.h      |  34 -----------
 pasta.c      |   7 +--
 pif.c        |  32 +++++++---
 repair.c     |  18 +++---
 tap.c        |  15 ++---
 tcp.c        |  46 +++++++++------
 tcp_conn.h   |   8 +--
 tcp_splice.c |  26 ++++----
 udp.c        |   2 +-
 udp_flow.c   |  23 ++++++--
 util.c       |  28 +--------
 util.h       |   6 +-
 vhost_user.c |  14 ++---
 vu_common.c  |   2 +-
 21 files changed, 403 insertions(+), 248 deletions(-)
 create mode 100644 epoll_ctl.c
 create mode 100644 epoll_ctl.h

-- 
2.51.0