wxj/dpdk-ws-client
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
feature/multi-core
dpdk-ws-client/CLAUDE.md
T

6.6 KiB
Raw Permalink Blame History

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Build

Requires DPDK 24.11 built with meson at ~/share/dpdk-study/sources/dpdk-24.11/build.

make

The binary is ./build/dpdk_ws_client. The Makefile uses pkg-config (with PKG_CONFIG_PATH pointing to DPDK's uninstalled .pc files) to get compiler/linker flags. The binary includes an RPATH pointing to the DPDK lib directory, so no LD_LIBRARY_PATH is needed at runtime. Key flags: -DALLOW_EXPERIMENTAL_API, -Wl,--allow-multiple-definition, -O2 -g.

Test/benchmark binaries (standalone, no DPDK needed)

gcc -O2 -o test/test_ws_protocol test/test_ws_protocol.c -lpthread
gcc -O2 -o test/bench_tcp test/bench_tcp.c -lpthread

Architecture

This is a DPDK user-space TCP/IP stack with a WebSocket client (RFC 6455) layered on top. It runs entirely in userspace — no kernel TCP.

Lcore split

  • lcore0 (main thread): DPDK RX/TX burst from NIC, WS receive polling, application logic (main.c)
  • lcore1 (pkt_process): Protocol processing loop — TCP state machine, ARP/ICMP, timer maintenance, TX output engine (tcp_core.c)

Data crosses lcores via two rte_ring SP/SC queues: ring->in (RX → pkt_process) and ring->out (pkt_process → TX).

Layer stack (bottom-up)

DPDK rte_eth_rx_burst / rte_eth_tx_burst
  → pkt_process dispatch: ARP / ICMP / TCP (ng_tcp_process)
    → TCP 11-state FSM (tcp_input.c) + congestion control (tcp_output.c) + timers (tcp_timer.c)
      → stream->rcvbuf / stream->sndbuf (rte_ring per-TCB)
        → POSIX-like socket API: nsocket / nconnect / nsend / nrecv / nclose (tcp_socket.c)
          → WebSocket: ws_connect (HTTP Upgrade) / ws_send_frame / ws_recv_frame (ws_client.c)
            → Application (main.c)

Critical design details

  • Blocking I/O via condition variables: nconnect() and nrecv() block the app thread using pthread_cond_wait. The pkt_process lcore signals stream->cond when state changes or data arrives. nconnect() additionally services DPDK RX/TX inline while waiting for the handshake to complete.
  • TCB lookup is O(1): 256-bucket hash table on the 4-tuple (sip, dip, sport, dport). Additionally, an fd → TCB direct array (fd_to_tcb[]) provides O(1) lookup from socket fd.
  • Client→Server frames are MASKED, Server→Client frames are unmasked (RFC 6455). Masking key is 4 random bytes XORed over the payload.
  • TCP send path: nsend() segments into MSS-sized ng_tcp_fragment structs, enqueues to stream->sndbuf ring. ng_tcp_out() (called from pkt_process) dequeues them, builds Ethernet/IP/TCP headers (with option bytes from fragment->option[]), resolves MAC via ARP table, sends to ring->out.
  • TCP receive path: ng_tcp_input() validates sequence numbers against the receive window, handles in-order data by enqueueing to stream->rcvbuf and waking nrecv(). Out-of-order segments go to ofo_head sorted linked list (max 64 entries), drained when the gap fills.
  • TCP timers checked per-iteration in pkt_process: RTO with exponential backoff, delayed ACK (200ms), keepalive (2h idle), zero-window probe (1s interval, max 10).
  • Congestion control: RFC 5681 slow start / congestion avoidance / fast retransmit (3 dupacks) / fast recovery. cwnd initialized to 2 MSS (~2920 bytes).
  • ARP: Linked-list ARP table in arp.h. ARP requests rate-limited to 10/s per destination. Missing MAC causes the fragment to be re-enqueued to sndbuf.
  • Globals: gDpdkPortId, gLocalIp (default 192.168.0.115), gSrcMac. Singletons: tcpInstance(), ringInstance().

Key source files

File Purpose
src/main.c Entry point, EAL init, cmdline parsing, main loop (RX/TX + WS poll + ping)
src/tcp_core.h All structs (TCB, fragment, hash table), constants, function declarations
src/tcp_core.c Hash table, packet encoding, ARP/ICMP helpers, pkt_process loop, port init, ng_tcp_out
src/tcp_input.c TCP option parsing, sequence validation, 11-state FSM, out-of-order queue
src/tcp_output.c Congestion control (slow start/CA/fast retx/recovery), option building
src/tcp_timer.c RTT estimation (RFC 6298), RTO with backoff, delayed ACK, keepalive, ZWP
src/tcp_socket.c POSIX-like socket API with condition-variable blocking
src/ws_client.c WebSocket handshake, frame encode/decode (masked send, unmasked recv), ping/pong
src/ws_client.h WebSocket opcodes, API declarations
src/arp.h ARP table (linked list), LL_ADD/LL_REMOVE macros

Tests

  • test/test_ws_protocol.c — WebSocket protocol test using kernel TCP sockets. Connects to a WS server (Python or Java), tests handshake, echo, subscribe/push. No DPDK needed. Usage: ./test_ws_protocol [server_ip] [port]
  • test/bench_tcp.c — Microbenchmarks for the data-path hot paths: ring buffer SP/SC throughput, hash table lookup, WS frame encode/decode, memcpy baseline. CPU-only, no NIC.
  • test/ws_test_server.py — Python async WS echo/subscribe server for manual testing. Dependencies: websockets. Usage: python3 test/ws_test_server.py [port]

Runtime

# 1. Hugepages
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
mkdir -p /mnt/huge && mount -t hugetlbfs nodev /mnt/huge

# 2. Bind NIC (use DPDK 24.11 usertools)
modprobe vfio-pci
sudo python3 ~/share/dpdk-study/sources/dpdk-24.11/usertools/dpdk-devbind.py --bind=vfio-pci <pci_addr>

# 3. Run (RPATH baked in, no LD_LIBRARY_PATH needed)
./build/dpdk_ws_client -l 0-1 -n 2 -- -a <SERVER_IP> -p <PORT> -w /ws

# With subscription and local IP:
./build/dpdk_ws_client -l 0-1 -n 2 -- \
    -a 192.168.1.100 -p 8080 -w /ws \
    -s '{"action":"subscribe","topic":"market_data"}' \
    -l 10.0.0.10

CLI parameters (after --)

Flag Default Description
-a, --addr 192.168.1.100 Server IPv4
-p, --port 8080 Server port
-w, --path /ws WebSocket path
-s, --subscribe (none) Subscribe JSON message
-k, --keepalive 30 Ping interval in seconds (0=off)
-l, --local-ip 192.168.0.115 Local/stub IPv4

Port init

ng_init_port() configures a single port (index 0) with 1 RX queue (128 descriptors) and 1 TX queue (512 descriptors), promiscuous mode enabled. DPDK port ID is the global gDpdkPortId (default 0). In DPDK 24.11, max_rx_pkt_len was replaced by mtu in rte_eth_rxmode.

AWS deployment notes

  • Use c5n.large+ (ENA PMD) or c5n.metal (SR-IOV)
  • Verify ENA PMD is enabled: grep ENA $RTE_SDK/x86_64-native-linux-gcc/.config
  • Security groups must allow the target TCP port
Reference in New Issue View Git Blame Copy Permalink