119 lines
6.6 KiB
Markdown
119 lines
6.6 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Build
|
|
|
|
Requires DPDK 24.11 built with meson at `~/share/dpdk-study/sources/dpdk-24.11/build`.
|
|
|
|
```bash
|
|
make
|
|
```
|
|
|
|
The binary is `./build/dpdk_ws_client`. The Makefile uses `pkg-config` (with `PKG_CONFIG_PATH` pointing to DPDK's uninstalled `.pc` files) to get compiler/linker flags. The binary includes an RPATH pointing to the DPDK lib directory, so no `LD_LIBRARY_PATH` is needed at runtime. Key flags: `-DALLOW_EXPERIMENTAL_API`, `-Wl,--allow-multiple-definition`, `-O2 -g`.
|
|
|
|
### Test/benchmark binaries (standalone, no DPDK needed)
|
|
|
|
```bash
|
|
gcc -O2 -o test/test_ws_protocol test/test_ws_protocol.c -lpthread
|
|
gcc -O2 -o test/bench_tcp test/bench_tcp.c -lpthread
|
|
```
|
|
|
|
## Architecture
|
|
|
|
This is a **DPDK user-space TCP/IP stack** with a **WebSocket client (RFC 6455)** layered on top. It runs entirely in userspace — no kernel TCP.
|
|
|
|
### Lcore split
|
|
|
|
- **lcore0 (main thread)**: DPDK RX/TX burst from NIC, WS receive polling, application logic (`main.c`)
|
|
- **lcore1 (pkt_process)**: Protocol processing loop — TCP state machine, ARP/ICMP, timer maintenance, TX output engine (`tcp_core.c`)
|
|
|
|
Data crosses lcores via two `rte_ring` SP/SC queues: `ring->in` (RX → pkt_process) and `ring->out` (pkt_process → TX).
|
|
|
|
### Layer stack (bottom-up)
|
|
|
|
```
|
|
DPDK rte_eth_rx_burst / rte_eth_tx_burst
|
|
→ pkt_process dispatch: ARP / ICMP / TCP (ng_tcp_process)
|
|
→ TCP 11-state FSM (tcp_input.c) + congestion control (tcp_output.c) + timers (tcp_timer.c)
|
|
→ stream->rcvbuf / stream->sndbuf (rte_ring per-TCB)
|
|
→ POSIX-like socket API: nsocket / nconnect / nsend / nrecv / nclose (tcp_socket.c)
|
|
→ WebSocket: ws_connect (HTTP Upgrade) / ws_send_frame / ws_recv_frame (ws_client.c)
|
|
→ Application (main.c)
|
|
```
|
|
|
|
### Critical design details
|
|
|
|
- **Blocking I/O via condition variables**: `nconnect()` and `nrecv()` block the app thread using `pthread_cond_wait`. The pkt_process lcore signals `stream->cond` when state changes or data arrives. `nconnect()` additionally services DPDK RX/TX inline while waiting for the handshake to complete.
|
|
- **TCB lookup is O(1)**: 256-bucket hash table on the 4-tuple (sip, dip, sport, dport). Additionally, an `fd → TCB` direct array (`fd_to_tcb[]`) provides O(1) lookup from socket fd.
|
|
- **Client→Server frames are MASKED**, Server→Client frames are unmasked (RFC 6455). Masking key is 4 random bytes XORed over the payload.
|
|
- **TCP send path**: `nsend()` segments into MSS-sized `ng_tcp_fragment` structs, enqueues to stream->sndbuf ring. `ng_tcp_out()` (called from pkt_process) dequeues them, builds Ethernet/IP/TCP headers (with option bytes from `fragment->option[]`), resolves MAC via ARP table, sends to ring->out.
|
|
- **TCP receive path**: `ng_tcp_input()` validates sequence numbers against the receive window, handles in-order data by enqueueing to `stream->rcvbuf` and waking `nrecv()`. Out-of-order segments go to `ofo_head` sorted linked list (max 64 entries), drained when the gap fills.
|
|
- **TCP timers** checked per-iteration in pkt_process: RTO with exponential backoff, delayed ACK (200ms), keepalive (2h idle), zero-window probe (1s interval, max 10).
|
|
- **Congestion control**: RFC 5681 slow start / congestion avoidance / fast retransmit (3 dupacks) / fast recovery. cwnd initialized to 2 MSS (~2920 bytes).
|
|
- **ARP**: Linked-list ARP table in `arp.h`. ARP requests rate-limited to 10/s per destination. Missing MAC causes the fragment to be re-enqueued to sndbuf.
|
|
- **Globals**: `gDpdkPortId`, `gLocalIp` (default `192.168.0.115`), `gSrcMac`. Singletons: `tcpInstance()`, `ringInstance()`.
|
|
|
|
### Key source files
|
|
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `src/main.c` | Entry point, EAL init, cmdline parsing, main loop (RX/TX + WS poll + ping) |
|
|
| `src/tcp_core.h` | All structs (TCB, fragment, hash table), constants, function declarations |
|
|
| `src/tcp_core.c` | Hash table, packet encoding, ARP/ICMP helpers, pkt_process loop, port init, `ng_tcp_out` |
|
|
| `src/tcp_input.c` | TCP option parsing, sequence validation, 11-state FSM, out-of-order queue |
|
|
| `src/tcp_output.c` | Congestion control (slow start/CA/fast retx/recovery), option building |
|
|
| `src/tcp_timer.c` | RTT estimation (RFC 6298), RTO with backoff, delayed ACK, keepalive, ZWP |
|
|
| `src/tcp_socket.c` | POSIX-like socket API with condition-variable blocking |
|
|
| `src/ws_client.c` | WebSocket handshake, frame encode/decode (masked send, unmasked recv), ping/pong |
|
|
| `src/ws_client.h` | WebSocket opcodes, API declarations |
|
|
| `src/arp.h` | ARP table (linked list), LL_ADD/LL_REMOVE macros |
|
|
|
|
## Tests
|
|
|
|
- **`test/test_ws_protocol.c`** — WebSocket protocol test using kernel TCP sockets. Connects to a WS server (Python or Java), tests handshake, echo, subscribe/push. No DPDK needed. Usage: `./test_ws_protocol [server_ip] [port]`
|
|
- **`test/bench_tcp.c`** — Microbenchmarks for the data-path hot paths: ring buffer SP/SC throughput, hash table lookup, WS frame encode/decode, memcpy baseline. CPU-only, no NIC.
|
|
- **`test/ws_test_server.py`** — Python async WS echo/subscribe server for manual testing. Dependencies: `websockets`. Usage: `python3 test/ws_test_server.py [port]`
|
|
|
|
## Runtime
|
|
|
|
```bash
|
|
# 1. Hugepages
|
|
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
|
|
mkdir -p /mnt/huge && mount -t hugetlbfs nodev /mnt/huge
|
|
|
|
# 2. Bind NIC (use DPDK 24.11 usertools)
|
|
modprobe vfio-pci
|
|
sudo python3 ~/share/dpdk-study/sources/dpdk-24.11/usertools/dpdk-devbind.py --bind=vfio-pci <pci_addr>
|
|
|
|
# 3. Run (RPATH baked in, no LD_LIBRARY_PATH needed)
|
|
./build/dpdk_ws_client -l 0-1 -n 2 -- -a <SERVER_IP> -p <PORT> -w /ws
|
|
|
|
# With subscription and local IP:
|
|
./build/dpdk_ws_client -l 0-1 -n 2 -- \
|
|
-a 192.168.1.100 -p 8080 -w /ws \
|
|
-s '{"action":"subscribe","topic":"market_data"}' \
|
|
-l 10.0.0.10
|
|
```
|
|
|
|
### CLI parameters (after `--`)
|
|
|
|
| Flag | Default | Description |
|
|
|------|---------|-------------|
|
|
| `-a, --addr` | `192.168.1.100` | Server IPv4 |
|
|
| `-p, --port` | `8080` | Server port |
|
|
| `-w, --path` | `/ws` | WebSocket path |
|
|
| `-s, --subscribe` | (none) | Subscribe JSON message |
|
|
| `-k, --keepalive` | `30` | Ping interval in seconds (0=off) |
|
|
| `-l, --local-ip` | `192.168.0.115` | Local/stub IPv4 |
|
|
|
|
## Port init
|
|
|
|
`ng_init_port()` configures a single port (index 0) with 1 RX queue (128 descriptors) and 1 TX queue (512 descriptors), promiscuous mode enabled. DPDK port ID is the global `gDpdkPortId` (default 0). In DPDK 24.11, `max_rx_pkt_len` was replaced by `mtu` in `rte_eth_rxmode`.
|
|
|
|
## AWS deployment notes
|
|
|
|
- Use `c5n.large`+ (ENA PMD) or `c5n.metal` (SR-IOV)
|
|
- Verify ENA PMD is enabled: `grep ENA $RTE_SDK/x86_64-native-linux-gcc/.config`
|
|
- Security groups must allow the target TCP port
|