1738 lines
62 KiB
Markdown
Executable File
1738 lines
62 KiB
Markdown
Executable File
# uNOS 详细设计说明书
|
||
|
||
> **版本**: v2.0 — 生产级架构 (RSS 多核 + Graph Pipeline + 内联 TCP)
|
||
> **日期**: 2026-07-29
|
||
> **关联文档**: `uNOS_REQUIREMENTS.md`（需求分析）、`lazy-puzzling-rocket.md`（开发执行计划）
|
||
|
||
---
|
||
|
||
## 目录
|
||
|
||
1. [设计概述](#一设计概述)
|
||
2. [架构设计](#二架构设计)
|
||
3. [双产品设计](#三双产品设计)
|
||
4. [数据面详细设计](#四数据面详细设计)
|
||
5. [控制面详细设计](#五控制面详细设计)
|
||
6. [自研 TCP/IP 协议栈详细设计](#六自研-tcpip-协议栈详细设计)
|
||
7. [容器网络详细设计](#七容器网络详细设计)
|
||
8. [管理面详细设计](#八管理面详细设计)
|
||
9. [高可用详细设计](#九高可用详细设计)
|
||
10. [构建系统设计](#十构建系统设计)
|
||
11. [测试设计](#十一测试设计)
|
||
12. [接口契约](#十二接口契约)
|
||
|
||
---
|
||
|
||
## 一、设计概述
|
||
|
||
### 1.1 设计目标
|
||
|
||
设计并实现一个**自研用户态网络操作系统（uNOS）**，核心特性：
|
||
|
||
- **自研 TCP/IP 协议栈**：从 RFC 793 状态机到 CUBIC 拥塞控制，全部手写
|
||
- **DPDK 高性能数据面**：L2/L3 线速转发 + NAT + ACL + QoS
|
||
- **双产品架构**：同一份源码，编译出 Cloud Edge（T140 容器网关）和 Home CPE（J3455 家庭网关）
|
||
- **容器网络支持**：memif 零拷贝接入容器
|
||
- **FRR 动态路由集成**：BGP/OSPF 路由自动同步到数据面
|
||
- **完整管理面**：CLI + REST API + Web Dashboard
|
||
|
||
### 1.2 产品矩阵
|
||
|
||
> ⚠ 2026-09-07 更新: Cloud Edge 硬件(T140)已售,该产品线**暂缓**;Home CPE 为唯一在研产品,硬件已升级(X540-T2 10G × 2,8GB RAM)。模块拆解与路线对齐见 `dpdk_study/uNOS_MODULE_ROADMAP.md`。
|
||
|
||
| | uNOS Cloud Edge ⏸ | uNOS Home CPE ▶ |
|
||
|---|---|---|
|
||
| 目标硬件 | Dell T140(已售,暂缓) | J3455B-ITX(Celeron, **8GB**) |
|
||
| 网卡 | X540-T2 10G × 2 + E1G42ET 1G × 4 | **X540-T2 10G × 2** |
|
||
| 容器支持 | ✅ memif + CNI | ❌ |
|
||
| BGP | ✅ full table | ❌ |
|
||
| OSPF | ✅ | ✅ stub |
|
||
| HA | ✅ VRRP | ❌ |
|
||
| 应用层服务 | ✅ HTTP/DNS/LB | ❌ |
|
||
| 编译宏 | `PRODUCT_CLOUD_EDGE` | `PRODUCT_HOME_CPE` |
|
||
|
||
### 1.3 设计原则
|
||
|
||
1. **零拷贝优先**：数据面路径上不产生任何 mbuf 拷贝
|
||
2. **编译时剪裁**：产品差异通过 `#ifdef` 在编译时决定，不在运行时判断
|
||
3. **固定 Pipeline**：转发路径不动态改变模块注册顺序，保证分支预测友好
|
||
4. **Cache Line 对齐**：所有热路径数据结构按 64B 对齐，避免 false sharing
|
||
5. **错误即崩溃**：数据面关键路径不检查返回值（如 mbuf 分配），由预分配 mempool 保证；控制面优雅处理所有错误
|
||
|
||
---
|
||
|
||
## 二、架构设计
|
||
|
||
### 2.1 系统分层
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────┐
|
||
│ Management Plane │
|
||
│ CLI (vtysh) │ REST API │ Web Dashboard │ Prometheus │
|
||
│ │ │
|
||
│ Redis DB (配置存储 + Pub/Sub) │
|
||
└──────────────────────────┬──────────────────────────────┘
|
||
│ Unix socket / TCP
|
||
┌──────────────────────────┼──────────────────────────────┐
|
||
│ Control Plane (lcore 0) │
|
||
│ │
|
||
│ Config Manager │ Route Manager │ Interface Manager │
|
||
│ ARP/NDP │ DHCP │ Health Monitor │
|
||
│ CNI Manager ◄── Cloud Edge only │
|
||
│ │ │
|
||
│ IPC (rte_ring) │
|
||
└──────────────────────────┼──────────────────────────────┘
|
||
│
|
||
┌──────────────────────────┼──────────────────────────────┐
|
||
│ Data Plane (lcore 1) │
|
||
│ │
|
||
│ ┌────────────────────────────────────────────────────┐ │
|
||
│ │ Forwarding Pipeline │ │
|
||
│ │ RX → Parser → Classifier → ACL → LPM → NAT → QoS │ │
|
||
│ │ │ │ │
|
||
│ │ ▼ │ │
|
||
│ │ TX (physical / memif) │ │
|
||
│ └────────────────────────────────────────────────────┘ │
|
||
│ │
|
||
│ ┌────────────────────────────────────────────────────┐ │
|
||
│ │ 自研 TCP/IP Stack (lcore 2) │ │
|
||
│ │ TCP State Machine → Congestion → Retransmit → TX │ │
|
||
│ └────────────────────────────────────────────────────┘ │
|
||
└──────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### 2.2 lcore 分配
|
||
|
||
#### Cloud Edge (T140, E-2236 6C/12T)
|
||
|
||
```
|
||
Core 0 ─→ lcore 0: Control Plane (main thread, 不参与数据面轮询)
|
||
├── Config Manager (YAML/Redis)
|
||
├── Route Manager (FRR FPM Netlink)
|
||
├── Interface Manager (UP/DOWN/MTU/Stats)
|
||
├── ARP/NDP Manager
|
||
├── CNI Manager (memif create/delete, IPAM)
|
||
├── Health Monitor (rte_timer 5s)
|
||
├── CLI + REST Server (基于自研 TCP 栈)
|
||
└── Flow Table GC Worker (全局老化)
|
||
|
||
Core 1-2 ─→ lcore 1-2: Data Plane Workers (RSS 对称，每个核独立)
|
||
每个 worker 的数据面:
|
||
┌─────────────────────────────────────────────┐
|
||
│ RX (本 worker 独占的 RSS queues) │
|
||
│ │ │
|
||
│ ┌─────────┴──────────┐ │
|
||
│ │ Graph Pipeline │ ← 独立转发实例 │
|
||
│ │ (per-core graph) │ │
|
||
│ └─────────┬──────────┘ │
|
||
│ │ │
|
||
│ TX (本 worker 独占的 TX queues) │
|
||
│ │
|
||
│ Per-core 状态 (无需锁): │
|
||
│ - Flow Cache (rte_hash) │
|
||
│ - Connection Track Table │
|
||
│ - TCP Stack (集成在 Graph 中) │
|
||
│ - ARP Cache (只读，控制面写入) │
|
||
│ - LPM Route Table (只读，控制面写入) │
|
||
└─────────────────────────────────────────────┘
|
||
|
||
RSS 保证同一五元组永远落在同一个 worker 上，
|
||
因此 per-core 的状态表天然无锁。
|
||
|
||
Core 3 ─→ lcore 3: TCP App Server (从数据面 worker 中接应用层流量)
|
||
└── HTTP Server / DNS Server / TCP Proxy
|
||
通过 per-core ring 从数据面 TCP 栈接收应用层请求
|
||
|
||
Core 4 ─→ (非 DPDK) FRR 进程 (bgpd + ospfd)
|
||
└── taskset 绑定，通过 Netlink FPM 推送路由到控制面
|
||
|
||
Core 5 ─→ (非 DPDK) 可观测 + Redis + 系统
|
||
└── Prometheus exporter / sFlow / 日志 / Redis Server
|
||
|
||
HT 线程 (6 个) ─→ 内核中断 + 辅助任务
|
||
└── isolcpus 隔离数据面核心 (core 1-3)
|
||
```
|
||
|
||
#### Home CPE (J3455, 4 cores)
|
||
|
||
```
|
||
Core 0 ─→ lcore 0: Control Plane (main thread)
|
||
├── Config Manager / Route Manager (ospfd FPM)
|
||
├── ARP Manager / DHCP Client / Health Monitor
|
||
└── CLI (Unix socket)
|
||
|
||
Core 1 ─→ lcore 1: Data Plane Worker (单核，1G 线速足够)
|
||
└── 物理口 (82576 × 2) RSS queue 0
|
||
Graph Pipeline: L2/L3/NAT/ACL/QoS
|
||
Per-core Flow Cache + Connection Track Table
|
||
(不包含 TCP 栈 — Home CPE 不跑应用层服务)
|
||
|
||
Core 2 ─→ (非 DPDK) FRR ospfd
|
||
|
||
Core 3 ─→ 空闲 / Prometheus 轻量 exporter
|
||
```
|
||
|
||
### 2.3 模块依赖图
|
||
|
||
```
|
||
┌─────────────────┐
|
||
│ main.c │
|
||
│ EAL init │
|
||
│ lcore launch │
|
||
└────────┬────────┘
|
||
│
|
||
┌────────────────┼────────────────┐
|
||
▼ ▼ ▼
|
||
┌──────────┐ ┌──────────┐ ┌──────────────┐
|
||
│pipeline.c│ │control.c │ │tcp_main.c │ ← Cloud Edge only
|
||
│(lcore 1) │ │(lcore 0) │ │(lcore 2) │
|
||
└────┬─────┘ └────┬─────┘ └──────┬───────┘
|
||
│ │ │
|
||
┌──────┼──────┐ ┌────┼──────┐ ┌─────┼──────┐
|
||
▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
|
||
parser l3fwd nat arp dhcp iface ipv4 tcp_state http
|
||
l2fwd acl qos ndp frr db icmp tcp_cong dns
|
||
(Redis) frag tcp_retx proxy
|
||
│ tcp_flow
|
||
┌─────┘ tcp_socket
|
||
▼
|
||
Redis Server
|
||
(external process)
|
||
```
|
||
|
||
---
|
||
|
||
## 三、双产品设计
|
||
|
||
### 3.1 编译时产品选择
|
||
|
||
```meson
|
||
# meson_options.txt
|
||
option('product', type: 'combo',
|
||
choices: ['cloud_edge', 'home_cpe'],
|
||
value: 'cloud_edge',
|
||
description: 'Target product: cloud_edge (T140) or home_cpe (J3455)')
|
||
|
||
# src/meson.build
|
||
product = get_option('product')
|
||
|
||
product_flags = []
|
||
if product == 'cloud_edge'
|
||
product_flags += '-DPRODUCT_CLOUD_EDGE'
|
||
sources += files(
|
||
'cni/memif.c',
|
||
'cni/cni_api.c',
|
||
'cni/ipam.c',
|
||
'app/http_server.c',
|
||
'app/dns_server.c',
|
||
'app/tcp_proxy.c',
|
||
'stcp/tcp_socket.c',
|
||
'stcp/tcp_txrx.c',
|
||
'ha/vrrp.c',
|
||
)
|
||
elif product == 'home_cpe'
|
||
product_flags += '-DPRODUCT_HOME_CPE'
|
||
# Home CPE excludes: CNI, HTTP Server, DNS, TCP Proxy,
|
||
# VRRP, and full TCP socket layer
|
||
endif
|
||
```
|
||
|
||
### 3.2 模块编译矩阵
|
||
|
||
```
|
||
模块 Cloud Edge Home CPE 说明
|
||
──── ────────── ──────── ──
|
||
src/core/pipeline.c ✅ ✅ 共享
|
||
src/core/ipc.c ✅ ✅ 共享
|
||
src/core/main.c ✅ ✅ 共享 (lcore 分配不同)
|
||
src/dp/parser.c ✅ ✅ 共享
|
||
src/dp/l2fwd.c ✅ ✅ 共享
|
||
src/dp/l3fwd.c ✅ ✅ 共享
|
||
src/dp/nat.c ✅ ✅ 共享
|
||
src/dp/acl.c ✅ ✅ 共享
|
||
src/dp/qos.c ✅ ✅ 共享
|
||
src/cp/arp.c ✅ ✅ 共享
|
||
src/cp/iface.c ✅ ✅ 共享
|
||
src/cp/dhcp.c ❌ ✅ Home CPE only
|
||
src/cp/frr_sync.c ✅ ✅ 共享 (协议不同)
|
||
src/cp/db.c ✅ ✅ 共享
|
||
src/stcp/ipv4.c ✅ ❌ Cloud Edge only
|
||
src/stcp/tcp_state.c ✅ ❌ Cloud Edge only
|
||
src/stcp/tcp_cong.c ✅ ❌ Cloud Edge only
|
||
src/stcp/tcp_retx.c ✅ ❌ Cloud Edge only
|
||
src/stcp/tcp_flow.c ✅ ❌ Cloud Edge only
|
||
src/stcp/tcp_socket.c ✅ ❌ Cloud Edge only
|
||
src/cni/memif.c ✅ ❌ Cloud Edge only
|
||
src/cni/cni_api.c ✅ ❌ Cloud Edge only
|
||
src/cni/ipam.c ✅ ❌ Cloud Edge only
|
||
src/app/http_server.c ✅ ❌ Cloud Edge only
|
||
src/app/dns_server.c ✅ ❌ Cloud Edge only
|
||
src/app/tcp_proxy.c ✅ ❌ Cloud Edge only
|
||
src/mgmt/cli.c ✅ ✅ 共享 (命令集不同)
|
||
src/mgmt/rest.c ✅ ❌ Cloud Edge only
|
||
src/ha/vrrp.c ✅ ❌ Cloud Edge only
|
||
src/telemetry/prometheus.c ✅ ✅ 共享 (指标集不同)
|
||
```
|
||
|
||
### 3.3 关键差异点
|
||
|
||
```c
|
||
// src/core/main.c — lcore 分配
|
||
|
||
void unos_launch_lcores(void) {
|
||
#ifdef PRODUCT_CLOUD_EDGE
|
||
// T140: 6+ cores, full allocation
|
||
rte_eal_remote_launch(dataplane_forward_loop, NULL, 1); // lcore 1
|
||
rte_eal_remote_launch(tcp_stack_loop, NULL, 2); // lcore 2
|
||
rte_eal_remote_launch(memif_rx_loop, NULL, 3); // lcore 3 (optional)
|
||
// lcore 0: control plane (main thread)
|
||
// lcore 4-5: FRR + observability (external processes pinned)
|
||
#else
|
||
// J3455: 4 cores, minimal allocation
|
||
rte_eal_remote_launch(dataplane_forward_loop, NULL, 1); // lcore 1
|
||
// lcore 0: control plane (main thread)
|
||
// lcore 2: FRR ospfd (external process pinned)
|
||
// lcore 3: idle
|
||
#endif
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 四、数据面详细设计（生产级：RSS 多核 + Graph Pipeline）
|
||
|
||
### 4.0 架构决策
|
||
|
||
```
|
||
为什么不用固定 Pipeline（v1.0 设计）:
|
||
❌ 所有包走完全部阶段，即使不需要
|
||
❌ 单核转发，无法线性扩展
|
||
❌ TCP 栈在独立 lcore，跨核 IPC 引入延迟 + cache miss
|
||
|
||
为什么用 RSS + Graph Pipeline（v2.0 生产级）:
|
||
✅ RSS 把流量按五元组分散到多核，每个核独立运行，无锁、
|
||
线性扩展，加核就加吞吐
|
||
✅ Graph DAG 按包类型走不同路径，ARP 包跳过 NAT，
|
||
本地包直接进 TCP 栈，不经过多余阶段
|
||
✅ TCP 栈作为 Graph 节点内联在数据面，本机包不跨核
|
||
✅ Per-core Flow Cache 跳过 LPM 查找（首包查路由，
|
||
后续包命中 cache 直接转发）
|
||
✅ 对标 VPP (FD.io) 的 Graph Node 架构，业界验证过
|
||
```
|
||
|
||
### 4.1 Graph Pipeline 核心数据结构
|
||
|
||
```c
|
||
// src/dp/graph.h
|
||
|
||
#define MAX_NEXT_NODES 8
|
||
#define NODE_NAME_LEN 32
|
||
|
||
// 前向声明
|
||
struct graph_node;
|
||
|
||
// Graph Node 处理函数签名:
|
||
// 输入: mbuf 数组 + 包计数
|
||
// 输出: 每个包走哪个 next_nodes[idx] 的索引数组
|
||
// 返回: 实际处理的包数
|
||
typedef uint16_t (*graph_node_process_fn_t)(
|
||
struct rte_mbuf **mbufs,
|
||
uint16_t nb_pkts,
|
||
uint8_t *next_indices // [nb_pkts]: 每个包的下一跳
|
||
);
|
||
|
||
struct graph_node {
|
||
char name[NODE_NAME_LEN];
|
||
graph_node_process_fn_t process;
|
||
uint16_t next_count;
|
||
struct graph_node *next_nodes[MAX_NEXT_NODES];
|
||
char *next_labels[MAX_NEXT_NODES]; // 分支标签
|
||
|
||
// 统计 (per-lcore, cacheline 对齐)
|
||
uint64_t pkts_processed;
|
||
uint64_t cycles_spent;
|
||
uint64_t pkts_dropped;
|
||
} __rte_cache_aligned;
|
||
|
||
// Graph 路由辅助宏: 让 process_fn 返回的 next_indices 更有语义
|
||
enum {
|
||
NEXT_DROP = 0, // 丢包
|
||
NEXT_TX = 1, // 直接发送
|
||
NEXT_L2 = 2, // L2 转发
|
||
NEXT_L3 = 3, // L3 转发
|
||
NEXT_NAT = 4, // NAT 转换
|
||
NEXT_QOS = 5, // QoS 调度
|
||
NEXT_TCP = 6, // 进 TCP 栈
|
||
NEXT_ARP = 7, // ARP 处理
|
||
};
|
||
|
||
// 每个 worker 拥有独立的 Graph 实例
|
||
struct graph_worker {
|
||
struct rte_mbuf *rx_bufs[BURST_SIZE]; // 收包暂存
|
||
struct rte_mbuf *tx_bufs[RTE_MAX_ETHPORTS][BURST_SIZE];
|
||
uint16_t tx_counts[RTE_MAX_ETHPORTS];
|
||
|
||
struct rte_mbuf *tcp_rx_bufs[BURST_SIZE]; // TCP 栈的收包队列
|
||
|
||
// Per-core 状态表（无锁——RSS 保证流的亲和性）
|
||
struct flow_cache *flow_cache; // 流缓存
|
||
struct conntrack_table *ct_table; // 连接跟踪
|
||
struct tcp_stack *tcp_stack; // TCP 协议栈实例
|
||
|
||
// 全局只读状态（控制面写入，数据面只读）
|
||
struct rte_lpm *lpm_ipv4; // 路由表 (RSS workers 共享)
|
||
struct arp_cache *arp_cache; // ARP 表 (RSS workers 共享)
|
||
struct acl_engine *acl; // ACL (RSS workers 共享)
|
||
} __rte_cache_aligned;
|
||
```
|
||
|
||
### 4.2 Graph 拓扑（完整转发图）
|
||
|
||
```
|
||
┌─────────┐
|
||
│ RX │
|
||
│ (NIC RX │
|
||
│ queues)│
|
||
└────┬────┘
|
||
│
|
||
┌────┴────┐
|
||
│ Parser │ ← 所有包的入口
|
||
│ L2-L4 │
|
||
└────┬────┘
|
||
│
|
||
┌───────┼───────────┐
|
||
▼ ▼ ▼
|
||
dst_is_me ARP pkt L2 (MAC查表)
|
||
│ │ │
|
||
▼ ▼ ▼
|
||
┌────────┐ ┌────────┐ ┌─────────┐
|
||
│Classify│ │ARP │ │L2 Fwd │
|
||
│本机流 │ │Handler │ │MAC学习 │
|
||
└───┬────┘ └───┬────┘ └────┬────┘
|
||
│ │ │
|
||
┌────┬────┤ ▼ ▼
|
||
▼ ▼ ▼ ┌────────┐ ┌─────────┐
|
||
TCP DNS HTTP │ TX │ │ TX │
|
||
Stack Server Server └────────┘ └─────────┘
|
||
│
|
||
▼ (TCP栈处理后)
|
||
┌──────────┐
|
||
│ TX │
|
||
│ (本机发包)│
|
||
└──────────┘
|
||
|
||
┌─────────────────┐
|
||
│ dst != me │ ← 转发路径
|
||
│ (非本机IP) │
|
||
└────────┬────────┘
|
||
│
|
||
┌───────┴───────┐
|
||
▼ ▼
|
||
┌──────────┐ ┌──────────┐
|
||
│Flow Cache│ │ LPM │ ← 首包走 LPM，后续命中 cache
|
||
│(per-core)│ │(路由查找)│
|
||
└────┬─────┘ └────┬─────┘
|
||
│ │
|
||
│ miss │
|
||
└──────┬───────┘
|
||
▼
|
||
┌──────────┐
|
||
│ ACL │ ← 安全策略
|
||
└────┬─────┘
|
||
│
|
||
┌──────────┼──────────┐
|
||
▼ ▼ ▼
|
||
┌────────┐ ┌──────────┐ ┌──────┐
|
||
│ NAT │ │ No NAT │ │ DENY │
|
||
│ SNAT/ │ │(纯转发) │ │(丢弃)│
|
||
│ DNAT │ └────┬─────┘ └──┬───┘
|
||
└───┬────┘ │ │
|
||
│ │ rte_pktmbuf_free()
|
||
└─────┬─────┘
|
||
▼
|
||
┌──────────┐
|
||
│ ARP │ ← 查下一跳 MAC
|
||
│ Lookup │
|
||
└────┬─────┘
|
||
│
|
||
▼
|
||
┌──────────┐
|
||
│ QOS │ ← 入队 + 调度
|
||
│8队列调度 │
|
||
└────┬─────┘
|
||
│
|
||
▼
|
||
┌──────────┐
|
||
│ TX │
|
||
└──────────┘
|
||
```
|
||
|
||
### 4.3 数据面核心循环（每个 RSS worker 独立运行）
|
||
|
||
```c
|
||
// src/dp/worker.c — 每个 lcore worker 的主循环
|
||
|
||
static __rte_always_inline int
|
||
dataplane_worker_loop(void *arg) {
|
||
struct graph_worker *gw = (struct graph_worker *)arg;
|
||
struct rte_mbuf *pkts[BURST_SIZE];
|
||
uint8_t next_indices[BURST_SIZE];
|
||
uint16_t nb_pkts;
|
||
|
||
while (likely(!force_quit)) {
|
||
// ==== STEP 1: RX — 轮询本 worker 独占的 RSS queues ====
|
||
uint16_t total_rx = 0;
|
||
for (int port = 0; port < gw->port_count; port++) {
|
||
nb_pkts = rte_eth_rx_burst(port, gw->queue_id, pkts + total_rx,
|
||
BURST_SIZE - total_rx);
|
||
if (nb_pkts == 0) continue;
|
||
|
||
// 预取第一层 Graph Node（parser_node）的数据
|
||
for (int i = 0; i < nb_pkts; i++)
|
||
rte_prefetch0(rte_pktmbuf_mtod(pkts[total_rx + i], void *));
|
||
total_rx += nb_pkts;
|
||
}
|
||
if (unlikely(total_rx == 0)) continue;
|
||
|
||
// ==== STEP 2: Graph Traversal — 每个包沿 DAG 路径前进 ====
|
||
// 用 SIMD 批次追踪每个包当前所在的 node
|
||
struct graph_node *current_nodes[BURST_SIZE]; // 每个包当前在哪个 node
|
||
uint16_t active_mask = 0; // bitmask: 还有包没走完
|
||
|
||
// 所有包起点: Parser Node
|
||
struct graph_node *first = &gw->graph->parser_node;
|
||
for (int i = 0; i < total_rx; i++) {
|
||
current_nodes[i] = first;
|
||
active_mask |= (1 << i);
|
||
}
|
||
|
||
while (active_mask) {
|
||
// 按当前 node 分组 → 同一 node 的包 batch 处理
|
||
for (int batch_start = 0; batch_start < total_rx; ) {
|
||
if (!(active_mask & (1 << batch_start))) {
|
||
batch_start++;
|
||
continue;
|
||
}
|
||
|
||
// 收集所有在同一个 node 的包
|
||
struct graph_node *batch_node = current_nodes[batch_start];
|
||
uint16_t batch_nb = 0;
|
||
struct rte_mbuf *batch_pkts[BURST_SIZE];
|
||
|
||
for (int i = batch_start; i < total_rx; i++) {
|
||
if (current_nodes[i] == batch_node) {
|
||
batch_pkts[batch_nb++] = pkts[i];
|
||
active_mask &= ~(1 << i); // 从活跃集移除
|
||
}
|
||
}
|
||
|
||
// 对该 node 的包 batch 处理
|
||
if (batch_node->next_count == 0) {
|
||
// 终端节点 (DROP / TX): 直接释放或发送
|
||
if (batch_node == &gw->graph->tx_node)
|
||
gw_tx_burst(gw, batch_pkts, batch_nb);
|
||
else // drop_node
|
||
for (int i = 0; i < batch_nb; i++)
|
||
rte_pktmbuf_free(batch_pkts[i]);
|
||
} else {
|
||
// 内部节点: 调用 process_fn
|
||
uint16_t processed = batch_node->process(
|
||
batch_pkts, batch_nb, next_indices);
|
||
|
||
// 更新每个包的下一跳
|
||
for (int i = 0; i < processed; i++) {
|
||
uint8_t nidx = next_indices[i];
|
||
if (nidx < batch_node->next_count) {
|
||
current_nodes[batch_start + i] =
|
||
batch_node->next_nodes[nidx];
|
||
active_mask |= (1 << (batch_start + i));
|
||
} else {
|
||
// 错误: 丢包
|
||
rte_pktmbuf_free(batch_pkts[i]);
|
||
}
|
||
}
|
||
// 处理失败的包
|
||
for (int i = processed; i < batch_nb; i++)
|
||
rte_pktmbuf_free(batch_pkts[i]);
|
||
}
|
||
|
||
// 统计
|
||
batch_node->pkts_processed += batch_nb;
|
||
batch_start = batch_start + batch_nb; // 跳过已分组的包
|
||
}
|
||
|
||
// 重新调整 active 包的索引 (释放掉的包对应的位置清空)
|
||
total_rx = compact_active_packets(pkts, current_nodes, total_rx,
|
||
&active_mask);
|
||
}
|
||
|
||
// ==== STEP 3: TX — 批量发送所有出端口队列 ====
|
||
for (int port = 0; port < gw->port_count; port++) {
|
||
if (gw->tx_counts[port] > 0) {
|
||
uint16_t sent = rte_eth_tx_burst(port, gw->queue_id,
|
||
gw->tx_bufs[port],
|
||
gw->tx_counts[port]);
|
||
if (unlikely(sent < gw->tx_counts[port])) {
|
||
for (int i = sent; i < gw->tx_counts[port]; i++)
|
||
rte_pktmbuf_free(gw->tx_bufs[port][i]);
|
||
}
|
||
gw->tx_counts[port] = 0;
|
||
}
|
||
}
|
||
}
|
||
return 0;
|
||
}
|
||
```
|
||
|
||
### 4.4 Per-Core Flow Cache（关键优化）
|
||
|
||
每个 RSS worker 拥有独立的 Flow Cache，避免对 Established 流的重复 LPM 查找：
|
||
|
||
```c
|
||
// src/dp/flow_cache.h
|
||
|
||
// Flow Cache 条目
|
||
struct flow_cache_entry {
|
||
uint32_t src_ip;
|
||
uint32_t dst_ip;
|
||
uint16_t src_port;
|
||
uint16_t dst_port;
|
||
uint8_t protocol;
|
||
uint8_t action; // ACTION_FWD / ACTION_NAT / ACTION_DROP
|
||
uint8_t out_port; // 出端口
|
||
struct rte_ether_addr dst_mac; // 下一跳 MAC (不变时缓存)
|
||
uint32_t nat_ip; // NAT 后的地址 (不变时缓存)
|
||
uint16_t nat_port; // NAT 后的端口
|
||
uint64_t last_seen;
|
||
} __rte_cache_aligned;
|
||
|
||
// Flow Cache 查找 — 在 Parser 后直接调用
|
||
// 命中 → 跳过 LPM + ACL + NAT，直接填 MAC + 发往 QoS → TX
|
||
int flow_cache_lookup(struct flow_cache *fc, struct parser_result *pr,
|
||
struct flow_cache_entry *result) {
|
||
uint32_t hash_key = pr->src_ip ^ pr->dst_ip ^
|
||
(pr->src_port << 16 | pr->dst_port) ^ pr->protocol;
|
||
uint32_t bucket = hash_key & fc->mask;
|
||
|
||
// 开放寻址 (cuckoo hash, 最多查 2 个 bucket)
|
||
// 命中 → O(1), no locks
|
||
// miss → 走完整 LPM + ACL + NAT 路径，然后插入 cache
|
||
}
|
||
```
|
||
|
||
### 4.5 核心数据结构（不变部分，续用 v1.0 设计）
|
||
|
||
以下结构体保持 v1.0 设计，仅从固定 Pipeline 改为 Graph Node 使用：
|
||
|
||
#### Parser Result (见 v1.0 的 4.1.3)
|
||
|
||
```c
|
||
// src/dp/parser.h — 保持不变，但增加 graph_next 字段
|
||
struct parser_result {
|
||
// ... (同 v1.0 设计) ...
|
||
|
||
uint8_t graph_next; // 新增: Parser 根据包类型设置下一跳
|
||
// PARSER_NEXT_L2 (ether_type == 0x0800 && dst_mac is not me)
|
||
// PARSER_NEXT_L3 (ether_type == 0x0800 && dst_mac is me)
|
||
// PARSER_NEXT_ARP (ether_type == 0x0806)
|
||
// PARSER_NEXT_DROP (unknown protocol)
|
||
};
|
||
```
|
||
|
||
#### L3 Forwarder (见 v1.0 的 4.1.4)
|
||
|
||
```c
|
||
// 不变，但变为所有 RSS worker 共享的只读数据
|
||
// 控制面通过 IPC 更新 (RCU 模式: 控制面写入新副本，数据面原子切换指针)
|
||
```
|
||
|
||
#### NAT / Connection Track (见 v1.0 的 4.1.5)
|
||
|
||
```c
|
||
// Per-core 的 Connection Track Table
|
||
// 关键变化: 每个 RSS worker 拥有独立的 ct_table，因为
|
||
// RSS 保证同一五元组永远落在同一个 worker，所以无需跨核锁
|
||
#endif
|
||
|
||
---
|
||
|
||
## 五、控制面详细设计
|
||
|
||
### 5.1 IPC 消息协议
|
||
|
||
```c
|
||
// src/core/ipc.h
|
||
|
||
#define IPC_MSG_MAX_PAYLOAD 1024
|
||
#define IPC_RING_SIZE 2048
|
||
|
||
enum ipc_msg_type {
|
||
IPC_MSG_ROUTE_ADD = 1,
|
||
IPC_MSG_ROUTE_DEL,
|
||
IPC_MSG_INTERFACE_UP,
|
||
IPC_MSG_INTERFACE_DOWN,
|
||
IPC_MSG_ACL_ADD,
|
||
IPC_MSG_ACL_DEL,
|
||
IPC_MSG_NAT_CONFIG,
|
||
IPC_MSG_QOS_CONFIG,
|
||
IPC_MSG_STATS_REPORT,
|
||
IPC_MSG_HEALTH_REPORT,
|
||
#ifdef PRODUCT_CLOUD_EDGE
|
||
IPC_MSG_MEMIF_CREATE,
|
||
IPC_MSG_MEMIF_DELETE,
|
||
IPC_MSG_CNI_ADD,
|
||
IPC_MSG_CNI_DEL,
|
||
#endif
|
||
};
|
||
|
||
struct ipc_msg {
|
||
uint32_t type; // enum ipc_msg_type
|
||
uint32_t seq_num; // 序列号 (用于配对 request/response)
|
||
uint32_t payload_len; // payload 实际长度
|
||
uint32_t _pad;
|
||
uint64_t timestamp; // rdtsc
|
||
uint8_t payload[IPC_MSG_MAX_PAYLOAD];
|
||
} __rte_cache_aligned;
|
||
|
||
// 控制面 → 数据面: cp_to_dp_ring (MPSC: 控制面单生产者, 数据面单消费者)
|
||
// 数据面 → 控制面: dp_to_cp_ring (SPMC: 数据面单生产者, 控制面单消费者)
|
||
```
|
||
|
||
### 5.2 控制面主循环
|
||
|
||
```c
|
||
// src/cp/control.c
|
||
|
||
void control_plane_loop(void) {
|
||
while (!force_quit) {
|
||
// 1. 定时器处理 (ARP 老化, 统计收集, 健康检查)
|
||
rte_timer_manage();
|
||
|
||
// 2. 处理数据面上报 (统计 + 事件)
|
||
struct ipc_msg msg;
|
||
while (rte_ring_dequeue(dp_to_cp_ring, &msg) == 0) {
|
||
switch (msg.type) {
|
||
case IPC_MSG_STATS_REPORT:
|
||
handle_stats_report(&msg);
|
||
break;
|
||
case IPC_MSG_HEALTH_REPORT:
|
||
handle_health_report(&msg);
|
||
break;
|
||
}
|
||
}
|
||
|
||
// 3. 处理 Redis Pub/Sub 消息 (配置变更)
|
||
redis_handle_pubsub();
|
||
|
||
// 4. CLI 连接 (Unix socket accept)
|
||
cli_accept_connections();
|
||
|
||
// 5. FRR 路由同步 (Netlink socket poll, non-blocking)
|
||
frr_poll_netlink();
|
||
}
|
||
}
|
||
```
|
||
|
||
### 5.3 Redis DB Schema
|
||
|
||
```
|
||
CONFIG_DB (持久化):
|
||
ROUTE|<vrf>|<prefix> → {"nexthop":"10.0.0.1","iface":"wan0","metric":10}
|
||
INTERFACE|<name> → {"ip":"192.168.1.1/24","mtu":1500,"state":"up"}
|
||
ACL|<rule_id> → {"priority":100,"action":"permit","src":"10.0.0.0/8","dst":"0.0.0.0/0","proto":"tcp","dport":"80"}
|
||
NAT|<rule_id> → {"type":"snat","src":"10.0.0.0/8","nat_ip":"203.0.113.1"}
|
||
QOS|<port>|<queue_id> → {"scheduler":"sp","weight":0}
|
||
#ifdef PRODUCT_CLOUD_EDGE
|
||
CNI|<container_id> → {"ip":"10.244.1.5/24","memif_socket":"/tmp/unos-memif-abc123"}
|
||
#endif
|
||
|
||
STATE_DB (运行时, 不持久化):
|
||
ARP|<ip> → {"mac":"00:11:22:33:44:55","iface":"wan0","age":12}
|
||
CONNTRACK|summary → {"total":1234,"tcp":1000,"udp":200,"icmp":34}
|
||
INTERFACE|<name>|status → {"admin":"up","oper":"up","speed":10000}
|
||
|
||
STATS_DB (运行时):
|
||
PORT|<name>|stats → {"rx_bytes":1234567890,"tx_bytes":9876543210,"rx_pps":1234567}
|
||
#ifdef PRODUCT_CLOUD_EDGE
|
||
CONTAINER|<id>|stats → {"rx_bytes":1234,"tx_bytes":5678,"memif_errors":0}
|
||
#endif
|
||
```
|
||
|
||
---
|
||
|
||
## 六、自研 TCP/IP 协议栈详细设计
|
||
|
||
### 6.0 架构决策: TCP 栈在 Graph 中内联
|
||
|
||
```
|
||
v1.0 (弃用): TCP 栈在独立 lcore → 跨核 IPC → 额外拷贝 + cache miss
|
||
v2.0 (生产级): TCP 栈作为 Graph Node → 同一 lcore 直接函数调用 → 零跨核开销
|
||
|
||
Graph 路径:
|
||
物理口 RX → Parser → (dst_is_me?) → TCP Stack Node → App → TX
|
||
│
|
||
└── (dst_is_not_me) → L2/L3/ACL/NAT → TX
|
||
|
||
本机收包路径 (全程在同一个 worker lcore 内):
|
||
rte_eth_rx_burst → Parser → graph_next=NEXT_TCP
|
||
→ tcp_stack_process_node() ← 这就是 TCP 栈入口
|
||
内部:
|
||
1. IP 层处理 (分片重组, 校验)
|
||
2. TCP 层处理 (状态机, 序列号, 重传)
|
||
3. 应用层数据放入 recv buffer
|
||
4. 待发送数据构造 mbuf → TX ring
|
||
→ 应用层 (通过 tcp_recv() 获取数据)
|
||
|
||
同一个 mbuf 全程不拷贝，不跨核，不经过 rte_ring IPC。
|
||
```
|
||
|
||
### 6.1 TCP Stack Graph Node
|
||
|
||
```c
|
||
// src/stcp/tcp_node.c — TCP 栈作为 Graph 处理的实现
|
||
|
||
static uint16_t
|
||
tcp_stack_process_node(struct rte_mbuf **mbufs, uint16_t nb_pkts,
|
||
uint8_t *next_indices)
|
||
{
|
||
for (int i = 0; i < nb_pkts; i++) {
|
||
struct rte_mbuf *m = mbufs[i];
|
||
struct parser_result pr;
|
||
parser_process(m, &pr); // 快速解析
|
||
|
||
// === IP 层处理 ===
|
||
if (pr.ip_version == 4) {
|
||
if (unlikely(!ipv4_validate_checksum(m)))
|
||
goto drop;
|
||
// 分片重组
|
||
if (unlikely(ipv4_is_fragment(m))) {
|
||
m = ipv4_reassemble(m);
|
||
if (m == NULL) continue; // 还没收全
|
||
}
|
||
}
|
||
|
||
// === TCP 层处理 ===
|
||
struct tcp_control_block *tcb;
|
||
tcb = tcb_lookup(pr.src_ip, pr.dst_ip, pr.src_port, pr.dst_port);
|
||
|
||
if (unlikely(tcb == NULL && (pr.tcp_flags & TCP_FLAG_SYN))) {
|
||
// 新连接请求 → 查 LISTEN socket → 创建 TCB
|
||
tcb = tcp_listen_accept(&pr);
|
||
}
|
||
|
||
if (unlikely(tcb == NULL))
|
||
goto drop;
|
||
|
||
// HTTP Server / DNS 跑在 STCP Socket API 上
|
||
// 当 recv buffer 有数据 → 唤醒应用层 epoll → HTTP handler 调用 tcp_send() 回复
|
||
|
||
// 状态机转换
|
||
tcp_state_process(tcb, &pr, m);
|
||
|
||
// 如果 tcp_state_process 内部调用了 tcp_send()，TX mbuf 已经入队
|
||
// 如果是纯 ACK，不需要回复，直接释放
|
||
|
||
// 本机接收路径: mbuf 交给 recv buffer
|
||
// 本机发送路径: TCP 栈自己构造的 mbuf 在 TX ring 里，
|
||
// 在 graph traversal 的 TX 阶段发出
|
||
continue;
|
||
|
||
drop:
|
||
next_indices[i] = NEXT_DROP;
|
||
}
|
||
return nb_pkts;
|
||
}
|
||
|
||
// 这个函数注册为 Graph Node:
|
||
REGISTER_GRAPH_NODE(tcp_stack, tcp_stack_process_node, {
|
||
{ NEXT_TCP, "tcp_stack" }, // 回环 (重传/ACK)
|
||
{ NEXT_TX, "tx" }, // 出队发送
|
||
{ NEXT_DROP, "drop" }, // 错误丢包
|
||
});
|
||
```
|
||
|
||
### 6.2 TCB (TCP Control Block) — 不变
|
||
|
||
(参见 v1.0 设计，TCB 结构不变。区别仅在于 TCB 挂载在 per-worker 的 tcp_stack 实例中，而非全局哈希表)
|
||
|
||
### 6.3 TCP 状态机 — 不变
|
||
|
||
(11 状态转换表不变，参见 v1.0 6.2 节)
|
||
|
||
### 6.4 数据面 TCP 的收发包路径
|
||
|
||
```
|
||
收到 SYN (dst_ip == 本机 IP, dst_port == 80):
|
||
─────────────────────────────────────────────
|
||
物理口 RX:
|
||
→ Parser: ether_type=0x0800, protocol=TCP, dst_port=80, SYN=1
|
||
→ graph_next = NEXT_TCP
|
||
|
||
TCP Stack Node:
|
||
→ tcb_lookup() → NULL (新连接)
|
||
→ tcp_listen_accept() → 创建 TCB (状态=SYN_RCVD)
|
||
→ tcp_send_syn_ack(tcb) → 构造 SYN+ACK mbuf
|
||
→ mbuf 入 TX ring
|
||
→ 数据面 TX 阶段发出
|
||
|
||
收到 ACK (of SYN+ACK):
|
||
→ tcb_lookup() → 找到 TCB
|
||
→ tcp_state_process() → SYN_RCVD → ESTABLISHED
|
||
→ 无应用数据，不唤醒
|
||
|
||
收到 HTTP GET (ESTABLISHED 状态下):
|
||
→ tcp_state_process() → 验证 seq, 更新 rcv_nxt, 回复 ACK
|
||
→ 应用数据放入 tcb->rcv_buf
|
||
→ 唤醒 HTTP handler (通过 eventfd)
|
||
→ HTTP handler 调用 tcp_send(tcb->sockfd, response, len)
|
||
→ TCP 栈构造 DATA mbuf → TX ring → 发出
|
||
|
||
关闭连接:
|
||
→ 客户端发 FIN → tcp_state: ESTABLISHED → CLOSE_WAIT
|
||
→ HTTP handler 收到 EOF → close(sockfd)
|
||
→ tcp_send_fin(tcb) → tcp_state: CLOSE_WAIT → LAST_ACK → 发 FIN
|
||
→ 收到 ACK → CLOSED → 删除 TCB
|
||
```
|
||
|
||
```c
|
||
// src/stcp/tcb.h
|
||
|
||
#define TCB_HASH_BUCKETS 65536
|
||
#define TCB_MAX_CONNECTIONS 100000 // Cloud Edge
|
||
#define TCP_MSS_DEFAULT 1460
|
||
#define TCP_RTO_MIN_MS 200
|
||
#define TCP_RTO_MAX_MS 120000
|
||
#define TCP_2MSL_MS 120000 // TIME_WAIT duration
|
||
|
||
enum tcp_state {
|
||
TCP_CLOSED = 0,
|
||
TCP_LISTEN,
|
||
TCP_SYN_SENT,
|
||
TCP_SYN_RCVD,
|
||
TCP_ESTABLISHED,
|
||
TCP_FIN_WAIT_1,
|
||
TCP_FIN_WAIT_2,
|
||
TCP_CLOSE_WAIT,
|
||
TCP_CLOSING,
|
||
TCP_LAST_ACK,
|
||
TCP_TIME_WAIT,
|
||
};
|
||
|
||
struct tcp_control_block {
|
||
// === 四元组 ===
|
||
uint32_t src_ip;
|
||
uint32_t dst_ip;
|
||
uint16_t src_port;
|
||
uint16_t dst_port;
|
||
|
||
// === 状态 ===
|
||
enum tcp_state state;
|
||
|
||
// === 发送序列号 ===
|
||
uint32_t iss; // Initial Send Sequence
|
||
uint32_t snd_una; // 最老未确认的序列号
|
||
uint32_t snd_nxt; // 下一个要发送的序列号
|
||
uint32_t snd_wnd; // 发送窗口
|
||
uint32_t snd_wl1; // 上次窗口更新的 seq
|
||
uint32_t snd_wl2; // 上次窗口更新的 ack
|
||
|
||
// === 接收序列号 ===
|
||
uint32_t irs; // Initial Receive Sequence
|
||
uint32_t rcv_nxt; // 期望收到的下一个序列号
|
||
uint32_t rcv_wnd; // 接收窗口 (通告给对方)
|
||
uint32_t rcv_adv; // 上次通告的窗口右边界
|
||
|
||
// === 拥塞控制 ===
|
||
uint32_t cwnd; // 拥塞窗口 (MSS 为单位)
|
||
uint32_t ssthresh; // 慢启动阈值
|
||
uint32_t cwnd_cnt; // cwnd 增长计数器
|
||
|
||
// === RTT / RTO ===
|
||
uint32_t srtt_us; // Smoothed RTT (微秒)
|
||
uint32_t rttvar_us; // RTT 方差 (微秒)
|
||
uint32_t rto_us; // 当前 RTO (微秒)
|
||
uint64_t rto_deadline; // rte_rdtsc 到期时间
|
||
uint8_t backoff; // 指数退避次数 (0-6)
|
||
uint64_t last_rtt_ts; // 上次 RTT 测量时间
|
||
|
||
// === 重传 ===
|
||
uint32_t dup_ack_count; // 重复 ACK 计数 (快速重传)
|
||
uint32_t retrans_count; // 连续重传次数
|
||
uint64_t last_data_ts; // 最后发送数据的时间
|
||
|
||
// === Options ===
|
||
uint16_t mss; // 协商的 MSS
|
||
uint8_t wscale_shift; // Window scale shift
|
||
uint8_t sack_permitted; // 对端支持 SACK
|
||
|
||
// === 发送缓冲区 (环形) ===
|
||
uint8_t *snd_buf;
|
||
uint32_t snd_buf_size;
|
||
uint32_t snd_buf_head; // 写入指针
|
||
uint32_t snd_buf_tail; // 确认指针
|
||
|
||
// === 接收缓冲区 (环形) ===
|
||
uint8_t *rcv_buf;
|
||
uint32_t rcv_buf_size;
|
||
uint32_t rcv_buf_head; // 写入指针
|
||
uint32_t rcv_buf_tail; // 读取指针
|
||
|
||
// === Socket 层 ===
|
||
int sockfd;
|
||
uint8_t nonblocking;
|
||
void *user_data; // 应用层上下文
|
||
|
||
// === 链表 ===
|
||
struct tcp_control_block *next; // 哈希冲突链表
|
||
struct tcp_control_block *prev;
|
||
uint64_t tcb_id; // 唯一 ID
|
||
|
||
} __rte_cache_aligned;
|
||
```
|
||
|
||
### 6.2 TCP 状态机实现
|
||
|
||
```c
|
||
// src/stcp/tcp_state.c
|
||
|
||
// 事件类型
|
||
enum tcp_event {
|
||
TCP_EV_SYN,
|
||
TCP_EV_SYN_ACK,
|
||
TCP_EV_ACK,
|
||
TCP_EV_FIN,
|
||
TCP_EV_RST,
|
||
TCP_EV_TIMEOUT,
|
||
TCP_EV_CLOSE, // 应用层调用 close()
|
||
TCP_EV_SEND, // 应用层调用 send()
|
||
};
|
||
|
||
// 每个状态的事件处理表
|
||
typedef int (*tcp_state_handler_t)(struct tcp_control_block *tcb,
|
||
struct tcp_event event,
|
||
void *data);
|
||
|
||
// 状态转换表: [state][event] → new_state + handler
|
||
static const struct {
|
||
enum tcp_state new_state;
|
||
tcp_state_handler_t handler;
|
||
} tcp_transition[11][8] = {
|
||
// TCP_CLOSED
|
||
[TCP_CLOSED] = {
|
||
[TCP_EV_SYN] = { TCP_SYN_RCVD, tcp_handle_listen_syn },
|
||
[TCP_EV_SEND] = { TCP_SYN_SENT, tcp_handle_active_open },
|
||
},
|
||
// TCP_LISTEN
|
||
[TCP_LISTEN] = {
|
||
[TCP_EV_SYN] = { TCP_SYN_RCVD, tcp_handle_listen_syn },
|
||
[TCP_EV_RST] = { TCP_LISTEN, tcp_handle_listen_rst },
|
||
},
|
||
// TCP_SYN_SENT
|
||
[TCP_SYN_SENT] = {
|
||
[TCP_EV_SYN_ACK]= { TCP_ESTABLISHED, tcp_handle_syn_sent_syn_ack },
|
||
[TCP_EV_SYN] = { TCP_SYN_RCVD, tcp_handle_simultaneous_open },
|
||
[TCP_EV_RST] = { TCP_CLOSED, tcp_handle_rst },
|
||
[TCP_EV_TIMEOUT]= { TCP_CLOSED, tcp_handle_timeout },
|
||
},
|
||
// TCP_SYN_RCVD
|
||
[TCP_SYN_RCVD] = {
|
||
[TCP_EV_ACK] = { TCP_ESTABLISHED, tcp_handle_syn_rcvd_ack },
|
||
[TCP_EV_RST] = { TCP_LISTEN, tcp_handle_syn_rcvd_rst },
|
||
[TCP_EV_TIMEOUT]= { TCP_CLOSED, tcp_handle_timeout },
|
||
},
|
||
// TCP_ESTABLISHED
|
||
[TCP_ESTABLISHED] = {
|
||
[TCP_EV_FIN] = { TCP_CLOSE_WAIT, tcp_handle_established_fin },
|
||
[TCP_EV_CLOSE] = { TCP_FIN_WAIT_1, tcp_handle_close },
|
||
[TCP_EV_RST] = { TCP_CLOSED, tcp_handle_rst },
|
||
},
|
||
// TCP_FIN_WAIT_1
|
||
[TCP_FIN_WAIT_1] = {
|
||
[TCP_EV_ACK] = { TCP_FIN_WAIT_2, tcp_handle_fin_wait1_ack },
|
||
[TCP_EV_FIN] = { TCP_TIME_WAIT, tcp_handle_fin_wait1_fin },
|
||
[TCP_EV_FIN|TCP_EV_ACK] = { TCP_TIME_WAIT, tcp_handle_simultaneous_close },
|
||
[TCP_EV_RST] = { TCP_CLOSED, tcp_handle_rst },
|
||
},
|
||
// TCP_FIN_WAIT_2
|
||
[TCP_FIN_WAIT_2] = {
|
||
[TCP_EV_FIN] = { TCP_TIME_WAIT, tcp_handle_fin_wait2_fin },
|
||
[TCP_EV_RST] = { TCP_CLOSED, tcp_handle_rst },
|
||
},
|
||
// TCP_CLOSE_WAIT
|
||
[TCP_CLOSE_WAIT] = {
|
||
[TCP_EV_CLOSE] = { TCP_LAST_ACK, tcp_handle_close_wait_close },
|
||
[TCP_EV_RST] = { TCP_CLOSED, tcp_handle_rst },
|
||
},
|
||
// TCP_CLOSING
|
||
[TCP_CLOSING] = {
|
||
[TCP_EV_ACK] = { TCP_TIME_WAIT, tcp_handle_closing_ack },
|
||
[TCP_EV_RST] = { TCP_CLOSED, tcp_handle_rst },
|
||
},
|
||
// TCP_LAST_ACK
|
||
[TCP_LAST_ACK] = {
|
||
[TCP_EV_ACK] = { TCP_CLOSED, tcp_handle_last_ack_ack },
|
||
[TCP_EV_RST] = { TCP_CLOSED, tcp_handle_rst },
|
||
},
|
||
// TCP_TIME_WAIT
|
||
[TCP_TIME_WAIT] = {
|
||
[TCP_EV_TIMEOUT]= { TCP_CLOSED, tcp_handle_time_wait_timeout },
|
||
[TCP_EV_RST] = { TCP_CLOSED, tcp_handle_rst },
|
||
},
|
||
};
|
||
```
|
||
|
||
---
|
||
|
||
## 七、容器网络详细设计
|
||
|
||
### 7.1 memif 接口架构
|
||
|
||
```
|
||
┌───────────────────────────────────────────────────────┐
|
||
│ Host (uNOS) │
|
||
│ │
|
||
│ ┌──────────────────┐ ┌──────────────────┐ │
|
||
│ │ memif master │ │ memif master │ │
|
||
│ │ socket: /tmp/ │ │ socket: /tmp/ │ │
|
||
│ │ unos-memif-1 │ │ unos-memif-2 │ │
|
||
│ └────────┬─────────┘ └────────┬─────────┘ │
|
||
│ │ shared memory │ shared memory │
|
||
│ │ (hugepage) │ (hugepage) │
|
||
│ ┌────────┴───────────────────────┴─────────┐ │
|
||
│ │ uNOS Data Plane │ │
|
||
│ │ memif1 → Pipeline → memif2 │ │
|
||
│ │ (container-to-container: zero copy) │ │
|
||
│ └───────────────────────────────────────────┘ │
|
||
└───────────────────────────────────────────────────────┘
|
||
│ │
|
||
┌────┴────┐ ┌────┴────┐
|
||
│Container│ │Container│
|
||
│ A │ │ B │
|
||
│ (nginx) │ │ (redis) │
|
||
│ │ │ │
|
||
│ memif │ │ memif │
|
||
│ slave │ │ slave │
|
||
└─────────┘ └─────────┘
|
||
```
|
||
|
||
### 7.2 memif 接口管理
|
||
|
||
```c
|
||
// src/cni/memif.h
|
||
|
||
#define MAX_MEMIF_SOCKETS 128
|
||
#define MEMIF_SOCKET_DIR "/var/run/unos/memif/"
|
||
|
||
struct memif_port {
|
||
int id; // memif interface ID
|
||
char socket_path[256]; // /var/run/unos/memif/container-xxx
|
||
uint8_t role; // MEMIF_ROLE_MASTER (uNOS) / MEMIF_ROLE_SLAVE (container)
|
||
struct rte_mempool *mempool; // per-port mempool
|
||
struct rte_ring *rx_queue; // RX ring (shared memory)
|
||
struct rte_ring *tx_queue; // TX ring (shared memory)
|
||
|
||
// === uNOS data plane reference ===
|
||
uint8_t unos_port_id; // uNOS internal port ID
|
||
struct rte_ether_addr mac; // container side MAC
|
||
uint32_t ip; // container IP (from IPAM)
|
||
uint8_t prefix_len;
|
||
|
||
// === Stats ===
|
||
uint64_t rx_pkts;
|
||
uint64_t tx_pkts;
|
||
uint64_t rx_bytes;
|
||
uint64_t tx_bytes;
|
||
} __rte_cache_aligned;
|
||
|
||
struct memif_manager {
|
||
struct memif_port ports[MAX_MEMIF_SOCKETS];
|
||
int port_count;
|
||
struct rte_hash *port_hash; // socket_path → index
|
||
};
|
||
```
|
||
|
||
### 7.3 CNI Plugin 接口
|
||
|
||
```c
|
||
// src/cni/cni_api.h
|
||
|
||
// CNI 标准 JSON 结构
|
||
struct cni_args {
|
||
char container_id[64];
|
||
char netns_path[256];
|
||
char ifname[16]; // e.g. "eth0"
|
||
char args[1024]; // extra CNI_ARGS
|
||
};
|
||
|
||
struct cni_result {
|
||
char ip_addr[64]; // e.g. "10.244.1.5/24"
|
||
char gateway[32]; // e.g. "10.244.1.1"
|
||
char routes[4096]; // JSON array of routes
|
||
char dns[256]; // JSON object of DNS config
|
||
};
|
||
|
||
// CNI 操作
|
||
// /opt/cni/bin/unos-cni ADD < CNI_CONFIG_JSON
|
||
// /opt/cni/bin/unos-cni DEL < CNI_CONFIG_JSON
|
||
// /opt/cni/bin/unos-cni CHECK < CNI_CONFIG_JSON (optional)
|
||
|
||
// uNOS CNI 实现:
|
||
// ADD:
|
||
// 1. 解析 CNI JSON → 提取 container_id, netns_path, ifname
|
||
// 2. 连接 uNOS Unix socket (/var/run/unos/cni.sock)
|
||
// 3. 发送 IPC_MSG_CNI_ADD → uNOS 控制面:
|
||
// a. IPAM 分配 IP (10.244.x.y/24)
|
||
// b. 创建 memif master socket
|
||
// c. 返回 memif socket path + IP config
|
||
// 4. CNI plugin 创建 memif slave 在容器 netns 内
|
||
// 5. 返回 CNI result JSON
|
||
//
|
||
// DEL:
|
||
// 1. 连接 uNOS Unix socket
|
||
// 2. 发送 IPC_MSG_CNI_DEL → uNOS 控制面:
|
||
// a. 删除 memif socket
|
||
// b. 回收 IP 地址
|
||
// 3. CNI plugin 清理容器网络命名空间
|
||
```
|
||
|
||
### 7.4 IPAM 设计
|
||
|
||
```c
|
||
// src/cni/ipam.h
|
||
|
||
#define POD_CIDR "10.244.0.0/16" // 整体 Pod 网络
|
||
#define NODE_CIDR "10.244.0.0/24" // 本节点子网
|
||
#define GATEWAY_IP "10.244.0.1" // 网关 IP (uNOS 自己)
|
||
|
||
struct ipam_pool {
|
||
uint32_t subnet; // 10.244.0.0
|
||
uint8_t prefix_len; // 24
|
||
uint64_t allocated_bitmap[4];// 256 IPs bitmap
|
||
uint32_t allocated_count;
|
||
uint32_t last_allocated; // 上次分配的 IP (减少扫描)
|
||
};
|
||
|
||
struct ipam {
|
||
struct ipam_pool pool;
|
||
uint32_t gateway_ip; // 10.244.0.1
|
||
struct rte_hash *allocations; // container_id → IP
|
||
};
|
||
|
||
// IP 分配算法: bitmap + last_allocated 加速
|
||
static inline int
|
||
ipam_allocate(struct ipam *ipam, uint32_t *ip_out) {
|
||
int host_id = -1;
|
||
int start = ipam->pool.last_allocated;
|
||
|
||
// 从上次位置开始扫描 (利用局部性)
|
||
for (int i = 0; i < 254; i++) {
|
||
int idx = (start + i) % 254;
|
||
if (!(ipam->pool.allocated_bitmap[idx / 64] & (1ULL << (idx % 64)))) {
|
||
host_id = idx + 2; // skip .0 (network) and .1 (gateway)
|
||
break;
|
||
}
|
||
}
|
||
if (host_id < 0)
|
||
return -1; // pool exhausted
|
||
|
||
ipam->pool.allocated_bitmap[host_id / 64] |= (1ULL << (host_id % 64));
|
||
ipam->pool.allocated_count++;
|
||
ipam->pool.last_allocated = host_id;
|
||
*ip_out = ipam->pool.subnet | host_id;
|
||
return 0;
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 八、管理面详细设计
|
||
|
||
### 8.1 CLI 命令树
|
||
|
||
```
|
||
Command Tree (vtysh style):
|
||
─────────────────────────────
|
||
|
||
unos> enable
|
||
unos# show [running-config|interfaces|route|arp|nat|acl|qos|system|tech-support]
|
||
unos# configure terminal
|
||
unos(config)#
|
||
|
||
[interface <ifname>]
|
||
[ip address <ip/len>]
|
||
[ipv6 address <ipv6/len>]
|
||
[mtu <1500|9000>]
|
||
[shutdown|no shutdown]
|
||
[description <text>]
|
||
[switchport mode <access|trunk>]
|
||
[switchport access vlan <vlan_id>]
|
||
[switchport trunk allowed vlan <vlan_list>]
|
||
|
||
[router ospf]
|
||
[router-id <ip>]
|
||
[network <ip/len> area <area_id>]
|
||
[redistribute static]
|
||
|
||
[router bgp <asn>]
|
||
[neighbor <ip> remote-as <asn>]
|
||
[network <ip/len>]
|
||
[redistribute connected]
|
||
|
||
[ip route <prefix> <nexthop> [metric <n>]]
|
||
[ip route <prefix> null0] # 黑洞路由
|
||
|
||
[access-list <100-199> <permit|deny> <proto> <src> <dst> [eq <port>]]
|
||
|
||
[nat source <internal_ip/len> interface <ifname> overload] # SNAT
|
||
[nat destination <external_ip> <port> <internal_ip> <port>] # DNAT
|
||
|
||
[qos class <0-7> <priority|bandwidth> <value>]
|
||
|
||
#ifdef PRODUCT_CLOUD_EDGE
|
||
[container-network]
|
||
[cidr <ip/len>]
|
||
[show containers]
|
||
|
||
[load-balancer]
|
||
[backend add <name> <ip:port>]
|
||
[frontend add <name> <ip:port> backend <name>]
|
||
[algorithm <rr|leastconn|hash>]
|
||
|
||
[vrrp <vrid>]
|
||
[ip <virtual_ip>]
|
||
[priority <1-254>]
|
||
[preempt]
|
||
#endif
|
||
```
|
||
|
||
### 8.2 REST API 路由表
|
||
|
||
```
|
||
Method Path Controller
|
||
────── ──── ──────────
|
||
|
||
# 路由管理
|
||
GET /api/v1/routes list_routes
|
||
POST /api/v1/routes add_route
|
||
DELETE /api/v1/routes/{prefix} del_route
|
||
|
||
# 接口管理
|
||
GET /api/v1/interfaces list_interfaces
|
||
GET /api/v1/interfaces/{name} get_interface
|
||
PUT /api/v1/interfaces/{name} update_interface
|
||
POST /api/v1/interfaces/{name}/reload reload_interface
|
||
|
||
# ACL
|
||
GET /api/v1/firewall/rules list_acl_rules
|
||
POST /api/v1/firewall/rules add_acl_rule
|
||
DELETE /api/v1/firewall/rules/{id} del_acl_rule
|
||
|
||
# NAT
|
||
GET /api/v1/nat/rules list_nat_rules
|
||
POST /api/v1/nat/rules add_nat_rule
|
||
DELETE /api/v1/nat/rules/{id} del_nat_rule
|
||
GET /api/v1/nat/connections list_connections
|
||
GET /api/v1/nat/connections/summary connection_summary
|
||
|
||
# QoS
|
||
GET /api/v1/qos/classes list_qos_classes
|
||
PUT /api/v1/qos/classes/{id} update_qos_class
|
||
|
||
# 统计
|
||
GET /api/v1/stats/ports port_stats
|
||
GET /api/v1/stats/ports/{name} port_stats_detail
|
||
GET /api/v1/stats/system system_stats
|
||
|
||
# 系统
|
||
GET /api/v1/system/health health_check
|
||
POST /api/v1/system/config/export export_config
|
||
POST /api/v1/system/config/import import_config
|
||
|
||
#ifdef PRODUCT_CLOUD_EDGE
|
||
# 容器网络
|
||
GET /api/v1/containers list_containers
|
||
GET /api/v1/containers/{id} get_container
|
||
|
||
# 负载均衡
|
||
GET /api/v1/loadbalancer/backends list_lb_backends
|
||
POST /api/v1/loadbalancer/backends add_lb_backend
|
||
DELETE /api/v1/loadbalancer/backends/{name} del_lb_backend
|
||
#endif
|
||
```
|
||
|
||
---
|
||
|
||
## 九、高可用详细设计
|
||
|
||
### 9.1 VRRP 状态机
|
||
|
||
```c
|
||
// src/ha/vrrp.h
|
||
|
||
#define VRRP_VERSION 3 // RFC 5798
|
||
#define VRRP_ADVERT_INT 1000 // Advertisement interval (ms)
|
||
#define VRRP_MASTER_DOWN 3000 // Master down detection (3 × advert)
|
||
|
||
enum vrrp_state {
|
||
VRRP_STATE_INITIALIZE = 0,
|
||
VRRP_STATE_BACKUP,
|
||
VRRP_STATE_MASTER,
|
||
};
|
||
|
||
struct vrrp_instance {
|
||
uint8_t vrid; // Virtual Router ID
|
||
uint8_t priority; // 本节点优先级 (1-254)
|
||
uint8_t adv_interval; // Advertisement 间隔 (centiseconds)
|
||
uint8_t preempt; // 抢占模式
|
||
uint32_t virtual_ip;
|
||
uint8_t virtual_mac[6]; // 00:00:5E:00:01:{VRID}
|
||
enum vrrp_state state;
|
||
uint64_t master_down_deadline; // 超时时间 (rdtsc)
|
||
struct rte_ring *conn_sync_queue; // 连接表同步队列
|
||
};
|
||
|
||
// 状态转换:
|
||
// INIT → BACKUP (priority != 255)
|
||
// INIT → MASTER (priority == 255, IP owner)
|
||
// BACKUP → MASTER (master_down timer expired)
|
||
// MASTER → BACKUP (received higher priority advertisement)
|
||
```
|
||
|
||
### 9.2 连接表同步协议
|
||
|
||
```c
|
||
// src/ha/sync.h
|
||
|
||
struct conn_sync_msg {
|
||
uint8_t type; // SYNC_ADD / SYNC_DEL / SYNC_FULL
|
||
uint8_t version; // 协议版本
|
||
uint16_t count; // 批量条目数
|
||
uint32_t seq_num; // 序列号 (用于去重)
|
||
struct conntrack_key keys[32]; // 批量同步 (最多 32 条)
|
||
};
|
||
```
|
||
|
||
---
|
||
|
||
## 十、构建系统设计
|
||
|
||
### 10.1 项目结构
|
||
|
||
```
|
||
unos/
|
||
├── meson.build # 顶层构建
|
||
├── meson_options.txt # 产品选项
|
||
├── src/
|
||
│ ├── meson.build # 源码构建
|
||
│ ├── core/
|
||
│ │ ├── meson.build
|
||
│ │ ├── main.c
|
||
│ │ ├── pipeline.c / pipeline.h
|
||
│ │ ├── ipc.c / ipc.h
|
||
│ │ └── timer.c / timer.h
|
||
│ ├── dp/
|
||
│ │ ├── meson.build
|
||
│ │ ├── forward.c
|
||
│ │ ├── parser.c / parser.h
|
||
│ │ ├── l2fwd.c / l2fwd.h
|
||
│ │ ├── l3fwd.c / l3fwd.h
|
||
│ │ ├── nat.c / nat.h
|
||
│ │ ├── acl.c / acl.h
|
||
│ │ └── qos.c / qos.h
|
||
│ ├── cp/
|
||
│ │ ├── meson.build
|
||
│ │ ├── control.c / control.h
|
||
│ │ ├── arp.c / arp.h
|
||
│ │ ├── dhcp.c / dhcp.h
|
||
│ │ ├── iface.c / iface.h
|
||
│ │ ├── frr_sync.c / frr_sync.h
|
||
│ │ └── db.c / db.h
|
||
│ ├── stcp/
|
||
│ │ ├── meson.build # 仅 Cloud Edge
|
||
│ │ ├── stcp.h
|
||
│ │ ├── tcb.h
|
||
│ │ ├── tcp_state.c
|
||
│ │ ├── tcp_cong.c
|
||
│ │ ├── tcp_retx.c
|
||
│ │ ├── tcp_flow.c
|
||
│ │ ├── tcp_socket.c
|
||
│ │ ├── tcp_txrx.c
|
||
│ │ ├── tcp_option.c
|
||
│ │ ├── ipv4.c
|
||
│ │ ├── icmp.c
|
||
│ │ └── frag.c
|
||
│ ├── cni/
|
||
│ │ ├── meson.build # 仅 Cloud Edge
|
||
│ │ ├── memif.c / memif.h
|
||
│ │ ├── cni_api.c / cni_api.h
|
||
│ │ └── ipam.c / ipam.h
|
||
│ ├── app/
|
||
│ │ ├── meson.build # 仅 Cloud Edge
|
||
│ │ ├── http_server.c
|
||
│ │ ├── dns_server.c
|
||
│ │ └── tcp_proxy.c
|
||
│ ├── mgmt/
|
||
│ │ ├── meson.build
|
||
│ │ ├── cli.c / cli.h
|
||
│ │ └── rest.c / rest.h
|
||
│ ├── ha/
|
||
│ │ ├── meson.build # 仅 Cloud Edge
|
||
│ │ ├── vrrp.c / vrrp.h
|
||
│ │ └── sync.c / sync.h
|
||
│ ├── telemetry/
|
||
│ │ ├── meson.build
|
||
│ │ ├── prometheus.c / prometheus.h
|
||
│ │ └── stats.c / stats.h
|
||
│ └── common/
|
||
│ ├── meson.build
|
||
│ ├── types.h
|
||
│ ├── utils.c / utils.h
|
||
│ └── log.c / log.h
|
||
├── web/
|
||
│ ├── index.html
|
||
│ └── dashboard.js
|
||
├── scripts/
|
||
│ ├── setup.sh
|
||
│ ├── bench.sh
|
||
│ └── profile.sh
|
||
├── config/
|
||
│ ├── cloud_edge.yaml
|
||
│ └── home_cpe.yaml
|
||
├── test/
|
||
│ ├── meson.build
|
||
│ ├── test_parser.c
|
||
│ ├── test_lpm.c
|
||
│ ├── test_nat.c
|
||
│ └── test_tcp_state.c
|
||
├── docs/
|
||
│ ├── ARCHITECTURE.md
|
||
│ └── REQUIREMENTS.md
|
||
├── .gitignore
|
||
├── LICENSE
|
||
└── README.md
|
||
```
|
||
|
||
### 10.2 顶层 meson.build
|
||
|
||
```meson
|
||
# unos/meson.build
|
||
project('unos', 'c',
|
||
version: '0.1.0',
|
||
default_options: [
|
||
'c_std=c17',
|
||
'warning_level=3',
|
||
'buildtype=release',
|
||
'default_library=static',
|
||
],
|
||
)
|
||
|
||
add_project_arguments('-march=native', '-mtune=native', language: 'c')
|
||
|
||
# DPDK dependency
|
||
dpdk = dependency('libdpdk', required: true, static: true)
|
||
|
||
# Other dependencies
|
||
threads = dependency('threads')
|
||
numa = dependency('numa', required: true)
|
||
hiredis = dependency('hiredis', required: true)
|
||
|
||
subdir('src')
|
||
```
|
||
|
||
### 10.3 src/meson.build (产品条件编译)
|
||
|
||
```meson
|
||
# unos/src/meson.build
|
||
|
||
product = get_option('product')
|
||
|
||
# === Common sources (both products) ===
|
||
common_sources = files(
|
||
'core/main.c',
|
||
'core/pipeline.c',
|
||
'core/ipc.c',
|
||
'core/timer.c',
|
||
'dp/forward.c',
|
||
'dp/parser.c',
|
||
'dp/l2fwd.c',
|
||
'dp/l3fwd.c',
|
||
'dp/nat.c',
|
||
'dp/acl.c',
|
||
'dp/qos.c',
|
||
'cp/control.c',
|
||
'cp/arp.c',
|
||
'cp/iface.c',
|
||
'cp/frr_sync.c',
|
||
'cp/db.c',
|
||
'mgmt/cli.c',
|
||
'telemetry/prometheus.c',
|
||
'telemetry/stats.c',
|
||
'common/utils.c',
|
||
'common/log.c',
|
||
)
|
||
|
||
# === Product-specific sources ===
|
||
cloud_edge_sources = files(
|
||
'stcp/ipv4.c',
|
||
'stcp/icmp.c',
|
||
'stcp/frag.c',
|
||
'stcp/tcp_state.c',
|
||
'stcp/tcp_cong.c',
|
||
'stcp/tcp_retx.c',
|
||
'stcp/tcp_flow.c',
|
||
'stcp/tcp_socket.c',
|
||
'stcp/tcp_txrx.c',
|
||
'stcp/tcp_option.c',
|
||
'cni/memif.c',
|
||
'cni/cni_api.c',
|
||
'cni/ipam.c',
|
||
'app/http_server.c',
|
||
'app/dns_server.c',
|
||
'app/tcp_proxy.c',
|
||
'mgmt/rest.c',
|
||
'ha/vrrp.c',
|
||
'ha/sync.c',
|
||
)
|
||
|
||
home_cpe_sources = files(
|
||
'cp/dhcp.c',
|
||
)
|
||
|
||
# === Assemble ===
|
||
sources = common_sources
|
||
product_args = []
|
||
|
||
if product == 'cloud_edge'
|
||
sources += cloud_edge_sources
|
||
product_args += ['-DPRODUCT_CLOUD_EDGE']
|
||
c_args = product_args
|
||
elif product == 'home_cpe'
|
||
sources += home_cpe_sources
|
||
product_args += ['-DPRODUCT_HOME_CPE']
|
||
c_args = product_args
|
||
endif
|
||
|
||
executable('unos',
|
||
sources,
|
||
c_args: c_args,
|
||
dependencies: [dpdk, threads, numa, hiredis],
|
||
install: true,
|
||
)
|
||
|
||
# === CNI plugin binary (Cloud Edge only) ===
|
||
if product == 'cloud_edge'
|
||
executable('unos-cni',
|
||
files('cni/cni_api.c'),
|
||
dependencies: [],
|
||
install: true,
|
||
install_dir: '/opt/cni/bin',
|
||
)
|
||
endif
|
||
```
|
||
|
||
---
|
||
|
||
## 十一、测试设计
|
||
|
||
### 11.1 单元测试
|
||
|
||
```c
|
||
// test/meson.build — 每个模块对应一个测试
|
||
|
||
parser_test = executable('test_parser',
|
||
'test_parser.c', '../src/dp/parser.c', '../src/common/utils.c',
|
||
dependencies: [dpdk],
|
||
)
|
||
test('Parser Test', parser_test)
|
||
|
||
lpm_test = executable('test_lpm',
|
||
'test_lpm.c', '../src/dp/l3fwd.c',
|
||
dependencies: [dpdk],
|
||
)
|
||
test('LPM Test', lpm_test)
|
||
|
||
nat_test = executable('test_nat',
|
||
'test_nat.c', '../src/dp/nat.c', '../src/dp/parser.c',
|
||
dependencies: [dpdk],
|
||
)
|
||
test('NAT Test', nat_test)
|
||
|
||
tcp_state_test = executable('test_tcp_state',
|
||
'test_tcp_state.c', '../src/stcp/tcp_state.c', '../src/stcp/tcb.h',
|
||
dependencies: [dpdk],
|
||
)
|
||
test('TCP State Machine Test', tcp_state_test)
|
||
```
|
||
|
||
### 11.2 集成测试脚本
|
||
|
||
```bash
|
||
# scripts/bench.sh — 自动化压测
|
||
|
||
#!/bin/bash
|
||
echo "=== uNOS Performance Benchmark Suite ==="
|
||
|
||
# 1. L2 Forwarding (64B)
|
||
echo "[1/6] L2 Forwarding 64B line rate..."
|
||
dperf -c 0-1 -n 2 -- --forward-mode mac --pkt-size 64 --duration 30
|
||
# Expected: 1.488 Mpps (1G) / 14.88 Mpps (10G), 0% loss
|
||
|
||
# 2. L3 Forwarding (64B)
|
||
echo "[2/6] L3 Forwarding 64B..."
|
||
dperf -c 0-1 -n 2 -- --forward-mode lpm --pkt-size 64 --duration 30
|
||
|
||
# 3. NAT (64B)
|
||
echo "[3/6] NAT Forwarding 64B..."
|
||
# Start uNOS with NAT enabled, then:
|
||
dperf -c 0-1 -n 2 -- --pkt-size 64 --duration 30
|
||
|
||
# 4. QoS (mixed traffic)
|
||
echo "[4/6] QoS mixed traffic test..."
|
||
# Background: iperf3 -c target -t 30 -b 900M (big packets)
|
||
# Foreground: ping -i 0.001 target -c 10000 (small packets)
|
||
# Expected: ping p99 < 2ms even with background load
|
||
|
||
# 5. TCP Stack throughput
|
||
echo "[5/6] TCP Stack throughput..."
|
||
iperf3 -c localhost -p 8080 -t 30
|
||
# Expected: >= 800 Mbps (J3455) / >= 9.4 Gbps (T140)
|
||
|
||
# 6. HTTP Server QPS
|
||
echo "[6/6] HTTP Server QPS..."
|
||
wrk -t 2 -c 100 -d 30s http://localhost:8080/
|
||
# Expected: >= 50K (J3455) / >= 200K (T140)
|
||
```
|
||
|
||
---
|
||
|
||
## 十二、接口契约
|
||
|
||
### 12.1 内部模块 API 契约
|
||
|
||
| 生产者 → 消费者 | 接口 | 契约 |
|
||
|---------------|------|------|
|
||
| Pipeline | Stage | `typedef uint16_t (*process_fn)(mbuf**, uint16_t)` — 输入/输出都是 mbuf 数组，返回处理后的包数 |
|
||
| Parser | Rest | `parser_result_t` 写入 mbuf 私有区域，一次解析、多次使用 |
|
||
| L3Fwd | ARP | `arp_lookup(ip) → mac` — 非阻塞，miss 返回 NULL |
|
||
| NAT | Conntrack | `conntrack_lookup(key) → entry` — O(1) 平均，miss 返回创建的新 entry |
|
||
| Control | Dataplane | `ipc_send(msg)` — 非阻塞 ring enqueue |
|
||
| Dataplane | Control | `ipc_send(msg)` — 非阻塞 ring enqueue |
|
||
|
||
### 12.2 外部接口契约
|
||
|
||
| 接口 | 协议 | 格式 |
|
||
|------|------|------|
|
||
| CNI Plugin ↔ uNOS | Unix socket (SOCK_STREAM) | JSON |
|
||
| CLI ↔ uNOS | Unix socket | Text (vtysh) |
|
||
| REST ↔ uNOS | HTTP/1.1 / HTTP/2 | JSON |
|
||
| FRR ↔ uNOS | Netlink (AF_NETLINK) | Binary (FPM) |
|
||
| Redis ↔ uNOS | TCP (localhost:6379) | Redis Protocol (RESP) |
|
||
| Prometheus ↔ uNOS | HTTP | Text (Prometheus exposition format) |
|
||
|
||
---
|
||
|
||
> **文档结束**
|
||
>
|
||
> 版本: v1.0 | 日期: 2026-07-29
|
||
> 随项目推进持续更新设计细节
|