newbieandClaude
6d8c994dde
- VitePress static site for rendering all markdown documents - Auto-discovers .md files from dpdk-roadmap-v3 and dpdk-study-kb - Features: full-text search, dark mode, responsive design - Pages: roadmap, notes (Phase 0-3), product docs, knowledge base Co-Authored-By: Claude <noreply@anthropic.com>
252 lines
14 KiB
Markdown
Executable File
252 lines
14 KiB
Markdown
Executable File
# Day 001 (2026-07-21, 周一) · 主题: PCIe总线拓扑
|
||
|
||
## 📖 学习 (30min)
|
||
|
||
### PCIe体系结构
|
||
- PCIe是点对点串行总线，替代了旧的PCI/PCI-X并行总线
|
||
- 分层结构：Root Complex → PCI Bridge → Endpoint
|
||
- Root Complex集成在CPU中，是CPU和PCIe的桥梁
|
||
- 每个PCIe设备有唯一的BDF地址：Bus:Device.Function
|
||
|
||
### 关键概念
|
||
1. **Root Complex (RC)**: 连接CPU和PCIe总线，负责：
|
||
- CPU地址 ↔ PCIe地址转换
|
||
- 生成PCIe配置请求
|
||
- 管理中断
|
||
|
||
2. **PCI Bridge**: 扩展PCIe端口，像网络交换机
|
||
- 上游口(Upstream)连RC或其他Bridge
|
||
- 下游口(Downstream)连设备或其他Bridge
|
||
|
||
3. **Endpoint**: 最终设备
|
||
- 网卡(NIC)、显卡(GPU)、SSD(NVMe)
|
||
- 通过BAR(Base Address Register)暴露寄存器
|
||
|
||
4. **BDF地址**: Bus:Device.Function
|
||
- Bus: 8位，最多256条总线
|
||
- Device: 5位，每条总线最多32个设备
|
||
- Function: 3位，每个设备最多8个功能
|
||
|
||
## 🔧 动手 (60min)
|
||
|
||
### 1. 获取本机PCIe拓扑
|
||
|
||
```bash
|
||
root@ubuntu:~# lspci -tv
|
||
-[0000:00]-+-00.0 Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge
|
||
+-01.0-[01]--
|
||
+-07.0 Intel Corporation 82371AB/EB/MB PIIX4 ISA
|
||
+-07.1 Intel Corporation 82371AB/EB/MB PIIX4 IDE
|
||
+-07.3 Intel Corporation 82371AB/EB/MB PIIX4 ACPI
|
||
+-07.7 VMware Virtual Machine Communication Interface
|
||
+-0f.0 VMware SVGA II Adapter
|
||
+-10.0 Broadcom / LSI 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI
|
||
+-11.0-[02]--+-00.0 VMware USB1.1 UHCI Controller
|
||
| +-01.0 Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
|
||
| +-02.0 VMware USB2 EHCI Controller
|
||
| +-03.0 VMware SATA AHCI controller
|
||
| \-04.0 Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
|
||
+-15.0-[03]----00.0 VMware VMXNET3 Ethernet Controller
|
||
+-15.1-[04]--
|
||
...
|
||
```
|
||
|
||
**拓扑分析**:
|
||
- 这是VMware虚拟机环境
|
||
- 主桥：Intel 440BX/ZX/DX (很老的芯片组，1998年)
|
||
- 总线00: Root Complex + 传统PCI设备
|
||
- 总线01: 空桥（可能是PCIe扩展槽）
|
||
- 总线02: PCI Bridge 00:11.0 下游，连接4个设备
|
||
- 总线03: PCI Bridge 00:15.0 下游，连接VMXNET3网卡
|
||
|
||
### 2. 读取网卡详细信息 (02:01.0)
|
||
|
||
```bash
|
||
root@ubuntu:~# lspci -vvv -s 02:01.0
|
||
02:01.0 Ethernet controller: Intel Corporation 82545EM Gigabit Ethernet Controller (Copper) (rev 01)
|
||
DeviceName: Ethernet2
|
||
Subsystem: VMware PRO/1000 MT Single Port Adapter
|
||
Physical Slot: 33
|
||
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
|
||
Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
|
||
Latency: 0 (63750ns min), Cache Line Size: 64 bytes
|
||
Interrupt: pin A routed to IRQ 19
|
||
Region 0: Memory at fd5c0000 (64-bit, non-prefetchable) [size=128K]
|
||
Region 2: Memory at fdff0000 (64-bit, non-prefetchable) [size=64K]
|
||
Region 4: I/O ports at 2000 [size=64]
|
||
Expansion ROM at fd500000 [virtual] [disabled] [size=64K]
|
||
Capabilities: [dc] Power Management version 2
|
||
Capabilities: [e4] PCI-X non-bridge device
|
||
Kernel driver in use: e1000
|
||
Kernel modules: e1000
|
||
```
|
||
|
||
**关键信息解码**:
|
||
- **Vendor ID**: 0x8086 (Intel)
|
||
- **Device ID**: 0x100F (82545EM)
|
||
- **Class**: 0x020000 (Network Controller)
|
||
- **BAR0**: Memory at fd5c0000, 128KB (MMIO寄存器)
|
||
- **BAR2**: Memory at fdff0000, 64KB (可能是Flash或额外寄存器)
|
||
- **BAR4**: I/O ports at 2000, 64B (传统I/O端口)
|
||
- **IRQ**: 19 (传统INTx中断)
|
||
- **Driver**: e1000 (内核驱动)
|
||
|
||
### 3. 手动解码Config Space前16字节
|
||
|
||
Config Space前64字节结构：
|
||
```
|
||
Offset 0x00: Vendor ID (2B) + Device ID (2B)
|
||
Offset 0x04: Command (2B) + Status (2B)
|
||
Offset 0x08: Revision ID (1B) + Class Code (3B)
|
||
Offset 0x0C: Cache Line Size (1B) + Latency Timer (1B) + Header Type (1B) + BIST (1B)
|
||
Offset 0x10: BAR0 (4B/8B)
|
||
Offset 0x14: BAR1 (4B)
|
||
Offset 0x18: BAR2 (4B/8B)
|
||
Offset 0x1C: BAR3 (4B)
|
||
Offset 0x20: BAR4 (4B)
|
||
Offset 0x24: BAR5 (4B)
|
||
Offset 0x28: CardBus CIS Pointer (4B)
|
||
Offset 0x2C: Subsystem Vendor ID (2B) + Subsystem ID (2B)
|
||
Offset 0x30: Expansion ROM Base Address (4B)
|
||
Offset 0x34: Reserved (2B) + Capabilities Pointer (1B) + Reserved (1B)
|
||
Offset 0x38: Reserved (4B)
|
||
Offset 0x3C: Max Latency (1B) + Min Gnt (1B) + Interrupt Pin (1B) + Interrupt Line (1B)
|
||
```
|
||
|
||
**从lspci -vvv输出解码**:
|
||
- Vendor ID: 0x8086 (Intel Corporation)
|
||
- Device ID: 0x100F (82545EM Gigabit Ethernet)
|
||
- Revision ID: 0x01
|
||
- Class Code: 0x020000 (02=Network, 00=Ethernet, 00=generic)
|
||
- Cache Line Size: 64 bytes
|
||
- Latency Timer: 0
|
||
- Header Type: 0x00 (普通Endpoint)
|
||
- Interrupt Pin: INTA# (pin A)
|
||
- Interrupt Line: IRQ 19
|
||
|
||
## 📤 产出
|
||
|
||
- 笔记: `notes/2026-07-week1/Day001-2026-07-21-pcie-topology.md`
|
||
- 拓扑图: 见下方ASCII图
|
||
|
||
## ✅ 掌握检验
|
||
|
||
1. **它是什么？**
|
||
PCIe是CPU连接外设的高速串行总线，采用分层结构(Root Complex → Bridge → Endpoint)。本机是VMware虚拟机，模拟Intel 440BX芯片组。
|
||
|
||
2. **为什么需要它？**
|
||
- 替代PCI/PCI-X，提供更高带宽(PCIe 4.0 x16可达64GB/s)
|
||
- 点对点连接，避免总线竞争
|
||
- 支持热插拔和电源管理
|
||
- 本机虽然是传统PCI(66MHz)，但VMware模拟了PCIe桥接
|
||
|
||
3. **DPDK vs 内核对比？**
|
||
- **内核**: 通过PCIe枚举发现设备，e1000驱动运行在内核态，应用通过系统调用(socket)访问网络
|
||
- **DPDK**: 同样枚举PCIe设备，但用VFIO/UIO把设备映射到用户态，应用直接操作BAR0(fd5c0000)的MMIO寄存器，绕过内核协议栈
|
||
- **关键差异**: DPDK避免了系统调用和内存拷贝，延迟从微秒级降到纳秒级。本机的82545EM是DPDK e1000 PMD支持的网卡。
|
||
|
||
## 🐧 内核对照
|
||
|
||
- **内核PCIe枚举**: `drivers/pci/probe.c` 的 `pci_scan_bus()`
|
||
- 读取ACPI MCFG表获取PCIe配置空间基地址
|
||
- 遍历所有Bus/Device/Function，读取Vendor ID
|
||
- 发现有效设备后，分配 `struct pci_dev`
|
||
|
||
- **DPDK对应**: `drivers/bus/pci/linux/pci.c` 的 `rte_pci_scan()`
|
||
- 同样读取 `/sys/bus/pci/devices/` 或 MCFG
|
||
- 但把结果存在用户态数组，不调用内核驱动
|
||
- 最终通过 `rte_pci_probe()` 绑定到PMD
|
||
|
||
- **两者差异**:
|
||
- 内核：枚举 → 绑定内核驱动(e1000) → 创建net_device(eth0) → 注册网络协议栈
|
||
- DPDK：枚举 → 绑定VFIO/UIO → 用户态直接访问BAR0(fd5c0000) → 绕过内核
|
||
|
||
## 📊 PCIe拓扑图 (ASCII)
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ VMware Virtual Machine │
|
||
│ │
|
||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||
│ │ Intel 440BX/ZX/DX Host Bridge │ │
|
||
│ │ (00:00.0) │ │
|
||
│ │ Root Complex │ │
|
||
│ └──────────────────────────┬──────────────────────────────┘ │
|
||
│ │ │
|
||
│ ┌────────────────────┼────────────────────┐ │
|
||
│ │ │ │ │
|
||
│ ┌────┴────┐ ┌────┴────┐ ┌────┴────┐ │
|
||
│ │ 00:07.0 │ │ 00:11.0 │ │ 00:15.0 │ │
|
||
│ │ PIIX4 │ │ PCI │ │ PCI │ │
|
||
│ │ ISA │ │ Bridge │ │ Bridge │ │
|
||
│ └────┬────┘ └────┬────┘ └────┬────┘ │
|
||
│ │ │ │ │
|
||
│ ┌────┴────┐ ┌────┴────┐ ┌────┴────┐ │
|
||
│ │ 00:07.1 │ │ Bus 02 │ │ Bus 03 │ │
|
||
│ │ PIIX4 │ │ │ │ │ │
|
||
│ │ IDE │ │ ┌───────┴───┐ │ ┌───────┴───┐ │
|
||
│ └─────────┘ │ │ 02:00.0 │ │ │ 03:00.0 │ │
|
||
│ │ │ USB1.1 │ │ │ VMXNET3 │ │
|
||
│ ┌─────────┐ │ └───────────┘ │ │ Ethernet │ │
|
||
│ │ 00:07.3 │ │ ┌───────────┐ │ └───────────┘ │
|
||
│ │ PIIX4 │ │ │ 02:01.0 │ │ │
|
||
│ │ ACPI │ │ │ 82545EM │◄───┤ DPDK目标网卡 │
|
||
│ └─────────┘ │ │ e1000 │ │ (10Gb虚拟) │
|
||
│ │ │ BAR0: │ │ │
|
||
│ ┌─────────┐ │ │ fd5c0000 │ │ │
|
||
│ │ 00:07.7 │ │ └───────────┘ │ │
|
||
│ │ VMware │ │ ┌───────────┐ │ │
|
||
│ │ VMCI │ │ │ 02:02.0 │ │ │
|
||
│ └─────────┘ │ │ USB2.0 │ │ │
|
||
│ │ └───────────┘ │ │
|
||
│ ┌─────────┐ │ ┌───────────┐ │ │
|
||
│ │ 00:0f.0 │ │ │ 02:03.0 │ │ │
|
||
│ │ SVGA II │ │ │ SATA │ │ │
|
||
│ └─────────┘ │ └───────────┘ │ │
|
||
│ │ ┌───────────┐ │ │
|
||
│ ┌─────────┐ │ │ 02:04.0 │ │ │
|
||
│ │ 00:10.0 │ │ │ 82545EM │◄───┤ 第二块e1000 │
|
||
│ │ LSI │ │ │ e1000 │ │ │
|
||
│ │ SCSI │ │ └───────────┘ │ │
|
||
│ └─────────┘ └──────────────────┘ │
|
||
│ │
|
||
└──────────────────────────────────────────────────────────────┘
|
||
|
||
数据流路径:
|
||
┌─────┐ ┌─────────────┐ ┌─────────┐ ┌─────────────┐ ┌─────┐
|
||
│ CPU │───►│ Root Complex│───►│ PCI │───►│ 82545EM │───►│网线 │
|
||
│ │ │ (440BX) │ │ Bridge │ │ (02:01.0) │ │ │
|
||
└─────┘ └─────────────┘ │ 00:11.0 │ │ BAR0: │ └─────┘
|
||
└─────────┘ │ fd5c0000 │
|
||
│ 128KB MMIO │
|
||
└─────────────┘
|
||
|
||
DPDK数据面路径 (绕过内核):
|
||
┌─────────┐ ┌─────────────┐ ┌─────────────────────────────┐
|
||
│ DPDK App│───►│ VFIO/UIO │───►│ 直接读写BAR0 (fd5c0000) │
|
||
│ (用户态)│ │ (设备绑定) │ │ 控制网卡收发 │
|
||
└─────────┘ └─────────────┘ └─────────────────────────────┘
|
||
▲ │
|
||
│ ┌────────────────────────────────────┘
|
||
│ │ DMA (直接内存访问)
|
||
│ ┌────┴────┐
|
||
└───│ Hugepage │
|
||
│ 内存 │
|
||
└─────────┘
|
||
```
|
||
|
||
## 🔍 关键发现
|
||
|
||
1. **这是VMware虚拟机**: 440BX是1998年的芯片组，但VMware用PCIe桥接模拟了现代PCIe结构
|
||
2. **两块e1000网卡**: 02:01.0和02:04.0，都是Intel 82545EM，DPDK完美支持
|
||
3. **一块VMXNET3**: 03:00.0，VMware准虚拟化网卡，性能更好，DPDK也支持
|
||
4. **BAR0地址**: fd5c0000，这是物理地址，DPDK通过VFIO映射到用户态虚拟地址
|
||
5. **Cache Line Size**: 64字节，DPDK的mbuf(2048B)是32个cache line，对齐优化很重要
|
||
|
||
## 📚 参考资料
|
||
|
||
- 《PCIe体系结构导读》Ch.1-2
|
||
- 内核: `drivers/pci/probe.c`, `include/linux/pci.h`
|
||
- DPDK: `drivers/bus/pci/linux/pci.c`, `drivers/net/e1000/`
|
||
- Intel 82545EM datasheet: https://www.intel.com/content/www/us/en/products/sku/76774/intel-82545em-gigabit-ethernet-controller/specifications.html
|