PCI Express Basics
Richard Solomon LSI Corporation
Copyright 2010, PCI-SIG, All Rights Reserved
Acknowledgements
I would like to acknowledge the contributions of Ravi Budruk, Mindshare, Inc.
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
PCI Express Introduction
PCI Express architecture is a high performance, IO interconnect for peripherals in computing/communication platforms Evolved from PCITM and PCI-XTM architectures
Yet PCI Express architecture is significantly different from its predecessors PCI and PCI-X
PCI Express is a serial point-to-point interconnect between two devices Implements packet based protocol for information transfer Scalable performance based on number of signal Lanes implemented on the PCI Express interconnect
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
PCI Express Terminology
PCI Express Device A
Signal
Link Lane
Wire
PCI Express Device B
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
PCI Express Throughput
Link Width x1 PCIe 1.x BW (GB/s) PCIe 2.x BW (GB/s) PCIe 3.0 BW (GB/s) 0.5 1 2 x2 1 2 4 x4 2 4 8 x8 4 8 16 x12 6 12 24 x16 8 16 32 x32 16 32 64
Derivation of these numbers: 2.5 GT/s (PCIe 1.x), 5.0 GT/s (PCIe 2.x), or 8GT/s (PCIe 3.0) signaling in each direction 20% overhead due to 8b/10b encoding in 1.x and 2.x Aggregate bandwidth, implying traffic in both directions
PCIe Technical Seminar Copyright 2010, PCI-SIG, All Rights Reserved 5
PCI Express Features
Point-to-point connection Serial bus means fewer pins Scaleable: x1, x2, x4, x8, x12, x16, x32 Dual Simplex connection 2.5, 5.0 and 8.0 GT/s transfer/direction/s Packet based transaction protocol
PCIe Device A
PCIe Technical Seminar
Packet
Link (x1, x2, x4, x8, x12, x16 or x32)
Packet
Copyright 2010, PCI-SIG, All Rights Reserved
PCIe Device B
6
Differential Signaling
Electrical characteristics of PCI Express signal
Differential signaling
Transmitter Differential Peak voltage = 0.4 - 0.6 V Transmitter Common mode voltage = 0 - 3.6 V
DVcm D+ V Diffp
Two devices at opposite ends of a Link may support different DC common mode voltages
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
Additional Features
Switches used to interconnect multiple devices Packet based protocol Bandwidth and clocking Same memory, IO and configuration address space as PCI
Similar transaction types as PCI with additional message transaction
PCI Express Transactions include:
memory read/write, memory read lock, IO read/write, configuration read/write, message requests
Split transaction model for non-posted
Posted: Memory writes and messages Non Posted: Memory reads, config read/write, IO read /weite Completions: Read completions, write completions
http://zone.ni.com/devzone/cda/tut/p/id/3767
PCIe Technical Seminar Copyright 2010, PCI-SIG, All Rights Reserved 8
Additional Features
Data Integrity and Error Handling
RAS capable (Reliable, Available, Serviceable) Data integrity at: 1) Link level, 2) end-to-end
Virtual channels (VCs) and traffic classes (TCs) to support differentiated traffic or Quality of Service (QoS)
The ability to define levels of performance for packets of different TCs 8 TCs and 8 VCs available
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
Additional Features
Flow Control
No retry as in PCI
MSI style interrupt handling
Also supports legacy PCI interrupt handling in-band
Advanced power management
Active State PM PCI compatible PM
Message Signaled Interrupt (PCIe 2.2) Allows 1,2,4,8,16,32 interrrupts Message Signaled Interrupt X (PCIe 3.0) Allows up to 2048 interrrupts http://en.wikipedia.org/wiki/Message_Signaled_Interrupts
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
10
Additional Features
Hot Plug and Hot Swap support
Native No sideband signals
PCI compatible software model
PCI configuration and enumeration software can be used to enumerate PCI Express hardware PCI Express system will boot existing OS PCI Express supports existing device drivers New additional configuration address space requires OS and driver update
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
11
PCI Express Topology
CPU
Root Complex
Bus 0 (Internal) PCIe 1 PCIe 6 PCIe 7
Memory
PCIe PCIe 3 Switch Endpoint
PCIe 4 PCIe 5
PCIe Endpoint
Switch
PCIe Bridge To
PCI/PCI-X
Virtual PCI Bridge
Virtual PCI Bridge
Bus 2
Virtual PCI Bridge
Virtual PCI Bridge
PCIe Endpoint
Legacy Endpoint
PCI/PCI-X Bus 8
Legend PCI Express Device Downstream Port PCI Express Device Upstream Port
PCIe Technical Seminar Copyright 2010, PCI-SIG, All Rights Reserved 12
PCI Express System
Processor FSB PCI Express GFX GFX
Root Complex
DDR SDRAM
Slots
HDD
Serial ATA USB 2.0 LPC
PCI Express PCI IO Controller Hub (ICH) IEEE 1394
Slot
S IO
COM1 COM2
GB Ethernet
Add-In Add-In Add-In PCI Express
PCI Express Link
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
13
Transaction Types, Address Spaces
Request are translated to one of four transaction types by the Transaction Layer:
1. Memory Read or Memory Write. Used to transfer data from or to a memory mapped location The protocol also supports a locked memory read transaction variant. I/O Read or I/O Write. Used to transfer data from or to an I/O location These transactions are restricted to supporting legacy endpoint devices. Configuration Read or Configuration Write. Used to discover device capabilities, program features, and check status in the 4KB PCI Express configuration space. Messages. Handled like posted writes. Used for event signaling and general purpose messaging.
2.
3.
4.
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
14
PCI Express TLP Types
Description
Memory Read Request Memory Read Request Locked Access Memory Write Request IO Read Request IO Write Request Configuration Read Request Type 0 and Type 1 Configuration Write Request Type 0 and Type 1 Message Request without Data Payload Message Request with Data Payload Completion without Data (used for IO, configuration write completions and read completion with error completion status) Completion with Data (used for memory, IO and configuration read completions)
Abbreviated Name
MRd MRdLk MWr IORd IOWr CfgRd0, CfgRd1 CfgWr0, CfgWr1 Msg MsgD Cpl CplD
Completion for Locked Memory Read without Data (used for error status)
Completion for Locked Memory Read with Data
CplLk
CplDLk
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
15
Three Methods For Packet Routing
Each request or completion header is tagged as to its type, and each of the packet types is routed based on one of three schemes:
Address Routing ID Routing Implicit Routing
Memory and IO requests use address routing. Completions and Configuration cycles use ID routing. Message requests have selectable routing based on a 3-bit code in the message routing sub-field of the header type field.
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
16
Programmed I/O Transaction
Processor MRd Requester: -Step 1: Root Complex (requester) initiates Memory Read Request (MRd) -Step 4: Root Complex receives CplD MRd Switch A MRd CplD Switch B MRd Endpoint Endpoint Endpoint CplD Completer: -Step 2: Endpoint (completer) receives MRd -Step 3: Endpoint returns Completion with data (CplD)
17
Processor FSB
Root Complex CplD Switch C
DDR SDRAM
Endpoint
Endpoint
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
DMA Transaction
Processor Processor
FSB
Completer: -Step 2: Root Complex (completer) receives MRd -Step 3: Root Complex returns Completion with data (CplD) CplD Switch A CplD MRd
Root Complex MRd Switch C
DDR SDRAM
Switch B
CplD Endpoint
Endpoint MRd
Endpoint
Endpoint
Endpoint
Requester: -Step 1: Endpoint (requester) initiates Memory Read Request (MRd) -Step 4: Endpoint receives CplD
18
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
Peer-to-Peer Transaction
Processor FSB Processor
Root Complex
DDR SDRAM
CplD
Switch A CplD MRd Switch B CplD
MRd
MRd
Switch C MRd Endpoint
CplD
CplD Endpoint Completer: -Step 2: Endpoint (completer) receives MRd -Step 3: Endpoint returns Completion with data (CplD)
Endpoint MRd
Endpoint
Endpoint
Requester: -Step 1: Endpoint (requester) initiates Memory Read Request (MRd) -Step 4: Endpoint receives CplD
Copyright 2010, PCI-SIG, All Rights Reserved
PCIe Technical Seminar
19
PCI Express Device Layers
PCI Express Device A Device Core PCI Express Core Logic Interface
TX RX TX
PCI Express Device B Device Core PCI Express Core Logic Interface
RX
Transaction Layer Data Link Layer Physical Layer
Link
Transaction Layer Data Link Layer Physical Layer
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
20
TLP Origin and Destination
PCI Express Device A Device Core PCI Express Core Logic Interface
TX
TLP Transmitted
PCI Express Device B Device Core PCI Express Core Logic Interface
TX RX
TLP Received
RX
Transaction Layer Data Link Layer Physical Layer
Link
Transaction Layer Data Link Layer Physical Layer
TLP: Transaction Layer Packet
PCIe Technical Seminar Copyright 2010, PCI-SIG, All Rights Reserved 21
TLP Structure
Information in core section of TLP comes from Software Layer / Device Core Bit transmit direction
Start Sequence Header
1B 2B 3-4 DW
Data Payload
0-1024 DW
ECRC LCRC End
1DW 1DW 1B
Created by Transaction Layer
Appended by Data Link Layer
Appended by Physical Layer
DW: Double Word (32 bits)
PCIe Technical Seminar Copyright 2010, PCI-SIG, All Rights Reserved 22
DLLP Origin and Destination
PCI Express Device A Device Core PCI Express Core Logic Interface
TX RX TX
PCI Express Device B Device Core PCI Express Core Logic Interface
RX
Transaction Layer
DLLP Transmitted
Transaction Layer Data Link Layer Physical Layer
Link DLLP Received
Data Link Layer Physical Layer
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
23
DLLP Structure
Bit transmit direction
Start
1B
DLLP
4B
CRC
2B
End
1B
Data Link Layer
Appended by Physical Layer
ACK / NAK Packets Flow Control Packets Power Management Packets Vendor Defined Packets
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
24
Ordered-Set Origin and Destination
PCI Express Device A Device Core PCI Express Core Logic Interface
TX RX TX
PCI Express Device B Device Core PCI Express Core Logic Interface
RX
Transaction Layer Data Link Layer
Ordered-Set Transmitted
Transaction Layer Data Link Layer Physical Layer
Link Ordered-Set
Physical Layer
Received
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
25
Ordered-Set Structure
COM Identifier Identifier Identifier
Training Sequence One (TS1)
16 character set: 1 COM, 15 TS1 data characters
Training Sequence Two (TS2)
16 character set: 1 COM, 15 TS2 data characters
SKIP
4 character set: 1 COM followed by 3 SKP identifiers
Fast Training Sequence (FTS)
4 characters: 1 COM followed by 3 FTS identifiers
Electrical Idle (IDLE)
4 characters: 1 COM followed by 3 IDL identifiers
Electrical Idle Exit (EIEOS) (new to 2.0 spec)
PCIe Technical Seminar
16 characters
Copyright 2010, PCI-SIG, All Rights Reserved
26
Quality of Service
Processor PCI Express GFX GFX Endpoint Processor
Root Complex
DDR SDRAM
InfiniBand Switch Out-of-Box
InfiniBand
Endpoint
Switch
10Gb Ethernet Endpoint
Switch
Fiber Channel
RAID Disk array
Add-In
Slot
Switch
10Gb Ethernet Endpoint
PCI Express to-PCI Endpoint
SCSI Endpoint PCI
PCI Express Link
PCIe Technical Seminar
Video Camera
SCSI
S IO
COM1 COM2
Endpoint
IEEE 1394
Slots
Copyright 2010, PCI-SIG, All Rights Reserved
27
PCI Express Flow Control
Credit-based flow control is point-to-point based, not end-to-end
Buffer space available TLP VC Buffer
Transmitter
Receiver
Flow Control DLLP (FCx) Receiver sends Flow Control Packets (FCP) which are a type of DLLP (Data Link Layer Packet) to provide the transmitter with credits so that it can transmit packets to the receiver
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
28
ACK/NAK Protocol Overview
Transmit Device A
From Transaction Layer Tx
Receiver Device B
To Transaction Layer Rx
Data Link Layer
TLP
Sequence TLP LCRC
Data Link Layer
DLLP
ACK / NAK
DLLP
ACK / NAK
TLP
Sequence De-mux TLP LCRC
Replay Buffer Mux
De-mux Mux Error Check
Tx
Rx
DLLP
ACK / NAK
Tx
Rx
Link
TLP Sequence PCIe Technical Seminar TLP LCRC 29
Copyright 2010, PCI-SIG, All Rights Reserved
ACK/NAK Protocol: Point-to-Point
1a. Request 4b. ACK Requester 1b. ACK 4a. Completion Switch 2b. ACK 3a. Completion 2a. Request 3b. ACK Completer
ACK returned for good reception of Request or Completion NAK returned for error reception of Request or Completion
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
30
Interrupt Model: Three Methods
PCI Express supports three interrupt reporting mechanisms:
1. Message Signaled Interrupts (MSI)
Legacy endpoints are required to support MSI (or MSI-X) with 32- or 64-bit MSI capability register implementation
Native PCI Express endpoints are required to support MSI with 64-bit MSI capability register implementation
2. 3.
Message Signaled Interrupts - X (MSI-X)
Legacy and native endpoints are required to support MSI-X (or MSI) and implement the associated MSI-X capability register Native and Legacy endpoints are required to support Legacy INTx Emulation PCI Express defines in-band messages which emulate the four physical interrupt signals (INTA-INTD) routed between PCI devices and the system interrupt controller Forwarding support required by switches
Copyright 2010, PCI-SIG, All Rights Reserved 31
INTx Emulation.
PCIe Technical Seminar
Native and Legacy Interrupts
PCIe -
PCIe
PCIe
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
32
PCI Express Configuration Space
PCIe Technical Seminar
Copyright 2010, PCI-SIG, All Rights Reserved
33
Thank you for attending the PCIe Technical Seminar
For more information please go to www.pcisig.com
PCIe Technical Seminar Copyright 2010, PCI-SIG, All Rights Reserved 34
