PCI EXPRESS

'PCI Express', officially abbreviated as 'PCI-E' or 'PCIe', is a computer expansion card interface format. It was designed to replace PCI, PCI-X and AGP (graphics card interface).
PCIe is based around serial links called lanes. The PCIe 1.1 specification supports x1 (pronounced "by one"), x2, x4, x8, x16, and x32 lanes. In each lane, the most common version PCIe 1.1 carries 250 MB/s in each direction. Every lane of the PCIe is a full duplex link; capable of simultaneous transmit and receive. The PCIe 1.1 bus runs at 2.5 GHz. An explicit clock is not used; instead the data stream is encoded using the 8b/10b scheme, which ensures that there are sufficient transitions within a single 10 bit character to properly and reliably recover the clock. The astute reader will notice that there are 3 times more combinations than characters used. Some of these additional characters are discarded due to an insufficient number of edges within the 10 bit packet to extract the clock. Others are used to encode error commands. Some may be used to provide "DC balancing" so that the wire doesn't acquire an electrical charge. The remainder are simply not used. Therefore, each lane transmits 250 MB/s. The most number of lanes supported is x32, so 250MB/s x 32 x 2 (bi-directionality) is 16GB/s for a theoretical maximum transfer rate.
PCIe slots come in a variety of sizes referred to by the maximum lane count they support. A larger card will not fit in a smaller slot but a smaller card can be used in a larger slot. The number of lanes actually connected may be smaller than the number supported by the slot size. While a 16 lane card cannot be used in an 8 lane slot it can be used in a 16 lane slot with only 8 lanes connected. The number of lanes are "negotiated" during power-up or explicitly during operation.
Putting this into perspective, a single lane has nearly twice the data rate of normal PCI, a four lane slot has a comparable data rate to the fastest version of PCI-X 1.0, and an eight lane slot has a data rate comparable to the fastest version of AGP.

Contents

Overview

History

Hardware protocol summary

Physical Layer

Data transmission

Signaling rate

Data Link Layer

Transaction Layer

Form factors

Competing protocols

Outlook

PCI Express 2.0

PCI Express 3.0

See also

References

External links

Overview

The PCIe physical layer consists of a network of serial interconnects. A hub on the mainboard acts as a crossbar switch allowing point-to-point device interconnections to be rerouted on the fly. This dynamic point-to-point connection behavior leads to parallelism since more than one pair of devices may communicate with each other at the same time. (In contrast, older PC interfaces had all devices permanently wired to the same bus; therefore, only one device could talk at a time.) This is similar to the difference between conversing over a telephone where you can only call one person at a time, and conversing in a meeting, where you can talk to a person beside you directly. The format also allows channel grouping, where multiple lanes are bonded to a single device pair in order to provide higher bandwidth.
The bonded serial format was chosen over a traditional parallel format due to the phenomenon of timing skew. Timing skew is a direct result of the limitations imposed by the speed of light: when an electrical signal travels down a wire, it does so at a finite speed. Because different traces in an interface have different lengths, parallel signals transmitted simultaneously from a source arrive at their destinations at different times. When the interconnection clock rate rises to the point where the wavelength of a single bit exceeds this difference in path length, the bits of a single word do not arrive at their destination simultaneously, making parallel recovery of the word difficult. Thus, the speed of light, combined with the difference in length between the longest and shortest trace in a parallel interconnect, leads to a naturally imposed maximum bandwidth. Serial channel bonding avoids this issue by not requiring the bits to arrive simultaneously. PCIe is just one example of a general trend away from parallel buses to serial interconnects. For other examples, see HyperTransport, Serial ATA, USB, SAS or FireWire. The multichannel serial design also increases flexibility by allowing slow devices to be allocated fewer lanes than fast devices.
PCIe is supported primarily by Intel, which started working on the standard as the ''Arapahoe'' project after pulling out of the InfiniBand system. PCIe is intended to be used as a local interconnect only. It was designed to be software compatible with the preexisting PCI standard, making the conversion of PCI cards and systems to PCI Express as simple as replacing the physical layer without requiring a change to the supporting software. The increased bandwidth on PCI Express has led to unification, as it is fast enough to replace almost all existing internal buses, including AGP and PCI. Intel envisions a single PCI Express controller talking to all external devices in the future, as opposed to the northbridge/southbridge solution used in current machines.
Unlike preceding PC expansion interface standards, PCIe is a point-to-point "bus". This type of connection removes the need for "arbitrating" the bus or waiting for the bus to free. This means that while standard PCI-X (133 MHz 64 bit) and PCIe x4 have roughly the same data transfer rate, PCIe x4 will give better performance if multiple device pairs are communicating simultaneously or if communication within a single device pair is bidirectional.

History

While in development, PCI Express (PCIe) was referred to as 'Arapaho' or '3GIO' for ''3rd Generation I/O''.
PCIe is a technology which receives further development and improvement. The current standard version in general use at time of writing is PCIe 1.1; however, PCI-SIG announced the availability of the PCI Express Base 2.0 specification on 15 January 2007. PCIe 2.0 doubles the data rate of each lane from 250 MB/s to 500 MB/s. PCIe 2.0 is backward compatible with PCIe 1.1 as a physical interface slot and from within software, so older cards will still be able to work in machines fitted with this new version. Further information on PCIe 2.0 is detailed below.

Hardware protocol summary

The PCIe link is built around dedicated unidirectional couples of serial (1-bit), point-to-point connections known as "lanes". This is in sharp contrast to the PCI connection, which is a bus-based system where all the devices share the same bidirectional, 32-bit (or 64-bit), parallel bus.
PCI Express is a layered protocol, consisting of a Transaction Layer, a Data Link Layer, and a Physical Layer. The Physical Layer is further divided into a logical sublayer and an electrical sublayer. The logical sublayer is frequently further divided into a Physical Coding Sublayer (PCS) and a Media Access Control (MAC) sublayer (terms borrowed from the IEEE 802 model of networking protocol).

Physical Layer

The PCIe physical Layer interface is known by the acronym PIPE which stands for "Physical Interface for PCI Express".
At the electrical level, each lane utilizes two unidirectional LVDS pairs at 2.5 Gbit/s. Transmit and receive are separate differential pairs, for a total of 4 data wires per lane.


A connection between any two PCIe devices is known as a "link", and is built up from a collection of 1 or more lanes. All devices must minimally support single-lane (x1) link. Devices may optionally support wider links composed of 2, 4, 8, 12, 16, or 32 lanes. This allows for very good compatibility in two ways:

★ a PCIe card will physically fit (and work correctly) in any slot that is at least as large as it is (e.g. an x1 sized card will work in any sized slot);

★ a slot of a large physical size (e.g. x16) can be wired electrically with fewer lanes (e.g. x1 or x8) as long as it provides the power and ground connections required by the larger physical slot size.
In both cases, PCIe will negotiate the highest mutually supported number of lanes.
It is not possible to place a physically larger PCIe card (e.g. a 16x sized card) into a smaller slot, even though the two would be signal-compatible if it were possible.

Data transmission

PCIe sends all control messages, including interrupts, over the same links used for data. The serial protocol can never be blocked, so latency is still comparable to PCI, which has dedicated interrupt lines.
Data transmitted on multiple-lane links is interleaved, meaning that each successive byte is sent down successive lanes. The PCIe specification refers to this interleaving as "data striping." While requiring significant hardware complexity to synchronize (or deskew) the incoming striped data, striping can significantly increase the throughput of the link. Due to padding requirements, striping may not necessarily reduce the latency of small data packets on a link.
As with all high data rate serial transmission protocols, clocking information must be embedded in the signal. At the physical level, PCI Express utilizes the very common 8B/10B encoding scheme to ensure that strings of consecutive ones or consecutive zeros are limited in length. This is necessary to prevent the receiver from losing track of where the bit edges are. In this coding scheme every 8 (uncoded) payload bits of data are replaced with 10 (encoded) bits of transmit data, consuming an extra 25% of the overall electrical bandwidth.
Many other protocols (such as SONET) use a different form of encoding known as "scrambling" to embed clock information into data streams. The PCI Express specification also defines a scrambling algorithm, but it is used to reduce EMI (Electromagnetic interference) by preventing repeating data patterns in the transmitted data stream.

Signaling rate

First-generation PCIe is constrained to a single signaling rate of 2.5 Gbit/s. The PCI Special Interest Group (the industry organization that maintains and develops the various PCI standards) plans future versions adding signaling rates of 5 and 10 Gbit/s.

Data Link Layer

The Data Link Layer implements the sequencing of the Transaction Layer Packets (TLPs) that are generated by the Transaction Layer, data protection via a 32-bit cyclic redundancy check code (CRC, known in this context as LCRC) and an acknowledgment protocol (ACK and NAK signaling). TLPs that pass an LCRC check and a sequence number check result in an acknowledgment, or ACK, while those that fail these checks result in a negative acknowledgment, or NAK. TLPs that result in a NAK, or timeouts that occur while waiting for an ACK, result in the TLPs being replayed from a special buffer in the transmit data path of the Data Link Layer. This guarantees delivery of TLPs in spite of electrical noise, barring any malfunction of the device or transmission medium.
ACK and NAK signals are communicated via a low-level packet known as a data link layer packet, or DLLP. DLLPs are also used to communicate flow control information between the transaction layers of two connected devices, as well as some power management functions.

Transaction Layer

PCI Express implements split transactions (transactions with request and response separated by time), allowing the link to carry other traffic while the target device gathers data for the response.
PCI Express utilizes credit-based flow control. In this scheme, a device advertises an initial amount of credit for each of the receive buffers in its Transaction Layer. The device at the opposite end of the link, when sending transactions to this device, will count the number of credits consumed by each TLP from its account. The sending device may only transmit a TLP when doing so does not result in its consumed credit count exceeding its credit limit. When the receiving device finishes processing the TLP from its buffer, it signals a return of credits to the sending device, which then increases the credit limit by the restored amount. The credit counters are modular counters, and the comparison of consumed credits to credit limit requires modular arithmetic. The advantage of this scheme (compared to other methods such as wait states or handshake-based transfer protocols) is that the latency of credit return does not affect performance, provided that the credit limit is not encountered. This assumption is generally met if each device is designed with adequate buffer sizes.
First-generation PCIe is often quoted to support a data rate of 250 MB/s in each direction, per lane. This figure is a calculation from the physical signaling rate (2.5 Gbaud) divided by the encoding overhead (10bits/byte.) This means a 16 lane (x16) PCIe card would then be theoretically capable of 250
★ 16 = 4 GB/s in each direction. While this is correct in terms of data bytes, more meaningful calculations will be based on the usable data payload rate, which depends on the profile of the traffic, which is a function of the high-level (software) application and intermediate protocol levels.
Like other high data rate serial interconnect systems, PCIe has a protocol and processing overhead due to the additional transfer robustness (CRC and Acknowledgments). Long continuous unidirectional transfers (such as those typical in high-performance storage controllers) can approach >95% of PCIe's raw (lane) data rate. These transfers also benefit the most from increased number of lanes (x2, x4, etc.) But in more typical applications (such as a USB or Ethernet controller), the traffic profile is characterized as short data packets with frequent enforced acknowledgments. This type of traffic reduces the efficiency of the link, due to overhead from packet parsing and forced interrupts (either in the device's host interface or the PC's CPU.) This loss of efficiency is not particular to PCIe.

Form factors

★ Low height card

★ Mini Card: a replacement for the Mini PCI form factor (with x1 PCIe, USB 2.0 and SMBus buses on the connector)

★ ExpressCard: successor to the PC card form factor (with x1 PCIe and USB 2.0; hot-pluggable)

★ PCI Express ExpressModule: a hot-pluggable modular form factor defined for servers and workstations

★ XMC: similar to the CMC/PMC form factor (with x4 PCIe or Serial RapidI/O)

★ AdvancedTCA: a complement to CompactPCI for larger applications; supports serial based backplane topologies

★ AMC: a complement to the AdvancedTCA specification; supports processor and I/O modules on ATCA boards (x1,x2,x4 or x8 PCIe).

★ PCI Express External Cabling^[1]

★ Mobile PCI Express Module (MXM) A laptop graphics module specification created by NVIDIA.

★ Advanced eXpress I/O Module (AXIOM) graphics module design endorsed by ATI Technologies.

Competing protocols

Several communications standards have emerged based on high bandwidth serial architectures.
These include but are not limited to HyperTransport, InfiniBand, RapidIO, and StarFabric.
Essentially the differences are based on the tradeoffs between flexibility and extensibility vs. latency and overhead.
An example of such a tradeoff is adding complex header information to a transmitted packet to allow for complex routing (PCI Express is not capable of this).
This additional overhead reduces the effective bandwidth of the interface and complicates bus discovery and initialization software.
Also making the system hot-pluggable requires that software track network topology changes. Examples of buses suited for this purpose are InfiniBand and StarFabric.
Another example is making the packets shorter to decrease latency (as is required if a bus is to be operated as a memory interface). Smaller packets mean that the packet headers consume a higher percentage of the packet, thus decreasing the effective bandwidth. Examples of bus protocols designed for this purpose are RapidIO and HyperTransport.
PCI Express falls somewhere in the middle, targeted by design as a system interconnect (local bus) rather than a device interconnect or routed network protocol. Additionally, its design goal of software transparency constrains the protocol and raises its latency somewhat.

Outlook

As of 2006, PCI Express appears to be well on its way to becoming the new backplane standard in personal computers. There are several explanations for this, but the principal reason is that it was designed to be completely transparent to software developers — an operating system designed for PCI can boot in a PCI Express system without any code modification. Other secondary reasons include its enhanced performance and strong brand recognition.
Almost all of the high end graphics cards being released today (2007) from ATI and NVIDIA use PCI Express. NVIDIA uses the high bandwidth data transfer of PCIe for its newly developed Scalable Link Interface (SLI) technology, which allows two graphics cards of the same chipset and model number to be run at the same time, allowing increased performance. ATI has also developed a dual-GPU system based on PCIe called CrossFire.
Most new Gigabit Ethernet chips and some 802.11 wireless chips also use PCI Express. Other hardware such as RAID controllers and network cards are also starting to make the switch. Sound cards and other cards providing relatively slow interfaces are still mostly based on PCI as of 2007, but are expected to switch over in the following years.
ExpressCard has been introduced on several mid- to high-range laptops such as the Dell Precision and the MacBook Pro. The problem is many laptops have only one PC card slot and it is difficult to give that up for a new ExpressCard slot. Desktops do not have this problem as they have multiple slots and can more easily support PCI Express and the legacy PCI slots concurrently.

PCI Express 2.0

PCI-SIG announced the availability of the PCI Express Base 2.0 specification on 15 January 2007.^[2] PCIe 2.0 doubles the bus standard's bandwidth from 2.5 Gbit/s to 5 Gbit/s, meaning a x32 connector can transfer data at up to 16 GB/s in each direction. PCIe 2.0 is still compatible with PCIe 1.1, so older cards will still be able to work in machines with this new version.
The PCI-SIG also said PCIe 2.0 also features improvements to the point-to-point data transfer protocol and its software architecture.^[3]
In June 2007 Intel released the specification of the P35 chipset which does NOT support PCIe 2.0 only PCIe 1.1.^[4] Some people may be confused by the P35 block diagram^[5] which states the Intel P35 has a PCIe x16 graphics link (8 GB/s) and 6 PCIe x1 links (500 MB/s each), for simple verification one can view the P965 block diagram which shows the same number of lanes and bandwidth but was released before PCIe 2.0 was finalised. Intel's first PCIe 2.0 capable chipset will be the X38 and will be released in Q3 2007.^[6] AMD starts supporting PCIe 2.0 from its RD700 chipset series. NVIDIA has revealed that the MCP72 will be their first PCIe 2.0 equipped chipset.^[7]

PCI Express 3.0

In August 2007 PCI-SIG announced that PCI Express 3.0 will carry a bit rate of 8 gigatransfers per second. The spec will be backwards-compatible with existing PCIe implementations and a final spec is due in 2009. New features for PCIe 3.0 specification include a number of optimizations for enhanced signaling and data integrity, including transmitter and receiver equalization, PLL improvements, clock data recovery, and channel enhancements for currently supported topologies.^[8]
PCI-SIG, the Special Interest Group responsible for PCI Express industry-standard I/O technology, announced the approval of next generation of PCIe architecture, PCIe 3.0
Following a six-month technical analysis of the feasibility of scaling the PCIe interconnect bandwidth, PCI-SIG's analysis found out that 8 gigatransfers per second can be manufactured in mainstream silicon process technology, and can be deployed with existing low-cost materials and infrastructure, while maintaining full compatibility (with negligible impact) to the PCIe protocol stack.
PCIe 2.0 delivers 5GT/s but employed an 8b/10b encoding scheme which took 20 percent overhead on the overall raw bit rate. By removing the requirement for the 8b/10b encoding scheme, PCIe 3.0's 8GT/s bit rate effectively delivers double PCIe 2.0 bandwidth. The PCIe 3.0 specification will also introduce a number of optimizations for enhanced signaling and data integrity, including transmitter and receiver equalization, PLL improvements, clock data recovery, and channel enhancements for currently supported topologies. The final PCIe 3.0 specifications, including form factor specification updates, may be available by late 2009, and could be seen in products starting in 2010 and beyond.
"Experts in the PCIe Electrical Workgroup analyzed both 10GT/s and 8GT/s as target bit rates for the next generation of PCIe architecture, and after careful consideration of several factors, including power, implementation complexity and silicon area, recommended 8GT/s", said Al Yanes, PCI-SIG chairman. "This allows us to satisfy the next generation performance requirements for all existing PCIe applications while maintaining backward compatibility, and at the same time broadening the adoption of this pervasive technology into new and emerging applications and usage models."
PCI-SIG expects the PCIe 3.0 specifications to undergo rigorous technical vetting and validation before being released to the industry. This process, which was followed in the development of prior generations of the PCIe Base and various form factor specifications, includes the corroboration of the final electrical parameters with data derived from test silicon and other simulations conducted by multiple members of the PCI-SIG.

See also

★ Industry Standard Architecture (ISA)

★ Extended Industry Standard Architecture (EISA)

★ Micro Channel architecture (MCA)

★ NuBus

★ VESA Local Bus (VLB)

★ Peripheral Component Interconnect (PCI)

★ Accelerated Graphics Port (AGP)

★ List of device bandwidths (A useful listing of device bandwidths that include PCI Express)

★ CompactPCI

★ AdvancedTCA

★ Geneseo (A future generation PCI Express)

References

1. PCI Express External Cabling 1.0 Specification
2. — note that in this press release the term "aggregate bandwidth" refers to the sum of incoming and outgoing bandwidth; using this terminology the aggregate bandwidth of full duplex 100BASE-TX is 200 Mbit/s
3. PCI Express 2.0 final draft spec published
4. Intel® P35 Express Chipset Product Brief
5. First look - Intel P35 chipset
6. Intel P35: Intel's Mainstream Chipset Grows Up
7. NVIDIA "MCP72" Details Unveiled
8. PCI Express 3.0 Bandwidth: 8.0 Gigatransfers/s

External links

★ PCI-SIG, the industry organization that maintains and develops the various PCI standards

★ How Stuff Works - PCI Express

★ Publishers of CompactPCI and AdvancedTCA Systems Magazine

★ How to identify a PCI Express vs an AGP video card

★ PCI Express Specifications and White Papers (Membership required to download specifications)

★ Creating a Third Generation I/O Interconnect (PDF)

★ Intel Developer Network for PCI Express Architecture

★ PCI Express: An Overview

★ IBM Redbooks: Introduction to PCI Express

★ PCI Express Resources for System Designers

★ PCI Express Form-Factors

★ Dell whitepaper - PCI Express Technology Compares PCIe, PCI, PCI-X.

★ Implementing PCI Express HotPlug

★ PCI Express is Everywhere (published in RTC magazine)

★ The Final Frontier - PCIe on the backplane (published in CompactPCI magazine)

★ New-generation backplane interconnects. PCIe vs Ethernet (Electronic Products magazine)

★ Using PCIe in a variety of multiprocessor system configurations (Embedded Magazine)

★ Using PCI Express in next generation battlefield applications (VME Critical magazine)

★ New PCIe Bridge-Based Riser Cards Allow Connection to PCI-X Cards (Computer Technology Review)

★ When to use an off the shelf PCIe device versus FPGA (Embedded Computing magazine)

★ PCI Express 3.0 Bandwidth: 8.0 Gigatransfers/s (Extremetech.com)