PCI Express

PCI Express

Year created	2004
Created by	Intel
Width in bits	1-32
No. of devices	1 per slot
Speed	Per lane: v1.x: 250 MB/s v2.0: 500 MB/s v3.0: 1 GB/s
Style	Serial
Hotplugging interface	Depends on form factor
External interface	Yes, with PCI Express External Cabling
Website	pcisig.com

PCI Express (Peripheral Component Interconnect Express), officially abbreviated as PCIe, is a computer expansion card standard designed to replace the older PCI, PCI-X, and AGP standards. Introduced by Intel in 2004, PCIe (or PCI-E, as it is commonly called) is the latest standard for expansion cards that is available on mainstream personal computers

PCI Express is used in consumer, server, and industrial applications, both as a motherboard-level interconnect (to link motherboard-mounted peripherals) and as an expansion card interface for add-in boards. A key difference between PCIe and earlier PC buses is a topology based on point-to-point serial links, rather than a shared parallel bus architecture.

The PCIe electrical interface is also used in a variety of other standards, most notably the ExpressCard laptop expansion card interface.

Overview

A PCI Express x16 slot

A PCI Express x1 slot

Conceptually, the PCIe bus can be thought of as a 'high-speed serial replacement' of the older (parallel) PCI/PCI-X bus. At the software-level, PCIe preserves compatibility with PCI; a PCIe device can be configured and used in legacy applications and operating-systems which have no direct knowledge of PCIe's newer features. In terms of bus-protocol, PCIe communication is encapsulated in packets. The work of packetizing and depacketizing data and status-message traffic is handled by the transaction-layer of the PCIe port (described later.) Radical differences in electrical-signalling and bus-protocol require the use of a different mechanical formfactor and expansion connectors (and thus, new motherboards and new adapter-boards.)

PCIe devices communicate via a logical connection called an interconnect or link. A link is a point-to-point communication channel between 2 PCIe ports, allowing both to send/receive ordinary PCI-requests (configuration read/write, I/O read/write, memory read/write) and interrupts (INTx, MSI, MSI-X) At the physical level, a link is composed of 1 or more lanes; low-speed peripherals (such as a 802.11 Wi-Fi card) use only a single-lane (x1) link, while a graphics-adapter typically uses a much-wider (and faster) 16-lane link.

A lane is composed of a transmit and a receive pair of differential lines. Conceptually, the lane is a full-duplex byte stream, transporting packets containing the data in 8 bit 'byte' format, between the two endpoints of a link, in both directions simultaneously. Physical PCIe slots may contain from one to thirty-two lanes, in powers of two (1, 2, 4, 8, 16 and 32). Lane counts are written with an "x" prefix (e.g. "x16" represents a sixteen-lane card or slot), with x16 being the largest size in common use.

A PCIe card will fit into a slot of its physical size or bigger, but may not fit into a smaller PCIe slot. Some slots use open-ended sockets to permit physically longer cards and will negotiate the best available electrical connection. The number of lanes actually connected to a slot may also be less than the number supported by the physical slot size. An example is a x8 slot that actually only runs at x1; these slots will allow any x1, x2, x4 or x8 card to be used, though only running at the x1 speed. This type of socket is described as a "x8 (x1 mode)" slot, meaning it physically accepts up to x8 cards but only runs at x1 speed. The advantage gained is that a larger range of PCIe cards can still be used without requiring the motherboard hardware to support the full transfer rate — in so doing keeping design and implementation costs down.

Specs for each generation per lane

	Clock speed	Transfer rate	Overhead	Data rate
1.x	1.25 GHz	2.5 GT/s	20%	250 MB/s
2.0	2.5 GHz	5 GT/s	20%	500 MB/s
3.0	4 GHz	8 GT/s	0%	1 GB/s

In PCIe 1.x , each lane carries 250 MB/s. PCIe 2.0, released in late 2007, adds a Gen2-signalling mode, doubling the rate to 500 MB/s. PCIe 3.0, currently in development (for release around 2010), will add a Gen3-signalling mode, at 1 GB/s.

As a point of reference, a single-lane PCIe card has nearly twice as much bandwidth as the most common PCI interface, a 32-bit 33MHz PCI bus (133 MB/s). A PCIe x4 slot has bandwidth comparable to the fastest version of PCI-X 1.0 (64-bit 133MHz.) An eight-lane slot has a transfer rate comparable to the fastest version of AGP.

Design

PCIe, unlike previous PC expansion standards, is structured around point-to-point serial links, a pair of which (one in each direction) make up lanes; rather than a shared parallel bus. These lanes are routed by a hub on the main-board acting as a crossbar switch. This dynamic point-to-point behavior allows more than one pair of devices to communicate with each other at the same time. In contrast, older PC interfaces had all devices permanently wired to the same bus; therefore, only one device could send information at a time. This format also allows "channel grouping," where multiple lanes are bonded to a single device pair in order to provide higher bandwidth.

The number of lanes is "negotiated" during power-up or explicitly during operation. By making the lane count flexible a single standard can provide for the needs of high-bandwidth cards (e.g. graphics cards, 10 Gigabit Ethernet cards and multiport Gigabit Ethernet cards) while also being economical for less demanding cards.

Unlike preceding PC expansion interface standards, PCIe is a network of point-to-point connections. This removes the need for "arbitrating" the bus or waiting for the bus to be free and allows for full duplex communications. This means that while standard PCI-X (133 MHz 64 bit) and PCIe x4 have roughly the same data transfer rate, PCIe x4 will give better performance if multiple device pairs are communicating simultaneously or if communication within a single device pair is bidirectional.

Specifications of the format are maintained and developed by a group of more than 900 industry-leading companies called the PCI-SIG (PCI Special Interest Group).

Serial

The bonded serial format was chosen over a traditional parallel format due to the phenomenon of timing skew. Timing skew is a direct result of the limitations imposed by the speed of an electrical signal traveling down a wire, which it does at a finite speed. Because different traces in an interface have different lengths, parallel signals transmitted simultaneously from a source arrive at their destinations at different times. When the interconnection clock rate rises to the point where the wavelength of a single bit is less than this difference in path length, the bits of a single word do not arrive at their destination simultaneously, making parallel recovery of the word difficult. Thus, the speed of the electrical signal, combined with the difference in length between the longest and shortest trace in a parallel interconnect, leads to a naturally imposed maximum bandwidth. Serial channel bonding avoids this issue by not requiring the bits to arrive simultaneously. PCIe is just one example of a general trend away from parallel buses to serial interconnects. Other examples include Serial ATA, USB, SAS, FireWire and RapidIO. The multichannel serial design also increases flexibility by allowing slow devices to be allocated fewer lanes than fast devices.

History

While in development, PCIe was initially referred to as HSI (for High Speed Interconnect), and underwent a name change to 3GIO (for 3rd Generation I/O) before finally settling on its PCI-SIG name PCI Express. It was first drawn up by a technical working group named the Arapaho Work Group (AWG) which for initial drafts consisted of an Intel only team of architects. Subsequently the AWG was expanded to include industry partners.

PCIe is a technology under constant development and improvement. In 2004, Intel introduced PCIe 1.0, with a data rate of 250 MB/s and a transfer rate of 2.5 GT/s. Later, version 1.1 offered minor revisions to the specification. On 15 January 2007, the PCI-SIG announced the availability of the PCI Express Base 2.0 specification. This doubled the data rate of each lane from 250 to 500, and the transfer rate from 2.5 GT/s to 5 GT/s. PCIe 2.0 is backward compatible with PCIe 1.1 as a physical interface slot and from within software, so older cards will still be able to work in machines fitted with PCIe 2.0. The proposed PCIe 3.0 is scheduled for release around 2010.

Hardware protocol summary

The PCIe link is built around dedicated unidirectional couples of serial (1-bit), point-to-point connections known as lanes. This is in sharp contrast to the conventional PCI connection, which is a bus-based system where all the devices share the same bidirectional, 32-bit (or 64-bit), parallel bus.

PCI Express is a layered protocol, consisting of a Transaction Layer, a Data Link Layer, and a Physical Layer. The Data Link Layer is further divided to include a Media Access Control sublayer. The Physical Layer is further divided into a logical sublayer and an electrical sublayer. The PHY logical sublayer contains a Physical Coding Sublayer (PCS). (Terms borrowed from the IEEE 802 model of networking protocol.)

Physical Layer

The PCIe Physical Layer (PHY) (PCIEPHY , PCI Express PHY or PCIe PHY) specification is divided into two sub-layers, corresponding to electrical and logical specifications. The logical sublayer is sometimes further divided into a MAC (Media Access Control) sublayer and a PCS (Physical Coding Sublayer), although this division is not formally part of the PCIe specification. A specification published by Intel, the PHY Interface for PCI Express (PIPE) , defines the MAC/PCS functional partitioning and the interface between these two sub-layers. The PIPE specification also identifies the PMA (Physical Media Attachment) layer, which includes the Serializer/Deserializer (SerDes) and other analog circuitry; however, since SerDes implementations vary greatly among ASIC vendors, PIPE does not specify an interface between the PCS and PMA.

At the electrical level, each lane consists of two unidirectional LVDS or PCML pairs at 2.525 Gbit/s. Transmit and receive are separate differential pairs, for a total of 4 data wires per lane.

A connection between any two PCIe devices is known as a "link", and is built up from a collection of 1 or more lanes. All devices must minimally support single-lane (x1) link. Devices may optionally support wider links composed of 2, 4, 8, 12, 16, or 32 lanes. This allows for very good compatibility in two ways:

• a PCIe card will physically fit (and work correctly) in any slot that is at least as large as it is (e.g. an x1 sized card will work in any sized slot);
• a slot of a large physical size (e.g. x16) can be wired electrically with fewer lanes (e.g. x1, x4, or x8) as long as it provides the power and ground connections required by the larger physical slot size.

In both cases, PCIe will negotiate the highest mutually supported number of lanes.

Even though the two would be signal-compatible, it is not usually possible to place a physically larger PCIe card (e.g. a 16x sized card) into a smaller slot — though some motherboards have open-ended PCIe slots that will allow this.

The width of a PCIe connector is 8.8 mm, while the height is 11.25 mm, and the length is variable. The 'minor' half of the connector is 11.65 mm in length and contains 22 pins, while the length of the 'major' half is variable. The thickness of the card going into the connector is 1.8 mm.

Lanes	Pins Total	Pins in 'major' half	Total Length	Length of 'major' half
x1	36	14	25 mm	7.65 mm
x4	64	42	39 mm	21.65 mm
x8	98	76	56 mm	38.65 mm
x16	164	142	89 mm	71.65 mm

Data transmission

PCIe sends all control messages, including interrupts, over the same links used for data. The serial protocol can never be blocked, so latency is still comparable to conventional PCI, which has dedicated interrupt lines.

Data transmitted on multiple-lane links is interleaved, meaning that each successive byte is sent down successive lanes. The PCIe specification refers to this interleaving as "data striping". While requiring significant hardware complexity to synchronize (or deskew) the incoming striped data, striping can significantly reduce the latency of the n^th byte on a link. Due to padding requirements, striping may not necessarily reduce the latency of small data packets on a link.

As with all high data rate serial transmission protocols, clocking information must be embedded in the signal. At the physical level, PCI Express utilizes the very common 8b/10b encoding scheme to ensure that strings of consecutive ones or consecutive zeros are limited in length. This is necessary to prevent the receiver from losing track of where the bit edges are. In this coding scheme every 8 (uncoded) payload bits of data are replaced with 10 (encoded) bits of transmit data, causing a 20% overhead in the electrical bandwidth.

Many other protocols (such as SONET) use a different form of encoding known as "scrambling" to embed clock information into data streams. The PCIe specification also defines a scrambling algorithm, but it is used to reduce EMI (Electromagnetic interference) by preventing repeating data patterns in the transmitted data stream. In the yet to be released PCIe 3.0, scrambling fully replaces the 8b/10b encoding, eliminating the 20% overhead.

Data Link Layer

The Data Link Layer implements the sequencing of the Transaction Layer Packets (TLPs) that are generated by the Transaction Layer, data protection via a 32-bit cyclic redundancy check code (CRC, known in this context as LCRC) and an acknowledgment protocol (ACK and NAK signaling). TLPs that pass an LCRC check and a sequence number check result in an acknowledgment, or ACK, while those that fail these checks result in a negative acknowledgment, or NAK. TLPs that result in a NAK, or timeouts that occur while waiting for an ACK, result in the TLPs being replayed from a special buffer in the transmit data path of the Data Link Layer. This guarantees delivery of TLPs in spite of lectrical noise, barring any malfunction of the device or transmission medium.

ACK and NAK signals are communicated via a low-level packet known as a data link layer packet, or DLLP. DLLPs are also used to communicate flow control information between the transaction layers of two connected devices, as well as some power management functions.

Transaction Layer

PCI Express implements split transactions (transactions with request and response separated by time), allowing the link to carry other traffic while the target device gathers data for the response.

PCI Express utilizes credit-based flow control. In this scheme, a device advertises an initial amount of credit for each of the receive buffers in its Transaction Layer. The device at the opposite end of the link, when sending transactions to this device, will count the number of credits consumed by each TLP from its account. The sending device may only transmit a TLP when doing so does not result in its consumed credit count exceeding its credit limit. When the receiving device finishes processing the TLP from its buffer, it signals a return of credits to the sending device, which then increases the credit limit by the restored amount. The credit counters are modular counters, and the comparison of consumed credits to credit limit requires modular arithmetic. The advantage of this scheme (compared to other methods such as wait states or handshake-based transfer protocols) is that the latency of credit return does not affect performance, provided that the credit limit is not encountered. This assumption is generally met if each device is designed with adequate buffer sizes.

PCIe 1.x is often quoted to support a data rate of 250 MB/s in each direction, per lane. This figure is a calculation from the physical signaling rate (2.5 Gbaud) divided by the encoding overhead (10 bits per byte.) This means a sixteen lane (x16) PCIe card would then be theoretically capable of 250 MB/s * 16 = 4 GB/s in each direction. While this is correct in terms of data bytes, more meaningful calculations will be based on the usable data payload rate, which depends on the profile of the traffic, which is a function of the high-level (software) application and intermediate protocol levels.

Like other high data rate serial interconnect systems, PCIe has a protocol and processing overhead due to the additional transfer robustness (CRC and Acknowledgments). Long continuous unidirectional transfers (such as those typical in high-performance storage controllers) can approach >95% of PCIe's raw (lane) data rate. These transfers also benefit the most from increased number of lanes (x2, x4, etc.) But in more typical applications (such as a USB or Ethernet controller), the traffic profile is characterized as short data packets with frequent enforced acknowledgments. This type of traffic reduces the efficiency of the link, due to overhead from packet parsing and forced interrupts (either in the device's host interface or the PC's CPU.) This loss of efficiency is not particular to PCIe.

Alternate form factors

There are several other expansion card types derived from PCIe. These include:

• Low height card
• ExpressCard: successor to the PC card form factor (with x1 PCIe and USB 2.0; hot-pluggable)
• PCI Express ExpressModule: a hot-pluggable modular form factor defined for servers and workstations
• XMC: similar to the CMC/PMC form factor (with x4 PCIe or Serial RapidI/O)
• AdvancedTCA: a complement to CompactPCI for larger applications; supports serial based backplane topologies
• AMC: a complement to the AdvancedTCA specification; supports processor and I/O modules on ATCA boards (x1,x2,x4 or x8 PCIe).

PCI Express Mini Card

A WLAN PCI Express Mini Card and its connector.

MiniPCI and MiniPCI Express cards in comparison

PCI Express Mini Card (also known as Mini PCI Express, Mini PCIe, and Mini PCI-E) is a replacement for the Mini PCI form factor based on PCI Express. It is developed by the PCI-SIG. The host device supports both PCI Express and USB 2.0 connectivity, and each card uses whichever the designer feels most appropriate to the task. Most laptop computers built after 2005 are based on PCI Express and can have several Mini Card slots.

Physical Dimensions

PCI Express Mini Cards are 30 x 56 mm. There is a 52 pin edge connector, consisting of two staggered rows on a 0.8 mm pitch. Each row has 8 contacts, a gap equivalent to 4 contacts, then a further 18 contacts. A half-length card is also specified 30 x 31.9 mm. Cards have a thickness of 1.0 mm (excluding components).

Electrical Interface

PCI Express Mini Card edge connector provide multiple connections and buses:

• PCIe x1
• USB 2.0
• SMBus
• Wires to diagnostics LEDs for wireless network (i.e. WiFi) status on computer's chassis
• SIM card for GSM and WCDMA applications
• Future extension for another PCIe lane
• 1.5 and 3.3 Volt power

PCI Express External Cabling

PCI Express External Cabling (also known as External PCI Express or Cabled PCI Express) specifications were released by the PCI-SIG in February 2007.

Standard cables and connectors have been defined for x1, x4, x8, and x16 link widths, with a transfer rate of 250 MB/s per lane. The PCI-SIG also expects the norm will evolve to reach the 500 MB/s as found in PCI Express 2.0. The maximum cable length hasn't been determined yet.

External PCIe Video Cards

Theoretically, External PCIe could give a notebook the graphic power of a desktop, by connecting a notebook with any PCIe desktop video card (enclosed in its own external housing), however, only one finalized product and two concept products exist. All three deliver the power of the video card to external displays only, and all connect to a notebook through an ExpressCard interface which limits the bandwidth from an inserted x16 video card (4 GB/s in each direction), to just x1 (250 MB/s in each direction).

Additionally, Nvidia has developed Quadro Plex, external PCIe Video Cards that can be used for advanced graphic applications. These video cards require a PCI Express x8 or x16 slot, for the interconnection cable. AMD has recently announced the ATI XGP technology based on proprietary cabling solution which is compatible with PCIe signal transmissions.

There are now card hubs that one can connect to a laptop through an expresscard slot in development, though they are currently rare, obscure, or unavailable on the open market. These hubs can have full sized cards placed in them, but they can only support an external monitor and not the laptop monitor.

Competing protocols

Several communications standards have emerged based on high bandwidth serial architectures. These include but are not limited to InfiniBand, RapidIO, and StarFabric.

Essentially the differences are based on the tradeoffs between flexibility and extensibility vs. latency and overhead. An example of such a tradeoff is adding complex header information to a transmitted packet to allow for complex routing (PCI Express is not capable of this). This additional overhead reduces the effective bandwidth of the interface and complicates bus discovery and initialization software. Also making the system hot-pluggable requires that software track network topology changes. Examples of buses suited for this purpose are InfiniBand and StarFabric.

Another example is making the packets shorter to decrease latency (as is required if a bus is to be operated as a memory interface). Smaller packets mean that the packet headers consume a higher percentage of the packet, thus decreasing the effective bandwidth. Examples of bus protocols designed for this purpose are RapidIO and HyperTransport.

PCI Express falls somewhere in the middle, targeted by design as a system interconnect (local bus) rather than a device interconnect or routed network protocol. Additionally, its design goal of software transparency constrains the protocol and raises its latency somewhat.

Status

PCI Express has replaced AGP as the default interface for graphics cards on new systems. With a few exceptions, all graphics cards being released today (2009) from ATI and NVIDIA use PCI Express. NVIDIA uses the high bandwidth data transfer of PCIe for its Scalable Link Interface (SLI) technology, which allows multiple graphics cards of the same chipset and model number to be run in tandem, allowing increased performance. ATI also has developed a multi-GPU system based on PCIe called CrossFire. Eventually AMD and NVIDIA released motherboard chipsets which support up to four PCIe x16 slots, allowing tri-GPU and quad-GPU card configurations.

Uptake for other forms of PC expansion has been much slower and conventional PCI remains dominant. PCI Express is commonly used for disk array controllers, onboard gigabit ethernet and wi-fi but add-in cards are still generally conventional PCI, particularly at the lower end of the market. Sound cards, modems, serial port cards and other cards with low-speed interfaces are still nearly all conventional PCI. For this reason most motherboards supporting PCI Express offer conventional PCI slots as well.

ExpressCard has been introduced on several mid- to high-range laptops such as Apple's MacBook Pro line. Unlike desktops, however, laptops frequently only have one expansion slot. Replacing the PC card slot with ExpressCard slot means a loss in compatibility with PC-card devices.

PCI Express 2.0

PCI-SIG announced the availability of the PCI Express Base 2.0 specification on 15 January 2007. PCIe 2.0 doubles the bus standard's bandwidth from 0.25 GByte/s to 0.5 GByte/s, meaning a x32 connector can transfer data at up to 16 GByte/s for both videocards (SLI 2x etc). PCIe 2.0 has two 32 bits channels for each GPU (2x16), while the first version only has 1x16 and is operating at 2 GHz.

PCIe 2.0 motherboard slots are backward compatible with PCIe v1.x. PCIe 2.0 cards have good backwards compatibility, new PCIe 2.0 graphics cards are compatible with PCIe 1.1 motherboards, meaning that they will run on them using the available bandwidth of PCI Express 1.1. Overall, graphic cards or motherboards designed for v2.0 will be able to work with the other being v1.1 or v1.0.

The PCI-SIG also said that PCIe 2.0 features improvements to the point-to-point data transfer protocol and its software architecture.

In June 2007 Intel released the specification of the Intel P35 chipset which does not support PCIe 2.0 only PCIe 1.1. Some people may be confused by the P35 block diagram which states the Intel P35 has a PCIe x16 graphics link (8 GB/s) and 6 PCIe x1 links (500 MB/s each). For simple verification one can view the P965 block diagram which shows the same number of lanes and bandwidth but was released before PCIe 2.0 was finalized. Intel's first PCIe 2.0 capable chipset was the X38 and boards began to ship from various vendors (Abit, Asus, Gigabyte) as of October 21, 2007. AMD started supporting PCIe 2.0 with its RD700 chipset series and nVidia started with the MCP72. The specification of the Intel P45 chipset includes PCIe 2.0 .

PCI Express 3.0

In August 2007 PCI-SIG announced that PCI Express 3.0 will carry a bit rate of 8 gigatransfers per second. The spec will be backwards-compatible with existing PCIe implementations and a final spec is due in 2009. New features for PCIe 3.0 specification include a number of optimizations for enhanced signaling and data integrity, including transmitter and receiver equalization, PLL improvements, clock data recovery, and channel enhancements for currently supported topologies.

Following a six-month technical analysis of the feasibility of scaling the PCIe interconnect bandwidth, PCI-SIG's analysis found out that 8 gigatransfers per second can be manufactured in mainstream silicon process technology, and can be deployed with existing low-cost materials and infrastructure, while maintaining full compatibility (with negligible impact) to the PCIe protocol stack.

PCIe 2.0 delivers 5 GT/s but employed an 8b/10b encoding scheme which took 20 percent overhead on the overall raw bit rate. By removing the requirement for the 8b/10b encoding scheme (relying solely on the still-used scrambler), PCIe 3.0's 8 GT/s bit rate effectively delivers double PCIe 2.0 bandwidth. According to an official press release by PCI-SIG on August 8, 2007:

"The final PCIe 3.0 specifications, including form factor specification updates, may be available by late 2009, and could be seen in products starting in 2010 and beyond."

PCI-SIG expects the PCIe 3.0 specifications to undergo rigorous technical vetting and validation before being released to the industry. This process, which was followed in the development of prior generations of the PCIe Base and various form factor specifications, includes the corroboration of the final electrical parameters with data derived from test silicon and other simulations conducted by multiple members of the PCI-SIG.

See also

• Industry Standard Architecture (ISA)
• Extended Industry Standard Architecture (EISA)
• Micro Channel architecture (MCA)
• NuBus
• VESA Local Bus (VLB)
• PCI Local Bus
• Accelerated Graphics Port (AGP)
• List of device bandwidths (A useful listing of device bandwidths that include PCI Express)
• PCI-X

External links

• PCI-SIG, the industry organization that maintains and develops the various PCI standards
• Intel Developer Network for PCI Express Architecture