Link aggregation is a computer networking term to describe various methods of combining (aggregating) multiple network connections in parallel to increase throughput beyond what a single connection could sustain, and to provide redundancy in case one of the links fails.
Further umbrella terms used to describe the method include port trunking,link bundling, Ethernet/network/NIC bonding, or NIC teaming. These umbrella terms not only encompass vendor-independent standards such as Link Aggregation Control Protocol (LACP) for Ethernet defined in IEEE 802.1ax or the previous IEEE 802.3ad, but also various proprietary solutions.
Aggregation can be implemented at any of the lowest three layers of the OSI model. Examples of aggregation at layer 1 are power line (e.g. IEEE 1901) and wireless (e.g. IEEE 802.11) network devices that combine multiple frequency bands. OSI layer 2 (data link layer, e.g. Ethernet frame in LANs or multi-link PPP in WANs, Ethernet MAC address) aggregation typically occurs across switch ports, which can be either physical ports, or virtual ones managed by an operating system. Aggregation is also possible at layer 3 in the OSI model using round-robin scheduling, or based on hash values computed from fields in the packet header, or a combination of these two methods. Regardless of the layer on which aggregation occurs, the network load is balanced across all links. Most methods provide failover as well.
Combining can either occur such that multiple interfaces share one logical address (i.e. IP) or one physical address (i.e. MAC address), or it can be done such that each interface has its own address. The former requires that both ends of a link use the same aggregation method, but has performance advantages over the latter.
- 1 Description
- 2 IEEE link aggregation
- 3 Proprietary link aggregation
- 4 Linux bonding driver
- 5 Linux Team driver
- 6 Usage
- 7 Limitations
- 8 See also
- 9 References
- 10 External links
Link aggregation addresses two problems with Ethernet connections: bandwidth limitations and lack of resilience.
With regard to the first issue: bandwidth requirements do not scale linearly. Ethernet bandwidths historically have increased by an order of magnitude each generation: 10 Megabit/s, 100 Mbit/s, 1000 Mbit/s, 10,000 Mbit/s. If one started to bump into bandwidth ceilings, then the only option was to move to the next generation which could be cost prohibitive. An alternative solution, introduced by many of the network manufacturers in the early 1990s, is to combine two physical Ethernet links into one logical link via channel bonding. Most of these solutions required manual configuration and identical equipment on both sides of the aggregation.
The second problem involves the three single points of failure in a typical port-cable-port connection. In either the usual computer-to-switch or in a switch-to-switch configuration, the cable itself or either of the ports the cable is plugged into can fail. Multiple physical connections can be made, but many of the higher level protocols were not designed to failover completely seamlessly.
By the mid 1990s, most network switch manufacturers had included aggregation capability as a proprietary extension to increase bandwidth between their switches. But each manufacturer developed its own method, which led to compatibility problems. The IEEE 802.3 group took up a study group to create an inter-operable link layer standard in a November 1997 meeting. The group quickly agreed to include an automatic configuration feature which would add in redundancy as well. This became known as "Link Aggregation Control Protocol".
Initial release 802.3ad in 2000
As of 2000[update] most gigabit channel-bonding schemes use the IEEE standard of Link Aggregation which was formerly clause 43 of the IEEE 802.3 standard added in March 2000 by the IEEE 802.3ad task force. Nearly every network equipment manufacturer quickly adopted this joint standard over their proprietary standards.
Move to 802.1 layer in 2008
David Law noted in 2006 that certain 802.1 layers (such as 802.1X security) were positioned in the protocol stack above Link Aggregation which was defined as an 802.3 sublayer. This discrepancy was resolved with formal transfer of the protocol to the 802.1 group with the publication of IEEE 802.1AX-2008 on 3 November 2008.
Link Aggregation Control Protocol
Within the IEEE specification the Link Aggregation Control Protocol (LACP) provides a method to control the bundling of several physical ports together to form a single logical channel. LACP allows a network device to negotiate an automatic bundling of links by sending LACP packets to the peer (directly connected device that also implements LACP).
Advantages over static configuration
- Failover occurs automatically: When a link fails and there is (for example) a media converter between the devices, a peer system will not perceive any connectivity problems. With static link aggregation the peer would continue sending traffic down the link causing the connection to fail.
- Dynamic configuration: The device can confirm that the configuration at the other end can handle link aggregation. With Static link aggregation a cabling or configuration mistake could go undetected and cause undesirable network behavior.
LACP works by sending frames (LACPDUs) down all links that have the protocol enabled. If it finds a device on the other end of the link that also has LACP enabled, it will also independently send frames along the same links enabling the two units to detect multiple links between themselves and then combine them into a single logical link. LACP can be configured in one of two modes: active or passive. In active mode it will always send frames along the configured links. In passive mode however, it acts as "speak when spoken to", and therefore can be used as a way of controlling accidental loops (as long as the other device is in active mode).
In addition to the IEEE link aggregation substandards, there are a number of proprietary aggregation schemes including Cisco's EtherChannel and Port Aggregation Protocol, AVAYA's Multi-Link Trunking, Split Multi-Link Trunking, Routed Split Multi-Link Trunking and Distributed Split Multi-Link Trunking, ZTE's "Smartgroup", or Huawei's "EtherTrunk". Most high-end network devices support some kind of link aggregation, and software-based implementations – such as the *BSD lagg package, Linux' bonding driver, Solaris' dladm etc. – also exist for many operating systems.
Linux bonding driver
The Linux bonding driver provides a method for aggregating multiple network interface controllers (NICs) into a single logical bonded interface of two or more so called (NIC) slaves. The majority of modern Linux distributions (distros) come with a Linux kernel which has the Linux bonding driver integrated as a loadable kernel module and the ifenslave (if = (network) interface) user level control program pre-installed. The Linux bonding driver was originally programmed by Donald Becker. It came into use with the beowulf cluster patches for the Linux kernel 2.0.
Modes for the Linux bonding driver (network interface aggregation modes) are supplied as parameters to the kernel bonding module at load time. They may be given as command line arguments to the insmod or modprobe command, but are usually specified in a Linux distribution-specific configuration file. The behavior of the single logical bonded interface depends upon its specified bonding driver mode. The default parameter is balance-rr.
- Round-robin (balance-rr)
- Transmit network packets in sequential order from the first available network interface (NIC) slave through the last. This mode provides load balancing and fault tolerance.
Storage manufacturers like NetApp (uses a proprietary BSD OS) have deprecated the use of Round Robin on their products as it tends to cause issues related to packet loss. In such cases use of LACP (link aggregation control protocol) is favored.
- Active-backup (active-backup)
- Only one NIC slave in the bond is active. A different slave becomes active if, and only if, the active slave fails. The single logical bonded interface's MAC address is externally visible on only one NIC (port) to avoid distortion in the network switch. This mode provides fault tolerance.
- XOR (balance-xor)
- Transmit network packets based on [(source MAC address XOR'd with destination MAC address) modulo NIC slave count]. This selects the same NIC slave for each destination MAC address. This mode provides load balancing and fault tolerance.
- Broadcast (broadcast)
- Transmit network packets on all slave network interfaces. This mode provides fault tolerance.
- IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP)
- Creates aggregation groups that share the same speed and duplex settings. Utilizes all slave network interfaces in the active aggregator group according to the 802.3ad specification.
- Adaptive transmit load balancing (balance-tlb)
- Linux bonding driver mode that does not require any special network switch support. The outgoing network packet traffic is distributed according to the current load (computed relative to the speed) on each network interface slave. Incoming traffic is received by one currently designated slave network interface. If this receiving slave fails, another slave takes over the MAC address of the failed receiving slave.
- Adaptive load balancing (balance-alb)
- includes balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special network switch support. The receive load balancing is achieved by ARP negotiation. The bonding driver intercepts the ARP Replies sent by the local system on their way out and overwrites the source hardware address with the unique hardware address of one of the NIC slaves in the single logical bonded interface such that different network-peers use different MAC addresses for their network packet traffic.
Linux Team driver
The Linux Team driver provides an alternative to bonding driver. The main difference is that Team driver kernel part contains only essential code and the rest of the code (link validation, LACP implementation, decision making, etc.) is run in userspace as a part of teamd daemon.
Link aggregation offers an inexpensive way to set up a high-speed backbone network that transfers much more data than any one single port or device can deliver. Link aggregation also allows the network's backbone speed to grow incrementally as demand on the network increases, without having to replace everything and buy new hardware.
Most backbone installations install more cabling or fiber optic pairs than is initially necessary, even if they have no immediate need for the additional cabling. This is done because labor costs are higher than the cost of the cable, and running extra cable reduces future labor costs if networking needs change. Link aggregation can allow the use of these extra cables to increase backbone speeds for little or no extra cost if ports are available.
Order of frames
When balancing traffic, network administrators often wish to avoid reordering Ethernet frames. For example, TCP suffers additional overhead when dealing with out-of-order packets. This goal is approximated by sending all frames associated with a particular session across the same link. The most common implementations use L3 hashes (i.e. based on the IP address), ensuring that the same flow is always sent via the same physical link.
However, depending on the traffic, this may not provide even distribution across the links in the trunk. It effectively limits the client bandwidth in an aggregate to its single member's maximum bandwidth per session. Principally for this reason 50/50 load balancing is almost never reached in real-life implementations; around 70/30 is more usual. Advanced switches can employ an L4 hash (i.e. using TCP/UDP port numbers), which will bring the balance closer to 50/50 as different L4 flows between two hosts can make use of different physical links.
Efficiency of equipment
||This section possibly contains original research. (January 2013)|
Aggregation becomes inefficient beyond a certain bandwidth — depending on the total number of ports on the switch equipment. A 24-port gigabit switch with two 8-gigabit trunks is using sixteen of its available ports just for the two interswitch connections, and leaves only eight of its 1-gigabit ports for other devices. This same configuration on a 48-port gigabit switch leaves 32 1-gigabit ports available, and so it is much more efficient (assuming of course that those ports are actually needed at the switch location).
When a switch utilizes 40-50% of its ports for backbone trunking, upgrading to a switch with either more ports or a higher base-operating speed may be a better option than simply adding more switches, especially if the old switch can be re-used elsewhere on a less performance-critical part of the network.
Multiple switches may be utilized to optimize for maximum throughput in a multiple network switch topology, when the switches are configured in parallel as part of an isolated network between two or more systems. In this configuration, the switches are isolated from one another. One reason to employ a topology such as this is for an isolated network with many hosts (a cluster configured for high performance, for example), using multiple smaller switches can be more cost effective than a single larger switch. If access beyond the network is required, an individual host can be equipped with an additional network device connected to an external network; this host then additionally acts as a gateway. The network interfaces 1 through 3 of computer cluster node A, for example, are connected via separate network switches 1 through 3 with network interfaces 1 through 3 of computer cluster node B; there are no inter-connections between the network switches 1 through 3. The linux bonding driver mode typically employed in configurations of this type is balance-rr; the balance-rr mode allows individual connections between two hosts to effectively utilize greater than one interface's bandwidth.
Use on network interface cards
NICs trunked together can also provide network links beyond the throughput of any one single NIC. For example, this allows a central file server to establish an aggregate 2-gigabit connection using two 1-gigabit NICs trunked together. Note the data signaling rate will still be 1Gbit/s, which can be misleading depending on methodologies used to test throughput after link aggregation is employed.
Microsoft Windows does support native link aggregation starting from Windows Server 2012. For the previous Windows Server versions however, some manufacturers provide software for aggregation on their multiport NICs at the device-driver layer. Intel, for example, has released a package for Windows called Advanced Networking Services (ANS) to bind Intel Fast Ethernet and Gigabit cards.
Nvidia also supports "teaming" with their Nvidia Network Access Manager/Firewall Tool. HP also has a teaming tool for HP branded NICs which will allow for non-etherchanneled NIC teaming or which will also support several modes of etherchannel (port aggregation) including 802.3ad with LACP. In addition there is a basic layer-3 aggregation (available at least from Windows XP SP3), that allows servers with multiple IP interfaces on the same network to perform load balancing, and home users, with more than 1 internet connection, to increase connection speed by sharing the load on all interfaces.
Broadcom offers advanced functions via Broadcom Advanced Control Suite (BACS) via which the teaming-functionality of BASP (advanced server program) are available offering 802.3ad static lags, LACP and "smart teaming" which doesn't require any configuration on the switches to work. It is possible to configure teaming with BACS with a mix of NIC's from different vendors as long as at least one of them is Broadcom and the other NIC's do have the required capabilities to create teaming.
Linux and UNIX
Linux, FreeBSD, NetBSD, OpenBSD, Mac OS X, OpenSolaris and commercial Unix distributions such as AIX implement Ethernet bonding (trunking) at a higher level, and can hence deal with NICs from different manufacturers or drivers, as long as the NIC is supported by the kernel.
Citrix XenServer and VMware ESX have native support for link-aggregation. XenServer offers both static-LAG's as well as LACP. vSphere 5.1 (ESXi) now supports both static-LAG's and LACP natively with their virtual distributed switch.
For Microsoft's Hyper-V bonding or teaming isn't offered from the hyper-visor or OS-level, but the above mentioned methods for teaming under Windows applies to Hyper-V as well.
With modes balance-rr, balance-xor, broadcast and 802.3ad all physical ports in the link aggregation group must reside on the same logical switch, which in most scenarios will leave a single point of failure when the physical switch to which both links are connected goes offline. Modes active-backup, balance-tlb, and balance-alb can also be set up with two or more switches. But after failover (like all other modes), in some cases, active sessions may fail (due to ARP problems) and have to be restarted.
However, almost all vendors have proprietary extensions that resolve some of this issue: they aggregate multiple physical switches into one logical switch. As of 2009[update], the IEEE has not yet committed resources to standardize this feature. The Split multi-link trunking (SMLT) protocol allows multiple Ethernet links to be split across multiple switches in a stack, preventing any single point of failure, and additionally allowing all switches to be load balanced across multiple aggregation switches from the single access stack. These devices synchronize state across an Inter-Switch Trunk (IST) such that they appear to the connecting (access) device to be a single device (switch block) and prevent any packet duplication. SMLT provides enhanced resiliency with sub-second failover and sub-second recovery for all speed trunks (10 Mbit/s, 100 Mbit/s, 1,000 Mbit/s, and 10 Gbit/s) while operating transparently to end-devices.
Arista switches support the MLAG (multi link aggregation) protocol to exchange messages that describe their LACP states. Hosts can be connected to two different switches using LACP only and see one logical link.
In most implementations, all the ports used in an aggregation consist of the same physical type, such as all copper ports (10/100/1000BASE‑T), all multi-mode fiber ports, or all single-mode fiber ports. However, all the IEEE standard requires is that each link be full duplex and all of them have an identical speed (10, 100, 1,000 or 10,000 Mbit/s).
Many switches are PHY independent, meaning that a switch could have a mixture of copper, SX, LX, LX10 or other GBICs. While maintaining the same PHY is the usual approach, it is possible to aggregate a 1000BASE-SX fiber for one link and a 1000BASE-LX (longer, diverse path) for the second link, but the important thing is that the speed will be 1 Gbit/s full duplex for both links. One path may have a slightly longer transit time but the standard has been engineered so this will not cause an issue.
Ethernet aggregation mismatch
Aggregation mismatch refers to not matching the aggregation type on both ends of the link. Some switches do not implement the 802.1AX standard but support static configuration of link aggregation. Therefore link aggregation between similarly statically configured switches will work, but will fail between a statically configured switch and a device that is configured for LACP.
- "Chapter 5.4: Link Aggregation Control Protocol (LACP)". IEEE Std 802.1AX-2008 IEEE Standard for Local and Metropolitan Area Networks — Link Aggregation. IEEE Standards Association. 2008-11-03. p. 30. doi:10.1109/IEEESTD.2008.4668665.
- Tech Tips - Bonding Modes
- Guijarro, Manuel; Ruben Gaspar et al (2008). "Experience and Lessons learnt from running High Availability Databases on Network Attached Storage" (PDF). Journal of Physics: Conference Series. Conference Series (IOP Publishing) 119 (4): 042015. doi:10.1088/1742-6596/119/4/042015. Retrieved 2009-08-17. "Network bonding (also known as port trunking) consists of aggregating multiple network interfaces into a single logical bonded interface that correspond to a single IP address."
- "IEEE 802.3ad Link Bundling". Cisco Systems. 2007-02-27. Retrieved 2012-03-15.
- "IEEE 802 Trunking Tutorial". 1997-11-11. Retrieved 2013-08-13.
- IEEE 802.3ad Link Aggregration Task Force
- Law, David (2006-02-13). "IEEE 802.3 Maintenance" (PDF). p. 9. Retrieved 2009-08-18. "Proposal to move Link Aggregation to IEEE 802.1 •It is an 802.3 sublayer but it has to go above IEEE Std 802.1x"
- "IEEE SA - 802.1AX-2008 - IEEE Standard for Local and metropolitan area networks--Link Aggregation". Retrieved 2013-08-13.
- Link aggregation on Dell servers
- The Linux Foundation: Bonding
- libteam project
- http://grouper.ieee.org/groups/802/3/hssg/public/apr07/frazier_01_0407.pdf[unreliable source?]
- Intel Advanced Networking Services
- RandomAdapter: Core Services, on MS TechNet
- Load Balance Network Adapters, at PCTool's Registry Guide for Windows
- Broadcom Windows Management Applications, visited 8 July 2012
- VMware What's New in vSphere 5.1 networking, June, 2012. Visited 17 January 2013