ATA over Ethernet
ATA over Ethernet (AoE) is a network protocol developed by the Brantley Coile Company, designed for simple, high-performance access of block storage devices over Ethernet networks. It is used to build storage area networks (SANs) with low-cost, standard technologies.
AoE runs on layer 2 Ethernet. AoE does not use internet protocol (IP); it cannot be accessed over the Internet or other IP networks. In this regard it is more comparable to Fibre Channel over Ethernet than iSCSI.
With fewer protocol layers, this approach makes AoE fast and lightweight. It also makes the protocol relatively easy to implement, provides intrinsic security protections by virtue of its being non-routable, and offers linear scalability with high performance. The AoE specification is 12 pages compared with iSCSI's 257 pages.
- AoE Header Format:
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 0 | Ethernet Destination MAC Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 4 | Ethernet Destination (cont) | Ethernet Source MAC Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 8 | Ethernet Source MAC Address (cont) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 12 | Ethernet Type (0x88A2) | Ver | Flags | Error | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 16 | Major | Minor | Command | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 20 | Tag | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 24 | Arg | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
AoE has the IEEE assigned EtherType 0x88A2.
SATA (and older PATA) hard drives use the Advanced Technology Attachment (ATA) protocol to issue commands, such as read, write, and status. AoE encapsulates those commands inside Ethernet frames and lets them travel over an Ethernet network instead of a SATA or 40-pin ribbon cable. Although internally AoE uses the ATA protocol, it presents the disks as SCSI to the operating system. Also the actual disks can be SCSI or any other kind, AoE is not limited to disks that use the ATA command set. By using an AoE driver, the host operating system is able to access a remote disk as if it were directly attached.
The encapsulation of ATA provided by AoE is simple and low-level, allowing the translation to happen either at high performance or inside a small, embedded device, or both.
AoE is a layer 2 protocol running at the data-link layer, unlike other SAN protocols which run on top of layer 3 utilizing IP. While this reduces the significant processing overhead of TCP/IP, this means that routers cannot route AoE data across disparate networks (such as the Internet). Instead, AoE packets can only travel within a single local Ethernet storage area network (e.g., a set of computers connected to the same switch or in the same LAN Subnet or VLAN).
The non-routability of AoE is the only security mechanism (i.e., an intruder can't connect through a router—they must physically plug into the local Ethernet switch where Ethernet frame tunneling over routed networks is not in use). However, there are no AoE-specific mechanisms for password verification or encryption. The protocol provides for AoE targets such as Coraid Storage appliances, vblade and GGAOED to establish access lists ("masks") allowing connections only from specific MAC addresses (although these can be spoofed).
The AoE protocol provides a mechanism for host-based cooperative locking. When more than one AoE initiator is using an AoE target they must communicate to avoid interfering with one another as they read and write the config string data on the shared AoE device. Without this cooperation file-system corruption and data loss is likely, unless access is strictly read-only or a cluster file system is used).
One option provided by AoE is to use the storage device itself as the mechanism for determining specific host access. This is the AoE "config string" feature. The config string can record who is using the device, as well as other information. If more than one host tries to set the config string simultaneously, only one succeeds. The other host is informed of the conflict.
Operating system support
The following operating systems provide ATA over Ethernet (AoE) support:
|Linux||Native (2.6.11+)||Coraid |
|Windows||Third-party||StarWind Software AoE Initiator, WinTarget AoE Target, WinAoE, WinVBlock 
|Mac OS X 10.4 and up||Third-party||From 2006 through 2010, 2ºFrost Technologies developed proprietary software and marketed AoE storage solutions in the Windows and Mac markets.
|Mac OS X 10.5 and 10.6||Third-party||Small Tree Communications 
|FreeBSD||Third-party||Coraid  (outdated)|
|OpenBSD||Native (4.5 to 5.6)|
|Plan 9 from Bell Labs ||Native|
Linux target support
Linux can function as an AoE target using one of these independently developed implementations:
- vblade, a userspace daemon that is part of the aoetools package.
- kvblade, a Linux kernel module.
- ggaoed, a userspace daemon that takes advantage of Linux-specific performance features.
- qaoed, a multithreaded userspace daemon.
- aoede, a userspace daemon with experimental protocol extensions.
The Coraid  company offered an array of AoE SAN appliances under the EtherDrive brand, along with diskless gateways that add network-attached storage functionality, using the NFS or SMB protocols, to one or more AoE appliances.
In 2007, LayerWalker announced the world's first single-chip AoE hardware solution called miniSAN running at both Fast and Gigabit Ethernet grades. The miniSAN product family offers standard AoE server functions plus other management features that targets PC, consumer and SMB markets.
Data Fabrics Inc. offers low-cost internal and external GbE AoE enablers as well as desktop and rackmount AoE SAN appliances.
Although AoE is a simple network protocol, it opens up a complex realm of storage possibilities. To understand and evaluate these storage scenarios, it helps to be familiar with a few concepts.
Storage area networks
A SAN allows the physical hard drive to be removed from the server that uses it, and placed on the network. A SAN interface is similar in principle to non-networked interfaces such as SATA or SCSI. Most users will not use a SAN interface directly. Instead, they will connect to a server that uses a SAN disk instead of a local disk. Direct connection, however, can also be used.
When using a SAN network to access storage, there are several potential advantages over a local disk:
- It is easier to add storage capacity and the amount of storage is practically unlimited.
- It is easier to reallocate storage capacity.
- Data may be shared.
- Additionally, compared to other forms of networked storage, SANs are low-level and high performance
Using storage area networks
To use a SAN disk, the host must format it with a filesystem. Unlike a SATA or SCSI disk, however, a SAN hard drive may be accessed by multiple machines. This is a source of both danger and opportunity.
Traditional filesystems (such as FAT or ext3) are designed to be accessed by a single host, and will cause unpredictable behavior if accessed by multiple machines. Such filesystems may be used, and AoE provides mechanisms whereby an AoE target can be guarded against simultaneous access (see: Config String).
Shared disk file systems allow multiple machines to use a single hard disk safely by coordinating simultaneous access to individual files. These filesystems can be used to allow multiple machines access to the same AoE target without an intermediate server or filesystem (and at higher performance).
- S. Hopkins, B. Coile. ATA over Ethernet Specification (Technical report). The Brantley Coile Company, Inc.[dead link]
- Marshall Kirk McKusick, William N. Joy, Samuel J. Leffler and Robert S. Fabry. A Fast File System for UNIX (PDF) (Technical report). Computer Systems Research Group, Computer Science Division, Department of Electrical Engineering and Computer Science, University of California, Berkeley, Berkeley, CA 94720.[dead link]
- S. Hopkins, B. Coile (February 2009). "AoE (ATA over Ethernet)". Retrieved 2011-01-23.[dead link]
- Quanstrom, Erik. "aoecfg(8) - Linux man page". die.net. Retrieved 5 July 2011.
- Linux Support for EtherDrive (R) Storage[dead link]
- StarWind Free AoE Initiator for Microsoft Windows
- WinTarget AoE Target: A bootable, Free AoE Target for Microsoft Windows
- WinAoE Driver: A bootable, open source AoE initiator
- WinVBlock Driver: A WinAoE fork capable of booting AoE targets, file-backed disks and RAM disks created with MEMDISK and GRUB4DOS
- Small Tree - The Mac Network Experts
- Coraid Solaris Support[dead link]
- FreeBSD Support[dead link]
- OpenBSD -current changelog
- CORAID Support - EtherDrive ESX HBA Driver[dead link]
- Plan 9 from Bell Labs
- Coraid Inc.[dead link]
- LayerWalker Technology, Inc.
- ATA Over Ethernet: Putting Hard Drives on the LAN — Linux Journal (28 April 2005)
- ATA-over-Ethernet enables low-cost Linux-oriented SAN — LinuxDevices.com (23 June 2004)
- The ATA over Ethernet (AoE) Protocol — Linux Magazine (June 15, 2005)
- The ATA over Ethernet Protocol Official Document from Coraid Coraid.com (1 June 2009)[dead link]