exFAT

From Wikipedia, the free encyclopedia
Jump to: navigation, search
exFAT
Developer Microsoft
Full name Extended File Allocation Table
Introduced November 2006 (Windows Embedded CE 6.0)
Partition identifier MBR/EBR: 0x07 (same as for HPFS/NTFS)
BDP/GPT: EBD0A0A2-B9E5-4433-87C0-68B6B72699C7
Structures
Directory contents Table
File allocation bitmap, linked list
Bad blocks Cluster tagging
Limits
Max. volume size ca. 128 PiB, 512 TiB recommended[1]
Max. file size ca. 128 PiB (theoretical 16 EiB–1)[nb 1]
Max. number of files up to 2,796,202 per directory[2]
Max. filename length 255 UTF-16 characters
Allowed characters in filenames Unicode UTF-16 except U+0000 (NUL) through U+001F (US) / (slash) \ (backslash) : (colon) * (asterisk) ? (Question mark) " (quote) < (less than) > (greater than) and | (pipe)
Features
Dates recorded Creation, modified, last access
Date range 1980-01-01 to 2107-12-31
Date resolution 10 ms
Forks No
Attributes Read-only, hidden, system, subdirectory, archive
File system permissions ACL (Windows CE 6 only)
Transparent compression No
Transparent encryption No
Other
Supported operating systems Windows Embedded CE 6.0
Windows XP (including x64) SP2 and later (optional[1])
Windows Server 2003 SP2 (optional[1])
Windows Vista SP1 and later
Windows 7
Windows 8
Windows Server 2008
Windows Server 2008 R2
Linux (via FUSE or non-mainline kernel driver)
Mac OS X 10.6.5 and later

exFAT (Extended File Allocation Table) is a Microsoft file system optimized for flash drives.[3] It is proprietary and patented.[2]

exFAT can be used where the NTFS file system is not a feasible solution (due to data structure overhead), or where the file size limit of the standard FAT32 file system (that is, without FAT32+ extension[4]) is unacceptable.

Although the industry-standard FAT32 file system supports volumes up to 2 TiB, exFAT has been adopted by the SD Card Association as the default file system for SDXC cards larger than 32 GiB.

Overview[edit]

exFAT was first introduced as part of Windows CE,[when?] an embedded Windows operating system. Most of the vendors signing on for licenses of exFAT are either for embedded systems or device manufacturers that produce media that will be preformatted with exFAT. The entire FAT family, exFAT included, is used for embedded systems because it is lightweight and is better suited for solutions that have low memory and low power requirements, and can be implemented in firmware.

exFAT allows individual files larger than 4 GiB, facilitating long continuous recording of HD video which can exceed the 4 GiB limit in less than an hour. Current digital cameras using FAT32 will break the video files into multiple segments of approximately 2 or 4 GiB. With the increase of capacity and the increase of data being transferred, the write operation needs to be made more efficient. SDXC cards, running at UHS-1 have a minimum guaranteed write speed of 10 MBps and exFAT plays a factor in achieving that throughput through the reduction of the file system overhead in cluster allocation. This is achieved through the introduction of a cluster bitmap and elimination (or reduction) of writes to the FAT table. A single bit (flag) in the directory record is used when the file is contiguous to tell the exFAT file driver to ignore the FAT table.

exFAT is also supported in a number of media devices such as modern flat panel TVs,[5] media centers, and portable media players.[6]

Some vendors of flash media, including USB pen drives, compact flash (CF) and solid state drives (SSD) are shipping from the factory some of their high capacity media pre-formatted with the exFAT file system. For example, Sandisk ships their 256 GiB CF cards as exFAT.

Technical specialities[edit]

File name lookup[edit]

Like NTFS and HFS+, exFAT is a proprietary file system. However exFAT is protected under US patent law, and its initial application was issued on 10 July 2009 under the application number US2008168029.[7] On 12 November 2013, the patent was granted by the US patent office under US8583708. Microsoft had also applied for, and so far in some cases received additional US patents on separate components that are used within exFAT. On 27 November 2012, US patent US8321439 was granted for Quick File Name Lookup Using Name Hash, which is an algorithm used in exFAT to speed up file searches.[8] Since Microsoft has not officially released the entire exFAT specification, the Name Hash patent application is a key document in providing internal details in the understanding of the file system, since those details were revealed in Appendix A of the application.

File and cluster pre-allocation[edit]

Like NTFS, exFAT can pre-allocate disk space for a file by just marking arbitrary space on disk as 'allocated'. For each file, exFAT uses two separate 64-bit length fields in the directory: the Valid Data Length (VDL) which indicates the real size of the file, and the physical data length.

To provide improvement in the allocation of cluster storage for a new file, Microsoft incorporated a method to pre-allocate contiguous clusters and bypass the use of updating the FAT table and on 10 December 2013 the US patent office granted patent US8606830.[9] One feature of exFAT (used in the exFAT implementation within embedded systems) provides atomic transactions for the multiple steps of updating the file system metadata. The feature, called Transaction Safe FAT, or TexFAT, was granted a patent by the US patent office under US7613738 on 3 November 2009.[10]

Directory file set[edit]

exFAT and the rest of the FAT family of filesystems does not use indexes for filenames, unlike NTFS that uses B-trees for file searching. When a file is accessed, the directory must be sequentially searched until a match is found. For filenames shorter than 16 characters in length, one filename record is required but the entire file is represented by three 32-byte directory records. This is called a directory file set, and a 256 MiB sub-directory can hold up to 2,796,202 file sets. (If files have longer names, this number will decrease but this is the maximum based on the minimum 3 record file set.) To help improve the sequential searching of the directories (including the root) a hash value of the filename is derived for each file and stored in the directory record. When searching for a file, the file name is first converted to upper case using the upcase table (file names are case insensitive) and then hashed using a proprietary patented algorithm into a 16-bit hash value. Each record in the directory is searched by comparing the hash value. When a match is found, the filenames are compared to ensure that the proper file was located in case of collisions. This improves performance because only two characters have to be compared for each file. This significantly reduces the CPU cycles because most filenames are more than 2 characters in size and each compare is only on two characters until the intended file is located.

Metadata and checksums[edit]

exFAT introduces metadata integrity through the use of checksums. There are three checksums currently in use. The Volume Boot Record (VBR) is a 12 sector region that contains the boot records, BIOS Parameter Block (BPB), OEM parameters and the checksum sector. (There are two VBR type regions, the main VBR and the backup VBR.) The checksum sector is a checksum of the previous 11 sectors, with the exception of three bytes in the boot sector (Flags and percent used). This provides integrity of the VBR by determining if the VBR was modified. The most common cause could be a boot sector virus, but this would also catch any other corruption to the VBR. A second checksum is used for the upcase table. This is a static table and should never change. Any corruption in the table could prevent files from being located because this table is used to convert the filenames to upper case when searching to locate a file. The third checksum is in the directory file sets. Multiple directory records are used to define a single file and this is called a file set. This file set has metadata including the file name, time stamps, attributes, address of first cluster location of the data, file lengths, and the file name. A checksum is taken over the entire file set and a mismatch would occur if the directory file set was accidentally or maliciously changed. When the filesystem is mounted, and the integrity check is conducted, these hashes are verified. Mounting also includes comparison of the version of the exFAT filesystem by the driver to make sure the driver is compatible with the filesystem it is trying to mount, and to make sure that none of the required directory records are missing (for example, the directory record for the upcase table and Allocation Bitmap are required and the filesystem can't run if they are missing). If any of these checks fail, the filesystem should not be mounted, although in certain cases it may mount read-only.

The file system provides extensibility through template based metadata definitions using generic layouts and generic patterns.[11][12]

Restrictive licensing and software patents[edit]

Microsoft has not officially released the complete exFAT file system specification and a restrictive license from Microsoft is required in order to make and distribute exFAT implementations. Microsoft also asserts software patents on exFAT which make it difficult to re-implement its functionality in a compatible way without violating a large percentage of them.[13] This renders the implementation, distribution, and use of exFAT as a part of free or open-source operating systems or of commercial software, for which the vendors could not obtain a license from Microsoft, legally difficult, especially in countries that recognize United States software patents. Although exFAT is now widely supported in consumer electronic devices[citation needed] and Mac OS X, initially these could only handle FAT12/FAT16/FAT32, which formerly rendered exFAT (and flash memory formatted with it) impractical as a universal exchange format. Linux support for exFAT is still limited.[14][15][16][17][18] As of September 2014, a working implementation under Filesystem in User space exists.[19]

Features[edit]

The specifications, features, and requirements of the exFAT file system include:

  • File size limit of 128 PiB–1 (theoretical 16 EiB–1) bytes.[nb 1] (limited by volume size), raised from 4 GiB−1 bytes in a standard FAT32 file system.[1] Therefore, for the typical user, seamless interoperability between Windows and OS X platforms for files in excess of 4 GiB. (The open FAT32+[4] specification proposes an extension how to store files up to 256 GB on otherwise backward-compatible FAT32 volumes as well. This extension is available in some versions of DR-DOS so far, but is trivial to implement in other FAT32 implementations as well.)
  • Scalability to large disk sizes: ca. 128 PiB[20][nb 2] maximum, 512 TiB recommended maximum, raised from the 32-bit limit (2 TiB for a sector size of 512 bytes) of standard FAT32 partitions.[21]
  • Free space allocation and delete performance improved due to introduction of a free space bitmap.
  • Timestamp granularity of 10 ms for Create and Modified times (but not as fine as NTFS's 100 ns).[2]
  • Timestamp granularity for Last Access time to double seconds (FAT was date only)
  • Optional support for UTC timestamps (starting with Vista SP2).[22]
  • Optional support for access control lists (not currently supported in Windows Desktop/Server versions).[23]
  • Optional support for TexFAT, a transactional file system standard (optionally WinCE activated function, not supported in Windows Desktop/Server versions).
  • Support for up to 2,796,202 files per directory only.[2][nb 3] Microsoft documents a limit of 65,534 files per sub-directory for their FAT32 implementation, but other operating systems have no special limit for the number of files in a FAT32 directory. FAT32 implementations in other operating systems allow an unlimited number of files up to the number of available clusters (that is, up to 268,304,373 files on volumes without long filenames).[nb 4]
  • Boundary alignment offset for the FAT table
  • Boundary alignment offset for the data region
  • Provision for OEM-definable parameters to customize the file system for specific device characteristics.[13][24]
  • On disk storage of file Valid Data Length (VDL) through the use of two distinct lengths fields which can be used to pre-allocate a file.
  • Maximum number of files on volume ca. 2^32-11 (up from ca. 2^28-11[nb 4] in standard FAT32)
  • Cluster size up to 32 MiB.[25]
  • Metadata integrity with checksums[clarification needed]
  • Template based metadata structures[clarification needed]
  • Removal of the physical "." and ".." directory entries that appear in subdirectories
  • exFAT does not support short (8.3 format) filenames
  • Windows XP and Windows Vista require certain updates for exFAT support.[26][27][28]
  • Windows Vista is unable to use exFAT drives for ReadyBoost. Windows 7 removes this limitation, enabling ReadyBoost caches larger than 4 GiB.[29]
  • The standard exFAT implementation is not journaled and only uses a single file allocation table and free space map. FAT file systems instead used alternating tables, as this allowed recovery of the file system if the media was ejected during a write (which occurs frequently in practice with removable media). The optional TexFAT component adds support for additional backup tables and maps, but may not be supported.

Adoption[edit]

exFAT is supported in Windows XP and Windows Server 2003 with update KB955704,[1] Windows Embedded CE 6.0, Windows Vista with Service Pack 1,[30] Windows Server 2008,[31] Windows 7, Windows 8, Windows Server 2008 R2 (except Windows Server 2008 Server Core), Mac OS X Snow Leopard starting from 10.6.5,[32] Mac OS X Lion, OS X Mountain Lion, and OS X Mavericks.

Companies can integrate exFAT into a specific group of consumer devices, including cameras, camcorders, and digital photo frames for a flat fee. Mobile phones, PCs, and networks have a different volume pricing model.[3]

Microsoft has entered into licensing agreements with BlackBerry,[33] Panasonic, Sanyo, Sony, Canon, Aspen Avionics,[34] and BMW.[35]

A FUSE-based implementation named fuse-exfat, or exfat-fuse, with read/write support is available for FreeBSD and multiple Linux distributions.[36][37][38][39] A kernel implementation has also been released, written by Samsung.[40] It was initially released on GitHub unintentionally,[41] and later released officially by Samsung in compliance with the GPL.[42] None of the solutions can become an official part of Linux due to the patent encumbered status of the exFAT filesystem.[citation needed]

Proprietary read/write solutions licensed and derived from the Microsoft exFAT implementation are available for Android,[43] Linux, and other operating systems from Paragon Software Group and Tuxera.

XCFiles (from Datalight) is a proprietary, full-featured implementation, intended to be portable to 32-bit systems.[44] Rtfs (from EBS Embedded Software) is a full-featured implementation for embedded devices.[45]

Two experimental, unofficial solutions are available for DOS. The loadable USBEXFAT driver requires Panasonic's USB stack for DOS and only works with USB storage devices; the open-source EXFAT executable is an exFAT filesystem reader, and requires the HX DOS extender to work.[46] There are no native exFAT real-mode DOS drivers, which would allow usage of, or booting from exFAT volumes.

Mac OS X Snow Leopard 10.6.5 and later can create, read, write, verify, and repair exFAT file systems.[47]

See also[edit]

Notes[edit]

  1. ^ a b Although Microsoft published a different value in KB955704, the file size is in bytes and is stored as a 64-bit number. The largest theoretical file size would be 16 EiB−1 byte, the same as in NTFS. However, since the true theoretical maximum volume size under the current specification cannot exceed 128 PiB, a file can never reach that file length. Another reason for the limit is that current IDE/ATA disk addressing is LBA-48, and uses a 48 bit block address to address a sector. A 512 byte sector size is represented by 2^9, which makes the maximum addressable file system 2^57 (48+9, you add the exponents) - which is 128 PiB. In other words, the 128 PiB limit on the architecture is a hardware restriction. This scenario does not include AF (4k sectors) and exFAT is limited to 128 PiB regardless of sector size based on the specification.
  2. ^ The theoretical maximum volume size is defined by up to 2^32-11 possible clusters with up to 2^25-1 bytes per cluster = ca. 128 PiB. The size is currently also limited by the LBA48 addressing scheme, as with a 512 byte sector size, only 2^48 * 512 = 2^57 bytes = 128 PiB can be addressed.
  3. ^ This limit applies because the maximum directory size is 256 MiB.
  4. ^ a b 268,304,373 files = 2^28 - 11 reserved clusters - 131,072, the minimum number of 64 KB clusters occupied for the 268,435,445 directory entries (á 32 bytes) without VFAT LFNs, which are required for 268,435,445 files with sizes between 1 and 65535 bytes. With VFATs, the 131,072 number must be multiplied by 21 (worst case), which would result in 265,682,933 files instead.

References[edit]

  1. ^ a b c d e "KB955704". 2009-01-27. "Description of the exFAT file system driver update package [for Windows XP and Windows Server 2003]" 
  2. ^ a b c d US 8321439  contains Microsoft exFAT specification (revision 1.00)
  3. ^ a b Marius Oiaga (2009-12-11). "Microsoft Licenses Windows 7's exFAT Flash File Format". Softpedia.com. 
  4. ^ a b Davis, Jeremy; Kuhnt, Udo; Georgiev, Luchezar (2007). "FATPLUS.TXT". FAT+. Draft revision 2. Retrieved 2013-11-25. 
  5. ^ "exFAT support on Sony". Retrieved 2013-01-06. 
  6. ^ Hamm, Jeff (2009). "Extended FAT File System". Paradigm Solutions. Retrieved 2013-04-09. 
  7. ^ US Patent Application 2008168029 , "Extensible File System";
  8. ^ US 8321439 , "Quick File Name Lookup Using Name Hash'"
  9. ^ US 8606830 , "Contiguous File Allocation in an Extensible File System'"
  10. ^ US 7613738 , "FAT Directory Structure for use in Transaction Safe File System'"
  11. ^ US Patent Application 2009164440 , "Quick Filename Lookup Using Name Hash"; Microsoft Corp; contains exFAT specification revision 1.00. See Tables 8, 12, 13, 14, 15, 16, & 17
  12. ^ US Patent Application 2008168029 , "Extensible File System"; States in Abstract and elsewhere that directory records can be custom defined.
  13. ^ a b "exFAT File System Licensing Program". Retrieved 2011-01-04. 
  14. ^ http://askubuntu.com/questions/370398/how-to-get-exfat-working-in-ubuntu-13-10
  15. ^ http://www.phoronix.com/scan.php?page=news_item&px=MTQ2ODc
  16. ^ http://phoronix.com/forums/showthread.php?81642-Native-Linux-Kernel-Module-Is-Out-For-Microsoft-exFAT
  17. ^ http://news.softpedia.com/news/Native-exFAT-Support-Finally-Arrives-on-Linux-365770.shtml
  18. ^ http://linuxg.net/native-support-for-exfat-filesystems-on-linux-will-be-added-soon/
  19. ^ https://code.google.com/p/exfat/
  20. ^ File System Functionality Comparison
  21. ^ "Limitations of the FAT32 File System in Windows XP". Microsoft. 2007-12-01. Retrieved 2011-02-11. 
  22. ^ Nash, Mike (2008-10-24). "Windows Vista Service Pack 2 Beta". The Windows Blog. Retrieved 2009-10-02. 
  23. ^ "A Second Shot: Windows Vista SP1". Retrieved 2013-11-05. 
  24. ^ "OEM Parameter Definition with exFAT (Windows Embedded CE 6.0)". Retrieved 2013-11-25. 
  25. ^ Upper limit of cluster size of exFAT file system.
  26. ^ Support for exFAT under Windows XP.
  27. ^ Information about support for exFAT under Windows Vista.
  28. ^ Windows Vista service pack 1 to support exFAT file system.
  29. ^ "exFAT Versus FAT32 Versus NTFS". 2008-02-27. Retrieved 2009-10-02. 
  30. ^ Brandon LeBlanc (2007-08-28). "Vista SP1 Whitepaper". Microsoft. Retrieved 2007-08-28. 
  31. ^ "Adding Hard Disk Drives". Microsoft. Retrieved 2009-09-15. 
  32. ^ "Mac OS X 10.6.5 Notes: exFAT Support, AirPrint, Flash Player Vulnerability Fixes". Retrieved 2013-11-25. [unreliable source?]
  33. ^ "Microsoft Licenses exFat to Research In Motion". 
  34. ^ "Microsoft Signs Patent Licensing Agreement With Aspen Avionics". 
  35. ^ "Microsoft Signs exFAT Licensing Agreement With BMW". 
  36. ^ "exFAT fs and Linux". Retrieved 2009-09-28. [unreliable source?]
  37. ^ "exFAT fs on FUSE". Retrieved 2010-05-15. 
  38. ^ "exFAT fs on linux UBUNTU". Retrieved 2012-10-11. 
  39. ^ "exFAT in FreeBSD". Retrieved 2013-04-08. 
  40. ^ "Open Source Release Center". Samsung. Retrieved 2013-11-25. 
  41. ^ Corbet, Jonathan (2013-07-24). "The exfiltrated exFAT driver". LWN.net. Retrieved 2013-11-25. 
  42. ^ "Conservancy Helps Samsung Resolve GPL Compliance Matter Amicably". Software Freedom Conservancy. 2013-08-16. Retrieved 2013-11-25. 
  43. ^ Clarke, Gavin (2012-08-08). "Sharp cuts exFAT deal with Microsoft for Android mobes". The Register. Retrieved 2012-08-12. 
  44. ^ "XCFiles". Retrieved 2010-06-21. 
  45. ^ "Rtfs". Retrieved 2011-01-19. 
  46. ^ "exFAT". 2011-02-02. 
  47. ^ "fsck_exfat(8) Mac OS X Manual Page". Retrieved 2013-11-25. 

External links[edit]