Talk:File Allocation Table

fragmentation - flush to disk

Removed this paragraph

MS-DOS also did not offer a system call which would allow applications to make sure a particular file has been completely written to disk in the presence of deferred writes (cf. fsync in Unix or DosBufReset in OS/2). Disk caches on MS-DOS were operating on disk block level and were not aware of higher-level structures of the file system. In this situation, cheating with regard to the real progress of a disk operation was most dangerous.

I agree with the comment elsewhere that the FAT article is not the place for a discussion of DOS.

But I removed this because in the presence of a MS DOS cache (which allowed defered writes), the 21-68 system call (commit file) would flush the cache.

This behaviour was different on Windows 98, but Win 98 did not use DOS for the file system. —Preceding unsigned comment added by 218.214.18.240 (talk) 05:18, 18 September 2010 (UTC)

Need a table or chart of Fat version, max partition size, cluster size

Article seems to need a a table or chart of Fat version, max partition size, cluster size. —Preceding unsigned comment added by 71.131.14.212 (talk) 16:18, 14 October 2010 (UTC)

Done. Took me some hours, because I didn't know that FAT12 hex. FF6 + FAT16 hex. FFF6 are not the maximal cluster numbers, but reserved. Now it's your turn, transform the stupid preformatted size limits into a proper wikitable... :-) –89.204.137.161 (talk) 01:55, 4 July 2011 (UTC)

At some point in time the article should settle on the arguably correct FF5 / FFF5 / 0FFF FFF5 limits instead of the more obscure FF0 / FFF0 / 0FFF FFF0 limits for FAT12 / FAT16 / FAT32, respectively. IIRC the ...0 limits existed for Amiga (or something), so that is no patent nonsense. But nevertheless it can't be "correct", otherwise there would be "impossible" maximal cluster numbers FF1..FF5 / FFF1..FFF5 / 0FFF FFF1 .. 0FFF FFF5 not discussed in the standards. –82.113.99.129 (talk) 15:03, 14 July 2011 (UTC)

A less known feature of the file attribute byte

The table in section: http://en.wikipedia.org/wiki/File_Allocation_Table#Directory_table states about the attribute byte at offeset 0x0b in the directory entry about bit 6, mask 0x40: 'Device (internal use only, never found on disk)'. Well, it can be found on some disks, especially when the were formatted by me :-) I have found out that when Windows (NT or newer) finds a file with this attribute bit set, it completely denies access to this file -- reading, writing, executing, deletion or attribute change is completely prohibited by the kernel.

I think this is a very useful feature: all my memory sticks and FAT formatted external drives contain an empty file named 'autorun.inf' with this attribute set in the root directory. Some Windows systems are configured to run the autorun.inf file without even asking the user, which is a perfect way to infect the system with malware.

Maybe this behaviour should be mentioned somewhere in this article? -- 153.97.93.25 (talk) 12:14, 17 December 2010 (UTC)

If somebody has a reference for this feature, e.g., if Ralf Brown's Interrupt List mentions it, please add it. –82.113.99.129 (talk) 14:51, 14 July 2011 (UTC)

FAT32 - DOS scandisk utility

Under section 1.6 FAT32 the following should be completely removed from the article:

Additionally, there are numerous reports that the DOS scandisk utility provided with Windows 98 (scandisk.exe) and even Windows 98 itself can in fact operate on FAT32 volumes with cluster counts much higher than the 4.177 million claimed by Microsoft, some of which have been in the 40 to 120 million range, thereby indicating that the scandisk utility can likely operate on volumes up to the upper limit of the FAT32 specifications.[18]

Reason

There are not numerous reports about this, this is a completely unsubstantiated claim made by one anonymous individual on a Usenet group. The individual has not presented his credentials and his test methodology or any verifiable reproducible test results. His proof lies in the simple claim that "I ran scandisk and it started and ran..." however he does not know what, if anything, that scandisk did and he doesn't know if scandisk examined all the files and clusters on the disk. The scandisk utility supplied with Windows 98 is a 16-bit application and due to immutable 16-bit memory block limits it simply cannot properly load the FAT to do a proper check if it contains more than 4,177,920 clusters, this is explained as such by Microsoft:

[Quote]

The ScanDisk tool included with Microsoft Windows 95 and Microsoft Windows 98 is a 16-bit program. Such programs have a single memory block maximum allocation size of 16 MB less 64 KB. Therefore, The Windows 95 or Windows 98 ScanDisk tool cannot process volumes using the FAT32 file system that have a FAT larger than 16 MB less 64 KB in size. A FAT entry on a volume using the FAT32 file system uses 4 bytes, so ScanDisk cannot process the FAT on a volume using the FAT32 file system that defines more than 4,177,920 clusters (including the two reserved clusters).

[end quote] http://support.microsoft.com/kb/184006

The poster's conclusion is that "...the scandisk utility can likely operate on volumes up to the upper limit of the FAT32 specifications...". In all likeliness the tests performed by the Usenet poster probably only loaded a truncated portion of the FAT or it may only have appeared to run properly while leaving a large portion of the disk unchecked. Without a verifiable test method the information in the post is specious at best. As a matter of fact, if one digs a bit in the same Usenet archive one will find that the claims of this particular poster have been questioned on several occasions, for example: ( http://groups.google.com/group/microsoft.public.win98.gen_discussion/browse_thread/thread/7bd36a399830299f/c5d72eb3d733d761?#c5d72eb3d733d761 ) and that the poster could not ever satisfactorily answer to his critics, this leaves me wondering as to why this information was ever inserted in the article and whom actually inserted this unproven information into the Wikipedia article to begin with.

Should we believe and trust the information provided by Microsoft or should we embark on a slippery slope and trust information made by anonymous unverifiable Usenet posters instead? It doesn't seem to me that Usenet posts made by single anonymous persons and containing no solidly verifiable source or citation would meet the high standards of accuracy required to make it in a Wikipedia article, these kind of unverified or unverifiable Usenet posts are highly dubious sources for any organization that strives to present accurate information to its readers. For these reasons I move that the information be stricken from the article.

MidnightTwo (talk) 00:12, 17 June 2011 (UTC)

If you find errors please just be bold and fix them. It's a wiki, undo/revert are simple. –82.113.99.129 (talk) 15:13, 14 July 2011 (UTC)

FAT32 - Improper terminology (and more factual errors!)

Section 1.6 contains the following:

"All versions of Windows prior to XP-SP1 and 2000-SP4 lacked inherent support for 48-bit LBA drive access because of a limitation in their 32-bit protected-mode IDE driver, ... "

Protected Mode Driver is terminology generally applied to hybrid Windows 9x operating systems, Windows NT operating systems are pure 32-bit operating systems and Protected Mode Driver is a redundant term when applied to these operating systems, the term is generally never used in the context of Windows NT operating system, on these systems they are simply drivers. This incorrect terminology also comes from the anonymous Usenet post claiming that scandisk runs properly on disks with more than 4,177,920 clusters.

Paragraphs 2, 3, 4 & 5 need to be removed or substantially pruned as they contain unproven or inaccurate and misleading information:

On Windows 95/98, due to the version of Microsoft's DOS-mode SCANDISK utility included with these operating systems being a 16-bit application, the FAT structure is not allowed to grow beyond 4 177 920 (< 222) clusters, placing the volume limit at 127.5 GiB (≈137 GB).[16][17] The FAT32 drive formatting tools provided by Microsoft (fdisk and format) are thus designed to scale the cluster size upwards as the volume size increases, thereby preventing the cluster-count from exceeding 4.177 million clusters - and only reaching that number on volumes with a size of 128 GB with 32 kB cluster size. Most or all appraisals of the efficiency of the FAT32 file system are rooted in this behavior, as they point out that FAT32 increases the cluster size with the volume size and therefore small file storage can become very inefficient when larger cluster sizes are used (eg - 32 kB). This behavior, however, is unnecessary in many cases. Large FAT32 volumes can in fact be created with smaller cluster sizes (eg 4 kB) given the use of alternate drive preparation tools.

Additionally, there are numerous reports that the DOS scandisk utility provided with Windows 98 (scandisk.exe) and even Windows 98 itself can in fact operate on FAT32 volumes with cluster counts much higher than the 4.177 million claimed by Microsoft, some of which have been in the 40 to 120 million range, thereby indicating that the scandisk utility can likely operate on volumes up to the upper limit of the FAT32 specifications.[18] A limitation in original versions of Windows 98/98SE's Fdisk utility causes it to incorrectly report disk sizes over 64 GiB.[19] A corrected version is available from Microsoft, but it cannot partition drives larger than 512 GiB (≈550 GB).[20]

Windows 98 (and presumably Windows ME) has been shown to be able to run from and correctly operate with volumes exceeding 128 GB (up to 1.5 TB in some reports) as well as with FAT32 volumes with 40 to 120 million clusters, although the native drive maintenance tools (scandskw.exe, defrag) have upper limits that are not yet known - but are reported to exceed 20 million clusters when using Windows ME versions of these tools. The installation program and filesystem creation tool supplied with Windows 2000, XP, and later imposes a limitation of 32 GiB.[21] However, these systems can read and write to FAT32 file systems of any size. This limitation is by design and according to Microsoft was imposed because many tasks on a very large FAT32 file system become slow and inefficient.[16][22] This limitation can be bypassed by using third-party formatting utilities or by using the FORMAT command with the /FS:FAT32 switch from the command line.[23]

All versions of Windows prior to XP-SP1 and 2000-SP4 lacked inherent support for 48-bit LBA drive access because of a limitation in their 32-bit protected-mode IDE driver, meaning that the maximum disk size for (parallel) ATA disks is 128 GiB (≈137 GB),[24] without using alternate drivers. Windows XP became fully 48-bit LBA capable with SP1 in 2002, but Microsoft did not release a patch for the 32-bit protected-mode drivers for Windows 98/ME (ESDI_506.PDR) even though those OS's were still being fully supported by Microsoft at that time. Intel did release an alternate IDE driver for win-9x/me (Intel Application Accelerator) that provides full 48-bit LBA operability for the 800-series chipsets, and several individuals and user groups have modified Microsoft's EDSI_506.PDR driver to make it 48-bit LBA compliant for Windows 98. Most or all third-party drivers for SATA controllers are 48-bit LBA compliant, even when used under Windows 98/ME. All versions of DOS that are FAT-32 aware are also 48-bit LBA capable, so long as 48-bit LBA is supported by the underlying hardware (ie - motherboard / BIOS).

In particular this statement in paragraph 2:

The FAT32 drive formatting tools provided by Microsoft (fdisk and format) are thus designed to scale the cluster size upwards as the volume size increases, thereby preventing the cluster-count from exceeding 4.177 million clusters - and only reaching that number on volumes with a size of 128 GB with 32 kB cluster size. Most or all appraisals of the efficiency of the FAT32 file system are rooted in this behavior, as they point out that FAT32 increases the cluster size with the volume size and therefore small file storage can become very inefficient when larger cluster sizes are used (eg - 32 kB). This behavior, however, is unnecessary in many cases. Large FAT32 volumes can in fact be created with smaller cluster sizes (eg 4 kB) given the use of alternate drive preparation tools.

This is grossly inaccurate. While it is true that larger clusters lead to less efficient storage space the main inefficiency on large FAT32 volumes does not come from the larger cluster size but rather it comes from the large cluster count and the ridiculously large FAT structure required to keep track of it all, as explained by Raymond Chen:

[Quote]

FAT used linear searching and threaded file allocation information in a linked list. All these linear-time data structures were acceptable when the FAT file system was developed—at a time when floppy drives were the primary storage device. Today, however, multi-gigabyte drives are the norm and these linear-time algorithms fall apart due to the large amount of disk access required to carry out a single operation. Furthermore, the high cluster count means a ridiculously large file allocation table; as a result, even the simple act of computing how much free disk space is available can take over a minute. That’s why the first time you type the dir command, there is a long pause at the end of the directory listing: the file system is busy calculating the disk free space to display at the bottom of the directory listing.

[end quote]

This is also touched on here: http://www.pcguide.com/ref/hdd/file/partFAT32-c.html

This FAT size affect performance because of the linear manner in which the FAT is accessed and processed whereas the high count of smaller 4K cluster isn't a concern on the NTFS MFT B-tree structure.

The paragraphs in question also demonstrate a poor technical writing style which is not in keeping with Wikipedia's high standards, for example:

All versions of Windows prior to XP-SP1 and 2000-SP4...

A more appropriate construction would be:

All Windows versions prior to Windows 2000 SP4 and Windows XP SP1... The writing style in the section pales when compared to other similar Wikipedia articles such as http://en.wikipedia.org/wiki/Windows_2000 and http://en.wikipedia.org/wiki/Windows_XP

And of course it goes without saying that the information in the above statement is also incorrect! Support for 48-bit LBA became available for Windows 2000 with Service Pack 3, not SP4!

In another paragraph the writer states:

"Windows 98 (and presumably Windows ME) has been shown..."

Another glaring example of writing or information which is not in keeping with Wikipedia's high standards. Technically correct information doesn't run on "presumptions", it runs on accurate and verifiable information, none of which is present in Paragraphs 2, 3, 4 & 5, .

Much of the information in section 1.6 was edited and added by a single person and includes information from citation 18, all of the information from this specious source needs to be revisited and confirmed by knowledgeable persons or it need to be stricken from the article. On second thought, all of it needs to be removed, period! It is all bogus and without supporting citations, a complete fabrication from a single anonymous Usenet poster!

MidnightTwo (talk) 01:39, 17 June 2011 (UTC)

Removed factually incorrect self published information on June 22, 2011 ~~ (broken signature apparently for MidnightTwo)

JFTR, protected mode is proper Intel or x86 (above 80186) or OS/2 terminology, but as you said it makes no sense for NT drivers, as there are no real mode or VxD drivers for NT — not counting NTVDM 16bit character mode DOS drivers. Writing "presumably ME" is sloppy, but features introduced in win 98 in fact presumably also exist in win ME. Not talking about it might be better than using a weasel word, but it's no evidence for nonsense. Likewise "2000-SP4" or "XP-SP1" is not the best style, but otherwise clear, and by itself no evidence for nonsense. If there are any FAT32 limits below 2 TB I'm interested in verified technical details. –82.113.99.129 (talk) 16:06, 14 July 2011 (UTC)

FATX

Please merge the FATX stub into the corresponding subsection here, a "fresher" epoch instead of 1980-01-01 does not justify a separate article. –89.204.137.133 (talk) 21:47, 17 July 2011 (UTC)

Done. Two references (Linux Xbox project) resulted in a server error today, please check this. –89.204.152.53 (talk) 03:44, 16 August 2011 (UTC)

I'm not sure it was a good idea to merge the articles, since FATX (FATX16 and FATX32) are different enough from FAT16 and and FAT32 to make it impossible to sufficiently describe them in this article. The "basic ideas" are the same, but most of the data structures are different, so FATX should really have it's own article in order not to confuse people (and also to avoid confusion with FAT16X and FAT32X, the official names for FAT16 and FAT32 partition IDs to enforce usage of LBA instead of CHS or mixed CHS/LBA access). I think it is okay to have a short abstract in the FAT article (as we have right now for FATX and exFAT), but better details should be discussed in separate articles. --Matthiaspaul (talk) 19:32, 9 January 2012 (UTC)

Technical

An assessment of this article was requested. I have tagged the article as being too technical for most readers to access. I have started to move some of the technical minutia to the new Notes section. I have lowered the assessment from B to C. --Kvng (talk) 22:23, 5 December 2011 (UTC)

This passage is a particular offender: The name originates from the usage of a table which centralizes the information about which areas belong to files, are free or possibly unusable, and where each file is stored on the disk. To limit the size of the table, disk space is allocated to files in contiguous groups of hardware sectors called clusters.

Not only is this extremely vague if you don't already understand what file systems do, it's flat wrong: the whole point of the linked allocation scheme used in FAT is that the clusters allocated to a file do not have to be (and often aren't) contiguous!

I am working on a substantial rewrite. I agree the article can be made far more layman accessible and much of the technical information can be displaced to a notes section.

Valency (talk) 23:28, 10 December 2011 (UTC)

Actually, the sectors within a cluster are contiguous -- though the clusters within a file might not be... AnonMoos (talk) 16:31, 11 December 2011 (UTC)

Well, that's another issue: it confuses sectors with clusters! A cluster is a kind of allocation unit, but a sector in this context usually means the smallest addressable storage unit on a disk (or other storage medium), whereas a "cluster" in FAT terminology is the smallest addressable storage unit within the file system, and may be composed of several (yes, contiguous) hardware sectors. As it is, the reader is told a cluster is a kind of sector, itself composed of sectors, and that it's a "hardware sector", when in fact it's sectors that are hardware sectors, clusters are are software (file system) unit, which are called "clusters" to avoid confusion with hardware sectors, and each cluster is made up of several hardware sectors... you can see the problem!

Still working on that revision. -- Valency (talk) 06:56, 12 December 2011 (UTC)

We shouldn't use the term "hardware sectors" (whatever that may be) here. In the context of many PC operating systems, the allocatable units below the file system driver, as offered by the disk device driver, are so called logical sectors. The allocatable units as they are used by the driver working on the system BIOS INT 13h CHS or LBA interface or directly on the disk I/O hardware interface are called physical sectors. File systems don't normally deal with physical sectors. The device driver translates and optionally combines physical sectors into logical sectors for the file system to use. Logical sectors can have the same size as physical sectors, but can also be larger. In the case of a volume on a non-partitioned medium, accessed via LBA, (a so called "super-floppy") the logical and physical sector numbers can be the same (except for nomenclature that physical CHS sector numbers at BIOS INT 13h level start with sector 1, whereas physical LBA sectors at BIOS INT 13h level as well as logical sector numbers start with number 0), otherwise they differ.

I assume by "hardware sector" the original author meant physical sectors, but perhaps he even meant the "on-medium" sectors inside the drive which are typically somewhat larger (for parity/ECC info, etc.) and may be organized quite differently internally.

For FAT volumes, logical sectors may have sizes of 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192 or 16384 bytes. Some broken file system implementations can only deal with logical sectors of 512 bytes, though. On PC compatible machines, logical sector sizes below 512 bytes are seldomly used. 128 byte physical sectors could be found on some archaic floppy disks, and on PC compatible machines they are typically only found on small memory drives. Even on memory drives, logically sector sizes below 128 bytes are extremely rare (I have seen them only in lab conditions). Logical sector sizes larger than 512 bytes have once been very important to overcome the volume size limits before the introduction of FAT16B with DOS 3.31. There have been various OEM DOS 2.x/3.x versions, which worked with so called logical sectored FATs. On these systems, the physical sector size was still 512 bytes, but the device driver combined them into larger logical sectors for the file system driver to work with. The downside of this approach has been significantly increased memory consumption for the sector buffering inside the device driver (however, this is no issue for operating systems which are not bound to run in real mode any more).

Larger physical sector sizes than 512 bytes have been used on various optical drives (however, most of the time not formatted with a FAT file system), but they have become important again in recent years, as physical sector sizes are increasing beyond 512 bytes (at present up to 4096 bytes). --Matthiaspaul (talk) 11:19, 12 December 2011 (UTC)

I have reworked the offending section to hopefully give a better idea of why the filesystem is named FAT and how it works in general, without going into implementation specific details. --Matthiaspaul (talk) 20:46, 11 December 2011 (UTC)

Thanks for your effort, this is much better in terms of accuracy, but now I think it's too much of an infodump to go at the top of the page. In my draft I just say:

The FAT file system derives its name from its central structure, the File Allocation table, which uses linked allocation to keep track of the storage space allocated to each file.

and leave the technical details for later. I'm new at editing, and I'm not quite ready to work the live article yet, but I would move this at the least. --Valency (talk) 07:07, 12 December 2011 (UTC)

Hm, but your description would apply to most any file system, don't you think? I find it quite important to describe what is "special" about FAT compared to other file systems.

While it is always good to make something easier accessible to anyone, a layman will ever only get a very basic understanding of what it is and how it works, unless he really dedicates time and energy to actually understand the inner workings and over time gets an idea of the various implications. Risking that not everyone will be able to understand everything, some degree of technical detail will be unavoidable in an article about a file system, I think.

In fact, I find the article in its current form rather superficial with lots of things presented inaccurately and some important details missing. This article may give enough information to help someone find a lost file on a FAT volume on a mainstream PC, however, it does not discuss essentails for someone who would want to implement a FAT repair tool or a FAT driver, so that it becomes fully compatible with all existing FAT volumes in the wild, I'm afraid. --Matthiaspaul (talk) 12:40, 12 December 2011 (UTC)

It would be nice to step back...way back...and give a bit of an overview of the consequences of the FAT file system. Files on disk grow in cluster-size chunks, so you want small clusters. You only have 12, 16, or 32 bits in each FAT entry, so you want big clusters for big disk drives. Clusters can be anywhere on the disk, so after a while the "cluster chain" weaves all over the disk surface, causing the notorious file fragmentation (hands up, everyone who ever ran defrag utilities? Keep your hands up if defrag ran overnight?). It's a singly-linked list, so if you lose one cluster in the chain, you lose the rest of the file. If you zero out a directory entry and lose the pointer to the first cluster of a file, that file space gets orphaned and can't be reused until some utility program notices that there's a chain with no corresponding directory entry - the so called "lost clusters" which CHKDSK would try to turn into files for you. You erase a file by setting all its FAT entries to 0, marking them as free space...even though the data is still in the sectors, you've destroyed the mapping into a file. This is why you can still read data on an erased disk. There's FAT in memory and FAT on the disk and woe betide you if you pull the disk out before its copy of the FAT has been synchronized with the copy in memory. Etc. Let's get the lay of the forest before we start reciting the names of the caterpillars on the leaves of the trees. We all too quickly get into details of Binford 6100 DOS and its all-Roman-numeral file naming convention...

We also have very long and detailed technical comments hidden in commented-out text inside the infobox. If this is important, it should be in a table or in the main text somewhere and visible - and referenced. --Wtshymanski (talk) 05:25, 7 January 2012 (UTC)

Date resolution

Recently revisions to the infobox content have "date_resolution = 2 seconds (10 ms with VFAT)". I thought granularity of access timestamp was 1 day and of modification timestamp was 2s, even in recent formats, and I cannot find any smaller-granularity fields in the revised tables in File Allocation Table#Directory table. The 10ms VFAT refinement seems to apply only to creation timestamp. Should the infobox entry be changed?

If so, then I would also suggest that finer modification timestamp granularity (and also recording of UTC timezone offsets) are two advantages of exFAT over FAT32, contrary to the statement that in File Allocation Table#exFAT that exFAT "offers no practical benefits over FAT32".

I also thought that exFAT has cluster alignment characteristics better suited to the large internal storage block sizes of solid-state drives.

— Richardguk (talk) 06:25, 23 December 2011 (UTC)

I noticed that too, but was waiting for the flurry of article updates to subside before looking in detail. You summarize the time precision I last found in the references and have implemented in my FAT drivers. exFAT, in my opinion, does not belong here. —EncMstr (talk) 06:49, 23 December 2011 (UTC)

That's right. Thanks for spotting this. 10ms resolution is only for creation time, which is an optional part of the specs. Last modified time resolution is still 2s, access date is still dates only.

Regarding exFAT, I agree, except for a small intro it does not belong into this FAT article, because exFAT is considerably different in many ways. It really is a filesystem of its own, and already has its own article, anyway. --Matthiaspaul (talk) 09:47, 23 December 2011 (UTC)

Thanks for the feedback and for updating both articles. — Richardguk (talk) 12:37, 23 December 2011 (UTC)

Length

This is now a book-length article. Isn't it time to move all the fascinating details to a Wikibook and make this into an encyclopedia overview? --Wtshymanski (talk) 14:21, 12 January 2012 (UTC)

I actually came to this page to find these "fascinating details". James Murray 80.176.88.36 (talk) 12:15, 16 January 2012 (UTC)