Sparse file
From Wikipedia, the free encyclopedia
.svg)
In computer science, a sparse file is a type of computer file that attempts to use file system space more efficiently when blocks allocated to the file are mostly empty. This is achieved by writing brief information (metadata) representing the empty blocks to disk instead of the actual "empty" space which makes up the block, using less disk space. The full block size is written to disk as the actual size only when the block contains "real" (non-empty) data.
When reading sparse files, the file system transparently converts metadata representing empty blocks into "real" blocks filled with zero bytes at runtime. The application is unaware of this conversion.
Most modern file systems support sparse files, including most Unix variants and NTFS[1], but notably not Apple's HFS+. Sparse files are commonly used for disk images, database snapshots, log files and in scientific applications.
Contents |
[edit] Creating sparse files in Unix
If executed manually, the Unix command:
dd if=/dev/zero of=sparse-file bs=1 count=0 seek=5M
Will create a file of five megabytes in size, but with no data stored on disk (only metadata).
[edit] Detecting sparse files in Unix
Sparse files have different apparent and actual file sizes. This can be detected by comparing the output of:
du -s -B1 --apparent-size sparse-file
and:
du -s -B1 sparse-file
[edit] Copying sparse files in Unix
Normally, the GNU version of cp is good at detecting whether a file is sparse, so it suffices to run:
cp sparse-file new-file
and new-file will be sparse. However, GNU cp does have a --sparse=WHEN option[2]. This is especially useful if a sparse-file has somehow become non-sparse (i.e. the empty blocks have been written out to disk in full). Disk space can be recovered by doing:
cp --sparse=always formerly-sparse-file recovered-sparse-file
It should be noted that not all cp implementations support the --sparse option and will in all cases expand sparse files, like e.g. FreeBSD's cp. A viable alternative on those systems is to use rsync with its own --sparse option[3] instead of cp.
[edit] Advantages
The advantage of sparse files is that storage is only allocated when actually needed: disk space is saved, and large files can be created even if there is insufficient free space on the file system.
[edit] Disadvantages
Disadvantages are that sparse files may become fragmented; file system free space reports may be misleading; filling up file systems containing sparse files can have unexpected effects; and copying a sparse file with a program that does not explicitly support them may copy the entire, uncompressed size of the file, including the sparse, mostly zero sections which are not on disk—losing the benefits of the sparse property in the file.
[edit] See also
[edit] References
- ^ Giampaolo, Dominic (1999). Practical File System Design with the Be File System. Morgan Kaufmann. ISBN 1-55860-497-9
- ^ GNU cp: accept new option to control creation of sparse files
- ^ rsync: handle sparse files efficiently
