glob (programming)

From Wikipedia, the free encyclopedia
Jump to navigation Jump to search

In computer programming, glob patterns specify sets of filenames with wildcard characters. For example, the Unix Bash shell command mv *.txt textfiles/ moves (mv) all files with names ending in .txt from the current directory to the directory textfiles. Here, * is a wildcard standing for "any string of characters" and *.txt is a glob pattern. The other common wildcard is the question mark (?), which stands for one character.

In addition to matching filenames, globs are also used widely for matching arbitrary strings. In this capacity a common interface is fnmatch.

Origin[edit]

A screenshot of the original 1971 Unix reference page for glob – note the owner is dmr, short for Dennis Ritchie.

The glob command, short for global, originates in the earliest versions of Bell Labs' Unix.[1] The command interpreters of the early versions of Unix (1st through 6th Editions, 1969–1975) relied on a separate program to expand wildcard characters in unquoted arguments to a command: /etc/glob. That program performed the expansion and supplied the expanded list of file paths to the command for execution.

Later, this functionality was provided as a C library function, glob(), used by programs such as the shell. It is usually defined based on a fnmatch() function, which tests for whether a string matches a given pattern. Both functions are a part of POSIX: the functions defined in POSIX.1 since 2001, and the syntax defined in POSIX.2.[2][3] The idea of defining a separate match function started with wildmat (wildcat match), a simple library to match strings against Bourne Shell globs.

Traditionally globs do not match hidden files or directories in the form of Unix dotfiles, so that * only matches any non-hidden file. To match hidden files, one must explicitly specify the dot prefix in the form of .*.

Syntax[edit]

The most common wildcards are *, ?, and […].

Wildcard Description Example Matches Does not match
* matches any number of any characters including none Law* Law, Laws, or Lawyer GrokLaw, La, or aw
*Law* Law, GrokLaw, or Lawyer. La, or aw
? matches any single character ?at Cat, cat, Bat or bat at
[abc] matches one character given in the bracket [CB]at Cat or Bat cat or bat
[a-z] matches one character from the (locale-dependent) range given in the bracket Letter[0-9] Letter0, Letter1, Letter2 up to Letter9 Letters, Letter or Letter10

In all cases the path separator character (/ on Unix or \ on Windows) will never be matched.

Unix-like[edit]

On Unix-like systems *, ? is defined as above while […] has two additional meanings:[4][5]

Wildcard Description Example Matches Does not match
[!abc] matches one character that is not given in the bracket [!C]at Bat, bat, or cat Cat
[!a-z] matches one character that is not from the range given in the bracket Letter[!3-5] Letter1, Letter2, Letter6 up to Letter9 and Letterx etc. Letter3, Letter4, Letter5 or Letterxx

The ranges are also allowed to include pre-defined character classes, euivalence classes for accented characters, and collation symbols for hard-to-type characters. They are defined to match up with the brackets in POSIX regular expressions.[4][5]

Unix globbing is handled by the shell per POSIX tradition. Globbing is provided on filenames at the command line and in shell scripts.[6] The POSIX-mandated case statement in shells provides pattern-matching using glob patterns.

Some shells (such as the C shell and Bash) support additional syntax known as alternation or brace expansion. Because it is not part of the glob syntax, it is not provided in case. It is only expanded on the command line before globbing.

The Bash shell also supports the following extensions:[7]

  • Extended Globbing (extglob): allows other pattern matching operators to be used to match multiple occurrences of a pattern enclosed in parentheses, essentially providing the missing kleene star and alternation for describing regular languages. It can be enabled by setting the extglob shell option. This option came from ksh93.[8] The GNU fnmatch and glob has an identical extension.[2]
  • globstar: allows ** on its own as a name component to recursively match any number of layers of non-hidden directories.[8] Also supported by the JS libraries and Python's glob.

Windows and DOS[edit]

The dir command with a glob pattern in IBM PC DOS

Windows shells, following DOS, do not traditionally perform any glob expansion in arguments passed to external programs. Shells may use an expansion for its own builtins:

  • Windows PowerShell has all the common syntax defined as stated above without any additions.[9]
  • COMMAND.COM and cmd.exe have most of the common syntax with some limitations: There is no […] and for COMMAND.COM the * may only appear at the end of the pattern, not at the beginning.

Windows and DOS programs receive a long command-line string instead of argv-style parameters, and it is their responsibility to perform any splitting, quoting, or glob expansion. There is technically no fixed way of describing wildcards in programs since they are free to do what they wish. Two common glob expanders include:[10]

  • The Microsoft C Runtime (msvcrt) command-line expander, which only supports ? and *.[11] Both ReactOS (crt/misc/getargs.c) and Wine (msvcrt/data.c) contain a compatible open-source implementation of __getmainargs, the function operating under-the-hood, in their core CRT.
  • The Cygwin and MSYS dcrt0.cc command-line expander, which uses the unix-style glob() routine under-the-hood, after splitting the arguments.

SQL[edit]

The SQL LIKE operator has an equivalent of ? and *. There is no equivalent of […].

Common wildcard SQL wildcard
? _
* %

Standard SQL uses a glob-like syntax for simple string matching in its LIKE operator. The percent sign (%) matches zero or more characters, and the underscore matches exactly one character. The term "glob" is not generally used in the SQL community, however. Many implementations of SQL have extended the LIKE operator to allow a richer pattern-matching language incorporating elements of regular expressions.

Some proprietary extensions such as Transact-SQL provide the […] functionality, e.g., [characters] and [^characters].[12]

Compared to regular expressions[edit]

Globs do not include syntax for the Kleene star which allows multiple repetitions of the preceding part of the expression; thus they are not considered regular expressions, which can describe the full set of regular languages over any given finite alphabet.[13]

Common wildcard Equivalent regular expression
? .
* .*

Globs attempt to match the entire string (for example, S*.DOC matches S.DOC and SA.DOC, but not POST.DOC or SURREY.DOCKS), whereas regular expressions match a substring unless the expression is enclosed with ^ and $ (so the equivalent of S*.DOC is ^S.*\.DOC$).[nb 1]

Implementing as regular expressions[edit]

The original Mozilla proxy auto-config implementation, which provides a glob-matching function on strings, uses a replace-as-RegExp implementation as above. The bracket syntax happens to be covered by regex in such an example.

Python's fnmatch uses a more elaborate procedure to replace the pattern to a regular expression.[14]

Implementations[edit]

Beyond their uses in shells, globs patterns also find use in a variety of programming languages, mainly to process human input.

A glob-style interface for returning files and/or an fnmatch-style interface for matching strings are found in the following programming languages:

  • D has a globMatch function in the std.path module.[15]
  • JavaScript has a library called minimatch which is used internally by npm, and micromatch, a purportedly more optimized, accurate and safer globbing implementation used by babel and yarn.[16][17]
  • Go has a Glob function in the filepath package.[18]
  • Java has a Files class containing methods that operate on glob patterns.[19]
  • Haskell has a Glob package with the main module System.FilePath.Glob. The pattern syntax is based on a subset of Zsh’s. It tries to optimize the given pattern and should be noticeably faster than a naïve character-by-character matcher.[20]
  • Perl has both a glob function (as discussed in Larry Wall's book Programming Perl) and a Glob extension which mimics the BSD glob routine.[21] Perl's angle brackets can be used to glob as well: <*.log>.
  • PHP has a glob function.[22]
  • Python has a glob module in the standard library which performs wildcard pattern matching on filenames,[23] and an fnmatch module with functions for matching strings or filtering lists based on these same wildcard patterns.[14] Guido van Rossum, author of the Python programming language, wrote and contributed a glob routine to BSD Unix in 1986.[24] There were previous implementations of glob, e.g., in the ex and ftp programs in previous releases of BSD.
  • Ruby has a glob method for the Dir class which performs wildcard pattern matching on filenames.[25] Several libraries such as Rant and Rake provide a FileList class which has a glob method or use the method FileList.[] identically.
  • SQLite has a GLOB function.
  • Tcl contains a globbing facility.[26]

See also[edit]

Notes[edit]

  1. ^ Strictly, . does not match a newline. To match newlines, the equivalents are [\s\S] and [\s\S]* or similar complementary pairs, respectively.

References[edit]

  1. ^ "First Edition Unix manual 'Miscellaneous' section (PDF)" (PDF). Archived from the original (PDF) on 2000-08-29. Retrieved 2011-05-11.
  2. ^ a b fnmatch(3) – Linux Programmer's Manual – Library Functions
  3. ^ glob(3) – Linux Programmer's Manual – Library Functions
  4. ^ a b "The Open Group Base Specifications Issue 7 IEEE Std 1003.1, 2013 Edition, 2.13. Pattern Matching Notation".
  5. ^ a b "Linux Programmer's Manual, GLOB(7)".
  6. ^ The "Advanced Bash-Scripting Guide, Chapter 19.2: Globbing" (Mendel Cooper, 2003) has a concise set of examples of filename globbing patterns.
  7. ^ "Bash globs". greg's bash knowledgebase. Retrieved 2019-11-25.
  8. ^ a b "Pattern Matching". Bash Reference Manual.
  9. ^ "Supporting Wildcard Characters in Cmdlet Parameters". Microsoft. Microsoft Developer Network.
  10. ^ "Wildcard Expansion". Microsoft Developer Network. 2013.
  11. ^ "Wildcard Expansion". docs.microsoft.com.
  12. ^ "LIKE (Transact-SQL)".
  13. ^ Hopcroft, John E.; Motwani, Rajeev; Ullman, Jeffrey D. (2000). Introduction to Automata Theory, Languages, and Computation (2nd ed.). Addison-Wesley.
  14. ^ a b "Lib/fnmatch.py". Python. 2019-11-24. Retrieved 2019-11-24.
  15. ^ "std.path - D Programming Language - Digital Mars". dlang.org. Retrieved 2014-09-08.
  16. ^ "isaacs/minimatch". GitHub. Retrieved 2016-08-10.
  17. ^ "jonschlinkert/micromatch". GitHub. Retrieved 2017-04-04.
  18. ^ "Package filepath - The Go Programming Language". Golang.org. Retrieved 2011-05-11.
  19. ^ "File Operations". Oracle. Retrieved 2013-12-16.
  20. ^ "Glob-0.7.4: Globbing library". Retrieved 2014-05-07.
  21. ^ "File::Glob - Perl extension for BSD glob routine". perldoc.perl.org. Retrieved 2011-05-11.
  22. ^ "glob - Manual". PHP. 2011-05-06. Retrieved 2011-05-11.
  23. ^ "10.7. glob — Unix style pathname pattern expansion — Python v2.7.1 documentation". Docs.python.org. Retrieved 2011-05-11.
  24. ^ "'Globbing' library routine". Archived from the original on 2007-12-19. Retrieved 2011-05-11.
  25. ^ "Class: Dir". Ruby-doc.org. Retrieved 2011-05-11.
  26. ^ "TCL glob manual page". Retrieved 2011-11-16.

Cite error: A list-defined reference named "winWcExp" is not used in the content (see the help page).