1# doublestar
2
3Path pattern matching and globbing supporting `doublestar` (`**`) patterns.
4
5[](https://pkg.go.dev/github.com/bmatcuk/doublestar/v4)
6[](https://github.com/bmatcuk/doublestar/releases)
7[](https://github.com/bmatcuk/doublestar/actions)
8[](https://codecov.io/github/bmatcuk/doublestar?branch=master)
9[](https://github.com/sponsors/bmatcuk)
10
11## About
12
13#### [Upgrading?](UPGRADING.md)
14
15**doublestar** is a [golang] implementation of path pattern matching and
16globbing with support for "doublestar" (aka globstar: `**`) patterns.
17
18doublestar patterns match files and directories recursively. For example, if
19you had the following directory structure:
20
21```bash
22grandparent
23`-- parent
24 |-- child1
25 `-- child2
26```
27
28You could find the children with patterns such as: `**/child*`,
29`grandparent/**/child?`, `**/parent/*`, or even just `**` by itself (which will
30return all files and directories recursively).
31
32Bash's globstar is doublestar's inspiration and, as such, works similarly.
33Note that the doublestar must appear as a path component by itself. A pattern
34such as `/path**` is invalid and will be treated the same as `/path*`, but
35`/path*/**` should achieve the desired result. Additionally, `/path/**` will
36match all directories and files under the path directory, but `/path/**/` will
37only match directories.
38
39v4 is a complete rewrite with a focus on performance. Additionally,
40[doublestar] has been updated to use the new [io/fs] package for filesystem
41access. As a result, it is only supported by [golang] v1.16+.
42
43## Installation
44
45**doublestar** can be installed via `go get`:
46
47```bash
48go get github.com/bmatcuk/doublestar/v4
49```
50
51To use it in your code, you must import it:
52
53```go
54import "github.com/bmatcuk/doublestar/v4"
55```
56
57## Usage
58
59### ErrBadPattern
60
61```go
62doublestar.ErrBadPattern
63```
64
65Returned by various functions to report that the pattern is malformed. At the
66moment, this value is equal to `path.ErrBadPattern`, but, for portability, this
67equivalence should probably not be relied upon.
68
69### Match
70
71```go
72func Match(pattern, name string) (bool, error)
73```
74
75Match returns true if `name` matches the file name `pattern` ([see
76"patterns"]). `name` and `pattern` are split on forward slash (`/`) characters
77and may be relative or absolute.
78
79Match requires pattern to match all of name, not just a substring. The only
80possible returned error is `ErrBadPattern`, when pattern is malformed.
81
82Note: this is meant as a drop-in replacement for `path.Match()` which always
83uses `'/'` as the path separator. If you want to support systems which use a
84different path separator (such as Windows), what you want is `PathMatch()`.
85Alternatively, you can run `filepath.ToSlash()` on both pattern and name and
86then use this function.
87
88Note: users should _not_ count on the returned error,
89`doublestar.ErrBadPattern`, being equal to `path.ErrBadPattern`.
90
91
92### MatchUnvalidated
93
94```go
95func MatchUnvalidated(pattern, name string) bool
96```
97
98MatchUnvalidated can provide a small performance improvement if you don't care
99about whether or not the pattern is valid (perhaps because you already ran
100`ValidatePattern`). Note that there's really only one case where this
101performance improvement is realized: when pattern matching reaches the end of
102`name` before reaching the end of `pattern`, such as `Match("a/b/c", "a")`.
103
104
105### PathMatch
106
107```go
108func PathMatch(pattern, name string) (bool, error)
109```
110
111PathMatch returns true if `name` matches the file name `pattern` ([see
112"patterns"]). The difference between Match and PathMatch is that PathMatch will
113automatically use your system's path separator to split `name` and `pattern`.
114On systems where the path separator is `'\'`, escaping will be disabled.
115
116Note: this is meant as a drop-in replacement for `filepath.Match()`. It assumes
117that both `pattern` and `name` are using the system's path separator. If you
118can't be sure of that, use `filepath.ToSlash()` on both `pattern` and `name`,
119and then use the `Match()` function instead.
120
121
122### PathMatchUnvalidated
123
124```go
125func PathMatchUnvalidated(pattern, name string) bool
126```
127
128PathMatchUnvalidated can provide a small performance improvement if you don't
129care about whether or not the pattern is valid (perhaps because you already ran
130`ValidatePattern`). Note that there's really only one case where this
131performance improvement is realized: when pattern matching reaches the end of
132`name` before reaching the end of `pattern`, such as `Match("a/b/c", "a")`.
133
134
135### GlobOption
136
137Options that may be passed to `Glob`, `GlobWalk`, or `FilepathGlob`. Any number
138of options may be passed to these functions, and in any order, as the last
139argument(s).
140
141```go
142WithFailOnIOErrors()
143```
144
145If passed, doublestar will abort and return IO errors when encountered. Note
146that if the glob pattern references a path that does not exist (such as
147`nonexistent/path/*`), this is _not_ considered an IO error: it is considered a
148pattern with no matches.
149
150```go
151WithFailOnPatternNotExist()
152```
153
154If passed, doublestar will abort and return `doublestar.ErrPatternNotExist` if
155the pattern references a path that does not exist before any meta characters
156such as `nonexistent/path/*`. Note that alts (ie, `{...}`) are expanded before
157this check. In other words, a pattern such as `{a,b}/*` may fail if either `a`
158or `b` do not exist but `*/{a,b}` will never fail because the star may match
159nothing.
160
161```go
162WithFilesOnly()
163```
164
165If passed, doublestar will only return "files" from `Glob`, `GlobWalk`, or
166`FilepathGlob`. In this context, "files" are anything that is not a directory
167or a symlink to a directory.
168
169Note: if combined with the WithNoFollow option, symlinks to directories _will_
170be included in the result since no attempt is made to follow the symlink.
171
172```go
173WithNoFollow()
174```
175
176If passed, doublestar will not follow symlinks while traversing the filesystem.
177However, due to io/fs's _very_ poor support for querying the filesystem about
178symlinks, there's a caveat here: if part of the pattern before any meta
179characters contains a reference to a symlink, it will be followed. For example,
180a pattern such as `path/to/symlink/*` will be followed assuming it is a valid
181symlink to a directory. However, from this same example, a pattern such as
182`path/to/**` will not traverse the `symlink`, nor would `path/*/symlink/*`
183
184Note: if combined with the WithFilesOnly option, symlinks to directories _will_
185be included in the result since no attempt is made to follow the symlink.
186
187### Glob
188
189```go
190func Glob(fsys fs.FS, pattern string, opts ...GlobOption) ([]string, error)
191```
192
193Glob returns the names of all files matching pattern or nil if there is no
194matching file. The syntax of patterns is the same as in `Match()`. The pattern
195may describe hierarchical names such as `usr/*/bin/ed`.
196
197Glob ignores file system errors such as I/O errors reading directories by
198default. The only possible returned error is `ErrBadPattern`, reporting that
199the pattern is malformed.
200
201To enable aborting on I/O errors, the `WithFailOnIOErrors` option can be
202passed.
203
204Note: this is meant as a drop-in replacement for `io/fs.Glob()`. Like
205`io/fs.Glob()`, this function assumes that your pattern uses `/` as the path
206separator even if that's not correct for your OS (like Windows). If you aren't
207sure if that's the case, you can use `filepath.ToSlash()` on your pattern
208before calling `Glob()`.
209
210Like `io/fs.Glob()`, patterns containing `/./`, `/../`, or starting with `/`
211will return no results and no errors. This seems to be a [conscious
212decision](https://github.com/golang/go/issues/44092#issuecomment-774132549),
213even if counter-intuitive. You can use [SplitPattern] to divide a pattern into
214a base path (to initialize an `FS` object) and pattern.
215
216Note: users should _not_ count on the returned error,
217`doublestar.ErrBadPattern`, being equal to `path.ErrBadPattern`.
218
219### GlobWalk
220
221```go
222type GlobWalkFunc func(path string, d fs.DirEntry) error
223
224func GlobWalk(fsys fs.FS, pattern string, fn GlobWalkFunc, opts ...GlobOption) error
225```
226
227GlobWalk calls the callback function `fn` for every file matching pattern. The
228syntax of pattern is the same as in Match() and the behavior is the same as
229Glob(), with regard to limitations (such as patterns containing `/./`, `/../`,
230or starting with `/`). The pattern may describe hierarchical names such as
231usr/*/bin/ed.
232
233GlobWalk may have a small performance benefit over Glob if you do not need a
234slice of matches because it can avoid allocating memory for the matches.
235Additionally, GlobWalk gives you access to the `fs.DirEntry` objects for each
236match, and lets you quit early by returning a non-nil error from your callback
237function. Like `io/fs.WalkDir`, if your callback returns `SkipDir`, GlobWalk
238will skip the current directory. This means that if the current path _is_ a
239directory, GlobWalk will not recurse into it. If the current path is not a
240directory, the rest of the parent directory will be skipped.
241
242GlobWalk ignores file system errors such as I/O errors reading directories by
243default. GlobWalk may return `ErrBadPattern`, reporting that the pattern is
244malformed.
245
246To enable aborting on I/O errors, the `WithFailOnIOErrors` option can be
247passed.
248
249Additionally, if the callback function `fn` returns an error, GlobWalk will
250exit immediately and return that error.
251
252Like Glob(), this function assumes that your pattern uses `/` as the path
253separator even if that's not correct for your OS (like Windows). If you aren't
254sure if that's the case, you can use filepath.ToSlash() on your pattern before
255calling GlobWalk().
256
257Note: users should _not_ count on the returned error,
258`doublestar.ErrBadPattern`, being equal to `path.ErrBadPattern`.
259
260### FilepathGlob
261
262```go
263func FilepathGlob(pattern string, opts ...GlobOption) (matches []string, err error)
264```
265
266FilepathGlob returns the names of all files matching pattern or nil if there is
267no matching file. The syntax of pattern is the same as in Match(). The pattern
268may describe hierarchical names such as usr/*/bin/ed.
269
270FilepathGlob ignores file system errors such as I/O errors reading directories
271by default. The only possible returned error is `ErrBadPattern`, reporting that
272the pattern is malformed.
273
274To enable aborting on I/O errors, the `WithFailOnIOErrors` option can be
275passed.
276
277Note: FilepathGlob is a convenience function that is meant as a drop-in
278replacement for `path/filepath.Glob()` for users who don't need the
279complication of io/fs. Basically, it:
280
281* Runs `filepath.Clean()` and `ToSlash()` on the pattern
282* Runs `SplitPattern()` to get a base path and a pattern to Glob
283* Creates an FS object from the base path and `Glob()s` on the pattern
284* Joins the base path with all of the matches from `Glob()`
285
286Returned paths will use the system's path separator, just like
287`filepath.Glob()`.
288
289Note: the returned error `doublestar.ErrBadPattern` is not equal to
290`filepath.ErrBadPattern`.
291
292### SplitPattern
293
294```go
295func SplitPattern(p string) (base, pattern string)
296```
297
298SplitPattern is a utility function. Given a pattern, SplitPattern will return
299two strings: the first string is everything up to the last slash (`/`) that
300appears _before_ any unescaped "meta" characters (ie, `*?[{`). The second
301string is everything after that slash. For example, given the pattern:
302
303```
304../../path/to/meta*/**
305 ^----------- split here
306```
307
308SplitPattern returns "../../path/to" and "meta*/**". This is useful for
309initializing os.DirFS() to call Glob() because Glob() will silently fail if
310your pattern includes `/./` or `/../`. For example:
311
312```go
313base, pattern := SplitPattern("../../path/to/meta*/**")
314fsys := os.DirFS(base)
315matches, err := Glob(fsys, pattern)
316```
317
318If SplitPattern cannot find somewhere to split the pattern (for example,
319`meta*/**`), it will return "." and the unaltered pattern (`meta*/**` in this
320example).
321
322Note that SplitPattern will also unescape any meta characters in the returned
323base string, so that it can be passed straight to os.DirFS().
324
325Of course, it is your responsibility to decide if the returned base path is
326"safe" in the context of your application. Perhaps you could use Match() to
327validate against a list of approved base directories?
328
329### ValidatePattern
330
331```go
332func ValidatePattern(s string) bool
333```
334
335Validate a pattern. Patterns are validated while they run in Match(),
336PathMatch(), and Glob(), so, you normally wouldn't need to call this. However,
337there are cases where this might be useful: for example, if your program allows
338a user to enter a pattern that you'll run at a later time, you might want to
339validate it.
340
341ValidatePattern assumes your pattern uses '/' as the path separator.
342
343### ValidatePathPattern
344
345```go
346func ValidatePathPattern(s string) bool
347```
348
349Like ValidatePattern, only uses your OS path separator. In other words, use
350ValidatePattern if you would normally use Match() or Glob(). Use
351ValidatePathPattern if you would normally use PathMatch(). Keep in mind, Glob()
352requires '/' separators, even if your OS uses something else.
353
354### Patterns
355
356**doublestar** supports the following special terms in the patterns:
357
358Special Terms | Meaning
359------------- | -------
360`*` | matches any sequence of non-path-separators
361`/**/` | matches zero or more directories
362`?` | matches any single non-path-separator character
363`[class]` | matches any single non-path-separator character against a class of characters ([see "character classes"])
364`{alt1,...}` | matches a sequence of characters if one of the comma-separated alternatives matches
365
366Any character with a special meaning can be escaped with a backslash (`\`).
367
368A doublestar (`**`) should appear surrounded by path separators such as `/**/`.
369A mid-pattern doublestar (`**`) behaves like bash's globstar option: a pattern
370such as `path/to/**.txt` would return the same results as `path/to/*.txt`. The
371pattern you're looking for is `path/to/**/*.txt`.
372
373#### Character Classes
374
375Character classes support the following:
376
377Class | Meaning
378---------- | -------
379`[abc123]` | matches any single character within the set
380`[a-z0-9]` | matches any single character in the range a-z or 0-9
381`[125-79]` | matches any single character within the set 129, or the range 5-7
382`[^class]` | matches any single character which does *not* match the class
383`[!class]` | same as `^`: negates the class
384
385## Performance
386
387```
388goos: darwin
389goarch: amd64
390pkg: github.com/bmatcuk/doublestar/v4
391cpu: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
392BenchmarkMatch-8 285639 3868 ns/op 0 B/op 0 allocs/op
393BenchmarkGoMatch-8 286945 3726 ns/op 0 B/op 0 allocs/op
394BenchmarkPathMatch-8 320511 3493 ns/op 0 B/op 0 allocs/op
395BenchmarkGoPathMatch-8 304236 3434 ns/op 0 B/op 0 allocs/op
396BenchmarkGlob-8 466 2501123 ns/op 190225 B/op 2849 allocs/op
397BenchmarkGlobWalk-8 476 2536293 ns/op 184017 B/op 2750 allocs/op
398BenchmarkGoGlob-8 463 2574836 ns/op 194249 B/op 2929 allocs/op
399```
400
401These benchmarks (in `doublestar_test.go`) compare Match() to path.Match(),
402PathMath() to filepath.Match(), and Glob() + GlobWalk() to io/fs.Glob(). They
403only run patterns that the standard go packages can understand as well (so, no
404`{alts}` or `**`) for a fair comparison. Of course, alts and doublestars will
405be less performant than the other pattern meta characters.
406
407Alts are essentially like running multiple patterns, the number of which can
408get large if your pattern has alts nested inside alts. This affects both
409matching (ie, Match()) and globbing (Glob()).
410
411`**` performance in matching is actually pretty similar to a regular `*`, but
412can cause a large number of reads when globbing as it will need to recursively
413traverse your filesystem.
414
415## Sponsors
416I started this project in 2014 in my spare time and have been maintaining it
417ever since. In that time, it has grown into one of the most popular globbing
418libraries in the Go ecosystem. So, if **doublestar** is a useful library in
419your project, consider [sponsoring] my work! I'd really appreciate it!
420
421Thanks for sponsoring me!
422
423## License
424
425[MIT License](LICENSE)
426
427[SplitPattern]: #splitpattern
428[doublestar]: https://github.com/bmatcuk/doublestar
429[golang]: http://golang.org/
430[io/fs]: https://pkg.go.dev/io/fs
431[see "character classes"]: #character-classes
432[see "patterns"]: #patterns
433[sponsoring]: https://github.com/sponsors/bmatcuk