1[](https://pkg.go.dev/github.com/charlievieth/fastwalk)
2[](https://github.com/charlievieth/fastwalk/actions/workflows/macos.yml)
3[](https://github.com/charlievieth/fastwalk/actions/workflows/linux.yml)
4[](https://github.com/charlievieth/fastwalk/actions/workflows/windows.yml)
5
6# fastwalk
7
8Fast parallel directory traversal for Golang.
9
10Package fastwalk provides a fast parallel version of [`filepath.WalkDir`](https://pkg.go.dev/io/fs#WalkDirFunc)
11that is \~2.5x faster on macOS, \~4x faster on Linux, \~6x faster on Windows,
12allocates 50% less memory, and requires 25% fewer memory allocations.
13Additionally, it is \~4-5x faster than [godirwalk](https://github.com/karrick/godirwalk)
14across OSes.
15
16Inspired by and based off of [golang.org/x/tools/internal/fastwalk](https://pkg.go.dev/golang.org/x/tools@v0.1.9/internal/fastwalk).
17
18## Features
19
20* Fast: multiple goroutines stat the filesystem and call the
21 [`filepath.WalkDirFunc`](https://pkg.go.dev/io/fs#WalkDirFunc) callback concurrently
22* Safe symbolic link traversal ([`Config.Follow`](https://pkg.go.dev/github.com/charlievieth/fastwalk#Config))
23* Same behavior and callback signature as [`filepath.WalkDir`](https://pkg.go.dev/path/filepath#WalkDir)
24* Wrapper functions are provided to ignore duplicate files and directories:
25 [`IgnoreDuplicateFiles()`](https://pkg.go.dev/github.com/charlievieth/fastwalk#IgnoreDuplicateFiles)
26 and
27 [`IgnoreDuplicateDirs()`](https://pkg.go.dev/github.com/charlievieth/fastwalk#IgnoreDuplicateDirs)
28* Extensively tested on macOS, Linux, and Windows
29
30## Usage
31
32Usage is the same as [`filepath.WalkDir`](https://pkg.go.dev/io/fs#WalkDirFunc),
33but the [`walkFn`](https://pkg.go.dev/path/filepath@go1.17.7#WalkFunc)
34argument to [`fastwalk.Walk`](https://pkg.go.dev/github.com/charlievieth/fastwalk#Walk)
35must be safe for concurrent use.
36
37Examples can be found in the [examples](./examples) directory.
38
39<!-- TODO: this example is large move it to an examples folder -->
40
41The below example is a very simple version of the POSIX
42[find](https://pubs.opengroup.org/onlinepubs/007904975/utilities/find.html) utility:
43```go
44// fwfind is a an example program that is similar to POSIX find,
45// but faster and worse (it's an example).
46package main
47
48import (
49 "flag"
50 "fmt"
51 "io/fs"
52 "os"
53 "path/filepath"
54
55 "github.com/charlievieth/fastwalk"
56)
57
58const usageMsg = `Usage: %[1]s [-L] [-name] [PATH...]:
59
60%[1]s is a poor replacement for the POSIX find utility
61
62`
63
64func main() {
65 flag.Usage = func() {
66 fmt.Fprintf(os.Stdout, usageMsg, filepath.Base(os.Args[0]))
67 flag.PrintDefaults()
68 }
69 pattern := flag.String("name", "", "Pattern to match file names against.")
70 followLinks := flag.Bool("L", false, "Follow symbolic links")
71 flag.Parse()
72
73 // If no paths are provided default to the current directory: "."
74 args := flag.Args()
75 if len(args) == 0 {
76 args = append(args, ".")
77 }
78
79 // Follow links if the "-L" flag is provided
80 conf := fastwalk.Config{
81 Follow: *followLinks,
82 }
83
84 walkFn := func(path string, d fs.DirEntry, err error) error {
85 if err != nil {
86 fmt.Fprintf(os.Stderr, "%s: %v\n", path, err)
87 return nil // returning the error stops iteration
88 }
89 if *pattern != "" {
90 if ok, err := filepath.Match(*pattern, d.Name()); !ok {
91 // invalid pattern (err != nil) or name does not match
92 return err
93 }
94 }
95 _, err = fmt.Println(path)
96 return err
97 }
98 for _, root := range args {
99 if err := fastwalk.Walk(&conf, root, walkFn); err != nil {
100 fmt.Fprintf(os.Stderr, "%s: %v\n", root, err)
101 os.Exit(1)
102 }
103 }
104}
105```
106
107## Benchmarks
108
109Benchmarks were created using `go1.17.6` and can be generated with the `bench_comp` make target:
110```sh
111$ make bench_comp
112```
113
114### Darwin
115
116**Hardware:**
117```
118goos: darwin
119goarch: arm64
120cpu: Apple M1 Max
121```
122
123#### [`filepath.WalkDir`](https://pkg.go.dev/path/filepath@go1.17.7#WalkDir) vs. [`fastwalk.Walk()`](https://pkg.go.dev/github.com/charlievieth/fastwalk#Walk):
124```
125 filepath fastwalk delta
126time/op 27.9ms ± 1% 13.0ms ± 1% -53.33%
127alloc/op 4.33MB ± 0% 2.14MB ± 0% -50.55%
128allocs/op 50.9k ± 0% 37.7k ± 0% -26.01%
129```
130
131#### [`godirwalk.Walk()`](https://pkg.go.dev/github.com/karrick/godirwalk@v1.16.1#Walk) vs. [`fastwalk.Walk()`](https://pkg.go.dev/github.com/charlievieth/fastwalk#Walk):
132```
133 godirwalk fastwalk delta
134time/op 58.5ms ± 3% 18.0ms ± 2% -69.30%
135alloc/op 25.3MB ± 0% 2.1MB ± 0% -91.55%
136allocs/op 57.6k ± 0% 37.7k ± 0% -34.59%
137```
138
139### Linux
140
141**Hardware:**
142```
143goos: linux
144goarch: amd64
145cpu: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
146drive: Samsung SSD 970 PRO 1TB
147```
148
149#### [`filepath.WalkDir`](https://pkg.go.dev/path/filepath@go1.17.7#WalkDir) vs. [`fastwalk.Walk()`](https://pkg.go.dev/github.com/charlievieth/fastwalk#Walk):
150
151```
152 filepath fastwalk delta
153time/op 10.1ms ± 2% 2.8ms ± 2% -72.83%
154alloc/op 2.44MB ± 0% 1.70MB ± 0% -30.46%
155allocs/op 47.2k ± 0% 36.9k ± 0% -21.80%
156```
157
158#### [`godirwalk.Walk()`](https://pkg.go.dev/github.com/karrick/godirwalk@v1.16.1#Walk) vs. [`fastwalk.Walk()`](https://pkg.go.dev/github.com/charlievieth/fastwalk#Walk):
159
160```
161 filepath fastwalk delta
162time/op 13.7ms ±16% 2.8ms ± 2% -79.88%
163alloc/op 7.48MB ± 0% 1.70MB ± 0% -77.34%
164allocs/op 53.8k ± 0% 36.9k ± 0% -31.38%
165```
166
167### Windows
168
169**Hardware:**
170```
171goos: windows
172goarch: amd64
173pkg: github.com/charlievieth/fastwalk
174cpu: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
175```
176
177#### [`filepath.WalkDir`](https://pkg.go.dev/path/filepath@go1.17.7#WalkDir) vs. [`fastwalk.Walk()`](https://pkg.go.dev/github.com/charlievieth/fastwalk#Walk):
178
179```
180 filepath fastwalk delta
181time/op 88.0ms ± 1% 14.6ms ± 1% -83.47%
182alloc/op 5.68MB ± 0% 6.76MB ± 0% +19.01%
183allocs/op 69.6k ± 0% 90.4k ± 0% +29.87%
184```
185
186#### [`godirwalk.Walk()`](https://pkg.go.dev/github.com/karrick/godirwalk@v1.16.1#Walk) vs. [`fastwalk.Walk()`](https://pkg.go.dev/github.com/charlievieth/fastwalk#Walk):
187
188```
189 filepath fastwalk delta
190time/op 87.4ms ± 1% 14.6ms ± 1% -83.34%
191alloc/op 6.14MB ± 0% 6.76MB ± 0% +10.24%
192allocs/op 100k ± 0% 90k ± 0% -9.59%
193```