1// Copyright 2018 The Go Authors. All rights reserved.
2// Use of this source code is governed by a BSD-style
3// license that can be found in the LICENSE file.
4
5/*
6Package packages loads Go packages for inspection and analysis.
7
8The Load function takes as input a list of patterns and return a list of Package
9structs describing individual packages matched by those patterns.
10The LoadMode controls the amount of detail in the loaded packages.
11
12Load passes most patterns directly to the underlying build tool,
13but all patterns with the prefix "query=", where query is a
14non-empty string of letters from [a-z], are reserved and may be
15interpreted as query operators.
16
17Two query operators are currently supported: "file" and "pattern".
18
19The query "file=path/to/file.go" matches the package or packages enclosing
20the Go source file path/to/file.go. For example "file=~/go/src/fmt/print.go"
21might return the packages "fmt" and "fmt [fmt.test]".
22
23The query "pattern=string" causes "string" to be passed directly to
24the underlying build tool. In most cases this is unnecessary,
25but an application can use Load("pattern=" + x) as an escaping mechanism
26to ensure that x is not interpreted as a query operator if it contains '='.
27
28All other query operators are reserved for future use and currently
29cause Load to report an error.
30
31The Package struct provides basic information about the package, including
32
33 - ID, a unique identifier for the package in the returned set;
34 - GoFiles, the names of the package's Go source files;
35 - Imports, a map from source import strings to the Packages they name;
36 - Types, the type information for the package's exported symbols;
37 - Syntax, the parsed syntax trees for the package's source code; and
38 - TypeInfo, the result of a complete type-check of the package syntax trees.
39
40(See the documentation for type Package for the complete list of fields
41and more detailed descriptions.)
42
43For example,
44
45 Load(nil, "bytes", "unicode...")
46
47returns four Package structs describing the standard library packages
48bytes, unicode, unicode/utf16, and unicode/utf8. Note that one pattern
49can match multiple packages and that a package might be matched by
50multiple patterns: in general it is not possible to determine which
51packages correspond to which patterns.
52
53Note that the list returned by Load contains only the packages matched
54by the patterns. Their dependencies can be found by walking the import
55graph using the Imports fields.
56
57The Load function can be configured by passing a pointer to a Config as
58the first argument. A nil Config is equivalent to the zero Config, which
59causes Load to run in LoadFiles mode, collecting minimal information.
60See the documentation for type Config for details.
61
62As noted earlier, the Config.Mode controls the amount of detail
63reported about the loaded packages, with each mode returning all the data of the
64previous mode with some extra added. See the documentation for type LoadMode
65for details.
66
67Most tools should pass their command-line arguments (after any flags)
68uninterpreted to the loader, so that the loader can interpret them
69according to the conventions of the underlying build system.
70See the Example function for typical usage.
71
72*/
73package packages // import "golang.org/x/tools/go/packages"
74
75/*
76
77Motivation and design considerations
78
79The new package's design solves problems addressed by two existing
80packages: go/build, which locates and describes packages, and
81golang.org/x/tools/go/loader, which loads, parses and type-checks them.
82The go/build.Package structure encodes too much of the 'go build' way
83of organizing projects, leaving us in need of a data type that describes a
84package of Go source code independent of the underlying build system.
85We wanted something that works equally well with go build and vgo, and
86also other build systems such as Bazel and Blaze, making it possible to
87construct analysis tools that work in all these environments.
88Tools such as errcheck and staticcheck were essentially unavailable to
89the Go community at Google, and some of Google's internal tools for Go
90are unavailable externally.
91This new package provides a uniform way to obtain package metadata by
92querying each of these build systems, optionally supporting their
93preferred command-line notations for packages, so that tools integrate
94neatly with users' build environments. The Metadata query function
95executes an external query tool appropriate to the current workspace.
96
97Loading packages always returns the complete import graph "all the way down",
98even if all you want is information about a single package, because the query
99mechanisms of all the build systems we currently support ({go,vgo} list, and
100blaze/bazel aspect-based query) cannot provide detailed information
101about one package without visiting all its dependencies too, so there is
102no additional asymptotic cost to providing transitive information.
103(This property might not be true of a hypothetical 5th build system.)
104
105In calls to TypeCheck, all initial packages, and any package that
106transitively depends on one of them, must be loaded from source.
107Consider A->B->C->D->E: if A,C are initial, A,B,C must be loaded from
108source; D may be loaded from export data, and E may not be loaded at all
109(though it's possible that D's export data mentions it, so a
110types.Package may be created for it and exposed.)
111
112The old loader had a feature to suppress type-checking of function
113bodies on a per-package basis, primarily intended to reduce the work of
114obtaining type information for imported packages. Now that imports are
115satisfied by export data, the optimization no longer seems necessary.
116
117Despite some early attempts, the old loader did not exploit export data,
118instead always using the equivalent of WholeProgram mode. This was due
119to the complexity of mixing source and export data packages (now
120resolved by the upward traversal mentioned above), and because export data
121files were nearly always missing or stale. Now that 'go build' supports
122caching, all the underlying build systems can guarantee to produce
123export data in a reasonable (amortized) time.
124
125Test "main" packages synthesized by the build system are now reported as
126first-class packages, avoiding the need for clients (such as go/ssa) to
127reinvent this generation logic.
128
129One way in which go/packages is simpler than the old loader is in its
130treatment of in-package tests. In-package tests are packages that
131consist of all the files of the library under test, plus the test files.
132The old loader constructed in-package tests by a two-phase process of
133mutation called "augmentation": first it would construct and type check
134all the ordinary library packages and type-check the packages that
135depend on them; then it would add more (test) files to the package and
136type-check again. This two-phase approach had four major problems:
1371) in processing the tests, the loader modified the library package,
138 leaving no way for a client application to see both the test
139 package and the library package; one would mutate into the other.
1402) because test files can declare additional methods on types defined in
141 the library portion of the package, the dispatch of method calls in
142 the library portion was affected by the presence of the test files.
143 This should have been a clue that the packages were logically
144 different.
1453) this model of "augmentation" assumed at most one in-package test
146 per library package, which is true of projects using 'go build',
147 but not other build systems.
1484) because of the two-phase nature of test processing, all packages that
149 import the library package had to be processed before augmentation,
150 forcing a "one-shot" API and preventing the client from calling Load
151 in several times in sequence as is now possible in WholeProgram mode.
152 (TypeCheck mode has a similar one-shot restriction for a different reason.)
153
154Early drafts of this package supported "multi-shot" operation.
155Although it allowed clients to make a sequence of calls (or concurrent
156calls) to Load, building up the graph of Packages incrementally,
157it was of marginal value: it complicated the API
158(since it allowed some options to vary across calls but not others),
159it complicated the implementation,
160it cannot be made to work in Types mode, as explained above,
161and it was less efficient than making one combined call (when this is possible).
162Among the clients we have inspected, none made multiple calls to load
163but could not be easily and satisfactorily modified to make only a single call.
164However, applications changes may be required.
165For example, the ssadump command loads the user-specified packages
166and in addition the runtime package. It is tempting to simply append
167"runtime" to the user-provided list, but that does not work if the user
168specified an ad-hoc package such as [a.go b.go].
169Instead, ssadump no longer requests the runtime package,
170but seeks it among the dependencies of the user-specified packages,
171and emits an error if it is not found.
172
173Overlays: The Overlay field in the Config allows providing alternate contents
174for Go source files, by providing a mapping from file path to contents.
175go/packages will pull in new imports added in overlay files when go/packages
176is run in LoadImports mode or greater.
177Overlay support for the go list driver isn't complete yet: if the file doesn't
178exist on disk, it will only be recognized in an overlay if it is a non-test file
179and the package would be reported even without the overlay.
180
181Questions & Tasks
182
183- Add GOARCH/GOOS?
184 They are not portable concepts, but could be made portable.
185 Our goal has been to allow users to express themselves using the conventions
186 of the underlying build system: if the build system honors GOARCH
187 during a build and during a metadata query, then so should
188 applications built atop that query mechanism.
189 Conversely, if the target architecture of the build is determined by
190 command-line flags, the application can pass the relevant
191 flags through to the build system using a command such as:
192 myapp -query_flag="--cpu=amd64" -query_flag="--os=darwin"
193 However, this approach is low-level, unwieldy, and non-portable.
194 GOOS and GOARCH seem important enough to warrant a dedicated option.
195
196- How should we handle partial failures such as a mixture of good and
197 malformed patterns, existing and non-existent packages, successful and
198 failed builds, import failures, import cycles, and so on, in a call to
199 Load?
200
201- Support bazel, blaze, and go1.10 list, not just go1.11 list.
202
203- Handle (and test) various partial success cases, e.g.
204 a mixture of good packages and:
205 invalid patterns
206 nonexistent packages
207 empty packages
208 packages with malformed package or import declarations
209 unreadable files
210 import cycles
211 other parse errors
212 type errors
213 Make sure we record errors at the correct place in the graph.
214
215- Missing packages among initial arguments are not reported.
216 Return bogus packages for them, like golist does.
217
218- "undeclared name" errors (for example) are reported out of source file
219 order. I suspect this is due to the breadth-first resolution now used
220 by go/types. Is that a bug? Discuss with gri.
221
222*/