1// Copyright 2018 The Go Authors. All rights reserved.
2// Use of this source code is governed by a BSD-style
3// license that can be found in the LICENSE file.
4
5/*
6Package packages loads Go packages for inspection and analysis.
7
8NOTE: THIS PACKAGE IS NOT YET READY FOR WIDESPREAD USE:
9 - The interface is still being revised and is likely to change.
10 - The implementation depends on the Go 1.11 go command.
11 - We intend to finalize the API before Go 1.11 is released.
12
13The Load function takes as input a list of patterns and return a list of Package
14structs describing individual packages matched by those patterns.
15The LoadMode controls the amounts of detail about the loaded packages.
16
17The patterns are used as arguments to the underlying build tool,
18such as the go command or Bazel, and are interpreted according to
19that tool's conventions.
20
21The Package struct provides basic information about the package, including
22
23 - ID, a unique identifier for the package in the returned set;
24 - GoFiles, the names of the package's Go source files;
25 - Imports, a map from source import strings to the Packages they name;
26 - Types, the type information for the package's exported symbols;
27 - Syntax, the parsed syntax trees for the package's source code; and
28 - TypeInfo, the result of a complete type-check of the package syntax trees.
29
30(See the documentation for type Package for the complete list of fields
31and more detailed descriptions.)
32
33For example,
34
35 Load(nil, "bytes", "unicode...")
36
37returns four Package structs describing the standard library packages
38bytes, unicode, unicode/utf16, and unicode/utf8. Note that one pattern
39can match multiple packages and that a package might be matched by
40multiple patterns: in general it is not possible to determine which
41packages correspond to which patterns.
42
43Note that the list returned by Load (LoadAllSyntax in this case)
44only contains the packages matched by the patterns. Their dependencies
45can be found by walking the import graph using the Imports fields.
46
47The Load function can be configured by passing a non-nil Config struct as
48the first argument. If you pass nil for the Config Load will
49run in LoadAllSyntax mode, collecting the maximal amount of information
50it can.
51See the documentation for type Config for details.
52
53As noted earlier, the Config.Mode controls increasing amounts of detail
54about the loaded packages, with each mode returning all the data of the
55previous mode with some extra added. See the documentation for type LoadMode
56for details.
57
58Most tools should pass their command-line arguments (after any flags)
59uninterpreted to the loader, so that the loader can interpret them
60according to the conventions of the underlying build system.
61For example, this program prints the names of the source files
62for each package listed on the command line:
63
64 package main
65
66 import (
67 "flag"
68 "fmt"
69 "log"
70
71 "golang.org/x/tools/go/packages"
72 )
73
74 func main() {
75 flag.Parse()
76 pkgs, err := packages.Load(nil, flag.Args()...)
77 if err != nil {
78 log.Fatal(err)
79 }
80 for _, pkg := range pkgs {
81 fmt.Print(pkg.ID, pkg.GoFiles)
82 }
83 }
84*/
85package packages // import "golang.org/x/tools/go/packages"
86
87/*
88
89Motivation and design considerations
90
91The new package's design solves problems addressed by two existing
92packages: go/build, which locates and describes packages, and
93golang.org/x/tools/go/loader, which loads, parses and type-checks them.
94The go/build.Package structure encodes too much of the 'go build' way
95of organizing projects, leaving us in need of a data type that describes a
96package of Go source code independent of the underlying build system.
97We wanted something that works equally well with go build and vgo, and
98also other build systems such as Bazel and Blaze, making it possible to
99construct analysis tools that work in all these environments.
100Tools such as errcheck and staticcheck were essentially unavailable to
101the Go community at Google, and some of Google's internal tools for Go
102are unavailable externally.
103This new package provides a uniform way to obtain package metadata by
104querying each of these build systems, optionally supporting their
105preferred command-line notations for packages, so that tools integrate
106neatly with users' build environments. The Metadata query function
107executes an external query tool appropriate to the current workspace.
108
109Loading packages always returns the complete import graph "all the way down",
110even if all you want is information about a single package, because the query
111mechanisms of all the build systems we currently support ({go,vgo} list, and
112blaze/bazel aspect-based query) cannot provide detailed information
113about one package without visiting all its dependencies too, so there is
114no additional asymptotic cost to providing transitive information.
115(This property might not be true of a hypothetical 5th build system.)
116
117This package provides no parse-but-don't-typecheck operation because most tools
118that need only untyped syntax (such as gofmt, goimports, and golint)
119seem not to care about any files other than the ones they are directly
120instructed to look at. Also, it is trivial for a client to supplement
121this functionality on top of a Metadata query.
122
123In calls to TypeCheck, all initial packages, and any package that
124transitively depends on one of them, must be loaded from source.
125Consider A->B->C->D->E: if A,C are initial, A,B,C must be loaded from
126source; D may be loaded from export data, and E may not be loaded at all
127(though it's possible that D's export data mentions it, so a
128types.Package may be created for it and exposed.)
129
130The old loader had a feature to suppress type-checking of function
131bodies on a per-package basis, primarily intended to reduce the work of
132obtaining type information for imported packages. Now that imports are
133satisfied by export data, the optimization no longer seems necessary.
134
135Despite some early attempts, the old loader did not exploit export data,
136instead always using the equivalent of WholeProgram mode. This was due
137to the complexity of mixing source and export data packages (now
138resolved by the upward traversal mentioned above), and because export data
139files were nearly always missing or stale. Now that 'go build' supports
140caching, all the underlying build systems can guarantee to produce
141export data in a reasonable (amortized) time.
142
143Test "main" packages synthesized by the build system are now reported as
144first-class packages, avoiding the need for clients (such as go/ssa) to
145reinvent this generation logic.
146
147One way in which go/packages is simpler than the old loader is in its
148treatment of in-package tests. In-package tests are packages that
149consist of all the files of the library under test, plus the test files.
150The old loader constructed in-package tests by a two-phase process of
151mutation called "augmentation": first it would construct and type check
152all the ordinary library packages and type-check the packages that
153depend on them; then it would add more (test) files to the package and
154type-check again. This two-phase approach had four major problems:
1551) in processing the tests, the loader modified the library package,
156 leaving no way for a client application to see both the test
157 package and the library package; one would mutate into the other.
1582) because test files can declare additional methods on types defined in
159 the library portion of the package, the dispatch of method calls in
160 the library portion was affected by the presence of the test files.
161 This should have been a clue that the packages were logically
162 different.
1633) this model of "augmentation" assumed at most one in-package test
164 per library package, which is true of projects using 'go build',
165 but not other build systems.
1664) because of the two-phase nature of test processing, all packages that
167 import the library package had to be processed before augmentation,
168 forcing a "one-shot" API and preventing the client from calling Load
169 in several times in sequence as is now possible in WholeProgram mode.
170 (TypeCheck mode has a similar one-shot restriction for a different reason.)
171
172Early drafts of this package supported "multi-shot" operation
173in the Metadata and WholeProgram modes, although this feature is not exposed
174through the API and will likely be removed.
175Although it allowed clients to make a sequence of calls (or concurrent
176calls) to Load, building up the graph of Packages incrementally,
177it was of marginal value: it complicated the API
178(since it allowed some options to vary across calls but not others),
179it complicated the implementation,
180it cannot be made to work in TypeCheck mode, as explained above,
181and it was less efficient than making one combined call (when this is possible).
182Among the clients we have inspected, none made multiple calls to load
183but could not be easily and satisfactorily modified to make only a single call.
184However, applications changes may be required.
185For example, the ssadump command loads the user-specified packages
186and in addition the runtime package. It is tempting to simply append
187"runtime" to the user-provided list, but that does not work if the user
188specified an ad-hoc package such as [a.go b.go].
189Instead, ssadump no longer requests the runtime package,
190but seeks it among the dependencies of the user-specified packages,
191and emits an error if it is not found.
192
193Overlays: the ParseFile hook in the API permits clients to vary the way
194in which ASTs are obtained from filenames; the default implementation is
195based on parser.ParseFile. This features enables editor-integrated tools
196that analyze the contents of modified but unsaved buffers: rather than
197read from the file system, a tool can read from an archive of modified
198buffers provided by the editor.
199This approach has its limits. Because package metadata is obtained by
200fork/execing an external query command for each build system, we can
201fake only the file contents seen by the parser, type-checker, and
202application, but not by the metadata query, so, for example:
203- additional imports in the fake file will not be described by the
204 metadata, so the type checker will fail to load imports that create
205 new dependencies.
206- in TypeCheck mode, because export data is produced by the query
207 command, it will not reflect the fake file contents.
208- this mechanism cannot add files to a package without first saving them.
209
210Questions & Tasks
211
212- Add GOARCH/GOOS?
213 They are not portable concepts, but could be made portable.
214 Our goal has been to allow users to express themselves using the conventions
215 of the underlying build system: if the build system honors GOARCH
216 during a build and during a metadata query, then so should
217 applications built atop that query mechanism.
218 Conversely, if the target architecture of the build is determined by
219 command-line flags, the application can pass the relevant
220 flags through to the build system using a command such as:
221 myapp -query_flag="--cpu=amd64" -query_flag="--os=darwin"
222 However, this approach is low-level, unwieldy, and non-portable.
223 GOOS and GOARCH seem important enough to warrant a dedicated option.
224
225- How should we handle partial failures such as a mixture of good and
226 malformed patterns, existing and non-existent packages, successful and
227 failed builds, import failures, import cycles, and so on, in a call to
228 Load?
229
230- Do we need a GeneratedBy map that maps the name of each generated Go
231 source file in GoFiles to that of the original file, if known, or "" otherwise?
232 Or are //line directives and "Generated" comments in those files enough?
233
234- Support bazel, blaze, and go1.10 list, not just go1.11 list.
235
236- Handle (and test) various partial success cases, e.g.
237 a mixture of good packages and:
238 invalid patterns
239 nonexistent packages
240 empty packages
241 packages with malformed package or import declarations
242 unreadable files
243 import cycles
244 other parse errors
245 type errors
246 Make sure we record errors at the correct place in the graph.
247
248- Missing packages among initial arguments are not reported.
249 Return bogus packages for them, like golist does.
250
251- "undeclared name" errors (for example) are reported out of source file
252 order. I suspect this is due to the breadth-first resolution now used
253 by go/types. Is that a bug? Discuss with gri.
254
255- https://github.com/golang/go/issues/25980 causes these commands to crash:
256 $ GOPATH=/none ./gopackages -all all
257 due to:
258 $ GOPATH=/none go list -e -test -json all
259 and:
260 $ go list -e -test ./relative/path
261
262- Modify stringer to use go/packages, perhaps initially under flag control.
263
264- Bug: "gopackages fmt a.go" doesn't produce an error.
265
266- If necessary, add back an IsTest boolean or expose ForTests on the Package struct.
267 IsTest was removed because we couldn't agree on a useful definition.
268
269*/