diff --git a/.agents/skills/builtin-skills/SKILL.md b/.agents/skills/builtin-skills/SKILL.md new file mode 100644 index 0000000000000000000000000000000000000000..991f61866b64ce19133765908aed813cf51ad9c8 --- /dev/null +++ b/.agents/skills/builtin-skills/SKILL.md @@ -0,0 +1,39 @@ +--- +name: builtin-skills +description: Guide for adding and modifying builtin skills embedded in the Crush binary. Use when creating a new builtin skill, editing an existing one, or understanding how the embedded skill system works (internal/skills/builtin/). +--- + +# Builtin Skills + +Crush embeds skills directly into the binary via `internal/skills/builtin/`. +These are always available without user configuration. + +## How It Works + +- Each skill lives in `internal/skills/builtin//SKILL.md`. +- The tree is embedded at compile time via `//go:embed builtin/*` in + `internal/skills/embed.go`. +- `DiscoverBuiltin()` walks the embedded FS, parses each `SKILL.md`, and + sets paths with the `crush://skills/` prefix (e.g., + `crush://skills/jq/SKILL.md`). +- The View tool resolves `crush://` paths from the embedded FS, not disk. +- User skills with the same name override builtins (last occurrence wins + in `Deduplicate()`). + +## Adding a New Builtin Skill + +1. Create `internal/skills/builtin//SKILL.md` with YAML + frontmatter (`name`, `description`) and markdown instructions. The + directory name must match the `name` field. +2. No extra wiring needed — `//go:embed builtin/*` picks up new + directories automatically. +3. Add a test assertion in `TestDiscoverBuiltin` in + `internal/skills/skills_test.go` to verify discovery. +4. Build and test: `go build . && go test ./internal/skills/...` + +## Existing Builtin Skills + +| Skill | Directory | Description | +|-------|-----------|-------------| +| `crush-config` | `builtin/crush-config/` | Crush configuration help | +| `jq` | `builtin/jq/` | jq JSON processor usage guide | diff --git a/.agents/skills/shell-builtins/SKILL.md b/.agents/skills/shell-builtins/SKILL.md new file mode 100644 index 0000000000000000000000000000000000000000..4ba4c4a2e35e3a7f5321c51ea9a35f5383cbabef --- /dev/null +++ b/.agents/skills/shell-builtins/SKILL.md @@ -0,0 +1,51 @@ +--- +name: shell-builtins +description: Guide for adding and modifying shell builtin commands in Crush's embedded shell (internal/shell/). Use when creating a new builtin command, editing an existing one, or understanding how commands are intercepted in the mvdan/sh interpreter. +--- + +# Shell Builtins + +Crush's shell (`internal/shell/`) uses `mvdan.cc/sh/v3` for POSIX shell +emulation. Commands can be intercepted before they reach the OS by adding +**builtins** — functions handled in-process. + +## How Builtins Work + +Builtins live in `Shell.builtinHandler()` in `internal/shell/shell.go`. +This is an `interp.ExecHandlerFunc` middleware registered in +`execHandlers()` **before** the block handler, so builtins run even for +commands that would otherwise be blocked. + +The handler is a switch on `args[0]`. Each case either handles the command +inline or delegates to a helper function. + +## Adding a New Builtin + +1. **Add the case** to the switch in `builtinHandler()` in `shell.go`. +2. **Get I/O from the handler context**, not from `os.Stdin`/`os.Stdout`. + This ensures the builtin works with pipes and redirections: + ```go + case "mycommand": + hc := interp.HandlerCtx(ctx) + return handleMyCommand(args, hc.Stdin, hc.Stdout, hc.Stderr) + ``` +3. **Implement the handler** in its own file (e.g., + `internal/shell/mycommand.go`). The function signature should accept + args, stdin, stdout, and stderr: + ```go + func handleMyCommand(args []string, stdin io.Reader, stdout, stderr io.Writer) error { + // args[0] is the command name ("mycommand"), args[1:] are arguments. + // Write output to stdout, errors to stderr. + // Return nil on success, or interp.ExitStatus(n) for non-zero exit codes. + } + ``` +4. **Return values**: return `nil` for success, `interp.ExitStatus(n)` for + non-zero exit codes. Write error messages to `stderr` before returning. +5. **No extra wiring needed** — `builtinHandler()` is already registered + in `execHandlers()`. + +## Existing Builtins + +| Command | File | Description | +|---------|------|-------------| +| `jq` | `jq.go` | JSON processor using `github.com/itchyny/gojq` | diff --git a/go.mod b/go.mod index a4ac1911cd4f75c6057d70f7731edc4769b37ff7..794095bd4bea16ceff5703ec8b61e3ac726e26e0 100644 --- a/go.mod +++ b/go.mod @@ -44,6 +44,7 @@ require ( github.com/go-git/go-git/v5 v5.17.2 github.com/google/uuid v1.6.0 github.com/invopop/jsonschema v0.13.0 + github.com/itchyny/gojq v0.12.19 github.com/joho/godotenv v1.5.1 github.com/jordanella/go-ansi-paintbrush v0.0.0-20240728195301-b7ad996ecf3d github.com/lucasb-eyer/go-colorful v1.4.0 @@ -142,6 +143,7 @@ require ( github.com/gorilla/websocket v1.5.3 // indirect github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect github.com/inconshreveable/mousetrap v1.1.0 // indirect + github.com/itchyny/timefmt-go v0.1.8 // indirect github.com/jackmordaunt/icns/v3 v3.0.1 // indirect github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect github.com/josharian/intern v1.0.0 // indirect diff --git a/go.sum b/go.sum index 89f13f804c3a67480271181ed89e8a403e0b312b..f7a861295e913cf0db8d254420bbb4c812275135 100644 --- a/go.sum +++ b/go.sum @@ -250,6 +250,10 @@ github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2 github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= github.com/invopop/jsonschema v0.13.0 h1:KvpoAJWEjR3uD9Kbm2HWJmqsEaHt8lBUpd0qHcIi21E= github.com/invopop/jsonschema v0.13.0/go.mod h1:ffZ5Km5SWWRAIN6wbDXItl95euhFz2uON45H2qjYt+0= +github.com/itchyny/gojq v0.12.19 h1:ttXA0XCLEMoaLOz5lSeFOZ6u6Q3QxmG46vfgI4O0DEs= +github.com/itchyny/gojq v0.12.19/go.mod h1:5galtVPDywX8SPSOrqjGxkBeDhSxEW1gSxoy7tn1iZY= +github.com/itchyny/timefmt-go v0.1.8 h1:1YEo1JvfXeAHKdjelbYr/uCuhkybaHCeTkH8Bo791OI= +github.com/itchyny/timefmt-go v0.1.8/go.mod h1:5E46Q+zj7vbTgWY8o5YkMeYb4I6GeWLFnetPy5oBrAI= github.com/jackmordaunt/icns/v3 v3.0.1 h1:xxot6aNuGrU+lNgxz5I5H0qSeCjNKp8uTXB1j8D4S3o= github.com/jackmordaunt/icns/v3 v3.0.1/go.mod h1:5sHL59nqTd2ynTnowxB/MDQFhKNqkK8X687uKNygaSQ= github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOlocH6Fxy8MmwDt+yVQYULKfN0RoTN8A= diff --git a/internal/shell/jq.go b/internal/shell/jq.go new file mode 100644 index 0000000000000000000000000000000000000000..ceac574df13c97befa05e8817b91a8d928a96a11 --- /dev/null +++ b/internal/shell/jq.go @@ -0,0 +1,277 @@ +package shell + +import ( + "encoding/json" + "fmt" + "io" + "os" + "strings" + + "github.com/itchyny/gojq" + "mvdan.cc/sh/v3/interp" +) + +const jqUsage = `jq - Go implementation of jq (gojq 0.12.19 builtin) + +Synopsis: + %% echo '{"foo": 128}' | jq '.foo' + +Usage: + jq [OPTIONS] [FILTER] [FILE...] + +Options: + -r, --raw-output output raw strings + -j, --join-output implies -r with no newline delimiter + -c, --compact-output output without pretty-printing + -s, --slurp read all inputs into an array + -n, --null-input use null as input value + -e, --exit-status exit 1 when the last value is false or null + -R, --raw-input read input as raw strings + --arg name value set a string value to a variable + --argjson name value set a JSON value to a variable + -h, --help display this help +` + +// handleJQ implements the jq builtin using gojq. It supports a subset of jq +// flags: -r (raw output), -c (compact output), -s (slurp), -n (null input), +// -e (exit status), -R (raw input), and --arg name value. +// +// Note that this is somewhat of a reimplmentation of the CLI of the glorious +// github.com/itchyny/gojq, and we'd ideally get the CLI exposed upstream to +// avoid this falling out of sync. +func handleJQ(args []string, stdin io.Reader, stdout, stderr io.Writer) error { + var ( + rawOutput bool + compact bool + slurp bool + nullInput bool + exitStatus bool + rawInput bool + joinOutput bool + argNames []string + argValues []any + ) + + // Parse flags and extract the query. + var queryStr string + var fileArgs []string + i := 1 // skip "jq" + for i < len(args) { + arg := args[i] + switch { + case arg == "-h" || arg == "--help": + fmt.Fprint(stdout, jqUsage) + return nil + case arg == "-r" || arg == "--raw-output": + rawOutput = true + case arg == "-j" || arg == "--join-output": + joinOutput = true + rawOutput = true + case arg == "-c" || arg == "--compact-output": + compact = true + case arg == "-s" || arg == "--slurp": + slurp = true + case arg == "-n" || arg == "--null-input": + nullInput = true + case arg == "-e" || arg == "--exit-status": + exitStatus = true + case arg == "-R" || arg == "--raw-input": + rawInput = true + case arg == "--arg": + if i+2 >= len(args) { + fmt.Fprintf(stderr, "jq: --arg requires name and value\n") + return interp.ExitStatus(2) + } + argNames = append(argNames, "$"+args[i+1]) + argValues = append(argValues, args[i+2]) + i += 2 + case arg == "--argjson": + if i+2 >= len(args) { + fmt.Fprintf(stderr, "jq: --argjson requires name and value\n") + return interp.ExitStatus(2) + } + var val any + if err := json.Unmarshal([]byte(args[i+2]), &val); err != nil { + fmt.Fprintf(stderr, "jq: invalid JSON for --argjson %s: %s\n", args[i+1], err) + return interp.ExitStatus(2) + } + argNames = append(argNames, "$"+args[i+1]) + argValues = append(argValues, val) + i += 2 + case arg == "--": + i++ + // Remaining args are file arguments. + for i < len(args) { + fileArgs = append(fileArgs, args[i]) + i++ + } + continue + case strings.HasPrefix(arg, "-") && queryStr != "": + fmt.Fprintf(stderr, "jq: unknown option: %s\n", arg) + return interp.ExitStatus(2) + default: + if queryStr == "" { + queryStr = arg + } else { + fileArgs = append(fileArgs, arg) + } + } + i++ + } + + if queryStr == "" { + queryStr = "." + } + + query, err := gojq.Parse(queryStr) + if err != nil { + fmt.Fprintf(stderr, "jq: %s\n", err) + return interp.ExitStatus(3) + } + + opts := []gojq.CompilerOption{ + gojq.WithEnvironLoader(os.Environ), + } + if len(argNames) > 0 { + opts = append(opts, gojq.WithVariables(argNames)) + } + + code, err := gojq.Compile(query, opts...) + if err != nil { + fmt.Fprintf(stderr, "jq: %s\n", err) + return interp.ExitStatus(3) + } + + // Build input values. + inputs, err := readInputs(stdin, fileArgs, nullInput, rawInput, slurp) + if err != nil { + fmt.Fprintf(stderr, "jq: %s\n", err) + return interp.ExitStatus(2) + } + + var lastFalsy bool + for _, input := range inputs { + iter := code.Run(input, argValues...) + for { + v, ok := iter.Next() + if !ok { + break + } + if err, ok := v.(error); ok { + fmt.Fprintf(stderr, "jq: %s\n", err) + return interp.ExitStatus(5) + } + if exitStatus { + lastFalsy = v == nil || v == false + } + if err := writeValue(stdout, v, rawOutput, compact, joinOutput); err != nil { + return err + } + } + } + + if exitStatus && lastFalsy { + return interp.ExitStatus(1) + } + return nil +} + +// readInputs reads JSON (or raw) input values from stdin or files. +func readInputs(stdin io.Reader, files []string, nullInput, rawInput, slurp bool) ([]any, error) { + if nullInput { + return []any{nil}, nil + } + + var readers []io.Reader + if len(files) > 0 { + for _, f := range files { + file, err := os.Open(f) + if err != nil { + return nil, err + } + defer file.Close() + readers = append(readers, file) + } + } else { + readers = []io.Reader{stdin} + } + + var vals []any + for _, r := range readers { + data, err := io.ReadAll(r) + if err != nil { + return nil, err + } + + if rawInput { + lines := strings.Split(string(data), "\n") + if slurp { + vals = append(vals, strings.Join(lines, "\n")) + } else { + for _, line := range lines { + if line != "" || !slurp { + vals = append(vals, line) + } + } + } + continue + } + + // Decode potentially multiple JSON values from the stream. + dec := json.NewDecoder(strings.NewReader(string(data))) + var streamVals []any + for { + var v any + if err := dec.Decode(&v); err != nil { + if err == io.EOF { + break + } + return nil, fmt.Errorf("parse error: %w", err) + } + streamVals = append(streamVals, v) + } + + if slurp { + vals = append(vals, streamVals) + } else { + vals = append(vals, streamVals...) + } + } + + if len(vals) == 0 { + return []any{nil}, nil + } + return vals, nil +} + +// writeValue writes a single jq output value. +func writeValue(w io.Writer, v any, raw, compact, join bool) error { + if raw { + if s, ok := v.(string); ok { + if _, err := fmt.Fprint(w, s); err != nil { + return err + } + if !join { + _, err := fmt.Fprint(w, "\n") + return err + } + return nil + } + } + + var bs []byte + var err error + if compact { + bs, err = gojq.Marshal(v) + } else { + bs, err = json.MarshalIndent(v, "", " ") + } + if err != nil { + return err + } + if _, writeErr := w.Write(bs); writeErr != nil { + return writeErr + } + _, err = fmt.Fprint(w, "\n") + return err +} diff --git a/internal/shell/shell.go b/internal/shell/shell.go index aabd79f2082ea96e43ac9f4bb4f36f31ad743961..5ca0a25b57d5e80778a6fa95bdd3eb991638db4b 100644 --- a/internal/shell/shell.go +++ b/internal/shell/shell.go @@ -244,6 +244,25 @@ func (s *Shell) blockHandler() func(next interp.ExecHandlerFunc) interp.ExecHand } } +func (s *Shell) builtinHandler() func(next interp.ExecHandlerFunc) interp.ExecHandlerFunc { + return func(next interp.ExecHandlerFunc) interp.ExecHandlerFunc { + return func(ctx context.Context, args []string) error { + if len(args) == 0 { + return next(ctx, args) + } + + // Builtins. + switch args[0] { + case "jq": + hc := interp.HandlerCtx(ctx) + return handleJQ(args, hc.Stdin, hc.Stdout, hc.Stderr) + default: + return next(ctx, args) + } + } + } +} + // newInterp creates a new interpreter with the current shell state func (s *Shell) newInterp(stdout, stderr io.Writer) (*interp.Runner, error) { return interp.New( @@ -307,6 +326,7 @@ func (s *Shell) execStream(ctx context.Context, command string, stdout, stderr i func (s *Shell) execHandlers() []func(next interp.ExecHandlerFunc) interp.ExecHandlerFunc { handlers := []func(next interp.ExecHandlerFunc) interp.ExecHandlerFunc{ + s.builtinHandler(), s.blockHandler(), } if useGoCoreUtils { diff --git a/internal/skills/builtin/jq/SKILL.md b/internal/skills/builtin/jq/SKILL.md new file mode 100644 index 0000000000000000000000000000000000000000..11f5a03957a9be07adaa9702d57e92b3fb2bfb19 --- /dev/null +++ b/internal/skills/builtin/jq/SKILL.md @@ -0,0 +1,97 @@ +--- +name: jq +description: Process, filter, transform, generate, and construct JSON data using the jq command-line processor. Crush includes a built-in jq implementation (powered by gojq) available in the bash tool. Use when the user needs to query, filter, reshape, extract, create, or construct JSON — including API responses, config files, log output, or any structured data. Also use when generating JSON from scratch with jq -n, or when helping users write or debug jq expressions. +--- + +# jq — Built-in JSON Processor + +Crush ships a built-in `jq` command (via `github.com/itchyny/gojq`) available +in the bash tool. No external binary is required. + +## Supported Flags + +| Flag | Description | +|------|-------------| +| `-r`, `--raw-output` | Output strings without quotes | +| `-j`, `--join-output` | Like `-r` but no trailing newline | +| `-c`, `--compact-output` | One-line JSON output | +| `-s`, `--slurp` | Read all inputs into an array | +| `-n`, `--null-input` | Use `null` as input (ignore stdin) | +| `-e`, `--exit-status` | Exit 1 if last output is `false` or `null` | +| `-R`, `--raw-input` | Read each line as a string, not JSON | +| `--arg name value` | Bind `$name` to a string value | +| `--argjson name value` | Bind `$name` to a parsed JSON value | + +File arguments after the filter are also supported: `jq '.foo' file.json`. + +## Differences from Standard jq + +The built-in uses gojq, which is a pure-Go jq implementation. Key +differences: + +- **No object key ordering** — keys are sorted by default; `keys_unsorted` + and `-S` are unavailable. +- **Arbitrary-precision integers** — large integers keep full precision + (addition, subtraction, multiplication, modulo, division when divisible). +- **String indexing** — `"abcde"[2]` returns `"c"`. +- **Not supported** — `--ascii-output`, `--seq`, `--stream`, + `--stream-errors`, `-f`/`--from-file`, `--slurpfile`, `--rawfile`, + `--args`, `--jsonargs`, `input_line_number`, `$__loc__`, some regex + features (backreferences, look-around). +- **YAML** — gojq supports `--yaml-input`/`--yaml-output` but the + built-in does not currently expose these flags. + +## Common Patterns + +Extract a field: +```sh +echo '{"name":"crush"}' | jq '.name' +``` + +Filter an array: +```sh +echo '[1,2,3,4,5]' | jq '[.[] | select(. > 3)]' +``` + +Reshape objects: +```sh +echo '{"first":"Ada","last":"Lovelace"}' | jq '{full: (.first + " " + .last)}' +``` + +Use variables: +```sh +echo '{}' | jq --arg host localhost --argjson port 8080 '{host: $host, port: $port}' +``` + +Slurp multiple JSON values: +```sh +echo '{"a":1}{"b":2}' | jq -s '.' +``` + +Compact output for piping: +```sh +echo '{"a":1}' | jq -c '.a += 1' +``` + +Raw string output: +```sh +echo '["one","two","three"]' | jq -r '.[]' +``` + +Process a file: +```sh +jq '.dependencies | keys' package.json +``` + +Null input for constructing JSON: +```sh +jq -n --arg msg hello '{"message": $msg}' +``` + +## Tips + +- Pipe jq output to other commands: `jq -r '.url' data.json | xargs curl` +- Chain filters with `|` inside the expression, not shell pipes. +- Use `try` to suppress errors on missing keys: `jq 'try .foo.bar'` +- Use `// "default"` for fallback values: `jq '.name // "unknown"'` +- Use `@csv`, `@tsv`, `@base64`, `@html`, `@uri` for format strings. diff --git a/internal/skills/skills_test.go b/internal/skills/skills_test.go index e8ae7b72c8cec704229472c90f110994a55a0380..fc91b5e4541ffb9fa8ef3e958089240a2187c9c8 100644 --- a/internal/skills/skills_test.go +++ b/internal/skills/skills_test.go @@ -424,6 +424,19 @@ func TestDiscoverBuiltin(t *testing.T) { } } require.True(t, found, "crush-config builtin skill not found") + + var foundJQ bool + for _, s := range discovered { + if s.Name == "jq" { + foundJQ = true + require.Equal(t, "crush://skills/jq/SKILL.md", s.SkillFilePath) + require.Equal(t, "crush://skills/jq", s.Path) + require.NotEmpty(t, s.Description) + require.NotEmpty(t, s.Instructions) + require.True(t, s.Builtin) + } + } + require.True(t, foundJQ, "jq builtin skill not found") } func TestDeduplicate(t *testing.T) {