1# Soft Serve Development Documentation
  2
  3## Important Instructions
  4
  5- Use the gopls MCP tooling as much as appropriate. It provides dense, highly-relevant symbolic information. Before making any edit, use the workspace tool, check the relevant APIs and references if necessary, and _then_ proceed with edits.
  6
  7## Code Style Guidelines
  8
  9### Formatting
 10
 11- **Go files**: Use tabs for indentation
 12- **Other files**: Use 2 spaces for indentation
 13- Use LF line endings
 14- Insert final newline
 15- Trim trailing whitespace
 16- UTF-8 character encoding
 17
 18### Go Code Style
 19
 20- Format with `gofumpt` and `goimports` (configured in `.golangci.yml`)
 21- Use standard Go naming conventions (PascalCase for exported, camelCase for unexported)
 22- Wrap exported types with helper methods rather than exposing underlying library types directly
 23- Implement standard interfaces where appropriate (e.g., `sort.Interface`)
 24
 25### Comments
 26
 27- Exported functions, types, and methods must have godoc comments
 28- Comments should end with proper punctuation (enforced by `godot` linter)
 29- Start comments with the name of the item being documented
 30
 31### Error Handling
 32
 33- Wrap errors with context using `wrapcheck` patterns
 34- Check all error returns (enforced by various linters)
 35- Close resources properly (enforced by `bodyclose`, `sqlclosecheck`, `rowserrcheck`)
 36
 37## How to Write Unit Tests
 38
 39### Tooling
 40
 41- Unit tests: Go's built-in `testing` package with `matryer/is` for assertions
 42- Integration tests: `testscript` framework (see `testscript/` directory)
 43
 44### Running Tests
 45
 46```bash
 47# Run all tests
 48go test ./...
 49
 50# Run tests for specific package
 51go test ./pkg/backend
 52
 53# Run integration tests
 54cd testscript && go test -v
 55
 56# Run with PostgreSQL
 57SOFT_SERVE_DB_DRIVER=postgres \
 58SOFT_SERVE_DB_DATA_SOURCE="postgres://postgres:postgres@localhost/postgres?sslmode=disable" \
 59go test ./...
 60
 61# Disable race detection (for slower systems)
 62SOFT_SERVE_DISABLE_RACE_CHECKS=1 go test ./...
 63```
 64
 65### Writing Unit Tests
 66
 67- Place test files alongside the code being tested (e.g., `tree_test.go` next to `tree.go`)
 68- Use `_test.go` suffix for test files
 69- Use `matryer/is` for simple, readable assertions
 70- Mock databases using transactions that rollback in deferred functions
 71- Test both SQLite and PostgreSQL code paths for database-related changes
 72- Use `pkg/test/test.go` helpers for fixtures (e.g., `test.RandomPort()`)
 73
 74### Writing Integration Tests
 75
 76- Add `.txtar` files to `testscript/testdata/`
 77- Each test spins up a complete Soft Serve instance with randomized ports
 78- Available commands: `soft`, `usoft`, `git`, `ugit`, `ui`, `uui`, `curl`, `mkfile`, `readfile`, `envfile`
 79- Use `ensureserverrunning`/`ensureservernotrunning` to wait for server state
 80
 81## Development Commands and Environment
 82
 83### Build and Development Commands
 84
 85#### Building
 86
 87```bash
 88go build -o soft ./cmd/soft
 89```
 90
 91#### CSS Cache-Busting
 92
 93When editing CSS files in `pkg/web/static/`, increment the version query parameter in `pkg/web/templates/base.html`:
 94- `overrides.css?v=1` → `overrides.css?v=2`
 95- `syntax.css?v=1` → `syntax.css?v=2`
 96
 97### Environment Variables
 98
 99All configuration can be set via environment variables with the `SOFT_SERVE_` prefix:
100- `SOFT_SERVE_DATA_PATH`: Data directory location (default: `data`)
101- `SOFT_SERVE_CONFIG_LOCATION`: Custom config file path
102- `SOFT_SERVE_INITIAL_ADMIN_KEYS`: SSH keys for initial admin (setup only, newline-separated)
103- `SOFT_SERVE_SSH_LISTEN_ADDR`: SSH server address (default: `:23231`)
104- `SOFT_SERVE_HTTP_LISTEN_ADDR`: HTTP server address (default: `:23232`)
105- `SOFT_SERVE_GIT_LISTEN_ADDR`: Git daemon address (default: `:9418`)
106- `SOFT_SERVE_DB_DRIVER`: Database driver (`sqlite` or `postgres`)
107- `SOFT_SERVE_DB_DATA_SOURCE`: Database connection string
108- `SOFT_SERVE_NO_COLOR`: Disable color output (set to any truthy value)
109- `SOFT_SERVE_DEBUG`: Enable debug logging (enables auth callback logging)
110- `SOFT_SERVE_VERBOSE`: Enable verbose logging (logs DB queries, SSH envs/keys; requires `SOFT_SERVE_DEBUG` to also be set)
111
112Environment variables always override config file settings via `cfg.ParseEnv()`.
113
114## Advanced Features and Storage
115
116### TUI Architecture
117
118The TUI (`pkg/ui/`) is built with Bubble Tea v2 and organized as:
119- `pages/`: Main views (repo browser, file viewer, log viewer, refs viewer)
120- `components/`: Reusable UI elements (code viewer, tabs, header, footer, viewport)
121- `common/`: Shared state and utilities including `common.Common` struct passed to all components
122- `styles/`: Lipgloss style definitions
123- `keymap/`: Keyboard shortcuts
124
125The TUI runs in PTY sessions and uses the BubbleTea middleware (`bm.MiddlewareWithProgramHandler`).
126
127### Git Protocol Version
128
129The git daemon and SSH/HTTP handlers support git protocol v2. The protocol version is negotiated by the client:
130- HTTP: Client sends `Git-Protocol` header (e.g., `Git-Protocol: version=2`)
131- SSH: Client sets `GIT_PROTOCOL` in session environment
132- Git daemon: Client specifies protocol in pktline extra parameters
133
134The server passes the `GIT_PROTOCOL` environment variable through to git processes.
135
136## Architecture and Core Components
137
138### Project Overview
139
140Soft Serve is a self-hostable Git server with SSH/HTTP/Git protocol support, featuring a terminal UI (TUI), Git LFS support, webhooks, and access control. It's built with Go and uses Bubble Tea for the TUI.
141
142### Core Components
143
1441. **Backend** (`pkg/backend/`): Central orchestrator that manages all business logic. The Backend struct ties together the database, store, config, cache, and task manager. It's the primary interface for repository, user, and settings operations. Key responsibilities:
145   - Repository CRUD operations with cache management
146   - User authentication and authorization
147   - Hook execution (implements `hooks.Hooks` interface)
148   - LFS object management
149   - Webhook event creation
150
1512. **Store** (`pkg/store/`): Database abstraction layer implementing repository, user, collaborator, LFS, access token, webhook, and settings persistence. The actual implementation lives in `pkg/store/database/`. This is a pure data layer with no business logic.
152
1533. **Database** (`pkg/db/`): Low-level database wrapper around `sqlx` supporting both SQLite (default) and PostgreSQL. Provides transaction management and migrations (in `pkg/db/migrate/`). Uses context-aware logging when verbose mode is enabled.
154
1554. **Config** (`pkg/config/`): Configuration management supporting both YAML files and environment variables. All `SOFT_SERVE_*` env vars override config file settings. Config is immutable once loaded and passed through context.
156
1575. **Git Operations** (`git/` package): Git command execution and repository manipulation using `aymanbagabas/git-module`. Handles commits, tags, references, patches, trees, and server-side hooks. This is a separate top-level package, not under `pkg/`.
158
159### Server Protocols
160
1611. **SSH Server** (`pkg/ssh/`): Provides both interactive TUI access and git protocol operations. Uses a middleware chain pattern:
162   - `AuthenticationMiddleware`: Validates SSH public key fingerprint matches approved key
163   - `ContextMiddleware`: Injects config, backend, store, DB, logger into session context
164   - `CommandMiddleware`: Routes non-PTY sessions to CLI commands using Cobra
165   - `LoggingMiddleware`: Logs connection, commands, and duration
166   
167   CLI commands live in `pkg/ssh/cmd/` as Cobra commands.
168
1692. **HTTP Server** (`pkg/web/`): Serves git HTTP protocol, Git LFS endpoints, and provides basic auth using access tokens. Uses Gorilla Mux for routing with these controllers:
170   - `GitController`: Git smart HTTP protocol
171   - `HealthController`: Health check endpoint
172   - Also uses middleware chain for compression, recovery, logging, and context
173
1743. **Git Daemon** (`pkg/daemon/`): Native git:// protocol support for anonymous read access. Implements connection pooling with configurable max connections and timeouts.
175
1764. **Stats Server**: Prometheus metrics endpoint for monitoring.
177
178### Access Control
179
180The `pkg/access/` package implements a four-level access system defined as iota constants:
181- `NoAccess` (0): Denies all access
182- `ReadOnlyAccess` (1): Can read public repos
183- `ReadWriteAccess` (2): Full control of repos
184- `AdminAccess` (3): Full server control
185
186Anonymous access is controlled by two settings:
187- `allow-keyless`: Whether to allow connections without SSH keys (affects HTTP/git:// protocols)
188- `anon-access`: Default access level for anonymous users (defaults to read-only)
189
190Access evaluation order:
1911. Admin users (via `initial_admin_keys` or user.is_admin flag) → AdminAccess
1922. Repository owner (user_id matches repo.user_id) → ReadWriteAccess  
1933. Explicit collaborators → their assigned level
1944. Authenticated users → read-only for public repos
1955. Anonymous users → `anon-access` setting (if `allow-keyless` is true)
196
197## Data Flow and Git Hooks
198
199### Key Data Flow
200
2011. **Context Propagation**: Config, Backend, Store, and DB are passed through `context.Context` using type-safe context helpers (e.g., `config.FromContext(ctx)`). Each package defines its own context key as an unexported type to prevent collisions.
202
2032. **Initialization Sequence** (see `cmd/cmd.go:InitBackendContext`):
204   ```
205   Config → DB → Store → Backend → Context
206   ```
207   
208   This happens in `InitBackendContext` which is used as a PersistentPreRunE hook in commands that need backend access.
209
2103. **Request Flow**:
211   ```
212   SSH/HTTP/Git Protocol → Middleware → Backend → Store → Database
213                                   ↓
214                                Cache (LRU, 1000 items)
215   ```
216
2174. **SSH Command Flow**:
218   ```
219   SSH Session → CommandMiddleware → Cobra Command → Backend Method → Store → DB
220   ```
221   
222   Non-PTY sessions are routed to CLI commands. PTY sessions go to the BubbleTea TUI.
223
224### Git Hooks System
225
226Git hooks use a clever two-tier system:
227
2281. **Hook Dispatcher** (generated in each repo's `hooks/` directory): Bash script that reads stdin, then calls all executables in `hooks/<hookname>.d/` directory, passing both stdin and arguments. Exits on first non-zero exit code.
229
2302. **Soft Serve Hook** (in `hooks/<hookname>.d/soft-serve`): Calls back into the `soft` binary via `soft hook <hookname>` command, which invokes Backend methods that implement the `hooks.Hooks` interface.
231
232Hook generation (`pkg/hooks/gen.go`):
233- Creates hook dispatcher at `hooks/<hookname>` 
234- Creates `hooks/<hookname>.d/` directory
235- Generates `hooks/<hookname>.d/soft-serve` that calls `${SOFT_SERVE_BIN_PATH} hook <hookname>`
236- Supports: pre-receive, update, post-receive, post-update
237
238This design allows:
239- Multiple hooks per event (run in alphabetical order from `.d/` directory)
240- User-defined hooks alongside Soft Serve's built-in hooks in each repo's `.d/` directory
241
242### Hook Environment Variables
243
244Git hooks receive environment variables set in the git daemon, SSH middleware, or HTTP handler:
245- `SOFT_SERVE_REPO_NAME`: Repository name (without .git)
246- `SOFT_SERVE_REPO_PATH`: Full path to repository  
247- `SOFT_SERVE_PUBLIC_KEY`: The authorized SSH key (in authorized_keys format)
248- `SOFT_SERVE_USERNAME`: Username for authenticated users
249- `SOFT_SERVE_BIN_PATH`: Path to soft binary (defaults to "soft" if not set)
250- `SOFT_SERVE_HOST`: Hostname from git protocol handshake
251- `SOFT_SERVE_LOG_PATH`: Path to hooks log file
252- `GIT_DIR`: Standard git variable
253- `GIT_PROTOCOL`: Git protocol version and capabilities
254
255The backend reads `SOFT_SERVE_PUBLIC_KEY` and `SOFT_SERVE_USERNAME` to identify users in hooks.
256
257## Database Schema and Migrations
258
259### Database Schema
260
261Key tables (see `pkg/db/models/`):
262- `users`: User accounts with admin flags, created_at, updated_at
263- `public_keys`: SSH public keys linked to users (one-to-many)
264- `repos`: Repository metadata with user_id owner (required)
265- `collabs`: Repository collaborators with access levels (many-to-many users↔repos)
266- `settings`: Server-wide settings (key-value store)
267- `lfs_objects`: Git LFS object metadata linked to repos
268- `lfs_locks`: Git LFS file locks with path and user
269- `access_tokens`: User access tokens for HTTP auth with optional expiry
270- `webhooks`: Repository webhook configurations with secret and events
271- `webhook_events`: Webhook event type subscriptions (many-to-many webhooks↔events)
272- `webhook_deliveries`: Webhook delivery attempts with request/response logs
273
274Important schema notes:
275- `repos.user_id` is `NOT NULL` (legacy repos are assigned to first admin during migration)
276- `public_keys` has a unique constraint on the key content (prevents duplicate keys)
277- `collabs` has a composite unique key on (user_id, repo_id)
278- `lfs_objects` has a composite unique key on (oid, repo_id)
279- `lfs_locks` has a composite unique key on (repo_id, path)
280- `webhook_events` has a composite unique key on (webhook_id, event)
281- Foreign key constraints are enabled and critical for data integrity
282
283### Database Migrations
284
285Migrations live in `pkg/db/migrate/` with separate SQL files for SQLite and PostgreSQL:
286- `NNNN_migration_name_sqlite.up.sql` / `.down.sql`
287- `NNNN_migration_name_postgres.up.sql` / `.down.sql`
288- Migration number prefixes must be unique and sequential
289- Go files (`NNNN_migration_name.go`) can handle data migrations
290
291The migration system in `migrate.go` automatically runs pending migrations on startup.
292
293### Transaction Patterns
294
295Always use `db.TransactionContext` for database operations:
296
297```go
298err := d.db.TransactionContext(ctx, func(tx *db.Tx) error {
299    // All operations use tx, not db
300    return d.store.CreateRepo(ctx, tx, name, ...)
301})
302return db.WrapError(err) // Wraps db errors as proto errors
303```
304
305The transaction automatically:
306- Commits on `nil` return
307- Rolls back on error return
308- Handles `sql.ErrTxDone` gracefully
309
310### Common Gotchas
311
312#### SQLite Foreign Keys
313
314SQLite requires `_pragma=foreign_keys(1)` in the DSN. The default config includes this, but custom configs must include it:
315```
316soft-serve.db?_pragma=busy_timeout(5000)&_pragma=foreign_keys(1)
317```
318
319Without this, cascade deletes won't work and referential integrity is not enforced.
320
321#### Transaction Context
322
323When creating database transactions, use `TransactionContext(ctx, fn)` to properly propagate context with logger and other values. The `Transaction(fn)` method uses `context.Background()` which loses context values.
324
325## Development Workflow Guides
326
327### Adding a New SSH Command
328
3291. Create command file in `pkg/ssh/cmd/` (e.g., `mycommand.go`)
3302. Define Cobra command with `RunE` function
3313. Use `checkIfReadable`, `checkIfCollab`, or `checkIfAdmin` for authorization
3324. Add command to `CommandMiddleware` in `pkg/ssh/middleware.go`
3335. Test with testscript in `testscript/testdata/`
334
335### Adding a New Database Table
336
3371. Create model in `pkg/db/models/`
3382. Create migration files in `pkg/db/migrate/` (both SQLite and Postgres)
3393. Add store interface methods to `pkg/store/` and implement in `pkg/store/database/`
3404. Update `pkg/backend/` to expose operations
3415. Test migrations with both databases
342
343### Adding a New Webhook Event
344
3451. Define event type in `pkg/webhook/event.go`
3462. Create event constructor in `pkg/webhook/` (e.g., `NewMyEvent`)
3473. Call `webhook.SendEvent(ctx, event)` where event should fire
3484. Add test in `testscript/testdata/`
349
350### Working with Git Operations
351
352All git operations should:
353- Use the `git` package, not exec.Command directly
354- Set appropriate timeouts (`CommandOptions.Timeout`)
355- Pass context for cancellation
356- Handle `git.ErrInvalidRepo` for non-existent repos
357- Use `git.EnsureWithin()` to prevent directory traversal attacks
358
359## Patterns and Conventions
360
361### Context Usage
362
363Always retrieve dependencies from context rather than passing them explicitly:
364```go
365cfg := config.FromContext(ctx)
366be := backend.FromContext(ctx)
367dbx := db.FromContext(ctx)
368store := store.FromContext(ctx)
369logger := log.FromContext(ctx)
370user := proto.UserFromContext(ctx)
371```
372
373### Repository Naming
374
375- User-facing repo names omit the `.git` extension (e.g., `dotfiles`)
376- On-disk repos are always `<name>.git` (e.g., `dotfiles.git`)
377- Repos are stored in `<data_path>/repos/<name>.git`
378- Nested repos are supported (e.g., `user/project.git` → `repos/user/project.git`)
379- Always use `utils.SanitizeRepo()` before using user input as repo name
380- Always validate with `utils.ValidateRepo()` before creating/renaming repos
381
382### Cache Invalidation
383
384The backend maintains an LRU cache (1000 items) of repository objects. **Critical**: Always invalidate cache after mutations:
385
386```go
387// Delete cache before Db transaction
388d.cache.Delete(name)
389
390return d.db.TransactionContext(ctx, func(tx *db.Tx) error {
391    // ... database operations
392})
393```
394
395Cache invalidation happens in:
396- `DeleteRepository`: After deletion
397- `RenameRepository`: For old name
398- `SetDescription`, `SetPrivate`, `SetHidden`, `SetProjectName`: Before update
399
400Failure to invalidate cache will cause stale data to be served until cache eviction.
401
402### SSH Authentication Flow
403
4041. Client connects with public key
4052. `PublicKeyHandler` checks if key belongs to a user, stores fingerprint in `Permissions.Extensions["pubkey-fp"]`
4063. If all public keys fail, `KeyboardInteractiveHandler` checks `allow-keyless` setting
4074. `AuthenticationMiddleware` validates that the public key fingerprint in context matches the actual auth key (gossh doesn't guarantee this)
408
409This double-check prevents a security issue where gossh's PublicKeyCallback doesn't guarantee the approved key is the one actually used.
410
411### Task Manager for Long-Running Operations
412
413Repository imports run asynchronously using `pkg/task`:
414
415```go
416tid := "import:" + name
417d.manager.Add(tid, func(ctx context.Context) error {
418    // Long-running operation
419    return nil
420})
421
422done := make(chan error, 1)
423go d.manager.Run(tid, done)
424err := <-done
425```
426
427Tasks are identified by string ID and automatically cleaned up on completion. The manager respects context cancellation.
428
429### Repository Path Construction
430
431Always use `backend.repoPath()` or construct paths consistently:
432```go
433// Correct
434rp := filepath.Join(cfg.DataPath, "repos", repoName+".git")
435
436// Wrong - missing .git suffix
437rp := filepath.Join(cfg.DataPath, "repos", repoName)
438```
439
440The `.git` suffix is mandatory for git to recognize bare repositories.