From 54b848d60630c3c6d9eadd6c5839dae8af8f1274 Mon Sep 17 00:00:00 2001 From: Amolith Date: Tue, 17 Feb 2026 10:07:43 -0700 Subject: [PATCH] feat: add auditing-repositories skill --- README.md | 19 +- skills/auditing-repositories/SKILL.md | 164 ++++++++++++++++++ .../references/llm-security.md | 53 ++++++ 3 files changed, 232 insertions(+), 4 deletions(-) create mode 100644 skills/auditing-repositories/SKILL.md create mode 100644 skills/auditing-repositories/references/llm-security.md diff --git a/README.md b/README.md index 4b7834af25158bdf017832796cc94276ac7d0e96..32f50f25cb33b4993968499888406869e366c3b7 100644 --- a/README.md +++ b/README.md @@ -226,6 +226,17 @@ Token breakdown: Total: 14533 tokens ⚠️ Body exceeds recommended 5000 token budget! +=== auditing-repositories === + +Token breakdown: + Name: 20 tokens + Description: 140 tokens + Body: 2840 tokens (156 lines) + References: + llm-security.md 1138 tokens + ─────────────────────────────────────────────── + Total: 4138 tokens + === authoring-skills === Token breakdown: @@ -438,10 +449,10 @@ Token breakdown: SUMMARY ============================================================ -Skills: 21 -Metadata: 3054 tokens -Combined bodies: 55795 tokens -Overall: 179935 tokens +Skills: 22 +Metadata: 3214 tokens +Combined bodies: 58635 tokens +Overall: 184073 tokens Validation errors: 0 Largest skills (by total tokens): diff --git a/skills/auditing-repositories/SKILL.md b/skills/auditing-repositories/SKILL.md new file mode 100644 index 0000000000000000000000000000000000000000..ddbd319a737891b479bd7bcd2624db06c6fe828c --- /dev/null +++ b/skills/auditing-repositories/SKILL.md @@ -0,0 +1,164 @@ +--- +name: auditing-repositories +description: Audits open source repositories for security, privacy, and unexpected behavior. Use when asked to audit, review, or assess a repo or codebase for vulnerabilities, privacy or telemetry concerns, suspicious behavior, dependency or supply chain risk, or OSS due diligence. +license: AGPL-3.0-or-later +metadata: + author: Amolith +--- + +Be objective and evidence-based — acknowledge what's well-designed, and don't invent issues to appear thorough. Each finding needs a source→sink path, unsafe default, or concrete misconfiguration as evidence. + +## Guardrails + +- **Prefer static inspection.** Don't execute untrusted artifacts. If running anything (linters, `npm audit`, `cargo audit`), document exactly what and why. +- **Avoid speculative findings.** Each issue needs concrete evidence, not hypotheticals. + +## Workflow + +1. **Understand the project**: Read README, docs, and config to learn what the software claims to do +2. **Identify the stack**: Languages, frameworks, build systems, deployment targets +3. **Map attack surface**: Locate entry points and trust boundaries — CLI args, HTTP routes, RPC handlers, webhooks, config/env, file format parsers, plugins, deserialization +4. **Analyze systematically**: Work through each analysis area below, starting with highest risk +5. **Check dependencies**: Review dependency manifests for known vulnerabilities and supply chain risk +6. **Review build and release pipeline**: Install scripts, CI workflows, build-time downloads, release signing +7. **LLM-specific checks**: If the project involves LLMs, AI assistants, or MCP servers, DO NOT FORGET to apply [references/llm-security.md](references/llm-security.md) +8. **Write the report**: Use the output format below; rate each finding + +Assess from both an external attacker and malicious insider perspective. Focus on practical, realistic threats — not theoretical edge cases requiring unrealistic exploitation scenarios. + +## Analysis Areas + +### Correctness / Unexpected Behavior + +- Does the software function as described? Any undocumented behaviors? +- Verify claims in README/docs against actual implementation + +### Security + +- Authentication/authorization weaknesses +- Input validation and injection (SQL, command, XSS, CSRF, SSRF, path traversal) +- Insecure cryptographic implementations or misuse +- Unsafe deserialization (YAML, JSON, pickle), archive extraction (zip-slip), template engines +- Race conditions and concurrency issues +- Hardcoded credentials or secrets +- Insecure default configurations +- Memory safety concerns (C/C++/Rust unsafe blocks): bounds checks, use-after-free, unsafe FFI + +### Privacy + +- What data is collected, how it's stored, and where it's sent +- Data retention policies and third-party sharing +- Tracking mechanisms: cookies, persistent identifiers, telemetry, analytics SDKs +- Whether data collection is disclosed to users and proportionate to functionality + +### Network Communications + +- All endpoints contacted during operation +- Authentication and encryption of connections +- Types of data transmitted; interception risk +- Whether each communication is necessary for core functionality +- Unexpected or suspicious outbound connections + +### Permissions and Sandboxing + +- System permissions requested (filesystem, network, hardware) +- Whether permissions align with stated functionality +- API scopes and integration breadth +- Running as root, setuid, container escape potential +- Background services, startup hooks, file permission modes +- Temp directory safety, symlink attacks + +### Code Quality (Security-Relevant) + +- Error handling: fail-open vs fail-closed +- Logging of sensitive data; secret leakage in logs or crash dumps +- Debug endpoints exposing config or secrets +- Input sanitization at boundaries +- Secure coding patterns (or lack thereof) + +### Dependencies + +- Known vulnerabilities in direct and transitive dependencies +- Supply chain risks (unmaintained, single-maintainer, typosquat) +- Unnecessary dependencies increasing attack surface +- Dependency pinning and verification practices + +### Build, Release, and Maintenance + +- Install scripts (`postinstall`, `Makefile`, `setup.py`, cargo build scripts, GitHub Actions) +- Downloading binaries at build time +- Signed releases, checksums, provenance +- Versioning policy, security patch cadence, dependency update automation + +## Risk Rating + +For each finding, provide: + +- **Severity**: Critical / High / Medium / Low / Informational +- **Impact**: What breaks or leaks if exploited +- **Likelihood**: How plausible is exploitation in practice +- **Complexity**: How difficult is exploitation (Low / Medium / High) +- **Affected components**: File paths with short code snippets as evidence + +## Output Format + +```markdown +# Due diligence for [Project Name] + +## Executive summary + +[2–3 sentences: overall posture, most significant findings] + +## Scope and methodology + +[What was analyzed, what approach was taken] + +## Findings + +### 🔴 Critical / High + +#### [Finding title] + +**Severity:** Critical | **Likelihood:** High | **Complexity:** Low +**Location:** `path/to/file:42` + +**Description** + +[What the issue is, with evidence] + +**Impact** + +[What an attacker could achieve] + +**Recommendation** + +[Specific fix] + +### 🟠 Medium + +[Same structure] + +### 🟡 Low / Informational + +[Same structure] + +## Dependency summary + +| Dependency | Version | Known CVEs | Risk | +| ----------- | ------- | ------------- | ------ | +| example-lib | 1.2.3 | CVE-XXXX-YYYY | Medium | + +## Recommendations + +[Prioritized list of actions] + +## Strengths + +[What's done well — acknowledge good security practices] + +## Not assessed + +[Areas outside the scope of this audit or not reachable via static analysis] +``` + +Omit empty severity sections. If the project is well-built and secure, say so clearly. diff --git a/skills/auditing-repositories/references/llm-security.md b/skills/auditing-repositories/references/llm-security.md new file mode 100644 index 0000000000000000000000000000000000000000..be6ca3114a869b35c695de2f586ce10798475548 --- /dev/null +++ b/skills/auditing-repositories/references/llm-security.md @@ -0,0 +1,53 @@ +## Prompt Injection + +- Examine all text entering LLM context windows: README, docs, comments, string literals, tool descriptions, parameter descriptions +- Identify hidden Unicode characters, zero-width spaces, or encoded payloads +- Check for instructions attempting to hide information from users or manipulate assistant behavior + +## Data Exfiltration + +- Markdown image links with suspicious query parameters (can leak context via URL) +- External resource loading that could exfiltrate conversation history or system prompts +- Webhook configurations and logging that capture LLM interactions +- Image-based prompt injection techniques +- Outbound HTTP calls triggered by user-controlled content (SSRF via prompt) + +## Context Window Pollution + +- Is all injected content necessary for stated functionality? +- Excessive or suspicious prose in tool/function descriptions +- Hidden instructions in metadata or configuration files + +## System Access Patterns + +- File system access to sensitive paths (browser profiles, SSH keys, credential stores) +- Environment variable reads that could capture secrets +- System command execution and privilege escalation vectors +- Temporary file creation with sensitive content and cleanup practices +- Whether the tool requests broader filesystem or network access than its stated purpose requires + +## Tool and Function Analysis + +- Do tools perform only their stated functions? +- Hidden instructions embedded in tool or parameter descriptions +- Return values formatted to inject instructions into downstream context +- Error messages that disclose system internals or file paths +- Review error messages and return value formatting for context manipulation + +## Retrieval and Tool-Output Poisoning (RAG) + +- Poisoning of retrieval corpora (docs, issues, wiki) that gets injected into context +- Trusting tool output or search results as instructions rather than data +- "Data-to-instructions" boundary: ensure retrieved text is treated as data, not policy +- Cross-tool injection: output from one tool containing instructions that another tool would execute + +## Tool Capability Scoping + +- Are tool definitions minimal and least-privilege? +- Structured schemas preferred over free-form tool calls + +## Prompt/Completion Retention + +- Where are prompts and completions stored? (telemetry, remote logging, analytics) +- Redaction practices for secrets and PII in logs +- Multi-tenant isolation if the agent is hosted or shared