Overview
SkillScan is an evidence-first static scanner for agent skills, MCP servers, npm packages, VS Code extensions, GitHub Actions, repositories, and pasted code. It gives each finding a severity, source location, snippet, and remediation note, then turns the combined evidence into a plain-language install verdict. The product is deliberately deterministic: its claims can be traced to visible rules and committed fixtures rather than an opaque model judgment.
The Challenge
Agent tooling crosses unusually powerful trust boundaries. A useful skill may ask for shell, filesystem, network, and credential access—the same capabilities a malicious package needs. Simple keyword matching produces noise, while a reassuring score can imply more certainty than static analysis earns. The product had to surface dangerous combinations quickly without turning a first-pass scan into a safety badge.
Constraints
- •Every public claim must be reproducible from code, tests, fixtures, or the returned scan evidence.
- •The scanner cannot execute untrusted artifacts as part of analysis.
- •A clear result must still communicate static analysis limits and preserve the manual-review gate.
- •Submitted source should not become a retained application dataset; sensitive work needs a local path.
Decision Log
Problem
A model-generated verdict would be flexible but difficult to reproduce or audit.
Decision
Built a deterministic rule engine whose findings retain category, severity, file, line, snippet, and remediation.
Tradeoff
Novel attacks require new rules, and intent cannot be inferred as freely as it could be with a model.
Impact
A developer can inspect why a score changed and test the exact behavior locally.
Problem
Package and extension metadata alone can look benign while executable artifacts contain the actual risk.
Decision
Fetch and unpack supported npm and OpenVSX artifacts, then scan their relevant source and manifest files.
Tradeoff
Artifact limits and provider availability can make a scan partial.
Impact
Results describe the code being installed, with explicit warnings when evidence is incomplete.
Problem
A numeric score can accidentally read as permission to install.
Decision
Pair the score with Block install, Manual review, or Looks clear, plus an explicit explanation of what a clear result cannot prove.
Tradeoff
The interface is more cautious than scanners that optimize for a simple green badge.
Impact
The product keeps the human decision visible at the moment of highest confidence.
Approach
1. Model capabilities, not product labels
Rules focus on consequential behavior—secret access, outbound requests, shell execution, destructive commands, persistence, filesystem reach, unsafe package hooks, and CI permissions—so the same engine can inspect several artifact types.
2. Preserve evidence through the pipeline
Repository and registry adapters normalize files into one scanner input. Findings keep their origin and the API reports partial fetches rather than silently treating missing files as clean.
3. Tune both sides of the error
Thirteen committed malicious fixtures exercise known hostile combinations. A separate 300-file local corpus catches rules that punish ordinary application and documentation code.
4. Make the cautious path the usable path
The web interface accepts a URL or direct source, returns a quick verdict, and keeps detailed categories expandable. The same scanner is available through the repository and a reusable Codex skill for local-first review.
Outcome
SkillScan now runs 29 deterministic checks across direct code, agent skills, repositories, package artifacts, extensions, and CI workflows. Its 13 adversarial tests pass, its public method and privacy boundary are visible in the product, and every result keeps manual review in the loop.
Proof points
29
Deterministic checks
13 / 13
Malicious fixtures
300 files
False-positive corpus
Learnings
- →The most useful scanner output is not a score; it is a short path from suspicion to source evidence.
- →False-positive work is part of the security model because noisy tools teach people to ignore them.
- →Partial evidence must be a first-class result state, not a footnote.
- →A product can be decisive about dangerous behavior while staying honest about what it cannot prove.
Anti-Patterns Avoided
- ×Calling a clean static result proof that an artifact is safe.
- ×Hiding evidence behind a single proprietary risk score.
- ×Executing untrusted packages in order to decide whether they are trustworthy.
- ×Sending local source to a model when deterministic analysis can run first.
Next Iterations
- →Export machine-readable findings for CI and code-review workflows.
- →Expand the real-incident corpus and measure rule precision over time.
- →Improve branch, monorepo, minified-code, and dependency-chain coverage.
Get In Touch
If you want to talk about similar work, email me.
Contact is the simplest place to start.
Next project
Default Index