Overview
SkillScan answers the question every agent tool should raise before you install it: what can this actually do on my machine? It's a static scanner I designed and built for agent skills, MCP servers, npm packages, VS Code extensions, repositories and pasted code. One TypeScript engine powers a public web app, a published npm CLI and a GitHub Action, and every finding comes with a severity, its location in the source, a snippet and a fix. JSON and SARIF exports let the same evidence travel into CI and code review.
The Challenge
Agent tools sit across unusually powerful trust boundaries. A genuinely useful skill might ask for shell, filesystem, network and credential access, which is exactly what a malicious package wants too. Keyword matching drowns you in noise, and a reassuring score claims more certainty than static analysis can earn. It had to surface the dangerous combinations fast, without turning a first-pass scan into a safety badge.
Constraints
- •Every public claim must be reproducible from code, tests, fixtures, or the returned scan evidence.
- •The scanner cannot execute untrusted artifacts as part of analysis.
- •A clear result must still communicate static analysis limits and preserve the manual-review gate.
- •Submitted source should not become a retained application dataset; sensitive work needs a local path.
How it works
Problem
A model-generated verdict would be flexible but difficult to reproduce or audit.
Decision
Built a deterministic rule engine whose findings retain category, severity, file, line, snippet, and remediation.
Novel attacks require new rules, and intent cannot be inferred as freely as it could be with a model.
Impact
A developer can inspect why a score changed and test the exact behavior locally.
Problem
Package and extension metadata alone can look benign while executable artifacts contain the actual risk.
Decision
Validate each fetch destination and redirect, bound downloads and decompression, and prioritize manifests and executable entrypoints before file caps. Inspect text by content rather than extension.
Artifact limits and provider availability can make a scan partial.
Impact
Archive padding, disguised scripts, unsafe redirects, and decompression limits have regression coverage. Missing evidence forces review instead of a clean result.
Problem
A numeric score can accidentally read as permission to install.
Decision
Pair the score with Block install, Manual review, or Looks clear, plus an explicit explanation of what a clear result cannot prove.
The interface is more cautious than scanners that optimize for a simple green badge.
Impact
The product keeps the human decision visible at the moment of highest confidence.
Approach
1. Model capabilities, not product labels
Rules focus on consequential behavior—secret access, outbound requests, shell execution, destructive commands, persistence, filesystem reach, unsafe package hooks, and CI permissions—so the same engine can inspect several artifact types.
2. Preserve evidence through the pipeline
Repository and registry adapters normalize files into one scanner input. Findings keep their origin and the API reports partial fetches rather than silently treating missing files as clean.
3. Tune both sides of the error
The release passes 54 regression tests and blocks all 13 committed malicious fixtures. Benign controls check ordinary runtime names, lockfiles, and documentation; these are regression results, not a real-world detection rate.
4. Make the cautious path the usable path
The web interface accepts a URL or pasted source. The published npm CLI scans local files without uploading them, and the GitHub Action shares its exit codes and reports. Explicit exclusions are disclosed; incomplete scans exit with a distinct status even when no finding threshold is selected.
Outcome
SkillScan is live at skillscan.ninetynine.digital, with @maxwellyoung/skillscan 0.1.0 published to npm and a GitHub Action tested against the public package. Hosted CI passes the 54-test regression suite, CLI packaging checks, and web build. The same 29 checks power each entry point, and a partial scan cannot produce a clean verdict. Static rules still miss unfamiliar patterns and do not certify an artifact as safe.
Proof points
29
Deterministic checks
13 / 13
Malicious fixtures
54 / 54
Regression tests
Learnings
- →The most useful scanner output is not a score; it is a short path from suspicion to source evidence.
- →False-positive work is part of the security model because noisy tools teach people to ignore them.
- →Partial evidence must be a first-class result state, not a footnote.
- →A product can be decisive about dangerous behavior while staying honest about what it cannot prove.
Anti-Patterns Avoided
- ×Calling a clean static result proof that an artifact is safe.
- ×Hiding evidence behind a single proprietary risk score.
- ×Executing untrusted packages in order to decide whether they are trustworthy.
- ×Sending local source to a model when deterministic analysis can run first.
Next Iterations
- →Expand the real-incident corpus and measure precision beyond synthetic fixtures.
- →Improve minified-code and transitive-dependency coverage.
- →Evaluate shared rate limits for larger public deployments.
Get In Touch
If you want to talk about similar work, email me.
Contact is the simplest place to start.
Next project
Default Index