How It Works
What Makes It Adaptive
1. Fingerprint-Based Soft-404 Detection
Before scanning, the engine requests 3 random non-existent paths and extracts response attributes (status code, content-type, headers, body hash, content-length ranges). Only attributes stable across all 3 samples become the baseline signature. During scanning, responses matching this signature are discarded as false positives. When an unknown response pattern appears, a 4-strategy wildcard validation (prefix, suffix, extension, middle) confirms whether the discovery is real or a new soft-404 variant - and learns the new pattern.2. Observed Collection System
Four data pools grow continuously during the scan:
Every newly discovered directory is probed with ALL observed values as high-priority tasks. When a new extension is found for the first time, it triggers tasks across ALL known directories.
3. JavaScript Intelligence
Two layers of JS analysis feed endpoints back into the discovery queue:- jstangle (embedded binary): Deobfuscates bundled JS, resolves string concatenation, traces variable assignments, and extracts
fetch()/XMLHttpRequest/$.ajaxcall sites into full HTTP request specs. - Spider extractors: Parse inline
<script>tags and JS string literals for URL patterns.
4. Dynamic Wordlist Growth
Response bodies are tokenized (content-type-aware for HTML, JSON, JS, CSS) to extract candidate words. These feed into the observed name pool and are replayed against every directory.5. Recursive Directory Expansion
When a file is found at/a/b/c/file.txt, the engine extracts /a/, /a/b/, /a/b/c/ as directories to test. Each new directory triggers its own full task set (wordlists + observed + modules).
6. Immutable Build-Output Awareness
SPA frameworks (Next.js/Nuxt/Vite/webpack/CRA) emit content-hash fingerprinted bundles — e.g.main-5cf96b0d57f7f579.js, 938-50137dfb3187f5b2.js, CRA main.073c9bfa.chunk.js — into immutable build-output directories. Discovery detects these fingerprinted filenames (a -/./_-delimited token of ≥8 hex chars containing at least one a-f letter, so plain version numbers and timestamps don’t match) and skips recursive wordlist/observed brute-forcing — plus backup/numeric derivations off the files themselves — into any directory that holds them. This is framework-agnostic (no hardcoded directory list), with a cheap fast-path for the well-known roots (/_next/static/, /_nuxt/, /_app/immutable/). The bundles the spider actually found are still fetched and scanned; a hand-authored /static/ with real assets and genuine endpoints like /api/... stay fully discoverable.
Task Types
Auto-fuzz on low-yield targets. FUZZ fuzzing is normally opt-in (
--fuzz-wordlist, --intensity deep, or a discovery-only run discover). On a full scan it also turns on automatically when the preceding Spidering phase comes up empty-handed — it found very few records, or it bounced off-host to an SSO/login wall. Such sites frequently still expose hidden routes that don’t redirect, so vigolium brute-forces them anyway. The auto-fuzz always targets the original -t host (the off-host SSO/identity-provider domain is excluded from scope), and a console line announces it when it kicks in. Disable with discovery.auto_fuzz_low_yield: false.Deduplication
Multiple layers prevent redundant work during the crawl:- Task-level: FNV-1a hash prevents duplicate task enqueueing
- Request-level: Cache prevents sending the same HTTP request twice
- URL-level: DiskSet tracks processed URLs
- Body-level: Hash prevents re-analyzing identical responses with jstangle
- Directory/file trackers: Prevent re-processing the same discovery
Output deduplication
Before discoveries are saved and handed to the scanner, two more layers run over the collected results:- Exact body dedup: Records are keyed on
host + status + response-body hash. Identical bodies served across different paths collapse to one (the shortest path is kept). The hash covers only the response body — not the raw response — so volatile headers (Date,Set-Cookie) and header ordering don’t defeat it. - Near-identical cluster cap: A backstop for catch-all / SPA targets that answer
200with the same page for every path (where each response differs by a few bytes or words, so the exact hash and soft-404 detection can’t collapse them). Records are grouped into clusters byhost + status + content-type, with body size and word count each within 0.5%; at mostdedup_cluster_caprecords (default 10) are kept per cluster, preferring the shallowest paths. Because the band is relative, small distinct responses (e.g. JSON API endpoints) need a near-exact match to cluster and are never collapsed — only large near-identical pages are capped.
The cluster cap is on by default. Set
discovery.dedup_cluster_cap: 0 to disable it, or raise/lower the value to keep more/fewer representatives per cluster. It trims both the report and the downstream dynamic-assessment workload, since the capped records are never emitted to the scanner.Built-In Modules
YAML-configured modules trigger specialized tasks when matching directories are found:Supporting Systems
WAF/CDN block warning. The first time scan traffic to a host comes back as a WAF/CDN block (a captcha, bot-detection, or challenge page), Vigolium prints a one-time-per-host
[waf-block-detected] warning to stderr (and the session log) noting the edge is filtering traffic and results for that host may be incomplete. Detection runs on the shared HTTP requester, so it fires across every phase (discovery, spidering, dynamic-assessment); the dedup means at most one line per host regardless of how many requests are blocked. Suppressed under --silent.Integration with Vigolium
Deparos runs as an input source (DeparosDiscoverySource) in the scanning pipeline. Each discovery is converted to an httpmsg.HttpRequestResponse and fed to the executor as a work item - where it flows through active and passive vulnerability scanning modules.