Skip to main content
Spitolas is a state-machine-driven web crawler that drives a real Chromium browser to discover application states through user-like interactions (clicking, form filling, iframe traversal). All resulting HTTP traffic is captured at the CDP level and fed into vigolium’s scanning pipeline.

​
How It Works

​
Core Concepts

​
States and the State Graph

A State is a DOM snapshot identified by SHA256(strippedDOM)[:16]. The State Graph is a directed graph where nodes are states and edges are actions that caused transitions. Navigation between states uses Dijkstra’s shortest path (with Yen’s K-shortest as fallback). Near-duplicate detection uses normalized Levenshtein distance (threshold: 10%). For large DOMs (>10K chars), a sampling-based distance is used for performance.

​
Actions and Candidate Elements

Candidate clickable elements are discovered via CSS selectors (a, button, [onclick], [role=button], input[type=submit], framework-specific bindings like [ng-click], [v-on:click], etc.). Each candidate becomes an Eventable (graph edge) once fired, linking a source state to a target state with an event type (click, hover, enter).

​
Fragments (Visual Page Segmentation)

Pages are decomposed into Fragments - DOM regions identified by XPath, bounding box, subtree size, and content hash. Two modes:
  • Landmark (default): fast DOM-based extraction
  • VIPS: vision-based page segmentation with multi-pass decreasing thresholds
Fragment comparison uses the APTED tree edit distance algorithm. The Fragment Manager tracks exploration status across duplicate/equivalent fragments and uses candidate influence scoring to prioritize which states to explore next.

​
Form Handling

The Form Handler detects and fills forms with smart value generation:
  • Field-name-aware values (email, password, phone, URL, etc.)
  • Constraint-aware generation (respects pattern, min/max, minlength/maxlength)
  • Pairwise fallback when filling all inputs at once fails
  • File upload support with type-aware file selection
Filling a form only exercises its inputs — the endpoints behind those forms and links stay invisible until something submits them. After a page settles, the crawler runs a set of bounded, default-on priming passes so those routes enter discovery:
  • GET-form submission (SubmitGetForms) — synthesizes the submit URL for each filled GET form and fetches it, so the resulting query-string endpoints are captured as traffic.
  • Same-origin POST-form submission (SubmitPostForms) — exercises JS-driven POST endpoints (a common home for XXE and body/cookie SQLi) by triggering their submit handlers. Both submission passes share SubmitFormMaxVariants, a cap on how many distinct submit variants are fetched per page.
  • Anchor-link priming (AnchorLinkPriming, capped by AnchorLinkMaxAssets) — on each settled page, fetches same-origin anchor-link targets so their routes are pulled into discovery even when nothing clicks them.
  • DOM-quiescence settle (WaitDOMQuiescence) — before harvesting a page, the settle waits for the DOM to stop mutating, so late-rendered forms and links are seen instead of being missed mid-render.
These passes are default-on and bounded. The form-submission passes are gated off by the same --no-forms flag that disables form filling. Destructive forms and paths are deliberately skipped — delete/logout actions, _method-override forms, and similar state-changing submits are never fetched.

​
DOM-based XSS (Browser-Confirmed)

As the final step of the crawl, the browser crawler runs a browser-confirmed DOM-based XSS probe over the client-side routes it already visited. It reuses the live browser and the routes discovered during the crawl rather than sending a separate fuzzing pass:
  • A marker-reflection gate first checks whether a route reflects an injected marker into the client-rendered DOM.
  • Only then does an execution canary run — the payload must actually fire in the browser (a real script execution, not just a textual reflection) before anything is reported, which keeps this free of the reflection false positives that plague raw-response DOM-XSS heuristics.
Confirmed hits are emitted as a finding titled “DOM-based XSS (Browser-Confirmed)” (module id spider-dom-xss), with severity reported by the browser-accurate probe. The probe is bounded — at most ~16 confirmations under a ~45-second budget — so it corroborates real DOM-XSS on reflected client routes without ever dominating the crawl.

​
Default-credential login attempts

To reach areas hidden behind a login, the crawler can try a short list of common default credentials against a confirmed local login form so the browser session stays authenticated and the crawl continues into the now-unlocked area. This pass is discovery-focused, not a brute-force, and is heavily constrained:
  • It runs only against a form the crawler has confirmed is a local login (a single password field + an identity field + a submit, posting to an in-scope host — never an external IdP).
  • It tries a small, documented default list (admin:admin, admin:123456, …) plus any identity the crawl registered earlier. The identity field’s type is respected (email inputs get email-style defaults).
  • It is negative-control gated: an improbable random pair must be rejected first, or the whole spray is abandoned (so a form that “accepts” anything can’t produce a false “logged in”).
  • It is single-flighted per host and capped (≤ 10 attempts, never a wordlist), so it cannot lock accounts.
  • No finding is emitted — the attempts are captured as ordinary traffic and any unlocked pages simply become more crawlable surface.
The pass is off by default and the native-scan runner auto-enables it by intensity: balanced uses the minimal list, deep uses the full documented list. An ordinary crawl leaves it off, keeping active login attempts explicit.

​
Session carry-forward

While crawling, the browser often clears an interstitial WAF/bot challenge (a JS or cookie challenge) or picks up session cookies. Those cleared cookies and the browser’s User-Agent are carried forward into the subsequent discovery and active-scanning phases, so content discovery and module probes reuse the already-cleared session instead of tripping the challenge again.
  • On by default whenever --spider runs; scoped per host to the same host the browser visited. Disable with --no-carry-browser-session.
  • Respects your own -H/--header overrides — an explicitly supplied header always wins.
  • The User-Agent is pinned from the browser only when you haven’t set a non-default UA yourself.

​
Exploration Strategies

​
Browser Management

  • Embedded binaries: ships Chromium (macOS/Windows/Linux) and ungoogled-Chromium (Linux). Extracted on first run, cached by version.
  • Headless mode: uses headless=new when extensions are loaded (supports Chrome extensions unlike legacy headless).
  • Extensions: loaded via --load-extension (e.g., uBlock Origin Lite for ad blocking during crawl).
  • Security flags disabled for crawling: --disable-web-security, --ignore-certificate-errors, --allow-running-insecure-content.
  • Pool: multiple browser instances with round-robin selection.

​
Network Capture

Traffic is captured at the browser level (not page level) via CDP events, covering all tabs, popups, and iframes:
  1. NetworkRequestWillBeSent → record request
  2. NetworkResponseReceived → record response headers
  3. NetworkLoadingFinished → fetch response body
Hash-based deduplication prevents duplicate records. A cleanup loop removes stale pending requests (>15s). Captured traffic is converted to httpmsg.HttpRequestResponse and saved via the RecordSaver interface with source "spidering".

​
Termination Conditions

The crawl stops when any of these are met:
  • Maximum states discovered
  • Maximum duration elapsed
  • Maximum crawl depth reached
  • Maximum consecutive failures
  • No more candidate actions to explore
  • Context cancellation

​
Entry Point

Returns SpiderResult with: states discovered, actions executed/failed, forms submitted, duration, and records saved.