Skip to main content
Spitolas is a state-machine-driven web crawler that drives a real Chromium browser to discover application states through user-like interactions (clicking, form filling, iframe traversal). All resulting HTTP traffic is captured at the CDP level and fed into vigolium’s scanning pipeline.

​
How It Works

​
Core Concepts

​
States and the State Graph

A State is a DOM snapshot identified by SHA256(strippedDOM)[:16]. The State Graph is a directed graph where nodes are states and edges are actions that caused transitions. Navigation between states uses Dijkstra’s shortest path (with Yen’s K-shortest as fallback). Near-duplicate detection uses normalized Levenshtein distance (threshold: 10%). For large DOMs (>10K chars), a sampling-based distance is used for performance.

​
Actions and Candidate Elements

Candidate clickable elements are discovered via CSS selectors (a, button, [onclick], [role=button], input[type=submit], framework-specific bindings like [ng-click], [v-on:click], etc.). Each candidate becomes an Eventable (graph edge) once fired, linking a source state to a target state with an event type (click, hover, enter).

​
Fragments (Visual Page Segmentation)

Pages are decomposed into Fragments - DOM regions identified by XPath, bounding box, subtree size, and content hash. Two modes:
  • Landmark (default): fast DOM-based extraction
  • VIPS: vision-based page segmentation with multi-pass decreasing thresholds
Fragment comparison uses the APTED tree edit distance algorithm. The Fragment Manager tracks exploration status across duplicate/equivalent fragments and uses candidate influence scoring to prioritize which states to explore next.

​
Form Handling

The Form Handler detects and fills forms with smart value generation:
  • Field-name-aware values (email, password, phone, URL, etc.)
  • Constraint-aware generation (respects pattern, min/max, minlength/maxlength)
  • Pairwise fallback when filling all inputs at once fails
  • File upload support with type-aware file selection
Filling a form only exercises its inputs — the endpoints behind those forms and links stay invisible until something submits them. After a page settles, the crawler runs a set of bounded, default-on priming passes so those routes enter discovery:
  • GET-form submission (SubmitGetForms) — synthesizes the submit URL for each filled GET form and fetches it, so the resulting query-string endpoints are captured as traffic.
  • Same-origin POST-form submission (SubmitPostForms) — exercises JS-driven POST endpoints (a common home for XXE and body/cookie SQLi) by triggering their submit handlers. Both submission passes share SubmitFormMaxVariants, a cap on how many distinct submit variants are fetched per page.
  • Anchor-link priming (AnchorLinkPriming, capped by AnchorLinkMaxAssets) — on each settled page, fetches same-origin anchor-link targets so their routes are pulled into discovery even when nothing clicks them.
  • DOM-quiescence settle (WaitDOMQuiescence) — before harvesting a page, the settle waits for the DOM to stop mutating, so late-rendered forms and links are seen instead of being missed mid-render.
These passes are default-on and bounded. The form-submission passes are gated off by the same --no-forms flag that disables form filling. Destructive forms and paths are deliberately skipped — delete/logout actions, _method-override forms, and similar state-changing submits are never fetched.

​
Frontier seeding (robots.txt and sitemap.xml)

A link-following crawl can only reach what something links to. Before the main loop starts, the crawler reads the host’s robots.txt and sitemap.xml and folds every in-scope location it finds into the graph — which is how routes that nothing on the site links to still enter discovery. Ironically, robots.txt disallow rules are often the most interesting entries: they name the paths the operator would rather nobody browsed. Seeding is a whole-crawl pass, not a landing-page step: it navigates to other locations and needs scope adoption already settled, so it runs after the index state is initialized and before the crawl loop. Locations are recorded up to a cap (1000), but only a subset (25) are actually browsed into states. Those are chosen by rounding over path shapes — identifier-looking segments are collapsed, so a sitemap dominated by one template (/product/1, /product/2, … ×900) cannot spend the entire allowance inside it. The rest are requested and recorded as traffic without being crawled. On by default. The run reports how many locations were declared and how many were browsed. A rendered SPA never serves its route list as static HTML — the URLs live in inline scripts, JSON blobs, and commented-out markup. After a page settles, the crawler scrapes URL-like strings out of the document’s inline script and HTML comments, keeps the same-origin ones, and fetches them so they land in the traffic corpus. On by default and budgeted against the same dedup set the anchor primer uses, with its own per-call allowance so neither primer can silently consume the other’s. Once the allowance is spent, later states skip the scan entirely rather than paying a full-document walk to learn there is nothing left.

​
Self-registration (self_register)

On an application with open registration, the entire authenticated surface is invisible to an unauthenticated crawl. self_register lets the crawl complete a public signup form and continue as the account it creates, turning that surface into ordinary crawlable content. It is heavily constrained, along the same lines as the default-credential pass:
  • Single-flighted per host, and it never submits to an off-host action.
  • It fills through the normal form handler, so the identity it creates lands in the shared fill context and is reused by any later login form — whether or not registration itself logged the browser in.
  • It stays off by default, because registering is a write. --intensity deep enables it; setting spidering.self_register: true forces it on at any intensity.

​
Crawl graph output (graph_output_dir)

Captured traffic records what was requested. The crawl graph records how the crawler got there — which action on which state, the selector that finds that element again, and the form values used. That is what makes a run reproducible and lets a later pass re-reach a specific state without rediscovering the path to it. Set spidering.graph_output_dir to a directory and each host’s graph is written into it as its own file. It is a directory rather than a path because one browser session reuses a single base config across all of a host’s seeds — a precomputed path would be whatever the first target implied, and every later seed would overwrite it. The DOM itself is deliberately excluded from the dump; the selector and form values on each edge are the point, since they are what make a transition replayable.

​
DOM-based XSS (Browser-Confirmed)

As the final step of the crawl, the browser crawler runs a browser-confirmed DOM-based XSS probe over the client-side routes it already visited. It reuses the live browser and the routes discovered during the crawl rather than sending a separate fuzzing pass:
  • A marker-reflection gate first checks whether a route reflects an injected marker into the client-rendered DOM.
  • Only then does an execution canary run — the payload must actually fire in the browser (a real script execution, not just a textual reflection) before anything is reported, which keeps this free of the reflection false positives that plague raw-response DOM-XSS heuristics.
Confirmed hits are emitted as a finding titled “DOM-based XSS (Browser-Confirmed)” (module id spider-dom-xss), with severity reported by the browser-accurate probe. The probe is bounded — at most ~16 confirmations under a ~45-second budget — so it corroborates real DOM-XSS on reflected client routes without ever dominating the crawl.

​
Default-credential login attempts

To reach areas hidden behind a login, the crawler can try a short list of common default credentials against a confirmed local login form so the browser session stays authenticated and the crawl continues into the now-unlocked area. This pass is discovery-focused, not a brute-force, and is heavily constrained:
  • It runs only against a form the crawler has confirmed is a local login (a single password field + an identity field + a submit, posting to an in-scope host — never an external IdP).
  • It tries a small, documented default list (admin:admin, admin:123456, …) plus any identity the crawl registered earlier. The identity field’s type is respected (email inputs get email-style defaults).
  • It is negative-control gated: an improbable random pair must be rejected first, or the whole spray is abandoned (so a form that “accepts” anything can’t produce a false “logged in”).
  • It is single-flighted per host and capped (≤ 10 attempts, never a wordlist), so it cannot lock accounts.
  • No finding is emitted — the attempts are captured as ordinary traffic and any unlocked pages simply become more crawlable surface.
The pass is off by default and the native-scan runner auto-enables it by intensity: balanced uses the minimal list, deep uses the full documented list. An ordinary crawl leaves it off, keeping active login attempts explicit.

​
Session carry-forward

While crawling, the browser often clears an interstitial WAF/bot challenge (a JS or cookie challenge) or picks up session cookies. Those cleared cookies and the browser’s User-Agent are carried forward into the subsequent discovery and active-scanning phases, so content discovery and module probes reuse the already-cleared session instead of tripping the challenge again.
  • On by default whenever --spider runs; scoped per host to the same host the browser visited. Disable with --no-carry-browser-session.
  • Respects your own -H/--header overrides — an explicitly supplied header always wins.
  • The User-Agent is pinned from the browser only when you haven’t set a non-default UA yourself.

​
Exploration Strategies

​
Browser Management

  • Embedded binaries: ships Chromium (macOS/Windows/Linux) and ungoogled-Chromium (Linux). Extracted on first run, cached by version.
  • Headless mode: uses headless=new when extensions are loaded (supports Chrome extensions unlike legacy headless).
  • Extensions: loaded via --load-extension (e.g., uBlock Origin Lite for ad blocking during crawl).
  • Security flags disabled for crawling: --disable-web-security, --ignore-certificate-errors, --allow-running-insecure-content.
  • Pool: multiple browser instances with round-robin selection.

​
Network Capture

Traffic is captured at the browser level (not page level) via CDP events, covering all tabs, popups, and iframes:
  1. NetworkRequestWillBeSent → record request
  2. NetworkResponseReceived → record response headers
  3. NetworkLoadingFinished → fetch response body
Hash-based deduplication prevents duplicate records. A cleanup loop removes stale pending requests (>15s). Captured traffic is converted to httpmsg.HttpRequestResponse and saved via the RecordSaver interface with source "spidering".

​
Termination Conditions

The crawl stops when any of these are met:
  • Maximum states discovered (max_states, default 1500)
  • Maximum duration elapsed (max_duration, default 30m)
  • Maximum crawl depth reached (max_depth, default 6)
  • Maximum consecutive failures (max_consecutive_fails, default 100)
  • No more candidate actions to explore
  • Context cancellation
max_depth and max_states were unlimited before v0.3.8. Both exist to bound how the crawl spends its max_duration: without a depth bound a link-dense site can sink the whole budget into one deep branch and never revisit the breadth near the seed, and without a state bound a template that mints a state per row (a paginated table, a calendar) spends it on near-identical pages. Set either to 0 in spidering to restore unlimited.
The end-of-crawl follow-up and retry sweeps are gated on the stop reason: both are no-ops when the crawl stopped on its budget, the state cap, or a failure streak. A streak means the application isn’t answering, which no retry fixes.

​
Entry Point

Returns SpiderResult with: states discovered, actions executed/failed, forms submitted, duration, and records saved.