Back to case studies

Case Study 02 · Full-Stack · Active

FleetForge

A browser-based studio for building, testing and running web data-collection jobs. A local language model suggests page selectors, and each suggestion is checked before use.

RoleSole author: architecture, back end, front end and the LLM integration
FleetForge Visual Recon: a quotes demo site beside the selector map, an author element picked, CSS and XPath candidates and a data preview.
Selectors are picked by eye on a live render instead of written by hand.
FleetForge Fleet Status: a running mission with two demo targets, one completed and one running, above a run history of completed runs with no errors.
Fleet Status shows a mission's targets live, with pause, cancel and run history.
FleetForge Selector Mapping: quotes demo page beside selector roles grouped by phase, each with its XPath and a match count; one role shows zero matches.
Each selector is checked against the page, so a selector that matches nothing is flagged.
FleetForge Recon Base Studio step 5 of 6, Extraction Test: passed with every row complete, a found/missing table per field and sample quotes.
A blueprint must pass a live extraction test before it can be activated.
FleetForge Intel Dashboard: item totals, source, campaign and target filters, CSV and JSON export, and collected rows from a public books demo site.
Collected data lands in one searchable table that exports to CSV or JSON.

Problem

Every new data source usually means another bespoke scraper, with the same orchestration bugs written again. The slowest step is not even the code: it is a person picking CSS and XPath selectors off a page by hand.

Solution

Scrapers became configuration: a blueprint is JSON run by a shared engine, authored in a visual wizard that loads the target page into a safe embedded render so selectors can be picked by eye. A self-hosted language model proposes a selector for each named role, and each proposal is checked against the page before it is accepted. Celery runs the resulting missions under a concurrency cap with managed browser lifecycles.

Highlights

  • Compresses the page before prompting: drops scripts, styles, SVG and other non-content tags, keeps a short attribute allowlist and caps data attributes per element, so a real page fits in a usable prompt.
  • Model output is never trusted as-is: suggestions are validated against the page for a fixed set of named roles.
  • The LLM client uses only the standard library and speaks an OpenAI-compatible API, so it works with Ollama or vLLM without new dependencies.
  • Requests use JSON mode at low temperature, retry once with backoff, and raise typed errors for a malformed reply or a missing provider.
  • Separate unit, integration and end-to-end suites, including browser lifecycle under concurrency, with every dependency pinned exactly.

Outcome

Adding a source stopped requiring Python. The LLM assist is a real feature: specified in a design document first, covered by tests, and switched off cleanly when no provider is configured.

How the Work Ran

Solo, run in numbered delivery phases that show in both the commit history and the stylesheet names. Conventional commit prefixes throughout. Design documents were written before the features they describe, including the selector-assist plan, and an architecture document lives next to the code.

Tech

  • Celery
  • Ollama
  • Redis
  • Selenium
  • Django
  • Python
  • SASS
Case Study
Open full page