Case Study 03 · Full-Stack · In Progress

Price Intelligence Platform

A Django platform that reads the product catalog from an ERP through configurable connectors, collects marketplace prices on a schedule, and keeps price history split by month.

RoleSole developer: data model, integrations, collection engine and dashboard

Competitive Dashboard with summary tiles and a product table of fictional Acme, Globex, Initech and Vandelay parts, our price, competitor price and trend. — The dashboard compares each catalog product's price with marketplace prices.

Product detail for a fictional Acme transfer belt: a three-month price trend chart of our price against four fictional competitors. — Price history shows how each competitor's price moved over time against ours.

Scraper Builder Studio, step 2 Selectors: a public books demo site rendered in Visual Recon beside XPath selectors grouped by search and listing phase. — New marketplace scrapers are built in a wizard as configuration, not new code.

Scrape Monitor: five fictional competitor scrapers selected, product scope options, a Start Scrape button and a history of completed runs. — The Scrape Monitor picks which sources and products to collect and logs every run.

Pricing Rules: strategy cards with Beat Lowest by Buffer selected, condition-tier discounts and a live preview of the suggested price for a demo product. — Pricing rules preview a suggested price before anything is saved.

Problem

Pricing staff needed to know what other marketplaces charge for the same parts, across the whole catalog, without checking by hand. The catalog also had to stay in step with the ERP, and a price is only useful with its history behind it.

Solution

A Django and Celery platform built around configuration rather than code. An ERP studio defines connectors in a wizard (auth, endpoints, joins, field mapping, transforms) and runs scheduled, incremental, read-only syncs. Scheduled browser collection matches marketplace listings to catalog products with a confidence score and appends price snapshots to a month-partitioned PostgreSQL table.

Highlights

Incremental ERP sync: the cursor is pushed into the OData filter and only advances after a successful run, and rows stream end to end into chunked bulk upserts so memory stays bounded.
API clients honour Retry-After on throttling and service errors, refresh the token once when a request is rejected, and prefetch pages in parallel while still yielding them in order.
Price history is an append-only table partitioned by month, with commands to create future partitions and prune old ones, plus trigram, partial and composite indexes shipped as migrations.
Celery fans collection out across a capped browser pool, with cooperative pause, resume and cancel and cleanup of orphaned browser processes.
A no-code scraper builder checks every server-side fetch against private and metadata addresses to block SSRF, and connector credentials are encrypted at rest.

Outcome

The platform is built and tested, including an isolated test environment that never writes to real price history. Most new ERP connectors and marketplace scrapers can now be added as configuration rather than new code. A beta rollout is waiting on stakeholder prerequisites, so it is not in use yet.

How the Work Ran

Run as an internal epic with a definition of done I wrote, for pricing and catalog stakeholders. Delivered in phases: a hard-coded ERP spike, then database-driven connectors and the studio, then a join engine, then removal of the legacy adapter. Network-dependent and ERP-dependent tests sit behind flags, apart from the offline suite. A runbook and a qualification audit were done before stakeholder review.

Tech

Celery
Redis
Selenium
Django
OData
PostgreSQL
Python
REST APIs