Case Study 04 · Back-End · Active
Product Data and Content Enrichment Pipelines
Python pipelines that gather product data, match it against ERP records in PostgreSQL, and publish checked product content through a guarded sync.
Problem
There was no established process for enriching the product data already in the ERP. That data feeds the e-commerce product pages directly, so outdated details kept piling up on the site. A bulk import was not a safe fix either: a part that fails an exact match is not proof that it is new, so a naive load would create duplicates and corrupt the catalog.
Solution
Two pipelines built on one collection engine. The product-data pipeline loads collected parts into a purpose-built PostgreSQL schema, reads the ERP's existing items through a read-only OData client, and reconciles the two into review-grouped import files. The content-enrichment pipeline renders and validates product descriptions, links kit components to the e-commerce catalog, and writes the result to the ERP through a guarded REST sync.
Highlights
- An adaptive worker fleet decides each tick whether to spawn, shed or hold browser workers: it learns the host's real capacity, backs off when the source slows, paces spawns and replaces dead workers, and long runs checkpoint and resume.
- Each table declares its grain and key, per-run observations are kept apart from canonical parts for provenance, and all DDL and loads are idempotent upserts built with safe SQL composition.
- Identity resolution is an ordered rule cascade with its own tests and no database or network inside it; ambiguous matches go to human review instead of import.
- ERP writes are safe by default: dry-run, an environment-name confirmation before any write, skip-if-unchanged upserts, and read-back verification by content hash.
- API clients use sliding-window rate limits, retry only on transient statuses while honouring Retry-After, refresh OAuth tokens once, and trip a circuit breaker after repeated failures.
Outcome
The enrichment sync was proven against an ERP sandbox, then qualified against the production ERP, with every write audited. The product-data collectors and database are complete; importing their output is at the field-mapping stage, agreed field by field with business owners.
How the Work Ran
Run as ticketed epics with business stakeholders. ERP walkthroughs became a field-level mapping spec that tracks each field as confirmed, needing a decision or missing a source. When batch tests showed one collection approach would not cover enough, it was re-planned with stakeholders. Every ERP write went to a sandbox first and reached the real system only after it had been demoed to stakeholders.
Tech
- Selenium
- OData
- PostgreSQL
- Python
- REST APIs
- SQL