Case Study 05 · Cloud & DevOps · Completed
Event-Driven Product Image Pipeline on AWS
An AWS pipeline that runs when an image is uploaded and attaches it to the right product in the e-commerce catalog, plus an audit tool that keeps storage and catalog in step.
Problem
Staff handled product images by hand across several separate systems, which took hours of work for very little output. Nothing kept the image bucket and the catalog from drifting apart either.
Solution
Staff upload photos through a browser portal and the upload is the trigger: S3 raises an event, EventBridge routes it to an SQS queue, and a Lambda starts a containerised worker on ECS Fargate for each batch. The worker reads product data from the ERP and attaches each image, with alt text built from that data, to the matching catalog product, working from metadata only and never the image bytes. A separate S3 auditing tool compares the bucket with the images the catalog actually uses and reports or removes orphans.
Highlights
- Built for at-least-once delivery: duplicate events collapse, audit writes are upserts, and catalog images are updated in place, never deleted before a replacement succeeds.
- A second Lambda listens for stopped ECS tasks and finalises the audit records of any container that died mid-run.
- The audit database runs on Amazon RDS with versioned up and down migrations, TLS-verified connections and separate least-privilege roles for migrations and runtime.
- Structured JSON logs in CloudWatch carry correlation IDs that join each Lambda invocation to its container run.
- The auditing tool throttles itself from the API's remaining-request header, honours Retry-After, and offers a dry-run preview and a max-delete cap that aborts on a surprising result.
Outcome
Validated end to end against a sandbox store, cut over to the real store and handed to the staff who use it, with training. The workers run only while there is work, and every task run and per-image result lands in a PostgreSQL audit trail.
How the Work Ran
Started as a written proposal with an architecture diagram and a refactor plan, followed by an evaluation of the AWS services and a monthly cost forecast before rollout. Built in phases: container and registry, then the audit database, then the upload portal and auditing tool. The portal went through rounds of UX review with its users before handoff. The infrastructure was built from a step-by-step runbook that records the reasoning on visibility timeouts, batching and duplicate delivery.
Tech
- AWS Lambda
- Amazon ECS
- Amazon EventBridge
- Amazon S3
- Docker
- AWS
- PostgreSQL
- Python
- REST APIs