I inherited a data extraction tool nobody had validated. It was silently losing 18% of the data.
What it does
Large operations team across multiple regions. They track weekly performance — who handled what, how much, whether it was resolved. The numbers feed into resource planning, coaching, and reporting.
The pipeline pulls activity records from an API, applies transformation rules (filtering, deduplication, attribution), and produces structured reports in JSON, dashboard views, and CSV exports.
What I inherited
A script written by someone who'd left. It ran weekly, pulled data from an API, processed it, and produced a report. People filed those reports upstream. Nobody cross-checked the numbers.
I compared expected record counts against actual output. Consistent ~18% shortfall. Three compounding bugs:
- A dedup function grouped by a property that didn't exist on the objects — every key resolved to
undefined, collapsing N entries into 1 - A sort comparator returned booleans instead of numbers — non-deterministic ordering across engines
- The API query filter was missing entire categories — some commented out during development, never restored
On top of that, work done through automation posted under a shared account. The pipeline only checked individual authorship. All automated work was invisible.
What I built
flowchart LR
A1["REST API"] --> B1["Filter"]
A2["Personnel API"] --> B1
B1 --> B2["Deduplicate"]
B2 --> B3["Attribute"]
B3 --> C1["JSON"]
B3 --> C2["Dashboard"]
B3 --> C3["CSV"]
Three stages. Extraction pulls from two independent sources — activity data and team membership. Transformation applies filtering, deduplication, and attribution. Output goes to multiple formats.
Extraction uses paginated async iteration. The team source is a filtered API call. Multiple query modes handle different metric types with different date semantics.
Transformation runs each record through a multi-step filter:
flowchart TD
IN["Record"] --> F1{"Known\ncontributor?"}
F1 -->|No| X1["Drop"]
F1 -->|Yes| F2{"Noise?"}
F2 -->|Yes| X2["Drop"]
F2 -->|No| F3{"In range?"}
F3 -->|No| X3["Drop"]
F3 -->|Yes| F4{"Duplicate?"}
F4 -->|Yes| X4["Drop"]
F4 -->|No| OK["Keep"]
Attribution determines who gets credit. Direct when the author is identified. Falls back to the assigned worker when the record was produced by automation — the system posts under a shared account, so you need a different signal.
The bugs
Silent data loss. The dedup grouped records by a field that didn't exist. Every iteration wrote to acc[undefined]. Only the last item survived — one record per entity regardless of activity. Fixed with Map-based dedup on a derived key.
Non-deterministic sort. A boolean comparator where sort() expects a number. false coerces to 0 (equal), so items that should precede get treated as identical. Fixed with numeric subtraction.
Missing categories. The query filter was incomplete. Valid records were invisible — commented out during development or never added. Reconciled against the source of truth.
Migration
Currently browser-based — someone triggers it manually each week. The primary data source is being deprecated, which forced a rethink. Designing a migration to a scheduled cloud function:
flowchart TD
CRON["Schedule"] --> FN["Function"]
FN --> SRC1["Warehouse"]
FN --> SRC2["New API"]
SRC1 --> PROC["Transform"]
SRC2 --> PROC
PROC --> OUT1["Storage"]
PROC --> OUT2["Notify"]
Serverless function on a weekly cron. Dual data source — warehouse for metadata, new API for record detail. Versioned output in object storage. Notifications on completion or failure. Under $1/month.
What I learned
Nobody was checking the output. The tool produced numbers, the numbers went into reports, the reports got filed. It took comparing expected counts to actuals — a basic sanity check — to surface a problem that had been compounding for months.
Now every business rule is documented in a decision record. Each ambiguity was resolved with the people who own the metric definitions. The next person who opens this code knows what it's supposed to do and why.