Deckard
Contact
Contact

Data that arrives checked

Data engineering and pipelines

Data from each system is copied into one central database, checked on the way in, with a named person for every failure. Clean sales and customer data shows which customers are due to reorder and what to offer them.

Start a conversationSee it step by step
Nightly orders feedFresh
  1. 01:12Orders file lands on SFTPPicked up on arrival
  2. 01:13Columns checked, 42,180 rowsStaged before anything reaches the warehouse
  3. 01:198 of 9 tests passThe one that fails: every order has a total
  4. 01:24analytics.orders writtenOne table for its 3 readers
  5. held14 orders held for order opsNo total on any; each kept with its test

The run needed no retries and added late rows from earlier nights.

Four sources in, three uses out

Data pipeline: four source feeds, each checked where it enters, merge into one tested model and one table, analytics.orders, which dashboards, the finance export and alerts all read Four feeds, CRM, POS, Sensors and the nightly orders file, each pass a check at the edge; 14 order rows missing a total are held there with the check they failed. The checked rows merge into one tested model, orders_clean, with its joins, types and 9 checks, which writes one table, analytics.orders, 42,166 rows. Dashboards, the finance export and alerts all read that table. Moving dots show rows flowing through.
  1. Bad rows stop on arrivalEach feed is checked as it comes in, and a failing row is held with its test.
  2. Cleaned once, in one placeRecords are matched and cleaned in one tested step.
  3. Three uses, one tableDashboards, the finance export and alerts read the same table.

See it work

Last night’s run, start to finish

What each data feed includes

Cleanup steps are written in dbt and run on a schedule by Dagster or Airflow.

  1. 01Data pulled from apps, databases, files, partner feeds (EDI) and live streams
  2. 02Cleanup steps kept as code and reviewed before they run
  3. 03Schedules and retries, with a map of where each number came from
  4. 04Monitoring that tells late data from wrong data
  5. 05Documentation written automatically from the code
  6. 06Past data reloaded while reports stay online
Yours at the end

orders-pipeline/

models/The cleanup code for each table

tests/9 checks on orders

dagster/Jobs, schedules and retries

lineage/Where each number comes from

The nightly orders feed, row by row

TodayA file left on a shared server

Picked up by a script one person wrote

01Collect
Duplicates0 dropped
Column changesStop the load
Source fileKept 30 days
02Clean
Clean copyIts own table
Last changeReviewed, Sep 18
Traced fromFile to dashboard
03Load
Rows written42,166
Run time12 minutes
Retries3, 5 minutes apart
OutThe orders tableFresh
The old scriptRetired
DocsRebuilt every run
HistoryBack to 2023

By industry

Meters, carrier files and bank feeds, checked on arrival

Capacity reconciliationHourly
Read the asset and patch record
Cabinets, U, ports, owners
Read branch-circuit meters
Per feed, per cabinet
33 cabinets disagree with billing
Held for a human, with both sides shown
4Publish the sales capacity view
Same numbers as operations

The feed stops when two records disagree.

Data centers and colocation

Partner feeds3 of 4 on time
Carrier status, EDI 214
3,412 events, on time00:40
Truck pings joined to loads
Matched on trailer and time1,106 loads
Warehouse file, due 23:00
Schema unchanged22:51
One carrier file is 2 hours late
Earlier loads left untouchedPaged

Each load’s history is one search, whoever sent it.

Logistics and supply chain

Bank feed for the closeDaily, 05:00
Bank and card files in
Balance matches yesterday’s close05:00
Transactions joined to billing
One table per account, lineage kept05:04
2 duplicate transactions
Both copies held from the ledgerTest failed
Rest of the day posted
Ready before the team logs in05:09

Tests on the data run before anything reaches the ledger.

Accounting and finance

How it goes live

  1. 01 · Optional

    Every data source

    Every source, how it arrives and what broke last quarter

  2. 02 · Optional

    Your worst feed first

    Rebuilt with checks and alerts

  3. 03 · Data pipelines

    All feeds

    Schedules, tracing and checks across every source

  4. 04 · Ongoing

    On call for the data

    Late or wrong data alerts us, or your team

Related services

Once it loads: reports, alerts and a direct link

Read from the orders table for the Monday revenue numbers.

Dashboards and BI

A file arriving after 01:30 alerts whoever is on call.

Infrastructure monitoring and SRE

Moved off file drops to a direct link with the partner.

System integration and APIs

All solutions

Which feed breaks most often?

You get a rebuild plan: tests, alerts and an owner for each feed.

Start a conversation