Lightweight ETL: Moving Data Without a Data Team

Raj runs a SaaS company with forty employees. His stack includes Shopify for e-commerce, Stripe for billing, HubSpot for CRM, Intercom for support, and a custom warehouse app that exports CSVs. Every Monday, his operations lead downloads five different reports, copies them into a master spreadsheet, and spends three hours building the weekly metrics deck. The deck is always three days stale by the time the leadership team sees it.

He doesn't have a data team. He has a part-time ops lead, a freelance bookkeeper, and a growing suspicion that he's making decisions with yesterday's news. He looked at Fivetran and Snowflake, but the combined cost would hire another engineer. He looked at Airflow, but the learning curve looked like a cliff. What he needs is a lightweight ETL pipeline — something one person can build, run, and fix without a certification in distributed systems.

You can build exactly that with dlt, Python, and Postgres. The stack is boring, reliable, and small enough that a single engineer can operate it alone. No data team required.

The Data Team Gap

Modern businesses generate data in a dozen places. The website logs events. The payment processor tracks revenue. The support tool records conversations. The warehouse counts inventory. Each system is excellent at its job and terrible at talking to the others.

The traditional answer is to hire data engineers. They build pipelines in Airflow, model data in dbt, and serve it through Looker or Tableau. This works beautifully at scale. It also costs $400K in salaries before you see your first dashboard. For a company with forty employees, that's madness.

Small businesses have data needs — they need unified reporting, they need alerts when metrics drift, they need to stop copy-pasting between systems . but they don't have data team budgets. They need tools that fit in a single brain. Tools that break loudly when they break, so someone notices before the quarter-end report is due.

A small business with five data sources and one engineer doesn't need a data team. It needs a pipeline that runs at 6 AM and breaks loudly when it breaks.

What dlt Gets Right

The library in question, dlt (short for data load tool), handles the boring parts of ETL so you don't have to. Connect to a source, define what you want, and dlt handles extraction, normalization, and loading. The magic is in the defaults.

Incremental loads. Instead of pulling everything every time, dlt tracks what it already loaded and fetches only what's new. A Shopify store with fifty thousand orders doesn't need all fifty thousand on every run. It needs the two hundred that arrived since yesterday. dlt does this automatically once you configure the cursor field.

Schema evolution. Sources change. Shopify adds a column. Stripe renames a field. Traditional pipelines break when this happens. dlt detects the change, updates the destination schema, and logs what happened. The pipeline keeps running while you decide if the new field matters.

The Python decorator. A complete dlt pipeline can be a twenty-line function with a @dlt.source decorator. Compare that to two hundred lines of Airflow DAG boilerplate. For simple extractions — API to database, CSV to table . the decorator approach is laughably productive. You're writing business logic, not orchestration code.

It won't replace Airflow for complex DAGs with fifty interdependent tasks. It won't replace Fivetran if you have twenty sources and zero engineers. But for the gap between "we have some data sources" and "we need a data team," dlt is exactly right.

The One-Engineer Stack

The full lightweight stack has four parts. Each part is replaceable, but together they're a complete data platform that fits in one person's head.

dlt for extraction. Pull from APIs, databases, or files. Handle pagination, retries, and rate limits without writing the glue yourself. The Python-based approach means you can transform data mid-pipeline with plain code — no SQL required if you don't want it.

Postgres for storage. Relational, reliable, and queryable by everyone. Analysts know SQL. Engineers know SQL. Even some executives know enough SQL to be dangerous. Postgres also handles JSONB for semi-structured data, which covers most modern API responses without requiring a document database. If you want the full case for keeping it simple, read our take on using Postgres for business applications.

cron for scheduling. Not Kubernetes. Not a workflow engine. Just cron. A line in crontab that runs your script at 6 AM. When something breaks, you check one log file. When you need to pause, you comment out one line. Simplicity is a feature at this scale.

One Grafana alert for monitoring. The alert checks that the pipeline ran, that it loaded the expected number of rows, and that no row is older than it should be. If the alert fires, you investigate. If it doesn't, you sleep. That's the entire monitoring strategy.

This stack won't impress anyone at a data conference. It will, however, deliver a unified dashboard every morning without requiring a standup meeting about pipeline health.

A Real Pipeline Walkthrough

Let's walk through Raj's actual problem. He needs Shopify orders, Stripe charges, and Intercom conversations in one place so his ops lead can stop copy-pasting.

The dlt pipeline starts with three sources. The Shopify source uses the REST API, paginates through orders, and tracks the updated_at field for incremental loads. The Stripe source does the same with charges. The Intercom source is slightly messier — it returns conversations as nested JSON . but dlt flattens the structure into relational tables automatically.

All three sources write to the same Postgres database. dlt creates schemas for each source: shopify_orders, stripe_charges, intercom_conversations. It handles the schema evolution when Shopify adds a new field called tax_lines. The pipeline doesn't break. It just logs the change and keeps going.

A simple SQL view joins the three tables on customer email. The view calculates weekly revenue by channel, support ticket volume by product, and average time to first response. A lightweight web app — maybe Streamlit, maybe a simple React dashboard . serves this view to the leadership team.

Altogether, the pipeline from extraction to dashboard is under three hundred lines of Python. One engineer wrote it in a week. One engineer maintains it. When Raj eventually hires a data team, they'll replace parts of this stack. But they'll start with working pipelines, not a blank slate.

When to Stop Adding Tools

The hardest part of lightweight ETL is resisting the urge to add tools. Every data blog tells you that real pipelines need orchestration, lineage tracking, and a semantic layer. They're not wrong. They're just describing a different scale.

You need Airflow when you have dozens of pipelines with complex dependencies — when task B can't start until task A finishes, and task C needs both. You need dbt when your transformations are complex enough that SQL alone becomes unmaintainable. You need Fivetran when you have twenty sources and no engineer to maintain custom extractors.

Most small businesses never reach these thresholds. A typical FDE engagement might connect five to ten sources, run ten to twenty SQL transformations, and serve three dashboards. That's comfortably inside the dlt-and-cron zone. Adding Airflow at this stage is like hiring a sommelier for a pizza night. Nice, but not necessary.

The signal to graduate is pain. When your cron schedule becomes a tangle of timing dependencies, consider Airflow. When your SQL views sprawl across fifty files, consider dbt. When your sources outnumber your engineers, consider Fivetran. Until then, keep it light.

Monitoring the Quiet Pipeline

Silent failures are the nightmare of lightweight ETL. A pipeline that runs every night and fails quietly is worse than no pipeline at all, because it trains people to trust bad data. The first sign of trouble is usually a CEO asking why the numbers look weird.

You need three monitoring layers. First, did the pipeline run? A simple cron job wrapper that logs start and end times. If the log is missing, the alert fires. Second, did it load the expected volume? Compare yesterday's row count to the thirty-day average. A fifty percent drop usually means the source API changed, not that business vanished. Third, is the data fresh? The oldest row in the table should never be more than twenty-five hours old for a daily pipeline.

These checks take twenty lines of Python and one Grafana alert. They're not sophisticated. They're sufficient. And they catch the failures that matter before anyone in leadership notices.

If you're building your first pipeline, start with the FDE's default tool stack — it covers the fundamentals without the bloat. For a step-by-step walkthrough of a first build, see building your first data pipeline. For data collection that goes beyond APIs, browser automation for data collection can fill gaps that official integrations miss. And once your pipelines are running, monitoring lightweight applications becomes the discipline that keeps them honest.

The best data pipeline is the one you forget about because it just works. Build light, monitor carefully, and add complexity only when the current stack screams for mercy.