ETL vs Data Pipeline

Quick verdict

AWS defines a data pipeline as "a series of processing steps to prepare enterprise data for analysis", and states directly that an ETL pipeline "is a special type of data pipeline". The difference is scope. A pipeline may simply extract and load data with no transformation, transform it after loading (ELT), process events continuously as they arrive, or send curated data back into operational applications.

ETL is the older, more specific pattern. Microsoft's Azure Architecture Center describes it as a data integration process that consolidates data from diverse sources into a unified data store, with the transformation phase applying business rules in a specialised engine, often through staging tables. Typical transformation work includes filtering, sorting, aggregating, joining, cleaning, deduplicating and validating data.

Side by side

Key differences

Data pipeline is the umbrella, ETL is one pattern

Most data pipelines in an analytics stack fall into a handful of patterns. Microsoft's architecture guidance and AWS's definition between them cover all of these:

• ETL: transform in a separate engine, then load.
• ELT: load raw data first, then transform inside the target using its compute. Microsoft notes that ELT differs from ETL "solely in where the transformation takes place".
• Extract and load only (replication): copy tables or files as they are, for example into a data lake.
• Change data capture (CDC): read each insert, update and delete from a database log soon after it happens; Debezium, for example, describes itself as "an open source distributed platform for change data capture".
• Streaming: process events continuously as they arrive, through a broker such as Kafka and a stream processor such as Apache Flink.
• Reverse ETL: move modelled data from the warehouse back into operational tools such as a CRM or marketing platform.

A real platform usually combines several: CDC into a lake, ELT models in the warehouse, a streaming path for alerts and a reverse ETL sync to the CRM. Calling all of that "the ETL" is common shorthand, but it hides decisions that matter.

Batch versus streaming

AWS describes batch pipelines as running infrequently, often off-peak, using high computing power for a short period, and stream pipelines as running continuously on a sequence of small data packets for low-latency analytics. Classic ETL is batch: a nightly or hourly job reads a window of data and processes it as a set. Microsoft contrasts this with streaming, which "processes data as it arrives", and lists the reliability concerns that come with it: checkpointing for at-least-once processing, idempotent transformations to cope with duplicates, watermarks for late events, and dead letter queues.

In our view, most reporting needs are met by batch or frequent micro-batch pipelines. Streaming earns its extra complexity when a decision must be taken within seconds, such as fraud checks, operational alerts or live logistics dashboards.

Ingestion versus orchestration

A pipeline has two different jobs that are often confused. Ingestion (or data movement) is the work of reading from sources and writing to targets: connectors, schema changes, incremental state. Orchestration (Microsoft calls it control flow) decides what runs, in what order and what happens on failure; Microsoft describes control flow as enforcing processing order through precedence constraints, with data flows executed as tasks within it.

Tools tend to specialise. Fivetran and Airbyte are ingestion tools; Apache Airflow describes itself as "an open-source platform for developing, scheduling, and monitoring workflows", and its documentation positions it for batch workflows rather than streaming. Many teams therefore run both: an ingestion tool to land data, and an orchestrator to trigger syncs, run transformations and tests in order, and alert on failure. See Apache Airflow vs Airbyte for when one is enough. Integration suites such as Talend and Informatica include both functions, which is part of why they are larger products.

What a transformation step looks like

Whatever the pattern, the transformation is often plain SQL. This step loads a daily sales summary from raw orders that an ingestion tool has already landed, written for PostgreSQL (15 or later for MERGE):

-- PostgreSQL: rebuild yesterday's rows in a reporting table
DELETE FROM reporting.daily_sales
WHERE  sales_date = CURRENT_DATE - 1;

INSERT INTO reporting.daily_sales (sales_date, country, orders, revenue)
SELECT o.order_date,
       c.country,
       COUNT(*),
       SUM(o.total_amount)
FROM   raw.orders o
JOIN   raw.customers c ON c.customer_id = o.customer_id
WHERE  o.order_date = CURRENT_DATE - 1
  AND  o.status <> 'cancelled'
GROUP  BY o.order_date, c.country;

Deleting and re-inserting one day makes the step idempotent: if the orchestrator retries it, the result is the same. When rows can change after the fact, a MERGE that updates matching rows and inserts new ones does the same job:

-- PostgreSQL 15+: upsert the latest customer attributes
MERGE INTO reporting.customers t
USING raw.customers s
   ON t.customer_id = s.customer_id
WHEN MATCHED THEN
  UPDATE SET email = s.email, country = s.country
WHEN NOT MATCHED THEN
  INSERT (customer_id, email, country)
  VALUES (s.customer_id, s.email, s.country);

Run inside the warehouse after loading, this is ELT. Run in a staging database or ETL engine before the result is loaded into the warehouse, the same logic is ETL. The SQL barely changes; the architecture does.

Choosing a design

Rather than starting from the label, start from four questions: how fresh the data must be (daily, hourly, seconds), where transformation should run (a separate engine, the warehouse, a stream processor), whether raw data must be kept for reprocessing, and who maintains the pipeline (data engineers writing code, or analysts using a managed service). Microsoft suggests ETL when the target is constrained, complex rules need a specialised engine, or compliance requires curated staging before loading, and ELT when the target has elastic compute and you want to keep raw data.

For concrete product choices, see Fivetran vs Talend, Airbyte vs Talend, Airbyte vs Matillion, Fivetran vs Matillion, Matillion vs Talend and Talend vs Informatica, or the guide to the best ETL tools. For where the data ends up, see Data warehouse vs database.

Pricing and licensing

This is a conceptual comparison, so there is no single price. What changes is the cost model. A classic ETL setup costs a transformation engine or integration suite (often licensed by capacity or processing units) plus the servers or cloud runtime it uses. Broader data pipelines spread cost across several services: managed ingestion is commonly billed by data volume or rows changed, warehouses bill for the compute used by ELT transformations, streaming platforms bill for brokers and processing capacity, and orchestrators cost either infrastructure (self-hosted) or a managed service fee.

The individual comparisons give each vendor's billing unit and any dated example price. In every design, budget for the people who build and run the pipelines, and for warehouse compute used by transformations, which is easy to overlook.

Pricing checked on the vendors' official pages on 7 October 2026. Prices change; confirm before buying.

Where each one leads

Limitations

When to choose each

When neither is right

• You only need occasional analysis of a few exports: loading files into an in-process engine is simpler; see DuckDB vs pandas.
• Reports come from one application database: a read replica or reporting views may be enough before any pipeline; see Data warehouse vs database.
• You are choosing between transforming before or after loading: that is the narrower question covered in ETL vs ELT.

Final recommendation

Frequently asked questions

Sources

• AWS: What is a data pipeline?
• Microsoft Learn: Extract, transform, load (ETL) (Azure Architecture Center)
• Apache Airflow documentation
• Debezium
• PostgreSQL documentation: MERGE
• PostgreSQL documentation: Logical replication

Checked October 2026.

How we research comparisons: our editorial method.