Lesson 2 of 8 · 11 min · The Node.js runtime

Streams: handling large data in Node.js

Covers: Node.js 22 and 24 LTS. Updated 13 Nov 2026. Part of the Full Stack Developer roadmap.

What you will learn

  • What a stream is and why it uses less memory than loading everything
  • The four stream types: readable, writable, duplex and transform
  • Connect streams safely with pipeline, and what backpressure is
  • Stream files, HTTP uploads and downloads, and compress data
  • Read streams with for await and know the web-streams equivalents

In simple words

Imagine moving a swimming pool of water to another place. You could try to lift the whole pool at once, which is impossible, or you could use a hose and let the water flow through. A stream is the hose: data moves in small pieces from a source to a destination, so you can process a 5 GB file or a long download without ever holding all of it in memory.

Words to know

  • Stream: a sequence of data pieces (chunks) that is read or written over time.
  • Chunk: one piece of data in a stream, often a buffer of bytes.
  • Backpressure: a signal from a slow consumer telling the producer to pause.
  • Pipe: connecting the output of one stream to the input of another.

Why streams

Reading a file in one call loads all of it into memory:

import { readFile } from "node:fs/promises";
const text = await readFile("orders-2gb.csv", "utf8");     // needs 2 GB of memory (and may fail): don't

With streams, memory use stays small and constant, and the first results appear before the last byte is read. Streams matter for large files, uploads and downloads, video and audio, log processing, compression, database exports, and proxying one service's response to another. HTTP requests and responses, sockets, files, process.stdin and process.stdout are all streams.

The four kinds

Type Meaning Examples
Readable a source you read from fs.createReadStream, HTTP request on a server, HTTP response on a client
Writable a destination you write to fs.createWriteStream, HTTP response on a server, process.stdout
Duplex both readable and writable a TCP socket
Transform a duplex that changes data as it passes zlib.createGzip(), encryption, CSV parsers

Reading and writing a file with streams

import { createReadStream, createWriteStream } from "node:fs";
import { pipeline } from "node:stream/promises";
import { createGzip } from "node:zlib";

await pipeline(
  createReadStream("orders.csv"),
  createGzip(),                                 // compress each chunk as it passes
  createWriteStream("orders.csv.gz")
);
console.log("done");

pipeline from node:stream/promises connects streams, forwards errors, destroys all streams on failure and returns a promise. Use it instead of the older .pipe(), which does not propagate errors or clean up. Memory stays small however large the file is.

Backpressure

If a fast reader feeds a slow writer (a big file to a slow network), data would pile up in memory. Backpressure solves that: a writable stream's write() returns false when its internal buffer is full, and the producer should pause until the drain event. pipeline and .pipe() handle this automatically, which is another reason to use them. If you write by hand:

function writeMany(writable, lines) {
  let i = 0;
  function write() {
    while (i < lines.length) {
      const ok = writable.write(lines[i++] + "\n");
      if (!ok) { writable.once("drain", write); return; }     // wait until the buffer empties
    }
    writable.end();
  }
  write();
}

Ignoring write()'s return value in a loop is a classic cause of memory blow-ups.

Streaming over HTTP

Download a large file without loading it:

import { createServer } from "node:http";
import { createReadStream } from "node:fs";
import { stat } from "node:fs/promises";
import { pipeline } from "node:stream/promises";

createServer(async (req, res) => {
  if (req.url === "/report.csv") {
    const info = await stat("./report.csv");
    res.writeHead(200, {
      "Content-Type": "text/csv",
      "Content-Length": info.size,
      "Content-Disposition": 'attachment; filename="report.csv"',
    });
    await pipeline(createReadStream("./report.csv"), res).catch(() => res.destroy());
  } else {
    res.writeHead(404).end();
  }
}).listen(3000);

Save an upload by piping the request body straight to disk (with size limits and authentication in real systems):

await pipeline(req, createWriteStream(`./uploads/${crypto.randomUUID()}.bin`));

In Express, res.sendFile and res.download stream files for you, and libraries such as busboy or multer handle multipart uploads in streams. In Express you can also pipeline(stream, res) for generated data, and set Content-Type first. For very large downloads, support the Range header for resumes (res.sendFile does).

Server-Sent Events and chunked responses are streams too: res.write(data) repeatedly, then res.end().

Transform streams

A transform reads chunks, changes them and passes them on. Write one to process data on the fly:

import { Transform } from "node:stream";

const upperCase = new Transform({
  transform(chunk, encoding, callback) {
    callback(null, chunk.toString().toUpperCase());          // callback(error, outputChunk)
  },
});

await pipeline(process.stdin, upperCase, process.stdout);

Remember a chunk is a buffer of bytes, not a line or a character: a chunk can end in the middle of a line or a multi-byte character (such as Hindi text in UTF-8). For lines, use node:readline (below); for text decoding across chunk boundaries, use setEncoding("utf8") or a TextDecoderStream, not chunk.toString() on each chunk.

Reading lines and async iteration

Readable streams are async iterable:

import { createReadStream } from "node:fs";
import { createInterface } from "node:readline";

const lines = createInterface({ input: createReadStream("orders.csv"), crlfDelay: Infinity });

let total = 0;
for await (const line of lines) {                     // one line at a time, constant memory
  const [id, customer, amount] = line.split(",");
  if (id !== "id") total += Number(amount);
}
console.log("Total:", total);

for await (const chunk of stream) consumes any readable. Streams are also built from generators with Readable.from:

import { Readable } from "node:stream";

async function* rows() {
  for (let i = 1; i <= 1_000_000; i++) yield `${i},order-${i}\n`;     // produce lazily
}
await pipeline(Readable.from(rows()), createWriteStream("big.csv"));

For proper CSV (quotes, commas inside fields) use a parser library (csv-parse) as a transform, not split(",").

Object mode

By default streams carry bytes or strings; with objectMode: true they carry arbitrary JavaScript objects, handy for pipelines of records:

const toRecords = new Transform({
  objectMode: true,
  transform(line, _enc, cb) { const [id, name] = line.split(","); cb(null, { id: Number(id), name }); },
});

Errors and cleanup

  • With pipeline, an error in any stream rejects the promise and destroys the others; wrap in try/catch.
  • With plain .pipe(), you must attach error handlers to every stream, or you get unhandled errors and leaked file descriptors.
  • Always end writable streams (end()), and destroy them on errors.
  • Use AbortSignal with pipeline(..., { signal }) to cancel.
  • A client that disconnects mid-download should stop the server work: pipeline destroys the source when the response closes.

Web streams

Browsers and Node.js share the Web Streams API (ReadableStream, WritableStream, TransformStream), also what fetch uses for response.body. Node.js converts between them:

import { Readable } from "node:stream";

const response = await fetch("https://example.com/large.csv");
await pipeline(Readable.fromWeb(response.body), createWriteStream("large.csv"));   // stream a download to disk

New code written for both browser and Node.js can use web streams; for server-only code, Node.js streams and pipeline remain the most convenient.

When not to use streams

Small data (a configuration file, a JSON of a few KB) is simpler with readFile and JSON.parse. Streams add complexity, so use them when data is large, unbounded or arrives over time. Remember that JSON.parse needs the whole text; for huge JSON use a streaming parser (stream-json) or newline-delimited JSON (NDJSON), with one record per line, which streams naturally.

Common mistakes

  • Reading large files with readFile and running out of memory.
  • Using .pipe() without error handlers or pipeline.
  • Ignoring backpressure when writing in a loop.
  • Splitting text on chunk boundaries (broken lines or characters).
  • Forgetting to end or destroy streams, leaking file handles.
  • Parsing CSV with split(",").
  • Streaming an upload with no size limit or authentication.

Try it yourself

  1. Create a 500 MB test file and compare memory use (process.memoryUsage()) of readFile and createReadStream.
  2. Compress a file with pipeline, then decompress it with createGunzip().
  3. Count the total of a CSV column with readline and for await.
  4. Serve a large file over HTTP and watch the browser download it while memory stays flat.
  5. Write a transform that converts each line of a log file to JSON, and pipe it to a new file.

Check yourself

Why use streams for large files?

They process data in small chunks, so memory use stays small and constant and processing can start before the whole file is read.

What is backpressure?

A mechanism where a slow consumer signals the producer to pause (writes return false until drain), preventing unbounded buffering in memory.

Why prefer pipeline to .pipe()?

pipeline propagates errors, cleans up all streams on failure and returns a promise, while .pipe() leaves error handling and cleanup to you.

How do you read a large text file line by line?

Create a read stream and iterate it with readline.createInterface and for await...of.

Interview questions

What are streams in Node.js and what types exist?

Interfaces for handling data in chunks over time. The four types are readable, writable, duplex and transform; they are event emitters and async iterables, and many Node.js APIs (files, HTTP, sockets, zlib) use them.

How would you process a 10 GB log file in Node.js?

Create a read stream, process it line by line with readline or a transform inside pipeline, keep memory constant, handle errors and backpressure, and write results incrementally.

What is the difference between a Buffer and a stream?

A buffer is a fixed block of bytes in memory; a stream is a flow of many such chunks over time, letting you handle data without holding it all at once.

Summary

  • Streams move data in chunks, keeping memory low and enabling early processing.
  • Use pipeline from node:stream/promises to connect streams safely; it handles errors and backpressure.
  • Stream files, uploads and downloads; use readline and for await for lines, transforms for processing and web streams via Readable.fromWeb.
  • Reserve streams for large or continuous data; use proper parsers for CSV and JSON.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Node.js and Express backends for Full Stack Developer and scores every answer, with what to fix.