Lesson 2 of 8 · 11 min · The Node.js runtime
Streams: handling large data in Node.js
What you will learn
- What a stream is and why it uses less memory than loading everything
- The four stream types: readable, writable, duplex and transform
- Connect streams safely with
pipeline, and what backpressure is - Stream files, HTTP uploads and downloads, and compress data
- Read streams with
for awaitand know the web-streams equivalents
In simple words
Imagine moving a swimming pool of water to another place. You could try to lift the whole pool at once, which is impossible, or you could use a hose and let the water flow through. A stream is the hose: data moves in small pieces from a source to a destination, so you can process a 5 GB file or a long download without ever holding all of it in memory.
Words to know
- Stream: a sequence of data pieces (chunks) that is read or written over time.
- Chunk: one piece of data in a stream, often a buffer of bytes.
- Backpressure: a signal from a slow consumer telling the producer to pause.
- Pipe: connecting the output of one stream to the input of another.
Why streams
Reading a file in one call loads all of it into memory:
import { readFile } from "node:fs/promises";
const text = await readFile("orders-2gb.csv", "utf8"); // needs 2 GB of memory (and may fail): don't
With streams, memory use stays small and constant, and the first results appear before the last byte is read. Streams matter for large files, uploads and downloads, video and audio, log processing, compression, database exports, and proxying one service's response to another. HTTP requests and responses, sockets, files, process.stdin and process.stdout are all streams.
The four kinds
| Type | Meaning | Examples |
|---|---|---|
| Readable | a source you read from | fs.createReadStream, HTTP request on a server, HTTP response on a client |
| Writable | a destination you write to | fs.createWriteStream, HTTP response on a server, process.stdout |
| Duplex | both readable and writable | a TCP socket |
| Transform | a duplex that changes data as it passes | zlib.createGzip(), encryption, CSV parsers |
Reading and writing a file with streams
import { createReadStream, createWriteStream } from "node:fs";
import { pipeline } from "node:stream/promises";
import { createGzip } from "node:zlib";
await pipeline(
createReadStream("orders.csv"),
createGzip(), // compress each chunk as it passes
createWriteStream("orders.csv.gz")
);
console.log("done");
pipeline from node:stream/promises connects streams, forwards errors, destroys all streams on failure and returns a promise. Use it instead of the older .pipe(), which does not propagate errors or clean up. Memory stays small however large the file is.
Backpressure
If a fast reader feeds a slow writer (a big file to a slow network), data would pile up in memory. Backpressure solves that: a writable stream's write() returns false when its internal buffer is full, and the producer should pause until the drain event. pipeline and .pipe() handle this automatically, which is another reason to use them. If you write by hand:
function writeMany(writable, lines) {
let i = 0;
function write() {
while (i < lines.length) {
const ok = writable.write(lines[i++] + "\n");
if (!ok) { writable.once("drain", write); return; } // wait until the buffer empties
}
writable.end();
}
write();
}
Ignoring write()'s return value in a loop is a classic cause of memory blow-ups.
Streaming over HTTP
Download a large file without loading it:
import { createServer } from "node:http";
import { createReadStream } from "node:fs";
import { stat } from "node:fs/promises";
import { pipeline } from "node:stream/promises";
createServer(async (req, res) => {
if (req.url === "/report.csv") {
const info = await stat("./report.csv");
res.writeHead(200, {
"Content-Type": "text/csv",
"Content-Length": info.size,
"Content-Disposition": 'attachment; filename="report.csv"',
});
await pipeline(createReadStream("./report.csv"), res).catch(() => res.destroy());
} else {
res.writeHead(404).end();
}
}).listen(3000);
Save an upload by piping the request body straight to disk (with size limits and authentication in real systems):
await pipeline(req, createWriteStream(`./uploads/${crypto.randomUUID()}.bin`));
In Express, res.sendFile and res.download stream files for you, and libraries such as busboy or multer handle multipart uploads in streams. In Express you can also pipeline(stream, res) for generated data, and set Content-Type first. For very large downloads, support the Range header for resumes (res.sendFile does).
Server-Sent Events and chunked responses are streams too: res.write(data) repeatedly, then res.end().
Transform streams
A transform reads chunks, changes them and passes them on. Write one to process data on the fly:
import { Transform } from "node:stream";
const upperCase = new Transform({
transform(chunk, encoding, callback) {
callback(null, chunk.toString().toUpperCase()); // callback(error, outputChunk)
},
});
await pipeline(process.stdin, upperCase, process.stdout);
Remember a chunk is a buffer of bytes, not a line or a character: a chunk can end in the middle of a line or a multi-byte character (such as Hindi text in UTF-8). For lines, use node:readline (below); for text decoding across chunk boundaries, use setEncoding("utf8") or a TextDecoderStream, not chunk.toString() on each chunk.
Reading lines and async iteration
Readable streams are async iterable:
import { createReadStream } from "node:fs";
import { createInterface } from "node:readline";
const lines = createInterface({ input: createReadStream("orders.csv"), crlfDelay: Infinity });
let total = 0;
for await (const line of lines) { // one line at a time, constant memory
const [id, customer, amount] = line.split(",");
if (id !== "id") total += Number(amount);
}
console.log("Total:", total);
for await (const chunk of stream) consumes any readable. Streams are also built from generators with Readable.from:
import { Readable } from "node:stream";
async function* rows() {
for (let i = 1; i <= 1_000_000; i++) yield `${i},order-${i}\n`; // produce lazily
}
await pipeline(Readable.from(rows()), createWriteStream("big.csv"));
For proper CSV (quotes, commas inside fields) use a parser library (csv-parse) as a transform, not split(",").
Object mode
By default streams carry bytes or strings; with objectMode: true they carry arbitrary JavaScript objects, handy for pipelines of records:
const toRecords = new Transform({
objectMode: true,
transform(line, _enc, cb) { const [id, name] = line.split(","); cb(null, { id: Number(id), name }); },
});
Errors and cleanup
- With
pipeline, an error in any stream rejects the promise and destroys the others; wrap intry/catch. - With plain
.pipe(), you must attacherrorhandlers to every stream, or you get unhandled errors and leaked file descriptors. - Always end writable streams (
end()), and destroy them on errors. - Use
AbortSignalwithpipeline(..., { signal })to cancel. - A client that disconnects mid-download should stop the server work:
pipelinedestroys the source when the response closes.
Web streams
Browsers and Node.js share the Web Streams API (ReadableStream, WritableStream, TransformStream), also what fetch uses for response.body. Node.js converts between them:
import { Readable } from "node:stream";
const response = await fetch("https://example.com/large.csv");
await pipeline(Readable.fromWeb(response.body), createWriteStream("large.csv")); // stream a download to disk
New code written for both browser and Node.js can use web streams; for server-only code, Node.js streams and pipeline remain the most convenient.
When not to use streams
Small data (a configuration file, a JSON of a few KB) is simpler with readFile and JSON.parse. Streams add complexity, so use them when data is large, unbounded or arrives over time. Remember that JSON.parse needs the whole text; for huge JSON use a streaming parser (stream-json) or newline-delimited JSON (NDJSON), with one record per line, which streams naturally.
Common mistakes
- Reading large files with
readFileand running out of memory. - Using
.pipe()without error handlers orpipeline. - Ignoring backpressure when writing in a loop.
- Splitting text on chunk boundaries (broken lines or characters).
- Forgetting to end or destroy streams, leaking file handles.
- Parsing CSV with
split(","). - Streaming an upload with no size limit or authentication.
Try it yourself
- Create a 500 MB test file and compare memory use (
process.memoryUsage()) ofreadFileandcreateReadStream. - Compress a file with
pipeline, then decompress it withcreateGunzip(). - Count the total of a CSV column with
readlineandfor await. - Serve a large file over HTTP and watch the browser download it while memory stays flat.
- Write a transform that converts each line of a log file to JSON, and pipe it to a new file.
Check yourself
Why use streams for large files?
They process data in small chunks, so memory use stays small and constant and processing can start before the whole file is read.
What is backpressure?
A mechanism where a slow consumer signals the producer to pause (writes return false until drain), preventing unbounded buffering in memory.
Why prefer pipeline to .pipe()?
pipeline propagates errors, cleans up all streams on failure and returns a promise, while .pipe() leaves error handling and cleanup to you.
How do you read a large text file line by line?
Create a read stream and iterate it with readline.createInterface and for await...of.
Interview questions
What are streams in Node.js and what types exist?
Interfaces for handling data in chunks over time. The four types are readable, writable, duplex and transform; they are event emitters and async iterables, and many Node.js APIs (files, HTTP, sockets, zlib) use them.
How would you process a 10 GB log file in Node.js?
Create a read stream, process it line by line with readline or a transform inside pipeline, keep memory constant, handle errors and backpressure, and write results incrementally.
What is the difference between a Buffer and a stream?
A buffer is a fixed block of bytes in memory; a stream is a flow of many such chunks over time, letting you handle data without holding it all at once.
Summary
- Streams move data in chunks, keeping memory low and enabling early processing.
- Use
pipelinefromnode:stream/promisesto connect streams safely; it handles errors and backpressure. - Stream files, uploads and downloads; use
readlineandfor awaitfor lines, transforms for processing and web streams viaReadable.fromWeb. - Reserve streams for large or continuous data; use proper parsers for CSV and JSON.
Going deeper
- Node.js: Stream API: readable, writable, duplex and transform streams.
- Node.js: Backpressuring in streams: why and how.
- Node.js: stream/promises pipeline: the recommended connector.
- Node.js: readline: reading line by line.
See how much of this you can explain
A free mock interview asks you new questions on Node.js and Express backends for Full Stack Developer and scores every answer, with what to fix.