node.js: create a connected writable and readable stream pair

ONLINECAST

2 min read

·

06-10-2024

4

0

Share

Mastering Node.js Streams: Creating a Connected Writable and Readable Pair

Working with streams in Node.js offers an efficient way to handle large amounts of data without overwhelming your memory. Often, you need a way to connect a writable stream to a readable stream to enable seamless data flow. This article explores the fundamental concepts of streams and provides a clear guide on creating a connected writable and readable stream pair.

Understanding the Problem:

The challenge lies in establishing a channel for data to flow between two streams without intermediaries. We want to write data to the writable stream and read it from the readable stream, ensuring a direct connection.

Scenario and Code:

Let's consider a scenario where we want to process data from a file and write the processed data to another file. We can achieve this using two streams: a fs.createReadStream for reading from the source file and a fs.createWriteStream for writing to the destination file. The original code might look like this:

const fs = require('fs');

const readStream = fs.createReadStream('input.txt');
const writeStream = fs.createWriteStream('output.txt');

readStream.on('data', (chunk) => {
  // Process data chunk here
  writeStream.write(chunk); 
});

readStream.on('end', () => {
  writeStream.end();
});

This approach works, but it involves manually handling data chunks and writing them to the destination stream. This can become cumbersome with complex processing logic.

A Stream-Based Solution:

Node.js offers a powerful solution using the stream.Duplex class. A Duplex stream acts as both a writable and readable stream, allowing us to establish a direct connection between the two.

const { Duplex } = require('stream');

class MyDuplex extends Duplex {
  constructor(options) {
    super(options);
    this.buffer = [];
  }

  _write(chunk, encoding, callback) {
    this.buffer.push(chunk);
    callback();
  }

  _read(size) {
    const chunk = this.buffer.shift();
    if (chunk) {
      this.push(chunk);
    } else {
      this.push(null); // signal end of stream
    }
  }
}

const myDuplex = new MyDuplex();
const readStream = fs.createReadStream('input.txt');
const writeStream = fs.createWriteStream('output.txt');

readStream.pipe(myDuplex).pipe(writeStream);

Explanation:

1. MyDuplex Class: We create a custom Duplex class to manage the data flow.
2. _write() Method: This method receives data chunks from the writable side. We store these chunks in a buffer for processing.
3. _read() Method: This method is called when the readable side requests data. We retrieve and push data chunks from the buffer.
4. pipe() Method: The pipe() method connects the streams seamlessly. Data written to readStream flows directly to myDuplex, and then to writeStream.

Key Benefits:

• Simplicity: The Duplex class simplifies the process of creating connected streams.
• Efficiency: The pipe() method handles the data flow automatically, eliminating manual handling of chunks.
• Flexibility: The MyDuplex class can be customized to implement various data processing logic.

Additional Tips:

• Use stream.Transform for streams that modify data in transit.
• Implement error handling and logging to monitor the stream's performance.
• Consider using third-party libraries like through2 for easier stream manipulation.

Conclusion:

Understanding Node.js streams, especially the Duplex class, empowers you to build efficient and flexible data processing pipelines. By creating connected writable and readable streams, you can establish direct data flow without intermediaries, making your applications more efficient and scalable.

Resources:

• Node.js Stream Documentation
• Through2 Library