Node.js certification JSNAD - Readable streams, Writable streams, Duplex streams, Transform streams and Passthrough

Last updated Oct 24, 2021 Published Jan 16, 2021

The content here is under the Attribution 4.0 International (CC BY 4.0) license

I already wrote about streams in PHP and how to use them, in the same way this post focuses on the Node.js streams. The aim here is to cover the streams in Node.js, the concepts are fundamentally the same, but each language has its implementation detail.

This post is based on the content provided by the hey(node) list to prepare for the nodejs certification exam (JSNAD) [1]

Benefits of streams

Streams are a collection of data that isn’t available all at once, instead, we can work on that, piece by piece, it improves performance and also avoids the spiking of the server resources. For example, for files, with streams we don’t need to read all the content in memory at once. Reading the entire file at once can lead to high consumption of the memory.

Streams type

  1. Readable streams: A stream for read data, a source of input. You can’t write
  2. Writable stream: A stream for writing data, the destination of the data to write to
  3. Duplex stream: This one implements both, writable and readable
  4. Transform: Like Duplex, but transforms the data passing through
  5. Passthrough: A special type of Transform that, does not change the data passing through

Stream events

Streams can be consumed using events, as such readable and writable streams have their events to handle data. This section describes the differences between each event used by Node.js streams and also lists open-source projects that are using streams.

Readable stream events

  • data: Emitted when a chunk of data arrives
  • readable: Emitted when data is waiting for being consumed
  • end: Emitted when there is no more data available
  • error: An error has occurred

Writable stream events

  • drain: The writable streams buffer has been cleared and is waiting for more data
  • finish: All data has been written
  • error: An error occurred while writing the data

[2] in its examples folder [3] uses the event streams as described in the previous section, the following code snippet was extracted from the commented source and modified to depict the Readable stream usage:

const fs = require('fs')
const csv = require('csv')

console.time('stream_file')

fs.readFile('./dump.csv', (error, content) => {
  if (error) {
    throw error
  }

  const generator = csv.parse(content)
  
  generator.on('readable', function() {
    while(data = generator.read()){
      console.log(data)
    }
  });
  
  generator.on('end', function() {
    console.timeEnd('stream_file')
    console.log('finished')
  });

  generator.on('error', function() {
    console.error('something went wrong')
  })
})

Even though the readFile loads the content of the file at once in memory, this example depicts the usage of the events: readable, end and error. Keep in mind that this approach depicts the usage of the event only, but, nodejs offers the pipe as well.

Appendix

[4] did a live stream about nodejs and data stream, it is a deep dive into the implementation. It’s recommended for developers who prefer a video-style approach.

References

  1. [1]J. Church, “Node.js Certification Study Guide by Hey Node - What is a node.js Stream,” 2021 [Online]. Available at: https://heynode.com/tutorial/what-stream. [Accessed: 16-Jan-2021]
  2. [2]W. David, “node-csv,” 2021 [Online]. Available at: https://github.com/adaltas/node-csv. [Accessed: 16-Jan-2021]
  3. [3]W. David, “node-csv,” 2021 [Online]. Available at: https://github.com/adaltas/node-csv/blob/master/samples/stream.js. [Accessed: 16-Jan-2021]
  4. [4]J. Duque, “[MNTD] Node.js ¡En Vivo! - Real-Time Streaming Data - Streams,” 2021 [Online]. Available at: https://www.youtube.com/watch?v=mLYsxM0Yp2A&t=1165s. [Accessed: 18-Apr-2021]
Made with 💖 by @MatheusMarabesi