Node.js certification JSNAD - Readable streams, Writable streams, Duplex streams, Transform streams and Passthrough
The content here is under the Attribution 4.0 International (CC BY 4.0) license
I already wrote about streams in PHP and how to use them, in the same way this post focuses on the Node.js streams. The aim here is to cover the streams in Node.js, the concepts are fundamentally the same, but each language has its implementation detail.
This post is based on the content provided by the hey(node) list to prepare for the nodejs certification exam (JSNAD) [1]
Benefits of streams
Streams is a collection of data which isn’t available all at once, instead, we can work on that, piece by piece, it improves performance and also avoids the spiking the server resources. For example, for files, with streams we don’t need to read all the content in memory at once. Reading the entire file at once can lead to high consume on the memory.
Streams type
- Readable streams: A stream for read data, a source of input. You can’t write
- Writable stream: A stream for write data, the destination of the data to write to
- Duplex stream: This one implements both, writable and readable
- Transform: Like Duplex, but transforms the data passing through
- Passthrough: A special type of Transform but, does not change the data passing through
Stream events
Streams can be consumed using events, as such readable and writable streams have its own events to handle data. This section describes the differences of each event used by Node.js streams and also list open source projects that are using streams.
Readable stream events
- data: Emitted when a chunk of data arrives
- readable: Emitted when there is data waiting for being consumed
- end: Emitted when there is no more data available
- error: An error has occurred
Writable stream events
- drain: The writable streams buffer has been cleared and is waiting for more data
- finish: All data has been written
- error: An error occurred while writing the data
[2] in its examples folder [3] uses the event streams as described in the previous section, the following code snippet was extracted from the commented source and modified to depict the Readable stream usage:
const fs = require('fs')
const csv = require('csv')
console.time('stream_file')
fs.readFile('./dump.csv', (error, content) => {
if (error) {
throw error
}
const generator = csv.parse(content)
generator.on('readable', function() {
while(data = generator.read()){
console.log(data)
}
});
generator.on('end', function() {
console.timeEnd('stream_file')
console.log('finished')
});
generator.on('error', function() {
console.error('something went wrong')
})
})
Even though,the readFile loads the content of the file at once in memory,
this example depicts the usage of the events: readable
, end
and error
.
Keep in mind that this approach depicts the events usage only, but, nodejs
offers the pipe
as well.
Appendix
[4] did a live stream about nodejs and data stream, it is a deep dive in the implementation. I it’s recommended for developers that prefer a video style approach.
References
- [1]J. Church, “Node.js Certification Study Guide by Hey Node - What is a node.js Stream,” 2021 [Online]. Available at: https://heynode.com/tutorial/what-stream. [Accessed: 16-Jan-2021]
- [2]W. David, “node-csv,” 2021 [Online]. Available at: https://github.com/adaltas/node-csv. [Accessed: 16-Jan-2021]
- [3]W. David, “node-csv,” 2021 [Online]. Available at: https://github.com/adaltas/node-csv/blob/master/samples/stream.js. [Accessed: 16-Jan-2021]
- [4]J. Duque, “[MNTD] Node.js ¡En Vivo! - Real-Time Streaming Data - Streams,” 2021 [Online]. Available at: https://www.youtube.com/watch?v=mLYsxM0Yp2A&t=1165s. [Accessed: 18-Apr-2021]
Table of contents
Got a question?
If you have question or feedback, don't think twice and click here to leave a comment. Just want to support me? Buy me a coffee!