Photo byTom Parkes on Unsplash
Abstract
In my previous blog post Breaking Chains with Pipelines in Modern JavaScript, I explained how modern JavaScript features, in particular iterators and generators, can be used to implement a tiny yet powerful iteration library. Such a library makes it possible to process and transform both built-in collections, such as arrays and maps, as well as user-defined collections, in a functional manner.
And thanks to iterators and generators, the implementation is highly efficient - computing values on demand while avoiding allocation of temporary containers. That post also highlights how the upcoming JavaScript pipeline operator makes it easy to both read and write code implemented using that library.
In this followup post I take this approach to a whole new level, showing how it can be extended to not only handle collections of items but also sequences of events. In other words, it can iterate both over containers of elements in (memory) space, and over elements generated dynamically in time.
You may know this approach as Functional Reactive Programming (FRP), as implemented by libraries such as RxJS. Unlike such libraries, the library I implement is exceedingly simple and small, yet implements all the expected functionality. This is thanks to the use of new JavaScript features, especially asynchronous iterators and generators, and the for await statement. In fact, much of the code is nearly identical to the library shown in the previous post.
Introducing Asynchronous Iterators and Generators
JavaScript iterators provide a generic mechanism to traverse collections of items. This makes it possible to create simple code that can work with almost any type of collection, both built-in and user-defined. For example, the following code prints out all the values stored in a collection, and will work with a collection of type Array or of type Map, or of a type I implement myself, as long as it has a proper [Symbol.iterator] method
Alternatively, I can implement this using the library described in the previous post, as:
Internally, that library uses the exact same iteration mechanism.
But suppose that instead of a collection of items, I need to process a sequence of events that are generated sequentially, one after the other. And that I don’t want to wait until all the events are generated; rather, I want to process them immediately as they arrive. For example, such events might be user interactions with a UI element, such as clicks on a button, or data received over the network, a packet at a time.
I want to be able to handle such data in the same way as items in a collection. In the past this required using third-party libraries, such as RxJS. And while such libraries provide a lot of power, they can also have significant overhead, both in terms of code size and complexity, and in terms of cognitive overhead required to learn them.
Wouldn’t it be great if such functionality was built into JavaScript itself, and could be implemented directly using language primitives? Well, now it can, thanks to asynchronous iterators and generators!
Asynchronous iterators work like regular iterators, with a few notable differences:
The asynchronous iterator instance is obtained by invoking the [Symbol.asyncIterator] method, instead of [Symbol.iterator]
Rather than returning an object that has a value and done properties, as per the iteration protocol, the next method of an asynchronous iterator returns a promise, which resolves to an object that has these properties
As an implicit aspect of the asynchronous iteration protocol, the next promise is not requested until the previous one resolves
Assuming object sequence implements asynchronous iteration, we can process the events that flow through it, as they arrive, using the following simple code:
This code is almost identical to the code at the beginning of this section, with the minor addition of the await keyword. But the behavior is very different in that the code inside the loop will be invoked asynchronously, as events arrive.
Here is an example of how this mechanism can be used to count from 0 to 9, with a delay of one second between each output:
This code is low-level in that the asynchronous iterator is created and managed explicitly. An easier approach would be to use an async generator to create such an iterator, as it’s easier to use a regular generator for creating synchronous iterators, for example:
An extra benefit of using for await is that if the specified object we are iterating over doesn’t implement the [Symbol.asyncIterator] method, for await will automatically try to fallback to [Symbol.iterator], and use a synchronous iterator instead.
In a scenario like that, for await behaves like a regular for of. This means that the same iteration code can be used for both asynchronous sequences and regular collections.
The Case For Asynchronous Iteration Methods
Having said all that, what we really want is to be able to use iteration methods for, is to process the values provided by an asynchronous iterator - for the same reasons that we prefer to use iteration methods for processing collections.
Combined with the pipe operator (see my previous blog post for details about this upcoming extension to the JavaScript language), this would enable us to write the following code to print out only the even numbers provided by a numeric sequence:
As with regular iteration methods, the benefits are that you can deconstruct complex operations into a sequence of simple operations, and that each such operation is specified in a declarative manner: what you want to achieve, rather than the details of how to achieve it.
Unfortunately, while JavaScript does provide both asynchronous iterators and generators, it does not provide such asynchronous iteration methods. So, let’s build them ourselves!
Implementing The First Asynchronous Iteration Methods
Some of the most basic iteration methods are map, filter, and forEach. Here is the implementation of these functions in the synchronous library described in the previous post:
It turns out that enhancing these methods to support asynchronous sequences is very easy to do in modern JavaScript:
Essentially, the only changes required are the async keyword in front of the function declarations, and the await keyword after for.
That’s really all that is needed! With this in hand, the sample code from the previous section - filtering and logging a sequence of numbers generated over time - just works. This goes to show how powerfully simple the abstraction for asynchronous behavior is in modern JavaScript.
And because these methods all use for await, they also work when src is a regular container, like Array, Map, or String. This is because, as explained above, for await falls back to using regular iterators if asynchronous iterators are unavailable.
This means that this library can actually be used as a replacement for the library presented in my previous post: a single library that can handle both synchronous and asynchronous scenarios.
Creating Asynchronous Event Emitters
It’s great that we’ve so easily created methods that can operate on event sequences, when implemented using asynchronous iterators. But where do the events originally come from? And how do we create asynchronous iterators for them? At the beginning of this post I provided two examples of such event emitters - timedSequence and timedGen - but both were highly specialized for a very simple scenario.
What we need is a mechanism for transforming almost any event source into an emitter which can provide asynchronous iterators.
The popular FRP library RxJS introduces the concept of observables. Observables implement a subscription model, in which consumers explicitly register to receive notifications. (There is even talk of adding some sort of observable mechanism into the JavaScript language itself.) For our simple JavaScript library we need a similar mechanism, but one that is more aligned with asynchronous iterators. Specifically what we need is a mechanism that transforms a sequence of events, provided using a callback function, into an async generator.
Here is the most complicated bit of code in this blog post, which converts a function that uses a callback to provide a sequence of values, into a sequence that supports asynchronous iteration:
The emitter argument is the function that receives a callback and generates the values that will then be provided via the asynchronous iterator, by invoking the callback with values as arguments, one at a time.
To handle a scenario in which values are generated by emitter faster than they can be consumed via the iterator, they are pushed into the values array, which functions as a temporary storage. The promise valuesAvailable is used to indicate when there are values available in the temporary storage.
These values are then yielded one-at-a-time through the asynchronous iterator to a consumer, using the yield* instruction. (yield* is a useful helper that iterates over its operand - in this case the array of pending values - and yields each value returned by it.)
One tricky bit of this code is the initialization of the valuesAvailable promise. The init function passed to the promise constructor saves the resolve callback of the promise (provided as the argument r) into the resolve variable. It also clears the temporary storage values, by resetting it to an empty array.
With fromCallback function in hand, we can easily create an asynchronous generator from a DOM event listener:
And now we can manipulate a stream of events in a functional and readable manner, for example:
This code will output the number of times the button has been clicked so far. As an exercise, see if you can create an implementation of the reduce method, in addition to filter and map implemented above. (You can find an implementation of reduce in the CodePen that I link to below.)
Summary
Using a very small amount of modern JavaScript code, we’ve created a super-simple library that implements functionality which is similar to that provided by heavyweight FRP libraries, such as RxJS. It uses the asynchronous capabilities that are now built-in into JavaScript itself, which makes it both lightweight and performant.
The use of the pipeline operator, which is not yet a part of standard JavaScript, makes the code easy to follow, and has the additional advantage of facilitating tree-shaking as a means to reduce code size.
You can find the complete library implementation in this CodePen. Coupled with some demo code, it’s still less than 160 lines of unminified code! The complete library includes some additional functionality that I will cover in an upcoming post, such as the ability to process events from the same source through multiple streams (like multiple subscribers on a single observable), and how to connect this library to Readable Streams, such as those returned by the Fetch API through the body property of a response object.
It’s important to emphasize that even the complete library code in the CodePen is not yet a fully-featured library that you should use in production. Certainly, it’s not yet an alternative to FRP libraries such as RxJS. But it does highlight the powerful, asynchronous capabilities that are now a part of JavaScript itself, and the way in which such libraries might be implemented in the future.
This post was written by Dan Shappir
You can follow him on Twitter
For more engineering updates and insights:
Visit us on GitHub
Subscribe to our YouTube channel