We love telling stories behind the daily challenges we face and how we solve them. But we also love hearing about the insights, experiences and the lessons learned by prominent voices in the global community.
In this new series of 1-on-1 interviews, our very own Wix Engineers talk directly with some of the most inspiring minds in the tech industry. From software engineering to architecture, open source, and personal growth.
In this third installment in the series Wix’s Dean Shub spoke with the principal engineer of Lululemon Zack Jackson.
When you’re done here, don’t forget to check out the rest of the series:
Want to know what the future of Webpack is going to be? Or maybe you are looking for that newest, fastest way to build js code? Now, that’s just 5% of the in-depth technical subjects and topics Zack and Dean go into in this fascinating conversation, covering everything from module federation, to monorepos, to so much more. Enjoy!
Dean: How did you come to be a Webpack maintainer?
Zack: I believe it started probably around 2015. I don't know if you are familiar with the React server-side rendering stuff - basically, there are two separate builds in all of that, that kind of creates it, and you correlate the server build to the client build and that's how you server-side-render all your chunks out. I was originally involved in the first project that achieved that in React code.
Part of that was a Webpack plugin for CSS. And we would figure out when the CSS for the chunk would load, and when the other scripts for the chunk would load, and then we would actually, you know, make the page hydrate when those modules got loaded.
Overtime, what we ended up doing was just adapting that to run on its own. And the big thing that it had was hot reloading, which mini-CSS didn't have at the time. And so how I ended up actually getting involved with the Webpack Foundation was I ended up merging that project with mini-CSS to bring hot reloading and a couple other features to mini-CSS. From there, I just continued working with Webpack and more of the dependencies from there. But yeah, I've always just had an interest in compilers. It's just a lot of power that you can augment from them, that you would usually struggle to try and implement at a higher level like in a framework or in an engineering pattern that you can kind of automate to some extent through the compiler.
Dean: I promise we'll get to compilers in a bit, but let me start first by asking about module federation - how would you explain this feature?
Zack: I would say that module federation is a way to import pieces of code from independently deployed applications at runtime. So if I wanted to, say, import a feature or share a dependency between two standalone applications where there’s the ability for them to overlap (let's say they're both React applications, something like that), they could share one single dependency of React. Or the big one was if I'm building up micro frontends or something like that and I need the ability to just import a feature or an actual component that another team owns, instead of having to go over to npm I could just import it at runtime through module federation.
Dean: So as you see it, is it a solution for a scaling problem or for a performance problem?
Zack: I would say that the performance is a side effect of the design, it was mostly geared towards solving the scaling issue.
Dean: How did you come to realize that such a feature is needed? What was the real world use case? Zack: I was working at Fiverr at the time and we had about 150 micro frontends that we had to manage. And we were managing them with things like Webpack externals - so redux / react, putting it in an externa chunk... We had some challenges when we would go from React 15 to React 16 at the time - how would you get an external vendor chunk with two different types of React in it? If the external chunk goes missing, the pages stop working... It was just a very brittle kind of solution - especially when you scale it up. It wasn't elegant.
But there was no better solution, really, out there at the time. So that got me thinking that there must be some way where I can just import the 20 kilobytes of code that I actually need from this bundle - because I already have React or already have Lodash or already have, you know, 80 percent of whatever else I need - if I just want those 20 kilobytes that's the page code or the feature code, there must be away to import it.
And so that's kind of what started the journey - around late 2018, that's when I started to really research it, and it took about two years to actually get something out there.
Dean: There's a lot of solutions - like monoliths and monorepos - and I recently found out there's a concept called virtual monorepo where you get like the git feature of submodules and do a submodule of gits and then have a Webpack that aggregates every other Webpack in order to solve that exact issue. What do you think about such solutions instead of module federation?
Zack: I think that there's still, even with monorepos, you still have at least 2 or 3 challenges that have cropped up for me around them. One is that there’s still the bottleneck of there being just a single branch, so even if you solve many other things... If you run tests or something like that - for instance, we saw this on one repo we had where there were about 800 deploys a day to it and the tests take about 15 minutes, so you're sitting there and updating the branch, updating the branch, trying to get your tests to pass before the next merge comes along and you just get stuck in this loop. Or it could take like an hour to actually merge the PR if you're not watching it the whole time.
So I do think that there still are some challenges around the mono repo space just in terms of the actual bottlenecks that you get in the merge pipeline. But... I use a mixture of it. So I still use federation, but I also use monorepos. And what I try to do with them is create a construct that’s almost like a sub-app system. So I’ll have a server instance or something like that and then I'll have the sub-applications that live on it in the monorepo. And that lets me scale horizontally and vertically, because once that repo gets too large, it's already set up kind of like a package internally. Meaning I can easily move it into another monorepo or an empty monorepo, and just keep on developing inside there. Because everything is kind of structured to be somewhat independent.
So it lets me increase the size of the repo, quickly add more teams onto an existing infrastructure that we have... But if I need to split it up, it's still kind of designed to be split up and go off into the more granular multirepo approach. So that's mostly how I try to gear it.
For instance, I use Rush Stack on a couple of things, which is quite nice for things like versioning monorepos and managing change logs and running builds and finding the difs, but still, it takes a lot of time to run part of the pipeline, it's not extremely fast, and if you have a lot of tests on there it can just get slower. So still I haven't found a truly scalable solution that's not expensive to implement that could solve, you know, fifty teams working on a single repo.
Dean: Let us know when you find one (smiling). So about module federation - would you say there is a difference between back end usage of it and front end?
Zack: Not entirely. I think that there is more immediate use for the front end because that's where we've historically had the limits around bundle size or have some things bundled and it can only be done in a specific way. On the back end we have more luxury around not having to worry about the latency issues as much, or it's easier to do microservices in back end because bandwidth isn't a major problem. You can just set up another container with all the dependencies and it's running.
How the actual mechanics of it work though are pretty much the same. If I do server-side rendering or something like that through federation, it all works the same. But backend-wise there's definitely some very interesting things that you can do with it - newer patterns that come out... For example, you could stream the code from, say, github's API, if you wanted, directly into a back end service and not have any S3 buckets or any kind of “deployment stage”. It can be transparent directly from GitHub - and Webpack can read that code in runtime right off of the GitHub API. Which offers some very interesting deployment mechanisms…
I also think that the ability to use federation with web assembly has some pretty interesting opportunities for polyglot development and orchestration of polyglot code.
Dean: Would you say there is a use case where you should not use module federation?
Zack: You probably don't want to do it unless you have a need for it. Unless you have the scaling challenges where you actually need federated code, it's not something you necessarily want to add on as an overhead.
What I would suggest is that if somebody thinks that they might - to try and just design the application to be self-sufficient or not to be heavily-bound, and then at a later stage decide if you actually need federated code. And if you do, then it won't be as much of a challenge to handle the architectural changes needed to support how federation can dynamically load different parts of your code.
When you have a wrapping application, or something like, where you generally develop a lot of dependencies higher up the tree, if we’re talking about React, you could have providers or something like that. But if you're pulling in a component to an app that doesn't have those providers at all - there's no way to encapsulate that, so you would want to bear in mind some of those patterns when developing with a federated app.
Dean: Because it can give you a blind eye about this stuff, yes... I did promise to get back to compilers - there's a new thing on the block which is called esbuild. Can you share your opinion on it?
Zack: I'm a big fan of esbuild. We actually got federation partially working on esbuild. It is really quick, I've enjoyed it a lot. When I need something done fast, even using Webpack, I often swap up Babel with the esbuild loader. Just because it's efficient, it's fast, and it's really nice to see tooling go beyond Javascript. That does create some kind of accessibility challenges for general Javascript engineers, but I think it is a good step for some of these lower-level tools to consider. Just to capitalize on the performance and the multithreaded capabilities that other languages have more out of the box.
Dean: So you don't use esbuild as a standalone tool, you use it as a loader from Webpack, right? Zack: Yes. Everything I do is Webpack-based. Just because generally with anything I do, I need more flexibility. Often Webpack gets a reputation of “something that spits out Javascript” and you think about it almost like it was, basically, Babel with a couple extra features or something like that. But the reality of it is that a lot of these bundlers and transpilers, they just handle converting code at some speed. Whereas what I'm mostly leveraging in all of my work is the orchestration of that runtime that they have on top of them.
So something like esbuild is really good for doing a one-to-one translation, but it doesn't have a parser that's accessible or a dependency factor, or anything like that. So all I can do is some plugins and things like that but I can't create complex rules or orchestrate a runtime easily just with esbuild.
Dean: You have just sort of answered my next question already, which was going to be if you thought it was an easy alternative for wrapping...
Zack: It really depends on what you're trying to do with it. If all you need is something to build code, then happily. But if you have really complex requirements or you want any kind of orchestration, it gets a little bit more complicated to do without the help of that extra API layer that Webpack has.
Dean: There's a library called Loadable Components which lazily-loads React components. So basically it does the same stuff as module federation. Why wouldn't you include the API of Loadable Components and achieve that as well?
Zack: Loadable Components works inside of an application where it already has access to the Webpack chunk graphs. And the big challenge that we have with federated code is that you don't know (or you don't have access to) their Webpack graphs when you're executing that code. You would need a way to manually go and loop the graphs together. And that's one of the reasons why Loadable doesn't work with federation.
We do have solutions for this, they’re just nor part of the Loadable Components spec. I don't think that we have a whole lot publicly available about this, but in some of our federation group repos and things like that we have developed solutions where you can preload the code ahead of time, or we've got server-side rendering systems in place now where we render it similar to React server components. We render it on an edge node and the edge node will return which scripts and CSS came with it and then we use a similar kind of mechanism where it goes through context and we push all the chunks up in the context and render them out at the top of the app.
So very similar to how Loadable Components collect or flush chunk extraction works. The main challenges that we have here are several. One is how Babel processes the import. Usually it gives it a name or a magic comment, so it can track down that file elsewhere and push some name into a map. The problem comes when it’s a federated import - it can't push the name the same way it could in a normal Webpack build, because Webpack transforms those modules differently then it would a normal dynamic import.
So there would need to be some special adjustments made in order to detect if this is an exposed module or not. And if it is, then we wouldn’t mutate the name or change the import. Often what it does is just turn it into a factory object, so instead of it being your dynamic import it's like an object with async, sync and a key. And on the server it runs the require synchronously and in the browser it will run the dynamic import synchronously - but with federation you would need to have your app set up in a certain way to run that synchronous require statement. You'll have to have an async pipeline somewhere up the application stream to actually do that.
It gets a little bit more complicated there because of how the imports are changed and because we don't have access to the actual chunk manifest. So in a case of, say, app1/button, we don't know what app1/button is in the compiled chunk because the server isn't aware of what it is either. That's the big challenge we have with something like Loadable Components. It totally can be done, it's just the case that it hasn't been supported by most of the async React ecosystems that are out there.
Dean: We actually did something very similar to what you just described, writing custom code for Wix to support it. Zack: It's not terribly hard to do, it's just the case that unless there is a big motivation to do it, it takes longer for these things to trickle down into open source. Especially on something as new as this.
Generally speaking, when a large company solves a problem, it most probably solves it internally. And contributing back is just more complicated. But also most of the time whatever solutions they put together might be more leaning toward the system that they have wrapped around it rather than a very agnostic function like Loadable Components - which works in a very repeatable way and it's not very specific to any architecture pattern.
Dean: Module federation has a feature which is called Required Version, which actually gives us the ability to use semantic versioning to load shared dependencies. How do you tackle the development experience where people actually develop with one version of the dependency but in production they get a different version? How would you tackle such an issue?
Zack: This is a problem that is still not fully addressed. What I personally do is just use strict versions on everything. None of my packaged JSONs have a caret symbol in front of them - and so if they don't have that little caret, then Webpack will enforce the version at its exact number. And then I use Renovate bot or something like that across all my repos to keep everything in sync.
The only things that I cater for would be, you know, if I have a React version and I already use singleton on it, then I just set required version to false. If I'm using, say, a nextJS application, I don't really have much choice on what React is available, I can't change it out or anything like that because of how Next is built - so in some of those cases I just say “whichever copy of React is on the page, that's the copy that I’m gonna have to use”, regardless of if it's “my” version or not, I have no choice. So there is some give and take around that aspect.
I would say good end-to-end tests will probably be the best strategy. And, you know, trust the SemVers that you use - which is harder in the open source space, or even internally, if you have internal libraries... A patch version can actually change some prop name and end up breaking something, but I think it's just a risk issue with how we semantically version things. There's always going to be some variance there unless you really want to go for a one-for-one tightly-locked-down dependency tree.
Dean: How do you see future plans for module federation? Where do you see it going?
Zack: I could see it helping a lot in orchestration. I've seen some very interesting uses of it. Like instead of pulling the data to where the code is hosted, pull the code to where the data is hosted for faster execution. So using federation on the backend, to load code into a specific backend where the database is right there instead of loading the data wherever the code is and kind of proxying everything around it, which is quite interesting.
Some of the other cool things that I've seen are to do with data models. Because it's all in Javascript, you can transmit the data model between different flows and you can transmit it with methods and with other things on it, instead of having to serialize it or use it only as data - you can pass it in and out of other systems with a set of methods attached to the actual data model itself. And then serialize it at some point if you needed to store it, or something like that.
I see some very interesting things in backend architecture, hexagonal architecture, that can be unlocked. For front end, some of the interesting places I see it going would be where we'd have something like a runtime log file - ability to change the versions that are used, or the nested versions of the federated modules used. Even control the shared modules - so if you have three apps on the page and they're each, say, a different patch version of React, a user interface can be created that says “cool, on this app it consumes these 3 other apps, here's the diversions that they have deployed” and you can go into UI and change which version all of the applications use. Say, you’d make them use React 17.02 from 17.01 and kind of retrofit the dependency graph at runtime either for sharing modules or for remote modules. I think that will be a very interesting one.
I think a lot in the orchestration space is going to blossom, A/B testing as well. Anything where you're changing things at runtime based on some decisioning engine or backend. I think there is a lot of room for growth in the marketplace for this type of adaptive front end, where you could do chaos engineering or something with the ability to roll back code at runtime. You could deploy something as a canary and it would start throwing errors at which point you could send a notification or an alert somewhere and it could immediately roll back the change on a specific app and notify the team of it. So instead of having to create SevOne or having to actually go and wait for your rollback point to kick in, you could immediately respond to real user telemetry data about what's going on. And if they refresh the page you could potentially alter how the Webpack runtime starts.
So I think there is a lot to unpack there in terms of what's possible. I've only played around with this concept a little bit, but I think that it has a lot of potential for how we can rollout code and how we can manage code and distribute it safely.
Dean: I'd like to see that in action... Let's finish off with some inspiration - share with us your insights about the future of dev tooling.
Zack: That is a good one... I think we are at a place where dev tools are starting to become a lot more important, especially as applications are getting more and more complicated. The importance of having well-integrated tools and deeper insights of your application is starting to become more centerfold, especially for scaling, things like that.
Just to give an example of this, some of the big dev tooling things that I've done would be developing company-specific Chrome extensions or Webpack plugins, then integrating into, say, services that have to do with reporting what applications are consuming across the board, or, you know, some other aspect like that.
And I think that these well-integrated tools are going to be quite critical for managing the growing applications. At some point they're eventually going to get too large and you’re going to have to start breaking things down into some kind of microservice setup, whether it's federation or some other setup... esmodules or something like that.
I do think that it's going to get to a point where we have to break them down into their own build pipelines. And with that comes a much bigger need for strong dev tools, better developer feedback loops, things like that. And I think that this is all an area where tooling can definitely grow a lot.
I also see dev tools starting to kind of trickle over a little bit more into the low level languages, so we're seeing a little bit more rust-based tools, we've got SWC, esbuild out in the world, and we've seen the performance improvements that those have had already.
It would be great to see a lot more of the build chain tools, things like minification would be nice. Which, I believe, is already handled by some of them, but it'd be great to see a lot more of our slower compile tools and build tools begin to leverage these new rust-based systems. I think that it’s still a place where there’s a lot of room to grow - in the speed of what we're doing.
Or even just the multi-threading of it - it would be very nice to see Javascript tools take on multi-threaded capabilities.
Dean: Thank you very much for all the answers and we hope to talk to you again soon!
Zack: Thank you so much for this, it was great!
Bio - Zack Jackson:
Creator of Module Federation, principal engineer at lululemon. Specializing in distributed software orchestration at runtime.
For more engineering updates and insights:
Join our Telegram channel
Visit us on GitHub
Subscribe to our YouTube channel
Comments