Posts Tagged ‘javascript’

A brief and incomplete history of JavaScript bundlers

Ever since I read Malte Ubl’s proposal for a JavaScript bundle syntax, I’ve been fascinated by the question: does JavaScript need a “bundle” standard?

Unfortunately that question will have to wait for another post, because it’s much more complicated than what I can cover here. But to at least make the first tentative stabs at answering it, I’d like to explore some more basic questions:

  • What is a JavaScript bundler?
  • What purpose do bundlers serve in the modern webdev stack?

To try to answer these questions, I’d like to offer my historical perspective on what are (arguably) the two most important bundlers of the last five years: Browserify and Webpack.

A bundle of bamboo

A bundle of bamboo, via Wikipedia

What is a bundle?

Conceptually, a JavaScript bundle is very simple: it’s a collection of multiple scripts, combined into a single file. The original bundler was called +=, i.e. concatenation, and for a long time it was all anyone really needed. The whole point was to avoid the 6-connections-per-origin limit and the built-in overhead of HTTP/1.1 connections by simply jamming all your JavaScript into a single file. Easy-peasy.

Disregarding some interesting but ultimately niche bundlers such as GWT, RequireJS, and Closure Compiler, concatenation was still the most common bundler until very recently. Even fairly modern scaffolding tools like Yeoman were still recommending concatenation as the default bundler well into 2013, using lightweight tools such as usemin.

It was only really when Browserify hit the scene in 2013 did non-concatenation bundlers start to go mainstream.

The rise of Browserify

Interestingly, Browserify wasn’t originally designed to solve the problem of bundling. Instead, it was designed to solve the problem of Node developers who wanted to reuse their code in the browser. (It’s right there in the name: “browser-ify” your Node code!)

Screenshot of Browserify homepage from 2013

Screenshot of the original Browserify homepage from January 2013 (via the Internet Archive)

Before Browserify, if you were writing a JavaScript module that was designed to work in both Node or the browser, you’d have to do something like this:

var MyModule = 'hello world';

if (typeof module !== 'undefined' && module.exports) {
  module.exports = MyModule;
} else {
  (typeof self !== 'undefined' ? self : window).MyModule = MyModule;
}

This works fine for single files, but if you’re accustomed to Node conventions, it becomes aggravating that you can’t do something like this:

var otherModule = require('./otherModule');

Or even:

var otherPackage = require('other-package');

By 2014, npm had already grown to over 50,000 modules, so the idea of reusing those modules within browser code was a compelling proposition. The problem Browserify solved was thus twofold:

  1. Make the CommonJS module system work for the browser (by crawling the dependency tree, reading files, and building a single bundle file).
  2. Make Node built-ins and conventions (process, Buffer, crypto, etc.) work in the browser, by implementing polyfills and shims for them.

This second point is an often-overlooked benefit of the cowpath that Browserify paved. At the time Browserify debuted, many of those 50,000 modules weren’t written with any consideration for how they might run in the browser, and Node-isms like process.nextTick() and setImmediate() ran rampant. For Browserify to “just work,” it had to solve the compatibility problem.

What this involved was a lot of effort to reimplement nearly all of Node’s standard library for the browser, tackling the inevitable issues of cross-browser compatibility along the way. This resulted in some extremely battle-tested libraries such as events, process, buffer, inherits, and crypto, among others.

If you want to understand the ridiculous amount of work that had to go into building all this infrastructure, I recommend taking a look at Calvin Metcalf’s series on implementing crypto for the browser. Or, if you’re too faint of heart, you can instead read about how he helped fix process.nextTick() to work with Sinon or avoid bugs in oldIE’s timer system. (Calvin is truly one of the unsung heroes of JavaScript. Look in your bundle, and odds are you will find his code in there somewhere!)

All of these libraries – buffer, crypto, process, etc. – are still in wide use today via Browserify, as well as other bundlers like Webpack and Rollup. They are the magic behind why new Buffer() and process.nextTick() “just work,” and are a big part of Browserify’s success story.

Enter Webpack

While Browserify was picking up steam, and more and more browser-ready modules were starting to be published to npm, Webpack rose to prominence in 2015, buoyed by by the popularity of React and the endorsement of Pete Hunt.

Webpack and Browserify are often seen today as solutions to the same problem, but Webpack’s initial focus was a bit different from Browserify’s. Whereas Browserify’s goal was to make Node modules run in the browser, Webpack’s goal was to create a dependency graph for all of the assets in a website – not just JavaScript, but also CSS, images, SVGs, and even HTML.

The Webpack view of the world, with multiple types of assets all treated as part of the dependency graph

The Webpack view of the world, via “What is Webpack?”

In contrast to Browserify, which was almost dogmatic in its insistence on Node compatibility, Webpack was cheerful to break Node conventions and introduce code like this:

require('./styles.css');

Or even:

var svg = require('svg-url?limit=1024!./file.svg');

Webpack did this for a few different reasons:

  1. Once all of a website’s assets can be expressed as a dependency graph, it becomes easy to define “components” (collections of HTML, CSS, JavaScript, images, etc.) as standalone modules, which can be easily reused and even published to npm.
  2. Using a JavaScript-based module system for assets means that Hot Module Replacement is easy and natural, e.g. a stylesheet can automatically update itself by injection and replacement into the DOM via script.
  3. Ultimately, all of this is configurable using loaders, meaning you can get the benefits of an integrated module system without having to ship a gigantic JavaScript bundle to your users. (Although how well this works in practice is debatable.).

Because Browserify was originally the only game in town, though, Webpack had to undergo its own series of compatibility fixes, so that existing Browserify-targeting modules could work well with Webpack. This wasn’t always easy, as a JavaScript package maintainer of the time might have told you.

Out of this push for greater Webpack-Browserify compatibility grew ad-hoc standards like the node-browser-resolve algorithm, which defines what the "browser" field in package.json is supposed to do. (This field is an extension of npm’s own package.json definition, which specifies how modules should be swapped out when building in “browser mode” vs “Node mode.”)

Closing thoughts

Today, Browserify and Webpack have largely converged in functionality, although Browserify still tends to be preferred by old-school Node developers, whereas Webpack is the tool of choice for frontend web developers. Up-and-comers such as Rollup, splittable, and fuse-box (among many others) are also making the frontend bundler landscape increasingly diverse and interesting.

So that’s my view of the great bundler wars of 2013-2017! Hopefully in a future blog post I’ll be able to cover whether or not bundlers like Browserify and Webpack demonstrate the need for a “standard” to unite them all.

Feel free to weigh in on Twitter or on Mastodon.

How to write a JavaScript package for both Node and the browser

This is an issue that I’ve seen a lot of confusion over, and even seasoned JavaScript developers might have missed some of its subtleties. So I thought it was worth a short tutorial.

Let’s say you have a JavaScript module that you want to publish to npm, available both for Node and for the browser. But there’s a catch! This particular module has a slightly different implementation for the Node version compared to the browser version.

This situation comes up fairly frequently, since there are lots of tiny environment differences between Node and the browser. And it can be tricky to implement correctly, especially if you’re trying to optimize for the smallest possible browser bundle.

Let’s build a JS package

So let’s write a mini JavaScript package, called base64-encode-string. All it does is take a string as input, and it outputs the base64-encoded version.

For the browser, this is easy; we can just use the built-in btoa function:

module.exports = function (string) {
  return btoa(string);
};

In Node, though, there is no btoa function. So we’ll have to create a Buffer instead, and then call buffer.toString() on it:

module.exports = function (string) {
  return Buffer.from(string, 'binary').toString('base64');
};

Both of these should provide the correct base64-encoded version of a string. For instance:

var b64encode = require('base64-encode-string');
b64encode('foo');    // Zm9v
b64encode('foobar'); // Zm9vYmFy

Now we’ll just need some way to detect whether we’re running in the browser or in Node, so we can be sure to use the right version. Both Browserify and Webpack define a process.browser field which returns true, whereas in Node it’s falsy. So we can simply do:

if (process.browser) {
  module.exports = function (string) {
    return btoa(string);
  };
} else {
  module.exports = function (string) {
    return Buffer.from(string, 'binary').toString('base64');
  };
}

Now we just name our file index.js, type npm publish, and we’re done, right? Well, this works, but unfortunately there’s a big performance problem with this implementation.

Since our index.js file contains references to the Node built-in process and Buffer modules, both Browserify and Webpack will automatically include the polyfills for those entire modules in the bundle.

From this simple 9-line module, I calculated that Browserify and Webpack will create a bundle weighing 24.7KB minified (7.6KB min+gz). That’s a lot of bytes for something that, in the browser, only needs to be expressed with btoa!

“browser” field, how I love thee

If you search through the Browserify or Webpack documentation for tips on how to solve this problem, you may eventually discover node-browser-resolve. This is a specification for a "browser" field inside of package.json, which can be used to define modules that should be swapped out when building for the browser.

Using this technique, we can add the following to our package.json:

{
  /* ... */
  "browser": {
    "./index.js": "./browser.js"
  }
}

And then separate the functions into two different files, index.js and browser.js:

// index.js
module.exports = function (string) {
  return Buffer.from(string, 'binary').toString('base64');
};
// browser.js
module.exports = function (string) {
  return btoa(string);
};

After this fix, Browserify and Webpack provide much more reasonable bundles: Browserify’s is 511 bytes minified (315 min+gz), and Webpack’s is 550 bytes minified (297 min+gz).

When we publish our package to npm, anyone running require('base64-encode-string') in Node will get the Node version, and anyone doing the same thing with Browserify or Webpack will get the browser version. Success!

For Rollup, it’s a bit more complicated, but not too much extra work. Rollup users will need to use rollup-plugin-node-resolve and set browser to true in the options.

For jspm there is unfortunately no support for the “browser” field, but jspm users can get around it in this case by doing require('base64-encode-string/browser') or jspm install npm:base64-encode-string -o "{main:'browser.js'}". Alternatively, the package author can specify a “jspm” field in their package.json.

Advanced techniques

The direct "browser" method works well, but for larger projects I find that it introduces an awkward coupling between package.json and the codebase. For instance, our package.json could quickly end up looking like this:

{
  /* ... */
  "browser": {
    "./index.js": "./browser.js",
    "./widget.js": "./widget-browser.js",
    "./doodad.js": "./doodad-browser.js",
    /* etc. */
  }
}

So every time you want a browser-specific module, you’d have to create two separate files, and then remember to add an extra line to the "browser" field linking them together. And be careful not to misspell anything!

Also, you may find yourself extracting individual bits of code into separate modules, merely because you wanted to avoid an if (process.browser) {} check. When these *-browser.js files accumulate, they can start to make the codebase a lot harder to navigate.

If this situation gets too unwieldy, there are a few different solutions. My personal favorite is to use Rollup as a build tool, to automatically split a single codebase into separate index.js and browser.js files. This has the added benefit of de-modularizing the code you ship to consumers, saving bytes and time.

To do so, install rollup and rollup-plugin-replace, then define a rollup.config.js file:

import replace from 'rollup-plugin-replace';
export default {
  entry: 'src/index.js',
  format: 'cjs',
  plugins: [
    replace({ 'process.browser': !!process.env.BROWSER })
  ]
};

(We’ll use that process.env.BROWSER as a handy way to switch between browser builds and Node builds.)

Next, we can create a src/index.js file with a single function using a normal process.browser condition:

export default function base64Encode(string) {
  if (process.browser) {
    return btoa(string);
  } else {
    return Buffer.from(string, 'binary').toString('base64');
  }
}

Then add a prepublish step to package.json to generate the files:

{
  /* ... */
  "scripts": {
    "prepublish": "rollup -c > index.js && BROWSER=true rollup -c > browser.js"
  }
}

The generated files are fairly straightforward and readable:

// index.js
'use strict';

function base64Encode(string) {
  {
    return Buffer.from(string, 'binary').toString('base64');
  }
}

module.exports = base64Encode;
// browser.js
'use strict';

function base64Encode(string) {
  {
    return btoa(string);
  }
}

module.exports = base64Encode;

You’ll notice that Rollup automatically converts process.browser to true or false as necessary, then shakes out the unused code. So no references to process or Buffer will end up in the browser bundle.

Using this technique, you can have any number of process.browser switches in your codebase, but the published result is two small, focused index.js and browser.js files, with only the Node-related code for Node, and only the browser-related code for the browser.

As an added bonus, you can configure Rollup to also generate ES module builds, IIFE builds, or UMD builds. For an example of a simple library with multiple Rollup build targets, you can check out my project marky.

The actual project described in this post (base64-encode-string) has also been published to npm so that you can inspect it and see how it ticks. The source code is available on GitHub.

The struggles of publishing a JavaScript library

If you’ve done any web development in the past few years, then you’ve probably typed something like this:

$ bower install jquery

Or maybe even:

$ npm install --save lodash

For anyone who remembers the dark days of combing Github for jQuery plugins, this is a miracle. But as with all software, somebody had to write that code in order for you to be able to download it. And in the case of tools like Bower and npm, somebody also had to do the legwork to publish it. This is one of their stories.

The Babelification of JavaScript

I tweeted this recently:

I got some positive feedback, but I also saw some incredulous responses from people telling me I only need to support npm and CommonJS, or more snarkily, that supporting “just JavaScript” is good enough. As a fairly active open-source JavaScript author, though, I’d like to share my thoughts on why it’s not so simple.

The JavaScript module ecosystem is a mess these days. For module definitions, we have AMD, UMD, CommonJS, globals, and ES6 modules 1. For distribution, we have npm, Bower, and jspm, as well as CDNs like cdnjs, jsDelivr, and Github itself. For translating between Node and browser code, we have Browserify, Webpack, and Rollup.

Supporting each of these categories comes with its own headaches, but before I delve into that, here’s my take on how we got into this morass in the first place.

What is a JS module?

For the longest time, JavaScript didn’t have any commonly-accepted module system, so the most straightforward way to distribute your code was as a global variable. jQuery plugins also worked this way – they would just look for the global window.$ or window.jQuery and hook themselves onto that.

But thanks largely to Node and the influx of people who care about highfalutin computer-sciencey stuff like “not polluting the global namespace,” we now have a lot more ways of modularizing our code. npm is famous for using CommonJS, with its module.exports and require(), whereas other tools like RequireJS use an alternative format called AMD, known for its define() and asynchronous loading. (It’s never ceased to confuse me that RequireJS is the one that doesn’t use require().) There’s also UMD, which seeks to harmonize all of them (the “U” stands for “universal”).

In practice, though, there’s no good “universal” way to distribute your code. Many libraries try to dynamically determine at runtime what kind of environment they’re in (here’s a pretty gnarly example), but this makes modularizing your own code a headache, because you have to repeat that boilerplate anywhere you want to split up your code into separate files.

More recently, I’ve seen a lot of modules migrate to just using CommonJS everywhere, and then bundling it up for distribution with Browserify. This can be fraught with its own difficulties though, if you aren’t aware of the subtleties of how your code gets consumed. For instance, if you use Browserify’s --standalone flag (-s), then your code will get built as an AMD-ready, UMD-ready, and globals-ready bundle file, but you might not think to add it as a build step, because the stated use of the --standalone flag is to create a global variable 2.

However, my new personal policy is to use this flag everywhere, even when I can’t think of a good global variable name, because that way I don’t get issues filed on me asking for AMD support or UMD support. (Speaking of which, it still tickles me that someone had to actually open an issue asking me to support a supposedly “universal” module system. Not so universal after all, is it!)

Package managers and pseudo-package managers

So let’s say you go the CommonJS + Browserify route: now you have an interesting problem, which is that you have both a “source” version and a “distributed” version of your code. (Commonly these are organized into a src/lib folder and a dist folder, but those are just conventions.) How do you make sure your users get the right one?

npm is a package manager that expects CommonJS modules, so typically in your package.json, you set the "main" key to point to whatever your source "src/index.js" file is. Bower, however, expects a bundle file that can be directly included as a <script> tag, so in that case you’ll want to set the "main" inside the bower.json to point instead to your "dist/mypackage.js" or "dist/mypackage.min.js" file. jspm complicates things further by defaulting to npm’s package.json file while actually expecting non-CommonJS modules, but you can override that behavior by including a {"jspm": "main": "dist/mypackage.js"}} in your package.json. Whew! We’re all done, right?

Not so fast. As it turns out, Bower isn’t really a package manager so much as a CLI over Github. What that means is that you actually need to check your bundle files into Git, to ensure that those dist/ files are available to Bower users. At the same time, you’ll have to be very cognizant not to check in anything you don’t want people to download, because Bower’s "ignore" list doesn’t actually avoid downloading anything; it just deletes the ignored files after they’re downloaded, which can lead to some enormous Bower downloads. Couple this with the fact that you’re probably also juggling .gitignore files and .npmignore files, and you can end up with some fairly complicated release scripts!

Of course, many users will also just download your bundle file from Github. So it’s important to be consistent with your Git tags, so that you can have a nice tidy Github releases page. As it turns out, Bower will also depend on those Git tags to determine what a “release” is – actually, it flat-out ignores the "version" field in bower.json. To make sense of all this complexity, our policy with PouchDB is to just do an explicit commit with the version tag that isn’t even a part of the project’s main master branch, purely as a “release commit” for Bower and Github.

What about CDNs?

Github discourages using their hosted JavaScript files directly from <script> tags (in fact their HTTP headers make it impossible), so often users will ask if they can consume your library via a CDN. CDNs are also great for code snippets, because you can just include a <script> tag pointing to the latest CDN release. So lots of libraries (including PouchDB) also support jsDelivr and cdnjs.

You can add your library manually, but in my experience this is a pain, because it usually involves checking out the entire source for the CDN (which can be many gigabytes) and then opening a pull request with your library’s code. So it’s better to follow their automated instructions so that they can automatically update whenever your code updates. Note that both jsDelivr and cdnjs rely on Git tags, so the above comments about Github/Bower also apply.

Correction: Both jsDelivr and cdnjs can be configured to point to npm instead of Github; my mistake! The same applies to jspm.

Browser vs Node

For anyone who’s written a popular JavaScript library, the situation inevitably arises that someone tries to use your Node-optimized library in the browser, or your browser-optimized library in Node, and invariably they run into issues.

The first trick you might employ, if you’re working with Browserify, is to add if/else switches anytime you want to do something differently in Node or the browser:

function md5(str) {
  if (process.browser) {
    return require('spark-md5').hash(str);
  } else {
    return require('crypto').createHash('md5').update(str).digest('hex');
  }
}

This is convenient at first, but it causes some unexpected problems down the line.

First off, you end up sending unnecessary Node code to the browser. And especially if the Browserified version of your dependencies is very large, this can add up to a lot of bytes. In the example above, Browserifying the entire crypto library comes out to 93KB (after uglify+gzip!), whereas spark-md5 is only 2.6KB.

The second issue is that, if you are using a tool like Istanbul to measure your code coverage, then properly measuring your coverage in Node can lead to a lot of /* istanbul ignore next */ comments all over the place, so that you can avoid getting penalized for browser code that never runs.

My personal method to avoid this conundrum is to prefer the "browser" field in package.json to tell Browserify/Webpack which modules to swap out when building. This can get pretty complicated (here’s an example from PouchDB), but I prefer to complicate my configuration code rather than my JavaScript code. Another option is to use Calvin Metcalf’s inline-process-browser, which can automatically strip out process.browser switches 3.

You’ll also want to be careful when using Browserify transforms in your code; any transforms need to be a regular dependency rather than a devDependency, or else they can cause problems for library users.

Wait, you tried to run my code where?

After you’ve solved Node/browser switching in your library, the next hurdle you’ll likely encounter is that there is some unexpected bug in an exotic environment, often due to globals.

One way this might manifest itself is that you expect a global window variable to exist in the browser – but oh no, it’s not there in a web worker! So you check for the web worker’s self as well. Aha, but NW.js has both a Node-style global and browser-style window as global variables, so you can’t know in advance which other globals (such as Promise or console) are attached to which! Then you can get into even stranger environments like iOS’s JSCore (which is used by React Native), or Electron, or Qt WebKit, or Rhino/Nashorn, or Java FXWebView, or Adobe Air…

If you want to see what kind of a mess this can create, check out these lines of code from Lodash, and weep for poor John-David Dalton!

My own solution to this issue is to never ever check for window or global or anything like that if I can avoid it, and instead use typeof whatever === 'undefined' to check. For instance, here’s my typical Promise shim:

function PromiseShim() {
  if (typeof Promise !== 'undefined') {
    return Promise;
  }
  return require('lie');
}

Trying to access a global variable that doesn’t exist is a runtime error in most JavaScript environments, but using the typeof check will prevent the error.

Browserify vs Webpack

Most library authors I know tend to prefer Browserify for building JavaScript modules, but especially with the rise of React and Flux, Webpack is increasingly becoming a popular option.

Webpack is mostly consistent with Browserify, but there are points of divergence that can lead to unexpected errors when people try to require() your library from Webpack. The best way to test is to simply run webpack on your source CommonJS file and see if you get any errors.

In the worst case, if you have a dependency that doesn’t build with Webpack, you can always tell users to specify a custom loader to work around the issue. Webpack tends to give more control to the end-user than Browserify does, so the best strategy is to just let them build up your library and dependencies however they need to.

Enter ES6

This whole situation I’ve described above is bad enough, but once you add ES6 to the mix, it gets even more complicated. ES6 modules are the “future-proof” way of authoring JavaScript, but as it stands, there are very few tools that can consume ES6 directly, including most versions of Node.

(Yes, even if you are using Node 4.x with its many lovely ES6 features like Promises and arrow functions, there are still some missing features, like spread arguments and destructuring, that are not supported by V8 yet.)

So, what many ES6 authors will do is add a "prepublish" script to build the ES6 source into a version consumable by Node/npm (here’s an example). (Note that your "main" field in package.json must point to the Node-ready version, not the ES6 version!) Of course, this adds a huge amount of additional complexity to your build script, because now you have three versions of your code: 1) source, 2) Node version, and 3) browser version.

When you add an ES6 module bundler like Rollup, it gets even hairier. Rollup is a really cool bundler that offers some big benefits over Browserify and Webpack (such as smaller bundle sizes), but to use it, it expects your library’s dependencies to be exported in the ES6 format.

Now, because npm normally expects CommonJS, not ES6 modules, there is an informal “jsnext:main” field that some libraries use to point to their ES6 source. Usage is not very widespread, though, so if any of your dependencies don’t use ES6 or don’t have a "jsnext:main", then you’ll need to use Rollup’s --external flag when bundling them so that it knows to ignore them.

"jsnext:main" is a nice hack, but it also brings up a host of unanswered questions, such as: which features of ES6 are supported? Is it a particular stage of recommendation for the spec, ala Babel? What about popular ES7 features that are already starting to creep into codebases that use Babel, such as async/await? It’s not clear, and I don’t think this problem will be resolved until npm takes a stance one way or the other.

Making sense of this mess

At the end of the day, if your users want your code bad enough, then they will find a way to consume it. In the worst case scenario, they can just copy-paste your code from Github, which is how JavaScript was consumed for many years anyway. (StackOverflow was a decent package manager long before cooler kids like npm and Bower came along!)

Many folks have advised me to just support npm and CommonJS, and honestly, for my smaller modules I’m doing just that. It’s simply too much work to try to support everything at once. As an example of how complicated it is, I’ve created a hello-javascript module that only contains the code you need to support all the environments above. Hopefully it will help someone trying to figure out how to publish to multiple targets.

If you happen to be thinking about hopping into the world of JavaScript library authorship, though, I recommend starting with npm’s publishing guide and working your way up from there. Trying to support every JavaScript user on the planet is an ambitious proposition, and you don’t want to wear yourself out when you’re having enough trouble testing, writing documentation, checking code coverage, triaging issues, and hey – at some point, you’ll also need to write some code!

But as with everything in software, the best advice is to focus on the user and all else will follow. Don’t listen to the naysayers who tell you that Bower users are “wrong” and you’re doing them a favor by “educating” them 4. Work with your users to try to support their use case, and give them alternatives if they’re unsatisfied with your current publishing approach. (I really like wzrd.in for on-demand Browserification.)

To me, this is somewhat like accessibility. Some users only know Bower, not npm, or maybe they don’t even understand the difference between the two! Others might be unfamiliar with the command line, and in that case, a big reassuring “Download” button on a github.io page might be the best way to accommodate them. Still others might be power users who will try to include your ES6 code directly and then Browserify it themselves. (Ask those users for a pull request!)

At the end of the day, you are giving away your labor for free, so you shouldn’t feel obligated to bend over backwards for anybody. But if your driving motivation is to make your code as usable as possible for other people, then I’d say you can’t go wrong by supporting the two most popular options: direct downloads for casual users, and npm/CommonJS for power users. If your library grows in popularity, you can always worry about the thousand and one other methods later. 5

Thanks to Calvin Metcalf, Nick Colley, and Colin Skow for providing feedback on a draft of this post.

Footnotes


1. I’ve seen no compelling reason to call it “ES2015,” except to signal my own status as a smarty-pants. So I don’t.

2. Another handy tool is derequire, which can remove all require()s from your bundle to ensure it doesn’t get re-interpreted as a CommonJS module.

3. Calvin Metcalf pointed out to me that you can also work around this issue by using crypto sub-modules, e.g. require('crypto-hash'), or by fooling Browserify via require('cryp' + 'to').

4. With npm 3, many developers are starting to declare Bower to be obsolete. I think this is mostly right, but there are still a few areas where Bower beats npm. First off, for isomorphic libraries like PouchDB, an npm install can be more time-consuming and error-prone than a bower install, due to native LevelDB dependencies that you’ll never need if you’re only using PouchDB on the frontend. Second, not all libraries are publishing their dist/ code to npm, meaning that former Bower users would have to learn the whole Browserify/Webpack stack rather than just include a <script> tag. Third, not all Bower modules are even on npm – Ionic framework is a popular one that springs to mind. Fourth, there’s the social cost of migrating folks from Bower to npm, throwing away a wealth of tutorials and accumulated knowledge in the process. It’s not so simple to just tell people, “Okay, now start using npm instead of Bower.”

5. I’ve ragged a lot on the JavaScript community in this post, but I still find authoring for JavaScript to be a very pleasurable experience. I’ve been a consumer of Python, Java, and Perl modules, as well as a publisher of Java modules, and I still find npm to be the nicest to work with. The fact that my publish process is as simple as npm version patch|minor|major plus a npm publish is a real dream compared to the somewhat bureaucratic process for asking permission to publish to Maven Central. (If I ever have to see the Sonatype Nexus web UI again, I swear I’m going to hurl.)

JavaScript development and the paradox of choice

There are a lot of folks trying to sell their miracle cure for the problem of writing efficient, testable, maintainable JavaScript. And there’s an equal number of folks decrying the proliferation of almost-there libraries and flash-in-the-pan frameworks.

Bootstrap. Backbone. Handlebars. Angular. I’ve spent so much time hearing snatches of conversation about these tools, and trying to make sense of them, that after awhile it all starts to sound like some crazy beat poetry.

Listen:

angular backbone bootstrap cordova handlebars lawnchair underscore jasmine karma testacular grunt yeoman blueprint ember bower require sencha dojo mootools phonegap modernizr prototype meteor…

If you shouted that on a street corner while wielding a bottle of bourbon, you wouldn’t look out of place. I’ve seen the best minds of my generation destroyed trying to understand this mess.

Police Chief Wiggum and a raving derelict, from Simpsons episode 3F02.

Pictured: a seasoned JavaScript developer.

A good JavaScript is hard to find

Part of the reason there are so many snake-oil salesmen is that the cure is so badly needed. Web development is both 1) hard and 2) absolutely crucial. Facebook and Gmail have set the bar high enough that nowadays everyone expects beautiful, responsive, browser-based applications that take milliseconds to download and work on every rectangular-shaped device you can throw at it. It’s a tall order.

And the reason it feels like snake oil is that none of these tools solves the entire problem. I’ve tried many of them, hoping that I had finally found the JavaScript silver bullet, and I’ve always felt vaguely disappointed afterwards. The medicine tastes good going down, I get excited watching YouTube tutorials and reading GitHub pages and coding in a new paradigm, and then afterwards I still end up sweating feverishly over the Chrome Developer Tools, trying to center a disobedient div or figure out why my event isn’t firing. I exchange one set of problems for another.

And then I lay awake at night wondering, “Well, maybe instead of JQuery UI, I should have used YUI or Bootstrap or…”? Then it’s back for another dose of the same old medicine.

Grandpa Simpson selling some 'revitalizing tonic,' from Simpsons episode 2F07.

Step right up and put some fury in your JQuery, some zest in your CSS!

Another world is possible

This situation really frustrates me, primarily because I come from a Java background. And in Java Land, the platform is mature enough that there’s a basic suite of components that have emerged as the brain-dead, obvious solutions to common problems.

  • Need to test your app? Duh, use JUnit.
  • Need basic HTTP operations? Double duh, use Apache HTTP Client.
  • Need ORM? What are you, stupid? Use Hibernate.
  • Need a package manager? Cripes, it’s the 21st century: use Maven. Or Ivy, if you want something even simpler.

And if you use modern Java-based frameworks like Android or Grails, you’ll see that a lot of these third-party tools are already baked in: e.g. JUnit and HTTP Client for Android; Ivy, Hibernate, and JUnit for Grails. New Java developers pick up stuff like JUnit without thinking about it, as if it were just part of the language. And it practically is.

Even Java itself is mature enough that I’ve honestly felt satisfied since Java 6, and haven’t seen much need to upgrade. String switches in Java 7? Yawn, I’ve been using Enums since Java 5. Lambdas in Java 8? No need, Google Guava has me covered.

No silver JS bullet

JavaScript, on the other hand, is anything but mature. There is no “obvious” choice for third-party components – with the exception of JQuery, which is so omnipresent nowadays that it almost is JavaScript.

But aside from JQuery, there’s no one-stop solution that everyone rallies behind. For each of my “easy” questions for Java above, you get a forest of forked decision trees in JavaScript:

Mr. Burns contemplates Ketchup vs. Catsup, from Simpsons episode 2F07.

Ketchup or catsup? Karma or Selenium?

The paradox of choice

When you’ve got dozens of popular frameworks, many of them with overlapping or even conflicting goals, the choices can be overwhelming. And even after you choose one, it’s easy to end up second-guessing yourself and fretting endlessly over your decision. It’s a familiar case of the phenomenon popularly known as the paradox of choice.

So what’s a poor JavaScript developer to do?

Let’s say, for instance, that your boss tells your team that you need to write a mobile webapp. Do you choose JQuery Mobile, Sencha Touch, or Dojo Mobile? And what if you need to write a regular data-driven Ajax app? Do you choose Angular, Ember, or Backbone? Each of them has a snazzy self-laudatory website and fierce partisans on Stack Overflow. Looks like you’ve got some reading to do!

I’m new to web development, but I’ve come to believe that the only surefire solution to the problem of competing frameworks is to try them all. Not for a mission-critical project, of course – instead, you should just write a stub app. That way, you’ll discover each framework’s strengths and shortcomings, you’ll understand the problems it’s trying to solve, and you’ll be able to make an informed decision when it really counts.

In my opinion, it’s better to have three developers on your team take a week to write stub apps in three different frameworks, rather than blindly embark down a single path based on the attractiveness of a documentation page or the charisma of a YouTube evangelist.

My own stub app

I decided to try this approach recently with three frameworks I was curious about – Angular, Bootstrap, and PhoneGap. They seemed to have orthogonal goals, so in theory they should play nicely together.

My objective was to write a webapp with nice MVC features (Angular) that would look pretty (Bootstrap) and could work as a native Android or iOS app (PhoneGap). For the task itself, I chose to write an end-of-game score calculator for one of my favorite Euro-style board games, Imperial. This had the benefit of being a well-defined problem that scratched a personal itch, and plus it gave me something to show off to my board gamer buddies.

For the feature specifications, the usual suspects applied. I needed to persist user data, because presumably users would want to see their saved games, or resume a game if they accidentally closed the tab. The design had to be responsive in order to accomodate multiple screen sizes, because you could imagine using this app in your browser as well as on a smartphone. It had to support deep-linking, because what if you wanted to share the game results with your friends? And of course, the UI had to present the data in a useful way: who’s in first place, who came in second, are there any ties, etc.

When I first described this project to one of my coworkers, his reaction was “that sounds like way more than a stub app.” Which is true – as soon as you exceed a certain level of complexity, you run into interesting problems, for which the frameworks are supposed to provide useful solutions. This is exactly the point of writing the app.

The end result of this experiment is the Imperial Score Calculator. It’s available as both a mobile-friendly webapp and an Android app (iOS version coming soon). And of course the source code is on GitHub.

Imperial Score Calculator desktop-sized screenshot

Imperial Score Calculator

I’ve learned something today

In the end, I’m very satisfied with the project. Not because the app itself is the best I’ve ever written (it’s not), but because it taught me some hard lessons that I’ll take with me to my next web project. For instance, here are some of the lessons learned:

  • Bootstrap does not magically make everything responsive. Do not design for the desktop and then hope that when you resize the viewport everything will “just work.” Some assembly required.
  • Angular is a godsend. It’s as if someone stepped out of a time machine and showed us what HTML6 will look like, today. I initially wrote the app in JQuery; a naïve Angular rewrite resulted in about 20% less code.
  • That being said, Angular does not instantly replace JQuery, unless you really grok directives. I still had to fall back on the good ol’ $ from time to time.
  • Lawnchair is a cool idea, but it’s poorly documented, and the asynchronous approach means you can’t save user data in the onbeforeunload event. In the end, I just went with LocalStorage.
  • PhoneGap is awesome. But man oh man, do not try debugging it without Weinre, unless you like pulling your hair out.

These are all opinions that I hold after working on this app. And I don’t expect you, dear reader, to swallow any of them just based on my say-so. The only way you can learn these lessons is to build a stub app yourself.

And perhaps you’ll have a totally different experience and come to totally different conclusions. Your mileage may vary. But you won’t know until you take the car out for a test drive.

Imperial Score Calculator mobile-sized screenshot

I ended up using a completely different layout for the mobile version.

Conclusion

JavaScript development is hard. The community is going through some growing pains, with everyone defending their cherished framework. The only solution to this problem of fragmentation and “There’s More Than One Million Ways To Do It” is time.

I do see some rays of hope in projects like Meteor and Yeoman, which are very opinionated meta-frameworks that attempt to combine multiple “best of class” JavaScript solutions into one easy package for web developers. In a sense, they’re trying to solve the problem that’s already been solved in Java Land.

But since Java Land is an increasingly irrelevant, fading power next to the ascendant hegemony that is the People’s Republic of JavaScript, the solution can’t come soon enough. In the meantime, I’ll keep writing stub apps.