Archive for the ‘Webapps’ Category

What I’ve learned about accessibility in SPAs

Over the past year or so, I’ve learned a lot about accessibility, mostly thanks to working on Pinafore, which is a Single Page App (SPA). In this post, I’d like to share some of the highlights of what I’ve learned, in the hope that it can help others who are trying to learn more about accessibility.

One big advantage I’ve had in this area is the help of Marco Zehe, an accessibility expert who works at Mozilla and is blind himself. Marco has patiently coached me on a lot of these topics, and his comments on the Pinafore GitHub repo are a treasure trove of knowledge.

So without further ado, let’s dive in!

Misconceptions

One misconception I’ve observed in the web community is that JavaScript is somehow inherently anti-accessibility. This seems to stem from a time when screen readers did not support JavaScript particularly well, and so indeed the more JavaScript you used, the more likely things were to be inaccessible.

I’ve found, though, that most of the accessibility fixes I’ve made have actually involved writing more JavaScript, not less. So today, this rule is definitely more myth than fact. However, there are a few cases where it holds true:

divs and spans versus buttons and inputs

Here’s the best piece of accessibility advice for newbies: if something is a button, make it a <button>. If something is an input, make it an <input>. Don’t try to reinvent everything from scratch using <div>s and <span>s. This may seem obvious to more seasoned web developers, but for those who are new to accessibility, it’s worth reviewing why this is the case.

First off, for anyone who doubts that this is a thing, there was a large open-source dependency of Pinafore (and of Mastodon) that had several thousand GitHub stars, tens of thousands of weekly downloads on npm, and was composed almost entirely of <div>s and <span>s. In other words: when something should have been a <button>, it was instead a <span> with a click listener. (I’ve since fixed most of these accessibility issues, but this was the state I found it in.)

This is a real problem! People really do try to build entire interfaces out of <div>s and <span>s. Rather than chastise, though, let me analyze the problem and offer a solution.

I believe the reason people are tempted to use <div>s and <span>s is that they have minimal user agent styles, i.e. there is less you have to override in CSS. However, resetting the style on a <button> is actually pretty easy:

button {
  margin: 0;
  padding: 0;
  border: none;
  background: none;
}

99% of the time, I’ve found that this was all I needed to reset a <button> to have essentially the same style as a <div> or a <span>. For more advanced use cases, you can explore this CSS Tricks article.

In any case, the whole reason you want to use a real <button> over a <span> or a <div> is that you essentially get accessibility for free:

  • For keyboard users who Tab around instead of using a mouse, a <button> automatically gets the right focus in the right order.
  • When focused, you can press the Space bar on a <button> to press it.
  • Screen readers announce the <button> as a button.
  • Etc.

You could build all this yourself in JavaScript, but you’ll probably mess something up, and you’ll also have a bunch of extra code to maintain. So it’s best just to use the native semantic HTML elements.

SPAs must manually handle focus and scroll position

There is another case where the “JavaScript is anti-accessibility” mantra has a kernel of truth: SPA navigation. Within SPAs, it’s common for JavaScript to handle navigation between pages, i.e. by modifying the DOM and History API rather than triggering a full page load. This causes several challenges for accessibility:

  • You need to manage focus yourself.
  • You need to manage scroll position yourself.

For instance, let’s say I’m in my timeline, and I want to click this timestamp to see the full thread of a post:

Screenshot of Pinafore timeline showing a post from me with an arrow pointing to the timestamp

When I click the link and then press the back button, focus should return to the element I last clicked (note the purple outline):

Screenshot showing the same image as above, but with a focus outline around the timestamp

For classic server-rendered pages, most browser engines [1] give you this functionality for free. You don’t have to code anything. But in an SPA, since you’re overriding the normal navigation behavior, you have to handle the focus yourself.

This also applies to scrolling, especially in a virtual list. In the above screenshot, note that I’m scrolled down to exactly the point in the page from before I clicked. Again, this is seamless when you’re dealing with server-rendered pages, but for SPAs the responsibility is yours.

Easier integration testing

One thing I was surprised to learn is that, by making my app more accessible, I also made it easier to test. Consider the case of toggle buttons.

A toggle button is a button that can have two states: pressed or not pressed. For instance, in the screenshot below, the “boost” and “favorite” buttons (i.e. the circular arrow and the star) are toggle buttons, because it’s possible to boost or favorite a post, and they start off in unboosted/unfavorited states.

Screenshot of a Mastodon post in Pinafore with the favorite (star) button pressed and the boost button unpressed

Visually, there are plenty of styles you can use to signal the pressed/unpressed state – for instance, I’ve opted to make the colors darker when pressed. But for the benefit of screen reader users, you’ll typically want to use a pattern like the following:

<button type="button" aria-pressed="false">
  Unpressed
</button>
<button type="button" aria-pressed="true">
  Pressed
</button>

Incidentally, this makes it easier to write integration tests (e.g. using TestCafe or Cypress). Why rely on classes and styles, which might change if you redesign your app, when you can instead rely on the semantic attributes, which are guaranteed to stay the same?

I observed this pattern again and again: the more I improved accessibility, the easier things were to test. For instance:

  • When using the feed pattern, I could use aria-posinset and aria-setsize to confirm that the virtual list had the correct number of items and in the correct order.
  • For buttons without text, I could test the aria-label rather than the background image or something that might change if the design changed.
  • For hidden elements, I could use aria-hidden to identify them.
  • Etc.

So make accessibility a part of your testing strategy! If something is easy for screen readers to interpret, then it’ll probably be easier for your automated tests to interpret, too. After all, screen reader users might not be able to see colors, but neither can your headless browser tests!

Subtleties with focus management

After watching this talk by Ian Forrest and playing around with KaiOS, I realized I could make some small changes to improve keyboard accessibility in my app.

As pointed out in the talk, it’s not necessarily the case that every mouse-accessible element also needs to be keyboard-accessible. If there are redundant links on the page, then you can skip them in the tabindex order, so a keyboard user won’t have to press Tab so much.

In the case of Pinafore, consider a post. There are two links that lead to the user’s profile page – the profile picture and the user name:

Screenshot showing a Mastodon post with red rectangles around the user avatar and the user name

These two links lead to exactly the same page; they are strictly redundant. So I chose to add tabindex="-1" to the profile picture, giving keyboard users one less link to have to Tab through. Especially on a KaiOS device with a tiny d-pad, this is a nice feature!


 

In the above video, note that the profile picture and timestamp are skipped in the tab order because they are redundant – clicking the profile picture does the same thing as clicking the user name, and clicking the timestamp does the same thing as clicking on the entire post. (Users can also disable the “click the entire post” feature, as it may be problematic for those with motor impairments. In that case, the timestamp is re-added to the tab order.)

Interestingly, an element with tabindex="-1" can still become focused if you click it and then press the back button. But luckily, tabbing out of that element does the right thing as long as the other tabbable elements are in the proper order.

The final boss: accessible autocomplete

After implementing several accessible widgets from scratch, including the feed pattern and an image carousel (which I described in a previous post), I found that the single most complicated widget to implement correctly was autocompletion.


 

Originally, I had implemented this widget by following this design, which relies largely on creating an element with aria-live="assertive" which explicitly speaks every change in the widget state (e.g. “the current selected item is number 2 of 3”). This is kind of a heavy-handed solution, though, and it led to several bugs.

After toying around with a few patterns, I eventually settled on a more standard design using aria-activedescendant. Roughly, the HTML looks like this:

<textarea
  id="the-textarea"
  aria-describedby="the-description"
  aria-owns="the-list"
  aria-expanded="false"
  aria-autocomplete="both"
  aria-activedescendant="option-1">
</textarea>
<ul id="the-list" role="listbox">
  <li 
    id="option-1"
    role="option" 
    aria-label="First option (1 of 2)">
  </li>
  <li
    id="option-2"
    role="option" 
    aria-label="Second option (2 of 2)">
  </li>
</ul>
<label for="the-textarea" class="sr-only">
  What's on your mind?
</label>
<span id="the-description" class="sr-only">
  When autocomplete results are available, press up or down 
  arrows and enter to select.
</span>

Explaining this pattern probably deserves a blog post in and of itself, but in broad strokes, what’s happening is:

  • The description and label are offscreen, using styles which make it only visible to screen readers. The description explains that you can press up or down on the results and press enter to select.
  • aria-expanded indicates whether there are autocomplete results or not.
  • aria-activedescendant indicates which option in the list is selected.
  • aria-labels on the options allow me to control how it’s spoken to a screen reader, and to explicitly include text like “1 of 2” in case the screen reader doesn’t speak this information.

After extensive testing, this was more-or-less the best solution I could come up with. It works perfectly in NVDA on the latest version of Firefox, although sadly it has some minor issues in VoiceOver on Safari and NVDA on Chrome. However, since this is the standards-based solution (and doesn’t rely on aria-live="assertive" hacks), my hope is that browsers and screen readers will catch up with this implementation.

Update: I managed to get this widget working in Chrome+NVDA and Safari+VoiceOver. The fixes needed are described in this comment.

Manual and automated accessibility testing

There are a lot of automated tools that can give you good tips on improving accessibility in your web app. Some of the ones I’ve used include Lighthouse (which uses Axe under the hood), the Chrome accessibility tools, and the Firefox accessibility tools. (These tools can give you slightly different results, so I like to use multiple so that I can get second opinions!)

Screenshot of a post in Pinafore with the Firefox accessibility tools open in a side panel

However, I’ve found that, especially for screen reader accessibility, there is no substitute for testing in an actual browser with an actual screen reader. It gives you the exact experience that a screen reader user would have, and it helps build empathy for what kinds of design patterns work well for voice navigation and which ones don’t. Also, sometimes screen readers have bugs or slightly differing behavior, and these are things that accessibility auditing tools can’t tell you.

If you’re just getting started, I would recommend watching Rob Dodson’s “A11ycasts” series, especially the tutorials on VoiceOver for macOS and NVDA for Windows. (Note that NVDA is usually paired with Firefox and VoiceOver is optimized for Safari. So although you can use either one with other browsers, those are the pairings that tend to be most representative of real-world usage.)

Personally I find VoiceOver to be the easiest to use from a developer’s point of view, mostly because it has a visual display of the assistive text while it’s being spoken.

Screenshot of VoiceOver in Safari on macOS showing text at the bottom of the screen representing what's spoken

NVDA can also be configured to do this, but you have to know to go into the settings and enable the “Speech Viewer” option. I would definitely recommend turning this on if you’re using NVDA for development!

Screenshot of Speech Viewer in NVDA on Firefox showing lines of text representing what's spoken

Similar to testing screen readers, it’s also a good idea to try Tabing around your app to see how comfortable it is with a keyboard. Does the focus change unexpectedly? Do you have to do a lot of unnecessary Tabing to get where you want? Are there any handy keyboard shortcuts you’d like to add?

For a lot of things in accessibility, there are no hard-and-fast rules. Like design or usability in general, sometimes you just have to experience what your users are experiencing and see where you can optimize.

Conclusion

Accessibility can be challenging, but ultimately it’s worth the effort. Working on accessibility has improved the overall usability of my app in a number of ways, leading to unforeseen benefits such as KaiOS arrow key navigation and better integration tests.

The greatest satisfaction, though, comes from users who are happy with the work I’ve done. I was beyond pleased when Marco had this to say:

“Pinafore is for now by far the most accessible way to use Mastodon. I use it on desktop as well as iOS, both iPhone & iPad, too. So thank you again for getting accessibility in right from the start and making sure the new features you add are also accessible.”
Marco Zehe, October 21 2019

Thank you, Marco, and thanks for all your help! Hopefully this blog post will serve as a way to pay your accessibility advice forward.

Thanks to Sorin Davidoi, Thomas Wilburn, and Marco Zehe for feedback on a draft of this post.

Footnotes

1. In the course of writing this article, I was surprised to learn that, for server-rendered pages, pressing the back button restores focus to the previously-clicked element in Firefox, Safari, and Edge (EdgeHTML), but not Chrome. I found a webcompat.com bug describing the browser difference, I’ve gone ahead and filed a bug on Chrome.

The joy and challenge of developing for KaiOS

Photo of my hand holding a Nokia 8110 4G running Pinafore on KaiOS

Recently I spent some time getting Pinafore to run on a KaiOS device. Overall, I found it to be challenging but enjoyable. In this post, I’ll talk about some of the tricks that helped me to work with this curious little feature phone.

Why KaiOS? Well, I guess I’ve always been a bit of a gadget geek. I’ve been developing for Android since the early days of the HTC Dream, I managed to get my Nexus 5 to triple-boot into Android, Firefox OS, and Ubuntu Touch, and I’ve also played around with the Amazon Fire Phone, Windows Phone, iPod Touch… You get the idea.

Recently though, I’ve found the mobile landscape a bit boring. Android and iOS are the two big players, and it’s been that way for nearly a decade. Firefox OS, Blackberry OS, Windows Phone, and other would-be contenders have long since bitten the dust.

That’s why I’m excited about KaiOS. It’s a platform that’s growing surprisingly fast and is actually based on Firefox OS (may it rest in peace). It’s especially popular in India, where it’s already the second-most popular mobile OS after Android.

KaiOS can run regular websites, but it can also run “packaged” apps, ala Firefox OS. So how can an aspiring KaiOS developer get started?

Step one: buying a development phone

I like to test on real devices, because I want to experience my app with real-world hardware and performance constraints, as an actual user would. I decided to get the Nokia 8110 4G because it was available on Amazon for $70.

Note that this phone only supports AT&T in the U.S., so if you plan on using it as your actual personal phone, you may be out of luck.

When you first unbox the phone, you’ll have about 5 minutes’ worth of setup screens, after which you’re ready to go. I’d also recommend going into the settings and upgrading the OS, since mine needed an upgrade right away.

Next steps: development environment

The best resources I’ve found for KaiOS development are the official developer guide and Paul Kinlan’s quick start guide. Paul’s guide is the best place to start, since it’s short and will get you up and running quickly.

Paul says that he could only get WebIDE to work in Firefox 48, but I found that Firefox 59 also worked (per the KaiOS developer guide). So let’s use that.

First you’ll want to download Firefox 59 for your OS (in my case, Ubuntu) via Mozilla’s download site. Sadly I could not get Firefox 59 to run alongside the latest Firefox (which is my go-to browser), and I also found that Firefox 59 tried to aggressively update itself, so every time I reopened it, it would be running the latest version. To work around that, you can run this script:

rm -fr firefox profile
mkdir profile
tar -xjf firefox-59.0.3.tar.bz2
./firefox/firefox-bin -profile profile

This will delete Firefox’s stored data and restart it with a fresh profile every time. It’s handy for quickly restarting Firefox!

After you input the “secret” code that Paul mentions (*#*#33284#*#*), you should see a little bug icon in the corner:

Screenshot of KaiOS home screen showing a little bug icon in the corner

At this point you should be able to run:

adb devices

…and see your device in the list of devices:

List of devices attached
4939400 device

If adb isn’t running, you can run adb kill-server && adb start-server to restart it.

This didn’t work for me out of the box – I got an error saying it couldn’t connect to my device due to faulty udev rules. Luckily the KaiOS developer guide has a “Setting USB access” script that will fix that.

After running that script, I used Paul’s trick of setting up adb forwarding:

adb forward tcp:6000 localfilesystem:/data/local/debugger-socket

After this, you should be able to connect in Firefox WebIDE by clicking “Remote Runtime.”

Screenshot of Firefox WebIDE with Remote Runtime selected and localhost:6000 entered

One thing I noticed is that occasionally my device would get disconnected from WebIDE. In that case, you can just forward to another port:

adb forward tcp:6001 localfilesystem:/data/local/debugger-socket

…and then reconnect WebIDE to the new port, and it should work.

Next steps: actually writing code

As far as I can tell, KaiOS is based on Firefox 48. You can tell by the user agent string:

Mozilla/5.0 (Mobile; Nokia_8110_4G; rv:48.0) Gecko/48.0 Firefox/48.0 KAIOS/2.5.1

This means that you won’t necessarily have all the latest-and-greatest JavaScript or Web API features. Here’s a brief list of what I found didn’t work:

  • async functions
  • ServiceWorker
  • Intl

Update: it turns out ServiceWorker is supported! But you have to enable a permission in the manifest. Thanks, Fabrice!

As a quick gut-check of whether your app will run on KaiOS, you can try downloading Firefox 48, loading your website, and seeing if it works. (I found this easier than trying to debug on KaiOS right out of the gate.) You can use the same script above to run Firefox 48 with a fresh profile.

Once your code is sufficiently backwards-compatible (I found that @babel/preset-env with the default settings plus an Intl polyfill was good enough for me as a starting point), you have two options for testing your app:

  • as a packaged app
  • as a hosted app

(If you’ve done Firefox OS development before, you’ll find these options familiar.)

I tried both techniques, but found that the hosted app was easier for quick testing. If your development machine and phone are on the same WiFi network, then you can create a manifest.webapp file per KaiOS’s documentation and then load a URL like so:

http://192.168.1.2:4002/manifest.webapp

Screenshot of WebIDE showing a local manifest URL entered into the Hosted App popup

(If you’re not sure what IP address, to use, run ifconfig -a | grep inet.)

Then you should see your app listed, and you can run it on the device by clicking the “play” button:

Screenshot of WebIDE showing Pinafore as a hosted app

You can also try the packaged-app route, but I found that it didn’t play well with the routing library in my framework of choice (Sapper) due to the extra /index.html. Plus, if I ever actually submit this app to the KaiStore, I’ll probably go with the hosted route since it’s less maintenance work.

Optimizing for KaiOS

In terms of web development, there are a few good things to be aware of when developing for KaiOS, and in particular for the Nokia 8110 4G.

First off, the available screen size is 240×294 pixels, which you can check by using window.innerWidth in JavaScript, @media (max-width: 240px) in CSS, etc. If you set "fullscreen": true to hide the status bar, you can get slightly more breathing room: 240×320.

I had already optimized my app for the iPhone 4’s screen size (which is 320×480), but I found I had to do extra work to make things show up correctly on an even tinier screen.

Screenshot comparing Pinafore on an iPhone 4 versus a smaller Nokia 8110 4G

The iPhone 4 is already pretty small, but it’s a giant compared to the Nokia 8110 4G.

Next, the input methods are quite limited. If you’ve actually taken the time to make your webapp keyboard-accessible, then you should be most of the way there. But there are still some extra optimizations you may have to make for KaiOS.

By default, the and arrows will typically scroll the page up or down. I decided to use the and arrows to change the focus – i.e. to act as proxies for the Tab and Shift + Tab keys. I wrote some fairly simple logic to navigate through the focusable elements on the page, taking special care to allow the arrow keys to work properly inside of text inputs and modal dialogs.

I also found that the focus outlines were a bit hard to see on the small screen, so I scaled them up for KaiOS and added some extra styling to make the focus more obvious.

Some apps may opt to display an onscreen cursor (this seems to be the default behavior of the web browser), but I found that to be a bit of a clumsy experience. So I think managing the focus is better.

 

Other KaiOS web developer hurdles

The next most difficult thing about KaiOS is still the lack of modern browser standards. For instance, I couldn’t get SVG <use> tags to work properly, so I fell back to inlining each SVG as a workaround.

CSS Grid is (mercifully) supported, so I didn’t have to backport my Grid logic. Since Grid is not supported in Firefox 48, this hints to me that KaiOS must have tweaked the Firefox OS code beyond what Firefox 48 can support. But I haven’t done the full rundown of all the supported features. (And I still find it helpful to use Firefox 48 as a minimum bar for a quick test target.)

As with all web development, this is the area where you’ll probably have to roll up your sleeves and test for what’s supported and what’s not. But overall I find KaiOS to be much easier to develop for than, say, IE11. It’s much further along on the standards track than a truly legacy browser.

Once you’ve gotten past those hurdles, though, the result can be surprisingly good! I was impressed at how well this tiny feature phone could run Pinafore:

 
(Much of the credit, I’m sure, is due to the excellent SvelteJS framework.)

Conclusion

Building for KaiOS is fun! The device is quirky and lightweight, and it makes me excited about mobile development in a way that I haven’t felt in a while. In terms of development, it feels like a nice compromise between a feature phone with limited hardware (small screen, no touch, limited key input) and a smartphone OS with cutting-edge web technologies (it has CSS Grid! And String.prototype.padStart()! It’s not so bad!).

However, I’m not a huge fan of app stores – I don’t like having to agree to the ToS, going through a review process, and ultimately subjecting myself to the whims of a private corporation. But KaiOS is neat, it’s cute, and it’s growing in developing markets, so I think it’s a worthy venue for development.

And unlike fully proprietary development platforms, you’re not writing throwaway code if KaiOS ever goes away. Since it’s ultimately just a webapp, improvements I’ve made to Pinafore for keyboard accessibility and small screens will accrue to any other device that can browse the web. Other than the manifest.webapp file, there’s nothing truly specific to KaiOS that I had to do to support it.

Overall, I find it fun and refreshing to build for something like KaiOS. It helps me see the web platform from a new perspective, and if nothing else, I’ve got a new gadget to play with.

Progressive enhancement isn’t dead, but it smells funny

Update: this blog post sparked a lively debate. You may want to read the responses from Laurie Voss, Jeremy Keith, Aaron Gustafson, and Christian Heilmann.

Progressive enhancement is a touchy subject. It can be hard to discuss dispassionately because, like accessibility, it’s often framed as an issue of empathy and compassion:

The insinuation is that if you don’t practice progressive enhancement, then maybe you’re just a careless elitist, building slick websites for the wealthy 1% who have modern browsers and fast connections, and offering only a sneering “let them eat cake” to everybody else. Using progressive enhancement can be seen as a moral decision as much as a technical one.

So what exactly is progressive enhancement? At the risk of grossly oversimplifying things, here are the two major interpretations I’ve seen:

  1. Broad version: start with a baseline of functionality, enhance upwards based on capabilities.
  2. Narrow version: start with HTML, then add CSS, then add JavaScript.

In this post, I’d like to argue that, while the broad version of progressive enhancement is still enormously useful, the narrow version doesn’t make much sense in the modern context of web applications running on smartphones in evergreen browsers. It doesn’t make sense for the western world, it doesn’t make sense for the developing world, and it doesn’t make sense given the state of web tooling. It is a holdover from a bygone era, repeated endlessly by those who have not recognized that the world has moved on without it, and publicly unchallenged by those who have already privately (although perhaps guiltily) abandoned it.

Before making my case, though, let’s explore the meaning of “progressive enhancement” a bit more.

What even is progressive enhancement?

Ask 10 different people, and you’ll likely get 10 different definitions of progressive enhancement. One of the main points of contention, though, is around whether or not a website should work without JavaScript.

In a poll by Remy Sharp, he says, “out of 800 responses, 25% said that progressive enhancement was making the site work without JavaScript.” This viewpoint is apparently shared by PE advocates who disable JavaScript and are disturbed by what they see. (Spoiler alert: many top websites do not bother to make their core functionality work without JavaScript.)

There are plenty of progressive enhancement “moderates,” though, who don’t take such a hard-line stance. Jake Archibald says “each phase of the enhancement needs a user,” and that sometimes a no-JS version wouldn’t have any users at all. Paul Lewis is a big fan of progressive rendering for performance reasons, and given the popularity of server-side React, Ember FastBoot, and Angular 2 universal JavaScript, I’d say plenty of web developers agree with them.

For many proponents of progressive enhancement, however, the issue of JavaScript remains a “magic line that must not be crossed.” I discovered this myself when I somewhat clumsily crossed this line, live on stage at Fronteers Conference in Amsterdam. I had a slide in my talk that read:

In 2016, it’s okay to build a website that doesn’t work without JavaScript.

To me, and to the kind of JavaScript-focused crowd I run with, this isn’t such a controversial statement. For the majority of websites I’ve built in my career, the question of how it functions without JavaScript has been largely irrelevant (except from a performance perspective).

However, it turned out that Fronteers was perhaps the crowd least likely to be amenable to this particular message. When I showed this slide, all hell broke loose:

The condemnation was as swift as it was vocal. Many prominent figures in the web community – Eric Meyer, Sara Soueidan, Jen Simmons – felt compelled not only to disagree with me, but to disagree loudly and publicly. Subtweets and dot-replies ran rampant. As one commentator put it, “you’d swear you had killed a baby panda the way some people react.”

Now, I have nothing against these folks personally. (In fact, I’m a big fan of Jen Simmons’ Web Ahead podcast, and of Sara Soueidan’s articles.) But the fact that their reaction wasn’t just of disagreement but of anger or exasperation is worth dissecting. I believe it harks back to what I said earlier about progressive enhancement being conflated with access – the assumption is that I’m probably just some privileged white dude advocating for a kind of web design that leaves anyone who’s less fortunate out in the cold.

Is that really true, though? Is JavaScript actually harmful for a certain segment of web users? As Jake Archibald pointed out, it’s not really about users who have disabled JavaScript, so who exactly are we helping when we make our websites work without it?

Progressive enhancement for the next billion

Tom Dale (who once famously declared progressive enhancement to be dead, but has softened since then) has a fabulous talk that pretty much cemented my thinking on progressive enhancement, so this next section owes a huge debt to him.

As Benedict Evans has noted, the next billion people who are poised to come online will be using the internet almost exclusively through smartphones. And if Google’s plans with Android One are any indication, then we have a fairly good idea of what kind of devices the “next billion” will be using:

  • They’ll mostly be running Android.
  • They’ll have decent specs (1GB RAM, quad-core processors).
  • They’ll have an evergreen browser and WebView (Android 5+).
  • What they won’t have, however, is a reliable internet connection.

In a world where your lowest common denominator is a very capable browser with a modern JavaScript engine, running on a smartphone that would have been classified as desktop-class ten years ago, but the network is now the bottleneck, what does that mean for progressive enhancement?

Simple: it means that, if you care about those users, you should be focusing on offline-first, i.e. treating the network as an enhancement. After the first load (which yes, should be server-rendered via isomorphic JavaScript), you’ll want to run as much logic as possible on the user’s device so that it can operate autonomously – regardless of whether the network conditions are good, bad, or nonexistent. And today, the way we accomplish this on the web is by using IndexedDB and Service Workers, i.e. with JavaScript.

Personally, I’ve found this method remarkably effective for building performant progressive web apps. I find that, by starting with a baseline of a low-end Android phone, throttled to 2G or 3G, and using that as my primary test device, I can build a web application that works quite well on a variety of hardware, browser, and network conditions. Optimizing for such devices tends to naturally lead to a heavily client-side approach, because by avoiding network round-trips the UI interactions become snappy and app-like. And thanks to advances in JavaScript frameworks, it’s easier than ever to move UI logic from the client to the server (using Node.js), in order to achieve a fast initial render.

The insight of offline-first is that, when you optimize for conditions where the network is unavailable, you end up making a better experience for everyone, even those on blazing-fast connections. The local cache is nearly always faster than the network, and even users on supposed “4G” connections will occasionally experience some amount of 2G speeds or offline, so the local cache is a good bet for them as well.

Offline-first is a form of progressive enhancement that directly targets the baseline experience that a high-quality progressive web app ought to support, rather than focusing on the more reductionist mindset of “first HTML, then CSS, then JavaScript.”

Truly robust web apps

Tom Dale and I aren’t the only ones who have come to this conclusion. The Chrome team has been pushing both for offline-first and the app shell architecture, which advocates for a server-rendered “shell” that then manages most of the actual app content using JavaScript. This is the way that most progressive web apps are being built these days, including applications designed for folks in developing countries, by folks in developing countries.

To demonstrate, here are screenshots of the progressive web apps Housing.com (India), Konga (Nigeria), and Flipkart (India), each with JavaScript deliberately disabled. What you’ll notice is that the authors of these apps have elected to show their script-less users an endless loading state. The “no-JS” case is clearly irrelevant to them, even if the offline case is not. (Each of these apps uses a Service Worker to cache data offline, and works fabulously well when JavaScript is enabled.)

Screenshots of Housing.com, Konga, and Flipkart without JavaScript

Screenshots of Housing.com, Konga, and Flipkart without JavaScript.

Now, you might argue, as Jeremy Keith has in “Regressive web apps,” that maybe these folks have been led astray by Google’s cheerleading for the app-shell model, and in fact it’d be nice to see some examples of PWAs that don’t require JavaScript. In his words:

“I hope we’ll see more examples of Progressive Web Apps that don’t require JavaScript to render content.”

My question to Jeremy, however, is: why? Why is it considered an unqualified good to make a website that works without JavaScript? Is it possible that this mindset – “start with HTML, add CSS, sprinkle on JavaScript” – is only a best practice in a world of incapable browsers (such as IE6) hooked up to stable desktop connections, and now that we’re in a world of smart, JavaScript-enabled browsers with unreliable connections, we need to re-evaluate our thinking?

I help maintain an open-source project called PouchDB, which enables offline storage and synchronization using (you guessed it) JavaScript. One of the more interesting projects PouchDB has been involved in was the fight against Ebola in West Africa, where it was used in an Angular app to store patient data offline (including symptom and treatment details, as well as photos), which were then synced to the server whenever a connection was re-established. In a region of the world where the network was neither fast nor reliable, this was a key feature.

Now, even with some very clever usage of AppCache, there’s no way the authors of this app could have built such an offline experience without JavaScript. And yet, it was perfectly adapted for the task at hand, and I’m proud that PouchDB actually played a role in stamping out Ebola. For anyone who is convinced that “first HTML, then CSS, then JavaScript” is the best approach for users in developing countries, I’d encourage them to actually talk to folks building apps for that demographic, and ask them if they don’t find offline-first (with JavaScript!) to be a more effective strategy.

My assertion is that, because of the reality of network and device conditions in those countries, the “HTML-first” approach has become almost completely obsolete (with the minor exception of server-side rendering), and the offline-first approach now reigns supreme. In those markets, PWAs as currently promoted are a big hit, which is clear from a fascinating Opera interview with developers in Nigeria, a Google report by Flipkart on their increased engagements with PWAs, and similar feedback from Konga.

The web is a big tent

On the other hand, I wouldn’t be so presumptuous as to say that I’ve unlocked The One True Way™ to build websites. I don’t believe every website needs to be an offline-first JavaScript app, any more than I believe every website needs to be an HTML5 canvas game, or a static blog, or a WebGL sandbox world, or whatever weirdo WebVR thing I might be strapping onto my face in the future.

When I said that in 2016 it’s okay to build a site that requires JavaScript, what I’m getting at is this: by 2016, the web has fundamentally changed. It’s expanded. It’s blossomed. There are more people building more kinds of websites than ever before, and no one-size-fits-all set of “best practices” can cut the mustard anymore.

There’s still plenty of room on the web for sites that rely primarily on HTML and CSS; in many cases, it’s still the best choice! Especially if it’s better suited to the skill set of your team, or if you’re primarily focused on static content, then “traditional” progressively-enhanced sites are a no-brainer. Such sites are certainly easier to manage and maintain than client-side webapps, and plus you often get accessibility and performance for free. Declarative languages like HTML and CSS are also easier to learn than imperative ones like JavaScript, and in many cases they’re also more robust. There are lots of good reasons to choose this architecture.

Different sites are optimized for different use cases, and I wouldn’t be so presumptuous as to tell folks that they all need to be building websites exactly the way I like them built. I certainly don’t think we should be chiding people for not building websites that work without JavaScript, or to make damning statements like this one:

“Pages that are empty without JS: dead to history, unreliable for search results, and thus ignorable. No need to waste time reading or responding.

This attitude, and others like it, stem from a set of myths about JavaScript that web developers have internalized based on conditions of the past. These days, JavaScript actually does run on search engines, in screen readers, and even in Opera Mini (for a strict but fair 5 seconds). JavaScript is a well-established part of the web platform – and unlike Flash (to which it’s often unflatteringly compared) JavaScript is standardized to ensure it will pass the test of time. Expending extra effort to make your website work without JavaScript is often not only fruitless; in the cases I mentioned above with PWAs, it can actually lead to a poorer user experience.

But besides just finding these attitudes wrong, I find them toxic. Any community that’s eager to tear each other down at the slightest whiff of unorthodoxy is not a community that I want to be a part of. I want the web to be a place where we celebrate each other’s accomplishments, where we remain ever curious and inquisitive and questioning, and where above all else, we make newcomers (who might not know everything already!) feel welcome. That’s the web I love – a big tent that’s always growing bigger.

Final thoughts

We as a community need to realize that the question of “JavaScript – yes or no?” is less about access and ubiquity, and more about performance and robustness. Only then can we stop all this ugly shaming and vitriol towards those who happen to choose JavaScript as their primary tool for building for the web. I believe that, once the moral and emotional dimension is removed, the issue of JavaScript can be more clearly viewed as just another tradeoff among the many tradeoffs we inevitably make when we build websites. So next time your gut instinct is to blame and shame: try to ask and understand instead.

And to the advocates of progressive enhancement: if you still believe requiring JavaScript is a universally bad idea, then don’t hesitate to voice that opinion! Any idea deserves to be evaluated in a public forum – that’s the only way we can suss out the good ideas from the bad ones. But purely on a strategic level, I would invite you to make your case in a less caustic and condescending way. Communities that welcome outsiders with open arms are usually better at winning converts than those that sneer and denigrate (something about flies and honey and vinegar and all that).

So if you believe in empathy – if you believe that the web is about building up good experiences for everyone, regardless of their background or ability – then I would encourage you to demonstrate that empathy, especially towards those you disagree with. Thankfully, I will admit that even those at Fronteers Conference who disagreed with me were perfectly polite and respectful in person; often these things only get out of hand on Twitter, which is not famous for enabling subtlety or nuance. So keep that in mind, and try to remember the human behind the keyboard.

The web is for everyone. The web platform is for everyone. Let’s keep it that way.

Thanks to Tom Dale, Jan Lehnardt, and Addy Osmani for reviewing a draft of this blog post.

Also, apologies to folks whose tweets I called out in this post, but I consider turnabout to be fair play. 😉 And to clarify: Sara Soueidan was perfectly courteous and respectful towards me online; I wouldn’t lump any of her comments in with the “caustic” ones.

The title is a reference to Frank Zappa.

Introducing the Cordova SQLite Plugin 2

TL;DR: I rewrote the Cordova SQLite Plugin; it’s faster and better-tested. Try it out!

For better or worse, WebSQL is still a force to be reckoned with in web development. Although the spec was deprecated over 5 years ago, it still lives on, mostly as a fallback from its more standards-friendly successor, IndexedDB. (See LocalForage, PouchDB, IndexedDBShim, and YDN-DB for popular examples of this.)

Thankfully, this WebSQL-as-polyfill practice is becoming less and less necessary, as pre-Kitkat Android slowly fades into memory, and Safari fixes its lingering IndexedDB issues. That said, there is still good reason to doubt that web developers will be able to safely hop onto the IndexedDB bandwagon anytime soon, at least without fallbacks.

For one, it’s unclear when the fixes from WebKit will be released in Safari proper (and how soon we can stop worrying about old versions of Safari). Secondly, although Safari’s “modern IndexedDB” rewrite has resolved many of its gnarliest bugs, their implementation is still around 50x slower (!) than WebSQL, even in the WebKit nightlies. (It depends on the use case, but see my database comparison tool for a demonstration of batch insert performance).

Even more saddening for the web platform as a whole is that, despite being blessed with no less than three storage engines (LocalStorage, WebSQL, and IndexedDB), many developers are still electing to go native for their storage needs. The Cordova SQLite plugin (which mimics the WebSQL API via native access to SQLite) remains a popular choice for hybrid developers, and may even be influencing the decision to go hybrid.

As a proponent of web standards, I’ve always felt a bit uneasy about the SQLite Plugin. However, after struggling with the alternatives, I must admit that it does have some nice properties:

  1. It works in iOS’s WKWebView, the successor to UIWebView, which boasts better performance but unfortunately dropped WebSQL support.
  2. It allows unlimited storage in iOS: no hard cutoff after 50MB.
  3. It allows durable storage – i.e. the browser cannot start arbitrarily deleting data when disk space runs low. This is something neither IndexedDB or WebSQL can provide until the Durable Storage API has shipped (and no browser currently has). If you think this isn’t a real problem in practice, watch this talk.
  4. It offers the ability to bundle prepopulated database files within the app, avoiding the overhead of initializing a large database at startup.

So while IndexedDB is definitely the future of storage on the web (how many years have we been saying that?), the SQLite Plugin still has its place.

I’ve actually contributed to the project before, but over the past couple years I’ve found myself unable to keep up with the changing project direction, and from my vantage point on PouchDB, I’ve watched several regressions, breaking changes, and API complexities creep into the project. I wanted to contribute, but I think my goals for the SQLite Plugin differed too much from that of the current maintainer.

So I did what’s beautiful in open source: I forked it! Actually I mostly rewrote it, while taking some snippets here and there, but in spirit it’s a fork. The new library, which I’ve creatively christened SQLite Plugin 2, diverges from its forebear in the following ways:

  1. It (mostly) just implements the WebSQL spec – no extra API complexity where possible. Under the hood, node-websql is used to maximize code reuse.
  2. It’s heavily tested – I ported over 600 tests from node-websql and PouchDB, which I’ve verified pass on Android 4.0+ and iOS 8+.
  3. In order to keep the footprint and API surface small, it only uses the built-in SQLite APIs on Android and iOS, rather than bundling SQLite itself with the plugin.

In all other ways, it works almost exactly the same as the original SQLite Plugin, on both iOS and Android. (For Windows Phone, cordova-plugin-websql already has us covered.)

Performance test

I didn’t set out to write the fastest possible WebSQL shim, but I figured folks would be interested in how my remake stacks up against the original. So I put together a small benchmark.

Again, these tests were borrowed from PouchDB: one test mostly involves reads, and the other mostly involves writes. As it turns out, PouchDB “writes” are not purely INSERTs, and PouchDB reads are not simple SELECTs (due to the CouchDB-style revision model), but hopefully this test should serve as a pretty good representation of what an actual app would do.

Each test was run 5 times with 1000 iterations each, with the median of the 5 runs taken as the final result. The test devices were a 6th generation iPod Touch running iOS 9.3.1 and a Nexus 5X running Android 6.0.1. For completeness, I also tested against pure WebSQL.

Here are the results:

SQLite Plugin 2 benchmark

SQLite Plugin 2 Original SQLite Plugin WebSQL
Writes (iOS) 29321ms 30374ms 21764ms
Reads (iOS) 8004ms 9588ms 3053ms
Writes (Android) 29043ms 33173ms 23806ms
Reads (Android) 8172ms 11540ms 7277ms

And a summary comparing SQLite Plugin 2 to the competition:

vs Original SQLite Plugin vs WebSQL
Writes (iOS) 3.59% faster 25.77% slower
Reads (iOS) 19.79% faster 61.86% slower
Writes (Android) 14.22% faster 22% slower
Reads (Android) 29.19% faster 12.3% slower

(Full results are available in this spreadsheet.)

As it turns out, SQLite Plugin 2 actually outperforms the original SQLite Plugin by quite a bit, which I credit to a smaller data size when communicating with the native layer, as well as some minor optimizations to the way SQLite itself is accessed (e.g. avoiding calculating the affected rows for a straight SELECT query).

Of course, one should also note that pure WebSQL is much faster than either plugin. This doesn’t surprise me; any Cordova plugin will always be at a disadvantage to straight WebSQL, due to the overhead of serializing the messages that are sent between the WebView and the native layer. (N.B.: just because something is “native” doesn’t necessarily mean it’s faster!)

Furthermore, if you’re storing large binary data (images, audio files, etc.), the performance will probably get even worse relative to regular WebSQL, since that large data needs to be encoded as a string (base64 or otherwise) when sent to the native side. In those cases, the most efficient choice is undoubtedly IndexedDB on Android and WebSQL on iOS, since Safari IndexedDB lacks Blob support and is already quite slow as-is. (Both PouchDB and LocalForage will intelligently store Blobs in this manner, preferring built-in Blob support where available.)

So please, heed some advice from the author himself: avoid this plugin whenever possible. Unless you absolutely need WKWebView support, unlimited storage, durable storage, or prepopulated databases, just use regular IndexedDB or WebSQL instead. Or at the very least, try to architect your app so that you can easily swap in a more standards-based approach in the future (i.e., IndexedDB!). LocalForage, PouchDB, and YDN-DB are great libraries for this, since they largely abstract away the underlying storage engine.

Conclusion

Hopefully the SQLite Plugin 2 will serve as a useful tool for hybrid developers, and can help ease the transition to the rosy future where IndexedDB and Durable Storage are well-supported in every browser. Until then, please try it out, file bugs, and let me know what you think!

High-performance Web Worker messages

Update: this blog post was based on the latest browsers as of early 2016. Things have changed, and in particular the benchmark shows that recent versions of Chrome do not exhibit the performance cliff for non-stringified postMessage() messages as described in this post.

In recent posts and talks, I’ve explored how Web Workers can vastly improve the responsiveness of a web application, by moving work off the UI thread and thereby reducing DOM-blocking. In this post, I’ll delve a bit more deeply into the performance characteristics of postMessage(), which is the primary interface for communicating with Web Workers.

Since Web Workers run in a separate thread (although not necessarily a separate process), and since JavaScript environments don’t share memory across threads, messages have to be explicitly sent between the main thread and the worker. As it turns out, the format you choose for this message can have a big impact on performance.

TLDR: always use JSON.stringify() and JSON.parse() to communicate with a Web Worker. Be sure to fully stringify the message.

I first came across this tip from IndexedDB spec author and Chrome developer Joshua Bell, who mentioned offhand:

We know that serialization/deserialization is slow. It’s actually faster to JSON.stringify() then postMessage() a string than to postMessage() an object.

This insight was further confirmed by Parashuram N., who demonstrated experimentally that stringify was a key factor in making a worker-based React implementation that improved upon vanilla React. He says:

By “stringifying” all messages between the worker and the main thread, React implemented on a Web Worker [is] faster than the normal React version. The perf benefit of the Web Worker approach starts to increase as the number of nodes increases.

Malte Ubl, tech lead of the AMP project, has also been experimenting with postMessage() in Web Workers. He had this to say:

On phones, [stringifying] is quickly relevant, but not with just 3 or so fields. Just measured the other day. It is bad.

This made me curious as to where, exactly, the tradeoffs lie with stringfying messages. So I decided to create a simple benchmark and run it on a variety of browsers. My tests confirmed that stringifying is indeed faster than sending raw objects, and that the message size has a dramatic impact on the speed of worker communication.

Furthermore, the only real benefit comes if you stringify the entire message. Even a small object that wraps the stringified message (e.g. {msg: JSON.stringify(message)}) performs worse than the fully-stringified case. (These results differ between Chrome, Firefox, and Safari, but keep reading for the full analysis.)

Test results

In this test, I ran 50,000 iterations of postMessage() (both to and from the worker) and used console.time() to measure the total time spent posting messages back and forth. I also varied the number of keys in the object between 0 and 30 (keys and values were both just Math.random()).

Clarification: the test does include the overhead of JSON.parse() and JSON.stringify(). The worker even re-stringifies the message when echoing it back.

First, here are the results in Chrome 48 (running on a 2013 MacBook Air with Yosemite):

Chrome 48 test results

And in Chrome 48 for Android (running on a Nexus 5 with Android 5.1):

Nexus 5 Chrome test results

What’s clear from these results is that full stringification beats both partial stringification and no-stringification across all message sizes. The difference is fairly stark on desktop Chrome for small messages sizes, but this difference start to narrow as message size increases. On the Nexus 5, there’s no such dramatic swing.

In Firefox 46 (also on the MacBook Air), stringification is still the winner, although by a smaller margin:

Firefox test results

In Safari 9, it gets more interesting. For Safari, at least, stringification is actually slower than posting raw messages:

Safari test results

Based on these results, you might be tempted to think it’s a good idea to UA-sniff for Safari, and avoid stringification in that browser. However, it’s worth considering that Safari is consistently faster than Chrome (with or without stringification), and that it’s also faster than Firefox, at least for small message sizes. Here are the stringified results for all three browsers:

Stringification results for all browsers

So the fact that Safari is already fast for small messages would reduce the attractiveness of any UA-sniffing hack. Also notice that Firefox, to its credit, maintains a fairly consistent response time regardless of message size, and starts to actually beat both Safari and Chrome at the higher levels.

Now, assuming we were to use the UA-sniffing approach, we could swap in the raw results for Safari (i.e. showing the fastest times for each browser), which gives us this:

Results with the best time for each browser

So it appears that avoiding stringification in Safari allows it to handily beat the other browsers, although it does start to converge with Firefox for larger message sizes.

On a whim, I also tested Transferables, i.e. using ArrayBuffers as the data format to transfer the stringified JSON. In theory, Transferables can offer some performance gains when sending large data, because the ArrayBuffer is instantly zapped from one thread to the other, without any cloning or copying. (After transfer, the ArrayBuffer is unavailable to the sender thread.)

As it turned out, though, this didn’t perform well in either Chrome or Firefox. So I didn’t explore it any further.

Chrome test results, with arraybuffer

Firefox results with arraybuffer

Transferables might be useful for sending binary data that’s already in that format (e.g. Blobs, Files, etc.), but for JSON data it seems like a poor fit. On the bright side, they do have wide browser support, including Chrome, Firefox, Safari, IE, and Edge.

Speaking of Edge, I would have run these tests in that browser, but unfortunately my virtual machine kept crashing due to the intensity of the tests, and I didn’t have an actual Windows device handy. Contributions welcome!

Correction: this post originally stated that Safari doesn’t support Transferables. It does.

Update: Boulos Dib has gracious run the numbers for Edge 13, and they look very similar to Safari (in that raw objects are faster than stringification):

Edge 13 results

Conclusion

Based on these tests, my recommendation would be to use stringification across the board, or to UA-sniff for Safari and avoid stringification in that browser (but only if you really need maximum performance!).

Another takeaway is that, in general, message sizes should be kept small. Firefox seems to be able to maintain a relatively speedy delivery regardless of the message size, but Safari and Chrome tend to slow down considerably as the message size increases. For very large messages, it may even make sense to save the data to IndexedDB from the worker, and then simply fetch the saved data from the main thread, but I haven’t verified this idea with a benchmark.

The full results for my tests are available in this spreadsheet. I encourage anybody who wants to reproduce these results to check out the test suite and offer a pull request or the results from their own browser.

And if you’d like a simple Web Worker library that makes use of stringification, check out promise-worker.

Update: Chris Thoburn has offered another Web Worker performance test that adds some additional ways of sending messages, like MessageChannels. Here are his own browser results.

How to think about databases

As a maintainer of PouchDB, I get a lot of questions from developers about how best to work with databases. Since PouchDB is a JavaScript library, and one with fairly approachable documentation (if I do say so myself), many of these folks tend toward the more beginner-ish side of the spectrum. However, even with experienced developers, I find that many of them don’t have a clear picture of how a database should fit into their overall app structure.

The goal of this article is to lay out my perspective on the proper place for a database within your app code. My focus will be on the frontend – e.g. SQLite in an Android app, CoreData in an iOS app, or IndexedDB in a webapp – but the discussion could apply equally well to a server-side app using MongoDB, MySQL, etc.

What is a database, anyway?

I have a friend who recently went through a developer bootcamp. He’s a smart guy, but totally inexperienced with JavaScript (or any kind of coding) before he started. So I found his questions endlessly fascinating, because they reminded me what it was like learning to code.

Part of his coursework was on MongoDB, and I recall spending some time coaching him on Mongoose queries. As I was explaining the concepts to him, he got a little frustrated and asked, “What’s the point of a database, anyway? Why do I need this thing?”

For a beginner, this is a perfectly valid question. You’ve already spent a long time learning to work with data in the form of objects and arrays (or “dictionaries” and “lists,” or whatever your language calls them), and now suddenly you’re told you need to learn about this separate thing called a “database” that has similar kinds of operations, but they’re a lot more awkward. Instead of your familiar for-loops and assignments, you’re structuring queries and defining schemas. Why all the overhead?

To answer that question, let’s take a step back and remember why we have databases in the first place.

#1 goal of a database: don’t forget stuff

When you create an object or an array in your code, what you have is data:

var array = [1, 2, 3, 4, 5];

This data feels tangible. You can iterate through it, you can print it out, you can insert and remove things, and you can even .map() and .filter() it to transform it in all sorts of interesting ways. Data structures like this are the raw material your code is made of.

However, there’s an ephemeral side to this data. We happen to call the space that it lives in “memory” or “RAM” (Random Access Memory), but in fact “memory” is kind of a nasty misnomer, because as soon as your application stops, that data is gone forever.

You can imagine that if computers only had memory to work with, then computer programs would be pretty frustrating to use. If you wanted to write a Word document, you’d need to be sure to print it out before you closed Word, because otherwise you’d lose your work. And of course, once you restarted Word, you’d have to laboriously type your document back in by hand. Even worse, if you ever had a power outage or the program crashed, your data would vanish into the ether.

Thankfully, we don’t have to deal with this awful scenario, because we have hard disks, i.e. a place where data can be stored more permanently. Sometimes this is called “storage,” so for instance when you buy a new laptop with 200GB of storage but only 8GB of RAM, you’re looking at the difference between disk (or storage) and memory (or RAM). One is permanent, the other is fleeting.

So if disk is so awesome, why don’t computers just use that? Why do we have RAM at all?

Well, the reason for that is that there’s a pretty big tradeoff in speed between “storage” and “memory.” You’ve felt it if you’ve ever copied a large file to a USB stick, or if you’ve seen an old low-RAM machine that look a long time to switch between windows. That’s called paging, and it’s when your computer runs out of RAM, so it starts hot-swapping between RAM and disk.

Latency numbers every programmer should know

Latency numbers, visualized.

This performance difference cannot be overstated. If you look at a chart of latency numbers every programmer should know, you’ll see that reading 1MB sequentially from memory takes about 250 microseconds, whereas reading 1MB from disk is 20 milliseconds. If those numbers both sound small, consider the scale: if 250 microseconds were the time it took to brush your teeth (5 minutes, if you listen to your dentist!), then 20 milliseconds would be 4.6 days, which is enough time to drive east-to-west across North America, with plenty of breaks in between.

And if you think reading 1MB from SSD is much better (1 millisecond), then consider that in our toothbrush-scale, it would still be 5.5 hours. That’s the time it would take for you to fly from New York to San Francisco, which is quite a bit shorter than our road trip, but still something you’d need to pack your bags for.

In a computer program, the kind of operations you can “get away with” in the toothbrush-scale of 5 minutes are totally different than what you can do in 5 hours or 4 days. This is the difference between a snappy application and a sluggish application, and it’s also at the heart of how you should be thinking about databases within your app.

Storage vs memory

Let’s move away from toothbrushes for a moment and try a different analogy. This is the one I find most useful myself when I’m writing an app.

Memory (including objects, arrays, variables, etc.) are like the counter space in your kitchen when you’re preparing a meal. You have all the tools available to you, you can quickly chop your carrots and put them into a bowl, you can mix the onions with the celery, and all of these things can be done fairly quickly without having to move around the kitchen.

Storage, on the other hand (including filesystems and databases) are like the freezer. It’s a place where you put food that you know you’re going to need later. However, when you pull it out of the freezer, there’s often a thawing period. You also don’t want to be constantly opening your freezer to pull ingredients in and out, or your electric bill is going to go through the roof! Plus, your food will probably end up tasting awful.

Probably the biggest mistake I see beginners make when working with databases is that they want to treat their freezer like their counter space. They want their application data to be perfectly mirrored in their database schemas, and they don’t want to have to think about where their food comes from – whether it’s been sitting on the counter for a few seconds, or in the freezer for a few days.

This is at the root of a lot of suffering when working with databases. You either end up constantly reading things in and out of disk, which means that your app runs slowly (and you probably blame your database!), or you have to meticulously manage your schemas and painstakingly migrate your data whenever anything in your in-memory representation changes.

Unfortunately, this idea that we can treat our databases like our RAM is a by-product of the ORM (Object-Relational Mapping) mentality, which in my opinion is one of the most toxic and destructive ideas in software engineering, because it sells you a false vision of hope. The ORM salesman promises that you can work with your in-memory objects and make them as fancy as you like, and then magically those objects will be persisted to the database (exactly as you left them!), and you’ll never even have to think about what a database is or how you’re accessing it.

In my experience, this is never how it works out with ORMs. It may seem easy at first, but eventually your usage of the database will become inefficient, and you’ll have to drop down into the murky details of the ORM layer, figure out the queries you wish it were doing, and then try to guess the incantation needed to make it perform that query. In effect, the promise of not having to think about the database is a sham, because you just end up just having to learn two layers: the database layer and the ORM layer. It’s a classic leaky abstraction.

Even if you manage to tame your ORM, you usually end up with a needlessly complex schema format, as the inflexibility of working with stored data collides with the needs of a flexible in-memory format. You might find that you wind up with a SQLite table with 20 columns, merely because your class has 20 variables – even if none of those 20 columns are ever used for querying, and in fact are just wasted space.

This partially explains the attraction of NoSQL databases, but I believe that even without rigid schemas, this problem of the “ORM mindset” remains. Mongoose is a good example of this, as it tries to mix JavaScript and MongoDB in a way that you can’t tell where one starts and the other ends. Invariably, though, this leads developers to hope that their in-memory format can exactly match their database format, which leads to irreconcilable situations (such as classes with behavior) or slowdowns (such as over-fetching or over-storing).

All of this is pretty abstract, so let me take some concrete examples from a recent app I wrote, Pokedex.org, and how I carefully modeled my database structure to maximize performance. (If you’re unfamiliar with Pokedex.org, you may want to read the introductory blog post.)

Case study: Pokedex.org

The first consideration I had to make for Pokedex.org was which database to use in the first place. Without going into the details of browser databases, I ended up choosing two:

  • LocalForage, because it has a simple key-value API that’s good for storing application state.
  • PouchDB, because it has good APIs for working with larger datasets, and can serve as an offline-first layer in front of Cloudant or CouchDB.

PouchDB can also store key-value data, so I might have used it for both. However, another benefit of LocalForage is that the bundle size is much smaller (8KB vs PouchDB’s 45KB). And in my case I had three JavaScript bundles (one for the service worker, one for the web worker, and one for the main JavaScript app), so I didn’t want to push 45KB down the wire three times. Hence I chose LocalForage for the simple stuff.

Pokedex.org database usage

Pokedex.org database usage

You can see what kind of data I stored in LocalForage if you go into the Chrome Dev Tools on Pokedex.org and open the “Resources” tab. You’ll see I’m using it to store the ServiceWorker data version (so it knows when to update), as well as "informedOffline", which just tells me whether I’ve already shown the dialog that says, “Hey, this app works offline.” If I had more app data to store (such as the user’s favorite Pokémon, or how many times they’ve opened the app), I might store that in LocalForage.

PouchDB, however, is responsible for storing the majority of the Pokémon data – i.e. the 649 species of monsters, their stats, and their moves. So this is much more interesting.

First off, you’ll notice that as you type into the search bar, you immediately get a filtered list showing Pokémon that match your search string. This is a simple prefix search, so if you type “bu” you will see “Bulbasaur” and “Butterfree” amongst others.

 

This search bar is super fast, and it ought to be, because it’s supposed to respond to user input. There’s a debounce on the actual <input> handler, but in principle every keystroke represents a database query, meaning that there’s a lot of data flying back and forth.

I considered using PouchDB for this, but I decided it would be too slow. PouchDB does offer built-in prefix search, but I don’t want to have to go back and forth to IndexedDB (i.e. disk) for every keystroke. So instead, I wrote a simple in-memory database layer that stores Pokémon summary data, i.e. only the things that are necessary to show in the list, which happens to be their name, number, and types. (The sprite comes from a CSS class based on their number.)

To perform the search itself, I just used a sorted array of String names, with a binary search to ensure that lookups take O(log n) time. If the list were larger, I might try to condense it as a trie, but I figured that would be overkill for this app.

For a small amount of data, this in-memory strategy works great. However, when you click on a Pokémon, it brings up a detail page with stats, evolutions, and moves, which is much too large to keep in memory. So for this, I used PouchDB.

 

Given that I am the primary author of PouchDB map/reduce, relational-pouch, and pouchdb-find, you may be surprised to learn that I didn’t use any of them for this task. Obviously I put a lot of care into those libraries, and I do think they’re useful for beginners who are unsure how to structure their data. But from a performance standpoint, none of them can beat the potential gains from rolling your own, so that’s what I did.

In this case, I used my knowledge of IndexedDB performance subtleties to get the maximum possible throughput in the shortest amount of time. Essentially, what I did was split up my data into seven separate PouchDB databases, representing seven different IndexedDB databases on disk:

  • Monster basic data
  • Monster descriptions
  • Monster evolutions
  • Monster supplemental data (anything not covered above)
  • Types
  • Monster moves
  • Moves

The first four all use IDs based on the number of the Pokémon (e.g. Bulbasaur is 1, Ivysaur is 2, etc.), and map to data such as evolutions, stats, and descriptions. This means that tapping on a Pokémon involves a simple key-value lookup.

The reason I segmented this data into multiple databases is because IndexedDB happens to do a lot of transaction-level blocking at the database level. If you have the luxury of specifying separate IndexedDB objectStores, you can allow your databases queries to run in parallel under the hood, but in the case of PouchDB all of the objectStores are predefined (due to the CouchDB-style revision semantics written on top of IndexedDB).

In practice, this usually means that read/write operations (such as the initial import of the data) will run sequentially unless you use separate PouchDB objects. Sequential is bad – we want the database to do as much work as quickly as possible – so I avoided using one large PouchDB database. (If you were using a lower-level library like Dexie, though, you could use a single database with separate objectStores and probably get a similar result.)

So when you tap on a Pokémon, the app fires off six concurrent get() requests, which the underlying IndexedDB layer is free to run in parallel. This is why you barely have to wait at all to see the Pokémon data, although it helps that I have a snazzy animation while the lookup is in progress. (Animations are a great way to mask slow operations!) The query is also run in a web worker, which is why you won’t see any UI blocking from IndexedDB during database interactions.

Pokémon's type(s) determine its strengths/weaknesses

A Pokémon’s type(s) determine its strengths/weaknesses relative to other types

Now, two of the six requests I described above are for a Pokémon’s “type” information, which merit some explanation. Each Pokémon has up to two types (e.g. Fire and Water), and types also have strengths and weaknesses relative to each other: Fire beats Grass, Water beats Fire, etc. The “types” database contains this big rock-paper-scissors grid, which isn’t keyed by Pokémon ID like the other four, but rather by type name.

However, since the type names of each Pokémon are already available in-memory (due to the summary view), the queries for a Pokémon’s strengths and weaknesses can be fired off in parallel with the other queries. And since they’re equally simple get() requests, they take about the same amount of time to complete. This was a nice side effect of my previous in-memory optimizations.

The last two databases are a bit trickier than the others, and are quite relation-y. I called these the “monster moves” and “moves” databases, and I modeled their implementation after relational-pouch (although I didn’t feel the need to use relational-pouch itself).

 

Essentially, the “monster moves” database contains a mapping from monster IDs to a list of learned moves (e.g. Bulbasaur learns Razor Leaf at level 27), while the “moves” database contains a mapping from move IDs to information about the move (e.g. Razor Leaf has a certain power, accuracy, and description). If you’re familiar with SQL, you might recognize that I would need a JOIN to combine this data together, although in my case I just did the join explicitly myself, in JavaScript.

Since this is a many-to-many relationship (Pokémon can learn many moves, and moves can be learned by many Pokémon), it would be prohibitively redundant to include the “move” data inside the “monster move” database – that’s why I split it apart. However, the relational query (i.e. the JOIN) has a cost, and I saw it while developing my app – it takes nearly twice as long to fetch the full “moves” data (75ms on a Nexus 5X) as it does to fetch the more basic data (40ms – these numbers are much larger on a slow device). So what to do?

Well, I pulled off a sleight-of-hand. You’ll notice that, especially in a mobile browser, the list of Pokémon moves is “below the fold.” Thus, I can simply load the above-the-fold data first, and then lazily fetch the rest before the user has scrolled down. On a fast mobile browser, you probably won’t even notice that anything was fetched in two stages, although on a huge monitor you might be able to glimpse it. (I considered adding a loading spinner, but the “moves” data was already fast enough that I felt it was unnecessary.)

So there you have it: the queries that ought to feel “instant” are done in memory, the queries that take a bit longer are fetched in parallel (with an animation to keep the eye busy), and the queries that are super slow are slipped in below-the-fold. This is a subtle ballet with lots of carefully orchestrated movements, and the end result is an app that feels pretty seamless, despite all the work going on behind the scenes.

Conclusion

When you’re working with databases, it’s worthwhile to understand the APIs you’re dealing with, and what they’re doing under the hood. Unfortunately, databases are not magic, and there’s no abstraction in the world (I believe) that can obviate the need to learn at least a little bit about how a database works.

So when you’re using a database, be sure to ask yourself the following questions:

  1. Is this database in-memory (Redis, LokiJS, MemDOWN, etc.) or on-disk (PouchDB, LocalForage, Lovefield, etc.)? Is it a mix between the two (e.g. LevelDB)?
  2. What needs to be stored on disk? What data should survive the application being closed or crashing?
  3. What needs to be indexed in order to perform fast queries? Can I use an in-memory index instead of going to disk?
  4. How should I structure my in-memory data relative to my database data? What’s my strategy for mapping between the two?
  5. What are the query needs of my app? Does a summary view really need to fetch the full data, or can it just fetch the little bit it needs? Can I lazy-load anything?

Once you’ve answered these questions, you can write an app that is fast, responsive, and doesn’t lose user data. You’ll also make your own job easier as a programmer, if you try to maintain a good grasp of the differences between your in-memory data (your counter space) and your on-disk data (your freezer).

Nobody likes freezer burn, but spoiled meat that’s been left on the counter overnight is even worse. Understand the difference between the two, and you’ll be a master chef in no time.

Notes

Of course there are more advanced topics I could have covered here, such as indexes, sync, caching, B-trees, and on and on. (We could even extend the metaphor to talk about “tagging” food in the freezer as an analogy for indexing!) But I wanted to keep this blog post small and focused, and just communicate the bare basics of the common mistakes I’ve seen people make with databases.

I also apologize for all the abstruse IndexedDB tricks – those probably merit their own blog post. In particular, I need to provide some experimental data to back up my claim that it’s better to break up a single IndexedDB database into multiple smaller ones. This trick is based on my personal experience with IndexedDB, where I noticed a high cost of fetching and storing large monolithic documents, but I should do a more formal study to confirm it.

Thanks to Nick Colley, Chris Gullian, Jan Lehnardt, and Garren Smith for feedback on a draft of this blog post.

The struggles of publishing a JavaScript library

If you’ve done any web development in the past few years, then you’ve probably typed something like this:

$ bower install jquery

Or maybe even:

$ npm install --save lodash

For anyone who remembers the dark days of combing Github for jQuery plugins, this is a miracle. But as with all software, somebody had to write that code in order for you to be able to download it. And in the case of tools like Bower and npm, somebody also had to do the legwork to publish it. This is one of their stories.

The Babelification of JavaScript

I tweeted this recently:

I got some positive feedback, but I also saw some incredulous responses from people telling me I only need to support npm and CommonJS, or more snarkily, that supporting “just JavaScript” is good enough. As a fairly active open-source JavaScript author, though, I’d like to share my thoughts on why it’s not so simple.

The JavaScript module ecosystem is a mess these days. For module definitions, we have AMD, UMD, CommonJS, globals, and ES6 modules 1. For distribution, we have npm, Bower, and jspm, as well as CDNs like cdnjs, jsDelivr, and Github itself. For translating between Node and browser code, we have Browserify, Webpack, and Rollup.

Supporting each of these categories comes with its own headaches, but before I delve into that, here’s my take on how we got into this morass in the first place.

What is a JS module?

For the longest time, JavaScript didn’t have any commonly-accepted module system, so the most straightforward way to distribute your code was as a global variable. jQuery plugins also worked this way – they would just look for the global window.$ or window.jQuery and hook themselves onto that.

But thanks largely to Node and the influx of people who care about highfalutin computer-sciencey stuff like “not polluting the global namespace,” we now have a lot more ways of modularizing our code. npm is famous for using CommonJS, with its module.exports and require(), whereas other tools like RequireJS use an alternative format called AMD, known for its define() and asynchronous loading. (It’s never ceased to confuse me that RequireJS is the one that doesn’t use require().) There’s also UMD, which seeks to harmonize all of them (the “U” stands for “universal”).

In practice, though, there’s no good “universal” way to distribute your code. Many libraries try to dynamically determine at runtime what kind of environment they’re in (here’s a pretty gnarly example), but this makes modularizing your own code a headache, because you have to repeat that boilerplate anywhere you want to split up your code into separate files.

More recently, I’ve seen a lot of modules migrate to just using CommonJS everywhere, and then bundling it up for distribution with Browserify. This can be fraught with its own difficulties though, if you aren’t aware of the subtleties of how your code gets consumed. For instance, if you use Browserify’s --standalone flag (-s), then your code will get built as an AMD-ready, UMD-ready, and globals-ready bundle file, but you might not think to add it as a build step, because the stated use of the --standalone flag is to create a global variable 2.

However, my new personal policy is to use this flag everywhere, even when I can’t think of a good global variable name, because that way I don’t get issues filed on me asking for AMD support or UMD support. (Speaking of which, it still tickles me that someone had to actually open an issue asking me to support a supposedly “universal” module system. Not so universal after all, is it!)

Package managers and pseudo-package managers

So let’s say you go the CommonJS + Browserify route: now you have an interesting problem, which is that you have both a “source” version and a “distributed” version of your code. (Commonly these are organized into a src/lib folder and a dist folder, but those are just conventions.) How do you make sure your users get the right one?

npm is a package manager that expects CommonJS modules, so typically in your package.json, you set the "main" key to point to whatever your source "src/index.js" file is. Bower, however, expects a bundle file that can be directly included as a <script> tag, so in that case you’ll want to set the "main" inside the bower.json to point instead to your "dist/mypackage.js" or "dist/mypackage.min.js" file. jspm complicates things further by defaulting to npm’s package.json file while actually expecting non-CommonJS modules, but you can override that behavior by including a {"jspm": "main": "dist/mypackage.js"}} in your package.json. Whew! We’re all done, right?

Not so fast. As it turns out, Bower isn’t really a package manager so much as a CLI over Github. What that means is that you actually need to check your bundle files into Git, to ensure that those dist/ files are available to Bower users. At the same time, you’ll have to be very cognizant not to check in anything you don’t want people to download, because Bower’s "ignore" list doesn’t actually avoid downloading anything; it just deletes the ignored files after they’re downloaded, which can lead to some enormous Bower downloads. Couple this with the fact that you’re probably also juggling .gitignore files and .npmignore files, and you can end up with some fairly complicated release scripts!

Of course, many users will also just download your bundle file from Github. So it’s important to be consistent with your Git tags, so that you can have a nice tidy Github releases page. As it turns out, Bower will also depend on those Git tags to determine what a “release” is – actually, it flat-out ignores the "version" field in bower.json. To make sense of all this complexity, our policy with PouchDB is to just do an explicit commit with the version tag that isn’t even a part of the project’s main master branch, purely as a “release commit” for Bower and Github.

What about CDNs?

Github discourages using their hosted JavaScript files directly from <script> tags (in fact their HTTP headers make it impossible), so often users will ask if they can consume your library via a CDN. CDNs are also great for code snippets, because you can just include a <script> tag pointing to the latest CDN release. So lots of libraries (including PouchDB) also support jsDelivr and cdnjs.

You can add your library manually, but in my experience this is a pain, because it usually involves checking out the entire source for the CDN (which can be many gigabytes) and then opening a pull request with your library’s code. So it’s better to follow their automated instructions so that they can automatically update whenever your code updates. Note that both jsDelivr and cdnjs rely on Git tags, so the above comments about Github/Bower also apply.

Correction: Both jsDelivr and cdnjs can be configured to point to npm instead of Github; my mistake! The same applies to jspm.

Browser vs Node

For anyone who’s written a popular JavaScript library, the situation inevitably arises that someone tries to use your Node-optimized library in the browser, or your browser-optimized library in Node, and invariably they run into issues.

The first trick you might employ, if you’re working with Browserify, is to add if/else switches anytime you want to do something differently in Node or the browser:

function md5(str) {
  if (process.browser) {
    return require('spark-md5').hash(str);
  } else {
    return require('crypto').createHash('md5').update(str).digest('hex');
  }
}

This is convenient at first, but it causes some unexpected problems down the line.

First off, you end up sending unnecessary Node code to the browser. And especially if the Browserified version of your dependencies is very large, this can add up to a lot of bytes. In the example above, Browserifying the entire crypto library comes out to 93KB (after uglify+gzip!), whereas spark-md5 is only 2.6KB.

The second issue is that, if you are using a tool like Istanbul to measure your code coverage, then properly measuring your coverage in Node can lead to a lot of /* istanbul ignore next */ comments all over the place, so that you can avoid getting penalized for browser code that never runs.

My personal method to avoid this conundrum is to prefer the "browser" field in package.json to tell Browserify/Webpack which modules to swap out when building. This can get pretty complicated (here’s an example from PouchDB), but I prefer to complicate my configuration code rather than my JavaScript code. Another option is to use Calvin Metcalf’s inline-process-browser, which can automatically strip out process.browser switches 3.

You’ll also want to be careful when using Browserify transforms in your code; any transforms need to be a regular dependency rather than a devDependency, or else they can cause problems for library users.

Wait, you tried to run my code where?

After you’ve solved Node/browser switching in your library, the next hurdle you’ll likely encounter is that there is some unexpected bug in an exotic environment, often due to globals.

One way this might manifest itself is that you expect a global window variable to exist in the browser – but oh no, it’s not there in a web worker! So you check for the web worker’s self as well. Aha, but NW.js has both a Node-style global and browser-style window as global variables, so you can’t know in advance which other globals (such as Promise or console) are attached to which! Then you can get into even stranger environments like iOS’s JSCore (which is used by React Native), or Electron, or Qt WebKit, or Rhino/Nashorn, or Java FXWebView, or Adobe Air…

If you want to see what kind of a mess this can create, check out these lines of code from Lodash, and weep for poor John-David Dalton!

My own solution to this issue is to never ever check for window or global or anything like that if I can avoid it, and instead use typeof whatever === 'undefined' to check. For instance, here’s my typical Promise shim:

function PromiseShim() {
  if (typeof Promise !== 'undefined') {
    return Promise;
  }
  return require('lie');
}

Trying to access a global variable that doesn’t exist is a runtime error in most JavaScript environments, but using the typeof check will prevent the error.

Browserify vs Webpack

Most library authors I know tend to prefer Browserify for building JavaScript modules, but especially with the rise of React and Flux, Webpack is increasingly becoming a popular option.

Webpack is mostly consistent with Browserify, but there are points of divergence that can lead to unexpected errors when people try to require() your library from Webpack. The best way to test is to simply run webpack on your source CommonJS file and see if you get any errors.

In the worst case, if you have a dependency that doesn’t build with Webpack, you can always tell users to specify a custom loader to work around the issue. Webpack tends to give more control to the end-user than Browserify does, so the best strategy is to just let them build up your library and dependencies however they need to.

Enter ES6

This whole situation I’ve described above is bad enough, but once you add ES6 to the mix, it gets even more complicated. ES6 modules are the “future-proof” way of authoring JavaScript, but as it stands, there are very few tools that can consume ES6 directly, including most versions of Node.

(Yes, even if you are using Node 4.x with its many lovely ES6 features like Promises and arrow functions, there are still some missing features, like spread arguments and destructuring, that are not supported by V8 yet.)

So, what many ES6 authors will do is add a "prepublish" script to build the ES6 source into a version consumable by Node/npm (here’s an example). (Note that your "main" field in package.json must point to the Node-ready version, not the ES6 version!) Of course, this adds a huge amount of additional complexity to your build script, because now you have three versions of your code: 1) source, 2) Node version, and 3) browser version.

When you add an ES6 module bundler like Rollup, it gets even hairier. Rollup is a really cool bundler that offers some big benefits over Browserify and Webpack (such as smaller bundle sizes), but to use it, it expects your library’s dependencies to be exported in the ES6 format.

Now, because npm normally expects CommonJS, not ES6 modules, there is an informal “jsnext:main” field that some libraries use to point to their ES6 source. Usage is not very widespread, though, so if any of your dependencies don’t use ES6 or don’t have a "jsnext:main", then you’ll need to use Rollup’s --external flag when bundling them so that it knows to ignore them.

"jsnext:main" is a nice hack, but it also brings up a host of unanswered questions, such as: which features of ES6 are supported? Is it a particular stage of recommendation for the spec, ala Babel? What about popular ES7 features that are already starting to creep into codebases that use Babel, such as async/await? It’s not clear, and I don’t think this problem will be resolved until npm takes a stance one way or the other.

Making sense of this mess

At the end of the day, if your users want your code bad enough, then they will find a way to consume it. In the worst case scenario, they can just copy-paste your code from Github, which is how JavaScript was consumed for many years anyway. (StackOverflow was a decent package manager long before cooler kids like npm and Bower came along!)

Many folks have advised me to just support npm and CommonJS, and honestly, for my smaller modules I’m doing just that. It’s simply too much work to try to support everything at once. As an example of how complicated it is, I’ve created a hello-javascript module that only contains the code you need to support all the environments above. Hopefully it will help someone trying to figure out how to publish to multiple targets.

If you happen to be thinking about hopping into the world of JavaScript library authorship, though, I recommend starting with npm’s publishing guide and working your way up from there. Trying to support every JavaScript user on the planet is an ambitious proposition, and you don’t want to wear yourself out when you’re having enough trouble testing, writing documentation, checking code coverage, triaging issues, and hey – at some point, you’ll also need to write some code!

But as with everything in software, the best advice is to focus on the user and all else will follow. Don’t listen to the naysayers who tell you that Bower users are “wrong” and you’re doing them a favor by “educating” them 4. Work with your users to try to support their use case, and give them alternatives if they’re unsatisfied with your current publishing approach. (I really like wzrd.in for on-demand Browserification.)

To me, this is somewhat like accessibility. Some users only know Bower, not npm, or maybe they don’t even understand the difference between the two! Others might be unfamiliar with the command line, and in that case, a big reassuring “Download” button on a github.io page might be the best way to accommodate them. Still others might be power users who will try to include your ES6 code directly and then Browserify it themselves. (Ask those users for a pull request!)

At the end of the day, you are giving away your labor for free, so you shouldn’t feel obligated to bend over backwards for anybody. But if your driving motivation is to make your code as usable as possible for other people, then I’d say you can’t go wrong by supporting the two most popular options: direct downloads for casual users, and npm/CommonJS for power users. If your library grows in popularity, you can always worry about the thousand and one other methods later. 5

Thanks to Calvin Metcalf, Nick Colley, and Colin Skow for providing feedback on a draft of this post.

Footnotes


1. I’ve seen no compelling reason to call it “ES2015,” except to signal my own status as a smarty-pants. So I don’t.

2. Another handy tool is derequire, which can remove all require()s from your bundle to ensure it doesn’t get re-interpreted as a CommonJS module.

3. Calvin Metcalf pointed out to me that you can also work around this issue by using crypto sub-modules, e.g. require('crypto-hash'), or by fooling Browserify via require('cryp' + 'to').

4. With npm 3, many developers are starting to declare Bower to be obsolete. I think this is mostly right, but there are still a few areas where Bower beats npm. First off, for isomorphic libraries like PouchDB, an npm install can be more time-consuming and error-prone than a bower install, due to native LevelDB dependencies that you’ll never need if you’re only using PouchDB on the frontend. Second, not all libraries are publishing their dist/ code to npm, meaning that former Bower users would have to learn the whole Browserify/Webpack stack rather than just include a <script> tag. Third, not all Bower modules are even on npm – Ionic framework is a popular one that springs to mind. Fourth, there’s the social cost of migrating folks from Bower to npm, throwing away a wealth of tutorials and accumulated knowledge in the process. It’s not so simple to just tell people, “Okay, now start using npm instead of Bower.”

5. I’ve ragged a lot on the JavaScript community in this post, but I still find authoring for JavaScript to be a very pleasurable experience. I’ve been a consumer of Python, Java, and Perl modules, as well as a publisher of Java modules, and I still find npm to be the nicest to work with. The fact that my publish process is as simple as npm version patch|minor|major plus a npm publish is a real dream compared to the somewhat bureaucratic process for asking permission to publish to Maven Central. (If I ever have to see the Sonatype Nexus web UI again, I swear I’m going to hurl.)

IndexedDB, WebSQL, LocalStorage – what blocks the DOM?

Update June 2019: This blog post was written in 2015. The benchmarks are out-of-date. WebSQL is still deprecated and in fact being removed from iOS Safari. Web Workers are not necessarily a panacea. I’d recommend using IndexedDB for large data, and LocalStorage for small amounts of data that you need synchronous access to.

When it comes to databases, a lot of people just want to know: which one is the fastest?

Never mind things like memory usage, the CAP theorem, consistency, read vs write speed, test coverage, documentation – just tell me which one is the fastest, dammit!

This mindset is understandable. A single number is easier to grasp than a big table of features, and it’s fun to make grand statements like “Redis is 20x faster than Mongo.” (N.B.: I just made that up.)

As someone who spends a lot of time on browser databases, though, I think it’s important to look past the raw speed numbers. On the client side especially, the way you use a database, and how it interacts with the JavaScript environment, has a big impact on something more important than performance: how your users perceive performance.

In this post, I’m going to take a look at various browser databases with regard not only to their speed, but to how much they block the DOM.

TLDR: IndexedDB isn’t nearly the performance home-run that many in the web community think it is. In my tests, I found that it blocked the DOM significantly in Firefox and Chrome, and was slower than both LocalStorage and WebSQL for basic key-value insertions.

Browser database landscape

For the uninitiated, the world of browser databases can be a confusing one. Lawnchair, PouchDB, LocalForage, Dexie, Lovefield, LokiJS, AlaSQL, MakeDrive, ForerunnerDB, YDN-DB – that’s a lot of databases!

As it turns out, though, the situation is much simpler than it appears on the surface. In fact, there are only three ways of storing data in the browser:

Every “database” listed above uses one of those three under the hood (or they operate in-memory). So to understand browser storage, you only really need to understand LocalStorage, WebSQL, and IndexedDB 1.

LocalStorage is a lightweight way to store key-value pairs. The API is very simple, but usage is capped at 5MB in many browsers. Plus the API is synchronous, so as we’ll see later, it can block the DOM. Browser support is very good.

WebSQL is an API that is only supported in Chrome and Safari (and Android and iOS by extension). It provides an asynchronous, transactional interface to SQLite. Since 2010, it has been deprecated in favor of IndexedDB.

IndexedDB is the successor to both LocalStorage and WebSQL, designed to replace them as the “one true” browser database. It exposes an asynchronous API that supposedly avoids blocking the DOM, but as we’ll see below, it doesn’t necessarily live up to the hype. Browser support is extremely spotty, with only Chrome and Firefox having fully usable implementations.

Now, let’s run a simple test to see when and how these APIs block the DOM.

Thou shalt not block the DOM

JavaScript is a single-threaded programming environment, meaning that synchronous operations are blocking. And since the DOM is synchronous, this means that when JavaScript blocks, the DOM is also blocked. So if any operation takes longer than 16ms, it can lead to dropped frames, which users experience as slowness, “stuttering,” or “jank.”

This is the reason that JavaScript has so many asynchronous APIs. Just imagine if your entire page was frozen during every AJAX request – wouldn’t the web be an awful user experience if it worked that way! Hence the profusion of programming constructs like callbacks, promises, event listeners, and the like.

To demonstrate DOM blocking, I’ve put together a simple demo page with an animated GIF. Whenever the DOM is blocked, Kirby will stop his happy dance and freeze in place.

Try this experiment: go to that page, open up the developer tools, and enter the following code:

for (var i = 0; i < 10000; i++) {console.log('blocked!')}

You’ll see that Kirby freezes for the duration of the for-loop:

 

This affects more than just animated GIFs; any JavaScript animation or DOM operation, such as adding or modifying elements, will also be blocked. You can’t even select a radio button; the page is totally unresponsive. The only animations that are unaffected are hardware-accelerated CSS animations.

Using this demo page, I tested four ways of of storing data: in-memory, LocalStorage, WebSQL, and IndexedDB. The test inserts a given number of “documents,” which are just unstructured JSON keyed by a string ID. I made a YouTube video showing my results, but the rest of the article will summarize my findings.

In-memory

Not surprisingly, since any synchronous code is blocking, in-memory operations are also blocking. You can test this in the demo page by choosing “regular object” or “LokiJS” (which is an in-memory database). The DOM blocks during long-running inserts, but unless you’re dealing with a lot of data, you’re unlikely to notice, because in-memory operations are really fast.

To understand why in-memory is so fast, a good resource is this chart of latency numbers every programmer should know. Or I can give you the TLDR, which I’m happy to be quoted on:

“Disk is about a bazillion times slower than memory, and the network is about a bazillion times slower than that.”

— Nolan Lawson

Of course, the tradeoff with in-memory is that your data isn’t saved. So let’s look at some ways of writing data that will actually survive a browser refresh.

LocalStorage

In all three of Chrome, Firefox, and Edge, LocalStorage fully blocks the DOM while you’re writing data 2. The blocking is a lot more noticeable than with in-memory, since the browser has to actually flush to disk.

This is pretty much the banner reason not to use LocalStorage. Even if the API only takes a few hundred milliseconds to return after inserting 10000 records, you’ll notice that the DOM might block for a long time after that. I assume this is because these browsers cache LocalStorage to memory and then batch their write operations (here’s how Firefox does it), but in any case the UI still ends up looking janky.

In Safari, the situation is even worse. Somehow the DOM isn’t blocked at all during LocalStorage operations, but on the other hand, if you insert too much data, you’ll get a spinning beach ball of doom, and the page will be permanently frozen. I’ve filed this as a bug on WebKit.

WebSQL

We can only test this one in Chrome and Safari, but it’s still pretty instructive. In Chrome, WebSQL actually blocks the DOM quite a bit, at least for heavy operations. Whereas in Safari, the animations all remain buttery-smooth, no matter what WebSQL is doing.

This should fill you with a sense of foreboding, as we start to move on to the supposed savior of client-side databases, IndexedDB. Aren’t both WebSQL and IndexedDB asynchronous? Don’t they have nothing to do with the DOM? Why should they block DOM rendering at all?

I myself was pretty shocked by these results, even though I’ve worked extensively with these APIs over the past two years. But let’s keep going further and see how deep this rabbit hole goes…

IndexedDB

If you try that demo page in Chrome or Firefox, you may be surprised to see that IndexedDB actually blocks the DOM for nearly the entire duration of the operation 3. In Safari, I don’t see this behavior at all (although IndexedDB is painfully slow), whereas in Edge I see the occasional dropped frame.

In both Firefox and Chrome, IndexedDB is slower than LocalStorage for basic key-value insertions, and it still blocks the DOM. In Chrome, it’s also slower than WebSQL, which does blocks the DOM, but not nearly as much. Only in Edge and Safari does IndexedDB manage to run in the background without interrupting the UI, and aggravatingly, those are the two browsers that only partially implement the IndexedDB spec.

This was a pretty shocking find, so I promptly filed a bug both on Chrome and on Firefox. It saddens me to think that this is just one more reason web developers will have to ignore IndexedDB – what with the shoddy browser support and the ugly API, we can now add the fact that it doesn’t even deliver on its promise of beating LocalStorage at DOM performance.

Web workers FTW

I do have some good news: IndexedDB works swimmingly well in a web worker, where it runs at roughly the same speed but without blocking the DOM. The only exception is Safari, which doesn’t support IndexedDB inside a worker.

So that means that for Chrome and Firefox, you can always offload your expensive IndexedDB operations to a worker thread, where there’s no chance of blocking the UI thread. In my own tests, I didn’t see a single dropped frame when using this method.

It’s also worth acknowledging that IndexedDB is the only storage option inside of a web worker (or a service worker, for that matter). Neither WebSQL nor LocalStorage are available inside of a worker for any of the browsers I tested; the localStorage and openDatabase globals just aren’t there. (Support for WebSQL used to exist in Chrome and Safari, but has since been removed.)

Test results

I’ve gathered these results into a consolidated table, along with the time taken in milliseconds as measured by a simple Date.now() comparison. All tests were on a 2013 MacBook Air; Edge was run in a Windows 10 VirtualBox. “In-memory” refers to a regular JavaScript object (“regular object” in the demo page). Between each test, all browser data was cleared and the page refreshed.

Take these raw numbers with the grain of salt. They only account for the time taken for the API in question to return successfully (or finish the transaction, in the case of IndexedDB and WebSQL), and they don’t guarantee that the data was durably written or that the DOM wasn’t blocked after the operation completed. However, it is interesting to compare the speed across browsers, and it’s pretty consistent with what I’ve seen from working on PouchDB over the past couple of years.

Number of insertions   1000     10000     100000     Blocks?     Notes  
Chrome 47
   In-memory 4 10 217 Yes
   LocalStorage 18 527 4725 Yes
   WebSQL 45 213 1927 Partially Blocks a bit at the beginning
   IndexedDB 64 572 5372 Yes
   IndexedDB
     in a worker
66 604 6108 No
Firefox 43
   In-memory 1 12 152 Yes
   LocalStorage 19 177 1950 Yes Froze significantly after loop finished
   IndexedDB 114 823 8849 Yes
   IndexedDB
     in a worker
132 1006 9264 No
Safari 9
   In-memory 2 8 100 Yes
   LocalStorage 6 41 418 No 10000 and 100000 crashed the page
   WebSQL 26 173 1557 No
   IndexedDB 1093 10658 117790 No
Edge 20
   In-memory 7 19 331 Yes
   LocalStorage 198 4624 N/A Yes 100000 crashed the page
   IndexedDB 315 5657 28662 Slightly A few frames lost at the beginning
   IndexedDB
     in a worker
985 2881 24236 No

 

Edit: The LocalStorage results are inaccurate, because there was a bug in the test suite causing it to improperly store the JavaScript objects as '[object Object]' rather than using JSON.stringify(). After the fix, LocalStorage performs more poorly.

Key takeaways from the data:

  1. WebSQL is faster than IndexedDB in both Chrome (~2x) and Safari (~100x!) even though I’m inserting unstructured JSON with a string key, which should be IndexedDB’s bread and butter.
  2. LocalStorage is slightly faster than IndexedDB in all browsers (disregarding the crashes).
  3. IndexedDB is not significantly slower when run in a web worker, and never blocks the DOM that way.

Again, these numbers wasn’t gathered in a super rigorous way (I only ran the tests once; didn’t average them or anything), but it should give you an idea of what kind of behavior you can expect from these APIs in different browsers. You can run the demo page yourself to try to reproduce my results.

Conclusion

Running IndexedDB in a web worker is a nice workaround for DOM slowness, but in principle it ought to run smoothly in either environment. Originally, the whole selling point of IndexedDB was that it would improve upon both LocalStorage and WebSQL, finally giving web developers the same kind of storage power that native developers have enjoyed for the past several years.

IndexedDB’s awkward asynchronous API was supposed to be a bitter medicine that, if you swallowed it, would pay off in terms of performance. But according to my tests, that just isn’t the case, at least with IndexedDB’s two flagship browsers, Chrome and Firefox.

I’m still hopeful that browser vendors will resolve all these issues with IndexedDB, although with the spec being over five years old, it sure feels like we’ve been waiting a long time. As someone who does both native and web development for a living, I’m tired of reciting a list of reasons why the web “isn’t quite there yet.” And IndexedDB has been too high on that list for too long.

IndexedDB was the web’s chance to finally get local storage right. It was the chosen one. It was supposed to lead us out of the morass of half-baked solutions and provide the best and fastest way to work with data on the client side. It’s come a long way, but I’m still waiting for it to make good on that original promise.

Footnotes


1: Yes, I’m ignoring cookies, the File API, window.name, SessionStorage, the Service Worker cache, and probably a few other oddballs. There are actually lots of ways of storing data (too many in my opinion), but all of them have niche use cases except for LocalStorage, WebSQL, and IndexedDB.


2: In this article, when I say “Chrome,” “Firefox,” “Edge,” and “Safari”, I mean Chrome Canary 47, Firefox Developer Edition 43, Edge 20, and WebKit Nightly 10600.8.9. All tests were run on a 2013 MacBook Air; Edge was run in Windows 10 using Virtual Box.


3: This blog post used to suggest that the DOM was blocked for the entire duration of the transaction, but after an exchange with Ben Kelly I changed the wording to “nearly the entire duration.”


Thanks to Dale Harvey for providing feedback on a draft of this blog post.

Safari is the new IE 2: Revenge of the Linkbait

The things I write rarely have broad appeal. I tend to write about weird esoteric stuff like IndexedDB and WebSQL, maybe throwing the normals a bone with something about CSS animations. I’m not some kind of thought-leader.

My “Safari is the new IE” article, however, got shared incessantly, was picked up by Ars Technica, and attracted the attention of people a lot smarter than me on both sides of the ensuing debate. Having no less than Don Melton call me out on Twitter has pretty much been the highlight of my career so far. It’s been overwhelming, to say the least.

I thoroughly enjoyed the resulting debate, though, and yes, there are valid arguments on both sides. So I’d like to pick up where I left off, and respond to my detractors while doubling-down on my claim that Safari is acting as an anchor in the web community. I’ll also end on a hopeful note, with some suggestions for reconciliation with Apple.

And since, judging by the response, people tend not to read much further than the first few paragraphs (or even the title!), I’ll keep the most important points up top, so nobody can miss them.

“Linkbait,” Safari != IE, etc.

I wrote that article in a mad rush the morning after I got back from EdgeConf. It was the culmination of a lot of frustration I’ve had with Safari over the past year or so, compounded by the fact that I just got back from a conference where I would have loved to pick the brain of somebody from Apple about it.

To be honest, I penned the whole thing in one go before settling on the title, which was based on something funny Calvin Metcalf had said in a presentation. Yes, the title was snappy, and yes, it was a bit sensational. But hey, nobody would be talking about it if I had called it “Meditations on the uncanny parallels between…” I choose headlines that grab attention; welcome to journalism 101.

So I find the “linkbait” accusations to be a boring argument, because it’s an argument that can be had without discussing the content of the article at all. More interesting were the arguments that the IE analogy is flawed, because Safari doesn’t have anything like ActiveX and Apple didn’t walk away from Safari for 5 years. All very good points. There’s also a case to be made that Android is more reminiscent of the old IE days, what with so many devices frozen in time on ancient versions of WebKit. (Although Lollipop, default Chrome, and Crosswalk have helped a lot lately.)

That being said, my point was to compare Safari to IE in terms of 1) not keeping up with new standards, 2) maintaining a culture of relative secrecy, and 3) playing a monopolistic role, by not allowing other rendering engines on iOS. Those accusations are pretty undeniable.

Web Components, Shadow DOM

I basically pulled these two examples out of a hat, because they were oft-discussed at EdgeConf, but I couldn’t find where Apple had even commented on them. Of course, such information is buried in mailing list discussions where I should have done the research to find it. (“Read the mailing list” is the “RTFM” of the W3C community.)

So yes, it was unfair of me to say that Apple “has shown no public interest” in those specs. I corrected the article and personally apologized to Ryosuke Niwa, who has been very active in the design of Web Components and Shadow DOM. Mea culpa; I made a dumb mistake.

The “user-centric web”

This point was made in a rebuttal article by Rene Ritchie, and although I enjoyed reading it, I don’t find the argument very persuasive. It boils down to a false dichotomy, saying that Apple can either focus on user-facing features (like speed and battery life) or new web APIs, but it can’t do both.

Other browser vendors, though, seem to be able to keep up with Google’s relentless pace (another counterargument), so I don’t understand how Apple, with its heaps of moolah, can’t do the same. And to those saying “it’s not the money, it’s finding the right people,” well, then maybe there’s a valid question about why Apple can’t find enough good browser developers. Something must have driven Google away when they forked WebKit into Blink (taking most of the committers with them), and I wonder if that same thing is keeping away new talent. But that’s a story I suspect only the WebKit/Blink developers really know.

From my own interactions with the WebKit developers, I can only conclude that they are sincere folks who take an extreme pride in their work. However, it’s become clear to me that Apple has different priorities, and that those priorities are limiting the potential of the WebKit team. One need only glance at the Safari 9 release notes to see both a paucity of new features, as well as a focus on proprietary and user-facing features. In terms of standards, a one-year release from Safari is comparable to two releases from Chrome, representing about 3 months of work. That’s a shockingly deep deficit.

(Proprietary or user-facing: force touch events, AirPlay, Picture in Picture, pinned tab icons, secure extensions, shared links, content blocking. Standards: scroll snapping, filters, ES6, unprefixing. Other: responsive design mode, Web Inspector overhaul, SFSafariViewController.)

As for Ritchie’s other argument that Apple is shying away from “native-hopeful” web features that “don’t make sense,” given the miraculous speed boosts that both WebKit and Chrome have demonstrated recently, I doubt anyone on either team actually shares that opinion. The web can reach 60 FPS, and where it can’t, that’s an argument for more progress in the web, not less.

IndexedDB, my idée fixe

Since nobody else brought this up, I’ll offer what I think is the best counterpoint to my article. What I basically did was take Apple’s utter bungling of IndexedDB, add in their lack of clear signals and iron grip on iOS, and extrapolate that Safari is dragging us back into some kind of new web Dark Ages. Okay! It sounds a bit hyperbolic, I’ll agree.

However, you’ll have to forgive me if I fixate on IndexedDB. While CSS features like scroll snap points and position:sticky are nice, I happen to think the thing that stores user data is pretty damned important. Imagine writing an iOS/Android app without SQLite/CoreData, and you’ll understand how badly web devs need consistent IndexedDB support. The fact that Apple messed it up so catastrophically shows a carelessness in their approach to the new, “appy” web.

Of course, that statement exposes my own bias in this debate, which I’ll gladly divulge. Namely: I’m an Android developer who’s tired of writing apps for a single platform, and wants webapps that can compete with native apps. That’s why I contribute to PouchDB – because although the IndexedDB spec is 5 years old, you still need a mountain of shims to get anything done with it. It’s a nasty situation, but PouchDB makes it bearable.

Even with tools like PouchDB, though, it’s still maddeningly difficult for web developers to create anything resembling a native app. And for that, the blame usually falls square on Safari, and especially mobile Safari. For instance, when PouchDB users ask me how much data they can store on iOS, and they find out it’s capped at 50MB with a modal popup after 5MB, they often say, “Thanks, but I’ll write a native app instead.” Yet when I complain about this stuff to Apple, they mostly shrug their shoulders.

Working on mobile webapps, I often find myself reaching for Safari polyfills (e.g. FastClick, which I cannot believe we still need), or relying on years-old standards that are in sore need of a replacement (WebSQL, AppCache, touch icons). Whereas at the same time, I’m seeing constant innovation from other browsers to improve the state of the “appy” web: Service Worker from Google, pointer events from Microsoft, and just about everything Mozilla has done on Firefox OS.

Gestures (touch-action). Vibration API. Ambient Light API. WebRTC (or Microsoft’s version). Device Orientation API. Permissions API. There’s a laundry list of things that would make my life easier as a webapp developer, and the one unifying feature is that their CanIUse page has two big red columns under “iOS” and “Safari.” You can add Web Manifests and Push API if you want to talk about stuff that’s still in the planning phase (not by Apple, though).

So yeah, by the raw HTML5Test numbers, Safari isn’t so far behind. But in the stuff that matters to me as a mobile and webapp developer, they’ve got a lot of catching up to do. And based on their priorities and release cycle, I’m not confident that they’re even keeping pace.

Engaging the community

The response to my article from web developers has been telling. Amidst all the “hear hear”s and their own tales of woe with Safari, you could detect that web developers just have an intense distrust of Apple. It’s revealing how quickly they started a petition against Apple, which to be honest made me kinda uncomfortable.

Let’s explore those feelings a bit, though. Why are web developers so wary of Apple? I have my own answer, and it touches on some of the most important points I raised in the article.

The web is increasingly becoming an open community. JavaScript is the most popular language on Github, and web developers are drowning in conferences, meetups, and hackathons. I regularly attend several meetups in New York City, many of which did not even exist a year ago. There are so many of these things nowadays, I can attend nearly one a week.

Through these community events and my activity on Github, I’ve had the pleasure of meeting and collaborating with people from Mozilla, Microsoft, and Google. Sometimes they’ve even come out of the blue to propose a pull request to PouchDB, or ask for feedback on IndexedDB. And yet, I’ve never once seen an Apple employee at any of these events, nor have I ever worked with any of them on an open-source project. My sole interaction with Apple has been through their bugtracker (or on Twitter, poking them to look at their bugtracker).

With even Microsoft being all chummy these days, the web has become an open party, but Apple is frequently tossing their invitation in the trash. This extends beyond the official discussions in the W3C mailing lists. As IE engineer Jacob Rossi put it, conferences like EdgeConf are “a crucial part of participating in standards.” As with any human endeavor, some of the most important discussions about the web platform are happening in hallways, at restaurants, and in face-to-face meetings. Apple can’t complain about getting cut out of the standards process, if they won’t even show up to join the conversation.

Apple’s lack of engagement with the broader web community has also been damaging to their reputation. By not making the effort to attend meetups, write blog posts, or set up forums for feedback, it’s no wonder developers are left with the impression that Apple doesn’t care about the web. Even just sending a developer evangelist to a few meetups to smile, nod, and answer questions politely would do wonders for their public perception.

Furthermore, Apple’s lack of boots on the ground at everyday developer soirées means that they’re increasingly out of touch with what developers want from the web platform. The fact that so many meetups and conferences have sprung up recently, and the fact that the web is fast becoming the world’s most advanced cross-platform application runtime are not isolated incidents. Developers like myself are getting excited about the web precisely because it’s supplanting all the old application paradigms. But as I pointed out above, the “appy” aspects of the web are exactly where Safari tends to falter.

Conclusion

Personally what I want out of this whole debate is for Apple to realize that the web is starting to move on without them, and that their weird isolationism and glacial release cycle are not going to win them any favors in this new, dynamic web community. I don’t even want them to open up iOS to other browsers nearly as much as I want them to just start coming to events and talking with web developers. Once they hear our gripes and see the frustration in our eyes as we describe how much we’ve struggled to support their browser, I think they’ll be motivated by empathy alone to start fixing these problems.

I do regret the generalizations and errata in my original post. However, I don’t regret starting the debate, because clearly I’ve touched a nerve at Apple, while casting a light on the widening divide between them and the rest of the web community. Maybe a harsh diatribe is exactly what Apple needs to shake them out of their complacency, even if it loses me a few friends in Cupertino.

So what I’m saying is this: next time I’m at a conference, I hope someone from Apple comes up to me, takes off a glove, and slaps me right in the face. Number one, because I kinda deserve it for being a jerk to them, and number two, because that would mean they’re finally coming to conferences.

Thanks to Jan Lehnardt, Chris Gullian, and Dale Harvey for reviewing a draft of this post.

Safari is the new IE

Update: This post represents my views before I joined the Microsoft Edge team, and before I got involved with browsers or web standards in general. I leave it up as a historically interesting artifact.

Last weekend I attended EdgeConf, a conference populated by many of the leading lights in the web industry. It featured panel talks and breakout sessions with a focus on technologies that are just now starting to emerge in browsers, so there was a lot of lively discussion around Service Worker, Web Components, Shadow DOM, Web Manifests, and more.

EdgeConf’s hundred-odd attendees were truly the heavy hitters of the web community. The average Twitter follower count in any given room was probably in the thousands, and all the major browser vendors were represented – Google, Mozilla, Microsoft, Opera. So we had lots of fun peppering them with questions about when they might release such-and-such API.

There was one company not in attendance, though, and they served as the proverbial elephant in the room that no one wanted to discuss. I heard them referred to cagily as “a company in California” or “a certain fruit company.” Their glowing logo illuminated nearly every laptop in the room, and yet it seemed like nobody dared speak their name. Of course I’m talking about Apple.

I think there is a general feeling among web developers that Safari is lagging behind the other browsers, but when you go to a conference like EdgeConf, it really strikes you just how wide the gap is. All of the APIs I mentioned above are not implemented in Safari, and Apple has shown no public interest in them. (Correction: actually, they have.) When you start browsing caniuse.com, the list goes on and on.

Even when Apple does implement newer APIs, they often do it halfheartedly. To take an example close to my heart, IndexedDB was proposed more than 5 years ago and has been available in IE, Firefox, and Chrome since 2012. Apple, on the other hand, didn’t release IndexedDB until mid-2014, and when they did, they unveiled a bafflingly incompetent implementation that was so bad, it’s been universally derided as unusable. (LocalForage, PouchDB, and YDN-DB, the major IndexedDB wrappers, all ignore Safari’s version and fall back to WebSQL.)

Now, after one year, Apple has fixed a whopping two bugs in IndexedDB (out of several), and they’ve publicly stated that they don’t find much value in working on it, because they don’t see “a huge use.” Well duh, nobody’s going to use IndexedDB if the browser support is completely broken. (Microsoft, I’m looking at you too.)

It’s hard to get insight into why Apple is behaving this way. They never send anyone to web conferences, their Surfin’ Safari blog is a shadow of its former self, and nobody knows what the next version of Safari will contain until that year’s WWDC. In a sense, Apple is like Santa Claus, descending yearly to give us some much-anticipated presents, with no forewarning about which of our wishes he’ll grant this year. And frankly, the presents have been getting smaller and smaller lately.

In recent years, Apple’s strategy towards the web can most charitably be described as “benevolent neglect.” Although performance has been improving significantly with JSCore and the new WKWebView, the emerging features of the web platform – offline storage, push notifications, and “installable” webapps – have been notably absent on Safari. It’s tempting to interpret this as a deliberate effort by Apple to sabotage any threats to their App Store business model, but a conspiracy seems unlikely, since that part of the business mostly breaks even. Another possibility is that they’re just responding to the demands of iOS developers, which largely amount to 1) more native APIs and 2) Swift, Swift, Swift. But since Apple is pretty good at keeping a lid on their internal process, it’s anyone’s guess.

The tragedy here is that Apple hasn’t always been a web skeptic. As recently as 2010, back when Steve Jobs famously skewered Flash while declaring that HTML5 is the future, Apple was a fierce web partisan. Lots of the early features that helped webapps catch up to native apps – ApplicationCache, WebSQL, touch events, touch icons – were enthusiastically implemented by WebKit developers, and many even originated at Apple.

Around that same time, when WebSQL was deprecated in favor of IndexedDB, you’ll even find mailing list arguments where Apple employees vigorously defended WebSQL as a must for performant web applications. Reading the debates, I sense a lot of bitterness from Apple after IndexedDB won out. The irony here is that Apple nearly gave us the tools to undermine their own proprietary platform, but by rejecting WebSQL, we gave them an opportunity to rethink their strategy and put the brakes on any new progress in web APIs.

I find Application Cache, which will probably soon be deprecated in favor of Service Worker, to be a similar story. It gained wide browser support at a time when Apple was still interested in the web, but unfortunately it turned out to be a rushed, half-baked solution to the problem. I worry that Service Worker might suffer the same fate as IndexedDB, if Apple continues to lag behind the pack.

At this point, we in the web community need to come to terms with the fact that Safari has become the new IE. Microsoft is repentant these days, Google is pushing the web as far as it can go, and Mozilla is still being Mozilla. Apple is really the one singer in that barbershop quartet hitting all the sour notes, and it’s time we start talking about it openly instead of tiptoeing around it like we’re going to hurt somebody’s feelings. Apple is the most valuable company in the world; they can afford to take a few punches.

So what can we do, when one of the major browser vendors is stuck in the 2010 model, and furthermore has a total monopoly on iOS (because no, “Chrome for iOS” is not really Chrome), showing a brazenness even beyond that of 90’s-era Microsoft? I see three major coping mechanisms:

  1. Stick with what worked in 2010, and use polyfills to support Safari. This is a strategy I highlighted in my opening talk for the frontend data panel, where I showed that you can get nearly the same features as Service Worker by using AppCache and PouchDB (which falls back to WebSQL on Safari). This approach should appeal to the vast majority of web developers, who tend to hit the snooze button on new technologies until they’re available cross-browser. On the other hand, it’s also a good way to coddle Apple and give them no incentive to step up their game.

  2. Use technologies like Service Worker that don’t work on Safari, and consider it a progressive enhancement. Alex Russell made a great point about this during the “installable webapps” breakout, arguing that if we create a large body of free webapps that use Service Worker, and which work fabulously well on Android but only meh on iOS, then it will be in Apple’s interest to suck it up and support the API. Unfortunately, while this would be the best outcome for the web community as a whole, it’s going to be hard to convince developers to write code that only reaches half their audience.

  3. Contribute to WebKit. The core of Safari is still, after all, an open-source project, so there’s no practical reason why anyone with the C++ chops couldn’t roll up their sleeves and implement the new APIs themselves. (The absurdity of giving free labor to the richest company on earth isn’t lost on me, but we’re talking desperate times here.) The major problem I see with this approach is that WebKit is not Safari, and Apple could still decide to not deploy WebKit features to their flagship browser. To circle back to IndexedDB again, it was fully implemented by Google well before the Blink fork, and for several years, Apple could have just flipped a bit in their build script to include Google’s implementation, but instead they decided to waffle for a few years and then replace it with their own broken version. (Update: see note on IndexedDB below.) There’s no guarantee they wouldn’t do the same thing for other outside contributions.

So in summary, I don’t know what the right solution is. I’ve engaged many of the WebKit developers on Twitter, and I’ve even done the hard work of writing reproducible test cases and trying out their beta software so I can give them early warning. (Yes, I fork over $200 a year to Apple, for the privilege of testing their own software for them.) And frankly I’ve grown bitter, because most of the bugs I’ve reported have languished, with little response other than a link to their internal Radar tracker.

I appreciate the work that many of the WebKit developers have been doing (Brady Eidson has been particularly patient and helpful), but at this point it seems to me that the best strategy toward Apple may be the stick rather than the carrot. So I’m inclined to take up Alex Russell’s solution outlined in #2 above, and to start promoting the adoption of new web technologies – Safari support be damned.

If we can start building a vibrant ecosystem of web applications where Apple is not invited, then maybe they’ll be forced to pull a Microsoft and make their own penitent walk to Canossa. Otherwise we’ll have to content ourselves with living in the web of 2010, with Safari replacing IE as the blue-tinged icon that fills web developers with dread.

Update on IndexedDB: I’m told by Ryosuke Niwa at Apple that Google’s IndexedDB was not so easily usable in Safari, even pre-fork. So it wasn’t a build flag. However, in a private discussion with a Google employee, I’m told that the IPC layer was abstracted to a degree that it shouldn’t have been too difficult for Safari to use. In any case, it’s true that Apple had something close to a fully working implementation years before they shipped their inferior version.

You can comment on Hacker News, on Ars Technica, and on Twitter. Thanks to Jan Lehnardt, Dave Myers, Beckie Choi, and Julian Applebaum for providing feedback on a draft of this blog post.