Read the Tea Leaves Software and other dark arts, by Nolan Lawson

Update June 2019: This blog post was written in 2015. The benchmarks are out-of-date. WebSQL is still deprecated and in fact being removed from iOS Safari. Web Workers are not necessarily a panacea. I’d recommend using IndexedDB for large data, and LocalStorage for small amounts of data that you need synchronous access to.

When it comes to databases, a lot of people just want to know: which one is the fastest?

Never mind things like memory usage, the CAP theorem, consistency, read vs write speed, test coverage, documentation – just tell me which one is the fastest, dammit!

This mindset is understandable. A single number is easier to grasp than a big table of features, and it’s fun to make grand statements like “Redis is 20x faster than Mongo.” (N.B.: I just made that up.)

As someone who spends a lot of time on browser databases, though, I think it’s important to look past the raw speed numbers. On the client side especially, the way you use a database, and how it interacts with the JavaScript environment, has a big impact on something more important than performance: how your users perceive performance.

In this post, I’m going to take a look at various browser databases with regard not only to their speed, but to how much they block the DOM.

TLDR: IndexedDB isn’t nearly the performance home-run that many in the web community think it is. In my tests, I found that it blocked the DOM significantly in Firefox and Chrome, and was slower than both LocalStorage and WebSQL for basic key-value insertions.

Browser database landscape

For the uninitiated, the world of browser databases can be a confusing one. Lawnchair, PouchDB, LocalForage, Dexie, Lovefield, LokiJS, AlaSQL, MakeDrive, ForerunnerDB, YDN-DB – that’s a lot of databases!

As it turns out, though, the situation is much simpler than it appears on the surface. In fact, there are only three ways of storing data in the browser:

• LocalStorage
• WebSQL
• IndexedDB

Every “database” listed above uses one of those three under the hood (or they operate in-memory). So to understand browser storage, you only really need to understand LocalStorage, WebSQL, and IndexedDB ¹.

LocalStorage is a lightweight way to store key-value pairs. The API is very simple, but usage is capped at 5MB in many browsers. Plus the API is synchronous, so as we’ll see later, it can block the DOM. Browser support is very good.

WebSQL is an API that is only supported in Chrome and Safari (and Android and iOS by extension). It provides an asynchronous, transactional interface to SQLite. Since 2010, it has been deprecated in favor of IndexedDB.

IndexedDB is the successor to both LocalStorage and WebSQL, designed to replace them as the “one true” browser database. It exposes an asynchronous API that supposedly avoids blocking the DOM, but as we’ll see below, it doesn’t necessarily live up to the hype. Browser support is extremely spotty, with only Chrome and Firefox having fully usable implementations.

Now, let’s run a simple test to see when and how these APIs block the DOM.

Thou shalt not block the DOM

JavaScript is a single-threaded programming environment, meaning that synchronous operations are blocking. And since the DOM is synchronous, this means that when JavaScript blocks, the DOM is also blocked. So if any operation takes longer than 16ms, it can lead to dropped frames, which users experience as slowness, “stuttering,” or “jank.”

This is the reason that JavaScript has so many asynchronous APIs. Just imagine if your entire page was frozen during every AJAX request – wouldn’t the web be an awful user experience if it worked that way! Hence the profusion of programming constructs like callbacks, promises, event listeners, and the like.

To demonstrate DOM blocking, I’ve put together a simple demo page with an animated GIF. Whenever the DOM is blocked, Kirby will stop his happy dance and freeze in place.

Try this experiment: go to that page, open up the developer tools, and enter the following code:

for (var i = 0; i < 10000; i++) {console.log('blocked!')}

You’ll see that Kirby freezes for the duration of the for-loop:

This affects more than just animated GIFs; any JavaScript animation or DOM operation, such as adding or modifying elements, will also be blocked. You can’t even select a radio button; the page is totally unresponsive. The only animations that are unaffected are hardware-accelerated CSS animations.

Using this demo page, I tested four ways of of storing data: in-memory, LocalStorage, WebSQL, and IndexedDB. The test inserts a given number of “documents,” which are just unstructured JSON keyed by a string ID. I made a YouTube video showing my results, but the rest of the article will summarize my findings.

In-memory

Not surprisingly, since any synchronous code is blocking, in-memory operations are also blocking. You can test this in the demo page by choosing “regular object” or “LokiJS” (which is an in-memory database). The DOM blocks during long-running inserts, but unless you’re dealing with a lot of data, you’re unlikely to notice, because in-memory operations are really fast.

To understand why in-memory is so fast, a good resource is this chart of latency numbers every programmer should know. Or I can give you the TLDR, which I’m happy to be quoted on:

“Disk is about a bazillion times slower than memory, and the network is about a bazillion times slower than that.”

— Nolan Lawson

Of course, the tradeoff with in-memory is that your data isn’t saved. So let’s look at some ways of writing data that will actually survive a browser refresh.

LocalStorage

In all three of Chrome, Firefox, and Edge, LocalStorage fully blocks the DOM while you’re writing data ². The blocking is a lot more noticeable than with in-memory, since the browser has to actually flush to disk.

This is pretty much the banner reason not to use LocalStorage. Even if the API only takes a few hundred milliseconds to return after inserting 10000 records, you’ll notice that the DOM might block for a long time after that. I assume this is because these browsers cache LocalStorage to memory and then batch their write operations (here’s how Firefox does it), but in any case the UI still ends up looking janky.

In Safari, the situation is even worse. Somehow the DOM isn’t blocked at all during LocalStorage operations, but on the other hand, if you insert too much data, you’ll get a spinning beach ball of doom, and the page will be permanently frozen. I’ve filed this as a bug on WebKit.

WebSQL

We can only test this one in Chrome and Safari, but it’s still pretty instructive. In Chrome, WebSQL actually blocks the DOM quite a bit, at least for heavy operations. Whereas in Safari, the animations all remain buttery-smooth, no matter what WebSQL is doing.

This should fill you with a sense of foreboding, as we start to move on to the supposed savior of client-side databases, IndexedDB. Aren’t both WebSQL and IndexedDB asynchronous? Don’t they have nothing to do with the DOM? Why should they block DOM rendering at all?

I myself was pretty shocked by these results, even though I’ve worked extensively with these APIs over the past two years. But let’s keep going further and see how deep this rabbit hole goes…

IndexedDB

If you try that demo page in Chrome or Firefox, you may be surprised to see that IndexedDB actually blocks the DOM for nearly the entire duration of the operation ³. In Safari, I don’t see this behavior at all (although IndexedDB is painfully slow), whereas in Edge I see the occasional dropped frame.

In both Firefox and Chrome, IndexedDB is slower than LocalStorage for basic key-value insertions, and it still blocks the DOM. In Chrome, it’s also slower than WebSQL, which does blocks the DOM, but not nearly as much. Only in Edge and Safari does IndexedDB manage to run in the background without interrupting the UI, and aggravatingly, those are the two browsers that only partially implement the IndexedDB spec.

This was a pretty shocking find, so I promptly filed a bug both on Chrome and on Firefox. It saddens me to think that this is just one more reason web developers will have to ignore IndexedDB – what with the shoddy browser support and the ugly API, we can now add the fact that it doesn’t even deliver on its promise of beating LocalStorage at DOM performance.

Web workers FTW

I do have some good news: IndexedDB works swimmingly well in a web worker, where it runs at roughly the same speed but without blocking the DOM. The only exception is Safari, which doesn’t support IndexedDB inside a worker.

So that means that for Chrome and Firefox, you can always offload your expensive IndexedDB operations to a worker thread, where there’s no chance of blocking the UI thread. In my own tests, I didn’t see a single dropped frame when using this method.

It’s also worth acknowledging that IndexedDB is the only storage option inside of a web worker (or a service worker, for that matter). Neither WebSQL nor LocalStorage are available inside of a worker for any of the browsers I tested; the localStorage and openDatabase globals just aren’t there. (Support for WebSQL used to exist in Chrome and Safari, but has since been removed.)

Test results

I’ve gathered these results into a consolidated table, along with the time taken in milliseconds as measured by a simple Date.now() comparison. All tests were on a 2013 MacBook Air; Edge was run in a Windows 10 VirtualBox. “In-memory” refers to a regular JavaScript object (“regular object” in the demo page). Between each test, all browser data was cleared and the page refreshed.

Take these raw numbers with the grain of salt. They only account for the time taken for the API in question to return successfully (or finish the transaction, in the case of IndexedDB and WebSQL), and they don’t guarantee that the data was durably written or that the DOM wasn’t blocked after the operation completed. However, it is interesting to compare the speed across browsers, and it’s pretty consistent with what I’ve seen from working on PouchDB over the past couple of years.

Edit: The LocalStorage results are inaccurate, because there was a bug in the test suite causing it to improperly store the JavaScript objects as '[object Object]' rather than using JSON.stringify(). After the fix, LocalStorage performs more poorly.

Key takeaways from the data:

1. WebSQL is faster than IndexedDB in both Chrome (~2x) and Safari (~100x!) even though I’m inserting unstructured JSON with a string key, which should be IndexedDB’s bread and butter.
2. LocalStorage is slightly faster than IndexedDB in all browsers (disregarding the crashes).
3. IndexedDB is not significantly slower when run in a web worker, and never blocks the DOM that way.

Again, these numbers wasn’t gathered in a super rigorous way (I only ran the tests once; didn’t average them or anything), but it should give you an idea of what kind of behavior you can expect from these APIs in different browsers. You can run the demo page yourself to try to reproduce my results.

Conclusion

Running IndexedDB in a web worker is a nice workaround for DOM slowness, but in principle it ought to run smoothly in either environment. Originally, the whole selling point of IndexedDB was that it would improve upon both LocalStorage and WebSQL, finally giving web developers the same kind of storage power that native developers have enjoyed for the past several years.

IndexedDB’s awkward asynchronous API was supposed to be a bitter medicine that, if you swallowed it, would pay off in terms of performance. But according to my tests, that just isn’t the case, at least with IndexedDB’s two flagship browsers, Chrome and Firefox.

I’m still hopeful that browser vendors will resolve all these issues with IndexedDB, although with the spec being over five years old, it sure feels like we’ve been waiting a long time. As someone who does both native and web development for a living, I’m tired of reciting a list of reasons why the web “isn’t quite there yet.” And IndexedDB has been too high on that list for too long.

IndexedDB was the web’s chance to finally get local storage right. It was the chosen one. It was supposed to lead us out of the morass of half-baked solutions and provide the best and fastest way to work with data on the client side. It’s come a long way, but I’m still waiting for it to make good on that original promise.

Footnotes

1: Yes, I’m ignoring cookies, the File API, window.name, SessionStorage, the Service Worker cache, and probably a few other oddballs. There are actually lots of ways of storing data (too many in my opinion), but all of them have niche use cases except for LocalStorage, WebSQL, and IndexedDB.


2: In this article, when I say “Chrome,” “Firefox,” “Edge,” and “Safari”, I mean Chrome Canary 47, Firefox Developer Edition 43, Edge 20, and WebKit Nightly 10600.8.9. All tests were run on a 2013 MacBook Air; Edge was run in Windows 10 using Virtual Box.


3: This blog post used to suggest that the DOM was blocked for the entire duration of the transaction, but after an exchange with Ben Kelly I changed the wording to “nearly the entire duration.”


Thanks to Dale Harvey for providing feedback on a draft of this blog post.