Posts Tagged ‘accessibility’

Introducing emoji-picker-element: a memory-efficient emoji picker for the web

Screenshot of emoji-picker-element, an emoji picker, in light and dark mode, with grids of emoji faces

Emoji pickers are ubiquitous. It seems that every social media and messaging app needs to have a little grid of cartoon faces you can click on.

There’s nothing inherently wrong with emoji (they’re fun! they’re popular! they make communication livelier!). But the way they’re currently used on the web is wasteful.

The emoji picker landscape

The main problem is that there are a lot of emoji: 1,814 as of the latest version, Emoji v.13.0. That doesn’t even include the skin tone variants (or combinations of skin tones), and each of those emoji has associated shortcodes, tags, ASCII emoticons, etc. And the way most web-based emoji pickers work, all of this data is stored in-memory.

A popular JavaScript library to manage emoji data, emojibase, currently offers two JSON files for English: the main one, which is 854kB, and the “compact” one, which is 543kB. Now imagine that, for every browser tab you have open, each website is potentially loading over a half a megabyte of data, just to show a little grid of emoji!

The median web page is now around 2MB, according to the Internet Archive. Hopefully any emoji picker data is lazy-loaded, but either way, half a megabyte is a big chunk in any reasonable perf budget.

You could say that these websites should ditch the custom picker and just ask people to use the built-in emoji picker for their OS. The problem is that a lot of people don’t know that these exist. (How many Mac users know about Cmd+Ctrl+Space? How many Windows users have memorized Win+.?) Some OSes don’t even have a native emoji picker (try Ubuntu or stock Android for an example). Even ignoring these problems, websites often need to tweak the emoji picker, such as adding their own custom emoji (e.g. Discord, Slack, Mastodon).

Screenshot of the built-in emoji pickers on macOS and Windows

The built-in emoji pickers on macOS and Windows. Ask a non-techie if they’ve ever even seen these.

Shouldn’t browser vendors offer a standard emoji picker element? Something like <input type="emoji">? I’ve actually proposed this in the past, but I’m not aware of any browser vendors that were interested in picking it up.

Also, standardizing an emoji picker would probably require coordination between the JavaScript (TC39) and web (W3C/WHATWG) standards, as you’d ideally want a JavaScript-based API for querying Intl-specific emoji data (e.g. to show autocompletions) in addition to the actual picker element. The degree of collaboration between the browser vendors and the standards bodies would have to be fairly involved, so it seems unlikely to happen soon.

Starting from scratch

When I first wrote Pinafore, I didn’t want to deal with this situation at all. I found the whole idea of custom emoji pickers to be absurd – just use the one built into the OS!

When I realized how unreliable the OS-provided emoji pickers were, though, and after considering the need for Mastodon custom emoji, I eventually decided to use emoji-mart, a React component that Mastodon also uses. For various reasons, though, I grew frustrated with emoji-mart (even as I contributed to it), and I mused about how I might build my own.

What would the ideal web-based emoji picker be like? I settled on a few requirements:

  1. Data should be stored in IndexedDB, not in-memory. The Unicode Consortium is never going to stop adding emoji, so at some point keeping all the emoji and their metadata in-memory is going to become unsustainable (or at least, unwieldy).
  2. It should be a custom element. Web components are a thing; using an emoji picker should be as simple as dropping <emoji-picker></emoji-picker> into your HTML.
  3. It should be lightweight. If every website is going to use their own emoji picker, then it should at least have a low JavaScript footprint.
  4. It should be accessible. Accessibility shouldn’t be an afterthought; the emoji picker should work well for screen reader users, keyboard users – everyone.

So against my better judgment, I embarked on the arduous task: building a full emoji picker, the way I thought it should be built. Today it’s available on npm, and you can try out a demo version here. I call it (somewhat presumptuously): emoji-picker-element.

Design

emoji-picker-element follows the vision I set out above. Usage can be as simple as:

<emoji-picker></emoji-picker>

<script type=module>
  import 'https://unpkg.com/emoji-picker-element'
</script>

Under the hood, emoji-picker-element will download the emojibase data, parse it, and store it in IndexedDB. (By default, it fetches from jsdelivr.) The second time the picker loads, it will just do a HEAD request and check the ETag to see if anything has changed.

This is a bonus of using IndexedDB: we can avoid downloading, parsing, and processing the emoji data a second time, because it’s already available on-disk! Following offline-first principles, emoji-picker-element also lazily updates in the background if the data has changed.

 

emoji-picker-element shows only native emoji (no spritesheets), in the categories defined by emojibase. If any emoji are not supported by the OS or browser, then those will be hidden.

Like most emoji pickers, you can also do a text search, set a skin tone, and see a list of frequently-used emoji. I also added support for custom emoji with custom categories, which is an important feature for Mastodon admins who want to add some pizzazz and personality to their instance.

To keep the bundle size small and the runtime performance fast, I’m using Svelte 3 to implement the picker. The tree-shaken Svelte “runtime” is bundled with the component, so consumers don’t have to care what framework I’m using – it “just works.”

Also, I’m using Shadow DOM, which keeps styles nicely encapsulated, while also offering a neat styling API using CSS variables:

emoji-picker {
  --num-columns: 6;
  --emoji-size: 14px;
  --border-color: black;
}

(If you’re wondering why this works, it’s because CSS variables pierce the shadow DOM.)

Evaluating

So how well did I do? The most important consideration in my mind was performance, so here’s how emoji-picker-element stacks up.

Memory usage

This was the most interesting one to me. Was my hypothesis correct, that storing the emoji data in IndexedDB would reduce the memory footprint?

Using the new Chrome measureMemory() API, I compared four pages:

  • A “blank” HTML page
  • The same page with emoji-picker-element loaded
  • The same page with the emojibase “compact” JSON object loaded
  • The same page with the emojibase full JSON object loaded

Here are the results:

Scenario Bytes Relative to blank page
blank 635 kB 0 B
picker 1.38 MB 744 kB
compact 1.43 MB 792 kB
full 1.77 MB 1.13 MB

As you can see, emoji-picker-element takes up less memory than merely loading the “compact” JSON itself! That’s all the HTML, CSS, and JavaScript needed to render the component, and it’s already smaller than the emoji data alone.

Note that the size of a JSON file (in this case, 854kB for the full data and 543kB for the compact data) is not the same as the memory usage when it’s parsed into a JavaScript object. That’s why it’s important to actually parse the JSON to get the true memory usage.

Disk usage

Given that emoji-picker-element moves data out of memory and into IndexedDB, you might also wonder how much space it takes up on disk. There are three major IndexedDB implementations in browsers (Chrome/Chromium, Firefox/Gecko, and Safari/WebKit), so I wrote a script to calculate the IndexedDB disk usage. Here it is:

Browser Disk usage
Chrome 896kB
Firefox 1.26MB
GNOME Web (WebKit) 1.77MB

(Note that these values may be a bit inconsistent – it seems that all browsers do some amount of compacting over time. These are the lowest I saw.)

These numbers are not small, but they make sense given the size of the emoji data, and the fact that the database contains an index for text search. I find it to be a reasonable tradeoff given the memory savings.

Runtime performance

To calculate the load time, I measure from the beginning of the custom element’s constructor to the requestPostAnimationFrame (polyfilled) after rendering the first set of emoji and favorites bar.

On a Nexus 5 (released in 2013) running Chrome on Android 6, median of 5 iterations, I got the following numbers:

Type Duration
First load (uncached) 2982ms
Subsequent load (cached) 259ms

(This was running on a local network – I’m focusing on CPU performance more than network performance.)

Looking at a Chrome trace of first load shows that the Nexus 5 spends about 200ms rendering the initial view (i.e. the “loading” state), 200ms downloading the data on the local intranet, 400ms processing it, 650ms inserting it into IndexedDB, roughly another second of IndexedDB transaction costs, then 250ms rendering the final result:

Annotated screenshot of a Chrome timeline trace showing the durations described above

In the above screenshot, the total render took 2.76s.

On second load, most of those IndexedDB costs are gone, and we merely fetch from IDB and render:

Another Chrome timeline screenshot, this one showing a much shorter duration (~430ms)

In this case, the total render took 434ms.

Interestingly, because it’s IndexedDB, we can actually move the initial data loading to a web worker, which frees up the main thread. In that case, the initial load looks like this:

Screenshot of Chrome timeline showing a similar trace to the "first load", but now the IndexedDB costs are on the worker thread and the main thread is largely idle

The total time in this case was about 2.5s. Clearly, we’ve just shifted the costs from one thread to another (we haven’t made anything faster), but it’s pretty neat that we can avoid jank this way! In the end, only about 300ms of work happens on the main thread.

This points to a nice application-level optimization: assuming the emoji picker is lazy-loaded, the app can proactively spin up a worker thread to populate the database. This would ensure that the emoji data is already ready when the user first opens the picker.

Bundle size

Currently the bundle size of emoji-picker-element is 39.66kB minified, and 12.3kB minified+Brotli. Those are both a bit too large for my taste, but they do compare favorably with other emoji pickers on npm: the smallest one I could find with a similar feature set is interweave-emoji-picker, which is 40.7kB minified.

Of course, the other emoji pickers don’t also include the entire runtime for their framework. The reason I’m bundling Svelte with the component is 1) Svelte’s runtime is quite small already, 2) it’s also tree-shaken to only include what you need, and 3) I’m assuming that most folks out there aren’t using Svelte, since it’s still an up-and-coming framework.

That said, I do have a separate build for those who are already using Svelte 3, and in that case the bundle size is about 11kB smaller (minified). Either way, emoji-picker-element should ideally be lazy-loaded!

Pitfalls and future work

Not everything went swimmingly well with this project, so I have some ideas for where it can improve in the future.

The woes of native emoji

First off, using native emoji is easier said than done. No OS has Emoji v13 yet, and unless you’re on an Apple device, it’s unlikely that you even have Emoji v12. (Only 19% of Android devices are running Android 10+, which has Emoji v12, whereas 75% of iOS devices are running iOS 13.2+, which has Emoji v12.1.) Or your OS may have incomplete emoji (Microsoft’s choice to use two-letter codes for flags is… odd at best). If you’re using Chrome on Ubuntu, you have no native color emoji at all, unless you install a separate package.

GitLab has a great post detailing all the headaches of supporting native emoji. It involves rendering emoji to canvas and checking the rendered color, as well as handling edge cases such as ligatures that may appear as “double emoji” when rendered improperly. (E.g. “person with red hair” may appear as a person with a floating wig of red hair next to them.) In short, it’s a mess.

"Person" emoji on the left and "red hair" emoji on the right

“Person with red hair” when rendered incorrectly.

However, my goal with this project is to skate to where the puck is going, not where it is. My hope is that browsers and OSes will get their acts together, and start to broadly support native color emoji without hacks or workarounds. (If they don’t, more websites than just emoji-picker-element will appear broken! So they have every incentive to do this.)

Plus, I don’t have to jump through as many hoops as GitLab, because I’m not concerned about supporting every emoji under the sun. (If I were, I would just use a spritesheet!) So my current strategy is a sniff test – check a representative emoji from each emoji version, and hide the entire set if one emoji is unsupported. This isn’t perfect, but I believe it’s good enough for most use cases. And it should improve over time as native emoji support improves.

Shadow DOM

This could be a blog post in and of itself, but shadow DOM is a double-edged sword. On the one hand, I love how easy it is to encapsulate CSS, and to offer a styling API with CSS variables. On the other hand, I found that any libraries that manage focus – e.g. focus-visible, a11y-dialog, or my own arrow-key-navigation – had to be tweaked to account for shadow DOM. Otherwise, the focus behavior didn’t work properly.

I won’t go into the nitty-gritty details (see my Mastodon thread for that). Long story short: none of these libraries worked with shadow DOM, except for focus-visible, which required some manual work. So for a11y-dialog I forked it, and for arrow-key-navigation I had to implement shadow DOM support. Kind of annoying, when all I really wanted was style encapsulation!

But again: I’m trying to skate to where the puck is going. My hope is that eventually browsers will iron out problems with the shadow DOM API, and make it easier for libraries to implement focus traps, custom focus hotkeys, etc.

Conclusion

Building emoji-picker-element was not easy. Properly rendering native emoji required arcane knowledge of emoji support on various platforms and browsers, as well as how to detect it. Add in skin tone variants (which may or may not be supported, even if the base emoji is supported), and it can get very complicated very fast.

I’m deeply indebted to the work of those who came before me: emojibase for providing the emoji data, Emojipedia for being an inexhaustible resource on the subject, and if-emoji and GitLab for demystifying how to detect emoji support.

Of course, I’m also grateful for all the existing emoji pickers that I drew inspiration from – it’s nice not to have to reinvent the wheel! Most emoji pickers have converged on a strikingly similar design, so I didn’t feel the need to be particularly creative with my own design. (Although I do think some of my CSS animations are nice.)

I still believe that the emoji picker is probably something browsers should handle themselves, to avoid the inevitable bloat of every web page loading their own picker. But for the time being, there should at least be a web-based emoji picker that’s as lightweight and performant as possible. emoji-picker-element is my best attempt at that vision.

You can discuss this post on the fediverse and on Lobste.rs. Here is the code and the demo.

Linux on the desktop as a web developer

I’ve been using Ubuntu as my primary home desktop OS for the past year and a half, so I thought it would be a good time to write up my experiences. Hopefully this will be interesting to other web developers who are currently using Mac or Windows and may be Linux-curious.

Photo of a Dell laptop with an Ubuntu background and a monitor, keyboard, and mouse

My basic setup. Dell XPS 13, Kensington trackball mouse (yes I’m a weirdo who likes trackballs), Apple magic keyboard (I still prefer the feel), and a BenQ monitor (because I play some games where display lag matters)

Note: in this post, I’m mostly going to be talking about Ubuntu. I’ve played with other Linux distros, but I stick with Ubuntu because if I have a problem, I can Google it and find an answer 99.9% of the time.

Some history

I first switched to Linux in 2007, when I was at university. At the time I perceived it to be a huge step-up over Windows Vista (so much faster! and better for programmers!), but it also came with plenty of headaches:

  • WiFi didn’t work out of the box. I had to use ndiswrapper to wrap Windows drivers.
  • Multi-monitor and presentations were terrible. Every time I used xrandr I knew I would suffer.
  • Poor support for a lot of consumer applications. I recall running Netflix on Firefox in Wine because this was the only way to get it to work.

Around 2012 I switched to Mac – mostly because I noticed that every web developer giving a conference talk was using one. Then I became a dual Windows/Mac user when I joined Microsoft in 2016, and I didn’t consider Linux again until after I left Microsoft in 2018.

I’m happy to say that none of my old Linux headaches exist anymore in 2020. On my Dell XPS 13 (which comes with Ubuntu preinstalled), WiFi and multi-monitor work out-of-the-box. And since it seems everything is either an Electron app or a website these days, it’s rare to find a consumer app that doesn’t support Linux. (At least, the ones I care about; I’m sure you can find a counter-example!) The biggest gripe I have nowadays is with fonts, which is a far cry from fiddling with WiFi drivers.

OK so enough history, let’s talk about the good and the bad about Linux in 2020, from a web developer’s perspective.

The command line

I tend to live and breathe on the command line, and for me the command line on Linux is second-to-none.

The main reason should be clear: if you’re writing code that’s going to run on a server somewhere, that server is probably going to run Linux. Even if you’re not doing much sysadmin stuff, you’re probably using Linux to run your test and CI infrastructure. So eventually your code is going to have to run on Linux.

Using Linux as your desktop machine just makes things that much simpler. All my servers run Ubuntu, as do my Travis CI tests, as does my desktop. I know that my shell scripts will run exactly the same on all these environments. If you’ve ever run into headaches with subtle differences between the Mac and Linux shell (e.g. incompatible versions of grep, tar, and sed with slightly different flags, so you have to brew install coreutils and use ggrep and gtar… ugh), then you know what I’m talking about.

If you’re a Mac user, the hardest part about switching to the Linux terminal is probably just going to be the iTerm keyboard shortcuts you’ve memorized to open tabs and panes. I found the simplest solution was to use tmux instead. As an added bonus, tmux also runs on Mac, so if you learn the keyboard shortcuts once, you can use them everywhere. I set my terminal to automatically open tmux on startup.

Screenshot of tmux running in Ubuntu with a few panes open

Ah, the command line on Linux. Feels like home.

Since the command line is the main selling point of Linux (IMO), it’s tempting to just use Windows with the Windows Subsystem for Linux instead. This is definitely a viable option, and totally reasonable, especially if there’s that one Windows program you really need (or you’re a PC gamer). For me, though, I don’t do much PC gaming, and my experience with WSL is that although compatibility was excellent, the performance was poor. npm install would take orders of magnitude more time on WSL compared to the equivalent Mac or Linux machine. (Keep in mind I was using WSL back in 2016-2018 though, and I’m told it’s improved since then.)

Still, for me, I just don’t find Windows to my taste. The UI has always felt slow and clunky to me, which may just be my perception, although when I read blog posts like this one from Bruce Dawson I feel a bit vindicated. (Right-clicking taskbar icons is slow! Why?) In any case, Ubuntu starts up fast, the system updates are quick and unobtrusive, and it’s not missing any must-have apps for me. So I run 100% Ubuntu, no dual-boot even.

Web development

For the average web developer, most of the stuff you need is going to work just fine on Linux. You can run Chrome and VS Code (or WebStorm, my preference), and all your command-line utilities like node and npm will work the same. You can use nvm to manage Node versions. So far, so good.

As a web developer, the biggest issue I’ve run into is not having a quick way to test all three major browser engines – Blink (Chrome), Gecko (Firefox), and WebKit (Safari). Especially now that Edge has gone Chromium and the Trident/EdgeHTML lineage is slowly dying out, it’s really attractive that, with a Mac, you can test all three major browser engines without having to switch to another machine or use a tool like BrowserStack.

On Linux of course we have Chrome and Firefox, and those run mostly the same as they do on a Mac, so they fit the bill just fine. For WebKit we even have GNOME Web (aka Epiphany Browser), but I only consider it “okay” as a stand-in for Safari. It doesn’t support some of the Safari-specific APIs (e.g. backdrop filter, Apple Pay, etc.), and it’s terribly slow, but it’s good for a quick gut-check to see if some bit of code will run well on Safari or not.

Screenshot of Gnome Web on HTML5Test showing a score of 432

GNOME Web on HTML5Test. Safari it is not.

Unfortunately for me, I don’t consider that “good enough,” especially since the vast majority of Safari users are on iOS, so that’s the platform you really want to test. And here is where Linux runs into its biggest drawback from a web developer’s perspective: debugging iOS Safari.

If you want to debug Chrome or Firefox on Android – no problem. adb runs just fine on Linux, you can run chrome:inspect on Chrome or Remote Debugging on Firefox, and it all works great. For iOS Safari, though, the best option you have is remotedebug-ios-webkit-adapter, which uses ios-webkit-debug-proxy under the hood.

Essentially this is an elaborate suite of tools that makes iOS Safari kinda-sorta look like Chrome, so that you can use the Chrome DevTools to debug it. The most amazing thing about it is… it actually works! As long as you can get the odd dependencies running correctly, you’ll have your familiar Chrome DevTools attached to an iOS device.

Screenshot of debugging example.com in iOS Safari in Chrome DevTools on Linux, showing some iOS-specific APIs in the console

Believe it or not, you can actually debug iOS Safari from Linux.

If you have a spare iPhone or iPod Touch laying around, this is not a bad option. But it’s still a far cry from the streamlined experience on a Mac, where you can quickly run an iOS Simulator with any iOS version of your choice, and where Safari debugging works out-of-the-box.

For accessibility testing, it’s a similar story. Of course Firefox and Chrome have accessibility tools, but they’re no substitute for VoiceOver on Mac or NVDA on Windows. Linux does have the Orca screen reader, but I don’t see much point in testing it, since it’s not representative of actual screen reader usage. Especially given that screen readers may have bugs or quirks, I prefer testing the real deal. So I keep a Mac Mini and cheap Windows desktop around for this reason.

Conclusion

So in short, using Linux as your desktop environment if you’re a web developer is pretty great. You probably won’t miss much, as soon as you rewire your brain to get the keyboard shortcuts right.

I find that the main things I miss these days are some of Apple’s best built-in apps, such as Preview or Garage Band. I love Preview for taking a quick screenshot and drawing arrows and boxes on it (something I do surprisingly often), and I haven’t found any good substitutes on Linux. (I use Pinta, which is okay.) Other apps like ImageOptim also have no Linux version.

So if you depend on some Mac-only apps, or if you need best-in-class Safari and iOS debugging, then I wouldn’t recommend Linux over Mac. If your main focus is accessibility, it also might not be sufficient for you (although something like Assistiv Labs may change this calculus). But for everything else, it’s a great desktop OS for web development.

Thanks to Ben Frain for asking about my Linux experiences and inspiring this blog post.