Living with an open-source phone

A few months ago, I decided to make a radical change to my smartphone lifestyle. I wanted a phone that would limit the amount of personal data I was broadcasting to third parties – namely Apple or Google – and decided to run a more-or-less vanilla version of Android, without any Google Play Services (i.e. no Google Maps, GMail, Google Play, or Google Search apps).

I’ve been living with this setup for a few months, and to my surprise, it’s really not that bad. It doesn’t feel like a return to the Stone Age, nor does it feel like I’ve sacrificed all the niceties of a smartphone to the point where I’m carrying around a glorified flip phone.

However, it’s a bit non-obvious how to get all of this stuff to work, and especially how to get by with a Android phone that doesn’t have the Google Play Store. So in this post, I’d like to talk a bit about my smartphone setup, in the hope that it might help someone who’s looking to make a similar switch.

Choosing an Android ROM

There are various flavors of Android out there: you can choose LineageOS (the spiritual successor to CyanogenMod), Paranoid Android, CopperheadOS, or you can even just build AOSP (the Android Open Source Project) from source.

After fumbling around with building AOSP from source (which was much more difficult than I thought – lots of requirements for the build environment, CPU horsepower, and storage), I also tried LineageOS and CopperheadOS, and eventually settled on LineageOS.

Why LineageOS? Well, mostly because it was easy, it offered fast security updates, and I like the minimalist interface and built-in apps. LineageOS was also familiar to me, as I had previously used Cyanogen for many years. (I forgot how much I enjoyed the small creature comforts, such as long-pressing the volume up/down keys to skip tracks!)

Screenshot of default homescreen on LineageOS

Default homescreen on LineageOS

I also needed a recovery image in order to install the ROM, and it seems that TWRP has become the de-facto standard these days, replacing the venerated ClockworkMod.

As it turns out, none of these options will result in a 100% open-source phone, as you’ll still be running the vendor binaries for your particular phone. But this is about as close as you can get to a smartphone that runs only free/libre open-source software.

Choosing a phone and installing a ROM

This isn’t going to be a guide to unlocking or rooting your Android phone. The process is different for every phone, and it would take too long to describe all the various steps. Suffice it so say that it’s still a painful process, and your choice of phone can either make it easier or harder.

I’ve been rooting, unlocking, and tinkering with Android phones for a long time, since the days of the HTC Dream and HTC Magic around 2009/2010. Since then, I’ve worked as an Android and mobile web developer, and I’ve become very comfortable with tools like the Android SDK, adb, fastboot, and Android Studio. And yet, to this day I still find installing custom ROMs to be a frustrating and time-consuming experience. I wish it were easier for casual folks to do this stuff.

The Nexus line of phones (now Pixel) have always been the easiest to customize, and for many years I stuck with the tried-and-true Nexus 5, which is still a surprisingly capable phone despite having been released in 2013. (I swear it ran faster than my 5X, which also kicked the bucket after only a year or two of use.) However, my aging Nexus 5 just couldn’t cut the mustard anymore due to hardware issues (the USB port was too loose; it had become tricky to charge it), and so I decided to buy a new phone instead.

I settled on the Samsung Galaxy S5, mostly because I could get it for cheap (<$200 on Amazon) and with full support for my carrier (T-Mobile). In retrospect, choosing a non-Nexus device made my life a lot harder, and after several hours of research on unlocking Samsung phones (including building Heimdall from source on a Windows machine, because the prebuilt binaries were out of date but the software was too old to build on a Mac), I finally had my LineageOS phone up and running.

The second thing you'll want to do is ensure that your device is encrypted, which you can enable in the security settings. Unfortunately this ended up making my phone unable to start the OS, but after booting into recovery and doing a factory reset, I had both encryption and the OS up and running just fine.

Screenshot of my LineageOS home screen

My LineageOS home screen

Getting apps

Without the Google Play Store, you'll have to use F-Droid, which has the added benefit of only hosting free and open-source apps. In fact, a couple of my own apps were previously on there (Catlog and KeepScore), and as far as I can tell, they were built directly from source. (I’m not sure why they were removed; possibly because I stopped maintaining them.)

To get F-Droid, you just download it directly from f-droid.org. You'll also have to allow installation from "unknown sources" in the security settings.

For the odd app that isn't available on F-Droid, you can also use the Yalp Store, which can either use your Google account or an ephemeral account to download apps from the Play Store. It’s not clear to me whether or not this violates the Google Play Terms of Service, though, so proceed with caution.

Messaging

I use Signal as my default SMS app and for most messaging with family and friends. Unfortunately it isn't available on F-Droid, but you can download it directly from the Signal website, or use the Yalp Store technique above.

Update: if you’re wondering how to verify the SHA256 fingerprint of the APK downloaded from the Signal website, here’s an example.

Like most Android apps, Signal normally uses Firebase Cloud Messaging (FCM, formerly GCM) to send push notifications. If you don't have Google Play Services installed, Signal will offer to switch to a less battery-efficient background polling mechanism.

I've used this mechanism for months and only had occasional problems where a large backlog of old messages was suddenly delivered all at once. It also hasn't had a visible impact on battery life. (I usually end the day with half a charge left; the GS5 has a great battery!)

Web browser

For a web browser, I mainly use Firefox via the FFUpdater tool, which is clunky but gets the job done. I’ve set DuckDuckGo as my default search engine, and I have uBlock Origin installed as an add-on, which undoubtedly makes my browsing faster and easier on the battery.

Occasionally though, I do find sites that don’t run so will with mobile Firefox, and for that I use Auto Updater for Chromium, which automatically installs Chromium and updates it. This is a bit nicer than FFUpdater because it does its work silently in the background, rather than requiring you to manually check for updates.

The built-in browser doesn’t support “Add to Homescreen,” so I find it fairly useless.

Maps

I use OsmAnd~, which is nice because it allows you to download maps in advance for offline use. Unfortunately it’s still nowhere near as feature-complete as Google Maps, so if you’re looking for something to help you navigate your car, you may be out of luck. (Update: actually, OsmAnd~ supports turn-by-turn navigation.)

I also frequently use maps.google.com in Firefox, which works surprisingly well. Besides some minor performance issues, it’s fairly indistinguishable from the native app for basic directions, bus timetables, and store lookups.

Screenshot of Google Maps running in Firefox

Google Maps running in Firefox

Living in Seattle, one of my favorite apps is also OneBusAway, which provides up-to-date arrival times for Seattle buses. Unfortunately this app requires Google Play Services in order to work, so I’ve had to do without.

Weather

Speaking of Seattle, it’s also important for me to know how much it’s going to rain in a given day. (Spoiler alert: a lot.)

This happens to be one of those things about LineageOS that’s a bit non-intuitive. To get weather to work, you have to go to the “extras” page and download a weather provider. (In my case, I’m using OpenWeatherMap.)

Unfortunately, though, even after installing the weather provider, I couldn’t get the built-in “cLock” widget to show the weather. (It keeps saying “network geolocation is disabled,” even though it’s not.) So I ended up installing Forecast widgets, which gets its data from the National Weather Service rather than the built-in weather provider.

Maybe this is just some bug with my installation or with the hardware itself, but in any case I’m satisfied with this workaround. The Forecast widget looks and works fine.

Screenshot of weather and time widget on my homescreen

Weather and time widget on my homescreen

Keyboard

The only non-standard features I really want from a keyboard are 1) emoji support, and 2) swipe input.

Unfortunately I couldn’t find any open-source keyboard that can do both of these things, and the AOSP keyboard doesn’t seem to support either. The closest I found was AnySoftKeyboard, which at least has an emoji screen. It doesn’t allow you to search for emoji, though, which is a bit frustrating.

Screenshot of sending an emoji via AnySoftKeyboard and Signal

Sending an emoji via AnySoftKeyboard and Signal

Overall, the keyboard experience has been my least favorite part of the LineageOS experience. I didn’t realize how much I had become accustomed to swipe input until I had to revert back to tapping, which feels to me like hunt-and-peck. Input is extremely laborious and slow, although on the plus side I’m spending less time texting, so maybe that’s a good thing.

Update: actually, AnySoftKeyboard does support searching emoji! You just need to type : and then the search term.

Music and podcasting

I’m a fairly old-school music consumer. I prefer to listen to albums from start to finish, and I never really got into services like Pandora or Spotify. Instead, I buy my MP3s from Amazon (old habit, my entire library is there) and then sync them from my desktop computer to my phone using adb-sync.

For browsing and listening to my music library, I’ve found Vanilla Music to be fantastic. It has a cover art downloader, the interface is minimal and clean, and it can play an album from start to finish, which is all I really ask.

Screenshot of Vanilla Music with some of my albums

Vanilla Music with some of my albums

Oh, and of course Vanilla Music sets the cover art on the lock screen. It’s the little things that count.

I’m also something of a podcast addict, so I use AntennaPod to download and listen to podcasts. I’ve tried several podcast apps on F-Droid, and I found this one to be the easiest and most reliable overall. I particularly like that it allows me to search on iTunes, since some podcasts can be hard to find elsewhere.

Social media

I stopped using Twitter, but if you absolutely must, their mobile website is not bad in either Firefox or Chromium. I’m told Facebook’s mobile website is also pretty functional.

I spend most of my social media time these days on Mastodon, and as it turns out the mobile websites work perfectly fine in both Firefox and Chrome, so you can just pin it to your homescreen. There’s also Tusky if you prefer the native app experience.

Ridesharing

I don’t use Uber, but Lyft has a mobile webapp that works just as well as their native app. So when I need a ride, I just open ride.lyft.com in Firefox. It’s a PWA, and as far as I can tell it works just as well as the native Android app.

Somewhat bafflingly, though, if you go to lyft.com directly, the site may try to route you to their driver portal or to the app store instead.

Screenshot of Lyft's website offering "Lyft in Firefox,"

Lyft’s website may offer “Lyft in Firefox,” which is the PWA

In case you get confused, you may need to look for the “Lyft in Firefox” link. To me, this seems like a bizarre way of saying “Lyft for the web,” but you do you, Lyft.

Email, calendar, and contacts

I switched to FastMail recently for my primary email account, and I’ve been really happy with the service. It’s fast, it’s simple, and it has a clear business model where I pay them 5 bucks a month and they host my email. Simple.

To get my email on my phone, I use K-9 Mail, as I’ve found it a bit more reliable and feature-rich than the built-in Email app. If you prefer a simpler interface, though, the built-in app works fine too.

FastMail also supports calendar and contact sync, and this is where it gets a bit trickier. If you’ve ever connected an email client to your email server by manually typing the IMAP and SMTP settings (which, incidentally, you’ll have to do for K-9 email), then it’ll feel pretty familiar. FastMail supports CalDav and CardDav, and so to hook these up to my phone I used DavDroid.

Honestly this was probably the most tedious process of setting up my LineageOS phone. It involved creating app passwords for each client (I use separate passwords for email, calendar, and contacts, in the interests of security), and manually typing in the server names and ports for the various FastMail endpoints. This was a long and error-prone process, but in the end I do have full email, calendar, and contact sync, so I can’t complain.

Passwords and two-factor auth

For nearly a decade I’ve stored my passwords using the Joel Spolsky method, which is to use PasswordGorilla on desktop and PasswdSafe on Android. I like this method because it’s simple, it works cross-platform, and I maintain control of the password file.

For two-factor authentication (other than basic SMS), there’s FreeOTP, which essentially takes the place of Google Authenticator.

The web works for everyone

One of my main reasons for switching to a non-Google Android phone was to see how capable the web is as an application delivery system. And aside from the native apps listed above and some minor utilities (e.g. a barcode scanner and a notes widget), I do pretty much everything in a web browser.

Media sources like Hacker News, Ars Technica, and others don’t need an app – you can just pin a website to your home screen. (Although for Hacker News, I use the excellent hn.premii.com.) The main exception for me is NewsBlur, which I access via the open-source client app.

For videos, both YouTube and Vimeo also work great as mobile webapps. In Firefox, they can run both in fullscreen mode and in the background.

For file sync, I use Dropbox’s mobile webapp for quick file downloads from my Dropbox account and Firefox Send for sending more ephemeral files. I need a better solution to backing up photos, though; for now I’m just using adb-sync to sync to my Dropbox folder on the desktop.

I also do work at Microsoft, and although I have a personal policy of not syncing my work email to my phone, I can make a temporary exception by loading outlook.office.com in Firefox, which works surprisingly well. For those odd moments when I need to send an email or check the location of a meeting, it gets the job done.

As a guitarist, I also needed a way to tune my guitar, and sadly the excellent g-strings is both paid and closed source, so it was a no-go. So I use Paul Lewis’ guitar tuning webapp instead, which is a good substitute.

Screenshot of guitar-tuner.appspot.com

Me singing a bit off-key into a guitar tuner

Turns out the web of 2017 is capable of quite a bit, from hailing a taxi to tuning your guitar!

Conclusion

I hope I’ve demonstrated in this blog post that it’s not only possible to use an open-source phone without any of the pervasive tracking, spyware, or bloatware that we’ve come to expect from most smartphones, but it’s actually quite viable and even enjoyable.

When I booted up my Galaxy S5 for the first time, I was immediately greeted with a barrage of ads for Samsung and T-Mobile services and upsells, all in the 5 minutes it took to turn on USB debugging so I could install custom software and mercifully silence all of the nagware. My clean and stripped-down LineageOS setup is about as far from that horrendous out-of-the-box experience as you can get.

Note though, that this is the setup that works for me, and your mileage may vary depending on how much you rely on various software and services. If you use GMail and GDocs, you may be better off just using a Google-flavored Android phone. And if you’re bought into the iTunes and iCloud ecosystem, it may be more trouble than it’s worth to switch from an iPhone.

Also, despite my attempts to de-mystify some of the less obvious parts of this setup, I’m in no way claiming that any of this is accessible to the non-geek, non-hacker crowd. The mere process of installing LineageOS is so far beyond the capabilities of the average non-techie that it actually fills me with a kind of despair.

This is probably a subject for an entire post, but I’m becoming concerned that my friends and I in the open-source, pro-privacy hacker scene are building a two-tiered world, where the tech elite can successfully install and configure software that maintains their security and privacy, whereas the average person is stuck either paying a premium for a privacy-respecting but closed-source iPhone, or paying a reasonable price for an Android phone where their security and privacy are far from assured.

That said, maybe if more of us inhabit this (admittedly neophyte-unfriendly) world, then maybe we can work to make it more accessible to those of us who don’t know how to use a command line and have no patience to type out IMAP server URLs into their smartphone. Let’s hope so.

Why I’m deleting my Twitter account

When I first got on the Internet back in the 90’s, it felt like a cool underground rock concert. Later on, it seemed like a vast public library, maybe with a nice skate park nearby. Today it feels more like a shopping mall. The transition happened so gradually that I barely noticed it.

Hanging out with your friends at the mall can be fun. But it can also be tiring. You’re constantly surrounded by ads, cheery salespeople are trying to get you to buy stuff, and whatever you eat in the food court is probably not great for your health.

For the past few years, I’ve subsisted on a media diet that mostly came from Twitter, consisting of “snackable” news articles with catchy headlines, shareable content with wide appeal (baby koala cuddles baby cat, how cute!), and righteous outrage at whatever horrible political thing was happening that day.

Twitter was often the first thing I looked at when I picked up my phone in the morning, and the last thing I browsed late into the night, endlessly flicking my thumb over the feed in the hope that something good would pop up. The light of the smartphone was often the only thing illuminating my bedroom before I finally turned in (always much too late).

All of this content – cat pictures, articles, memes, political hysteria – came streaming into my eyeballs in a rapid and seemingly random order, forcing my brain to make sense of the noise, to find patterns in the data. It’s addictive.

But the passivity of it, and the endless searching for something good to watch, meant that for me Twitter had essentially become television. Browsing Twitter was no more edifying than flipping through channels. At the end of a long, multi-hour session of Twitter-surfing, I could barely recall a single thing I had read.

Social media as public performance

Twitter is unlike television in a few crucial aspects, though. First off, the content is algorithmically selected, so whatever I’m seeing is whatever Twitter has determined to be most likely to keep my eyes on the screen. It’s less like I’m surfing through channels and more like the TV is automatically flipping from channel to channel, reading my eye movement and facial expressions to decide what to show next.

Second, Twitter has become an inescapable part of my professional life. My eight thousand-odd Twitter followers are a badge of honor, the social proof that I am an important person in my field and worthy of admiration and attention. It also serves as a measure of my noteworthiness in comparison to others. If someone has more followers than me, then they’re clearly more important than I am, and if they have less, well then maybe they’re an up-and-comer, but they’re certainly not there yet.

(This last statement may sound crass. But any avid Twitter user who hasn’t sized someone up by their follower count is either lying to themselves, or is somehow immune to the deep social instincts that mark us as primates.)

For the kinds of professionals who go to conferences, give public talks, and write blog posts, Twitter serves as a sort of “Who’s Who,” except that everyone is ranked by a single number that gives you a broad notion of their influence and prominence.

I’m sure many of my friends from the conference and meetup scene will look at my announcement of deleting my Twitter account as a kind of career suicide. Clearly Nolan’s lost his mind. He’ll never get invited to a conference again, or at the very least he won’t be given top billing. (Conference websites usually list their speakers in descending order of Twitter followers. How else can you tell if a speaker is worth listening to, if you don’t know their follower count?)

Much of that is probably true. I used to get a lot of conference invites via Twitter DMs, and those definitely won’t be rolling in anymore. Also, anyone who wants to judge my influence by a single number is going to have a hard time: they’ll have to piece it together from blog posts and search results instead. Furthermore, my actual influence will be substantially reduced, as most of the hits to my blog currently come from Twitter.

Why I’m done with Twitter

Thing is, I just don’t care anymore. I’ve spent years pouring my intellectual and emotional labor into Twitter, and for countless reasons ranging from harassment to Nazis to user-hostile UI, platform, and algorithm choices, they’ve demonstrated that they don’t deserve it. I don’t want to add value to their platform anymore.

To me, the fact that Twitter is so deeply embedded into so many people’s professional lives is less a reason to resign myself to keep using it, and more a reason to question and resist its dominance. No single company should have the power to make or break someone’s career.

Twitter has turned a wide variety of public and quasi-public figures – from Taylor Swift to a dude who speaks at tech conferences – into brand ambassadors for Twitter, and that ought to worry us. Despite what it claims, Twitter is not a neutral platform. It’s an advertising company with a very specific set of values, which it expresses both in how it optimizes for its core constituents (advertisers) and how it implements its moderation policies (poorly).

Well, it may indeed be career suicide for Taylor Swift to abandon her Twitter account, but for a (very) minor public figure like myself, it’s a small sacrifice to make to knock Twitter down a peg. My career will survive, and my mental health can only improve by spending less time flicking a smartphone screen into the late hours of the night.

That’s why I’m deleting my account rather than just signing out. I want my old tweets to disappear from threaded conversations, from embeds in blog posts – anything that’s served from twitter.com. I want to punch a hole in Twitter’s edifice, even if it’s a small one.

I’ve backed up my tweets so that anyone who wants to see them still can. I’m also still fairly active on Mastodon, and as always, folks can follow me via my blog’s RSS feed or contact me via email.

This isn’t me saying goodbye to the Internet – this is me saying goodbye to the shopping mall. But you can still find me at the rock concert, in the public library, and in the park.

What is Mastodon and why is it better than Twitter

Mastodon is a Twitter alternative that recently released version 2.0 and has been steadily growing over the past year. It’s also a project that I’ve been dedicating an inordinate amount of my time to since last April – helping write the software, running my own instance, and also just hanging out. So I’d like to write a bit about why I think it’s a more humane and ethical social media platform than Twitter.

Much of the discussion around Mastodon centers on the fact that the flagship instance explicitly bans Nazis. This is true, and it remains a great selling point for Mastodon, but it also kind of misses the point. Mastodon isn’t a single website run by a single company with a single moderation policy. It’s a piece of open-source software that anybody can use, which in practice means it’s a network of independent websites that can run things however they like.

There is no company behind Mastodon. There’s no “Mastodon, Inc.” Mastodon doesn’t have a CEO. The code is largely written by a 24-year old German dude who lives off Patreon donations, even though he’s a very talented web developer and could probably make a lot more money if he joined the industry. He works on Mastodon because it’s his passion.

What this means is that if someone wanted to take Mastodon’s code and build a competing service, they could do so trivially in a matter of minutes. And they do. The original instance, mastodon.social, isn’t the only server – in fact, it’s not even the biggest one anymore. There are over a thousand active instances, and it’s become easy enough that Masto.host can even create one at the click of a button.

In practice, though, these Mastodon instances don’t compete with each other so much as they form a giant constellation of interconnected communities. Users from any server can read, follow, and reply to users on another server, assuming neither of the two servers is blocking the other.

The closest analogy is email: if you use Gmail, you can still communicate with someone who uses Outlook.com and vice-versa, because they both rely on the same underlying system (email). Through its own underlying systems, Mastodon (as well as compatible software like Friendica, GNU Social, and postActiv) forms a network of independent sites referred to as the “fediverse,” or federation of servers.

Why this is better than Twitter

The problem with Twitter is that its incentives are completely misaligned with those of its users. Twitter makes its money from advertising, which means that its goal is to keep your eyes glued to the screen for as long as possible, and to convince you to interact with ads. Its goal is not to keep you safe from harassment, or to ban dangerous extremists, or to ensure your psychological well-being. Its goal is to make advertisers money by selling them an engaged audience.

This is why Twitter will never #BanTrump, even though many have called for it after he began threatening North Korea on the platform. From Twitter’s perspective, Donald Trump increases engagement. Donald Trump gets eyeballs. If Donald Trump started a nuclear war on Twitter then hey, all the better, because Twitter would get a massive boost in traffic, at least right up until the point the bombs started raining down. Twitter even uses Trump in some of its advertising, which gives you an idea of how they feel about him.

Mastodon, by contrast, isn’t run on advertising. Well, instances could add advertising if they wanted to, but I’m not aware of any that do. Most of them, including the flagship, are run on donations from their users. Others get a bit more creative: cybre.space, for instance, allows free signups for one hour each day, but if you donate you can get an instant invite. capitalism.party is an interesting experiment where every signup costs $5. social.coop is run as a co-op. The possibilities are endless, since the underlying code is open-source.

What these instances all have in common is that they’re not driven by the insatiable appetite of marketers for clicks and engagement – instead, their goal is to make as warm and hospitable a place for their users as possible. The incentives of the people who run the platform are aligned with the incentives of the users.

Ultimately, this is why Mastodon instances can implement the kinds of moderation policies that their users clamor for (including banning Nazis). Most instances only have a few dozen to a few thousand active users, and they’re often organized based on shared interests, languages, or nationalities. This means that each instance tends to be small enough and like-minded enough that they can have fairly nitpicky moderation policies (or policies that adapt to local laws and customs), and it’s not too overwhelming for a small group of sympathetic and highly-motivated admins to handle.

Privacy and respect for the user

There are a lot of other benefits to Mastodon’s lack of an advertising model. For one, as a Mastodon user you’re not subjecting yourself to the adware, spyware, and bloatware that we’ve come to expect from much of the modern web. To see what I mean, here’s a screenshot of my instance, toot.cafe, compared to Twitter.com.

creenshots of Twitter vs Mastodon, showing Twitter loading 3.48MB of JS vs 990.84KB on toot.cafe

Besides the refreshing lack of advertising on the Mastodon site (and toot.cafe’s charming purple theme), you might observe that Mastodon is loading less than a meg of JavaScript, whereas Twitter loads a generous 3.5MB. A lot of that extra heft is probably just standard web bloat, but if you have an ad blocker or tracker blocker installed, then you can see another dimension to the story.

Screenshot of Ghostery showing 4 trackers blocked on Twitter.com vs 0 for toot.cafe

According to Ghostery, Twitter.com is loading 4 separate trackers, including Google Analytics, TellApart, Twitter Analytics, and Twitter Syndication. (Those last 3 are all owned by Twitter, so who knows why they need 3 separate trackers for each.) Whereas on the Mastodon site, Ghostery found 0 total trackers.

Screenshot of uBlock origin showing 14 requests blocked for Twitter vs 0 for toot.cafe

Looking at uBlock Origin, we can see it needed to block 14 requests on Twitter.com, or 9% of the total. On the Mastodon site, though, uBlock didn’t need to block anything.

Beyond the lack of ads and trackers, though, these privacy benefits accrue to the data you share with the website itself. On Twitter, you’re handing over your tweets, browsing habits, and photo metadata to a large VC-funded company that makes no bones in its privacy policy about all the various ways it feels entitled to use and sell that data. The terms of service also make it clear that once you post something, Twitter can do whatever it wants with it.

Screenshot of https://twitter.com/en/tos#usContent starting from "By submitting, posting or displaying Content..."

A snippet of Twitter’s terms of service.

Now compare this to Mastodon. On Mastodon, image metadata is stripped by default, links show up as (wait for it) actual links instead of tracking redirects, and some instances even go so far as to specify in their terms of service that you’re not relinquishing any copyright over your content and your data will never be sold.

Screenshot of mastodon.art's guidelines, saying "All content is ⓒ each artist & cannot be distributed or used without prior permission by the respective Mastodon.ART artist."

A snippet of mastodon.art‘s terms of service.

It’s such a far cry from the way we’re used to being treated by online services, with their massive legalese-laden EULAs stripping us of the right to do anything beyond gripe at the rough way we’re being manhandled, that using Mastodon can almost feel like browsing a web from a parallel universe.

So Mastodon is a paradise, right?

I’m not going to pretend that Mastodon is devoid of moderation problems. Yes, the flagship instance bans Nazis and other malcontents, as do most of the other large instances (including my own). There are plenty of instances with their own policies, though, and there’s nothing in the software to prevent them from doing so. So if you want to use an instance that harbors Nazis, or even just libertarians or free-speech advocates, then you can certainly find them.

As you can imagine, though, a right-wing instance that brags about its tolerance toward fascists is not likely to get along with a left-wing instance that bills itself as “anticapitalist”. Thus you will find lots of instances that block each other, creating a situation where you might discover vastly different content and vastly different people depending on which instance you sign up with.

This goes beyond straightforward disagreements between the political left and right. Every so often in the Mastodon community, a serious conflict will arise between instances. Often it starts because two users on two different instances got into a fight with each other, the admins got involved, and they disagreed on how to resolve the dispute. Sometimes it’s the admins themselves who started the fight. Either way, the admins end up criticizing or disavowing each other, the public timeline gets filled with debates on who’s right or wrong, and ultimately one group of instances may decide to block or silence another group. We call this “the discourse.”

“The discourse” tends to flare up every month or so, and when it does there’s usually a lot of moaning about how much drama there is on the fediverse. This lasts for a day or two and then things go back to normal, albeit with a slightly more bifurcated community than we started with.

Discourse and disintegration

I don’t enjoy “the discourse,” and I tend to agree with folks who argue that it could be alleviated if Mastodon had better tools for resolving inter-admin conflicts. I don’t think this problem can ever be completely eliminated, though. Human beings are just naturally inclined to seek the company of those they agree with and shun those they disagree with. This has the unfortunate effect of creating filter bubbles, but it turns out human beings also have a boundless appetite for filter bubbles, as evidenced by the churches, clubs, meetups, and political parties where we seek those who are similar to us and give a cold shoulder to outsiders.

I don’t believe it’s Mastodon’s job to correct the problems caused by the right to free association. But Mastodon could improve the process of communities splitting into smaller, more harmonious networks of people with shared values and mutual tolerance for one another.

Furthermore, a lot of these disputes boil down to a difference of opinion over what constitutes harassment, abuse, hate speech, etc. So in a way, “the discourse” can be seen as a testament to the seriousness with which these subjects are treated on Mastodon. Instance admins care so much about the well-being of their users and protecting them from disturbing content, that they routinely argue and even block each other over the best way to implement it.

Now compare that situation to Twitter. On Twitter, there’s one moderation policy, and if you don’t like it: tough. Whereas on Mastodon, if you don’t like your instance’s policy, you can always switch to another one. (And there’s work in progress to make that migration easier.)

Conclusion

Mastodon is not perfect. The software is still rough in some places, the underlying protocols (OStatus and ActivityPub) are still getting hammered out at the W3C, and the community devolves into tiresome bickering more often than I’d like.

But I still have more faith in Mastodon than I do in Twitter, whose user growth has flatlined and whose profits are nonexistent, and thus will have to resort to increasingly desperate measures to satisfy its investors, who are still waiting for a sweet return on investment for all those eyeballs they bought. I expect this will mean more promoted tweets, more ways to promote tweets, and ultimately less value for Twitter’s users, as they become increasingly drowned in a sea of brand accounts trying to sell them a hamburger, fake news trying to swing an election, and bots trying to do who knows what. Meanwhile the harassment problem will never be Twitter’s main priority, despite what their CEO says, because as long as controversy and conflict are good for grabbing eyeballs, they’re good for Twitter’s bottom line.

The main reason I’m hopeful about Mastodon is that it’s an opportunity to learn from Twitter’s mistakes and to experiment with fresh ideas for improving social media. For instance, how about disabling public follower counts, since they can make us feel like we’re living in a Black Mirror episode where everyone’s self-worth is determined by a single number? (In fact witches.town already does this; every user’s number is a cheeky 666.) Or how about removing the “quote-repost” feature, since we saw the nasty dog-piling it enabled on Twitter? Or how about adding features that encourage users to log off every once in a while, so that social media doesn’t turn into an addictive slot machine?

All of these things are possible in Mastodon, because the code is open-source and the servers belong to the users. We can even tinker with these ideas at the instance level, to test how something pans out at the small scale before bringing it to a wider audience. Instead of Twitter’s one-size-fits-all approach, we can tailor social media to fit the needs of every community, with local admins who are motivated to help because they’re moderating a small group of like-minded people rather than 300 million of them.

Mastodon can feel like a return to another time, when the web was small and it felt possible to actually have an impact on the websites we use every day. But it’s also a glimpse into the post-Twitter future that we need, if we want to have control over our data, our minds, and our public discourse.

Interested in Mastodon? Check out joinMastodon.org or instances.social for help finding an instance to join. If you’re not sure, I’d recommend toot.cafe (my own), cybre.space (cyberpunk themed), mastodon.art (for artists), awoo.space (focus on safety), or for general interests: mastodon.social, icosahedron.website, or octodon.social.

Burnout and Twitter fatigue

I suppose it should come as no surprise that the author of “What it feels like to be an open-source maintainer” is suffering from burnout, but to be honest it caught me off-guard. Even after writing it, I thought I was just going through a rough patch, but in retrospect it’s impossible to call what I’ve been feeling by any other name.

In the six months since I wrote that post, I’ve largely abandoned my involvement in PouchDB as well as dozens of other open-source projects. I’ve stepped away from the social telephone that only brings bad news and let the GitHub notifications pile up on my doorstep. (Over a thousand now, but who’s counting?) I’ve stopped going to conferences and only written one new blog post, which had actually been in draft for several months and so barely qualifies. In short, I display all the symptoms of burnout.

I could rehash the same material from that blog post, but really it’s only part of the story. Dealing with a constant flood of negative attention on your open-source projects is enough to wear anybody down, but that’s only a proximate cause. This story is best told against a backdrop of general malaise and world-weariness, which perhaps other folks in the tech industry can identify with.

In this post, I’m going to talk about one major stage of my burnout, which was the end of my love affair with Twitter. There’s more to cover on the subject of burnout, but hopefully this will serve as a good starting point.

2016: the terrible, horrible, no good, very bad year

It’s become an internet cliché to say that 2016 was a terrible year, but last year’s US election truly shocked me. It also shook me out of a lot of my complacency over the direction of the tech industry and of the world in general. I consider myself a centrist (or perhaps center-left), but the election of a man as obviously vile and odious as Donald Trump left me, like many others, grasping for explanations.

2016 marked a worldwide rise in authoritarianism and illiberalism, and it’s hard to pin down to a single cause. (For a full review, The Retreat of Western Liberalism comes closest.) A lot of ink has already been spilled on the subject, but what I feel most qualified to talk about is the role technology played in the election. It wasn’t a starring role, but if 2016 were a theater play, technology might have fit as the mischievous trickster, sowing confusion and distrust and ultimately helping the baddies on their way.

Of course, what I’m talking about is social media. Twitter is well-known as Donald Trump’s favorite megaphone, and Facebook, for its part, also helped to spread rumors, lies, and propaganda that not only boosted extremists and conspiracy theorists, but also led to a general erosion of public trust in our media and institutions. We no longer have a common set of facts we can all agree upon; we only have bombast and hearsay, shouted across the political divide. Integrity and truth no longer matter; only who can get the sharpest zingers and the juiciest headlines that deliver clicks and eyeballs.

None of this should be surprising given what both Twitter and Facebook are. Fundamentally, they are marketing tools designed to promote the most virulent memes (in Dawkins’ sense of the word, as a catchy idea that travels well) and thus maximize “engagement” and “sharing.” Their algorithms act as a force of natural selection on a candidate population of quips, one-liners, clickbait, and repartees to select only those best-suited to tap into our basest human emotions, so that they get shared and reposted and thus infect the maximum number of brains possible.

Facebook and Twitter are platforms where subtlety and nuance get lost in a sea of paranoid accusations, wild hyperbole, vicious put-downs, and smug preaching-to-the-choir. They are platforms where the sensationalist headline gets thousands of retweets, and the sheepish retraction (if it exists) gets less than a hundred. (Guess which one people remember.) In short, they are platforms that are tailor-made for marketing, and thus for its public-sector twin, propaganda.

With social media, it’s as if the Axis powers no longer had to distribute leaflets via airplane, but could instead just whip up some catchy headlines like “You won’t believe why the Allies can’t win the war” and “11 reasons the resistance movement is a flop.” Toss enough of those into social media – it doesn’t even matter which ones are true or false, and you can tweak the headlines as you go – and you’ve got a recipe for a public discourse that sounds less like that of an informed democracy and more like the swirling cacophony of everybody’s id talking at the same time.

Screenshot from "Dunkirk" showing soldier holding leaflet saying "we surround you"

Credit: Dunkirk

I work for a browser vendor, and I’ve even contributed code to a JavaScript library that appears on Twitter’s frontend (LocalForage, used on Twitter’s mobile site). So in 2016 I came to feel implicitly responsible for the building of a technical infrastructure that, at this moment, looks to me like the grotesque monster at the end of Akira slouching its way towards Bethlehem to devour what’s left of the civilized world. It’s not exactly a vision that makes me feel chipper when I’m getting ready for work in the morning.

Twitter fatigue

More than anything, 2016 was the year when I started to question my Twitter addiction, and to resolve to use it less and less. Today I don’t even have the Twitter app installed, and I barely visit the mobile site either. Most of my idle thoughts go to Mastodon instead (which is a subject for another post).

In 2016, the tenor of Twitter seemed to me to change dramatically for the worse. Suddenly, everything became politicized. Somehow the newfound eagerness to howl bloody murder at one’s political opponents translated into an eagerness to howl at everyone about everything. Increasingly I felt I was watching people break off into rival factions and hurl insults at each other.

Perhaps I felt this most directly when I was attacked for a slide in a presentation I gave at Fronteers Conf saying “In 2016, it’s okay to build a website that doesn’t work without JavaScript.” I already covered that surreal experience in this post, but essentially a photo of me made the rounds on social media, and without even knowing who was pictured or what the context of the talk was, people decided I needed to be taken down a peg. And so I watched my Twitter feed devolve into a nasty pile-on of insults, snark, and condemnation, all because I had expressed a controversial opinion on (wait for it) a programming language.

I have nothing against Fronteers Conf, and I appreciate that many of the conference organizers and attendees defended me on Twitter when the discussion started to take a turn for the ugly. But honestly, this incident is one of the main reasons I stopped speaking at conferences.

At tech conferences, you’re encouraged to “interact” via Twitter, which is a platform not known for valuing nuance or context. So instead of my 40-minute talk being about, you know, the talk, its defining moment was a bite-sized nugget blasted out on social media, as well as the ensuing uproar. (I noticed that most of the audience had their eyes glued to their phones for the last 20 minutes after that slide, so certainly what I said in those moments didn’t matter.)

Essentially, everyone on Twitter decided that it was time for the Two Minutes Hate, and mine was the face that needed to be snarled at. I got a small taste of the Justine Sacco experience, and frankly it felt awful. Moreover, it didn’t feel like a good trade-off for the hours poured into preparing a talk, practicing it, and flying out to the conference venue.

This experience has also made me ambivalent about all the other moral crusades that seem to be carried out regularly on Twitter these days. Did the target really deserve it? Or was it just a clumsy gaffe brought about by jetlag, a half-hearted joke that landed badly, or a poorly-worded slide (mea culpa), and so maybe we should cut them some slack?

Or, even if they deserved it, is all the wailing and pearl-clutching really good for the rest of us? It certainly earns us likes and retweets, which can feel like virtue in the heat of the moment. But the more I see these dog-piles, the more I get the sense that what we’re feeling isn’t the noble élan of the virtuous, but instead the childish glee of having the unpopular kid on the playground laying crumpled at your feet and the crowd at your back. (Or the relief of being in the crowd, and not on the ground getting dirt kicked into your face.)

Breaking up with Twitter is hard to do

In the tech industry, though, quitting Twitter is no easy feat. Especially if you work in developer relations or evangelism, Twitter might be a fundamental part of your job description: promoting your company’s content, answering questions, “engaging the community,” etc. Your follower count, especially if it numbers in the thousands or tens of thousands, may also be seen as a job asset that’s hard to give up.

Twitter itself also serves as a personal identifier in an industry where schmoozing and networking can be critical career boosters. At developer conferences, speakers often introduce themselves using their Twitter handle (which might be discreetly emblazoned on every slide) and then end the talk by imploring their audience to follow them on Twitter. It’s not even necessary to say “Twitter” or use the Twitter logo; if you put an @-sign in front of it, there’s no confusion about which social network you’re referring to.

In blog posts, it’s also common to link to someone’s Twitter handle when you mention them by name. Forget your personal blog or myname.comtwitter.com/myname acts as your unique identifier, and the preferred way of discovering your work, contacting you, and judging your “influencer” status by your number of followers. Think of it as your personal baseball card, complete with your photo, team (company) affiliation, and key career stats.

Twitter is so inescapably embedded in the tech industry, especially in my corner of it (JavaScript/web/whatever you want to call it) that it can be maddening if you’ve resolved to quit it. For better or worse, it’s where news hits first, it’s where software releases and blog posts are announced, and it’s how you keep up with friends and colleagues in the industry.

The irony is not lost on me that, if I even want people to read this blog post, I’m going to need to post it to Twitter. Such is the degree of Twitter’s dominance, at least in the crowd I run with. To some extent, if you’re not on Twitter, you’re just not part of the conversation.

Alternatives do exist. For a hot moment last April, I was hoping that Mastodon (ad-free, open-source, no algorithmic timeline) might build up enough momentum to overtake Twitter, but unfortunately that wave seems to have crested. That said, I’m still betting on it in the form of running my own Mastodon server, despite having few illusions that it can eclipse Twitter anytime soon. But again, that’s a subject for another post.

One thing my personal exodus toward Mastodon did teach me, though, is that I can live without social media. Not only am I spending more time on Mastodon than on Twitter these days, but I’m spending less time on social media in general, to which I credit both sleeping better and feeling less anxious and distracted all the time. No more staying up late to read about the scandal du jour or hearing hot takes on the latest political calamity that, this time, will surely bring about the end of the world. No more watching people tear each other down over petty grievances, or hearing the sanctimonious echo-chamber sermons of those utterly convinced of their own righteousness.

With less time devoted to social media, I’m also enjoying more non-tech activities to help heal my burnout: spending time with my wife, visiting friends and family, getting back into cycling, reading books, playing guitar. I’m hoping that eventually I’ll feel well enough to hop back onto the GitHub treadmill, although it’s tough because it’s only when I stepped off that I realized how fast the dang thing was going.

Advice for Twitter expats

What are the good alternatives to Twitter? Besides Mastodon (which is really more of a friendly chat room these days), I’ve found you can get decent tech news from:

  • Hacker News (Yeah yeah, just don’t read the comments. Or read n-gate for a satirical take.)
  • Lobsters (Like HN, but more technical.)
  • EchoJS (A bit spammy, but occasionally there are gems.)
  • The webdev and javascript subreddits (Again, the comments don’t come recommended.)

I don’t find these sites to be a full supplement for Twitter, but maybe that’s a good thing. The less time I spend on Twitter, the less I expose my mind to toxic brain-viruses, the less I feel goaded by some titillating comment to respond with my own mental detritus (thereby stimulating others to do the same), and the less I contribute to the propaganda machine that is mayyybe bringing about the fall of civilization. All of which is pure goodness to a burned-out techie like me.

A brief and incomplete history of JavaScript bundlers

Ever since I read Malte Ubl’s proposal for a JavaScript bundle syntax, I’ve been fascinated by the question: does JavaScript need a “bundle” standard?

Unfortunately that question will have to wait for another post, because it’s much more complicated than what I can cover here. But to at least make the first tentative stabs at answering it, I’d like to explore some more basic questions:

  • What is a JavaScript bundler?
  • What purpose do bundlers serve in the modern webdev stack?

To try to answer these questions, I’d like to offer my historical perspective on what are (arguably) the two most important bundlers of the last five years: Browserify and Webpack.

A bundle of bamboo

A bundle of bamboo, via Wikipedia

What is a bundle?

Conceptually, a JavaScript bundle is very simple: it’s a collection of multiple scripts, combined into a single file. The original bundler was called +=, i.e. concatenation, and for a long time it was all anyone really needed. The whole point was to avoid the 6-connections-per-origin limit and the built-in overhead of HTTP/1.1 connections by simply jamming all your JavaScript into a single file. Easy-peasy.

Disregarding some interesting but ultimately niche bundlers such as GWT, RequireJS, and Closure Compiler, concatenation was still the most common bundler until very recently. Even fairly modern scaffolding tools like Yeoman were still recommending concatenation as the default bundler well into 2013, using lightweight tools such as usemin.

It was only really when Browserify hit the scene in 2013 did non-concatenation bundlers start to go mainstream.

The rise of Browserify

Interestingly, Browserify wasn’t originally designed to solve the problem of bundling. Instead, it was designed to solve the problem of Node developers who wanted to reuse their code in the browser. (It’s right there in the name: “browser-ify” your Node code!)

Screenshot of Browserify homepage from 2013

Screenshot of the original Browserify homepage from January 2013 (via the Internet Archive)

Before Browserify, if you were writing a JavaScript module that was designed to work in both Node or the browser, you’d have to do something like this:

var MyModule = 'hello world';

if (typeof module !== 'undefined' && module.exports) {
  module.exports = MyModule;
} else {
  (typeof self !== 'undefined' ? self : window).MyModule = MyModule;
}

This works fine for single files, but if you’re accustomed to Node conventions, it becomes aggravating that you can’t do something like this:

var otherModule = require('./otherModule');

Or even:

var otherPackage = require('other-package');

By 2014, npm had already grown to over 50,000 modules, so the idea of reusing those modules within browser code was a compelling proposition. The problem Browserify solved was thus twofold:

  1. Make the CommonJS module system work for the browser (by crawling the dependency tree, reading files, and building a single bundle file).
  2. Make Node built-ins and conventions (process, Buffer, crypto, etc.) work in the browser, by implementing polyfills and shims for them.

This second point is an often-overlooked benefit of the cowpath that Browserify paved. At the time Browserify debuted, many of those 50,000 modules weren’t written with any consideration for how they might run in the browser, and Node-isms like process.nextTick() and setImmediate() ran rampant. For Browserify to “just work,” it had to solve the compatibility problem.

What this involved was a lot of effort to reimplement nearly all of Node’s standard library for the browser, tackling the inevitable issues of cross-browser compatibility along the way. This resulted in some extremely battle-tested libraries such as events, process, buffer, inherits, and crypto, among others.

If you want to understand the ridiculous amount of work that had to go into building all this infrastructure, I recommend taking a look at Calvin Metcalf’s series on implementing crypto for the browser. Or, if you’re too faint of heart, you can instead read about how he helped fix process.nextTick() to work with Sinon or avoid bugs in oldIE’s timer system. (Calvin is truly one of the unsung heroes of JavaScript. Look in your bundle, and odds are you will find his code in there somewhere!)

All of these libraries – buffer, crypto, process, etc. – are still in wide use today via Browserify, as well as other bundlers like Webpack and Rollup. They are the magic behind why new Buffer() and process.nextTick() “just work,” and are a big part of Browserify’s success story.

Enter Webpack

While Browserify was picking up steam, and more and more browser-ready modules were starting to be published to npm, Webpack rose to prominence in 2015, buoyed by the popularity of React and the endorsement of Pete Hunt.

Webpack and Browserify are often seen today as solutions to the same problem, but Webpack’s initial focus was a bit different from Browserify’s. Whereas Browserify’s goal was to make Node modules run in the browser, Webpack’s goal was to create a dependency graph for all of the assets in a website – not just JavaScript, but also CSS, images, SVGs, and even HTML.

The Webpack view of the world, with multiple types of assets all treated as part of the dependency graph

The Webpack view of the world, via “What is Webpack?”

In contrast to Browserify, which was almost dogmatic in its insistence on Node compatibility, Webpack was cheerful to break Node conventions and introduce code like this:

require('./styles.css');

Or even:

var svg = require('svg-url?limit=1024!./file.svg');

Webpack did this for a few different reasons:

  1. Once all of a website’s assets can be expressed as a dependency graph, it becomes easy to define “components” (collections of HTML, CSS, JavaScript, images, etc.) as standalone modules, which can be easily reused and even published to npm.
  2. Using a JavaScript-based module system for assets means that Hot Module Replacement is easy and natural, e.g. a stylesheet can automatically update itself by injection and replacement into the DOM via script.
  3. Ultimately, all of this is configurable using loaders, meaning you can get the benefits of an integrated module system without having to ship a gigantic JavaScript bundle to your users. (Although how well this works in practice is debatable.).

Because Browserify was originally the only game in town, though, Webpack had to undergo its own series of compatibility fixes, so that existing Browserify-targeting modules could work well with Webpack. This wasn’t always easy, as a JavaScript package maintainer of the time might have told you.

Out of this push for greater Webpack-Browserify compatibility grew ad-hoc standards like the node-browser-resolve algorithm, which defines what the "browser" field in package.json is supposed to do. (This field is an extension of npm’s own package.json definition, which specifies how modules should be swapped out when building in “browser mode” vs “Node mode.”)

Closing thoughts

Today, Browserify and Webpack have largely converged in functionality, although Browserify still tends to be preferred by old-school Node developers, whereas Webpack is the tool of choice for frontend web developers. Up-and-comers such as Rollup, splittable, and fuse-box (among many others) are also making the frontend bundler landscape increasingly diverse and interesting.

So that’s my view of the great bundler wars of 2013-2017! Hopefully in a future blog post I’ll be able to cover whether or not bundlers like Browserify and Webpack demonstrate the need for a “standard” to unite them all.

Feel free to weigh in on Twitter or on Mastodon.

What it feels like to be an open-source maintainer

Outside your door stands a line of a few hundred people. They are patiently waiting for you to answer their questions, complaints, pull requests, and feature requests.

You want to help all of them, but for now you’re putting it off. Maybe you had a hard day at work, or you’re tired, or you’re just trying to enjoy a weekend with your family and friends.

But if you go to github.com/notifications, there’s a constant reminder of how many people are waiting:

screenshot showing 403 unread GitHub notifications

When you manage to find some spare time, you open the door to the first person. They’re well-meaning enough; they tried to use your project but ran into some confusion over the API. They’ve pasted their code into a GitHub comment, but they forgot or didn’t know how to format it, so their code is a big unreadable mess.

Helpfully, you edit their comment to add a code block, so that it’s nicely formatted. But it’s still a lot of code to read.

Also, their description of the problem is a bit hard to understand. Maybe this person doesn’t speak English as a first language, or maybe they have a disability that makes it difficult for them to communicate via writing. You’re not sure. Either way, you struggle to understand the paragraphs of text they’ve posted.

Wearily, you glance at the hundreds of other folks waiting in line behind them. You could spend a half-hour trying to understand this person’s code, or you could just skim through it and offer some links to tutorials and documentation, on the off-chance that it will help solve their problem. You also cheerfully suggest that they try Stack Overflow or the Slack channel instead.

The next person in line has a frown on their face. They spew out complaints about how your project wasted 2 hours of their life because a certain API didn’t work as advertised. Their vitriol gives you a bad feeling in the pit of your stomach.

You don’t waste a lot of time on this person. You simply say, “This is an open-source project, and it’s maintained by volunteers. If there’s a bug in the code, please submit a reproducible test case or a PR.”

The next person has run into a very common error, with an easy workaround. You know you’ve seen this error a few times before, but can’t quite recall where the solution was posted. Stack Overflow? The wiki? The mailing list? After a few minutes of Googling, you paste a link and close the issue.

The next person is a regular contributor. You recognize their name from various community forums and sibling projects. They’ve run into a very esoteric issue and have proposed a pull request to fix it. Unfortunately the issue is complicated, and so their PR contains many paragraphs of prose explaining it.

Again, your eye darts to the hundreds of people still waiting in line. You know that this person put a lot of work into their solution, and it’s probably a reasonable one. The Travis tests passed, and so you’re tempted to just say "LGTM" and merge the pull request.

However, you’ve been burned by that before. In the past, you’ve merged a PR without fully evaluating it, and in the end it led to new headaches because of problems you failed to foresee. Maybe the tests passed, but the performance degraded by a factor of ten. Or maybe it introduced a memory leak. Or maybe the PR made the project too confusing for new users, because it excessively complicated the API surface.

If you merge this PR now, you might wind up with even more issues tomorrow, because you broke someone else’s workflow by solving this one person’s (very edge-casey) problem. So you put it on the back burner. You’ll get to it later when you have more time.

The next person in line has found a new bug, but you know that it’s actually a bug in a sibling project. They’re saying that this is blocking them from shipping their app. You know it’s a big problem, but it’s one of many, and so you don’t have time to fix it right now.

You respond that this looks like a genuine issue, but it’s more appropriate to open in another repo. So you close their issue and copy it into the other repo, then add a comment suggesting where they might look in the code to start fixing it. You doubt they’ll actually do so, though. Very few do.

The next person just says “What’s the status on this?” You’re not sure what they’re talking about, so you look at the context. They’ve commented on a lengthy GitHub thread about a long-standing bug in the project. Many people disagreed on the proper solution to the problem, so it generated a lot of discussion.

There are more than 20 comments on this particular issue, and it would take you a long time to read through them all to jog your memory. So you merely respond, “Sorry, this issue has been open for a while, but nobody has tackled it yet. We’re still trying to understand the scope of the problem; a pull request could be a good start!”

The next person is just a GreenKeeper bot. These are easy. Except that this particular repo has fairly flaky tests, and the tests failed for what looks like a spurious reason, so you have to restart them to pass. You restart the tests and try to remind yourself to look into it later after Travis has had a chance to run.

The next person has opened a pull request, but it’s on a repo that’s fairly active, and so another maintainer is already providing feedback. You glance through the thread; you trust the other maintainer to handle this one. So you mark it as read and move on.

The next person has run into what appears to be a bug, and it’s not one you’ve ever seen before. But unfortunately they’ve provided scant details on how the problem actually occurred. What browser was it? What version of Node? What version of the project? What code did they use to reproduce it? You ask them for clarification and close the tab.

The constant stream

After a while, you’ve gone through ten or twenty people like this. There are still more than a hundred waiting in line. But by now you’re feeling exhausted; each person has either had a complaint, a question, or a request for enhancement.

In a sense, these GitHub notifications are a constant stream of negativity about your projects. Nobody opens an issue or a pull request when they’re satisfied with your work. They only do so when they’ve found something lacking. Even if you only spend a little bit of time reading through these notifications, it can be mentally and emotionally exhausting.

Your partner has observed that you’re always grumpy after going through this ritual. Maybe you found yourself snapping at her for no reason, just because you were put in a sour mood. “If doing open source makes you so angry, why do you even do it?” she asks. You don’t have a good answer.

You could take a break; in fact you’ve probably earned it by now. In the past, you’ve even taken vacations of a week or two from GitHub, just for your own mental health. But you know that that’s exactly how you ended up in this situation, with hundreds of people patiently waiting.

If you had just kept on top of your GitHub notifications, you’d probably have a more manageable 20-30 to deal with per day. Instead you let them pile up, so now there are hundreds. You feel guilty.

In the past, for one reason or another, you’ve really let issues pile up. You might have seen an issue that was left unanswered for months. Usually, when you go back to address such an issue, the person who opened it never responds. Or they respond by saying, “I fixed my problem by abandoning your project and using another one instead.” That makes you feel bad, but you understand their frustration.

You’ve learned from experience that the most pragmatic response to these stale issues is often just to say, “I’m closing old issues. Please reopen if this is still a problem for you or if you can provide more details.” Usually there is no response. Sometimes there is, but it’s just an angry comment about how they were made to wait for so long.

So nowadays you try to be more diligent about staying on top of your notifications. Hundreds of people waiting in line are far too many. You long for that line to get down to a hundred, or a dozen, or even the mythical inbox zero. So you press on.

Attracting new contributors

After triaging enough issues like this, even if you eventually reach inbox zero, you might still end up with a large backlog of open bugs and pull requests. Labeling can help – for instance, you might label issues as “needs reproducing” or “has test case” or “good first patch.” The “good first patch” ones can be especially helpful, since they often attract new contributors.

However, you’ve noticed that often the only issues that attract new contributors are the very easy ones, the ones where the effort to document the issue and explain how to fix it outweighs the effort to just fix it yourself. You create some of these issues, because you know it’s a worthy goal to get new people involved in open source, and you feel good when the pull request author tells you, “This was my first contribution to an open-source project.”

But you know it’s very unlikely that they’ll ever come back; usually these folks don’t become regular contributors or maintainers. You wonder if you did something wrong, if there’s something more you could have done to onboard new maintainers and help lighten your load.

One of your projects is nearly self-sustaining. You haven’t touched it in years, but there’s a group of maintainers who respond to every issue and PR, so you don’t have to. You’re enormously grateful to these maintainers. But you have no idea what you did to get so many contributors to this project, whereas other projects wind up as your responsibility and yours alone.

Looking ahead

You’re reluctant to create new projects, because you know it will just increase your maintenance burden. In fact, there’s a perverse effect where, the more successful you are, the more you get “punished” with GitHub notifications.

You can still recall the thrill of creation, the joy of writing a new project from scratch and solving a previously-unsolved problem. But now you weigh that joy against the knowledge that any new project will necessarily steal time from old projects. You wonder if it it’s time to formally deprecate one of your old repos, or to mark it as unmaintained.

You wonder how much longer this can go on before you just burn out. You’ve considered doing open source as your day job, but from talking with folks who actually do open source for a living, you know that this usually means permission to work on a specific open-source project as your day job. That doesn’t help you much, because you have dozens of projects across various domains, which are all vying for your time.

What you want most of all is to have more projects that maintain themselves. You try to follow all the best practices: you have a CONTRIBUTING.md and a code of conduct, you enthusiastically hand out owner privileges to anyone who submits a high-quality PR. It’s exhausting to do this for every project, though, so you’re not as diligent as you wish you could be.

You feel guilty about that too, since you know open source is frequently regarded as an exclusive club for privileged white males, like yourself. So you worry that you’re not doing enough to help fix that problem.

More than anything, you feel the guilt: the guilt of knowing that you could have helped someone solve their problem, but instead you let their issue rot for months before closing it. Or the guilt of knowing that someone opened their first pull request ever on your repo, but you didn’t have time to respond to it, and because of that, you may have permanently discouraged them from open source. You feel guilty for the work that you do, for the work that you didn’t do, and for not recruiting more people to share in your unhappy guilt-ridden experience.

Putting it all together

Everything I’ve said above is based on my own experiences. I can’t claim to speak for all people who do open-source software, but this is what it feels like to me.

I’ve been doing open source for a long time (roughly seven years), and I’ve been reluctant to complain about any of this, because I worried it could be perceived as melodramatic whining from someone who ought to know better. After all, isn’t this situation one of my own making? I could walk away from GitHub whenever I want; I have no obligations to anyone.

Also, shouldn’t I be grateful? My work on open source has helped give me my standing in the community. I get invitations to speak at conferences. I have thousands of Twitter followers who listen to what I have to say and hold my opinion in high esteem. Arguably, I got my job at Microsoft because of my experience in open source. Who am I to complain?

And yet, I’ve known many others in positions similar to mine who have burned out. Folks who enthusiastically merged pull requests, fixed issues, and wrote blog posts about their projects, before vanishing without a trace. For some of these people, I don’t even bother opening issues on their repos, because I know they won’t respond. I don’t hold it against them, but I worry that I’ll share their fate.

I’ve already taken plenty of self-care measures. I don’t use the GitHub notification interface anymore – I use email filters, so that I can categorize my notifications based on project (unmaintained ones get ignored) or type of notification (at-mentions and threads I’ve commented on usually deserve higher priority). Since it’s email, this also helps me work offline and manage everything in one place.

Often I will get emails out of the blue asking for support on a project that I’ve long stopped maintaining (I still get at least one per month about this one, for instance), and usually I just don’t even respond to those. I also tend to ignore comments on my blog posts, responses to Stack Overflow answers, and mailing list questions. I aggressively un-watch repos that I feel someone else is doing a good enough job of maintaining.

One reason this situation is so frustrating is that, increasingly, I find that issue triage takes time away from the actual maintenance of a project. In other words, I often only have enough time to read through an issue and say, “Sorry, I don’t have time to look at this right now.” Just the mere act of responding can take up a majority of the time I’ve set aside for open source.

Issue templates, GreenKeeper, Travis, travis_retry, Coveralls, Sauce Labs… there are so many technical solutions to the problems of open-source maintenance, and I’m grateful to all of them. There’s no way I’d be able to keep my head on straight if I didn’t have these automated tools. But at some point you run up against issues that are more social problems than technical problems. One human being just doesn’t scale. I’m not even in the top 100 npm maintainers and I’m already feeling the squeeze; I can’t imagine how those hundred people must feel.

I’ve already told my partner that, if and when we decide to start having kids, I will probably quit open source for good. I just can’t see how I’ll be able to make the time for both raising a family and doing open source. I anticipate that ultimately this will be the solution to my problem: the nuclear option. I just hope it comes in a positive form, like starting a new chapter of my life, and not in a negative form, like unceremoniously burning out.

Closing thoughts

If you’ve read this far and are interested in the problems plaguing open-source communities and potential solutions, you may want to look into “Roads and Bridges” by Nadia Eghbal. It’s probably the clearest-eyed and most thorough analysis of the problem.

I’m also open to suggestions, although keep in mind that I’m very reluctant to mix money and labor in my open-source projects (for perhaps silly idealistic reasons). But I have seen it work well for other projects.

Note that despite all the negativity expressed above, I still feel that open source has been a valuable addition to my life, and I don’t regret any of it. But I hope this is a useful window into how it can feel to be a victim of your own success, and to feel weighed down by all the work left undone.

If there’s one thing I’ve learned in open source, it’s this: the more work you do, the more work gets asked of you. There is no solution to that problem that I’m aware of.

Please feel free to comment on Twitter and to read responses by Mikeal Rogers and Jan Lehnardt.

How to write a JavaScript package for both Node and the browser

This is an issue that I’ve seen a lot of confusion over, and even seasoned JavaScript developers might have missed some of its subtleties. So I thought it was worth a short tutorial.

Let’s say you have a JavaScript module that you want to publish to npm, available both for Node and for the browser. But there’s a catch! This particular module has a slightly different implementation for the Node version compared to the browser version.

This situation comes up fairly frequently, since there are lots of tiny environment differences between Node and the browser. And it can be tricky to implement correctly, especially if you’re trying to optimize for the smallest possible browser bundle.

Let’s build a JS package

So let’s write a mini JavaScript package, called base64-encode-string. All it does is take a string as input, and it outputs the base64-encoded version.

For the browser, this is easy; we can just use the built-in btoa function:

module.exports = function (string) {
  return btoa(string);
};

In Node, though, there is no btoa function. So we’ll have to create a Buffer instead, and then call buffer.toString() on it:

module.exports = function (string) {
  return Buffer.from(string, 'binary').toString('base64');
};

Both of these should provide the correct base64-encoded version of a string. For instance:

var b64encode = require('base64-encode-string');
b64encode('foo');    // Zm9v
b64encode('foobar'); // Zm9vYmFy

Now we’ll just need some way to detect whether we’re running in the browser or in Node, so we can be sure to use the right version. Both Browserify and Webpack define a process.browser field which returns true, whereas in Node it’s falsy. So we can simply do:

if (process.browser) {
  module.exports = function (string) {
    return btoa(string);
  };
} else {
  module.exports = function (string) {
    return Buffer.from(string, 'binary').toString('base64');
  };
}

Now we just name our file index.js, type npm publish, and we’re done, right? Well, this works, but unfortunately there’s a big performance problem with this implementation.

Since our index.js file contains references to the Node built-in process and Buffer modules, both Browserify and Webpack will automatically include the polyfills for those entire modules in the bundle.

From this simple 9-line module, I calculated that Browserify and Webpack will create a bundle weighing 24.7KB minified (7.6KB min+gz). That’s a lot of bytes for something that, in the browser, only needs to be expressed with btoa!

“browser” field, how I love thee

If you search through the Browserify or Webpack documentation for tips on how to solve this problem, you may eventually discover node-browser-resolve. This is a specification for a "browser" field inside of package.json, which can be used to define modules that should be swapped out when building for the browser.

Using this technique, we can add the following to our package.json:

{
  /* ... */
  "browser": {
    "./index.js": "./browser.js"
  }
}

And then separate the functions into two different files, index.js and browser.js:

// index.js
module.exports = function (string) {
  return Buffer.from(string, 'binary').toString('base64');
};
// browser.js
module.exports = function (string) {
  return btoa(string);
};

After this fix, Browserify and Webpack provide much more reasonable bundles: Browserify’s is 511 bytes minified (315 min+gz), and Webpack’s is 550 bytes minified (297 min+gz).

When we publish our package to npm, anyone running require('base64-encode-string') in Node will get the Node version, and anyone doing the same thing with Browserify or Webpack will get the browser version. Success!

For Rollup, it’s a bit more complicated, but not too much extra work. Rollup users will need to use rollup-plugin-node-resolve and set browser to true in the options.

For jspm there is unfortunately no support for the “browser” field, but jspm users can get around it in this case by doing require('base64-encode-string/browser') or jspm install npm:base64-encode-string -o "{main:'browser.js'}". Alternatively, the package author can specify a “jspm” field in their package.json.

Advanced techniques

The direct "browser" method works well, but for larger projects I find that it introduces an awkward coupling between package.json and the codebase. For instance, our package.json could quickly end up looking like this:

{
  /* ... */
  "browser": {
    "./index.js": "./browser.js",
    "./widget.js": "./widget-browser.js",
    "./doodad.js": "./doodad-browser.js",
    /* etc. */
  }
}

So every time you want a browser-specific module, you’d have to create two separate files, and then remember to add an extra line to the "browser" field linking them together. And be careful not to misspell anything!

Also, you may find yourself extracting individual bits of code into separate modules, merely because you wanted to avoid an if (process.browser) {} check. When these *-browser.js files accumulate, they can start to make the codebase a lot harder to navigate.

If this situation gets too unwieldy, there are a few different solutions. My personal favorite is to use Rollup as a build tool, to automatically split a single codebase into separate index.js and browser.js files. This has the added benefit of de-modularizing the code you ship to consumers, saving bytes and time.

To do so, install rollup and rollup-plugin-replace, then define a rollup.config.js file:

import replace from 'rollup-plugin-replace';
export default {
  entry: 'src/index.js',
  format: 'cjs',
  plugins: [
    replace({ 'process.browser': !!process.env.BROWSER })
  ]
};

(We’ll use that process.env.BROWSER as a handy way to switch between browser builds and Node builds.)

Next, we can create a src/index.js file with a single function using a normal process.browser condition:

export default function base64Encode(string) {
  if (process.browser) {
    return btoa(string);
  } else {
    return Buffer.from(string, 'binary').toString('base64');
  }
}

Then add a prepublish step to package.json to generate the files:

{
  /* ... */
  "scripts": {
    "prepublish": "rollup -c > index.js && BROWSER=true rollup -c > browser.js"
  }
}

The generated files are fairly straightforward and readable:

// index.js
'use strict';

function base64Encode(string) {
  {
    return Buffer.from(string, 'binary').toString('base64');
  }
}

module.exports = base64Encode;
// browser.js
'use strict';

function base64Encode(string) {
  {
    return btoa(string);
  }
}

module.exports = base64Encode;

You’ll notice that Rollup automatically converts process.browser to true or false as necessary, then shakes out the unused code. So no references to process or Buffer will end up in the browser bundle.

Using this technique, you can have any number of process.browser switches in your codebase, but the published result is two small, focused index.js and browser.js files, with only the Node-related code for Node, and only the browser-related code for the browser.

As an added bonus, you can configure Rollup to also generate ES module builds, IIFE builds, or UMD builds. For an example of a simple library with multiple Rollup build targets, you can check out my project marky.

The actual project described in this post (base64-encode-string) has also been published to npm so that you can inspect it and see how it ticks. The source code is available on GitHub.