Archive for November, 2013

Introducing the SuperSaiyanScollView: super-fast sectioned lists for Android

Update: This blog post is now out-of-date. Please see the official documentation on GitHub.

Say you’re writing an Android app, and you have a ListView you’d like to divide into sections:

Sectioned list views.

Sectioned list views.

Normally, you’d need to write a custom ListAdapter, where you define the resources yourself and juggle two different types of View. Not fun.

public class MyBoringAdapter<Foo> extends ArrayAdapter<Foo> {

    // constructors...
    
    public int getViewTypeCount() {
      return 2;
    }
    
    public int getItemViewType(int pos) {
      return isHeader(pos) ? 0 : 1;
    }

    // more boilerplate...
    
    public View getView(int pos, View view, ViewGroup parent) {
        if (isHeader(pos)) {
            // sigh
        } else {
            // so tired of this crap
        }
    }
}

Now, what happens if you want to change the ordering? Or add new sections? Or add fast-scroll overlays?

It’s one of the most common UI patterns in Android, and yet (surprisingly) it’s still a pain to implement. Nothing in the stock Android SDK provides this functionality.

Enter the SuperSaiyanScrollView (dramatic gong sound). It’s a standalone library that you can easily import into any Android app, and it seamlessly adds fast-scrolling, sorting, and sectioning. Plus, it looks great on both Android 4.0 and pre-4.0 devices, on tablets and phones.

SuperSaiyanScrollView on HTC Magic (Eclair) and Galaxy Nexus (Jelly Bean)

SuperSaiyanScrollView on HTC Magic (Eclair) and Galaxy Nexus (Jelly Bean)

Why “Super Saiyan”? Because:

  1. I made it, so I get to name it.
  2. It’s super-fast, super-powerful, and it kicks (stock) Android’s ass.
Their power levels are definitely over 9000.

Their power levels are definitely over 9000.

Usage

The SuperSaiyanScrollView code attemps to be as unobtrusive as possible. To use it, you just need to wrap your existing ListView in a SuperSaiyanScrollView and your existing Adapter in a SectionedListAdapter.

In your layout XML file, add a SuperSaiyanScrollView around your ListView:

    <com.nolanlawson.supersaiyan.widget.SuperSaiyanScrollView
      android:id="@+id/scroll"
      android:layout_width="match_parent"
      android:layout_height="match_parent">

      <ListView
        android:id="@android:id/list"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:scrollbars="none"
        />

    </com.nolanlawson.supersaiyan.widget.SuperSaiyanScrollView>

(I like to set android:scrollbars="none", to remove the omnipresent gray scrollbars and stick with the “fast” blue scrollbars.)

Next, wrap your existing Adapter (e.g. an ArrayAdapter) in a SectionedListAdapter. The SectionedListAdapter uses a fluent “builder” pattern, similar to AlertDialog.Builder:

    SectionedListAdapter<MyCoolAdapter> adapter = 
        SectionedListAdapter.Builder.create(this, myCoolAdapter)
        .setSectionizer(new Sectionizer<MyCoolListItem>(){

          @Override
          public CharSequence toSection(MyCoolListItem item) {
            return item.toSection();
          }
        })
        .sortKeys()
        .sortValues()
        .build();

To include the SuperSaiyanScrollView in your Android app, simply follow these steps:

  1. Check out the code from GitHub:

    git clone https://github.com/nolanlawson/SuperSaiyanScrollView.git
    
  2. If you use Eclipse/ADT, go to Import -> Existing Android Code -> and choose the SuperSaiyanScrollView/library/ folder.
  3. If you use Proguard, add the following to your proguard.cfg:

    -keep class com.nolanlawson.supersaiyan.widget.** { *; }
    

For more information on importing library projects, read this section of the Android developer guide.

Examples

I’m going to walk through some short examples, which should demonstrate the simplicity and flexibility of the SuperSaiyanScrollView. The source code for these apps is included in the GitHub project, and you can download the APKs here:

Example #1: Countries

In this example, we have a list of countries, which we’d like to sort by continent. The finished app looks like this:

Example 1: Countries

We have a simple Country object:

public class Country {

  private String name;
  private String continent;

  /* getters and setters ... */
  
  @Override
  public String toString() {
    return name;
  }
}

We use a basic ArrayAdapter<Country> to display the countries:

ArrayAdapter<Country> adapter = new ArrayAdapter<Country>(
        this, 
        android.R.layout.simple_spinner_item, 
        countries);

Next, we wrap it in a SectionedListAdapter. In this case, we’d like to section countries by their continent, sort the continents by name, and sort countries by name:

    sectionedAdapter = 
        SectionedListAdapter.Builder.create(this, adapter)
        .setSectionizer(new Sectionizer<Country>(){

          @Override
          public CharSequence toSection(Country input) {
            return input.getContinent();
          }
        })
        .sortKeys()
        .sortValues(new Comparator<Country>() {
          
          public int compare(Country lhs, Country rhs) {
            return lhs.getName().compareTo(rhs.getName());
          }
        })
        .build();

A Sectionizer is a simple callback that provides a section name for the given list item. In your own code, this might be a HashMap lookup, a database query, or a simple getter (as in this example).

Notice also that the keys (i.e. the section titles) and the values (i.e. the list contents) can be sorted independently, or not sorted at all. By default, they’re sorted according to the input order.

Now, let’s try to change the sections dynamically! In the action bar, the user can switch between alphabetic sorting and continent sorting:

alphabetic sorting vs. continent sorting

To do so, we first get a reference to the SuperSaiyanScrollView:

SuperSaiyanScrollView superSaiyanScrollView = 
    (SuperSaiyanScrollView) findViewById(R.id.scroll);

Then, we call the following function whenever the user chooses alphabetic sorting:

  private void sortAz() {

    // use the built-in A-Z sectionizer
    sectionedAdapter.setSectionizer(
        Sectionizers.UsingFirstLetterOfToString);

    // refresh the adapter and scroll view
    sectionedAdapter.notifyDataSetChanged();
    superSaiyanScrollView.refresh();
  }

Notice that the SectionedListAdapter and SuperSaiyanScrollView need to be informed whenever their content changes.

Next, when the user switches back to continent sorting, we call this function:

  private void sortByContinent() {

    // use the by-continent sectionizer
    sectionedAdapter.setSectionizer(new Sectionizer<Country>(){

          @Override
          public CharSequence toSection(Country input) {
            return input.getContinent();
          }
        });

    // refresh the adapter and scroll view
    sectionedAdapter.notifyDataSetChanged();
    superSaiyanScrollView.refresh();
  }

Notice that you never need to call adapter.sort() or Collections.sort() yourself. The SectionedListAdapter handles everything. And it does so without ever modifying the underlying adapter, which means that view generation is lightning-fast.

Example #2: Pokémon

This example shows off some of the advanced functionality of the SuperSaiyanScrollView. We have three different sortings, the size of the overlay box changes to fit the text size, and we can dynamically hide both the overlays and the section titles.

alphabetic vs by-region sorting

First off, the size of the overlay can be configured in XML. In this example, we start off with a single-letter alphabetic sorting, so we want the overlays to be a bit smaller than normal.

Add a namespace to the root XML tag in your layout XML:

<RelativeLayout
  ...
  xmlns:myapp="http://schemas.android.com/apk/res/com.example.example1"
  ...
  >
</RelativeLayout>

Next, use values prefixed with ssjn_ to define the size of the overlay:

<com.nolanlawson.supersaiyan.widget.SuperSaiyanScrollView
  ...
  myapp:ssjn_overlaySizeScheme="normal">

  <ListView
    ...
    />

</com.nolanlawson.supersaiyan.widget.SuperSaiyanScrollView>

I include the built-in schemes small (for one letter), normal (for most use cases), and large and xlarge (for longer section titles). Section titles of up to two lines (separated by \n) are supported.

Small, normal, large, and xlarge overlays in my AMG Geneva app.

Small, normal, large, and xlarge overlays in my AMG Geneva app.

If you want, you can also manually specify the font size, width, height, and text color yourself:

<com.nolanlawson.supersaiyan.widget.SuperSaiyanScrollView
  ...
  myapp:ssjn_overlayWidth="400dp"
  myapp:ssjn_overlayHeight="200dp"
  myapp:ssjn_overlayTextSize="12sp"
  myapp:ssjn_overlayTextColor="@android:color/black" >

  <ListView
    ...
    />
</com.nolanlawson.supersaiyan.widget.SuperSaiyanScrollView>

Now, in the Java source, we have a PocketMonster object:

public class PocketMonster {

  private String uniqueId;
  private int nationalDexNumber;
  private String type1;
  private String type2;
  private String name;
  
  /* getters and setters */

  @Override
  public String toString() {
    return name;
  }
}

We have a simple PocketMonsterAdapter to define how the monsters are displayed in the list:

public class PocketMonsterAdapter 
    extends ArrayAdapter<PocketMonster> {
  
  // Constructors...
  
  @Override
  public View getView(int pos, View view, 
      ViewGroup parent) {
    
    PocketMonster monster = 
        (PocketMonster) getItem(pos);
    
    /* Create and style the view... */

    return view;
  }
}

We wrap this adapter in a SectionedListAdapter that, by default, sections and sorts everything alphabetically:

    adapter = SectionedListAdapter.Builder.create(this, subAdapter)
        .setSectionizer(Sectionizers.UsingFirstLetterOfToString)
        .sortKeys()
        .sortValues(new Comparator<PocketMonster>(){

          @Override
          public int compare(PocketMonster lhs, 
                PocketMonster rhs) {
            return lhs.getName().compareToIgnoreCase(
                rhs.getName());
          }})
        .build();

Notice that we call both sortKeys() and sortValues(), because we want both the section titles and the Pokémon to be ordered alphabetically. Since PocketMonster does not implement Comparable, we defined a custom Comparator.

Now let’s say we want to organize the Pokémon by region:

Pokémon sorted by region.

Some quick background: Pokémon are ordered by their “national ID,” an integer value that starts at 1 (Bulbasaur) and goes up to 718 (Zygarde). Every time Nintendo releases a new generation of Pokémon games, they add about 100 new monsters, set the game in a new “region,” and sell about a bazillion new Pokémon toys.

So basically, we can determine the regions from the Pokémon’s ID. We’ll define a new
Sectionizer, which is called when the user selects “sort by region”:

  private void sortByRegion() {
    adapter.setSectionizer(new Sectionizer<PocketMonster>() {

      @Override
      public CharSequence toSection(PocketMonster input) {
        int id = input.getNationalDexNumber();
        
        // Kanto region will appear first, followed 
        // by Johto, Hoenn, Sinnoh, Unova, and Kalos
        if (id <= 151) {
          return "Kanto (Generation 1)";
        } else if (id <= 251) {
          return "Johto (Generation 2)";
        } else if (id <= 386) {
          return "Hoenn (Generation 3)";
        } else if (id <= 493) {
          return "Sinnoh (Generation 4)";
        } else if (id <= 649) {
          return "Unova (Generation 5)";
        } else {
          return "Kalos (Generation 6)";
        }
      }
    });

    // uses the nat'l pokedex order, since 
    // that's the original input order
    adapter.setKeySorting(Sorting.InputOrder);
    adapter.setValueSorting(Sorting.InputOrder);
    scrollView.setOverlaySizeScheme(
        OverlaySizeScheme.Large);

    // refresh the adapter and scroll view
    adapter.notifyDataSetChanged();
    scrollView.refresh();
  }

Notice that we’ve changed the key and value sorting to Sorting.InputOrder, because now we want to order Pokémon by their national IDs, which was the order the data was read in. (A custom Comparator would have also done the trick.) Additionally, we’ve increased the size of the overlay to accommodate the longer section text.

Now, let’s say we want to organize Pokémon by type. Each Pokémon has at least one elemental type (such as “fire” or “water”), but some have two. Ideally we would like to list Pokémon in multiple categories, so they could appear multiple times in the list.

To do so, we will define a MultipleSectionizer instead of a regular Sectionizer:

  private void sortByType() {
    adapter.setMultipleSectionizer(
        new MultipleSectionizer<PocketMonster>() {

      @Override
      public Collection<? extends CharSequence> toSections(
          PocketMonster monster) {
        String type1 = monster.getType1();
        String type2 = monster.getType2();

        if (!TextUtils.isEmpty(type2)) { // two types
          return Arrays.asList(type1, type2);
        } else { // one type
          return Collections.singleton(type1);
        }
      }
    });
    adapter.setKeySorting(Sorting.Natural);
    adapter.setValueSorting(Sorting.InputOrder);
    scrollView.setOverlaySizeScheme(OverlaySizeScheme.Normal);

    // refresh the adapter and scroll view
    adapter.notifyDataSetChanged();
    scrollView.refresh();
  }

Notice that the key sorting has again changed, this time to Sorting.Natural, which simply sorts alphabetically. Value sorting has changed to Sorting.InputOrder, because we’ve decided to sort Pokémon by their national IDs.

This works as expected:

Pokémon sorted by type.

Notice that Charizard appears in both in the “Fire” and “Flying” sections, since he has two types.

This example app also shows how you can disable the section titles or section overlays, just in case you don’t like them. These values can also be set during the Builder chain, using hideSectionTitles() and hideSectionOverlays().

comparison of hiding overlays and hiding section titles

You can read the Java documentation for more information about customizing the SuperSaiyanScrollView and the SectionedListAdapter.

Summary

The SuperSaiyanScrollView is a cool new library, and you should be using it. File bugs on me ‘n’ stuff, if there are any missing features you’d like to have.

S3 bucket listing that’s easier on the eyes

Update: I learned that Shrub exists. It’s much nicer than what I hacked up in an hour!

This is just a quick one.

I host a lot of public files on an Amazon S3 bucket. It’s my main mechanism for publishing releases of my open-source software.

So I was amazed to discover recently that S3 doesn’t have an easy way to just… show all the files. Like, not even a basic directory listing, which you could easily get with an Apache server. Just nothing.

Directory listing in Apache

This is all I wanted.

Well, that’s not entirely true. There is this ancient sample code from Amazon, made in 2008, that I found frozen in ice. But it looks like crap.

Amazon's standard S3 directory listing

“$folder$”? Seriously?

So I made a better one, using Bootstrap for styling. Below is a screenshot, and here it is in action.

My pretty Bootstrap S3 index.html

Much better.

To use it, just download the index.html file from the GitHub page and drop it into the root of your public S3 bucket. That’s it!

As an aside, isn’t it awesome how easy web development has become, thanks to modern tools like Bootstrap, JQuery, and Handlebars? That file from Amazon used 174 lines of Javascript, whereas mine is only 99. Of course I have three external dependencies, but I use CDNs, so you probably won’t notice a difference in performance. How cool is that?

CouchDB doesn’t want to be your database. It wants to be your web site.

I’d like to talk to you today about Couch apps. No, not CouchApps. No, not necessarily CouchApps either. The phrase has been bandied around a lot, so it’s worth explaining what I mean: I’m talking about webapps that exclusively use CouchDB for their backend, whether or not they’re actually hosted within CouchDB and regardless of how they’re built.

Yes, this is a thing people are actually trying to do, and no, it’s not crazy. The purpose of this article is to explain why.

First off, some background: CouchDB is a NoSQL database (or key-value store, as the cool kids say) written in Erlang. It is probably the origin of this joke. Nobody who uses CouchDB cares that it is written in Erlang, though, because the big selling point is that you can interact with it using Javascript, JSON, and plain ol’ HTTP. It is “a database for the web,” the first of its kind.

CouchDB: it’s a database, right?

When I first started using CouchDB, I tried to treat it like any other database. I looked for connectors based on the language I was using: Ektorp for Java, AnyEvent::CouchDB for Perl, Nano for Node. And I used the web interface (bewilderingly called “Futon”) as I would a query browser – neat for debugging, but not much else. The fact that it ran in a web browser just kinda seemed like a gimmick.

Recently, though, when I was working on a Node app that didn’t go anywhere but was a fun diversion, I came across this quote by Couch apostle J. Chris Anderson:

Because CouchDB is a web server, you can serve applications directly [to] the browser without any middle tier. When I’m feeling punchy, I like to call the traditional application server stack “extra code to make CouchDB uglier and slower.”

Suddenly, I realized what CouchDB was all about.

No wait, CouchDB is a miracle

See, here I was, using client-side Javascript to talk to Express to talk to Node to talk to Nano to talk to Couch, and at each step I was converting parameter names from underscores to camel case (or whatever my petty hangups are), all the while introducing bugs as I tried to make each layer fit nicely with the next one. And I had a working web server right in front of me! CouchDB! Why not just call it directly, you fool?! (I shout at myself in hindsight.)

I think the reason a lot of developers, like myself, might have missed this epiphany is that we’re used to treating databases as, well, databases. Whether it’s MongoDB or MySQL or Oracle, you gotta have your JDBC connector for Java and perhaps an ORM layer or maybe you just give up on Hibernate and write all the database objects yourself, so half of your code is getters and setters, but that’s OK, because that’s how we abstract the database.

You see, you can’t just have your peanut butter and jelly sandwich! You need an interface between the bread and the peanut butter, and an abstraction layer between the peanut butter and the jelly, and don’t even get me started on the jelly and the bread! What, you want your bread to get soggy?

As a programmer, I’m so used to treating databases as this other, alien thing that needs to be handled with latex gloves, separately from my application code, that reaching for the nearest library has become a reflex.

But you don’t need that with CouchDB. Because… it’s just HTTP. Any extra layers just give you another API to learn.

CouchDB is the web done right

And in fact, CouchDB is better than HTTP, because CouchDB actually fulfills the promise of what RESTful services were supposed to be, instead of the kludges we’ve come to expect. Look! DELETE actually deletes things! POST isn’t just what you use when you need to send more data than a GET allows! And HEAD and PUT are actually useful, instead of just being trivia to impress your friends at dinner parties — “Oh, did you know that there are actually more HTTP commands than just GET and POST?” “Oh, how fascinating!”

You see, once you set aside your preconceived notion of what a database is supposed to be, you can actually get rid of all your fancy connectors and just use a standard HTTP library. (I like Requests for Python.) You can even use the network debugger in a browser window to see how CouchDB does everything. It’s all just AJAX!

And then, if you make it this far down the rabbit hole, you might notice that CouchDB actually has a user authentication database, with password hashing. You might also notice that it’s even got roles and privileges and administrator controls. And that’s when you realize, with fascinated horror, the most insidious thing about CouchDB:

CouchDB doesn’t want to be your database; it wants to be your web site.

And finally, this is where we come back to the subject of Couch apps. A Couch app is just a pure HTML/CSS/Javascript application, with only CouchDB as its backend, and this is the intended use case for CouchDB.

Now, think about what this proposition means to you as a developer. The web is moving more and more towards rich, client-side applications — we’ve had jQuery for years, and now we even have MVC with platforms like Ember, Knockout, and AngularJS. If CouchDB does user authentication (it’s got a “signup” button right on the home page, for crying out loud), paging, indexing, full-text search, geo data, and it all speaks HTTP, well… what does that actually leave us to do on the server?

Take a long look in the mirror, and really ask yourself! And yes, for those of you who do machine learning and scientific computing and business intelligence, I can already see you raising your hands, but for the rest of us who get paid to write Twitter clones, the answer is: not much. Your average CRUD app can magically transform into a PGPD app (PUT, GET, POST, DELETE), you can throw it up on CouchDB with some nice HTML and CSS to style it, and be at your local brewpub by 3. Or maybe you could just send the default Futon interface to the client and tell them you wrote it.

Futon interface in CouchDB

“See, it’s a collaborative document editor, and the dude on the Couch is a lazy writer…”

Now, this is the dream. And CouchDB, as it stands in 2013, actually gets us pretty damn far toward that dream. The app I’m releasing this week, Ultimate Crossword, is a testament to that. It’s a pure Couch app that only cheats by using Solr for full-text search (because I was too lazy to learn the Lucene plugin). It’s got user accounts, data aggregation, and even continuous syncing between the client and server thanks to the wonderful PouchDB.

Building this site gave me a lot of insight into what’s possible with a Couch app. However, I also got a reality check about where CouchDB still falls short of achieving the dream. I’ve got four big complaints:

1) No per-document read privileges

This is a big one. CouchDB has three basic security modes:

  • Everyone can do everything.
  • Some people can write (some documents), everyone can read (all documents).
  • Some people can write (some documents), some people can read (all documents).

If you want to give users exclusive read access to certain documents, you have to create a separate database for each user. And unfortunately, CouchDB has no feature to do this automatically. So you need a process on the server with administrative privileges to do it, breaking the pure “Couch app” ideal. Then, if you want to aggregate the data, you actually need another process to sync to a separate database, and… well, it just gets messy. I’m strongly rooting for this feature to show up in a future CouchDB release.

2) No password recovery.

This is a feature that users have come to expect from modern web sites. And despite all its security flaws (in that it makes your email a single point of failure), it seems here to stay.

Now, CouchDB can store arbitrary data in the users table (like email addresses), and you can even do custom validation. But for the whole “give us your email, and we’ll send you a new password” thing, you’re on your own.

On the bright side, the passwords are all salted and PBKDF2-hashed, so no attacker has much to gain from cracking your Couch.

3) No database migration.

This is a big one for me, although I wonder if I’m the only one. Since my early days of Java development, I’ve appreciated having Liquibase so I could track my database schema changes in version control.

In theory, CouchDB should be ideal for something like this, since it versions everything, and even its views (aka indexes) are their own documents. But I haven’t found a good recipe for managing this yet. For the time being, I just keep a series of Python scripts that create the databases.

4) Views are not indexes, and documents are not tables.

One of the nice things about SQL databases as a development paradigm is the flexibility of the SQL language itself. Decided you wanna sort by dogsLastName instead of favoritePokemon? No problem, we’ll just add an index. Too much data getting sent across the wire? No big deal, we’ll just SELECT the fields we need, instead of SELECT(*).

In CouchDB, you can’t do a WHERE and you can’t just SELECT the fields you want. Any query that’s not simply fetching a whole document by its ID requires a view, and those are costly to create. I’ve worked with Couch databases containing millions of documents, and rebuilding a view would often take days. I’d have a coworker ask me to add a new filter criterion for a view, and on Friday I’d say, “Okay, it’ll be ready by Monday.” For the Ultimate Crossword app, I stupidly decided to use CouchDB to crunch the data itself, and I ended up needing five separate Couch servers running on solid state drives in order to process it in in a reasonable amount of time. (CouchDB is best thought of as a single-process application. It’s append-only, so it uses one process per database file.)

Also, the fact that you can’t SELECT arbitrary fields means you need to start thinking about how much data you want to send over the wire with each document, and how to threshold it. I found myself structuring my database into a summary/detail format early on, and modeling the documents very tightly to the user interface, in ways that just made me feel icky.

Database purists, of course, would say that this is where the latex gloves are supposed to come out. But I think that if CouchDB simply had a better system for managing migrations (see #3) and/or faster view creation, this would be a non-issue. I’d also love it if the output of a view could be put into its own database, so I could have endlessly kaleidoscoping views of my data. One more for the wishlist!

Conclusion

Despite these drawbacks, I still think CouchDB has a lot of potential to revolutionize the way people write webapps. I certainly still plan to use it for quick hacking (hell, the crossword app only took me ten days to write), and Couch’s append-only design means I’ll never have to worry about my data getting corrupted. (It’s been proudly touted as “the Honda Accord of databases.”)

But for all its developers’ humility, CouchDB is a really exciting technology. When you step back and look at it, it’s a daring, crazy proposition, a bold statement about how awesome web development would be if we could just let it be the web. It’s a raving streetside lunatic, grabbing random people by the shoulders and screaming at them with frantic urgency: “We don’t need the server anymore! We only need the database! The database is the server!”

In short, CouchDB is an expression of an ideal, a fantastical tale of science fiction told by wide-eyed dreamers. And if there’s one truth about wide-eyed dreamers, it’s this: with hindsight, their predictions either seem delusional, or inevitable.

(Psssst! Go check out my Ultimate Crossword app! It’ll make you feel bad about your user authentication!)

Update: I decided to remove the CouchDB user authentication from the Ultimate Crossword app (I realized it was irresponsible to let people collaboratively “solve” the puzzle), but it’s still a pure Couch app!