Web sockets with Socket.io, Node.js, and Nginx: port 80 considered harmful

TL;DR: web sockets are more widely supported on port 443 (via SSL) than port 80. Check that your ISP actually supports the port you’re using at WebSocketsTest.com.

I just wasted a good half-day trying to debug a problem with web sockets in the Socket.io plugin for Node.js running on Nginx 1.4.1.

Running the Node app locally on port 3000, everything worked swell. However, as soon as I deployed it to my production server, I was seeing about a 10-second startup in my app. This occurred in four different browsers – Chrome, Chromium, Firefox, and Safari.

Each one was reporting a 502 Bad Gateway response from the server. So since the client assumed web sockets were not being supported, Socket.io was falling back on XHR polling, which is less speedy.

The Node logs said:

warn: websocket connection invalid
info: transport end (undefined)

The Nginx logs said:

2013/05/21 10:41:00 [error] 6117#0: *8 upstream prematurely closed 
    connection while reading response header from upstream [...]

Neither was very helpful. So after reading several forums and blog posts online, and adding lots of tweaks and hacks that didn’t work, my Nginx site configuration bloated up to this:

# node.js running locally on port 3000
upstream node {
    server 127.0.0.1:3000;
    keepalive 256; # not necessary
}


# the public nginx server instance running on port 80
server {
    listen 80;
    server_name <mysite.com>; # domain of my site
    access_log /var/log/nginx/<mysite>.access.log;
    error_log /var/log/nginx/<mysite>.error.log;
    
    # supposedly prevents 502 bad gateway error;
    # ultimately not necessary in my case
    large_client_header_buffers 8 32k;

    # run the app on the root directory
    location / {

        # the following is required for WebSockets
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;
 
        # supposedly prevents 502 bad gateway error;
        # ultimately not necessary in my case
        proxy_buffers 8 32k;
        proxy_buffer_size 64k;
        
        # the following is required
        proxy_pass http://node;
        proxy_redirect off;
 
        # the following is required as well for WebSockets
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        tcp_nodelay on; # not necessary
    }
 }

Since web sockets worked fine when running Node locally, I assumed it was a problem with Nginx. What eventually clued me in was that even running Node directly on port 80 yielded the same error.

Finally I realized that the problem was with my Internet provider themselves – they were disallowing web sockets connections on port 80! WebSocketsTest.com turned out to be an invaluable resource in debugging this.

Incidentally, it would have worked if I had used SSL on port 443. According to WebSocketsTest’s aggregate data, port 443 is supported about 89% of the time, compared to 78% for port 80.

I tested a few public Wifi spots around Geneva and can confirm that this figure seems about right. I would be curious to know if support varies by geographic region, but WebSocketsTest doesn’t publish that data.

Lesson #1: even if you’re using a modern browser, your Internet provider may not be up to snuff. Check it first, before you wreck your brain!

Lesson #2: If you plan on using web sockets in a production environment, SSL is apparently the way to go. Not only is it secure, but it’s also better supported (as of 2013).

12 responses to this post.

  1. Hi Nolan,
    We also have the same issue! We are running node js/socket.io on nginx 1.4.1 but we are using only http (port 80), request are always falling into XHR polling. When you make it port 443 request, does it solve your issue?

    Reply

    • Hi there. As I said, it depends on the ISP of your client, but you should have better luck with SSL+443. Use WebSocketsTest to see what your client accepts.

      Reply

  2. You saved my life man! Deploying a socket.io app i was going mad…

    Reply

  3. I think we’ve got the issue too on a live app. So many “warn: websocket connection invalid”. This is not happening at all locally, hard to track what’s going on. We are using nodejitsu.

    What do you think of having the server running on a custom port?

    io.connect(‘http://www.myserver.com:3000’);

    Would that help with the ISP?

    Reply

    • My guess is that, if an ISP is blocking port 80, they’ll also block any non-standard port. (I’d be happy to be corrected, though!)

      For the time being, it seems that the best solution is just to use port 443 everywhere, in which case it will still only work 89% of the time. Support for web sockets is not great in 2013.

      Reply

      • Is there any reason that could create all these “warn: websocket connection invalid” beside ISP blocking web sockets? I’m completely unable to reproduce this problem everywhere I tried. Kind of frustrating, I can’t find solutions if I can’t reproduce it myself. I only see these messages in the remote server log, that’s about as far as I went…

      • I’m not sure; I didn’t write the software. In any case, this is the way that socket.io is supposed work: try method #1, warn if invalid, try method #2, rinse, repeat. So I imagine you shouldn’t worry too much about the warnings.

        As for reproducing this problem “in the wild,” you’re probably out of luck on that one, unless you want to try to configure your local network to block certain ports.

  4. If I configure the server script as you suggested, so what I’m going to do on the web side. Can I used something like ws://mywebsite.com:8080 to connect to the server.

    Thank you,

    Reply

  5. pandora sale do you want to add new ones in next few weeks
    zanotti homme 2014 http://www.ccpfrs.fr/homme/

    Reply

  6. Posted by Philipp Richter on August 11, 2015 at 2:34 AM

    Thank you so much! The nginx configuration was driving me insane until I stumbled upon your “set_header” config options which ended up making it all work!

    Reply

  7. I know exactly how you must have felt, I’ve spent about 4 days trying to solve the problem with nginx before directly running my flask server on port 80. I moved to SSL as a work around. Such a headache! Thanks for the websocketstest website, that’s great.

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: