Category: Linux/Unix

WebSockets from Ember, Rails, Nginx with AWS ElastiCache Redis

When we finally upgraded our Rails application to v5, plans were put in place to take advantage of its WebSockets capabilities.

Locally on development machines, things seem to chug along smoothly until it all broke down on our staging stack.

Here’s the basic setup:

* Rails 5.0.x
* Ember 2.x (using Ember Cable for WebSockets)
* Puma
* Nginx (for reverse proxying Ember calls to Rails + serving Ember app)
* AWS ElastiCache Redis
* Postgres hosted on AWS RDS
* EC2 instance + ELB + custom security group… etc.
* Entire stack is orchestrated with Chef running on AWS OpsWorks

I didn’t think the set up was anything out of the ordinary per se. The only exception is probably my choice of using ElastiCache instead of having an EC2 server running our own Redis instance (or having it hosted at Redis Labs or something). But other than that, it’s pretty vanilla.

Immediately we rant into trouble having Ember making a proper connection. We kept seeing this error:

WebSocket connection to 'ws://staging.domain.name/cable' failed: Connection closed before receiving a handshake response

First thing first, we had to make sure our /cable endpoint was proxied properly so that Rails can answer the calls. So in our main Nginx config file, we added:

1
2
3
4
5
6
7
8
9
10
location /cable {
    proxy_set_header   X-Real-IP $remote_addr;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   Host $http_host;
    proxy_set_header   Upgrade $http_upgrade;
    proxy_set_header   Connection "Upgrade";
    proxy_redirect     off;
    proxy_http_version 1.1;
    proxy_pass         http://app_server;
}

It still didn’t quite work.

1
2
3
4
5
I, [2017-03-22T18:46:39.166623 #4175]  INFO -- : Started GET "/cable" for 127.0.0.1 at 2017-03-22 18:46:39 +0000
I, [2017-03-22T18:46:39.167699 #4175]  INFO -- : Started GET "/cable/" [WebSocket] for 127.0.0.1 at 2017-03-22 18:46:39 +0000
E, [2017-03-22T18:46:39.167837 #4175] ERROR -- : Request origin not allowed: https://staging.domain.name
E, [2017-03-22T18:46:39.167950 #4175] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE:)
I, [2017-03-22T18:46:39.168064 #4175]  INFO -- : Finished "/cable/" [WebSocket] for 127.0.0.1 at 2017-03-22 18:46:39 +0000

Obviously we need to specify in Rails that our Staging subdomain, along with its https protocol should be allowed. So we added that to our Rails /config/environments/staging.rb file.

config.action_cable.allowed_request_origins = %w(https://staging.domain.name staging.domain.name)

Once we squared that away, we were still left with that dreadful error on line #4 above.

Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE:)

Somehow, the $http_upgrade variable in Nginx is simply not being set despite browser’s request. As it turns out, WebSocket traffic isn’t quite HTTP/HTTPS. The devil is in the details of the AWS ELB (load balancer).

I simply swapped out HTTPS with SSL. And BOOM! Like that, it worked:

Successfully upgraded to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: websocket)

Now connection was being made properly. Except for this one little thing: The browser had to keep reinitialize the connection every few seconds. And we noticed the UI simply wasn’t being updated despite changes to data. Sigh…

Next stop was to test if the EC2 instance itself was actually talking to AWS ElastiCache. There are a couple of ways to try this:

In Rails:

1
2
3
4
5
irb(main):001:0> redis = Redis.new(:host => 'my-elasticache-instance-identifier.0001.usw2.cache.amazonaws.com', :port => 6379)
=> #< redis client v3.3.3 for redis://my-elasticache-instance-identifier.0001.usw2.cache.amazonaws.com:6379/0>
irb(main):002:0> redis.ping
Redis::CannoConnectError: Error connecting to Redis on my-elasticache-instance-identifier.001.usw2.cache.amazonaws.com:6379 (Redis::TimeoutError)
...

Using Telnet:

1
2
3
$ telnet my-elasticache-instance-identifier.0001.usw2.cache.amazonaws.com 6379
Trying /ip ADDRESS/...
# eventually times out

OMFG. WHAT ELSE IS WRONG NOW??!!

As it turns out, the devil, again, is in the details. This time, it’s in AWS Security Groups.

AWS resources are only allowed to talk to each other if said resources are within the same security groups (among other things). In this case, AWS ElastiCache has a default security group, which we’re not allowed to change (not to my knowledge anyway). So the trick was to make sure that our OpsWorks Layers also have been assigned the default security group. Now, OpsWorks documentation says that you must reboot each instance within the layer for the new security group settings to kick in. But the truth is I had to spawn brand new instances for the new security group to stick. But once that was done, everything finally started to gel:

1
2
3
4
5
6
irb(main):001:0> redis = Redis.new(:host => 'my-elasticache-instance-identifier.0001.usw2.cache.amazonaws.com', :port => 6379)
=> #<redis client v3.3.3 for redis://my-elasticache-instance-identifier.0001.usw2.cache.amazonaws.com:6379/0>
irb(main):002:0> redis.ping
=> "PONG"
...
</redis>

I hate technology sometimes.

References:
* https://blog.jverkamp.com/2015/07/20/configuring-websockets-behind-an-aws-elb/ (cached in PDF)

Changing AWS RDS PostreSQL database name


Working with AWS RDS isn’t always the easiest. That’s the limitations that come with hosted solutions. /shrugs/

For example, you can’t change master username once an instance has been spawned, not even on an instance restored from a snapshot.

But I did figure out how to change a Postgres RDS instance’s database name. To do this, we’ll need terminal connection to the database:

1
2
$ psql -U james_holden -h my-database-identifier.abc123.us-west-2.rds.amazonaws.com -d donnager
$ password for james_holden: # provide your db password here

At this point, if you attempted to change the database name, Postgres will complain immediately:

1
2
donnager=> ALTER DATABASE donnager RENAME TO rocinate;
donnager=> ERROR:  current database cannot be renamed

However, if you get out of the connected db, switch to the standard “postgres” db, it’s all fair game:

1
2
3
4
5
6
7
donnager=> \c postgres
psql (9.5.5, server 9.5.4)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
You are now connected to database "postgres" as user "james_holden".
postgres=> ALTER DATABASE donnager RENAME TO rocinate;
ALTER DATABASE
postgres=> \q

Peace is restored again.

Temporarily Stopping McAfee AntiMalware Agent on the Mac

McAfee’s AntiMalware product can be a real resource hog for some people. Fortunately there’s a way to simply disable it via the command line. It can then be easily re-enabled the same way.

I made an alias in my profile to take care of that:

alias kill-mcafee='sudo /usr/local/McAfee/AntiMalware/VSControl stopoas'

Re-enabling it is just as easy:

alias unkill-mcafee='sudo /usr/local/McAfee/AntiMalware/VSControl startoas'

Deploying Chef without a Chef Server

chef-logo

Chef is a great tool. I really love it. But to take full advantage of it, you need a Chef server (either build one yourself or have it hosted with Opscode (which we did at my last job). For small businesses and/or personal server(s) of very small scale (I’d suggest up to 3 to 5 servers at most), a nifty tool like LittleChef can really be quite useful. But if servers you manage ever grows more than a handful, I’d highly recommend hosted service like Chef to keep your sanity.

Minor Server Hiccups

After we moved all the data from our old Texas ServerBeach machine to the new one in L.A., the new server kept on rebooting itself sporadically. Loz and I monitored and went through the logs for almost two days but couldn’t find anything wrong with it (especially after Loz installed a couple of nice monitoring software). Let me tell ya, having a server that reboots itself whenever it wanted to was stressful.

Good thing the tech support guys at ServerBeach did their thing and swapped out the box for us within an hour of submitting a support ticket. And then 40 minutes after that, we were back in business. I gotta say, I agree with Lawrence, ServerBeach has been pretty good with the kind of service we’ve received so far!

On a similar note, I had a client whose dedicated server was (and I assume still is) hosted with EV1 Servers (now part of The Planet, a company Loz and I were thinking about using but went with ServerBeach instead). And their service was just awful. I received more support from peer users in their discussion board than from EV1’s tech support (all I got was a templated response which offended me quite a bit for the kind of money my client was paying them).

Life is good again.

VNC on Fedora Core 6

Setting up VNC on a Linux box is always such a hassle. But I had to do it again today so that I don’t have to deal with two sets of mic and keyboards when I am testing my work in VMWare.

Thank goodness for this article that I saved a bunch of time having to figure things out myself. Maybe in the [always] good old college days, I’d have done that. But now time is of utmost importance with an active child in the house and all…

The only hiccups I got was that the above setup failed to discuss the issue with firewall and iptables. To allow outside access (even within the same home network), a port must be open to allow the traffic to flow through. There are a couple of ways to do that:

Adding a rule through a GUI in Gnome:

1. Go to

1
System -> Administration -> Security Level and Firewall

2. (enter the appropriate password when prompted)
3. Under the

1
Firewall Options

tab, click on the white arrow at the bottom that’s labeled

1
Other ports

4. Click on

1
Add

and enter the port number you are allowing access; in my case, it was

1
5902

for display number 2. (Leave the

1
protocol

at default, which should be

1
tcp

5. Click

1
OK

all the way out, and iptables should have been restarted with the rule in place

A geekier way to do it is through command prompt:

1.

1
sudo vi /etc/sysconfig/iptables

2. Add the following line to the rule:

1
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 5902 -j ACCEPT

3. Restart iptables by issuing:

1
sudo /etc/rc.d/init.d/iptables restart

Another thing that many Linux novice (like myself) don’t quite grasp is the fact that Linux’s GUI is not at all tied to the operating system. You can have Gnome, KDE, Flux or Blackbox as GUI options installed on the same OS. And you can switch around as you please upon setting the preferred desktop and log/in again. When you are viewing the remote system through VNC, you can do exactly the same thing — you can define what type of GUI you want to see as you launch your preferred VNC viewer client:

Edit

1
/home/_vnc_user_name_/.vnc/xstartup

as such:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
#!/bin/sh

# Uncomment the following two lines for normal desktop:
# unset SESSION_MANAGER
# exec /etc/X11/xinit/xinitrc

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
#xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" -e ./menu &
#twm &
#fluxbox &
gnome-session &
#startkde &

Noticed I commented out (#) some desktop GUI options at the bottom except Gnome. Basically you can use any one of them anytime as long as you restart the

1
vncserver

after you’ve made the changes.

Upgrading to Fedora Core 6

I finally got around to upgrading my Dell to Fedora Core 6 using yum. My last attempt failed pretty miserably mainly because of some stupid package dependencies I couldn’t figure out how to resolve (something to do with

1
initscripts

, which was kind of what happened in my attempt to upgrade from FC4 to FC5). So instead of dealing with it, I simply shut it up… Here are the steps:

1. me@localhost$

1
rpm -ev fedora-release --nodeps

2. me@localhost$

1
rpm -Uvh [url-to-fc6-release-notes]

3. me@localhost$

1
rpm -Uvh [url-to-fc6-release]

4. me@localhost$

1
yum clean all

5. me@localhost$

1
yum -y update

By step 5, everything should’ve gone flawlessly, except, again, Fedora threw a fit on dependency problems such as

1
bg5ps

,

1
gtkhtml

and a couple of other packages. Again, instead of dealing with resolving them, I just removed them:

1. me@localhost$

1
yum remove bg5ps

2. …. (repeat step 1 until all problematic dependencies are removed)

Then everything was good. And as it turns out, the new kernel that came with FC6 (2.6.18-1.2798.fc6-i686) is capable of handling dual cpu systems, unlike kernels before that where I had to specifically update multi-processor capable kernels (kernel-smp). I wasted about 20 minutes trying to find/update something that never existed!

Finally, after reboot, I got my nice FC6 all up and running. Life was good except now VMWare refuses to launch. So I had to reinstall it because of the new kernel I installed. Unfortunately, VMWare couldn’t locate the new kernel’s c header library (usually installed at

1
/usr/src/`uname -r`/include

). But it wasn’t there!!! Even though I had the latest kernel, its c header library still could not be found! After some research, I finally was able to resolve that by force installing

1
kernel

and

1
kernel-devel

rpm files:

me@localhost$

1
 rpm -Uvh --force [url-to-kernel-rpm-package]

Issue the command above for both the current

1
kernel

and

1
kernel-devel

Once VMWare got its kernel c header library, it was happy. And life was good again….

My next project with that machine is to upgrade its CPUs with a couple of nice Pentium 3 Slot 1 modules at the maximum clock speed the machine can handle (which is at around 1.1Ghz — fast compared to 450mhz I currently have). These types of CPUs are pretty hard to come by because Intel abandoned Slot 1 architecture a long time ago.

Fedora Core 6 upgrade made possible by this discussion thread at fedoraforum.org. VMWare troubleshooting and resolution made possible by this discussion thread on VMWare’s community site and this discussion thread on fedoraforum.org.

More Geek Talk

Mike Neely discovered something pretty freaking awesome: It’s called Meebo. It’s basically Adiumx (or Trillian for you Windowz folks, gaim for you Linux dorks) except it’s web based. It supports quite a bit of stuff that you are accustomed to using desktop apps.

Speaking of apps, Neely revealed the other day that he’s doing some cocoa programming! I am so envious of him having the initiative and drive to pick up cocoa/objective-c. This reminds me that I ought to get back into desktop programming!

While I am on the topic of “desktop” programming, I tried upgrading my Fedora Core 5 to the latest Core 6 last night but failed. I guess it’s too new that various issues aren’t widely discussed yet (bless Google). I’ll probably give it another try another time. Strangely though, the audio in FC5 just magically started working all by itself — I spent days downloading and compiling various drivers for it to no avail. And that piece of sh*t just decided to work all of a sudden? WTF?! This is yet another reason Linux is far from being ready for mainstream consumption — lack of usability for everyday-users.

Upgrading to Fedora Core 5

After having successsfully upgraded my spared Dell box to Fedora Core 5, I messed up the boot partition and had to do it all over again. So I decided I might as well document it here since the processs wasn’t quite as smooth as one’d hope.

UPGRADING FEDORA CORE 4 TO CORE 5 USING YUM
1. First, make sure

1
yum

is up to date. If you are going to upgrade your system with it, you might as well make sure the tool is up to pars.

me@localhost$

1
yum -y upgrade yum

This is going to download a bunch of other stuff other than

1
yum

. So be patient.

2a. Next, make sure sure your

1
kernel

is up to date as well. Or else when you upgrade to FC5, it will throw a bunch of errors like this:

1
2
Error: Package initscripts needs kernel < 2.6.12, this is not available.
  Error: Package kudzu needs kernel < 2.6.13, this is not available.

So here we go:

me@localhost$

1
 yum -y upgrade kernel

For multi-CPU systems, do the following instead:
me@localhost$

1
yum -y upgrade kernel-smp

2b. Just to be on the safe side, some packages may need to be removed before upgrading. This script should tell you what needs to go:

me@localhost$

1
perl -ne 'print "$1\n" if ((/Error: Missing Dependency:.*is needed by package (.*)$/) || (/Error: Package (.*?) needs.*, this is not available./))' /tmp/yum_upgrade | sort | uniq

3. Reboot. Make sure you boot into the latest

1
kernel

or else you are going to have the same problems as I mentioned before.

4a. Now delete any old

1
kernels

you have still on your system. First, let’s see what’s there:

me@localhost$

1
rpm -q kernel kernel-smp kernel-devel kernel-smp-devel | sort

4b. Delete old

1
kernels

:

me@localhost$

1
rpm -e kernel-version-number

I read somewhere that you should delete

1
kernels

by using

1
rpm

since it also tidies up your bootloader file for you.

Optional: Update

1
rpm

packages:

me@localhost$

1
rpm --rebuilddb

5. Now we are ready to get the FC5 upgrade package:

me@localhost$

1
2
3
rpm -Uvh <a href="http://download.fedora.redhat.com/pub/fedora/linux/core/5/i386/os/Fedora/RPMS/fedora-release-5-5.noarch.rpm" target="_blank">http://download.fedora
.redhat.com/pub/fedora/linux/core/5/i386/os
/Fedora/RPMS/fedora-release-5-5.noarch.rpm</a> (this code wraps for cosmetic reasons. But you should copy/paste this as if it's one unbroken line.)

6. Finally. Show time. Let’s upgrade this puppy…

me@localhost$

1
yum -y upgrade

Now sit tight and wait. This could take a while depending on your Internet connection. It took me roughly 5-7 hours on a moderately fast DSL.

7. Once everything is downloaded and installed, reboot.

8. It’s probably a good idea to keep your old

1
kernels

. But I deleted my FC4 kernels.

INSTALLING VMWARE 5.5.1.x
Nothing is EVER easy on Linux. The same assumption (and proof) goes to updating VMWare after the FC5 upgrade. VMWare complained about not being able to find “the directory of C header files”. To resolve this problem, you must download the

1
<a href="http://ftp.cvut.cz/vmware/vmware-any-any-update101.tar.gz" target="_blank">vmware-any-any-update101.tar.gz</a>

and run it. Everything will hum just fine after this update is applied to VMWare.

Much thanks to: brandonhutchinson.com and LinuxSky.com (Simplified Chinese only)

The Blast from the Past

Being in the web development business, it’s plain impossible to develop sites without having to check your work in Internet Explorer for bugs (not that my code is buggy, but that Internet Explorer rendering engines are simply not compliant to standards). So I finally broke down, booted up my Fedora Core 4 Linux on my 6-year-old Dell (running dual Pentium 3 @ 450Mhz!), fussed with VMWare and finally managed to install Windows 2000 and Windows XP.

Oh, how I dreaded having to touch Windows again having stayed away from it for almost a year and a half. Just the installation of Windows 2000 felt like playing with Mac OS 9 again… So primitive and ancient. I especially “loved” the restarts after each service pack, patches to service packs and then hotfixes to the patches that were originally released to fix the service packs… etc. I wasted an entire evening just installing and patching Windows in VMWare. Nothing says “I love my life” like wasting time installing Windows OS!

The upside, though, is that I can perhaps install Internet Explorer 7 beta to see what the fuss is all about (yoohoo! Tabbed browsing… only 2 years late!)…. And then maybe if I feel adventurous enough, I’ll even download Windows Vista Public Beta and try to install it in VMWare (which I doubt will even install given that both the host and guest hardware are pretty damn old… even if by some miracle Vista installs, it’d probably take a few hours just to boot up… ha!).

I hope this is the last time I’ll have to deal with installing Windows ever again…

Why Linux Will Never Make It to the Mainstream

To get ready for a Linux class I am taking, I have to put Linux back on the old Dell I have abandoned ever since moving Grace to the Mac. I decided to try the latest flavors out there — Ubuntu 5.10 and Fedora Core 4.

Linux has been great as a server environment for me. But I have been frustrated with it since the Redhat 8.0 days (precursor to the current Fedora project). Nothing has been intuitive in almost every aspect of the Linux wold. Everything from application naming convention (WTF is “Gimp” if you don’t Google it?), install, setup to the simplest things such as application upgrades. I have picked up quite a bit of Linux just from Googling and learning everything myself with occassional Q&A sessions with Kyung. But it never seems to get any easier.

Ubuntu
Start with Ubuntu, the latest and the greatest and the most recent “king of ease of use”. Sure enough, the entire installation and setup process was seamlessly fast — took less than 3 hours with everything installed and done. It was as easy to install as the Windows installation process, I thought. And all it took was one 700MB disc. Easy! I liked the experience so far.

But it sucked when it came to upgrading Firefox from the default 1.07 to the latest 1.5.0.1. I had to download Firefox, extract the file to some other directory, make symbolic links in one directory while changing other files and variables to make everything happy. Screw that. I tried using apt-get to update the software automatically, but then I was too lazy to figure out where to fix the repository list so that it’d get the latest Firefox, so I ditched Ubuntu.

Fedora Core 4
On with Fedora. I have used it quite a bit before. But that didn’t make the experience any easier. Upgrading Firefox was a bit simpler. But I still had to go through similar hoops. The main stumbling block on Fedora was trying to get VNC server to work properly (so that I can free up the monitor for my laptop). To demonstrate how much crap I had to do to make it work, here’s what I did…

1. Install VNC server by issuing this command

1
yum -y install vncserver

2. Check to see if vnc server has the right runlevel:

1
/sbin/chkconfig --list | grep vnc

;
but I got this:

1
vncserver 0: off 1: off 2: off 3: off 4: off 5: off 6: off

(hint: runlevel 5 should be

1
on

)
3. Change runlevel:

1
/sbin/chkconfig --level 5 vncserver

4. In

1
~/.vnc/xstartup

file, uncomment these two lines:

1
2
unset SESSION_MANAGER
exec /etc/X11/xinit/xinitrc

5. From the menu, go to “Applications > Preferences > Remote Desktop” and change options to allow remote users to control the workstation
6. Change rules in the iptables file (Linux version of a firewall) in

1
/etc/sysconfig/iptables

by adding a new rule:

1
-A INPUT -p tcp -m tcp --dport 5900 -j ACCEPT

(5900 being the port I need to open to allow VNC to accept communications through that port)
7. Restart iptables:

1
sudo /etc/init.d/iptables restart

8. Start VNC server:

1
sudo /etc/init.d/vncserver start

Yeah sure, I have everything in one place now. It sure took me some work and digging to put all the pieces together. Can an average Joe get VNC to work not knowing all this? The answer is obviouly a NO. And I rest my case… Unless things drastically change in simplifying the environment, it will never be in the mainstream.

UPDATE 06/14/2006: More info on how to set screen resolutions for vncviewer.