Strangeness
I was recently wrestling with an obscure JDBC bug in Sequel. I was seeing an error connecting to the PostgreSQL database but it was being rescued by the Sequel JDBC adapter and hidden a little bit. (If you’re curious to see the fix it’s over on Github). Anyway, the original error led me to believe that the JDBC adapter for postgres wasn’t being loaded at all.
What to do?
What else? Get on the #jruby IRC channel on Freenode. Seriously, when you’re running JRuby and you come across a problem you go to there. Charles, Nick, and Tom (JRuby core) are always lurking around and very willing to help. Charles told me to look at $” and see what was loaded.
$”
A magic-pre-defined variable in Ruby (and JRuby) – see what I did there with the title? According to zenspider’s QuickRef:
$" The array contains the module names loaded by require.
Huh. Okay, cool. As a bonus JRuby throws the loaded jars in there as well. ( I really should learn all of these, come to think it, so should you. )
So what
You take a look at that array and you’ll see all of the modules names currently loaded. Good stuff. Here is the part from the project that helped me resolve my issue:
jdbc/postgres.rb
postgresql-8.4-702.jdbc4.jar
arjdbc/postgresql/connection_methods.rb
active_record/connection_adapters/postgresql_adapter.rb
arjdbc/postgresql/adapter.rb
I was able to see that the postgresql jar was loaded along with the supporting cast of characters. This led me to the #sequel IRC channel where Jeremy Evans and I were able to figure this little guy out. Trying to troubleshoot a potential jar loading problem in JRuby? $” is your friend. (and so is IRC it seems!)
A word to the wise: pg 0.10.0 doesn’t want to compile on OpenSolaris.
The Stack
One of our production servers is running OpenSolaris version OpenSolaris 2008.11 snv_101b_rc2 X86, Ruby 1.8.7 (2010-08-16 patchlevel 302) [i386-solaris2.11], and PostgreSQL 8.3.4. We use Capistrano for deployment with some bundler goodness thrown in the mix.
The Problem
Here’s what we see when bundler tries to build the gem:
ERROR: Failed to build gem native extension. (Gem::Installer::ExtensionBuildError)
/usr/local/rubies/ruby-1.8.7-p302/bin/ruby extconf.rb
checking for pg_config... yes
checking for libpq-fe.h... yes
checking for libpq/libpq-fs.h... yes
checking for PQconnectdb() in -lpq... no
checking for PQconnectdb() in -llibpq... no
checking for PQconnectdb() in -lms/libpq... no
Can't find the PostgreSQL client library (libpq)
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers. Check the mkmf.log file for more
details. You may need configuration options.
Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/usr/local/rubies/ruby-1.8.7-p302/bin/ruby
--with-pg
--without-pg
--with-pg-config
--without-pg-config
--with-pg-dir
--without-pg-dir
--with-pg-include
--without-pg-include=${pg-dir}/include
--with-pg-lib
--without-pg-lib=${pg-dir}/lib
--with-pqlib
--without-pqlib
--with-libpqlib
--without-libpqlib
--with-ms/libpqlib
--without-ms/libpqlib
Oh yeah! Good stuff right there.
The Investigation
At first, I thought the problems were to do with the development libraries being missing. This is frequently the case, and the error message pretty much says just that. Wrong – libpq was (and is) present. In fact, prior to the pg gem, we had the postgres gem running and it had the same dependencies. Next up: must be a mixup in the deployment situation. Not so. Capistrano and bundler both checked out okay.
After my searches on The Google provided ample proof that compilation on a Solaris box can be tricky, my next suspect was a wonky compiler toolchain. Nope, toolchain was just fine. Heck, I even checked the rbconfig.rb to make sure. That’s right, I’m thorough.
The (Temporary) Solution
On a lark, and after a day of trying all kinds of weird little tweaks … I figured, what about 0.9 ??? A quick look through the commits on 0.10.0 showed some recent work on the build system. Why not? Quick, change the Gemfile:
source 'http://rubygems.org' gem 'rails', '3.0.3' gem 'pg', '~> 0.9.0' # That '~>' operator tells Bundler/Rubygems to allow gem updates to # the least significant portion of the version number for that gem. # In the Gemfile snippet above, I am signifying that I will allow any pg version # in the 0.9.x branch. Had I specified '~> 0.9', bundler would install # the newest pg gem with a major version of '0.'.
Then
> bundle install > git add . > git commit -m "Attempt to use pg 0.9" > git push > cap deploy
SUCCESS!
I just opened this issue on the pg project and will update this space when we come up with a solution. For now, just install the pg gem from the 0.9.x version.
Epilogue
You might be wondering why I would care if I’m running this particular gem for Postgres connectivity. There are three postgres gems out there that I know of: pg, postgres, and postgres-pr. Let’s take a look at some stats from rubygems.org:
gem version date downloads ---------------- ----------------- ------------- ---------- postgres (native) 0.7.9.2008.01.28 Jan 28, 2008 49K postgres-pr (ruby) 0.6.3 Dec 14, 2009 35K pg (native) 0.10.0 Dec 01, 2010 210K
Even taking the subjective versioning and download numbers, I want us to use a library that is actively maintained and well used. Michael Granger is all over pg and having the most downloads by far, gives me the warm fuzzy that errors will have been encountered and (hopefully) fixed. Since we run on a platform that can use the native gems, pg is the obvious choice – for us. Of course, as they say, your mileage may vary.
UPDATE Dec. 15, 2010:
The issue has been updated – looks like this isn’t an isolated problem.
Disclaimer: Eventhough we aren’t deploying the Custodian Console under GlassFish, we spent some time playing and this overview may help some lost soul out there. (And some of the knowledge was transferrable to our current deployment method.)
GlassFish
GlassFish v3 is a reference implementation of Java EE 6. Quite a mouthful. For Rails developers: it uses some of the latest and greatest technology available on the Java 6 Enterprise Edition, and that means the deployment environment is going to need JDK 6 (Java Development Kit), which clocks in between 42-63MB.
The GlassFish application server has first class support for two deployment methods for Rails applications:
- Directory-based
- WAR-based
(Technically, you could use the GlassFish gem and deploy your applications this way, but it seems to be the same style of deployment you all know and love with mongrel/thin/unicorn/etc – and who wants to read another article about that?)
The differences between the deployment methods are subtle but important: a WAR can have JRuby bundled with it, whereas you’ll need to install JRuby separately with directory deployment. A WAR deploy can get you closer to a self-contained application, if that is your goal. Additionally, with the WAR file, you can drop it into the GlassFish autodeploy directory and the application server will automagically detect it and deploy. (We use Warbler, a library that will take a Rails application and package it up in a WAR with JRuby and jruby-rack, both of which you need.)
A note on asadmin: Most of what you do with GlassFish will be done using the asadmin command. In fact, it seems to be the hub of the GlassFish universe.
Installing GlassFish
Installation is a simple affair – just head over to GlassFish v3 Downloads and pick your poison. There is a GUI installer for Windows, Solaris, Linux and MacOSX. Thankfully, there is also a platform independent zip file.
GlassFish Architecture
Here is where it gets interesting(?) … *deep breath* This GlassFish “server” can house multiple domains, which in turn, have many server instances, and these server instances are where your applications are deployed. *whew!* A good resource for truly understanding the GlassFish architecture is GlassFish Administration by Xuekun Kou (also on Safari Books Online). Luckily, with the out-of-the-box install you get a default domain (domain1) and a default server instance. Remember that GlassFish is intended to do all the fancy things that a Java application server can do, so you can have clustering, multiple domains, multiple applications running in a single domain, the same application running on multiple server instances – pretty much whatever you can think of.
Deploy That App!
So, GlassFish is installed. You need asadmin to get things going.
> asadmin start-domain
This command will, um, start the domain. (If you had the war file in autodeploy, or if the application had been previously deployed, it would be deployed/started during domain initialization.) GlassFish features a fairly complete administrative UI which is available at http://yourserver:4848. The admin site takes a little bit to start up as GlassFish doesn’t load anything until it is requested, this includes your Rails application. That first request is a biggie! From the admin site you can monitor performance, view logs, change settings, and all kinds of fun stuff. Take a look around, and ask yourself, “Do I really need any/all of this?”.
Deploy from a directory or “It sounds easy but it may be the same dance you do now”
Remember, if you aren’t bundling JRuby with your application you’ll need to tell GlassFish where it is …
> asadmin configure-jruby-container --jruby.home=/path/to/jruby-x.x.x
and you’ll need to have all the gems installed, etc, etc. (Sound familiar?) In this case, it would be wise to have your gems vendored in your Rails application. From the directory above your rails application run:
> $GLASSFISH_HOME/bin/asadmin deploy your-rails-app/
This will deploy the application to your server at http://localhost:3000/your-rails-app. Any errors will be available in $GLASSFISH_HOME/domains/domain1/logs/server.log
Deploy from a WAR or “It sounds easy and it is“
Assuming you’ve got your Rails application all WAR’d up (using Warbler, of course). Warbler ensures that JRuby, jruby-rack, and all your gems are included.
> asadmin deploy /path/to/your-rails-app.war
This will deploy your application to your server at http://localhost:3000/your-rails-app. Same as above. Notice you didn’t have to do anything to tell GlassFish about JRuby or the gems. Nice!
Alternatively, you can drop your-rails-app.war in $GLASSFISH_HOME/domains/domain1/autodeploy and GlassFish will do the rest.
Making your app the root application (and changing the deployed app name)
You may want your rails application to be available at http://localhost:3000/ (or the root of the server). Again, asadmin is your friend:
> asadmin deploy --contextroot / /path/to/your-rails-app.war
Now your application is served at the root of the GlassFish server instance.
Want to change the name of the deployed application?
> asadmin deploy --name myapp /path/to/your-rails-app.war
Now you have http://localhost:3000/myapp
Our thoughts
As mentioned earlier, GlassFish would seem to shine when used for internal application deployments where you want to leverage existing infrastructure by hosting multiple Rails application within a single GlassFish install or using the GlassFish clustering options. Imagine using Capistrano to just push up the newest WAR and having GlassFish autodeploy that, no more stopping and starting your webservers! The monitoring and administration would also provide you with insight that isn’t provided in other Rails installation methods.
We played with GlassFish for a bit and decided that for our purposes it was a bit heavy. What were we looking for?:
- easy deployment (not just for internal techies)
- ability to provide a zip file with everything needed in it, no install needed
- no admin UI
- deploy without dependence on a central command (asadmin anyone?) – we could have used autodeploy here
- smallest download possible
- least amount of external dependencies (GlassFish requires the JDK and your run-of-the-mill Rails app isn’t going to need all that!)
Keep in mind our Custodian Console is a Rails application installed at customer sites, we do not always have direct access to the servers so we need deployment to be fire and forget. This led us to pursue Tomcat as an alternative and this set up will be the topic of a future post.
