Breakfast of champions

Have you ever had a really fun day? A day that starts like this:

Feb 24, 2011 10:50:42 PM org.apache.catalina.core.ApplicationContext log
SEVERE: unable to create shared application instance
org.jruby.rack.RackInitializationException: stack level too deep
from c:/console/ROOT/WEB-INF/gems/gems/bundler-1.0.10/lib/bundler/resolver.rb:348:in `catch'
from c:/console/ROOT/WEB-INF/gems/gems/bundler-1.0.10/lib/bundler/resolver.rb:348:in `resolve_requirement'
from c:/console/ROOT/WEB-INF/gems/gems/bundler-1.0.10/lib/bundler/resolver.rb:299:in `resolve'
from c:/console/ROOT/WEB-INF/gems/gems/bundler-1.0.10/lib/bundler/resolver.rb:298:in `reverse_each'
from c:/console/ROOT/WEB-INF/gems/gems/bundler-1.0.10/lib/bundler/resolver.rb:298:in `resolve'
from c:/console/ROOT/WEB-INF/gems/gems/bundler-1.0.10/lib/bundler/resolver.rb:349:in `resolve_requirement'
from c:/console/ROOT/WEB-INF/gems/gems/bundler-1.0.10/lib/bundler/resolver.rb:348:in `catch'
from c:/console/ROOT/WEB-INF/gems/gems/bundler-1.0.10/lib/bundler/resolver.rb:348:in `resolve_requirement'
... 213 levels...
from file:/C:/console/ROOT/WEB-INF/lib/jruby-rack-1.0.3.jar!/vendor/rack-1.2.1/rack/builder.rb:46:in `initialize'
from <script>:2:in `new'
from <script>:2

This was how my morning looked a week or two ago when I was starting up an installation of our Custodian’s Console on a Windows server. Take another look at that stack trace. (full trace here) It looks like bundler is causing some trouble in this app, right? In fact, it reads like bundler is in some sort of recursive blackhole and JRuby just gives up – as it should. Let’s see what Google has to say about this.

First search: JRuby, bundler and this error message

The results for my first search include a Warbler issue. WARBLER-21. In the comments, Nick Sieger recommends increasing the stack size:

“We’ve heard increased reports of “stack level too deep” with Bundler lately. In fact, we bumped up the default stack size to -Xss2048k in JRuby master as of ea20ac17. Can you try running Warbler as follows:

jruby -J-Xss2048k -S warble ...

Sounds perfect. This could be the answer… annnnnd nope. I try making these changes with no effect. In fact, I try passing in stack size changes in just about every conceivable way. (In the Tomcat docs they point out that you can also pass in an increased stack size with the -JvmSs option to the service itself.)

Hrm. Case of bad Google-Fu?

What about JRuby itself? Or Bundler?

I’m able to find another mention of this problem pretty quickly. This JRuby issue speaks about the *exact* same error. No suggested solution but to upgrade JRuby to 1.6 RC1 and bundler to latest (1.0.10)

Okay. I’ll bite.

I upgrade bundler. Same error.

Then I try using JRuby 1.6 RC1. No love.

I’m going to have to dig deeper – it could be 3 or 4 pages of Google search results before I find my answer!

Liferay?

After several pages of results and no luck I stumble across some postings on a message board (yuck!) for something called Liferay. Never heard of it. Turns out it has something to do with “market leading”, “portals” and “enterprises”. Oh, and message boards.

One such post talks about the same error I’m having and a working solution. Sort of.

I found out some further information. I am running Tomcat 6 as a service through the 64-bit Tomcatw.exe (procrun implementation). If I use the “jvm” startmode I get the above error. If I use the “java” startmode I don’t. Wanting to know why still.

That poor guy made the last post in that thread. I really dislike message boards.

Startmode, what’s that all about? Turns out that Tomcat Tomcat uses “procrun” to allow it to run as a service on Windows. Ahhhh, a mention of Windows. I like where this is going.

“Procrun is a set of applications that allow Windows users to wrap (mostly) Java applications (e.g. Tomcat) as a Windows service. The service can be set to automatically start when the machine boots and will continue to run with no user logged onto the machine.”

From the procrun docs:

“startmode” is here:

One of jvm, Java or exe. The modes are:
jvm – start Java in-process. Depends on jvm.dll, see –Jvm.
Java – same as exe, but automatically uses the default Java executable, i.e. %JAVA_HOME%\bin\java.exe. Make sure JAVA_HOME is set correctly, or use –JavaHome to provide the correct location. If neither is set, procrun will try to find the default JDK (not JRE) from the Windows registry.
exe – run the image as a separate process

Confirmed. Starting in ‘java’ startmode allows the console to start up but shutdown via the service console is now broken and the process runs as java.exe instead of tomcat6.exe. That’s a start but not really a long-term solution. Why would this be working when running as a separate process?

That question leads me to spend several hours trying to figure out if my parameters are actually being passed into the JVM. If you are trying to do the same let me save you some time. Edit the line in service.bat that looks like this:"%EXECUTABLE%" //IS//%SERVICE_NAME% to include --LogLevel Debug at the end. This little change will dump lots of useful information to the catalina or commons-daemon log.

If I set the startmode to java I can get things to run, just not the way we need or want. Time to try something else.

Back to the Liferay messageboards! Hooray.

Here is a post that actually proposes a solution to the problem:

The last post says:

“Tomcat 6 ships with v1.0.2 of Commons Daemon (in the form of tomcat6.exe and tomcat6w.exe). It seems this version is not compatible on certain combinations of 64-bit Windows and JDK6.

To resolve this, simply download v1.0.4 from http://commons.apache.org/daemon/download_daemon.cgi, replace tomcat6.exe and tomcat6w.exe with prunsrv.exe and prunmgr.exe respectively, rename to tomcat6.exe and tomcat6w.exe and run service.bat.”

THAT’S what I’m talking about! I follow the proposal and it works. Neat-o.

Onward to a permanent solution

The newest procrun (v1.0.5) works just fine on Windows Server 2008 64bit and 32bit. Although I can’t find the specific bug fix in daemons (procrun) JIRA, I can confirm that it works. I see some vague references to a reorg and some other moves happening in the code but nothing concrete. I’d rather not have to maintain versions of procrun along with Tomcat so I go hunting for Tomcat updates … Tomcat has since released 6.0.32 which includes commons daemon (procrun) v1.0.5 since version 6.0.30 – the console is currently on 6.0.29 so an upgrade fixes the problem.

If you are fighting with these kinds of issues with JRuby and Tomcat (or just Tomcat) it would behoove you to try a newer version of Tomcat first, they update regularly and pull in upstream changes for other components. You can check the Tomcat 6.0 changelog for details. For instance, the procrun update message went something like this:

Update to Commons Daemon 1.0.5. (mturk)

Good hunting!

 

2 Responses to “Tomcat 6 – stack level too deep error on Windows”

  1. Unic Man says:

    You saved my days! I struggled for this at least for 2 days!

    Thanks a lot!

    PS
    I can’t post on liferay forum. If you have access, I would appreciate if you could thank them on my behalf!

  2. tillsc says:

    I had the problem with bundler 1.0.18, rails 3.1 and Tomcat 6.0.20 (win32) too. A simple tomcat upgrade to 6.0.33 did the job.

    Thank you very much for your writeup and the sharing of your knowledge!

Leave a Reply