Tomcat Shutdown Issues Caused by an Embedded Infinispan Cache

Posts, Programming, Web Development

I was recently stumped by some shutdown issues in a Tomcat-hosted Java webservice. The server just wouldn’t shut down cleanly. After (most of) a day running round in circles, I tracked the issue down to embedded Infinispan - and specifically JGroups.

I’m writing the solution up here in the hopes someone might find it useful.

Background

To ensure a proper resource cleanup during JVM shutdown, Infinispan has built in shutdown hooks. Tomcat (and most other Application Servers) have their own shutdown hooks, so it’s desirable to turn off Infinispan’s built-in shutdown hooks to prevent any contention issues. Via infinispan.xml:

<infinispan
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:7.0 http://www.infinispan.org/schemas/infinispan-config-7.0.xsd"
    xmlns="urn:infinispan:config:7.0">

   <cache-container default-cache="default" shutdown-hook="DONT_REGISTER">
       <local-cache name="xml-configured-cache">
          <eviction strategy="LIRS" max-entries="10" />
       </local-cache>
   </cache-container>

</infinispan>

The Issue

When Infinispan running in a clustered configuration, it uses JGroups for network communications.

By unregistering the shutdown hooks (as above), non-daemon JGroups threads are left running inside the Application Server - even after a shutdown call. This prevents the Application Server process from terminating - it needs to be killed. From jstack, the culprit threads are:

  • INT-1
  • INT-2
  • multicast receiver
  • unicast-receiver
  • TransferQueueBuilder

The question: how do we stop these runaway threads and allow the Application Server to shutdown gracefully?

The Fix

The fix is fairly simple.

Being a Servlet Container, Tomcat conforms to the Java Servlet API. We can use the ServletContextListener Interface to initiate a graceful Infinispan shutdown when Tomcat shutdown is initiated. Using Spring:

@WebListener
public class ContextFinalizer implements ServletContextListener {

    ...

    @Override
    public void contextDestroyed(ServletContextEvent servletContextEvent) {
        DefaultCacheManager cacheManager = (DefaultCacheManager) ContextLoaderListener.getCurrentWebApplicationContext().getBean("cacheManager"); // my cache manager bean is called "cacheManager"
        cacheManager.stop();
    }
}

That’s all there is to it!

Moral of the Story

When you unregister Infinispan shutdown hooks via DONT_REGISTER, always make sure to manually stop the Infinispan caches via CacheManager.stop().

References

Comments