Properly Shutting down an ExecutorService

Datetime:2016-08-23 02:12:08          Topic: Spring  Tomcat  Lucene           Share

I was recently working on an application that used hibernate for db access and lucene for storing and searching text. This combination was really great. It made searching really fast. And tomcat was used as the application container.

Sometimes I used to get this exception on redeploy:

java.lang.IllegalStateException: Pool not open
        at org.apache.commons.pool.BaseObjectPool.assertOpen(BaseObjectPool.java:123)
        at org.apache.commons.pool.impl.GenericObjectPool.returnObject(GenericObjectPool.java:898)
        at org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:80)
        at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:180)
        at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.closeConnection(LocalDataSourceConnectionProvider.java:95)

And also tomcat used to complain that it could not stop a thread:

SEVERE: The web application [/MyApp] appears to have started a thread named [myspringbean-thread-1] but has failed to stop it.This is very likely to create a memory leak

Here is the bean that created a thread to read some data from db and use lucene to index it.

I used ExecutorService to start the Thread on @PostConstruct. Here is the method that started the thread

@PostConstruct
public void init() {

  BasicThreadFactory factory = new BasicThreadFactory.Builder()
        .namingPattern("myspringbean-thread-%d").build();
  executorService =  Executors.newSingleThreadExecutor(factory);
  executorService.execute(new Runnable() {

   @Override
   public void run() {
    try {
     doTask();
    } catch (Exception e) {
     logger.error("indexing failed", e);
    }
   }
  });

  executorService.shutdown();
}

And the @PreDestroy I used the executor service to shutdown. This was the first try that caused the above errors. This seemed to me that it should work fine. But it didn't.

@PreDestroy
public void beandestroy() {
  if(executorService != null){
    executorService.shutdownNow();
  }
}

After fighting for some time with the above errors, I came to the conclusion that the bean is destroyed and the above thread that was created was still running. So I started checking all the calls from this thread to find out if the interrupt was somwhere lost. But came to know that if the thread is doing some IO operations it is better to wait for them to finish before trying to shutdown the thread.

So I came up with the below @PreDestroy method and it works fine. Here the thread waits for a second for the IO to complete and then shuts it down. And the thread itself checks for stopthread variable to not do any more IOs.

Instead of the stopThread variable an alternative would be to use Thread.currentThread().isInterrupted() in the thread loop. But for using this flag you need to be absolutely sure that the code you are calling in the thread is not resetting the interrupt status.

@PreDestroy
public void beandestroy() {
  this.stopThread = true;
  if(executorService != null){
   try {
    // wait 1 second for closing all threads
    executorService.awaitTermination(1, TimeUnit.SECONDS);
   } catch (InterruptedException e) {
    Thread.currentThread().interrupt();
   }
  }
 }

Here is the complete Spring bean.

package com.tak.package;

import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import org.apache.commons.lang3.concurrent.BasicThreadFactory;
import org.apache.logging.log4j.Logger;
import org.springframework.stereotype.Component;

@Component
public class MySpringBean {

 private static final Logger logger = org.apache.logging.log4j.LogManager
   .getLogger(MySpringBean.class);

 private ExecutorService executorService;
 private volatile boolean stopThread = false; 

 @PostConstruct
 public void init() {

  BasicThreadFactory factory = new BasicThreadFactory.Builder()
        .namingPattern("myspringbean-thread-%d").build();

  executorService =  Executors.newSingleThreadExecutor(factory);
  executorService.execute(new Runnable() {

   @Override
   public void run() {
    try {
     doTask();
    } catch (Exception e) {
     logger.error("indexing failed", e);
    }
   }
  });
  executorService.shutdown();
 }

 private void doTask()  {
  logger.info("start reindexing of my objects");
  List<MyObjects> listOfMyObjects = new MyClass().getMyObjects();

  for (MyObjects myObject : listOfMyObjects) {
   if(stopThread){ // this is important to stop further indexing
    return;
   }

   DbObject dbObjects = getDataFromDB();
   indexDbObjectsUsingLucene(dbObjects);
  }
 }

 @PreDestroy
 public void beandestroy() {
  this.stopThread = true;

  if(executorService != null){
   try {
    // wait 1 second for closing all threads
    executorService.awaitTermination(1, TimeUnit.SECONDS);
   } catch (InterruptedException e) {
    Thread.currentThread().interrupt();
   }
  }
 }
}

Conclusion

The exception stacktrace in dbcp was caused because the spring bean with executorservice was using the datasource bean to access the db. On using shutdownnow () the bean destroy was done but the thread was still running. And since the bean destroy was done all the beans accessed by this  bean were also destroyed. But the thread was still running and tried to access the spring datasource bean which was already destroyed and leading to the exception. So the pattern of also setting a boolean variable on shutdown and waiting for termination gives the programmer a means to stop the processing being done so that the thread can exit gracefully. And the spring bean is also destroyed only when all the threads started by this bean have exited.

Please drop your comments if you have any questions or suggestions.





About List