Testing Database Code Using JUnit

This post outlines the basic setup that you may need to use HSQL in JUnit, Spring, Maven environment to test database code when you are using JdbcTemplate directly instead of Hibernate.

Step 1: Include the maven dependencies in your pom:

<dependency>
    <groupId>hsqldb</groupId>
    <artifactId>hsqldb</artifactId>
    <version>1.8.0.7</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>commons-collections</groupId>
    <artifactId>commons-collections</artifactId>
    <version>3.2</version>
</dependency>
<dependency>
    <groupId>commons-dbcp</groupId>
    <artifactId>commons-dbcp</artifactId>
    <version>1.2.2</version>
</dependency>

Step 2: Setup bean config that will be used by our test (save as database.xml and put it in the resources folder):

<bean id="dataSource"
        class="org.apache.commons.dbcp.BasicDataSource">
    <property name="driverClassName" 
        value="org.hsqldb.jdbcDriver"/>
    <property name="url" 
        value="jdbc:hsqldb:mem:aname"/>
    <property name="username" value="sa"/>
    <property name="password" value=""/>
    <property name="defaultAutoCommit" value="false"/>
</bean>
<bean id="jdbcTemplate" 
        class="org.springframework.jdbc.core.JdbcTemplate">
    <property name="dataSource" ref="dataSource"/>
</bean>

Step 3: Write the test case(s)

You may find the HSQL documentation (supported data types in chapter 9) and the api doc on JdbcTemplate useful as well.

Of course things will not be that simple due to one simple reason: HSQL will most probably not support all the data types used by your DB (e.g. "bytea" , an array of bytes, is supported by PostGreSQL DB but not by HSQL). But it definitely a headstart and a good way to test your DB.

Cutting Corners

When you are making systems that need to be very fast, every small bit helps. Some tips (thanks to Andrew) to fine tune performance:

• If you know that a value will not change in its lifetime, mark it as final (even those inside a method call). This will ensure that compiler accesses them faster. eg:
  

  final int someValueToRefer;
  
  public void aMethod(final Object obj, final int funcValToRefer){
      final int aMethodValueThatWontChange = 1234;
  }

• Use message queues between systems you want to truly decouple. This makes sure that one part doesn't hold the other to ransom


• Use System.currentTimeMillis() if all you need is current time instead of new Date()


• Save dates in DB as long values instead of date objects


• When using a cache, save minimal information (saving a unique ID instead of the whole object, for example)


• Use connection pooling: for databases, thread pools, messaging queues


• Use synchronization blocks instead of full method synchronization


• When there is too much information to process, it may be desirable to first "group" similar chunks and process them in batches (for example, when you are sending stock alerts to you star trading team, you may want to group all the alerts by the time they are expected to go off)


• When there is a boolean to return, one can generally avoid creating one explicitly. eg:
  

  boolean wasFound = false;
  
  for(MyObject currObj:objectList){
       if(currObj.getProperty().equals(valueLookingFor)){
          wasFound=true;
          break;
      }
  }
  
  return wasFound;

  can be replaced by:
  

  for(MyObject currObj:objectList){
       if(currObj.getProperty().equals(valueLookingFor)){
          return true;
      }
  }
  
  return false;



• Similarly, when we have a function that finds something and returns it, when you find it in a loop return right there instead of bothering to assign it to some var and break. eg:
  

  MyObject retVal= null;
  
  for(MyObject currObj:objectList){
       if(currObj.getProperty().equals(valueLookingFor)){
          retVal=currObj;
          break;
      }
  }
  
  return retVal;

  can be replaced by:
  

  for(MyObject currObj:objectList){
       if(currObj.getProperty().equals(valueLookingFor)){
          return currObj;
      }
  }
  
  return null;

A Simple Cache That Works

In our last post we saw how to make a custom multiple write-single read class. In this post we see how to use basic caching to help us process information selectively. Caching is a complex thing, ie if you go to finer details. However, we needed something simple that we can log the way we want to (and know for sure that it works). I started with some complicated AOP that caches method results to only find using a basic cache to be more efficient.

There are three basic steps to using cache in the spring environment:

1. Configure the ehcache.xml and put it in your resources folder
EHCache Config

2. Put the required spring configuration, injecting the cache in the required bean
Spring Cache Config

3. Use the cache in the bean as you deem fit:
The multiple write, single read bean that uses caching
A helper bean that stores data being passed around

And, of course, you would need the ehcache and spring cache dependencies if you are using maven:

<dependency>
    <groupId>net.sf.ehcache</groupId>
    <artifactId>ehcache</artifactId>
    <version>1.5.0</version>
    <scope>compile</scope>
</dependency>
<dependency>
    <groupId>spring-cache</groupId>
    <artifactId>spring-cache</artifactId>
    <version>2.4.1</version>
</dependency>

Multiple Write and Single Read Mechanism

Just imagine: you are a successful financial company with a big entourage of traders slogging it out throughout the day. They make money for you and you would want to build a system that works as hard to help them. Sheer muscle may get you some where, but sooner or later you are going into big problems associated with building a distributed system that handles a huge-huge volume of data. It can be a hard to tame, resource hungry beast.

Lets say you want it to process the data supplied in a known format and send alerts/emails to all the concerned traders (some rule based events). You also want it to be accessible over the web to your team sitting in different corners of the country. On top of it, the speed is of paramount importance.

My team has solved a similar problem and it has been one hell of a experience. Don't know from where I should try to collate my experience. In each article I will summarize a problem we faced and the solutions that we opted for.

We were facing a nasty problem with our messaging system. Our system had three queues where messages were posted to and read from. Each of the queue had multiple consumers and the posting was done using a pooled connection factory (ActiveMQ). The problem was that after some time the consumers would refuse to pick any fresh messages from the queue, as if they were stuck forever. We suspected it to be an ActiveMQ bug, but at the end it turned out to be a problem with our threading and caching. Our threads were getting dead-locked blocking the ports from which a consumer could read anything. We were also trying to cache the whole method call instead of the results. Got it rectified and learnt a few more things about the threaded systems and caching. In the process learned quite a few things from my manager and colleagues. Thanks to Andrew and Ben.

First, we wanted to use a ReadWriteLock mechanism in which a number of threads write to a list for a particular amount of time after which another thread processes all the data written. When the data is being read, no thread can write to the common pool and the reader should allow any thread that was already writing to complete before attempting to read. We opted to use nuts-and-bolts multi-threading instead of delving deeper into java's lock mechanisms. Here is a generic class that can do that for you:

MultiWriteSingleRead.java

In the next post we will explore using a cache on the same class so as we process information selectively.

The Return of the Persistent Quartz Job

Finally I managed to understand the persistent job and make it work reliably: for your stateful job and the trigger volatility should NOT be true. Even setting durability to true can cause problems as it would persist orphan jobs that we may not need. If the intention is to just prevent loss of jobs during system re-starts, it is enough to set requests recovery to true. That's it. Simple is beautiful :).

Caching Method Invocation Results

Worked on setting up a cache for a method invocation fetching results from a webservice using Spring Cache AOP and EHCache in a Spring-Maven environment.Long live sourceforge! The spring cache java doc is pretty helpful and this article by Pieter Coucke expands on what you read in the java doc. You can also have a look at my complete configuration (which works) and is based on the article and the java doc.

And if you are using Maven, you would need to add dependencies for ehcache and spring cache.

<dependency>
    <groupId>net.sf.ehcache</groupId>
    <artifactId>ehcache</artifactId>
    <version>1.5.0</version>
</dependency>
<dependency>
    <groupId>spring-cache</groupId>
    <artifactId>spring-cache</artifactId>
    <version>2.4.1</version>
</dependency>

Another Way of Starting ActiveMQ Broker in Spring

I had posted a blog about running an ActiveMQ broker in Spring. Recently, I found an alternative way to start an ActiveMQ message broker in spring environment by injecting a bean based on org.apache.activemq.xbean.XBeanBrokerService:

<bean id="broker" class="org.apache.activemq.xbean.XBeanBrokerService">
   <property name="useJmx" value="true"/>
   <!-- this tells it to create a filesystem store -->
   <property name="persistent" value="true"/>
   <property name="brokerName" value="mySpringBroker"/>
   <!--the queues/topics that this broker starts with-->
   <property name="destinations">
       <list>
           <ref bean="destination1"/>
           <ref bean="destination2"/>
       </list>
   </property>
   <property name="transportConnectors">
       <list>
          <!-- an in vm connector which is nice and fast -->
          <bean class="org.apache.activemq.broker.TransportConnector">
             <property name="uri" value="vm://localhost"/>
          </bean>
          <!-- a connection for external clients specifying the IP of machine deploying the broker, something like tcp://10.10.10.10:61636. Here read from a properties file but can be hard-coded as well-->
          <bean class="org.apache.activemq.broker.TransportConnector">
             <property name="uri" value="${broker.transportExternalConnectorUri}"/>
          </bean>
       </list>
   </property>
</bean>

<bean id="destination1" class="org.apache.activemq.command.ActiveMQQueue">
    <constructor-arg value="com.saveenkumar.myQueue1"/>
</bean>

<bean id="destination2" class="org.apache.activemq.command.ActiveMQQueue">
    <constructor-arg value="com.saveenkumar.myQueue2"/>
</bean>

In fact,the package org.apache.activemq.xbean provides various options (beans) to start the broker in a spring environment.