Brainstorming Basics

A question asked is if this idea is unique. Yes and no is the answer. No, as a information systems are being built in one form or another. Yes, as there is no comprehensive information management and analysis system. For example, Joomla is unique as it gives a completely different perspective on building and managing website. Similarly, the project that I am proposing is unique in its vision and application.

Is it even possible to make information systems generalist when people use them so differently? Indeed, this problem is challenging. Nevertheless, in my experience, it is not only possible but is also desirable.

Am I not worried that somebody might steal my idea? Not really. In fact, if somebody can build such a system I may use it to make more applications. On the other hand, since this application that I intend to build may end up as open-source, it may have some appeal eventually.

The first step is to start defining the key modules. Let us brainstorm the objects that we can see in the system

Data Handler: A data handler that reads data in a standardized format and then stores, updates or deletes it as need be

Information: The central class that would act as a controller or central point of directing a particular information set

Webpage: A simple html page that we wish to use display various data interpretation elements

Report: A static report

Graph: Tools to generate static graphs on data set

Tables: Tools to generate smart tables on data set

Filters: Filters for graph or table data

Dashboard: Tools to generate a group of one or more dynamic graphs, smart tables and filters

Rules: Conditional treatment of incoming data

Alert: A notification broadcasted based on some rules defined on data

Logger: Logger that takes care of information logging

Support: A UI to dynamically manage the information system and see the system state visually

Each word in this list is inevitably linked to many more. As I work on it (time permitting) one definition a time, let us see where we reach. To reach the skeleton stage, we will define minimal (and final) goals for each list as we reach them.

A Need For Generic Information Systems

Information systems are critical to any business. But what do I mean by information systems? I broadly define them as software entities that read data and perform some useful function on the same. These useful functions can include:

- Data visualization (reports & graphs)
- Aid manual data analysis and model making (with rich data dashboards)
- Generate event based alerts for stakeholders (email, sms, automated call, fax)
- Rule-based data analysis

Further, these systems can be either at real-time or passive.

The umpteen number of permutations mean infinite possibilities. Nevertheless, it is fair to assume that such systems should have some common governing principles as they perform a fairly similar job. The current market has a plethora of information system software that are sector specific and address a small subset according to needs of its users. Yet I have not seen an information engine that addresses all these concerns in a way that is generic and powerful enough. There are a lot of pieces that address some part of the puzzle, but what we are missing is something that takes into account the whole picture.

Such a project will be complex in scope and diverse in application. A modular approach that does not force user to take what he does not need is warranted. It should save the development cost for such systems for a wide array of sectors.

I would propose (naturally) a java based solution using mix of spring, maven, Apache libraries, Flex, OpenMQ, Quartz and PostGres. Making it work efficiently for real-time systems and the scale of the project are two immediate challenges that I see. Making it open-source and charging for elite training/support is the way to go; unless somebody has a better idea :D. In coming days, I will try to put down the basic pegs to work towards. Ta till then!

Writing DOM to XML File on Disk

Just wrote a small program to write a DOM to an XML on the disk using Xerces. The sample class can be seen here. If you are using maven, don't forget to add xerces dependencies:

        <dependency>
            <groupId>xerces</groupId>
            <artifactId>xmlParserAPIs</artifactId>
            <version>2.6.2</version>
        </dependency>
        <dependency>
            <groupId>apache-xerces</groupId>
            <artifactId>xercesImpl</artifactId>
            <version>2.9.1</version>
        </dependency>

For more details, refer Processing XML with Java by Elliotte Rusty Harold

Testing Database Code Using JUnit

This post outlines the basic setup that you may need to use HSQL in JUnit, Spring, Maven environment to test database code when you are using JdbcTemplate directly instead of Hibernate.

Step 1: Include the maven dependencies in your pom:

<dependency>
    <groupId>hsqldb</groupId>
    <artifactId>hsqldb</artifactId>
    <version>1.8.0.7</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>commons-collections</groupId>
    <artifactId>commons-collections</artifactId>
    <version>3.2</version>
</dependency>
<dependency>
    <groupId>commons-dbcp</groupId>
    <artifactId>commons-dbcp</artifactId>
    <version>1.2.2</version>
</dependency>

Step 2: Setup bean config that will be used by our test (save as database.xml and put it in the resources folder):

<bean id="dataSource"
        class="org.apache.commons.dbcp.BasicDataSource">
    <property name="driverClassName" 
        value="org.hsqldb.jdbcDriver"/>
    <property name="url" 
        value="jdbc:hsqldb:mem:aname"/>
    <property name="username" value="sa"/>
    <property name="password" value=""/>
    <property name="defaultAutoCommit" value="false"/>
</bean>
<bean id="jdbcTemplate" 
        class="org.springframework.jdbc.core.JdbcTemplate">
    <property name="dataSource" ref="dataSource"/>
</bean>

Step 3: Write the test case(s)

You may find the HSQL documentation (supported data types in chapter 9) and the api doc on JdbcTemplate useful as well.

Of course things will not be that simple due to one simple reason: HSQL will most probably not support all the data types used by your DB (e.g. "bytea" , an array of bytes, is supported by PostGreSQL DB but not by HSQL). But it definitely a headstart and a good way to test your DB.

Cutting Corners

When you are making systems that need to be very fast, every small bit helps. Some tips (thanks to Andrew) to fine tune performance:

• If you know that a value will not change in its lifetime, mark it as final (even those inside a method call). This will ensure that compiler accesses them faster. eg:
  

  final int someValueToRefer;
  
  public void aMethod(final Object obj, final int funcValToRefer){
      final int aMethodValueThatWontChange = 1234;
  }

• Use message queues between systems you want to truly decouple. This makes sure that one part doesn't hold the other to ransom


• Use System.currentTimeMillis() if all you need is current time instead of new Date()


• Save dates in DB as long values instead of date objects


• When using a cache, save minimal information (saving a unique ID instead of the whole object, for example)


• Use connection pooling: for databases, thread pools, messaging queues


• Use synchronization blocks instead of full method synchronization


• When there is too much information to process, it may be desirable to first "group" similar chunks and process them in batches (for example, when you are sending stock alerts to you star trading team, you may want to group all the alerts by the time they are expected to go off)


• When there is a boolean to return, one can generally avoid creating one explicitly. eg:
  

  boolean wasFound = false;
  
  for(MyObject currObj:objectList){
       if(currObj.getProperty().equals(valueLookingFor)){
          wasFound=true;
          break;
      }
  }
  
  return wasFound;

  can be replaced by:
  

  for(MyObject currObj:objectList){
       if(currObj.getProperty().equals(valueLookingFor)){
          return true;
      }
  }
  
  return false;



• Similarly, when we have a function that finds something and returns it, when you find it in a loop return right there instead of bothering to assign it to some var and break. eg:
  

  MyObject retVal= null;
  
  for(MyObject currObj:objectList){
       if(currObj.getProperty().equals(valueLookingFor)){
          retVal=currObj;
          break;
      }
  }
  
  return retVal;

  can be replaced by:
  

  for(MyObject currObj:objectList){
       if(currObj.getProperty().equals(valueLookingFor)){
          return currObj;
      }
  }
  
  return null;

A Simple Cache That Works

In our last post we saw how to make a custom multiple write-single read class. In this post we see how to use basic caching to help us process information selectively. Caching is a complex thing, ie if you go to finer details. However, we needed something simple that we can log the way we want to (and know for sure that it works). I started with some complicated AOP that caches method results to only find using a basic cache to be more efficient.

There are three basic steps to using cache in the spring environment:

1. Configure the ehcache.xml and put it in your resources folder
EHCache Config

2. Put the required spring configuration, injecting the cache in the required bean
Spring Cache Config

3. Use the cache in the bean as you deem fit:
The multiple write, single read bean that uses caching
A helper bean that stores data being passed around

And, of course, you would need the ehcache and spring cache dependencies if you are using maven:

<dependency>
    <groupId>net.sf.ehcache</groupId>
    <artifactId>ehcache</artifactId>
    <version>1.5.0</version>
    <scope>compile</scope>
</dependency>
<dependency>
    <groupId>spring-cache</groupId>
    <artifactId>spring-cache</artifactId>
    <version>2.4.1</version>
</dependency>

Multiple Write and Single Read Mechanism

Just imagine: you are a successful financial company with a big entourage of traders slogging it out throughout the day. They make money for you and you would want to build a system that works as hard to help them. Sheer muscle may get you some where, but sooner or later you are going into big problems associated with building a distributed system that handles a huge-huge volume of data. It can be a hard to tame, resource hungry beast.

Lets say you want it to process the data supplied in a known format and send alerts/emails to all the concerned traders (some rule based events). You also want it to be accessible over the web to your team sitting in different corners of the country. On top of it, the speed is of paramount importance.

My team has solved a similar problem and it has been one hell of a experience. Don't know from where I should try to collate my experience. In each article I will summarize a problem we faced and the solutions that we opted for.

We were facing a nasty problem with our messaging system. Our system had three queues where messages were posted to and read from. Each of the queue had multiple consumers and the posting was done using a pooled connection factory (ActiveMQ). The problem was that after some time the consumers would refuse to pick any fresh messages from the queue, as if they were stuck forever. We suspected it to be an ActiveMQ bug, but at the end it turned out to be a problem with our threading and caching. Our threads were getting dead-locked blocking the ports from which a consumer could read anything. We were also trying to cache the whole method call instead of the results. Got it rectified and learnt a few more things about the threaded systems and caching. In the process learned quite a few things from my manager and colleagues. Thanks to Andrew and Ben.

First, we wanted to use a ReadWriteLock mechanism in which a number of threads write to a list for a particular amount of time after which another thread processes all the data written. When the data is being read, no thread can write to the common pool and the reader should allow any thread that was already writing to complete before attempting to read. We opted to use nuts-and-bolts multi-threading instead of delving deeper into java's lock mechanisms. Here is a generic class that can do that for you:

MultiWriteSingleRead.java

In the next post we will explore using a cache on the same class so as we process information selectively.