Marketing Triumphs Technology?

I have heard quite a few business leaders implying that marketing always triumphs technology. I do not dispute the importance of marketing. Nevertheless, working in technology business myself, I have no doubts that in the long-run technology is as important as marketing. Putting one above the other is to disparage one at the cost of other.

It is possible to use logic to exhalt any department. For example, the Business Development department can as easily assert that it is the sales force that gets in the money and is truly in touch with the truths at the ground level, unlike marketing managers that make a sweet sounding theoretical plans in their ivory towers. Technology, Marketing and Sales have their own place in the company structure. Comparing one with the other is absurd, unwarranted and potentially destructive.

Many technology guys are more interested in developing and creating. Innovation cannot come to them easily unless they are given a chance to interact with customers and involved with the business side of things. Barricading engineers & scientists in laboratories and complaining about lack of innovation is ironic, to say the least. Without customer interaction innovation is difficult to achieve. Marketing itself can be highly destructive if it operates in isolation from the customer. Sales without customer focus will kill the product eventually. Some people talk of innovation as if it is a commodity or a God given gift. My experience tells me that innovation is a result of closeness to customer and willingness to experiment.

Innovation may not be about wearing white coats, but it is not about wearing a tie and a suit (with a marketing hat) either. Innovation is an attitude born out of a company's culture and its people. Period.

Speeding Up Technology: Thinking People

In the previous two articles I have talked about the need for a vision with an ambition and the need to cultivate leaders. I have talked about the importance of policy. It, however, strikes me that who will make policy? Policy is the product created by people. So we have read about the importance of hiring the right people. But who will hire the right people? Only somebody who believes in it as a concept and is willing to implement it in practice. Moreover, it does not end here and the company needs to be ready with at least ambition-vision + cultivate leadership mindset so as it can make full use of the right people and retain them. Bottom-up changes are generally painful and often viewed with suspicion by almost everybody at top. Top-down changes without getting people involved are viewed similarly by people who are not party to it. The next pre-requisite, then, is a lot of communication and flexibility within the limits of the clearly defined vision/ policy.

The organization is defined by its people. Nevertheless, people cannot operate in a vacuum. Having a leader that nurtures such people and inculcates a sense of pride and ownership in them without duping them is a rarity, and can be a supremely powerful force in the growth of a company.

What has to be remembered, though, is that no policy, no vision can be immutable. If there is one certainty in this universe, it is change. More so in the technical world. The playing field can change completely in few years at best and few months at worse, such is the speed of change. The only defence can be having the right people and cultivating in them a spirit of learning and leadership. I would call it a "nursery" of leaders, hiring brilliant people and preparing them to rise

Speeding Up Technology: Cultivating Leaders

Most people understand leadership as power and/or authority and there is a lot of talk about the "strength" and "vision" of a leader. However, it is easy to mistake bullying & stubbornness for strength and demagogy & rhetoric for vision. The difference is too subtle and short-term results too similar to cause any alarm.

The generally accepted way to make a new product at greatest speed is to place some brilliant developers under an able Senior Manager or Product Architect. Now, it can take two directions from this basic set up. One way is that the manager just focuses on the current product and executes the project in the style of "Command and Conquer" by making things simpler for his team and spelling out clear instructions to execute the project. If the team struggles, he dives right in and leads from the front ensuring the correct course for the ship. All sailors learn a lot by seeing the able captain in action and there is tremendous growth. He can encourage his sailors to learn and grow.

All is happy in the world till the point, till one of the sailor learns enough to attempt to captain the ship, perhaps a smaller ship. The captain may view it as abject desertion or, worse, covert mutiny. Or he may provide the testing ground for the willing sailor as he knows that the company will eventually have many ships and he cannot steer all of them alone. In this context, the vision and growth strategy of company assumes paramount importance. If there is no growth potential, it will definitely be either desertion or mutiny (with accompanying unpleasantness). Else the business will gain another leader and grow.

This also makes further demand on the captain to actively recognize, cultivate and have an outlet for promising young sailors. Else he should hire sailors who do not harbour such ambitions and are happy sticking their nose only to their immediate work. Since no company can provide an unlimited supply of opportunities, one has to be ready to let go of talented people if they cannot neither find optimum use of the same nor think of ways to cultivate them further.

A company trusts only the best to build the nuts and bolts of their product. If it cannot make a strategy to retain them, it is a shame. This can not only lead to loss of talent but can also disrupt progress of the product due to loss of expertise and problems associated with hiring. Hence, it is in the best interest of the project to give a thought about this before starting. Here, as I said, the vision and policy of a company may be the ultimate deciding factor. At least in the initial life of a company, an entrepreneurial team that generates more entrepreneurial teams can be a powerhouse of growth and innovation. That it will affect speed positively over the long run is but one of the many side-effects of a well-thought and flexible company policy.

Speeding Up Technology: Ambition with Vision

Till this point we talked only about technology and its impact on the project without taking into account the manager, the leadership and the company. These elements bring in the human element and its pitfalls to the job. For this one reason, I feel behaviour sciences need to be explored more not only wrt economics but practically all domains of management. My articles do not claim to be any authority but reflect my point of view developed by working, reading and observing.

In our first article we talked that from managerial point of view, the main stumbling blocks can be:

• poor choice of technology
• poor choice of development methodology
• lack of focus on long-term vision with complete focus on speed

We shall explore them. But even before that, a lot depends on the company vision and the corresponding leadership policy.

Company vision defines the existence of a corporation. Why does the corporation exist in the first place? How ambitious and flexible is it towards growth? I have seen a lot of family and privately owned businesses stagnating as their sole purpose is to earn money for its owners, as a stable source of income or to do some limited good work in its sphere. There are yet others that come up due to the Government's policy changes and new legislation or those that manage to entrench themselves in a monopolistic niche. One thing strikingly common is that many of these ventures aim to neither be the "best" nor have a long term growth plan. There is no vision to the ambition. In Jim Collins words, these are companies who have figured out a way to be good but have not thought of being great by choice or chance. In cases when they do, some truly amazing companies or professions result from that.

The problems, to name a few, that arise from company's lack of vision for its growth are as follows:

• A small company does not have any use for too many leaders. This would generally translate into a senior rung of management with a big vacuum beneath. This can blow in the company's face in time
• Lack of growth will ultimately frustrate ambitious & able executives and employees alike, who may ultimately leave for better avenues. This is especially true for the Sales force
• A "family" atmosphere and nepotism can take roots more easily in a family run enterprise, seriously affecting the company's long-term future
• Ultimately competition will come calling, and if they are growth oriented, it will prove disastrous for the parent company

Having said that, there are some advantages to be had by working for a small company in its setting up phase:

• it will give you the opportunity to interact with and witness all aspects of business. This may prove critical in your growth as a leader who has a strong general awareness
• you get to see the whole product development lifecycle very closely, again adding to your leadership potential
• if (a big if) you can affect the company to make the leap and it succeeds, the dividends are yours to have
• in the initial growth phase learning is very profound and comprehensive due to the chaos associated with starting a venture, these things may be hard to learn in any big company and will serve you well when you want to do something on your own

Summarizing, in general limiting of vision ultimately reduces the company's efficiency and affects its supply of leaders. This is true at least for the technology industry where change is too rapid for comfort of the conservative. Ambition with vision can be the currency of speed. If your employees know that they are not a "cog in the wheel" but are actually contributing to something important that will affect at least their lives positively, it will help to speed up work and check attrition. A good work-life balance, an understanding manager, "just" compensations and good employees facilities can be the icing on the cake, but it is not the cake that can satiate the hunger. I feel that nothing can replace the electric, palpitating combination of vision and ambition.

Speeding Up Technology: Testing

Nothing can replace good development, and a good unit testing is a part of good development. I will not go as far as recommending hardcore test-driven development, but robust unit testing never hurts.

Pre-JUnit, having a "main" method in every Java class to be tested was the way I went about testing. In came web applications and testing this way suddenly felt difficult. Then a combination of JUnit, JMock and Spring enabled me to test my code more meaningfully. For multi-threaded systems where even debugging into the running code is virtually impossible and Unit testing can help only so much, intelligent use of logging can help a lot to both verify functionality and spot potential bugs.

Change is our constant friend, and unit tests enable your team to keep track of their key assumptions and all the parts of the code affected by any change in the code. Barring this, the team will inevitably waste time due to avoidable, simple bugs. This makes more sense as a company cannot be sure about who will change the code in future. Documentation, code-comments and well-written code can help a new developer to understand many things, but none of them can replace well-thought unit tests that capture the essential purpose, assumptions and functionality of the class in a very practical way. It also helps the developer to think about his code more deliberately and deeply. Overall, this adds to efficiency gains in terms of product development speed and code quality.

Speeding Up Technology: Choosing the Right IDE

People who were born with 'Eclipse' in their hand may not be able to appreciate the woes of a programmer trying to be productive with 'Edit Plus' or 'vim'. Many managers grossly under-estimate the productivity gains that can be had by using the right IDE. Not only that, development productivity for many current technologies can be improved significantly if their IDEs see significant improvement.

If you do not mind spending, go ahead and buy 'Idea'. If you are broke, 'Net Beans' would make a very good choice. I know a lot of people would be screaming 'Eclipse' by this point of time. 'Eclipse' is, no doubt, the IDE with greatest potential. But if you are developing a web application, want to get up-to-speed asap and integrated with most of the needed Java and J2EE technologies for the project(Maven, Subversion, Database, Servers, Messaging etc.) in no time, I would recommend 'Net Beans'. If you value super customization over speed, 'Eclipse' is your baby. Eclipse can also serve as a fabulous starting point for anybody who wants to develop an IDE.

The tricky bit will come into play if you wish to develop a rich internet application. This would certainly need either JavaScript or Flex. Personally, I feel that there is no IDE that effectively addresses this problem of building a super-efficient dashboard on top of a Java based backend. In this case I would recommend using Eclipse because the Flex IDE is built on top of Eclipse. You may even use Net Beans and Eclipse in conjunction while you take time to configure Eclipse to do the complete bidding.

Speeding Up Technology: Reinventing The Wheel

Re-usability is the holy-grail of any large-scale production industry. IT is no different. This re-usability primarily manifests itself on the technical front in form of programming paradigms, reusable components and frameworks.

Programming Paradigms

Programming paradigms include computing concepts like algorithms, data structures, object oriented programming, aspect oriented programming, design pattern et al. These are fundamentals that enable programmers to 'reuse' experience. It is surprising how many developers are unaware about these things. If the team has any member who completely lacks this knowledge, training or replacement is in order as these are probably the basic pre-requisites for hiring.

Reusable Components

Reusable components can be taken from open-source communities, licensed from third party or developed in-house. In open source, Google and Apache Foundation are perhaps the biggest and most reliable contributors. There are also a lot of third party development houses that may have developed the needed components and can save the company a lot of time. It is always a good idea to not only be aware about these initiatives but to actively track them. Joining organizations like Association for Computing Machinery and attending technology seminars can help in keeping you aware and agile. Buying a library, using an open-source library or building on a tried and tested library are acceptable, and even desirable, ways of speeding up development.

Developing reusable components in-house, however, is a more tricky business. Each company, knowingly or unknowingly, does try to build upon what it knows. It is a good start but can never be enough. What is really needed is an active initiative to consciously try to build components, if not frameworks, at each step. A forum to share best practices and components across the company can multiply the benefits and speed up development significantly. In fact, bigger the company, more the benefits. The only point to be kept in mind is that this initiative should neither detract nor confuse the main objective of building the product. The danger of "initiative" turning into another bureaucratic process is an equally scary scenario.

Frameworks

The final component is using frameworks. We have seen Ruby-on-Rails, Struts, Spring et al assuming significant importance. Most of them are practical implementation of programming paradigms and demonstrate the effectiveness (and weaknesses) of the underlying paradigms. Hence, frameworks are at one higher level of abstraction than either the programming paradigms or the components. This means that efficiency gains can be significantly higher if the framework is chosen correctly. In my experience, Spring makes an excellent choice for complex, multi-tiered web applications.

In our next article we talk about IDEs.

Speeding Up Technology Projects

Speeding up technology projects can be done in various ways. I feel that there can be two fundamental stumbling blocks: technological and managerial.

Technologically, I assume that the team has hired reasonably bright developers with a decent, if not perfect, understanding of programming basics. After that,the potential bottle-necks can be:

• reinventing the wheel: programmers spend way too much time doing stuff that is already done by somebody
• IDE: a good IDE can significantly speed up the speed of development
• poor testing

From managerial point of view, the main stumbling blocks can be:

• poor choice of technology
• poor choice of development methodology
• lack of focus on long-term vision with complete focus on speed

These may be self-evident to most of you who have had some experience in IT. Nevertheless, each point is much more complicated than it appears and merits some discussion.

These issues can come to haunt a team at any point due to the extremely dynamic nature of information technology. To be ever-vigilant is the only answer. What makes it especially challenging is the fact that many problems can persist themselves without making themselves evident for a long time; and by that time the rot has set in with a vengeance and changes are pretty hard to implement. In a series of articles I shall have a closer look at these issues in perspective of my experience and observations and chronicle the same. Please feel free to give your feedback and views wrt same.

Specifying SSL Certificate and Keystore Location & Password

A small reminder about how to configure the location and password for your keystore with Tomcat. By default Tomcat will look for your keystore (with the file name .keystore) in the home directory with the default password. It is possible to change the filename, password, and the location that Tomcat looks for the keystore by configuring the SSL Connector in "server.xml" (in conf folder in tomcat). "keystoreFile" is the parameter specifying key location and "keyPass" would specify the corresponding password. the finished config may look something like:

<Connector port="8443" 
    protocol="HTTP/1.1" SSLEnabled="true"
    maxThreads="150" scheme="https" secure="true"
    clientAuth="false" sslProtocol="TLS" 
    keystoreFile="/home/user_name/my_domain.key" 
    keypass="my_keystore_password"/>

This, however, will not help you if you are using an SSL socket. This is just for making an https connection. If you want to control only keystore and truststore name and password for your socket, you need to give the system properties the values by either specifying the same in vm parameters or setting it in the code:

Using VM parameters:

-Djavax.net.ssl.keyStore=keystore.jks 
-Djavax.net.ssl.keyStorePassword=abc123 
-Djavax.net.ssl.trustStore=truststore.jks 
-Djavax.net.ssl.trustStorePassword=abc123

In Code:

Properties properties = System.getProperties();
properties.put("javax.net.ssl.keyStore", keyStoreName);
properties.put("javax.net.ssl.keyStorePassword", keyStorePassword);
properties.put("javax.net.ssl.trustStore", trustStoreName);
properties.put("javax.net.ssl.trustStorePassword", trustStorePassword);

For a better understanding of SSL Sockets, please refer to the JSSE Reference Guide. Nevertheless, this is not the end of your woes if you want to control the location as well. This would make sense if you wished to package your application with some default stores so as it can potentially work right out of the box. Here is a piece of code that can help you to do exactly that: Managing store locations

Brainstorming Basics

A question asked is if this idea is unique. Yes and no is the answer. No, as a information systems are being built in one form or another. Yes, as there is no comprehensive information management and analysis system. For example, Joomla is unique as it gives a completely different perspective on building and managing website. Similarly, the project that I am proposing is unique in its vision and application.

Is it even possible to make information systems generalist when people use them so differently? Indeed, this problem is challenging. Nevertheless, in my experience, it is not only possible but is also desirable.

Am I not worried that somebody might steal my idea? Not really. In fact, if somebody can build such a system I may use it to make more applications. On the other hand, since this application that I intend to build may end up as open-source, it may have some appeal eventually.

The first step is to start defining the key modules. Let us brainstorm the objects that we can see in the system

Data Handler: A data handler that reads data in a standardized format and then stores, updates or deletes it as need be

Information: The central class that would act as a controller or central point of directing a particular information set

Webpage: A simple html page that we wish to use display various data interpretation elements

Report: A static report

Graph: Tools to generate static graphs on data set

Tables: Tools to generate smart tables on data set

Filters: Filters for graph or table data

Dashboard: Tools to generate a group of one or more dynamic graphs, smart tables and filters

Rules: Conditional treatment of incoming data

Alert: A notification broadcasted based on some rules defined on data

Logger: Logger that takes care of information logging

Support: A UI to dynamically manage the information system and see the system state visually

Each word in this list is inevitably linked to many more. As I work on it (time permitting) one definition a time, let us see where we reach. To reach the skeleton stage, we will define minimal (and final) goals for each list as we reach them.

A Need For Generic Information Systems

Information systems are critical to any business. But what do I mean by information systems? I broadly define them as software entities that read data and perform some useful function on the same. These useful functions can include:

- Data visualization (reports & graphs)
- Aid manual data analysis and model making (with rich data dashboards)
- Generate event based alerts for stakeholders (email, sms, automated call, fax)
- Rule-based data analysis

Further, these systems can be either at real-time or passive.

The umpteen number of permutations mean infinite possibilities. Nevertheless, it is fair to assume that such systems should have some common governing principles as they perform a fairly similar job. The current market has a plethora of information system software that are sector specific and address a small subset according to needs of its users. Yet I have not seen an information engine that addresses all these concerns in a way that is generic and powerful enough. There are a lot of pieces that address some part of the puzzle, but what we are missing is something that takes into account the whole picture.

Such a project will be complex in scope and diverse in application. A modular approach that does not force user to take what he does not need is warranted. It should save the development cost for such systems for a wide array of sectors.

I would propose (naturally) a java based solution using mix of spring, maven, Apache libraries, Flex, OpenMQ, Quartz and PostGres. Making it work efficiently for real-time systems and the scale of the project are two immediate challenges that I see. Making it open-source and charging for elite training/support is the way to go; unless somebody has a better idea :D. In coming days, I will try to put down the basic pegs to work towards. Ta till then!

Writing DOM to XML File on Disk

Just wrote a small program to write a DOM to an XML on the disk using Xerces. The sample class can be seen here. If you are using maven, don't forget to add xerces dependencies:

        <dependency>
            <groupId>xerces</groupId>
            <artifactId>xmlParserAPIs</artifactId>
            <version>2.6.2</version>
        </dependency>
        <dependency>
            <groupId>apache-xerces</groupId>
            <artifactId>xercesImpl</artifactId>
            <version>2.9.1</version>
        </dependency>

For more details, refer Processing XML with Java by Elliotte Rusty Harold

Testing Database Code Using JUnit

This post outlines the basic setup that you may need to use HSQL in JUnit, Spring, Maven environment to test database code when you are using JdbcTemplate directly instead of Hibernate.

Step 1: Include the maven dependencies in your pom:

<dependency>
    <groupId>hsqldb</groupId>
    <artifactId>hsqldb</artifactId>
    <version>1.8.0.7</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>commons-collections</groupId>
    <artifactId>commons-collections</artifactId>
    <version>3.2</version>
</dependency>
<dependency>
    <groupId>commons-dbcp</groupId>
    <artifactId>commons-dbcp</artifactId>
    <version>1.2.2</version>
</dependency>

Step 2: Setup bean config that will be used by our test (save as database.xml and put it in the resources folder):

<bean id="dataSource"
        class="org.apache.commons.dbcp.BasicDataSource">
    <property name="driverClassName" 
        value="org.hsqldb.jdbcDriver"/>
    <property name="url" 
        value="jdbc:hsqldb:mem:aname"/>
    <property name="username" value="sa"/>
    <property name="password" value=""/>
    <property name="defaultAutoCommit" value="false"/>
</bean>
<bean id="jdbcTemplate" 
        class="org.springframework.jdbc.core.JdbcTemplate">
    <property name="dataSource" ref="dataSource"/>
</bean>

Step 3: Write the test case(s)

You may find the HSQL documentation (supported data types in chapter 9) and the api doc on JdbcTemplate useful as well.

Of course things will not be that simple due to one simple reason: HSQL will most probably not support all the data types used by your DB (e.g. "bytea" , an array of bytes, is supported by PostGreSQL DB but not by HSQL). But it definitely a headstart and a good way to test your DB.

Cutting Corners

When you are making systems that need to be very fast, every small bit helps. Some tips (thanks to Andrew) to fine tune performance:

• If you know that a value will not change in its lifetime, mark it as final (even those inside a method call). This will ensure that compiler accesses them faster. eg:
  

  final int someValueToRefer;
  
  public void aMethod(final Object obj, final int funcValToRefer){
      final int aMethodValueThatWontChange = 1234;
  }

• Use message queues between systems you want to truly decouple. This makes sure that one part doesn't hold the other to ransom


• Use System.currentTimeMillis() if all you need is current time instead of new Date()


• Save dates in DB as long values instead of date objects


• When using a cache, save minimal information (saving a unique ID instead of the whole object, for example)


• Use connection pooling: for databases, thread pools, messaging queues


• Use synchronization blocks instead of full method synchronization


• When there is too much information to process, it may be desirable to first "group" similar chunks and process them in batches (for example, when you are sending stock alerts to you star trading team, you may want to group all the alerts by the time they are expected to go off)


• When there is a boolean to return, one can generally avoid creating one explicitly. eg:
  

  boolean wasFound = false;
  
  for(MyObject currObj:objectList){
       if(currObj.getProperty().equals(valueLookingFor)){
          wasFound=true;
          break;
      }
  }
  
  return wasFound;

  can be replaced by:
  

  for(MyObject currObj:objectList){
       if(currObj.getProperty().equals(valueLookingFor)){
          return true;
      }
  }
  
  return false;



• Similarly, when we have a function that finds something and returns it, when you find it in a loop return right there instead of bothering to assign it to some var and break. eg:
  

  MyObject retVal= null;
  
  for(MyObject currObj:objectList){
       if(currObj.getProperty().equals(valueLookingFor)){
          retVal=currObj;
          break;
      }
  }
  
  return retVal;

  can be replaced by:
  

  for(MyObject currObj:objectList){
       if(currObj.getProperty().equals(valueLookingFor)){
          return currObj;
      }
  }
  
  return null;

A Simple Cache That Works

In our last post we saw how to make a custom multiple write-single read class. In this post we see how to use basic caching to help us process information selectively. Caching is a complex thing, ie if you go to finer details. However, we needed something simple that we can log the way we want to (and know for sure that it works). I started with some complicated AOP that caches method results to only find using a basic cache to be more efficient.

There are three basic steps to using cache in the spring environment:

1. Configure the ehcache.xml and put it in your resources folder
EHCache Config

2. Put the required spring configuration, injecting the cache in the required bean
Spring Cache Config

3. Use the cache in the bean as you deem fit:
The multiple write, single read bean that uses caching
A helper bean that stores data being passed around

And, of course, you would need the ehcache and spring cache dependencies if you are using maven:

<dependency>
    <groupId>net.sf.ehcache</groupId>
    <artifactId>ehcache</artifactId>
    <version>1.5.0</version>
    <scope>compile</scope>
</dependency>
<dependency>
    <groupId>spring-cache</groupId>
    <artifactId>spring-cache</artifactId>
    <version>2.4.1</version>
</dependency>

Multiple Write and Single Read Mechanism

Just imagine: you are a successful financial company with a big entourage of traders slogging it out throughout the day. They make money for you and you would want to build a system that works as hard to help them. Sheer muscle may get you some where, but sooner or later you are going into big problems associated with building a distributed system that handles a huge-huge volume of data. It can be a hard to tame, resource hungry beast.

Lets say you want it to process the data supplied in a known format and send alerts/emails to all the concerned traders (some rule based events). You also want it to be accessible over the web to your team sitting in different corners of the country. On top of it, the speed is of paramount importance.

My team has solved a similar problem and it has been one hell of a experience. Don't know from where I should try to collate my experience. In each article I will summarize a problem we faced and the solutions that we opted for.

We were facing a nasty problem with our messaging system. Our system had three queues where messages were posted to and read from. Each of the queue had multiple consumers and the posting was done using a pooled connection factory (ActiveMQ). The problem was that after some time the consumers would refuse to pick any fresh messages from the queue, as if they were stuck forever. We suspected it to be an ActiveMQ bug, but at the end it turned out to be a problem with our threading and caching. Our threads were getting dead-locked blocking the ports from which a consumer could read anything. We were also trying to cache the whole method call instead of the results. Got it rectified and learnt a few more things about the threaded systems and caching. In the process learned quite a few things from my manager and colleagues. Thanks to Andrew and Ben.

First, we wanted to use a ReadWriteLock mechanism in which a number of threads write to a list for a particular amount of time after which another thread processes all the data written. When the data is being read, no thread can write to the common pool and the reader should allow any thread that was already writing to complete before attempting to read. We opted to use nuts-and-bolts multi-threading instead of delving deeper into java's lock mechanisms. Here is a generic class that can do that for you:

MultiWriteSingleRead.java

In the next post we will explore using a cache on the same class so as we process information selectively.

The Return of the Persistent Quartz Job

Finally I managed to understand the persistent job and make it work reliably: for your stateful job and the trigger volatility should NOT be true. Even setting durability to true can cause problems as it would persist orphan jobs that we may not need. If the intention is to just prevent loss of jobs during system re-starts, it is enough to set requests recovery to true. That's it. Simple is beautiful :).

Caching Method Invocation Results

Worked on setting up a cache for a method invocation fetching results from a webservice using Spring Cache AOP and EHCache in a Spring-Maven environment.Long live sourceforge! The spring cache java doc is pretty helpful and this article by Pieter Coucke expands on what you read in the java doc. You can also have a look at my complete configuration (which works) and is based on the article and the java doc.

And if you are using Maven, you would need to add dependencies for ehcache and spring cache.

<dependency>
    <groupId>net.sf.ehcache</groupId>
    <artifactId>ehcache</artifactId>
    <version>1.5.0</version>
</dependency>
<dependency>
    <groupId>spring-cache</groupId>
    <artifactId>spring-cache</artifactId>
    <version>2.4.1</version>
</dependency>

Another Way of Starting ActiveMQ Broker in Spring

I had posted a blog about running an ActiveMQ broker in Spring. Recently, I found an alternative way to start an ActiveMQ message broker in spring environment by injecting a bean based on org.apache.activemq.xbean.XBeanBrokerService:

<bean id="broker" class="org.apache.activemq.xbean.XBeanBrokerService">
   <property name="useJmx" value="true"/>
   <!-- this tells it to create a filesystem store -->
   <property name="persistent" value="true"/>
   <property name="brokerName" value="mySpringBroker"/>
   <!--the queues/topics that this broker starts with-->
   <property name="destinations">
       <list>
           <ref bean="destination1"/>
           <ref bean="destination2"/>
       </list>
   </property>
   <property name="transportConnectors">
       <list>
          <!-- an in vm connector which is nice and fast -->
          <bean class="org.apache.activemq.broker.TransportConnector">
             <property name="uri" value="vm://localhost"/>
          </bean>
          <!-- a connection for external clients specifying the IP of machine deploying the broker, something like tcp://10.10.10.10:61636. Here read from a properties file but can be hard-coded as well-->
          <bean class="org.apache.activemq.broker.TransportConnector">
             <property name="uri" value="${broker.transportExternalConnectorUri}"/>
          </bean>
       </list>
   </property>
</bean>

<bean id="destination1" class="org.apache.activemq.command.ActiveMQQueue">
    <constructor-arg value="com.saveenkumar.myQueue1"/>
</bean>

<bean id="destination2" class="org.apache.activemq.command.ActiveMQQueue">
    <constructor-arg value="com.saveenkumar.myQueue2"/>
</bean>

In fact,the package org.apache.activemq.xbean provides various options (beans) to start the broker in a spring environment.

The Wierd Persistent Quartz Job

I recently faced a problem with getting quartz to persist the jobs in case of a scheduler shut down. Penning down how it got solved (with Ben's help). 

My initial scheduler config looked like this.

Also, when I configured my stateful job, I set volatility and request recovery to true besides putting in job class, name and group. Sadly, the job never-ever persisted when the system re-started. 

Miraculously, this started working fine when config was replaced by this configuration (only scheduler config is different).

Plus when I configured the stateful job, I set durability to true and always put in job class, name and group. Of course you can add what you want to the data map (as long as it is Serializable). 

What I don't understand is that even when I have not set either volatility or request recovery to true, how setting only durability to true takes care of persisting the Job between shutdowns and startups. Whatever it takes to make it work... Quirky but may be helpful to you if you face a similar problem. Cheers.

Database Connection Pooling in Hibernate

I was looking for a way to control connection pooling in Hibernate inside a Spring environment. God bless Apache foundation for their amazing code helps, as I found the answers in DBCP. Which-ever hibernate session factory you use, use the following data source bean in your configuration xml to control connection pooling:

    <bean id="internalDataSource" class="org.apache.commons.dbcp.BasicDataSource">
        <property name="driverClassName" value="org.postgresql.Driver"/>
        <property name="url" value="jdbc:postgresql://127.0.0.1/myDatabase"/>
        <property name="username" value="root"/>
        <property name="password" value="abc123"/>
        <property name="maxIdle" value="5"/>
        <property name="maxActive" value="25"/>
        <property name="minIdle" value="5"/>
        <property name="initialSize" value="5"/>
    </bean>

For all options and details, refer the API for BasicDataSource. And, if you are a maven user, don't forget to include the dependency:

        <dependency>
            <groupId>commons-dbcp</groupId>
            <artifactId>commons-dbcp</artifactId>
            <version>1.2.2</version>
        </dependency>

 

Which IDE?

IDEs can be critical to productivity. As a developer, which is the one that I prefer?

There are three IDEs that I have used for development:  Idea, Eclipse and NetBeans.

Idea costs money, but it is worth it. Developing enterprise applications with it is easier than with other IDEs. Nevertheless it is not cheap and other IDEs are fast catching up with it.

Eclipse is too much customizable for its own good, making the "setting up of eclipse" so as it works with a plethora of commonly used enterprise technologies almost an art. Hence the learning curve can be sharp. If you are in a hurry to get the project running and don't have much experience with Eclipse , I would not recommend Eclipse. Eclipse is a good (and free) IDE and can be a developer's joy. Nevertheless it can be a pain for somebody new to it and not willing to spend so much effort customizing it.

I would have completely ruled out NetBeans before its version 6. In the past I saw it as a good Java GUI builder and thats it. But the recent release is impressive. It takes into account a lot of problems that are commonly faced using disparate technologies in enterprise development. It is configurable but comes with a lot of intelligent defaults that can get you up-to-speed pretty quickly. Still not as good as Idea, but is a good contender if you are not willing to spend. 

Parsing CSV File

I just wanted to read a simple CSV file in the resources folder and parse it in my spring application. A simple class to do the same. csvFileName is a string property injected at runtime.

CSV Reader

Errata: My good friend and colleague Ben has rightly pointed out availability of generic csv parsers that can make life easier. I am exploring Java CSV and Flatpack as suggested by Ben.

Java CSV API
Java CSV Sample