Mar 12, 2012

Maven Profile -conditionally exclude junits


Our dataaccess layer is dependent on special software components that are not available in every developer’s sandbox. By default we exclude these tests, but we still need our continuous integration environment (Bamboo) to run all the tests.  
We were able to leverage Maven’s Profile to conditionally exclude tests.

We created a maven property in our pom: exclude.tim.tests

<exclude.tim.tests>%regex[com.somecompany.dataaccess.usermanager.dao.*Test.*]</exclude.tim.tests>



We used this property in exclude configuration.
<plugin>
       <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-surefire-plugin</artifactId>
       <version>2.12</version>
       <configuration>
              <excludes>
                     <exclude>${exclude.tim.tests}</exclude>
              </excludes>
       </configuration>
</plugin>

Running, mvn clean install excludes these tests.

To include the tests, we created a profile IncludeSpecialTests and configured the test exclusion property with a dummy value (NothingToExclude)
<profiles>
  <profile>
    <id>IncludeSpecialTests </id>
    <properties>
      <exclude.tim.tests>NothingToExclude</exclude.tim.tests>
    </properties>
  </profile>
</profiles>

Running mvn install  runs all the tests.
mvn clean install –PIncludeSpecialTests

Mar 6, 2012

Injecting Mockitto mocks using Spring @Autowired


We are using @Autowired (by Type) for dependency management of our multi-layer/multi-tiered Spring 3 project. Here is how our interface/implementation look,

public interface UserProfileService {

       public abstract UserProfileDO getUser(String userId) throws UserNotFoundException;

}

@Service
public class UserProfileServiceImpl extends BaseService implements UserProfileService, InitializingBean
{

       @Autowired
       private UserProfileDAO userprofileDao;

       @Override
       public void afterPropertiesSet() throws Exception
       {
              Assert.notNull(userprofileDao, "UserProfileDAO not injected.");
       } 
      
       @Override
       public UserProfileDO getUser(String userId) throws UserNotFoundException
       {
              IMPersonEntity person;
              try
              {
                     person = userprofileDao.getUser(userId);
              }
              catch (DataAccessException e)
              {
                     throw new UserNotFoundException(userId);
              }
              if (person == null)
                     throw new UserNotFoundException(userId);

              UserProfileDO userProfile = (UserProfileDO) objectConversion(person, UserProfileDO.class);
              return userProfile;
       }

       public void setUserprofileDao(UserProfileDAO userprofileDao)
       {
              this.userprofileDao = userprofileDao;
       }

}

UserProfileService has two dependent beans
·         UserProfileDAO: Needs to be  mocked. This is a data-access layer which is implementated using  IBMs Tivoli APIs and LDAP.
·         dozerObjectMapper: This dependency is from the BaseService, which converts the Entities into DomainObject. We don’t intend to mock this.

To configure Mockitto, I  started by using @Mock annotation but in my test I needed a mix of @Autowired and @Mock. I  don’t think MockitoJUnit44Runner loads the beans from ContextConfiguration. Also, I didn’t find any special value on using @Mock annotation (This is my first mockitto test, maybe I will realize its value later).

@RunWith(MockitoJUnit44Runner.class)
@ContextConfiguration("classpath:mock-userprofile-service-context.xml")
public class UserProfileServiceTest implements InitializingBean
{
       @Autowired    private UserProfileService userDetailsService;
       @Mock         private UserProfileDAO mockUserDetailsDao;

Other issue with my test was that my ServiceInterface (UserProfileService) did not have setter for DAO ( as per desing), that means I cannot set my mockObject on the service. I overcame this problem by using (abusing) Reflection.
So, I changed my test to switched back to SpringJUnit4ClassRunner (removed @Mock annotation) and I used reflection to set  mockObject to the testService.

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration("classpath:mock-userprofile-service-context.xml")
public class UserProfileServiceTest implements InitializingBean
{
       @Autowired    
private UserProfileService userDetailsService;
       private UserProfileDAO mockUserDetailsDao = mock(UserProfileDAO);

@Test
       public void getValidUser() throws UserNotFoundException, EntityNotFoundException, MultipleEntitiesFoundException
       {
              IMPerson mockPerson = new IMPerson();
              mockPerson.setDisplayName("test cca");
              when(mockUserDetailsDao.getUser("cca_test_user")).thenReturn(mockPerson);
ReflectionTestUtils.setField(userDetailsService, "userProfileDAO", mockUserDetailsDao);
              UserProfileDO user = userDetailsService.getUser("cca_test_user");
              assertNotNull(user);
              Assert.assertEquals("test cca", user.getDisplayName());

       }

This got me going, I could run tests with mock objects. I still wanted to get rid of reflection and find a way to Autowire mockObject (maintain consistency across application).
So,  I moved my mockbean creation to  test context and had @Autowired in service detect the required mock Dao (no more Reflection). I still needed a reference to mock bean in my test, to configure mock conditions.
This is how my final test context looks, with mock bean creation.

mock-userprofile-service-context.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-2.5.xsd">


<bean  id="mockUserProfileDao" class="org.mockito.Mockito" factory-method="mock">
    <constructor-arg value="com.company.dataaccess.userprofile.dao.UserProfileDAO" />
</bean>

<bean class="com.company.service.userprofile.UserProfileServiceImpl"/>
<bean class="org.dozer.DozerBeanMapper"/>

</beans> 
This is how my  final test looks

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration("classpath:mock-userprofile-service-context.xml")
public class UserProfileServiceTest
{ 
       @Autowired
       private UserProfileService userDetailsService;
       @Autowired
       private UserProfileDAO mockUserDetailsDao;
      

       @Test
       public void getValidUser() throws UserNotFoundException, EntityNotFoundException, MultipleEntitiesFoundException
       {
              IMPerson mockPerson = new IMPerson();
              mockPerson.setDisplayName("test cca");
              when(mockUserDetailsDao.getUser("cca_test_user")).thenReturn(mockPerson);
              UserProfileDO user = userDetailsService.getUser("cca_test_user");
              assertNotNull(user);
              Assert.assertEquals("test cca", user.getDisplayName());

       }
       @Test
       @ExpectedException(UserNotFoundException.class)
       public void getMultipleInValidUser() throws UserNotFoundException, EntityNotFoundException, MultipleEntitiesFoundException
       {
              when(mockUserDetailsDao.getUser("multipleUserids")).thenThrow(new MultipleEntitiesFoundException());
              userDetailsService.getUser("multipleUserids");
              fail();

       }
       @Test
       @ExpectedException(UserNotFoundException.class)
       public void getInValidUser() throws UserNotFoundException, EntityNotFoundException, MultipleEntitiesFoundException
       {
              when(mockUserDetailsDao.getUser("nulluser")).thenReturn(null);
              userDetailsService.getUser("nulluser");
              fail();

       }
      
}

Happy testing J

Feb 28, 2012

JIRA filter for Daily Scrum

Here is a way to configure a simple JIRA filter to help identify items that have to workon (touched) since last scrum.
In the Issue Navigator create a query
  • Select the appropriate project 
  • Go to Dates and Time section
    • Set Update From -1d To 0
  • Save the Filter (ex DailyScrumFilter)

Now you can use this filter in Agile (grasshopper) tab.
  • Create a new Context 
    • Open the Context dropdown  (the dropdown next to Project dropdown)
    • Click New link (Open a window)
      •  Give a name to the context ex "DailyScrumContext"
      •  Under filters select the DailyScrum filter
      •  You can even give some criteria for Highlight (I highlighed the inProgress issues)
      • Save the Context (you can check the sharing if you want others to use this context).
Now you can use DailyScrumContext during your scrums to see all the items that were modified since last scrum and highlight the once in Progress.

Feb 27, 2012

Encrypt/Decrypt Properties using PropertyPlaceholderConfigurer

Spring allows for configuring context bean properties by directly injecting property values from properties file through PropertyPlaceholderConfigurer. At high level this is how it is configured

In the context, you will need to configure a property-placeholder with location poitning to the required properties file. Then we need to provide the property key for the require bean property .

<context:property-placeholder location="classpath:my.properties"/>
<bean id="myBean" class="MyBeanClass">
<property name="userid"
value=" ${userid} " />
<property name="password" value="${password}" />
</bean>

where my.properties will have
user=myuserid
password=mypassword.

works great.
Now, if I the password is encrypted in the properties file and needs to be decrypt it before setting it as the bean property.

There are various approaches to doing it like Over-riding mybean which would decrypt the password during initialization or write a custom ProperOverrideConfigurer to decrypt the property.

There is one another way that can be use, by creating a custom PropertyPlaceholderConfigurer
(extends  PropertyPlaceholderConfigurer  ) and override convert method.

example:

public class DecryptPropertyConfigurer extends PropertyPlaceholderConfigurer
{

@Override
protected void convertProperties(Properties props)
{
Enumeration<?> propertyNames = props.propertyNames();
while (propertyNames.hasMoreElements()) {
String propertyName = (String) propertyNames.nextElement();
String propertyValue = props.getProperty(propertyName);

String convertedValue = decrypt(propertyValue);
if (!ObjectUtils.nullSafeEquals(propertyValue, convertedValue)) {
props.setProperty(propertyName, convertedValue);
}
}
}

}

Now, move the properties that need to be decrypted to a new property file
my_encrypted.propeties
password=myencryptedpassword

and configure this property file in the custome PropertyPlaceholder (you will still need to configure the regular property place holder with my.properties)
sothe spring context would look like


<context:property-placeholder location="classpath:my.properties"/>
<bean id="myBean" class="MyBeanClass">
<property name="userid"
value=" ${userid} " />
<property name="password" value="${password}" />
</bean>




<bean class="DecryptPropertyConfigurer">
<property name="location" value="classpath: my_encrypted.propeties"/>
</bean>



Sep 11, 2011

NoSQL - Rapid Application Development Platform




Lately, I came across a lot of discussion around NoSQL's slow enterprise adoption, this prompted me to blog my experience in trying to add NoSQL in our enterprise’s toolset. 

We are working on a program to modernize our systems/applications using Web2.0/Restful services and in the process build a highly automated agile development platform. 

As part of this modernization, we came across certain use-cases that required extensible schema support. The rigid schema of our legacy systems lead to highly complex solutions to handle these use-cases. 


The uses-cases we came across were, shopping cart with extensible line items and a highly customizable customer's metadata repository. The customization of the metadata ranged from number of fields, definition of fields and the data to which the fields can be associated with. This customer metadata repository is a large data-set (2.3 billion records) spread across large set of vertically partitioned databases (80+). Our end-users can also search on these metadata fields through our Web2.0/RESTful application. We are using Apache Solr 3 (migrating from Oracle text) for our text search solution.

In legacy system, the schema flexibility was achieved by adding few spare fields in various tables or by using key-value schema design. Either of the approaches are not elegant. We also considered storing metadata as blob with customizable fields and their definitions, which again, wasn’t the optimum use of RDBMS. 


This need for schema flexibility lead us to NoSQL. 


We are indexing all of the metadata in Apache Solr. Solr was giving us high performance near-real time searching at the same time providing a consolidated (80+ databases) and denormalized (documented oriented) view of the data. So my first instinct was to consider Solr as the NoSQL solution. However, we are still in the middle of the migration from Oracle and that option looks pre-matured.

Moving on to the real NoSQL products (there are so many of them), I considered Apache Cassandra, it seemed most matured and adopted amongst all NoSQL solutions. At the time, I did not find any HTTP interface for it and using its Java API did not look all that easy to get started. Moreover, Apache Cassandra came across as a BigData solution for high volume and velocity data, geared more for social media type scenarios. We had more of a denormalization and unstructured data-storage problem.

Next in the line was Apache CouchDB, it had an HTTP / JSON interface and was easy to get started with. Implementing a PoC with CouchDB was quite easy, I already had Rest Services with data as JSON generated from JAXB . For persistence, I just did HTTP Post with the JSON document and unique rowid. There was no database schema design, no ORM layer, JPA, transaction or any of the persistence stuff need for RDBMS. 

Flexible schema and HTTP interface in itself, seemed a big reason to use NoSQL, in additional there was all the goodness of scalability and replication. This got me thinking about the price that applications have been paying for a Rigid Schema and JDBC/ODBC driver based db interface. 

The data access layer is usually the most complex layer of any application (instead of business logic layer), mostly due to the complexities of ORM, transaction/concurrency management etc. Also, to get any meaningful data from normalized schema, an application needs to perform joins, which adds to the complexity ex. Order Line in itself does not give any meaning unless it is joined with Order Header. 

I think, normalization and schema-design should be an optimization step and not an upfront design activity. In initial stages of application development  one should start will NoSQL database, once all the data storage and retrieval needs for an application are built, one can design an optimized schema.


Another aspect of schema design is that in modern architectures, system-data is not directly exposed to external entities. It is abstracted by a services or data-access layer for centralized enterprise data-access. In legacy systems, enterprises were trying to build a centralized enterprise database with standard schema that provided a consistent data format for all enterprise applications. As various systems and applications were directly accessing the data, there was a need for rigid schema.


I think, enterprises will be more interested in NoSQL if it is presented as low cost, rapid application development platform that provides business agility and cost savings due to reduced application complexity. 

In my case, even though management realized the value of NoSQL, at the time we did not go ahead with it, for all the various reasons discussed in this discussion thread

To state a few, at the time NoSQL solution did not have the maturity and enterprise readiness (support, management, monitoring etc.).
The overall organization impact in having a NoSQL platform was not well understood, ie the skillset change, changes in responsibility of organizational units (application developer, middleware admins, database admins/data modelers). 
There was some concern of having an open source product in the most-secured enterprise layer, most enterprises are still slow in adopting OpenSource. 
There was also lack of success stories of enterprises using NoSQL, the few that were there, were mostly in the context of BigData and Social media type problems.
  
Since then, there has been a lot of progress and maturity in NoSQL technology and some of the above concerns are now addressed. 

NoSQL is an impressive technology and can provide a lot of value to enterprises; it just needs a Don Draper to make that push.