Remote Client Web Performance Monitor

Selenium is an excellent open source tool for automated functional testing of web applications. We use it along with JBehave for automation and  Behavior Driven Development of our large-scale browser-based application.  The tests are run from Atlassian Bamboo and we use Sauce Labs for the cross browser testing. This setup allows for a full regression testing of the application on all supported browsers with very little manual testing. This automation, along with high-coverage unit-test suite (Junit/Mockito) and puppet-based one-click deployment, is core to our continues improvement & delivery platform.

We are also using selenium to measure Page Load time of our Single Page Web application (ExtJS4.0).  Selenium WebDriver provides more reliable page load time then Selenium RC, as it uses browser's native support for automation as oppose to injecting JavaScript.

This extends Selenium ability to act as an automated performance testing tool, it can  validate ( or just log) performance of the application, along with performing functional validation.

We went a step further with Selenium implementation, were we used its WebDriver to run tests on any remote client browser to understand the real performance of the application. The user is given a test URL that runs the application via Selenium webdriver on the client browser and collect required diagnostic data.

Selenium has a remote server, which allows the selenium tests to run on remote browser. However, the remote server needs to be installed and running on the client machine, and we did not want that. We needed a reverse solution, where selenium tests run on a remote server and the application runs on the local browser.

This was solved by creating a simple web app (servlet) that can identify  its browser (using user-agent request header) and based on browser type, invoke appropriate webDriver. It would then run the application and collect load time metrics. This metric is sent via HTTP/JSON to CouchDB. I am a big fan of  CouchDB, it provides rapid application development and  schema flexibility for tool’s feature extension. 

I am searching a way to reuse my existing Selenium scripts for Load Testing, maybe run them through JMeter and capture response time in CouchDB. For now, we are using HP LadRunner with AJAX extension and it is functioning well.

Client-Tier Performance Tuning of Single Page Applications

JavaScript (Browser-based/Single Page) applications are new generation applications that tap on the browser’s processing power to provide rich user experience and fast user interaction. However, these applications do have slower initial page load then traditional applications. As  client-tier is handling more processing in these apps, it plays a bigger role in application responsiveness; client-factors like browser, network, processing power can slow down a request even before it ever reaches the server, even a well-performing back-end, might give a bad performance to certain clients. Along with server optimization ( lean architecture), these applications require client-tier performance tuning to have fast application response time. 

A browser’s main function is to get requested web resources, parse the received content, execute JavaScript and display the content in its window. It has a browser engine, a rendering engine, a JavaScript engine and networking component to fetch content from server resources via HTTP requests. The HTTP request to fetch a resource involves a DNS name lookup, SSL negotiations, setting up TCP connections, transmitting HTTP requests, content download, and fetching resources from cache. Browser processing (rendering/JavaScript execution) is a single threaded process, except network operation, which is done in parallel. However, browsers do have a limit on the number of parallel network connections to a domain, varies from 2 -6 connections. Browsers have their own implementation of these components which is why response time and rendering of same content differs based on browsers.

In order for JavaScript to be executed on the browser it first needs to be transferred JavaScript source code from the server to the browser. This not only cause delays due to network latency, but also leads to synchronous execution of the page. 

When browser processes a downloaded JavaScript resource (script tag), it blocks all other JavaScript/CSS resources till that particular JavaScript is parsed and executed. 
This becomes a major issue when using AJAX toolkit that have large JavaScript libraries.
Also, browser-based applications have large application specific JavaScript code. Usually this code is modularized into smaller more manageable JS files. This means there is a large JavaScript code, distributed in small JS files that need to be transferred from the server to the browser.

So download and processing of JavaScript code itself increases network latency and can be a major bottleneck during page load.

The most efficient solution for this is to combine multiple JavaScript/CSS files into a single large file, which is compressed and minified before being transferred to the browser. 

There are several frameworks available for this, we used JAWR. JAWR provides server side configuration to combine multiple files (JS or CSS) into a single file called bundle. These bundles are generated during server startup and are invoked by calling JAWR tag libs replacing <script> tags. JAWR also applies minification and compression to the bundles. 

We bundled all of the application JS code together with ExtJS into a single JS bundle. 
We created another bundle for our client-side translation code: I8N JavaScript resource bundles and ExtJS locales. The bundle for each language had to be defined separately, otherwise properties would be overwritten. JAWR does have an I8N message generator that can optimize translation bundling, but we did not use it because we had a custom solution for client-side translation. 
We used OWA for client-side event tracking. It had few JavaScript files that need to be included in the application pages to allow OWA to track the events on that page and send the events to a centralized OWA server. We had to do some custom coding, but were able to include OWA JS in our application JAWR bundles. 
We had a single CSS bundle for all the CSS files in the application. This reduced the number of sever calls from the page and also there was less blocking for JS execution. 
We saw a major performance boost to application page load with this change.

The next performance tuning was to manage the number of HTTP requests from the page. As browsers have limit to maximum parallel requests that it can send to a single domain, this can block the AJAX requests on the page and cause delays.

Typically, most of resources on a web page are images, so we need to find ways to reduce network calls for images. The easiest approach is to cache images, because they rarely change. We configured cache control header to cache images and other non-changing static content on the browser. We also use Content Delivery Network –Akamai as a Web Proxy cache and for Geo-accelerates content delivery.

For no caching (first application run) situation, one of the approaches  to reduce the number of images related HTTP calls that we tried was JAWR Image Sprite, we faced few UI complexities with this approach and did not use it.

We were severing static content and dynamic content from the same domain, so we tried to move the images into a separate domain (cookie-less and non-SSL). This lead to mixed content (HTTPS/HTTP) issue, which is not secure and certain browser presents a user prompt for this. Changing image's domain to be secure (HTTPS) increased the SSL negotiation time which negated any gain that we got by increasing parallel network calls from the browser.

We finally implemented a solution to  perloaded/ pre-cache the images in a previous page (login page). The images are preloaded and avaliable in the browser cache before the page that displayed them was called. This reduced the number of HTTP calls on the main page. Also we moved the images to an un-authenticated host, which reduced load on the back-end authenticated server and we reduced a network hop. This gave us a major performance improvement, especially in IE browsers. This might not a generic solution, but I guess using Image Sprites should give the same results.

The images need to be optimized during UX design time, to have smaller size and proper format. However, at times, one might get more latency for smaller images than for larger ones. This can happen when the client gets asymmetric bandwidth and the content response size is not proportionately bigger than the request size as expected by asymmetric (upload: download) ratio.

Along with these changes, we had also tuned back-end (HTTPD/tcServer) to handle large number of concurrent HTTP requests by configuring KeepAlive, Connection timeout and maxThreads/maxClients. On tcServer, we configure Non-Blocking IO connector which is optimized to handle large number of concurrent HTTP connections.

With all these optimizations, our large scale enterprise web application now has a high-performing and rich front-end with lean and scalable back-end.

Note: Checkout Google Chrome Frame if you need to provide new functionlity on legacy browsers.

Custom MBean configuration for Connection Pool monitoring

We ran into some interesting problems in configuring Hyperic to monitor JDBC connection pool via JMX. I would like to share the details, hoping it would save time for others.

When the connection pool and data sources are configured in the container using JNDI, Hyperic is able to identify pool MBeans automatically. However, if the pool configuration is within the application (war) and not in the container, MBeans will not be visible in Hyperic. To make them available in Hyperic one will need to configure JMX plugin for Hyperic.

Depending on the connection pool, you might need to register the MBeans explicitly. Some connection pools (c3p0) have auto-registered MBeans, so if you use them, the MBean will be auto-registered and become available to JMX tools like JConsole. However, these MBeans have dynamic object name that changes on server restart. This would be a problem in cases where you need a static MBean name to be able to identify it for setting up monitors and alerts. Hyperic does have a patch to allow monitoring MBeans with dynamic name http://communities.vmware.com/thread/389027, but this patch needs to be installed on the enterprise tcServer installation. 

If you do not wish to use the patch, you will need to register connection pool MBean explicitly with a static name.

Registering (export) the MBean with JMX server can be tricky and can lead to sever connection leak problem. One needs to be careful in selecting the MBean that should be exposed. Typically, the datasource beans itself has all the required connection attributes that one needs for monitoring like number of connection Idle/Used. But this datasource bean also has additional methods like getConnection*, that actually borrows a connection from the pool. 

If this data source is directly registered, and then monitored by JConsole (or likes), it may cause these getConnection* methods to be invoked and thus lead to connections being  borrowed from the pool without ever being released. Tools like JConsole are configured to display all the exposed attributes for the registered MBean in its attribute tab by calling the assessors  to get the attribute value. If the bean had assessors like getConnection*, Jconsole will invoke them, causing connections to be borrowed.

To avoid this connection leakage problem and have a static MBean name that can be used for monitoring and alerts, create a custom MBean. Configure this custom bean to have just the attributes that are needed for monitoring and alerts (caution: do not extend the data source bean).

Here is the JMX configuration for a tomcat - pool that is working well for us through JConsole and Hyperic.

       <beans:bean id="exporterOracle"   class="org.springframework.jmx.export.MBeanExporter" lazy-init="false">

              <beans:property name="beans">

                     <beans:map>

                           <beans:entry key="bean:name=OracleConnectionMBean" value="#{dataSource.getPool().getJmxPool()}"/>        

                     </beans:map>

              </beans:property>

       </beans:bean>

<beans:bean id="dataSource" class="org.apache.tomcat.jdbc.pool.DataSource "

              destroy-method="close">

              <beans:property name="driverClassName"   value="${oracle.db.driver}" />

              <beans:property name="url" value="${oracle.db.url}" />

              <beans:property name="username" value="${oracle.db.user}" />

              <beans:property name="password" value="${oracle.db.password}" />

              <beans:property name="jmxEnabled" value="true" />

              <beans:property name="initialSize" value="${oracle.pool.initialsize}" />

              <beans:property name="maxActive" value="${oracle.pool.maxactive}" />      

              <beans:property name="maxIdle" value="${oracle.pool.maxIdle}" /> <!-- maxIdle needs to be same as maxActive -->

              <beans:property name="testOnBorrow" value="true"></beans:property> <!-- Needs to be set to prevent Connection timeout exception -->         

              <beans:property name="validationQuery" value="select 1 from dual" /> <!-- This needs to be set if testofBorrow is set -->

       </beans:bean>

Check that the MBean name is static "bean:name=OracleConnectionMBean" and bean that is exposed is specific JMXPool interface of the datasource (dataSource.getPool().getJmxPool()), not the datasource itself.

Protecting Web Resources

We are working on a modernization program, as part of which we have built a new enterprise Portal that replaces existing Websphere Portal. This new Portal is a simple Web20 application with ExtJS4.0/Spring/Restful services running on the VMware Tc Server.

This migration required replacing Webspere Portal components with light-weight open source products.  However, WebSEAL was one product that was impressively lightweight and helped us solve some complex integration problems.

WebSEAL is the resource manager that acts as a reverse Web proxy. It receives HTTP/HTTPS requests from a Web browser and delivering content from its own Web server or from junctioned back-end Web application servers. Web Requests passing through WebSEAL are authorized by the Tivoli Access Manager.

We were already using WebSEAL for authentication, single sign-on and high level HTTP URL authorization. In new Portal, we extended its use for integrating third party application with Portal.

Websphere Portal (WSP)  used WebClipper technology  to integrate and render third-party applications. WebClipper runs from within the portal server and manage session and identity across Portal and third-party apps. This solution was highly complex and created tight-coupling between app and the Portal.

In the new Portal we replaced WebClipper with WebSEAL-based integration, where we created a WebSEAL junction for the application, and used the secured junction URL to be rendered the app as  ExtJS Tabs within the Portal. WebSEAL provided secure rendering and session management for the app & Portal. WebSEAL also passed authentication token to the app, as trust HTTP Headers, eliminating the need to engage Portal server. 

We even used WebSEAL as an operational tool to control user traffic at run time.  We plan to use WebSEAL to display error pages or redirect traffic to a specific web resources. While this can be done on any webserver, WebSEAL allows the redirection based on fine grain permissions/ACLs.

WebSEAL is an excellent product, I am not sure if there is any open source product that can provide the same capabilities. Its only limitation is that it is tightly coupled with Tivoli Access management and enforces security policies against just Tivoli Access Manager, also that it is not open source.

Spring Security does provide similar web security against any policy manager and can be configured as an HTTP reserve proxy. However, Spring Security protects web resources within its own web context and needs an application server. If you need to protect third-party web resources/application using Spring security, you will need to stream the HTTP request and attach security headers to it. For protecting simple webservices  this approach will work. But for protecting web applications, streaming can create complexities. Moreover, you will need to serve application’s static content through the app server.

As the trend of HTTP interface grows, products  provide HTTP/JSON interfaces, ex Solr, CoucbBase , so will the need to non-instrusively secure web-resources . I would like to know how others are solving this problem.

Sonar with mysql

I wanted to share few details on configuring Sonar to use mysql. It is quite easy, just follow documentation available at  http://docs.codehaus.org/display/SONAR/Installing+Sonar

On sonar startup, if you get this error 

Cause: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Unknown database 'sonar')

It might mean that sonar database is not configured properly. Try creating the database by running the scripts  available @ https://github.com/SonarSource/sonar/tree/master/sonar-application/src/main/assembly/extras/database/mysql.

Now on maven side , you need to add this to the setting.xml

<profile>

            <id>sonar</id>

            <activation>

                <activeByDefault>true</activeByDefault>

            </activation>

            <properties>

                <!-- SERVER ON A REMOTE HOST -->

               <sonar.jdbc.url>jdbc:mysql://localhost:3306/sonar?useUnicode=true&amp;characterEncoding=utf8</sonar.jdbc.url>

                <sonar.jdbc.driverClassName>com.mysql.jdbc.Driver</sonar.jdbc.driverClassName>

                <sonar.jdbc.username>sonar</sonar.jdbc.username>

                <sonar.jdbc.password>sonar</sonar.jdbc.password>

                <sonar.host.url>http://localhost:9000</sonar.host.url>

            </properties>

 </profile>

 If you get the error ‘Cannot load JDBC driver class’ on running mvn sonar:sonar  

 check for the driver class name. 

If driver class is derby or H2, it means the mysql configuration is not picked by from setting.xml. Try giving profile for sonar explicitly and including the path to setting.xml, this would ensure that the mysql settings is pulled up by maven.

If however the driver class name in the error is 'com.mysql.jdbc.Driver', this means that the profile is read properly. This error is mostly because of incorrect JDBC url; eventhough the eror messages makes you think that you need to copy the driver file somewhere for maven to pickup. 

In my case I had my jdbc url as jdbc:mysql://localhost:3306/sonar, when I changed to jdbc:mysql://localhost:3306/sonar?useUnicode=true&amp;characterEncoding=utf8 everything worked fine.

Single Page Applications: Fat Client - Slim App

I have been working on Single Page Applications (SPA) for some time now. These applications have not just changed the user interface design, they have also changed the server architecture; server becomes lean and client picks up extra weight. 

Single Page Application provides rich, fast and interactive user interface, they emulate a desktop application in look/feel and server interaction. These applications do not require a page load between user interactions, UI components of the page are loaded up front and the user sees the page instantly. The browser continues to load the page while making multiple asynchronous server calls to gather page data. With these concurrent asynchronous severs calls; the overall page load is faster too. 

Traditional web applications follow a request-based model where the server responds to a request by selecting, generating & delivering the next view. The server gathers all the data needed to display the next page and hands it to the UI.

In SPA, UI has the responsibility of selecting, generating and delivering the next view. It makes fine-grained service call to gather data needed to respond to user events.

In traditional web application, server managed the client state through HTTPSession or state-full session beans. In SPA, this responsibility moved to UI, user’s state is maintained browser-side.

Now, that the server no longer manages state or performs view control, it just provides data-feeds to UI in response to the user events. The server is a simple stateless web service application. Such applications have a simple cluster configuration; there is no overhead of session replication or sticky session.

In my case, I did not go completely stateless; I had a state aware Application Tier in front of the stateless services tier. This tier is a Spring MVC web application, which interacts with UI/ExtJS and proxies’ service calls to the domain services (Apache CXF).  It is deployed in the VMware cluster with sticky session and domain services are deployed in a simple cluster.

These stateless web service applications have a low memory footprint which helps in simplifying the JVM configuration.

SPA store page data in browser and UI makes service calls to get delta data. As the data is already in browser the number of service calls are less. Also, the service calls are fine grained, so the network bandwidth consumption is low. 

Single Page Applications make server architecture simple and lean. However because of chatty UI-service interactions, SPA requires certain server configurations that were not needed in the traditional web application. I plan to write about it in my next blog. 

Security annotations in Spring Controller methods

There are few things you need to know for securing your controller methods using security annotations.You will need to first configure global-method-security in you context

<sec:global-method-security    secured-annotations="enabled" pre-post-annotations="enabled"/>

If you method annotations are @Secured set secured-annotations=enabled and if you plan to use @PreAuthorize or @PostAuthorize enable pre-post-annotations=enabled.

The placement of this tag is also important, you need to place this tag in your mvc-context.xml instead of security-context.xml, your mvc-context.xml is the one configured for the DispatcherServlet.

Web.xml

<context-param>

              <param-name>contextConfigLocation</param-name>

              <param-value>/WEB-INF/spring/mvc-context.xml /WEB-INF/spring/security-context.xml</param-value>

       </context-param>

<!-- Processes application requests -->

       <servlet>

              <servlet-name>appServlet</servlet-name>

              <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>

              <init-param>

                     <param-name>contextConfigLocation</param-name>

                     <param-value>/WEB-INF/spring/mvc-context.xml</param-value>

              </init-param>

              <load-on-startup>1</load-on-startup>

       </servlet>

There are suggestions to place global-method-security tag after component-scan tag in your context, but I didn't think it made any difference in my case.

<context:component-scan base-package="com.ironmountain.imconnect" />

<sec:global-method-security    secured-annotations="enabled" pre-post-annotations="enabled"/>

After this, you might hit Add CGLIB to the class path exception during initialization:

Initialization of bean failed; nested exception is org.springframework.aop.framework.AopConfigException: Cannot proxy target class because CGLIB2 is not available. Add CGLIB to the class path or specify proxy interfaces.

If so, you will need to add dependency for CGLIB.

The reason you need CGLIB is that, by default, Spring uses Java Dynamic Proxies to implement Aspect Oriented Programming support. This requires you to "program to interfaces rather than classes".

But we like to leverage component-scan wiring for controllers using @Controller annotation, which means controller are concrete classes and not implementation classes.

Meaning we need to generate class level proxies for security annotated classes, which requires CGLIB dependency.

So all this got my security annotation to work for access denied scenarios, but for  success scenarios I still got this error after the method was properly executed.

No mapping found for HTTP request with URI [/imchome/app/users] in DispatcherServlet with name 'appServlet'

My controller was declared as

@Controller

@RequestMapping(value = "/users")

public class ManageUsersController implements InitializingBean

{

And I had configured Tuckey’s URLrewriting filter to clean the URL from /imchome/app/users to /imchome/users.

My Controller worked perfectly fine, if I removed the secured annotations.

After hours of debugging, I realized that the problem was that my controller was implementing InitlizingBean interface.

By implementing InitlizingBean interface my controller became an implementation class, which should be wired through interface using spring’s dynamic proxy. Since I was creating the controller using component-scan as a concerete class, it was giving problem.

My solution was to not have my controller implement InitlizingBean.

So to get the security annotation to work (Spring AOP), you either need to have an implementation class created through its interface using Spring’s dynamic proxy or have a concrete class which generates a dynamic class proxy using CGLIB. PS: even if you implement a blank interface, you will need to follow the first option.

Portal - Crushed under its own weight

Enterprise Portal is a web interface that provides single-entry for the enterprise’s products and services to its end users.Sounds like a simple web-page with links to enterprise’s applications, but then you hear about Portal Technology, Portal Server, Portal Standards, Portal Developers …., and  the scariest  of all "Portal Cost". It makes you wonder, how can a simple webpag,e be so outrageously expensive and highly complex.

It may have something to do with the way Portal Products are sold. Portal vendor sell Portal as packaged bundle of tools that allow enterprises to quickly develop and deploy their portals. A typical Portal toolset consists of tools for Content Management, Identity & Access Management, Personalization & Customization, Federation and Integration.

 All these tools do sound like something an enterprise would need, I am not sure i fthat is always the case.The fact is that inspite of the high investment in Portal, most enterprises are far from reaping its benefits. 

·         Not so quick development and deployment: Due to high cost and bulky infrastructure, there are usually limited shared test environments, causing projects to contend for testing time and wait for availability of test environment. It is hard to make all portal components available on developer’s workstation, this impacts development and unit testing time, impacting project schedule and code quality.

·         Limited Skillset availability : Portal technology requires specialized skillset that are hard to find. Also, in most cases there is need for engaging product consultants, which adds to the cost and can be time consuming.

·         Not so easy: The so called out-of-box and ready to use portal features don’t look as easy when it comes time to implement them. Application Integration is one of the basic capabilities of most Portal products, but technologies for integrating non Portal apps like WebClipper and WSRP are way too complex to implement. There are times when vendors themselves discourage to use these solutions.

·         Not real use-cases that can leverage Portal paid-for features like Personalization.

If one is to compare the investment in Portal vs the features that enterprises actually leverage, it becomes apparent that existing Portal is way too complex solution for the problem in hand (single entry point).

It has been over a decade since Portal products came to market; a lot has changed since, especially in open source world. There are better technology options for quick development and deployment of portal features. Some suggestions of open source alternatives for building your own portal on a regular Application server (I used VMWare’s tcServer):

Web2.0 toolkits –ExtJS -Provide easy and customizable User Interface, which is way cooler and easy to implement then Portlets. It comes with an excellent Portlet APIs that will have you Portal up in no time, even on an Apache WebServer, thaz too skinny..
ExtJS also have great support for application integration (alternative of Portlet)-  ExtJS Tab Panel allows you to open a browser-like tab within the your application, which can be managed independently. 
We also used ExtJS iFrame Architecture to embed an existing app (alternative of WebClipper/WSRP)

Spring Security provides an excellent option for Identity and Access Management.

Spring MVC/ REST services – Rendering services for UI.

Federated Search – Solr is an excellent solution for federated search.

Content Management - Alfresco

I think enterprises should re-assess their ROI from Portal investment against these Open Source alternatives. Enterprises are in a better position to avail the flexibility of selecting just the right tools that meet their specific Enterprise Portal needs and thus keep the their cost low and stay lean.