How To move to Spring Boot 2 and Flyway 5 from Spring Boot 1.5 and Flyway 3

I had to figure this out the hard way, as soon as we upgraded our project from Spring Boot 1.5 to Spring Boot 2, all our existing migrations written using Flyway 3 started bombing.

After reading the docs I realised that the migration schema used by Flyway 3 isn't compatible with Flyway 5. What's more, there is not direct upgrade path from Flyway 3 to Flyway 5.

According to the official documentation we need to do the following:

• First upgrade your 1.5.x Spring Boot application to Flyway 4 (4.2.0 at the time of writing), see the instructions for Maven and Gradle. 

• Once your schema has been upgraded to Flyway 4, upgrade to Spring Boot 2 and run the migration again to port your application to Flyway 5.

This essentially means we will have to do two releases, which I didn't really want to do. Another approach to getting this fixed is described here. I haven't tried it but it should work.

For me, all I wanted was to get my project upgraded and working. I had a few migrations but I also had lots of DB snapshots which we could use to restore the old database. Hence, I decided to take the easy way out. I looked for ways to ignore the existing migrations, in the Flyway documentation we were able to find two settings that could be used. 

# Whether to automatically call baseline when migrate is executed against a non-empty schema with no schema history
# table. This schema will then be initialized with the baselineVersion before executing the migrations.
# Only migrations above baselineVersion will then be applied.
# This is useful for initial Flyway production deployments on projects with an existing DB.
# Be careful when enabling this as it removes the safety net that ensures
# Flyway does not migrate the wrong database in case of a configuration mistake! (default: false)
flyway.baselineOnMigrate=true

# The version to tag an existing schema with when executing baseline. (default: 1)
flyway.baselineVersion=20200131132000

As the documentation suggest, basically it will ignore all the migrations till the version 20200131132000 and thats exactly what we wanted to do.

When using these settings with Spring Boot you need to append "spring." to them hence, the final settings added in application.properites file are:

spring.flyway.baselineOnMigrate=true
spring.flyway.baselineVersion=20200131132000

That did it for us, the project was up and running with Spring Boot 2 and Flyway 5. After this point if we wrote new migrations those worked as well.

How to setup a fixed shutdown sequence with Spring

Recently, I was faced with an interesting problem.

• Our application server was listening to a queue
• The queue used to receive messages from different sources
• The requirement was very simple, when the application server is going down (this could happen because of many reasons, may be because we brought the site down, or we are deploying an updated software on to our application server) we do not want to miss any message from the queue.

To achieve this we just need to obey the following rules when the application server is going down.

• We should stop consuming any further messages from the queue.
• We should wait for the processing to complete for all messages that have already started processing.
• After these two rules are satisfied we should continue with the shutdown of the application server.

To implement these rules successfully, we need a way to hook into the shutdown sequence of our application server.  Our application was using Spring and I had a feeling that Spring must have a way to do this pretty easily.

After googling for a while, I found the perfect solution for our situation.  The solution was pretty simple and extremely elegant.

How Do They Do It!

Spring framework provides a callback interface called SmartLifeCycle.  This interface has a bunch of lifecycle methods that get called when the application server is booting up or shutting down.

All we need to do is to create a class that implements this interface and register it with the Spring context.  Spring guarantees to notify this bean about various events that occur during the lifecycle of the application server.  

This interface provides methods like, start, stop, isRunning etc. please refer the JavaDocs for more details.

To implement the above rules all we did was to create a bean called ApplicationLifeCycle that implemented the SmartLifeCycle interface.  I could implement the above rules in the stop method of the bean.

However, this still kind of solved only part of the problem.  I also wanted to make sure that my bean class ApplicationLifeCycle was the first class that was notified when the shutdown was initiated.  This was the missing link.  

The SmartLifeCycle interface provides a solution to this problem as well.  This interface extends another interface called Phased.  The Phased interface exposes a method called getPhase which returns an int value.

From the documentation it was not very clear what was the purpose of this method and what should be the value returned from this method.  Turns out that, this is an extremely useful method.  Spring uses this method to determine the order in which the beans will be notified about the life cycle events.  

• Let's say we have two beans ApplicationLifeyCycle1 and ApplicationLifeCycle2.  
• These beans both implement the SmartLifeCycle interface.
• ApplicationLifeyCycle1 returns a value 10 from the getPhase method
• ApplicationLifeyCycle2 returns a value 20 from the getPhase method
• When the application server starts, spring will invoke the ApplicationLifeCycle1 "start" callback before calling the ApplicationLifeCycle2 "start" callback method.
• When the application server stops, spring will invoke the ApplicationLifeCycle2 "stop" callback before calling the ApplicationLifeCycle1 "stop" callback method.

At this point, we had found our missing link.  All we wanted to do was make sure that our ApplicationLifeCycle class gets notified that the server is shutting down before anyone else.  All we had to do was return a higher value from the getPhase method.  We ended up returning the highest value possible i.e. Integer.MAX_VALUE and that was it!

Once we did this, Spring guaranteed that our bean will be the first bean to be notified when the server is going down (this is exactly what we wanted), while it will be the last bean to be notified when the server is starting (which we didn't really care about).

The sample implementation of SmartLifeCycle interface looks like this

Thats all folks, until we meet again happy coding!

How To Use IIS7 as Front End to Java Web Servers Like Tomcat and Jetty

While working on a project, I was faced with a situation where, I needed to use IIS server as the front end for the Jetty web server. 

I had a web application developed using Java running on Jetty.  It was time to host it on a public server, I already had a hosting provider and a hosting plan setup (from one of my previous projects) the only problem was, this was a Windows 2012 server running IIS8 on it.  There was already another .NET website hosted on IIS on this server.  I just wanted to add my web application written in Java to this setup.  Well, I know, not a very ideal situation, but I needed to get this done and get it done fast enough!

I thought for a few moments about what are my options here

• I could install Apache or some other HTTP server and use it to front my Jetty
• I could use expose Jetty or some other Servlet Container directly to the world (Note that, this was not a mission critical application with huge load and ultra high availability or anything like that)
• I could use IIS to front my Jetty server

As I mentioned earlier, I needed to get this done as fast as I could.  So out of all the options that came to my mind, last one sounded like the best or at least the fastest option.

I googled around a bit to find out, what was the least painful way to front Jetty with IIS.  There were numerous options that surfaced, I tried a few options like using the JK Connector with ISAPI filter but the one that I ended up using was - Application Request Routing (ARR).  ARR is a free Web extension for IIS 7+.  

It was pretty straightforward to configure ARR on IIS 7+ to front any Java Based web server (or any server for that matter).  In this post I am going to demonstrate the steps needed to configure ARR on IIS7 so that IIS can be used to front any Java based web server.

How Do They Do It?

The process can be divided into 3 easy steps.

• Install ARR
• Configure ARR
• Test the setup

Step - 1 - Install ARR

ARR is available as a free download from  http://www.iis.net/downloads/microsoft/application-request-routing URL.  Download and install this on the machine that has IIS 7+.

After the installation if you open up the IIS Manager (inetmgr) tool, you should see a new section called Server Farms. 

Server Farm Section Added to IIS

As the name suggests you can do a lot more (like setting up server farms) with ARR, but for now lets just focus on getting using IIS to front Jetty.

Step - 2 - Configure ARR

• This is where the meat of the solution lies.  Let's start by creation a Server Farm.  Right click on the Server Farm section and select Create Server Farm...

Create Server Farm Section

• This should open up a wizard that will ask you some details about the Server Farm.  First step is to give a name to the server farm.  In my setup both the IIS and Jetty were on the same physical machine, I called the server farm localhost.  You can call it Blah or Lady Gaga or Tom Hanks or anything else, just go crazy!

Give a name to the server farm

• Next step would ask you a little more details about the server where we need to forward the request.  Just enter the IP address of you machine.  Please also make sure that, you change the port number in the configuration to match the port number where Jetty is listening for requests.  The port number settings are hidden under the Advanced Settings... link.  In my case Jetty was listening on port 8085.

Provide IP address with port of server where we need to forward the requests

• Once you are done changing the port number, click Add on the same dialog.  This should add the server to the table below with status as Online.  Once the server is added, you should be all set to Click Finish on the wizard.

Server added to the farm

• Another confirmation popup would appear which states that, IIS Manager can create a URL rewrite rule to route all incoming request.., just click No here.  We will add the rule ourselves.

• After clicking Finish, you should have a view very similar to this one

Server Farm Added

• On this page click the Routing Rules icon.  This should take you to a page where we can configure the Routing Rules.  On this page make sure that Use URL Rewrite to inspect incoming requests is checked.  If it is unchecked, then check it and click Apply on the right side.

Check Use URL Rewrite to inspect incoming requests

• Click on the URL Rewrite... link under the Advanced Routing section on the right side.  This would open up a page similar to this one.

URL Rewrite Page

• Click on Add Rule(s)... link on the right side.  On clicking the link it would ask you about what type of rule template you want to start with, just select the Blank rule under the Inbound rules section and click OK.

New Blank Rule

• Next comes a longish page where we need to configure our rule.  Do not worry, its pretty straightforward.

New Rule Form

• Just enter the name of the rule as GitShah (again, go crazy here!). Change the Using drop down value from Regular Expressions to Wildcards.  Now the only thing left is to enter a Pattern.

Rule name with Wildcard selected

• Click the Test pattern next to the Pattern text box.  You should be presented with a dialog where we can test the URL pattern.  
• Basically in this dialog we need to inform ARR about which URL should it match and where should it forward the request to.  In my case, I wanted to forward any request coming to http://localhost/gitshah to http://localhost:8085/gitshah (served by Jetty).  Hence, any request that starts with /gitshah I wanted to forward to the Jettey server.  
• This can be done by entering /gitshah* into the Patter (wildcard) text field.  To test whether our understanding is correct, just enter a few patterns in the Input data to test: text box and click the Test button.

Testing the pattern

• You will notice that in the table below, it shows two rows {R:0} and {R:1}, they are nothing but parts of the test URL that matched the pattern.  {R:0} contains the entire string and {R:1} holds the value other than the pattern itself.  Click Close when you are satisfied with the pattern you want.  It will ask you whether you want to save the pattern, Click Yes.

• Next you only need to change just more one thing on this page, change the Action type drop down value from Rewrite to Roue to Server Farm

Action value changed to Route to Server Farm

• That's about all the configuration we need.  Click Apply on the right side and navigate Back to Rules.  You should see a view similar to this one

New rule created

 Step - 3 - Test the setup

All we now need to do is navigate to the URL http://localhost/gitshah/.  If everything is configured correctly (and luck is on your side!), we should see a page that is served from the Jetty web server.

Page served by IIS in Front of Jetty

Success! 

As we can see its pretty simple with ARR to use IIS as Front End to any Java Web Server.  That said, it might not be the best way of doing it.  Depending on your scenario please choose the optimal option.

Fixing the java.lang.ClassNotFoundException com.test.model.SomeModelClass in loader dalvik.system.PathClassLoader on Android

Recently, I had faced a very interesting issue on one of my Android project.  The issue was the dreadful java.lang.ClassNotFoundException.

The Android App, that was facing the ClassNotFoundException was tested on multiple devices.  Some of the devices that we tested the app on were

• Motorola Droid
• Motorola Droid X
• HTC Desire
• HTC WildFire
• Dell Streak
• Nexus One
• Samsung Galaxy 3
• LG Optimus One

The app worked fine on all devices without any problems at all.  We decided to go live with the App on Android Market.

Everything looked great till one day when one of the QA's downloaded the app on HTC MyTouch 4G.  The app refused to work on this device.  The app threw ClassNotFoundExceptions.  The stack trace looked like

After looking at the stack trace, I was thinking, Hmm this is weird, the app works on all these devices without an issue.  Moreover, the app cannot compile if the class com.test.model.SomeModelClass was not found.  Then why it doesn't work on HTC MyTouch 4G?

My initial reaction was, may be the app was not downloaded correctly from the Market.  May be the downloaded APK was corrupt?

I requested the QA's to uninstall the app and re-install it again from the market.  I was hoping that this could fix the issue.

But software development has taught me one thing

Nothing changes, If Nothing changes

Yep, the reinstall didnt work.  The app was still throwing the ClassNotFoundException.

We were using ProGuard.  Proguard is a free Java class file shrinker, optimizer, obfuscator, and preverifier.  Another thought that crossed my mind was, Is ProGuard removing the class com.test.model.SomeModelClass?

I created the APK without running ProGuard on it.  Requested the QA to test it.

That didn't work either.  ProGuard was not at fault.

By now, I was really confused.  What could be the issue with the app?  I thought, may be, if I try to load the class explicitly as the first thing in the app, then would it fix the issue?  

I updated the APK, loaded the class explicitly when the app starts, requested the QA to retest the APK.

Nope, that didn't fix the issue.  Interesting thing that I noticed about about this change was, the class loads successfully when I load it explicitly, but the class is not found when the app really needs it at a later stage!

History has shown us, class loader issues are not easy to fix.

I was quickly running out of options.  People were downloading the app on their device from the Market and may be some of them were facing the ClassNotFoundException issue with the app.  I needed a solution and that too quickly!  We didn't want to make a wrong first impression.  What should I try next?

At the back of my mind, I was sure that its some sort of a class loader issue.  When I explicitly load the class its found, but when the class is needed later in the app, its not found.  Surely something to do with a different class loader.  I decided to go back to basics, let's to do it step by step.

First step, 

When does the app need the class?

The app makes a HTTP call to get some information.  The information is returned in form of JSON responses.  As mentioned in my previous post, I was using Gson to parse the JSON response.  Gson is an excellent library with it comes to JSON manipulation.

Gson converts JSON response into Java Model Objects.  This is the point when the Java Model Class com.test.model.SomeModelClass is loaded.  The problem lies here, Gson tries to load the class com.text.model.SomeModelClass dynamically and its unable to find it!

Why Gson does not find the class?

The real confusion in my mind was, why Gson is able to load the class on all other devices but HTC MyTouch 4G?  My reasoning to that was, may be, Gson is loaded in a different class loader in HTC MyTouch 4G?  Is that even possible?  

After some intense googling and trying our numerous things, I finally figured out what was issue and what was solution.

This issue occurs only when all of the following conditions are met

• The App uses Gson to manipulate JSON
• The device is HTC Desire HD or MyTouch 4G on T-Mobile

The issue is HTC - The device manufacturer, also uses Gson library and they have made it public.  

Somehow, when the JVM is loaded on such devices, the Gson is also loaded along with it.  Because of this, the Gson library that is bundled with the app never gets loaded.  

When the global Gson tries to load classes from our domain model, it fails to find them.  This is because, the class loader that loaded the global Gson library does not have our app on its classpath!

That explained what the problem was, let's see how we can fix it!

The Fix

The fix was pretty simple.  We need to change the package name of the Gson library so that the global Gson library does not interfere with the local Gson library.  To change the package name of local Gson library, I used JarJar project.

I followed the following steps to change the package name of the local Gson library

• Download jarjar project.
• Put jarjar-1.1.jar and gson-1.6.jar in the same directory
• Create a new text file in this directory, lets call it rules.txt.
• Write the following line in the rules.txt file: rule com.google.gson.** com.google.myproject.@1
• Open a command prompt and execute the command, java -jar jarjar.jar process rules.txt gson-1.6.jar myproject-gson-1.6.jar.
• Replaced the gson library reference in the project with myproject-gson-1.6.jar
• Update the imports to use the new package name
• Compile and build the new APK
• Requested the QA to test the updated APK

To look at more details visit this link.

And that was it, It worked!

It was such a relief!  I am glad that, I faced this issue, makes me love the job that I am doing!

How to work with JSON on Android - Part - 2

As mentioned in my previous post, Gson is an excellent lightweight opensource library to do JSON manipulations.  I have used this library on the Android platform, it works great!

We have seen that, converting JSON to Java model objects and Java model Objects to JSON is pretty straight forward with Gson.

In this post lets look at the various hooks that Gson provides to deal with not so straight forward scenarios.

Lets take an example.  Lets say you want to convert the JSON string value "08:00" into Java model Time object.  A very valid scenario right?

Basically, we need to serialize and deserialize the Time object in a special way.  Gson provides a way in which we can hookup our own serializer and deserializer for a given type.  This serializer and deserializer will be invoked to convert the Java model object into JSON String and JSON string into Java model object.

The approach taken by Gson to do this task is pretty natural and easy to follow.  Great job guys!

Lets look at the code to put things in perspective.

The Time class has two fields one to represent the hour and other represents the minute of the day.  Let's say that, we want the JSON representation of Time objects to be of in the format "HH:MM".  Let's first look at the code to convert Java model Time object into JSON

Gson provides an interface called JsonSerializer.  We can provide an implementation of JsonSerializer to serialize the Time object in the way we want.  The code to do this would look like

Ok so the serializer does not look very complex. How do we tell Gson to use TimeSerializer when it serializes the Time object?

Turns out, Gson providers a class called GsonBuilder.  This class can be used to register custom serializers and deserializers with Gson.  GsonBuilder class has the responsibility to construct an instance of Gson with registered serializers and deserializers.  Let's look at the code

Does not look too difficult right?

Serialization done, let's look at deserialization.  Any guesses?

I am sure you got it right.  Gson provider an interface called JsonDeserializer.  We have implement this interface to provide a custom deserializer for the Time Java model class.  Let's look at the deserializer code.

The code of deserializer looks pretty simple. The code to integrate the deserializer is exactly the same as the code to integrate the serializer.

Thats all folks! Gson is a really good framework to manipulate JSON.

We have effectively used it in Android projects. If you have to do JSON manipulation in your project, I would suggest to have a look at Gson for sure!

How to work with JSON on Android

Recently, I have been working quite a bit on Android.  In my experience, there are lot of times when one has to deal with JSON while building apps for Android.

In one of my projects, I had a situation where, I had to invoke a web-service which returned JSON response.  JSON response has to be parsed, information has to be extracted from it and shown to the user.

The JSON response returned by the web-service was pretty big, with lots of fields, objects, lists and lists of objects.  In sort, the JSON response was pretty complex, it would have been a lot of work to extract information out of it, construct domain objects and then use it to show the information on the GUI.

Android provides a package org.json which has class like JSONObject, JSONArray, etc which help us parse the JSON responses.  I could have used those class and parsed the JSON response, extracted the information out of it, converted that into domain objects and used them to show it on the GUI.  But I am a lazy developer!  I am always in an hunt for ways to reduce my work!

I was in no mood to write the such a lot of code just to parse the JSON response.  Over and above it, there are multiple problems with writing such code

• Quite some code just to parse the JSON response
• Code has to be tested very well.  We have to make sure that it works in all situations.
• This approach is error prone.  For example, if we decide to change the JSON keys, a lot of code have to be updated and there is no compile time check to find out errors if any while updating the keys.

So what is the alternate?  Simple, search for a library that does the job for you :)

The perfect library for the task is Gson.  Gson is an opensource project that helps us convert JSON to JavaBeans or JavaBeans to JSON using one line of code!

This effectively means we could convert the JSON response to our model Objects with one line of code.  Isn't it awesome!

How do they do it?

An example explains things much faster.  So lets not waste any time and head straight to see Gson in action.

Lets say that the JSON response returned by the service looked like

As we can see, the web services returns us the searched Address and Points of Interest (POI) around the searched address in the JSON response.

We want to convert this JSON response into the POIResponse Java domain object.  Lets look at how the POIResponse class looks like

The model class are pretty standard.  They just have fields with the same name as the keys in the JSON response.  Notice that we do have complex objects like Address and POI and list of POI in POIResponse class.

For Gson to work only one condition has to meet and that is, the name of fields in the model class should be same as the JSON keys.  This is based on the convention over configuration principal.

Also notice that we do not have getters and setters in our domain classes.  Gson uses reflection to set the values directly into the fields.  Gson find private fields as well.

Lets look at the code to convert the JSON response into model Java object.

Thats all we need to do to convert JSON response into POIResponse object!  As promised its just one line of code.

Gson has excellent support for Generics.  Notice that, we did not have to caste the return type of fromJson method to POIResponse.

Gson can convert POIResponse to JSON with the same ease.

The string reference json holds the JSON representation of the poiResponse instance.

Gson is an excellent lightweight library when it comes to working with JSON.

In the next post I will show how Gson provides enough hooks to provide custom serializers and deserializers.

Till then, stay tuned!

The deadly Deadlock - Fixing the issue with java.util.Calendar and java.util.TimeZone

All computer engineers learn about a condition called Deadlock in college.

What is deadlock?

This is what Wikipedia has to say about Deadlock:

"A deadlock is a situation where in two or more competing actions are each waiting for the other to finish, and thus neither ever does.  In computer science, Coffman deadlock refers to a specific condition when two or more processes are each waiting for each other to release a resource, or more than two processes are waiting for resources in a circular chain"

The article gives quiet some information about deadlock.  No! don't worry I am not going to explain all that.  There are lots of resources that will explain what is deadlock and why it occurs in a much better way.

We have all learned about the deadlock and know why does it occurs.  But have we ever faced a deadlock situation in our code?

The technology, enterprise stack, databases, we use have matured so much, that we hardly ever face a deadlock situation in production code.  But, I was fortunate enough to face a deadlock situation in one of my recent projects.

Of course the deadlock I faced happened when the app went live i.e. in production environment.  

I don't know about others, but its not a good situation to be in.  Since the deadlock was only detected in production environment, situation becomes very critical.  Numerous mails are exchanged (some of them ugly), everyone wants to know what is causing the deadlock and Me (poor soul) had no idea why it occurred :)

Moving on, how did we know that there was a deadlock?

Symptoms of deadlock

• Performance of the app degrades quite a bit
• CPU usages shoots up to 100% and stays there!
• Users complain that the site is not responding and they are not able to do a certain action.
• To get things back to normal servers have to be restarted!

When we noticed these symptoms in production for the first time, we thought it was an one off case, we restarted the server and decided we would try and find a reason for this performance problem.

I tried to find all possible causes of this performance issue but without any luck.  All possible causes were ruled out, because we were using all possible best practices, we were using a standard J2EE stack, spring, hibernate, lazy loading, optimized queries etc.  

I was not read to believe myself but, I suspected that the app might be a victim of deadlock!  

Our app was deployed in a Weblogic cluster.  We had configured Weblogic to throw an exception when it detects that a thread is running for more than 10 min (We had not long running background processes which would take more than 600 sec to finish).  And bang!  After a few days, Weblogic did throw an exception stating that a certain thread was running for 666 sec which is more than the threshold of 600 sec!

Following is the part of stack trace

Looking at the stack trace, I was a bit surprised, I was expecting a deadlock situation, when app tries to interact with the database. Thats what my impression about deadlock was! It occurs mostly when we interact with the database.

But, I was proved wrong by my own code!  In our app the deadlock occurred in memory!

We nailed it down to a function called DateRange.countNumberOfFullDays.  As the name suggests this functions calculates the number of full days between two dates!

The code was not complex at all.  And this was the least expected place where a deadlock could occur.

As you can see the actual code of DateRange.countNumberOfFullDays does not do anything special!  Then what was causing the deadlock?

Googling a little, I found that there are various issues with the Java Calendar and TimeZone classes in Java 1.5.  Visit http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6199320 and http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6232022 for more information.

These bugs were fixed in Java 1.6 but, unfortunately our production servers use Java 1.5.  Damn!  Why why why, production server always lag behind the pace at which technology evolves.

The solution:

There is an excellent open source library which provides a parallel and better implementation of Calender and TimeZone api's.  Its called Joda Time

I used this library, changed the code of countNumberOfFullDays to look like

As we can see in this simple example, Joda time provides a very simple and intuitive Date and Calendar api.

Since I implemented this change (on 04/01/2011) we have not faced a deadlock situation yet! Fingers crossed!

OK, enough about the deadlock situation that I faced.  I want to know from you guys, have you ever faced a deadlock situation in production code?  If yes then, what was the situation and how did you fix it?

The thrill of being a developer - Fixing the problem of weird URL's

There are days when as a developer, I feel bored, bored to death, angry, happy, sad and many other emotions.  But there are a few days in life of a developer, when you feel truly thrilled!

Recently, one such thrilling day occurred in my life.

In one of the project, we were facing a weird and interesting issue in production.  The application server would get weird requests from the browser.  The weird thing about these request was that, the parts of the requested URL would be valid but, the end part would contain some weird html tags!  Since the resource would not be found on the server, app server would log the request and return a 404 error page.

For e.g. one such log message was:

The request that browser sent was:

The above URL is to fetch jquery.maskedinput.min.js file.  

Generally when an issue is reported by the QA/Client, I already have some idea as to what could have caused the issue.  But, this issue was special.  When I first saw these error, I was completely blank.  I had no idea what so ever why would something like this every happen!

We were using a pretty common Java stack.  Spring MVC and Hibernate, app was deployed on Weblogic server.  I had used these frameworks in so many other projects, but never ever faced an issue like this.

The most frighting part was, I had no idea what was the end user impact.  Was the user logged off?  Was he shown any error?  Did he see a garbled page?  No idea at all!

Coming back to the issue.

Yes believe it or not, the above request is to fetch the jquery masked input javascript! Notice that the beginning part of the URL is valid, but what gets added at the end is really not expected.

From where did

text get appended to the URL?

Digging a little deeper, I found something totally bizarre.  Turns out that the text

is present in the same page after some 4096 bytes!

How is it possible that something that comes after a few thousand bytes gets appended to the request URL of jquery.maskedinputments.min.js?

After two and half days of hard core googling, trying hundreds of different things with the script tag, the textarea tag, using different permutation combinations, and countless hours of analyzing the server logs over and over again, finally I have found a credible explanation to weird URL’s issue we were facing.

It happens because of a bug in IE8’s Lookahead Downloader.  The problem has nothing to do with the application!

What are you talking about?  Explain me in detail:

Following is an extract from this URL: http://blogs.msdn.com/b/ieinternals/archive/2009/07/27/bugs-in-the-ie8-lookahead-downloader.aspx which describes the bug in full detail.

---

Lookahead Downloader is used to quickly scan the page as it comes in, looking for the URLs of resources which will be needed later in the rendering of the page (specifically, JavaScript files). The lookahead downloader runs ahead of the main parser and is much simpler-- its sole job is to hunt for those resource urls and get requests into the network request queue as quickly as possible.

The problem here is that there are a number of tags which will cause the parser and lookahead downloader to restart scanning of the page from the beginning. One such tag is the META HTTP-EQUIV Content-Type tag which contains a CHARSET directive. Since the CHARSET specified in this tag defines what encoding is used for the page, the parser must restart to ensure that is parsing the bytes of the page in the encoding intended by the author. Unfortunately, IE8 has a bug where the restart of the parser may cause incorrect behaviour in the Lookahead downloader, depending on certain timing and network conditions.

The incorrect behavior occurs if your page contains a JavaScript URL which spans exactly the 4096th byte of the HTTP response. If such a URL is present, under certain timing conditions the lookahead downloader will attempt to download a malformed URL consisting of the part of the URL preceding the 4096th byte combined with whatever text follows the 8192nd byte, up to the next quotation mark.  Web developers encountering this problem will find that their logs contain requests for bogus URLs with long strings of URLEncoded HTML at the end.

Impact on the end user:

Generally this has no direct impact on the visitor's experience, because when the parser actually reaches a tag that requires a sub-download, if the speculative downloader has not already requested the proper resource, the main parser will at that time request download of the proper resource.  Hence,

• The visitor will not notice any problems like script errors, etc
• The visitor will have a slightly slower experience when rendering the page because the speculative requests all "miss"
• IIS/Apache logs will note requests for non-existent or incorrect resources

The fix:

The fix is simple.  We needed to apply a IE8 Cumulative Update (KB980182) patch on the client machine’s which have this problem.

---

When I read this post, I was just stunned!  Problems like these truly blow your mind away!

Today, when I think about the problem, I realize that, its because of problems like these, I love being a Developer!

True developers, solve Real problems!

How to run selenium tests as part of maven build phase

Recently I came across a very simple requirement.  Our project used Maven as the build tool and we wanted to run Selenium tests as part of the build.  The idea was, QA's write the automated selenium tests, these should run as part of the integration-test build phase. 

There are multiple advantages of doing this:

• We could run these Selenium tests on our Continuous Integration server (Hudson in our case) as part of build process.  This means, if because of some check-in, a selenium tests fails, the build fails and hence someone has to fix it!
• Developers and QA's can easily run these functional tests whenever they want.  For e.g. if developers want to make sure that the code they are about to checkin does not break any existing functionality they can run the functional tests before checkin with just one command.

So how do they do it?

Lets try and reverse engineer the problem.  To run selenium tests we need to do the following things

1. Compile the app
2. Run unit-tests
3. Create a war file of the application
4. Run an tomcat server (or any other servlet container you are using)
5. Deploy the war file created in step 3 on the server started in step 4
6. Start the selenium server
7. Fire the selenium tests

Looks like a lot of work to me!  But don't worry we will see in a moment that its not as difficult as it sounds.

Lets do this step-by-step:

Lets break down the problem.  If you look at the above mentioned 7 steps.  We can classify them in four broad categories

1. Compile the application and package it as a war file. (steps 1-3)
2. Run the tomcat server (steps 4-5)
3. Run the selenium server (step 6)
4. Run the selenium tests (steps 7-8)   

Problem looks simpler now?  Lets try and slove the above four problems separately first.  Later  we will see how we can solve all of them using just one command.

Solving the step 1:

Assuming that you already have the pom.xml setup to properly compile and run unit-tests.  We need to add the configuration in pom.xml to package the application as war file.  This is what we have to do

Find the tag called <packaging> in pom.xml and set the value of the tag to war instead of jar.

Then, we need to tell maven about how the war file will be generated.  This is done using the maven-war-plugin.  The sample configuration looks like this:

Of course for all this to work you should have a proper directory structure and web.xml.  But I will assume that you already have that in place.

At this point if you run the following command:

You will get a war file ready to be deployed in any servlet container.

Next, Solving the step 2:

To run an embedded tomcat server we have to add the following configuration to pom.xml

Now, when we run the following command, we will have a running tomcat server with our application deployed on it.


Next, Solving the step 3:

To start the selenium server from command prompt we need to do two things.

• Add the dependency of selenium drivers to the app.  This is required so that we could use selenium api to programmatically write selenium tests.  This is done using the following code:

• Add the selenium plug-in so that we can start the selenium server from the command prompt.  This is done using the following code:

After this we can start the selenium server using the following command:


Next, Solving the step 4:

Now, we are ready to run the tests simply run the tests using the following command:

We have our selenium tests running as part of our maven build!  To do this we have to fire three commands from three command prompts/terminal windows.  One running tomcat, another running selenium server and the third running the tests.

All this is good but we don't want three command prompts running.  We just want one command that does it all!

And this is exactly what we are going to see next!

So as you already know that Maven is build around phases, we are going to run our functional tests (selenium tests) in the integration-test phase.

To run the selenium tests in the integration-test phase we have to do the following:

1. Start the tomcat server
2. Deploy the war
3. Start the selenium server
4. Fire the function tests

Here's how we do steps 1 and 2:

Above configuration means that following things in plain english:

• Start the tomcat6 server
• Deploy the app during the pre-integration-test phase.  
• Stop the server during the post-integration-test phase.

You will have to define an environment variable CATALINA_HOME, which points to the tomcat server installation directory on your machine.

Next up, Step 3 do this:

Above configuration states the following:

• Start the selenium server during the pre-integration-test.
• Stop the server during the post-integration-test.

Next, Step 4: 

To run the functional tests during the integration-test phase, do the following (I promise this is the last bit of XML that you see on this post!)

This configuration does the following:

• Tells maven-surefire-plugin to run the unit tests during the test phase but exclude the classes in the functional folder.
• Run the test class in the functional folder during the integration-test phase.

And that's it my friend!  Now when you run the following command:

1. Application will compile
2. Unit tests will run
3. War file will be created
4. Tomcat server will be started 
5. war file will be deployed on the tomcat started in step 4
6. Selenium server will be started
7. Functional tests will be fired
8. And we will live happily ever after!

How to append text and images to an existing PDF document

Requirements:

Recently in one of my projects we had a requirement that, we had to append some legal text and an image (in BMP, PNG or JPEG format) to an existing PDF invoice.

Now PDF is not a text format.  We can't simply add text and an image to the pdf document, by appending their bytes to the PDF file.  It requires special api's for manipulation. 

How do we do it:

iText framework has excellent support for manipulating PDF documents.  It can create, read, edit and append PDF documents.  Its really powerful!

But today we are specifically interested in adding some text and image to an existing PDF document.

Here's the code that can do this:

The generated PDF looks like this:

That's it!  The end result is we have a shining new PDF called "invoice-appended.pdf", which holds the content of the existing PDF and new information (text and image) appended to it.

How to convert HTML to PDF

Recently I was faced with a situation where, two days before the project was supposed to go into system testing phase, client had a change of mind (sounds familiar?  Sometimes I feel this happens quiet often).

In our project, we where showing an invoice as html, which the users could print.  Everything was fine, client was happy with the html invoice till the system testing was about to start.  Then one fine day, client decides they need the invoice as a PDF and not as an HTML.

Well, well, we had to deliver this in about a day.  The invoice was not really straight forward.  It changed based on different situations.

The project was build using Java technologies.

We had three choices:

• Scrap all the effort (development and testing) that we had put in to get the HTML invoice in the format desired by the client.   Use some ugly api's to generate the PDF that would look similar to the HTML invoice we already had in place - Painful approach which would require a lot of effort.
• Design a jasper report (jrxml) which would look exactly like the HTML invoice, then use jasper api's to render the designed report as PDF - Relatively sensible approach.  Jasper is extremely powerful (no doubts about that!), but people who have used Jasper in past would agree, designing a report in Jasper is not the easiest of the things.  It would take some time.  Moreover, the look and feel of the generated PDF might not be exactly like the HTML invoice.  In this approach also we might have to scrap all efforts we had put in generating the HTML invoice.
• Find a way to convert HTML invoice directly into PDF - An awesome approach!  The effort we had put in to get the HTML invoice in the right format would not be wasted and everyone would live happily ever after!

I being a lazy developer, always choose the easiest approach.  Hence, I decided to give the approach #3 a shot.  Tried finding some frameworks that could convert HTML directly into PDF.

Some were good, some were bad and some were ugly.


But when I was trying these frameworks, I had a WOW moment!

Found a framework that would take the HTML and convert that into the exact same PDF!  It used the styles that were included in the HTML document and the generated PDF looked exactly like the HTML invoice!  I was thinking, Man this is Awesome!

The framework is called Flying Saucer

Internally Flying Saucer uses iText to genearte the pdf.  iText is really powerfull peace of software, it can genearte, read, edit, append pdf documents.

Lets look at the code to convert an HTML document directly into a PDF.

How do they do it:


That's it!  Pass the URL of the HTML invoice (we want to convert into PDF) to the create method.  Have a look at the generated PDF invoice file.

I was so amazed to look at the PDF file, it looked exactly like the HTML invoice.

To add to the joy, PDF is a paged medium.  Which means it could have page numbers, footers and headers.

Flying Saucer supports CSS 2.1 standards for paged medium and recognizes @page attribute in the CSS.  Hence, if you want page numbers at the bottom right cornor of your generated PDF, simply include the following styles in your HTML to be converted into PDF.

Simply, isn't it!
Note:  To convert HTML to PDF using Flying Saucer, your HTML should be a valid XML

Update: I have uploaded the sample code to convert html document in PDF here.

Dont be in love with one language, be in love with the concepts

If you are like me, who does programming for living, if you are very passionate about one language and would never ever think of switching to any other language?

Then this blog post if for you. 

During my early years as programmer, I was very passionate about Java and only Java (does not mean that, I am any less passionate about Java today).  I wanted to do programming only in Java.  I thought I can do anything and everything in Java (which is true even today).

I was very apprehensive to do programming in any other language.

Then due to a project requirement I had to work on C++.  Because of that, I started appreciating Java even more.

How elegantly it does memory allocation and garbage collection etc.  I started appreciating the high productivity I could get using Java.


After that, I got an opportunity to work on Ruby on Rails.  I started admiring all the dynamic feature of Ruby language.  Started appreciating closures in Ruby and learned how a few lines of code can be expressed in just on single line.  Started admiring how simple, a modern web framework could be. 

Then came along .Net.  In .Net we had used a dependency injection framework called Castle.  I had used Spring framework from the Java world in the past, because of this I could apply same concepts in Castle framework.  This meant, I was productive from Day 1.  Effectively no learning curve involved!

I realized that as time passed, C# and Java have converged to each other.  Both these languages have learned from each other and implemented some very good features.

Then, I started working on Grails.  I was amazed to see that, Grails and Ruby on Rails are almost 90% similar.  They are so similar to each other in concepts and implementation that, I find the only difference is the syntax!

Again, was able to apply the concepts, that I had learned while doing the Ruby on Rails project in Grails.  End result productive from Day 1.


After all these years, I have learned one thing and that is, all these languages be it Java, Ruby, Groovy, C# etc. be it anything, they implement same concepts over and over and over again.

Now I feel, If someone is a good programmer it actually does not matter which language they use to achieve a given task.

Language and Framework are just tools to achieve a particular task. 

In the olden days car manufactures used old tools to manufacture cars and now as the tools and technologies improve they use better tools and manufacture wonderful cars!

Applying similar analogy we just have to keep updating the tools that we use so that we can create good efficient software.

That's why I keep saying to programmers, don't be in love with one language, be in love with the concepts!