My Journey with Mule

Stage 1 : Drag & Drop, Enterprise Integration Patterns , Oh So Cool!!

Stage 2 : What the **** ¬†$$$!!!??? IDE Sucks, weird behavior , silent xml config alteration ….How can anyone work on this!!! Going back to my core java apps…

Stage 3 (Used to all the quirks of Mule Studio) : Hmmm…I see the point..not so bad after all !!! Just built an entire state machine system app with DB integration / HTTP & TCP Endpoints in 90 mins!

Stage 4: Wait, they hid all the cool features in EE edition ūüė¶ … I am supposed to debug my app only with log messages!! Thats almost as bad as the time when I used to write EJB code! hmmmm….start looking for alternative? How is that spring integration project coming along?¬†

Adding Functional power in Java programs

I am digging the Functional style power in my everyday regular programming. I was implementing some failover connection setup in a pure Java code and thought about adding the ugly java command pattern as an alternative to closure. At the end, the code turned up not so bad so I thought I would share it.

Problem:

We have two Quentin Server – Server A & Server B (A kind of recording server used in broadcast industry). The QuentinApi abstracts the interaction with the server which maintains some persistent TCP connection. Now the requirement is, we have to keep talking to Server A. However, if the connection is not valid / or it went down for some reason, we need to switch back to Server B.

Here is how the Quentin Api Looks like:

/**
 * 
 * Api to create/update/delete clips on Quantel. Clips are actually created from the channel. The quantel system has a pool of server. 
 * Every server has a fixed number of channel. 
 * 
 * @author smoinuddin
 *
 */
public interface QuentinApi {
	
	public String createClip(String clipName, int server, int channel, DateTime startTime, DateTime endTime) throws QuantelServerLookupFailedException, QuentinApiException;
	
	public void updateClip(String clipName, int server, int channel, DateTime startTime, DateTime endTime) throws QuentinApiException;
	
	public void cancelRecording(String clipName, int server, int channel) throws QuentinApiException;
      ...... // some more similar methods....	
}

Now all of these method should have the feature of failure detection and should revert to Server B whenever there is a problem. I have solved similar problem before with Java using Dynamic Proxy and Spring AOP, but I thought using Functional Closure style is simpler and easier to understand.

public class QuentinApiFailoverProxy implements QuentinApi {
	private static final Logger LOGGER = LoggerFactory.getLogger(QuentinApiFailoverProxy.class);
	
	private QuentinApiImpl quantelServerA;
	private QuentinApiImpl quantelServerB;
	
	private QuentinApiImpl currentConnection;
	
	public QuentinApiFailoverProxy(QuentinApiImpl quantelServerA,
			QuentinApiImpl quantelServerB) {
		super();
		this.quantelServerA = quantelServerA;
		this.quantelServerB = quantelServerB;
		
		this.currentConnection = quantelServerA;
	}
	
	private abstract class QuantelApiCall<T> {
		abstract T execute() throws QuentinApiException;
		
		public  String toString() {
			return null;
		}
	}
	
	private <T> T execute(QuantelApiCall<T> call) throws QuentinApiException{
		if(call.toString() != null) {
			LOGGER.debug("***executing Call: {}", call);
		}
		try {
			if(currentConnection.validate()) {
				return call.execute();
			} else {
				LOGGER.warn("Could not validate current connection: " + currentConnection);
				throw new QuantelCommunicationException("Could not validate current connection: " + currentConnection);
			}
		} catch (QuantelServerLookupFailedException | 
				QuantelCommunicationException | 
				NullPointerException| 
				org.omg.CORBA.COMM_FAILURE |
				org.omg.CORBA.OBJECT_NOT_EXIST e) {
			
			LOGGER.warn("Corba Failed to talk to {}", currentConnection, e);
			resetConnections();
			LOGGER.warn("Trying to talk to {}", currentConnection);
			
			return call.execute();
		} finally {
			if(call.toString() != null) {
				LOGGER.debug("***finished executing call: " + call);
			}
		}
	}
	
	private synchronized void resetConnections() throws QuantelCommunicationException {
		boolean validCurrentConnection = false;
		try {
			currentConnection.reset();
			if(currentConnection.validate()) {
				validCurrentConnection = true;
			}
		} catch (Exception e) {	}
		
		if(!validCurrentConnection) {
			currentConnection = getAlternative();			
			currentConnection.reset();
			if(!currentConnection.validate()) {
				LOGGER.warn("Failed to talk to main connection & alternative connection . Alternative connection just tried = {}" , currentConnection);
				throw new QuantelCommunicationException("Failed to talk to main connection & alternative connection . Alternative connection just tried: " + currentConnection);
			}
		} else {
			LOGGER.warn("re-initiated the current connection to {}");
		}
		
	}

	private QuentinApiImpl getAlternative() {
		if(currentConnection.equals(quantelServerA)) {
			return quantelServerB;
		} else {
			return quantelServerA;
		}
	}    

	@Override
	public String createClip(final String clipName, final int serverId, final int channel,
			final DateTime startTime,final  DateTime endTime)
			throws QuantelServerLookupFailedException, QuentinApiException {
		
		return execute(new QuantelApiCall<String>() {
			@Override
			String execute() throws QuentinApiException {
				return getCurrentConnection().createClip(clipName, serverId, channel, startTime, endTime);
			}

			@Override
			public String toString() {
				return "createClip: " + clipName ;
			}});		
	}

	@Override
	public void updateClip(final String clipName,final  int server,final  int channel,
			final DateTime startTime, final DateTime endTime) throws QuentinApiException {
		execute(new QuantelApiCall<Object>() {
			@Override
			Object execute() throws QuentinApiException {
				getCurrentConnection().updateClip(clipName, server, channel, startTime, endTime);
				return null;
			}

			@Override
			public String toString() {
				return "Update Clip: " + clipName;
			}});	
	}

	private QuentinApiImpl synchronized  getCurrentConnection() {
		return currentConnection;
	}

.................. and so on.
}

Notice how the execute method wraps each method call and adds exception handling, reset connection, try alternative connection feature. Introducing the QuantelApiCall lets us bundle our code and pass it as an argument to another method (Code as object). Again the execute() call does not take any parameter, instead it uses static binding functionality with the local variable to carry them over to the new context.

One thing you need to be cautious is to use the getCurrentConnection() in the closures to enable java dynamic binding. If I used the currentConnection instead, the currentConnection at the time of ‘Closing’ (aka Creating the closure object) would be used everytime you call execute(). But methods are always dynamically bounded and will be reevaluated with every call of the execute().

Again, the code can be called by multiple thread at the same time so you need synchronization in the reset() call (You don’t want two threads trying to reset the connection at the same time). Other interesting thing is you need ‘synchronized’ on the getCurrentConnection to avoid Thread data caching issue. I have noticed a lot of java developer mistakenly think synchronized is only used for mutual exclusion but it is also necessary for visibility of the data.

Will scala become obsolete with java 8?

I have been investing a lot of my free time in Scala lately. Although I have been injecting groovy heavily in all my projects whenever I get a chance, I miss the safety of Type system (too much java does that to you).

Even though I had some previous Erlang experience and background knowledge of Actor Model, I must admit Scala has a steeper learning curve than I thought.It requires significant time and energy invested before you can do anything useful with it. I can imagine I’d have a hard time if I wanted to push it to a team of java devs. On the other hand, Groovy is too easy to adopt, you write your java code in a script and in no time fall in love with the power of functional language with Java goodies in your hand.

That brings the question in my mind, will scala be obsolete with java 8? A lot of scala goodies will face serious competition with java 8. Developers will eventually adopt those feature sooner or later if they want to keep their job but I don’t see mass adoption of scala. Don’t get me wrong, I am loving scala and no doubt its an elegant language but I don’t see it as a successful complement language of Java with java 8 in the picture. On the other hand, I can imagine myself doing all my scripting / hacking jobs in Groovy as a replacement of my broken perl skills.

Some taste of functional programming with groovy

In my day job, we have to do a lot of system integration, a lot of it consists consuming external feeds , process it and feed it to some other system. We use Mule a lot for this. I was writing a groovy script in mule to parse a xml feed and extract some data, here is how my functional programming investment helped me do it in an elegant way. The XML to parse

<sport reportName="aflRoundMatches" jobId="72451003">

  <aflRoundMatches roundNumber="3">
    <match matchId="10370301" matchStatus="complete" matchName="Dockers vs Bombers" matchNumber="1" isPreseason="false" isFinal="false" homeSquadId="60" homeSquadName="Fremantle" awaySquadId="50" awaySquadName="Essendon" venueId="70" venueName="Patersons Stadium" venueTimezone="Australia/Perth" utcStartTime="2013-04-12T10:40:00+00:00" localStartTime="2013-04-12T18:40:00+08:00" period="4" periodCompleted="4" periodSeconds="1877" homeSquadGoals="9" homeSquadBehinds="14" homeSquadScore="68" awaySquadGoals="10" awaySquadBehinds="12" awaySquadScore="72"/>
    <match matchId="10370302" matchStatus="complete" matchName="Kangaroos vs Swans" matchNumber="2" isPreseason="false" isFinal="false" homeSquadId="100" homeSquadName="North Melbourne" awaySquadId="160" awaySquadName="Sydney Swans" venueId="2" venueName="Blundstone Arena" venueTimezone="Australia/Hobart" utcStartTime="2013-04-13T03:45:00+00:00" localStartTime="2013-04-13T13:45:00+10:00" period="4" periodCompleted="4" periodSeconds="1873" homeSquadGoals="13" homeSquadBehinds="14" homeSquadScore="92" awaySquadGoals="20" awaySquadBehinds="11" awaySquadScore="131"/>
    <match matchId="10370303" matchStatus="complete" matchName="Demons vs Eagles" matchNumber="3" isPreseason="false" isFinal="false" homeSquadId="90" homeSquadName="Melbourne" awaySquadId="150" awaySquadName="West Coast Eagles" venueId="40" venueName="MCG" venueTimezone="Australia/Melbourne" utcStartTime="2013-04-13T04:10:00+00:00" localStartTime="2013-04-13T14:10:00+10:00" period="4" periodCompleted="4" periodSeconds="1878" homeSquadGoals="13" homeSquadBehinds="5" homeSquadScore="83" awaySquadGoals="27" awaySquadBehinds="15" awaySquadScore="177"/>
    <match matchId="10370304" matchStatus="complete" matchName="Giants vs Saints" matchNumber="4" isPreseason="false" isFinal="false" homeSquadId="1010" homeSquadName="GWS Giants" awaySquadId="130" awaySquadName="St Kilda" venueId="150" venueName="Manuka Oval" venueTimezone="Australia/Canberra" utcStartTime="2013-04-13T06:40:00+00:00" localStartTime="2013-04-13T16:40:00+10:00" period="4" periodCompleted="4" periodSeconds="1949" homeSquadGoals="10" homeSquadBehinds="8" homeSquadScore="68" awaySquadGoals="21" awaySquadBehinds="14" awaySquadScore="140"/>
    <match matchId="10370305" matchStatus="playing" matchName="Cats vs Blues" matchNumber="5" isPreseason="false" isFinal="false" homeSquadId="70" homeSquadName="Geelong Cats" awaySquadId="30" awaySquadName="Carlton" venueId="190" venueName="Etihad Stadium" venueTimezone="Australia/Melbourne" utcStartTime="2013-04-13T09:40:00+00:00" localStartTime="2013-04-13T19:40:00+10:00" period="3" periodCompleted="2" periodSeconds="421" homeSquadGoals="13" homeSquadBehinds="6" homeSquadScore="84" awaySquadGoals="10" awaySquadBehinds="8" awaySquadScore="68"/>
    <match matchId="10370306" matchStatus="playing" matchName="Suns vs Lions" matchNumber="6" isPreseason="false" isFinal="false" homeSquadId="1000" homeSquadName="Gold Coast Suns" awaySquadId="20" awaySquadName="Brisbane Lions" venueId="81" venueName="Metricon Stadium" venueTimezone="Australia/Brisbane" utcStartTime="2013-04-13T09:40:00+00:00" localStartTime="2013-04-13T19:40:00+10:00" period="3" periodCompleted="2" periodSeconds="773" homeSquadGoals="7" homeSquadBehinds="8" homeSquadScore="50" awaySquadGoals="7" awaySquadBehinds="14" awaySquadScore="56"/>
    <match matchId="10370307" matchStatus="scheduled" matchName="Tigers vs Bulldogs" matchNumber="7" isPreseason="false" isFinal="false" homeSquadId="120" homeSquadName="Richmond" awaySquadId="140" awaySquadName="Western Bulldogs" venueId="190" venueName="Etihad Stadium" venueTimezone="Australia/Melbourne" utcStartTime="2013-04-14T03:10:00+00:00" localStartTime="2013-04-14T13:10:00+10:00" period="0" periodCompleted="0" periodSeconds="0" homeSquadGoals="0" homeSquadBehinds="0" homeSquadScore="0" awaySquadGoals="0" awaySquadBehinds="0" awaySquadScore="0"/>
    <match matchId="10370308" matchStatus="scheduled" matchName="Magpies vs Hawks" matchNumber="8" isPreseason="false" isFinal="false" homeSquadId="40" homeSquadName="Collingwood" awaySquadId="80" awaySquadName="Hawthorn" venueId="40" venueName="MCG" venueTimezone="Australia/Melbourne" utcStartTime="2013-04-14T05:20:00+00:00" localStartTime="2013-04-14T15:20:00+10:00" period="0" periodCompleted="0" periodSeconds="0" homeSquadGoals="0" homeSquadBehinds="0" homeSquadScore="0" awaySquadGoals="0" awaySquadBehinds="0" awaySquadScore="0"/>
    <match matchId="10370309" matchStatus="scheduled" matchName="Power vs Crows" matchNumber="9" isPreseason="false" isFinal="false" homeSquadId="110" homeSquadName="Port Adelaide" awaySquadId="10" awaySquadName="Adelaide Crows" venueId="10" venueName="AAMI Stadium" venueTimezone="Australia/Adelaide" utcStartTime="2013-04-14T06:40:00+00:00" localStartTime="2013-04-14T16:10:00+09:30" period="0" periodCompleted="0" periodSeconds="0" homeSquadGoals="0" homeSquadBehinds="0" homeSquadScore="0" awaySquadGoals="0" awaySquadBehinds="0" awaySquadScore="0"/>
  </aflRoundMatches>
</sport>

The Groovy script:

new XmlSlurper().parseText(payload).aflRoundMatches.match
		.findAll { ['playing','postmatch'].contains(it.@matchStatus.text()) }
		.collect { match -> 
			[
			'matchId': match.@matchId.text() , 
			'homeSquadId' : match.@homeSquadId.text() , 
            'matchStatus' : match.@matchStatus.text(),
            'awaySquadId' : match.@awaySquadId.text(), 
		    'HomeTeam/ScoreTotal_C' : match.@homeSquadScore.text(),         
            'AwayTeam/ScoreTotal_C' : match.@awaySquadScore.text() ,            
            'Period_Active' : match.@period.text().toInteger() >= match.@periodCompleted.text().toInteger() ? 0 : 1
            ]
		}

Here,¬†new XmlSlurper().parseText(payload).aflRoundMatches.match gives me all the <match> elements of the xml, on that I run a filter for only the ‘playing’ & ‘postmatch’ games (incase you are wondering, its for pushing match scores on Live Television for the playing/postmatch games). After that I run a ‘collect’ method to convert the <match> elements into a Map<attribute, value> for that match. At the end of process I get a List<Map<attribute, Value>> Imagine doing this in Java! This is pure transition of data from one state to another through functions. Coming from years of Object Oriented background, I had to really struggle with this concept as we are so used to think in terms of Nouns aka Classes. Here is a really interesting read about it: http://steve-yegge.blogspot.com.au/2006/03/execution-in-kingdom-of-nouns.html I have some more advanced example which I will save for the next blog post.

Currying with Groovy

I was writing some functional test case with Cucumber for JVM and opted to use Groovy for my step¬†definitions. While I am not yet comfortable enough to start any big project with an untyped language like groovy,¬†¬†its super cool for writing test cases. Specially when we write functional test cases, we want to write as less code as possible and achieve as much as possible. I think groovy is a perfect candidate for that. I am also writing my shell/perl scripts with Groovy now. Call me old school but I still like my java ūüôā

Anyway, I was fetching the XML from the server and was asserting various attributes on it. Here is how the power of functional programming made it elegant.

def xmlFetchedFromWebservice = '''
<SchedRemote> <!-- Long Form: Room, Service, and Resource -->
 <WorkOrderDetails ConflictLevel="0" SEQNUM="232323">
 <Fields>
 <Field STAT="10" />
 <Field STATUSEX='0'/>
 </Fields>
 </WorkOrderDetails>
 </SchedRemote>
'''

def xmlFind = { xmlFields, fieldName ->
 def field = xmlFields.find {
 return it['@' + fieldName] != null
 }
 return field?.( '@' + fieldName)
}
def selectXmlFields = {xmlText -> new XmlParser().parseText(xmlText).WorkOrderDetails.Fields.Field}

def workOrder = xmlFind.curry(selectXmlFields(xmlFetchedFromWebservice))

assert workOrder('STAT') == 10
assert workOrder('STATUSEX') == 0

//..... many more assersions like this

I refactored some of the implementation details , In a nutshell We have two part task, 1st select the correct section from xml and then perform a search on that section to select the field value.

So once I get the xml from webservice, I select the right fields with selectXmlFields and curry the result in the xmlFind method.

Now, the resulting function can be used almost as object like fashion. The core concept of functional programming which took me quite a while to grasp is that functions are the building blocks, not the objects. If I wanted to do the same thing with java, may be I would create a WorkOrder class, parse the XML in the constructor and use workOrder object. In functional language, I am using the function composition from the currying to perform the same thing. In other words, I am using curry here to bind xmlFind to the perticular workorder instance. How cool is that?