programming

Why twitter needs to be so fast???!!!

sajid2045 design, productivity, programming Leave a comment

Ok, first I must confess, I am not a active twitter user, but I am a regular follower about their System Architecture which always fascinates me. Also they have done some great contribution to the OSS world with products like :

Bootstrap , CloudHopper , Twitter-Server etc.

https://github.com/twitter

Its amazing to see what started as a ruby on rails application turns into this Mega-Structure and does things at a scale that I wouldn’t even believe is possible.
I was watching this excellent presentation in infoQ:

http://www.infoq.com/presentations/real-time-twitter

And I’v been wondering, why Twitter needs to be so fast! I have always had a interest in HighFrequency-LowLatency applications and I understand their importance in Telecom / HFT banking market, but does twitter really need it? Working in broadcast industry, I realised when we say a LIVE Game, its actually a 7 sec delayed FEED, and I don’t hear anyone complaining about it. So would it make a big difference if I got my tweets updates 30 sec later rather than sub-second? I can only imagine how much extra energy / development / money had to be spent to gain sub-second or millisecond level latency.

So the question is, was it actually a business requirement or they did it just because they can ? Or the sub-second latency came as a bonus of their superbly scalable design?

Pragmatic Thinking & Learning

sajid2045 productivity, programming, scrum, Uncategorized Leave a comment

I just started reading the book “Pragmatic Thinking & Learning” published from the Pragmatic Programmers. Although I only read only the first few chapters so far, I am so excited about it that I thought I should write a line or two about it. If you are in Software development and want to learn the dynamics of software development and software development teams, you must read this book. This is by far the most impressive non technical book I have read so far.

Here is a snippet to give you some test:

“
Dreyfus at Work: Herding Racehorses and Racing Sheep :

In one of the Dreyfus studies, the researchers did exactly that. They took seasoned airline pilots and had them draw up a set of rules for the novices, representing their best practices. They did, and the novices were able to improve their performance based on those rules. But then they made the experts follow their own rules. It degraded their measured performance significantly.

”

– How true! Can you have the same set of rules for your top developer who is at “Expert/Proficient” level and the average developer who is @ “Novice/Beginner” level ? How can you explain intuition to the developer who needs a recipe? (If you don’t know what I am talking about, go read the book). This is one of the top challenges I have faced adopting SCRUM coming from a strict hierarchical team management. In SCRUM everyone from Novice to Expert seems to have the same “voice” and sometimes it just holds you back. But hey, the efficiency of a SCRUM team is often measured by the efficiency of the weakest node of the team! So thats fair enough 🙂

Lessons learned in Software Development (2) – Abstraction

sajid2045 design, OOAD, productivity, programming 1 Comment

Long ago during our weekly tech meeting @ Therap, we had this intriguing discussion, “What is the most crucial skill for a software developer?”. It was quite interesting to see how different people at different skill level had different opinion about the topic. I remember my answer was “Algorithm” (Back then I was into solving ACM style problem). Other inputs were “Language”, “Communication”, “Optimization”, “Clean Code” etc. However, we all agreed finally that it was “Design skills” that matters most.

My view on this topic also changed from time to time as I grew older and played different roles. I tried to master java (with two java certification under my belt) when I thought language was the most important, I memorized all design patterns from GOF books and tried to apply them when I thought design was the most important one.

Well now a days, I feel like I got the bigger picture. I think it is the power of “Abstraction” that can turn you from a good Software engineer to a Great one. You can find the importance of abstraction everywhere. When we were tought object oriented design at our schools, we first relate the word “Abstraction” as one of the three attributes of Object Oriented Programming. Well, it has much bigger meaning than that. Abstraction is not only part of  detailed design or Object design, Its the most important part when you want to architect big systems.

When I started my career, I used to get overwhelmed by big problems, big tasks, big projects. I tried to address them all together and made a mess of things. Now a days, I try to apply abstraction in such scenarios. Whenever I get big problems, I try to divide them into smaller pieces. “Divide and conquer”–Thats the trick. If you can abstract away the smaller problems and focus on the big picture, at the end of the day, you are going to achieve the big goal. If you are working in a team, you can put your team members to work with each of the abstractions.

I am not going to talk about the Abstraction @ code level. There are enough materials on them. But here is the key, If you can abstract away the components/responsibility/code, you will always be safe from big disaster and achieve the bigger goal.

Few words about “Comments”

sajid2045 books, design, programming, Refactoring Leave a comment

I’v been reading this great book “Clean Code: A Handbook of Agile Software Craftsmanship” By Robert C. Martin. This book is going to be another classic like the “Code Complete” or “Refactoring“. Anyway, I liked and agreed so much with the author about what he wrote about comments that I couldn’t resist sharing a few lines from the book —-

“The proper use of comments is to compensate for our failure to express ourself in code. Note that I used the word failure. I meant it. Comments are always failures. We must have them because we cannot always figure out how to express ourselves without them, but their use is not a cause for celebration.
So when you find yourself in a position where you need to write a comment, think it through and see whether there isn’t some way to turn the tables and express yourself in code. Every time you express yourself in code, you should pat yourself on the back. Every time you write a comment, you should grimace and feel the failure of your ability of expression.”

When we learned programming in our college, we were told by our teachers that you should always write comment! Even when we joined job, our experienced seniors also told us to write comments! It took me a long time to realize that comments should only be used to document some domain knowledge, writing api/library documentation, not to describe code. If you use comments too frequently in every piece of code you write, you should read all three books I mentioned earlier.

Your code should speak for itself. Period.

Lessons learned from Software Development – Part 1

sajid2045 design, productivity, programming Leave a comment

When I started my career, the first thing I made up my mind about is I have to “Stand on the shoulders of giants”. From the very beginning, I tried to learn from the best coding practices, design patterns, case studies, books etc etc. In fact, there are so much materials around that one has to be overwhelmed! But there are some lessons that I had to learn the hard way. Although they are in the books, I didn’t realize them until I made the mistakes over and over again.

In this series of blog post, I’ll try to share them with the rest of the world. Note that these are not my philosophy, I just think they are the most important among all other principles.

Lesson 1 : Pareto’s Law – The 80/20 principle:

While the 80/20 principle is applicable to a lot of areas, I am only talking about the software modules and codes. If you are involved in a big project, You will see only 20% of the code is performing the core task, the rest 80% is just supporting codes. If you have 10 components in the system, only 2 of them is actually performing the task, the rest are just there to help the 20% (This applies if you group your functionality in a proper way rather than scattering them all around).

We software developers often overlook this, we often put our focus in the wrong place. One thing you need to think about is the Return of Investment. If you claim to be an engineer, one thing that an engineer does is Optimization and Trade Off. Say you are developing a plugin framework – the framework itself is much more important than the plugin. If you have limited time and you try to put equal effort on both of them, you will get none of them right. We always work with deadlines, we never get infinite time, so we have to effectively use that time.

Perfection takes a lot more time than getting something done. You have to make wise decision about which areas you need to perfect and which areas just need to work. Abstraction is the key here. If you have good abstraction, you can always go back and improve your code. I was watching this presentation by Eric Evans about “Strategic Domain Driven Design” for large project where he talked about the same thing. I was quite surprised how our view seemed kind of same! Well, Great men think alike 😉

Anyways, the point I am trying raise is that , we developers become too much obsessed with our code and technology and forget to think about the Deadline, estimation and all these stuff. If you don’t think about ROI (Return of Investment), your project is going to suffer from it. This will even make more sense when you think about Agile development, Changes is requirement and “Throw-away System” from The Mythical Man-Month(Another lesson I’ll talk about).

In the next post, I will talk about the Lesson 2: Abstraction

UPDATE: How to Hotswap classes in the running jvm

sajid2045 hotdeploy, hotswap class, java, productivity, programming Tags: 2 Comments

If you have read my previous blog post about How to Hotswap classes in the running jvm, I am sure you will find Java Rebels even more interesting. It supports full class reloading support at runtime with method refactoring and all the stuffs that hotswap.jar could not do. Trust me, Java development has never been more fun. And if you are into spring development, it even supports spring configuration reloading without restarting the whole spring application context! Could it be better!!! I think its a must have feature for every java developer. I don’t know why sun is not incorporating this in the JVM? Just because java is not a scripting based language like php/ruby , doesn’t mean we can’t have fun like those guys! Speaking of fun, you can have some from this Java Rebel cartoon also

Only downside is JavaRebel is not free. There is a small fee that you have to pay. But let me tell you its worth it. And if you don’t trust me, you can always use the 30 Day trial version (although it will only take 3 days for you to become addicted to it).

NOTE: I don’t work for java rebel.I wish they took me in 🙂 it would be fun to develop such cool features that eases the life of so many developers.

How to get the calling method information in Java

sajid2045 Getting the name of the calling method, java, programming Tags: Leave a comment

I was wondering around in a huge code base trying to figure out whats going on.
I wouldn’t say it was a bad code base, but it was done by a single geeky programmer who never cared about the readability because I guess he never thought anyone else would ever try to read it. Anyway, I could see all these meaningful log messages coming out from it but I didn’t know from where. I think he never heard of log4j or anything like that but luckily he used a class like this:

public class LogHelper{    
   public static void log(String log){

      System.out.println(log);  
   }
}

and used it like this:

public class X  { 
     public void doA() { 
        Logger.log("doing A"); 

        //..         
        //..         
        Logger.log("done doing A");
     }

     public void doB() {
	Logger.log("doing B");
        // 
        //

        Logger.log("done doing B");
     }
}

So he called this static log method from all over the code base to do the logging.
And the log looked like this:

   doing A
   doing B
   done doing B
   done doing A

I desperately wanted to know which class/method/line was calling this method like I am used to in log4j.

Then I thought the calling Thread has this information. The question is how can I get it! After playing a bit with stack information, I came up with this:

public static String getCallingMethodInfo() {  
  StackTraceElement[] stackTrace ;

  try{
      throw new Exception();  
  }
  catch (Exception e) { 
    stackTrace = e.getStackTrace();
  }

  if(stackTrace != null && stackTrace.length >= 2) {

     StackTraceElement s = stackTrace[2];  
  
     if(s != null) {

        return s.getClassName() + ".(" + s.getMethodName() +"):["+ s.getLineNumber() + "] -:";

     } 
  }

  return null;
}

So finally the log method looked like this:

public class LogHelper {  
 public static void log(String log){

      String caller = getCallingMethodInfo();  
      System.out.println(caller + ": " + log);

   }
}

So, finally the logs looked like this:

  X.(doA):[5]-: doing A
  X.(doB):[14]-: doing B
  X.(doA):[9]-: done doing A
  X.(doB):[18]-: done doing B

In jdk 1.5 you can actually get the stack information like this:

Thread.currentThread().getStackTrace()

but unfortunately I was working with a 1.4 codebase. But Throwing an exception and catching it back did the trick 🙂

Having said all that, It will be an insane idea to put this code to production. But can be very helpful when you are trying to understand the code.