The Developer’s Pit

(This blog post was inspired from a discussion I had with Moti Gust earlier this week, Moti and I have been working jointly on various development projects since early 90’s)

 

Model-View-Controller (MVC) is a corner stone Design Pattern that in recent years received more and more acceptance. MVC is a “solution” to various problems raised when presentation code, business logic code and data access code are mixed together. MVC separates a UI driven application into three components: View - the presentation code (which renders the contents of the model), Controller - translates between user actions (received through the View) and actions performed by the model and Model - representing the enterprise data / business rules that communicate with the data model. The following image (taken from here) summarizes the concepts in this design pattern:

 

 

There are some frameworks that endorse MVC as the proper way to build UI applications, such as (part of the Castle Project)  and recently Microsoft has been looking into integrating MVC into the .NET Framework 3.5, lots of info on that around the web, one such link is this.

MVC is a great concept, it provides away to separate between pure UI code, UI logic and application logic. The separation allows the UI developer to be de-coupled from the application logic, all he needs is a good, robust API and to separate between the graphical / rendering part of his presentation code (the View) and the link to the business logic (the Controller). It even allows another developer, who did not originally wrote the rendering part to tie between the model and the presentation. It gives a lot of flexibility. But Dragons be here - is that really the solution we’re looking for?

Before going into the real-life example which flared this whole discussion, I think it’s worth a while to pause and think about Design Patterns. There are many ways to categorize and segment Design Patterns, I won’t get into all that, but if you look the MVC, Publish-Subscribe Design Pattern and Patterns like Singleton something very evident pops up : Singleton is implementation driven, it’s practicable (while some would argue that it’s overly used incorrectly [which I agree to, but we'll leave that for a later post, maybe]), its a technical way to write you code, it’s a tactical solution. While MVC (and others) are different, they are CONCEPTS, they are a way to think of information driven SYSTEM, it’s a strategical solution. But what MonoRail and Microsoft are doing is to endorse  MVC as a framework, rather than system’s architecture solution, and in my mind this falls short. If your development group is well disciplined and the design is good, you don’t really need a framework to uphold the rules - there are many other ways to address that. What is missing in my mind (and I’ll try to prove it with a real-time example) are system architecture solutions that address real-life application development & deployment cycles that address mid to large size projects with MVC. Instead we get some development frameworks suitable to a small group of developers working on smaller scale projects, at time it seems more of a university type of solution than issues we tackle in real life.

So after all that, let’s get to the problem at hand. JAJAH is a medium to large size software project with a wide variety of  technologies, we’re also a global company with Marketing in the US and R&D, Operations and other departments in Israel. One of our hurdles is that content changes should be close to the Marketing team, while today every request has to flow to the R&D team and be part of the release cycle. This causes extra work (tidious, not-so-interesting work for our killer developers) on the R&D team, and does not provide quick enough response to the Marketing team. Hence we need some Content Management System where we can have a not-so-expensive HTML guy next to the marketing team making content changes. This is practically MVC. However, we need different release cycles, the View should be developed and deployed in a different way than the rest of the system. It’s not just separating everything to different projects, DLLs etc - it’s a whole different system. We need to QA the content, having it in a staging environment and have proper roll-out procedures. We also want to limit the changes that the HTML guy can make in the system, we don’t want to give the guy full control over all rendering aspects, just certain parts. We used to play around with Offermatica (recently acquired by Omniture), but while it’s a really cool tool, it’s mostly focused around A/B tests and is not suitable as a heavy duty content management system.

MVC is cool, and it makes a lot of sense, but what we actually want is M-V-C, totally separating the components for that a framework is not enough. For that MonoRails is just not good enough since it keep everything together. What we need is some type of a Content Management System, coupled with a release cycle & deployment management, and same applies to the Model. The controller is a different story, at times we would like to couple it to the Model and times to the View release cycle.

My point in all this is that Design Patterns should not be regarded as framework solutions pre-say, but rather as design concepts being applied to the high level system architecture and not necessarily at the development level.

Amichay

On January 2008 I wrote a blog entry about Google’s Android and Social Network API’s under the title ‘Developer is King’ . Recently a friend directed me to Google’s new initiative : . Google is actually not the first to provide and host a network based solution for developers, a well known solution is . But while Amazon provides Web Services for storage, payment and others, Google actually took this a step forward - Google is taking this way beyond Web Services and provide a full blown Application Engine running on top Google’s monstrous infrastructure.

I think this is quite ingenious on Google’s part, they provide a fully hosted application environment, allowing developers to enjoy Google’s infrastructure for storage, load balancing and scale, authentication and various other API’s. Google’s MapReduce and GFS probably serve as an infrastructure together with their monitoring, and hosting technologies.

Each application is running in it’s own secured Sandbox allowing distribution between multiple servers and distributed web requests. Currently only Python programming language is supported, but assuming this takes off more are likely to be supported.

This move on Google’s part should attract application developers to use Google technologies as well as create an eco-system around Google web technologies. This is a bold move that will probably take a while to mature but if you’re a web application developer this is certainly something to look at.

Thinking back, it was Larry Ellison, the legendary CEO of Oracle, who coined the term Network Computer (NC)  in the mid 90’s, while the initiative itself did not really take off it created a lot of buzz that later lead into ‘thin clients’. If Google will play this right, thin applications will emerge : applications that take into consideration their distributed and scalable nature, without worrying about the complicated setup, storage and hosting environments.

While it’s hard for me to know how strategic it is for Google internally, and how they intend to push this, if at all, I think this could make a difference in the constantly evolving  computation world. I can only bow to what Google as a software giant is trying to push. If I were Microsoft I would do some serious thinking, mostly since .NET - Microsoft’s leading environment is severally lacking an application engine, but I will leave that for a later post…

Amichay

Kylie Minogue song (not that I’m such a great fan), third paragraph writes:

"you know it’s all in your head

You better put that business to bed

By your fair hands of design you met with

The monster in your mind"

when it comes to iPhone - this is so true! not that I think that Kylie is such a techno-prophet …

Apple did it again, and it’s all for the sake of the short sighed stock price and the word on the (Wall) Street. Though the writing is already on the Wall - trying to squeeze much of a platform for the sake of the product revenues is not a long-term-winning-strategy. Apple had a taste of that during the Mac-PC wars, and they lost even though their ‘fair hands of the design’ where all over the place. Google has already figured it out by targeting the echo-system, Apple however is taking a different approach. iPhone Bluetooth is blocked beyond using the headset, iPhone is locked to certain US carriers, iPhone SDK is limited.

Basically Apple is trying not to cannibalize their short term earnings from the iPhone and try to squeeze as much as possible from the product / buzz, but this is a short sighted vision. I would have liked Apple to take the lead not by having copycats grab concepts and idea from their iPhone product, but provide a winning open platform with rich API’s to really change the mobile world.

There are over billion mobile devices around the globe - aside the fashion statement iPhone’s target of 10M devices is negligible.

Apple please don’t do it again!

Amichay

 

Took me a lot of time to realize that nobody really understands networks, many say they do, many more think they do - but very few actually know what they are talking about. Specially when it gets to setting up massive data centers. At the beginning of March 2008 we plan at JAJAH to turn the lights on our new data center in New-York City. In case you want to visit us you can probably find our IT & DBA teams here around the beginning on March, but don’t piss them off - they’ve been working day and night for the past couple of months so they might be a little cranky by the time you get there.

This post is greatly influenced by our head of engineering Alon and his quest for the flawless data center.  Alon is not alone in this quest (hhm…) , this undertaking is joined by our system administrators, storage and other experts all working to get the perfect, scalable and robust solution to serve our growing user’s community as well as providing a sound base for JAJAH business growth. But before we pin the medals let me share with you one bizarre experience that led me to the belief that network is more of an ART than Science. 

Alon setup site-to-site VPN between our existing data-centers and the new NY data-center. However, when our DBA tried to transfer data from a remote database to our new monster DB machine in NY, the connection would simply hung. Alon and his team looked everywhere, checked the ISP connection, Firewall cluster, routers, BGP setup, MTU, blades center set-up, DB cluster, OS, installed and re-installed every piece of hardware - you name it - but nothing worked and time was running out. Our DBA started to get agitated (big guys - you don’t want to upset them), we had to look for a work around. We found the most bizarre workaround ever - TCP OFFLOADING.

TCP Offloading is a great feature which moves TCP stack processing from the main CPU to the network interface. Works great and improves performance, but sometimes causes problems. Just by chance we discovered that if you disable TCP Offloading and move the network processing back to the main processor things start to work well for us. Dotan, our head DBA was smiling again. Yet another day in the office.

Although we could not prove the performance lose by disabling TCP Offloading we knew that we paid for the feature, might as well get the damn thing to work. So while our DBA team was back on track we started to investigate deeper. We found two things which you don’t always read about in the school text books, but you run into in daily life.

First some background. there are two important mechanisms that play a significant role when two sites have to be connected over the web (specially when firewalls are involved): MSS - the TCP maximal packet size between two networks, and MTU size. MSS is specially important when trying not to fragment packets (for obvious reasons packet segmentation is a problem when it comes to encrypted networks). The other important mechanism is PMTUD which stands for Path MTU Discovery - an automated mechanism for MTU size negotiation when sending traffic around the web. A nice article from CISCO that explain about MSS negotiation and path MTU discovery  can be found here. 

Few things you have to be aware of when setting the MSS & PMTU:

• MSS - MSS is negotiated between the two end-points, but it is fixed in nature - the smaller of the MSS will be used by both sides.
• While MSS is set between the two ends point, PMTUD is set according to the route between the two end points.
• MTU is automatically selected using ICMP. Namely, one side sends packets in certain MTU and the other side returns notification using ICMP protocol. However, ICMP is also used with the common ‘ping’ utility (ping is actually sending ICMP and waiting for the ICMP echo to return). Because of potential ‘denial of service’ attacks many security officers block ICMP messages (Alon is also our security officer, the guy would block HTTP if we’re not looking), and this many hamper PMTUD. You need to make sure your firewall rules are set correctly to allow the relevant ICMP messages to be sent between the two VPN sites. Your access list should look something like:

• For PMTUD to work you need to make sure the DF (’dont fragment‘) bit is set in the routers.

Once Alon got the MSS correctly configured, and proper rules where applied to allow PMTUD -  we could enable again the TCP Offloading. Go figure.

Amichay

p.s.

Ori, Alon, Nir, Ilya, Eran, Dotan, Dani - this one is for you…

I write this blog entry some 10688m above a place called Yakaterinburg , flying back from Tokyo. This is my third visit to Japan in the past few months, working with our Japanese partners. A lot has been said and written about technology in Asia, and particularly in Japan. "Everyone" knows that Japanese like gadgets, but in my short visits I’ve gained some more insights into Mobile technology in Japan that makes me admire the Japanese way of thinking. A quick comment before I start, one thing which fascinated me in Japan, and I’m quite aware that this could be just my personal conditioned experiences, is that influences of tradition are more "felt" in Japan than other places I visited. While Japan is clearly far off from the Samurai days, I could still sense some of the culture and history in contemporarily Japan. Unlike places where the cultural roots seems to have faded. 

Back to technology, here are some things I picked up:

Mobile Devices are bulkier than the small, round edges devices I’m used to (I am a big fan of Nokia) : Devices in Japan tend to be big, mostly because of screen size (see below). That took me by surprise, I always thought that Japanese customers prefer small devices.

• Mobile TV - Japanese devices have the capability to receive TV broadcasts. To do that, relevant devices are equipped with a special antenna. People actually site down in trains and watch TV broadcasts.  Quality is very good. Apparently Location Based Services are also underway meaning that users receive certain local broadcasts when located in certain areas such as a community TV and others (there are some other interesting applications, but let’s leave it at that).
• There’s an interesting feature to Mobile TV - and that’s connectivity to web based information. When I was in CES earlier this year, I saw all these science fiction home TVs where one day you might be able to browse the web while watching TV (side note: why do all these demo use cases always converge to watching "Sex and the City?"). While it may still be science fiction on home usage in the US, it’s a reality in Mobile TV in Japan. That’s really cool, you can see special links under the Mobile TV screen or relevant web based tickers.
• I’m not sure why people would want to watch TV on their mobiles, on the other hand  I hardly watch TV at all (just too many things to do in real life), so I’m probably not the target customer. But fact is that this technology is commercially available in Japan makes them in my mind way more advanced than Western mobile users. I also think that Mobile Advertisement will likely to pick this up. Current Ad solutions I saw are not targeted, there’s an interesting market opportunity there in my mind.
• Mobile Data connections in Japan are far more advanced than we be found in Europe and certainly the US. HSDPA of 7.2Mb is nearly five times faster than in my home country (it’s also different frequency), not to mention hardly any packet drops (and I checked…). I also found it interesting that WiFi networks are not as common as they are in the West, seems as if people rely much more on mobile data networks than wireless devices. This really caught me by surprise.
• SMS / Text Messaging is uncommon in Japan, I saw large portion of the people in the Tokyo underground fiddle around with the mobile devices either playing, or writing messages, at first I was sure they are SMSing, but they are not - they are sending and receiving emails. While push-email in the West is mostly used by business people, in Japan everybody send and receive emails to their mobile phones. Always wired society.
• While Japan is governed by huge corporations, I found the underlying VoIP infrastructure already deployed by Telcos quite advanced as well as VoIP capabilities I did not see anywhere else. Since dealing with large, heavy corporations I expected them to be some what laid-back when it comes to VoIP, I was wrong. But for obvious business confidentiality issues I can’t elaborate.

If you do get a chance - try visiting Japan. It’s not all about technology, the people are very welcoming, food is great, culture is interesting, service is superb and there are some great hiking places in the mountains (wish I could do more of that).

Till next time

Amichay

I’m a big fan of Martin Fowler and his work. Recently I came by an interesting panel from InfoQ ‘07 about Modifiability in Agile development () . While I concur with many of the things said, one thing I would have expect to have in that panel, and did not hear - is focus on writing software which is easy to maintain and support.

 

A lot has been said about Test-Driven development and how important it is to understand the problem domain and start by writing the top-down test plans and embed it into the software development process. That is quite right, but the problem domain does not only covers the end-user, it also covers the people who need to maintain and support the software. Let there be "Operations Driven Development" an augmentation to "Test Driven Development".

 

Some background: JAJAH Operations team works 24 by 7 (some actually say it’s 26 by 8, but that’s a different story) with the sole purpose of delivering the best possible VOIP quality. This covers web servers, application servers as well as telephony and networking equipment. That’s kind of complicated, marrying all these technologies together and than make sure they all play nicely together. That’s where JAJAH NOC (Network Operations Center) comes into play, tracking every part of the system, around the clock, constantly. These hard working guys don’t care much about Modifiability, they care about Maintainability.

 

Hence, the consumer who picks up the bill is only one part of the equation when it comes to software development of mission critical system (and others as well). As software developers, we should not only think on how to make sure our software is tested according to the product understanding, but also how do we as developers bring out software that other can keep on running.

Developers should be well conscience about software monitoring, alerts, logging and error reporting. Providing good monitoring, error reporting and exception handling so other people can perform root cause analysis and react is not a science - it’s an Art. It’s also a sign of a mature developer.

 

I’ve been looking into error reporting Design Patterns and found surprisingly too few of those. I wonder why.

 

Amichay

For me it all started back in 1998. I helped co-found a company called CyBook Inc, with three good friends; one of them is my wife, who since than decided to follow other venues, another currently works as JAJAH VP R&D and the last one runs his own freelance business now. We teamed up with an energetic fellow from the ‘Silicon Valley’ and together started to follow our dream: create a device that will replace printed books. While the topic of electronic books is interesting (and I will probably return to this in future blog entries) I wanted to focus on one of the three slogans I learned during the short livelihood of CyBook: "Brave New World" (BNW), "Not Invented Here" (NIH) Syndrome and "Customer Is King".

When I proceeded to my next venture, the concept of "Customer Is King" crystallized and formed our way of thinking, especially since we were in the Internet consumer business. Few years passed during which I closely witnessed the struggle of the titans: Microsoft, Oracle and SAP, and understood what makes Microsoft different than the rest of the pack: Developers, Millions of them, anywhere around the globe: so many people sitting around, hammering their keyboards, writing software on, with or for Microsoft environments. It is the developers who wheel the industry; not only the customers who consume the services and products we make.

When Google released “Android” a few weeks ago, I was puzzled and started to look deeper into both the platform and their motives. It was than that I realized that the industry focus has changed: we are no longer the knights who say "Customer Is King", we are now the nights who say "Developer Is King".

The era of API’s is here.

The new battle ground of the giants is now Mobile platforms and Google realized that the only way to sway around Microsoft strength is to attack their core: the developers’ community. "Go back to the source" say the Architect(s) at Google while releasing more API’s and trying to convert Microsoft developers into Google developers. It is not a coincidence in my mind the Google released two different development platforms nearly simultaneously: "Open Social" for the social network applications and "Android", a new Linux based platform for Mobile development.

It is without doubt that many companies are putting a lot of effort into releasing more and more API’s and putting more and more emphasis on the developers’ community - Developer is King.

 

Amichay