Be a different fool

How-Tos for implementation of Open Source Portal and ECM Projects

2010-03-29T02:15:00.000-07:00

It is always a big challenge when you want to extend your current web presence and manageability of your content produced by your existing legacy applications. There are a variety of different implementations on how to achieve this goal. In this post I will try to list some of the main criteria and constraints and evaluate their impact and order of magnitude on successful execution of such projects.

1. Available budget and time (this includes current license fees for proprietary solutions)

2. Volume of new requirements

3. Non functional requirements

4. Complexity of current applications

5. Volume of Legacy Data

6. Technologies used in the current applications

7. Size of client organization, on board technical team and existing technical knowledge

8. Foreseeable vision in the industry and its impact on you

Based on the constraints above and some additional requirements companies decide about their choice of framework or application.

In fact, your budget and window of time is often the main driver of most of the projects. But how can you receive a solution in a tight window with an already limited budget? It makes sense to adopt an already developed framework and build on top of that to cut time and cost. So, how are you supposed to train your team (if you have any) to pick up that framework? The answer is easy; you need creditable system integrators and reliable open source solution providers. Hence, you will have your project delivered and concurrently train the team during implementation.

Alright, let’s say you are pretty convinced about dancing with open source Portal/ECM technologies. Obviously it is not going to end up like an arranged marriage because you should not pay an arm and a leg for it. You are just dating some open source solution providers, just like high school days. The whole idea is to not anchor yourself.

Now the question is how could an open source Portal/ECM solution help you in satisfying your requirements? To be quite frank, open source frameworks may require light to heavy customization in order to satisfy some of your requirements. What makes them quite compelling is that being open source makes it possible to extend and customize them. Indeed, volume of your requirements should be balanced with your time and budget.

Currently there are a variety of different Open Source Portal and CMS frameworks in the market, frameworks built on Python, Java, PHP, C++, ROR, and basically any language or technology out there. Most of these frameworks come with tons of OOTB (Out of the Box) features and cool stuff. However, often most of these freebies must be polished and re-skinned to match your requirements. In fact, assessing these frameworks and matching their OOTB features with the requirement list is the first thing you need to do.

After benchmarking your requirements with available frameworks or having someone to do it, you can shortlist a handful of products for your project. Next step is to categorize the requirements in respect to the shortlisted frameworks as below:

1. Requirement is satisfied by OOTB feature: The selected framework will contain the code and services.

2. Requirement will be satisfied by extending the OOTB feature(s): The customizations and extensions should be done in an extensible pluggable way, I will describe after this section.

3. Requirement is custom and must be developed. This type of requirement could be broken into the following sub categories:

a. Singular requirement: For instance custom Admin, dashboard, contact us page, custom Portlets, unique user attributes and etc. Basically anything that is unique and not repeatable: Such requirements should be implemented as external services or components as much as possible. You do not need to bind the developed components to any frameworks. This is vital for near future extensions. Especially when you consider the trend of technology where services will be hosted on the cloud in near future.

b. Repeatable requirements: For instance email templates, Product Pages, Order entities, Workflows, Search function, etc. Basically anything that is repeatable or could be generalized to be reusable: For such requirements a custom framework must be designed containing generic extensible services configurable for your repeatable requirements. This is imperative especially with big projects where amount of time put in designing and creating such framework pays off in reducing overall production time.

After or even before this analysis you should compare the shortlisted frameworks with your current in house expertise. Basically a framework is discarded if it does not match your in house knowledge. Let’s say your team knows PHP like the back of their hand in which case, it would not be wise to pick a Java based framework. But what if you are coming from a big corporate using several technologies, what should be your approach? What if you want to implement a Portal or ECM aggregating legacy services or content from different sources each built on a different technology?

To answer this question it is better to analyze the impact of complexity of current legacy applications and their data on architecture of the solution first.

You may think your new solution must fully replace the existing legacy applications and the current data should be migrated. I would agree if you are using a solution from the eighties or you want to get rid of licensing fees of proprietary products. Otherwise, most of the time it is not necessary or even wrong to fully replace the existing systems. Let me categorize the cases where you need to replace a product:

1. The current product is using an old technology and you receive no in house or external support for it. In other words, it is impossible to extend, alter, or even maintain it.

2. The current product is an expensive proprietary solution and you need to cut cost.

In other cases it is much better to wire all or some parts of the old legacy applications with the new Portal or ECM solution. It is recommended to do the wiring through Web Service, REST, CMIS or other widely supported interfaces. By the way, this kind of wiring is affordable if the new Portal or ECM is an Open Source one.

Integration of old and new applications through adaptable and extensible interfaces helps you in cutting cost and reducing production time. In a nutshell, my idea is about maximizing utilization of useful components and increasing their re-usability by wrapping them with configurable interfaces.

So far we have discussed different types of requirements and techniques to implement them and how to integrate applications together. It is very important to execute the integration via pluggable, extensible and reconfigurable interfaces. Following these design patterns enables you to be able to box solutions in atomic packages and make them independent of each other. All you need to do is to make sure that they can communicate with each other. This way your dependency on technologies will be reduced and your near future customizations could be done much easier.

As mentioned before it is really important what Portal or ECM technology is selected. On the other hand, it is vital to have a flexible Portal and ECM backbone for hosting different types of technologies and content. It is better recognized, if your company is serving a big volume of data produced by different applications. In a nutshell, considering the pace of change in the industry you always need to be able to reuse your existing services instead of renewing them constantly.

You should be hearing cloud computing, cloud services, social networking and things like that every day. The fact is you need to be ready to publish your services to the cloud or be able to consume services from it. In order to do that, you need to focus on the integration level and extension of the current services more than recreating services with new technologies.

As illustrated in the diagram below generic modules and components should be loosely coupled to each other so in future extensions you would go through smaller changes.

To recap, it is important to be technology agnostic and be flexible to interact with components developed in any language. As depicted in the diagram above it is highly imperative to separate the components from each other and make them as reusable as possible. In order to achieve such goal the following steps are necessary to be considered.

1. Have the right vision about the future of your requirements

2. Prioritize your requirements based on your budget and time and cross check them with the framework you select.

3. Do not fall in love with any technology, they change every day.

4. Develop the components in standalone packages and modules. Even avoid developing them within the open source Portal and ECM stack. You do not want to marry them either. Remember, you are just dating them for maximum five years. Standalone packages could later be reconfigured and reused over and over. But if you change the framework chances of re-usability drop drastically.

5. Try to maximize usage of the existing services and save money and time for more important requirements. Try to componentize existing functions and equip them with configurable, extensible and reusable interfaces.

6. Execute the project by the right people who know the open source stack and technologies inside out. Do not worry to outsource the project. You always can have your people trained on the job during project implementation.

7. Try to separate rendition of your content from its processing and population. The only thing you need to display in a Portal is the rendered content not the way it is processed. In other words package the rendering layer separately. The view or rendering layer should fetch data from other components through the discussed interfaces.

8. Do not used heavy technologies in the view layer. Try to use lightweight ones instead. Your Portlets should be designed as shell rendering your already processed content by other modules. If you execute this correctly, you could render your services in any Portal, or application.

9. Keep your content components separate from the other components and access them through generic and configurable services. You do not need to be bound to one particular CMS framework. Remember you are dating.

10. The middle layer and business logic services should be template agnostic. The rendering layer is responsible for templating the processed content. The business layer services should be designed in such a way to be consumable by any Portal (for instance iGoogle, iTunes, Facebook, etc.) with minimum effort.

Finally, I need to mention that I have exercised this idea for several big clients and it has worked out pretty well. My concept is pretty easy, do not waste your time by repeating yourself, look for already developed code and try to make it reusable and extensible. Ain't it easy?

Why is web UI development slow?

2008-07-17T00:32:00.000-07:00

Have you ever been a team member in a web framework or web product development? You may have noticed although UI prototyping is a fast process but UI integration is disappointingly slow. Have you thought why does this happen? Is it because of not using MVC design pattern and templating technologies? Is it because of poor code quality, tight time frame or ever changing requirements?

To answer the above question, I would say it is more because of the sluggish process of inserting the framework components and tags into the UI pages. Especially when you try to see the changes as the UI is being translated inserting MVC tags and components into the view pages. As many of us have experienced UI designers and usability people can freely preview UI pages as they are being built. Well, at the other hand you may have noticed that a developer can only preview the translated UI pages when they are served by an application server. That really slows down the process. Each time you should wait as the application is built and deployed, again wait for the app server to come up and then finally checking your change. That is just a waste of time, sucks, nah?

So let’s talk about a solution to make the UI construction faster and easier? In this post I am going to introduce the first part of new discipline for UI development by smart manipulation of the existing technologies. That is true, no new technology.

I have learned in time that solutions must be simple not complex; complex solutions are always more error prone than the simple ones. To realize the necessity of the change for UI construction let’s answer to the questions below:

● What if UI designers could develop UI separately from developers and test them with mock data?

● What if UI designers could still leverage from easy preview features of their IDE’s while constructing the final UI pages using only HTML, CSS and Java Script?

● What if backend developers could integrate the UI to their code simply by introducing some touch points (Think of touch points as URLs, yes I mean RESTful API here)?

● What if the developed UI could be interoperable and be easily integrated with any backend technology like Python, .Net, Java, etc. (Relying on XML and Ajax for interfacing)?

Well, we can continue with questions but I think it is better to see how they all could be possible. It is simple just follow the following steps:

● Avoid using any non HTML tags (JSF, ASP, JSP, Struts, etc.) in the view layer pages; just use plain HTML with Java script. This helps for faster development and easy page previewing during construction. Simply there will be no need to wait for the app server to be restarted or even waiting for hot deployment.

● During unit development and testing, simply test the UI with some static mock data, whether it is text or XML. I personally recommend using XML for complex data types. It is better for standardization and reusability and extensibility of your code both in the backend and front end.

● For any Ajax based component follow a consistent RESTful API. It means that in the backend the integration points should provide XML based RESTful APIs. This helps the ultimate view layer abstraction from the backend technology. This would also lead you to become adaptable with future ESB, SDO and Metamodel integrations.

● Try to package and proxy XML de-serialization and parsing in reusable and reconfigurable Java Script functions. As an example, study Dojo, JSON, JQuery, etc. In fact, there are frameworks like IceFaces or DWR that perform XML to Java Script data serialization and de-serialization out of the box. Eventually, after creating a dozen of such reusable and reconfigurable functions, then most of your new requirements would be satisfied by an already developed function.

Basically, there is always a tendency that opposes any direction or discipline change from the current state to a new and unexplored state. To dump it down, there is inertia for quitting the View chunk of the MVC technologies, and simply hopping on this UFO. In fact, this discipline is not a new technology. Instead, it is a simple trick for accelerating the development process by abstracting the view layer from the underlying ones and making it more interoperable, cluster-able, cache-able and reusable. The Lego parts have always been there and this time we are not playing based on the given plot, but we want to play it above the age limit.

Indeed, I have to elaborate this discipline more and describe it by tangible and solid examples. I believe each of the bullets above could be a post for itself. I will definitely expand them with detailed and comparative samples in the coming days.

By the way, I am looking for a name for this discipline, what about MASK?

Enhanced Object Pool

2008-06-18T04:01:00.000-07:00

The classic design pattern for an object pool is to define a synchronized List (or rarely a Map) object that stores initialized objects in it. Each time a consumer requests an object the controller Thread selects a free object from the list, marks it as busy and returns it to the consumer. For releasing the object back to the pool the consumer calls the controller Thread and subsequently it passivates the object or reconfigures it or simply sets its status back to free (based on the complexity of the object). The controller Thread is also responsible for object initialization if there is object starvation in the pool. It is worthy to remind that most of the common patterns prescribe a block size for object initialization rather than creating only one object.

// The conventional borrow method
public synchronized Object borrow() {
   if (pool.size == 0) {
      createObjects();
   }
   Object freeObject = pool.getNextElement();
   return freeObject;
}

If an object consumer has to wait for object initialization then the pool design pattern becomes meaningless. To avoid any delay in responding back to the greedy object consumption, instead of losing all resources and then initializing a number of them, set a bottom limit other than zero so if the pool hits this limit it initializes more objects, so it always has ready objects. Likely, in order to control memory and process management (if objects are Threads) set an upper limit, so each time the pool has more than a certain number of objects it passivates the excessive ones.

// The enhanced borrow method
public synchronized Object borrow() {
   if (pool.size <= limit && initializingFlag == SLEEP) {
      // a separate Thread
      objectInitializerThread().resume();
   }
   Object freeObject = pool.getNextElement();
   return freeObject;
}

It is important to note that a separate Thread must be responsible for Object initialization and if the initialization process is in progress then do not exhaust the resources again. This requires a heedful calculation for the bottom limit and block size adjustment.

Now if the snippets are compared it is obvious that the second one is more responsive than the first one. It is worthy to note again that based on your specification and requirements the bottom limit and the block size numbers must be configured in a way to avoid any object starvation in the pool.

Next I will present a formula for adjusting these numbers and afterward will unveil a series about the new Client Server architecture based on Ajax, RIA, REST, SOA and ESB concepts.