LiteBytes from Lustratus Research

Integrating the silos: Not only an IT problem

Talking about siloed application is such a cliche in IT infrastructure circles that we can sometimes forget one of the main reasons for siloed applications is siloed organisations.  Furthermore, siloed organisations are there for many good reasons - alternatives exist full of matrices and dotted lines but they do not suit every business, nor are without their own problems.  McKinsey Quaterly recently covered all of this in a report (you need to register for free to read it) which focuses on financial services and highlights the particular business drivers which are forcing greater cross-silo integration:

"Yet many [financial services] institutions find the going tough as clients demand—and increasingly expect—integrated services, including (among other things) advice, the raising of capital, and risk-management solutions."

There is no simple hierarchical organisation that can deliver this level of integration (and this is also noted).  Therefore, the only solution is to promote and encourage strong formal and informal networks - individuals cooperating across organisational boundaries to address customer requirements. I am sure that IT person reading this who is also thinking about SOA will find all of this very familiar.    And SOA clearly supports these human networks at an application level by enabling easy connectivity between the silos. 

In fact, most of the report is about how to map these networks and how to identify the good and prune the bad:

"In our experience, more networking isn’t always better—hence the inability of broad forums to add value by themselves. Indeed, one firm we know raised its productivity 25 percent by eliminating interactions that didn’t add value."

And clearly there is an IT dimension to this:

"One global investment bank, for example, has put a good deal of money and energy into a knowledge-management system that enables managers from different functions and product groups to access expertise and build on existing knowledge rather than reinventing the wheel."

While I am being a little unfair in picking on this one example in the paper, this point and a general emphasis on top-down enablement jars a little.  I am sure Web2.0 proponents would feel that this is starting at the wrong end:  The "right" end being make it much easier to create and develop networks by encouraging use of tools like wikis and blogs.  This is the right end because forcing managers into the role of informal network creators is clearly oxymoronic - and counter-productive in my own experience.  Furthermore in most cases, the level of informal networks within firms is low and random and therefore requires something more fundamental than 'getting to know the team across the corridor' drinks receptions. 

SOA itself also requires the same type of human networks of individuals cooperating to support service reuse and good governance.  Again, the wikis and blogs have an important role to play either in parallel to or embedded within registries and other governance tools - but that is a bigger topic for another occasion.

Ronan

SOA and Conway's law

Jim Webber's blog reminded me of the existence of Conway's Law which seems particularly relevant to the challenges SOA governance attempts to address.  For those of you who haven't come across this law, two forms of it are...

"Any piece of software reflects the organizational structure that produced it."

or a more techie version:

"If you have four groups working on a compiler, you'll get a 4-pass compiler."

The first is a good encapsulation of the reality of working in enterprise IT:  And SOA governance attempts to optimise the way in which software reflects the organisational structure through building expertise, capturing and formulating successful patterns of use, promoting 'good behaviour' and so on.

The second statement of Conway's law, while being more jokey, also brings in implications of human nature: If you set up teams with separate responsibilities, expect them to collaborate sufficiently to complete the job but also expect them to carve out their areas of control to the potential detriment of the overall solution.  This is true with all professions - however mixing in the stronger tendency to 'not invented here' with IT and the problem becomes even bigger.  The SOA equivalent of the 4-pass compiler is poor rates of reuse and multiple versions of what is essentially the same service.  Overcoming this again requires good governance - promoting communication between teams and incenting reuse - and good technology to support it.

Ronan

SOA, governance and the Trough of Disillusionment™

All shiny new things in IT go through a cycle of boom and bust before becoming an established and useful part of the enterprise IT world.  This cycle is as much about the need for marketing departments and the press to have something new to write about as it is about ‘real’ issues.  The pattern is so well established that Gartner even has graphic to capture where technologies are in what they call the hype cycle – the lowest point before technologies become generally understood and used is called the trough of disillusionment.

Service Oriented Architecture is no exception and Brenda Michelson of Elemental Links even celebrates the possible arrival of SOA at the trough of disillusionment as a good sign!  Moreover, I suspect that with SOA the trough may be worse than with most because it is more fundamental to the enterprise than ‘point’ technologies and attempts to address the fundamental issues of alignment with business strategic and IT agility.

I contrast SOA with point technologies because SOA is actually about changing the way IT solutions are created, deployed and maintained – not providing a single (however big) solution to a single (however big) business problem.  In essence, it is about enterprise architecture (and architects) and how this fits into the whole business.  This means that with SOA there should be no such thing as a stand alone project.  A stand alone project is inherently suboptimal as it removes the opportunity to reuse existing services and to expose further services.   It may take organisations many years to get there but this is the final destination.  This means that the organisational impacts of SOA (both human and technological) are in the long run the most important aspect of SOA. 

What has obfuscated this point to a degree is that most discussions around these issues are gathered into the term governance.  Unfortunately, using the term governance has put off many of the people who should be interested – perhaps governance suggests discussions on how to organise committees!   Furthermore, the term has to a degree been appropriated by vendors with products which assist in the technological aspects of governance.  And finally, there isn’t a single clean solution to governance as it is inherently tied into the way each organisation does its business which makes it hard for the press to get to grips with it.

All of which doesn’t take away the fact that governance is where SOA success will happen or not.  This of course does not mean that we now need a big-bang SOA Governance investment.  To quote Fill Bowen from IBM speaking at a presentation/discussion on governance hosted by the SOA Consortium at its European event back in June:   

“your SOA governance [should be] based on the level of SOA effort. So putting a
Cadillac SOA governance system in place when your SOA effort is targeted at just
trying to walk seems to be a little bit of overkill.”

Ronan

Sun becomes JAVA (for the NASDAQ at least)

Sun has taken the strange (to my mind) step of changing their NASDAQ stock symbol from SUNW (apparently it stood for "Stanford University Network Workstation," and according to Jonathan Schwartz in a blog item, where he comments on the change, the SUNW symbol "heralds back to Sun's cherished roots (in academia)"). The reason given by Mr Schwartz is:

"To be very clear, this isn't about changing the company name or focus ... But we [Sun] are no longer simply a workstation company, nor a company whose products can be limited by one category - and Java does a better job of capturing exactly that sentiment than any other four letter symbol. Java means limitless opportunity - for our software, systems, storage, service and microelectronics businesses. And for the open source communities we shepherd. What a perfect ticker."

Ticker symbols are sometimes seen as status symbols in the US - in particular having a single character ticker symbol on the NY Stock Exchange (for instance AT&T's symbol is "T") has a certain cache. However in general stock symbols are 4 letter abbreviations of the company name (goog for google, msft for microsoft etc). However I can't think of another company that uses the ticker symbol to make a marketing statement as this seems to be (that Sun = Java = Sun).

Without any inside knowledge one could speculate that there has been another round of internal discussion about whether to rename the company to Java and it has ended in this compromise. What is not in doubt is that Sun remains on the horns of its long standing dilemma: how to get the maximum commercial benefit from its creation of Java without undermining Java success. Frankly, I can't see changing the stock ticker symbol making much difference either commercially or on the investor front.

Ronan

CICS users of SOA get a bit of a headache

IBM has many SOA tools in its armoury, and has made it quite clear that SOA is a key focus for the company. I find it disappointing, therefore, that IBM seems to have slipped up a bit on its normally strong record of providing onward compatibility with its SOA solutions. Users of IBM's power-house mainframe transaction processing product, CICS, have a bit of a headache coming - CICS TS 3.1 offered many useful tools for using CICS resources as part of an overall service-oriented architecture, including the Service Flow Feature. However, as IBM rolls out the new CICS TS 3.2 level of the product, it turns out that according to IBM, service flows built for the CICS TS 3.1 environment wont work with TS 3.2. The IBM announcement letter states:

CICS Service Flow Feature:

The level of CICS Service Flow Feature that is provided with CICS TS V3.2 cannot be used with CICS TS V3.1. The CICS Service Flow Feature of CICS TS V3.1 cannot be used with CICS TS V3.2. Customers who have implemented service flows using the CICS Service Flow Feature of CICS TS V3.1 must migrate the service flow models created to the CICS Service Flow Feature for CICS TS V3.2 and regenerate the service flow executable for deployment to CICS TS V3.2. Tools to assist in migration of flow models created using the CICS Service Flow Feature of CICS TS V3.1 will be delivered when the CICS Service Flow Feature of CICS TS V3.2 is available. Service flow executables generated using the CICS Service Flow Feature of CICS TS V3.1 will not run under the CICS Service Flow Feature of CICS TS V3.2. Forward binary compatibility between offerings is the usual migration approach that IBM strives to offer. In this instance, as the result of the maturing of the implementation to deliver enhanced serviceability, forward migration at the model level is the required approach.

Although IBM says it will offer tools to help, I think it is a bit of a poor show that onward compatibility could not be maintained. Customers who have used the CICS service flow feature will not welcome this headache.

The Year of ESB-ability

Recently, I wrote about the ESB (enterprise service bus) maturing at a basic functionality level, and how the focus was swinging around to the '-abilities' such as scalability, availability, usability, manageability, etc.. There is also a free paper on this subject, available from the Lustratus web store, available here.

I noticed my post generated a number of responses from various people. These were all on a theme which I found quite amusing. The accusation was that ESBs are NOT mature - but the posts then went on to demonstrate exactly what I was saying. Take, for example, the illustrious Jame McGovern's excellent blog on Enterprise Architecture. In his wrap-up of links for 2007-08-05. James discusses my post :

"Another industry analyst gets it twisted by stating: the ESB concept has matured, the functionality checklists are of less value - everyone ticks most of the boxes which holds true if you decide to ignore enterprise security considerations."

David failed to consider more carefully what I said about basic ESB functionality. My point was that now that ESBs in general can support the basic functions of an ESB - pass messages from component to component with mediation services available in-flight for enrichment etc, support key standards such as web services etc - that attention is turning to the functionality surrounding these basic operations. That is, the characteristics and all those functions to actually make the thing usable in a production environment. If David had gone on to read the Lustratus paper I referenced, he would have seen that security was classified as an honourary 'ability' and is indeed one of the types of functions that people are now demanding.

Other posters fell into the same trap. So, I think we are actually all in riotous agreement. I may not agree with the next bit of David's post about the need to support such standards as SPML, XACML and WS-Federation, but that is largely because I can never keep up with the stream of new standards, and as most people know who read this blog regularly, I have a highly cynical view to many standards - particularly to WS ones outside of the obvious and more mature ones, as I outlined in my post on WS-Madness.

Steve

Three best chance principles for building successful SOA programs

During this quieter summer period, I thought it would be worthwhile to restate some of principles that seem to improve the chance of success of a SOA program.  The usual health warning must apply:  There can be no one size fits all approach as SOA must adapt to fit each organization’s history and culture.  And with that caveat out of the way...

1. Identify the strategic benefits of the SOA program

SOA can sometimes be associated with the label of middleware and described as ‘plumbing’ with its associations of little added value but potentially high cost (although I have never quite understood this analogy as most people put good plumbing as a high priority in their homes!).  Compounding this problem, SOA programs can fall into the trap of offering only benefits perceived as intangible by non-IT senior management such as infrastructure improvement or enhanced agility.  Such benefits are not only hard to measure, they are harder to sustain through the inevitable ups and downs of an enterprise wide adoption of SOA.

To overcome this problem, it is essential that these longer term benefits are clearly associated with business benefits.  Otherwise, SOA programs will gather dust on the CIO’s desk or be condemned while partially implemented as a waste of time and money.   Some examples of these longer term benefits might be:

• Reduced cost of rolling out new business processes: Automating business processes is often a costly and time consuming process, particularly across organizational boundaries.  SOA eases this task by providing a single architecture spanning these boundaries and reducing the cost of assembling new business processes.
• Ability to track and react to regulatory changes:  Many organizations operate in industries which are forced to comply with multiple regulatory requirements which are moving towards real-time and evolving on an almost constant basis.  Regulation typically requires enhanced auditing of activity, control of process, and reporting – all of which can be addressed within the scope of SOA.

2. Leverage the pragmatic potential of SOA to deliver early benefits with reduced risk

SOA as an architectural rather than technology driven concept is capable of delivering individual projects on an as-needed basis within a consistent framework which ensures cohesiveness across projects over the longer term.  Some observers initially recommended an approach which is front loaded in terms of initial investment in expertise and design.  This is both unnecessary and unlikely to yield the full potential of SOA as a key premise underpinning SOA is that the organization is changing and therefore any initial design will be at risk of becoming out of date from the moment it is created.  Given that this is the case, target specific tactical challenges while taking cognizance of the fact that the solutions provided must be combined later into an enterprise-wide system. 

3. Make organizational and role changes to ensure long term success

It is possible to execute an individual SOA project within the existing organizational structures and role changes: the designated project team takes ownership for the SOA project and delivers this project as it would any other project.  However, this approach carries two major risks which has proven fatal to some SOA programs:

• Failure to develop expertise and control: Scaling any strategic program that spans many participating units such as SOA requires the expertise to be propagated across the organization as is the case with any strategic change program.  Furthermore in the case of SOA where services will be reused across project and organizational boundaries, it is essential that good practises and controls (collectively termed governance) are in place and maintained throughout the program.  This is best done through a designated expertise centre as part of an architecture function within IT.
• Failure to achieve service reuse: SOA success requires the reuse of services and the level of service reuse is a key metric of success.  High reuse rates can only be achieved if an individual is identified who is responsible within the business line or IT function (depending on the service in question) for driving the rate of reuse for that individual service.  In my opinion, this requires individuals to not only to be tasked but also incented to drive up reuse of their portfolio of services.
  

Ronan

Red Hat: The last ESB start-up?

As part of my work prior to updating of “Best of breed ESB” paper, I was recently briefed by Red Hat about their current ESB and SOA related projects which will become a supported ‘product’ called JBOSS Enterprise SOA Platform towards the end of the year.  This puts them rather late in the game when compared to both open source and closed source offerings.  Combined with the focus of the briefing on functionality over value, it felt very much like I was talking to the last ESB start-up to appear. 

As a characterisation this should not be taken a wholly negative:  For a company the size of Red Hat with its strong JBoss application server franchise, it would not have been surprising if they had simply done a me-too ESB to match that available from other stack vendors.  Instead they are putting together a much more interesting set of capabilities but lack some others that will required for most serious enterprise use.  On the positive, they seem to have a good grasp on the importance of data within SOA including both their MetaMatrix acquisition– once it has been open sourced - and pipeline based transformation (allowing complex manipulation of messages in-flight often required within deployed SOAs).  Counter-balancing this, there is still much to do around service life-cycle and in particular how to support service reuse.  I won’t attempt to go into much more detail at this time until the report is completed.

Ronan

3 rules for purchasing enterprise middleware

JP Morgenthal has come up with 3 rules when buying an enterprise application.  While I am not sure he necessarily excludes integration software/middleware, I thought I would have a go at drawing up three more (semi-tongue in cheek) rules.

JP’s rules are (shortened from their original form):

“1. There will always be a customization requirement: You're not purchasing Quicken. If you have special requirements for how that software will be used in your business, expect that some level of customization is going to be needed.
2. You're buying the company and its people: At the end of the day, most enterprise software applications can be made to do anything you want--that part is a matter of time and cost
3. Going with the leader is not always the best decision:  These companies’ offerings are, often times, bloated, cost more to customize and maintain and cost more to procure simply because they know they are the leader.”

My rules for integration software/middleware are:

1. There should always be a more bleeding edge alternative to your approach; otherwise you are too bleeding edge.  To put it another way, if you are thinking about an approach that is the newest thing on the block then you need to be sure that it does something pretty amazing for your organization to counter-balance the risk of being the proving ground for something so new.

2. The customization will be much greater than the software budget.  JP politely played this one way, way down.  Any enterprise software project spends much, much more money on implementation than on integration software licenses.  If you remember that many integration projects are part of a larger application project, the comparison between integration software versus total project cost escalates again.  Unfortunately, the relatively small size of the integration software line item in the budget does not reflect the reality that picking the wrong tool for the job here can cause the whole project to fail.

3. There are always many ways to skin the integration cat.  There have always been a number of different approaches to solving any middleware problem.  Just think of the regularly erupting REST versus Web Services debates or EDA versus SOA or even .Net versus J2EE.  Picking the correct cat skinning approach is far more complex as there are rarely clean answers. You must consider organization history, culture, legacy, appetite for risk, available skills etc etc.  This was very apparent on the recent Integration Consortium call on EDA and SOA as it was clear that EDA and SOA were often both needed in any given organization. 

Ronan

SOA for the scientific community - a practical example

I was intrigued to discover a discussion paper from the University of Southampton in the UK about how SOA is helping the scientific community. The paper, entitled 'A Collaborative Orthopaedic Research Environment', describes how SOA has been used to enable a Virtual Research Environment for orthopaedic researchers to collaborate in the design, analysis and dissemination of experiments.

What I found most interesting were the reasons for using SOA as the base architecture for this environment. The key objective, based on input from the specialists who will use the system, was to provide an easy way to share scientific data and results from collaborative research. However, it was deemed essential that the system could also evolve based on the changing requirements of the user community. One example of change that the paper gives is that of knowing which of a wide range of protocols will be followed for a particular clinical trial. Not only do these vary considerably, but it appears they are also susceptible to changes in regulations.

The solution decided to utilize a coarse level of services, essentially using just four - to manage the trial-related data, analyse it, submit and disseminate related research articles and support discussion forums. However, through the reliance on SOA, these services are flexible and extensible, making it much easier to address the changing needs of this particular scientific discipline. In addition, the services are reusable, and some could therefore be usable for other scientific areas.

It's good to see SOA being used to effectively address clear user needs in this way.

Steve