LiteBytes from Lustratus Research

Morgan Stanley goes SOA

Morgan Stanley was recently talking about its major SOA investment spanning the last 18 months.  Implemented in its Global Wealth Management division , the justification for SOA was to give advisors and clients an integrated view of more accurate information, more quickly.  As with most SOA projects of this scale, SOA isn't the whole story.  Morgan Stanley had to first upgrade the network infrastructure to give its branch network the bandwidth required, then implement SOA on top and finally use ETL to convert the data flowing through their service bus into the dashboards the advisors look at (and a customer portal).

Interesting aspects of the coverage include:

• The programme was a strategic initiative to make this division competitive now and into the future.  Technology is simply the enabler to that goal.
• A best of breed approach was taken to the technology: IBM for the SOA layer, Informatica for the ETL.
• Web Services were used extensively (In the drive to distinguish between SOA and Web Services, it is easy to dismiss Web Services.  They can and often are part of the SOA technology stack)
• The project included major rewrites of existing applications - to consolidate where possible and also to update so that they could be plugged into the SOA framework.
• The common problem of changing the development culture away from write-it-all-ourselves was stressed.

Ronan

Is your legacy integration just a veneer?

Summer seems to be a time for reading through that huge pile of interesting articles and magazines that you can only find time to look at on vacation. On flicking through my own mountain of stuff, I came across the Q2 edition of Financial-i, a magazine targeted at the Financial Services industry. One point to jump out at me was a comment from Paul Joynt of Nordea, a Scandinavian finance house.  Paul was pointing out that SOA does not necessarily solve the problems associated with legacy integration.

The article, 'SOA - is it worth the effort', is available from the Financial-i site if you register, but Paul comments that covering a legacy system with a wrapper "so it looks like what you want" still leaves problems with the next level of change, because "it's only a veneer".

I think Paul has hit on an important point here. Different vendors in the SOA space have different approaches to addressing the problem of integrating legacy systems. Some will simply 'hand off' the request for legacy information to a tool from the legacy supplier - in the case of IBM mainframes this might mean using WebSphereMQ as the bridge, for instance. Others might approach the problem in some sort of screen-scraping or other interface simulation approach, where the legacy application is fooled into thinking it is running in its normal mode of operations. Yet more may generate code-based wrappers for each individual need, to be executed whenever a particular service is required.

To me, this all sounds too much like veneer in Paul's terms. Although this might address immediate needs, future changes will continue to generate substantial additional work and the generation of more and more 'special-case' code and wrappers.

Instead, the best of breed legacy integration solution should embrace SOA and integration rather than try to fool it with wrappers designed to seal off the legacy world from the outside. Legacy integration should be about making the legacy system a full and active participant in the service definition and execution. For example, orchestration should be possible both outside and within the legacy environment. Services should be built with full participation from both sides. By taking this approach, the best of breed legacy integration tool will ensure that future changes will become easier, quicker, cheaper and more reliable.

For more information on the whole subject of legacy integration, specifically in the case of mainframe systems, Lustratus offers a free paper on the subject.

Steve

Integrating the silos: Not only an IT problem

Talking about siloed application is such a cliche in IT infrastructure circles that we can sometimes forget one of the main reasons for siloed applications is siloed organisations.  Furthermore, siloed organisations are there for many good reasons - alternatives exist full of matrices and dotted lines but they do not suit every business, nor are without their own problems.  McKinsey Quaterly recently covered all of this in a report (you need to register for free to read it) which focuses on financial services and highlights the particular business drivers which are forcing greater cross-silo integration:

"Yet many [financial services] institutions find the going tough as clients demand—and increasingly expect—integrated services, including (among other things) advice, the raising of capital, and risk-management solutions."

There is no simple hierarchical organisation that can deliver this level of integration (and this is also noted).  Therefore, the only solution is to promote and encourage strong formal and informal networks - individuals cooperating across organisational boundaries to address customer requirements. I am sure that IT person reading this who is also thinking about SOA will find all of this very familiar.    And SOA clearly supports these human networks at an application level by enabling easy connectivity between the silos. 

In fact, most of the report is about how to map these networks and how to identify the good and prune the bad:

"In our experience, more networking isn’t always better—hence the inability of broad forums to add value by themselves. Indeed, one firm we know raised its productivity 25 percent by eliminating interactions that didn’t add value."

And clearly there is an IT dimension to this:

"One global investment bank, for example, has put a good deal of money and energy into a knowledge-management system that enables managers from different functions and product groups to access expertise and build on existing knowledge rather than reinventing the wheel."

While I am being a little unfair in picking on this one example in the paper, this point and a general emphasis on top-down enablement jars a little.  I am sure Web2.0 proponents would feel that this is starting at the wrong end:  The "right" end being make it much easier to create and develop networks by encouraging use of tools like wikis and blogs.  This is the right end because forcing managers into the role of informal network creators is clearly oxymoronic - and counter-productive in my own experience.  Furthermore in most cases, the level of informal networks within firms is low and random and therefore requires something more fundamental than 'getting to know the team across the corridor' drinks receptions. 

SOA itself also requires the same type of human networks of individuals cooperating to support service reuse and good governance.  Again, the wikis and blogs have an important role to play either in parallel to or embedded within registries and other governance tools - but that is a bigger topic for another occasion.

Ronan

The data deluge: new surf-board required?

Another report (this time from Tower Group) is highlighting the likely increase in data volumes that enterprises will need to handle over the next couple of years - a 900% increase in financial market data by 2012.  Of course, increasing amounts of data flowing around the enterprise is hardly new (in the mid 1990s, I remember being told in hushed tones at one of the baby bells that their databases were several terabytes in size).  Each time enterprises have been faced with such an increase, decisions have been made about what needs to be done:  Is the focus on integration, storage or analysis and can we use existing technologies or do we need/want new technologies? 

In the case of The Tower Group  report, the focus is on financial services and in particular the increase in market data handling requirements caused by new regulations (the EU’s MiFiD and US’s Reg NMS – both intended to create fairer markets).   Specifically, there is a regulatory requirement to store the data (in a manner which can be used as evidence if there is ever a compliance issue).  As these regulations relate to best price execution, this in itself will be onerous because every single published quote available when each trade is executed must be stored as well as all other relevant data associated with the trade.

Similar scenarios (where there is a flood of data or a potential one) exist in many industries – from the online games with millions of players generating massive amounts of data to security monitoring devices for government agencies to RFID tags.  However, it is too easy to react by announcing that the data deluge is coming and some new technological surfboard is needed. 

For a start, in many cases the data may have little potential value and storage may not even be needed.  But of course this will not always be the case.  To quote from the Bob’s guide coverage of the press release:

“Regulatory compliance will pave the way for firms to completely automate the trading process,” added Price. “Once all the players have done so, the winner in the hunt for liquidity will be whoever can process the data the fastest.”

To put it another way, in the case of market data there may be business justification for doing more than storage for forensic reasons either in the realm of integration or analysis of the data streams.   In my other examples there will also be opportunities – again in the realm of integration or analysis. 
The second hurdle is whether new technology is needed. 

Simply saying that there will be an increase in data and a need to do something with it is not enough to justify new technology as old technologies may continue to scale.  Going back to my baby bell experience, this was around the time when Object databases were being touted as the relational db killer – as only Object databases could possibly handle that size of database. 

In the case of the “data deluge”, the real-time processing of streams (when the value of the analysis is high enough) is certainly one area where a new approach seems needed – and is an area targeted by the likes of StreamBase and Progress' Apama products.  For analysing the increasing massive data warehouses and for integration, I think the jury is still out – although the increasing complexity within the data may throw a spanner in the works.

Ronan   

Examples of SOA successes in smaller banks

Two stories that demonstrate success with SOA in smaller financial service organisations show how it can be used to support very different business objectives.  Obviously, SOA is not a certain or easy win and require a degree of organisational and technological change to be successful.  However, it is nice to see some success with SOA emerging in organisations apart from the usual suspects of Wachovia or HSBC.

The first is EBS, a firm with 10% of the Irish home loan market that wanted to automate its relationship with its network of brokers.   EBS recognised that its mortgage products were actually made up of a set of different products (core loan, home insurance, life policy, legals etc), each of which potentially required separate systems to complete processing.  The final SOA system is able to integrate across all of these systems and across to the brokers.  The biggest change/benefit highlighted by David Yeates, the senior manager of IT architecture at EBS, is actually the mythical alignment of business with IT – although he says it a little differently:

“A huge amount of business knowledge exists in the IT department, but that tacit knowledge remained there,” Yeates says of the old way of working. “It [SOA] turns the IT department on its head: suddenly, from being in the back room, you have to be at the front. The issues for IT departments are absolutely enormous. IT isn’t a thing that’s holding people back anymore.”

Yeates also mentions another benefit of their SOA deployment – not only does it reduce time to market for new products, it also allow them to white label other providers’ products.  Which neatly leads onto the second example of SOA success: Webster Bank,  profiled by IBS Publishing.  Webster came to SOA from a very different direction:  It wanted to move away from its own core banking systems – which were under strain – and outsource to use Fidelity’s systems. 

The CTO, John Kershner states that the decision was whether to spend ‘x million dollars’ on point-to-point interfaces that would be difficult to support and maintain or take that money and invest it in new technology, to create an SOA infrastructure with the associated lower maintenance costs. 

Again, the article claims that Webster has had pretty much unlimited success with the project.  Reflecting what is a common experience, Kershner highlights the biggest challenge as managing and coordinating as many as 30 different project teams over 15 to 18 months.

Ronan

Complex Event Processing (CEP) and SOA: Sitting side by side in finance

A perennial SOA discussion is whether it is a universal architecture or whether there will be complementary or even competing architectures.  This article in Water’s Online focuses on this issue in the context of where SOA is being used and more importantly not being used.  While the tone of the article is probably negative to SOA, it is interesting to notice how established SOA is in the large investment banks with quotes featured from both HSBC and Bear Stearns.

To briefly cover the argument:  SOA has obvious overheads if messages are encoded in XML and used synchronously.  This makes it unusable in some of the trading and market data applications where the volumes of data are huge and speed is everything.  In this space, Complex Event Processing (CEP) tools such as those from Progress Apama and Streambase are clearly more suitable than SOA based on business processes and synchronous processing models.  To quote a well balanced view from a senior IT person at Bear Stearns:

"The more glamorous parts of the trading system typically involve super high-performance networks, data-distribution models, and technologies such as CEP that are 'in line' with information coming from and going to the markets," says Moschetti [chief architecture officer at Bear Stearns]. "But other aspects of the system, such as entitlements, configuration, loading of baskets, lookups on reference data, and so on, can use what the industry conceives as SOA—that is, Web services, XML over message busses, and so on," he adds.

Therefore, it isn’t a matter of one approach dominating the other is some strange Darwinian struggle.  For industries where the processing requirements include real time streams (and in particular where the stream volume and complexity exceeds what ‘static’ approaches based on variations on data warehousing can handle), CEP will have a bright future.  That means at the moment: front office in financial services, some government security applications and Massive Multiplayer Online Gaming (as shown by Streambase’s recent announcement).   Apart from that CEP has an important role to play in control and monitoring within the SOA layer as Steve has previously blogged about.

Ronan

Notes from the bleeding edge: Data models, Financial derivatives and the future of SOA.

Golden Source, a vendor of enterprise data management solutions for the financial services industry, has just released a report  highlighting the major problems associated with the management of data connected with some classes of financial instruments (those known as Over The Counter or OTC derivatives).   In particular, it focuses on the strain placed on the existing data models by the need to roll-out new types of financial products ever more quickly.  The relevance to SOA and enterprise integration is two-fold:

• Data models are core to SOA, whether defined implicitly within services or explicitly as part of a separate data modelling exercise.  As the scope of SOA grows within an organisation, the challenge is to balance the need to create global models (and hence reduce the need to translate between models) and the need for different data models reflecting real differences between business units (and hence provide powerful transformation capabilities in the infrastructure).  In the case of business-to-business integration, as is the case with financial trading, there is a need for an agreed data model which is used to exchange messages between the two parties.  FpML is a leading example of such a B2B format which addresses derivatives in particular.

• As the level of integration increases, there are more messages flowing through the network and more complexity in data models and hence more complexity in the transformation between data models.  An objective of SOA is to deliver agility – to allow business to change unhampered by the traditionally slow moving integration capability.  Therefore as SOA grows, the data modelling and transformation challenges move centre stage.

The market in complex financial derivatives (bundled combinations of multiple types of financial instrument designed to have specific risk and return profiles) shows a potential future for SOA in terms of data complexity and rate of change.  However, it should be noted that it is unusual in a number of ways: as with all financial instruments, they are entirely virtual; with a fast moving and growing market, advantage is firmly with the organisation that can create and roll-out a new type of derivative and they are highly complex in their structure.  Nevertheless, the challenges faced in this problem domain should be indicative of the challenges that will be faced the most advanced SOA adopters.  The key findings of the Golden Source commissioned report were:

• Most of the manual data management effort goes into fixing the 1%-2% problem cases.   In the SOA context, this statistic should focus the mind and justify the business case for prioritising data modelling in any wide-scale SOA programme.
• Flexibility is essential in creating data models capable of changing as business changes and markets become more complex.  Data modelling is not about making a perfect representation of the business frozen in time – it is about making something sufficient for today which is built to change and extend as the business changes.
• While there is a clear need for industry wide collaboration in defining base models, the report notes scepticism that it will actually happen due to lack of incentives for each institution to share information.  While there are huge economic drivers towards standardisation within financial services, the natural suspicion and lack of trust between competitors has meant that the significant progress towards standards has taken a long time and has been painful process.  Therefore, it is reasonable to assume that most other industries will take a lot longer.  In the SOA context, this means that organisations should not wait for standards to emerge but rather focus back onto addressing their own needs and build data models in such a way to ensure easy integration to other organisation’s models. 

Ronan

Is consolidation impacting on SOA innovation? Banking consolidation that is!

There has been a steady stream of SOA related start-ups being acquired over the last two years – most recently LogicBlaze by IONA.  Some commentators are beginning to suggest that SOA is fundamentally unsuitable territory for start-ups.  I agree to a degree that SOA start-ups have a harder job today than their enterprise infrastructure software ancestors of 10 years ago – but I think it is less about SOA and more about changes in a key industry that used to support such start-ups: financial services. 

To start with the argument raised by some against SOA start-ups such as Randy Heffner of Forrester who was recently quoted in SearchWebServices as saying: 

“a proliferation of small companies doing innovation it works against the goal of having a coherent platform with broad and rich functionality."

And

"Innovation is one major value point that has to be balanced against the need and requirements of building a coherent software infrastructure platform for the next generation of applications,"

I am far from convinced of this argument on two counts:  if SOA is going to be truly a enterprise wide architecture it must span technology stacks and hence a single coherent software infrastructure platform may be hard to achieve and secondly SOA by its nature should ease the plug-and-play of products from different vendor into it.  Of course, as Tony Baer of onStrategies points out in the same article, customers like ‘one throat to choke’ but they also like the solution least likely to result in a throat choking requirement.

Which begs the question why aren’t SOA start-ups succeeding at the rate one might have expected given the popularity of SOA.  I suspect one contributory factor may be the continuing consolidation among the global investment banks based in Wall Street and the City of London.  This was the traditional birthing ground for so many software infrastructure products and vendors.  This consolidation means that there is much less room for SOA start-ups to build before breaking out into other industries.  From my own personal experience of being involved with companies selling into the investment banks from the mid-nineties on, I can say that the task has got harder but more importantly the list of possible targets has got much much shorter. 

Some examples: eleven years ago JP Morgan Chase Bank One was 6 banks (JP Morgan, Chase Manhatten, Chemical Bank, Banc One of Ohio and First Chicago NBD) of which at least 3 would have been on the target list of any start-up; Deutsche Bank took over Banker’s Trust in the late nineties; Union Bank of Switzerland merged with Swiss Bank Corporation to form UBS in 1998, in 2006 Mellon and Bank of New York agreed to merge and today JP Morgan is rumoured to be considering an offer for Barclays which is in turn attempting to merge with ABN AMRO – taking at least one more target off the list. 

Of course, it can be argued that telecoms (another popular starting point for infrastructure software) has recovered in the last couple of years and other verticals could also pick up the slack with government in particular now a bigger buyer of integration products  That is certainly true but government is often even harder and slower for a start-up to sell into than banking.  Having managed to sell into both when I was running PolarLake, I am a good position to compare the scars!

Ronan