In recent years, as financial services firms have moved closer and closer to Straight Through Processing, there has been an increased focus on the concept of moving data as a message from application to application, or from firm to firm, rather than moving the data in files.  This increase in the use of messages has driven the need for flexible applications that can reformat and reroute these messages so that they can be easily read and understood by the appropriate systems.

• Messaging involves individual transactions that are sent from point to point as they occur.  The message is a single transaction that is sent by itself to another application for further processing, while a file typically contains many transactions. 
• Messaging is also associated with real-time processing, where the actual processing occurs as soon as the transaction is received.  The opposite of real-time processing is batch processing, where a file of transactions is held until the system is ready to process them. 

It is clear that an order for a trade must be sent and processed immediately, and therefore messaging and real-time processing are appropriate.  It is also clear that in today's environment we can easily process transactions such as dividends in an overnight batch, so not every transaction needs to be converted to real-time processing.

Most financial services firms have a technology architecture that evolved as users defined their changing needs, specific applications were acquired and as new technologies were developed.  These architectures consist of a mix of batch and real-time systems on various platforms.  Efficient and high quality processing increasingly requires the movement of data in the form of a message between these legacy applications, new applications, other firms and industry utilities. This increase in the use of messaging has increased the need for flexible applications that can reformat and reroute these messages so that they can be read by the appropriate systems.

In the past, programmers wrote specific code that generated the application's output in a specified format and told the network where to send it.  This worked fine when there were only a few applications and a few end points; however, as the number of both has increased, it has become more and more difficult to maintain the hard coded instructions, and the need for a flexible application increased.  The new category of application has been called Middleware, and is increasingly called Enterprise Application Integration (EAI). 

History of Connectivity

Most firms have several different applications and databases that have been connected over the years by a series of separate programs that reformat and move and the data between applications.

Figure 1.  Typical connections between applications

In most firms, applications have been added on an ad hoc basis over the years without a unifying architecture, and each addition resulted in a new set of isolated connection programs.  Making these connections is complex, since each program must overcome a variety of issues, including:

Batch vs. Real Time

Connecting the files produced and used by batch systems with the messages produced and used by real-time systems involves additional coding when connections are developed separately.

Data Availability

Frequently, firms find that the data they need to complete a message when they move from faxes and verbal communication to electronic messaging is not available in electronic form.

Technology

Data that is defined and processed in one system often must be transformed before it is used by another system.

Timing

If the data is produced by the source system before it is needed by the target system, it must be stored until it is needed.

The biggest problems in improving connectivity between applications are that the connections already exist and they work.  It is difficult to cost-justify replacing all of them when there is no immediate benefit.  And, whenever a new single connection is required, it is cheaper to hard code a single connection than to add middleware.

But, it becomes increasingly difficult to maintain these links, and adding or changing one application can be extremely difficult and time consuming.  As new technolgies and techniques have been developed, applications have been constructed and connected in many different ways.  And, since firms usually acquire applications over a period of time, most firms have a mix of platforms and even architectures.  Therefore, any connectivity solution must work with all of these existing approaches.

A typical firm might have evolved into a complex structure, as represented in Figure 2.

Figure 2.  Possible firm architecture

The solution to this connectivity problem has been the development of a new category of application, called Middleware

Middleware Overview

The Middleware layer of code sits between the processing applications and the network and performs three primary functions: reformatting, routing and protocol connectivity.  Middleware eliminates the need to hard code each of the connections between applications or firms. 

Figure 3.  Middleware Domain

Even with Middleware, firms must develop a way to extract the required information from one or more files.  The extracted information is then routed by the Middleware to a specific formatting routine that is linked to a protocol module that establishes the technical requirements for the data transfer.

Reformatting, which can include intelligence for transforming and enriching data, is the process of changing data in one format, such as SWIFT into another format, such as FIX.  It can include something as simple as changing the data in one file format directly to another format in a new file, or it can include complex transformations such as taking data from multiple files with different formats, enriching it with data from multiple files or tables that reside on multiple platforms, and creating a message in a totally different format.

Routing processes are needed to determine where specific messages should be sent, either internally or externally.  This is usually based upon some rules that evaluate the content or address of the file or message to determine where it should be sent.

In addition to routing and formatting, systems developers have had to contend with various telecommunications protocols, which describe the technical points of connectivity.  To provide true Middleware functionality, an application must also support the transfer of a message from one protocol to another while routing and formatting. This delivery process, which may include some of the process that is needed for transporting the data, can be included in the Middleware application or be in a separate program that works with the Middleware application.  For instance, a Middleware application can provide the routing intelligence while using a transport service such as IBM's MQ Series to perform the actual transport, or the transport control processes can be built into the Middleware application itself.  And, the distribution process can either be Conversational, Request and Reply, Publish and Subscribe or Store and Forward.

In summary, the intelligence embedded in a Middleware application can cover several different areas, including:

The use of Middleware will continue to expand as real-time processing and messaging become more prevalent, as developers realize that it is not economical to hard code every interface between applications and firms, and as the number of different points of internal and external connectivity continues to increase. 

Middleware Positioning

Middleware has been designed to support several different types of application architectures, as shown in the following chart.

Figure 4.  Application Architectural examples

And, as previously stated, firms may not have a single architecture.  As third-party applications have been acquired, or as firms merged, separate architectures may have been required, and a firm may now have multiple architectures running simultaneously.  Middleware must be able to support all of these architectures and provide a structured way to interconnect them.

To be successful, Middleware must support these architectures within three standard environments, which are the production, development and test environments.

To connect these applications, Middleware moves data between applications, files and data bases on a single operating system and works with the network transport layer to move information to other platforms or operating systems.

Figure 5.  Platform connectivity

More specifically, Middleware fits in the OSI (Open Systems Interconnection) model as shown in the following example.

Figure 6.  OSI Model and Middleware

Middleware supports the application, presentation and session layers, and may involve some portion of the transport layer.  The application, presentation and session layers can contain:

• Fourth Generation Languages (4GL)
• Object Request Brokers (ORB)
• Transaction Processing (TP) Monitors
• Remote Procedure Calls (RPC) such as DCE, T-RPC, and Netview
• Message Systems

The Middleware domain is always between the application and the operating system and network services, but the exact area covered is defined slightly differently in the OSI model and in the TSP/IP model.

Figure 7.  Middleware in OSI and TCP/IP

One of the reasons that it is difficult to precisely categorize Middleware is that its use and scope varies, depending upon the view-point of various IT professionals.  Data base administrators see SQL, developers concentrate on tools and languages and standards, and network administrators see and emphasize different aspects of what Middleware can and does do.

Developers may see Middleware as processing-based, while network managers will tend to see it as message-based.  And, the application can be used for intra-enterprise (A2A), inter-enterprise (B2B) and consumer oriented (B2C) businesses.

In the following sections, Middleware is examined in several categories, including Middleware Types, Communication Styles and Functions.

Types of Middleware

There are three different major types of Middleware: Message-Based, RPC-Based and Object-Based.

Message-Based Middleware (MOM)

Message-Based Middleware was initially created to enable developers to move data packaged as messages between independent applications on heterogeneous platforms and operating systems, and across disparate networks with a guarantee of delivery. To meet this objective, Middleware has the following high level characteristics:

• Available on multiple platforms (Hardware, Networks, Protocol Differences, Architectures, Operating Systems, Databases , and Other Application Services)
• Reliable data transfer
• Adaptable bandwidth
• Multiple communication structures
• Naming conventions
• Move data as transactions

The definition of Middleware expanded to include data transformation as more complex opportunities were defined.

Message-oriented Middleware exchanges data reliably and securely between applications on disparate platforms and networks, either through message queues or on a Message Bus.

With the queue-based process, there isn't any direct connection between the different applications, and the sender does not have to have any knowledge of the receiving applications.   These services are usually connectionless, are based on explicit send/receive verbs and require the application program to use more aspects of message buffer preparation and interpretation.

By accessing a communication bus that is common to several applications, Message-Oriented Middleware simplifies development since the programmer is not affected by the network protocols and does not have to make calls to operating system functions.

Characteristics of Message-Oriented Middleware (MOM)

The primary characteristics of Message-Oriented Middleware are:

• Messages can either be sent synchronously or asynchronously.  Synchronous messages require the application to send a message and receive a positive acknowledgement from the receiving application before it processes the next transaction.  By also allowing asynchronous transmissions, the Middleware frees the sending application from waiting for the response.  It can continue to process at its own speed.
• Message delivery is guaranteed since the Middleware accepts ownership for a message and controls its progress until it is complete.

MOM has been designed to operate on multiple platforms, such as UNIX, NT, IBM MVS, etc. This permits interoperability between applications that run on totally different platforms.

• Messages can be sent with different priority classifications so that the receiving system can process the transactions in the order that is important to it, rather than in the order that the messages were sent.
• Messages can be sent as broadcasts, where a sending application creates one message that is received by multiple systems.

Advantages and Disadvantages of Message-Based Middleware

Middleware provides several advantages:

• The guarantee of delivery removes a very large burden from the programmer.  Once the message has been accepted by the sending application, the sending application does not have to do anything further to ensure delivery.
• Middleware can be used at the level of traditional programming, or it can be used in an object oriented environment.  This provides a wide spectrum of programming opportunities and makes support simpler.

There are also some disadvantages with Middleware:

• There are no firm standards that all Middleware products must employ, therefore if a firm uses multiple Middleware vendors, gateways between the vendors will have to be constructed.  This is increasingly important as firms increase the level of integration between their front and back offices.
• Each Middleware product has been designed with some differences that make it difficult to select between the alternative vendors.  Some provide an extremely open environment that provides the programmer with a wide array of choices, and others provide a very simple user interface with limited programmer access.  Each extreme is valuable for certain circumstances.

Categories of Message-Oriented Middleware

There are three main types of MOM[1].

• The most common form of the messaging concept includes electronic mail systems[2] or general-purpose enterprise messaging services such as Microsoft's Exchange Server.
  

MOM message movers[3] are very different from enterprise messaging systems. While conversational, MOMs provide high-speed, generally connectionless transport with a non-blocking sender.

• MOM message-queuing Middleware[4] moves messages at high speeds and includes  message storing.

Remote Procedure Call (RPC) Middleware

Many DBMSs build RPC processing into their applications, so this form of Middleware may exist in different forms throughout the firm.

In the standard RPC, the sending application is blocked while it awaits the reply.  Some service providers are building in techniques to allow unblocked transactions.  RPCs are always synchronized.

RPC is sometimes used in a general sense to describe Request / Reply Middleware with basic data-type translation and connection-oriented communication services and can also refer to products that use an Interface Definition Language (IDL) to describe the argument lists for outgoing and incoming parameters.

Some products are relatively unbundled[5], some others are primarily database gateways[6] that incorporate RPC software, and some DBMS vendors[7] offer general-purpose RPCs.

RPCs can also be included with network operating systems[8] and other network computing environments[9].  RPCs, or RPC-like functionality, can be embedded in several different products, such as TP Monitors[10], development and run-time environments[11], multipurpose Middleware[12], and application packages[13].

Object Request Broker (ORB) Middleware

Object Request Broker Middleware[14] goes beyond Message-Based Middleware by connecting the applications at a higher level than at the data element level.  ORBs connect at the business logic level by using defined standards and connecting objects, such as customers, accounts and transactions.

The ORB approach works the best when a firm is establishing an entirely new architecture and is either acquiring several new applications or is building applications internally.

CORBA, Dcom and Java Beans are examples of the standards that can be used in an ORB environment.  CORBA tends to be used more often in the financial services sector since most firms have a variety of platforms, and do not reply exclusively on Microsoft solutions, which would be better served by Dcom.  Internet-based applications would generally use Java Beans.

Because most financial services firms already have an extensive inventory of systems, most are not pushing in the direction of ORB as an enterprise-wide solution, and the Middleware vendors are slow to add this functionality.  There is some tendency to use CORBA when individual applications are created internally.

Object request brokers with a CORBA IDL[15] are a category of Middleware that is useful for object-based, or component computing. Most ORBs are typically slow and are not yet mature tools.  They generally do not yet have the necessary security, management, and transaction management features, and do not have a proven track record for production applications.

As an improved version of an RPC, ORBs have the potential to reduce the complexity of heterogeneous, distributed computing. ORBs offer program-to-program communication and integration with a call-and-return process and an object-based discipline.   ORBs connect programs more dynamically than most traditional Middleware, but they are not yet ready to replace other forms of Middleware.

Middleware Communication Styles

Distributed function Middleware has four basic communication styles.

Conversational

The conversational approach is generally implemented through lower-level communication APIs [16], and since it is always synchronous both applications must be available at the same time for the transfer to take place. Generally non-blocking, the sender establishes a connection to the receiver, sends and receives messages, and explicitly terminates the electronic conversation.

Request/Reply

The Request / Reply approach is generally implemented through Remote Procedure Calls (RPC) services which move across a network to another system. An RPC in a program creates and sends a request to the receiver (server), which performs the desired function and returns the results. Generally blocked, RPC services can have features that enable unblocked senders.

RPCs are always synchronous, where the requester (client) maintains state information while the server does not, and the server is generally reusable by another requester.

Publish /Subscribe

Messaging involves a simple, one way transfer of information. Generally unblocked, the sender can continue to process other information after the message has been sent.  The message itself holds any of the required state information.

The message is passed along the network and if it is needed by an application, the application accepts the message.  The method directs a message to the intended recipient either by pushing data out to the network (publish) or by pulling it in from the network (subscribe).

Store and Forward

The store-and-forward approach, uses an intermediate place to store messages (message queues) while they are being transported. A queue is a file or database of messages that are awaiting delivery.

With queuing, the sending application sends the message to the Middleware application, which stores it in a queue that can be located anywhere on the network.  The delivery of the message can be delivered immediately or delayed for seconds or longer period of time.

Generally unblocked, message queuing is asynchronous and the message maintains state information for communication.

These four different communication styles are often used by businesses in different ways, as shown in the following chart.

Middleware Application Functions

In addition to the basic functions of formatting, routing and protocol converting, there are several different functions that can be provided by Middleware applications:

Compression

Large files can be automatically compressed in order to reduce transmission time.

Encryption

Messages and files can be automatically encrypted for security purposes.

Control services

Data can be automatically put into queues, messages can be bound, etc...

Application Programming Interfaces (API)

APIs and Adapters (format and gateway)

Administration facilities

Transaction management

Access security, volume and other statistics

Core services include logical name management through directories, distributed memory management, gateways to external systems and server-to-server communications.

This can also include discrete functions such as a message repository, audit trail, archive, analyses, etc.

Work flow

Workflow applications direct messages to and one of several recipients according to either conditions or policies that are codified in rules that are stored in the application.

Middleware that incorporates workflow functionality will be increasingly important in message brokering because of its ability to centralize processing flows and to interact directly with end users. The ability to control flow varies from a basic form where the application can only support statically defined, one-way, immediate message delivery to a more sophisticated form that supports rich, explicit flow control with sophisticated, rules-based processing.

Most of this category of Middleware[17] is used for application-to-application and has several very useful features, including:

• Ability to connect people to each other through features such as e-mail and electronic folders
• Ability define and manage complex workflows
• Ability to deal with unstructured information such as paper documents and images
• Ability to support interface standards such as WAPI, ODMA and MAPI-WF 

Workflow Middleware also has some weaknesses, including:

• Less ability to handle complex transformation
• Slower response times
• Slower throughput

Distributed transactions

While modern databases perform this function as a part of their own functionality, Middleware can offer the ability to manage simultaneous updates to multiple databases under transaction control.

Middleware Applications

Middleware is important in financial services to extend the life of the wide array of financial services that exist and to allow firms to link the various "best of breed" applications that firms need to acquire.  It also makes it possible for firms to move closer to Straight Through Processing, which will be essential as settlement periods are shortened around the world.

Middleware can support internal, as well as external connectivity.

Application to Application Middleware

Increasingly, firm need to easily connect internal applications in order to increase their ability to achieve straight through processing.  This connectivity can be accomplished by using message queuing and a message broker.  Once the data is in the queue managed by the message broker, it can be immediately forwarded or held until the appropriate time.

Firms can also send messages using a pure Publish and Subscribe methodology, which would involve sending the message from the message broker along a Message Bus, where the message would be "grabbed" by whichever application needs the information.

External Connectivity

In order to effective conduct business in the future, financial firms will have to increasingly communicate electronically with counter-parties, vendors and the infrastructure, globally.

This external communication requires that all of the industry participants be able to reciprocate, and that each firm prepare their internal capabilities.  The following graphic shows one method of moving messages via queues across platforms to a central message broker that enriches and reformat the message so that it can be send through SWIFT interface device to the SWIFT network, and back. [18]

Standards

The securities industry must seamlessly integrate applications that may operate on different platforms, using different languages or varying standards. The trend has four primary elements:

• Data repositories that contain document-type definitions as well as data definitions
• Message mapping business rules for complex transformations
• Extensible Markup Language (XML) as a standard
• Enabling technology like XML parsers and agents

XML

 As XML matures, vendors are developing tools that support the securities industry. The most widely use form of financial XML standard is FIXML.

FIXML

As an XML-derived language, FIXML messages can be validated by standard parsers and take advantage of the following points, according to FIX: [19]

Session and application layers that evolve independently

The separation of application and session layers creates opportunities to break away from the traditional methods of implementing FIX. Session and security levels can develop unconstrained without affecting application level messages. A new security model could be embraced or the session layer could be implemented using off-the-shelf technology.

• Improved communication
• A DTD-based protocol language (one based on XML) offers improved communication over written documentation alone.
• Improved evolution and extensibility
• A DTD-based language provides a better communication model for software and users.
• Structure and content validation
• DTD-based structural and content validation eliminates imprecision over current procedural-based software implementations.
• Evolutionary, not revolutionary
• XML offers an evolutionary model for future improvements to the FIX protocol, allowing the protocol to evolve naturally and in a controlled way.
• Reduce duplication of effort

XML is becoming widely used and well-respected in the software industry. Basing FIX on XML will provide commonality with other initiatives outside the FIX community and help to reduce duplication of effort in achieving mutual goals. In forums such as EDI, S.W.I.F.T., OFX, and ISITC, dictionaries are being built that describe the varieties of price, quantity, security, etc. XML allows these groups to share results and save effort by leveraging each other's work. Over time, this may also establish a basis for convergence and interoperability of these various message standards. The ultimate benefit will be a common data dictionary for the financial community that helps pave the way for straight through processing.

Evaluating Middleware

Selecting between Middleware alternatives requires knowledge in various areas since different processing applications have different features, and there are many subtle but critical differences in the various Middleware applications that are available. Middleware applications may be designed to support:

• Application-to-application communication
• Application-to-DBMS or
• DBMS-to-DBMS services
• External networks

When evaluating Middleware, the most important areas to consider are:

Primary criteria (Reliability, performance, management and scalability)
The number of:

• Users (concurrent and total)
• Files and data bases
• Programs

Availability (24 x 7)
Type of applications to be connected (batch, real-time, etc.)
Variety of protocols, platforms, operating systems, etc
Architectural tiers (two, three, etc.)
Volume of data to move (Current and future)
Timeliness requirements for data movement
Internal vs. external routing
Complexity of transformations
Technical capability of internal staff
Budget
Time frame to implement

Implementing Middleware

There are several strategies that firms may use to implement Middleware 

Layering with a Super-API

Some firms today see a Middleware application just as they do any other application, and do not wish to become dependent upon the vendor.  To avoid this, these firms create a higher level API that interfaces to the Middleware's APIs.

This maintains their independence, but can make it difficult to use the full functionality of the Middleware application and may introduce some level of runtime overhead

Service-Oriented Architecture

A service-oriented architecture requires applications to be designed as a collection of business functions, that are encapsulated and available to any qualified requester. These services are granular and allow developers to have significant flexibility when they design new applications

Message Queuing

Although Middleware is reaching the point where is can support synchronous cooperative processing, asynchronous Middleware using store-and-forward messaging in queues, is simple, intuitive and stable.

Message Brokers

Message brokers allow firms to develop an infrastructure that can integrate legacy systems with new and third party applications and offer other advantages, such as transforming messages and providing store and forward capabilities.

Summary

The need for application integration is more likely to increase in the future than to decrease, and while IT departments, on average are spending as much as 1/3 of their budget on integrating and maintaining the integration links, the need for a long term architectural solution will be in demand.