Max Dolgicer

Application integration requires an architecture, or blueprint, that an organization follows when implementing the technology.

Although common data stores would be ideal, businesses found that it was practical only for applications in the same or related systems. This is because the data in different systems is both technically and semantically different.

Data synchronization was an attempt to loosely integrate data by reconciling duplicate data versions. Although it allowed each application domain to operate with its own data model and DBMS, the dynamic nature of application development made the model impractical. In addition. the redundant version of data was often inconsistent or out of date.

Other architectures rely more on the ability of applications to communicate with each other without forcing application owners to agree on common data stores and common application interfaces. This is where application integration, with its underlying middleware products, comes in. The ability of one application to send requests to multiple applications and, when necessary, receive replies back from these applications requires communication middleware.

Communication middleware comes in many flavors – socket based communication, RPCs, messaging and so on. Because application integration often means asynchronous, time-independent communication, asynchronous messaging is becoming a de facto standard to facilitate application-to-application communication. However, in most instances, messaging only provides communication plumbing, a foundation for application integration.

Take IBM's MQSeries, for example. MQSeries has become the dominant infrastructure for many corporations, but it does not deliver the higher level of services needed to efficiently cross application boundaries. MQSeries is a sophisticated pipe capable of delivering messages in a reliable fashion. However, like any communication pipe, it is not application contentaware. In order for applications to "talk" to each other, they have to agree on application interfaces and the meaning of these interfaces.

Unfortunately, many applications have been developed or purchased without any specific architecture in place. For years, the traditional approach to application integration consisted of a patchwork of one-to-one interfaces between applications. However, this approach introduces maintenance complexities that increase exponentially with each application that joins the architecture. In addition, the extent of any "side effects" may not be known until a transaction fails.

To solve this problem, users began looking for a more practical approach to application integration. This is where message broker-based architecture "snaps in". Message brokers provide the means to mediate between the different application formats that applications "speak" the different invocation semantics that deal with how the applications talk to each other. They also furnish an approach to business process flow control outside traditional program logic.

Message Brokers: A History

The term "message broker" was coined by the Gartner Croup in early 1996. At that time, some very specialized approaches existed to integrate disparate applications so that they could exchange information automatically. The technology was mainly used in the healthcare industry (the HL7 standard) and the financial industry(SWIFT standard). There were also a variety of products on the market that provided some kind of application integration capability.

The original papers on message brokers tried to evolve a general concept based on a combination of selected functionality of message broker-like products, requirements gathered from end-user organizations and an analysis of the direction of the middleware market. At that time, the following definition for a message broker was stated: 'A Message broker is a logical hub that copies and resends messages to one or more destinations. It is an intelligent third party - a true broker - between information sources and information consumers so that communication is an independent, sharable function rather than something conducted between only two parties."

However, due to a lack of commercial products that cover the functionality of a message broker solution, the only choice was traditional communication middleware (MOM systems, CORBA Object Request Brokers and so on) that provided diverse communication services between application programs. Because application integration encompasses many more services than just plain program-to-program communication, many companies expend a significant effort in developing required services, thus developing their own homegrown message brokers. Because most of these services are generic by nature, they are not part of the application layer - not a business an end-user company wants to be in. In fact, some internal integration projects became commercial message broker products that are on the market today.

There is a substantial difference between product development and development of a customer-specific solution; therefore, message brokers that have been developed with an intent to be a product generally have a substantially better architecture. Nevertheless, the situation has changed as more commercial message broker products have emerged on the market. There are a number of products on the market today that call themselves message brokers, although in reality message brokers span many product categories that often have very little in common with each other. In fact, in many peoples' opinion, message broker is not a product, but a concept. Regardless of whether this is true or not, there is a basic set of services a message broker should perform.

Message brokers should enable program-to-program communication between disparate applications that always require some sort of communication middleware. First generation message broker products had their proprietary communication middleware built-in. However, most message broker vendors quickly realized that communication middleware is not the business they wanted to compete in, and today, most products support communication middleware with IBM's MQSeries at the top of the list. It should also be noted that some message broker products are based on the publish/subscribe model, while others support the message queuing model.

Support for a communication model has a direct impact on the architecture of the message broker itself. While some message brokers facilitate a hub-and-spoke application architecture in which the message queuing model of communication is the norm, others are based on the Message Bus model where publish/subscribe communication fits the best. Ideally message brokers should support both communication models and offer both architectures. It is also a mistake to assume that message brokers mandate real-time program-to-program interaction. Message brokers should support diverse modes of input, including bulk-oriented input media, as well as more interactive media such as messages. The same applies to support of diverse media output. Message brokers should be able to support interactive, as well as batch, processing.

Modes of Processing

Message brokers externalize (offload) some logic from application programs by providing additional value-added services to the basic communication capabilities. In fact, a message broker can be viewed as a logical hub that acts as an intelligent intermediary between application systems capable of hosting at least some of the services essential to application integration. Examples of such services are intelligent message transformation, message storage, and configuration and monitoring of applications using message brokers. Additional important services include support for different adapters that range from specific packaged applications such as SAP R/3, PeopleSoft, and Baan, to general purpose adapters such as the 3270 protocol, and databases such as DB2 and Oracle.

Instead of developing the rules that govern message transformation between two applications and placing them inside the respective application programs, message brokers allow externalization of this logic and place it in a hub controlled by a message broker. Because a substantial percent age of an application developers' time is spent developing transformation logic, the ability to enable such a transformation via GUI-based mapping or a scripting language inside the message broker is a giant step toward programless application integration.

Buy vs. Build: Message Broker Light

Message broker products differ in their capabilities, platform support, architecture, and industry and technology focus.

However, many have two items in common: complexity and price. This brings up the following question: Buy or build? In my experience, no commercial message broker product provides all of the required functionality. Therefore, the choice is actually buy and build vs. just build.

Does your company have few supported platforms, message transformation rules governing application protocols that are not too complex, and a need to watch expenses? If so, you might want to build a homegrown "Message Broker Light" on a top of purchased communication middleware. Message Broker Light supports rules-based message transformation; intelligent content-based routing of messages between applications, including one-to-one and one-to-many routing; message repository; and distributed workflow. Companies that embark on such an initiative could also build Message Broker Light incrementally. where different services expose well-defined interfaces and where each service is not tightly coupled with other services.

Conceptually, message brokering provides a layer of abstraction into which purchased or homegrown software may "plug in." it also resolves the problem of "black box" information by providing a framework for distributing, routing and determining the flow of the application's messages, as well as providing services to transform and secure the messages. By defining a space in which ROI can be easily justified and explained in enterprise application integration, message. brokers are strong catalysts for legitimizing middleware technologies.