This chapter will provide you with a general overview of the application integration process, and the benefits of BizTalk Server in regard to this process. We will start by looking at some common situations that require application integration, and identify some common problems that BizTalk Server can solve for us. We'll survey the major features and tools made available to us by BizTalk Server, and orient ourselves for the rest of our journey through this book.

When you have finished this chapter, you will have an understanding of the role and features of BizTalk Server. In particular, you will appreciate the following:

• The increasing need for application integration tools in corporate development
• The role that open standards for information interchange can play in application integration
• The features and scope of Microsoft BizTalk Server
• How BizTalk Server uses a data-driven mechanism for coordinating application integration
• The interprocess and internetworking protocols that are essential to application integration with BizTalk Server

Benefits of Enterprise Application Integration

Let's remind ourselves of why we should want to perform Enterprise Application Integration (EAI). Applications, in particular mission-critical applications that are used to run the enterprise, represent chunks of proven functionality. At the highest levels, they perform the functions that represent what the organization does in the business sense. They represent ordering supplies, scheduling manufacturing, fulfilling customer orders, and reconciling the finances of the company.

These business applications work in isolation in many companies. Since, in reality, the business processes do not occur in isolation, manual steps exist to integrate the applications. These steps introduce delay and add cost to the end product or service of the organization. Anything that can be done to replace manual integration with interprocess coordination is likely to speed up the company and reduce overhead. We want to go from the industrial revolution to Internet time.

Carrying this idea a step further, companies are themselves steps in a complete process. Few companies do everything to move a product from raw materials to finished goods. Companies operate in a web of business relationships, with one supplying the other with what they need. Here, as with processing within a single organization, companies mostly interact through manual processes. However, if an inventory application in one company can place an order directly with the order entry process of a trusted supplier, the entire process can be streamlined. Essentially, this is EAI carried out between companies, and it is often referred to as Business-to-Business e-commerce (B2B).

Finding a way to carry out EAI without having to rewrite mission-critical applications is the goal of this book. Microsoft BizTalk Server is a product that can perform this task for you, by coordinating the flow of message document files between applications. This is illustrated below:

Microsoft BizTalk Server is a product that performs application integration, using messaging.

Data from one application can be transmitted to BizTalk Server over some communications protocol, such as HTTP, and BizTalk can forward it to another application in another data format over another protocol. It does this based on internal configurations supplied by business analysts and programmers. It is a general-purpose server customized to a particular task through the provision of data configurations.

However, the presence of multiple data formats and communications protocols in such a system suggests some challenges that BizTalk Server must overcome before it can implement application integration. Let us turn now to take a look at those challenges.

hallenges of Enterprise Application Integration

In this section we are going to look at some of the inherent problems we encounter if we wish to integrate applications within a business process. First, let's compare and contrast a typical multi-application business system with the simple client-server model with which we are all familiar.

Simple Client-Server Model

If you have ever programmed a client-server system, you have actually performed application integration. Neither of the applications can fulfill the requirements of the system by itself; they must be able to exchange data. The two programs must be made to share data reliably. Without a predictable, efficient, and robust protocol for exchanging data, the client-server system cannot be made to work.

The simplicity of the client-server model produces some distinct advantages in comparison with the sort of application integration we will focus on in this book. Client-server has just two entities sending data back and forth. Although one entity - the server - can communicate with one or more clients concurrently, there are only two roles in the system. A party is either a client or a server. A single interprocess communication technology is embedded into both applications in the client-server model. The two programs are designed from the start around a protocol suited to the overall system. Typically, both applications are programmed by the same people, or at the very least, the two programming teams are in close communication. In short, the client and server applications were intended to work together, so application integration is relatively simple.

Distributed Business Systems

Contrast the client-server model with the problem of building a distributed business system involving multi-part data flow (known as workflow). Information comes into the system through some client interface, and one or more applications perform processing on the data. These programs may act directly on the data, or they may perform supporting functions as a side effect of the data passing through the system.

Often, many of the applications are in existence before the distributed system is implemented. In one way, this is a desirable situation, as we are then dealing with well-tested building blocks. However, integrating these applications poses several challenges:

• Incompatible protocols and data formats
• Workflow design and error handling
• Monitoring the workflow

Let's take a look at each of these potential problem areas.

Incompatible Protocols and Data Formats

First, we have the problem of incompatible protocols. If the applications were built independently, it is highly unlikely that they will use the same technology for interprocess communication. An application enabled for DCOM cannot speak directly to another application designed for HTTP. Some older applications may not be equipped for direct interprocess communication at all - they may expect to find a data file on a local disk.

We also encounter the problem of incompatible data formats. This may be a question of mismatched low-level data type representation between dissimilar computers, or a higher-level issue of mismatched data structures.

Mismatched data types may arise because one computer uses a different binary representation for some data type - numeric types are particularly troublesome - from another computer. Structures that are logically identical will be physically different at a low level of representation.

Dissimilar data structures involve two programming teams choosing different data structures to represent the same body of data. For example, one program might use a hierarchical structure well suited to programmatic use. Another might serialize its data in fixed length fields, a system oriented to saving data to disk or performing bulk data exchanges through files. Whatever the case, application integration usually needs some facility for translating between data formats. This facility must correctly map one item of data into another and back again if two programs are to communicate.

Workflow Design and Error Handling

Application integration is not complete when two programs are made to communicate with one another. We have to deal with data flow and error handling. Implementing a non-trivial process usually requires the connection of multiple applications in an appropriate sequence. The architect of such a system must also consider how the process can fail and provide for alternative paths through the system.

Consider a simple retail purchase at a web site:

1. The site searches and displays a catalog
2. The user selects purchase
3. The site accesses an inventory control system to confirm availability and mark the product for selection
4. An order price is calculated (this may involve shipping costs, sales taxes, and promotional discounts)
5. An e-mail acknowledgement is formatted and sent
6. A shipping invoice is generated so that warehouse fulfillment workers can complete and check the order

If you have the luxury of starting this business from scratch and have to write custom systems to support it you have little need for extra application integration. If, on the other hand, you are enabling web solutions for an existing business, you will probably have to coordinate the activities of several programs.

Also, the path I have just described is the path the process will take if everything works correctly. If the site is to be robust, though, the architect will have to consider various contingencies, such as:

• What if the product is back ordered, or perhaps discontinued?
• Is the product compatible with all forms of shipping?
• What happens if the acknowledgement is bounced at the receiving domain?

Different applications may have to be invoked in different sequences to account for all the contingencies that are possible in the system, as shown in the diagram below:

In other words, we need an integrated system that is flexible, and has built-in error-handling.

Monitoring the Workflow

Even when all the individual applications have been stitched together to implement the desired workflow, system administrators must be able to monitor the process during run-time, to manually verify the status of any particular item of work and intervene if necessary. They should also be able to adjust the parameters governing each step in the workflow. Considerations include:

• Should some steps be processed in batches, and if so, at what intervals?
• How long should one process wait for a reply from another before raising an error?

In other words, it is not enough to build an integrated system. We must be able to monitor and control it, as well.

Summarizing the Challenges of Enterprise Application Integration

The task of application integration poses several challenges. Efficiently overcoming them requires a set of tools and technologies. The challenges include:

• Defining one or more data formats for the exchange of data
• Defining the logical exchange of data between two applications
• Implementing the physical exchange of data, accounting for dissimilar protocols and asynchronous exchanges
• Defining ideal workflow processes
• Identifying error conditions and processing exceptions, and defining workflow to handle these cases
• Monitoring and operating the integrated system effectively and efficiently

These are the challenges that BizTalk Server is designed to overcome. When we discuss the various parts of the product later, you will see that there are different tools and services that address each of these areas. Before we get into that, however, let's look at some scenarios that use EAI so that you can see some ways in which these challenges arise.