Understanding BizTalk message flow
When a message is received by BizTalk, it is received via an Adapter; always. Adapters are the endpoints of any BizTalk system. They are the points of contact to the outside world; the gateways if you will, into the realm that is BizTalk. These endpoints could be File, HTTP, DB2, or any of the dozens of transports supported by BizTalk out of the box.
Upon arrival, all messages go to the Message Box, but to get there they must follow a specific sequence. The following is the list of actors involved in receiving a message listed in the order in which they process the message:
- Adapter.
- Pipeline: chain of Pipeline Components.
- Map.
- Message Box.
This flow is depicted in the following figure:
Before we discuss the Message Box itself, it is worth describing a little more of what these components do. The message is clearly what we are interested in receiving. The Adapter is the communication endpoint that does the physical receiving of the bytes. Pipelines process the stream of data received, so that it can be made more usable for later components. The pipeline contains components within it that can provide services like decoding (such as decryption), disassembly, validation, and party resolution (determining who sent the message).
This is almost like an international arrival when flying. The adapter is the airplane, the pipeline is the way you get to customs, with components of jet bridge and hallway, the signs and queues are the map, and the customs booth is the Message Box. The customs agent really doesn't care if you flew in a Boeing 777 or an Airbus 330, or even how you got to the booth; they care about letting you into the country.
As this chain of processing completes, the message is mapped and streamed into the Message Box.
The Message Box
The Message Box has been the central construct behind every version of BizTalk Server since 2004. The Message Box is the place where everything in BizTalk happens. Messages that are received from adapters are put into the Message Box. Messages sent out of BizTalk are retrieved from the Message Box in a pull fashion by send ports and orchestrations. These sent messages are pulled by their subscribers. Understanding this complex series of steps is very useful, but you can also skip this section for now and come back later, if you prefer.
The Message Box is based on the concept of queues. In fact, the first versions of BizTalk used MSMQ internally to organize their work queues. MSMQ is really a fantastic technology, but BizTalk's solution to queuing is far more scalable and appropriate for the type of work BizTalk performs.
Each queue consists of several tables that are as follows:
- Queue
- Suspended queue
- Scheduled queue
- Message Ref Count Log
- Dequeued batched
In a nod towards our future discussions, these tables exist for each Host in the BizTalk group; the significance of which will be addressed later in this chapter.
When a message arrives at the Message Box—during insert to the actual database—the Subscription Manager checks the subscription list to determine subscribers for that message. Then each subscriber gets a queued "work item" inserted into its queue. If there are many they each get a work item, but the message itself is not actually duplicated, they simply get a reference to the original message from which to perform their work. This reduces duplication and increases performance. It all works because messages are immutable in BizTalk. That is, you cannot change a message once it has been received. Even at times when it may look like you are changing an existing message, you are not; you're creating a new message.
Each subscriber then retrieves its queued work items, marks them as retrieved, and processes them. When they are completed (be they an orchestration or a send port) they are marked as completed and the reference counts are adjusted to reflect this. Other maintenance tasks perform the necessary clean-up later on.
If no subscribers are found, the message is still inserted into the Message Box, but as a failed message, which is a special type of message that simply stays in the Message Box.
All of this may sound a little complicated, and to some extent it is, but it is the key to BizTalk's scalability. As users of BizTalk, and I mean this as developers or administrators, we are completely unaware of all this going on. It simply works. It is an infrastructure provided to allow us to focus on delivering real value in our solutions. The Message Box construct is the core of BizTalk that makes all of this possible.
Other BizTalk databases
Beyond the Message Box there are quite a few databases involved in BizTalk. They each fill a specific role in the platform to enable it to be the scalable, distributed product that it is. The BizTalk databases are briefly introduced as follows:
- BizTalkMgmtDb: This is the management database that holds all artifacts that are part of a BizTalk solution. Ports, Maps, Schemas, Orchestrations; they all go into this database. Although some are actually loaded from the Global Assembly Cache (GAC: the .NET component registry), this is where BizTalk knows what to load from the GAC. If the Message Box is the heart of BizTalk, the management database is the brain to a large extent. The management database is also where runtime information and statistics are kept to allow BizTalk to self-tune or throttle itself.
- BizTalkRuleEngineDb: This database is for the Business Rules Engine, which stores and manages policies as well as vocabularies.
- BizTalkDTADb: This is the tracking database that records all events which take place within BizTalk.
- BAMPrimaryImport: This database stores all initial Business Activity Monitoring (BAM) data. BAM data arrives here from the Message Box and can also be offloaded to the BAM Archive database later on, in order to further reduce the burden on system runtime.
- BAMArchive: This database is an exact mirror, structurally, of the BAM Primary Import database. This database is used to offload Business tracking data that is outside the lifetime of online requirement, but is not yet able to be deleted. Custom reports, or even the BAM Portal, can be pointed at this data source to allow indefinite storage.
- BAMStarSchema: This database is used for the analysis services feature of BAM.
- SSODB: This is the Single Sign on Service database that holds configuration information for BizTalk groups.
- BAMAlertsNSMain: This database contains information on how notification services connect to BAM, such as the protocols used for notification delivery and the version of the BizTalk database.
- BAMAlertsApplication: This database is the real body of notifications. It stores subscription parameters as well as records of all notification deliveries.
Add to these the Message Box, which we've already covered, UDDI and the ESB databases and you are looking at almost a dozen databases that are part of the core BizTalk product. This may sound excessive if you come from a background of single large databases, but it makes perfect sense and is part of why BizTalk can scale so well. As a system administrator you're free and even encouraged to move these databases to different servers and instances. If your applications are tracking heavy, you can move all the BAM databases to their own servers so as not to interfere with live transactions.