Monitoring
Monitoring is a critical part of all software solutions, but even more so with Middleware, Integration, and Service Orientation. These types of solutions inherently do not have a user interface so knowing 'what is happening' becomes critical. The ability to view data and transactions as they happen becomes important both from a technical, as well as from a business operations standpoint. No solution is complete without monitoring. I repeat that point, no solution is complete without monitoring capability. Monitoring can generally fall into two types: technical or platform, which would be covered by tools like SCOM and commonly referred to as tracking and business or solution monitoring.
Unfortunately most solutions don't have good monitoring or instrumentation built into them in the first place. This, however, is one place where BizTalk Server really shines. The entire solution stack, all the way through the adapters and message box, support advanced monitoring centered on BAM—Business Activity Monitoring. In most cases, BAM can be added to a solution after it is completed without requiring code changes. This is actually the preferred method as monitoring is rarely an upfront requirement; though it certainly should be.
Why BAM?
BAM is a powerful set of tools that, when used properly, creates the infrastructure we need to gather, analyze, and present data about our business solutions without writing a single line of code. This can even include complex statistics and aggregations. The BAM tools in BizTalk will create the infrastructure needed to provide monitoring. This includes tables that BizTalk will then automatically populate with the tracked data analysis cubes used for more complex derivatives, and also maintenance of SSIS packages, to age or archive that data and to process analysis cubes. These are all designed to scale and built with a care and detail it would be very difficult to replicate on our own. That said, like most of BizTalk, they also represent a good model to be inspired by, if the need ever arises. In fact, BAM can be used without BizTalk at all and any WCF service can have BAM added without recompiling the service.
Importantly, although BAM has a lot of similarities to data warehousing, it is not a data warehouse and not meant to replace your data warehouse. Data warehouses are created for exploring your data. Generally speaking, in BAM you aren't exploring that deeply although you can to a certain extent. BAM is, however, a great way to feed your data warehouse and it is a great way to provide both business monitoring, as the name implies, as well as operational monitoring.
BizTalk also has its own tracking information in the DTA tracking database and this can be good for emergency operational tracking or viewing some information about running transactions. This data is really meant for BizTalk, however, and generally this type of tracking should not be relied upon as it encourages inappropriately tight coupling. A term I have often used, resulting in many giggles, is inappropriate intimacy. This data is also purged periodically and requires administrator permission to BizTalk (or at least operator) just to be viewed. It is also prone to bloat in even moderate volume environments if not purged often, which it is critically important to do. For this reason, BAM should be used for all tracking and monitoring wherever possible.
Understanding BAM concepts
There are several ways to work with BAM, including a .NET API, but I believe the best way is to keep it simple and as originally intended by the tools. I believe this approach results in a more loosely coupled solution than other methods such as the BAM API. Coupling is a slippery slope and it is something we must struggle to avoid all the time. There certainly are times when the BAM API is needed, but I've run into only two in all my time with BizTalk.
The best way to avoid coupling and produce excellent tracking is via the Excel Plug-In and the Tracking Profile Editor (primarily the TPE). This is covered in great detail in the second half of this text in the context of our development storyline, but I will briefly present the concepts here. Please refer to the appropriate sections for a detailed walk through of each concept.
Instrumenting a BizTalk solution is done in three steps: creating the structure of what you want to track (an activity), creating the way you want to present the data from this activity (a view), and binding the activity to the solution via a profile.
Creating a BAM activity
A BAM activity defines the base data that we wish to track. This is broken down into milestones and business data. Milestones are timestamps, points in time when events happen, such as a message received or sent. Business data can be any text, integer, or decimal that is in a message. Although something like Order Date in a received message is a date, it is not a milestone, it is actually business data; whereas Order Received would be a milestone representing when BizTalk actually received the order message.
You should track all pertinent information that you will need to report against in BAM, but do keep in mind that there is such a thing as too much information. BizTalk will handle this fine and the SSIS packages that are created to manage the data will help keep the database bloating to a minimum, but you should really track only the data you're actually interested in. The activity represents the database structure that will be created to hold our tracking data.
Creating a BAM view
Creating a BAM view allows us to select which activity data we would like to present in a display. We can combine data from multiple activities or display only some of the tracked fields. We may want to do this because IT operations staff may want to know things like the number of orders or time to process, while business operations staff may be more interested in the total dollar amounts of orders (that is, teams will be interested in different metrics and creating different views is a great way to provide each with what they need).
In addition to selecting what we want to see we can create durations between milestones, groups that allow any one in a set of milestones to represent a logical milestone (such as when a branch can be taken, but both sides result in a "complete" status), and create aliases so that specific user groups can see different labels for the same data. We can also make logical progression markers out of milestones so that we can assign labels to each milestone. This would be useful if we wanted to show the current progress of a long-running process that has multiple milestones.
Perhaps most extraordinarily, we can also create dimensions and measures with which to drive deeper analysis. These result in multidimensional analysis cubes from our data, for which the BAM tools will then create the infrastructure. All of this will be covered in detail later. Measures are data aggregations such as count, sum, average, maximum, and minimum values, while dimensions can be time, data, ranges, and the aforementioned progress. Taken together, these allow us to provide our users with complex and expressive views into business transaction data.
Creating a BAM tracking profile
The tracking profile is the glue that binds an activity (and its views) to a BizTalk solution. These are created in the Tracking Profile Editor (or implemented via the API, which I've already explained I believe is best to avoid). The activity and views define what we want to track, and the profile defines how we want to track it; that is to say at which points in our solution we want to connect data passing through BizTalk to our tracking profile. The TPE is a simple graphical tool that allows us to bind these components together. The complete figure if the BAM activity and profile in action is displayed before.
In this diagram, we can see how the profile ties the running solution to our activity, which represents the tables into which the data will flow.
Advanced BAM concepts
Although all of these are covered in detail this is an explanation of some of the more critical, advanced BAM concepts covered in the second half of this book.
Continuation
A continuation is a way to connect disparate parts of a business process into a single tracking record. Suppose a business process involved receiving a purchase order that is sent to an ERP system, then receiving the order confirmation from the ERP later on, asynchronously. This solution involves two disparate steps, but they are both part of the same business process—processing a sales order. A BAM continuation would allow us to connect these two by the use of a common information token, like an order number.
Subtly and importantly a continuation is expected to continue at some point, so they should be designed with care.
Relationship
A relationship in BAM allows two different activities to be linked together. Suppose in addition to the previously mentioned "purchase order received" that we also have "purchase order cancellations" sent via a different mechanism. These are two different business processes, a sale and a cancellation, so they should be two different activities. They are, however, related because the cancellation is of a previous sale. This is accomplished via a BAM relationship that also uses a specific piece of data, like the continuation though in a different way, to perform its duty.
Document reference URL
The document reference URL is perhaps not an advanced concept, but it is useful and allows us to add links to file shares or better still to SharePoint to link documents into our monitoring profile. If these purchase orders were submitted via an InfoPath form a link to the archived original form within SharePoint could be provided for fast and easy reference.