Visual Studio solution structure
Now we'll see how to use Visual Studio to implement the structure that has just been introduced. These suggestions are based on both my experiences as well as that of others within the BizTalk community. I always use this basic template presented in the following image for every solution and it has served me well for years.
First always start with a blank Visual Studio solution, as this will allow you to name the solution in a way that represents the functional area, the solution addresses, and the projects within the solution in a manner that reflects their purpose within that solution; such as Maps, ExternalSchemas, and so on. It is important to note that within the solution, the projects can have short names like Maps instead of MyIntegration.Maps. The name of the solution in Visual Studio is unrelated to the names of the assemblies or namespaces. Once you have your blank solution you can add new projects to it by right clicking the solution and selecting Add Project.
Generally, I like to break my solutions into the six projects as shown in the following screenshot. I find that this meets the needs of most solutions, but it is not a hard rule. Some solutions will have more, others less, but they will all be a similar variation of this concept. Each project is explained below. Importantly, only start with the projects you need and simply add others as the need arises. A Visual Studio template can be found here: http://biztalk2010patterns.com/documents/templates/solutionstructure and can be used to generate solutions with all these base projects already created, complete with references.
Projects
The following are the typical projects in a BizTalk Visual Studio solution and a brief explanation of the role that they serve.
External Schemas (.xsd files)
This project contains all of the schemas that are sent or received by the BizTalk solution. Port-level mapping, a later recommendation, ensures no dependencies are leaked into or out of the BizTalk solution. This project would include schemas generated by adapter wizards, SOAP, or WCF references, flat file formats that we send or receive, and EDI schemas. It is vital that these be treated as what they really are: external artifacts (that is external dependencies). Even if you own the source system, don't assume that the rate of change between two systems will be equal. You may end up with schemas that closely match internal schemas, but the flexibility you gain is certainly worth the very small cost. This project contains no references to other projects in the solution as it represents the endpoint or edge of our solution.
Internal Schemas (.xsd files)
The InternalSchemas project contains all the schemas used internally by a BizTalk solution. These are schemas that are never exposed to any other systems outside of BizTalk and define entities within the actual solution. Every external schema should have a corresponding internal schema or translate to part of a composite internal schema. These schemas are commonly referred to as canonical schemas. This project also contains no references to other projects in the solution. It represents messages that will be used internally by orchestrations, business rules, direct subscription routing, and so on. Property schemas would also exist here.
Maps (.btm files)
The Maps project contains all maps within the solution that translate internal to external schemas. It references the InternalSchemas and ExternalSchemas projects and nothing else (with the exception of custom functoid assemblies). This project is the guardian that prevents external dependencies from permeating a BizTalk solution. Critically, every external schema should have a map translating either to or from internal types as needed. Again, this ensures the flexibility that is so vital to successful distributed systems. The orchestrations assembly should never reference the maps assembly as it would require the orchestration package to now have an indirect reference to the external schemas project, and therefore bypass this carefully constructed separation we are trying to enforce. It is true that occasionally you need a map in an orchestration, but that map should exist in the orchestrations assembly or in an internal assembly of its own, perhaps named InternalMaps, not in the maps assembly, as this would violate our separation of concerns.
Pipelines (.btp files)
All pipeline components (assemblers and disassemblers) are grouped in this project to make testing and maintenance easier. This project should reference the ExternalSchemas project or the InternalSchemas project as needed. If you want to provide a very robust isolation, you could create internal and external pipeline projects, but I really only recommend this if you need to call a pipeline from within an orchestration to avoid leaking a dependency into your orchestrations layer. The Pipelines project will contain BizTalk pipeline files (.btp) and also reference any assemblies that contain custom pipeline components if they exist. Pipelines are perhaps the most underutilized and most misunderstood part of BizTalk.
Many useful and complex solutions can be created in BizTalk using only pipelines and adapters.
Pipeline components (.cs files)
Not all solutions will need a Pipeline Components project, but when they do, pipeline components, which are .NET classes, should be in their own project to continue our isolation paradigm. If the pipeline component being used is commonly reused throughout your enterprise, you can just put the component in third-party assemblies (covered shortly) and have a dedicated solution just for the pipeline component. Pipeline components should be flexible and reusable and thus be in their own solutions. This can also avoid assembly locking problems in Visual Studio.
Orchestrations (.odx files)
This project contains all orchestrations used in the solution. It references the internal schemas project and possibly the pipelines project (if you're using pipelines from within an orchestration). As depicted earlier, orchestrations are the top layer of the BizTalk solution layer model. They should generally be unaware of any outside systems or artifacts. Failure to preserve this isolation principle results in tighter coupling creeping into your solution. Even when you call a web service from an orchestration, you're not actually calling the service from an orchestration. The message box functions as an intermediary. Care must be taken not to tie your orchestrations to the specific endpoints that you will eventually connect them to.
Libraries (C#, resources, and so on)
Any custom classes or utilities that are used by your solution should also be broken into their own projects. It is important to not let these bleed dependencies into your solution. If you have some custom components to do processing in an orchestration and some for custom functoids or pipeline components these should be broken into their own distinct projects. Isolating functionality into separate assemblies gives us flexibility to make changes without having side effects on other parts of the solution. It will also simplify our versioning and patching options.
Testing (.xml, .dtd, .cs files)
This project hosts all unit and functional tests and their supporting data. The only direct references should be to the utility assemblies (libraries) and to the testing frameworks (for example, Microsoft UnitTesting/NUnit and BizUnit). This also ensures a clean separation of tests and artifacts from the other parts of a solution. Normally there is only one testing project in the entire solution and it can be used to test BizTalk as well as .NET artifacts.
Non-project artifacts
There are also several solution-level folders I like to use to organize my solutions. Solution-level folders in Visual Studio are logical folders, not physical directories, so I like to create physical directories that match them to keep everything more organized. These are generally:
- Third-party assemblies: For storing any external assemblies and components the solution may need to utilize. The BizUnit assembly used by the solution testing should go here.
- Bindings: To store deployment bindings used in each of the environments you will have in your solution. I generally have Local Development, Integration, Staging, and Production. You may have more or less depending upon your organization and its needs.
- Build: Used to hold build scripts and other build artifacts like assembly signing keys. I also put my CI scripts into the build folder for safe keeping and source control.
- Policies: This solution folder will hold business rule policies and vocabularies and will help you keep them organized and in source control.
- Tracking: This folder will hold all BAM-related artifacts like activity definitions and tracking profiles.
It is important to not only structure your solution correctly, but to ensure that the build is as self-contained as possible. Although some environments will have only a single BizTalk developer working on a solution, it is vital that solutions contain all the artifacts necessary to build them within source control. This is a time saver if your workstation dies or your team size grows, or better yet you win the lottery and decide to take some well-deserved time off. I would like to validate that I am accomplishing this by building the solution on another developer workstation to ensure the build is portable, repeatable, and reliable. The following tips will help you to achieve these goals.
Tip
Always use relative path locations for all artifacts including:
- Keys used to sign assemblies. Don't browse to these; that will hard code an absolute path to the key file and the chances of another developer's workstation being set up exactly the same are slim. Worse still the browse feature in Visual Studio will likely make a local copy of the key file in the project folder. You really only need one key for an entire solution:
<PropertyGroup> <AssemblyOriginatorKeyFile>..\Build\PurchaseOrder.snk</AssemblyOriginatorKeyFile> </PropertyGroup>
- References to other assemblies and projects in the solution. Always use project references, not absolute references, when adding references to other BizTalk projects in the solution. After adding references to third-party assemblies (and if it is not compiled in this solution, it is a third-party assembly) edit the project file manually, it is simply an MSBuild file, and correct the absolute path. This is easy to test as your solution will no longer build correctly if the path is invalid.
- Paths of test files should also be made relative. For instance in BizUnit tests it is easy enough to simply paste in the path to the sample input file, but now that path is hard coded, use a relative path instead.
Tip
Bad idea:
D:\Projects\BizTalkPatterns\InvoiceProcessing\UnitTests\TestData\PurchaseOrder.xml
Good idea:
..\..\UnitTests\TestData\PurchaseOrder.xml
This will make it easier to build the solution on multiple machines and also make it easier to use a CI server to perform automated building and testing of the solution. Not doing this will result in solutions that only properly build on one developer's workstation. A good guideline is that you should be able to compress a solution folder and send it to another developer (or Microsoft Support Services, should the need arise) and not have to spend any time getting the solution to build or deploy. I was once given what I consider to be a great complement by Microsoft Support Services when a solution I needed support for built and deployed cleanly with no changes at all directly from a zip file. Even the support engineer was amazed.
Remember not all solutions will have these projects listed above; some will have less, others more, but keep in mind the separation of concerns brought up in the discussion about layers in a BizTalk solution. Later on I'll cover some specific examples of when to combine which parts of this guidance.
Motivations for solution structure
It is natural and healthy to ask why we should go through all this trouble (ignore that creating projects in a solution is very simple; even more so with a template). Earlier we covered the need to control dependencies and this is a certain way to do that, but there are other reasons as well. You may find yourself needing to update in-flight orchestrations or maps in a solution with such in-flight orchestrations that you cannot terminate and cannot wait for completion. Perhaps it's a simple fix and you haven't quite worked out your versioning strategy, or the business demands an emergency resolution (a misplaced decimal point or rounding error would be good motivations). With separate assemblies you are free to deploy just the changes necessary, rather than all assemblies and artifacts at once as would be the case with a single project solution, which would also require a full stop and termination of any in-flight instances.
Building solutions in a separable manner is a good practice that allows administrators and operators more control over the solution once it is out of development. This is critical to the success of any enterprise software system and even more so to BizTalk solutions.