Wednesday, April 09, 2014

Sentinet – Service Virtualization Part 4 - BizTalk Server

In the previous post I have discussed the protocol mediation between SOAP and REST services by the Sentinet product. I also demonstrated how Sentinet provides monitoring capabilities for service and API message exchanges. In this post I would like to continue my exploration of the Sentinet and share my experiences of using Sentinet together with the BizTalk Server.

In this scenario I want to expose the data of my LOB systems through the BizTalk Server. The BizTalk Adapter Pack includes WCF LOB Adapters that will give me access to LOB systems like SAP, Oracle eBusiness suite, and databases like SQL Server or Oracle. After BizTalk Send Port is configured with the WCF LOB Adapter to interface with LOB system, I can provide BizTalk with inbound endpoints that will expose my LOB data as a SOAP or REST service. I can expose BizTalk service endpoints using regular BizTalk WCF Adapters with bindings for HTTP or SOAP protocols. The access to these endpoints must be secured with authentication and authorization – and that can be challenging! In case when BizTalk Server resides behind a DMZ but access to the LOB data is meant to be provided externally from the outside (which is a very typical case), you will need to think about using a reverse proxy.

You could mitigate these challenges by using service virtualization concept implemented in the Sentinet. By placing a Sentinet Node in front of the BizTalk WCF service endpoints you can delegate most (if not all) of the authentication/authorization logic to a virtual service hosted in the Sentinet Node. By using Sentinet you can also make use of the other service virtualization features it offers, like protocol mediation, routing, and monitoring.

In the diagram below you see a Sentinet Node in front a BizTalk hosted endpoint that accepts messages and routes them a send port that communicates with a LOB system (in this case just a SQL Database).


You can setup security of the service endpoint hosted in BizTalk by configuring WCF Adapter. In this scenario you can delegate external security (authentication/authorization of the end users) to the virtual service hosted in the Sentinet Node and keep BizTalk endpoint configured with any WCF Adapter you want regardless of the required external security (for example, you can use WCF-WSHttp adapter with internal windows integrated security). By means of using virtual service you can now leverage the diverse security capabilities offered by Sentinet, where security of the virtual service can be controlled dynamically and remotely via Sentinet browser-based console. You can also make use of multiple identity types such as Username/Password, Kerberos/NTLM, X.509, SAML, binary security tokens and custom identity types over message-level or transport-level security models. Authorization can be enforced by using Sentinet Access Rules designer.

When you have to change BizTalk endpoint security, you have to reconfigure it with a different WCF Adapter, or at minimum you have to reconfigure adapter itself. Sentinet provides BizTalk services with far more flexibility by offering many different authentication/authorization schemes without any need to reconfigure BizTalk artifacts. Besides that, you can configure multiple virtual endpoints with different bindings to provide access to a single BizTalk endpoint, while BizTalk endpoint itself provides access to an LOB system or a database. Basically, Sentinet Node acts as a special gateway that can host any number of diverse virtual services. Once you have exposed your data and/or process through a BizTalk endpoint you will never have to change it to provide access to it for different consumers. The Sentinet will provide that for you through its configuration without any code. You can also create multiple virtual services with multiple endpoints supporting various bindings, authentication and authorization methods exceeding the capabilities of a single BizTalk adapter.

In the previous posts I have demonstrated how to create a Sentinet virtual service. To provide end users with more ways to access LOB data through a virtual service, you need to create multiple inbound virtual endpoints like indicated by the Sentinet screenshot below. Each inbound virtual endpoint will be configured with its own security, while they all route messages to the same BizTalk endpoint.


Now I will walk through creating multiple endpoints with various bindings and security mechanisms to demonstrate the capabilities Sentinet offers. In this and the upcoming posts I will create five different virtual endpoints configured with different bindings and security mechanism. All five virtual endpoints will route messages to the same single BizTalk endpoint.

In this post I will start with a simple (unsecured) endpoint configured with SOAP 1.2 over SSL and no user credentials. Then I will add a secure endpoint configured with SOAP 1.1 over SSL that requires username/password client credentials. In the next posts I will continue with the endpoint that requires client X509 certificate, SAML Token (Claims), and yet another endpoint configured with the Microsoft Azure Service Bus. Note, that while I will be complicating use cases by adding and changing security scenarios, I will not be “touching” my BizTalk Server configuration that will remain configured with the same single endpoint and the same single BizTalk Adapter. In other words, my BizTalk application will be given agility to adapt for continued changes, and it will be provided with new capabilities (such as to be a "claims-aware” application) with no configuration or code changes.


The first endpoint will be an endpoint that uses transport security (https) with SOAP 1.2 binding.


I named this endpoint SearchTimes and I applied to this endpoint the policy that I named Secure HTTPS. Policy configuration is visualized by another Sentinet screenshot below.


The second endpoint will be configured with basicHttp and transport-level security (https) that requires username/password client credentials.


I named this endpoint RunningResults and I applied to this endpoint the policy that I named Steef. Policy configuration is visualized by another Sentinet screenshot below.


After both virtual endpoints are created, I add an Access Control that will manage authorization of the messages sent to these endpoints. Access Control in the Sentinet is driven by the Access Rules that are created in the Sentinet Access Rules Designer. I will use Everyone Access Rule that will provide access to anyone who sends messages over HTTPS to SearchTimes endpoint, while SimpleAuth access rule will require specific username/password. The latter can be created by adding a new access rule in the Sentinet Repository, then using drag and drop of the Username/Password element from the Identity conditions of the Sentinet Access Rules Designer. Provide specific username and password and save your access rule.

Note that Sentinet Access Rule can be configured with many username/password combinations, where specific username/password identities may be stored in the Sentinet Repository, or in Active Directory, or in your own custom database, or in any other external identity system.


After you created Access Rules you assign them to the virtual service Access Control. You simply drag and drop the Access Rule on the virtual endpoint and hit the Save button. You will now see Access Rules applied to virtual endpoints SearchTimes and RunningRules like it is shown in the screenshot below.


To test both virtual endpoints I will use SoapUI. You export the WSDL of your virtual service and save it on disk. Open SoapUI and create a new SOAP Project, specify project name and navigate to the WSDL file saved on disk. Click Ok and the project will create operations for each web service endpoint. Double click the SearchTimes endpoint, click Request 1 and fill in the parameters.


Hit the green play button and the request will be sent to the endpoint of the virtual service. Request will be sent through the virtual service to the BizTalk endpoint (receive location).


You can see all the details of this message exchange through the Sentinet Monitoring and optional messages recording. You will see request message sent by the SoapUI and recorded by the Sentinet virtual service. You can also see complete response that is returned by the virtual service to the SoapUI client application.

The SearchTimes endpoint is using HTTPS transport to encrypt and digitally sign messages on the wire. This does not really mean the use cases represents secure communication because the client identity is not required for this message exchange and anybody can send a message to the SearchTimes endpoint. By adding the RunningRules virtual endpoint I make it a secure endpoint by forcing the user of the service endpoint to provide credentials. In SoapUI navigate to the RunningRules endpoint, double click the endpoint, click Request 1 and fill in the parameters. Hit play and you will see SOAP Fault coming back to SoapUI client.

<s:Envelope xmlns:s="">
<faultcode xmlns:a="">a:InvalidSecurity</faultcode>
<faultstring xml:lang="en-US">An error occurred when verifying security for the message.</faultstring>

In the monitoring of Sentinet I will see recorded request and response. Moreover, you will be able to see that virtual service did not even try to forward request to the BizTalk endpoint, rather it immediately generated SOAP Fault and returned it to SoapUI. The reason for that is clear - I did not provide username/password in the SoapUI. As a result of that, Sentinet virtual service rejected my request with the SOAP Fault, and it did not even “bother” my BizTalk endpoint.


So I have to provide credentials when making my request to the virtual service. This can be specified in the SoapUI behind the Auth tab.


If I hit the Play button now I will receive a response with the data I want. In the Sentinet monitoring I can see the message (request) sent to the virtual service endpoint.


What’s interesting is in the Usernametoken element:

<wsse:UsernameToken wsu:Id="UsernameToken-9">
<wsse:Password Type="">test</wsse:Password>
<wsse:Nonce EncodingType="">O11AOMFb+wfUqTn9nB7CBQ==</wsse:Nonce>

So far I have demonstrated two different endpoints of the virtual service that sits in front of the BizTalk application. As you can see I did not create any code. Everything is done through Sentinet configuration. In the next part I will continue with X509 virtual endpoint followed by parts on SAML/Claims and Service Bus virtual endpoints.



Saturday, April 05, 2014

Windows Azure BizTalk Service REST API

The BizTalk Service in Windows Azure can be managed through a REST API. The API provides programmatic access to much of the functionality available through the Management Portal. With the REST API for managing BizTalk Services you can perform the following operations:
  • Register BizTalk Service
  • Create or Update a BizTalk Service
  • Get Cloud Service
  • Get Cloud Services
  • Get BizTalk Service Properties
  • Suspend BizTalk Service
  • Resume BizTalk Service
  • Restart BizTalk Service
  • Poll on an Async Operation on BizTalk Service
  • Sync Access Control Keys
  • Delete BizTalk Service
  • Backup BizTalk Service
  • Restore Azure BizTalk Service from Backup
In this post I like to show how you can leverage the REST API to manage BizTalk Service by means of a windows form application.


To send request messages through the operations of the API you will need to have authentication in place. The REST API like the Windows Azure Service Management API uses mutual authentication of management certificates over SSL to ensure that a request made to the service is secure. Note that anonymous requests to the API are not allowed. Therefor requests made to the REST API for managing WABS requires that a management certificate is associated with your subscription. This certificate can be self-signed using the makecert command line tool or a commercial one obtained through VeriSign or Go Daddy. See the MSDN article Create and Upload a Management Certificate for Windows Azure.

REST API for BizTalk Services

The REST API for BizTalk Services is documented on MSDN. Each operation mentioned in the introduction is documented. The Request URI, URI Parameters, Request Header, Request Message, Request Elements, Response, Status Code, Response Header, and Response Body of each operation is described.

A common operation to perform before any other operation is the Get Cloud Services. This general operation will provide you with the details of your BizTalk Service(s). The Get Cloud Services will get all BizTalk Services within all cloud services. With the data provided by these operations you can perform any other operations as it will provide you with a common piece information the cloud-service-name. This is the unique name for the cloud service that hosts your BizTalk Service.

Get Cloud Services operation

To perform the Get Cloud Services operation from the REST API in a Windows Form you need to provide the subscription id, which is your Azure subscription ID. You can find this id by going to the settings page of your Windows Azure Portal.

The request will be as follows:

The response will look like:

As you see the response provides a list of resources i.e. BizTalk Services and its details like State, Edition, and tracking details.

Get BizTalk Service Properties

Sending a request to the Get BizTalk Service Properties operation will provide you the properties of a specific Windows Azure BizTalk Service. You specify in your request the subscription id of your Azure Account, Cloud Service Name (which can be obtained through the Get Cloud Services operation) and the resource name (name of one of your BizTalk Service(s)).

The request will have the following format:

The response might look like:

You see all the details of the request BizTalk Service in the response message.

Delete BizTalk Service

Sending a request to the Delete BizTalk Service operation will result in the deletion of the BizTalk Service. You specify in your request the subscription id of your Azure Account, Cloud Service Name (which can be obtained through the Get Cloud Services operation) and the resource name (name of one of your BizTalk Service(s)).

The request will have the following format:


The response might look like:

The status code is 202, which means the operation was successful. You can call to check the asynchronous result through GET operation with URL provided Request URI.

If you check within the Windows Azure Portal under BizTalk Services you will see that the specified BizTalk Service is no longer present.

This post I demonstrated how you can leverage the REST API to manage the BizTalk Services. As an example a windows form application implements a few of the REST API operations for WABS. Three operations were discussed yet you can implement any of the other operations in a similar way using .NET code.

There are different ways you can leverage the API other than using a Windows Form Application. You could choose to use ASP.NET or a Mobile application. The REST API for BizTalk Service is created by Microsoft to enable you to perform various operations on the BizTalk Services. Microsoft offers various other REST API’s to manage other services offered in Windows Azure.

The code of the WABS Manager can be obtained through MSDN Code Gallery: Windows Azure BizTalk Services Manager

Note that not all the operations have been implemented in the tool. You reuse the code to create your own full fledged WABS Manager.



Friday, April 04, 2014

BizTalk Developer Productivity with BTSG NoS AddIn

A dear friend of mine has been working hard on a developer productivity add in for BizTalk projects in Visual Studio 2012. He name is Nino Crudele a Microsoft Integration MVP from Italy.


Steef-Jan, Sandro, and Nino Palazzo Gotico in Piacenza, October 2013.

The add in is called BTSG NoS. Nino started with the add in creation started last year after he thought it was time someone cared about developer productivity in BizTalk. Since there has been not any enhancements the last years in the BizTalk Server product regarding productivity Nino took matters in his own hands.

The add in is almost ready for a public beta. At the moment the add in is vigorously tested by fellow Microsoft Integration MVP Sandro Perreira. And he recently created a few post about the add in:

More posts/documentation on the add in will follow. I myself had the opportunity to look at a stable version of the add in and I must say it the user experience is amazing and intuitive. The Installation is straightforward and it is easy to add in Visual Studio through the Add-in Manager.
There is a good set of features available in the add in that will make live easier for a BizTalk developer. You can for instance select your BizTalk project in your solution and choose reflector.


It will then show you a dialog with information of the solution.


This is just an example of the many feature the add in contains. All of them are target to increase the productivity of the developer.

Nino has done an outstanding job creating this add in and make it available for community with next couple of weeks.So keep a watch on Nino blog. A video of his presentation during the London BizTalk Summit 2014 will be available soon. See BizTalk360 BizTalk Summit 2014, London – Videos.



Wednesday, April 02, 2014

First book on WABS is there!

Windows Azure BizTalk Services was released November 2013 and now within six months’ time there is a book about available. It has been written by Karthik Bharathy a Lead Program Manager in the BizTalk product group and longtime BizTalk veteran and Microsoft Integration MVP Jon Fancey.


I have had the pleasure of being one the technical reviewers of this book and I must say the authors have done an excellent job. Is this a biased opinion. No, because:
  • I have been working extensively with WABS and its predecessors EAI/Labs (beta). So I was able to put the thumbscrews on both of them, which resulted in valuable feedback for the book
  • of the Microsoft BizTalk Product Group involvement, and they are very committed to it,
  • Scott Guthrie endorses it and wrote a forward.
WABS provides EAI or B2B services through the cloud and both are covered in the book. It is a promising technology with the increase demand for integration. The landscape we know today has changed with the rise of the cloud (internet), devices and on premise growth of the IT-systems (including mainframe, servers, applications, and services). Integration of it is key and so is having an integration service in the cloud to support new hybrid scenarios.
You will learn the following while reading it from cover to cover:
  • Use the EAI and B2B features of BizTalk Services
  • Connect with Line-Of-Business systems in your datacenter on-premises
  • Create bridges and configure them to process and route messages
  • Design transforms
  • Using custom code
  • Address and diagnose problems
  • Operations using the API
  • Migrate from BizTalk Server to BizTalk Services

If you are new to WABS or want to learn more than I definitely recommended it. Even I learned quite a few thing during the review of the chapters. You can download a sample chapter in case you are interested. It is about bridges. You can buy the book from Packt, Amazon, or Barnes & Noble.

In case you are interested in more content than you can visit the TechNet Wiki article: Windows Azure BizTalk Services Resources on the TechNet Wiki.



Monday, March 24, 2014

Windows Azure BizTalk Services: Pulling Messages from a Service Bus Queue

Windows Azure BizTalk Services (WABS) provides capabilities for EAI and B2B in the cloud. This relative new service was made available for customers in November 2013. WABS is a cloud integration platform or integration platform as a service or IPaaS. Characteristic of IPAAS is that you build your integration on premise, deploy in the cloud, where it is hosted in a service (set of dedicated hardware). The service is offered through a subscription model for operating the service in the cloud (for BizTalk Services this means Windows Azure hence the name Windows Azure BizTalk Services).

Currently there are quite some players offering that service like Dell, Attunity, MuleSoft, TIBCO, and so on. Microsoft is currently pushing the service hard to catch up with some of these players and try to quicken/hasten the maturity of the product. Therefore, Microsoft has committed to have a release cadence for new features every three months. The recent release was in February 2014, which added new features for WABS. These new features are:
  • EDIFACT Protocol Support and X12 Schema Updates
  • Pulling Messages from Service Bus Queues and Topics
  • Service Bus Shared Access Signatures (SAS) support with Service Bus Queues and Topics
  • BizTalk Adapter Services No Longer Needs SQL On Premises
  • Backup and Restore Support
  • Operation Log Support
This post will dive into the new feature of pulling messages from a Service Bus Queue. When a developer designs and build a bridge he/she can now choose two new sources i.e. Service Bus Queues and Topics. A Bridge with WABS is a workflow that processes a message pulled from a source and delivered to a destination (see picture below).

Benefits of pulling messages from Service Bus queues and topics are:
  • Durable messaging
  • Leveraging the support for multiple protocols
Durable messaging
The ability to pull messages from Service Bus queues and topics increases the durability of a bridge solution. You can leverage the pub-sub mechanism of the topics and subscriptions. It is even possible to have multiple bridges pulling messages from different subscriptions and have them send to multiple destinations. A single bridge will not enable you to do that.
Leveraging multiple protocols
Another benefit is that you as a developer will have more input channels for the bridge besides FTP, and HTTP (REST). Indirectly through the Service Bus Queues you could have support for AMQP.


To explore the new capability of pulling message from the Service Bus we will explore the following scenario. A payroll company wants to offer a service in Windows Azure BizTalk Services for organizations to send their payroll data for pay rolling. Organizations that like to outsource their pay rolling can connect to this service in Azure. Basically the organizations can send their payroll data to the service in any kind of format and structure they want. The payroll company will provide a bridge to provide connectivity to the source the organization pushes its data to. This can be FTP(S), HTTP or the Service Bus Queue or Topic (subscription). The data will be picked up from the source and processed to one unified format to be routed to a LOB system or database for a later pay roll run that the payroll company will do. Below you will find a diagram detailing the scenario.

I have used this scenario also in my talk during the BizTalk Summit 2014 in London discussing the manageability of WABS.

Building the solution

The solution will be a bridge for an organization that requires to push out its payroll data to the payroll company. In this case the data is pushed to a Service Bus Queue and the Bridge will pull the data from the queue, transform it to a format that enables a table operation in SQL Server on premise. To build this solution the following parts need to build, configured and/or created:
  • a LOB Target that needs to be configured for access to SQL Server through a LOB Relay endpoint,
  • generation of a schema for the table operation (insert operation),
  • a mapping,
  • a bridge design,
  • connection of the source to the bridge and to the destination,
  • configuration of the source, the bridge and the destination. 
When solution is build it needs to be deployed to the BizTalk Service in the cloud.

Create a BizTalk Service Project

To build a WABS solution you need to create a BizTalk Service project: Open Visual Studio and on the File menu, point to New, and then click Project. Use the details in the following table to create the project and then click Ok:

Installed Templates:  Click BizTalk Services, then click BizTalk Service.
Name:  Specify a name.
Location:  Set this to a location on your computer.
Create directory for solution: Select this if you want this solution to have a separate folder in Windows Explorer.

LOB Target/Relay

A part of the Windows Azure BizTalk Services is on-premise infrastructure. By downloading the WABS SDK you can set up that infrastructure. By infrastructure in this context is addition of WABS templates in Visual Studio 2012 and installation/configuration of the BizTalk Adapter Service. This service is basically a IIS wrapper around the adapter pack that provides access to on-premise LOB systems. The BizTalk Adapter Services can be viewed as a gateway to premise LOB systems that can be securely accessed indirectly through WABS. The concept of BizTalk Adapter Service is to ability to create kind of endpoints to operation on your LOB systems. You create a LOB Target, which is your on-premise Line-of-Business (LOB) system and the operations (like SELECT or INSERT) exposed to your client applications (a Bridge). During the configuration of a LOB Target you create a LOB Relay, which is a URL that provides a connection to the cloud using Service Bus Relays.
For installation of the BizTalk Adapter Service see TechNet Wiki article BizTalk Adapter Service Installation.
Creating a LOB Target/Relay
A LOB Target is created through Visual Studio. You have to perform the following steps to create the LOB Target:
  1. In the BizTalk Service project, open Server Explorer, right-click BizTalk Adapter Service and expand.
  2. You will need to authenticate to further expand to LOB Types.
  3. Provide credentials to access the BizTalk Adapter Service.

  4. Click Ok. You can now expand LOB Types, right-click SQL, and select Add SQL Target.
  5. You will see a Wizard pop-up and the first screen will be a welcome screen.

  6. Click Next and configuration process of LOB target will start.
  7. In the Connection parameters tab you specify the connection details of SQL Server and the credentials.

  8. Click Next.
  9. In the Operations tab you can select object, operations and add them to the selected operations.

  10. Click Next.
  11. In the Runtime Security tab you specify the way you like the authentication to LOB target i.e. SQL Server.

  12. Click Next.
  13. In the Deployment tab you specify LOB Relay. You can select an existing relay or specify a new one.

  14. To create a new LOB Relay, enter the following details: Service Bus Namespace
    Specify the Service Bus namespace on which the LOB relay endpoint is created.
    Service Bus
    Issuer Name

    Specify the issuer name for the Service Bus namespace
    Service Bus
    Issuer Secret

    Specify the issuer secret for the Service Bus namespace
    LOB Relay Path
    Enter a name for the relay.
    LOB Target Sub-path
    Enter a sub-path to make this target unique. 
    Target runtime URL
    This read-only property displays the URL where the relay is deployed on Service Bus. This is the path where you could send a message to be inserted into the on-premises SQL Server. In the payroll scenario, this is where the bridge routes the message.
  15. Click Next.
  16. The summary tab will appear and you review your LOB Target/Relay setup.

  17. By click Create the LOB target/relay will be created. What basically will happen is that LOB target will be created in the BizTalk Adapter Service and is created as an application in IIS.

    Generate schema for the insert operation
    When a LOB Target is created you can generate the schema(s) for specified operation(s). The following steps describe how to generate the schema for the insert operation:
    1. In the BizTalk Service project, in the Server Explorer, right click the LOB target you created, and then click Add Schemas To Payrollservice....
    2. A dialog will appear, where you specify the credentials, folder name, ...

    3. Click Ok.
    4. The schema(s) will be added to the BizTalk Service project.

      Generate schema for request

      In BizTalk Service project you can create the schema for the incoming request message containing the payroll data. The data is delivered per employee. The schema can be created as visualized by screenshot below.

      Create Mapping from source schema to destination schema (XML Transform)

      The following steps show how to transform the incoming request message (source) to a request message for LOB Target (destination):
      1. In Visual Studio, right click the your project, point to Add, and then click New Item.
      2. In the Add New Item dialog box, select Map, specify the map name as ".trfm", and then click OK.
      3. In the Solution Explorer, double click the ".trfm" file to open the Transform. On the Transform surface, select the source schema to Request.xsd and the destination schema to xxxxx_PAYROLLSERVICE_payrolldata_TableOperation.dbo.Employee.xsd.
      4. Drag lines between the various elements in the source and destination schema like below:

        Configure a one-way Bridge

        The following steps describe how to configure the one-way bridge with a source and destination. You select the MessageFlowItinerary.bcs and:
        1. Drag and drop the LOB Target Entity from the BizTalk Adapter Service onto the design surface (destination).
        2. Drag and drop XML one-way bridge onto the design surface.
        3. Finally drag and drop Service Bus Queue source to the design surface (source).
        4. Double click the XML One-Way Bridge on the itinerary designer.
        5. On the XML One-Way Bridge design surface, within the Message Types box, click the add icon [+] to open the Message Type Picker dialog box.
        6. In the Message Type Picker dialog box, from the Available message types box, select the Request (PayRolldata) message type, click the right arrow icon to associate the request schema with the XML One-Way Bridge, and then click OK
        7. Configure the bridge to use the Transform created earlier. Within the Transform stage, select the Xml Transform activity, and then from the Properties window click the ellipsis button (…) against the Maps property to open the Map Selection dialog box.
        8. Save the bridge configuration and go back to the Bridge Configuration designer surface
        Configure the Service Bus Queue source
        The source of the bridge needs to be configured (connectivity) to enable the bridge to receive or pull a message from. For the service bus queue you need to specify the connection string in the following format:
        Endpoint=sb://<your service bus name>;SharedSecretIssuer=<issuer name>;SharedSecretValue=<issuer secret>
        Beside the connection string you specify the name (entity) of the source and name of the queue.

        Configure the LOB Target Destination
        Once you have dragged a LOB Target on the design surface you will see a <your lob target>.config appear. You can double click this file in the solution explorer. This file resembles a kind of binding file where you need to specify the credentials of your LOB Target Relay. These are service bus credentials.  After specifying those in the file you can save it.

        Connect the components on the bridge design surface
        After configuration of the source, bridge and destination it is time to connect the source to the bridge and the bridge to the destination. The following steps describe how to connect the components and connection configuration from the XML One-way bridge to the LOB Target:
        1. From the Toolbox, select Connection, and drag-drop the mouse pointer from the Payroll Queue component (source) to the bridge component to connect the two.
        2. Select a Connection again from the Toolbox, and drag-drop the mouse pointer from the bridge component to the LOB Target/Relay component to connect the two. 
        3. Set the filter condition on the connection between the bridge and the LOB Relay entity.
        4. Click the connection between XML One-Way Bridge and the LOB Relay entity.
        5. In the Properties window, click the ellipsis (…) button for Filter Condition.
        6. In the Route Filter Configuration dialog box, set the filter condition to Match All. This ensures that all the messages that are processed through the bridge are routed to the LOB entity.
        7. Click Ok.
        8. Set the Route action so that the outgoing message to the LOB application has the appropriate SOAP action header.
        9. Open Server Explorer and navigate to the SQL LOB Relay created earlier. Right click the relay, click Properties, and for the Operations property, copy the value of the first operation (insert). This value denotes the value of the SOAP action header that must be set on the message that is routed to the LOB Relay.
        10. On the Bridge Configuration surface, click the connection between XML One-Way Bridge and the LOB Relay entity.
        11. In the Properties window, click the ellipsis (…) button for Route Action. In the Route Actions dialog box, click Add to open the Add Route Action dialog box. In the Add Route Action dialog box, do the following:
        12. Under Property (Read From) section, select Expression, and then paste the value that you copied earlier.
        13. Under Destination (Write To) section, set the Type to SOAP and the Identifier to Action. The dialog box resembles the following:

        14. Click OK in the Add Route Action dialog box to add the route action. Click OK in the Route Actions dialog box and then click Save to save changes to an BizTalk Service project. The end-to-end message flow resembles the following:

        15. Save change to the project.

        Build, deploy and Test the solution

        You have finished creating the application. In this steps, you will build and deploy the application under your BizTalk Services subscription. Subsequently you will use the Service Bus Explorer to test your solution.

        To build and deploy the solution

        To build and deploy your solution you can perform the following steps:
        1. In Visual Studio, right click your bridge solution, and then click Build Solution.
        2. Once the build succeeds, right click your solution, and then click Deploy Solution.
        3. In the deployment window, the Deployment Endpoint is a read-only property and the value is derived from the BizTalk Service URL/Namespace set in the message flow surface. However, you must provide the ACS Namespace for BizTalk Services, Issuer Name, and Shared Secret.

        4. Click Deploy. The Visual Studio Output pane displays the deployment progress and result. The URL where the bridge is deployed is also displayed in the Output pane. For this scenario, the bridge is deployed at

        Test the solution

        To test the solution you can use the Windows Azure Service Bus Explorer. This tool can be used to sent a message to a service bus queue. The following steps will describe the way to test the solution:
        1. Open the Service Bus Explorer.
        2. Connect to the appropriate namespace within the service bus where the queue resides that the bridge will pull the messages from.
        3. Right click the queue and select send messages.
        4. Paste/Load a message payload in the Message Text pane.

        5. Click Start.
        6. Message will be sent to the queue.
        7. You can see the log displaying the result.

        8. In SQL Server you can view the Employee table to verify if the data is present in the table.

        The addition of queues and topics as sources to WABS has extended the flexibility and ability to create durable cloud solutions. However, WABS will need more adapters (sources/destinations) to be able to compete with some of the other IPaaS vendors like MuleSoft, who offers a great deal of adapters. Hence more versatile integration capabilities.

        This post demonstrated the use of pulling messages from a Service Bus Queue. This ability can add durability to your cloud integration solution using WABS. The walk through of the payroll scenario demonstrated the ease of creating a bridge solution. Most of the configuration is done through Visual Studio a main component currently to build, deploy and manage a bridge solution.

        The dependency of Visual Studio for building WABS can be considered undesirable. Changes to the configuration of sources and destinations for a bridge has to be done in Visual Studio accordingly and then redeployed to the BizTalk Service hosting the bridge. Therefore, managing some parts of a bridge solution like the configuration of sources and destinations is preferable better off in a different tool like a management console/web application or within Windows Azure itself. Since this is still the first release after the go-live of WABS last year in November you can expect improvement of the service in many areas including the manageability.



        Monday, March 10, 2014

        BizTalk Services Bridges/Operations – Troubleshooting

        Windows Azure BizTalk Services (WABS) is general since November 2013 as a service within Windows Azure. Recently it has been enhanced with new features like backup/restore.

        This new service in Windows Azure is meant to provide EAI or B2B services through the cloud. The EAI Service enables you to exchange data through different protocols and transform it to and from different formats. Similar to what the on premise BizTalk offers through mapping and routing. The B2B services offer Businesses to exchange data between their partners. You can view it as a new way of EDI data exchange other than a value added network (VAN).

        During provisioning of a BizTalk Service you need to specify tracking database (Windows Azure SQL Database), and a storage account for monitoring and archiving.

        The tracking data can be useful for diagnostic purposes and to monitor what messages goes through BizTalk Service.

        The Tracking Database can be extended to meet your own tracking requirements, see Windows Azure BizTalk Services EAI Bridges – Diagnostics. You can also use the tracking data for troubleshooting combined with the debug logs stored in your storage account. The recent February release included integration with the operation logs a part of the Windows Azure Management Service. This gives you the ability to troubleshoot issue you might face with the BizTalk Service itself.

        Storage account

        Provisioning of a BizTalk Service includes a step, where you specify a storage account. In the final tab called Specify monitoring/archiving settings you specify the storage account. You either create a new or use an existing storage account to store monitoring and archiving data. Note that this storage account can be used by only one BizTalk Service! During provisioning the following will happen:
        • A storage account will be created in case an existing storage account has not been assigned,
        • Blob containers (blobs, tables) will be created.
        After provisioning you could access what’s available in these containers using the server explorer. Within Visual Studio, navigate to Windows Azure in the Server Explorer, select Storage. You subsequently need to log in and then you can navigate to your storage account.
        You can view what's inside the blobs and tables. By right clicking on the blobs you can see what is inside.
        By right clicking on the tables you can view the contents.


        Each message send to a BizTalk Service, for instance to a bridge endpoint a response will be sent back. Regardless if it is successful or a failure. The header of the response includes a tracking ID. This ID can be used in the Tracking Messages feature in Windows Azure BizTalk Services Portal to troubleshoot. You can use the BizTalk Service explorer to send a message to a bridge and see what the tracking id is.
        In the Windows Azure Portal under Tracking you can see the information regarding the message being sent to the Bridge Endpoint.
        You can select the event and click Details in the lower pane of the portal.
        This metadata is retrieved from the database residing in Windows Azure SQL Database.
        You can view more details through debug logs, which are available in the WADLogsTable. You can select the details from the table and paste it in excel. Below you see the what is under the message column, which can be interesting to see what's in this case happens in the bridge.
        The debug logs in the table contain the following information:
        • Loading an assembly
        • Adding or updating an artifact, like a Transform
        • Adding, updating, or deleting Bridge Configuration
        • Message submitted to the Bridge pipeline
        • Bridge stages, including their Begin Execute and End Execute events
        • Faults
        Note: The debug logs are only available during the development process. In each row under you will this kind of data: 
        Partition Key column 6,35239E+17
         Row Key 1d0c1f327e9c4b6e9a7e8ed405732638__IntegrationRole___IntegrationRole_IN_0___0000000001652031490___WADLogsLocalQuery TimeStamp 12/29/2013 3:14:52 PM
         EventTickCount 635228233283452000 
        Deployment ID 1d0c1f327e9c4b6e9a7e8ed405732638
         RoleInstance IntegrationRole IntegrationRole_IN_0
         EventID 30008 
        Pid 3216
         Tid 104
        Message column Stage validator execute called.
        RequestId                         =             f3d60453-a63a-44e9-811a-7ac47e1b111d. TrackId                            =             f3d60453-a63a-44e9-811a-7ac47e1b111d. ServiceNamespace           =             default.
         RuntimeUrl                     =             /xmlrequestreplybridge1.; TraceSource 'Microsoft-Integration-PipelineService' event  

        Operation Logs

        The integration of BizTalk Services to the operation logs of the Management Service has been an enhancement of the February 2014 release. You can view the operations performed on the BizTalk Service, see MSDN Operations Tracked Using Azure Management Services. image

        You can click any operation and view the details. You can do so by selecting a specific row and click Details from the bottom of the page.


        This post has provided some insight how to troubleshoot a BizTalk Service Bridge. There are two main sources, where you can find data for troubleshooting. That is, through tracking database in the Windows Azure SQL Database, which contains the data that is visible in tracking messages feature of the BizTalk Services Portal. And the storage account that has blobs and tables. The WADLogsTable contains debug logs you can access to have a clear view of what is going inside the bridge. Besides these two sources there is the operations logs within the Management Service that can give you details about specific BizTalk Service operations.