Service Bus Messaging: Queues, SB-Messaging Adapter
Like some of you I have been experimenting with the Service Bus Queues, Topics and subscriptions. The information from Microsoft gives you a straight forward insight how to send messages to queues, topics and subscriptions using .NET. The first step to interact with the Service Bus Queues, Topics and subscriptions is by creating an account with Windows Azure through a trail or buy option. The next step is creating a service namespace through the Windows Azure Portal followed by obtaining the management credentials you need when setting up Service Bus Connection string. To create a solution that can interact with the Service Bus you need to have a reference in your Visual Studio project to the Microsoft.ServiceBus.dll on your machine, which is a part of the Windows Azure SDK. In your code template (for instance C# form) you will add two using statements:
The client application is a .NET Windows Form application that will send messages to a queue in Windows Azure Service Bus. An application in BizTalk Server 2010 R2 has a port with a receive location configured with SB-Messaging Adapter. With the port a mapping will be performed to Insert Operation Message format. A send port has a subscription on messages with the SQL Insert format. The send port will send the messages to SQL Server 2008 R2 resulting in inserts in a SQL Table.
The client application
The client application is a Windows Form application. It has one form, where details can be filled for a runner like Name, number and time. I use the participants textbox to simulate a load to the Service Bus (see the latency section).
When I hit submit after filling in participants (i.e. 1), name, number and time the runner information will be sent a message to the Service Bus Queue.
The Service Bus Queue
Service Bus queue can be created through code, the service bus explorer within Visual Studio 2012, the Service Bus Explorer created by Paolo Salvatori (see my post Visual Studio Service Bus Explorer versus Standalone Service Bus Explorer) or through Windows Azure Portal.
BizTalk Server 2010 R2 Solution
The BizTalk Server solution consists of a schema of incoming messages from the Service Bus (runner message), a map and schema’s generated through Adapter Metadata Utility (i.e. Consume Adapter Service Add-in in Visual Studio 2012, see also TechNet Wiki article on Adapter Pack). Below you can see the artifacts created in a BizTalk project in Visual Studio 2012
Runner Schema
SQL Insert Operation Schema
Map
The solution is deployed to the BizTalk runtime and configured through the BizTalk Administration Console. The interesting part is the configuration of the receive location using the SB-messaging adapter.
Configuring the SB-Messaging Adapter
The first tab for configuring the SB-Messaging is setting the URL for Queue (see picture of the Service Bus Queue: QUEUE URL). In the Second Tab the Authentication you provide the management credentials of the namespace (BTUGNLD).
The final tab properties you can enable promotion of Brokered Message Properties.
The configuration of the receive location is straight forward as you can see. The information on configuring this adapter in a receive location can be found on MSDN: How to Configure an SB-Messaging Receive Location.
Running the solution
When I run the client application a form will appear, where I can fill in the details of a runner.
After filling in the detail I hit submit. When a message is send I can see the duration of sending the message from the client to Service Bus Message Queue. On average it takes less than a second. On my VM in Windows Azure I query the RegisteredTimes table and will see the details of the runner.
Latency
Sending one message to Service Bus queue takes less than a second. Sending a 100 messages (setting participants to 100) will take about 2 seconds.
As soon as the client is finished with sending the messages I instantly check the database in my VM and query number of records in the table. All the records are there. So it's safe to say that latency to have all messages pulled from the queue and inserted in SQL table is around the same time (i.e. duration to send to queue and few milliseconds later). With number of messages increasing the average duration (latency) is far less than a second to send a message to queue and then pick it up from queue to insert the message (data) into a table.
Throughput
Now 100000 messages would take about an 2 hours (extrapolate with measured data below) to be inserted in SQL Server table.
Usage Overview
Below you can see the number of messages going through the Service Bus Queue “registeredtimes”. There is a latency of a few minutes before actual number are shown (it is near real time).
Overall I am very impressed with this solution. Messages sent so quickly to a Service Bus queue and are then picked up by BizTalk Server 2010 R2 using the SB-Messaging adapter to store data in SQL Server 2008 R2. Image that I would like to register all the finishing times of a marathon with 50000 participants or all finishing times including passing the 5, 10, 15, 20, 25 ,30, 35 and 40 km mark. That would accumulate in 450.000 messages. This can be quite some time (a few hours, see throughput) so scaling out with more queues, tune SQL Server and BizTalk would be a way to shorten the time.
You could of course use a queue for 5, 10, 15, 20, 25 ,30, 35, 40 km and finish mark. Or you could use topics and subscriptions for this. In my next post I will discuss a solution that will support this scenario using queues and topics/subscriptions including some fine tuning of the BizTalk / SQL Server. Interesting to see the latency and throughput then.
Stay tuned!
- using Microsoft.ServiceBus;
- using Microsoft.ServiceBus.Messaging;
The client application is a .NET Windows Form application that will send messages to a queue in Windows Azure Service Bus. An application in BizTalk Server 2010 R2 has a port with a receive location configured with SB-Messaging Adapter. With the port a mapping will be performed to Insert Operation Message format. A send port has a subscription on messages with the SQL Insert format. The send port will send the messages to SQL Server 2008 R2 resulting in inserts in a SQL Table.
The client application
The client application is a Windows Form application. It has one form, where details can be filled for a runner like Name, number and time. I use the participants textbox to simulate a load to the Service Bus (see the latency section).
When I hit submit after filling in participants (i.e. 1), name, number and time the runner information will be sent a message to the Service Bus Queue.
The Service Bus Queue
Service Bus queue can be created through code, the service bus explorer within Visual Studio 2012, the Service Bus Explorer created by Paolo Salvatori (see my post Visual Studio Service Bus Explorer versus Standalone Service Bus Explorer) or through Windows Azure Portal.
BizTalk Server 2010 R2 Solution
The BizTalk Server solution consists of a schema of incoming messages from the Service Bus (runner message), a map and schema’s generated through Adapter Metadata Utility (i.e. Consume Adapter Service Add-in in Visual Studio 2012, see also TechNet Wiki article on Adapter Pack). Below you can see the artifacts created in a BizTalk project in Visual Studio 2012
Runner Schema
SQL Insert Operation Schema
Map
The solution is deployed to the BizTalk runtime and configured through the BizTalk Administration Console. The interesting part is the configuration of the receive location using the SB-messaging adapter.
Configuring the SB-Messaging Adapter
The first tab for configuring the SB-Messaging is setting the URL for Queue (see picture of the Service Bus Queue: QUEUE URL). In the Second Tab the Authentication you provide the management credentials of the namespace (BTUGNLD).
The final tab properties you can enable promotion of Brokered Message Properties.
The configuration of the receive location is straight forward as you can see. The information on configuring this adapter in a receive location can be found on MSDN: How to Configure an SB-Messaging Receive Location.
Running the solution
When I run the client application a form will appear, where I can fill in the details of a runner.
After filling in the detail I hit submit. When a message is send I can see the duration of sending the message from the client to Service Bus Message Queue. On average it takes less than a second. On my VM in Windows Azure I query the RegisteredTimes table and will see the details of the runner.
Latency
Sending one message to Service Bus queue takes less than a second. Sending a 100 messages (setting participants to 100) will take about 2 seconds.
| Number of Messages | Duration (Latency) in seconds | Messages/Sec |
| 1 | 0.665 | 1.50 |
| 10 | 1.59 | 6.29 |
| 100 | 9.15 | 10.92 |
| 200 | 14.39 | 13.89 |
| 500 | 35.84 | 13.95 |
| 1000 | 68.07 | 14.69 |
| 10000 | 707 | 14.14 |
| 100000 | ~7142 | ~14 |
As soon as the client is finished with sending the messages I instantly check the database in my VM and query number of records in the table. All the records are there. So it's safe to say that latency to have all messages pulled from the queue and inserted in SQL table is around the same time (i.e. duration to send to queue and few milliseconds later). With number of messages increasing the average duration (latency) is far less than a second to send a message to queue and then pick it up from queue to insert the message (data) into a table.
Throughput
Now 100000 messages would take about an 2 hours (extrapolate with measured data below) to be inserted in SQL Server table.
Usage Overview
Below you can see the number of messages going through the Service Bus Queue “registeredtimes”. There is a latency of a few minutes before actual number are shown (it is near real time).
Overall I am very impressed with this solution. Messages sent so quickly to a Service Bus queue and are then picked up by BizTalk Server 2010 R2 using the SB-Messaging adapter to store data in SQL Server 2008 R2. Image that I would like to register all the finishing times of a marathon with 50000 participants or all finishing times including passing the 5, 10, 15, 20, 25 ,30, 35 and 40 km mark. That would accumulate in 450.000 messages. This can be quite some time (a few hours, see throughput) so scaling out with more queues, tune SQL Server and BizTalk would be a way to shorten the time.
You could of course use a queue for 5, 10, 15, 20, 25 ,30, 35, 40 km and finish mark. Or you could use topics and subscriptions for this. In my next post I will discuss a solution that will support this scenario using queues and topics/subscriptions including some fine tuning of the BizTalk / SQL Server. Interesting to see the latency and throughput then.
Stay tuned!
Comments
Did you play around with the prefecth count property. I assume your not doing ordered delivery in BizTalk? if so setting the prefetch count would probably improve your throughput if messages were recieved in batches of 200 as an example.
What do you think?
Regards
Mike