Real World Azure: April 4th Iselin, NJ

On Wednesday April 4th, I will be presenting “Real World Azure: Migrating e-Commerce to Azure”, a case study reflecting on my experiences leading the team that migrated www.plccenter.com from on-premise to Azure.

Abstract:

Jim Priestley, will review his team’s experiences migrating www.plccenter.com from an on premise ASP.Net 2.0 application to an Azure hosted MVC 3 application that now serves over 30,000 unique visitors daily. The session will include details of the implementation including storage, performance, payment processing considerations, and tuning SQL Azure for high speed searching.

The registration link is thru meetup.com at http://www.meetup.com/NJAzureUserGroup/events/57552512/

Joining Microsoft!

On March 12th, I embarked on a new journey…

I’ve joined Microsoft as an Azure Technical Solution Professional in the Mid-Atlantic District.

In this role I work with enterprise customers across multiple segments to help them leverage Windows Azure in their organizations.

This blog will remain a place for me to share my thoughts and code snippets.

This blog continues to be my personal opinion and does not represent my current or past employer’s views in any way.

Jim

Reading CRM 2011 RemoteExecutionContext from Azure Service Bus Brokered Messages

Summary – This article demonstrates using Windows Azure Service Bus SDK 1.6 to read Brokered Messages using “PeekLock” to ensure the “At-Least-Once” delivery pattern. The key to this process is understanding that the default CRM 2011 Plug-In for Azure Service Bus will wrap the RemoteExecutionContext in a Soap Envelope, so the standard binary de-serialization method in Azure SDK Sample will not work for these messages.

Getting CRM 2011 Messages onto the Azure Service Bus Queue

In my previous article, Connecting CRM 2011 to Azure Service Bus Queues, I reviewed how to set up Service Bus using the Windows Azure Management Portal, and walked through configuring CRM 2011 Online, to send a message to the Service Bus each time a new Contact is created in CRM. This article will show how to read those messages from the Queue and de-serialize them back into the CRM objects.

Creating a Simple Queue Listener

We will create a simple console app to read all the messages from the Queue, output some properties to the Console and then exit. In a production application, you would add exception management and logging, security key secret storage, and perhaps host this in a Windows service.

1. Using Visual Studio 2010, create a new c# console application project.

2. Edit the project properties and change the Target Framework to “Full 4.0 Framework”, this is important, as the Azure Service Bus SDK 1.6 relies on System.Servicemodel.Web which is not included in the “Client” framework that is selected by default for a console application.

3. Use the NuGet Package Manager Console to install the latest Azure Service Bus SDK“ using the command “Install-Package WindowsAzure.ServiceBus”  (NuGet is a package management tool for Visual Studio, for more information see this post by Scott Hanselman, or go to http://NuGet.org )

4. Add a reference to Microsoft.Xrm.Sdk from the CRM SDK.

5. Add the needed Using Statements to the class.

 

6. For this sample, I am using constants for Namespace, Issuer, Key, and QueueName. THIS NOT BEST PRACTICES and should NEVER be done in production code. See this article, MSDN Magazine - "Crypto Services and Data Security in Windows Azure" ,for guidance on key management.

 

7. Create Queue Client

 

8. Read Each Message until queue is empty. (Note: The max timeout for open connections is one minute, we are using 10 seconds here.) This loop will read each message, and then exit when there are no messages after a 10 second wait.

 

9. Process each message. Note that the content type is XML Stream, not the binary object for the messages that the CRM Online Plug-In places in the Queue.

 

In the sample below, note the DecodeBrokeredMessage function. Using TextMessageEncodingBindingElement from the System.Servicemodel.Channels library, we create an encoder. The body of our BrokeredMessage is an XML stream, containing a Message containing our CRM RemoteExecutionContext object.. With the encoder, we convert the Stream to a Message, and then cast the body of the Message back to the RemoteExecutionContext.

 

The RemoteExecutionContext object is the standard CRM object that is wrapped around any Messages sent to the Queue. It contains the EntityName, EntityID and CRM Action (Create, Update Delete) that triggered the sending of the object. It also contains the CRM Entity, and can be configured to contain the Pre-Execution, Post-Execution or both snapshots from the CRM save pipeline. For more information on using the RemoteExecutionContext in your integration solution, refer to the SDK Documentation. 

Connecting CRM 2011 to Azure Service Bus Queues

Summary – This article describes how to send messages from Microsoft CRM 2011 to Microsoft Azure Service Bus Topics and Queues, using CRM SDK 5.08 and Azure SDK 1.6

Configuring Azure Service Bus Queues, Topics and Subscriptions

Azure Account

An Azure account is required to use Service Bus Queues, Topics and Subscriptions. Information is available at https://www.windowsazure.com on creating an account and pricing. Free trial accounts, and special free accounts for MSDN subscribers are also available.

Create a Service Bus Namespace

Once you have created an Azure account, you will need to create a Service Bus Namespace. In the Azure Management Portal select “Service Bus, Access Control and Caching” on the lower left side of the screen, then expand “Services” at the top left, and choose “Service Bus”. Any existing Service Bus Namespaces you have provisioned will be listed in the main window of the portal. Select “New” on the ribbon bar and you will be presented with the pop-up window below:

You need to choose a unique name for your Namespace and then press “Check Availability”, then choose an Azure datacenter to host your Service Bus, and associate it to one of your Azure subscriptions for billing purposes.

Queues, Topics and Subscriptions

In late 2011, Queues, Topics and Subscriptions went live on Windows Azure Service Bus. A Service Bus is a message-oriented middleware pattern that facilitates building loosely-coupled distributed applications. Azure Queues support reliable queuing of messages for later consumption. Azure Topics are an extension of Queues specifically intended to support the publish/subscribe pattern. These queuing and pub/sub support asynchronous de-coupling and load-balancing scenarios in distributed systems.

Queues are First In, First Out durable message containers. Multiple consumers can connect to the same Queue to create a load-balanced “competing consumer” pattern. In general, queued messages are expected to be delivered in the order received, and only once.

Queues support two methods for consumers to read messages. In “ReceiveAndDelete” mode, the message is deleted from the Queue as soon as the consumer reads it. In “PeekLock” mode, the message is hidden from other consumers after being read by the consumer. The consumer must explicitly mark the message “Complete” or “Abandon” within the lock timeout period configured on the connection to the Queue. If the message lock timeout is reached without a “Complete”, or the consumer calls “Abandon” then the message is returned to the Queue. “PeekLock” processing is often called “At Least Once” processing. In systems where it is critical that every message is processed, this is a good pattern to follow because faults in the consumer will re-queue the message for another attempt. It is important when using this pattern that the client have proper compensation processes to handle possible duplicate or out of sequence messages. The “MessageID” property on Service Bus messages can be used to track which messages have been processed in the consumer.

When using “PeekLock” with “competing consumer” it is critical that all messages and consumer processes are idempotent, as one consumer may abandon a message back to the queue while another consumer completes the message after it, and a third consumer later re-attempts the first message after the second has completed.

Topics differ from Queues in that they offer a one to many “publish/subscribe” pattern for the messages. Messages are written to a Topic the same way they are written to Queues. The difference is that Topics are write-only. Subscriptions are setup in Azure through the portal or code, and the Subscription is where the consumer connects to read the messages. Each Subscription represents a unique path for the distribution of messages. Subscriptions can also be set with filters so they only receive a subset of the messages.

Subscriptions support both “ReceiveAndDelete” and “PeekLock” consumers just like Queues, and all the same rules regarding competing consumers apply to each Subscription.

An example of when to use Subscriptions instead of Queues: Suppose you have a CRM installation configured to send messages to a Service Bus Queue each time a change is made to an Account entity, and then a listener application on Azure reads the message and updates a customer portal database in SQL Azure. The Queue model would support scale-out by having multiple consumers reading the messages from the Queue using the competing consumer pattern. Later, you decide that you would also like to post a copy of each message to a data warehouse. Suppose the portal application needs the data to update your website as soon as possible, but the data warehouse only needs an hourly update. You could add an additional Subscription to the Topic receiving the Account entity messages. The new Subscription would receive a durable copy of each message from that point forward that is posted to the the Topic.

Topics must have at least one Subscription. You will receive an error if you post a message to a Topic with no Subscriptions.

Setting Up a New Queue or Topic

To set up a Queue in Azure, click on Service Bus under Services in the portal menu. Expand the drop-down next to your Azure Subscription name and you will see the new Service Bus Namespace that we created above.

Clicking “New Queue”, on the ribbon, while the Queues list is highlighted will pop-up the Queue configuration window:

Clicking “New Topic” on the ribbon will pop-up the Topic configuration window:

Note how the options related to reading messages are not on the Topic as it is a write-only object.

As noted above, a Topic must have an active Subscription to receive messages. Click “New Subscription” while the Topic is highlighted to create one:

This is what the portal looks like after creating a Queue, Topic and Subscription:

Configuring CRM to Send Messages to an Azure Service Bus Queue

Using the Service Bus Plug-In for Microsoft CRM 2011

Microsoft provides a plug-in for CRM that will send messages to a Service Bus Queue or Topic. Messages sent to Service Bus using the Microsoft provided plug-in use Windows Azure Access Control Service (ACS) for claims-based authentication to the Queue. The configuration requires that the CRM server has a valid publicly signed certificate for ACS to trust the CRM server as an authorized client. There are directions on how to configure the certificate for an on-premise CRM 2011 installation here. For this walk-through, I will use a trial account for CRM 2011 Online available here. CRM Online provides a public certificate that we can use for ACS.

ACS 2.0 and Microsoft CRM 2011

The CRM SDK documentation provides a walkthrough for configuring ACS 1.0 to work with CRM. We created new Queues, in this walkthrough, after the release of ACS 2.0, so ACS 2.0 was chosen automatically as the security model for our Namespace by the Azure Management Portal. This SDK article which describes how to register an Azure-aware plug-in actually also contains the details on how to use the Plug-In registration tool to configure ACS.

Here is a summary of the steps:

1. Download and unzip the CRM 2011 SDK.
2. Install the correct version of Windows Identity Foundation for your version of Windows.
3. Build the Plug-In Registration tool using Visual Studio 2010. The solution file is located at “sdk\tools\pluginregistration\pluginregistrationtool.sln”.
4. Run the Plug-in Registration Tool.
5. Click “Create New Connection”.
6. Open CRM 2011 in your browser.
7. Go to Settings->Customizations->Developer Resources this page contains all the CRM Discovery Service URL, Organization Unique Name, and a link to download the certificate you will need to configure ACS.
8. In the Plug-In tool, name the connection in the Label field, key in the Discovery URL from CRM, and enter your CRM User Name. (Windows for on-prem or Live ID for Online) then choose connect.
9. Double-click on the organization name so that the right side panel of the Tool loads your Organization.
10. In the tab for you Organization, click Register->Register New Service Endpoint.
11. The key fields here are Solution Namespace = to our Azure Namespace, Path = to the name of our Queue or Topic, and Queue as the Contract Type.
12. Click “Save & Configure ACS”. The ACS Configuration window will open.
13. Enter the Management Key from your Azure Management Portal. (Click on the NameSpace, then click View for the Default Key. Click Copy to Clipboard in the pop-up window, and then paste the value into the ACS Configuration window of the Plug-in tool.)
14. If using CRM Online, click Download Certificate on the Developer Resources page. Then upload the certificate into the ACS Configuration window.
15. For CRM Online, the certificate Issuer Name is crm.dynamics.com. Enter your certificate issuer name in to the ACS Configuration window and then click Configure ACS.
16. If you have configured it correctly, there will be a log displaying the changes that the Plug-in tool has made to your ACS on Azure. Close the ACS Configuration window.
17. Click Save & Verify Authentication on the Service Endpoint Details Window.
18. You should see this window indicating that you have successfully authorized CRM to post to the Queue. Click Close.
19. Click Save on the Service Endpoint Details window and it will close. You should now see something similar to this in the main window of the Plug-in Tool:
20. Click the Service Endpoint so that it is highlighted, then click Register->Register New Step. You will see the Register New Step window.
21. CRM uses message types for Create, Update, Delete, etc. To send all Create messages for the Contact entity, Type “Create” in the Message field and “Contact” in the Primary Entity field. Note that the Event Handler is defaulted to our Service Bus.

Confirming that CRM is Sending Messages to Service Bus

CRM uses an Asynchronous task to send messages to Azure Service Bus. It can take up to 10 minutes for a message to post to the Queue depending upon how busy your server is. To confirm that the messages are being sent:

1. Open CRM and Navigate to the Contacts list
2. Fill in the required fields and click Save and Close.
3. Open Setting->System Jobs

If the message was sent you should see something similar to the following:

If the job fails, the notes of the system job object will contain the stack trace.

Once you are comfortable that your messages are flowing correctly, you can go back to the Plug-in Tool, and check the box “Delete Async Operation if Status Code = Successful” this will purge the job from the System Jobs table if successful.

Confirming the Messages are in the Service Bus Queue

From the Management Portal for Azure, you cannot directly see the message data in your Queue, but the Portal will list the length of the Queue in Messages and Bytes.

1. Open the Azure Management Portal
2. Click on your Queue in the Service Bus Area
3. Scroll the Properties Window on the right until you see the Queue Length field

The number listed is the number of Messages in your Queue. If you already had the Portal open when you sent the message from CRM, you may to hit Refresh on the ribbon to get the Queue Length to update.

Reading Messages from the Queue and De-Serializing the CRM Entity

The sample provided in the CRM SDK works in ReadAndDelete mode, but does not seem to work in PeekLock mode.

PeekLock requires the use of the BrokeredMessage object, rather than the Message object used in the CRM SDK sample.

Using the samples provided with the Azure SDK, I have created a listener using PeekLock, but the object returned from the Brokered Message is an XML Stream instead of the expected IExecutionContext.

I’ll provide a sample that is able to de-serialize the XML Stream back to IExecutionContext, and then extract the CRM entity, in my next post.

Technorati Tags: Azure,Service Bus,CRM

Using the CRM 2011 Tracing Service to Capture Plugin Exception Details

Summary – This article describes using the CRM 2011 Tracing Service to log exception details from within a custom plugin. This method works for both on-premise and cloud hosted deployments of CRM 2011.

What is the CRM Tracing Service

Microsoft has updated the process for tracking exceptions in CRM 2011. The new tracing service can be used inside the Execute method of a custom CRM 2011 plug in. The Tracing Service will log all of the Tracing Messages that are sent from the custom plugin.  In the event the plugin throws an unhandled exception, the Tracing Service will publish the messages it has received to the appropriate error handler routine.  For synchronous plugins, the error messages are returned to the user in the error dialog in the CRM client. For asynchronous plugins, the error messages are posted in the details of the system job log, which is available for Administrators to review. The service

How to use the CRM Tracing Service

Using the Tracing Service is a fairly simple process.  The Execute method of the customer plugin class receives a parameter of type IServiceProvider. From this parameter, a reference to the Tracing Service can be obtained as follows:

 ITracingService tracingService = (ITracingService)
     serviceProvider.GetService(typeof(ITracingService));

Passing messages to the Tracing Service is done by calling the Trace method:

tracingService.Trace("ErrorMessage");

Limitations using the CRM Tracing Service

It is important to note that the messages passed to the Tracing Service are only reported to the user or logged, if the plugin throws an unhandled exception. If the Execute method of the plugin terminates normally, the Trace messages are all discarded.

In a future post I will document creating a custom logging Entity in CRM that will allow the posting of durable Trace messages from custom plugins.