Tag Archives: Azure

Creating a Basic Azure Web Job

In this article, I discussed the use of Azure functions; however, Web Jobs perform a similar task. Azure Functions are effectively an abstraction on top of Web Jobs – meaning that, while you have more control when using Web Jobs, there’s a little more to do when writing them.

This article covers the basics of Web Jobs, and has a walk-through for creating a very simple task using one.

Create a new Web Job

Once you create this project, you’ll need to fill in the following values in the app.config:

<configuration>
  <connectionStrings>
    <!-- The format of the connection string is "DefaultEndpointsProtocol=https;AccountName=NAME;AccountKey=KEY" -->
    <!-- For local execution, the value can be set either in this config file or through environment variables -->
    <add name="AzureWebJobsDashboard" connectionString="" />
    <add name="AzureWebJobsStorage" connectionString="" />
  </connectionStrings>

These can both be the same value, but they refer to where Azure stores it’s data.

AzureWebJobsDashboard

This is the storage account used to store logs.

AzureWebJobsStorage

This is the storage account used to store whatever the application needs to function (for example: queues or tables). In the example below, it’s where the file will go.

Storage accounts can be set-up from the Azure dashboard (more on this later):

A Basic Application

For this example, let’s take a file from a blob storage and parse it, then write out the result in a log. Specifically, we’ll take an XML file, and write the number of nodes into a log; here’s the file:

<test>
    <myNode>
    </myNode>
    <myNode>
    </myNode>
</test>

I think we’ll probably be looking for a figure around 2.

Blob Storage

Before we can do anything with blob storage, we’ll need a new storage area; create a new storage account:

Set the storage kind to “General Storage” (because we’re working with files); other than that, go with your gut.

Uploading

Once you’ve created the account, you’ll need to add a file – otherwise nothing will happen. You can do this in the web portal, or you can do it via a desktop utility that Microsoft provide: Storage Explorer.

I kind of expected this to take me to the web page mentioned… but it doesn’t! You have to navigate there manually:

http://storageexplorer.com

Install it… unless you want to upload your file using the web portal… in which case: don’t.

We can create a new container:

Now, we can see the storage account and any containers:

Now, you can upload a file from here (remember that you can do all this inside the Portal):

Once you’ve created this, go back and update the storage connection string (described above). You may also want to repeat the process for a dashboard storage area (or, as stated above, they can be the same).

Programmatically Downloading

Now we have a file in the directory, it can be downloaded via the WebJob; here’s a function that will download a file:

        public static async Task<string> GetFileContents(string connectionString, string containerString, string fileName)
        {
            CloudStorageAccount storage = CloudStorageAccount.Parse(connectionString);
            CloudBlobClient client = storage.CreateCloudBlobClient();
            CloudBlobContainer container = client.GetContainerReference(containerString);
            CloudBlob blob = container.GetBlobReference(fileName);

            MemoryStream ms = new MemoryStream();
            await blob.DownloadToStreamAsync(ms);
            ms.Position = 0;

            StreamReader sr = new StreamReader(ms);
            string contents = sr.ReadToEnd();
            return contents;
        }

The code to call this is here (note the commented out commands from the default WebJob Template):

        static void Main()
        {
            Console.WriteLine("Starting");

            var config = new JobHostConfiguration();

            if (config.IsDevelopment)
            {
                config.UseDevelopmentSettings();
            }

            //var host = new JobHost();

            string fileContents = AzureHelpers.GetFileContents(config.StorageConnectionString, "testblob", "test.xml").Result;
            Console.WriteLine(fileContents);

            // The following code ensures that the WebJob will be running continuously
            //host.RunAndBlock();

            Console.WriteLine("Done");
        }

Although this works (sort of – it doesn’t check for new files, and it would need to be run on a scheduled basis – “On Demand” in Azure terms), you don’t need it (at least not for jobs that react to files being uploaded to storage containers). WebJobs provide this functionality out of the box! There are a number of decorators that you can use for various purposes:

  • string
  • TextReader
  • Stream
  • ICloudBlob
  • CloudBlockBlob
  • CloudPageBlob
  • CloudBlobContainer
  • CloudBlobDirectory
  • IEnumerable<CloudBlockBlob>
  • IEnumerable<CloudPageBlob>

Here, we’ll use a BlobTrigger and accept a string. Moreover, doing it this way makes the writing to the log much easier, as there’s injection of sorts (at least I’m assuming that’s what it’s doing). Here’s what the complete solution looks like in the new paradigm:

        public static void ProcessFile([BlobTrigger("testblob/{name}")] string fileContents, TextWriter log)
        {            
            XmlDocument xmlDoc = new XmlDocument();
            xmlDoc.LoadXml(fileContents);            
            log.WriteLine($"Node count: {xmlDoc.FirstChild.ChildNodes.Count}");
        }

The key thing to notice here is that the function is static and public (the class it’s in needs to be public, too – even is that’s the Program class). The WebJob framework uses reflection to work out which functions it needs to run.

The other point to note is that I’m getting the parameter as a string – the article above details what you could have it as; for example, if you wanted to delete it afterwards, you’d probably want to use an ICloudBlob or something similar.

Anyway, it works:

The log file

Remember the storage area that we specified for the dashboard earlier? You should now see some new containers created in that storage area:

This has created a number of directories, but the one that we’re interested in is “output-logs” in the “azure-webjobs-hosts” container:

And here’s the log itself:

References

https://docs.microsoft.com/en-us/azure/app-service-web/web-sites-create-web-jobs

https://stackoverflow.com/questions/36610952/azure-webjobs-vs-azure-functions-how-to-choose

https://stackoverflow.com/questions/27580264/where-do-i-get-the-azurewebjobsdashboard-connection-string-information

http://www.hanselman.com/blog/IntroducingWindowsAzureWebJobs.aspx

https://stackoverflow.com/questions/24286214/where-are-azure-webjobs-blobinput-and-bloboutput-classes

https://docs.microsoft.com/en-us/azure/app-service-web/websites-dotnet-webjobs-sdk-storage-blobs-how-to

Deploying an Azure Recommendation Service Using an ARM Template

Azure provides a number of AI services out of the box. The recommendation service is one of these, and it’s part of the Azure Cognitive Services.

Why?

Deploying a new service to Azure is quite straightforward; for recommendations, you Navigate to the Portal and select a new service:

Then you select the various options one by one, and finally, you create the resource.

But, what if you want to create this in development, and then in test, and then again in production, or what if you want to deploy it again multiple times? Although it’s straightforward, putting data in this kind of form is prone to error – and it’s time consuming.

ARM Templates

Azure allows you to create a template, and to create your resource based on that. There are a number of ways to do this; ultimately, it’s just a JSON document, so you could open up notepad and just type one.

Here’s how I created it initially:

Create a new resource group:

However, this doesn’t seem to give you too much out of the box (there are templates, but recommendations isn’t one of them):

Fortunately, you can reverse engineer the deployment that you’ve already made:

Downloading this gives you everything you need to re-deploy:

Running the template

So, now you’ve got a JSON file, how do you tell Azure what to do? Powershell seems to be the answer of choice (as it is for so many things these days) for Microsoft.

You’ll need to change the execution policy first:

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Unrestricted

Then run the script:

Success

When it works, you’ll get something like this:

And here’s the service:

Errors

It would be a gross exaggeration to say this worked straight away for me; here’s the errors that I encountered, and how I resolved them.

Resource Group Name is null

New-AzureRmResourceGroupDeployment : 18:23:28 – Error: Code=InvalidDeploymentParameterValue; Message=The value of deployment parameter ‘accounts_TestRecommendations_name’ is null. Please specify the value or use the parameter reference. See https://aka.ms/arm-deploy/#parameter-file for details.
At C:\Users\Paul\Downloads\ExportedTemplate-pcm-dev\deploy.ps1:104 char:5
+ New-AzureRmResourceGroupDeployment -ResourceGroupName $resourceGr …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [New-AzureRmResourceGroupDeployment], Exception
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.NewAzureResourceGroupDeploymentCmdlet

Resolution

This is caused because the parameter is set to null by default. Change parameters.json:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "accounts_TestRecommendations_name": {
            "value": "testRecommendations1"
        }
    }
}

No connection

Login-AzureRmAccount : The browser based authentication dialog failed to complete. Reason: The server or proxy was not found.
At C:\Users\Paul\Downloads\ExportedTemplate-pcm-dev\deploy.ps1:71 char:1
+ Login-AzureRmAccount;
+ ~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Add-AzureRmAccount], AadAuthenticationFailedException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Profile.AddAzureRMAccountCommand

Resolution

This is caused by not having a connection to Azure… so the resolution is to connect.

Invalid parameter value

C:\Users\Paul\Downloads\ExportedTemplate-pcm-dev\deploy.ps1 : Cannot retrieve the dynamic parameters for the cmdlet.
Error parsing boolean value. Path ‘parameters.accounts_TestRecommendations_name.value’, line 6, position 22.
At line:1 char:1
+ .\deploy.ps1
+ ~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [deploy.ps1], ParameterBindingException
+ FullyQualifiedErrorId : GetDynamicParametersException,deploy.ps1

Resolution

In my first attempt to resolve the first error, I specified a name without quotes; i.e.:

            "value": testRecommendations1

This seems to cause Azure to consider this a boolean; the fix is pretty straightforward once you’ve worked out what it’s actually saying:

            "value": "testRecommendations1"

Error

New-AzureRmResourceGroupDeployment : 07:58:51 – Resource Microsoft.CognitiveServices/accounts ‘testRecommendations1’
failed with message ‘{
“error”: {
“code”: “CanNotCreateMultipleFreeAccounts”,
“message”: “Operation failed. Only one free account is allowed for account type ‘Recommendations’.”
}
}’
At C:\Users\Paul\Downloads\ExportedTemplate-pcm-dev\deploy.ps1:104 char:5
+ New-AzureRmResourceGroupDeployment -ResourceGroupName $resourceGr …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [New-AzureRmResourceGroupDeployment], Exception
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.NewAzureResourceGroupDeploymentCmdlet

Resolution

This was caused because my account would only allow me to have a single recommendation service at any time. So the fix is to delete existing recommendation account:

References

https://blogs.endjin.com/2015/07/using-azure-resource-manager-and-powershell-dsc-to-create-and-provision-a-vm/

https://blogs.endjin.com/2016/01/azure-resource-manager-authentication-from-a-powershell-script/

Using Azure Functions to Send an E-mail Alert from a Service Bus

In this post, I discussed creating an Azure service bus that sends an e-mail as an action once a message has expired; and in this post, I covered Azure functions and setting a basic one up.

These two pieces of functionality seem to be crying out to be together. After all, if your functionality to send an e-mail is in the cloud, you don’t have to worry about your server being down (which, if your message has expired, is a real possibility).

Create the Azure Function

The first thing to do is to create the Azure function to send an e-mail. Remember that we’ll be hooking into the service bus, and so we’ll create the function a little differently.

The first few steps are the same, though:

The new function is here:

We’ll create a custom function again:

Although this looks familiar from the last post, the next part does differ slightly. This time, we’ll set up a Service Bus Trigger:

This requires the connection string to your service bus…

As you can see above, the service bus connection is blank, and there are no possible entries… onto App Settings:

App Settings

On the App Settings tab, you can configure settings that pertain to your Azure Function App. Select “Manage App Settings”. Here we can set-up a connection string:

Now, we should be able to see that from the Function:

Does it work?

What does this function do out of the box?

Well, having populated the queue with 50 messages that time out after 30 seconds, the function kicked in and started logging that it was picking up messages after 30 seconds – so that’s a promising sign!

The messages are processed and removed from the dead letter queue. This process happens so quickly, it’s easy (as I did) to interpret this as a bug (i.e. messages are not being dead-lettered). However, as we can see from the function logs – they are.

This did, however, leave me with a concern that the messages were being disposed of before they had been successfully processed. To check this, I changed the function slightly:

So, it crashes correctly:

And here, safe and sound, are 50 freshly dead-lettered messages:

Function Code

Now we have a function, we need to make it send an e-mail… so we’ll need some code. Let’s start with what we created here.


using System;
using System.Threading.Tasks;
using System.Net.Mail;

public static void Run(string myQueueItem, TraceWriter log)
{
    log.Info($"Start C# ServiceBus queue trigger function processed message: {myQueueItem}");

    System.Net.Mail.MailMessage message = new System.Net.Mail.MailMessage();
    message.To.Add("to.address@hotmail.co.uk");
    message.Subject = "Message in queue has expired";
    message.From = new System.Net.Mail.MailAddress("from.address@hotmail.co.uk");
    message.Body = messageText;
    System.Net.Mail.SmtpClient smtp = new System.Net.Mail.SmtpClient("smtp.live.com");
    smtp.Port = 587;
    smtp.UseDefaultCredentials = false;
    smtp.Credentials = new System.Net.NetworkCredential("my.address@hotmail.co.uk", "p@ssw0rd");
    smtp.EnableSsl = true;
    smtp.Send(message);

    log.Info($"End C# ServiceBus queue trigger function processed message: {myQueueItem}");
}


This doesn’t work:

2017-06-27T16:47:56.928 Function started (Id=1188dbdb-4963-4e55-af5c-4be1f71a1ca5)
2017-06-27T16:47:56.928 Start C# ServiceBus queue trigger function processed message: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA32
2017-06-27T16:47:56.928 Function completed (Failure, Id=1188dbdb-4963-4e55-af5c-4be1f71a1ca5, Duration=0ms)
2017-06-27T16:47:57.147 Exception while executing function: Functions.ServiceBusQueueTriggerCSharp1. mscorlib: Exception has been thrown by the target of an invocation. f-ServiceBusQueueTriggerCSharp1__-1971403142: Cannot complete.
2017-06-27T16:47:57.557 Exception while executing function: Functions.ServiceBusQueueTriggerCSharp1. mscorlib: Exception has been thrown by the target of an invocation. f-ServiceBusQueueTriggerCSharp1__-1971403142: Cannot complete.

Debugging Azure

A quick side note on debugging Azure. There are a number of resources with details of how this should work on the web, and I’ll probably have a later post of my own experiences, but it’s a pretty flaky experience, and I ended up using trial and error to determine the issue.

Working code

using System;
using System.Threading.Tasks;

public static void Run(string myQueueItem, TraceWriter log)
{
    log.Info($"Start C# ServiceBus queue trigger function processed message: {myQueueItem}");

    System.Net.Mail.MailMessage message = new System.Net.Mail.MailMessage();
    
    message.To.Add("to.address@hotmail.co.uk");    
    message.Subject = "Message in queue has expired";    
    message.From = new System.Net.Mail.MailAddress("from.address@hotmail.co.uk");
    message.Body = myQueueItem;
        
    System.Net.Mail.SmtpClient smtp = new System.Net.Mail.SmtpClient("smtp.live.com");
    smtp.Port = 587;
    smtp.UseDefaultCredentials = false;
    smtp.Credentials = new System.Net.NetworkCredential("my.address@hotmail.co.uk", "p@ssw0rd");
    smtp.EnableSsl = true;
    smtp.Send(message);

    log.Info($"End C# ServiceBus queue trigger function processed message: {myQueueItem}");
}

So, the problem was just that I was referencing an unknown variable (messageText). I’m unsure exactly why I needed to travel to the mountains of Mordor to determine this – a simple error message in the online text would have sufficed.

The other issue that I faced was a security challenge; however, once I’d persuaded Azure that this really was me, everything sprung into life:

Credit Considerations

Unlike in previous posts where I’ve identified the Azure cost to be negligible, functions are the fastest way to use up credit I have found so far. Especially functions such as I’ve created here. I left the (non-working) function above active, but failing all night, and it used up over £40 worth of credit, continually trying, and failing, to process the dead-letter queue… I think the lights might even have dimmed in Redmond for a split second! The moral of the story is is: be careful when you’re debugging this – you can’t just leave at the end of the night with a function that doesn’t work, but is still active.

Summary

This concept is extremely compelling. I can have a service bus queue that is processed and monitored by an Azure function. If aliens land and steal the entire office, all the servers, dev PCs and programmers, this function will continue to run. There is obviously a mindset shift here, and it doesn’t make sense to move everything into this kind of model, but consider the possibilities; imagine a system that books holidays: it processes the customer request and adds it to a queue; the aeroplane booking system picks that from the queue and books the ticket on the plane, the car hire system takes the message to book a car, once they’re all complete they add respective messages to say so (but remain agnostic of each other), finally, if any one part of the system fails, an Azure function could sit there monitoring and cancel the whole lot. I’ve never worked in this kind of industry, so there’s a lot that I’ve probably not considered, but the essence is that you can have active functionality on (even catastrophic) failure – which is a brand new concept.

References

https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus

https://stackoverflow.com/questions/10043219/view-content-of-an-azure-service-bus-queue

Service Bus Explorer:

https://code.msdn.microsoft.com/Service-Bus-Explorer-f2abca5a

http://markheath.net/post/remote-debugging-azure-functions

Sending e-mails:

https://stackoverflow.com/questions/25216202/smtp-live-com-mailbox-unavailable-the-server-response-was-5-7-3-requested-ac

Azure Functions

Azure functions are Microsoft’s answer to “serverless” architecture. The concept behind Serverless Architecture being that you can create service functionality, but you don’t need to worry about a server. Obviously, there is one: it’s not magic; it’s just not your problem.

How?

Let’s start by creating a new Azure function app:

Once created, search “All resources”; you might need to give it a minute or two:

Next, it asks you to pick function type. In this case, we’re going to pick “Custom function”:

Azure then displays a plethora of options for creating your function. We’re going to go for “Generic Webhook” (and name it):

A Webhook is a http callback; meaning that you can use them in the same way as you would any other HTTP service.

This creates your function (with some default code):

We’ll leave the default code, and run it (because you can’t go wrong with default code – it always does exactly what you need… assuming what you need is what it does):

The right hand panel shows the output from the function. Which means that the function works; so, we now have a web based function that works… well… says hello world (ish). How do we call it?

Using the function

The function has an allocated URL:

Given that we have a service, and a connection URL; the rest is pretty straightforward. Let’s try to connect from a console application:

        static void Main(string[] args)
        {
            HttpClient client = new HttpClient();
            string url = "https://pcm-test.azurewebsites.net/api/pcm_GenericWebhookCSharp1?code=Kk2397soUoaK7hbxQa6qUSMV2S/AvLCvjn508ujAJMMZiita5TsjkQ==";

            var inputObject = new
            {
                first = "pcm-Test-input-first",
                last = "pcm-Test-input-last"
            };
            string param = JsonConvert.SerializeObject(inputObject);
            HttpContent content = new StringContent(param, Encoding.UTF8, "application/json");

            HttpResponseMessage response = client.PostAsync(url, content).Result;
            string results = response.Content.ReadAsStringAsync().Result;

            Console.WriteLine($"results: {results}");
            Console.ReadLine();
        }
    }

When run, this returns:

Conclusion

Let’s think about what we’ve just done here: we have set up a service, connected to that service from a remote source and returned data. Now, let’s think about what we haven’t done: any configuration; that is, other than clicking “Create Function”.

This “serverless” architecture seems to be the nth degree of SOA. If I wish, I can create one of these functions for each of the server activities in my application, they are available to anything with an internet connection. It then becomes Microsoft’s problem if my new website suddenly takes off and millions of people are trying to access it.

References

http://robertmayer.se/2016/04/19/azure-function-app-to-send-emails/

http://www.c-sharpcorner.com/article/azure-functions-create-generic-webhook-trigger/

Azure Service Bus – Send an e-mail on Message Timeout

A message queue has, in its architecture, two main points of failure; the first is the situation where a message is added to a queue, but never read (or at least not read within a specified period of time); this is called a Dead Letter, and it is the subject of this post. The second is where the message is corrupt, or it breaks the reading logic in some way; that is known as a Poison Message.

There are a number of reasons that a message might not get read in the specified time: the service reading and processing the messages might not be keeping up with the supply, it might have crashed, the network connection might have failed.

One possible thing to do at this stage, is to have a process that automatically notifies someone that a message has ended up in the dead letter queue.

Step One – specify a timeout

Here’s how you would specify a timeout on the message specifically:

           BrokeredMessage message = new BrokeredMessage(messageBody)
            {
                MessageId = id,
                TimeToLive = new TimeSpan(0, 5, 0)
            };

Or, you can create a default on the queue from the QueueDescription (typically this would be done when you initially create the queue:

                QueueDescription qd = new QueueDescription("TestQueue")
                {
                    DefaultMessageTimeToLive = new TimeSpan(0, 5, 0)
                };
                nm.CreateQueue(qd);

Should these values differ, the shortest time will be taken.

What happens to the message by default?

I’ve added a message to the queue using the default timeout of 5 minutes; here it is happily sitting in the queue:

Looking at the properties of the queue, we can determine that the “TimeToLive” is, indeed, 5 minutes:

In addition, you can see that, by default, the flag telling Service Bus to move the message to a dead letter queue is not checked. This means that the message will not be moved to the dead letter queue.

5 Minutes later:

Nothing has happened to this queue, except time passing. The message has now been discarded. It seems an odd behaviour; however, as with ReadAndDelete Locks there may be reasons that this behaviour is required.

Step Two – Dead Letters

If you want to actually do something with the expired message, the key is a concept called “Dead Lettering”. The following code will direct the Service Bus to put the offending message into the “Dead Letter Queue”:


                QueueDescription qd = new QueueDescription("TestQueue")
                {
                    DefaultMessageTimeToLive = new TimeSpan(0, 5, 0),
                    EnableDeadLetteringOnMessageExpiration = true
                };
                nm.CreateQueue(qd);

Here’s the result for the same test:

Step Three – Doing something with this…

Okay – so the message hasn’t been processed, and it’s now sat in a queue specially designed for that kind of thing, so what can we do with it? One possible thing is to create a piece of software that monitors this queue. This is an adaptation of the code that I originally created here:

        static void Main(string[] args)
        {
            System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
            sw.Start();

            if (!InitialiseClient())
            {
                Console.WriteLine("Unable to initialise client");
            }
            else
            {
                while (true)
                {
                    string message = ReadMessage("TestQueue/$DeadLetterQueue");

                    if (string.IsNullOrWhiteSpace(message)) break;
                    Console.WriteLine($"{DateTime.Now}: Message received: {message}");
                }
            }

            sw.Stop();
            Console.WriteLine($"Done ({sw.Elapsed.TotalSeconds}) seconds");
            Console.ReadLine();
        }

        private static bool InitialiseClient()
        {
            Uri uri = ServiceManagementHelper.GetServiceUri();
            TokenProvider tokenProvider = ServiceManagementHelper.GetTokenProvider(uri);

            NamespaceManager nm = new NamespaceManager(uri, tokenProvider);
            return nm.QueueExists("TestQueue");
        }

        private static string ReadMessage(string queueName)
        {
            QueueClient client = QueueManagementHelper.GetQueueClient(queueName, true);

            BrokeredMessage message = client.Receive();
            if (message == null) return string.Empty;
            string messageBody = message.GetBody<string>();

            //message.Complete();

            return messageBody;
        }

If this was all that we had to monitor the queue, then somebody’s job would need to be to watch this application. That may make sense, depending on the nature of the business; however, we could simply notify the person in question that there’s a problem. Now, if only the internet had a concept of an offline messaging facility that works something akin to the postal service, only faster…

        static void Main(string[] args)
        {
            System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
            sw.Start();

            if (!InitialiseClient())
            {
                Console.WriteLine("Unable to initialise client");
            }
            else
            {
                while (true)
                {
                    string message = ReadMessage("TestQueue/$DeadLetterQueue");

                    if (string.IsNullOrWhiteSpace(message)) break;
                    Console.WriteLine($"{DateTime.Now}: Message received: {message}");

                    Console.WriteLine($"{DateTime.Now}: Send e-mail");
                    SendEmail(message);
                }
            }

            sw.Stop();
            Console.WriteLine($"Done ({sw.Elapsed.TotalSeconds}) seconds");
            Console.ReadLine();
        }

        private static void SendEmail(string messageText)
        {
            System.Net.Mail.MailMessage message = new System.Net.Mail.MailMessage();
            message.To.Add("notification.address@hotmail.co.uk");
            message.Subject = "Message in queue has expired";
            message.From = new System.Net.Mail.MailAddress("my.address@hotmail.co.uk");
            message.Body = messageText;
            System.Net.Mail.SmtpClient smtp = new System.Net.Mail.SmtpClient("smtp.live.com");
            smtp.Port = 587;
            smtp.UseDefaultCredentials = false;
            smtp.Credentials = new System.Net.NetworkCredential("my.address@hotmail.co.uk", "passw0rd");
            smtp.EnableSsl = true;
            smtp.Send(message);
        }

In order to prevent a torrent of mails, you might want to put a delay in this code, or even maintain some kind of list so that you only send one mail per day.

References

https://docs.microsoft.com/en-us/dotnet/api/microsoft.servicebus.messaging.queuedescription.enabledeadletteringonmessageexpiration?view=azureservicebus-4.0.0#Microsoft_ServiceBus_Messaging_QueueDescription_EnableDeadLetteringOnMessageExpiration

https://www.codit.eu/blog/2015/01/automatically-expire-messages-in-azure-service-bus-how-it-works/

https://stackoverflow.com/questions/9851319/how-to-add-smtp-hotmail-account-to-send-mail

Implicitly Acknowledging a Message from Azure Service Bus

In this post I discussed receiving, processing and acknowledging a message using the Azure Service Bus. There are two ways to acknowledge a message received from the queue (which are common to all message broker systems that I’ve used so far). That is, you either take the message, process it, and then go back to the broker to tell it you’re done (explicit acknowledgement); or, you remove the queue and then process it (implicit acknowledgement).

Explicit Acknowledgement / PeekLock

If the message is not processed within a period of time, then it will be unlocked and returned to the queue to be picked up by the next client request.

The code for this is as follows (it is also the default behaviour in Azure Service Bus):

QueueClient queueClient = QueueClient.CreateFromConnectionString(connectionString, queueName, ReceiveMode.PeekLock);

Remember that, with this code, if you don’t call:

message.Complete();

Then you will repeatedly read the same message over and over again.

Implicit Acknowledgement / ReadAndDelete

Here, if the message is not processed within a period of time, or fails to process, then it is likely lost. So, why would you ever use this method of acknowledgement? Well, speed is the main reason; because you don’t need to go back to the server, you potentially increase the whole transaction speed; furthermore, there is clearly work involved for the broker in maintaining the state of a message on the queue, expiring the message lock, etc.

The code for the implicit acknowledgement is:

QueueClient queueClient = QueueClient.CreateFromConnectionString(connectionString, queueName, ReceiveMode.ReceiveAndDelete);

References

https://docs.microsoft.com/en-us/rest/api/servicebus/peek-lock-message-non-destructive-read

Reading a Message From an Azure Service Bus Queue

In this post. I documented how to create a new application using Azure Service Bus and add a message to the queue. In this post, I’ll cover how to read that post from the queue, and how to deal with acknowledging the receipt.

The Code

The code from this post can be found here.

The code uses a lot of hard coded strings and static methods, and this is because it makes it easier to see exactly what it happening and when. This is not intended as an example of production code, more as a cut-and-paste repository.

Reading a Message

Most of the code that we’ve written can simply be re-hashed for the receipt. First, initialise the queue as before:

            Uri uri = ServiceManagementHelper.GetServiceUri();
            TokenProvider tokenProvider = ServiceManagementHelper.GetTokenProvider(uri);

            NamespaceManager nm = new NamespaceManager(uri, tokenProvider);
            if (!nm.QueueExists("TestQueue")) return;

Obviously, if the queue doesn’t exist for reading, there’s limited point in creating it. The next step is to set-up a queue client:

            string connectionString = GetConnectionString();

            QueueClient queueClient = QueueClient.CreateFromConnectionString(connectionString, queueName);

            return queueClient;

The connection string is found here:

Finally, ask for the next message:

            BrokeredMessage message = queueClient.Receive();
            string messageBody = message.GetBody<string>();
            Console.WriteLine($"Message received: {messageBody}");

And we can see the contents of the queue:

If we run again:

We can see that, despite being read, the message is still sat in the queue:

Acknowledging the Message

To explicitly acknowledge a message, just calling the Complete method on the message object will work:

            BrokeredMessage message = queueClient.Receive();
            string messageBody = message.GetBody<string>();

            message.Complete();

            Console.WriteLine($"Message received: {messageBody}");

And the message is gone:

Summary and Cost

We now have a basic, working, message queue. But one thing that I always worry about with Azure is how much this costs. Let’s run a send and receive for 100 messages with the content: “test” as above.

The first thing is to change the code slightly so that it reads through all messages (not just the first):

                while (true)
                {
                    string message = ReadMessage("TestQueue");

                    if (string.IsNullOrWhiteSpace(message)) break;
                    Console.WriteLine($"Message received: {message}");
                }
        private static string ReadMessage(string queueName)
        {
            QueueClient client = QueueManagementHelper.GetQueueClient(queueName);

            BrokeredMessage message = client.Receive();
            if (message == null) return string.Empty;
            string messageBody = message.GetBody<string>();

            message.Complete();

            return messageBody;
        }

Then run this to clear the queue. By default, client.Receive has a default timeout, so it will pause for a few seconds before returning if there are no messages. This timeout is a very useful feature. Most of this post was written on a train with a flaky internet connection, and this mechanism provided a resilient way to allow communications to continue when the connection was available.

And change the send code:


            string message = Console.ReadLine();

            for (int i = 1; i <= 100; i++)
            {
                AddNewMessage("1", message, "TestQueue");
            }

Next, the current credit on my account:

Let’s run 100 messages:

That looks familiar. Let’s try 10,000:

I’ve added some times to this, too. It’s processing around 10 / second – which is not astoundingly quick. It’s worth mentioning again that this post was written largely on a train, but still, 10 messages per second means that 10K messages will take around 15 mins. It is faster when you have a reliable (non-mobile) internet connection, but still. Anyway, back to cost. 10K messages still showed up as a zero cost.

But, Azure is a paid service, so this has to start costing money. This time, I’m adding 1000 character string to send as a message, and sending that 100,000 times.

After this, the balance was the same; however, the following day, it dropped slightly to £36.94. So, as far as I can tell, the balance is updated based on some kind of job that runs each day (which means that the balance might not be updated in real-time).

I asked this question here.

The published pricing details are here, but it looks like you should be able to post around 500,000 messages before you start incurring cost (1M operations).

References

https://insidethecpu.com/2015/11/06/levaraging-azure-service-bus-with-c/

https://www.simple-talk.com/cloud/cloud-data/an-introduction-to-windows-azure-service-bus-brokered-messaging/

https://msdn.microsoft.com/en-gb/library/hh868041.aspx?f=255&MSPPError=-2147217396

https://stackoverflow.com/questions/14831281/how-does-the-service-bus-queueclient-receive-work

A C# Programmer’s Guide to Queues and Sending a Message with Azure Service Bus

I have previously written about message queue systems. The big two, as far as I can see, are Active MQ and RabbitMQ.

Microsoft have always had MSMQ*, but it’s not really a message broker as such (I believe that you can get similar behaviour using NServiceBus, but have never tried that myself). However, with Azure comes the Azure Service Bus.

The first thing that you need to do is set-up an Azure account. Note that Microsoft offer Azure as a paid service, and so this is not free. However, they also offer free trials and free Azure credit if you have an MSDN.

Log on to:

https://portal.azure.com

Namespace

Namespaces are an important concept in Azure. They basically allow you to split a single Azure account across many functions, but what that means is that everything you do relates to a specific namespace.

To add one, first, pick a pricing tier:

Make sure that your Namepsace isn’t taken:

You’ll then get an alert to say it worked:

If you refresh, you should now see your namespace:

Create Test Project

I always try to start with a console app when trying new stuf. Add NuGet reference:

It is my understanding that, as with ActiveMQ and RabbitMQ, these client libraries are an abstraction over a set of HTTP Post calls. In the case of Azure, I believe that, behind the scenes, it uses WCF to handle all this.

Using the Namespace

Using a message queue system such as RabbitMQ or ActiveMQ, you need a message queue server, and a URL that relates to it. However, one of the things Azure allows you to do is to abstract that; for example:

        static void Main(string[] args)
        {
            Console.WriteLine($"Getting service bus URI...");
            Uri uri = ServiceBusEnvironment.CreateAccessControlUri("pcm-servicebustest");
            Console.WriteLine($"Service Bus URI: {uri.ToString()}");
            Console.ReadLine();
        }

Tells me what the URI of the message queue broker is:

Adding a message to a queue

In order to do anything with a message queue in Azure, you need a token; effectively, this provides a level of security

Tokens

Get the key:

You can store these details in the app/web.config, or you can use them programmatically:

        private static TokenProvider GetTokenProvider(Uri uri)
        {
            Console.WriteLine($"Getting token...");
            TokenProvider tp = TokenProvider.CreateSharedAccessSignatureTokenProvider("RootManageSharedAccessKey", "JWh82nkstIAi4w5tW6MEj7GKQfoiZlwBYjHx9wfDqdA=");                                                

            Console.WriteLine($"Token {tp.ToString()}");
            return tp;
        }

Queues

Putting the above calls together, we can now create a queue in Azure:

        private static void CreateNewQueue(Uri uri, TokenProvider tokenProvider)
        {
            Console.WriteLine($"Creating new queue...");
            NamespaceManager nm = new NamespaceManager(uri, tokenProvider);

            Console.WriteLine($"Created namespace manager for {nm.Address}");
            if (nm.QueueExists("TestQueue"))
            {
                Console.WriteLine("Queue already exists");
            }
            else
            {
                Console.WriteLine("Creating new queue");
                QueueDescription qd = nm.CreateQueue("TestQueue");
            }
        }

Incidentally, the act of creating a queue appears to have cost £0.24 GBP. If you have MSDN, you should get £40 GBP credit each month (at the time of writing).

Now we have a queue, let’s put some messages on it.

Adding a message

        private static void AddNewMessage(string id, string messageBody, string queueName)
        {
            BrokeredMessage message = new BrokeredMessage(messageBody)
            {
                MessageId = id
            };

            string connectionString = GetConnectionString();
            
            QueueClient queueClient = QueueClient.CreateFromConnectionString(connectionString, queueName);
            queueClient.Send(message);
        }

The Connection String can be found here:

We can now see that a message has, indeed, been added to the queue:

At this time, this is about as much as you can see from this portal.

Errors

These are some errors that I encountered during the creation of this post, and their solutions.

System.UnauthorizedAccessException

System.UnauthorizedAccessException: ‘The token provider was unable to provide a security token while accessing ‘https://pcm-servicebustest-sb.accesscontrol.windows.net/WRAPv0.9/’. Token provider returned message: ‘The remote name could not be resolved: ‘pcm-servicebustest-sb.accesscontrol.windows.net”.’

The cause is not an invalid secret

That’s because this line:

TokenProvider tp = TokenProvider.CreateSharedSecretTokenProvider("RootManageSharedAccessKey", "jjdsjdsjk");

Gives the error:

System.ArgumentException: ‘The ‘issuerSecret’ is invalid.’

The fix…

This code is littered throughout the web:

TokenProvider tp = TokenProvider.CreateSharedSecretTokenProvider("RootManageSharedAccessKey", "jjdsjdsjk");

But the correct code was:

TokenProvider tp = TokenProvider.CreateSharedAccessSignatureTokenProvider("RootManageSharedAccessKey", "JWh82nkstIAi4w5tW6MEj7GKQfoiZlwBYjHx9wfDqdA=");                                                

System.ArgumentNullException: ‘Queue name should be specified as EntityPath in connectionString.’

Or: 40400: Endpoint not found.

Microsoft.ServiceBus.Messaging.MessagingEntityNotFoundException: ‘40400: Endpoint not found., Resource:sb://pcm-servicebustest.servicebus.windows.net/atestqueue. TrackingId:48de75d7-fb01-4fa9-b72e-20a5dc090a8d_G11, SystemTracker:pcm-servicebustest.servicebus.windows.net:aTestQueue, Timestamp:5/25/2017 5:23:27 PM

Means (obviously) that the following code:

QueueClient.CreateFromConnectionString(connectionString, queueName);

Either doesn’t have the queue name, or it is wrong.

References

https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-messaging-exceptions

https://blogs.msdn.microsoft.com/brunoterkaly/2014/08/07/learn-how-to-create-a-queue-place-and-read-a-message-using-azure-service-bus-queues-in-5-minutes/

https://stackoverflow.com/questions/18558299/servicebus-throws-401-unauthorized-error

https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-queues-topics-subscriptions

https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-how-to-use-topics-subscriptions

https://msdn.microsoft.com/en-us/library/jj542433.aspx?f=255&MSPPError=-2147217396

https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-multi-tier-app-using-service-bus-queues

https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-get-started-with-queues

* Microsoft probably haven’t ALWAYS had MSMQ. There was probably a time in the early 90’s where they didn’t have a message queue system at all.

Console Application Builds, But Will Not Run

While doing some testing recently, I created a new bog standard console application and, on pressing F5, nothing happened.

The project builds fine, but wouldn’t launch the console window.

Why (and how to fix)?

Well, I had installed the Azure Service Bus Client. Other than that, I can’t really say; however, the fix does kind of make sense:

Uncheck the “Prefer 32-bit” checkbox, and it all springs back to life!

Microsoft Cognitive Services – Text Recognition

Recently at DDD North I saw a talk on MS cognitive services. This came back and sparked interest in me while I was looking at some TFS APIs (see later posts for why). However, in this post, I’m basically exploring what can be done with these services.

The Hype

  • Language: can detect the language that you pass
  • Topics: can determine the topic being discussed
  • Key Phrases: key points (which I believe may equate to nouns)
  • Sentiment: whether or not what you are saying is good or bad (I must admit, I don’t really understand that – but we can try some phrases to see what it comes up with)

For some reason that I can’t really understand, topics requires over 100 documents, and so I won’t be getting that to work, as I don’t have a text sample big enough. The examples that they give in marketing seem to relate to people booking and reviewing holidays; and it feels a lot like these services are overly skewed toward that particular purpose.

Set-up

Register here:

https://www.microsoft.com/cognitive-services/

Registration is free (although I believe you need a live account).

cog1

Client

The internal name for this at MS is Project Oxford. You don’t have to install the client libraries (because they are just service calls), but you get some objects and helpers if you do:

cog2

Cognition

The following code is largely plagiarised from the links at the bottom of this page:

Here’s the Main function:

var requestDocs = PopulateDocuments();
Console.WriteLine($"-=Requests=-");
foreach (var eachReq in requestDocs.Documents)
{
    Console.WriteLine($"Id: {eachReq.Id} Text: {eachReq.Text}");
}
Console.WriteLine($"-=End Requests=-");
 
string req = JsonConvert.SerializeObject(requestDocs);
 
MakeRequests(req);
Console.WriteLine("Hit ENTER to exit...");
Console.ReadLine();

PopulateDocuments just fills the RequestDocument collection with some test data:


private static LanguageRequest PopulateDocuments()
{
    LanguageRequest requestText = new Microsoft.ProjectOxford.Text.Language.LanguageRequest();
    requestText.Documents.Add(
        new Microsoft.ProjectOxford.Text.Core.Document()
        { Id = "One", Text = "The quick brown fox jumped over the hedge" });
    requestText.Documents.Add(
        new Microsoft.ProjectOxford.Text.Core.Document()
        { Id = "Two", Text = "March is a green month" });
    requestText.Documents.Add(
        new Microsoft.ProjectOxford.Text.Core.Document()
        { Id = "Three", Text = "When I press enter the program crashes" });
    requestText.Documents.Add(
        new Microsoft.ProjectOxford.Text.Core.Document()
        { Id = "4", Text = "Pressing return - the program crashes" });
    requestText.Documents.Add(
        new Microsoft.ProjectOxford.Text.Core.Document()
        { Id = "5", Text = "Los siento, no hablo Enspanol" });
 
    return requestText;
}

As you can see, I dropped some Spanish in there for the language detection. The MakeRequests method and its dependencies:


static async void MakeRequests(string req)
{
    using (var client = new HttpClient())
    {
        client.BaseAddress = new Uri(BaseUrl);
 
        // Request headers.
        client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", AccountKey);
        client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
 
        // Request body. Insert your text data here in JSON format.
        byte[] byteData = Encoding.UTF8.GetBytes(req);
 
        // Detect key phrases:
        var uri = "text/analytics/v2.0/keyPhrases";
        string response = await CallEndpoint(client, uri, byteData);
        Console.WriteLine("Key Phrases");
        Console.WriteLine(ParseResponseGeneric(response));
 
        // Detect language:
        var queryString = HttpUtility.ParseQueryString(string.Empty);
        queryString["numberOfLanguagesToDetect"] = NumLanguages.ToString(CultureInfo.InvariantCulture);
        uri = "text/analytics/v2.0/languages?" + queryString;
        response = await CallEndpoint(client, uri, byteData);
        Console.WriteLine("Detect language");
        Console.WriteLine(ParseResponseLanguage(response));
 
        // Detect topic:
        queryString = HttpUtility.ParseQueryString(string.Empty);
        queryString["minimumNumberOfDocuments"] = "1";
        uri = "text/analytics/v2.0/topics?" + queryString;
        response = await CallEndpoint(client, uri, byteData);
        Console.WriteLine("Detect topic");
        Console.WriteLine(ParseResponseGeneric(response));
 
        // Detect sentiment:
        uri = "text/analytics/v2.0/sentiment";
        response = await CallEndpoint(client, uri, byteData);
        Console.WriteLine("Detect sentiment");
        Console.WriteLine(ParseResponseSentiment(response));
    }
}
private static string ParseResponseSentiment(string response)
{
    if (!string.IsNullOrWhiteSpace(response))
    {
        SentimentResponse resp = JsonConvert.DeserializeObject<SentimentResponse>(response);
        string returnVal = string.Empty;
 
        foreach (var doc in resp.Documents)
        {
            returnVal += Environment.NewLine +
                $"Sentiment: {doc.Id}, Score: {doc.Score}";
        }
 
        return returnVal;
    }
 
    return null;
}
 
private static string ParseResponseLanguage(string response)
{
    if (!string.IsNullOrWhiteSpace(response))
    {
        LanguageResponse resp = JsonConvert.DeserializeObject<LanguageResponse>(response);
        string returnVal = string.Empty;
        foreach(var doc in resp.Documents)
        {
            var detectedLanguage = doc.DetectedLanguages.OrderByDescending(l => l.Score).First();
            returnVal += Environment.NewLine +
                $"Id: {doc.Id}, " +
                $"Language: {detectedLanguage.Name}, " +
                $"Score: {detectedLanguage.Score}";
        }
        return returnVal;
    }
 
    return null;
}
 
private static string ParseResponseGeneric(string response)
{            
    if (!string.IsNullOrWhiteSpace(response))
    {
        return Environment.NewLine + response;                
    }
 
    return null;
}

The subscription key is given when you register (in the screen under “Set-up”). Keep an eye on the requests, too: 5000 seems like a lot, but when you’re testing, you might find you get through them faster than you expect.

Here’s the output:

cog3

Evaluation

So, the 5 phrases that I used were:

The quick brown fox jumped over the hedge

This is a basic sentence indicating an action.

The KeyPhrases API decided that the key points here were “hedge” and “quick brown fox”. It didn’t think that “jumped” was key to this sentence.

The Language API successfully worked out that it’s written in English.

The Sentiment API thought that this was a slightly negative statement.

March is a green month

This was a nonsense statement, but in a valid sentence structure.

The KeyPhrases API identified “green month” as being important, but not March.

The Language API successfully worked out that it’s written in English.

The Sentiment API thought this was a very positive statement.

When I press enter the program crashes

Again, a completely valid sentence, and with a view to my idea ultimate idea for this API.

The KeyPhrases API spotted “program crashes”, but not why. I found this interesting because it seems to conflict with the other phrases, which seemed to identify nouns only.

Again, the Language API knew this was English.

The sentiment API identified that this was a negative statement… which I think I agree with.

Pressing return – the program crashes

The idea here was, it’s basically the same sentence as above, but phrased differently.

The KeyPhrases API wasn’t fooled, and returned the same key phrase – this is good.

Still English, according to the Language API.

This is identified as a negative statement again, but oddly, not as negative as the previous one.

Los siento, no hablo Enspanol

I threw in a Spanish phrase because I felt the Language API hadn’t had much of a run.

The KeyPhrase API pulled out “hablo Espanol”, which based on my very rudimentary Spanish, means the opposite of that was said.

It was correctly identified as Spanish by the Language API.

The Sentiment API identified it as the most negative statement. Perhaps because it has the word “sorry” and “no” in it?

References

Sample code:

https://text-analytics-demo.azurewebsites.net/Home/SampleCode

https://elbruno.com/2016/04/13/cognitiveservices-text-analytics-api-new-operation-detect-key-topics-in-documents/

https://mrfoxsql.wordpress.com/2016/09/13/azure-cognitive-services-apis-with-sql-server-2016-clr/