Tag Archives: Recommendations

Debugging Recommendations Engine

Here I wrote about how to set-up and configure the MS Azure recommendations engine.

One thing that has become painfully apparent while working with recommendations is how difficult it is to work out what has gone wrong when you don’t get any recommendations. The following is a handy check-list for the next time this happens to me… so others may, or may not find this useful*:

1. Check the model was correctly generated

Once you have produced a recommendations model, you can access that model by simply navigating to it. The url is in the following format:

{recommendations uri}/ui

For example:

https://pcmrecasd4zzx2asdf.azurewebsites.net/ui

This gives you a screen such as this:

The status (listed in the centre of the screen) tells you whether the build has finished and, if so, whether it succeeded or not.

If the build has failed, you can select that row and drill into, and find out why.

In the following example, there is a reference in the usage data, to an item that is not in the catalogue.

Other reasons that the model build may fail include invalid, corrupt or missing data in either file.

2. Check the recommendation in the interface

In order to exclude other factors in your code, you can manually interrogate the model directly by simply clicking on the “Score” link above; you will be presented with a screen such as this:

In here, you can request direct recommendations to see how the model behaves.

3. Volume

If you find that your score is consistently returning as zero, then the issue may be with the volume of usage data that you have provided. 1k** rows of usage data is the sort of volume you should be dealing with; this statistic was based on a catalogue of around 20 – 30 products.

4. Distribution

The number of users matters – for the above figures, a minimum of 15** users was necessary to get any scores back. If the data sample is across too small a user base, it won’t return anything.

Footnotes

* Although this post is written by me, and is for my benefit, I stole much of its content from wiser work colleagues.

** Arbitrary values – your mileage may vary.

Azure Recommendations

Azure provides a number of pre-configured machine learning services out of the box. One of these (still in Beta at the time of writing this) is Recommendations. The idea being that it will try to work out what, given a list of items, you would prefer, based knowledge about your habits. There’s a lot of information on line about this, but briefly, it can work out what your preference is based a combination of your past activity, and the past activity of others that have shown an interest in the same item.

Obviously, the “items” could be products, films, aardvarks, or sheep; Azure doesn’t know anything about the content of what its recommending; if you’ve bought* “A” in the past, and 75% of everyone else that has bought “A” has also bought “B” then there’s a chance you’ll want “B”. “A” could be an apple, and “B” could be a pair of sunglasses; so obviously, you need to be careful about the data that you feed it.

Recommendations API

The first thing to note is that the Recommendations API represents an earlier attempt to implement this by Microsoft, and is due to be discontinued early next year (2018). If you try to use this to follow any of the online tutorials then you’ll get into a world of hurt.

Deploy Recommendations

The new method of creating a recommendations service is via a wizard (which, I believe behind the scenes, builds a custom ARM template). This is the start of the deployment, and gives you a screen similar to the following (once you’ve logged in):

As you can see, there’s clearly some re-branding in progress here; anyway, complete the form and create the service.

Another thing that has changed in the new version of this is that the free pricing tier has disappeared:

After a few screens, it starts the deployment process:

The next screen that is displayed shows all the connection strings and keys in one handy reference:

… they are just below this screenshot.

This should create four separate services:

Sample Project

Microsoft provides a sample project that should work out of the box (they actually provide more than one – some of which work better than others). This one uses AutoRest, but there’s another referenced at the bottom of this post.

In this project, open Recommendations.Sample.Program.cs and, at the top of the main function, enter the details that you noted after the creation of your service. If you didn’t note them, then you can still find them. You’ll notice that four separate services were created: AppService, StorageAccount, App Insights and an App Service Plan.

recommendationsEndPointUri

Is found in the URL of the App Service:

apiAdminKey

Is found in the Application Settings of the App Service:

connectionString

Is the connection string of the storage account:

If you run this now, you should find that it will process and score the recommendations:

So, for example, we can see from the results above that people that bought DHF-01159 are recommended to buy DHF-01055 (although it doesn’t seem very convinced).

Footnotes

* The term that Azure uses here is “Purchase”. Different actions have different weighting (configurable), but by default, you would assume that buying something is more important than, for example, clicking on it (“Click” is another action). These actions can mean anything you choose; in the sheep example above, “Purchase” might mean shearing, and “Click” might mean photographing.

References

http://pmichaels.net/2017/08/06/deploying-azure-recommendation-service-using-arm-template/

https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-overview-of-diagnostic-logs

https://docs.microsoft.com/en-us/azure/cognitive-services/cognitive-services-recommendations-ui-intro

https://gallery.cortanaintelligence.com/Tutorial/Recommendations-Solution

https://github.com/Microsoft/Cognitive-Recommendations-Windows

Deploying an Azure Recommendation Service Using an ARM Template

Azure provides a number of AI services out of the box. The recommendation service is one of these, and it’s part of the Azure Cognitive Services.

Why?

Deploying a new service to Azure is quite straightforward; for recommendations, you Navigate to the Portal and select a new service:

Then you select the various options one by one, and finally, you create the resource.

But, what if you want to create this in development, and then in test, and then again in production, or what if you want to deploy it again multiple times? Although it’s straightforward, putting data in this kind of form is prone to error – and it’s time consuming.

ARM Templates

Azure allows you to create a template, and to create your resource based on that. There are a number of ways to do this; ultimately, it’s just a JSON document, so you could open up notepad and just type one.

Here’s how I created it initially:

Create a new resource group:

However, this doesn’t seem to give you too much out of the box (there are templates, but recommendations isn’t one of them):

Fortunately, you can reverse engineer the deployment that you’ve already made:

Downloading this gives you everything you need to re-deploy:

Running the template

So, now you’ve got a JSON file, how do you tell Azure what to do? Powershell seems to be the answer of choice (as it is for so many things these days) for Microsoft.

You’ll need to change the execution policy first:

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Unrestricted

Then run the script:

Success

When it works, you’ll get something like this:

And here’s the service:

Errors

It would be a gross exaggeration to say this worked straight away for me; here’s the errors that I encountered, and how I resolved them.

Resource Group Name is null

New-AzureRmResourceGroupDeployment : 18:23:28 – Error: Code=InvalidDeploymentParameterValue; Message=The value of deployment parameter ‘accounts_TestRecommendations_name’ is null. Please specify the value or use the parameter reference. See https://aka.ms/arm-deploy/#parameter-file for details.
At C:\Users\Paul\Downloads\ExportedTemplate-pcm-dev\deploy.ps1:104 char:5
+ New-AzureRmResourceGroupDeployment -ResourceGroupName $resourceGr …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [New-AzureRmResourceGroupDeployment], Exception
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.NewAzureResourceGroupDeploymentCmdlet

Resolution

This is caused because the parameter is set to null by default. Change parameters.json:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "accounts_TestRecommendations_name": {
            "value": "testRecommendations1"
        }
    }
}

No connection

Login-AzureRmAccount : The browser based authentication dialog failed to complete. Reason: The server or proxy was not found.
At C:\Users\Paul\Downloads\ExportedTemplate-pcm-dev\deploy.ps1:71 char:1
+ Login-AzureRmAccount;
+ ~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Add-AzureRmAccount], AadAuthenticationFailedException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Profile.AddAzureRMAccountCommand

Resolution

This is caused by not having a connection to Azure… so the resolution is to connect.

Invalid parameter value

C:\Users\Paul\Downloads\ExportedTemplate-pcm-dev\deploy.ps1 : Cannot retrieve the dynamic parameters for the cmdlet.
Error parsing boolean value. Path ‘parameters.accounts_TestRecommendations_name.value’, line 6, position 22.
At line:1 char:1
+ .\deploy.ps1
+ ~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [deploy.ps1], ParameterBindingException
+ FullyQualifiedErrorId : GetDynamicParametersException,deploy.ps1

Resolution

In my first attempt to resolve the first error, I specified a name without quotes; i.e.:

            "value": testRecommendations1

This seems to cause Azure to consider this a boolean; the fix is pretty straightforward once you’ve worked out what it’s actually saying:

            "value": "testRecommendations1"

Error

New-AzureRmResourceGroupDeployment : 07:58:51 – Resource Microsoft.CognitiveServices/accounts ‘testRecommendations1’
failed with message ‘{
“error”: {
“code”: “CanNotCreateMultipleFreeAccounts”,
“message”: “Operation failed. Only one free account is allowed for account type ‘Recommendations’.”
}
}’
At C:\Users\Paul\Downloads\ExportedTemplate-pcm-dev\deploy.ps1:104 char:5
+ New-AzureRmResourceGroupDeployment -ResourceGroupName $resourceGr …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [New-AzureRmResourceGroupDeployment], Exception
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.NewAzureResourceGroupDeploymentCmdlet

Resolution

This was caused because my account would only allow me to have a single recommendation service at any time. So the fix is to delete existing recommendation account:

References

https://blogs.endjin.com/2015/07/using-azure-resource-manager-and-powershell-dsc-to-create-and-provision-a-vm/

https://blogs.endjin.com/2016/01/azure-resource-manager-authentication-from-a-powershell-script/