This is the third and final post in a short series on linking up Azure with GCP (for Christmas). In the first post, I set-up a basic Azure function that updated some data in table storage, and then in the second post, I configured the GCP link from PubSub into BigQuery.
In the post, we’ll square this off by adapting the Azure function to post a message directly to PubSub; then, we’ll call the Azure function with Santa’a data, and watch that appear in BigQuery. At least, that was my plan - but Microsoft had other ideas.
It turns out that Azure functions have a dependency on Newtonsoft Json 9.0.1, and the GCP client libraries require 10+. So instead of being a 10 minute job on Boxing day to link the two, it turned into a mammoth task. Obviously, I spent the first few hours searching for a way around this - surely other people have faced this, and there’s a redirect, setting, or way of banging the keyboard that makes it work? Turns out not.
The next idea was to experiment with contacting the Google server directly, as is described here. Unfortunately, you still need the Auth libraries.
Finally, I swapped out the function for a WebJob. WebJobs give you a little move flexibility, and have no hard dependencies. So, on with the show (albeit a little more involved than expected).
WebJob
In this post I described how to create a basic WebJob. Here, we’re going to do something similar. In our case, we’re going to listen for an Azure Service Bus Message, and then update the Azure Storage table (as described in the previous post), and call out to GCP to publish a message to PubSub.
Handling a Service Bus Message
We weren’t originally going to take this approach, but I found that WebJobs play much nicer with a Service Bus message, than with trying to get them to fire on a specific endpoint. In terms of scaleability, adding a queue in the middle can only be a good thing. We’ll square off the contactable endpoint at the end with a function that will simply convert the endpoint to a message on the queue. Here’s what the WebJob Program looks like:
public static void ProcessQueueMessage(
[ServiceBusTrigger("localsantaqueue")] string message,
TextWriter log,
[Table("Delivery")] ICollector<TableItem> outputTable)
{
Console.WriteLine("test");
log.WriteLine(message);
// parse query parameter
TableItem item = Newtonsoft.Json.JsonConvert.DeserializeObject<TableItem>(message);
if (string.IsNullOrWhiteSpace(item.PartitionKey)) item.PartitionKey = item.childName.First().ToString();
if (string.IsNullOrWhiteSpace(item.RowKey)) item.RowKey = item.childName;
outputTable.Add(item);
GCPHelper.AddMessageToPubSub(item).GetAwaiter().GetResult();
log.WriteLine("DeliveryComplete Finished");
}
Effectively, this is the same logic as the function (obviously, we now have the GCPHelper, and we’ll come to that in a minute. First, here’s the code for the TableItem model:
[JsonObject(MemberSerialization.OptIn)]
public class TableItem : TableEntity
{
[JsonProperty]
public string childName { get; set; }
[JsonProperty]
public string present { get; set; }
}
As you can see, we need to decorate the members with specific serialisation instructions. The reason being that this model is being used by both GCP (which only needs what you see on the screen) and Azure (which needs the inherited properties).
GCPHelper
As described here, you’ll need to install the client package for GCP into the Azure Function App that we created in post one of this series (referenced above):
Install-Package Google.Cloud.PubSub.V1 -Pre
Here’s the helper code that I mentioned:
public static class GCPHelper
{
public static async Task AddMessageToPubSub(TableItem toSend)
{
string jsonMsg = Newtonsoft.Json.JsonConvert.SerializeObject(toSend);
Environment.SetEnvironmentVariable(
"GOOGLE\_APPLICATION\_CREDENTIALS",
Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "Test-Project-8d8d83hs4hd.json"));
GrpcEnvironment.SetLogger(new ConsoleLogger());
PublisherClient publisher = PublisherClient.Create();
string projectId = "test-project-123456";
TopicName topicName = new TopicName(projectId, "test");
SimplePublisher simplePublisher =
await SimplePublisher.CreateAsync(topicName);
string messageId =
await simplePublisher.PublishAsync(jsonMsg);
await simplePublisher.ShutdownAsync(TimeSpan.FromSeconds(15));
}
}
I detailed in this post how to create a credentials file; you’ll need to do that to allow the WebJob to be authorised. The Json file referenced above was created using that process.
Azure Config
You’ll need to create an Azure message queue (I’ve called mine localsantaqueue):
I would also download the Service Bus Explorer (I’ll be using it later for testing).
GCP Config
We already have a DataFlow, a PubSub Topic and a BigQuery Database, so GCP should require no further configuration; except to ensure the permissions are correct.
The Service Account user (which I give more details of here needs to have PubSub permissions. For now, we’ll make them an editor, although in this instance, they probably only need publish:
Test
We can do a quick test using the Service Bus Explorer and publish a message to the queue:
The ultimate test is that we can then see this in the BigQuery Table:
Lastly, the Function
This won’t be a completely function free post. The last step is to create a function that adds a message to the queue:
[FunctionName("Function1")]
public static HttpResponseMessage Run(
[HttpTrigger(AuthorizationLevel.Function, "post")]HttpRequestMessage req,
TraceWriter log,
[ServiceBus("localsantaqueue")] ICollector<string> queue)
{
log.Info("C# HTTP trigger function processed a request.");
var parameters = req.GetQueryNameValuePairs();
string childName = parameters.First(a => a.Key == "childName").Value;
string present = parameters.First(a => a.Key == "present").Value;
string json = "{{ 'childName': '{childName}', 'present': '{present}' }} ";
queue.Add(json);
return req.CreateResponse(HttpStatusCode.OK);
}
So now we have an endpoint for our imaginary Xamarin app to call into.
Summary
Both GCP and Azure are relatively immature platforms for this kind of interaction. The GCP client libraries seem to be missing functionality (and GCP is still heavily weighted away from .Net). The Azure libraries (especially functions) seem to be in a pickle, too - with strange dependencies that makes it very difficult to communicate outside of Azure. As a result, this task (which should have taken an hour or so) took a great deal of time, and it was completely unnecessary.
Having said that, it is clearly possible to link the two systems, if a little long-winded.
References
https://blog.falafel.com/rest-google-cloud-pubsub-with-oauth/
https://github.com/Azure/azure-functions-vs-build-sdk/issues/107
https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus