Some time ago, I was working on a project using WCF RPC calls. We discovered that of the calls that we made, around 10 - 20% were failing. Once we discovered this, I implemented a retry policy to try and cope with the flaky network. I didn’t realise at the time (or maybe it didn’t exist at the time) that there’s a library out there that does that for you: Polly.
To test this out, let’s create a bog standard API Web Project; from Powershell create a directory; e.g.:
mkdir PollyDemo
Then create a Web API project:
dotnet new webapi -n UnreliableApi
Let’s add a client project console app:
static async Task Main(string[] args)
{
HttpClient httpClient = new HttpClient()
{
BaseAddress = new Uri("https://localhost:44368")
};
var result = await httpClient.GetAsync("weatherforecast");
if (result.IsSuccessStatusCode)
{
var output = await result.Content.ReadAsStringAsync();
Console.WriteLine(output);
}
}
This should call the API fine (using the default API project you get a weather forecaster). Let’s now introduce a level of entropy into the system. In the server, let’s add the following code:
static Random \_rnd = new Random();
[HttpGet]
public IEnumerable<WeatherForecast> Get()
{
// 1 in 3 chance of an error
if (\_rnd.Next(3) == 1)
{
throw new Exception("Bad stuff happened here");
}
var rng = new Random();
return Enumerable.Range(1, 5).Select(index => new WeatherForecast
{
Date = DateTime.Now.AddDays(index),
TemperatureC = rng.Next(-20, 55),
Summary = Summaries[rng.Next(Summaries.Length)]
})
.ToArray();
}
Now, if you run the client a few times, you’ll see that it’s occasionally failing (actually, from this screenshot, all you can really see if that it didn’t succeed):
We know that this error is transient - just needs an F5, so we can put some code into the client to handle this:
for (int i = 1; i <= 3; i++)
{
var result = await httpClient.GetAsync("weatherforecast");
if (result.IsSuccessStatusCode)
{
var output = await result.Content.ReadAsStringAsync();
Console.WriteLine(output);
break;
}
else
{
Console.WriteLine("Bad things happened... ");
}
}
Console.WriteLine("If we get here, we either succeeded or gave up ");
And we’re back in business:
The only problem here is that our code is suddenly a little cumbersome; we have a foreach loop around the code, and we’re risking each person implementing this doesn’t introduce a bug by not breaking when successful or whatever.
Before we introduce Polly as the solution, it’s probably worth mentioning that, whilst some errors can legitimately be fixed by simply trying again, in other cases this can be incredibly harmful. You should ensure that your code is idempotent. Imagine you’re creating a sales order on this call, and it fails after writing something to the database - you risk creating multiple sales orders!
Polly
Polly is basically a library built around this concept. It’s a NuGet package, so import it from here into your client. Let’s start by refactoring our code to call the service:
private static async Task<bool> GetTheWeather(HttpClient httpClient)
{
var result = await httpClient.GetAsync("weatherforecast");
if (result.IsSuccessStatusCode)
{
var output = await result.Content.ReadAsStringAsync();
Console.WriteLine(output);
return true;
}
else
{
Console.WriteLine("Bad things happened... ");
return false;
}
}
Now let’s call that using Polly:
static async Task Main(string[] args)
{
//https://localhost:44368/weatherforecast
HttpClient httpClient = new HttpClient()
{
BaseAddress = new Uri("https://localhost:44368")
};
var retryPolicy = Policy
.HandleResult<bool>(false)
.RetryAsync(3, (response, retryCount) =>
{
Console.WriteLine($"Received a response of {response.Result}, retrying {retryCount}.");
});
var result = await retryPolicy.ExecuteAsync(() => GetTheWeather(httpClient));
Console.WriteLine("If we get here, we either succeeded or gave up ");
}
All we’re doing here is telling Polly that we want a policy that acts on a return value of False (in theory, I imagine you could set this to something like .HandleResult(“Aardvark”) and have it retry while the method returned a value of Aardvark). RetryAsync sounds pretty obvious, but the Async part is very important, otherwise, you won’t be able to use ExecuteAsync… and you’ll spend a while wondering why (so I hear)!
ExecuteAsync is awaitable, so it will wrap the retry logic in this single line.
The advantage here is that you can define a retry policy for your application, or several retry policies for your application.