A message queue has, in its architecture, two main points of failure; the first is the situation where a message is added to a queue, but never read (or at least not read within a specified period of time); this is called a Dead Letter, and it is the subject of this post. The second is where the message is corrupt, or it breaks the reading logic in some way; that is known as a Poison Message.
There are a number of reasons that a message might not get read in the specified time: the service reading and processing the messages might not be keeping up with the supply, it might have crashed, the network connection might have failed.
One possible thing to do at this stage, is to have a process that automatically notifies someone that a message has ended up in the dead letter queue.
Step One - specify a timeout
Here’s how you would specify a timeout on the message specifically:
BrokeredMessage message = new BrokeredMessage(messageBody)
{
MessageId = id,
TimeToLive = new TimeSpan(0, 5, 0)
};
Or, you can create a default on the queue from the QueueDescription (typically this would be done when you initially create the queue:
QueueDescription qd = new QueueDescription("TestQueue")
{
DefaultMessageTimeToLive = new TimeSpan(0, 5, 0)
};
nm.CreateQueue(qd);
Should these values differ, the shortest time will be taken.
What happens to the message by default?
I’ve added a message to the queue using the default timeout of 5 minutes; here it is happily sitting in the queue:
Looking at the properties of the queue, we can determine that the “TimeToLive” is, indeed, 5 minutes:
In addition, you can see that, by default, the flag telling Service Bus to move the message to a dead letter queue is not checked. This means that the message will not be moved to the dead letter queue.
5 Minutes later:
Nothing has happened to this queue, except time passing. The message has now been discarded. It seems an odd behaviour; however, as with ReadAndDelete Locks there may be reasons that this behaviour is required.
Step Two - Dead Letters
If you want to actually do something with the expired message, the key is a concept called “Dead Lettering”. The following code will direct the Service Bus to put the offending message into the “Dead Letter Queue”:
QueueDescription qd = new QueueDescription("TestQueue")
{
DefaultMessageTimeToLive = new TimeSpan(0, 5, 0),
EnableDeadLetteringOnMessageExpiration = true
};
nm.CreateQueue(qd);
Here’s the result for the same test:
Step Three - Doing something with this…
Okay - so the message hasn’t been processed, and it’s now sat in a queue specially designed for that kind of thing, so what can we do with it? One possible thing is to create a piece of software that monitors this queue. This is an adaptation of the code that I originally created here:
static void Main(string[] args)
{
System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
sw.Start();
if (!InitialiseClient())
{
Console.WriteLine("Unable to initialise client");
}
else
{
while (true)
{
string message = ReadMessage("TestQueue/$DeadLetterQueue");
if (string.IsNullOrWhiteSpace(message)) break;
Console.WriteLine($"{DateTime.Now}: Message received: {message}");
}
}
sw.Stop();
Console.WriteLine($"Done ({sw.Elapsed.TotalSeconds}) seconds");
Console.ReadLine();
}
private static bool InitialiseClient()
{
Uri uri = ServiceManagementHelper.GetServiceUri();
TokenProvider tokenProvider = ServiceManagementHelper.GetTokenProvider(uri);
NamespaceManager nm = new NamespaceManager(uri, tokenProvider);
return nm.QueueExists("TestQueue");
}
private static string ReadMessage(string queueName)
{
QueueClient client = QueueManagementHelper.GetQueueClient(queueName, true);
BrokeredMessage message = client.Receive();
if (message == null) return string.Empty;
string messageBody = message.GetBody<string>();
//message.Complete();
return messageBody;
}
If this was all that we had to monitor the queue, then somebody’s job would need to be to watch this application. That may make sense, depending on the nature of the business; however, we could simply notify the person in question that there’s a problem. Now, if only the internet had a concept of an offline messaging facility that works something akin to the postal service, only faster…
static void Main(string[] args)
{
System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
sw.Start();
if (!InitialiseClient())
{
Console.WriteLine("Unable to initialise client");
}
else
{
while (true)
{
string message = ReadMessage("TestQueue/$DeadLetterQueue");
if (string.IsNullOrWhiteSpace(message)) break;
Console.WriteLine($"{DateTime.Now}: Message received: {message}");
Console.WriteLine($"{DateTime.Now}: Send e-mail");
SendEmail(message);
}
}
sw.Stop();
Console.WriteLine($"Done ({sw.Elapsed.TotalSeconds}) seconds");
Console.ReadLine();
}
private static void SendEmail(string messageText)
{
System.Net.Mail.MailMessage message = new System.Net.Mail.MailMessage();
message.To.Add("[email protected]");
message.Subject = "Message in queue has expired";
message.From = new System.Net.Mail.MailAddress("[email protected]");
message.Body = messageText;
System.Net.Mail.SmtpClient smtp = new System.Net.Mail.SmtpClient("smtp.live.com");
smtp.Port = 587;
smtp.UseDefaultCredentials = false;
smtp.Credentials = new System.Net.NetworkCredential("[email protected]", "passw0rd");
smtp.EnableSsl = true;
smtp.Send(message);
}
In order to prevent a torrent of mails, you might want to put a delay in this code, or even maintain some kind of list so that you only send one mail per day.
References
https://www.codit.eu/blog/2015/01/automatically-expire-messages-in-azure-service-bus-how-it-works/
https://stackoverflow.com/questions/9851319/how-to-add-smtp-hotmail-account-to-send-mail