Tag Archives: .net

Short Walks – Running an Extension Method on a Null Item

I came across this issue recently, and realised that I didn’t fully understand extension methods. My previous understanding was that an extension method was simply added to the original class (possible in the same manner that weavers work); However, a construct similar to the following code changed my opinion:

class Program
{
    static void Main(string[] args)
    {
        var myList = GetList();            
        var newList = myList.Where(
            a => a.IsKosher());
        var evaluateList = newList.ToList();
 
        foreach(var a in evaluateList)
        {
            Console.WriteLine(a.Testing);
        }
    }
 
    static IEnumerable<TestClass> GetList()
    {
        return new List<TestClass>()
        {
            new TestClass() {Testing = "123"},
            null
        };
    }
}
 
public class TestClass
{
    public string Testing { get; set; }
}
 
public static class ExtensionTest
{
    public static bool IsKosher(this TestClass testClass)
    {
        return (!string.IsNullOrWhiteSpace(testClass.Testing));
    }
}

As you can see from the code, GetList() returns a null class in the collection. If you run this code, you’ll find that it crashes inside the extension method, because testClass is null.

A note on Linq

If you’re investigating this in the wild, you might find it particularly difficult because of the was that Linq works. Even though the call to the extension method is on the line above, the code doesn’t get run until you actually use the result (in this case, via a ToList()).

New understanding

As I now understand it, extension methods are simply a nice syntactical way to use a static method. That is, had I simply declared my IsKosher method as a standard static method, it would behave exactly the same. To verify this, let’s have a look at the IL; here’s the IL for my function above:

IL Code for extension method

And here’s the IL for the same function as a standard static method:

IL code for static method

The only difference is the line at the top of the extension method calling the ExtensionAttribute constructor.

References

https://stackoverflow.com/questions/847209/in-c-what-happens-when-you-call-an-extension-method-on-a-null-object

Short Walks – Using CompilerService Arguments in an Interface

Until today, I thought that the following code would work:

class Program
{
    static void Main(string[] args)
    {
        ITest test = new Test();
        test.Log("testing");
        Console.ReadLine();
    }
}
 
interface ITest
{
    void Log(string text, string function = "");
}
 
class Test : ITest
{
    public void Log(string text, [CallerMemberName] string function = "")
    {
        Console.WriteLine($"{function} : text");
    }
}

And, by work, I mean output something along the lines of:

Main : testing

However; it actually outputs:

: testing

CompilerServiceAttributes need to be on the Interface, and not on the implementation

class Program
{
    static void Main(string[] args)
    {
        ITest test = new Test();
        test.Log("testing");
        Console.ReadLine();
    }
}
 
interface ITest
{
    void Log(string text, [CallerMemberName] string function = "");
}
 
class Test : ITest
{
    public void Log(string text, string function = "")
    {
        Console.WriteLine($"{function} : text");
    }
}

Why?

When you think about it, it does kind of make sense. Because you’re calling against the interface, the compiler injected value needs to be there; if you took the interface out of the equation, then the attribute needs to be on the class.

You live and learn!

Creating a Basic Azure Web Job

In this article, I discussed the use of Azure functions; however, Web Jobs perform a similar task. Azure Functions are effectively an abstraction on top of Web Jobs – meaning that, while you have more control when using Web Jobs, there’s a little more to do when writing them.

This article covers the basics of Web Jobs, and has a walk-through for creating a very simple task using one.

Create a new Web Job

Once you create this project, you’ll need to fill in the following values in the app.config:

<configuration>
  <connectionStrings>
    <!-- The format of the connection string is "DefaultEndpointsProtocol=https;AccountName=NAME;AccountKey=KEY" -->
    <!-- For local execution, the value can be set either in this config file or through environment variables -->
    <add name="AzureWebJobsDashboard" connectionString="" />
    <add name="AzureWebJobsStorage" connectionString="" />
  </connectionStrings>

These can both be the same value, but they refer to where Azure stores it’s data.

AzureWebJobsDashboard

This is the storage account used to store logs.

AzureWebJobsStorage

This is the storage account used to store whatever the application needs to function (for example: queues or tables). In the example below, it’s where the file will go.

Storage accounts can be set-up from the Azure dashboard (more on this later):

A Basic Application

For this example, let’s take a file from a blob storage and parse it, then write out the result in a log. Specifically, we’ll take an XML file, and write the number of nodes into a log; here’s the file:

<test>
    <myNode>
    </myNode>
    <myNode>
    </myNode>
</test>

I think we’ll probably be looking for a figure around 2.

Blob Storage

Before we can do anything with blob storage, we’ll need a new storage area; create a new storage account:

Set the storage kind to “General Storage” (because we’re working with files); other than that, go with your gut.

Uploading

Once you’ve created the account, you’ll need to add a file – otherwise nothing will happen. You can do this in the web portal, or you can do it via a desktop utility that Microsoft provide: Storage Explorer.

I kind of expected this to take me to the web page mentioned… but it doesn’t! You have to navigate there manually:

http://storageexplorer.com

Install it… unless you want to upload your file using the web portal… in which case: don’t.

We can create a new container:

Now, we can see the storage account and any containers:

Now, you can upload a file from here (remember that you can do all this inside the Portal):

Once you’ve created this, go back and update the storage connection string (described above). You may also want to repeat the process for a dashboard storage area (or, as stated above, they can be the same).

Programmatically Downloading

Now we have a file in the directory, it can be downloaded via the WebJob; here’s a function that will download a file:

        public static async Task<string> GetFileContents(string connectionString, string containerString, string fileName)
        {
            CloudStorageAccount storage = CloudStorageAccount.Parse(connectionString);
            CloudBlobClient client = storage.CreateCloudBlobClient();
            CloudBlobContainer container = client.GetContainerReference(containerString);
            CloudBlob blob = container.GetBlobReference(fileName);

            MemoryStream ms = new MemoryStream();
            await blob.DownloadToStreamAsync(ms);
            ms.Position = 0;

            StreamReader sr = new StreamReader(ms);
            string contents = sr.ReadToEnd();
            return contents;
        }

The code to call this is here (note the commented out commands from the default WebJob Template):

        static void Main()
        {
            Console.WriteLine("Starting");

            var config = new JobHostConfiguration();

            if (config.IsDevelopment)
            {
                config.UseDevelopmentSettings();
            }

            //var host = new JobHost();

            string fileContents = AzureHelpers.GetFileContents(config.StorageConnectionString, "testblob", "test.xml").Result;
            Console.WriteLine(fileContents);

            // The following code ensures that the WebJob will be running continuously
            //host.RunAndBlock();

            Console.WriteLine("Done");
        }

Although this works (sort of – it doesn’t check for new files, and it would need to be run on a scheduled basis – “On Demand” in Azure terms), you don’t need it (at least not for jobs that react to files being uploaded to storage containers). WebJobs provide this functionality out of the box! There are a number of decorators that you can use for various purposes:

  • string
  • TextReader
  • Stream
  • ICloudBlob
  • CloudBlockBlob
  • CloudPageBlob
  • CloudBlobContainer
  • CloudBlobDirectory
  • IEnumerable<CloudBlockBlob>
  • IEnumerable<CloudPageBlob>

Here, we’ll use a BlobTrigger and accept a string. Moreover, doing it this way makes the writing to the log much easier, as there’s injection of sorts (at least I’m assuming that’s what it’s doing). Here’s what the complete solution looks like in the new paradigm:

        public static void ProcessFile([BlobTrigger("testblob/{name}")] string fileContents, TextWriter log)
        {            
            XmlDocument xmlDoc = new XmlDocument();
            xmlDoc.LoadXml(fileContents);            
            log.WriteLine($"Node count: {xmlDoc.FirstChild.ChildNodes.Count}");
        }

The key thing to notice here is that the function is static and public (the class it’s in needs to be public, too – even is that’s the Program class). The WebJob framework uses reflection to work out which functions it needs to run.

The other point to note is that I’m getting the parameter as a string – the article above details what you could have it as; for example, if you wanted to delete it afterwards, you’d probably want to use an ICloudBlob or something similar.

Anyway, it works:

The log file

Remember the storage area that we specified for the dashboard earlier? You should now see some new containers created in that storage area:

This has created a number of directories, but the one that we’re interested in is “output-logs” in the “azure-webjobs-hosts” container:

And here’s the log itself:

References

https://docs.microsoft.com/en-us/azure/app-service-web/web-sites-create-web-jobs

https://stackoverflow.com/questions/36610952/azure-webjobs-vs-azure-functions-how-to-choose

https://stackoverflow.com/questions/27580264/where-do-i-get-the-azurewebjobsdashboard-connection-string-information

http://www.hanselman.com/blog/IntroducingWindowsAzureWebJobs.aspx

https://stackoverflow.com/questions/24286214/where-are-azure-webjobs-blobinput-and-bloboutput-classes

https://docs.microsoft.com/en-us/azure/app-service-web/websites-dotnet-webjobs-sdk-storage-blobs-how-to

NUnit TestCaseSource

While working on this project, I found a need to abstract away a base type that the unit tests use (in this instance, it was a queue type). I was only testing a single type (PriorityQueue); however, I wanted to create a new type, but all the basic tests for the new type are the same as the existing ones. This led me to investigate the TestCaseSource attribute in NUnit.

As a result, I needed a way to re-use the tests. There are definitely multiple ways to do this; the simplest one is probably to create a factory class, and pass in a string parameter. The only thing that put me off this is that you end up with the following test case:

        [TestCase("test", "test9", "test", "test2", "test3", "test4", "test5", "test6", "test7", "test8", "test9"]
        [TestCase("a1", "a", "a1", "b", "c", "d", "a"]
        public void Queue_Dequeue_CheckResultOrdering(
            string first, string last, params string[] queueItems)
        {

Becoming:

        [TestCase("PriorityQueue", "test", "test9", "test", "test2", "test3", "test4", "test5", "test6", "test7", "test8", "test9"]
        [TestCase("PriorityQueue2", "test", "test9", "test", "test2", "test3", "test4", "test5", "test6", "test7", "test8", "test9"]
        [TestCase("PriorityQueue", "a1", "a", "a1", "b", "c", "d", "a"]
        [TestCase("PriorityQueue2", "a1", "a", "a1", "b", "c", "d", "a"]
        public void Queue_Dequeue_CheckResultOrdering(
            string queueType, string first, string last, params string[] queueItems)
        {

This isn’t very scaleable when adding a third or fourth type.

TestCaseSource

It turns out that the (or an least an) answer to this is to use NUnit’s TestCaseSource attribute. The NUnit code base dog foods quite extensively, so that is not a bad place to look for examples of how this works; however, what I couldn’t find was a way to mix and match. To better illustrate the point; here’s the first test that I changed to use TestCaseSource:

        [Test]
        public void Queue_NoEntries_CheckCount()
        {
            // Arrange
            PQueue.PriorityQueue<string> queue = new PQueue.PriorityQueue<string>();

            // Act
            int count = queue.Count();

            // Assert
            Assert.AreEqual(0, count);
        }

Which became:

        [Test, TestCaseSource(typeof(TestableQueueItemFactory), "ReturnQueueTypes")]
        public void Queue_NoEntries_CheckCount(IQueue<string> queue)
        {
            // Arrange


            // Act
            int count = queue.Count();

            // Assert
            Assert.AreEqual(0, count);
        }

(For completeness, the TestableQueueItemFactory is here):

    public static class TestableQueueItemFactory
    {
        public static IEnumerable<IQueue<string>> ReturnQueueTypes()
        {
            yield return new PQueue.PriorityQueue<string>();
        }
    }

However, when you have a TestCase like the one above, there’s a need for the equivalent of this (which doesn’t work):

        [Test, TestCaseSource(typeof(TestableQueueItemFactory), "ReturnQueueTypes")]
        [TestCase("test", "test9", "test", "test2", "test3", "test4", "test5", "test6", "test7", "test8", "test9")]
        [TestCase("a1", "a", "a1", "b", "c", "d", "a")]
        public void Queue_Dequeue_CheckResultOrdering(string first, string last, params string[] queueItems)
        {

A quick look at the NUnit code base reveals these attributes to be mutually exclusive.

Compromise

By no means is this a perfect solution, but the one that I settled on was to create a second TestCaseSource helper method, which looks like this (along with the test):

        private static IEnumerable Queue_Dequeue_CheckResultOrdering_TestCase()
        {
            foreach(var queueType in TestableQueueItemFactory.ReturnQueueTypes())
            {
                yield return new object[] { queueType, "test", "test9", new string[] { "test", "test2", "test3", "test4", "test5", "test6", "test7", "test8", "test9" } };
                yield return new object[] { queueType, "a1", "a", new string[] { "a1", "b", "c", "d", "a" } };
            }
        }

        [Test, TestCaseSource("Queue_Dequeue_CheckResultOrdering_TestCase")]
        public void Queue_Dequeue_CheckResultOrdering(
            IQueue <string> queue, string first, string last, params string[] queueItems)
        {

As you can see, the second helper method doesn’t really help readability, so it’s certainly not a perfect solution; in fact, with a single queue type, this makes the code more complex and less readable. However, When a second and third queue type are introduced, the test suddenly becomes resilient.

YAGNI

At first glance, this may appear to be an example of YAGNI. However, in this article, Martin Fowler does state:

Yagni only applies to capabilities built into the software to support a presumptive feature, it does not apply to effort to make the software easier to modify.

Which, I believe, is what we are doing here.

References

http://www.smaclellan.com/posts/parameterized-tests-made-simple/

http://stackoverflow.com/questions/16346903/how-to-use-multiple-testcasesource-attributes-for-an-n-unit-test

https://github.com/nunit/docs/wiki/TestCaseSource-Attribute

http://dotnetgeek.tumblr.com/post/2851360238/exploiting-nunit-attributes-valuesourceattribute

https://github.com/nunit/docs/wiki/TestCaseSource-Attribute

Programmatically List Existing Tags Using The TFS API

Having looked into this for some time; I came up with the following method of extracting team project tags. I’m not for a minute suggesting this is the best way of doing this – but it does work. My guess is that it’s not a very scalable solution, as it’s doing a LOT of work.

As it was, I couldn’t find a way to directly query the tags, so instead, I’m going through all the work items, and picking the tags. I couldn’t even find a way to filter the work items that actually have tags; so here’s the query that I ended up with:

private static IEnumerable<string> GetAllDistinctWorkItemTags(string uri, string projectName)
{
    TfsTeamProjectCollection tfs;
 
    tfs = TfsTeamProjectCollectionFactory.GetTeamProjectCollection(new Uri(uri)); // https://mytfs.visualstudio.com/DefaultCollection
    tfs.Authenticate();
 
    var wis = new WorkItemStore(tfs);
 
    WorkItemCollection workItemCollection = wis.Query(
         " SELECT [System.Tags]" +
         " FROM WorkItems " +
         $" WHERE [System.TeamProject] = '{projectName}' ");                
 
    if (workItemCollection.Count == 0)
        return null;
 
    List<string> tags = new List<string>();
    foreach (WorkItem wi in workItemCollection)
    {
        if (string.IsNullOrWhiteSpace(wi.Tags)) continue;
 
        var splitTags = wi.Tags.Split(';');
        tags.AddRange(splitTags.ToList());                
    }
 
    return tags.Distinct();
}

From debugging, I strongly suspect that whatever you put in the “SELECT”, it returns the entire work item. I also, for the life of me, couldn’t work out a lambda query for parsing the tags.

The calling method is here:

public static IEnumerable<string> GetAllTags(string uri, string teamProject)
{
    var project = GetTeamProject(uri, teamProject);
    IEnumerable<string> tags = GetAllDistinctWorkItemTags(uri, teamProject);
 
    return tags;
}

I’ve listed GetTeamProject helper method before, but for the sake of completeness:


public static Project GetTeamProject(string uri, string name)
{
    TfsTeamProjectCollection tfs;
 
    tfs = TfsTeamProjectCollectionFactory.GetTeamProjectCollection(new Uri(uri)); // https://mytfs.visualstudio.com/DefaultCollection
    tfs.Authenticate();
 
    var workItemStore = new WorkItemStore(tfs);
    
    var project = (from Project pr in workItemStore.Projects
                       where pr.Name == name
                       select pr).FirstOrDefault();
    if (project == null)
        throw new Exception($"Unable to find {name} in {uri}");
 
    return project;
}

Here’s the output:

tags1

Notes on Tags

A couple of points on tags: firstly, tags seem to exist in a kind of transient state; that is, while something is tagged, the tag exists, but once you remove all instances of a tag (for example, if I removed “Tagtest1” from all work items in my team project, TFS would eventually (I believe after a couple of days) just delete the tag for me. Obviously, in my example, as soon as I did this, I would no longer find it. This might leave you thinking that there is a more efficient way of removing tags (that is, you should be able to access the transient store in some way).

The existence of this Visual Studio plug-in lends support to that idea. It allows you to maintain the tags within your team project. If you’re using tags in any kind of serious way then I’d strongly recommend that you try it.

Performance

This is doing a lot of (IMO) unnecessary work, so I tried a little performance test; using this post as a template, I created a lot of bugs:

tags2

As you can see, I created a random set of tags. One other point that I’m going to put here is that a TFS database with ~30K work items and no code whatsoever increases the size of the default collection DB to around 2GB:

tags3

Now I ran the GetAllTags with some timing on:

tags4

19 seconds, which seems like quite a reasonable speed to me for 13.5k tags.

GetFilesAsync is not returning all the files in the “My Documents” folder

The following code should return a list of all files in the “My Documents” folder:

    var files = await KnownFolders.DocumentsLibrary.GetFilesAsync();
    List<FileInformation> filesList = new List<FileInformation>();
    
    System.Diagnostics.Debug.WriteLine($"File count: {files.Count()}");

The returned count from this code is (in my case) 11. However, excluding folders, there are 61 files in that directory for me. When I iterate through the collection, it does indeed find 11.

After scratching my head for a while, I finally realised that the answer to my question was my own blog post. It only lists the allowed types.

Reading and Opening a Zip File in a UWP

Some years ago, I had an idea for an application, and the functionality of that application involved extracting the contents of a zip file. I spent a while trying to work out how to do this in VB6 programmatically, and finally got bored, and my app never happened (not that there was such a thing as an app at the time).

Recently, I thought that I might re-visit my idea. Things have moved on since VB6, and this is how to work with zip files in 2016.

The following code will allow you to access the files inside an archive:

 
public async Task GetZipFileInformation(Stream stream)
{
    System.IO.Compression.ZipArchive zip = new System.IO.Compression.ZipArchive(stream);
 
    var firstFile = zip.Entries.FirstOrDefault();
    if (firstFile != null)
    {
    …

Generally speaking, this is much easier that trying to automate PKZIP, or whatever we used to use back in 2002!

Acknowledging a Message in Active MQ

Following on from my previous post on Active MQ, I’m now going to explore creating a mechanism whereby the message can fail.

The main issue with the trial project was that it used an auto acknowledge:

using (ISession session = connection.CreateSession(AcknowledgementMode.AutoAcknowledge))

There are a number of considerations here; firstly, what if the message that you read errors – we want to retry; but secondly, what happens is the message repeatedly errors (this type of message is known as a poison message).

The Problem

Here’s the send code again from the last post:

string queueName = "TextQueue";

Console.WriteLine($"Adding message to queue topic: {queueName}");

string brokerUri = $"activemq:tcp://localhost:61616";  // Default port
NMSConnectionFactory factory = new NMSConnectionFactory(brokerUri);

using (IConnection connection = factory.CreateConnection())
{
    connection.Start();

    using (ISession session = connection.CreateSession(AcknowledgementMode.AutoAcknowledge))
    using (IDestination dest = session.GetQueue(queueName))
    using (IMessageProducer producer = session.CreateProducer(dest))
    {                     
        producer.DeliveryMode = MsgDeliveryMode.Persistent;

        var msg = session.CreateTextMessage();
        producer.Send(msg);
                                                
        Console.WriteLine($"Sent {text} messages");
        
    }
}            

Other than splitting the message out, I haven’t changed anything. Okay, so let’s run that and check the queue:

msgrec1

msgrec2

Now, I’m going to change the receive code slightly:

    string queueName = "TextQueue";
 
    string brokerUri = $"activemq:tcp://localhost:61616";  // Default port
    NMSConnectionFactory factory = new NMSConnectionFactory(brokerUri);
 
    using (IConnection connection = factory.CreateConnection())
    {
        connection.Start();
        using (ISession session = connection.CreateSession(AcknowledgementMode.AutoAcknowledge))
        using (IDestination dest = session.GetQueue(queueName))
        using (IMessageConsumer consumer = session.CreateConsumer(dest))
        {
            IMessage msg = consumer.Receive();
            if (msg is ITextMessage)
            {
                ITextMessage txtMsg = msg as ITextMessage;                        
 
                Console.WriteLine($"Received message: {txtMsg.Text}");
 
                message = txtMsg.Text;
 
                throw new Exception("Test"); // <-- May cause problems
                
 
                return true;
            }
            else
            {
                Console.WriteLine("Unexpected message type: " + msg.GetType().Name);
            }
        }                
    }

As you can see, there is now an issue in the code; for some reason, it is repeatedly throwing an error entitled “Test”. I can’t work out why (maybe I’ll post a question on StackOverflow later), but when I run that, despite crashing, the message is read, and the queue is now empty.

Obviously, this is an issue: if that message was “DebitBankAccountWith200000” then someone is going to wish that the person that wrote this code hadn’t automatically acknowledged it.

Firstly, how do we stop the auto acknowledge?

There are basically two alternatives to auto acknowledge (there are more, but we’ll only look at two here): client acknowledge, and transactional acknowledgement. I’ll leave transactional acknowledgement for another day.

Client Acknowledge

This method is basically the manual version. You’re telling ActiveMQ that you will, or will not acknowledge the message yourself. Now, let’s alter the receive code slightly:

using (ISession session = connection.CreateSession(AcknowledgementMode.ClientAcknowledge))
using (IDestination dest = session.GetQueue(queueName))
using (IMessageConsumer consumer = session.CreateConsumer(dest))
{
    IMessage msg = consumer.Receive();
    if (msg is ITextMessage)
    {
        ITextMessage txtMsg = msg as ITextMessage;                        

        Console.WriteLine($"Received message: {txtMsg.Text}");

        message = txtMsg.Text;

        throw new Exception("Test");
        msg.Acknowledge();
        

        return true;
    }
    else
    {
        Console.WriteLine("Unexpected message type: " + msg.GetType().Name);
    }
}                

As you can see, I’ve changed two main parts here; the first is that I’ve changed that AcknowledgmentMode to ClientAcknowledge and I’ve added a call to the acknowledge method on the message.

Now let’s re-run the send and receive and see what happens to the queue.

msgrec3

Unfortunately, I still haven’t worked out why it’s crashing, but here’s the queue; still safely with the message:

msgrec4

We had an error, it crashed, but because it was never acknowledged, it’s still safe and sound in the queue. When we run the receive again, hopefully the bug will have magically disappeared and the message will successfully process.

Poison Messages

The concept of a poison message is where the issue with the message, resides in the message; the situation described above is not a poison message because the message is fine; but code is erroring. Once the code above is fixed, the message can be processed; however, let’s have a look at a different error scenario; here’s some new receive code:

string queueName = "TextQueue";
 
string brokerUri = $"activemq:tcp://localhost:61616";  // Default port
NMSConnectionFactory factory = new NMSConnectionFactory(brokerUri);
 
using (IConnection connection = factory.CreateConnection())
{
    connection.Start();
    using (ISession session = connection.CreateSession(AcknowledgementMode.ClientAcknowledge))
    using (IDestination dest = session.GetQueue(queueName))
    using (IMessageConsumer consumer = session.CreateConsumer(dest))
    {
        IMessage msg = consumer.Receive();
        if (msg is ITextMessage)
        {
            ITextMessage txtMsg = msg as ITextMessage;                        
 
            Console.WriteLine($"Received message: {txtMsg.Text}");
 
            message = txtMsg.Text;
 
            if (message.First() != 't')
                throw new Exception("Message is invalid");
            msg.Acknowledge();
            
 
            return true;
        }
        else
        {
            Console.WriteLine("Unexpected message type: " + msg.GetType().Name);
        }
    }                
}

This time, we have some code that actually processes the message and, based on the contents, does something; in this case, it throws an error where the message doesn’t start with ‘t’. So, the rules are simple; messages start with ‘t’. Let’s run the send code again and try some messages:

msgrec5

And now let’s receive these messages (incidentally, while testing this, my notes on starting two projects might be useful):

msgrec6

Okay – so we’ve come across a message that we can’t process. This has yet to be acknowledged, so it’s still safe and sound in the queue. We’ll simply restart the listener and pick it up:

msgrec7

Ah – okay. So, we have a problem. “nexttest” is causing an error with the queue, but if we don’t acknowledge it, we’re going to keep picking it up and erroring.

The Antidote

Once we know that the message is causing a problem, we can send it to a special queue; here’s the code to capture the error:

ITextMessage txtMsg = msg as ITextMessage;
 
try
{
    Console.WriteLine($"Received message: {txtMsg.Text}");
 
    message = txtMsg.Text;
 
    if (message != null && message.First() != 't')
    {
        // The message has a problem, and so we need to file it away without losing it
 
        throw new Exception("Message is invalid");
    }
    msg.Acknowledge();
 
    return true;
}
catch
{
    ResendMsg(session, msg);
    msg.Acknowledge(); // Acknoweledge the message from the original queue
}

And here’s the new method, ResendMsg:


private static void ResendMsg(ISession session, IMessage msg)
{
    var deadLetterQueue = new Apache.NMS.ActiveMQ.Commands.ActiveMQQueue("ActiveMQ.DLQ");
    IMessageProducer producer = session.CreateProducer(deadLetterQueue);
    producer.Send(msg);
}

The first time this executes, it will throw and catch the error, and then resend to a dead letter queue:

msgrec8

Subsequent runs can proceed past the problem message, and the message itself remains intact:

msgrec9

Chaos Monkey – Part 3 – Consuming Memory

Continuing from previous posts on programs that generally do your machine no good at all, I thought it might be an idea to have a look what I could do to the available memory. The use case here being that you want to see how your application can function when in competition with either one high-memory process, or many smaller ones.

To accomplish this, we’re going to create a list of strings – since strings are notoriously bad for memory anyway. The first thing to note here is that a single character takes up 16 bits, which is 2 bytes.

The second this is how to check the system’s available memory:

        private static System.Diagnostics.PerformanceCounter ramCounter =
            new System.Diagnostics.PerformanceCounter("Memory", "Available MBytes");

        private static long GetRemainingMemory()
        {
            return ramCounter.RawValue;
        }

Finally, you need to be aware that you can only use up all the memory in your machine (assuming you have more than 2GB) if you run the app in x64 mode. If you have less then you probably don’t need this article to simulate what low memory feels like.

There is a pretty big caveat to doing this; once you actually run out of memory; it takes a good few minutes for the system to catch up; even when you terminate the process. Consequently, the code that I use allows you to specify a “remaining memory”; here’s the main function:

        static void Main(string[] args)
        {
            long remainingMemory = int.Parse(args[0]);

            // Determine how much memory there is
            long memoryLeft = GetRemainingMemory();
            Console.WriteLine("Consuming memory until {0} is left", remainingMemory);

            // Calculate how much memory to use
            long removeMemory = memoryLeft - remainingMemory;

            // Call the function to consume the memory
            Console.WriteLine("Consuming {0} memory", removeMemory);
            ConsumeMemory(removeMemory, 1000);

            // Free the memory
            Console.WriteLine("Press any key to free memory");
            Console.ReadLine();
            FreeMemory();

            Console.ReadLine();
        }

As you can see, it first determines what we have to play with, and then calls a function to consume it. The second parameter to ConsumeMemory allows you to specify the speed which it consumes memory. If you set this to 1 then the usage will be slow; however, if you set it higher than you want for the remaining memory then it may use too much. Also, it doesn’t seem to improve speed much after that anyway.

The ConsumeMemory() function looks like this:


        static void ConsumeMemory(long memoryToConsumeMB, int consumePerItt)
        {            
            long bitsPerMB = 1024 * 1024 * 8;
            // Single char 2 bytes (16 bits)
            long numCharsPerMB = (bitsPerMB / 16);
            long numChars = numCharsPerMB * memoryToConsumeMB;
            long counter = 1, chunk = 0;

            if (memoryToConsumeMB > GetRemainingMemory())
            {
                Console.WriteLine("Cannot consume {0} because there is only {1} left", 
                    memoryToConsumeMB, GetRemainingMemory());
            }

            counter = memoryToConsumeMB / consumePerItt;
            chunk = numCharsPerMB * consumePerItt;

            Console.WriteLine("Consuming {0} memory", memoryToConsumeMB);

            for (int i = 1; i <= counter; i++)
            {
                Console.WriteLine("Consuming {0} MB", chunk / numCharsPerMB);

                _str.Add(new string('_', (int)chunk));

                Console.WriteLine("Memory remaining: {0}", GetRemainingMemory());
            }
        }

So, we basically work out how much memory we’re using each iteration and just add to a list of strings each time. Here’s what it looks like when you run it as above:

Chaos1

The FreeMemory() function just releases the list and calls the GC:

        private static void FreeMemory()
        {
            _str = null;
            GC.Collect();
            ShowMemory();
        }

As you can see, it ramps up pretty quick. In this case I’m leaving 2GB.

Super Chaos Monkey Mode

Let’s try putting this in a loop and take out the prompts:


        static void Main(string[] args)
        {
            long remainingMemory = int.Parse(args[0]);

            while (true)
            {
                // Determine how much memory there is
                long memoryLeft = GetRemainingMemory();
                Console.WriteLine("Consuming memory until {0} is left", remainingMemory);

                // Calculate how much memory to use
                long removeMemory = memoryLeft - remainingMemory;

                // Call the function to consume the memory
                Console.WriteLine("Consuming {0} memory", removeMemory);
                ConsumeMemory(removeMemory, 1000);

                // Free the memory
                Console.WriteLine("Press any key to free memory");
                //Console.ReadLine();
                FreeMemory();
            }

            //Console.ReadLine();
        }

        private static void FreeMemory()
        {
            _str = new List<string>();
            GC.Collect();
            ShowMemory();
        }

Chaos2

A note on the GC

Okay – there are very few cases where the GC.Collect() should be called. But I believe this to be one of them. The reason being that, not calling it explicitly ends in the following:

Chaos3

Basically, by the time the garbage collection kicks in, you’re already allocating more memory, which affects the ebb and flow.

Super speed chaos

If you want very rapid consumption of memory, just alter the consume memory function as follows:


            //for (int i = 1; i <= counter; i++)
            Parallel.For(1, counter + 1, (i) =>
              {
                  Console.WriteLine("Consuming {0} MB", chunk / numCharsPerMB);

                  _str.Add(new string('_', (int)chunk));

                  Console.WriteLine("Memory remaining: {0}", GetRemainingMemory());
              });

Chaos4

Be very careful with this one, though. A slight bug in your code and you’ll need to do a hard reboot of your machine.