Category Archives: Console Applications

Manually Parsing a JSON String Using JSON.NET

How to manually parse a JSON string using JSON.NET.

Disclaimer

If you jump straight to the references, you will find a very similar set of information, and I strongly encourage people to do so. Additionally, this is probably not the most efficient way to achieve this.

Right, on with the show

Here’s the string that I’ll be parsing, and a little code stolen directly from the link at the bottom to show what it looks like:

static void Main(string[] args)
{
    string json = "{\"documents\":[{\"keyPhrases\":[\"Test new bug\"],\"id\":\"1\"}],\"errors\":[]}";
    JsonTextReader reader = new JsonTextReader(new StringReader(json));
    while (reader.Read())
    {
        if (reader.Value != null)
        {
            Console.WriteLine("Token: {0}, Value: {1}", reader.TokenType, reader.Value);
        }
        else
        {
            Console.WriteLine("Token: {0}", reader.TokenType);
        }
    }
 
    Console.ReadLine();
}

The output for this looks like:

Using this, it’s easier to create a routine to manually parse this. Each object can be tracked by using the Start and EndObject tags. Here’s my unit test to check this works:

[TestMethod]
public void TestJSONParse()
{
    // Arrange
    string json = "{\"documents\":[{\"keyPhrases\":[\"Test new bug\"],\"id\":\"1\"}],\"errors\":[]}";
    // Act
    var result = JsonHelper.ParseResponse(json);
 
    // Assert
    Assert.AreEqual(1, result.Count());
    Assert.AreEqual(1, result.Keys.First());
    string expectedPhrase = result.Values.First().First().ToString();
    Assert.AreEqual("Test new bug", expectedPhrase, false);
}

And here’s the code itself:

/// <summary>
/// Parse the following JSON
/// {"documents":[{"keyPhrases":["Test new bug"],"id":"1"}],"errors":[]}
/// </summary>
/// <param name="response"></param>
/// <returns></returns>
public static Dictionary<int, List<string>> ParseResponse(string response)
{
    Dictionary<int, List<string>> dict = new Dictionary<int, List<string>>();
    object readerValue;
 
    if (!string.IsNullOrWhiteSpace(response))
    {
        JsonTextReader reader = new JsonTextReader(new StringReader(response));
        int? currentValue = null;
        List<string> currentList = null;
 
        while (reader.Read())
        {
            readerValue = reader.Value;
 
            switch (reader.TokenType)
            {
                case JsonToken.PropertyName:                            
                    if (readerValue.ToString() == "id")
                    {
                        reader.Read();
                        currentValue = int.Parse(reader.Value.ToString());                                
                    }
                    else if (readerValue.ToString() == "keyPhrases")
                    {
                        // Do nothing
                    }
                    else if (readerValue.ToString() == "errors")
                    {
                        currentValue = null;
                    }
                    break;
 
                case JsonToken.String:                            
                    currentList.Add(reader.Value.ToString());
                    break;
 
                case JsonToken.StartArray:
                    currentList = new List<string>();
                    break;
 
                case JsonToken.StartObject:
                    currentList = null;
                    currentValue = null;
                    break;
 
                case JsonToken.EndObject:
                    if (currentValue.HasValue)
                    {
                        dict.Add(currentValue.Value, currentList);
                    }
                    break;
            }
        }
    }
 
    return dict;
}

It is messy, and it is error prone, and it would be better done by creating classes and serialising it; however, I’d never attempted to do this manually before, and it’s generally nice to do things the hard way, that way, you can appreciate what you get from these tools.

References

http://www.newtonsoft.com/json/help/html/ReadJsonWithJsonTextReader.htm

Finding Duplicate Values in a Dictionary Using C#

Due to a series of blog posts that I’m writing on TFS and MS Cognitive Services, I came across a requirement to identify duplicate values in a dictionary. For example, imagine you had an actual physical dictionary, and you wanted to find all the words that meant the exact same thing. Here’s the set-up for the test:

Dictionary<int, string> test = new Dictionary<int, string>()
{
    { 1, "one"},
    { 2, "two" },
    { 3, "one" },
    { 4, "three" }
};
DisplayDictionary("Initial Collection", test);

I’m outputting to the console at every stage, so here’s the helper method for that:


private static void DisplayDictionary(string title, Dictionary<int, string> test)
{
    Console.WriteLine(title);
    foreach (var it in test)
    {
        Console.WriteLine($"Key: {it.Key}, Value: {it.Value}");
    }
}

Finding Duplicates

LINQ has a special method for this, it’s Intersect. For flat collections, this works excellently, but no so well for Dictionaries; here was my first attempt:


Dictionary<int, string> intersect = test.Intersect(test)
    .ToDictionary(i => i.Key, i => i.Value);
DisplayDictionary("Intersect", intersect);

As you can see, the intersect doesn’t work very well this time (don’t tell Chuck).

Manual Intersect

The next stage then is to roll your own; a pretty straightforward lambda in the end:


var intersect2 = test.Where(i => test.Any(t => t.Key != i.Key && t.Value == i.Value))
    .ToDictionary(i => i.Key, i => i.Value);
DisplayDictionary("Manual Intersect", intersect2);
 

This works much better.

Chaos Monkey – Part 3 – Consuming Memory

Continuing from previous posts on programs that generally do your machine no good at all, I thought it might be an idea to have a look what I could do to the available memory. The use case here being that you want to see how your application can function when in competition with either one high-memory process, or many smaller ones.

To accomplish this, we’re going to create a list of strings – since strings are notoriously bad for memory anyway. The first thing to note here is that a single character takes up 16 bits, which is 2 bytes.

The second this is how to check the system’s available memory:

        private static System.Diagnostics.PerformanceCounter ramCounter =
            new System.Diagnostics.PerformanceCounter("Memory", "Available MBytes");

        private static long GetRemainingMemory()
        {
            return ramCounter.RawValue;
        }

Finally, you need to be aware that you can only use up all the memory in your machine (assuming you have more than 2GB) if you run the app in x64 mode. If you have less then you probably don’t need this article to simulate what low memory feels like.

There is a pretty big caveat to doing this; once you actually run out of memory; it takes a good few minutes for the system to catch up; even when you terminate the process. Consequently, the code that I use allows you to specify a “remaining memory”; here’s the main function:

        static void Main(string[] args)
        {
            long remainingMemory = int.Parse(args[0]);

            // Determine how much memory there is
            long memoryLeft = GetRemainingMemory();
            Console.WriteLine("Consuming memory until {0} is left", remainingMemory);

            // Calculate how much memory to use
            long removeMemory = memoryLeft - remainingMemory;

            // Call the function to consume the memory
            Console.WriteLine("Consuming {0} memory", removeMemory);
            ConsumeMemory(removeMemory, 1000);

            // Free the memory
            Console.WriteLine("Press any key to free memory");
            Console.ReadLine();
            FreeMemory();

            Console.ReadLine();
        }

As you can see, it first determines what we have to play with, and then calls a function to consume it. The second parameter to ConsumeMemory allows you to specify the speed which it consumes memory. If you set this to 1 then the usage will be slow; however, if you set it higher than you want for the remaining memory then it may use too much. Also, it doesn’t seem to improve speed much after that anyway.

The ConsumeMemory() function looks like this:


        static void ConsumeMemory(long memoryToConsumeMB, int consumePerItt)
        {            
            long bitsPerMB = 1024 * 1024 * 8;
            // Single char 2 bytes (16 bits)
            long numCharsPerMB = (bitsPerMB / 16);
            long numChars = numCharsPerMB * memoryToConsumeMB;
            long counter = 1, chunk = 0;

            if (memoryToConsumeMB > GetRemainingMemory())
            {
                Console.WriteLine("Cannot consume {0} because there is only {1} left", 
                    memoryToConsumeMB, GetRemainingMemory());
            }

            counter = memoryToConsumeMB / consumePerItt;
            chunk = numCharsPerMB * consumePerItt;

            Console.WriteLine("Consuming {0} memory", memoryToConsumeMB);

            for (int i = 1; i <= counter; i++)
            {
                Console.WriteLine("Consuming {0} MB", chunk / numCharsPerMB);

                _str.Add(new string('_', (int)chunk));

                Console.WriteLine("Memory remaining: {0}", GetRemainingMemory());
            }
        }

So, we basically work out how much memory we’re using each iteration and just add to a list of strings each time. Here’s what it looks like when you run it as above:

Chaos1

The FreeMemory() function just releases the list and calls the GC:

        private static void FreeMemory()
        {
            _str = null;
            GC.Collect();
            ShowMemory();
        }

As you can see, it ramps up pretty quick. In this case I’m leaving 2GB.

Super Chaos Monkey Mode

Let’s try putting this in a loop and take out the prompts:


        static void Main(string[] args)
        {
            long remainingMemory = int.Parse(args[0]);

            while (true)
            {
                // Determine how much memory there is
                long memoryLeft = GetRemainingMemory();
                Console.WriteLine("Consuming memory until {0} is left", remainingMemory);

                // Calculate how much memory to use
                long removeMemory = memoryLeft - remainingMemory;

                // Call the function to consume the memory
                Console.WriteLine("Consuming {0} memory", removeMemory);
                ConsumeMemory(removeMemory, 1000);

                // Free the memory
                Console.WriteLine("Press any key to free memory");
                //Console.ReadLine();
                FreeMemory();
            }

            //Console.ReadLine();
        }

        private static void FreeMemory()
        {
            _str = new List<string>();
            GC.Collect();
            ShowMemory();
        }

Chaos2

A note on the GC

Okay – there are very few cases where the GC.Collect() should be called. But I believe this to be one of them. The reason being that, not calling it explicitly ends in the following:

Chaos3

Basically, by the time the garbage collection kicks in, you’re already allocating more memory, which affects the ebb and flow.

Super speed chaos

If you want very rapid consumption of memory, just alter the consume memory function as follows:


            //for (int i = 1; i <= counter; i++)
            Parallel.For(1, counter + 1, (i) =>
              {
                  Console.WriteLine("Consuming {0} MB", chunk / numCharsPerMB);

                  _str.Add(new string('_', (int)chunk));

                  Console.WriteLine("Memory remaining: {0}", GetRemainingMemory());
              });

Chaos4

Be very careful with this one, though. A slight bug in your code and you’ll need to do a hard reboot of your machine.

Console Games – Snake – Part 5

Continuing on from my series of posts on writing a console game with my children, this post will cover the score and speed up the game a little to make it progressively harder. If you haven’t seen the earlier posts then start here.

What’s the score?

Let’s start with the score; first thing to do is create a variable to store it:

    class Program
    {
        private static int _length = 6;
        private static int _score = 0;

The way to increase the score is to eat food, so that’s quite straight-forward:


private static void DetectCollision(Position currentPos)
{
    …
    // Check if we've eaten the food
    if (_foodPosition.left == currentPos.left && _foodPosition.top == currentPos.top)
    {
        _length++;
        _score++;
        _foodPosition = null;
}

Nothing hugely complicated there. Finally, display the score:


private static void DrawScreen()
{
    Console.Clear();

    Console.SetCursorPosition(Console.WindowWidth - 3, Console.WindowHeight - 1);
    Console.Write(_score);

Speed

That’s the score; next we need to speed the game up. Currently we have an `UpdateGame()` method that determines how often the game is updated; here’s what it currently does:

        private static bool UpdateGame()
        {
            if (DateTime.Now < nextUpdate) return false;

            if (_foodPosition == null)
            {
                _foodPosition = new Position()
                {
                    left = _rnd.Next(Console.WindowWidth),
                    top = _rnd.Next(Console.WindowHeight)
                };
            }

            if (_lastKey.HasValue)
            {
                Move(_lastKey.Value);
            }

            nextUpdate = DateTime.Now.AddMilliseconds(500);
            return true;
        }

So, we can simply change the nextUpdate to use a variable that we already have; like this:

nextUpdate = DateTime.Now.AddMilliseconds(500 / (_score + 1));

Game Over

Okay, well, the eagle eyed among you may have noticed that game over just gives a runtime error; let’s try something a little more user friendly. First, we’ll create a variable to store whether the game is still in play:

        private static bool _inPlay = true;

Next, change the game loop to use this:

        static void Main(string[] args)
        {
            Console.CursorVisible = false;
            DrawScreen();
            while (_inPlay)
            {

And finally, change the `GameOver()` method:

        private static void GameOver()
        {
            _inPlay = false;
            Console.Clear();
            Console.WriteLine("Game over.");
            Console.ReadLine();
        }

Final word

I’m still working through this game, and with a catch game (which I’ll also post at some stage) with the children. The way that I’ve been addressing this is, after an initial explanation phase, asking the children to complete each small section; for example, in the above section, I would have asked them to complete three separate tasks: To create a new boolean variable, to use that variable in the while loop and to re-write the GameOver() function so that it sets the variable to false. Roughly speaking, the posts are arranged in small sections, and they could be treated as separate exercises.

Please leave a comment if you found any of these helpful, or with any suggestions for improvements.

If I get the time or the inclination, I might break these posts down into individual exercises and post that as well.

Detecting Recursion – JSON.Net Stack Overflow

Trying to save the game class of a game that I’m working on, I got a Stack Overflow error. I’m using MVVMCross, and my code looked like this:

        public void Save()
        {
            var jsonConv = Mvx.Resolve<IMvxJsonConverter>();
            string text = jsonConv.SerializeObject(this);
            FileHelper.SaveGameFile(text);
        }

The problem was that I got this error:

An unhandled exception of type ‘System.StackOverflowException’ occurred in mscorlib.dll

stackoverflow

I had a pretty good idea why. My game features a large population of “people”. Some of these people relate to each other; for example, they are parents, children, employers, etc. My guess was that I’d somehow messed up the creation routine and ended up with a recursive reference. (As it happened, far from messing it up, I hadn’t considered that a spouse relationship is recursive by definition!)

The Problem

The problem was that I had a starting population of 10,000 people. There are other ways to solve this: representing the reference between the classes as some kind of index, debugging the creation code, etc… However, I wanted to see if I could write a program to detect this.

My test program looks like this:

    class MyData
    {
        public MyData recursiveRef { get; set; }
        public string test { get;set; }
    }

    class Program
    {
        static void Main(string[] args)
        {
            List<MyData> data = new List<MyData>()
            {
                new MyData() {recursiveRef = null, test="non-recursive" },
                new MyData() {recursiveRef = null, test="recursive ref 1" },
                new MyData() {recursiveRef = null, test="recursive ref 2" },
                new MyData() {recursiveRef = null, test="recursive ref 3" },
                new MyData() {recursiveRef = null, test="recursive ref back to 1" }
            };
            data[1].recursiveRef = data[2];
            data[2].recursiveRef = data[3];
            data[3].recursiveRef = data[4];
            data[4].recursiveRef = data[1];

            SerialiseData(data);

            Console.ReadLine();
        }

        private static void SerialiseData(List<MyData> data)
        {
            JsonSerializerSettings settings = new JsonSerializerSettings()
            {
                ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Serialize,
                DateFormatHandling = DateFormatHandling.IsoDateFormat,
            };

            var ser = JsonConvert.SerializeObject(data, Formatting.None, settings);
            Console.Write(ser);
        }

Quickly running this will error with a stack overflow (to clarify, slowly running it will result in the same error!).

Detection Routine

Here’s the functions that I wrote to detect recursion:


        /// Itterate the list and call method CheckElement to analyse each element
        private static bool DetectRecursion<T>(IEnumerable<T> data)
        {
            Type typeT = typeof(T);

            foreach (T element in data)
            {
                if (CheckElement<T>(typeT, element, typeT, element))
                {
                    Console.WriteLine("Recursion found - exiting");
                    Console.ReadLine();
                    return true;
                }
            }

            return false;
        }

        private static List<object> recursionChain;

        /// Method recursively traverses the object to determine if there are any recursove references
        private static bool CheckElement<T>(Type baseType, T baseElement, Type checkType, object checkElement)
        {
            PropertyInfo[] piArr = checkType.GetProperties();
            foreach (PropertyInfo pi in piArr)
            {
                Console.WriteLine("Checking {0}, type {1}", pi.Name, pi.PropertyType);
                if (pi.PropertyType != baseType)
                    continue;

                var val = pi.GetValue(checkElement);
                if (val == null) continue;

                Console.WriteLine("Value for {0} is {1}", pi.Name, val.ToString());

                Type piType = val.GetType();

                if (piType == baseType && val.Equals(baseElement))
                {
                    return true;
                }

                if (CheckRecursionChain(val, piType)) return true;

                if (CheckElement<T>(baseType, baseElement, piType, val))
                {
                    Console.WriteLine("Successfully found recursive element {0}, {1}", piType.ToString(), val.ToString());
                    return true;
                }
            }

            return false;
        }

        /// Check the static recursion chain for a match
        private static bool CheckRecursionChain(object val, Type piType)
        {            
            if (Program.recursionChain != null && Program.recursionChain.Contains(val))
            {
                Console.WriteLine("Successfully found recursive element {0}, {1}", piType.ToString(), val.ToString());
                return true;
            }

            if (Program.recursionChain == null)
                Program.recursionChain = new List<object>();

            Program.recursionChain.Add(val);
            
            return false;
        }

This is, admittedly, not a simple or short piece of code; having said that, it doesn’t do anything complex, once you’re happy with the reflection, the rest is just a recursive tree.

To use this, simply call `DetectRecursion` in the above test program before calling serialize (or instead of).

...
DetectRecursion(data);

//SerialiseData(data);

...

Console Games – Snake – Part 2

Following on from this post, we were creating a snake game for the purpose of teaching a 9 year old to program. This post will not make sense without its predecessors.

Clean up the tail

The next part of this game is to get the snake game to tidy up after itself (to remove the last part of its own tail). This was done (for us) in two parts.


private static int _length = 3;        

private static void CleanUp()
{
    while (points.Count() > _length)
    {
        points.Remove(points.First());
    }
}

This is called from AcceptInput:

private static bool AcceptInput()
{
    …
    points.Add(currentPos);
    CleanUp();

    return true;
}

If you run that now, it simply does what it did before; the final part is to re-introduce the clear:


private static void DrawScreen()
{
    Console.Clear();
    foreach (var point in points)
    …

So now we have a 3 star snake game; try extending the length manually to play with it. It is strangely addictive, even in this immature state.

Console Games – Snake – Part 1

Based on this earlier post, we had a working console game. Admittedly it doesn’t do much, apart from allow you to move a star around the screen. In order to turn this into a snake game, the first thing to do is to no longer clear the screen:

        private static void DrawScreen()
        {
            //Console.Clear();
            Console.SetCursorPosition(_left, _top);
            Console.Write('*');            
        }

That gives us a snake – but you might notice that when you move left, it doesn’t ‘trail’. There is a possible workaround (albeit, not massively efficient – although remember that this is a game with the purpose of teaching programming).

First, create a struct to hold the position:

        private struct Position
        {
            public int left;
            public int top;
        }

This obviously could be a class. Next, create a list of these:

        private static List<Position> points = new List<Position>();

Here’s what the `AcceptInput` function looks like with the changes:

        private static bool AcceptInput()
        {
            if (!Console.KeyAvailable)
                return false;

            ConsoleKeyInfo key = Console.ReadKey();

            Position currentPos;
            if (points.Count != 0)
                currentPos = points.Last();
            else
                currentPos = GetStartPosition();

            switch (key.Key)
            {
                case ConsoleKey.LeftArrow:
                    currentPos.left--;
                    break;
                case ConsoleKey.RightArrow:
                    currentPos.left++;
                    break;
                case ConsoleKey.UpArrow:
                    currentPos.top--;
                    break;
                case ConsoleKey.DownArrow:
                    currentPos.top++;
                    break;

            }

            points.Add(currentPos);

            return true;
        }

        private static Position GetStartPosition()
        {
            return new Position()
            {
                left = 0,
                top = 0
            };
        }

But what about the tail

In traditional snake, the last element is removed each iteration, unless you `eat` something. Doing it the above way lends itself to this more easily. I’ll tackle that for the next post.

Writing a Game in a Console Application – Teaching Programming

Recently I began trying to teach my children some basics of programming. I’d previously tried teaching them using tools like Scratch, but they seemed to get distracted by the graphics whizzing around, and they forgot about the actual coding.

This time, I started up a Visual Basic Console Application, and took them through a couple of basic programs: guess the number, guess the favourite food and calculate prime numbers.

They quickly started to ask about games (by which they meant arcade games), but I didn’t want to jump into a game framework like XNA, as I felt that this time, they were actually understanding some of the constructs. My idea was that we could write a game using a console application.

In order to introduce this, I simply got them to swap the code they had been writing (for example):

        Dim name As String

        Console.WriteLine("hello, what is your name?")
        name = Console.ReadLine() ' Remember name here

For something like this:

        While True
            Dim test As ConsoleKeyInfo = Console.ReadKey()

            Console.Clear()

            If test.Key = ConsoleKey.LeftArrow Then
                Console.WriteLine("You have pressed the left arrow")

Which was going well – I had definitely piqued their interest. The subsequent conversation went something like this:

9 Year Old: So, how do we use this to move something around the screen?

Me: Err, well – we’ll do that next time!

The truth is I had no idea… but I do now, so I thought I’d write at least one post describing it. This is the it. Note that the code that follows in C#, and the preceding code is VB. That’s because I thought that a 9-year-old child would relate better to Visual Basic than C# initially, but I personally think in C# (I learnt to program using Spectrum Basic – and I think we’d have more children programming if you had to wait five minutes for a game to load).

Translation between the two is straight-foward, but you can always use this.

The hardest part

The hardest part in getting a console game working is displaying a character outside of the next position on the console; the trick here is:

Console.SetCursorPosition(left, top);

Extending this, here a function to position a character on the screen:

        private static void DrawScreen()
        {
            Console.Clear();
            Console.SetCursorPosition(_left, _top);
            Console.Write('*');
        }

And the function to read the input would be:

        private static void AcceptInput()
        {
            ConsoleKeyInfo key = Console.ReadKey();

            switch (key.Key)
            {
                case ConsoleKey.LeftArrow:
                    _left--;
                    break;
                case ConsoleKey.RightArrow:
                    _left++;
                    break;
                case ConsoleKey.UpArrow:
                    _top--;
                    break;
                case ConsoleKey.DownArrow:
                    _top++;
                    break;

            }

Finally, the main function and variables:

        private static int _left;
        private static int _top;

        static void Main(string[] args)
        {
            Console.CursorVisible = false;
            while (true)
            {
                DrawScreen();
                AcceptInput();
            }
        }

Is this efficient?

No – it’s not. You can improve this by using:

            if (Console.KeyAvailable)

I’m not planning to write Call Of Duty, but I will try and get them to optimize it slightly, just to get the idea across that the less code executed, the better.

Not much of a game?

No, it’s not. But it is the basis of every arcade game. We had a talk, and have a snake game, and a catch the falling characters game planned; both seem eminently doable based on this starting point.