Tag Archives: c#

NUnit TestCaseSource

While working on this project, I found a need to abstract away a base type that the unit tests use (in this instance, it was a queue type). I was only testing a single type (PriorityQueue); however, I wanted to create a new type, but all the basic tests for the new type are the same as the existing ones. This led me to investigate the TestCaseSource attribute in NUnit.

As a result, I needed a way to re-use the tests. There are definitely multiple ways to do this; the simplest one is probably to create a factory class, and pass in a string parameter. The only thing that put me off this is that you end up with the following test case:

        [TestCase("test", "test9", "test", "test2", "test3", "test4", "test5", "test6", "test7", "test8", "test9"]
        [TestCase("a1", "a", "a1", "b", "c", "d", "a"]
        public void Queue_Dequeue_CheckResultOrdering(
            string first, string last, params string[] queueItems)
        {

Becoming:

        [TestCase("PriorityQueue", "test", "test9", "test", "test2", "test3", "test4", "test5", "test6", "test7", "test8", "test9"]
        [TestCase("PriorityQueue2", "test", "test9", "test", "test2", "test3", "test4", "test5", "test6", "test7", "test8", "test9"]
        [TestCase("PriorityQueue", "a1", "a", "a1", "b", "c", "d", "a"]
        [TestCase("PriorityQueue2", "a1", "a", "a1", "b", "c", "d", "a"]
        public void Queue_Dequeue_CheckResultOrdering(
            string queueType, string first, string last, params string[] queueItems)
        {

This isn’t very scaleable when adding a third or fourth type.

TestCaseSource

It turns out that the (or an least an) answer to this is to use NUnit’s TestCaseSource attribute. The NUnit code base dog foods quite extensively, so that is not a bad place to look for examples of how this works; however, what I couldn’t find was a way to mix and match. To better illustrate the point; here’s the first test that I changed to use TestCaseSource:

        [Test]
        public void Queue_NoEntries_CheckCount()
        {
            // Arrange
            PQueue.PriorityQueue<string> queue = new PQueue.PriorityQueue<string>();

            // Act
            int count = queue.Count();

            // Assert
            Assert.AreEqual(0, count);
        }

Which became:

        [Test, TestCaseSource(typeof(TestableQueueItemFactory), "ReturnQueueTypes")]
        public void Queue_NoEntries_CheckCount(IQueue<string> queue)
        {
            // Arrange


            // Act
            int count = queue.Count();

            // Assert
            Assert.AreEqual(0, count);
        }

(For completeness, the TestableQueueItemFactory is here):

    public static class TestableQueueItemFactory
    {
        public static IEnumerable<IQueue<string>> ReturnQueueTypes()
        {
            yield return new PQueue.PriorityQueue<string>();
        }
    }

However, when you have a TestCase like the one above, there’s a need for the equivalent of this (which doesn’t work):

        [Test, TestCaseSource(typeof(TestableQueueItemFactory), "ReturnQueueTypes")]
        [TestCase("test", "test9", "test", "test2", "test3", "test4", "test5", "test6", "test7", "test8", "test9")]
        [TestCase("a1", "a", "a1", "b", "c", "d", "a")]
        public void Queue_Dequeue_CheckResultOrdering(string first, string last, params string[] queueItems)
        {

A quick look at the NUnit code base reveals these attributes to be mutually exclusive.

Compromise

By no means is this a perfect solution, but the one that I settled on was to create a second TestCaseSource helper method, which looks like this (along with the test):

        private static IEnumerable Queue_Dequeue_CheckResultOrdering_TestCase()
        {
            foreach(var queueType in TestableQueueItemFactory.ReturnQueueTypes())
            {
                yield return new object[] { queueType, "test", "test9", new string[] { "test", "test2", "test3", "test4", "test5", "test6", "test7", "test8", "test9" } };
                yield return new object[] { queueType, "a1", "a", new string[] { "a1", "b", "c", "d", "a" } };
            }
        }

        [Test, TestCaseSource("Queue_Dequeue_CheckResultOrdering_TestCase")]
        public void Queue_Dequeue_CheckResultOrdering(
            IQueue <string> queue, string first, string last, params string[] queueItems)
        {

As you can see, the second helper method doesn’t really help readability, so it’s certainly not a perfect solution; in fact, with a single queue type, this makes the code more complex and less readable. However, When a second and third queue type are introduced, the test suddenly becomes resilient.

YAGNI

At first glance, this may appear to be an example of YAGNI. However, in this article, Martin Fowler does state:

Yagni only applies to capabilities built into the software to support a presumptive feature, it does not apply to effort to make the software easier to modify.

Which, I believe, is what we are doing here.

References

http://www.smaclellan.com/posts/parameterized-tests-made-simple/

http://stackoverflow.com/questions/16346903/how-to-use-multiple-testcasesource-attributes-for-an-n-unit-test

https://github.com/nunit/docs/wiki/TestCaseSource-Attribute

http://dotnetgeek.tumblr.com/post/2851360238/exploiting-nunit-attributes-valuesourceattribute

https://github.com/nunit/docs/wiki/TestCaseSource-Attribute

WPF Performance – TextBlock

WPF typically doesn’t have many performance issues and, where it does, this can usually be fixed by virtualisation. Having said that, even if you never need to use this, it’s useful to know that you can eek that last ounce of performance out of the system.

Here’s a basic program that contains a TextBlock:

<Window x:Class="TextBlockTest.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
        xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
        xmlns:local="clr-namespace:TextBlockTest"
        mc:Ignorable="d"
        Title="MainWindow" Height="350" Width="525"
        x:Name="MainWindowView">
    <Grid>
        <ScrollViewer>
            <ItemsControl ItemsSource="{Binding BigList, ElementName=MainWindowView}" Margin="0,-1,0,1">
                <ItemsControl.ItemTemplate>
                    <DataTemplate>
                        <TextBlock Text="{Binding}"/>
                    </DataTemplate>
                </ItemsControl.ItemTemplate>
            </ItemsControl>
        </ScrollViewer>
    </Grid>
</Window>

Code behind:

using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;

namespace TextBlockTest
{
    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
        public ObservableCollection<string> BigList { get; set; }

        public MainWindow()
        {
            BigList = new ObservableCollection<string>();
            for (int i = 0; i <= 10000; i++)
            {
                BigList.Add($"Item {i}");
            }

            InitializeComponent();
        }
    }
}

Let’s, for a minute, imagine this is slow, and profile it:

The layout is taking most of the time. Remember that each control needs to be created, and remember that the TextBlock does slightly more than just display text:

The whole panel took 3.46s. Not terrible, performance, but can it be improved? The answer is: yes, it can! Very, very slightly.

Let’s create a Custom Control:

using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;

namespace FastTextBlock
{
   
    public class MyTextBlockTest : Control
    {
        private FormattedText _formattedText;

        static MyTextBlockTest()
        {
            //DefaultStyleKeyProperty.OverrideMetadata(typeof(MyTextBlockTest), new FrameworkPropertyMetadata(typeof(MyTextBlockTest)));
        }

        public static readonly DependencyProperty TextProperty =
             DependencyProperty.Register(
                 "Text", 
                 typeof(string),
                 typeof(MyTextBlockTest), 
                 new FrameworkPropertyMetadata(string.Empty, FrameworkPropertyMetadataOptions.AffectsMeasure,
                    (o, e) => ((MyTextBlockTest)o).TextPropertyChanged((string)e.NewValue)));

        private void TextPropertyChanged(string text)
        {
            var typeface = new Typeface(
                new FontFamily("Times New Roman"),
                FontStyles.Normal, FontWeights.Normal, FontStretches.Normal);

            _formattedText = new FormattedText(
                text, CultureInfo.CurrentCulture,
                FlowDirection.LeftToRight, typeface, 15, Brushes.Black);
        }


        public string Text
        {
            get { return (string)GetValue(TextProperty); }
            set { SetValue(TextProperty, value); }
        }

        protected override void OnRender(DrawingContext drawingContext)
        {
            if (_formattedText != null)
            {
                drawingContext.DrawText(_formattedText, new Point());
            }
        }

        protected override Size MeasureOverride(Size constraint)
        {
            //return base.MeasureOverride(constraint);

            return _formattedText != null
            ? new Size(_formattedText.Width, _formattedText.Height)
            : new Size();
        }
    }
}

Here’s the new XAML:

    <Grid>
        <ScrollViewer>
            <ItemsControl ItemsSource="{Binding BigList, ElementName=MainWindowView}" Margin="0,-1,0,1">
                <ItemsControl.ItemTemplate>
                    <DataTemplate>
                        <!--<TextBlock Text="{Binding}"/>-->
                        <controls:MyTextBlockTest Text="{Binding}" />
                    </DataTemplate>
                </ItemsControl.ItemTemplate>
            </ItemsControl>
        </ScrollViewer>
    </Grid>

Shaves around 10ms off the time:

Even more time can be shaved by moving up an element (that is, inheriting from a more base class than `Control`. In fact, `Control` inherits from `FrameworkElement`:

public class MyTextBlockTest : FrameworkElement

Shaves another 10ms off:

Conclusion

Clearly, this isn’t a huge performance boost, and in 99% of use cases, this would be massively premature optimisation. However, the time that this really comes into its own is where you have a compound control (a control that consists of other controls), and you have lots of them (hundreds). See my next post for details.

References:

https://social.msdn.microsoft.com/Forums/en-US/94ddd25e-7093-4986-b8c8-b647924251f6/manual-rendering-of-a-wpf-user-control?forum=wpf

http://www.codemag.com/article/100023

http://stackoverflow.com/questions/20338044/how-do-i-make-a-custom-uielement-derived-class-that-contains-and-displays-othe

http://stackoverflow.com/questions/42494455/wpf-custom-control-inside-itemscontrol

Getting Started With SignalR

SignalR is an open source framework allowing bi-directional communication between client and server. Basically, it uses a stack of technologies; the idea being that the Signalr framework will establish the “best” way to maintain a bi-directional data stream, starting with web sockets, and falling all the way back to simply polling the server.

The following gives the basics of establishing a web site that can accept Signalr, and a console app that can send messages to it.

Create project

Let’s go MVC:

Hubs

Hubs are the way in which the Signalr service communicates with its clients. Obviously, the term service here may not actually represent a service.

To add a hub class, select the project, right-click and “New Item..”:

This adds the file, along with new references:

The code above that gets added is:

public void Hello()
{
    Clients.All.hello();
}

Clients.All returns a dynamic type, so we lose intellisense at this point. It’s important that the signature of this method is exactly correct, and that it is decorated with the name of the hub, and that it is decorated with the name of the hub; so let’s replace with:


[HubName("MyHub1")]
public class MyHub1 : Hub
{
    public void Hello(string message)
    {
        Clients.All.Hello(message);
    }
}

Change Startup.cs:

public partial class Startup
{
    public void Configuration(IAppBuilder app)
    {
        ConfigureAuth(app);
 
        app.MapSignalR();
    }
}

For all this to actually do anything, the next thing to do is hook up the JavaScript:

$(function () {
    // Declare a proxy to reference the hub. 
    var hub = $.connection.MyHub1;
    // Create a function that the hub can call to broadcast messages.
    hub.client.hello = function (message) {
 
        alert("Hello");
    };
 
    
    $.connection.hub.start()
        .done(function () { console.log("MyHub1 Successfully Started"); })
        .fail(function () { console.log("Error: MyHub1 Not Successfully Started"); })
});

Effectively, once we receive a message, we’re just going to display an alert. Once the event handler is wired up, we try to start the hub.

Next, reference the required files in BundleConfig.cs:

bundles.Add(new ScriptBundle("~/bundles/signalr").Include(
    "~/Scripts/jquery-3.1.1.min.js").Include(
    "~/Scripts/jquery.signalR-2.2.1.js"));

These are referenced in _Layout.cshtml; remember also that, because SignalR references Jquery, you’ll need to remove other references to Jquery:


<title>@ViewBag.Title - My ASP.NET Application</title>
@Styles.Render("~/Content/css")
@Scripts.Render("~/bundles/modernizr")    
@Scripts.Render("~/bundles/signalr")    
<script type="text/javascript" src="~/signalr/hubs"></script>
<script type="text/javascript" src="~/Scripts/Notification.js"></script>

. . .

    </div>
    
    @Scripts.Render("~/bundles/bootstrap")
    @RenderSection("scripts", required: false)
</body>

Notes on Bundles

The purpose of bundling is to shrink the size of the bundled files. The idea being that small files make for a speedy web-site.

Console App

The next step is to create an application that can fire a notification to the page. In this case, I’m using a console app, just because I like to see everything working with console apps.

Start with a NuGet Reference:

The code:

class Program
{
    static void Main(string[] args)
    {
        Console.Write("Message: ");
        string message = Console.ReadLine();
 
        HubConnection connection = new HubConnection("http://localhost:4053/");
        IHubProxy hub = connection.CreateHubProxy("myHub1");
                    
        connection.Start().Wait();
        hub.Invoke<string>("Hello", message).Wait();            
 
        Console.WriteLine("Sent");
        Console.ReadLine();
    }
}

And that’s it – you should be able to send a message to the web site from the console app. The examples that are typically given elsewhere on the net are chat rooms, but this clearly has many more uses.

Some abstract notes that I made while researching this.

Adding:

Version 1

protected void Application_Start()
{
    AreaRegistration.RegisterAllAreas();
 
    >RouteTable.Routes.MapHubs(new HubConfiguration());
 
    FilterConfig.RegisterGlobalFilters(GlobalFilters.Filters);
…

Gives:

Severity Code Description Project File Line Source Suppression State
Error CS0619 ‘SignalRRouteExtensions.MapHubs(RouteCollection, HubConfiguration)’ is obsolete: ‘Use IAppBuilder.MapSignalR in an Owin Startup class. See http://go.microsoft.com/fwlink/?LinkId=320578 for more details.’ SignalRTest3 C:\Users\Paul\documents\visual studio 14\Projects\SignalRTest3\SignalRTest3\Global.asax.cs 18 Build Active

This was for v1 Signal R – superseded in 2.

CORS

During trying to get this working, the prospect of using CORS came up. This enables cross domain requests, which are typically prohibited.

Proxies

The generated Proxy can be viewed (navigate to http://localhost:4053/signalr/hubs):

 $.hubConnection.prototype.createHubProxies = function () {
        var proxies = {};
        this.starting(function () {
            // Register the hub proxies as subscribed
            // (instance, shouldSubscribe)
            registerHubProxies(proxies, true);
this._registerSubscribedHubs();
        }).disconnected(function () {
            // Unsubscribe all hub proxies when we "disconnect".  This is to ensure that we do not re-add functional call backs.
            // (instance, shouldSubscribe)
            registerHubProxies(proxies, false);
        });
proxies['MyHub1'] = this.createHubProxy('MyHub1'); 
        proxies['MyHub1'].client = { };
        proxies['MyHub1'].server = {
            hello: function (message) {
                return proxies['MyHub1'].invoke.apply(proxies['MyHub1'], $.merge(["Hello"], $.makeArray(arguments)));
             }
        };
return proxies;
    };

References:

https://www.asp.net/signalr/overview/guide-to-the-api/hubs-api-guide-javascript-client

https://docs.microsoft.com/en-us/aspnet/signalr/overview/getting-started/tutorial-getting-started-with-signalr

https://docs.microsoft.com/en-us/aspnet/signalr/overview/guide-to-the-api/hubs-api-guide-javascript-client

https://github.com/SignalR/SignalR/wiki/Faq

http://stackoverflow.com/questions/42108193/signalr-test-project-not-working-as-expected

http://www.jeffreyfritz.com/2015/05/where-did-my-asp-net-bundles-go-in-asp-net-5/

Scientist.Net

The purpose of the library is to allow you to try new code in a small sample of production usage – effectively, testing in production. The idea being that if you’re refactoring an important part of the system, you can re-write, and then call your new code on occasion; it’s logged and, should it reveal a major issue, can be simply switched off.

The first port of call is the GitHub repository:

Which adds this:

The following is some test code; there are two methods, an old, slow method, and a refactored new method:


class LegacyCode
{
    public void OldMethod1()
    {
        System.Threading.Thread.Sleep(1000);
        System.Console.WriteLine("This is old code");
    }
}
class RefactoredCode
{
    public void RefactoredNewMethod()
    {
        System.Console.WriteLine("RefactoredNewMethod called");
    }
}
static void Main(string[] args)
{
    System.Console.WriteLine("Start Test");
 
    for (int i = 1; i <= 100; i++)
    {
        Scientist.Science<bool>("Test", testNewCode =>
        {
            testNewCode.Use(() =>
            {
                new LegacyCode().OldMethod1();
                return true;
            });
            testNewCode.Try(() =>
            {
                new RefactoredCode().RefactoredNewMethod();
                return true;
            });
        });
    }
 
    System.Console.ReadLine();
}

In the code above you’ll notice that the call to Scientist looks a little forced – that’s because it insists on a return value from the experiments (and experiment being a trial of new code).

As you can see, Scientist is managing the calls between the new and old method:

One thing that wasn’t immediately obvious to me here was exactly how / what it does with this; especially given that the Try and Use blocks were not always appearing in a consistent order; the following test revealed it more clearly:

Because the order of the runs are randomly altered, I had assumed that which code was called was also randomly determined; in fact, both code paths are run. This is a hugely important distinction, because if you are changing data in one or the other, you need to factor this in.

Statistics

Scientist collects a number of statistics on the run; to see these, you need to implement an IResultPublisher; for example:

public class ResultPublisher : IResultPublisher
{
    public Task Publish<T, TClean>(Result<T, TClean> result)
    {
        System.Console.WriteLine($"Publishing results for experiment '{result.ExperimentName}'");
        System.Console.WriteLine($"Result: {(result.Matched ? "MATCH" : "MISMATCH")}");
        System.Console.WriteLine($"Control value: {result.Control.Value}");
        System.Console.WriteLine($"Control duration: {result.Control.Duration}");
        foreach (var observation in result.Candidates)
        {
            System.Console.WriteLine($"Candidate name: {observation.Name}");
            System.Console.WriteLine($"Candidate value: {observation.Value}");
            System.Console.WriteLine($"Candidate duration: {observation.Duration}");
        }
 
        return Task.FromResult(0);
    }
}

The code in here is executed for every call:

We’ve clearly sped up the call, but does it still do the same thing?

Matches… and mismatches

There’s a lot of information in the trace above. One thing that Scientist.Net does allow you to do is to compare the results of a function; let’s change the initial experiment a little:

public bool OldMethod1(int test)
{            
    System.Threading.Thread.Sleep(1000);
    System.Console.WriteLine("This is old code");
    return test >= 50;
}

public bool RefactoredNewMethod(int test)
{
    System.Console.WriteLine("RefactoredNewMethod called");
 
    return test >= 50;
}

for (int i = 1; i <= 100; i++)
{
    var result = Scientist.Science<bool>("Test", testNewCode =>
    {
        testNewCode.Use(() =>
        {
            return new LegacyCode().OldMethod1(i);                        
        });
        testNewCode.Try(() =>
        {
            return new RefactoredCode().RefactoredNewMethod(i);                        
        });
    });
}

Now we’re returning a boolean flag to say that the number is greater or equal to 50, and returning that. Finally, we need to change ResultPublisher (otherwise we won’t be able to see the wood for the trees:


public Task Publish<T, TClean>(Result<T, TClean> result)
{
    if (result.Mismatched)
    {
        System.Console.WriteLine($"Publishing results for experiment '{result.ExperimentName}'");
        System.Console.WriteLine($"Result: {(result.Matched ? "MATCH" : "MISMATCH")}");
        System.Console.WriteLine($"Control value: {result.Control.Value}");
        System.Console.WriteLine($"Control duration: {result.Control.Duration}");
        foreach (var observation in result.Candidates)
        {
            System.Console.WriteLine($"Candidate name: {observation.Name}");
            System.Console.WriteLine($"Candidate value: {observation.Value}");
            System.Console.WriteLine($"Candidate duration: {observation.Duration}");
        }
    }
 
    return Task.FromResult(0);
}

If we run that:

Everything is the same. So, let’s break the new code:


public bool RefactoredNewMethod(int test)
{
    System.Console.WriteLine("RefactoredNewMethod called");
 
    return test > 50;
}

Now we have a bug in the new code, so what happens:

We have a mismatch. The old code is now behaving differently, and so Scientist has identified this.

Summary

I came across this on this episode of .Net Rocks with Phil Haack. There are more features here, too – you can control the way the comparison works, categorise the results, and so forth.

References

http://haacked.com/archive/2016/01/20/scientist/

https://visualstudiomagazine.com/articles/2016/11/01/testing-experimental-code.aspx

https://github.com/github/Scientist.net

Building Block Game in Unity 3D

Not sure this qualifies as a game, but it’s a computerised version of the building blocks that you might give to a three-year-old. What can I say, it was a nice way to spend a Sunday afternoon !

Here’s what the finished game / program looks like:

The Script

There is only one script:

public class BehaviourScript : MonoBehaviour
{
    
    private Vector3 screenPoint;
    private Vector3 offset;
 
    void OnMouseDown()
    {
        screenPoint = Camera.main.WorldToScreenPoint(gameObject.transform.position);
        offset = gameObject.transform.position - Camera.main.ScreenToWorldPoint(new Vector3(Input.mousePosition.x, Input.mousePosition.y, screenPoint.z));
    }
 
    void OnMouseDrag()
    {
        Vector3 cursorPoint = new Vector3(Input.mousePosition.x, Input.mousePosition.y, screenPoint.z);
        Vector3 cursorPosition = Camera.main.ScreenToWorldPoint(cursorPoint) + offset;
 
        if (cursorPosition.y > 0)
        {
            transform.position = cursorPosition;
        }
    }
}

The Scene

Basically, the blocks are standard unit cubes with a wood texture, a rigid body and the above script attached:

Microsoft Cognitive Services – Text Recognition

Recently at DDD North I saw a talk on MS cognitive services. This came back and sparked interest in me while I was looking at some TFS APIs (see later posts for why). However, in this post, I’m basically exploring what can be done with these services.

The Hype

  • Language: can detect the language that you pass
  • Topics: can determine the topic being discussed
  • Key Phrases: key points (which I believe may equate to nouns)
  • Sentiment: whether or not what you are saying is good or bad (I must admit, I don’t really understand that – but we can try some phrases to see what it comes up with)

For some reason that I can’t really understand, topics requires over 100 documents, and so I won’t be getting that to work, as I don’t have a text sample big enough. The examples that they give in marketing seem to relate to people booking and reviewing holidays; and it feels a lot like these services are overly skewed toward that particular purpose.

Set-up

Register here:

https://www.microsoft.com/cognitive-services/

Registration is free (although I believe you need a live account).

cog1

Client

The internal name for this at MS is Project Oxford. You don’t have to install the client libraries (because they are just service calls), but you get some objects and helpers if you do:

cog2

Cognition

The following code is largely plagiarised from the links at the bottom of this page:

Here’s the Main function:

var requestDocs = PopulateDocuments();
Console.WriteLine($"-=Requests=-");
foreach (var eachReq in requestDocs.Documents)
{
    Console.WriteLine($"Id: {eachReq.Id} Text: {eachReq.Text}");
}
Console.WriteLine($"-=End Requests=-");
 
string req = JsonConvert.SerializeObject(requestDocs);
 
MakeRequests(req);
Console.WriteLine("Hit ENTER to exit...");
Console.ReadLine();

PopulateDocuments just fills the RequestDocument collection with some test data:


private static LanguageRequest PopulateDocuments()
{
    LanguageRequest requestText = new Microsoft.ProjectOxford.Text.Language.LanguageRequest();
    requestText.Documents.Add(
        new Microsoft.ProjectOxford.Text.Core.Document()
        { Id = "One", Text = "The quick brown fox jumped over the hedge" });
    requestText.Documents.Add(
        new Microsoft.ProjectOxford.Text.Core.Document()
        { Id = "Two", Text = "March is a green month" });
    requestText.Documents.Add(
        new Microsoft.ProjectOxford.Text.Core.Document()
        { Id = "Three", Text = "When I press enter the program crashes" });
    requestText.Documents.Add(
        new Microsoft.ProjectOxford.Text.Core.Document()
        { Id = "4", Text = "Pressing return - the program crashes" });
    requestText.Documents.Add(
        new Microsoft.ProjectOxford.Text.Core.Document()
        { Id = "5", Text = "Los siento, no hablo Enspanol" });
 
    return requestText;
}

As you can see, I dropped some Spanish in there for the language detection. The MakeRequests method and its dependencies:


static async void MakeRequests(string req)
{
    using (var client = new HttpClient())
    {
        client.BaseAddress = new Uri(BaseUrl);
 
        // Request headers.
        client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", AccountKey);
        client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
 
        // Request body. Insert your text data here in JSON format.
        byte[] byteData = Encoding.UTF8.GetBytes(req);
 
        // Detect key phrases:
        var uri = "text/analytics/v2.0/keyPhrases";
        string response = await CallEndpoint(client, uri, byteData);
        Console.WriteLine("Key Phrases");
        Console.WriteLine(ParseResponseGeneric(response));
 
        // Detect language:
        var queryString = HttpUtility.ParseQueryString(string.Empty);
        queryString["numberOfLanguagesToDetect"] = NumLanguages.ToString(CultureInfo.InvariantCulture);
        uri = "text/analytics/v2.0/languages?" + queryString;
        response = await CallEndpoint(client, uri, byteData);
        Console.WriteLine("Detect language");
        Console.WriteLine(ParseResponseLanguage(response));
 
        // Detect topic:
        queryString = HttpUtility.ParseQueryString(string.Empty);
        queryString["minimumNumberOfDocuments"] = "1";
        uri = "text/analytics/v2.0/topics?" + queryString;
        response = await CallEndpoint(client, uri, byteData);
        Console.WriteLine("Detect topic");
        Console.WriteLine(ParseResponseGeneric(response));
 
        // Detect sentiment:
        uri = "text/analytics/v2.0/sentiment";
        response = await CallEndpoint(client, uri, byteData);
        Console.WriteLine("Detect sentiment");
        Console.WriteLine(ParseResponseSentiment(response));
    }
}
private static string ParseResponseSentiment(string response)
{
    if (!string.IsNullOrWhiteSpace(response))
    {
        SentimentResponse resp = JsonConvert.DeserializeObject<SentimentResponse>(response);
        string returnVal = string.Empty;
 
        foreach (var doc in resp.Documents)
        {
            returnVal += Environment.NewLine +
                $"Sentiment: {doc.Id}, Score: {doc.Score}";
        }
 
        return returnVal;
    }
 
    return null;
}
 
private static string ParseResponseLanguage(string response)
{
    if (!string.IsNullOrWhiteSpace(response))
    {
        LanguageResponse resp = JsonConvert.DeserializeObject<LanguageResponse>(response);
        string returnVal = string.Empty;
        foreach(var doc in resp.Documents)
        {
            var detectedLanguage = doc.DetectedLanguages.OrderByDescending(l => l.Score).First();
            returnVal += Environment.NewLine +
                $"Id: {doc.Id}, " +
                $"Language: {detectedLanguage.Name}, " +
                $"Score: {detectedLanguage.Score}";
        }
        return returnVal;
    }
 
    return null;
}
 
private static string ParseResponseGeneric(string response)
{            
    if (!string.IsNullOrWhiteSpace(response))
    {
        return Environment.NewLine + response;                
    }
 
    return null;
}

The subscription key is given when you register (in the screen under “Set-up”). Keep an eye on the requests, too: 5000 seems like a lot, but when you’re testing, you might find you get through them faster than you expect.

Here’s the output:

cog3

Evaluation

So, the 5 phrases that I used were:

The quick brown fox jumped over the hedge

This is a basic sentence indicating an action.

The KeyPhrases API decided that the key points here were “hedge” and “quick brown fox”. It didn’t think that “jumped” was key to this sentence.

The Language API successfully worked out that it’s written in English.

The Sentiment API thought that this was a slightly negative statement.

March is a green month

This was a nonsense statement, but in a valid sentence structure.

The KeyPhrases API identified “green month” as being important, but not March.

The Language API successfully worked out that it’s written in English.

The Sentiment API thought this was a very positive statement.

When I press enter the program crashes

Again, a completely valid sentence, and with a view to my idea ultimate idea for this API.

The KeyPhrases API spotted “program crashes”, but not why. I found this interesting because it seems to conflict with the other phrases, which seemed to identify nouns only.

Again, the Language API knew this was English.

The sentiment API identified that this was a negative statement… which I think I agree with.

Pressing return – the program crashes

The idea here was, it’s basically the same sentence as above, but phrased differently.

The KeyPhrases API wasn’t fooled, and returned the same key phrase – this is good.

Still English, according to the Language API.

This is identified as a negative statement again, but oddly, not as negative as the previous one.

Los siento, no hablo Enspanol

I threw in a Spanish phrase because I felt the Language API hadn’t had much of a run.

The KeyPhrase API pulled out “hablo Espanol”, which based on my very rudimentary Spanish, means the opposite of that was said.

It was correctly identified as Spanish by the Language API.

The Sentiment API identified it as the most negative statement. Perhaps because it has the word “sorry” and “no” in it?

References

Sample code:

https://text-analytics-demo.azurewebsites.net/Home/SampleCode

https://elbruno.com/2016/04/13/cognitiveservices-text-analytics-api-new-operation-detect-key-topics-in-documents/

https://mrfoxsql.wordpress.com/2016/09/13/azure-cognitive-services-apis-with-sql-server-2016-clr/

Manually Parsing a JSON String Using JSON.NET

How to manually parse a JSON string using JSON.NET.

Disclaimer

If you jump straight to the references, you will find a very similar set of information, and I strongly encourage people to do so. Additionally, this is probably not the most efficient way to achieve this.

Right, on with the show

Here’s the string that I’ll be parsing, and a little code stolen directly from the link at the bottom to show what it looks like:

static void Main(string[] args)
{
    string json = "{\"documents\":[{\"keyPhrases\":[\"Test new bug\"],\"id\":\"1\"}],\"errors\":[]}";
    JsonTextReader reader = new JsonTextReader(new StringReader(json));
    while (reader.Read())
    {
        if (reader.Value != null)
        {
            Console.WriteLine("Token: {0}, Value: {1}", reader.TokenType, reader.Value);
        }
        else
        {
            Console.WriteLine("Token: {0}", reader.TokenType);
        }
    }
 
    Console.ReadLine();
}

The output for this looks like:

Using this, it’s easier to create a routine to manually parse this. Each object can be tracked by using the Start and EndObject tags. Here’s my unit test to check this works:

[TestMethod]
public void TestJSONParse()
{
    // Arrange
    string json = "{\"documents\":[{\"keyPhrases\":[\"Test new bug\"],\"id\":\"1\"}],\"errors\":[]}";
    // Act
    var result = JsonHelper.ParseResponse(json);
 
    // Assert
    Assert.AreEqual(1, result.Count());
    Assert.AreEqual(1, result.Keys.First());
    string expectedPhrase = result.Values.First().First().ToString();
    Assert.AreEqual("Test new bug", expectedPhrase, false);
}

And here’s the code itself:

/// <summary>
/// Parse the following JSON
/// {"documents":[{"keyPhrases":["Test new bug"],"id":"1"}],"errors":[]}
/// </summary>
/// <param name="response"></param>
/// <returns></returns>
public static Dictionary<int, List<string>> ParseResponse(string response)
{
    Dictionary<int, List<string>> dict = new Dictionary<int, List<string>>();
    object readerValue;
 
    if (!string.IsNullOrWhiteSpace(response))
    {
        JsonTextReader reader = new JsonTextReader(new StringReader(response));
        int? currentValue = null;
        List<string> currentList = null;
 
        while (reader.Read())
        {
            readerValue = reader.Value;
 
            switch (reader.TokenType)
            {
                case JsonToken.PropertyName:                            
                    if (readerValue.ToString() == "id")
                    {
                        reader.Read();
                        currentValue = int.Parse(reader.Value.ToString());                                
                    }
                    else if (readerValue.ToString() == "keyPhrases")
                    {
                        // Do nothing
                    }
                    else if (readerValue.ToString() == "errors")
                    {
                        currentValue = null;
                    }
                    break;
 
                case JsonToken.String:                            
                    currentList.Add(reader.Value.ToString());
                    break;
 
                case JsonToken.StartArray:
                    currentList = new List<string>();
                    break;
 
                case JsonToken.StartObject:
                    currentList = null;
                    currentValue = null;
                    break;
 
                case JsonToken.EndObject:
                    if (currentValue.HasValue)
                    {
                        dict.Add(currentValue.Value, currentList);
                    }
                    break;
            }
        }
    }
 
    return dict;
}

It is messy, and it is error prone, and it would be better done by creating classes and serialising it; however, I’d never attempted to do this manually before, and it’s generally nice to do things the hard way, that way, you can appreciate what you get from these tools.

References

http://www.newtonsoft.com/json/help/html/ReadJsonWithJsonTextReader.htm

Finding Duplicate Values in a Dictionary Using C#

Due to a series of blog posts that I’m writing on TFS and MS Cognitive Services, I came across a requirement to identify duplicate values in a dictionary. For example, imagine you had an actual physical dictionary, and you wanted to find all the words that meant the exact same thing. Here’s the set-up for the test:

Dictionary<int, string> test = new Dictionary<int, string>()
{
    { 1, "one"},
    { 2, "two" },
    { 3, "one" },
    { 4, "three" }
};
DisplayDictionary("Initial Collection", test);

I’m outputting to the console at every stage, so here’s the helper method for that:


private static void DisplayDictionary(string title, Dictionary<int, string> test)
{
    Console.WriteLine(title);
    foreach (var it in test)
    {
        Console.WriteLine($"Key: {it.Key}, Value: {it.Value}");
    }
}

Finding Duplicates

LINQ has a special method for this, it’s Intersect. For flat collections, this works excellently, but no so well for Dictionaries; here was my first attempt:


Dictionary<int, string> intersect = test.Intersect(test)
    .ToDictionary(i => i.Key, i => i.Value);
DisplayDictionary("Intersect", intersect);

As you can see, the intersect doesn’t work very well this time (don’t tell Chuck).

Manual Intersect

The next stage then is to roll your own; a pretty straightforward lambda in the end:


var intersect2 = test.Where(i => test.Any(t => t.Key != i.Key && t.Value == i.Value))
    .ToDictionary(i => i.Key, i => i.Value);
DisplayDictionary("Manual Intersect", intersect2);
 

This works much better.

Change a TFS Work Item Definition

Let’s assume that the built-in TFS standard templates are not sufficient for you. You’re in luck: TFS allows you to create your own custom work item. Let’s imagine that, for some reason, you want a work item type called “Defect”, rather than “Bug”. Here’s the process, based on the “Bug” work item type.

First thing is to open the command prompt in administrator mode, and navigate to a work directory; for example, the “Documents” folder
.
Then export the template work item type, like so (you can export the entire definition of all work items, but it becomes unmanageable):

witadmin exportwitd /collection:http://tlaptop:8080/tfs/DefaultCollection /p:"TFSSandbox" /n:Bug /f:Defect.xml

customwi1

Check your working directory:

customwi2

Now, open the XML file. I used Notepad++, but you can use Notepad or vi, or whatever. To change the work item, change the name attribute; once you’ve done this, you have a new work item type:

customwi3

Now, you can add the fields that you want. I’m adding a field called “pcm.CustomField”:

<?xml version="1.0" encoding="utf-8"?>
<witd:WITD application="Work item type editor" version="1.0" xmlns:witd="http://schemas.microsoft.com/VisualStudio/2008/workitemtracking/typedef">
  <WORKITEMTYPE name="Defect">
    <DESCRIPTION>Customised defect</DESCRIPTION>
    <FIELDS>
      <FIELD name="Iteration Path" refname="System.IterationPath" type="TreePath" reportable="dimension">
        <HELPTEXT>The iteration within which this bug will be fixed</HELPTEXT>
      </FIELD>
	  <FIELD name="New Field" refname="pcm.CustomField" type="Integer" />
      <FIELD name="Iteration ID" refname="System.IterationId" type="Integer" />

The field name attribute (“New Field” in this case) tells TFS how to refer to this field to the user. This is important, because if you forget that you’ve called it “New Field”, you might assume this hasn’t worked and start googling to find out why.

Now you’ve told TFS that you have a field to store; the next step is to add that field to the layout:

    <FORM>
      <Layout HideReadOnlyEmptyFields="true" HideControlBorders="true">
        <Group Margin="(4,0,0,0)">
          <Column PercentWidth="90">
            <Control FieldName="System.Title" Type="FieldControl" ControlFontSize="large" EmptyText="&lt;Enter title here&gt;" />
          </Column>
          <Column PercentWidth="10">
            <Control FieldName="System.ID" Type="FieldControl" ControlFontSize="large" />
          </Column>
        </Group>
     <Group Margin="(10,0,0,0)">
          <Column PercentWidth="60">
            <Control FieldName="pcm.CustomField" Type="FieldControl" Label="Custom Field Here" ControlFontSize="large" EmptyText="&lt;Custom field here&gt;" />
          </Column>		
    </Group>
        <Group Margin="(10,0,0,0)">

As you can see, we can move this about and pretty much show anything we want here. Next, re-import:

witadmin importwitd /collection:http://tlaptop:8080/tfs/DefaultCollection /p:TFSSandbox /f:Defect.xml

customwi5

Using the new Work Item Type

And now we have a new work item type:

customwi6

And the new field is available:

customwi7

Using The Field in a Query

customwi8

References

Tutorial on modifying Work Item States

Tutorial on customising TFS 2013

Detailed definition of the FIELD tag

Turotial on modifying the FIELD tag

Programmatically List Existing Tags Using The TFS API

Having looked into this for some time; I came up with the following method of extracting team project tags. I’m not for a minute suggesting this is the best way of doing this – but it does work. My guess is that it’s not a very scalable solution, as it’s doing a LOT of work.

As it was, I couldn’t find a way to directly query the tags, so instead, I’m going through all the work items, and picking the tags. I couldn’t even find a way to filter the work items that actually have tags; so here’s the query that I ended up with:

private static IEnumerable<string> GetAllDistinctWorkItemTags(string uri, string projectName)
{
    TfsTeamProjectCollection tfs;
 
    tfs = TfsTeamProjectCollectionFactory.GetTeamProjectCollection(new Uri(uri)); // https://mytfs.visualstudio.com/DefaultCollection
    tfs.Authenticate();
 
    var wis = new WorkItemStore(tfs);
 
    WorkItemCollection workItemCollection = wis.Query(
         " SELECT [System.Tags]" +
         " FROM WorkItems " +
         $" WHERE [System.TeamProject] = '{projectName}' ");                
 
    if (workItemCollection.Count == 0)
        return null;
 
    List<string> tags = new List<string>();
    foreach (WorkItem wi in workItemCollection)
    {
        if (string.IsNullOrWhiteSpace(wi.Tags)) continue;
 
        var splitTags = wi.Tags.Split(';');
        tags.AddRange(splitTags.ToList());                
    }
 
    return tags.Distinct();
}

From debugging, I strongly suspect that whatever you put in the “SELECT”, it returns the entire work item. I also, for the life of me, couldn’t work out a lambda query for parsing the tags.

The calling method is here:

public static IEnumerable<string> GetAllTags(string uri, string teamProject)
{
    var project = GetTeamProject(uri, teamProject);
    IEnumerable<string> tags = GetAllDistinctWorkItemTags(uri, teamProject);
 
    return tags;
}

I’ve listed GetTeamProject helper method before, but for the sake of completeness:


public static Project GetTeamProject(string uri, string name)
{
    TfsTeamProjectCollection tfs;
 
    tfs = TfsTeamProjectCollectionFactory.GetTeamProjectCollection(new Uri(uri)); // https://mytfs.visualstudio.com/DefaultCollection
    tfs.Authenticate();
 
    var workItemStore = new WorkItemStore(tfs);
    
    var project = (from Project pr in workItemStore.Projects
                       where pr.Name == name
                       select pr).FirstOrDefault();
    if (project == null)
        throw new Exception($"Unable to find {name} in {uri}");
 
    return project;
}

Here’s the output:

tags1

Notes on Tags

A couple of points on tags: firstly, tags seem to exist in a kind of transient state; that is, while something is tagged, the tag exists, but once you remove all instances of a tag (for example, if I removed “Tagtest1” from all work items in my team project, TFS would eventually (I believe after a couple of days) just delete the tag for me. Obviously, in my example, as soon as I did this, I would no longer find it. This might leave you thinking that there is a more efficient way of removing tags (that is, you should be able to access the transient store in some way).

The existence of this Visual Studio plug-in lends support to that idea. It allows you to maintain the tags within your team project. If you’re using tags in any kind of serious way then I’d strongly recommend that you try it.

Performance

This is doing a lot of (IMO) unnecessary work, so I tried a little performance test; using this post as a template, I created a lot of bugs:

tags2

As you can see, I created a random set of tags. One other point that I’m going to put here is that a TFS database with ~30K work items and no code whatsoever increases the size of the default collection DB to around 2GB:

tags3

Now I ran the GetAllTags with some timing on:

tags4

19 seconds, which seems like quite a reasonable speed to me for 13.5k tags.