Add Evaluation to ML.NET Model

November 21, 2020

I’ve recently been playing around with ML.NET. I’ve documented some of my escapades here.

One thing that I found was that, when trying to work out how effective the model created from the data was, I was manually rifling through the data that I had: having to essentially compare each result. As a result, I created EvaluateMLNet. It’s a small NuGet package that essentially does this for you.

Step 1 - Import the package

If you follow the previous post, you’ll find yourself in a position where you have a Model project and a ConsoleApp project:

In order to use the package, start by importing the NuGet package into the ConsoleApp project:



Install-Package EvaluateMLNet

Step 2 - Add the data

The next stage is to have some data to test your model against. Add this to your ConsoleApp project, and remember to set the Copy if Newer or Copy Always on the file.

Step 3 - Code

The main program will look something like this to begin with:



        static void Main(string[] args)
        {
            // Create single instance of sample data from first line of dataset for model input
            ModelInput sampleData = new ModelInput()
            {
                Season = @"2019-2020",
                Time = @"7:00pm",
                Home\_team\_name = @"Liverpool",
                Away\_team\_name = @"Norwich City",
                Referee = @"Michael Oliver",
                Stadium\_name = @"Anfield (Liverpool)",
            };

            // Make a single prediction on the sample data and print results
            var predictionResult = ConsumeModel.Predict(sampleData);

            Console.WriteLine($"Season: {sampleData.Season}");
            Console.WriteLine($"Time: {sampleData.Time}");
            Console.WriteLine($"Home\_team\_name: {sampleData.Home\_team\_name}");
            Console.WriteLine($"Away\_team\_name: {sampleData.Away\_team\_name}");
            Console.WriteLine($"Referee: {sampleData.Referee}");
            Console.WriteLine($"Stadium\_name: {sampleData.Stadium\_name}");
            Console.WriteLine($"\\n\\nPredicted home\_team\_score: {predictionResult.Score}\\n\\n");
            Console.WriteLine("=============== End of process, hit any key to finish ===============");
            Console.ReadKey();
        }

Instead of that, start by extracting the Predict method - that is everything after:



// Make a single prediction on the sample data and print results

This should give you:



        static void Main(string[] args)
        {
            // Create single instance of sample data from first line of dataset for model input
            ModelInput sampleData = new ModelInput()
            {
                Season = @"2019-2020",
                Time = @"7:00pm",
                Home\_team\_name = @"Liverpool",
                Away\_team\_name = @"Norwich City",
                Referee = @"Michael Oliver",
                Stadium\_name = @"Anfield (Liverpool)",
            };

            PredictData(sampleData);
        }

        private static float PredictData(ModelInput sampleData)
        {
            // Make a single prediction on the sample data and print results
            var predictionResult = ConsumeModel.Predict(sampleData);

            Console.WriteLine($"Season: {sampleData.Season}");
            Console.WriteLine($"Time: {sampleData.Time}");
            Console.WriteLine($"Home\_team\_name: {sampleData.Home\_team\_name}");
            Console.WriteLine($"Away\_team\_name: {sampleData.Away\_team\_name}");
            Console.WriteLine($"Referee: {sampleData.Referee}");
            Console.WriteLine($"Stadium\_name: {sampleData.Stadium\_name}");

            return predictionResult.Score;
        }

Note that we’re also returning the result of the prediction. In fact, that method only needs to return the result of the prediction - the Console.WriteLines are unnecessary.

Finally, replace the Main method with the following:



        static void Main(string[] args)
        {
            var runEvaluation = new RunEvaluation();
            var resultStats = runEvaluation.Run<ModelInput, float>("my-data-file.csv",
                "Predicted\_field\_name", PredictData, 0);

            Console.WriteLine("Results");
            Console.WriteLine("Total evaluated results: {0}", resultStats.EvaluatedCount);
            Console.WriteLine("Total success results: {0}", resultStats.SuccessCount);
            Console.ReadLine();            
        }

A few comments about this code:

1. The **"Predicted\_field\_name"** is the name of the field in the class **ModelInput**.  It's very likely to have a capitalised first letter.
2. My data is predicting a **float** - if yours is not then you'll need to change this.
3. The margin of error here is 0; that means that a prediction is only considered a success where it's within the same integer; for example, if the prediction was 1.3, then 1 and 2 would be considered a success, but 0 and 3 would not.

That’s it, the output will give you something like this:

Summary

I realise that this is feeding a very niche crowd, but hopefully it’ll save someone a Saturday afternoon.