Geplaatst door I’m working through a small list of questions related to machine learning. They’ve been nagging me for a long time, and it’s unlikely I’ll encounter them in practice. So let’s handle another one: “Creating a test set seemed easy at uni, but isn’t it actually quite difficult?”
At university in 2008, I learned machine learning. One of the steps we discussed was how to generate a test set. I remember a few scenario’s:
- Easy: just randomly sample some data
- Advanced: do stratified sampling for classification with imbalanced classes
- Timeseries: think carefully whether you need a slice of time, or a slice of cases/users/..
This post is not about these scenario’s, but about the sampling: what if I have bad luck in the sampling and get a bad test set?
Let’s be positive and reframe this to: “what makes a good test set”? To me, this seems like a difficult issue. When training a machine learning model, I’m trying to solve/improve some business problem. This requires a lot of domain knowledge, and a thorough understanding of the problem.
I recognize such understanding (or the gathering thereof) in the creation and selection of features, and in choosing a suitable algorithm/model family. But not in anything related to the creation of a test set. Which seems weird: it’s the final referee on the quality of a model.
For the rest of the post, let’s just keep things simple: assume that I’m dealing with a balanced binary classification task, and that there is no drift in data or in concept.
These are a few of the thoughts I’ve had over the years on creating a test set:
- I suppose when I have trained a model well, then I would be happy to change my initial seed and blindly deploy the model to production – without any further checks. Right?
- I’m trying to learn a pattern that captures the labels well. I don’t know the pattern, so how do I know if the sampling doesn’t break the pattern?
- I guess if the data set is large enough, it’s unlikely that I have bad luck – but still, I might.
- What if the data has clusters, and some of these are important for prediction. There are no guarantees that they are preserved in sampling.
- Same as with the clusters, but now with low-dimensional manifolds.
Maybe there are ways to deal with clusters, manifolds, and other structural properties of both sets. But still: does anyone do that in practice?
I’ve been ’talking’ to my new buddy Lumo about it. My gut feeling still tells me that it’s not necessarily easy to create a good test set. Perhaps I should be more pragmatic: just try some stuff, run it in production for a while, learn a few lessons, and try again. Or perhaps I should not 🙂
When writing this blog, I considered writing about ‘how to create a good test set’, but decided not to. I don’t do machine learning in practice, and going down that path will probably lead to a bunch of special cases, exceptions, etc. – which I’m not going to use anyway. Also, right now I’m mostly interested in investigating my own intuition on this topic.
Edit, a little while later. I found this great idea:
- After selecting a test set, recombine it with the rest and add a column ‘in_test_set’ (1/0). Then train a simple classifier for ‘in_test_set’, and check whether AUC~0.5. Clever!