Geplaatst door Do you know those small nagging questions that keep popping up? Things I want to look up, but never do? Well, I’m dealing with some of them. The first few are related to machine learning. I’m also using this as an excuse to continue my PKM activities in Obsidian.
Suppose you fit a machine learning model, and it learns to predict a certain label. During training, it achieves a score, and of course I do my best to improve this score as much as I can. In practice, I quit when the classifier generalizes sufficiently well, and then just put it to the test in real life.
But what if I had all the time and resources I wanted: what is the best score I can achieve?
Whenever I’m working on a real-life problem (which is not often, granted), I want to really understand it – deeply. I love talking to domain experts, and learning all the tiny ins and outs of their job. So it’s kind of annoying if stuff is happening that I don’t understand.
Prior to writing this blog, I couldn’t imagine a way to somehow estimate “the amount of intrinsic noise”. After sparring with Lumo for a while, I still don’t think it’s possible. Lumo only suggests techniques and tricks that indicate that I’m maxing out on the score, but nothing to estimate or prove it.
There is something called the Bayes Error Rate, defined on Wikipedia as “the lowest possible error rate for any classifier of a random outcome […] and is analogous to the irreducible error“. The trouble is: you can’t actually calculate it. It requires knowledge of the true conditional probabilities – i.e., the thing you’re trying to learn with ML (from a sample of data as well).
So, I guess it’s time to put this nagging question to rest 🙂
