Is there a way to set a threshold manually or other ways I could use to maximize my model RECALL ?

[piotr-bratek]

Hi,

I haven't found a solution which allows for changing model settings in a way which maximizes my model RECALL.
Let's say I need to distinguish as many fraudulent transactions as possible, so I don't care so much about precision.

Thx
P

[michalnurzynski]

Hi Piotr

I'm not sure if I got your point but both intelligent model optimization and hyperparameters optimization always aims for the best F1 and there is no way to change it. Not yet, anyway, since e.g. python equivalents for hyperparameters optimization have a “scoring” argument that takes the metric to optimize for during random search and grid search, so there is a room for that in AutoML as well.

But since AutoML shows Recall/Precision along with other scoring metrics after each training cycle, it's your task to choose which model suits you best. In your case, just choose the model with the best Recall and forget about other metrics.

Regards

M.

[piotr-bratek]

Let me give you an example. F1 is just an indicator and AutoML's goal is to maximize it.
However F1 balances between Recall and Precision.
Depending on your business case you may focused on one or the other.
For Example if we talk about some medial diagnosis, regarding some hard disease your goal is to minimize Miss Rate as no one wants to miss the fact that your patient is seriously sick.
That's why I'm less focused on precision. I should not miss anyone who is sick and I'm ready to pay for it with lower precision. That means I might want to change the threshold set by AutoML - perhaps optimal from a technical perspecitve (F1) but imperfect from a "business" perspective.

So my question is: can we somehow change the threshold manually?

[piotr-bratek]

In this case, it is very likely it will end with a selection of the highest recall. Yet my questions is a sort of binary classification - yes or no 😉 so I guess it is NO. There are some examples where it is not such a clear cut case so I was wondering if there is some secretly hidden "threshold slider" 😉

[michalnurzynski]

There's a lot of cases where one metric e.g. Recall is crucial, that's for sure, and it can be improved via manual model optimization. So it's not just a model "selection of the highest recall" but conscious changes of training dataset to improve that crucial metric in subsequent training cycles. 

[Msandeepkumar]

Hi Piotr,

Have you got the solution, Or other way to set the threshold cutoff manually.

Regards,

Sandeep Maurya

[igoralcantara]

It is an ugly workaround but you can define manually the threshold if you load the predicted probabilities and use an if statement to define the result, like:


if(predicted_class_1 >= 0.65, 'Yes', 'No')