Problem with Practice Exam question

[Rachael_F]

In the question:

****************************

Category: MLS – Modeling

A Machine Learning Specialist at a mobile network company is tasked to develop a model that predicts customers who are likely to cancel subscriptions. Since the customer churn costs about $15, the company plans on giving an $8 retention incentive for unhappy customers.

The model was evaluated on a test dataset of 100 customers. The confusion matrix for the model is given below:

Why is this model feasible to deploy in production?

****************************

I don’t think the model is feasible to deploy in production. I agree with all the calculations in the answer, but not with the conclusions.

If the company does NOT deploy the model, they would lose 11 customers (8 + 3 Actual Yes’s). At $15 for each lost customer they would lose $165. They would pay no retention incentives, so total loss = $165.

If the company does deploy the model they will pay $8 each to the 16 predicted yes’s (8 TP and 8 FP) so they would pay out $128. They would still lose 3 customers to churn (FN) at $15 per customer so they would lose $45 there. Even assuming every single customer who was paid a retention incentive stays, using the model to offer retention incentives results in a loss of $128 + $45 = $173, which is more than the $165 they would have lost if they didn’t deploy the model. Therefore it is not feasible to deploy this model in production. The real question is whether the cost of the FP is less than the savings of the TP. If they had lost their 8 TP customers they would have lost $15 each, instead they only lost $8 each on those customers by paying them the retention incentive, so they saved $56 on their TP customers. They paid $8 to each of the 8 FP customers so they lost $64 on their FP, which is more than the $56 they saved.

[Carlo-TutorialsDojo]

Hello Rachael_F,

Thank you for providing such an in-depth analysis of the question. I agree with what you said. I actually wrote this question, and it appears that I only focused on the cost of FN and FP and neglected to consider other variables. I apologize for this; it was a silly mistake on my part. It’s either the model’s accuracy is simply not high enough for the company to benefit or they’ll have to adjust the incentive. I’ll revise this item to address the problem.

Let me know if you have further questions.

Regards,

Carlo @ Tutorials Dojo