AIDA Excellence Lecture 39: Predicting Classifier Accuracy in the Wild

[@costadelle]

00:00 - Intro
01:17 - Predicting classifier accuracy
04:19 - Dataset Shift
05:22 - Causes of Dataset Shift
08:00 - Factorizing the Joint Probability Distribution
10:09 - Covariate Shift
11:47 - Prior Probability Shift
13:37 - Concept Shift
14:48 - Classifier Accuracy Prediction under DS
15:59 - A New Method for CAP under DS
18:02 - Observation 1
18:43 - Idea 1
21:37 - Observation 2
24:14 - Focus of the present work
24:48 - The 1 ×4 method
27:40 - The 2 ×2 method
29:43 - The 1 ×3 method
32:18 - QuAcc - Additional covariates
33:53 - Experimental Setup - A few experiments
36:48 - Experimental Setup - Datasets
39:14 - Experimental Setup - Baselines
40:06 - Experimental Setup - Optimisation
40:51 - Experimental Setup - Overall Results for Vanilla Accuracy
41:41 - Experimental Setup - Overall Results for F1
47:04 - Experimental Setup - Vanilla Accuracy on IMDB
48:09 - Experimental Setup - F1 on IMDB
48:43 - Vanilla Accuracy on IMDB - Naive
50:07 - Vanilla Accuracy on IMDB - RCA
50:23 - Vanilla Accuracy on IMDB - ATC
50:45 - Vanilla Accuracy on IMDB - DoC
50:57 - Vanilla Accuracy on IMDB - QuAcc(CC)
51:16 - Vanilla Accuracy on IMDB - QuAcc(SLD)
51:31 - Vanilla Accuracy on IMDB - QuAcc(KDEy)
51:53 - Lessons learned
53:45 - Questions
54:24 - Chen's question 1
58:11 - Chen's question 2
1:00:39 - Chen's question 3
01:05:31 - Pitas' question
01:07:31 - Chen's question 4