Title: ETH_Sem6 / DC / W06_ModelSelectionAndValidation

Q: Give a general pipeline for selection and validation of an ML model.
A: 1. Select the best ML algorithm for the problem at hand
2. Select the best hyperparameters for that algorithm
3. Validate that it performs well on new data
4. Validate the applicability of the model to the problem at hand

Q: What are the 3 important errors (which can be calculated) when testing an ML model?
A: - When the model is evaluated on the same data it was trained on: \(E_\text{train} = \frac{1}{n}\sum_{i=1}^n L(y_i, \hat{f}(x_i))\)
- When the model is evaluated on a test data set (X,Y): \(E_\text{test} = E\left[L\left(Y_{n+1}, \hat{f}(X_{n+1})\right)|(X,Y)\right]\)
- When the model is trained on different training sets and evaluated on different test sets (e.g. Cross-Validation): \(E_\text{Expected Prediction} = E\left[E_\text{test}\right]\)

Q: Resubstitution Evaluation
A: - Model is trained and evaluated on the same data set.
- \(E_\text{train}\) is taken as optimistic guess for the generalization (true) error.
- We cannot assess if the model will gerneralize to unseen data.
https://arxiv.org/pdf/1811.12808

Q: Hold-out train and test method (4)
A: - Data is split in test- and training set
- Training on train set and testing on test set
- The error form the hold-out method is on the pessimistic side, as the model was trained with a smaller dataset the the full one
- Problem: Only works if the model to use is already decided; Adjusting parameters according to the test data would mean data-snooping

Q: What is data-snooping? (3)
A: - If the model has effect on any step of the learning process, its ability to assess the outcome has been compromised.
- "If you torture the data long enough, it will confess"
- This is a common trap for which one can fall

Q: Train-validation-test holdout (4)
A: - In addition to test and training set, here the training set is further split in to training and validation sets
- The validation is used to select the best model (the best hyperparameters for a chosen algorithm)
- After the best performing model on the validation set has been found the train and validation sets can be merged to form a larger data set to train the best model on. This results in a less pessimistic prediction.
- After the performance is evaluated the model can be trained on the whole data to further lower the \(E_\text{test}\)
https://arxiv.org/pdf/1811.12808

Header: Which type of hyperparameter optimization is shown below the red mask?

Question: What are cons of the holdout method?
QType (0=kprim,1=mc,2=sc): 1
Q_1: Slightly optimistic
Q_2: Risk of overfitting on a single validation set
Q_3: No uncertainty estimates
Answers: 0 1 1
Extra 1: The error predicted by the holdout method is slightly pessimistic due to the smaller size of the training data set.

Q: Why should one do feature scaling of the data?
A: - Can increase model performance.
- Models that rely on distances (e.g. KNN) are strongly effected
- PCA also strongly effected, as large values lead to large eigenvalues

Q: Resampling
A: - Inital data is randomly sampled over and over again to generate more train/test splits.
- With each resampling the distribution of datapoints is slightly altered.
- As "[most] supervised learning algorithms for classification and regression as well as the performance estimates operate under the assumption that a dataset is representative of the
population that this dataset sample has been drawn from" (https://arxiv.org/pdf/1811.12808; v3) resampling is more applicable if the datasets are larger, as seen below.

Q: Monte Carlo cross-validation
A: - Make multiple random train/test splits
- Estimates generalization and standard error

Q: k-fold cross-validation
A: - Main Idea: Each sample in the data set should have the opportunity of being tested
- In k interations over the data set the data is split into k parts of which one is used for validation and the other \(k-1\) are used for training
- This approach should reduce the pessimistic bias which is larger if the dats set is smaller
- Nested CV is more expensive but can prevent overfitting on the validation set

Header: To which regime do the arrows point?

Q: What is the difference between validation curves and learning curves?
A: - For Validation Curves the x-axis it's the value of the hyperparameter
- For Learning Curves the x-axis it's the number of training examples (i.e data set size)

Q: State the "One-Standard Error Rule".
A: If multiple models are within one standard error, choose the simplest because it likely generalizes better

Q: Leave-one-out Cross-validation (3)
A: - The model is trained on all data points but one (\(n-1\)) and is then evaluated on the left-out data point. This is iteratively done for all \(n\) data points.
- Computationally more expensive (\(n\) iterations) but great for small data sets
- The very small validation "set" increases the variance of the estimated error.

Q: Underlying assumptions of most ML models which chemical data violates. (2)
A: - IID: Independent and identically distributed (IID) random variables
- Uniform Noise level for all data points homoscedasticity

Q: What are two common biases present in chemical data?
A: - Sampling Bias (Some datapoints have higher or lower propabilty of being sampled; https://en.wikipedia.org/wiki/Sampling_bias): Leads to non-representative experiments
- Reporting/Publication Bias: skews data away from negative results

Q: What is understood by an "Activity Cliff"?
A: - When a small change in the molecular structure brings about a large change in activity

Q: How can applicability be tested in chemical problems? (5)
A: - Perform Scaffold Split
- Perform Similarity Split
- Perform Time Split
- Compare dummy models
- Perform a permutation test with target data (\(y\)-scrambling)