Title: ETH_Sem6 / DC / W05_ClassicalML

Q: Three types of ML
A: 1. Supervised learning:
  - Regression, Classification, Time series prediction, Outlier detection
  - Labeled data, direct feedback, predict outcome/future
- Unsupervised learning:
  - Clustering, Dimensionality reduction, Anomaly detection
  - No labels/tragets, No feedback, For hidden structure in data
  - E.g. give SMILES with one atom covered as input -> Model predicts covered atom
- Reinforcement learning:
  - Decision process, Reward system, Learn series of actions

Q: Overview supervised algorithms (2, (3, 4))
A: - Linear Regression:
  - Ridge, Lasso, Logistic
- Non-linear:
  - KNN
  - Tree methods (decision trees, random forest)
  - Kernel methods (Support vector machines, Gaussian process regression, Kernel ridge)
  - Neural Networks

Q: Classification vs. regression
A: Predict a label vs. predict a value

Q: KNN
A: K Nearest Neighbors:
1. Choose integer \(k\) and distance metric \(d\) (function that defines distance with features as input)
2. Find the \(k\) nearest neighbors by \(d\)
3. Assign value based on majority vote or average

Q: Linear Regression
A: Find target \(y\) as weighted linear combination of features with intercept \(b\). \[y = w_1 x_1 + w_2 x_2 + \cdots + w_m x_m + b = \vec w \cdot \vec x\]

Q: Fitting linear regression coefficients
A: Minimize the MSE loss function \(L(w) \) analytically:
\[\dede{y}{w_j} L(w) = \frac{2}{n} \sum_{i=1}^n (\vec w \cdot \vec x^{(i)} - y^{(i)}) x_j^{(i)}\]
with the gradient:\[\grad_w L(w) = \begin{pmatrix} \dede{y}{w_0}L \\ \dede{y}{w_1} L \\ \vdots \\ \dede{y}{w_m} L \end{pmatrix} = \frac{2}{n} \vec{X} \cdot (\vec{X} \vec{w} - \vec{Y})\]Where the tensors \(\vec{X} = (\vec x^{(1)}, \dots, \vec x^{(n)})^T\) and \(\vec Y\) accordingly.
Or through gradient descent.

Q: Gradient descent
A: Update weights with gradient step of loss function scaled by small factor (learning rate \(\eta\)), to train on large datasets.
\[w^{\text{next}} = w^{\text{current}} + \eta \cdot \grad_w L(\vec w)\]

Q: Regression metrics (4)
A: - Mean squared error (MSE)
- square-root of MSE (RMSE)
- Mean absolute error (MAE)
- Pearson coefficient \(R^2 = 1 - \frac{RSS}{TSS}\), with \(RSS = \sum_{i=1}^n (y_i - \hat y_i)^2\), \(TSS = \sum_{i=1}^n(y_i - \bar y)^2\). \(R^2\) large is better

Q: Sources of uncertainty in predictions (3)
A: 1. Uncertainty of fit given the data - reducible error
2. Simple model might be far from the truth - model bias
3. Uncertainty from noise - irrreducible error

Q: Outlier identification metric example
A: Cook's distance:
\[D_i = \frac{1}{(p+1)MSE} \sum^n_{j=1} (\hat y_j - \hat y_j ^{(i)} )^2\]with threshold \(D_i > 4/n\).

Q: Alternative regression that is less sensitive to outliers
A: Robust regression, change loss function to eg. Huber loss:\[L(y, f(x)) = \begin{cases} (y - \hat y)^2 & \text{for} |y - \hat y| \leqslant \delta \\ 2 \delta |y - \hat y| - \delta^2 & \text{for} |y - \hat y| \gt \delta \end{cases}\]

Q: Curse of dimensionality
A: More features lead to overfitting and add noise and high risk of multicolinearity.

Q: Overfitting and underfitting
A: Model complexity needs to be neither too high, nor too low:
Example Classification:

Q: Bias and variance?
A: - Bias = How close on average to the target?
- Variance = How much spread around target?

Q: Consider two methods that approximate \(f(x) = \sin(\pi x)\) with two random points over many repeats:
1. Constant function \(y = b = \frac{y_1+y_2}{2}\)
2. Fitted linear function \(y = a x + b\)
Which method has the lower bias?
A: It's not always the less complex model that has a lower bias!

Q: Regularization + 2 Examples
A: - Explicit via penalty of high weights in loss function
- Implicit through other forms, eg. early stopping
General regularization formula with modified loss function
\[L'(y, f(X, w)) = L(y, f(X, w)) + \lambda R(f(X, w))\]
examples:
1. L2 or Tikhonov regularization: \(R(f(X, w)) = ||w||^2_2 = \sum_{j=1}^m w_j^2\) used in Ridge regression
2. L1 regularization: \(R(f(X, w)) = ||w||_1 = \sum_{j=1}^m |w_j|\) used in Lasso regression

Q: Elastic Net
A: Combination of scaled L1 and L2 regularization, interpolates between Ridge and Lasso.
\[L(w) = \frac{1}{n} \sum_{i=1}^n \left(y^{(i)} - \hat y^{(i)} \right)^2 + \lambda_1 ||w||_1 + \lambda_2 ||w||_2^2\]

Q: Which models below correspond to which regularization?
A: Ordinary least squares, B: Lasso, C: Ridge
A: 1 (most sparse model) - Lasso
2 - Ridge
3 (look at the scale!) - Ordinary least squares

Q: Use the training data to choose hyperparameters?
A: No, that will lead to overfitting! Use resampling methods like cross-validation (CV) or optimize on validation data.

Q: Why feature selection?
A: Removing non-informative features reduce computational and feature collection cost. Mostly useful for models that don't support regularization.
Plot accuracy vs. number of features selected to see which features are informative:

Q: Principal component analysis
A: - Linear transformation of features to orthogonal basis
- Obtained by eigenvalue decomposition, eg. singular value decomposition \(\Rightarrow\)eigenvalues \(w_i\) with scores \(t_i\)
\[\frac{1}{n-1} X^TX = WTW^T = W \begin{pmatrix} t_1 & & \\ & \ddots & \\ & & t_m \end{pmatrix} W^T\]

Q: What step is important before PCA?
A: Standardize the data, otherwise features with higher numerical values have seemingly higher importance.

Q: Partial Least Squares
A: PLS components both have high variance in \(X\) and correlate with \(y\)

Q: Most common linear regression method for classification
A: Logistic regression: Output is conditional probability \(p = p(y=1|x)\) of class 1

Question: For low-dim data I would choose
Title: (Opinionated) recommendation
QType (0=kprim,1=mc,2=sc): 2
Q_1: Elastic net preferred over Lasso
Q_2: Ridge (logistic) regression
Q_3: PLS
Q_4: PCR
Q_5: Pure linear regression
Answers: 0 1 0 0 0

Question: For high-dimensional data I would use
Title: (Opinionated) recommendation
QType (0=kprim,1=mc,2=sc): 2
Q_1: Ridge (logistic) regression
Q_2: Elastic net
Q_3: Lasso
Q_4: Sequential feature selection
Q_5: PCR
Answers: 0 1 0 0 0
Sources: https://onlinelibrary.wiley.com/doi/full/10.1111/insr.12469
Extra 1: Elastic net punishes strong outliers in multiple features stronger than ridge, but not as strong as ridge. Ridge could also be okay, but stay away from lasso: In a regime of linear sparsity, there exist a trade-off between false and true positive rates along the LASSO path, even when the design variables are stochastically independent.

Q: What means beyond linearity?
A: Include higher-dregree terms and cross-terms (interactions):\[y = w_1 x_1 + w_2 x_2 + w_3 x_1^2 + w_4 x_2^2 + w_5 x_1 x_2 + \dots\]

Q: Problems high degree polynominal regression
A: - Expensive
- Overfitting

Q: Decision trees
A: All samples start at a root node, go through split nodes to leaf nodes \(\Rightarrow\)Value/label of leaf node predicted
Good for noisy, high dimensional data, eg. medical treatment prediction.

Q: Train decision trees
A: Binary greedy splits minimize loss at each split, finding optimal feature \(k\) and threshold \(t_k\):
\[L(k, t_k) = \frac{n_{\text{left}}}{n_{\text{parent}}} L_{\text{left}} + \frac{n_{\text{right}}}{n_{\text{parent}}} L_{\text{right}}\]

Q: Problems/fixes decision trees
A: Low bias and high variance \(\Rightarrow\)easily overfit. Can be simplified by constraining training or by pruning
\(\Rightarrow\) Most important hyperparam is maximum depth

Q: Ensemble methods
A: Many models into one meta model, final prediction by voting. (regression \(\Rightarrow\) average, classification \(\Rightarrow\) majority)

Q: Random Forest
A: Decision trees with bagging and subset of features at each split:
Trees with high variance and low bias are averaged.
Important hyperparams: Number of trees (~100), maximum features \(\Rightarrow \)default almost always decent

Question: Which feature is the most important of these two?
Title: Feature importance
QType (0=kprim,1=mc,2=sc): 2
Q_1: AGE
Q_2: EXPERIENCE
Answers: 0 0 1
Extra 1: They are very much correlated! The fitting is sensitive to training data as revealed by resampling:
Lasso selects just one of them while Ridge would weigh them equally.

Question: How to fix colinearity?
QType (0=kprim,1=mc,2=sc): 0
Q_1: Drop all but one of the correlated features
Q_2: Drop all correlated features
Q_3: Keep only correlated features
Q_4: Choose a higher dimensional model
Q_5: Use a scaler to scale the input
Answers: 1 1 0 0 0

Q: Correlation map with python?
A: df = pd.read_csv("input.csv")
plt.figure(figsize=(12, 8))
sns.heatmap(df.corr().abs(), cmap="coolwarm")

Q: Leave-one-out cross validation
A: The validation set consists of only one datapoint and the training set of the rest iteratively for the entire dataset.
y_true, y_pred = [], []
for train_ix, val_ix in LeaveOneOut().split(X_small):
    est_ = make_pipeline(StandardScaler(), LinearRegression())
	  # Split data
    X_train_, X_val_ = X_small[train_ix, :], X_small[val_ix, :]
    y_train_, y_val_ = y.iloc[train_ix], y.iloc[val_ix]
	  # Fit model
    est_.fit(X_train_, y_train_)
	  # Evaluate model
    yhat = est_.predict(X_val_)
	  # Store
    y_true.append(y_val_.iloc[0])
    y_pred.append(yhat[0])
print("LOOCV Q2:", r2_score(y_true, y_pred))