Title: ETH_Sem6 / DC / W07_AdvancedML

Q: Gradient Boosting (5)
A: - Is an Ensemble method: Combine many models
- Strong learner: low bias; Weak learner: high bias
- Idea: Let each subsequent model learn on the residuals of the previous one and the overall prediction is taken as result.
- XGBoost is the most popular algorithm and very good for tabular data.
- Early Stopping: Training is stopped as soon as the performace on the validation set is constant or increases. This ensures that the model avoids overfitting and speeds up training. (https://scikit-learn.org/stable/auto_examples/ensemble/plot_gradient_boosting_early_stopping.html)

Q: Support Vector Models.
A: - The Idea is to "fit the widest Street" between classes by maximizing the margin (lane width) between the decision boundary and the nearest training examples (lane boundary) i.e the support vectors.
- Hard maxium margin: To the first data point only; Can result in SVM not working do to outliers.
- Soft margin: Volations of the margins are allowed (set by hyperparameter) but minimized
- Very sensitive to scale
- Model is more regularized when more violations are allowed.
- SV Regression: Maximize Points inbetween margins

How can SVM be fitted to non-linear data?

Q: Kernel Ridge Regression
A: - Combination of ridge regression (linear least square with L2-norm regularization) with a kernel function https://scikit-learn.org/stable/modules/generated/sklearn.kernel_ridge.KernelRidge.html
- The prediction given is a weighted sum of similarity to the training data (measured by the kernel function)

Q: What does the universal approximation theorem tell us?
A: Regardless of what function we are trying to learn, we know that a large enough MLP will be able to represent this function.  However, we are not guaranteed that the training algorithm will be able to learn that function

Q: Classical Perceptron
A: - Can be used for binary classification Tasks \((0, 1)\)
- The output is determined by the given equation: \(f(\mathbf{x}) = \theta(\mathbf{w}\cdot\mathbf{x} + b)\); with \(\theta\) the heaviside step function and \(\mathbf{w}\) and \(\mathbf{x}\) the vectors of weights and inputs respectively.

Q: Fully connected Feed Forward Neural Networks
A: - Input Layer - Hidden layer(s) - Output layer
- Multiple Layers of Perceptrons
- Each layers is composed of two computations: The Net Input \(\mathbf{z} = \mathbf{x}\mathbf{W}^T + \mathbf{b} \) and the activation (forms the output of the respective layer): \(\mathbf{a} = \sigma(\mathbf{z}) \)
- A variety of activation functions exist, choice can be seen as additional hyperparameter
- Optimal weight inizialization depends on activation function
- To optimize weights we do backpropagation: Minimize the Loss \(L(\mathbf{W}, \mathbf{b}) \) with respect to each weight using partial derivatives and the chain rule
- For example the for the output layer weight can be written as follows: \(\frac{\partial L}{\partial w_{1,1}^\text{(out)}}= \frac{\partial L}{\partial a_1^\text{(out)}}\times\frac{\partial a_1^\text{(out)}}{\partial z_1^\text{(out)}} \times \frac{\partial z_1^\text{(out)}}{\partial w_{1,1}^\text{(out)}}\) and we can use results from the forwad pass calculations
- With the Gradients for the loss with respect to each weight at hand the weights can be optimized using gradient decent:
- Full/Batch calculates all the gradients at once with the full data-set; Stochastic Gradient Descent: "picks a random instance in the training set at every step and computes the gradients based only on that single instance." (Hands-on ML, A. Géron, p.126); Mini-batch is the "combination": Takes "small random sets of instances" (Hands-on ML, A. Géron, p.129).
- One Epoch: One full batch has been considered
- Practical considerations: Learning Rate; Use scaled values; Be aware of local minima
- Impicit and Explicit regularization possible

Question: What is the in the "too small" and "too large" label graphs?
QType (0=kprim,1=mc,2=sc): 2
Q_1: The number of hidden layers is too large or too small.
Q_2: The learning rate is too large or too small.
Q_3: The features are not scaled.
Answers: 0 1 0

Q: What is Dropout in context of NNs?
A: Dropout: Only use a given percentage of random neurons. Prevents the NN from relying heavily on specific connections.

Q: Representation Learning?
A: NN learn sophisticated representations of data by reducing the dimensionality and untangling the features.