Title: ETH_Sem6 / DC / W08_GraphAndLanguageModels

Q: What is the rough idea of graph convolution?
A: - The information of node is iterativly updated: For features of node \(i\): \(\mathbf{x}'_i = \mathbf{x}_i\mathbf{W}_1+\sum_{j\in N(i)}\mathbf{x}_j\mathbf{W}_2\) or of the entire feature matrix: \(\mathbf{X}' = \mathbf{X}\mathbf{W}_1 + \mathbf{AXW}_2\)
- In the Message-passig Formalism: Convolutions are described as messages passed between the nodes:

Q: How do we get predictions from GNNs?
A: Number of convolutions is a hyperparameter

Q: How can molecules be represented as sequential data and which ML model is suited for sequential data?
A: - As SMILES strings in which all characters (atoms, bonds, ...) are tokens
- The full set of tokens in called vocabulary
- Recurrent neural networks can are especially suited for sequential data (SMILES, Kinetics, ...)

Q: Recurrent Neural Networks
A: - Difference to FFNN: The calculated activation \(\mathbf{h}^{(t)}\) at time point \(t\) is used as an input for the activation at time point \(t'\). So the Net Input depends on the weighted input from \(x\) at \(t\) and the weighted activation from previous neuron
- The output layer can be arranged in different ways tailored to the task at hand.
- Training with Backpropagation through time calculated with the chain rule
- Problem: Previously hidden states not accesible.
- Alternative: Long-Term-short-term memory

Q: What is attention in an RNN?
A: - It is the weight with which the activations of the hidden encoder states are weighted an then combined in the context vector: \(c_i = \sum_{j = 1}^T\alpha_{ij}h^\text{(j)}\)
- The attention weights can be calculated by an "inserted" Neural Network: The Alignment model

Q: The transformer
A: - The transformers use attention but the encoder and decoder no longer consist of RNNs.
- The encoder uses Mulit-head self-attention
- The decoder uses Masked self-attention

Q: What is self-attention?
A: - The attention weights for input \(\mathbf{x}^\text{(i)}\) are calculated based on all other input vectors \(\mathbf{x}^\text{(j)}\). The output \(\mathbf{z}^\text{(i)}\) is calculated based on all the input vectors weight by their respective attention.

Q: What is Multi-head attention?
A: Using multiple attention layers in paralell, each focusing on different aspects of the data. The resulting embeddings (\(\mathbf{z}\)) are concatenated afterwards before the resulting attention is used further

What is masked attention?

Q: What is cross-attention?
A: - Mix input and output embeddings

Q: Attention-augmented RNNs vs. transformer for large molecules?
A: Transformer models show better performace

Q: How can pre-trained transformers be used further?
A: - The pre-trained model is used to generate the features: Based on the embedding layer a classified model is trained.
- An output layer is added to the model and it is fine tuned using supervised learning and all parameters are updatable.

What is the difference between encoder- and decoder-only transformer models?