links - IntML

Info

  • exam
    • multiple choice, fully on paper
  • projects
    • 3/4 have to “pass” to be able to do the exam
    • 1min video explaining solution for each project needed, uploaded to polybox

Hilfsmittel IntroductionToMachineLearning

  • 2 A4 pages (1 sheet) handwritten / >11pt
  • calculator

https://mlstory.org/features.html

Exercises

-> FS2026_tasks

Projects

Vorlesung

timestamp 2026-02-27

todo completely useless - delete or rewrite

We have:

normal equation:

timestamp 2026-03-03

Variance might not be only because of noise, but also because there are not enough samples.

overfitting too sensitive to noise

timestamp 2026-03-04

  • Bias distance between average and
    • error from wrong assumptions in algorithm (“underfitting)
  • Variance how far is from
    • error from sensitivity to noise (“overfitting”)

-> increasing model complexity beyond point where training error , can lead to second decrease in generalization error (more than just u-curve)

splitting data

  • Train/Test split: Normally - test split, model is evaluated on test data
  • Train/Validation/Test split: e.g. , use validation set to tune hyperparameters, use test set to get unbiased estimate
  • K-Fold Cross-Validation (CV): Dividing the training data into subsets (folds). The model is trained times, each time using a different fold as the validation set and the remaining folds for training.

control model complexity Complex model (e.g. high-degree polynomial) might make weights large / osscilate. To counteract:

  1. Smaller degree
  2. Smaller number of monomials “active” by limiting -norm
  3. Limit norm

Lasso Regression ( Regularization) Adds a penalty proportional to the absolute value of the coefficients.

  • Effect: Induces sparsity, meaning it sets some coefficients to exactly zero, effectively performing feature selection.

Ridge Regression ( Regularization) Adds a penalty proportional to the square of the magnitude of coefficients.

  • Effect: Shrinks all coefficients toward zero but rarely makes them exactly zero.
  • Analytical Solution: .

todo why does lasso induce sparsity?

FeatureRidge (ℓ2​)Lasso (ℓ1​)
Penalty$\lambda \sum
SolutionClosed-formNumerical optimization
SparsityNo (coefficients )Yes (coefficients )
Use CasePreventing overfittingFeature selection

timestamp 2026-03-20

check whether kernel is valid:

  1. check symmetry
  2. decompose into known valid kernels (using linearity)
  3. use closure rule (sum/product/scaling/composition)
  4. if unsure, build a small kernel matrix counterexample

10. Neural networks

timestamp 2026-03-24

computation at each layer consists of three parts:

  • , trainable weights
  • , biases, which shift our functions (linear -> affine)
  • , activation functions which scale the values and introduces non-linearity -> allows the network to learn complex patterns
    • possible activation functions: ReLU, Sigmoid

forward propagation

  • initial input (input vector assigned to 0th layer):
  • hidden layers can be written as matrix-multiplication (pre-activation value , activation value ):
  • output layer (): linear combination of last hidden layer activations:

=> for a multilayer perceptron with a single hidden layer:

backwards propagation: https://xnought.github.io/backprop-explainer/

Dimensionality reduction

MethodWhat it doesBest used for…
PCAFlattens the data like a pancake.Simple data where things move in straight lines.
Kernel PCABends and twists the data before flattening it.Complex patterns like spirals or circles.
AutoencoderCompresses the data into a “secret code” and learns to unpack it.Very messy, high-tech data like photos or speech.

PCA PCA relies on the Covariance Matrix . This matrix tells us how every dimension in your data moves in relation to every other dimension.

  • The Matrix: .
  • The Magic (Eigenvectors): We solve .
    • (Eigenvector) is the direction of the “best angle” (the principal component).
    • (Eigenvalue) is a number representing the amount of spread (variance) in that direction.
  • The Projection: To turn your -dimensional point into a -dimensional point , you just multiply it by the top eigenvectors:

kernel PCA In standard PCA, we find eigenvectors of the data. But what if we first transformed the data into a crazy high-dimensional space using a function ? We can’t actually compute because it might be infinite-dimensional.

  • The Kernel Trick: Instead of calculating , we use a Kernel Function that calculates the “similarity” between two points directly.
  • The Kernel Matrix (): We build a matrix where every entry is the similarity between point and point .
  • The Solution: We find the eigenvectors of instead of .
  • The Projection: The new coordinates for a point are calculated by checking its similarity to all training points:

autoencoders This is an optimization problem. We have two functions: an Encoder and a Decoder .

  • The Bottleneck: . Here, is a nonlinear “gate” (like ReLU or Sigmoid) that allows the math to learn curves instead of just straight lines.
  • The Loss Function: We want the output to be as close to the input as possible:
  • The Training: gradient descent