Info

project

  • 5 ECTS
  • 8 pages (template provided), due January
  • project ideas: google docs
  • -> get results within one month (oct 12 - nov 20)
  • DPHPC: team of five + approved project (@2026-10-12)
  • DPHPC: first draft + results (@2026-11-20)
  • DPHPC: final project presentation to TAs (recorded) (@2026-12-14)
  • DPHPC: project report (@2027-01-08)

Exam DPHPC

120min, no aids; counts 50% towards the grade

Exercises

-> HS2026_tasks

SerieLösungKorrektur

OpenMP for pragma makes the loop iterator private, even if defined outside of it

Vorlesung

timestamp

Exercises

Note

  • exam will be on lecture material + recitation
  • most time should be spent on project, not assignments

Project

  • open-ended project
  • goal: understand parallelism types (patterns of distribution), find real-life examples for it, implement it efficiently (lightweight implementation); experimental project: problem-solving, finding problems worth solving instead of implementing

examples:

  1. large LLMs nowadays cannot fit even on the largest GPU - not enough RAM on GPU
    • solution: split model between several CPUs, GPUs (even matrices!)
    • data paralellism (split tokens) vs tensor parallelism (split weights)
    • no dependence between the batches if tokens are independent
  2. split tensors, and how to do it independently
    • one slab of matrix in each GPU
    • multiple (2d) slabs on GPU
  3. stencils/distributions -> distribution squares inefficient in stencils really big -> strips better for halo distribition
  4. matrix transpositions (e.g. matrix is rowwise strip, and each GPU has different strip) -> how to efficiently
  5. atomic simulation - giant grid/cube - how to communicate halos between simulations

-> ultimate goal: (AI can do that, too)

  • high-end clusters for final benchmarking

start with e.g. matrix multiplication: test different distributions

meetings:

  • in 2 weeks (end oktober)
  • mid november
  • end december

timestamp 2026-10-19 (?)

Rule 2: Specify the reason for only reporting subsets of standard benchmarks or applications or not using all system resources

  • Implies: Show results even if your code/approach stops scaling
  • -> If there is a good reason, mention it clearly (e.g. mathematically proven that running with processes>8 is too much overhead or memory-bound instead of bandwidth-bound -> not enough memory for >8 processes)

exam

no need to know MPI Collective Routines by heart (will give functions in exam, if needed); do need to know how to call them, though

MPI reduce could have multiple implementations, that MPI switches based on the

  • topology
  • network speed

timestamp 2026-10-26

do not summarize!

  • anomalies require discussion
  • write how often you measured

Rule 3:

  • Use the arithmetic mean only for summarizing costs.
  • Use the harmonic mean for summarizing rates. Rule 4: Avoid summarizing ratios (e.g., speedup); summarize the costs or rates that the ratios base on instead. Only if these are not available use the geometric mean for summarizing ratios. Rule 5: Rule 5: Report if the measurement values are deterministic. For nondeterministic data, report confidence intervals of the measurement (normally 95%) -> determine number of measurements based on data noisiness (3-100s of measurements)
TopologyTypeProsCons
BusDirectVery cheapOnly one source can communicate with a destination at any time; low bisection bandwidth
Fully ConnectedDirectAny source-destination pair can communicate simultaneouslyVery expensive; quadratic cost; does not scale; difficult layout on a PCB
CrossbarSwitch Building BlockCheaper than a fully connected network ( cables); much better performance than a busArea and cost remain quadratic
2D-MeshDirectSimple direct topology where each node acts as both a switch and a terminalHigh diameter
2D TorusDirectCuts the diameter of a mesh in half by adding wrap-around linksWires have vastly different lengths unless folded
Butterfly (-ary -fly)IndirectEnables construction of arbitrarily sized networks based on technology constraints(partially) Blocking network
- always goes left->right / right->left => only one path between each pair
ClosIndirectFlexible multi-stage switch configuration- Multi-stage management complexity
- Blocking
Fat Tree (-ary -tree)IndirectUses fatter links at higher levels; widely used in HPCComplex wiring requirements
DragonflyIndirect / Group-basedSeparates long and short links to reduce expensive long cables; heavily used in HPCComplex hierarchical design
Large noise (very slow if calculations spread over multiple racks)
Slim FlyGraph-theoreticApproaches the Moore bound for maximum network size given a fixed switch radix and diameterComplex construction rules based on Galois fields
-> Bandwith: smallest bisection (bisection: cut that sepaates grpah in half)
-> Diameter: largest hop count of the minimal paths for all source-destination pairs

timestamp 2026-11-02

little’s law can easily be packed into an exam task -> will almost definitively be asked at the exam

  • latency * throughput = concurrency

timestamp 2026-11-05

Questions

  • Why are exercise sessions independent of the lectures? Is the material from the exercise sessions exam relevant?