links - Design of Parallel and High-Performance Computing
lecture website: https://spcl.inf.ethz.ch/Teaching/2026-dphpc/ + moodle
Info
project
- 5 ECTS
- 8 pages (template provided), due January
- project ideas: google docs
- -> get results within one month (oct 12 - nov 20)
- DPHPC: team of five + approved project (@2026-10-12)
- DPHPC: first draft + results (@2026-11-20)
- DPHPC: final project presentation to TAs (recorded) (@2026-12-14)
- DPHPC: project report (@2027-01-08)
Exam DPHPC
120min, no aids; counts 50% towards the grade
Exercises
-> HS2026_tasks
| Serie | Lösung | Korrektur |
|---|---|---|
OpenMP for pragma makes the loop iterator private, even if defined outside of it
Vorlesung
Exercises
Note
- exam will be on lecture material + recitation
- most time should be spent on project, not assignments
Project
- open-ended project
- goal: understand parallelism types (patterns of distribution), find real-life examples for it, implement it efficiently (lightweight implementation); experimental project: problem-solving, finding problems worth solving instead of implementing
examples:
- large LLMs nowadays cannot fit even on the largest GPU - not enough RAM on GPU
- solution: split model between several CPUs, GPUs (even matrices!)
- data paralellism (split tokens) vs tensor parallelism (split weights)
- no dependence between the batches if tokens are independent
- split tensors, and how to do it independently
- one slab of matrix in each GPU
- multiple (2d) slabs on GPU
- stencils/distributions -> distribution squares inefficient in stencils really big -> strips better for halo distribition
- matrix transpositions (e.g. matrix is rowwise strip, and each GPU has different strip) -> how to efficiently
- atomic simulation - giant grid/cube - how to communicate halos between simulations
-> ultimate goal: (AI can do that, too)
- high-end clusters for final benchmarking
start with e.g. matrix multiplication: test different distributions
meetings:
- in 2 weeks (end oktober)
- mid november
- end december
timestamp 2026-10-19 (?)
Rule 2: Specify the reason for only reporting subsets of standard benchmarks or applications or not using all system resources
- Implies: Show results even if your code/approach stops scaling
- -> If there is a good reason, mention it clearly (e.g. mathematically proven that running with processes>8 is too much overhead or memory-bound instead of bandwidth-bound -> not enough memory for >8 processes)
exam
no need to know MPI Collective Routines by heart (will give functions in exam, if needed); do need to know how to call them, though
MPI reduce could have multiple implementations, that MPI switches based on the
- topology
- network speed
timestamp 2026-10-26
do not summarize!
- anomalies require discussion
- write how often you measured
Rule 3:
- Use the arithmetic mean only for summarizing costs.
- Use the harmonic mean for summarizing rates. Rule 4: Avoid summarizing ratios (e.g., speedup); summarize the costs or rates that the ratios base on instead. Only if these are not available use the geometric mean for summarizing ratios. Rule 5: Rule 5: Report if the measurement values are deterministic. For nondeterministic data, report confidence intervals of the measurement (normally 95%) -> determine number of measurements based on data noisiness (3-100s of measurements)
| Topology | Type | Pros | Cons |
|---|---|---|---|
| Bus | Direct | Very cheap | Only one source can communicate with a destination at any time; low bisection bandwidth |
| Fully Connected | Direct | Any source-destination pair can communicate simultaneously | Very expensive; quadratic cost; does not scale; difficult layout on a PCB |
| Crossbar | Switch Building Block | Cheaper than a fully connected network ( cables); much better performance than a bus | Area and cost remain quadratic |
| 2D-Mesh | Direct | Simple direct topology where each node acts as both a switch and a terminal | High diameter |
| 2D Torus | Direct | Cuts the diameter of a mesh in half by adding wrap-around links | Wires have vastly different lengths unless folded |
| Butterfly (-ary -fly) | Indirect | Enables construction of arbitrarily sized networks based on technology constraints | (partially) Blocking network - always goes left->right / right->left => only one path between each pair |
| Clos | Indirect | Flexible multi-stage switch configuration | - Multi-stage management complexity - Blocking |
| Fat Tree (-ary -tree) | Indirect | Uses fatter links at higher levels; widely used in HPC | Complex wiring requirements |
| Dragonfly | Indirect / Group-based | Separates long and short links to reduce expensive long cables; heavily used in HPC | Complex hierarchical design Large noise (very slow if calculations spread over multiple racks) |
| Slim Fly | Graph-theoretic | Approaches the Moore bound for maximum network size given a fixed switch radix and diameter | Complex construction rules based on Galois fields |
| -> Bandwith: smallest bisection (bisection: cut that sepaates grpah in half) | |||
| -> Diameter: largest hop count of the minimal paths for all source-destination pairs |
timestamp 2026-11-02
little’s law can easily be packed into an exam task -> will almost definitively be asked at the exam
- latency * throughput = concurrency
timestamp 2026-11-05
Questions
- Why are exercise sessions independent of the lectures? Is the material from the exercise sessions exam relevant?