direkt zum Inhalt springen

direkt zum Hauptnavigationsmenü

Sie sind hier

TU Berlin

Inhalt des Dokuments

Information: All of our modules are presented in English. Therefore, this page is only available in the English language. 

Module: Information Theory (40230)

About the Module

Students are presented with fundamental and advanced concepts in Information Theory. The module is comprised of two courses in sequence. In the first course, basic information measures and single-source/single-destination problems are presented. In the second course, multiuser network problems are presented. Information theory forms the theoretical foundation of communication networks , data compression, data storage, and source coding, such as video and audio coding. 

Further Information

Quick Facts

Two consecutive courses:
Information Theory (3 LP)
Network Information Theory (3 LP)
Duration of Module:
Two semesters
Total LP:
Giuseppe Caire
Type of examination:
Portfolio exam
Enrollment for exam:
Available to:
The course is open to students enrolled in any MSc in EE, CS, Mathematics and Physics. 
Prerequisite for participation to courses are a mathematical background at the level of beginning MS students in Electrical Engineering (multivariate calculus, Fourier and Laplace Transforms, signals and systems, linear algebra and notions of matrix theory).  
Module Year:
LP = Leistungspunkte/Credits


Module Content

Information Theory
  1.  Brief review of probability theory for discrete random variables.
  2. Definition and main properties of entropy, cross-entropy (information divergence) and mutual information, basic relations and convexity properties, data processing inequality, Fano inequality.
  3. Typical sequences and typical sets, fundamental lemmas of typicality.
  4. Data compression, Human codes, arithmetic codes, notion of universal lossless source coding (Lempel-Ziv algorithm).
  5. Channel coding, capacity of discrete memoryless channels, capacity-cost function, proofs of achievability and converse of the channel coding theorem.
  6. A short overview of channel coding and its applications.
  7. Brief review of continuous random variables, probability density functions, differential entropy.
  8. Gaussian channels, spectral efficiency, waterfilling power allocation with relation to bit-loading and OFDM.
  9. Rate-distortion theory and quantization, reverse waterfilling with relation to subband coding, successive refinement coding, with reference to scalable image and video coding.
  10. Advanced topics (time permitting): a selection of topics at the boundary of today's research will be presented. Examples are: non-stationary non-ergodic channels and sources via the "information spectrum methods", application to fading and MIMO channels, performance of codes in the finite length regime and "channel dispersion".



Network Information Theory
  1. Review of typical sequences and typical sets, fundamental lemmas of typicality with particular emphasis to the generalizations useful in the achievability proofs of network information theory.
  2. Multiple Access Channel (MAC).
  3. Superposition coding and application to the Broadcast Channel (BC).
  4. The Interference Channel (IC).
  5. The 2-user Gaussian IC: generalized degrees of freedom, linear deterministic models, optimality of treating interference as noise.
  6. The K-user Gaussian IC: degrees of freedom and Interference Alignment, topological Interference Alignment and relation to Index Coding, approximate optimality of Treating Interference as Noise.
  7. Channels with states (Gelfand-Pinsker Problem, Writing on Dirty Paper).
  8. Further results on the Broadcast Channel (Marton Region, El-Gamai and Nair outer bound).
  9. Source coding with side information (Slepian-Wolf and Wyner-Ziv coding).
  10. Wireline Relay Networks: Max-ow problem and the Max-ow min-cut theorem, connection to Network Coding.
  11. General Relay Networks: Cut-set bound.
  12. Deterministic model for relay networks and their capacity.
  13. Approximate capacity of Gaussian relay networks (Physical Layer Network Coding).



Module Supporting Material

Information Theory

The course is primarily based on lecture notes and problem sets with solutions. The following books can be used as a reference:
  •  T. Cover and J. Thomas, Elements of Information Theory, 2nd Ed., Wiley Interscience, 2006.


Network Information Theory

The course depends primarily on lecture notes. The following textbooks are strongly advised as a reference:
  • A. El Gamal and Y.{H. Kim, Network Information Theory, Cambridge University Press, 2011.
  • G. Kramer, Topics in Multi-User Information Theory, Foundations and Trends in Communications and Information Theory, NOW Publisher, Vol. 4, No. 45, 2008.


Zusatzinformationen / Extras


Schnellnavigation zur Seite über Nummerneingabe

COURSE - Information Theory

LV-Nr. 0432 L 654
Prof. G. Caire

COURSE - Network Information Theory

LV-Nr. 34331600 L 001
Prof. G. Caire