Programming on Parallel Machines; GPU, Multicore, Clusters and More (textbook)


“Why is this book different from all other parallel programming books?”

  • Suitable for either students or professionals.
  • Practical viewpoint:
    • There is very little theoretical analysis of parallel
      algorithms, such as O() analysis, maximum theoretical speedup,
      acyclic graphs and so on.
    • Extensive coverage of “wizardry” aspects, i.e. material known
      to experienced practitioners but generally not in books, such as
      coverage of loop iteration scheduling, memory effects of storing
      large arrays and so on.
    • Appendices cover systems background, crucial in applied work
      but always just “assumed” to be knowledge possessed by the readers.
    • Considerable attention is paid to techniques for debugging.
  • Uses the main parallel platforms—OpenMP, CUDA and
    MPI—rather than languages that at this stage are largely
    experimental, such as the elegant-but-not-yet-mainstream Cilk.
  • Starts with real parallel code right away in Chapter 1, with
    examples from pthreads, OpenMP and MPI.

Constantly evolving:

Like all my open source textbooks, this one is constantly
evolving. I continue to add new topics, new examples, more timing
analyses, and so on, and of course fix bugs and improve the exposition.

Prerequisites: The student must be easonably adept in
programming, and have math background through linear algebra. (An
appendix to the book reviews the parts of the latter needed for this


Creative Commons License
work is licensed under a Creative
Commons Attribution-No Derivative Works 3.0 United States License.

Copyright is retained by N. Matloff in all non-U.S. jurisdictions,
but permission to use these materials in teaching is still granted,
provided the authorship and licensing information here is displayed.
I would appreciate being notified if you use this book for teaching,
just so that I know the materials are being put to use, but this is
not required.


The book is freely available, subject to the conditions above, and can
be downloaded from


Norm Matloff
is a Professor of Computer Science at the University of
California, Davis. He was formerly a statistics professor at that
university. His research has included a number of diverse areas,
and he has been a recipient of the university’s Distinguished
Teaching Award. He was born and raised in Los Angeles, and
earned his PhD in theoretical mathematics (probability
theory/functional analysis) at UCLA.

Original URL:

Original article

Comments are closed.

Proudly powered by WordPress | Theme: Baskerville 2 by Anders Noren.

Up ↑

%d bloggers like this: