Welcome to the BlueFog tutorials!

In this repository, we've put together a collection of awesome Jupyter notebooks. These notebooks serve two purposes:

• Help readers understand the basic concepts and theories of the decentralized optimization.
• Help readers understand how to implement decentralized algorithms with the BlueFog library.

Learn how to write your first "hello world" program over the real multi-CPU system with BlueFog.

Learn how to achieve the globally averaged consensus among nodes in a decentralized manner.

Learn how to solve a general distributed (possibly stochastic) optimization problem in a decentralized manner.

Learn how to accelerate your decentralized (possibly stochastic) optimization algorithms with various bias-correction techniques.

Learn how to solve distributed optimization in a decentralized manner if the connected topology is directed or time-varying.

Learn how to solve a general distributed optimization problem with asynchronous decentralized algorithms.

Learn how to train a deep neural network with decentralized optimization algorithms.

This tutorial only contains the very basic concepts, algorithms, theories, and implementations for decentralized optimization. It misses many important recent progress in the algorithm development and theory in the decentralized optimization community. We hope you will consider using BlueFog in the experiment of your new decentralized algorithm and summarize your ideas into a Jupyter notebook tutorial.

The BlueFog Team involves several researchers and engineers that target to make decentralized algorithms practical for large-scale optimization and deep learning. We hope to bridge the gap between the theoretical progress of decentralized algorithms in the academia and the real implementation in the industry. We hope more researchers and engineers can join us to contribute to the community of decentralized optimization.

Faster Learning over Networks and BlueFog, BlueFog Team, invited talk at MLA, 2020 [slides]

Parallel, Distributed, and Decentralized optimization methods, Wotao Yin, Tutorial in ECOM2021, 2021 [Materials]

Feel free to share the BlueFog repo and this tutorial to anyone that has an interest. If you use BlueFog, please cite it as follows:

@software{bluefog2021_4616052,
  author       = {BlueFog Team},
  title        = {BlueFog: Make Decentralized Algorithms Practical For Optimization and Deep Learning},
  month        = Mar.,
  year         = 2021,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.4616052},
  url          = {https://doi.org/10.5281/zenodo.4616052}
}