These technologies have been used in many promising industry projects and scientific research, which has allowed machine learning to develop widely in recent years. Effectiveness and utilization of the power of modern computers are one of the core challenges that Deep Learning faces. However, Deep Learning models have inherent parallelism. Exploiting and applying this parallelism to hardware is one of the most exciting and important research directions that researchers of these models face. In the lecture a variety of topics in the context of parallelism and distribution in Deep Learning will be discussed.