Branch attention can be seen as a dynamic branch selection mechanism: which to pay attention to, used with a multi-branch structure.

PipeDream is an asynchronous pipeline parallel strategy for training large neural networks. It adds inter-batch pipelining to intra-batch parallelism to further improve parallel training throughput, helping to better overlap computation with communication and reduce the amount of communication when possible.

PipeDream

Branch attention

Training Very Deep Networks

A **Residual SRM** is a module for convolutional neural networks that uses a [Style-based Recalibration Module](https://paperswithcode.com/method/style-based-recalibration-module) within a [residual block](https://paperswithcode.com/method/residual-block) like structure. The Style-based Recalibration Module (SRM) adaptively recalibrates intermediate feature maps by exploiting their styles.

Source	Training Very Deep Networks
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com