asynchronous - Neural network accelerator executes layer by layer

Question

Modern accelerator such as GPU, when executes a forward pass on neural network, will it execute layer by layer? That is, will it finish ALL work of the previous layer, then starts to execute the next layer? I think the answer should also depend on software, but could someone just share some thoughts on this?

question from:https://stackoverflow.com/questions/66062260/neural-network-accelerator-executes-layer-by-layer

Answer 1

Waitting for answers