This knowledge can be helpful to know when analyzing performance bottlenecks. These are the least compute-intensive operations. These are the remaining operators: biases, dropout, activations, and residual connections. Softmax and layer normalization are less compute-intensive than tensor contractions, and involve one or more reduction operations, the result of which is then applied via a map. These operations are the most compute-intensive part of training a transformer. Linear layers and components of Multi-Head Attention all do batched matrix-matrix multiplications. Transformers architecture includes 3 main groups of operations grouped below by compute-intensity. To understand a bit better why this is the case let’s have look at a model’s operations and memory needs. What’s interesting is that we use much more memory than the size of the model. So ideally we want to tune the batch size to our model’s needs and not to the GPU limitations. However, a larger batch size can often result in faster model convergence or better end performance. We see that already a relatively small batch size almost fills up our GPU’s entire memory. First, we set up a few standard training arguments that we will use across all our experiments: So now we can start training the model and see how the GPU memory consumption changes. We get the same number as before and you can also see that we are using a V100 GPU with 16GB of memory. | GPU GI CI PID Type Process name GPU Memory | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M.

| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr.

Wise memory optimizer full 2020