Abstract
Training deep Convolutional Neural Networks (CNNs) presents unique
challenges, including the pervasive issue of elimination singularities,
consistent deactivation of nodes leading to degenerate manifolds within the
loss landscape. These singularities impede efficient learning by disrupting
feature propagation. To mitigate this, we introduce Pool Skip, an architectural
enhancement that strategically combines a Max Pooling, a Max Unpooling, a 3
times 3 convolution, and a skip connection. This configuration helps stabilize
the training process and maintain feature integrity across layers. We also
propose the Weight Inertia hypothesis, which underpins the development of Pool
Skip, providing theoretical insights into mitigating degradation caused by
elimination singularities through dimensional and affine compensation. We
evaluate our method on a variety of benchmarks, focusing on both 2D natural and
3D medical imaging applications, including tasks such as classification and
segmentation. Our findings highlight Pool Skip's effectiveness in facilitating
more robust CNN training and improving model performance.