Keras currently provides several activation-level regularization layers, including Dropout, GaussianDropout, AlphaDropout, and SpatialDropout. However, it does not provide a native layer for connection-level stochastic regularization.
DropConnect (Wan et al., 2013) introduced connection-level masking as an alternative to activation-level dropout. Despite its established formulation, practical implementations have remained relatively uncommon because efficient implementations often require generating per-sample masked weight matrices or specialized backend implementations.
I would like to propose a Dense layer that performs per-sample connection-level stochastic masking using standard Keras tensor operations. The implementation does not require custom CUDA kernels or TensorFlow runtime changes and can be expressed entirely as a Keras layer.
A working proof-of-concept implementation, documentation, and experimental evaluation are available:
The intention of this proposal is not to replace existing Dropout layers, but to provide native support for connection-level stochastic regularization within the Keras API.
This feature would benefit users interested in exploring connection-level stochastic regularization without maintaining custom layers or backend-specific implementations.
It would provide a native Keras implementation of an established regularization concept, making experimentation easier for researchers, educators, and practitioners while remaining consistent with existing Keras workflows.
Keras currently provides several activation-level regularization layers, including Dropout, GaussianDropout, AlphaDropout, and SpatialDropout. However, it does not provide a native layer for connection-level stochastic regularization.
DropConnect (Wan et al., 2013) introduced connection-level masking as an alternative to activation-level dropout. Despite its established formulation, practical implementations have remained relatively uncommon because efficient implementations often require generating per-sample masked weight matrices or specialized backend implementations.
I would like to propose a Dense layer that performs per-sample connection-level stochastic masking using standard Keras tensor operations. The implementation does not require custom CUDA kernels or TensorFlow runtime changes and can be expressed entirely as a Keras layer.
A working proof-of-concept implementation, documentation, and experimental evaluation are available:
The intention of this proposal is not to replace existing Dropout layers, but to provide native support for connection-level stochastic regularization within the Keras API.
This feature would benefit users interested in exploring connection-level stochastic regularization without maintaining custom layers or backend-specific implementations.
It would provide a native Keras implementation of an established regularization concept, making experimentation easier for researchers, educators, and practitioners while remaining consistent with existing Keras workflows.