- [[Skip connections]] across individual layers
	- Conditionally
- Soft gates
	- Learn vs carry
- Gradients propagate further
- Inspired by [[LSTM]] [[RNN]]s

![[highway-vs-residual.png]]
![[skip-connections 1.png]]