diff --git a/AI/Neural Networks/Activation Functions.md b/AI/Neural Networks/Activation Functions.md index a8e5860..f716bf7 100644 --- a/AI/Neural Networks/Activation Functions.md +++ b/AI/Neural Networks/Activation Functions.md @@ -53,7 +53,7 @@ Rectilinear - For deep networks - $y=max(0,x)$ - CNNs - - Breaks associativity of successive convolutions + - Breaks associativity of successive [[convolution]]s - Critical for learning complex functions - Sometimes small scalar for negative - Leaky ReLu diff --git a/AI/Neural Networks/CNN/CNN.md b/AI/Neural Networks/CNN/CNN.md index ad0d735..4363bab 100644 --- a/AI/Neural Networks/CNN/CNN.md +++ b/AI/Neural Networks/CNN/CNN.md @@ -15,7 +15,7 @@ # Full Connected [[MLP|Dense]] -- Move from convolutional operations towards vector output +- Move from [[Convolutional Layer|convolutional]] operations towards vector output - Stochastic drop-out - Sub-sample channels and only connect some to [[MLP|dense]] layers @@ -28,14 +28,14 @@ # Finetuning - Observations - - Most CNNs have similar weights in conv1 - - Most useful CNNs have several conv layers + - Most CNNs have similar weights in [[Convolutional Layer|conv1]] + - Most useful CNNs have several [[Convolutional Layer|conv layers]] - Many weights - Lots of training data - Training data is hard to get - Labelling - Reuse weights from other network -- Freeze weights in first 3-5 conv layers +- Freeze weights in first 3-5 [[Convolutional Layer|conv layers]] - Learning rate = 0 - Randomly initialise remaining layers - Continue with existing weights diff --git a/AI/Neural Networks/CNN/FCN/FCN.md b/AI/Neural Networks/CNN/FCN/FCN.md index 969920b..9715cf8 100644 --- a/AI/Neural Networks/CNN/FCN/FCN.md +++ b/AI/Neural Networks/CNN/FCN/FCN.md @@ -1,6 +1,6 @@ Fully [[Convolution]]al Network -Convolutional and up-convolutional layers with [[Activation Functions#ReLu|ReLu]] but no others (pooling) +[[Convolutional Layer|Convolutional]] and [[UpConv|up-convolutional layers]] with [[Activation Functions#ReLu|ReLu]] but no others (pooling) - All some sort of Encoder-Decoder Contractive → [[UpConv]] diff --git a/AI/Neural Networks/CNN/FCN/FlowNet.md b/AI/Neural Networks/CNN/FCN/FlowNet.md index 6516946..a3c44dc 100644 --- a/AI/Neural Networks/CNN/FCN/FlowNet.md +++ b/AI/Neural Networks/CNN/FCN/FlowNet.md @@ -7,7 +7,7 @@ Optical Flow ![[flownet.png]] -# Skip Connections +# [[Skip Connections]] - Further through the network information is condensed - Less high frequency information - Link encoder layers to [[upconv]] layers diff --git a/AI/Neural Networks/CNN/FCN/Highway Networks.md b/AI/Neural Networks/CNN/FCN/Highway Networks.md new file mode 100644 index 0000000..18499e4 --- /dev/null +++ b/AI/Neural Networks/CNN/FCN/Highway Networks.md @@ -0,0 +1,9 @@ +- [[Skip connections]] across individual layers + - Conditionally +- Soft gates + - Learn vs carry +- Gradients propagate further +- Inspired by [[LSTM]] [[RNN]]s + +![[highway-vs-residual.png]] +![[skip-connections 1.png]] \ No newline at end of file diff --git a/AI/Neural Networks/CNN/FCN/ResNet.md b/AI/Neural Networks/CNN/FCN/ResNet.md index 886f426..2acf9c5 100644 --- a/AI/Neural Networks/CNN/FCN/ResNet.md +++ b/AI/Neural Networks/CNN/FCN/ResNet.md @@ -12,14 +12,18 @@ # Design -- Skips across pairs of conv layers +- Skips across pairs of [[Convolutional Layer|conv layers]] - Elementwise addition - All layer 3x3 kernel - Spatial size halves each layer - Filters doubles each layer -- Fully convolutional +- [[FCN|Fully convolutional]] - No fc layer - - No pooling + - No [[Max Pooling|pooling]] - Except at end - No dropout +![[imagenet-error.png]] + +![[resnet-arch.png]] +![[resnet-arch2.png]] \ No newline at end of file diff --git a/AI/Neural Networks/CNN/FCN/Skip Connections.md b/AI/Neural Networks/CNN/FCN/Skip Connections.md index 9a01257..fa602f1 100644 --- a/AI/Neural Networks/CNN/FCN/Skip Connections.md +++ b/AI/Neural Networks/CNN/FCN/Skip Connections.md @@ -1,16 +1,16 @@ -- Output of conv, c, layers are added to inputs of upconv, d, layers +- Output of [[Convolutional Layer|conv]], c, layers are added to inputs of [[upconv]], d, layers - Element-wise, not channel appending - Propagate high frequency information to later layers - Two types - Additive - - Resnet - - Super-resolution auto-encoder + - [[ResNet]] + - [[Super-resolution]] auto-encoder - Concatenative - Densely connected architectures - DenseNet - - FlowNet + - [[FlowNet]] -![[skip-connections.png]] +![[STEM/img/skip-connections.png]] [AI Summer - Skip Connections](https://theaisummer.com/skip-connections/) -[Arxiv - Visualising the Loss Landscape](https://arxiv.org/abs/1712.09913)aaaaa \ No newline at end of file +[Arxiv - Visualising the Loss Landscape](https://arxiv.org/abs/1712.09913) \ No newline at end of file diff --git a/AI/Neural Networks/CNN/GAN/DC-GAN.md b/AI/Neural Networks/CNN/GAN/DC-GAN.md index d0a1cc2..372fe06 100644 --- a/AI/Neural Networks/CNN/GAN/DC-GAN.md +++ b/AI/Neural Networks/CNN/GAN/DC-GAN.md @@ -1,4 +1,4 @@ -Deep Convolutional [[GAN]] +Deep [[Convolution]]al [[GAN]] ![[dc-gan.png]] - Generator @@ -13,7 +13,7 @@ Deep Convolutional [[GAN]] - Discriminator - Contractive - Cross-entropy [[Deep Learning#Loss Function|loss]] - - Conv and leaky [[Activation Functions#ReLu|ReLu]] layers only + - [[Convolutional Layer|Conv]] and leaky [[Activation Functions#ReLu|ReLu]] layers only - Normalised output via [[Activation Functions#Sigmoid|sigmoid]] ## [[Deep Learning#Loss Function|Loss]] diff --git a/AI/Neural Networks/CNN/GAN/GAN.md b/AI/Neural Networks/CNN/GAN/GAN.md index 5ea2ea5..a94f424 100644 --- a/AI/Neural Networks/CNN/GAN/GAN.md +++ b/AI/Neural Networks/CNN/GAN/GAN.md @@ -1,4 +1,4 @@ -# Fully Convolutional +# Fully [[Convolution]]al - Remove [[Max Pooling]] - Use strided [[upconv]] - Remove [[MLP|FC]] layers diff --git a/AI/Neural Networks/CNN/UpConv.md b/AI/Neural Networks/CNN/UpConv.md index e69de29..c092e77 100644 --- a/AI/Neural Networks/CNN/UpConv.md +++ b/AI/Neural Networks/CNN/UpConv.md @@ -0,0 +1,35 @@ +- Fractionally strided convolution +- Transposed [[convolution]] + - Like a deep interpolation +- Convolution with a fractional input stride +- Up-sampling is convolution 'in reverse' + - Not an actual inverse convolution +- For scaling up by a factor of $f$ + - Consider as a [[convolution]] of stride $1/f$ +- Could specify kernel + - Or learn +- Can have multiple upconv layers + - Separated by [[Activation Functions#ReLu|ReLu]] + - For non-linear up-sampling conv + - Interpolation is linear + +![[upconv.png]] + +# Convolution Matrix +Normal + +![[upconv-matrix.png]] + +- Equivalent operation with a flattened input + - Row per kernel location +- Many-to-one operation + +![[upconv-matrix-result.png]] + +[Understanding transposed convolutions](https://www.machinecurve.com/index.php/2019/09/29/understanding-transposed-convolutions/) + +## Transposed +![[upconv-transposed-matrix.png]] +- One-to-many + +![[upconv-matrix-transposed-result.png]] \ No newline at end of file diff --git a/img/highway-vs-residual.png b/img/highway-vs-residual.png new file mode 100644 index 0000000..2fe350d Binary files /dev/null and b/img/highway-vs-residual.png differ diff --git a/img/imagenet-error.png b/img/imagenet-error.png new file mode 100644 index 0000000..83f0382 Binary files /dev/null and b/img/imagenet-error.png differ diff --git a/img/resnet-arch.png b/img/resnet-arch.png new file mode 100644 index 0000000..ef154f8 Binary files /dev/null and b/img/resnet-arch.png differ diff --git a/img/resnet-arch2.png b/img/resnet-arch2.png new file mode 100644 index 0000000..07d6329 Binary files /dev/null and b/img/resnet-arch2.png differ diff --git a/img/skip-connections 1.png b/img/skip-connections 1.png new file mode 100644 index 0000000..8dccc57 Binary files /dev/null and b/img/skip-connections 1.png differ diff --git a/img/upconv-matrix-result.png b/img/upconv-matrix-result.png new file mode 100644 index 0000000..542c70a Binary files /dev/null and b/img/upconv-matrix-result.png differ diff --git a/img/upconv-matrix-transposed-result.png b/img/upconv-matrix-transposed-result.png new file mode 100644 index 0000000..0840bad Binary files /dev/null and b/img/upconv-matrix-transposed-result.png differ diff --git a/img/upconv-matrix.png b/img/upconv-matrix.png new file mode 100644 index 0000000..baac680 Binary files /dev/null and b/img/upconv-matrix.png differ diff --git a/img/upconv-transposed-matrix.png b/img/upconv-transposed-matrix.png new file mode 100644 index 0000000..b41d3f4 Binary files /dev/null and b/img/upconv-transposed-matrix.png differ diff --git a/img/upconv.png b/img/upconv.png new file mode 100644 index 0000000..4ee0868 Binary files /dev/null and b/img/upconv.png differ