Andy Pack
c3ebefdd64
Affected files: .obsidian/workspace.json Food/Plans & Recipes.canvas Health/ADHD.md Health/Alexithymia.md Health/Anxiety.md Health/BWS.md Health/Worry Tree.md History/Britain.md Media/Listen.md Media/Watch.md Money/Assets/Asset.md Money/Assets/CFD.md Money/Assets/Derivative.md Money/Assets/Financial Instruments.md Money/Assets/Options.md Money/Assets/Security.md Money/Assets/Stocks.md Money/Econ.md Money/Equity.md Money/Inflation.md Money/Markets/Commodity.md Money/Markets/Markets.md Money/Markets/Types.md Money/Prices.md Money/To Learn.md Politcs/British/Prime Ministers.md Politcs/British/Tory.md Politcs/Neoliberalism/Thatcher.md Politcs/Now.md Projects/Dev Options.md STEM/AI/Neural Networks/Activation Functions.md STEM/AI/Neural Networks/MLP/Back-Propagation.md STEM/AI/Neural Networks/SLP/Least Mean Square.md STEM/AI/Pattern Matching/Markov/Markov.md STEM/Maths/Algebra.md STEM/Maths/Tensor.md STEM/Maths/Vector Operators.md Want/Clothes.md Want/House.md Want/Want.md Work/Board.canvas
86 lines
3.0 KiB
Markdown
86 lines
3.0 KiB
Markdown
---
|
||
tags:
|
||
- ai
|
||
- maths
|
||
---
|
||
- To handle overlapping classes
|
||
- Linearity condition remains
|
||
- Linear boundary
|
||
- No hard limiter
|
||
- Linear neuron
|
||
- Cost function changed to error, $J$
|
||
- Half doesn’t matter for error
|
||
- Disappears when differentiating
|
||
|
||
$$\mathfrak{E}(w)=\frac{1}{2}e^2(n)$$
|
||
- Cost' w.r.t to weights
|
||
$$\frac{\partial\mathfrak{E}(w)}{\partial w}=e(n)\frac{\partial e(n)}{\partial w}$$
|
||
- Calculate error, define delta
|
||
$$e(n)=d(n)-x^T(n)\cdot w(n)$$
|
||
$$\frac{\partial e(n)}{\partial w(n)}=-x(n)$$
|
||
$$\frac{\partial \mathfrak{E}(w)}{\partial w(n)}=-x(n)\cdot e(n)$$
|
||
- Gradient vector
|
||
- $g=\nabla\mathfrak{E}(w)$
|
||
- Estimate via:
|
||
$$\hat{g}(n)=-x(n)\cdot e(n)$$
|
||
$$\hat{w}(n+1)=\hat{w}(n)+\eta \cdot x(n) \cdot e(n)$$
|
||
|
||
- Above is a [feedforward](../Architectures.md) loop around weight vector, $\hat{w}$
|
||
- Behaves like low-pass filter
|
||
- Pass low frequency components of error signal
|
||
- Average time constant of filtering action inversely proportional to learning-rate
|
||
- Small value progresses algorithm slowly
|
||
- Remembers more
|
||
- Inverse of learning rate is measure of memory of LMS algorithm
|
||
- $\hat{w}$ because it's an estimate of the weight vector that would result from steepest descent
|
||
- Steepest descent follows well-defined trajectory through weight space for a given learning rate
|
||
- LMS traces random trajectory
|
||
- Stochastic gradient algorithm
|
||
- Requires no knowledge of environmental statistics
|
||
|
||
## Analysis
|
||
|
||
- Convergence behaviour dependent on statistics of input vector and learning rate
|
||
- Another way is that for a given dataset, the learning rate is critical
|
||
- Convergence of the mean
|
||
- $E[\hat{w}(n)]\rightarrow w_0 \text{ as } n\rightarrow \infty$
|
||
- Converges to Wiener solution
|
||
- Not helpful
|
||
- Convergence in the mean square
|
||
- $E[e^2(n)]\rightarrow \text{constant, as }n\rightarrow\infty$
|
||
- Convergence in the mean square implies convergence in the mean
|
||
- Not necessarily converse
|
||
|
||
## Advantages
|
||
- Simple
|
||
- Model independent
|
||
- Robust
|
||
- Optimal in accordance with $H^\infty$, minimax criterion
|
||
- _If you do not know what you are up against, plan for the worst and optimise_
|
||
- ___Was___ considered an instantaneous approximation of gradient-descent
|
||
|
||
## Disadvantages
|
||
- Slow rate of convergence
|
||
- Sensitivity to variation in eigenstructure of input
|
||
- Typically requires iterations of 10 x dimensionality of the input space
|
||
- Worse with high-d input spaces
|
||
|
||
- Use steepest descent
|
||
- Partial derivatives
|
||
|
||
- Can be solved by matrix inversion
|
||
- Stochastic
|
||
- Random progress
|
||
- Will overall improve
|
||
|
||
|
||
|
||
$$\hat{w}(n+1)=\hat{w}(n)+\eta\cdot x(n)\cdot[d(n)-x^T(n)\cdot\hat w(n)]$$
|
||
$$=[I-\eta\cdot x(n)x^T(n)]\cdot\hat{w}(n)+\eta\cdot x(n)\cdot d(n)$$
|
||
|
||
Where
|
||
$$\hat w(n)=z^{-1}[\hat w(n+1)]$$
|
||
## Independence Theory
|
||
|
||
|
||
 |