stem/AI/Neural Networks/Properties+Capabilities.md

Linearity

• Neurons can be linear or non-linear
• Network of non-linear neurons is non-linear
• Non-linearity is distributed
• Helpful if target signals are generated non-linearly

Input-Output Mapping

• Map input signal to desired response
  • Supervised learning
• Similar to non-parametric statistical inference
  • Non-parametric as in no prior assumptions
  • No probabilistic model

Adaptivity

• Synaptic weights
  • Can be easily retrained
• Stationary environment
  • Essential statistics can be learned
  • Model can then be frozen
• Non-stationary environments
  • Can change weights in real-time
  • In general, more adaptive = more robust
    • Not always though
    • Short time-constant system may be thrown by short-time spurious disturbances
    • Stability-plasticity dilemma
  • Not equipped to track statistical variations
  • Adaptive system
• Linear adaptive filter
  • Linear combiner
    • Single neuron operating in linear mode
  • Mature applications
  • Nonlinear adaptive filters
    • Less mature
• Environments typically considered pseudo-stationary
  • Speech stationary over short windows
• Retrain network at regular intervals to account for fluctuations
  • E.g. stock market
• Train network on short time window
  • Add new data and pop old
    • Slide window
  • Retrain network

Evidential Response

• Decisions are made with evidence not just declared
  • Confidence value

Contextual Information

• Knowledge represented by structure and activation weight
  • Any neuron can be affected by global activity
• Contextual information handled naturally

Fault Tolerance

• Hardware implementations
• Performance degrades gracefully with adverse conditions
• If some of it breaks, it won't cause the whole thing to break
  • Like a real brain

VLSI Implementability

• Very large-scale integration
  • Chips with millions of transistors (MOS)
  • E.g. microprocessor, memory chips
• Massively parallel nature
  • Well suited for VLSI

Uniformity in Analysis

• Are domain agnostic in application
  • Analysis methods are the same
  • Can share theories, learning algorithms

Neurobiological Analogy

• Design analogous to brain

  • Already a demonstrable fault-tolerant, powerful, fast, parallel processor

• To slight changes

  • Rotation of target in images
  • Doppler shift in radar

• Network needs to be invariant to these transformations

Invariance

1. Invariance by Structure
   • Synaptic connections created so that transformed input produces same output
   • Set same weight for neurons of some geometric relationship to image
     • Same distance from centre e.g.
   • Number of connections becomes prohibitively large
2. Invariance by Training
   • Train on different views/transformations
     • Take advantage of inherent pattern classification abilities
   • Training for invariance for one object is not necessarily going to train other classes for invariance
   • Extra load on network to do more training
     • Exacerbated with high dimensionality
3. Invariant Feature Space
   • Extract invariant features
     • Use network as classifier
   • Relieves burden on network to achieve invariance
     • Complicated decision boundaries
   • Number of features applied to network reduced
   • Invariance ensured
   • Required prior knowledge