Pyramidal-Inspired Network: An Evolutionarily Simple Model for Memory & Context

[Chip (OP)]

The Py-INN or Pyramidal-Inspired Neural Network: An Evolutionarily Simple Model for Memory and Context

1. Design Philosophy

This model is intentionally simple, but biologically inspired. 
It borrows key ideas from pyramidal neurons without trying to fully replicate the cortex.

Core goals:

• Improve context handling
  
• Strengthen short-term and working memory
  
• Stay computationally simple and easy to implement
  

We treat each artificial “pyramidal unit” as a neuron with two main input zones:

• Basal zone – local, bottom-up data
  
• Apical zone – global, top-down context
  

2. Basic Unit: The Pyramidal-Like Neuron

Each unit has:

• Basal input vector (x): current sensory or token-level data
  
• Apical input vector (c): context, history, or task signals
  
• Two internal weights: W_basal and W_apical
  
• A simple nonlinearity: e.g., ReLU or tanh
  

Activation:

• Compute basal drive: h_b = f(W_basal · x)
• Compute apical drive: h_a = f(W_apical · c)
• Combine: y = f(h_b + α · h_a)

Where α is a scalar that controls how strongly context (apical) modulates the neuron.

3. Layer Structure

We build three main layers:

• Sensory Layer (Basal-dominant) 
  Receives raw input (tokens, pixels, features). 
  Apical input is minimal or zero here.
  
  
• Context Layer (Apical-dominant) 
  Receives compressed history (e.g., previous hidden states, task embeddings). 
  Basal input is weaker; apical input is stronger.
  
  
• Integration Layer (Balanced) 
  Receives both basal (current input) and apical (context) and combines them. 
  This is where “pyramidal-like” integration happens.
  

4. Simple Memory Mechanism

To keep it evolutionarily simple, we use:

• Context vector c_t that persists across time steps
  
• Update rule: c_t = β · c_{t-1} + γ · y_t

Where:

• β controls how much old context is retained (memory)
• γ controls how much new activity is written into context

This is a minimal analogue of recurrent pyramidal networks and working memory:

• Old context is never fully erased
• New input gradually reshapes the context
• The apical zone always “sees” this evolving context

5. How It Addresses Memory and Context Limitations

Problem in current AI:

• Context is often treated as a static window or a single embedding
• Memory is either huge (transformer context) or shallow (simple RNNs)

What this model adds:

• Explicit separation of data (basal) and context (apical)
  
• Persistent context vector that evolves over time
  
• Context-modulated activation in each pyramidal-like unit
  

This allows:

• Better handling of long-range dependencies
• More stable representation of task or conversation state
• Simple, interpretable mechanism for “what matters right now”

6. Learning Rules (Kept Simple)

We use standard gradient-based learning, but with a small twist:

• Update W_basal mainly from prediction error on current input
  
• Update W_apical mainly from prediction error on context-sensitive outputs
  

Optionally:

• Regularize W_apical to avoid overfitting context
• Encourage sparsity in apical activations (only strong context should matter)

7. Evolutionary Simplicity

This model is “evolutionarily simplistic” because:

• It uses only two compartments (basal, apical) per unit
• It keeps memory as a single evolving context vector
• It relies on standard backprop, no exotic learning rules
• It can be implemented as a small modification of existing feedforward or recurrent networks

Yet it still captures key pyramidal neuron ideas:

• Dual input streams (data + context)
• Nonlinear integration of those streams
• Persistent, slowly changing context state

8. Summary

This pyramidal-inspired neural network:

• Adds a simple, biologically motivated way to handle context
• Provides a minimal memory mechanism without heavy architectures
• Respects the idea of basal vs apical integration from pyramidal neurons
• Is easy to implement and extend in existing AI frameworks