Master's Thesis | Computational Creativity & Cognitive Architecture | VUB AI Lab
This thesis presents an empirical comparison between PPM★ (Prediction by Partial Matching), a state-of-the-art statistical language modeling algorithm, and IDyOT (Information Dynamics of Thinking), a cognitive architecture designed to model human-like hierarchical learning and prediction.
Where PPM★ operates as a descriptive black box—achieving high prediction accuracy without explaining why—IDyOT provides an explanatory cognitive mechanism where each abstraction layer holds semantic meaning. This work documents the mathematical details of IDyOT's within-layer interactions, validates both implementations empirically, and identifies optimal configurations for cognitive modeling.
This research contributes to:
- Computational Creativity — Enabling AI systems that can articulate their creative reasoning
- Cognitive Science — Providing testable models of hierarchical prediction in the brain
- Artificial General Intelligence — Offering a cognitively-grounded alternative to scale-based approaches
This thesis occupies a unique position at the intersection of four major research domains:
┌─────────────────────────────────────────────────────────────────────────┐
│ PREDICTIVE PROCESSING FRAMEWORK │
│ (Friston's Free Energy Principle / Bayesian Brain) │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌────────────────┐ ┌────────────────┐ ┌─────────────────┐ │
│ │ COMPRESSION │ │ COGNITIVE │ │ COMPUTATIONAL │ │
│ │ THEORY │◄────►│ ARCHITECTURE │◄────►│ CREATIVITY │ │
│ │ (PPM★) │ │ (IDyOT) │ │ (Music/AI) │ │
│ └───────┬────────┘ └───────┬────────┘ └────────┬────────┘ │
│ │ │ │ │
│ └───────────┬───────────┴───────────┬────────────┘ │
│ ▼ ▼ │
│ ┌─────────────────────────────────────────┐ │
│ │ THIS THESIS │ │
│ │ Empirical bridge between PPM★ & IDyOT │ │
│ └─────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────┐ │
│ │ IDyOM (Music Cognition) │ │
│ │ Variable-order Markov chains for │ │
│ │ melodic expectation modeling │ │
│ └─────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
PPM (Prediction by Partial Matching) has been a gold standard in lossless data compression since the 1980s, achieving compression rates as low as 2.2 bits per character for English text. PPM★ extends this with unbounded context, making it highly effective for:
- Text compression (used in RAR, 7z, Zip formats)
- Language modeling for speech recognition
- Music cognition via IDyOM
Recent development (2024): The connection between prediction and compression is now being explored with Large Language Models (LLMs), with research showing "Language Modeling is Compression" (ICLR 2024).
IDyOM (Information Dynamics of Music) is the leading computational model for melodic expectation, using PPM★ internally to:
- Model how listeners form expectations about upcoming musical events
- Calculate Surprisal (unexpectedness of a note) and Entropy (prediction uncertainty)
- Combine Long-Term Models (lifelong musical exposure) with Short-Term Models (piece-specific learning)
IDyOM correlates strongly with experimental data from human participants and is widely used in empirical music cognition research.
IDyOT (Information Dynamics of Thinking) is a computational cognitive architecture by Prof. Geraint A. Wiggins that aims to provide an explanatory model of cognition, not just a descriptive one. Key principles:
| Aspect | IDyOT Approach |
|---|---|
| Foundation | Pre-conscious predictive loop driven by information efficiency |
| Core Operations | Segmentation → Categorization → Prediction → Abstraction |
| Memory | Hierarchical construction with meaningful semantic layers |
| Theory Basis | Aligns with Baars' Global Workspace Theory |
This thesis aligns with the broader Predictive Processing paradigm in cognitive science, which posits:
- The brain is fundamentally a "prediction machine"
- Cognition minimizes prediction error (or "free energy")
- Consciousness emerges from hierarchically organized, recursively updating generative models
IDyOT operationalizes these principles for music and language domains.
| Problem | PPM★ Approach | IDyOT Approach | Thesis Contribution |
|---|---|---|---|
| Context Selection | Ad-hoc: finds shortest deterministic context | Principled: hierarchical abstraction layers | Documents the mechanism formally |
| Semantic Meaning | None—context is purely statistical | Each layer holds meaningful abstraction | Provides visual & mathematical support |
| Black Box Problem | PPM★ in IDyOM is unexplained | IDyOT offers explanatory mechanism | Replaces black box with "magnifying glass" |
| Zero Probability | Escape mechanisms (Method C, D) | Gap identified—needs smoothing | Implements and documents solution |
This work provides:
- Empirical validation of IDyOT as a viable replacement for PPM★ in cognitive modeling
- Detailed documentation of within-layer interactions previously implicit in the literature
- Configuration analysis identifying optimal IDyOT variants for different use cases
- Foundation for future work in explainable music AI and cognitive creativity systems
Extended and documented the details of IDyOT memory implementation that were implicit in Wiggins (2019), providing both visual and mathematical support for the within-layer interactions.
Developed a "magnifying glass" metaphor/abstraction framework for analyzing IDyOT's information exchange within-layers, replacing the PPM★ "black box" with an explanatory cognitive model.
Successfully implemented both PPM★ and a version of IDyOT memory, enabling empirical comparison between: A descriptive model (PPM★) An explanatory cognitive model (IDyOT)
Validated online implementations of PPMC and PPM★C against established benchmarks (Moffat 1990, Cleary 1997, Bunton 1997).
Identified the optimal IDyOT configuration—one that does not use the Upward Distribution—which performed consistently better across different datasets in terms of KL divergence.
Identified a gap in IDyOT theory regarding the zero-probability problem and implemented smoothing techniques to address it.
How IDyOT memory compares with PPM★ algorithm Different alternatives for combining within-layer interaction information in IDyOT Which IDyOT configuration performs best against PPM★
Traditional approaches to cognitive modeling in music and language have relied on descriptive models—systems that accurately predict outcomes without explaining why those predictions work. PPM★ exemplifies this: it achieves excellent compression and prediction by finding deterministic contexts, but offers no insight into the cognitive mechanisms underlying human perception.
This thesis contributes to the paradigm shift toward explanatory models that:
- Propose hypothetical cognitive mechanisms
- Provide semantic meaning at each level of abstraction
- Align with empirical findings in cognitive neuroscience
Human cognition operates through hierarchical processing, where:
Higher Levels → Abstract concepts, long-term patterns
↕ bidirectional information flow
Lower Levels → Concrete percepts, immediate sensory input
IDyOT captures this through its layered memory structure:
- Bottom-up learning: Percepts are segmented and categorized into increasingly abstract concepts
- Top-down prediction: Higher layers modulate expectations at lower layers
- Within-layer interaction: The core mechanism documented in this thesis
This work has implications for:
| Domain | Application |
|---|---|
| Music AI | Explainable melodic generation systems |
| Natural Language Processing | Cognitively-grounded language models |
| Computational Creativity | Systems that can articulate "why" they create |
| Consciousness Studies | Empirical testing of predictive processing theories |
By providing formal documentation of IDyOT's within-layer interactions, this thesis enables future researchers to:
- Build on a well-documented cognitive architecture
- Test specific hypotheses about hierarchical prediction
- Compare cognitive models against human behavioral data
- Develop explainable AI systems grounded in cognitive science
While current AI systems—particularly Large Language Models (LLMs)—achieve remarkable performance through scale and statistical learning, they lack the architectural principles that enable human-like general intelligence. This thesis contributes to an alternative approach: building AGI through cognitively-grounded architectures that model how the brain actually works.
┌─────────────────────────────────────────────────────────────────────┐
│ TWO PATHS TO AGI │
├──────────────────────────────┬──────────────────────────────────────┤
│ SCALE-BASED (Current AI) │ COGNITION-BASED (This Research) │
├──────────────────────────────┼──────────────────────────────────────┤
│ • More parameters │ • Brain-inspired architecture │
│ • More training data │ • Hierarchical abstraction │
│ • Emergent capabilities │ • Predictive processing │
│ • Black-box reasoning │ • Explainable mechanisms │
│ • Task-specific fine-tuning │ • Domain-general learning │
└──────────────────────────────┴──────────────────────────────────────┘
IDyOT embodies key principles believed essential for AGI:
| AGI Requirement | How IDyOT Addresses It |
|---|---|
| Generalization | Domain-agnostic learning mechanism (works for music, language, any sequential domain) |
| Abstraction | Automatic hierarchical concept formation from raw percepts |
| Efficiency | Information efficiency principle (like the brain's metabolic constraints) |
| Prediction | Core mechanism is predictive—aligns with Free Energy Principle |
| Compositionality | Segments combine into higher-level structures with semantic meaning |
| Consciousness Potential | Compatible with Global Workspace Theory of consciousness |
Modern LLMs face a fundamental limitation: they cannot explain why they generate specific outputs. This "black box" problem is not merely inconvenient—it may represent a fundamental barrier to true general intelligence.
This thesis addresses this by:
- Documenting explicit mechanisms for prediction and abstraction
- Providing mathematical formalization of within-layer interactions
- Enabling hypothesis testing against human cognitive data
PPM★ (Descriptive) → IDyOT (Explanatory) → AGI (Future)
│ │ │
"What comes next?" "Why this prediction?" "What does it mean?"
│ │ │
High accuracy Mechanism exposed True understanding
No explanation Semantic hierarchy Conscious awareness?
This thesis is part of a larger research program aiming to:
- Replace black boxes with transparent, testable cognitive mechanisms
- Bridge the gap between statistical success (PPM★) and cognitive plausibility (IDyOT)
- Establish foundations for truly explainable AI systems
- Contribute to AGI through cognitively-grounded architecture rather than pure scale
Key Insight: If human-level intelligence emerges from the brain's predictive processing architecture, then modeling that architecture (as IDyOT does) may be a more direct path to AGI than scaling statistical models that lack such structure.