MSE-GLM: A Deterministic, Zero-Weight Graph Language Model, Step by Step

By Clifford Chivhanga · July 17, 2026 · github.com/fodokidza/mse_glm

Most language models today are a pile of weights nobody can fully read. MSE-GLM (Matrix-Structured Edge — Graph Language Model) is an attempt at the opposite: a language model with no neural weights, no embeddings, and no probability distributions. Every decision it makes during generation can be traced back to a specific row in a specific matrix, built from the training text itself. Nothing is learned in the gradient-descent sense — everything is counted, deduplicated, and indexed.

This post walks through every stage of the pipeline in the order it actually runs — tokenizing, building the graph, generating text, inferring beyond what was literally seen, and finally, a newer piece: figuring out what a cluster of interchangeable tokens actually means.

Step 1 — Tokenizing the corpus (tokenizer.py)

Everything starts with a from-scratch Byte Pair Encoding (BPE) tokenizer. No external tokenizer library, no pretrained vocabulary. Training text is lowercased, stripped of anything that isn't a letter, digit, or space, and split into sentences on . ! ? and newlines.

Four reserved ids anchor the vocabulary before any merges happen:

<PAD> = 0   reserved
<UNK> = 1   unknown character fallback
<BOS> = 2   prepended to every encoded sequence
<EOS> = 3   appended only during training

From there it's classic BPE: every character in the corpus becomes a one-character token, then the tokenizer repeatedly finds the most frequent adjacent pair of symbols across all words and merges it into a new token, until vocab_size is reached or there are no pairs left to merge. The merge list is stored in order and replayed identically at encode time, so a word is always split the same way it would have been split during training.

Two details matter for everything downstream:

• encode() always prepends <BOS> — this matters later because BOS is treated as a real, if artificial, "previous token" for the very first generation step.
• encode_for_training() additionally appends <EOS>, so every training sentence has an explicit, learnable end.

Step 2 — Building the graph (graph.py)

Once every training sentence is a sequence of token ids, MSE-GLM builds three matrices from those sequences. All three are array-backed (Python's array('i')) and CSR-indexed — the same compressed-sparse-row layout used in sparse linear algebra — so every lookup ("what comes after token X?") is O(1) rather than a scan.

2a. The Edge Matrix — what follows what

The simplest structure: every distinct bigram (token[i], token[i+1]) seen anywhere in training, deduplicated. Given a token, successors() returns every token that has ever directly followed it. This is the model's notion of "grammatically legal next token" — nothing more.

2b. The Bridge Matrix — triples, and where clusters come from

This is the heart of the system. For every 3-token window (source, bridge, target) in the training sequences, the Bridge Matrix stores a deduplicated triple. So for "the cat sat", the triple is (source=the, bridge=cat, target=sat).

Every triple additionally gets a cluster_id, assigned by a simple dual-axis rule:

• Bridge axis — if two or more triples share the same (source, target) pair, their bridge tokens are grouped into one cluster. Example: "the cat sat" and "the dog sat" share (the, sat), so cat and dog get the same cluster_id — they're interchangeable in that slot.
• Target axis — if two or more triples (not already grouped on the bridge axis) share the same (source, bridge) pair, their target tokens are grouped instead. Example: "cat is animal" and "cat is pet" share (cat, is), so animal and pet get grouped as things "cat" can be.
• Anything that doesn't fit either rule gets cluster_id = 0 — the "unclustered" bucket. (More on why that bucket turns out to matter later in this post.)

Alongside the triples, the Bridge Matrix keeps a t_index: a reverse map from every token to the (non-zero) cluster_ids it participates in as a bridge or target anywhere in the graph. This is what lets the system ask "does token A show up in the same kind of slot as token B, anywhere in the corpus?" without rescanning everything.

2c. The Relationship Matrix — lineage

The Relationship Matrix is deliberately thin: it stores only pairs of (triple_id, relationship_id), where a relationship_id is just the index of the training sentence a triple came from. It doesn't duplicate any triple content — it's a foreign key back into the Bridge Matrix's row order. A single triple can belong to several relationship_ids if the exact same 3-token pattern showed up in more than one training sentence.

This sounds like a small bookkeeping detail, but it's what makes generation deterministic rather than a popularity contest (see Step 3).

Step 3 — Generation: the two-stage lineage-vote pipeline (inference.py)

This is the part that replaces what a neural model would do with a softmax over a hidden state. MSE-GLM's InferenceEngine picks the next token through two ranked stages, both gated by a concept called lineage — the set of relationship_ids (training sentences) the generation path has stayed consistent with so far.

Stage 1 — current token authority

Given the current token, collect every legal successor from the Edge Matrix. For each candidate C, check whether a Bridge triple (source = current, bridge = C) exists whose relationship_id intersects the currently active lineage. Read as: "the current token only votes for a candidate it recognizes as its own bridge, and only if that recognition traces back to a sentence we're still consistent with."

• Exactly one candidate survives → that's the next token.
• A tie, but no previous token exists yet (start of generation) → pick the first candidate; all tied options are equally legitimate.
• A tie, and a previous token exists → fall through to Stage 2.

Stage 2 — previous token authority

First, the active lineage is narrowed to whatever relationship_ids the exact (previous, current) pair actually shares — this keeps the lineage tied to what those two tokens were literally trained on together, rather than a broader set inherited from earlier in generation. Then, among only the Stage 1 survivors, vote for candidate C if the exact triple (source = previous, bridge = current, target = C) exists with a matching relationship_id.

• One survivor → done.
• Still tied → genuinely ambiguous continuations; pick randomly among them.
• No survivors at all → fall back to a random pick among the Stage 1 survivors.

One rule holds throughout the whole generation: active_rels only ever narrows by intersection, never widens. A later, looser match can't undo an earlier, more specific one. If Stage 1 ever finds literally no legal successors, the model emits <EOS> immediately — it doesn't try to guess.

Every one of these decisions is inspectable. step() returns not just the chosen token but which stage decided it, which rule fired, and what the active lineage was at that moment — this is what makes the "explainable" half of "deterministic, explainable, zero-weight" true in practice, not just in the README.

Step 4 — Going beyond training data: Open Mode (experience.py)

Strict Mode only ever says what it directly saw. Open Mode adds a second, clearly separated set of matrices — the Experience Edge, Bridge, and Relationship matrices — built by a structural inference pass over the already-trained Bridge Matrix, using cluster substitutability rather than anything new observed in text.

Two expansion rules generate new, never-literally-seen triples:

• Rule 1 (bridge axis): if token X shares a cluster with bridge member B in slot (S, T), and B also bridges some other slot (S2, T2), then X is inferred to be able to bridge (S2, T2) too.
• Rule 2 (target axis): the mirror image — if X shares a cluster with target member T in slot (S, B), and T also targets some other slot (S2, B2), X can target (S2, B2) too.

Both rules are pure substitution logic over clusters already discovered in Step 2 — no new information is invented, only recombined. The three resulting matrices are saved separately (experience_edges.json, experience_bridges.json, experience_relationships.json) so Strict Mode never touches them, and Open Mode inference simply unions both sets of matrices at every lookup. Passing None for the experience matrices degrades an InferenceEngine straight back to Strict Mode with no special-casing required.

Step 5 — Reading the graph back: the analysis layer (analyse.py)

Because everything is stored as explicit, inspectable matrices, MSE-GLM ships a whole read-only analysis layer on top — it never touches generation, it only reads. Highlights:

• cluster_report — lists every non-zero cluster_id, its axis (bridge or target), and its members.
• token_similarity / infer_shared_role — given two or more tokens, intersects their t_index cluster memberships to find slots they're all interchangeable in, and predicts what would fill that shared slot.
• trace — runs generation for a prompt and prints the stage/rule/lineage behind every single token, exactly as step() returned it.

Step 6 — Naming the clusters: the Cluster Interpreter (CI) Matrix (interpret.py)

A cluster_id is just an integer. Knowing that cat, dog, and pig share cluster 1 doesn't tell you why. The Cluster Interpreter layer is a read-only pass — same category as analyse.py, never touches inference — that proposes a human-readable label for a cluster (e.g. naming {cat, dog, pig} as "animal"), gathering evidence from all three matrices built in Step 2:

1. Bridge Matrix, source axis (primary signal). Fix (bridge, target), let source vary: if cluster members are each the source of a triple ending in the same (bridge, target) pair — e.g. each of them is the source of an "is animal" triple — that target token becomes a candidate label. coverage is simply the fraction of members that reach it this way.
2. Edge Matrix, direct adjacency. A Bridge triple only guarantees the two adjacent bigrams exist, not a direct source→target bigram. So checking whether the corpus also states the fact a shorter way (skipping the bridge word) is genuinely separate evidence, not a restatement of signal 1.
3. Bridge Matrix, shared-role check. Does the candidate label itself show up, via t_index, as an interchangeable member of some other cluster alongside one of these members — a structural family resemblance found by a completely different route.
4. Relationship Matrix, robustness. How many distinct training sentences assert the specific fact behind a candidate. One sentence could be a fluke; several is more durable.

Every candidate carries an evidence_mask listing exactly which of these fired — deliberately not collapsed into one confidence float, because the signals are correlated (they're all read off the same training sequences to different degrees), not independent probabilistic votes.

A cluster can carry more than one label

Nothing requires exactly one interpreter per cluster. {cat, dog, pig} can be "animal" (full coverage, all three) and, separately, the {cat, dog} subset within it can independently support "pet" (partial coverage, since the corpus never calls a pig a pet) — both survive as long as each clears its own thresholds. build_interpreter_matrix returns one row per (cluster_id, interpreter) pair, not one best guess per cluster.

Mining the zero cluster

The dual-axis rule from Step 2 only clusters by fixed source. There's no rule for "fix bridge and target, let source vary" — so a set of tokens that are each the source of one categorical statement, but never otherwise share a context, gets no cluster_id at all. They sit in the cluster_id = 0 bucket individually, invisible to every tool in Step 5 and Step 6 so far.

discover_zero_cluster_groups mines that bucket directly, applying exactly the missing rule to the rows the first two rules left behind. Tested against a corpus engineered so cat, dog, and pig shared nothing else — different verbs, different lead-in words — it still recovered {cat, dog, pig} → "animal" purely from the unclustered bucket. It's deliberately scoped as a targeted recovery tool for that one gap, not a general quality upgrade: most of what sits at cluster_id = 0 is unclustered for good reason (one-off phrasing, low-frequency noise), and the candidate space it searches is much larger than the regular interpreter matrix.

Putting it together

None of this involves a single learned weight. Every output — a generated token, a cluster label, an inferred Open Mode triple — can be traced back to specific rows in specific matrices, built directly from the training text. That's the whole bet MSE-GLM is making: that a useful amount of language modeling can be done with counting, indexing, and explicit structure, instead of a black box.