Reflectron — Data & Tables

What this page is

This document holds the full data for the Reflectron comparison experiments: eight architecture arms tested on four classification datasets across ten seeds each, with 200 training epochs per run. It is intended as a companion to the blog post, which summarizes the results and interpretation; here you'll find the raw per-seed numbers, per-arm parameter counts, exact hyperparameters, and the methodology footnotes referenced from the post.

Every measurement below was produced by the C binaries under test/baseline_compare/, aggregated by debug/aggregate_baseline.sh. Timing figures are wall-clock milliseconds on a single-core CPU workload — comparative only, not benchmarks.

How the experiments were run

Datasets

Training hyperparameters

Seeds

1..10 per arm per dataset (see the RNG caveat below on why not 0..9).

Epochs

200 per seed. Every epoch performs full-batch shuffled SGD over the training set.

Split

Per-seed shuffle: index array shuffled, first N_TRAIN used for training, remainder for test. Wine and breast additionally z-score all features using train-set statistics, applied to both train and test before training begins.

Optimizer

Plain SGD via NetTrainerStep: w −= lr · ∇w per parameter. No momentum, no weight decay, no learning-rate schedule.

Loss

Cross-entropy over softmax outputs for all classification arms.

Learning rates

iris: 0.05 (MLP, N2S). all others: 0.005 (MLP, N2S). SS uses 0.001 across the board — SS is less stable at higher rates.

num_reflections

1 for every N2S run reported here (a two-stage graph). Deeper reflection stacks did not produce statistically distinguishable gains, and the extra parameters and runtime weren't worth it.

Output layer

All N2S and MLP runs use LinearOut: the final layer is a plain LinearNeuron layer (no Reflectron output stage). This keeps the softmax input's dynamic range clean.

Reproducibility

The C binaries are deterministic given a seed. Setting srand(seed) at the start of each run gives byte-for-byte identical parameter initialization and identical epoch shuffle orders — this is what lets the cross-unit and nocross arms run pairwise-comparable experiments. Different builds may show tiny floating-point differences due to compiler reassociation; the reported numbers are from gcc -O0 -Wall.

The eight arms

Each dataset was tested against every arm below. Arm labels appear verbatim in the CSVs and in every summary table on this page.

Iris uses a 2-layer shape (hidden → LinearOut); the others use 3-layer (hidden → hidden → LinearOut). SS param counts are dominated by the per-halflayer α arrays across features and interfaces.

All arms, all datasets, aggregate stats

Best per dataset (highest mean test accuracy) is highlighted. Ties are broken by lower std_test (fewer bad seeds).

Times are per-seed averages of the training loop only (no data-loading or eval). Iris timings are dominated by construction overhead — the per-epoch work is very small on 120 training samples.

Iris

Small architecture, small training set. Iris is where seed sensitivity is most visible: MLP_equal_neurons collapses on seed 9 (0.233 test); N2S_nocross collapses on seed 4 (0.333 test) but N2S_cross rescues it to 1.000. Best mean test: N2S_unique_cross (0.940 ± 0.100).

Circles (synthetic 3-ring)

Cleanly separable with 800 training samples. Nearly every arm reaches 1.000 training accuracy; test-set differences come from a handful of ambiguous ring-boundary samples per seed. SS is the outlier here — it collapses to ~0.73 on this dataset. Best mean test: N2S_unique_nocross (0.993 ± 0.020).

Wine (UCI)

13-feature classification with only 142 training samples after split. Every N2S arm hits 1.000 training accuracy; test-set variance is dominated by ~1-3 misclassified samples out of 36. Wine is where equal-params MLP lost ground vs equal-neurons MLP — extra capacity without more training data hurt slightly. Best mean test: N2S_cross tied with N2S_nocross (0.992 ± 0.013).

Breast cancer (UCI Wisconsin Diagnostic)

30 features, 455 training samples, binary classification. The most compute-intensive of the four (breast SS takes ~27 seconds per seed). All arms cluster tightly around 0.96-0.98; wins are ~1-2 correctly-classified additional test samples out of 114. Best mean test: N2S_cross (0.975 ± 0.012).

Things to know before reading too much into the numbers

The glibc srand(0) trap

Under glibc, srand(0) and srand(1) produce byte-for-byte identical RNG sequences — the seeds 0 and 1 map to the same internal state. Early runs used seeds 0..9 and I initially thought there was a bug in the training loop when seeds 0 and 1 showed identical results across every arm. There wasn't. All reported data uses seeds 1..10 to sidestep this. If you re-run the binaries in a different libc environment, you may see different absolute numbers but the arm-relative ordering should hold.

Small-seed budget for iris

The iris MLP mean of 0.650 at 10 seeds is misleadingly low. Running the same MLP configuration at 50+ seeds pulls the mean to ~0.95 (see the older iris_baseline.c). The 10-seed number sits in a fragile-initialization regime where individual bad seeds dominate the mean. The Reflectron rows on iris are more stable across the same seed budget, but readers should treat "N2S beats MLP by 0.28 on iris" as partly a small-sample artifact — the honest gap at higher seed counts is smaller.

Runtime is comparative, not absolute

Timing is single-threaded on one CPU with no vectorization tuning or arena reuse across seeds. Ratios between arms on the same dataset are meaningful (N2S is ~3-5× MLP, SS is ~5-20× MLP); absolute numbers are not benchmarks. Wall-clock variance across runs is 5-15% on the same binary.

Cross-unit modulation is training-active everywhere

Even on datasets where cross-unit's aggregate test accuracy delta is roughly zero (circles, wine, breast), the trained weights differ substantially between cross and nocross runs on every seed. On iris seed 1 — a "quiet" seed where both variants reach ~99% test accuracy — all 91 trainable parameters differ between the two runs, and the total L2 divergence is 111% of the nocross weight-vector norm. The two runs land on effectively the same decision boundary via completely different weight configurations. See debug/param_diff.c.

MLP_equal_params sometimes hurts

On wine and breast, MLP_equal_params has lower mean test accuracy than MLP_equal_neurons (0.978 → 0.969 on wine; 0.966 → 0.963 on breast). Extra parameters without extra training data can search a slightly worse local minimum on nearly-saturated tasks. This reinforces the N2S win: the SHARED-mode Reflectron isn't beating MLP only because it has more parameters — because equal-params MLPs are not better than equal-neurons MLPs on these datasets. Something about the Reflectron's inductive bias contributes independently of raw parameter count.