Hi everyone,
I’m an independent researcher working on a reproducible, open-source framework that measures how large language models express certainty — not just what they say.
After a year of cross-model experiments (GPT-5, Claude 4.5, Gemini, Grok), I found that AI systems naturally organize their answers into four distinct “epistemic types,” defined by measurable confidence ratios and convergence width.
We call this framework IRIS Gate — it’s basically a map of knowledge reliability for AI output.
Core Idea
Every multi-model run produces a numerical confidence pattern that separates cleanly into:
Type Confidence Ratio What it Means Human Action
0 – Crisis / Conditional ≈ 1.26 Known emergency logic, activates only with triggers Trust if trigger
1 – Facts ≈ 1.27 Established knowledge Trust
2 – Exploration ≈ 0.49 Emerging hypotheses Verify
3 – Speculation ≈ 0.11 Unverifiable/future claims Override
It’s like a real-time reliability gauge for AI reasoning.
The system self-labels outputs as “Trust / Verify / Override,” which makes scientific and technical research auditable instead of opaque.
What’s Been Done
• 7 full multi-model runs (49 convergence chambers, ~1,200 data points)
• Reproducibility bundle with SHA-256 checksums and citation metadata
• Validated against a real biomedical use-case (CBD mechanism discovery, including the VDAC1 paradox)
What I Need Help With
I’d love collaborators who can help with:
1. Independent replication of the confidence-ratio experiment on other models
2. Code review & architecture of the iris_orchestrator.py classifier (Python)
3. Statistical validation (is the 4-type separation significant under bootstrapping?)
4. Open-science publication / peer review advice
Everything is MIT-licensed and reproducible; repo + data bundles are ready.
Links
• GitHub: (add your repo link here)
• Full documentation: EPISTEMIC_MAP_COMPLETE.md
• Paper draft (preprint): link if available
Why It Matters
LLMs are powerful but opaque.
This project aims to give them an epistemic dashboard — a way to quantify what kind of knowing each answer represents.
If it works, it could improve safety, reproducibility, and trust in scientific AI systems.
⸻
If you’re into interpretability, epistemology, or reproducible AI science — come help kick the tires.
Hi everyone, I’m an independent researcher working on a reproducible, open-source framework that measures how large language models express certainty — not just what they say.
After a year of cross-model experiments (GPT-5, Claude 4.5, Gemini, Grok), I found that AI systems naturally organize their answers into four distinct “epistemic types,” defined by measurable confidence ratios and convergence width. We call this framework IRIS Gate — it’s basically a map of knowledge reliability for AI output.
Core Idea
Every multi-model run produces a numerical confidence pattern that separates cleanly into:
Type Confidence Ratio What it Means Human Action 0 – Crisis / Conditional ≈ 1.26 Known emergency logic, activates only with triggers Trust if trigger 1 – Facts ≈ 1.27 Established knowledge Trust 2 – Exploration ≈ 0.49 Emerging hypotheses Verify 3 – Speculation ≈ 0.11 Unverifiable/future claims Override
It’s like a real-time reliability gauge for AI reasoning. The system self-labels outputs as “Trust / Verify / Override,” which makes scientific and technical research auditable instead of opaque.
What’s Been Done • 7 full multi-model runs (49 convergence chambers, ~1,200 data points) • Reproducibility bundle with SHA-256 checksums and citation metadata • Validated against a real biomedical use-case (CBD mechanism discovery, including the VDAC1 paradox)
What I Need Help With
I’d love collaborators who can help with: 1. Independent replication of the confidence-ratio experiment on other models 2. Code review & architecture of the iris_orchestrator.py classifier (Python) 3. Statistical validation (is the 4-type separation significant under bootstrapping?) 4. Open-science publication / peer review advice
Everything is MIT-licensed and reproducible; repo + data bundles are ready.
Links • GitHub: (add your repo link here) • Full documentation: EPISTEMIC_MAP_COMPLETE.md • Paper draft (preprint): link if available
Why It Matters
LLMs are powerful but opaque. This project aims to give them an epistemic dashboard — a way to quantify what kind of knowing each answer represents. If it works, it could improve safety, reproducibility, and trust in scientific AI systems.
⸻
If you’re into interpretability, epistemology, or reproducible AI science — come help kick the tires.