Prepared: 2026-08-19. Method in one line: open I4R's public GitHub organisation, read every repo that could plausibly be tooling, and report what is actually there.
Read this before you trust the schema section below. The reproducibility assessment form this run went looking for lives on osf.io, which the egress proxy refused. The form's field structure was recovered indirectly — from the variable names, factor levels and recodes in the study's own
code/cleaning.R, which reads the teams' responses. That is a faithful reconstruction of what the form captured, not a reading of the form itself: exact question wording and scale anchors are unverified. Anyone confirming it should openosf.io/sz2g8/from a network that can reach it.
Paper. Brodeur et al., Proceedings of the National Academy of Sciences (2026). A randomised study of how team composition affects reproducibility assessment: teams were assigned to Human-Only, AI-Assisted, or AI-Led branches and set to work reproducing quantitative social-science papers.
Package. https://github.com/I4Replication/AI-Games — opened. R, MIT licence for code and CC-BY-4.0 for data. The repo carries data/ (including AI games.xlsx, prompts information, comparison papers, sentiment data), code/ (19 scripts plus master.R), output/, and a focus groups/ folder of qualitative material. The README gives a single entry point — source("code/master.R") from the repository root — and the master script installs packages, creates folders, cleans, and runs the analysis. Requires R >= 4.4.0 and 22 packages via pacman::p_load(). No runtime is stated.
Why it is worth reproducing. All input data ships in the repo, so nothing is gated: this is a complete package on public data with one command to run it, which is the profile the board most wants. It is also the corpus I4R has said it will benchmark its own engine against, so a reproduction here is legible to the people building the comparison. And the subject matter is the board's own: reproducing the paper about whether AI can assess reproducibility.
Completeness. verified_complete: yes — from the README, the file tree, and reading code/cleaning.R, not from executing anything. The usual caveat holds: pages were read through a fetch-and-summarise tool rather than a clone.
The engine is not public. I4R's organisation holds four public repos: AI-Games (R, the package above), utilities (Python, "several utility functions to help our processes"), AI-vertical (HTML), and I4R-First-Meta-Paper (Stata). There is no AI Replication Engine, no agent framework, no reproduction toolkit. Their public GitHub supports human-conducted replication; the engine described in their blog posts and grant applications is not released. Build-vs-adopt therefore stays open rather than resolved — with the note that they have said a toolkit is anticipated later in 2026.
The study's own result is a finding for this board. Brodeur et al. report that AI-assisted teams outperform AI-led teams, but not human-only teams — AI works best as a collaborative tool rather than an autonomous agent. That is direct evidence about the thing Night Shift is deciding how far to automate, and it favours the architecture already in place: blind two-reader extraction, reconciliation by script, verdicts assigned maintainer-side, a human merge. The line to hold is that an agent may produce numbers but must not certify its own verdict.
I4R's error taxonomy — the most useful thing found. cleaning.R reads the teams' responses under these fields, which together are the classification the form captured:
x1_major — major errorsx2_minor — minor errorsx3_pre_regression — errors before the regression stagex4_regression — errors in the regression itselfx5_transcription_post_regression — transcription errors after the regressionx6_false_positive — a reported error that turned out not to be onex7_error_known_by_i4r — the error was already knownalongside the reproduction outcome (reproduction, time_reproduction) and robustness (ran_robustness, good_robustness).
This is an axis the board does not currently have. Night Shift's inclusion_rule classes (target / restatement / parameter / excluded) answer which numbers count; its three-state schema answers did it reproduce. Neither answers what kind of error this was. Two of I4R's categories are worth adopting regardless of the rest: x6_false_positive, which is exactly the harm the second-run gate exists to prevent and is currently tracked nowhere, and x7_error_known_by_i4r, which separates a novel finding from a rediscovery.
Reached: github.com, raw.githubusercontent.com, i4replication.org. Refused by the proxy: osf.io (the template form and AI training materials), arxiv.org (the ReproRepo and ETH papers), www.builder.io. The OSF block is the one that cost something, and the workaround is stated at the top of this file rather than hidden.
(2026). A randomised study of how team composition affects reproducibility
assessment: teams were assigned to **Human-Only**, **AI-Assisted**, or **AI-Led**
branches and set to work reproducing quantitative social-science papers.
**Package.** https://github.com/I4Replication/AI-Games — **opened.** R, MIT
licence for code and CC-BY-4.0 for data. The repo carries `data/` (including
`AI games.xlsx`, prompts information, comparison papers, sentiment data),
`code/` (19 scripts plus `master.R`), `output/`, and a `focus groups/` folder of
qualitative material. The README gives a single entry point — `source("code/master.R")`
from the repository root — and the master script installs packages, creates
folders, cleans, and runs the analysis. Requires R >= 4.4.0 and 22 packages via
`pacman::p_load()`. No runtime is stated.
**Why it is worth reproducing.** All input data ships in the repo, so nothing is
gated: this is a complete package on public data with one command to run it,
which is the profile the board most wants. It is also the corpus I4R has said it
will benchmark its own engine against, so a reproduction here is legible to the
people building the comparison. And the subject matter is the board's own:
reproducing the paper about whether AI can assess reproducibility.
**Completeness.** `verified_complete: yes` — from the README, the file tree, and
reading `code/cleaning.R`, not from executing anything. The usual caveat holds:
pages were read through a fetch-and-summarise tool rather than a clone.
---
https://github.com/I4Replication/AI-Games
You do not need any of this: the Run button sets it up. These are the exact settings and brief the run uses, for checking it or running it by hand.
Nothing. It searches.
This is the environment file as it stands today, not a record of this run. The run predates environment capture, so what it actually had on hand was never recorded and cannot be recovered.
# Find brief — prior art, the Institute for Replication **Environment:** `night-shift-default`. The settings are on this mission's page. This run was not a broad sweep. It was one question, asked because the answer changes what the board should build: > I4R is publicly building an "AI Replication Engine" — three agents that > re-execute research code and compare against published results. Does it exist > in a form Night Shift can adopt, instead of building a parallel one? And is > there anything else in their public work worth taking? Two rules carried over from FIND-002 and FIND-003, because they are what make a row trustworthy: - **Never invent a paper, a title, or a package URL.** - **`package_read: yes` only if the package was actually opened in this run.** Found-by-search is not opened. Report what could not be reached, by name. A blocked host is a finding, not an omission to be quietly rounded away.