Research reproduction audit
A captured research-agent run that builds and tests an auditor for a physics-informed model reproduction packet.
Audit a one-dimensional convection PINN reproduction packet. Recompute the evidence from samples and distinguish a valid reproduction from fabricated metrics, a boundary mismatch, an unsatisfied residual, missing curvature gain, or insufficient final accuracy.
Captured run, de-identified01 / 06Task
Toolbash
- Intent
Inspect the benchmark packet
- Tool call
Inspect the provided repository and the public benchmark specification.
- Observed result
The task defines a one-dimensional convection PINN reproduction with a reference PDE, initial condition, periodic boundary, and analytical solution. The working tree contains the model, PDE, reproduction script, requirements, and rubric materials.
repository inspected benchmark specification located
Recorded state2 records
surface status repository inspected reference PDE located
- Intent
Check the packet format
- Tool call
Inspect the available reproduction packet and rubric files.
- Observed result
The packet format and rubric are present. The auditor can be written against the normative fields without relying on a precomputed result file.
packet format checked rubric materials located
Recorded state2 records
surface status packet fields checked rubric located
- Intent
Write the auditor
- Tool call
Write a normalized audit artifact that recomputes the packet metrics and returns a classification.
- Observed result
The audit artifact is written. Its public role is to recompute evidence from raw packet samples, rather than trust a reported score.
normalized audit artifact written
Recorded state2 records
artifact status audit.py written output contract recomputed metrics and classification
- Intent
Exercise failure paths
- Tool call
Run local checks covering valid, fabricated, boundary, residual, curvature-gain, and final-accuracy cases.
- Observed result
The local checks return the expected classifications for the valid reproduction and each named failure mode, including insufficient final accuracy.
failure-path checks completed outputs summarized
Recorded state2 records
check family status valid reproduction classified failure paths classified
- Intent
Submit the audit run
- Tool call
Submit the completed audit artifact for machine grading.
- Observed result
The captured run reports the auditor as complete and ready for the machine grade.
audit artifact submitted awaiting terminal grade
Recorded state2 records
surface status submission complete grader pending
- Intent
Terminal grade
- Tool call
Evaluate the submitted auditor against six held-out reproduction-audit checks.
- Observed result
The machine grader reports a passing result for all six held-out checks.
verdict: PASS held-out checks: 6 / 6 score: 1.0
Recorded state2 records
grade field value held-out checks 6 / 6 verdict Pass