Reproduction package

This package supports the paper From Scores to Stable Evidence: Evaluating Structured Outputs in a Stochastic ML Software Pipeline. It contains the four processed study subjects, 40 retained GAER embeddings, 800 KMeans partitions fitted with ten initializations each, the analysis code, and the compact result tables used by the manuscript.

The package has two purposes:

1. verify the reported results directly from immutable retained artifacts;
2. regenerate the analyses and paper figures without retraining the models.

The graph-autoencoder implementation is derived from GAER 1.1.0. Its BSD license and the CC0 data dedication are included as LICENSE and LICENSE-DATA.

Contents

• data/ contains the four processed node and typed-dependency datasets and the Lucene reference mapping;
• output/ contains five embedding runs per system and encoder, with twenty clustering runs per embedding;
• src/ contains the instrumented pipeline and analysis programs;
• analysis/ contains retained result tables and the 24-case qualitative audit;
• scripts/ contains artifact verification, ordered analysis regeneration, controlled RQ3 validation, the KMeans initialization sensitivity check, and paper-figure generation;
• figures/ contains the two regenerated vector figures.

Regenerable co-assignment matrices are deliberately omitted from the release. Running the full analysis creates approximately 223 MB of .npy intermediates under analysis/experiment2/coassociation/.

Environment

The experiments were run with Python 3.11 on macOS arm64. Dependencies are locked in uv.lock. With uv installed:

uv sync --extra cpu --frozen

The environment also defines a gpu extra for compatible CUDA systems. Different devices need not reproduce the retained embeddings byte for byte, so verification uses the archived embeddings and partitions.

Verify the retained artifact

From the package root:

uv run python scripts/verify_artifact.py

The verifier checks the input manifest, the 40-by-20 nested design, and the n_init=10 clustering metadata, qualitative-audit inventory, and headline RQ1--RQ3 results used in the manuscript.

Regenerate analyses and figures

uv run python scripts/run_analysis.py

This runs the analyses in dependency order, recreates the controlled RQ3 tables and vector figures, and finishes by running the artifact verifier. It takes about 30 minutes on a recent multi-core laptop. About half of that is scripts/ninit_variance_sensitivity.py, which reclusters all 40 retained embeddings twice to compare one against ten KMeans initializations per fit; run the other steps individually if that comparison is not needed.

The retained tables remain available for inspection without executing any code. analysis/experiment3/fidelity/audit_method.md documents the fixed selection and interpretation limits of the 24-case qualitative audit, while analysis/rq3_validation/README.md documents the dependence-aware RQ3 checks.

Data and scope

See DATA_PROVENANCE.md. Labels are excluded from representation learning and used only for external-fidelity evaluation. Lucene's 129 unmatched files remain in representation learning, clustering, stability, and coverage calculations, but not in label-based metrics.