Reproduction Results vs. Paper — Full Comparison

Mean over TSB-AD-M (n = 199). Δ = ours − paper (positive = higher than paper).

1. Overall (headline, Table 6.1)

Method	Source	ours AUC-PR	paper AUC-PR	Δ AUC-PR	ours AUC-ROC	paper AUC-ROC	Δ AUC-ROC
DeepAnT	`deepant/cnn.py`	0.350	0.33	+0.020	0.770	0.81	-0.040
RW	`rw/cnn_uns.py`	0.321	0.29	+0.031	0.703	0.73	-0.027
RW-1 (best-HP)	`rw/cnn_rw.py`	0.289	0.28	+0.009	0.725	0.75	-0.025

2. Per-collection (family) — ours (3 methods) + paper RW-1 + Δ

Paper per-family values (Tables 6.2/6.3) are reported for RW-1 only, so the Δ columns compare RW-1.

Family	n	DeepAnT PR/ROC	RW PR/ROC	RW-1 PR/ROC	paper RW-1 PR/ROC	Δ RW-1 PR	Δ RW-1 ROC
CATSv2	6	0.199/0.714	0.152/0.675	0.106/0.619	0.228/0.755	-0.122	-0.136
CreditCard	1	0.147/0.952	0.176/0.952	0.111/0.875	0.173/0.953	-0.062	-0.078
Daphnet	1	0.212/0.829	0.127/0.678	0.267/0.817	0.286/0.871	-0.019	-0.054
Exathlon	27	0.951/0.996	0.983/0.997	0.873/0.973	0.847/0.981	+0.026	-0.008
GECCO	1	0.454/0.927	0.400/0.911	0.639/0.959	0.621/0.979	+0.018	-0.020
GHL	24	0.016/0.592	0.021/0.403	0.033/0.665	0.013/0.564	+0.020	+0.101
Genesis	1	0.030/0.919	0.006/0.677	0.033/0.747	0.032/0.954	+0.001	-0.207
LTDB	5	0.336/0.662	0.212/0.595	0.248/0.642	0.253/0.611	-0.005	+0.031
MITDB	13	0.166/0.681	0.082/0.616	0.089/0.612	0.127/0.639	-0.038	-0.027
MSL	16	0.116/0.690	0.097/0.652	0.131/0.692	0.086/0.557	+0.045	+0.135
OPPORTUNITY	8	0.272/0.946	0.550/0.983	0.138/0.720	0.059/0.430	+0.079	+0.290
PSM	1	0.407/0.787	0.146/0.589	0.137/0.537	0.238/0.697	-0.101	-0.160
SMAP	27	0.146/0.677	0.071/0.601	0.119/0.635	0.095/0.578	+0.024	+0.057
SMD	22	0.396/0.853	0.366/0.766	0.233/0.714	0.317/0.790	-0.084	-0.076
SVDB	31	0.216/0.680	0.099/0.593	0.116/0.615	0.166/0.628	-0.050	-0.013
SWaT	2	0.516/0.798	0.515/0.813	0.444/0.764	0.227/0.689	+0.217	+0.075
TAO	13	0.996/0.999	1.000/1.000	0.995/0.996	1.000/1.000	-0.005	-0.004

3. Per-dataset (all 199) — AUC-PR / AUC-ROC per method

Δ(R1−DA) = RW-1 minus DeepAnT AUC-PR (per-dataset method gap). λ* = RW-1 best l1_weight.

#	dataset	DA PR	DA ROC	RW PR	RW ROC	R1 PR	R1 ROC	λ*	Δ(R1−DA) PR
1	Genesis_1	0.030	0.919	0.006	0.677	0.033	0.747	0.001	+0.004
2	MSL_1	0.145	0.704	0.018	0.628	0.136	0.594	0.001	-0.009
3	MSL_2	0.156	0.715	0.116	0.687	0.075	0.544	0.001	-0.081
4	MSL_3	0.104	0.694	0.073	0.627	0.146	0.764	0.01	+0.042
5	MSL_4	0.062	0.797	0.045	0.743	0.051	0.692	0.01	-0.011
6	MSL_5	0.100	0.529	0.090	0.532	0.108	0.588	1.0	+0.008
7	MSL_6	0.118	0.811	0.100	0.757	0.136	0.827	0.01	+0.018
8	MSL_7	0.193	0.791	0.121	0.685	0.103	0.618	0.001	-0.091
9	MSL_8	0.013	0.712	0.005	0.338	0.026	0.864	0.001	+0.013
10	MSL_9	0.029	0.442	0.026	0.409	0.033	0.522	0.001	+0.003
11	MSL_10	0.093	0.762	0.183	0.943	0.132	0.759	0.01	+0.039
12	MSL_11	0.154	0.766	0.114	0.700	0.206	0.863	0.01	+0.052
13	MSL_12	0.065	0.742	0.028	0.676	0.060	0.550	0.001	-0.005
14	MSL_13	0.104	0.766	0.186	0.911	0.151	0.731	0.01	+0.047
15	MSL_14	0.125	0.428	0.115	0.446	0.135	0.478	0.001	+0.010
16	MSL_15	0.202	0.601	0.146	0.570	0.232	0.780	0.01	+0.030
17	MSL_16	0.196	0.782	0.183	0.784	0.367	0.898	0.1	+0.172
18	Daphnet_1	0.212	0.829	0.127	0.678	0.267	0.817	0.001	+0.055
19	MITDB_1	0.241	0.695	0.192	0.631	0.165	0.622	1.0	-0.076
20	MITDB_2	0.036	0.736	0.027	0.681	0.032	0.558	0.01	-0.004
21	MITDB_3	0.091	0.654	0.017	0.447	0.024	0.521	0.001	-0.067
22	MITDB_4	0.101	0.530	0.039	0.408	0.053	0.535	0.001	-0.048
23	MITDB_5	0.075	0.711	0.035	0.740	0.028	0.577	0.01	-0.047
24	MITDB_6	0.192	0.707	0.131	0.726	0.128	0.704	0.001	-0.064
25	MITDB_7	0.573	0.845	0.179	0.787	0.151	0.812	0.001	-0.422
26	MITDB_8	0.197	0.611	0.123	0.572	0.111	0.567	0.001	-0.086
27	MITDB_9	0.119	0.669	0.010	0.648	0.012	0.526	0.01	-0.107
28	MITDB_10	0.066	0.719	0.038	0.554	0.033	0.668	0.001	-0.033
29	MITDB_11	0.348	0.799	0.204	0.776	0.339	0.782	0.001	-0.009
30	MITDB_12	0.076	0.632	0.045	0.510	0.058	0.583	0.001	-0.018
31	MITDB_13	0.037	0.549	0.029	0.531	0.025	0.506	0.01	-0.012
32	GHL_1	0.009	0.514	0.005	0.189	0.009	0.503	0.001	-0.000
33	GHL_2	0.007	0.424	0.007	0.407	0.090	0.847	0.1	+0.083
34	GHL_3	0.008	0.449	0.006	0.220	0.010	0.539	0.01	+0.002
35	GHL_4	0.012	0.455	0.007	0.116	0.013	0.490	0.001	+0.000
36	GHL_5	0.010	0.487	0.012	0.534	0.030	0.833	0.1	+0.020
37	GHL_6	0.009	0.546	0.014	0.692	0.009	0.525	0.1	-0.000
38	GHL_7	0.005	0.306	0.005	0.210	0.008	0.490	0.001	+0.002
39	GHL_8	0.006	0.580	0.003	0.168	0.007	0.600	0.1	+0.001
40	GHL_9	0.017	0.590	0.014	0.572	0.021	0.747	1.0	+0.004
41	GHL_10	0.024	0.808	0.005	0.070	0.009	0.497	0.01	-0.015
42	GHL_11	0.009	0.587	0.005	0.285	0.017	0.620	0.1	+0.008
43	GHL_12	0.020	0.883	0.015	0.844	0.187	0.988	0.1	+0.167
44	GHL_13	0.009	0.438	0.009	0.557	0.028	0.855	0.1	+0.019
45	GHL_14	0.047	0.918	0.035	0.886	0.022	0.823	0.1	-0.025
46	GHL_15	0.044	0.937	0.254	0.992	0.014	0.727	0.01	-0.030
47	GHL_16	0.016	0.538	0.007	0.121	0.028	0.755	0.01	+0.012
48	GHL_17	0.018	0.542	0.013	0.336	0.023	0.574	0.01	+0.005
49	GHL_18	0.026	0.502	0.015	0.095	0.025	0.476	0.001	-0.001
51	GHL_20	0.009	0.659	0.004	0.208	0.007	0.598	0.01	-0.002
52	GHL_21	0.007	0.652	0.003	0.293	0.011	0.796	0.1	+0.004
53	GHL_22	0.028	0.478	0.034	0.564	0.167	0.913	1.0	+0.139
54	GHL_23	0.030	0.698	0.017	0.468	0.030	0.650	1.0	-0.001
55	GHL_24	0.007	0.626	0.003	0.170	0.005	0.528	0.01	-0.002
56	GHL_25	0.018	0.600	0.021	0.670	0.017	0.578	0.1	-0.000
57	SMD_1	0.207	0.877	0.242	0.836	0.083	0.781	0.01	-0.124
58	SMD_2	0.432	0.932	0.295	0.829	0.253	0.692	0.001	-0.179
59	SMD_3	0.452	0.956	0.508	0.935	0.276	0.826	0.001	-0.177
60	SMD_4	0.295	0.765	0.148	0.607	0.147	0.626	0.001	-0.148
61	SMD_5	0.131	0.707	0.412	0.755	0.180	0.851	0.01	+0.049
62	SMD_6	0.469	0.902	0.411	0.758	0.277	0.871	0.01	-0.192
63	SMD_7	0.227	0.782	0.154	0.697	0.104	0.604	0.001	-0.123
64	SMD_8	0.248	0.759	0.214	0.650	0.259	0.641	0.001	+0.011
65	SMD_9	0.627	0.875	0.269	0.658	0.223	0.571	0.1	-0.403
66	SMD_10	0.292	0.842	0.163	0.672	0.094	0.598	0.001	-0.198
67	SMD_11	0.090	0.689	0.082	0.641	0.067	0.523	0.001	-0.024
68	SMD_12	0.459	0.884	0.455	0.850	0.289	0.821	0.001	-0.169
69	SMD_13	0.173	0.584	0.147	0.452	0.152	0.566	0.01	-0.021
70	SMD_14	0.316	0.881	0.371	0.846	0.206	0.691	0.001	-0.110
71	SMD_15	0.441	0.929	0.410	0.864	0.298	0.791	0.001	-0.143
72	SMD_16	0.512	0.907	0.406	0.799	0.240	0.765	0.01	-0.272
73	SMD_17	0.433	0.925	0.418	0.880	0.361	0.841	0.001	-0.072
74	SMD_18	0.372	0.928	0.356	0.645	0.152	0.639	0.001	-0.220
75	SMD_19	0.693	0.965	0.790	0.940	0.231	0.724	0.001	-0.462
76	SMD_20	0.956	0.993	0.898	0.984	0.876	0.957	0.001	-0.081
77	SMD_21	0.255	0.778	0.216	0.694	0.248	0.627	0.001	-0.006
78	SMD_22	0.628	0.912	0.686	0.857	0.108	0.691	0.01	-0.520
79	LTDB_1	0.588	0.805	0.294	0.672	0.344	0.753	1.0	-0.244
80	LTDB_2	0.139	0.482	0.135	0.469	0.230	0.553	0.001	+0.091
81	LTDB_3	0.425	0.613	0.427	0.625	0.420	0.601	0.001	-0.005
82	LTDB_4	0.463	0.675	0.191	0.542	0.235	0.613	0.001	-0.228
83	LTDB_5	0.062	0.735	0.011	0.669	0.011	0.690	0.001	-0.051
84	SVDB_1	0.084	0.610	0.030	0.558	0.040	0.535	0.01	-0.044
85	SVDB_2	0.159	0.642	0.069	0.597	0.066	0.584	0.001	-0.093
86	SVDB_3	0.141	0.638	0.092	0.566	0.116	0.607	0.001	-0.025
87	SVDB_4	0.513	0.761	0.133	0.639	0.145	0.623	0.001	-0.368
88	SVDB_5	0.116	0.554	0.110	0.544	0.115	0.557	0.001	-0.001
89	SVDB_6	0.043	0.693	0.017	0.530	0.029	0.555	0.01	-0.015
90	SVDB_7	0.567	0.833	0.106	0.575	0.099	0.578	0.001	-0.467
91	SVDB_8	0.169	0.669	0.163	0.662	0.177	0.689	0.001	+0.008
92	SVDB_9	0.435	0.797	0.129	0.716	0.126	0.664	0.001	-0.309
93	SVDB_10	0.167	0.626	0.161	0.627	0.241	0.670	0.001	+0.074
94	SVDB_11	0.144	0.653	0.056	0.563	0.057	0.630	0.001	-0.086
95	SVDB_12	0.092	0.563	0.036	0.522	0.051	0.530	0.01	-0.042
96	SVDB_13	0.046	0.571	0.040	0.533	0.054	0.530	0.01	+0.007
97	SVDB_14	0.435	0.735	0.349	0.703	0.407	0.729	0.001	-0.027
98	SVDB_15	0.142	0.608	0.102	0.536	0.112	0.570	0.001	-0.030
99	SVDB_16	0.284	0.740	0.200	0.660	0.203	0.720	0.001	-0.081
100	SVDB_17	0.365	0.706	0.038	0.549	0.072	0.640	0.001	-0.294
101	SVDB_18	0.235	0.728	0.145	0.658	0.175	0.675	0.001	-0.060
102	SVDB_19	0.348	0.832	0.115	0.666	0.141	0.583	0.01	-0.208
103	SVDB_20	0.076	0.639	0.029	0.537	0.036	0.596	1.0	-0.041
104	SVDB_21	0.240	0.756	0.064	0.649	0.090	0.679	0.001	-0.151
105	SVDB_22	0.168	0.634	0.149	0.598	0.136	0.581	0.001	-0.032
106	SVDB_23	0.134	0.715	0.021	0.583	0.109	0.639	0.01	-0.025
107	SVDB_24	0.179	0.694	0.028	0.532	0.060	0.645	0.001	-0.119
108	SVDB_25	0.327	0.798	0.179	0.759	0.178	0.780	0.001	-0.149
109	SVDB_26	0.187	0.667	0.134	0.589	0.145	0.607	0.001	-0.042
110	SVDB_27	0.103	0.686	0.040	0.510	0.052	0.610	0.001	-0.051
111	SVDB_28	0.398	0.769	0.189	0.671	0.191	0.620	0.001	-0.207
112	SVDB_29	0.012	0.503	0.010	0.456	0.012	0.516	0.01	-0.000
113	SVDB_30	0.384	0.754	0.128	0.629	0.150	0.646	0.001	-0.234
114	SVDB_31	0.005	0.498	0.006	0.473	0.008	0.463	0.001	+0.003
115	PSM_1	0.407	0.787	0.146	0.589	0.137	0.537	0.001	-0.270
116	TAO_1	0.993	0.998	1.000	1.000	0.992	0.995	0.001	-0.001
117	TAO_2	0.998	0.999	1.000	1.000	0.996	0.996	0.001	-0.002
118	TAO_3	0.995	0.998	1.000	1.000	0.994	0.995	0.001	-0.001
119	TAO_4	0.996	0.999	1.000	1.000	0.994	0.995	0.001	-0.002
120	TAO_5	0.994	0.998	1.000	1.000	0.992	0.995	0.001	-0.002
121	TAO_6	0.998	0.999	1.000	1.000	0.997	0.998	0.001	-0.001
122	TAO_7	0.996	0.999	1.000	1.000	0.995	0.996	0.001	-0.001
123	TAO_8	1.000	1.000	1.000	1.000	1.000	1.000	0.001	+0.000
124	TAO_9	0.993	0.998	1.000	1.000	0.992	0.993	0.001	-0.001
125	TAO_10	0.999	1.000	1.000	1.000	0.998	0.999	0.001	-0.000
126	TAO_11	0.997	0.999	1.000	1.000	0.995	0.996	0.001	-0.001
127	TAO_12	0.995	0.998	1.000	1.000	0.994	0.996	0.001	-0.000
128	TAO_13	0.998	0.999	1.000	1.000	0.997	0.998	0.001	-0.001
129	OPPORTUNITY_1	0.199	0.947	0.349	0.970	0.045	0.692	1.0	-0.155
130	OPPORTUNITY_2	0.347	0.921	0.519	0.963	0.090	0.529	1.0	-0.257
131	OPPORTUNITY_3	0.120	0.961	0.371	0.992	0.019	0.714	0.01	-0.102
132	OPPORTUNITY_4	0.231	0.928	0.567	0.985	0.102	0.775	1.0	-0.129
133	OPPORTUNITY_5	0.726	0.988	0.940	0.998	0.700	0.978	0.1	-0.026
134	OPPORTUNITY_6	0.221	0.966	0.692	0.995	0.027	0.634	0.01	-0.194
135	OPPORTUNITY_7	0.209	0.935	0.652	0.988	0.087	0.770	0.01	-0.122
136	OPPORTUNITY_8	0.119	0.922	0.313	0.975	0.037	0.668	0.01	-0.081
137	CreditCard_1	0.147	0.952	0.176	0.952	0.111	0.875	0.001	-0.036
138	CATSv2_1	0.076	0.648	0.050	0.558	0.046	0.552	0.001	-0.030
139	CATSv2_2	0.070	0.673	0.094	0.686	0.060	0.593	0.001	-0.010
140	CATSv2_3	0.366	0.789	0.321	0.764	0.170	0.633	0.001	-0.196
141	CATSv2_4	0.369	0.783	0.258	0.689	0.142	0.677	0.001	-0.228
142	CATSv2_5	0.181	0.642	0.127	0.590	0.133	0.611	0.001	-0.049
143	CATSv2_6	0.128	0.747	0.062	0.761	0.088	0.649	0.001	-0.040
144	SMAP_1	0.234	0.556	0.189	0.602	0.258	0.711	0.001	+0.025
145	SMAP_2	0.168	0.700	0.128	0.677	0.075	0.507	0.001	-0.093
146	SMAP_3	0.112	0.838	0.082	0.784	0.053	0.659	0.1	-0.058
147	SMAP_4	0.019	0.569	0.015	0.492	0.017	0.542	0.001	-0.002
148	SMAP_5	0.034	0.701	0.012	0.639	0.057	0.773	0.01	+0.023
149	SMAP_6	0.072	0.493	0.059	0.420	0.070	0.528	1.0	-0.002
150	SMAP_7	0.067	0.653	0.049	0.584	0.059	0.644	0.001	-0.008
151	SMAP_8	0.031	0.672	0.025	0.584	0.037	0.667	0.01	+0.006
152	SMAP_9	0.061	0.578	0.054	0.500	0.053	0.516	0.001	-0.008
153	SMAP_10	0.020	0.576	0.007	0.527	0.019	0.573	1.0	-0.001
154	SMAP_11	0.011	0.274	0.010	0.253	0.014	0.501	0.001	+0.003
155	SMAP_12	0.036	0.822	0.031	0.650	0.042	0.715	0.01	+0.006
156	SMAP_13	0.725	0.775	0.090	0.716	0.408	0.743	0.001	-0.317
157	SMAP_14	0.111	0.800	0.085	0.762	0.040	0.463	0.001	-0.071
158	SMAP_15	0.166	0.908	0.201	0.699	0.113	0.576	0.001	-0.052
159	SMAP_16	0.358	0.556	0.171	0.656	0.385	0.958	0.1	+0.027
160	SMAP_17	0.089	0.629	0.081	0.636	0.057	0.498	0.001	-0.032
161	SMAP_18	0.065	0.583	0.078	0.660	0.049	0.461	1.0	-0.016
162	SMAP_19	0.290	0.893	0.032	0.581	0.164	0.806	0.01	-0.126
163	SMAP_20	0.030	0.620	0.025	0.578	0.026	0.579	0.1	-0.005
164	SMAP_21	0.031	0.684	0.008	0.507	0.027	0.574	0.01	-0.004
165	SMAP_22	0.100	0.666	0.069	0.564	0.144	0.736	1.0	+0.044
166	SMAP_23	0.834	0.958	0.193	0.910	0.784	0.877	0.001	-0.049
167	SMAP_24	0.155	0.853	0.177	0.663	0.123	0.679	0.1	-0.032
168	SMAP_25	0.074	0.636	0.020	0.488	0.095	0.786	0.01	+0.022
169	SMAP_26	0.036	0.658	0.011	0.478	0.028	0.530	0.01	-0.008
170	SMAP_27	0.018	0.626	0.014	0.607	0.014	0.536	0.1	-0.004
171	SWaT_1	0.221	0.703	0.229	0.723	0.268	0.736	0.1	+0.047
172	SWaT_2	0.811	0.894	0.801	0.903	0.620	0.792	1.0	-0.192
173	GECCO_1	0.454	0.927	0.400	0.911	0.639	0.959	0.001	+0.184
174	Exathlon_1	0.896	0.995	0.921	0.997	0.942	0.992	0.1	+0.046
175	Exathlon_2	0.992	0.998	0.992	0.997	0.929	0.981	1.0	-0.063
176	Exathlon_3	0.955	0.997	0.996	0.999	0.939	0.988	0.1	-0.016
177	Exathlon_4	0.949	0.997	0.980	0.996	0.947	0.996	0.1	-0.002
178	Exathlon_5	0.965	0.996	0.986	0.997	0.749	0.945	0.1	-0.216
179	Exathlon_6	0.772	0.987	0.909	0.992	0.582	0.929	0.1	-0.190
180	Exathlon_7	0.955	0.997	0.996	0.999	0.939	0.988	0.1	-0.016
181	Exathlon_8	0.950	0.997	0.994	1.000	0.858	0.979	0.1	-0.093
182	Exathlon_9	0.955	0.997	0.996	0.999	0.939	0.988	0.1	-0.016
183	Exathlon_10	0.949	0.997	0.980	0.996	0.947	0.996	0.1	-0.002
184	Exathlon_11	0.949	0.997	0.980	0.996	0.947	0.996	0.1	-0.002
185	Exathlon_12	0.949	0.997	0.980	0.996	0.947	0.996	0.1	-0.002
186	Exathlon_13	0.955	0.997	0.996	0.999	0.939	0.988	0.1	-0.016
187	Exathlon_14	0.985	0.998	0.980	0.997	0.889	0.972	0.1	-0.096
188	Exathlon_15	0.965	0.996	0.986	0.997	0.749	0.945	0.1	-0.216
189	Exathlon_16	0.950	0.997	0.994	1.000	0.858	0.979	0.1	-0.093
190	Exathlon_17	0.965	0.996	0.986	0.997	0.749	0.945	0.1	-0.216
191	Exathlon_18	0.949	0.997	0.980	0.996	0.947	0.996	0.1	-0.002
192	Exathlon_19	0.949	0.997	0.980	0.996	0.947	0.996	0.1	-0.002
193	Exathlon_20	0.971	0.987	0.983	0.991	0.932	0.977	1.0	-0.039
194	Exathlon_21	0.997	1.000	0.997	1.000	0.638	0.850	0.001	-0.359
195	Exathlon_22	0.955	0.997	0.996	0.999	0.939	0.988	0.1	-0.016
196	Exathlon_23	0.949	0.997	0.980	0.996	0.947	0.996	0.1	-0.002
197	Exathlon_24	0.955	0.997	0.996	0.999	0.939	0.988	0.1	-0.016
198	Exathlon_25	0.965	0.996	0.986	0.997	0.749	0.945	0.1	-0.216
199	Exathlon_26	0.965	0.996	0.986	0.997	0.749	0.945	0.1	-0.216
200	Exathlon_27	0.955	0.997	0.996	0.999	0.939	0.988	0.1	-0.016

Notes

DeepAnT / RW: single-config full run. RW-1: per-dataset best-HP (max AUC-PR over l1_weight ∈ {1.0,0.1,0.01,0.001}).
RW-1 vs Table 6.5 best-HP (AUC-PR 0.35): Δ −0.061; vs Table 6.1 (0.28): Δ +0.009.
RW-1 best-HP has 0 frozen datasets (AUC-ROC≪0.5), down from 17 in the old single-config run.
Sources: TSB-AD/eval/metrics/multi/CNN (DeepAnT), multi_algofaithful/CNN_UNS (RW), multi_rw1_l1sweep/l1_*/CNN_RW (RW-1).

FilesExpand file tree

reproduction_comparison.md

Latest commit