[Bug]: ErrorHandlers with monitoring features sometimes cause vasprun.xml corruption due to abrupt termination (frequent on GPU VASP)

[simplecaryu]

Code snippet

What happened?

Description

I have encountered a recurrent issue regarding custodian error handlers that monitor VASP calculations periodically (specifically those with is_monitor=True). When these handlers detect anomalies during a run, they terminate the calculation to modify input files (e.g., INCAR) and restart.

However, after such an intervention, the final validation step fails because the vasprun.xml file from the terminated run is corrupted (truncated or missing closing tags). Even though the handler successfully intervened, the validation throws a parsing error similar to the following:

xml.etree.ElementTree.ParseError: mismatched tag: line 822, column 23
[2025-11-07 07:46:24,381] [ERROR] [custodian.py:398:run] Validation failed: VasprunXMLValidator

I have confirmed the following behaviors:

• The parsing error occurs specifically when an error handler terminates the VASP job early.

• Disabling the error handlers with is_monitor=True eliminates the issue.

• Modifying the code to perform a graceful termination using STOPCAR also resolves the issue completely.

These findings strongly suggest that the default abrupt termination mechanism triggered by these monitors is the root cause.

This behavior has been observed in the following environments:

• VASP Versions: Both GPU and CPU versions (Tested on VASP 6.5.1 and 6.4.3).

• Note: The error occurs very frequently with the GPU version of VASP. While it is rare with the CPU version, it does happen occasionally.

• Custodian Version: Tested on the latest version.

• Infrastructure: Occurs on both local sandbox environments and HPC computing centers.

Analysis & Suggestion

It seems the main culprit is the abrupt termination signal sent to VASP when a monitor triggers. This "hard kill" cuts VASP off right in the middle of writing vasprun.xml, leaving it incomplete.

To fix this, I implemented a graceful exit mechanism using STOPCAR (e.g., LABORT = .TRUE.) instead of killing the process immediately. This gives VASP enough time to close file handles and finish writing the XML tags. After this change, the parsing errors completely disappeared in all my tests.

I think it would be really useful to add a flag (e.g., graceful_termination=False by default) to the ErrorHandler or VaspErrorHandler. If enabled, the handler would try to stop the job via STOPCAR first and wait for the process to finish cleanly before resorting to a hard kill. This would be a huge lifesaver for users running VASP on GPUs where this I/O issue is frequent.

Version

2025.8.13

Which OS?

• MacOS
• Windows
• Linux

Log output

[Andrew-S-Rosen]

Hi @simplecaryu. Thanks for your message.

This is very similar, if not identical, to #384. Indeed, if the VASP process is killed, there is a high probability that the vasprun.xml is broken. Most users do not really care about this because the run failed, so why would one need a valid vasprun.xml. But the error message is certainly confusing for end users.

I'd like to ask: aside from a misleading error message, what is the motivation to resolve the error? I am not opposed. In fact, I am strongly in favor of your idea. But I'd like to understand your perspective.

I think it would be really useful to add a flag (e.g., graceful_termination=False by default) to the ErrorHandler or VaspErrorHandler. If enabled, the handler would try to stop the job via STOPCAR first and wait for the process to finish cleanly before resorting to a hard kill. This would be a huge lifesaver for users running VASP on GPUs where this I/O issue is frequent.

I agree that this would be a cleaner way to stop the process, at the cost of a small compute burden. I would be happy to review a PR on this.

[simplecaryu]

Sorry for the late response!

I think there might be a slight difference in how we understand the origin of the error. As you mentioned, the VasprunXMLValidator error typically occurs when Custodian cannot fix the underlying problem. (I actually didn't know that) In my case, however, I believe the issue arises even when Custodian could fix the error, but the VASP process termination itself corrupts vasprun.xml. My motivation is that sometimes a calculation succeeds if I gracefully terminate using STOPCAR, but fails if I don't. That failure rate is non-negligible in GPU HPC environments.

I would be happy to try preparing a PR with a graceful_termination option, though I also understand that it might be faster if the core developers implement it directly. (I used monkey patch to monitor wrapper)