custodian failed to correct the NBANDS: NBANDS seems to be too high.

[hongyi-zhao]

Hi there,

I tried to run the Using atomate2 with jobflow-remote example, but the two non-scf jobs from the flow failed with the following error:

werner@x13dai-t:~/Desktop/std_vasp_slurm/a6/e1/8c/a6e18c86-916f-40c5-a0ac-d26fe068d245_1$ tail -5 queue.err 
The number of bands has been changed
/home/werner/.pyenv/versions/3.11.1/envs/datasci/lib/python3.11/site-packages/custodian/vasp/handlers.py:732: UserWarning: NBANDS seems to be too high. The electronic structure may be inaccurate. You may want to rerun this job with a smaller number of cores.
  warnings.warn(
ERROR:custodian.custodian:VaspErrorHandler
ERROR:custodian.custodian:Unrecoverable error for handler: <custodian.vasp.handlers.VaspErrorHandler object at 0x14bebd7a0e50>

Based on my tries, all of the following revisions fixed the problem:

In [2]: from atomate2.vasp.flows.core import RelaxBandStructureMaker
   ...: from atomate2.vasp.powerups import add_metadata_to_flow
   ...: from jobflow_remote import submit_flow
   ...: from pymatgen.core import Structure
   ...: from atomate2.vasp.powerups import (
   ...:     update_user_incar_settings,
   ...:     update_user_potcar_functional,
   ...:     update_user_kpoints_settings,
   ...: )
   ...: 
   ...: #resources = {"nodes": 1, "partition": "micro", "time": "00:55:00", "ntasks": 48}
   ...: ncores = 16
   ...: resources = {"nodes": 1, "partition": "batch", "mem":"80g", "time": "00:55:00", "ntasks": ncores}
   ...: 
   ...: # construct a rock salt MgO structure
   ...: mgo_structure = Structure(
   ...:     lattice=[[0, 2.13, 2.13], [2.13, 0, 2.13], [2.13, 2.13, 0]],
   ...:     species=["Mg", "O"],
   ...:     coords=[[0, 0, 0], [0.5, 0.5, 0.5]],
   ...: )
   ...: 
   ...: # make a band structure flow to optimise the structure and obtain the band structure
   ...: bandstructure_flow = RelaxBandStructureMaker().make(mgo_structure)
   ...: bandstructure_flow = update_user_incar_settings(bandstructure_flow, {"NCORE": 4})
   ...: 
   ...: # (Optional) add metadata to the flow task document.
   ...: # Could be useful to filter specific results from the database.
   ...: # For e.g., adding material project ID for the compound, use following lines
   ...: bandstructure_flow = add_metadata_to_flow(
   ...:     flow=bandstructure_flow,
   ...:     additional_fields={"mp_id": "mp-190"},
   ...: )
   ...: 
   ...: submit_flow(
   ...: #    bandstructure_flow, worker="my_worker", resources=resources, project="my_project"
   ...:     bandstructure_flow, worker="vasp_slurm", exec_config="vasp_slurm", resources=resources, project="std"
   ...: )
Out[2]: ['255', '256', '257', '258', '259']

In [3]: from atomate2.vasp.flows.core import RelaxBandStructureMaker
   ...: from atomate2.vasp.powerups import add_metadata_to_flow
   ...: from jobflow_remote import submit_flow
   ...: from pymatgen.core import Structure
   ...: from atomate2.vasp.powerups import (
   ...:     update_user_incar_settings,
   ...:     update_user_potcar_functional,
   ...:     update_user_kpoints_settings,
   ...: )
   ...: 
   ...: #resources = {"nodes": 1, "partition": "micro", "time": "00:55:00", "ntasks": 48}
   ...: ncores = 16
   ...: resources = {"nodes": 1, "partition": "batch", "mem":"80g", "time": "00:55:00", "ntasks": ncores}
   ...: 
   ...: # construct a rock salt MgO structure
   ...: mgo_structure = Structure(
   ...:     lattice=[[0, 2.13, 2.13], [2.13, 0, 2.13], [2.13, 2.13, 0]],
   ...:     species=["Mg", "O"],
   ...:     coords=[[0, 0, 0], [0.5, 0.5, 0.5]],
   ...: )
   ...: 
   ...: # make a band structure flow to optimise the structure and obtain the band structure
   ...: bandstructure_flow = RelaxBandStructureMaker().make(mgo_structure)
   ...: bandstructure_flow = update_user_incar_settings(bandstructure_flow, {"KPAR": 4,
   ...:     "NCORE": 2})
   ...: 
   ...: # (Optional) add metadata to the flow task document.
   ...: # Could be useful to filter specific results from the database.
   ...: # For e.g., adding material project ID for the compound, use following lines
   ...: bandstructure_flow = add_metadata_to_flow(
   ...:     flow=bandstructure_flow,
   ...:     additional_fields={"mp_id": "mp-190"},
   ...: )
   ...: 
   ...: submit_flow(
   ...: #    bandstructure_flow, worker="my_worker", resources=resources, project="my_project"
   ...:     bandstructure_flow, worker="vasp_slurm", exec_config="vasp_slurm", resources=resources, project="std"
   ...: )
Out[3]: ['260', '261', '262', '263', '264']

In [4]: from atomate2.vasp.flows.core import RelaxBandStructureMaker
   ...: from atomate2.vasp.powerups import add_metadata_to_flow
   ...: from jobflow_remote import submit_flow
   ...: from pymatgen.core import Structure
   ...: from atomate2.vasp.powerups import (
   ...:     update_user_incar_settings,
   ...:     update_user_potcar_functional,
   ...:     update_user_kpoints_settings,
   ...: )
   ...: 
   ...: #resources = {"nodes": 1, "partition": "micro", "time": "00:55:00", "ntasks": 48}
   ...: ncores = 16
   ...: resources = {"nodes": 1, "partition": "batch", "mem":"80g", "time": "00:55:00", "ntasks": ncores}
   ...: 
   ...: # construct a rock salt MgO structure
   ...: mgo_structure = Structure(
   ...:     lattice=[[0, 2.13, 2.13], [2.13, 0, 2.13], [2.13, 2.13, 0]],
   ...:     species=["Mg", "O"],
   ...:     coords=[[0, 0, 0], [0.5, 0.5, 0.5]],
   ...: )
   ...: 
   ...: # make a band structure flow to optimise the structure and obtain the band structure
   ...: bandstructure_flow = RelaxBandStructureMaker().make(mgo_structure)
   ...: bandstructure_flow = update_user_incar_settings(bandstructure_flow, { "NBANDS": 32,
   ...:         #"KPAR": 4,    "NCORE": 2
   ...:         })
   ...: 
   ...: # (Optional) add metadata to the flow task document.
   ...: # Could be useful to filter specific results from the database.
   ...: # For e.g., adding material project ID for the compound, use following lines
   ...: bandstructure_flow = add_metadata_to_flow(
   ...:     flow=bandstructure_flow,
   ...:     additional_fields={"mp_id": "mp-190"},
   ...: )
   ...: 
   ...: submit_flow(
   ...: #    bandstructure_flow, worker="my_worker", resources=resources, project="my_project"
   ...:     bandstructure_flow, worker="vasp_slurm", exec_config="vasp_slurm", resources=resources, project="std"
   ...: )
Out[4]: ['265', '266', '267', '268', '269']

So, the following custodian codesnippet should be improved in this aspect to tackle such problems:

custodian/src/custodian/vasp/handlers.py

Lines 730 to 754 in aca839e

	if "auto_nbands" in self.errors and (nbands := self._get_nbands_from_outcar(directory)):
	outcar = load_outcar(os.path.join(directory, "OUTCAR"))
	if (nelect := outcar.nelect) and (nbands > 2 * nelect):
	self.error_count["auto_nbands"] += 1
	warnings.warn(
	"NBANDS seems to be too high. The electronic structure may be inaccurate. "
	"You may want to rerun this job with a smaller number of cores.",
	UserWarning,
	)
	elif nbands := vi["INCAR"].get("NBANDS"):
	kpar = vi["INCAR"].get("KPAR", 1)
	ncore = vi["INCAR"].get("NCORE", 1)
	# If the user set an NBANDS that isn't compatible with parallelization settings,
	# increase NBANDS to ensure correct task distribution and issue a UserWarning.
	# The number of ranks per band is (number of MPI ranks) / (KPAR * NCORE)
	if (ranks := outcar.run_stats.get("cores")) and (rem_bands := nbands % (ranks // (kpar * ncore))) != 0:
	actions.append({"dict": "INCAR", "action": {"_set": {"NBANDS": nbands + rem_bands}}})
	warnings.warn(
	f"Your NBANDS={nbands} setting was incompatible with your parallelization "
	f"settings, KPAR={kpar}, NCORE={ncore}, over {ranks} ranks. "
	f"The number of bands has been decreased accordingly to {nbands + rem_bands}.",
	UserWarning,
	)

.

See here for the related discussion.

Regards,
Zhao