SSCHA
How to parallel SSCHA+QE on two different computing nodes on a cluster? #156
-
|
Dear Development Team, I am currently working with a cluster comprised of multiple computing nodes. I have successfully configured SSCHA to run in conjunction with pw.x on a single computing node. For this, I've been using PBS for job scheduling and submission from the login node. However, when I attempt to run SSCHA+QE across two different computing nodes, I run into issues. Here's a detailed breakdown: I created an input file, nvt_para.py, for SSCHA, modifying it from the tutorial file nvt_local.py. I adjusted the 'command' value as follows: command = '~/software/openmpi/4.0.5/bin/mpiexec -n 8 ~/software/QE/6.6_openmpi/bin/pw.x -npool 2 -i PREFIX.pwi > PREFIX.pwo'The whole content of the file nvt_para.py is shown below: import sscha, sscha.Ensemble, sscha.SchaMinimizer, sscha.Relax, sscha.Utilities
import cellconstructor as CC, cellconstructor.Phonons
import cellconstructor.Structure, cellconstructor.calculators
import cellconstructor.calculators
from ase.calculators.espresso import Espresso
import numpy as np, matplotlib.pyplot as plt
import sys, os
#
# Initialize the DFT (Quantum Espresso) calculator for H3S
# The input data is a dictionary that encodes the pw.x input file namelist
input_data = {
'control' : {
# Avoid writing wavefunctions on the disk
'disk_io' : 'None',
# Where to find the pseudopotential
'pseudo_dir' : '.',
'outdir' : './outdir',
'wfcdir' : './wfcdir',
'tprnfor' : True,
'tstress' : True
},
'system' : {
# Specify the basis set cutoffs
'ecutwfc' : 35, # Cutoff for wavefunction
'ecutrho' : 350, # Cutoff for the density
# Information about smearing (it is a metal)
'input_dft' : 'blyp', # exchange-correlation functional
'occupations' : 'smearing',
'smearing' : 'mv',
'degauss' : 0.02
},
'electrons' : {
'conv_thr' : 1e-8
}
}
#
# the pseudopotential for each chemical element
pseudopotentials = {'H' : 'H.pbe-rrkjus_psl.1.0.0.UPF', 'S' : 's_pbe_v1.4.uspp.F.UPF'}
# the kpoints mesh and the offset
kpts = (8,8,8)
koffset = (1,1,1)
#
# Specify the command to call quantum espresso
command = '~/software/openmpi/4.0.5/bin/mpiexec -n 8 ~/software/QE/6.6_openmpi/bin/pw.x -npool 2 -i PREFIX.pwi > PREFIX.pwo'
#
# Prepare the quantum espresso calculator
calculator = CC.calculators.Espresso(input_data,
pseudopotentials,
command = command,
kpts = kpts,
koffset = koffset)
#calculator = Espresso(pseudopotentials = pseudopotentials, input_data = input_data,
# command = command, kpts = kpts, koffset = koffset)
TEMPERATURE = 300
N_CONFIGS = 50
MAX_ITERATIONS = 20
START_DYN = 'start_sscha'
NQIRR = 3
# Let us load the starting dynamical matrix
dyn = CC.Phonons.Phonons(START_DYN, NQIRR)
dyn.Symmetrize()
#
# Initialize the random ionic ensemble
ensemble = sscha.Ensemble.Ensemble(dyn, TEMPERATURE)
#
# Initialize the free energy minimizer
minim = sscha.SchaMinimizer.SSCHA_Minimizer(ensemble)
minim.set_minimization_step(0.01)
#
# Initialize the NVT simulation
relax = sscha.Relax.SSCHA(minim, calculator, N_configs = N_CONFIGS,
max_pop = MAX_ITERATIONS)
#
# Define the I/O operations
# To save info about the free energy minimization after each step
ioinfo = sscha.Utilities.IOInfo()
ioinfo.SetupSaving("minim_info")
relax.setup_custom_functions(custom_function_post = ioinfo.CFP_SaveAll)
#
# Run the NVT simulation (save the stress to compute the pressure)
relax.relax(get_stress = True)
#
# If instead you want to run a NPT simulation, use
# The target pressure is given in GPa.
#relax.vc_relax(target_press = 0)
#
# You can also run a mixed simulation (NVT) but with variable lattice parameters
#relax.vc_relax(fix_volume = True)
#
# Now we can save the final dynamical matrix
# And print in stdout the info about the minimization
relax.minim.finalize()
relax.minim.dyn.save_qe("sscha_T{}_dyn".format(TEMPERATURE))To submit the job, I crafted a script named sub.sh and used the following command to submit the job qsub sub.shThe file 'sub.sh' reads: #!/bin/bash
#PBS -N noname
#PBS -o LogPBS
#PBS -e LogPBS.err
#PBS -m ae
#PBS -l nodes=node1:ppn=16+node2:ppn=16
cd $PBS_O_WORKDIR
echo Job started at `date`
echo Directory is $PWD
echo This job runs on the following nodes:
cat $PBS_NODEFILE
#mpiexec -n 2 -machinefile $PBS_NODEFILE python nvt_para.py >out.log
mpiexec -n 2 -machinefile nodefile python nvt_para.py >out.logDespite these efforts, I found that all of the pw.x processes (2x8=16) spawn exclusively on the first computing node (node1), completely ignoring the second node (node2). node1
node2And modified the sub.sh command to: mpiexec -n 2 -machinefile nodefile python nvt_para.py >out.logHowever, only 8 pw.x processes appear on node1, while node2 remains inactive. Could you offer some guidance on how to run pw.x in parallel across multiple nodes under the control of SSCHA? Best regards! |
Beta Was this translation helpful? Give feedback.
Answered by
mesonepigreco
Aug 26, 2023
Replies: 3 comments 6 replies
-
|
Hi, you could try and use the cluster object and point it as localhost. Setting the cluster calculation is explained in the SSCHA documentation http://sscha.eu/Tutorials/tutorial_02_advanced_submission/, so you change the cluster to 127.0. 0.1 and that will loopback to itself. |
Beta Was this translation helpful? Give feedback.
4 replies
-
|
I agree with Diego. It would be better to submit each work as a job so that PBS automatically initializes the MPI workload to be correctly distributed across your nodes. There are two solutions:
cluster = sscha.Cluster.Cluster(hostname="localhost")
cluster.submit_command="sbatch"
cluster.submit_name="SBATCH"
cluster.terminal="/bin/bash"
cluster.v_nodes="-N "
cluster.v_cpu="-n "
cluster.v_account="-A "
cluster.v_time="--time="
cluster.v_memory="--mem="
cluster.v_partition="--partition="
cluster.v_qos = "--qos="
cluster.use_qos = FalseChange these variables in your submission script to match what you would use for PBS (for example cluster = sscha.Cluster.Cluster(hostname="localhost")
cluster.submit_command="qsub"
cluster.submit_name="PBS"
cluster.use_nodes = False
cluster.use_accout = False
cluster.use_time = False
# The others should be False by default
# And then add them as custom variables
#cluster.custom_params["-N"] = "noname"
#cluster.custom_params["-o" ] = "LogPBS"
#cluster.custom_params["-e"] = "LogPBS.err"
#cluster.custom_params["-m"] = "ae"
cluster.custom_params["-l nodes"] = "node1:ppn=16+node2:ppn=16"The problem with the lines I commented on is that they create a submission script like this: #PBS -N="noname"
[...]Which is different from what you want. However, since these are optional keywords, you can usually avoid them. See if it works; if it does not, send the submission script it generates and what it is doing wrong. |
Beta Was this translation helpful? Give feedback.
Answer selected by
hellolori
-
|
Dear mesonepigreco and diegomartinez2, Thank you for your suggestion to involve the cluster object. (1) I have prepared a Python script called "run_cluster.py" (directly copied from the tutorial document). import sscha, sscha.Ensemble
# Import the two python scripts for the cluster and espresso configurations
import espresso_calculator
import cluster
# Generate an ensemble with 10 configurations
dyn = CC.Phonons.Phonons("start_sscha", 3)
ensemble = sscha.Ensemble.Ensemble(dyn, 300)
ensemble.generate(10)
# Get the espresso and cluster configurations
espresso_config = espresso_calculator.get_calculator()
cluster_config = cluster.configure_cluster()
# Compute the ensemble
ensemble.compute_ensemble(espresso_config, cluster=cluster_config)
# Save the ensemble (using population 2 to avoid overwriting the other one)
ensemble.save("data", 2)(2) I have also made modifications to the file "cluster.py", which is required by "run_cluster.py". The modified "cluster.py" is as follows: import cellconstructor as CC, cellconstructor.Phonons
import sscha
import sscha.Cluster
import sys, os
def configure_cluster(cluster_workdir = "/work/sscha/"):
cluster = sscha.Cluster.Cluster(hostname = "localhost")
cluster.submit_command="qsub"
cluster.submit_name="PBS"
cluster.use_nodes = False
cluster.use_time = False
cluster.use_memory = True
cluster.ram = 180000
cluster.use_partition = False
cluster.custom_params["-l nodes"] = "node1:ppn=16+node2:ppn=16"
# cluster.custom_params["-q "] = "SECTION1"
home_workdir=os.path.join("$HOME", cluster_workdir)
scratch_workdir = os.path.join("$HOME", cluster_workdir, "/scratch_workdir")
cluster.workdir = home_workdir
cluster.add_set_minus_x = True # Avoid the set -x
cluster.load_modules = f"""
export OMP_NUM_THREADS=1
mkdir -p {scratch_workdir}
"""
def cp_files(lbls):
extrain = f"cd {scratch_workdir}\n"
extraout = "sleep 1\n"
for lbl in lbls:
extrain += f"cp {home_workdir}/{lbl}.pwi {scratch_workdir}/\n"
extraout += f"mv {scratch_workdir}/{lbl}.pwo {home_workdir}/\n"
print('hello')
return extrain, extraout
# Add the possibility to copy the input files
cluster.additional_script_parameters = cp_files
# Force to open a shell when executing ssh commands
# (Otherwise the cluster will not load the module environment)
cluster.use_active_shell = True
cluster.setup_workdir()
# Check the communication
if not cluster.CheckCommunication():
raise ValueError("Impossible to connect to the cluster.")
return cluster(3) Additionally, I have prepared a Python script called "espresso_calculator.py", which is also required by "run_cluster.py". #import cellconstructor as CC
import cellconstructor as CC, cellconstructor.Phonons
from cellconstructor.calculators import Espresso
import sys, os
import numpy as np
import cellconstructor.Structure, cellconstructor.calculators
# Import the DFT calculator
import cellconstructor.calculators
def get_calculator():
pseudopotentials = {
"H" : "H.pbe-rrkjus_psl.1.0.0.UPF",
"S" : "s_pbe_v1.4.uspp.F.UPF"
}
input_params = {
"control" : {
"tprnfor" : True,
"tstress" : True,
"disk_io" : "none",
'pseudo_dir' : '.',
'outdir' : './outdir',
'wfcdir' : './wfcdir',
'tprnfor' : True,
'tstress' : True
},
"system" : {
"degauss" : 0.03,
"smearing" : "mv",
"ecutwfc" : 35,
"ecutrho" : 350,
"occupations" : "smearing"
},
"electrons" : {
"conv_thr" : 1e-10
}
}
kpts = (4,4,4)
koffset = (1, 1, 1)
command = '~/software/openmpi/4.0.5/bin/mpiexec -n 4 ~/software/QE/6.6_openmpi/bin/pw.x -npool 2 -i PREFIX.pwi > PREFIX.pwo'
# calc = Espresso(pseudopotentials = pseudopotentials,
# input_data = input_params,
# kpts = kpts,
# koffset = koffset)
calc = CC.calculators.Espresso(input_data = input_params,
pseudopotentials = pseudopotentials,
command = command,
kpts = kpts,
koffset = koffset)
return calc(4) Finally, when I ran the "run_cluster.py" on the logging node, I encountered the following issue: Can you please advise me on this matter? Additionally, I need to submit the job to a specific partition named "SECTION1", which corresponds to a setting in the "sub.sh" file: How can I translate this setting for SSCHA? I note that the current setting cluster.custom_params["-q "] = "SECTION1" is incorrect, as it creates a submission script like the following: Best regards! |
Beta Was this translation helpful? Give feedback.
-
|
Hi,
The partition problem is easy as you may set: cluster.use_partition=True
cluster.v_partition = "-q "
cluster.partition_name = "SECTION1"Regarding the You can also remove the lines: def cp_files(lbls):
extrain = f"cd {scratch_workdir}\n"
extraout = "sleep 1\n"
for lbl in lbls:
extrain += f"cp {home_workdir}/{lbl}.pwi {scratch_workdir}/\n"
extraout += f"mv {scratch_workdir}/{lbl}.pwo {home_workdir}/\n"
print('hello')
return extrain, extraout
# Add the possibility to copy the input files
cluster.additional_script_parameters = cp_files
# Force to open a shell when executing ssh commands
# (Otherwise the cluster will not load the module environment)
cluster.use_active_shell = TrueThese are used to exploit a local scratch in the cluster, which I do not think is your case. Let us know if this is enough. |
Beta Was this translation helpful? Give feedback.
-
|
Hello mesonepigreco, I've noticed that I can successfully set up a virtual environment using Python 3.7 for SSCHA. However, I encounter problems when attempting to do so with a Python version higher than 3.7. I'm curious if the procedure I'm following is accurate. Let's consider the instance of Python 3.10. To proceed: conda create --name sscha_py3.10 -c conda-forge python=3.10(2) activate the environment conda activate sscha_py3.10(3) install the following package pip install -i https://pypi.mirrors.ustc.edu.cn/simple ase spglib quippy-aseUp to now, everything is OK. However, an error occurs when I try to install cellconstructor and python-sscha (4)install cellconstructor and python-sscha pip install -i https://pypi.mirrors.ustc.edu.cn/simple cellconstructor python-sschaThe whole output of the above command reads: |
Beta Was this translation helpful? Give feedback.
-
|
The line "skip targets (VXE FMA4 NEON XOP VSX2 VSX4 VX VXE2 AVX512_SKX VSX3 ASIMD VSX AVX512F) not part of baseline or dispatch-able features" suggest that your machine is not supported. Your system compiler does not support AVX-512. Maybe a solution is in the OpenBLAS FAQ. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
https://github.com/xianyi/OpenBLAS/wiki/faq#binutils |
Beta Was this translation helpful? Give feedback.
-
|
As a curiosity, this is GPT3.5 answer to the error The compilation error "gcc: error: unrecognized command line option '-mavx512vl'" means that the compiler is encountering an option ("-mavx512vl") that it does not recognize. This could be due to a few reasons:
To resolve this error, you can try the following steps:
It's worth noting that the specific error message you provided ("gcc: error: unrecognized command line option '-mavx512vl'") is a common error message that can occur in various situations, not just related to AVX-512 instructions. So, the steps provided above are general troubleshooting steps to resolve unrecognized option errors in GCC. Sources: |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
The response from GPT-3.5 is indeed helpful. The matter concerning the "unrecognized command line option '-mavx512vl'" can be resolved by updating the version of gcc. Consequently, SSCHA can be installed correctly using Python 3.10. By employing Python 3.10 to execute SSCHA, the problem related to the "get_native_id error" can also be addressed. With these steps completed, calculations for various structures can be submitted as separate jobs to the PBS queue. It seems that my current issue has been successfully resolved. However, I intend to explore an alternative approach outlined in the "3.1 Manual submission" section. I believe this method would be more convenient for me. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I agree with Diego. It would be better to submit each work as a job so that PBS automatically initializes the MPI workload to be correctly distributed across your nodes.
The problem is related to the nested calls of MPI, which confuses the system.
There are two solutions:
However, by default, the cluster module works with SLURM. To have it work with PBS, you can override the specific SLURM commands with the equivalent of PBS. For example, here are the total command configured for SLURM: