-------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[10158,1],105]) is on host: s35 Process 2 ([[10158,1],0]) is on host: s40 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s35:00000] *** An error occurred in MPI_Send [s35:00000] *** reported by process [665714689,105] [s35:00000] *** on communicator MPI_COMM_WORLD [s35:00000] *** MPI_ERR_INTERN: internal error [s35:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s35:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[40998,1],80]) is on host: s35 Process 2 ([[40998,1],0]) is on host: s40 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s35:00000] *** An error occurred in MPI_Send [s35:00000] *** reported by process [2686844929,80] [s35:00000] *** on communicator MPI_COMM_WORLD [s35:00000] *** MPI_ERR_INTERN: internal error [s35:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s35:00000] *** and MPI will try to terminate your MPI job as well) 48 more processes have sent help message help-mca-bml-r2.txt / unreachable proc 48 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal 18 more processes have sent help message help-mca-bml-r2.txt / unreachable proc 18 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal 16 more processes have sent help message help-mca-bml-r2.txt / unreachable proc 15 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal srun: error: s40: task 84: Exited with exit code 17 srun: error: s40: task 75: Exited with exit code 17 -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[31587,1],105]) is on host: s35 Process 2 ([[31587,1],0]) is on host: s40 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s35:00000] *** An error occurred in MPI_Send [s35:00000] *** reported by process [2070085633,127] [s35:00000] *** on communicator MPI_COMM_WORLD [s35:00000] *** MPI_ERR_INTERN: internal error [s35:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s35:00000] *** and MPI will try to terminate your MPI job as well) srun: error: s40: task 81: Exited with exit code 17 -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[39684,1],127]) is on host: s40 Process 2 ([[39684,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [2600730625,127] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[60229,1],85]) is on host: s40 Process 2 ([[60229,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [3947167745,85] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[35549,1],72]) is on host: s40 Process 2 ([[35549,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [2329739265,72] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[49780,1],102]) is on host: s40 Process 2 ([[49780,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [3262382081,102] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[22832,1],108]) is on host: s40 Process 2 ([[22832,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [1496317953,108] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[52605,1],77]) is on host: s40 Process 2 ([[52605,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [3447521281,77] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[10464,1],67]) is on host: s40 Process 2 ([[10464,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [685768705,67] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[29579,1],119]) is on host: s40 Process 2 ([[29579,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [1938489345,119] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[41022,1],120]) is on host: s40 Process 2 ([[41022,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [2688417793,120] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[3931,1],92]) is on host: s40 Process 2 ([[3931,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [257622017,92] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[22487,1],98]) is on host: s40 Process 2 ([[22487,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [1473708033,98] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[32247,1],75]) is on host: s40 Process 2 ([[32247,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [2113339393,75] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[34576,1],69]) is on host: s40 Process 2 ([[34576,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [2265972737,69] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[19037,1],121]) is on host: s40 Process 2 ([[19037,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [1247608833,121] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[35825,1],113]) is on host: s40 Process 2 ([[35825,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [2347827201,113] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[25254,1],125]) is on host: s40 Process 2 ([[25254,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [1655046145,125] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[30680,1],119]) is on host: s40 Process 2 ([[30680,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [2010644481,119] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[53670,1],126]) is on host: s40 Process 2 ([[53670,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [3517317121,126] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[54284,1],80]) is on host: s40 Process 2 ([[54284,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [3557556225,80] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[30300,1],79]) is on host: s40 Process 2 ([[30300,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[32400,1],107]) is on host: s40 Process 2 ([[32400,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [2123366401,107] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [1985740801,79] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[30268,1],82]) is on host: s40 Process 2 ([[30268,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [1983643649,82] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[35280,1],104]) is on host: s40 Process 2 ([[35280,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [2312110081,104] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[40202,1],123]) is on host: s40 Process 2 ([[40202,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [2634678273,123] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[10060,1],98]) is on host: s40 Process 2 ([[10060,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[11747,1],124]) is on host: s40 Process 2 ([[11747,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [769851393,124] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[47462,1],114]) is on host: s40 Process 2 ([[47462,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [3110469633,114] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [659292161,98] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[42875,1],114]) is on host: s40 Process 2 ([[42875,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [2809856001,114] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[3268,1],85]) is on host: s40 Process 2 ([[3268,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [214171649,85] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[35520,1],74]) is on host: s40 Process 2 ([[35520,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [2327838721,74] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[53334,1],72]) is on host: s40 Process 2 ([[53334,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [3495297025,72] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[25319,1],95]) is on host: s40 Process 2 ([[25319,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [1659305985,95] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[19866,1],113]) is on host: s40 Process 2 ([[19866,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [1301938177,113] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[50239,1],91]) is on host: s40 Process 2 ([[50239,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [3292463105,91] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[44757,1],78]) is on host: s40 Process 2 ([[44757,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [2933194753,78] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[18192,1],127]) is on host: s40 Process 2 ([[18192,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [1192230913,127] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[44301,1],66]) is on host: s40 Process 2 ([[44301,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [2903310337,66] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[52949,1],90]) is on host: s40 Process 2 ([[52949,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [3470065665,90] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) 8 more processes have sent help message help-mca-bml-r2.txt / unreachable proc 8 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal 2 more processes have sent help message help-mca-bml-r2.txt / unreachable proc 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal 2 more processes have sent help message help-mca-bml-r2.txt / unreachable proc 2 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[64988,1],83]) is on host: s40 Process 2 ([[64988,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [4259053569,83] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[26206,1],81]) is on host: s40 Process 2 ([[26206,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [1717436417,81] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) 2 more processes have sent help message help-mca-bml-r2.txt / unreachable proc 2 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[46611,1],65]) is on host: s40 Process 2 ([[46611,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [3054698497,65] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[54427,1],107]) is on host: s40 Process 2 ([[54427,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [3566927873,107] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[44483,1],102]) is on host: s40 Process 2 ([[44483,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [2915237889,102] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal 2 more processes have sent help message help-mca-bml-r2.txt / unreachable proc 2 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[57574,1],93]) is on host: s40 Process 2 ([[57574,1],0]) is on host: s35 BTLs attempted: self sm Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [s40:00000] *** An error occurred in MPI_Send [s40:00000] *** reported by process [3773169665,93] [s40:00000] *** on communicator MPI_COMM_WORLD [s40:00000] *** MPI_ERR_INTERN: internal error [s40:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [s40:00000] *** and MPI will try to terminate your MPI job as well) 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc srun: error: s35: tasks 3,10: Exited with exit code 17 srun: Job step aborted: Waiting up to 32 seconds for job step to finish. [2026-10-22T16:24:35.754] error: *** JOB 9917219 ON s35 CANCELLED AT 2026-10-22T16:24:35 DUE to SIGNAL Terminated *** srun: error: s35: tasks 23,34,36: Exited with exit code 1 srun: error: s35: tasks 0,2,5,7-9,11-16,18,20-22,24-28,30,32,37-38,42-43,45,51-56,58-59,61: Exited with exit code 1 srun: error: s35: task 49: Exited with exit code 1 srun: error: s35: task 62: Exited with exit code 1 -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886116@0,0] on node s40 Remote daemon: [prterun-s40-1886116@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886117@0,0] on node s40 Remote daemon: [prterun-s40-1886117@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886118@0,0] on node s40 Remote daemon: [prterun-s40-1886118@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886123@0,0] on node s40 Remote daemon: [prterun-s40-1886123@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886128@0,0] on node s40 Remote daemon: [prterun-s40-1886128@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886132@0,0] on node s40 Remote daemon: [prterun-s40-1886132@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886137@0,0] on node s40 Remote daemon: [prterun-s40-1886137@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886140@0,0] on node s40 Remote daemon: [prterun-s40-1886140@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886142@0,0] on node s40 Remote daemon: [prterun-s40-1886142@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886144@0,0] on node s40 Remote daemon: [prterun-s40-1886144@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886145@0,0] on node s40 Remote daemon: [prterun-s40-1886145@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886149@0,0] on node s40 Remote daemon: [prterun-s40-1886149@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886154@0,0] on node s40 Remote daemon: [prterun-s40-1886154@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886156@0,0] on node s40 Remote daemon: [prterun-s40-1886156@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886158@0,0] on node s40 Remote daemon: [prterun-s40-1886158@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886165@0,0] on node s40 Remote daemon: [prterun-s40-1886165@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886176@0,0] on node s40 Remote daemon: [prterun-s40-1886176@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886178@0,0] on node s40 Remote daemon: [prterun-s40-1886178@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886166@0,0] on node s40 Remote daemon: [prterun-s40-1886166@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- srun: error: s35: task 47: Exited with exit code 1 -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886138@0,0] on node s40 Remote daemon: [prterun-s40-1886138@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- -------------------------------------------------------------------------- PRTE has lost communication with a remote daemon. HNP daemon : [prterun-s40-1886134@0,0] on node s40 Remote daemon: [prterun-s40-1886134@0,1] on node s35 This is usually due to either a failure of the TCP network connection to the node, or possibly an internal failure of the daemon itself. We cannot recover from this failure, and therefore will terminate the job. -------------------------------------------------------------------------- srun: error: Timed out waiting for job step to complete [2026-10-22T16:25:06.172] error: common_file_write_content: unable to open '/sys/fs/cgroup/system.slice/slurmstepd.scope/s8FXM5JM8YZ100/step_batch/user/cgroup.freeze' for writing: Permission denied