Pipeline job ValidateCallsetTask failing with file already exists error

[mehshad]

I keep getting this error when I am running data load task for my project. I have tried with different datasets, references (both 37 and 38) and projects. I tried deleting the matrix table folder, but still the error persists

Hail: INFO: wrote matrix table with 7279144 rows and 3 columns in 6 partitions to /tmp/8e118849-1968-4a96-a831-ce16f7e38269.mt
ERROR: [pid 1] Worker Worker(salt=9444364660, workers=1, host=pipeline-runner-api-7c454844b6-g5pck, username=root, pid=1) failed ValidateCallsetTask(reference_genome=GRCh38, dataset_type=SNV_INDEL, run_id=20251123-081803-717131, sample_type=WGS, callset_path=/var/seqr/seqr-loading-temp/Patient-450.hard-filtered.vcf.gz, project_guids=["R0004_novel_test_batch"], skip_check_sex_and_relatedness=False, skip_expect_tdr_metrics=True, skip_validation=False, is_new_gcnv_joint_call=False)
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/luigi/worker.py", line 210, in run
new_deps = self._run_get_new_deps()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/luigi/worker.py", line 138, in _run_get_new_deps
task_gen = self.task.run()
^^^^^^^^^^^^^^^
File "/v03_pipeline/lib/tasks/base/base_update.py", line 20, in run
write(ht, self.output().path)
File "/v03_pipeline/lib/misc/io.py", line 332, in write
return t.write(destination_path, overwrite=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 2, in write
File "/usr/local/lib/python3.11/dist-packages/hail/typecheck/check.py", line 585, in wrapper
return original_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/hail/matrixtable.py", line 2810, in write
Env.backend().execute(ir.MatrixWrite(self._mir, writer))
File "/usr/local/lib/python3.11/dist-packages/hail/backend/spark_backend.py", line 217, in execute
raise err
File "/usr/local/lib/python3.11/dist-packages/hail/backend/spark_backend.py", line 209, in execute
return super().execute(ir, timed)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/hail/backend/backend.py", line 181, in execute
raise e.maybe_user_error(ir) from None
File "/usr/local/lib/python3.11/dist-packages/hail/backend/backend.py", line 179, in execute
result, timings = self._rpc(ActionTag.EXECUTE, payload)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/hail/backend/py4j_backend.py", line 221, in _rpc
raise fatal_error_from_java_error_triplet(
hail.utils.java.FatalError: HailException: RelationalSetup.writeMetadata: file already exists: /var/seqr/seqr-loading-temp/GRCh38/SNV_INDEL/imported_callsets/b0579a8c111de07ee25141e6b1a81c0c3e1315bf44b8d6592586c366330350d2.mt/rows

Java stack trace:
is.hail.utils.HailException: RelationalSetup.writeMetadata: file already exists: /var/seqr/seqr-loading-temp/GRCh38/SNV_INDEL/imported_callsets/b0579a8c111de07ee25141e6b1a81c0c3e1315bf44b8d6592586c366330350d2.mt/rows
at __C24120Compiled.__m24124begin_group_0_region6_11(Emit.scala)
at __C24120Compiled.__m24124begin_group_0(Emit.scala)
at __C24120Compiled.__m24122split_Block(Emit.scala)
at __C24120Compiled.apply(Emit.scala)
at is.hail.expr.ir.CompileAndEvaluate$.$anonfun$_apply$4(CompileAndEvaluate.scala:60)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:84)
at is.hail.expr.ir.CompileAndEvaluate$.$anonfun$_apply$2(CompileAndEvaluate.scala:60)
at is.hail.expr.ir.CompileAndEvaluate$.$anonfun$_apply$2$adapted(CompileAndEvaluate.scala:58)
at is.hail.backend.ExecuteContext.$anonfun$scopedExecution$1(ExecuteContext.scala:144)
at is.hail.utils.package$.using(package.scala:673)
at is.hail.backend.ExecuteContext.scopedExecution(ExecuteContext.scala:144)
at is.hail.expr.ir.CompileAndEvaluate$._apply(CompileAndEvaluate.scala:58)
at is.hail.expr.ir.CompileAndEvaluate$.$anonfun$apply$1(CompileAndEvaluate.scala:17)
at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:84)
at is.hail.expr.ir.CompileAndEvaluate$.apply(CompileAndEvaluate.scala:17)
at is.hail.expr.ir.TableWriter.apply(TableWriter.scala:51)
at is.hail.expr.ir.Interpret$.run(Interpret.scala:922)
at is.hail.expr.ir.Interpret$.alreadyLowered(Interpret.scala:66)
at is.hail.expr.ir.LowerOrInterpretNonCompilable$.evaluate$1(LowerOrInterpretNonCompilable.scala:20)
at is.hail.expr.ir.LowerOrInterpretNonCompilable$.rewrite$1(LowerOrInterpretNonCompilable.scala:59)
at is.hail.expr.ir.LowerOrInterpretNonCompilable$.apply(LowerOrInterpretNonCompilable.scala:64)
at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.transform(LoweringPass.scala:83)
at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$3(LoweringPass.scala:32)
at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:84)
at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$1(LoweringPass.scala:32)
at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:84)
at is.hail.expr.ir.lowering.LoweringPass.apply(LoweringPass.scala:30)
at is.hail.expr.ir.lowering.LoweringPass.apply$(LoweringPass.scala:29)
at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.apply(LoweringPass.scala:78)
at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1(LoweringPipeline.scala:21)
at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1$adapted(LoweringPipeline.scala:19)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:19)
at is.hail.expr.ir.CompileAndEvaluate$._apply(CompileAndEvaluate.scala:45)
at is.hail.backend.spark.SparkBackend._execute(SparkBackend.scala:578)
at is.hail.backend.spark.SparkBackend.$anonfun$execute$4(SparkBackend.scala:614)
at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:84)
at is.hail.backend.spark.SparkBackend.$anonfun$execute$3(SparkBackend.scala:609)
at is.hail.backend.spark.SparkBackend.$anonfun$execute$3$adapted(SparkBackend.scala:608)
at is.hail.backend.ExecuteContext$.$anonfun$scoped$3(ExecuteContext.scala:78)
at is.hail.utils.package$.using(package.scala:673)
at is.hail.backend.ExecuteContext$.$anonfun$scoped$2(ExecuteContext.scala:78)
at is.hail.utils.package$.using(package.scala:673)
at is.hail.annotations.RegionPool$.scoped(RegionPool.scala:13)
at is.hail.backend.ExecuteContext$.scoped(ExecuteContext.scala:65)
at is.hail.backend.spark.SparkBackend.$anonfun$withExecuteContext$2(SparkBackend.scala:411)
at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:55)
at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:62)
at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:397)
at is.hail.backend.spark.SparkBackend.execute(SparkBackend.scala:608)
at is.hail.backend.BackendHttpHandler.handle(BackendServer.scala:88)
at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:848)
at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:817)
at jdk.httpserver/sun.net.httpserver.ServerImpl$DefaultExecutor.execute(ServerImpl.java:201)
at jdk.httpserver/sun.net.httpserver.ServerImpl$Dispatcher.handle(ServerImpl.java:560)
at jdk.httpserver/sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:526)
at java.base/java.lang.Thread.run(Thread.java:829)

[bpblanken]

Hi @mehshad,

Would you mind sharing:

1. What filesystem you're running the pipeline on?
2. The directory permissions of the imported_callsets folder?
3. What happens on your filesystem when you do something like:

mt = hl.read_matrix_table('/tmp/8e118849-1968-4a96-a831-ce16f7e38269.mt') # the checkpointed matrix table from your stacktrace
mt.write('/tmp/test_matrix_table.mt')
mt.write('/tmp/test_matrix_table.mt', overwrite=True)

to test the overwrite method in isolation?

We explicitly use the overwrite=True flag to the .write() method. It's possible something is failing within that method, as reported in this issue.

[mehshad]

@bpblanken

1. Local filesystem mounted into Kubernetes via a PersistentVolumeClaim (PVC)
2. The pipeline-runner pod runs as root (no restrictive securityContext set), so anything it creates under /var/seqr owned by root.
3. It is able to overwrite

python3 - <<'PY'
import hail as hl

mt = hl.read_matrix_table('/var/seqr/seqr-loading-temp/GRCh37/SNV_INDEL/imported_callsets/db2a6ad8a2636a4d2c894b5e06d3e322f77c2f7420e2dcda15cd7258559842e9.mt') # the checkpointed matrix table from your stacktrace
print("Read OK")
mt.write('/var/seqr/seqr-loading-temp/GRCh37/SNV_INDEL/imported_callsets/test_matrix_table.mt')
print("First write OK")
mt.write('/var/seqr/seqr-loading-temp/GRCh37/SNV_INDEL/imported_callsets/test_matrix_table.mt', overwrite=True)
print("Overwrite write OK")
PY
Initializing Hail with default parameters...
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Running on Apache Spark version 3.5.5
SparkUI available at http://pipeline-runner-api-7c454844b6-g5pck:4041
Welcome to
__ __ <>__
/ // /__ __/ /
/ __ / _ `/ / /
// //_,/// version 0.2.133-4c60fddb171a
LOGGING: writing to /hail-20251130-0617-0.2.133-4c60fddb171a.log
Read OK
SLF4J: Failed to load class "org.slf4j.impl.StaticMDCBinder".
SLF4J: Defaulting to no-operation MDCAdapter implementation.
SLF4J: See http://www.slf4j.org/codes.html#no_static_mdc_binder for further details.
2025-11-30 06:17:32.192 Hail: INFO: wrote matrix table with 15269325 rows and 24 columns in 30 partitions to /var/seqr/seqr-loading-temp/GRCh37/SNV_INDEL/imported_callsets/test_matrix_table.mt
First write OK
2025-11-30 06:17:57.844 Hail: INFO: wrote matrix table with 15269325 rows and 24 columns in 30 partitions to /var/seqr/seqr-loading-temp/GRCh37/SNV_INDEL/imported_callsets/test_matrix_table.mt
Overwrite write OK

[bpblanken]

Interesting; so it does appear overwriting works in your environment.

My next suggestion is to try running the first few pipeline steps manually:

import luigi

from v03_pipeline.lib.core import DatasetType, ReferenceGenome, SampleType
from v03_pipeline.lib.tasks.write_imported_callset import WriteImportedCallsetTask

luigi.build(
        [
            WriteImportedCallsetTask(
                reference_genome=ReferenceGenome.GRCh38,
                dataset_type=DatasetType.SNV_INDEL,
                sample_type=SampleType.WGS,
                callset_path=/var/seqr/seqr-loading-temp/Patient-450.hard-filtered.vcf.gz, 
                project_guids=["R0004_novel_test_batch"],
                run_id='manual_run_0',
            )
        ],
    )

and

import luigi

from v03_pipeline.lib.core import DatasetType, ReferenceGenome, SampleType
from v03_pipeline.lib.tasks.validate_callset import ValidateCallsetTask

luigi.build(
        [
            ValidateCallsetTask(
                reference_genome=ReferenceGenome.GRCh38,
                dataset_type=DatasetType.SNV_INDEL,
                sample_type=SampleType.WGS,
                callset_path=/var/seqr/seqr-loading-temp/Patient-450.hard-filtered.vcf.gz, 
                project_guids=["R0004_novel_test_batch"],
                run_id='manual_run_0',
            )
        ],
    )

From what you've been seeing, I think we expect the first of these to succeed and write a matrix table, and the second to fail (Validating a callset runs in place and updates a "global" on the existing table, marking it as "validated"). Hopefully running the tasks in isolation will help illuminate why.

[mehshad]

You're right. The second task failed with the exact same error we've seen before

===== Luigi Execution Summary =====

Scheduled 4 tasks of which:

• 3 complete ones were encountered:
  • 1 UpdatedReferenceDatasetTask(reference_genome=GRCh38, dataset_type=SNV_INDEL, reference_dataset=gnomad_coding_and_noncoding)
  • 1 VCFFileTask(pathname=/var/seqr/seqr-loading-temp/Patient-450.hard-filtered.vcf.gz)
  • 1 WriteImportedCallsetTask(...)
• 1 failed:
  • 1 ValidateCallsetTask(...)

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

Java stack trace:
is.hail.utils.HailException: RelationalSetup.writeMetadata: file already exists: /var/seqr/seqr-loading-temp/GRCh38/SNV_INDEL/imported_callsets/b0579a8c111de07ee25141e6b1a81c0c3e1315bf44b8d6592586c366330350d2.mt/rows

[bpblanken]

A few more debugging ideas:

• Can you inspect the existing matrix table to make sure it is valid?

import hail
hl.read_matrix_table(' /var/seqr/seqr-loading-temp/GRCh38/SNV_INDEL/imported_callsets/b0579a8c111de07ee25141e6b1a81c0c3e1315bf44b8d6592586c366330350d2.mt').globals.show()

• You can try renaming your callset Patient-450.hard-filtered.vcf.gz -> Patient-450.hard-filtered.2.vcf.gz or something like that to just verify that there's nothing pathological with the directory as currently named (b0579a8c111de07ee25141e6b1a81c0c3e1315bf44b8d6592586c366330350d2 is a hash that includes the VCF path).

[mehshad]

It worked with
/var/seqr/seqr-loading-temp/GRCh37/SNV_INDEL/imported_callsets/test_matrix_table.mt
and
/var/seqr/seqr-loading-temp/GRCh37/SNV_INDEL/imported_callsets/db2a6ad8a2636a4d2c894b5e06d3e322f77c2f7420e2dcda15cd7258559842e9.mt

Also with some other old matrix tables. It is all working.

For this particular vcf, I get the following error

Error summary: HailException: MatrixTable is corrupted: /var/seqr/seqr-loading-temp/GRCh38/SNV_INDEL/imported_callsets/b0579a8c111de07ee25141e6b1a81c0c3e1315bf44b8d6592586c366330350d2.mt/metadata.json.gz is missing.

Regardless, the write error comes consistently for all files when running pipeline task from seqr

[bpblanken]

Gotcha. A few more suggestions.

Can you try cloning the repo and running this test (not all elements of the tests are included in the Docker image):

export ACCESS_PRIVATE_REFERENCE_DATASETS=1
export PYSPARK_SUBMIT_ARGS='--driver-memory 8G pyspark-shell' 
nosetests v03_pipeline/lib/tasks/update_variant_annotations_table_with_new_samples_test.py:UpdateVariantAnnotationsTableWithNewSamplesTaskTest.test_multiple_update_vat

If that doesn't work, you'll need to start adding hail checkpoint calls, potentially here and elsewhere. Hail pipelines tend to obfuscate the actual cause of the error and only output the final failure; checkpoints can help illuminate exactly where the issue is.

[mehshad]

Sorry for not getting back earlier. I had to remount the filesystem and it corrupted the pod executions. Good news is that these errors have stopped coming now and I am reaching WriteNewEntriesParquetTask where it fails with

2025-12-24 10:28:21 - java_gateway - ERROR - Exception while sending command.46]
Traceback (most recent call last):
File "/.venv/lib/python3.11/site-packages/py4j/clientserver.py", line 516, in send_command
raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/.venv/lib/python3.11/site-packages/py4j/java_gateway.py", line 1038, in send_command
response = connection.send_command(command)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.venv/lib/python3.11/site-packages/py4j/clientserver.py", line 539, in send_command
raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while sending or receiving

[bpblanken]

Hi @mehshad,

This looks like a Spark Driver dying from underneath Hail. One thing you can try is giving it more memory.