I'm training a YOLOX model for object detection using a dataset that includes:
- Annotated images (positive samples)
- Unannotated images (negative samples, i.e., no bounding boxes)
When I include negative images in the training dataset (images with no annotations), I consistently encounter the following CUDA error:
Assertion input_val >= zero && input_val <= one failed
From this trace:
../aten/src/ATen/native/cuda/Loss.cu:94: operator(): block: [0,0,0], thread: [0,0,0]
Assertion input_val >= zero && input_val <= one failed.
...
terminate called after throwing an instance of 'c10::DistBackendError'
what(): Process group watchdog thread terminated with exception: CUDA error: device-side assert triggered
The unannotated images in the .json file are only in the images section; they do not appear in the annotations section.
Is happening in the epoch 3 or later.
How can I safely include negative samples (images without annotations) in YOLOX training without triggering the CUDA assert?
I'm training a YOLOX model for object detection using a dataset that includes:
When I include negative images in the training dataset (images with no annotations), I consistently encounter the following CUDA error:
Assertion
input_val >= zero && input_val <= onefailedFrom this trace:
../aten/src/ATen/native/cuda/Loss.cu:94: operator(): block: [0,0,0], thread: [0,0,0]
Assertion
input_val >= zero && input_val <= onefailed....
terminate called after throwing an instance of 'c10::DistBackendError'
what(): Process group watchdog thread terminated with exception: CUDA error: device-side assert triggered
The unannotated images in the .json file are only in the images section; they do not appear in the annotations section.
Is happening in the epoch 3 or later.
How can I safely include negative samples (images without annotations) in YOLOX training without triggering the CUDA assert?