Skip to content

OOM error #151

@rvandewater

Description

@rvandewater

Task seems to have subjects. Tested on Slurm allocated machine with about 600 GB Ram and 8 cores. Ran with MEDS-DEV (so older version of MEDS-Tab presumably)

tqdm: false                                                                                                                                                                                                                                               16:19:32 [130/1884]
worker: 0                                                                                                                                                                                                                                                                    
loguru_init: false                                                 
log_dir: ${output_dir}/.logs/                                      
cache_dir: ${output_dir}/.cache                                                                                                                                                                                                                                              
tabularization:                                                    
  filtered_code_metadata_fp: ${output_dir}/metadata/codes.parquet                                                                                                                                                                                                            
  allowed_codes: null                                                                                                                 
  min_code_inclusion_count: 10                                                                                                                                                                                                                                               
  min_code_inclusion_frequency: null  
  min_correlation: null                                                                                                                                                                                                                                                      
  max_by_correlation: null        
  max_included_codes: 100                                                                                                                                                                                                                                                    
  do_compress: false                                                                                                                                                                                                                                                         
  window_sizes:                                                                                                                                                                                                                                                              
  - 7d                                                                                                                                                                                                                                                                       
  - 30d                                                                                                                                                                                                                                                                      
  aggs:                                                            
  - code/count                                                                                                                                                                                                                                                               
  - value/sum                                                                                                                         
  _resolved_codes: ${filter_to_codes:${tabularization.filtered_code_metadata_fp},${tabularization.allowed_codes},${tabularization.min_code_inclusion_count},${tabularization.min_code_inclusion_frequency},${tabularization.max_included_codes}}                             
input_code_metadata_fp: ${output_dir}/metadata/codes.parquet                                                                                                                                                                                                                 
output_tabularized_dir: ${output_dir}/tabularize                                                                                                                                                                                                                             
input_label_dir: /sc/home/robin.vandewater/datasets/meds/HIRID/labels/mortality/in_icu/first_24h                                                                                                                                                                             
name: tabularization                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                             
Paths: (checkbox indicates if it exists)                                                                                                                                                                                                                                     
  - input_code_metadata_fp: ✅ /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/metadata/codes.parquet                                                        
  - input_dir: ✅ /sc/home/robin.vandewater/datasets/meds/HIRID/data                                                                                                                                                                                                         
  - tabularization.filtered_code_metadata_fp: ✅ /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/metadata/codes.parquet                                                                                               
/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/MEDS_tabular_automl/describe_codes.py:63: PerformanceWarning: Determining the column names of a LazyFrame requires resolving its schema, whi
ch is a potentially expensive operation. Use `LazyFrame.collect_schema().names()` to get the column names without this warning.                                                                                                                                              
  if not df.columns == ["code", "count"]:                                                                                                                                                                                                                                    
2025-10-19 16:17:41.269 | INFO     | MEDS_tabular_automl.generate_ts_features:summarize_dynamic_measurements:112 - Generating Sparse matrix for Time Series Features                                                                                                         
2025-10-19 16:17:50.398 | INFO     | MEDS_tabular_automl.generate_summarized_reps:generate_summary:337 - Generating aggregation code/count for window_size 30d, with 653 columns.                                                                                            
2025-10-19 16:17:52.898 | INFO     | MEDS_tabular_automl.generate_summarized_reps:compute_agg:284 - Step 2: computing rolling windows and aggregating.                                                              
Traceback (most recent call last):                                                                                                                                                                                                                                           
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/bin/meds-tab-tabularize-time-series", line 8, in <module>                                                                                                        
    sys.exit(main())                                                                                                                                                                                                                                                         
             ^^^^^^                                                                                                                                                                                                                                                          
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/hydra/main.py", line 94, in decorated_main                                                                                          
    _run_hydra(                                                                                                                                                                                                                                                              
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra                                                                                  
    _run_app(                                                                                                                                                                                                                                                                
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/hydra/_internal/utils.py", line 465, in _run_app                                                                                    
    run_and_report(                                                                                                                                                                                                                                                          
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/hydra/_internal/utils.py", line 223, in run_and_report                                                                              
    raise ex                                                                                                                                                                                                                                                                 
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/hydra/_internal/utils.py", line 220, in run_and_report                    
    return func()                                                                                                                                                                                                                                                            
           ^^^^^^                                                                                                                                                                                                                                                            
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/hydra/_internal/utils.py", line 466, in <lambda>                                                                                    
    lambda: hydra.multirun(                                                                                                                                                                                                                                                  
            ^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                  
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/hydra/_internal/hydra.py", line 162, in multirun                                                                                    
    ret = sweeper.sweep(arguments=task_overrides)                                                                                                                                                                                                                            
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                            
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/hydra/_internal/core_plugins/basic_sweeper.py", line 177, in sweep        
    results = self.launcher.launch(batch, initial_job_idx=initial_job_idx)                                                                                                                                                                                                   
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                   
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/hydra_plugins/hydra_joblib_launcher/joblib_launcher.py", line 45, in launch                
    return _core.launch(                                                                                                                                                                                                                                                     
           ^^^^^^^^^^^^^                                                                                                                                                                                                                                                     
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/hydra_plugins/hydra_joblib_launcher/_core.py", line 103, in launch                                                                  
    runs = Parallel(**joblib_cfg)(                                                                                                                                                                                                                                           
           ^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                           
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/joblib/parallel.py", line 2072, in __call__        
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/joblib/parallel.py", line 2072, in __call__                                                                                         
    return output if self.return_generator else list(output)       
                                                ^^^^^^^^^^^^       
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/joblib/parallel.py", line 1682, in _get_outputs                                                                                     
    yield from self._retrieve()                                    
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/joblib/parallel.py", line 1784, in _retrieve                                                                                        
    self._raise_error_fast()                                                                                                          
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/joblib/parallel.py", line 1859, in _raise_error_fast                                                                                
    error_job.get_result(self.timeout)
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/joblib/parallel.py", line 758, in get_result                                                                                        
    return self._return_or_raise()
           ^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                           
  File "/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/.venv/lib/python3.11/site-packages/joblib/parallel.py", line 773, in _return_or_raise                                                                                  
    raise self._result                                                                                                                                                                                                                                                       
joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to k
ill the worker.                                                                                                                                                                                                                                                              
                                                                   
The exit codes of the workers are {SIGKILL(-9)}                                                                                                                                                                                                                              
Detailed tracebacks of the workers should have been printed to stderr in the executor process if faulthandler was not disabled.       
                                                                                                                                                                                                                                                                             
STDOUT:                                                                                                                                                                                                                                                                      
[2025-10-19 16:17:34,349][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/21.parquet exists; returning.                                                         
[2025-10-19 16:17:34,353][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/4.parquet exists; returning.                                                          
[2025-10-19 16:17:34,362][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/9.parquet exists; returning.                                                          
[2025-10-19 16:17:34,367][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/27.parquet exists; returning.                                                         
[2025-10-19 16:17:34,372][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/held_out/2.parquet exists; returning.                                                       
[2025-10-19 16:17:34,377][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/16.parquet exists; returning.
[2025-10-19 16:17:34,388][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/25.parquet exists; returning.                                                         
[2025-10-19 16:17:34,397][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/17.parquet exists; returning.                                                         
[2025-10-19 16:17:34,402][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/held_out/1.parquet exists; returning.                                                       
[2025-10-19 16:17:34,406][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/held_out/0.parquet exists; returning.                                                       
[2025-10-19 16:17:34,410][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/3.parquet exists; returning.                                                          
[2025-10-19 16:17:34,414][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/19.parquet exists; returning.                                                         
[2025-10-19 16:17:34,419][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/tuning/3.parquet exists; returning.                                                         
[2025-10-19 16:17:34,423][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/13.parquet exists; returning.
[2025-10-19 16:17:34,427][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/11.parquet exists; returning.                                                         
[2025-10-19 16:17:34,432][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/2.parquet exists; returning.                                                          
[2025-10-19 16:17:34,436][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/12.parquet exists; returning.                                                         
[2025-10-19 16:17:34,440][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/1.parquet exists; returning.                                                          
[2025-10-19 16:17:34,445][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/24.parquet exists; returning.                                                         
[2025-10-19 16:17:34,452][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/7.parquet exists; returning.                                                          
[2025-10-19 16:17:34,459][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/tuning/0.parquet exists; returning.                                                         
[2025-10-19 16:17:34,465][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/14.parquet exists; returning.                                                         
[2025-10-19 16:17:34,469][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/23.parquet exists; returning.                                                         
[2025-10-19 16:17:34,475][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/held_out/3.parquet exists; returning.                                                       
[2025-10-19 16:17:34,479][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/22.parquet exists; returning.                                                         
[2025-10-19 16:17:34,484][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/10.parquet exists; returning.                                                         
[2025-10-19 16:17:34,488][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/5.parquet exists; returning.
[2025-10-19 16:17:34,493][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/18.parquet exists; returning.                                                         
[2025-10-19 16:17:34,497][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/tuning/2.parquet exists; returning.                                                         
[2025-10-19 16:17:34,503][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/0.parquet exists; returning.                                                          
[2025-10-19 16:17:34,507][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/20.parquet exists; returning.                                                         
[2025-10-19 16:17:34,511][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/tuning/1.parquet exists; returning.                                                         
[2025-10-19 16:17:34,515][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/26.parquet exists; returning.                                                         
[2025-10-19 16:17:34,519][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/8.parquet exists; returning.                                                          
[2025-10-19 16:17:34,524][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/15.parquet exists; returning.                                                         
[2025-10-19 16:17:34,529][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.cache/train/6.parquet exists; returning.
[2025-10-19 16:17:34,536][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/metadata/codes.parquet exists; returning.                                                          
[2025-10-19 16:17:36,025][MEDS_transforms.mapreduce.utils][INFO] - /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/metadata/codes.parquet exists; returning.                                                          
[2025-10-19 16:17:39,514][HYDRA] Joblib.Parallel(n_jobs=-1,backend=loky,prefer=processes,require=None,verbose=0,timeout=None,pre_dispatch=2*n_jobs,batch_size=auto,temp_folder=None,max_nbytes=None,mmap_mode=r) is launching 1 jobs
[2025-10-19 16:17:39,514][HYDRA] Launching jobs, sweep output dir : /sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/.logs                                                                                             
[2025-10-19 16:17:39,514][HYDRA]        #0 : worker=0 input_dir=/sc/home/robin.vandewater/datasets/meds/HIRID/data output_dir=/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab do_overwrite=False input_label_dir=/sc/h
ome/robin.vandewater/datasets/meds/HIRID/labels/mortality/in_icu/first_24h tabularization.aggs=[code/count,value/sum] tabularization.window_sizes=[7d,30d] tabularization.max_included_codes=100                                                                             
[2025-10-19 16:17:41,120][MEDS_transforms.mapreduce.utils][INFO] - Registered lock at 2025-10-19 16:17:41.117620. Double checking no earlier locks have been registered.                                                                                                     
[2025-10-19 16:17:41,122][MEDS_transforms.mapreduce.utils][INFO] - Reading input dataframe from /sc/home/robin.vandewater/datasets/meds/HIRID/data/train/19.parquet                                                                                                  [0/1884]
[2025-10-19 16:17:41,265][MEDS_transforms.mapreduce.utils][INFO] - Read dataset
                                                                                                                                                                                                                                                                             
                                                                                                                                                                                                                                                                             
The above exception was the direct cause of the following exception:                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                             
Traceback (most recent call last):                                                                                                                                                                                                                                           
  File "/sc/home/robin.vandewater/conda3/envs/meds_dev_new/bin/meds-dev-model", line 7, in <module>
    sys.exit(main())                                                                                                                                                                                                                                                         
             ^^^^^^                                                                                                                   
  File "/sc/home/robin.vandewater/conda3/envs/meds_dev_new/lib/python3.12/site-packages/hydra/main.py", line 94, in decorated_main                                                                                                                                           
    _run_hydra(                                                                                                                                                                                                                                                              
  File "/sc/home/robin.vandewater/conda3/envs/meds_dev_new/lib/python3.12/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra                                                                                                                                   
    _run_app(                                                                                                                                                                                                                                                                
  File "/sc/home/robin.vandewater/conda3/envs/meds_dev_new/lib/python3.12/site-packages/hydra/_internal/utils.py", line 457, in _run_app                                                                                                                                     
    run_and_report(                                                                                                                                                                                                                                                          
  File "/sc/home/robin.vandewater/conda3/envs/meds_dev_new/lib/python3.12/site-packages/hydra/_internal/utils.py", line 223, in run_and_report                                                                                                                               
    raise ex                                                                                                                                                                                                                                                                 
  File "/sc/home/robin.vandewater/conda3/envs/meds_dev_new/lib/python3.12/site-packages/hydra/_internal/utils.py", line 220, in run_and_report                                                                                                                               
    return func()                                                                                                                                                                                                                                                            
           ^^^^^^                                                                                                                                                                                                                                                            
  File "/sc/home/robin.vandewater/conda3/envs/meds_dev_new/lib/python3.12/site-packages/hydra/_internal/utils.py", line 458, in <lambda>                                                                                                                                     
    lambda: hydra.run(                                                                                                                                                                                                                                                       
            ^^^^^^^^^^                                                                                                                                                                                                                                                       
  File "/sc/home/robin.vandewater/conda3/envs/meds_dev_new/lib/python3.12/site-packages/hydra/_internal/hydra.py", line 132, in run                                                                                                                                          
    _ = ret.return_value                                                                                                                                                                                                                                                     
        ^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                     
  File "/sc/home/robin.vandewater/conda3/envs/meds_dev_new/lib/python3.12/site-packages/hydra/core/utils.py", line 260, in return_value                                                                                                                                      
    raise self._return_value                                                                                                                                                                                                                                                 
  File "/sc/home/robin.vandewater/conda3/envs/meds_dev_new/lib/python3.12/site-packages/hydra/core/utils.py", line 186, in run_job                                                                                                                                           
    ret.return_value = task_function(task_cfg)                                                                                                                                                                                                                               
                       ^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                               
  File "/sc/home/robin.vandewater/projects/MEDS_DEV_NEW/src/MEDS_DEV/models/__main__.py", line 36, in main                                                                                                                                                                   
    raise ValueError(f"Failed to run {cfg.model} command {cmd}") from e                                                                                                                                                                                                      
ValueError: Failed to run meds_tab/tiny command meds-tab-describe \                                                                                                                                                                                                          
    "input_dir=/sc/home/robin.vandewater/datasets/meds/HIRID/data" "output_dir=/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab"                                                                                       
                                                                                                                                                                                                                                                                             
meds-tab-tabularize-static \                                                                                                                                                                                                                                                 
    "input_dir=/sc/home/robin.vandewater/datasets/meds/HIRID/data" "output_dir=/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab" \                           
    do_overwrite=False "input_label_dir=/sc/home/robin.vandewater/datasets/meds/HIRID/labels/mortality/in_icu/first_24h" \                                                                                                                                                   
    "tabularization.aggs=[code/count,value/sum]" \                                                                                                                                                                                                                           
    "tabularization.window_sizes=[7d,30d]" \                                                                                                                                                                                                                                 
    "tabularization.max_included_codes=100"                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                             
meds-tab-tabularize-time-series \                                                                                                                                                                                                                                            
    --multirun \                                                                                                                                                                                                                                                             
    worker="range(0,1)" \                                                                                                                                                                                                                                                    
    hydra/launcher=joblib \                                                                                                                                                                                                                                                  
    "input_dir=/sc/home/robin.vandewater/datasets/meds/HIRID/data" "output_dir=/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab" \                                                                                     
    do_overwrite=False "input_label_dir=/sc/home/robin.vandewater/datasets/meds/HIRID/labels/mortality/in_icu/first_24h" \                                                                                                                                                   
    "tabularization.aggs=[code/count,value/sum]" \                                                                                                                                                                                                                           
    "tabularization.window_sizes=[7d,30d]" \                                                                                                                                                                                                                                 
    "tabularization.max_included_codes=100"                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                             
meds-tab-xgboost \                                                                                                                                                                                                                                                           
    --multirun \                                                                                                                                                                                                                                                             
    "input_dir=/sc/home/robin.vandewater/datasets/meds/HIRID/data" "output_dir=/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab" \                                                                                     
    "output_model_dir=/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/results" "task_name=null" do_overwrite=False \
    "hydra.sweeper.n_trials=20" "hydra.sweeper.n_jobs=1" \
    "hydra.sweeper.max_failure_rate=0.5" \                                                                                            
    "input_tabularized_cache_dir=/sc/home/robin.vandewater/datasets/meds/HIRID/models/mortality/in_icu/first_24h/meds_tab/tiny/meds_tab/tabularize/" \
    "input_label_cache_dir=/sc/home/robin.vandewater/datasets/meds/HIRID/labels/mortality/in_icu/first_24h" \
    "tabularization.aggs=[code/count,value/sum]" \                                                                                    
    "tabularization.window_sizes=[7d,30d]" \
    "tabularization.max_included_codes=100" tabularization.min_code_inclusion_count=null
Completed dataset: HIRID, task: mortality/in_icu/first_24h                                                                                                                                                                                           

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions