Merge pull request #13 from radical-cybertools/improve/dragonv3

Improve tasks wait in Dragon

Author: AymenFJA

.github/workflows/docs.yml

@@ -2,7 +2,7 @@ name: Deploy Documentation
 
 on:
   push:
-    branches: [ main, master, feature/docs_polish ]
+    branches: [ main, master]
   workflow_dispatch:
 
 permissions:

CHANGELOG.md

@@ -0,0 +1,11 @@
+# Changelog
+
+## [Unreleased]
+
+- Removed ThreadPoolExecutor wait approach (`_wait_executor`) attribute, streamlining concurrency management.
+- Added `disable_batch_submission` boolean parameter to `DragonExecutionBackendV3.__init__` and `create` methods, allowing configuration of task submission strategy.
+- Introduced `_monitored_batches` dictionary and `_batch_monitor_thread` to manage and monitor active task batches.
+- Replaced the `_wait_for_batch` method with a new `_monitor_loop` (running in a dedicated thread) and `_process_batch_results` helper, implementing a polling-based mechanism for task completion.
+- Modified `_async_init` to start the `_batch_monitor_thread` during backend initialization.
+- Updated `submit_tasks` to support conditional task submission: tasks can now be submitted as individual batches (stream mode) or a single combined batch, based on the `_disable_batch_submission` flag.
+- Adjusted the `shutdown` method to properly manage the lifecycle of the new `_batch_monitor_thread`.

docs/about.md

@@ -1,6 +1,6 @@
 # About RHAPSODY
 
-RHAPSODY (**R**untime for **H**eterogeneous **AP**plications, **S**ervice **O**rchestration and **DY**namism) is a cutting-edge runtime system designed to address the evolving challenges of modern scientific computing. As computational workflows become increasingly complex, combining traditional HPC simulations with AI/ML workloads, there is a critical need for systems that can efficiently orchestrate heterogeneous tasks across diverse computing resources.
+RHAPSODY (<b>R</b>untime for <b>H</b>eterogeneous <b>AP</b>plications, <b>S</b>ervice <b>O</b>rchestration and <b>DY</b>namism) is a cutting-edge runtime system designed to address the evolving challenges of modern scientific computing. As computational workflows become increasingly complex, combining traditional HPC simulations with AI/ML workloads, there is a critical need for systems that can efficiently orchestrate heterogeneous tasks across diverse computing resources.
 
 ## Vision
 

docs/changelogs.md

@@ -0,0 +1 @@
+--8<-- "CHANGELOG.md"

docs/getting-started/configuration.md

@@ -55,12 +55,30 @@ backend = DragonExecutionBackendV3(
     name="dragon",
     num_workers=16,               # Total worker processes
     disable_telemetry=False,       # Enable/disable Dragon telemetry
-    disable_background_batching=False # Enable/disable background monitoring
+    disable_background_batching=False, # Enable/disable background monitoring
+    disable_batch_submission=False # Toggle batch vs individual submission
 )
 ```
 
+!!! note "Batch vs. Stream Submission"
+    The `disable_batch_submission` parameter controls how tasks are sent to the Dragon runtime:
+
+    - **Batch Mode** (`False`): High throughput. Groups tasks into a single multi-task batch.
+    - **Stream Mode** (`True`): Individual monitoring. Each task is submitted and tracked as a separate Dragon batch.
+
+    ```python
+    # Batch Mode (Recommended for many small tasks)
+    backend = await DragonExecutionBackendV3(disable_batch_submission=False)
+    await session.submit_tasks([t1, t2]) # Single submission
+
+    # Stream Mode (Better for isolated task tracking)
+    backend = await DragonExecutionBackendV3(disable_batch_submission=True)
+    await session.submit_tasks([t1, t2]) # Two separate submissions
+    ```
+
+
 !!! note "Dragon Versions"
-    While `DragonExecutionBackendV3` is recommended for most users due to its native batch integration, V1 and V2 are also available for legacy compatibility.
+    While `DragonExecutionBackendV3` is recommended for most users and will be always maintained, V1 and V2 are also available for legacy compatibility.
 
 ### RADICAL-Pilot Backend
 Large-scale HPC execution using RADICAL-Pilot.

mkdocs.yml

@@ -115,6 +115,7 @@ nav:
   - Project:
       - project/index.md
       - Roadmap: project/roadmap.md
+      - Changelog: changelogs.md
       - Publications: project/publications.md
       - NSF Award: project/nsf-award.md
       - Team: project/team.md