Motif

[Iznoanygod]

This pull request adds the Digital Twin and Inverse Design motifs as workflows, built on Asyncflow with Dragon Execution backend.
Workflows are tunable with the config file.
This was based on the Tutorial_reprod branch, which has an updated workflow miniapp kernel API and tutorial notebook.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the mini-app ecosystem by integrating two complex scientific workflows, Digital Twin and Inverse Design. These additions provide robust, configurable frameworks for advanced computational tasks, built upon an updated and more performant kernel API. The changes aim to enhance the platform's capability to model and execute sophisticated scientific applications efficiently, with a focus on modularity and performance analysis.

Highlights

• New Workflow Motifs: Introduced Digital Twin (Motif 4) and Inverse Design (Motif 3) as new workflows, leveraging Asyncflow with a Dragon Execution backend for enhanced flexibility and performance.
• Configurable Workflows: Enabled tunability of new workflows through dedicated YAML configuration files, allowing users to easily adjust parameters for simulation, training, and inference components.
• Kernel API Enhancements: Refactored the wfMiniAPI kernel with a new registry system, adding decorators for kernel annotation and introducing new functions like fftn and axpy_fast, along with top_k for active learning tasks.
• Performance Benchmarks: Added several CUDA benchmarks for core computational operations such as AXPY, FFT (1D and 3D), and matrix multiplication to evaluate kernel performance.
• Package Structure Updates: Updated the wfMiniAPI package structure, including a new src directory, MANIFEST.in, VERSION file, and revised setup.py for improved distribution and dependency management.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• examples/Tutorial/tasks/active_learn.py
  • Added a new Python script for active learning tasks, including argument parsing for sample size, batch size, device, and dense layer dimensions.
• examples/Tutorial/tasks/simulation.py
  • Added a new Python script for molecular dynamics simulation tasks, configurable with parameters like atom count, grid size, and logging frequency.
• examples/Tutorial/tasks/training.py
  • Added a new Python script for Bayesian ML training tasks, supporting configurable epochs, sample sizes, batch sizes, and device selection.
• examples/motif_3/config.yaml
  • Added a YAML configuration file defining parameters for the Inverse Design (Motif 3) simulation and training steps.
• examples/motif_3/miniapp_parallel.py
  • Added a new Python script implementing a parallel Inverse Design (Motif 3) mini-app using Asyncflow, with simulate and train function tasks.
• examples/motif_3/miniapp_simple.py
  • Added a new Python script implementing a simple Inverse Design (Motif 3) mini-app using Asyncflow, with iterative simulate and train tasks.
• examples/motif_3/utils/metrics.py
  • Added a utility file containing a Timer class and a logging function for performance measurement within Motif 3 applications.
• examples/motif_4/config.yaml
  • Added a YAML configuration file defining parameters for the Digital Twin (Motif 4) experiment, simulation, training, and inference steps.
• examples/motif_4/miniapp_async.py
  • Added a new Python script implementing an asynchronous Digital Twin (Motif 4) mini-app using Asyncflow, orchestrating experiment, simulation, training, and inference tasks.
• examples/motif_4/utils/metrics.py
  • Added a utility file containing a Timer class and a logging function for performance measurement within Motif 4 applications.
• wfMiniAPI/MANIFEST.in
  • Added a manifest file to include various file types for package distribution.
• wfMiniAPI/README.md
  • Renamed the wfMiniAPI/README file to wfMiniAPI/README.md for standard markdown formatting.
• wfMiniAPI/VERSION
  • Added a version file to specify the package version as '0.1'.
• wfMiniAPI/benchmark/axpy_benchmark.cu
  • Added a CUDA C++ benchmark for the AXPY operation using cuBLAS.
• wfMiniAPI/benchmark/axpy_naive_benchmark.cu
  • Added a CUDA C++ benchmark for a naive AXPY kernel implementation.
• wfMiniAPI/benchmark/fft_benchmark.cu
  • Added a CUDA C++ benchmark for 1D FFT using cuFFT.
• wfMiniAPI/benchmark/fft_benchmark_3d.cu
  • Added a CUDA C++ benchmark for 3D FFT using cuFFT.
• wfMiniAPI/benchmark/matmul_benchmark.cu
  • Added a CUDA C++ benchmark for matrix multiplication (SGEMM) using cuBLAS.
• wfMiniAPI/requirements-gpu.txt
  • Added a new requirements file listing Python dependencies specifically for GPU-enabled environments, including cupy and ROSE.
• wfMiniAPI/requirements.txt
  • Added a new requirements file listing base Python dependencies, including numpy, h5py, mpi4py, and radical-pilot.
• wfMiniAPI/setup.py
  • Updated the setup.py script to reflect the new package structure, add metadata, and manage dependencies more robustly, including GPU-specific extras.
• wfMiniAPI/src/wfMiniAPI/kernel.py
  • Renamed the file from wfMiniAPI/python/wfMiniAPI/kernel.py to wfMiniAPI/src/wfMiniAPI/kernel.py, integrated kernel annotation decorators, added fftn and axpy_fast functions, and implemented top_k functionality.
• wfMiniAPI/src/wfMiniAPI/registry.py
  • Added a new module to register, list, and time kernel functions, providing a structured way to manage and execute kernels.
• wfMiniAPI/src/wfMiniAPI/sim.py
  • Added a new module containing triad_kernel and reduction_kernel for simulation mini-apps, along with a run_miniapp function.
• wfMiniAPI/src/wfMiniAPI/timing.py
  • Added a new module demonstrating how to use the kernel registry for timing CPU and GPU kernel executions.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces new mini-applications and configuration files under examples/Tutorial/tasks/, examples/motif_3/, and examples/motif_4/, alongside new CUDA benchmark files. The wfMiniAPI library is refactored to include a new registry.py module for kernel management (annotation, listing, running, timing), and kernel.py is updated to integrate this registry, annotating existing kernels and adding new fftn, axpy_fast, and top_k functions. The review identified several critical issues, including a loop variable shadowing a function parameter in examples/motif_4/miniapp_async.py, undefined variables in wfMiniAPI/src/wfMiniAPI/sim.py, and an undefined sh_callout function in wfMiniAPI/setup.py. It also highlighted high-severity improvements related to unused function parameters across multiple mini-app scripts. Medium-severity feedback included unused imports, redundant logging calls, non-portable path concatenations, typos in help strings, an unused logger variable, hardcoded CUDA version in setup.py, a redundant GPU warmup loop in registry.py, misplacement of a timing script, and a lack of error checking in CUDA benchmark cleanup functions.

[AymenFJA]

since you are linking this to the main branch of ROSE. Does the mini apps support the new ROSE API?

[Iznoanygod]

Requirements have been changed, ROSE is not used in the framework or examples, instead Asyncflow, Rhapsody, and Dragon have been added

[AymenFJA]

@Iznoanygod The code that you are trying to merge is using an old version of radical.asyncflow. We need to do 2 things, please:

1. Migrate it to using AsyncFlow release + RHAPSODY
2. Make sure the active learn examples and any example that uses ROSE are tested against the new API of ROSE.

LMK, if you need any help of clairfication

[Iznoanygod]

@AymenFJA Code now uses current Asyncflow + RHAPSODY API, with Dragon Backend. The requirements now exclude ROSE as it is not needed, dragon, rhapsody, asyncflow added as requirements for the motif examples.