Skip to content

Add example: deploy Faster-Whisper as a custom model on a SageMaker Triton MME - #4888

Open
eldadcohen1 wants to merge 2 commits into
aws:defaultfrom
eldadcohen1:sm-triton_mme_faster_whisper
Open

Add example: deploy Faster-Whisper as a custom model on a SageMaker Triton MME#4888
eldadcohen1 wants to merge 2 commits into
aws:defaultfrom
eldadcohen1:sm-triton_mme_faster_whisper

Conversation

@eldadcohen1

Copy link
Copy Markdown

Adds an example showing how to deploy a pre-trained Faster-Whisper-Large-v3 model from Hugging Face as a custom model on an Amazon SageMaker multi-model endpoint (MME) backed by Triton, including building a custom Triton container and running audio-transcription inference.

Description of changes:

  • New example under deploy_and_monitor/sm-triton_mme_faster_whisper/ (notebook, README.md, and a sample audio file), following the current repository structure and sm-triton_* naming convention.
  • Registered the notebook in the Deploy and Monitor toctree in index.rst.
  • CI badge URLs and kernelspec (conda_python3) match the sibling Triton examples.

This supersedes #4608 / #4887, which targeted the legacy main branch and the old sagemaker-triton/ layout. This PR targets the current default branch and the new folder structure.

Testing is done:
Tested on AWS SageMaker.

Merge Checklist

  • I have read the CONTRIBUTING doc and adhered to the example notebook best practices
  • I have updated any necessary documentation, including READMEs
  • I have tested my notebook(s) and ensured it runs end-to-end
  • I have linted my notebook(s) and code using black-nb -l 100 {path}/{notebook-name}.ipynb

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…riton MME

Ports the multi-model-endpoint Triton custom-model (Faster-Whisper-Large-v3)
example into the new repository structure under deploy_and_monitor/, updates
the CI badge URLs and kernelspec to match sibling examples, and registers the
notebook in the Deploy and Monitor toctree.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@review-notebook-app

Copy link
Copy Markdown

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@eldadcohen1

Copy link
Copy Markdown
Author

@mollyheamazon can you please review and approve

@mollyheamazon

Copy link
Copy Markdown

The notebook uses backend: "python" in config.pbtxt, but there's no cell that writes the required model.py into the model directory. Could you add the %%writefile cell for it? Without it a reader can't reproduce end-to-end.

Another note: This notebook uses raw boto3 calls rather than the SageMaker Python SDK. It might be a better fit for awsdocs/aws-doc-sdk-examples (which hosts boto3 examples)

Addresses review feedback on aws#4888: config.pbtxt uses backend: "python"
but no cell wrote the required model.py. Adds a %%writefile cell that creates
model.py in the version directory, loading Faster-Whisper and returning the
transcript, so the example is reproducible end-to-end.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@eldadcohen1

eldadcohen1 commented Jun 21, 2026

Copy link
Copy Markdown
Author

Thanks @mollyheamazon!

1. Missing model.py - good catch, fixed in 6d68871. Added a %%writefile faster-whisper-large-v3/1/model.py cell that writes the Python-backend model into the version directory next to the weights. It loads WhisperModel in initialize, and in execute base64-decodes audio_array, transcribes with the given audio_lang, and returns transcript. faster-whisper is already pip-installed in the Dockerfile, so the example now reproduces end-to-end.

2. boto3 vs SageMaker SDK - the raw boto3 calls are intentional here: the example's focus is the low-level MME + custom-Triton-container flow (create_model / create_endpoint_config / create_endpoint with Mode: MultiModel), which the SDK abstracts away. Happy to defer to you, though - if you'd rather this live in aws-doc-sdk-examples, let me know and I'll move it.

@eldadcohen1

Copy link
Copy Markdown
Author

@jam-jee , @mollyheamazon fixed you comments, can you please take a look.

@eldadcohen1

Copy link
Copy Markdown
Author

@mollyheamazon any thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants