Context
An issue that arose from a previous discussion has actually found its way into one of my projects, and I would like to use Skelebot to solve it. The issue revolve around versioning different parts of a Data Science project separately.
First, I built a model with a particular version and published it to S3. This code was then pushed to a branch and merged to master with that version as the overall project version. Later, I needed to make some updates to the scoring process, so I did that and updated the patch version at the same time. Unfortunately now, this creates an issue where my Python package is now looking for that particular version number for the model artifact instead of the old one.
I don't need to re-build the model, that code is unchanged, but I also want to increment the version since the scoring code did get updated. Skelebot itself already provides a way of pulling artifacts with the Latest Compatible Version (LCV) that could help in this situation, but gets more complicated as you consider versioning the Python package, the training code, and the results of training (models, encoders, etc.) separately since each is capable of being updated independently.
Feature
One step toward a solution for this issue would be to allow Skelebot users to override the VERSION file value with a value placed directly in the skelebot.yaml file. This would at least allow for multiple versions of the same project if used in different skelebot env files (skelebot-env.yaml).
While this doesn't completely solve the problem, specifically for Python packages that contain training and scoring code, this does help to alleviate it quite a bit for non-package DS projects.
Acceptance Criteria
Context
An issue that arose from a previous discussion has actually found its way into one of my projects, and I would like to use Skelebot to solve it. The issue revolve around versioning different parts of a Data Science project separately.
First, I built a model with a particular version and published it to S3. This code was then pushed to a branch and merged to master with that version as the overall project version. Later, I needed to make some updates to the scoring process, so I did that and updated the patch version at the same time. Unfortunately now, this creates an issue where my Python package is now looking for that particular version number for the model artifact instead of the old one.
I don't need to re-build the model, that code is unchanged, but I also want to increment the version since the scoring code did get updated. Skelebot itself already provides a way of pulling artifacts with the Latest Compatible Version (LCV) that could help in this situation, but gets more complicated as you consider versioning the Python package, the training code, and the results of training (models, encoders, etc.) separately since each is capable of being updated independently.
Feature
One step toward a solution for this issue would be to allow Skelebot users to override the VERSION file value with a value placed directly in the skelebot.yaml file. This would at least allow for multiple versions of the same project if used in different skelebot env files (skelebot-env.yaml).
While this doesn't completely solve the problem, specifically for Python packages that contain training and scoring code, this does help to alleviate it quite a bit for non-package DS projects.
Acceptance Criteria
versionin the Skelebot yaml file should take precedence over the VERSION file in the project