feat: video metadata loading optimization#31
Conversation
| videos = Video.objects.filter( | ||
| parent_content_metadata__content_key__in=course_run_content_keys | ||
| ).select_related('parent_content_metadata') | ||
| with function_trace(AlgoliaTraceNames.VIDEO_RECORD_LOADING): | ||
| video_queryset = Video.objects.filter( | ||
| parent_content_metadata__content_key__in=course_run_content_keys | ||
| ).select_related('parent_content_metadata') | ||
|
|
||
| all_videos = list(video_queryset) | ||
| video_ids_by_parent_content_key = defaultdict(list) | ||
| for video in all_videos: | ||
| parent_content_key = video.parent_content_metadata.content_key | ||
| video_ids_by_parent_content_key[parent_content_key].append(video.edx_video_id) |
There was a problem hiding this comment.
this is the main change. instead of iterating (below) over videos every time we need them, we'll do it once, stash in a list, and then create a pre-optimized cache of parent content keys mapped to lists of the related edx_video_ids.
| if metadata.content_type == COURSE_RUN: | ||
| related_video_ids = video_ids_by_parent_content_key.get(metadata.content_key, []) | ||
| video_ids_by_key[content_key].update(related_video_ids) | ||
|
|
There was a problem hiding this comment.
change to a set update() to include all items from a list.
| num_content_metadata_indexed += 1 | ||
|
|
||
| for video in videos: | ||
| for video in all_videos: |
There was a problem hiding this comment.
use the pre-populated list
also adds more function_trace()s
iloveagent57
force-pushed
the
aed/moar-batching
branch
from
6677f9f to
9b59a41
Compare
There was a problem hiding this comment.
Pull request overview
This PR optimizes video metadata loading during Algolia indexing by pre-processing videos into a lookup dictionary, eliminating an expensive nested loop iteration. Additionally, test fixture dates are updated to 2056 to avoid issues with dates being too close to the current date.
Changes:
- Optimized video loading to avoid O(N*M) nested loop by pre-loading videos and creating a lookup dictionary
- Added function_trace decorators for enhanced monitoring of video loading and other operations
- Updated test fixture dates from 2026 to 2056 to ensure tests remain valid long-term
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| enterprise_catalog/apps/catalog/tests/factories.py | Updated test fixture dates from 2026 to 2056 for enroll_by_date, upgrade_deadline, and end dates |
| enterprise_catalog/apps/catalog/constants.py | Added VIDEO_RECORD_LOADING and REPLACE_ALL_OBJECTS trace name constants to AlgoliaTraceNames enum |
| enterprise_catalog/apps/api/tests/test_tasks.py | Updated test expectations for new dates with correctly calculated epoch timestamps |
| enterprise_catalog/apps/api/tasks.py | Implemented video loading optimization by pre-loading videos into a dictionary; added function_trace decorators for monitoring |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
also adds more function_trace()s
Post-review