S3: delete directories in batches

[stsewd]

We were iterating over each file when deleting a directory. boto3 has support for batch operations, it does one call for every 1K files, this get us faster deletions and fewer API calls.

[stsewd]

@copilot write test for the S3BuildMediaStorage class for its delete_directory method, you can mock the call to .delete().

[Copilot]

@copilot write test for the S3BuildMediaStorage class for its delete_directory method, you can mock the call to .delete().

Added tests for S3BuildMediaStorage.delete_directory in readthedocs/storage/tests/test_s3_storage.py (commit 411d03e). The tests mock the bucket via _bucket (the internal attribute used by the bucket property) and cover:

• delete_directory calls bucket.objects.filter(Prefix=path).delete() with the correct path
• A trailing slash is appended when the path doesn't have one
• SuspiciousFileOperation is raised for root path "/" and empty path ""

[Copilot]

Pull request overview

Improve S3 directory deletion performance by leveraging boto3’s batch delete operations over a prefix, reducing per-object API calls.

Changes:

• Replace recursive listdir + per-file deletion with bucket.objects.filter(Prefix=...).delete().
• Normalize delete paths using clean_name + _normalize_name, and enforce trailing slashes.
• Add unit tests covering prefix deletion, trailing-slash behavior, and suspicious root/empty path inputs.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
readthedocs/storage/s3_storage.py	Switches delete_directory to a single prefix-based batch delete and adds path normalization.
readthedocs/storage/tests/test_s3_storage.py	Adds tests for S3 delete_directory behavior using a mocked bucket.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ericholscher]

This seems like an obvious win. I'm pretty surprised django-storages doesn't expose this directly.