Skip to content

use compression.zstd#812

Open
dholth wants to merge 8 commits intomainfrom
811-compression-zstd
Open

use compression.zstd#812
dholth wants to merge 8 commits intomainfrom
811-compression-zstd

Conversation

@dholth
Copy link
Contributor

@dholth dholth commented Dec 11, 2025

Description

This library also does not require max_output_size when decompressing data without a zstandard size header.

Fix #811

Checklist - did you ...

  • Add a file to the news directory (using the template) for the next release's release notes?
  • Add / update necessary tests?
  • Add / update outdated documentation?

@dholth dholth self-assigned this Dec 11, 2025
@dholth dholth requested a review from a team as a code owner December 11, 2025 16:52
@github-project-automation github-project-automation bot moved this to 🆕 New in 🔎 Review Dec 11, 2025
@conda-bot conda-bot added the cla-signed [bot] added once the contributor has signed the CLA label Dec 11, 2025
# In one test reusing the context saves difference between .006s and .01s
# We could make this a threadlocal.
dctx = zstandard.ZstdDecompressor()
dctx = ZstdDecompressor()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest using the decompress function directly instead of creating dctx = ZstdDecompressor() first:

  1. dctx cannot be re-used
  2. If the compressed data is composed of several Zstandard frames, you will only decompress the first one

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Different than the other zstandard library.

"""
cache = cache.copy()
dctx = zstandard.ZstdDecompressor(max_window_size=ZSTD_MAX_SHARD_SIZE)
dctx = ZstdDecompressor()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

--file tests\requirements-s3.txt
--file ..\conda-libmamba-solver\dev\requirements.txt
${{ matrix.test-type == 'conda-libmamba-solver' && '--file ..\\conda-libmamba-solver\\tests\\requirements.txt' || '' }}
${{ matrix.python-version < '3.14' && 'backports.zstd' || '' }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
${{ matrix.python-version < '3.14' && 'backports.zstd' || '' }}
${{ matrix.python-version < '3.14' && 'backports.zstd' || '' }}

Hm, are we relying of lexicographical sorting for this comparison? I guess since the lower bound is 3.10 anyway we don't have issues with 3.9 > 3.14. Nice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed [bot] added once the contributor has signed the CLA

Projects

Status: 🆕 New

Development

Successfully merging this pull request may close these issues.

shards: use compression.zstd instead of zstandard-python

4 participants