Skip to content

Fix integer overflow in intermediate values in QuantizerWu.kt#197

Open
RuurdBijlsma wants to merge 1 commit intomaterial-foundation:mainfrom
RuurdBijlsma:main
Open

Fix integer overflow in intermediate values in QuantizerWu.kt#197
RuurdBijlsma wants to merge 1 commit intomaterial-foundation:mainfrom
RuurdBijlsma:main

Conversation

@RuurdBijlsma
Copy link

@RuurdBijlsma RuurdBijlsma commented Feb 28, 2026

There's integer overflow issues in the Kotlin implementation of QuantizerWu.

Example in the maximize function (original, wrong, code):

tempNumerator = (halfR * halfR + halfG * halfG + halfB * halfB).toDouble()

I put in an image to be quantized, and these are the values:

halfR = 6645743
halfG = 5064075
halfB = 3940401

These variables are all int 32, so the above formula would cause integer overflow (halfR * halfR is already too much for i32).

Similar cases in other places. I fixed it in this PR by converting to double earlier in the flow.

@google-cla
Copy link

google-cla bot commented Feb 28, 2026

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant