Use npz instead of pickle for Thai W2P weights by wannaphong · Pull Request #1328 · PyThaiNLP/pythainlp

wannaphong · 2026-03-13T22:01:22Z

This PR hardens corpus/model loading against unsafe pickle deserialization and updates Thai W2P to load .npz weights.

Changes:

Switch Thai W2P weights loading to a .npz-based format with allow_pickle=False.
Add is_unsafe_pickle_allowed() and re-export it via pythainlp and pythainlp.tools.
Update tests and docs/README/changelog to reflect the new unsafe-pickle opt-in behavior.
Passed code styles and structures
Passed code linting checks and unit test

Change np.load to disallow pickling for security.

wannaphong · 2026-03-13T22:02:11Z

@copilot Make np.load that disallow allow_pickle working.

Copilot · 2026-03-13T22:02:19Z

@wannaphong I've opened a new pull request, #1329, to work on those changes. Once the pull request is ready, I'll request review from you.

) * Initial plan * Fix np.load allow_pickle=False to work with .npz NpzFile format - Replace .item().get(key) with [key] dict-style access on NpzFile - Remove variables instance attribute; use local variable instead - Add type annotation for variables local var as np.lib.npyio.NpzFile - Add allow_pickle=False to embeddings.npy load in words_spelling_correction.py Co-authored-by: wannaphong <8536487+wannaphong@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: wannaphong <8536487+wannaphong@users.noreply.github.com>

wannaphong · 2026-03-13T22:12:27Z

@copilot How to fix this?

Traceback (most recent call last):
File "/home/runner/work/pythainlp/pythainlp/tests/compact/testc_util.py", line 48, in test_check_khuap_klam
self.assertTrue(check_khuap_klam("กราบ"))
File "/home/runner/work/pythainlp/pythainlp/pythainlp/util/khuap_klam.py", line 63, in check_khuap_klam
reading = pronunciate(word, engine="w2p").replace("\u0e3a", "")
File "/home/runner/work/pythainlp/pythainlp/pythainlp/transliterate/core.py", line 217, in pronunciate
from pythainlp.transliterate.w2p import pronunciate
File "/home/runner/work/pythainlp/pythainlp/pythainlp/transliterate/w2p.py", line 261, in
_THAI_W2P: "Thai_W2P" = Thai_W2P()
File "/home/runner/work/pythainlp/pythainlp/pythainlp/transliterate/w2p.py", line 96, in init
self._load_variables()
File "/home/runner/work/pythainlp/pythainlp/pythainlp/transliterate/w2p.py", line 103, in _load_variables
variables: "np.lib.npyio.NpzFile" = np.load(
File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/site-packages/numpy/lib/_npyio_impl.py", line 484, in load
return format.read_array(fid, allow_pickle=allow_pickle,
File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/site-packages/numpy/lib/format.py", line 822, in read_array
raise ValueError("Object arrays cannot be loaded when "
ValueError: Object arrays cannot be loaded when allow_pickle=False

Copilot · 2026-03-13T22:12:36Z

@wannaphong I've opened a new pull request, #1330, to work on those changes. Once the pull request is ready, I'll request review from you.

The thai_w2p corpus v0.2 is stored as a .npy pickled dict. Loading it with allow_pickle=False raises ValueError. Detect file format by extension: - .npz: use allow_pickle=False (secure, for future corpus versions) - .npy (legacy): use allow_pickle=True + dict validation Also add `import os` for os.path.splitext(). Co-authored-by: wannaphong <8536487+wannaphong@users.noreply.github.com>

Fix np.load allow_pickle=False breaking thai_w2p corpus loading

coveralls · 2026-03-14T03:23:53Z

coverage: 65.93% (-0.003%) from 65.933%
when pulling 500c6e5 on improve-w2p
into d9dec9e on dev.

bact · 2026-03-14T03:27:36Z

@copilot The dev branch is updated. Sync the changes.

Review all the changes in file loading, including the pickle handling. Pay attention to security issues.

When loading a pickle file, emit a warning.

Copilot · 2026-03-14T03:27:42Z

@bact I've opened a new pull request, #1331, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: bact <128572+bact@users.noreply.github.com>

This release focuses on security issues related to corpus file loading, including improved pickle handling and defensive file loading.

Disallow pickle in np.load, warn on legacy pickle loading, sync dev defensive file loading

Copilot

Pull request overview

Updates corpus/model loading paths to reduce unsafe deserialization risks, primarily by tightening np.load usage and hardening some corpus CSV parsing.

Changes:

Add .npz-first loading for Thai W2P weights and warn on legacy pickle-based .npy loading.
Set allow_pickle=False for embedding matrix loading in spelling correction.
Minor hardening/refactor of corpus CSV field validation and doc/changelog updates.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pythainlp/transliterate/w2p.py	Add `.npz` loading path and legacy `.npy` warning/pickle fallback for Thai W2P weights.
pythainlp/tag/_tag_perceptron.py	Update `load()` docstring to reflect JSON model format.
pythainlp/spell/words_spelling_correction.py	Disallow pickle when loading embeddings with `np.load`.
pythainlp/corpus/common.py	Refactor validation logic for CSV fields before processing/warning.
CHANGELOG.md	Add 5.3.1 entry and update compare links.

You can also share your feedback on Copilot code review. Take the survey.

pythainlp/transliterate/w2p.py

CHANGELOG.md

pythainlp/tag/_tag_perceptron.py

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR tightens security around model/corpus loading by disabling legacy pickle-based .npy weight loading by default, adding a .npz-first loading path for Thai W2P, and requiring explicit opt-in via PYTHAINLP_ALLOW_UNSAFE_PICKLE with warnings.

Changes:

Add .npz loading for Thai W2P weights and block legacy pickle-based .npy loading unless PYTHAINLP_ALLOW_UNSAFE_PICKLE is set (with warning).
Introduce is_unsafe_pickle_allowed() and re-export it via pythainlp and pythainlp.tools.
Update tests and documentation to reflect the new unsafe-pickle opt-in behavior.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/extra/testx_transliterate.py	Adjust W2P-related tests to opt into legacy pickle loading and add coverage for blocked/allowed behavior.
tests/core/test_transliterate.py	Minor formatting change.
tests/core/test_tools.py	Add unit tests for `is_unsafe_pickle_allowed()`.
tests/compact/testc_util.py	Opt into unsafe pickle for tests that trigger W2P-backed functionality.
pythainlp/transliterate/w2p.py	Implement `.npz`-first loading; gate legacy `.npy` pickle loading behind env var + warning.
pythainlp/tools/path.py	Add `is_unsafe_pickle_allowed()` helper.
pythainlp/tools/init.py	Re-export `is_unsafe_pickle_allowed()`.
pythainlp/tag/_tag_perceptron.py	Update docstrings/comments to reflect JSON model save/load.
pythainlp/spell/words_spelling_correction.py	Load embeddings with `allow_pickle=False`.
pythainlp/corpus/common.py	Make CSV field validation checks more explicit/defensive.
pythainlp/init.py	Re-export `is_unsafe_pickle_allowed()` at top-level.
README_TH.md	Document `PYTHAINLP_ALLOW_UNSAFE_PICKLE`.
README.md	Document `PYTHAINLP_ALLOW_UNSAFE_PICKLE`.
CHANGELOG.md	Add 5.3.1 entry and note improved pickle handling.

You can also share your feedback on Copilot code review. Take the survey.

Updated model name and refactored variable loading to use .npz format exclusively, removing legacy .npy handling.

wannaphong · 2026-03-14T05:49:50Z

@bact I was convert Thai W2P model to .npz file.

Removed unused imports from w2p.py

bact · 2026-03-14T06:05:01Z

@wannaphong I have removed most tests related to PYTHAINLP_ALLOW_UNSAFE_PICKLE then

Copilot

Pull request overview

This PR hardens corpus/model loading against unsafe pickle deserialization by default, adds an explicit opt-in gate via PYTHAINLP_ALLOW_UNSAFE_PICKLE, and updates Thai W2P to load .npz weights.

Changes:

Add is_unsafe_pickle_allowed() and re-export it via pythainlp and pythainlp.tools.
Switch Thai W2P weights loading to a .npz-based format with allow_pickle=False.
Update tests and docs/README/changelog to reflect the new unsafe-pickle opt-in behavior.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/core/test_transliterate.py	Minor formatting change at EOF.
tests/core/test_tools.py	Adds unit tests for `is_unsafe_pickle_allowed()`.
pythainlp/transliterate/w2p.py	Switches Thai W2P weight loading to `.npz` keys and disables pickle.
pythainlp/tools/path.py	Introduces `is_unsafe_pickle_allowed()` env-var gate.
pythainlp/tools/init.py	Re-exports `is_unsafe_pickle_allowed`.
pythainlp/tag/_tag_perceptron.py	Updates docstrings/comments to reflect JSON model persistence.
pythainlp/spell/words_spelling_correction.py	Hardens `np.load` by setting `allow_pickle=False`.
pythainlp/corpus/common.py	Tightens CSV field validation checks before processing.
pythainlp/init.py	Re-exports `is_unsafe_pickle_allowed` at top-level API.
README_TH.md	Documents `PYTHAINLP_ALLOW_UNSAFE_PICKLE`.
README.md	Documents `PYTHAINLP_ALLOW_UNSAFE_PICKLE`.
CHANGELOG.md	Adds 5.3.1 security notes and updates compare links.

You can also share your feedback on Copilot code review. Take the survey.

tests/core/test_transliterate.py

pythainlp/tools/path.py

pythainlp/transliterate/w2p.py

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

We no longer use pickle. Do not advertise this env var. Keep it internally for future use. (may remove in 6.0.0)

sonarqubecloud · 2026-03-14T06:28:30Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

bact · 2026-03-14T06:29:07Z

@wannaphong feel free to merge when you are ready

Update np.load to disallow allow_pickle

2c19966

Change np.load to disallow pickling for security.

Copilot AI mentioned this pull request Mar 13, 2026

Fix np.load allow_pickle=False to work correctly with .npz format #1329

Merged

2 tasks

Initial plan

48b1f0f

Copilot AI mentioned this pull request Mar 13, 2026

Fix np.load allow_pickle=False breaking thai_w2p corpus loading #1330

Merged

2 tasks

Copilot AI and others added 2 commits March 13, 2026 22:21

Merge pull request #1330 from PyThaiNLP/copilot/sub-pr-1328-again

90eaaca

Fix np.load allow_pickle=False breaking thai_w2p corpus loading

bact added corpus corpus/dataset-related issues security and removed corpus corpus/dataset-related issues labels Mar 14, 2026

bact added this to the 5.3.1 milestone Mar 14, 2026

Initial plan

d02d313

Copilot AI mentioned this pull request Mar 14, 2026

Disallow pickle in np.load, warn on legacy pickle loading, sync dev defensive file loading #1331

Merged

2 tasks

bact changed the title ~~[WIP] Update np.load to disallow allow_pickle~~ [WIP] Update np.load to handle allow_pickle Mar 14, 2026

bact changed the title ~~[WIP] Update np.load to handle allow_pickle~~ [WIP] Update np.load to handle pickle Mar 14, 2026

Copilot AI and others added 4 commits March 14, 2026 03:30

Merge branch 'dev' into copilot/sub-pr-1328

ec12c6e

Sync dev, add pickle warning, fix docstring and code style

a145055

Co-authored-by: bact <128572+bact@users.noreply.github.com>

Update CHANGELOG for release 5.3.1

8c5f125

This release focuses on security issues related to corpus file loading, including improved pickle handling and defensive file loading.

Merge pull request #1331 from PyThaiNLP/copilot/sub-pr-1328

053afe5

Disallow pickle in np.load, warn on legacy pickle loading, sync dev defensive file loading

bact requested a review from Copilot March 14, 2026 03:43

Copilot started reviewing on behalf of bact March 14, 2026 03:44 View session

Copilot AI reviewed Mar 14, 2026

View reviewed changes

pythainlp/transliterate/w2p.py Outdated Show resolved Hide resolved

pythainlp/transliterate/w2p.py Outdated Show resolved Hide resolved

CHANGELOG.md Outdated Show resolved Hide resolved

pythainlp/tag/_tag_perceptron.py Show resolved Hide resolved

Potential fix for pull request finding

44b4202

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

bact added 2 commits March 14, 2026 12:14

Fix import

db08a00

Set PYTHAINLP_ALLOW_UNSAFE_PICKLE for tone detector test

a8bfe51

bact marked this pull request as ready for review March 14, 2026 05:30

bact requested a review from Copilot March 14, 2026 05:30

Copilot started reviewing on behalf of bact March 14, 2026 05:30 View session

Copilot AI reviewed Mar 14, 2026

View reviewed changes

bact and others added 2 commits March 14, 2026 12:45

Refactor is_unsafe_pickle_allowed()

bec6fe1

Refactor model loading to use .npz format only

fb27b5d

Updated model name and refactored variable loading to use .npz format exclusively, removing legacy .npy handling.

wannaphong and others added 4 commits March 14, 2026 12:51

Clean up w2p.py by removing os and warnings imports

ddb78b6

Removed unused imports from w2p.py

Remove PYTHAINLP_ALLOW_UNSAFE_PICKLE tests

17f66ba

Update CHANGELOG.md

1caf8f6

Fix imports

7a731a6

bact changed the title ~~[WIP] Only load pickle when PYTHAINLP_ALLOW_UNSAFE_PICKLE is set~~ [WIP] Disallow pickle loading in np.load Mar 14, 2026

bact requested a review from Copilot March 14, 2026 06:08

Copilot started reviewing on behalf of bact March 14, 2026 06:09 View session

Copilot AI reviewed Mar 14, 2026

View reviewed changes

tests/core/test_transliterate.py Outdated Show resolved Hide resolved

pythainlp/tools/path.py Show resolved Hide resolved

pythainlp/transliterate/w2p.py Outdated Show resolved Hide resolved

bact and others added 3 commits March 14, 2026 13:13

Potential fix for pull request finding

a52bdf1

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

9cdd39e

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Update CHANGELOG.md

b8f39fb

bact approved these changes Mar 14, 2026

View reviewed changes

Remove PYTHAINLP_ALLOW_UNSAFE_PICKLE from doc

500c6e5

We no longer use pickle. Do not advertise this env var. Keep it internally for future use. (may remove in 6.0.0)

bact approved these changes Mar 14, 2026

View reviewed changes

wannaphong changed the title ~~[WIP] Disallow pickle loading in np.load~~ Disallow pickle loading in np.load Mar 14, 2026

bact changed the title ~~Disallow pickle loading in np.load~~ Use npz instead of pickle for Thai W2P weights Mar 14, 2026

wannaphong merged commit 6df48fb into dev Mar 14, 2026
28 of 29 checks passed

wannaphong deleted the improve-w2p branch March 14, 2026 06:36

Conversation

wannaphong commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wannaphong commented Mar 13, 2026

Uh oh!

Copilot AI commented Mar 13, 2026

Uh oh!

wannaphong commented Mar 13, 2026

Uh oh!

Copilot AI commented Mar 13, 2026

Uh oh!

coveralls commented Mar 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bact commented Mar 14, 2026

Uh oh!

Copilot AI commented Mar 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

wannaphong commented Mar 14, 2026

Uh oh!

bact commented Mar 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented Mar 14, 2026

Quality Gate passed

Uh oh!

bact commented Mar 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

wannaphong commented Mar 13, 2026 •

edited

Loading

coveralls commented Mar 14, 2026 •

edited

Loading