Migrate to uv, fix Cython for ARM, support Python 3.10-3.14 (#597)

* Migrate to uv, fix Cython extensions for ARM and modern Python

- Replace pip/requirements.txt/setup.cfg with uv and modern pyproject.toml
- Fix Cython 3.x compatibility: move extern declarations from _dice.pxd
  into _dice.pyx with C name aliasing to resolve name collisions
- Add language_level=3 directive to all .pyx files
- Replace deprecated pkg_resources with importlib.metadata
- Fix zero-length filter divide-by-zero edge case in _dice_x86.py
- Update CI: use uv, test Python 3.10-3.14, add ARM Linux runner
  (ubuntu-24.04-arm), add QEMU aarch64 wheel builds
- Native C extensions now build and load on ARM (NEON) and x86

* Fix char signedness mismatch on ARM Linux

Use int8_t (signed char) for Cython memoryview types instead of char,
since Python's array('b') is always signed char but plain char is
unsigned on ARM Linux. Cast to const char* at C function call sites.

* Remove Azure Pipeline configuration

The Azure DevOps account is no longer active. CI has been migrated
to GitHub Actions.

* Add mypy typecheck job to GitHub Actions

Adds mypy and types-setuptools as dev dependencies and a typecheck
job to the unit testing workflow, replacing the Azure Pipeline
static checks stage.

* Bump version to 0.16.0, update docs and changelog

- Version bump to 0.16.0 reflecting breaking changes (dropped Python
  3.8/3.9, Cython 3 requirement, build system migration)
- Update README: remove Azure badge, refresh install instructions for
  uv, update benchmark numbers with ARM results, update test output
- Add changelog entry for 0.16.0
- Clean up MANIFEST.in: remove stale _entitymatcher reference, add
  _multiparty_solving_inner.cpp

Author: hardbyte

.azurePipeline/cibuildwheel_steps.yml

@@ -13,35 +13,42 @@ jobs:
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
-        os: [ubuntu-20.04, windows-2019, macos-11]
+        os: [ubuntu-latest, windows-latest, macos-latest]
 
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
+
+      # QEMU needed for building Linux ARM wheels on x86 runners
+      - name: Set up QEMU
+        if: runner.os == 'Linux'
+        uses: docker/setup-qemu-action@v3
+        with:
+          platforms: arm64
 
       - name: Build wheels
-        uses: pypa/cibuildwheel@v2.11.1
-        # to supply options, put them in 'env', like:
+        uses: pypa/cibuildwheel@v2.23
         env:
-         CIBW_SKIP: '*-win32 *i686'
+         CIBW_SKIP: '*-win32 *i686 *-musllinux_*'
+         CIBW_ARCHS_LINUX: auto aarch64
          CIBW_ARCHS_MACOS: x86_64 arm64
 
-      - uses: actions/upload-artifact@v3
+      - uses: actions/upload-artifact@v4
         with:
-          name: binary-wheels
+          name: wheels-${{ matrix.os }}
           path: ./wheelhouse/*.whl
 
   build_sdist:
     name: Build source distribution
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
 
       - name: Build sdist
         run: pipx run build --sdist
 
-      - uses: actions/upload-artifact@v3
+      - uses: actions/upload-artifact@v4
         with:
-          name: binary-wheels
+          name: sdist
           path: dist/*.tar.gz
 
   upload_pypi:
@@ -51,12 +58,18 @@ jobs:
     # upload to PyPI on every release
     if: github.event.release && github.event.action == 'published'
     steps:
-      - uses: actions/download-artifact@v3
+      - uses: actions/download-artifact@v4
+        with:
+          pattern: wheels-*
+          merge-multiple: true
+          path: dist
+
+      - uses: actions/download-artifact@v4
         with:
-          name: binary-wheels
+          name: sdist
           path: dist
 
-      - uses: pypa/gh-action-pypi-publish@v1.4.2
+      - uses: pypa/gh-action-pypi-publish@v1.12
         with:
           user: __token__
           password: ${{ secrets.PYPI_API_TOKEN }}

.azurePipeline/unittest_wheel_steps.yml

@@ -14,35 +14,49 @@ jobs:
     name: Unittest Anonlink ${{ matrix.python }} ${{ matrix.os }}
     runs-on: ${{ matrix.os }}
     strategy:
+      fail-fast: false
       matrix:
-        os: [macos-latest, windows-latest, ubuntu-20.04]
-        python: ["3.8", "3.9", "3.10", "3.11"]
-        
+        os: [ubuntu-latest, macos-latest, windows-latest]
+        python: ["3.10", "3.11", "3.12", "3.13", "3.14"]
+        include:
+          # Native ARM Linux runner
+          - os: ubuntu-24.04-arm
+            python: "3.13"
+
     steps:
-      - uses: actions/checkout@v2
-      - name: Set up Python ${{ matrix.python }}
-        uses: actions/setup-python@v2
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v4
         with:
-          python-version: ${{ matrix.python }}
-      - name: Get full Python version
-        id: full-python-version
-        shell: bash
-        run: echo ::set-output name=version::$(python -c "import sys; print('-'.join(str(v) for v in sys.version_info))")
-      - name: Install dependencies
-        shell: bash
-        run: |
-          python -m pip install --upgrade pip
-          python -m pip install -U -r requirements.txt
-      - name: Build and install anonlink
-        shell: bash
+          enable-cache: true
+      - name: Install dependencies and build
+        run: uv sync --python ${{ matrix.python }} --extra test
+      - name: Verify native extensions loaded
         run: |
-          python -m pip install -e .
+          uv run python -c "
+          from anonlink.similarities import dice_coefficient
+          from anonlink.solving import greedy_solve
+          print(f'Similarity: {dice_coefficient.__module__}.{dice_coefficient.__name__}')
+          print(f'Solver: {greedy_solve.__module__}.{greedy_solve.__name__}')
+          assert 'accelerated' in dice_coefficient.__name__, 'Native dice extension not loaded'
+          assert 'native' in greedy_solve.__name__, 'Native solver extension not loaded'
+          "
       - name: Test with pytest
-        run: |
-          pytest -q
+        run: uv run pytest -q
       - name: Test with extended size 100k
-        if: ${{ matrix.python == 3.10}}
-        env: 
+        if: matrix.python == '3.13' && matrix.os == 'ubuntu-latest'
+        env:
           INCLUDE_100K: 1
-        run: |
-          pytest -q
+        run: uv run pytest -q
+
+  typecheck:
+    name: Typecheck
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v4
+        with:
+          enable-cache: true
+      - name: Install dependencies
+        run: uv sync --python 3.13
+      - name: Run mypy
+        run: uv run mypy anonlink --ignore-missing-imports

.github/workflows/build_wheels.yml

@@ -1,3 +1,16 @@
+0.16.0
+======
+
+- Migrate build system to uv with modern pyproject.toml. Remove requirements.txt and setup.cfg.
+- Fix Cython extensions for Cython 3.x compatibility. Native C++ extensions now build on all platforms.
+- Add ARM (aarch64/Apple Silicon) support for native C++ extensions using NEON intrinsics.
+- Fix char signedness mismatch on ARM Linux (``const char`` vs ``signed char``).
+- Fix divide-by-zero with zero-length filters in accelerated Dice comparison.
+- Replace deprecated ``pkg_resources`` with ``importlib.metadata``.
+- Add Python 3.12, 3.13, 3.14 support. Drop Python 3.8, 3.9.
+- Migrate CI from Azure Pipelines to GitHub Actions with ARM Linux runner.
+- Add mypy typecheck job to CI.
+
 0.15.3
 ======
 

.github/workflows/unittests.yml

@@ -5,6 +5,6 @@ global-exclude __pycache__/*
 global-include *.pyx
 global-include *.pxd
 include anonlink/solving/_multiparty_solving_inner.h
+include anonlink/solving/_multiparty_solving_inner.cpp
 include anonlink/similarities/libpopcount.h
-include anonlink/_entitymatcher*
 include anonlink/similarities/dice.cpp

CHANGELOG.rst

@@ -1,7 +1,4 @@
 
-.. image:: https://dev.azure.com/data61/Anonlink/_apis/build/status/data61.anonlink?branchName=main
-    :target: https://dev.azure.com/data61/Anonlink/_build/latest?definitionId=3&branchName=main
-
 .. image:: https://github.com/data61/anonlink/actions/workflows/unittests.yml/badge.svg?branch=main
     :target: https://github.com/data61/anonlink/actions/workflows/unittests.yml
 
@@ -30,79 +27,81 @@ Install a precompiled wheel from PyPi::
 
     pip install anonlink
 
-Or (if your system has a C++ compiler) you can locally install from source::
+Or install from source using `uv <https://docs.astral.sh/uv/>`__::
 
-    pip install -r requirements.txt
-    pip install -e .
+    uv sync
 
 
 Benchmark
 ---------
 
-You can run the benchmark with:
+You can run the benchmark with ``python -m anonlink.benchmark`` (or ``uv run python -m anonlink.benchmark``).
+
+The following results were obtained on an Apple M1 (ARM):
 
 ::
 
     $ python -m anonlink.benchmark
     Anonlink benchmark -- see README for explanation
     ------------------------------------------------
+    using 'greedy_solve_native' as solver and 'dice_coefficient_accelerated' as similarity metric
 
     Threshold: 0.5, All results returned
     Size 1 | Size 2 | Comparisons      | Total Time (s)          | Throughput
            |        |        (match %) | (comparisons / matching)|  (1e6 cmp/s)
     -------+--------+------------------+-------------------------+-------------
-      1000 |   1000 |    1e6  (50.73%) |  0.762  (49.2% / 50.8%) |     2.669
-      2000 |   2000 |    4e6  (51.04%) |  3.696  (42.6% / 57.4%) |     2.540
-      3000 |   3000 |    9e6  (50.25%) |  8.121  (43.5% / 56.5%) |     2.548
-      4000 |   4000 |   16e6  (50.71%) | 15.560  (41.1% / 58.9%) |     2.504
+      1000 |   1000 |    1e6  (48.94%) |  0.201  (59.2% / 40.8%) |     8.426
+      2000 |   2000 |    4e6  (49.95%) |  1.344  (37.1% / 62.9%) |     8.025
+      3000 |   3000 |    9e6  (50.11%) |  3.204  (36.0% / 64.0%) |     7.799
+      4000 |   4000 |   16e6  (49.86%) |  5.873  (35.3% / 64.7%) |     7.725
 
     Threshold: 0.5, Top 100 matches per record returned
     Size 1 | Size 2 | Comparisons      | Total Time (s)          | Throughput
            |        |        (match %) | (comparisons / matching)|  (1e6 cmp/s)
     -------+--------+------------------+-------------------------+-------------
-      1000 |   1000 |    1e6  ( 6.86%) |  0.170  (85.9% / 14.1%) |     6.846
-      2000 |   2000 |    4e6  ( 3.22%) |  0.384  (82.9% / 17.1%) |    12.561
-      3000 |   3000 |    9e6  ( 2.09%) |  0.612  (81.6% / 18.4%) |    18.016
-      4000 |   4000 |   16e6  ( 1.52%) |  0.919  (78.7% / 21.3%) |    22.135
-      5000 |   5000 |   25e6  ( 1.18%) |  1.163  (80.8% / 19.2%) |    26.592
-      6000 |   6000 |   36e6  ( 0.97%) |  1.535  (75.4% / 24.6%) |    31.113
-      7000 |   7000 |   49e6  ( 0.82%) |  1.791  (80.6% / 19.4%) |    33.951
-      8000 |   8000 |   64e6  ( 0.71%) |  2.095  (81.5% / 18.5%) |    37.466
-      9000 |   9000 |   81e6  ( 0.63%) |  2.766  (72.5% / 27.5%) |    40.389
-     10000 |  10000 |  100e6  ( 0.56%) |  2.765  (81.7% / 18.3%) |    44.277
-     20000 |  20000 |  400e6  ( 0.27%) |  7.062  (86.2% / 13.8%) |    65.711
+      1000 |   1000 |    1e6  ( 6.79%) |  0.064  (84.6% / 15.4%) |    18.503
+      2000 |   2000 |    4e6  ( 3.23%) |  0.134  (85.9% / 14.1%) |    34.651
+      3000 |   3000 |    9e6  ( 2.07%) |  0.220  (86.7% / 13.3%) |    47.213
+      4000 |   4000 |   16e6  ( 1.53%) |  0.310  (86.2% / 13.8%) |    59.837
+      5000 |   5000 |   25e6  ( 1.18%) |  0.414  (85.7% / 14.3%) |    70.435
+      6000 |   6000 |   36e6  ( 0.98%) |  0.524  (86.7% / 13.3%) |    79.239
+      7000 |   7000 |   49e6  ( 0.83%) |  0.636  (86.3% / 13.7%) |    89.303
+      8000 |   8000 |   64e6  ( 0.71%) |  0.794  (82.8% / 17.2%) |    97.306
+      9000 |   9000 |   81e6  ( 0.64%) |  0.894  (86.1% / 13.9%) |   105.184
+     10000 |  10000 |  100e6  ( 0.56%) |  1.034  (86.8% / 13.2%) |   111.325
+     20000 |  20000 |  400e6  ( 0.27%) |  2.679  (87.3% / 12.7%) |   170.965
 
     Threshold: 0.7, All results returned
     Size 1 | Size 2 | Comparisons      | Total Time (s)          | Throughput
            |        |        (match %) | (comparisons / matching)|  (1e6 cmp/s)
     -------+--------+------------------+-------------------------+-------------
-      1000 |   1000 |    1e6  ( 0.01%) |  0.009  (99.0% /  1.0%) |   113.109
-      2000 |   2000 |    4e6  ( 0.01%) |  0.033  (98.7% /  1.3%) |   124.076
-      3000 |   3000 |    9e6  ( 0.01%) |  0.071  (99.1% /  0.9%) |   128.515
-      4000 |   4000 |   16e6  ( 0.01%) |  0.123  (99.0% /  1.0%) |   131.654
-      5000 |   5000 |   25e6  ( 0.01%) |  0.202  (99.1% /  0.9%) |   124.999
-      6000 |   6000 |   36e6  ( 0.01%) |  0.277  (99.0% /  1.0%) |   131.403
-      7000 |   7000 |   49e6  ( 0.01%) |  0.368  (98.9% /  1.1%) |   134.428
-      8000 |   8000 |   64e6  ( 0.01%) |  0.490  (99.0% /  1.0%) |   131.891
-      9000 |   9000 |   81e6  ( 0.01%) |  0.608  (99.0% /  1.0%) |   134.564
-     10000 |  10000 |  100e6  ( 0.01%) |  0.753  (99.0% /  1.0%) |   134.105
-     20000 |  20000 |  400e6  ( 0.01%) |  2.905  (98.8% /  1.2%) |   139.294
+      1000 |   1000 |    1e6  ( 0.01%) |  0.003  (99.0% /  1.0%) |   312.191
+      2000 |   2000 |    4e6  ( 0.01%) |  0.011  (98.1% /  1.9%) |   356.224
+      3000 |   3000 |    9e6  ( 0.01%) |  0.026  (99.0% /  1.0%) |   347.011
+      4000 |   4000 |   16e6  ( 0.01%) |  0.045  (99.0% /  1.0%) |   358.368
+      5000 |   5000 |   25e6  ( 0.01%) |  0.071  (98.9% /  1.1%) |   356.423
+      6000 |   6000 |   36e6  ( 0.01%) |  0.098  (98.9% /  1.1%) |   370.163
+      7000 |   7000 |   49e6  ( 0.01%) |  0.133  (98.9% /  1.1%) |   373.096
+      8000 |   8000 |   64e6  ( 0.01%) |  0.172  (98.9% /  1.1%) |   377.015
+      9000 |   9000 |   81e6  ( 0.01%) |  0.218  (98.9% /  1.1%) |   374.817
+     10000 |  10000 |  100e6  ( 0.01%) |  0.272  (99.0% /  1.0%) |   371.551
+     20000 |  20000 |  400e6  ( 0.01%) |  1.053  (99.0% /  1.0%) |   383.731
 
     Threshold: 0.7, Top 100 matches per record returned
     Size 1 | Size 2 | Comparisons      | Total Time (s)          | Throughput
            |        |        (match %) | (comparisons / matching)|  (1e6 cmp/s)
     -------+--------+------------------+-------------------------+-------------
-      1000 |   1000 |    1e6  ( 0.01%) |  0.009  (99.0% /  1.0%) |   111.640
-      2000 |   2000 |    4e6  ( 0.01%) |  0.033  (98.6% /  1.4%) |   122.060
-      3000 |   3000 |    9e6  ( 0.01%) |  0.074  (99.1% /  0.9%) |   123.237
-      4000 |   4000 |   16e6  ( 0.01%) |  0.124  (99.0% /  1.0%) |   130.204
-      5000 |   5000 |   25e6  ( 0.01%) |  0.208  (99.1% /  0.9%) |   121.351
-      6000 |   6000 |   36e6  ( 0.01%) |  0.275  (99.0% /  1.0%) |   132.186
-      7000 |   7000 |   49e6  ( 0.01%) |  0.373  (99.0% /  1.0%) |   132.650
-      8000 |   8000 |   64e6  ( 0.01%) |  0.496  (99.1% /  0.9%) |   130.125
-      9000 |   9000 |   81e6  ( 0.01%) |  0.614  (99.0% /  1.0%) |   133.216
-     10000 |  10000 |  100e6  ( 0.01%) |  0.775  (99.1% /  0.9%) |   130.230
-     20000 |  20000 |  400e6  ( 0.01%) |  2.939  (98.9% /  1.1%) |   137.574
+      1000 |   1000 |    1e6  ( 0.01%) |  0.003  (98.9% /  1.1%) |   314.762
+      2000 |   2000 |    4e6  ( 0.01%) |  0.011  (98.7% /  1.3%) |   357.730
+      3000 |   3000 |    9e6  ( 0.01%) |  0.024  (98.9% /  1.1%) |   372.850
+      4000 |   4000 |   16e6  ( 0.01%) |  0.044  (98.9% /  1.1%) |   363.783
+      5000 |   5000 |   25e6  ( 0.01%) |  0.066  (98.9% /  1.1%) |   382.863
+      6000 |   6000 |   36e6  ( 0.01%) |  0.095  (98.9% /  1.1%) |   383.880
+      7000 |   7000 |   49e6  ( 0.01%) |  0.128  (98.9% /  1.1%) |   385.778
+      8000 |   8000 |   64e6  ( 0.01%) |  0.171  (98.9% /  1.1%) |   377.762
+      9000 |   9000 |   81e6  ( 0.01%) |  0.210  (99.0% /  1.0%) |   389.182
+     10000 |  10000 |  100e6  ( 0.01%) |  0.275  (99.0% /  1.0%) |   367.465
+     20000 |  20000 |  400e6  ( 0.01%) |  1.040  (99.0% /  1.0%) |   388.491
 
 
 The tables are interpreted as follows. Each table measures the throughput
@@ -135,24 +134,10 @@ matrix, which will be approximately `#comparisons * match% / 100`.
 Tests
 =====
 
-Run unit tests with `pytest`:
-
-::
-
-    $ pytest
-    ====================================== test session starts ======================================
-    platform linux -- Python 3.6.4, pytest-3.2.5, py-1.4.34, pluggy-0.4.0
-    rootdir: /home/hlaw/src/n1-anonlink, inifile:
-    collected 71 items
-
-    tests/test_benchmark.py ...
-    tests/test_bloommatcher.py ..............
-    tests/test_e2e.py .............ss....
-    tests/test_matcher.py ..x.....x......x....x..
-    tests/test_similarity.py .........
-    tests/test_util.py ...
+Run unit tests with ``pytest``::
 
-    ======================== 65 passed, 2 skipped, 4 xfailed in 4.01 seconds ========================
+    $ uv run pytest -q
+    9051 passed, 2 skipped in 12.87s
 
 To enable slightly larger tests add the following environment variables: