Discrepancy between Coverup-generated Tests and Pytest Execution

[keepingm]

Hi,

I am using Coverup to generate unit tests and running them with pytest to measure code coverage. However, I am encountering an issue where the number of tests generated by Coverup does not match the number of tests executed by pytest.

Here is the relevant part of the log:
`(mate_run) mgh@mgh:/dev/data/dataset/DevEval/python/hone$ coverup --package-dir src
Measuring coverage... 0.0%
Prompting gpt-4o for tests to increase coverage...
(in the following, G=good, F=failed, U=useless and R=retry)
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 17/17 [00:19<00:00, 1.12s/it, G=5, F=24, U=1, R=0, cost=$0.32]
Measuring coverage... 45.8%
Some modules seem to be missing: utils, hone
(mate_run) mgh@mgh:~/dev/data/dataset/DevEval/python/hone$ pytest --cov=src --cov-branch
/home/mgh/.conda/envs/mate_run/lib/python3.12/site-packages/pytest_isolate/plugin.py:70: RuntimeWarning: Failed to get GPU count using pynvml
warnings.warn("Failed to get GPU count using pynvml", RuntimeWarning)
=========================================================================================================================================== test session starts ===========================================================================================================================================
platform linux -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0
rootdir: /home/mgh/dev/data/dataset/DevEval/python/hone
configfile: pytest.ini
plugins: forked-1.6.0, metadata-3.1.1, isolate-0.0.13, repeat-0.9.4, anyio-4.12.1, cleanslate-1.0.7, cov-7.0.0, json-report-1.5.0
collected 9 items

tests/test_coverup_1.py . [ 11%]
tests/test_coverup_2.py . [ 22%]
tests/test_coverup_3.py ... [ 55%]
tests/test_coverup_4.py .. [ 77%]
tests/test_coverup_5.py .. [100%]

============================================================================================================================================= tests coverage ==============================================================================================================================================
____________________________________________________________________________________________________________________________ coverage: platform linux, python 3.12.12-final-0 _____________________________________________________________________________________________________________________________

Name Stmts Miss Branch BrPart Cover

src/hone.py 144 83 60 2 39%
src/utils/csv_utils.py 23 0 0 0 100%
src/utils/json_utils.py 9 9 2 0 0%

TOTAL 176 92 62 2 43%
============================================================================================================================================ 9 passed in 0.07s ============================================================================================================================================`

As shown, Coverup generates 17 tests, with 5 marked as "good", 24 as "failed", and 1 as "useless". However, when I run pytest, it only executes 9 tests.

There seems to be a discrepancy between the generated tests and the ones actually being executed by pytest. The failed and useless tests are not being executed, and I'm unsure why some of the generated tests are being ignored.

Could you provide any insight into why this might be happening? Is there a configuration issue, or is Coverup failing to generate executable tests for all cases? Any guidance on how to resolve this issue would be greatly appreciated!

Thanks!