feat: use an opinionated project structure for the testing code (#172)

* add Makefile and switch to pyproject.toml

* exclude package info in .gitignore

* add 'clean' target

* format the testing code

* fix type error in conftest.py

* remove requirements.txt

* switch CI to use new structure (no uv)

* add missing working dir

* add initial HACKING.md

* pytest keyword can just be the rule number

* improve robustness of Makefile

* update intro guidance and cross-links

* Add small section to README

* add guidance for code maintainers

* adjust HACKING.md

* bump action versions in pytest workflow

* add workflow to run the linter

* exclude more cache dirs in .gitignore

* reformat updated code after merging from upstream

* change duplicate function name in testing code

Author: dwilding

.github/workflows/pytest.yml

@@ -18,22 +18,26 @@ jobs:
         python-version: [ '3.10', '3.12' ] 
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@v6
 
       - name: Setup Python
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@v6
         with:
           python-version: ${{ matrix.python-version }}
           cache: 'pip'
 
-      - name: Install Python dependencies
+      - name: Create an environment with Python dependencies
+        working-directory: tests
         run: |
-          pip install -r tests/requirements.txt
+          python3 -m venv .venv
+          . .venv/bin/activate
+          pip install -e .
 
       - name: Run pytest suite
+        working-directory: tests
         env:
           # Set to 1 once manifest covers ALL style rules to enforce coverage.
           VALE_ENFORCE_COVERAGE: 0
         run: |
-          python -m pytest -q || python -m pytest -vv
-
+          . .venv/bin/activate
+          make run

.github/workflows/testing-code-lint.yml

@@ -0,0 +1,25 @@
+name: Lint the testing code
+
+on:
+  pull_request:
+    paths: [ tests/** ]
+    branches: [ main ]
+  # Allow manual trigger
+  workflow_dispatch: {}
+
+permissions:
+  contents: read
+
+jobs:
+  testing-code-lint:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+
+      - name: Lint the testing code
+        working-directory: tests
+        run: make lint

.gitignore

@@ -1,7 +1,8 @@
 node_modules
 coverage
 build/
+*.egg-info
 .venv
 .vscode
-.pytest_cache/
-tests/__pycache__/
+.*_cache/
+tests/__pycache__/

HACKING.md

@@ -0,0 +1,79 @@
+In this guide:
+
+- [Guidance for maintainers of the rules](#guidance-for-maintainers-of-the-rules)
+    - [Add test cases for a rule](#add-test-cases-for-a-rule)
+    - [Run all test cases](#run-the-test-cases)
+    - [Run selected test cases](#run-selected-test-cases)
+- [Guidance for maintainers of the testing code](#guidance-for-maintainers-of-the-testing-code)
+
+# Guidance for maintainers of the rules
+
+See first: [Introduction to Vale rule development](getting-started.md)
+
+## Add test cases for a rule
+
+Make sure that the rule has suitable test cases in [tests/data/manifest.yml](tests/data/manifest.yml).
+
+## Run all test cases
+
+We recommend that you first install [uv](https://docs.astral.sh/uv/). To install uv on Ubuntu:
+
+```
+sudo snap install astral-uv --classic
+```
+
+To run the test cases for every rule:
+
+- **If uv is installed**
+
+    ```text
+    make -C tests run
+    ```
+
+- **If uv is not installed**
+
+    ```text
+    cd tests
+    python3 -m venv .venv
+    . .venv/bin/activate
+    pip install -e .
+    make run
+    ```
+
+## Run selected test cases
+
+Behind the scenes, we're using [pytest](https://docs.pytest.org/en/stable/) to run each test case.
+
+To run the test cases for a particular rule, such as `003-Ubuntu-names-versions`:
+
+- **If uv is installed**
+
+    ```text
+    uv run --directory tests pytest -vv -k 003
+    ```
+
+- **If uv is not installed**
+
+    ```text
+    # (Provided the working dir is 'tests' and the virtual environment is active)
+    pytest -vv -k 003
+    ```
+
+# Guidance for maintainers of the testing code
+
+The code in the `tests` directory uses Python with [pytest](https://docs.pytest.org/en/stable/). We require the code to be well-formatted and pass static checks.
+
+Our tools of choice are:
+
+- [ruff](https://docs.astral.sh/ruff/) for formatting and checking code conventions
+- [pyright](https://microsoft.github.io/pyright/) for checking types
+
+If you've already installed ruff, you should be able to use it in the `tests` directory with no trouble. pyright is less straightforward, as it needs to be run in a virtual environment that contains the testing code's dependencies.
+
+Instead of manually running these tools, we strongly recommend that you install [uv](https://docs.astral.sh/uv/) and use `make` in the `tests` directory.
+
+| Command       | Purpose                                                       |
+|---------------|---------------------------------------------------------------|
+| `make format` | Use ruff to format the testing code                           |
+| `make lint`   | Use ruff to check code conventions and pyright to check types |
+| `make run`    | Use pytest to run the test cases for every rule               |

README.md

@@ -34,7 +34,11 @@ Anyone is welcome to submit a PR to add additional rules. However, no additions
 
 For a reference on rule syntax, see the Vale [documentation on Styles][Vale styles].
 
-If you are completely new to developing Vale rules, see this [introductory guide](https://github.com/canonical/documentation-style-guide/blob/8c7fee862b2258c692439ef430198e393bdc30c4/getting-started.md). 
+If you are completely new to developing Vale rules, see this [introductory guide](getting-started.md). 
+
+### Testing the rules
+
+This repo includes automated tests of the rules, which you can run locally, and which run in CI. See [Guidance for maintainers of the rules](HACKING.md).
 
 ### Using the rules
 

getting-started.md

@@ -6,11 +6,14 @@ The goal of this guide is to curate existing resources and provide pointers for
 * [Install the Vale CLI tool](#install-the-vale-cli-tool)
 * [Develop a rule](#develop-a-rule)
   * [Basics](#basics)
+  * [Add test cases](#add-test-cases)
 * [Regex](#regex)
-* [Test rules](#test-rules)
+* [Manually test the rules](#manually-test-the-rules)
   * [Test all rules](#test-all-rules)
   * [Test a specific rule](#test-a-specific-rule)
+  * [Resources](#resources-3)
 * [Troubleshooting](#troubleshooting)
+  * [Conditionally ignoring rules](#conditionally-ignoring-rules)
 
 
 ## Install the Vale CLI tool
@@ -67,6 +70,10 @@ There are several other extensions, each with a particular set of keys to fine-t
 
 You can now use the [documentation](https://vale.sh/docs/topics/styles/#extension-points) to find the extension point that best suits your rule, and see what parameters it requires.
 
+### Add test cases
+
+This repo includes automated tests of the rules, which you can run locally, and which run in CI. As you develop your rule, you should add test cases that exercise your rule. See [Guidance for maintainers of the rules](HACKING.md).
+
 ### Resources
 
 * Rule examples: 
@@ -96,9 +103,12 @@ The single quotes indicate the beginning and end of the regex, and the `.*?` qua
 - [Regex editor](https://regex101.com/): Online tool that helps you compose and test your expressions. I like how it highlights capture groups in different colors.
 - Before composing a complicated regex, remember to check for [existing rules](#resources) that might have done this already. 
 
-## Test rules
+## Manually test the rules
+
+> [!NOTE]
+> This repo includes automated tests of the rules, which you can run locally, and which run in CI. See [Guidance for maintainers of the rules](HACKING.md).
 
-Create a file in the root directory of the repository and fill it with text that you expect your rule to catch. It can be in any text format (`.txt`, `.md`, `.rst`...)
+To manually test your rule, create a file in the root directory of the repository and fill it with text that you expect your rule to catch. It can be in any text format (`.txt`, `.md`, `.rst`...)
 
 Besides the text file, the `vale` command needs a `vale.ini` config file. This file already exists in the root folder `documentation-style-guide/`. Vale will automatically find the `vale.ini` if you run the command in the same directory.
 

tests/Makefile

@@ -0,0 +1,36 @@
+# If we can, use uv (https://docs.astral.sh/uv/) to manage a virtual environment and run commands.
+# Otherwise, require an active virtual environment and directly run commands.
+uv := $(shell command -v uv 2>/dev/null)
+prerun :=
+runner :=
+ifneq ($(strip $(uv)),)
+runner := uv run
+else ifndef VIRTUAL_ENV
+prerun = $(error You have not activated a virtual environment)
+endif
+
+.DEFAULT_GOAL := help
+
+.PHONY: run format lint clean help
+
+run:
+	$(prerun)
+	$(runner) pytest -vv
+
+format:
+	$(prerun)
+	$(runner) ruff format
+
+lint:
+	$(prerun)
+	$(runner) ruff check
+	$(runner) ruff format --diff
+	$(runner) pyright
+
+clean:
+	@rm -rf *.egg-info .venv __pycache__ .pytest_cache .ruff_cache
+
+help:
+	@echo "Test the Vale rules                     make run"
+	@echo "Format the testing code                 make format"
+	@echo "Check the quality of the testing code   make lint"

tests/conftest.py

@@ -4,6 +4,7 @@
 Extensible: add new rules/cases via `tests/data/manifest.yml`.
 Set environment variable VALE_ENFORCE_COVERAGE=1 to enforce full rule coverage.
 """
+
 from __future__ import annotations
 
 import json
@@ -33,13 +34,15 @@ class ExpectedResult(BaseModel):
     severity: optional; if provided we assert all findings share this severity.
     message_regex: optional regex that all messages must match.
     """
+
     triggers: List[str] = Field(default_factory=list)
     severity: str | None = None
     message_regex: str | None = None
 
 
 class TestCase(BaseModel):
     """A single test case for a rule."""
+
     id: str
     filetypes: List[str]
     content: str
@@ -48,12 +51,14 @@ class TestCase(BaseModel):
 
 class RuleDefinition(BaseModel):
     """All test cases for a single Vale rule."""
+
     name: str
     cases: List[TestCase]
 
 
 class Manifest(BaseModel):
     """Root manifest model loaded from YAML."""
+
     rules: List[RuleDefinition]
 
     @classmethod
@@ -67,7 +72,7 @@ def from_yaml_dict(cls, data: dict) -> "Manifest":
         """
         rules_dict = data.get("rules", {})
         rules = [
-            {"name": rule_name, "cases": rule_data.get("cases", [])}
+            RuleDefinition(name=rule_name, cases=rule_data.get("cases", []))
             for rule_name, rule_data in rules_dict.items()
         ]
         return cls(rules=rules)
@@ -93,11 +98,7 @@ def _load_manifest() -> Manifest:
 def _discover_rule_ids() -> List[str]:
     if not os.path.isdir(STYLES_DIR):
         return []
-    return sorted(
-        f.rsplit(".", 1)[0]
-        for f in os.listdir(STYLES_DIR)
-        if f.endswith(".yml")
-    )
+    return sorted(f.rsplit(".", 1)[0] for f in os.listdir(STYLES_DIR) if f.endswith(".yml"))
 
 
 @pytest.fixture(scope="session")
@@ -176,7 +177,7 @@ def _run_vale(target_file: str, rule_id: str) -> List[ValeResult]:
     results: List[ValeResult] = []
     for h in raw_hits:
         span = None
-        if 'Span' in h and isinstance(h.get('Span'), list):
+        if "Span" in h and isinstance(h.get("Span"), list):
             span_list = h.get("Span")
             if isinstance(span_list, list) and len(span_list) == 2:
                 span = (span_list[0], span_list[1])
@@ -195,12 +196,15 @@ def _run_vale(target_file: str, rule_id: str) -> List[ValeResult]:
 def _iter_cases(manifest: Manifest) -> Iterable[Tuple[str, TestCase]]:
     return manifest.iter_cases()
 
+
 def _idfn(param):
     rule, case = param
     return f"{rule}::{case.id}"
 
 
 _ALL_CASES = list(_iter_cases(_load_manifest()))
+
+
 @pytest.fixture(params=_ALL_CASES, ids=_idfn)
 def case_definition(request):
     """Parametrized (rule_id, TestCase) tuple for each case in the manifest."""
@@ -250,10 +254,11 @@ def _assert_case(rule_id: str, case: TestCase, results: List[ValeResult]):
 
     # For tokens with multiplicity in EXPECTED, ensure counts are met.
     from collections import Counter
+
     exp_counts = Counter(expected_list)
     act_counts = Counter(actual_list)
     multiplicity_failures = [
-        f"{tok} (expected >= {exp_counts[tok]}, got {act_counts.get(tok,0)})"
+        f"{tok} (expected >= {exp_counts[tok]}, got {act_counts.get(tok, 0)})"
         for tok in exp_counts
         if exp_counts[tok] > 1 and act_counts.get(tok, 0) < exp_counts[tok]
     ]
@@ -278,10 +283,12 @@ def _assert_case(rule_id: str, case: TestCase, results: List[ValeResult]):
 def vale_runner():
     return _run_vale
 
+
 @pytest.fixture
 def assert_case():
     return _assert_case
 
+
 @pytest.fixture(scope="session")
 def manifest_schema(manifest: Manifest) -> Dict[str, Any]:
     """Provide the JSON schema for the Manifest (Draft generation by Pydantic)."""
@@ -306,6 +313,7 @@ def _test_markdown(markdown_input: str):
         check=True,
     )
 
+
 @pytest.fixture
 def test_markdown():
     return _test_markdown

tests/pyproject.toml

@@ -0,0 +1,21 @@
+[project]
+name = "test_style_guide"
+version = "0.0.1"  # The version number isn't meaningful unless we publish the package.
+requires-python = ">=3.10"
+dependencies = [
+    "docutils",
+    "pydantic>=2.0",
+    "pytest",
+    "PyYAML",
+    "rst2html",
+    "vale==3.13.0.0",
+]
+
+[dependency-groups]
+dev = [
+    "pyright",
+    "ruff",
+]
+
+[tool.ruff]
+line-length = 99