Skip to content

Commit 65600b3

Browse files
dsblankCopilot
dsblank
and
Copilot
authored
Add tests and checks (#10)
* Windows compatibility * Use Path constructs instead * Use parts not split * Fix PEP 8 violations * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix tests, imports * Add run tests actions * Added requirements.txt, pytest instructions and ini * Added requirements, update to Python 3.12 * Use get_path_parts * Update cometx/cli/reproduce.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add pre-commit configuration with flake8, mypy, black, and other tools * Remove security checks (bandit) from pre-commit configuration * Restore black and flake8 to automatic mode * Update tests/test_utils.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/test_utils.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/test_utils.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/test_utils.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/test_utils.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Rewrote get_path_parts() * Added some comments * Don't line-break strings * Update cometx/framework/wandb.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update cometx/utils.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update cometx/utils.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update cometx/framework/wandb.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Move import * Revert compatible changes * Revert compatible changes * Update .pre-commit-config.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update .pre-commit-config.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 parent 36e472d commit 65600b3

File tree

14 files changed

+379
-8
lines changed

14 files changed

+379
-8
lines changed

‎.flake8‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
[flake8]
2+
max-line-length = 88
3+
extend-ignore = E203
4+
exclude =
5+
.git,
6+
__pycache__,
7+
.venv,
8+
venv,
9+
.env,
10+
.eggs,
11+
*.egg,
12+
.pytest_cache,
13+
.coverage,
14+
build,
15+
dist,
16+
*.egg-info
17+
per-file-ignores =
18+
# Ignore missing docstrings in __init__.py files
19+
__init__.py:D104
20+
# Ignore unused imports in __init__.py files (often used for exports)
21+
__init__.py:F401

‎.github/workflows/test.yml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
name: Run Pytest Tests
2+
3+
on:
4+
pull_request:
5+
branches: [main, master]
6+
push:
7+
branches: [main, master]
8+
9+
jobs:
10+
test:
11+
runs-on: ubuntu-latest
12+
13+
steps:
14+
- name: Checkout code
15+
uses: actions/checkout@v4
16+
17+
- name: Set up Python
18+
uses: actions/setup-python@v5
19+
with:
20+
python-version: '3.12' # Latest stable Python version
21+
22+
- name: Install dependencies
23+
run: |
24+
python -m pip install --upgrade pip
25+
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
26+
27+
- name: Run tests
28+
run: |
29+
pip install pytest
30+
pytest

‎.pre-commit-config.yaml‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
repos:
2+
# Code formatting
3+
- repo: https://github.com/psf/black
4+
rev: 24.1.1
5+
hooks:
6+
- id: black
7+
language_version: python3
8+
args: [--line-length=88]
9+
10+
# Import sorting
11+
- repo: https://github.com/pycqa/isort
12+
rev: 5.13.2
13+
hooks:
14+
- id: isort
15+
args: [--profile=black, --line-length=88]
16+
17+
# Linting
18+
- repo: https://github.com/pycqa/flake8
19+
rev: 7.0.0
20+
hooks:
21+
- id: flake8
22+
args: [--max-line-length=88, --extend-ignore=E203, --extend-ignore=E501]
23+
24+
# Type checking
25+
- repo: https://github.com/pre-commit/mirrors-mypy
26+
rev: v1.8.0
27+
hooks:
28+
- id: mypy
29+
additional_dependencies: [types-requests, types-six]
30+
args: [--ignore-missing-imports]
31+
stages: [manual]
32+
33+
34+
35+
# General Python checks
36+
- repo: https://github.com/pycqa/pydocstyle
37+
rev: 6.3.0
38+
hooks:
39+
- id: pydocstyle
40+
args: [--convention=google]
41+
stages: [manual]
42+
43+
# Check for merge conflicts
44+
- repo: https://github.com/pre-commit/pre-commit-hooks
45+
rev: v4.5.0
46+
hooks:
47+
- id: check-merge-conflict
48+
- id: check-yaml
49+
- id: check-json
50+
- id: check-added-large-files
51+
- id: check-case-conflict
52+
- id: check-docstring-first
53+
- id: check-ast
54+
- id: check-symlinks
55+
- id: debug-statements
56+
- id: end-of-file-fixer
57+
- id: trailing-whitespace
58+
- id: check-toml
59+
- id: check-vcs-permalinks
60+
- id: mixed-line-ending
61+
- id: requirements-txt-fixer

‎MIGRATIONS.md‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ from the source, and then copy it to the destination Comet instance.
3434

3535
### Downloading Data
3636

37-
The first step in a migration is to use `cometx download`.
37+
The first step in a migration is to use `cometx download`.
3838
For example, to download from an existing Comet installation:
3939

4040
```shell
@@ -115,7 +115,7 @@ cometx download --from wandb stacey/yolo-drive/1dwb18ia
115115

116116
This will download the WandB run: https://wandb.ai/stacey/yolo-drive/runs/1dwb18ia
117117

118-
After download, the following `copy` commands will be relevant.
118+
After download, the following `copy` commands will be relevant.
119119

120120
#### Additional Download Flags
121121

@@ -167,4 +167,3 @@ COMET_URL_OVERRIDE=http://comet.b.com/clientlib \
167167
COMET_API_KEY=B-KEY \
168168
cometx copy <WORKSPACE> <NEW-WORKSPACE>
169169
```
170-

‎PRE_COMMIT_SETUP.md‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
# Pre-commit Setup for CometX
2+
3+
This document describes the pre-commit setup for the CometX repository.
4+
5+
## What's Included
6+
7+
The pre-commit configuration includes the following hooks:
8+
9+
### Automatic Hooks (run on every commit)
10+
- **Black**: Code formatting (88 character line length)
11+
- **isort**: Import sorting (compatible with Black)
12+
- **flake8**: Linting (with Black-compatible settings)
13+
- **pre-commit hooks**: Various general checks:
14+
- Merge conflict detection
15+
- YAML/JSON validation
16+
- Large file detection
17+
- Case conflict detection
18+
- Docstring placement
19+
- AST validation
20+
- Debug statement detection
21+
- End-of-file fixing
22+
- Trailing whitespace removal
23+
- TOML validation
24+
- VCS permalink checking
25+
- Mixed line ending detection
26+
- Requirements.txt sorting
27+
28+
### Manual Hooks (run only when explicitly called)
29+
- **mypy**: Type checking (basic mode)
30+
- **pydocstyle**: Documentation style checking
31+
32+
## Setup Instructions
33+
34+
1. Install development dependencies:
35+
```bash
36+
pip install -r requirements-dev.txt
37+
```
38+
39+
2. Install pre-commit hooks:
40+
```bash
41+
pre-commit install
42+
```
43+
44+
3. (Optional) Run the setup script:
45+
```bash
46+
./setup-pre-commit.sh
47+
```
48+
49+
## Usage
50+
51+
### Automatic Checks
52+
The automatic hooks will run every time you commit. If any fail, the commit will be blocked until the issues are fixed.
53+
54+
### Manual Checks
55+
To run the manual hooks:
56+
57+
```bash
58+
# Run all manual hooks
59+
pre-commit run --all-files --hook-stage manual
60+
61+
# Run specific manual hooks
62+
pre-commit run mypy --all-files
63+
pre-commit run pydocstyle --all-files
64+
```
65+
66+
### Running Specific Hooks
67+
```bash
68+
# Run all hooks on all files
69+
pre-commit run --all-files
70+
71+
# Run specific hooks
72+
pre-commit run black --all-files
73+
pre-commit run flake8 --all-files
74+
pre-commit run isort --all-files
75+
```
76+
77+
## Configuration Files
78+
79+
- `.pre-commit-config.yaml`: Main pre-commit configuration
80+
- `.flake8`: Flake8 linting configuration
81+
- `pyproject.toml`: Black, isort, and mypy configuration
82+
83+
## Current Status
84+
85+
✅ **Working Hooks:**
86+
- Black (code formatting)
87+
- isort (import sorting)
88+
- flake8 (linting)
89+
- All pre-commit general hooks
90+
91+
⚠️ **Manual Hooks:**
92+
- mypy (type checking) - set to manual due to existing type issues
93+
- pydocstyle (documentation) - set to manual due to extensive docstring issues
94+
95+
## Known Issues
96+
97+
1. **Line Length Violations**: Some files exceed the 88-character line limit
98+
2. **Type Annotations**: Some files have syntax errors in type annotations
99+
3. **Documentation**: Many functions and classes lack proper docstrings
100+
4. **Code Quality**: Focus on formatting, imports, and linting
101+
102+
## Next Steps
103+
104+
To improve the code quality:
105+
106+
1. **Fix line length violations**: Run `pre-commit run black --all-files` to auto-format
107+
2. **Fix type annotations**: Address mypy errors in `cometx/framework/comet/download_manager.py`
108+
3. **Add docstrings**: Gradually add proper documentation to functions and classes
109+
4. **Improve code quality**: Focus on formatting, imports, and linting issues
110+
111+
## Disabling Hooks
112+
113+
If you need to bypass pre-commit hooks temporarily:
114+
115+
```bash
116+
git commit --no-verify -m "Your commit message"
117+
```
118+
119+
**Note**: This should only be used in emergencies, not as a regular practice.

‎cometx/cli/copy.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,6 @@
7575

7676
from comet_ml import APIExperiment, Artifact, Experiment, OfflineExperiment
7777
from comet_ml._typing import TemporaryFilePath
78-
from comet_ml.utils import compress_git_patch
7978
from comet_ml.file_uploader import GitPatchUploadProcessor
8079
from comet_ml.messages import (
8180
GitMetadataMessage,
@@ -86,6 +85,7 @@
8685
SystemDetailsMessage,
8786
)
8887
from comet_ml.offline_utils import write_experiment_meta_file
88+
from comet_ml.utils import compress_git_patch
8989

9090
from ..api import API
9191
from ..utils import remove_extra_slashes

‎cometx/utils.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -111,8 +111,9 @@ def get_query_experiments(api, query_string, workspace, project_name):
111111
return api.query(workspace, project_name, query)
112112

113113

114-
def download_url(url, output_filename, width=None, height=None, timeout=5,
115-
headless=False):
114+
def download_url(
115+
url, output_filename, width=None, height=None, timeout=5, headless=False
116+
):
116117
"""
117118
Args:
118119
url: (str) the URL to download

‎pyproject.toml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
[tool.mypy]
2+
python_version = "3.8"
3+
warn_return_any = true
4+
warn_unused_configs = true
5+
disallow_untyped_defs = true
6+
disallow_incomplete_defs = true
7+
check_untyped_defs = true
8+
disallow_untyped_decorators = true
9+
no_implicit_optional = true
10+
warn_redundant_casts = true
11+
warn_unused_ignores = true
12+
warn_no_return = true
13+
warn_unreachable = true
14+
strict_equality = true
15+
show_error_codes = true
16+
17+
[[tool.mypy.overrides]]
18+
module = [
19+
"tests.*",
20+
"extras.*"
21+
]
22+
ignore_missing_imports = true
23+
disallow_untyped_defs = false
24+
25+
[tool.black]
26+
line-length = 88
27+
target-version = ['py38']
28+
include = '\.pyi?$'
29+
extend-exclude = '''
30+
/(
31+
# directories
32+
\.eggs
33+
| \.git
34+
| \.hg
35+
| \.mypy_cache
36+
| \.tox
37+
| \.venv
38+
| build
39+
| dist
40+
)/
41+
'''
42+
43+
[tool.isort]
44+
profile = "black"
45+
line_length = 88
46+
multi_line_output = 3
47+
include_trailing_comma = true
48+
force_grid_wrap = 0
49+
use_parentheses = true
50+
ensure_newline_before_comments = true
51+
52+
[tool.pydocstyle]
53+
convention = "google"

‎pytest.ini‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
[pytest]
2+
asyncio_default_fixture_loop_scope = function
3+
filterwarnings =
4+
ignore::DeprecationWarning:google.protobuf.*
5+
ignore::DeprecationWarning:jupyter_client.*
6+
ignore::DeprecationWarning:sentry_sdk.*

‎requirements-dev.txt‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Development dependencies
2+
-r requirements.txt
3+
4+
# Additional development tools
5+
black>=23.0.0
6+
flake8>=6.0.0
7+
8+
# Additional tools for pre-commit hooks
9+
isort>=5.13.0
10+
mypy>=1.0.0
11+
pre-commit>=3.0.0
12+
pydocstyle>=6.3.0
13+
tox>=4.0.0
14+
15+
# Type stubs for mypy
16+
types-requests
17+
types-six

0 commit comments

Comments
 (0)