Refactor into module architecture for easier expansion and maintenance (#59)

* Tidy up directory layout
* Switched builds to Podman
* Add run time output for svn log and fetch commands
* Dedupe git config file
* Fix the git config file newline issue
* Increase retry for psutils

Author: marcleblanc2

…ithub/workflows/docker-image-cleanup.yml .github/workflows/cleanup-images.yml.github/workflows/docker-image-cleanup.yml renamed to .github/workflows/cleanup-images.yml .github/workflows/docker-image-cleanup.yml renamed to .github/workflows/cleanup-images.yml

@@ -1,12 +1,8 @@
-name: Build and Push Docker Image to GitHub Container Registry
+name: Docker build and push to GitHub Container Registry
 
 on:
-  workflow_dispatch: # Creates button in web UI to run the workflow manually, shouldn't be needed
-  push:
-    tags:
-      - '**'
-    branches:
-      - main
+  workflow_dispatch: # Creates button in web UI to run the workflow manually
+  push: # All pushes
   pull_request:
     types:
       - opened
@@ -47,9 +43,9 @@ jobs:
       - name: Build and push
         uses: docker/build-push-action@v6
         with:
-          context: repo-converter/build
+          context: .
           platforms: linux/amd64,linux/arm64
-          file: repo-converter/build/Dockerfile
+          file: build/Dockerfile
           push: true
           sbom: true
           tags: |

.github/workflows/docker-image-build.yml .github/workflows/docker-build.yml.github/workflows/docker-image-build.yml renamed to .github/workflows/docker-build.yml

@@ -0,0 +1,42 @@
+name: Podman build and push to GitHub Container Registry
+
+on:
+  workflow_dispatch: # Creates button in web UI to run the workflow manually
+  push: # All pushes
+  pull_request:
+    types:
+      - opened
+      - reopened
+      - edited
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set env vars
+        run: |
+          BUILD_BRANCH="$(git rev-parse --abbrev-ref HEAD)"
+          BUILD_COMMIT="$(git rev-parse --short HEAD)"
+          BUILD_DIRTY="$(git diff --quiet && echo 'False' || echo 'True')"
+          BUILD_DATE="$(date -u +'%Y-%m-%d %H:%M:%S UTC')"
+          BUILD_TAG="$(git tag --points-at HEAD)"
+          ENV_FILE="build/.env"
+          {
+              echo "BUILD_BRANCH=${BUILD_BRANCH}"
+              echo "BUILD_COMMIT=${BUILD_COMMIT}"
+              echo "BUILD_DATE=${BUILD_DATE}"
+              echo "BUILD_DIRTY=${BUILD_DIRTY}"
+              echo "BUILD_TAG=${BUILD_TAG}"
+          } > "$ENV_FILE"
+
+      - name: Log in to container registry
+        run: echo "${{ secrets.GITHUB_TOKEN }}" | podman login -u "${{ github.repository_owner }}" --password-stdin ghcr.io
+
+      - name: Build and push image
+        run: |
+          podman build -f build/Dockerfile --format docker --platform linux/amd64 -t ghcr.io/sourcegraph/repo-converter:latest .
+          podman push ghcr.io/sourcegraph/repo-converter:latest

.github/workflows/podman-build.yml

@@ -1,13 +1,13 @@
 # Sourcegraph
+*.code-workspace
 config/cloud-agent-config.yaml
 config/cloud-agent-service-account-key.json
 config/config.yaml
-config/service-account-key.json
 config/repos-to-convert.yaml
-src-serve-root
-svn-repo-stats/*.csv
-svn-repo-stats/repos.txt
-svn-repo-stats/tmp-repo-metadata/
+config/service-account-key.json
+dev/stats/repos.txt
+logs/
+src-serve-root/
 
 # Byte-compiled / optimized / DLL files
 __pycache__/
@@ -18,23 +18,23 @@ __pycache__/
 *.so
 
 # Distribution / packaging
+.eggs/
+.installed.cfg
 .Python
+*.egg
+*.egg-info/
 develop-eggs/
 dist/
 downloads/
 eggs/
-.eggs/
 lib/
 lib64/
+MANIFEST
 parts/
 sdist/
+share/python-wheels/
 var/
 wheels/
-share/python-wheels/
-*.egg-info/
-.installed.cfg
-*.egg
-MANIFEST
 
 # PyInstaller
 #  Usually these files are written by a python script from a template
@@ -47,33 +47,33 @@ pip-log.txt
 pip-delete-this-directory.txt
 
 # Unit test / coverage reports
-htmlcov/
-.tox/
-.nox/
+.cache
 .coverage
 .coverage.*
-.cache
-nosetests.xml
-coverage.xml
-*.cover
-*.py,cover
 .hypothesis/
+.nox/
 .pytest_cache/
+.tox/
+*.cover
+*.py,cover
 cover/
+coverage.xml
+htmlcov/
+nosetests.xml
 
 # Translations
 *.mo
 *.pot
 
 # Django stuff:
 *.log
-local_settings.py
 db.sqlite3
 db.sqlite3-journal
+local_settings.py
 
 # Flask stuff:
-instance/
 .webassets-cache
+instance/
 
 # Scrapy stuff:
 .scrapy
@@ -132,11 +132,11 @@ celerybeat.pid
 # Environments
 .env
 .venv
+env.bak/
 env/
-venv/
 ENV/
-env.bak/
 venv.bak/
+venv/
 
 # Spyder project settings
 .spyderproject

.gitignore

@@ -1,4 +1,5 @@
 {
+    // VSCode debugpy configuration
     // Use IntelliSense to learn about possible attributes.
     // Hover to view descriptions of existing attributes.
     // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387

.vscode/launch.json

@@ -0,0 +1,25 @@
+# Agent Guidelines for Implementation Bridges Codebase
+
+## Build/Test Commands
+- Build and start all containers: `cd repo-converter/build && docker compose up -d --build`
+- View repo-converter logs: `cd repo-converter/build && ./build.sh logs`
+- Update requirements: `cd repo-converter/build && pipreqs --force --mode gt .`
+- Run single test: No tests in codebase
+
+## Code Style Guidelines
+- Python version: 3.13.2
+- Imports: Standard libs first, then third-party libs with URLs in comments
+- Variables: Snake case (e.g., `local_repo_path`)
+- Error handling: Use try/except blocks with specific exception types
+- Logging: Use the custom `log()` function with appropriate levels
+- Functions: Snake case for function names. Add docstrings (not yet implemented but mentioned in TODOs)
+- Security: Use `redact_password()` before logging, if the input contains a password
+- Documentation: Use Python best practices for docstrings
+- Environment variables: Set defaults with `os.environ.get("VAR", "default")`
+- Comments: Use `#` for comments, and add lots of comments
+
+The purpose of this script is to convert repos from Subversion to Git
+`repo-converter/build/run.py` is the only file in this repo which runs application code
+It runs in a Docker container, and is the entrypoint for the container
+
+The usage of this product is described to users in `repo-converter/README.md`

AGENT.md

@@ -0,0 +1,128 @@
+# Refactoring Plan for repo-converter/build/run.py
+
+## Overview
+
+The current `run.py` script (1500+ lines) performs repository conversion from SVN to Git but has several issues:
+
+- Very long file with monolithic structure
+- Many functions with multiple responsibilities
+- Complex error handling scattered throughout
+- No object-oriented design despite handling complex data
+- Incomplete features marked with TODOs
+- Duplicate code patterns
+
+## Phase 1: Code Organization
+
+### 1.1 Create Package Structure
+
+```
+repo_converter/
+├── __init__.py
+├── main.py              # Entry point, contains main() function
+├── config/
+│   ├── __init__.py
+│   ├── environment.py   # Environment variable handling
+│   └── yaml_config.py   # YAML configuration file parsing
+├── core/
+│   ├── __init__.py
+│   └── logging.py       # Custom logging functionality
+├── utils/
+│   ├── __init__.py
+│   ├── process.py       # Process management utilities
+│   └── security.py      # Password redaction functionality
+└── repositories/
+    ├── __init__.py
+    ├── base.py          # Base Repository class
+    ├── svn.py           # SVN repository handling
+    ├── git.py           # Git repository handling
+    └── tfs.py           # TFS repository handling (future)
+```
+
+### 1.2 Implement Base Classes
+
+- Create a `Repository` base class with common methods
+- Implement `SVNRepository`, `GitRepository`, and `TFSRepository` subclasses
+- Move configuration validation to each repository type
+
+## Phase 2: Refactor Functionality
+
+### 2.1 Configuration Management
+
+- Move environment variable loading to `config/environment.py`
+- Move YAML parsing to `config/yaml_config.py`
+- Implement proper type validation with clear error messages
+- Create unified configuration object merging both sources
+
+### 2.2 Logging Improvements
+
+- Refactor `log()` function to use Python's logging more effectively
+- Implement cleaner password redaction
+- Add log rotation for long-running instances
+
+### 2.3 Process Management
+
+- Refactor subprocess handling into a cleaner utility class
+- Improve zombie process detection and cleanup
+- Implement better error handling for process management
+
+## Phase 3: Repository Processing
+
+### 3.1 SVN Repository Handling
+
+- Break down `clone_svn_repo()` into smaller, focused methods
+- Implement proper state management for create/update/running
+- Add better error handling with specific error types
+- Fix batch processing logic
+
+### 3.2 Git Repository Handling
+
+- Implement proper Git repository functionality
+- Use GitPython more extensively
+
+### 3.3 Process Concurrency
+
+- Improve multiprocessing implementation
+- Add proper resource limiting and queuing
+- Implement better status tracking
+
+## Phase 4: Testing & Documentation
+
+### 4.1 Testing
+
+- Add unit tests for each module
+- Add integration tests for repository operations
+- Implement mock objects for external dependencies
+
+### 4.2 Documentation
+
+- Add proper docstrings to all classes and methods
+- Create usage documentation
+- Document configuration options
+
+## Phase 5: Implement TODOs
+
+After refactoring, implement the TODOs from the original file:
+
+1. Config file improvements
+2. SVN enhancements (timeouts, gitignore handling, etc.)
+3. Git SSH clone functionality
+4. Permissions improvements
+5. Add fetch interval configuration
+6. Process status improvements
+
+## Implementation Strategy
+
+1. Start with creating the directory structure and moving code
+2. Refactor one component at a time, ensuring functionality is preserved
+3. Add tests for each refactored component
+4. Keep the original script working until the refactored version is complete
+5. Implement a gradual migration strategy
+
+## Benefits
+
+- Improved maintainability through modular design
+- Better error handling and recovery
+- Clearer separation of concerns
+- Easier implementation of new features
+- More testable code structure
+- Better documentation

GAMEPLAN.md

@@ -0,0 +1 @@
+TODO

LICENSE

@@ -41,6 +41,6 @@ Docker compose also allows for easier upgrades, troubleshooting, monitoring, log
     - There are docker-compose.yaml and override files in a few different directories in this repo, separated by use case, so that each use case only needs to run `docker compose up -d` in one directory, and not fuss around with `-f` paths.
     - The only difference between the docker-compose-override.yaml files in host-ubuntu vs host-wsl is the src-serve-git container's name, which is how we get a separate `dnsName` for each.
     - If you're using the repo-converter:
-        - If you're using the pre-built images, `cd ./repo-converter && docker compose up -d`
-        - If you're building the Docker images, `cd ./repo-converter/build && docker compose up -d --build`
+        - If you're using the pre-built images, `cd ./deploy && docker compose up -d`
+        - If you're building the Docker images, `cd ./build && docker compose up -d --build`
         - Either of these will start all 3 containers: cloud-agent, src-serve-git, and the repo-converter

README.md

@@ -0,0 +1,15 @@
+# Include any files or directories that you don't want to be copied to your
+# container here (e.g., local build artifacts, temporary files, etc.).
+#
+# For more help, visit the .dockerignore file reference guide at
+# https://docs.docker.com/go/build-context-dockerignore/
+
+
+# Ignore everything
+*
+
+# Allow specific directories
+!/src
+
+# Allow specific files
+!/build/.env