Skip to content

Add test coverage reporting with grcov and codecov integration#61

Merged
linw1995 merged 12 commits intomainfrom
misc/coverage_tests
Aug 23, 2025
Merged

Add test coverage reporting with grcov and codecov integration#61
linw1995 merged 12 commits intomainfrom
misc/coverage_tests

Conversation

@linw1995
Copy link
Copy Markdown
Owner

@linw1995 linw1995 commented Aug 22, 2025

Summary

This PR introduces comprehensive test coverage reporting to the project using grcov (LLVM-based code coverage) and codecov.io integration. This is the first coverage implementation for the project.

Key Changes

New Coverage Infrastructure

  • grcov Integration: Add LLVM-based source coverage analysis with grcov
  • Multiple Report Formats: Generate HTML, Cobertura, and Markdown coverage reports
  • codecov.io Integration: Automated coverage upload to codecov.io with PR comments and trends
  • Development Scripts: New run-cov.sh and run-test.sh scripts for standardized testing workflows

Nix Flake Enhancements

  • Add grcov and LLVM tools to development dependencies
  • Create dedicated cov and test apps for easy coverage analysis: nix run .#cov and nix run .#test
  • Include coverage scripts as packages in development shell
  • Maintain full backwards compatibility with existing workflows

Test Infrastructure Improvements

  • Simplified Binary Management: Refactor integration test setup to remove complex one-time compilation logic
  • Better Target Directory Handling: Support CARGO_TARGET_DIR environment variable for coverage builds
  • Improved LSP Reliability: Increase LSP timeouts from 5s to 15s for more stable integration tests
  • Cleaner Test Code: Streamline integration test infrastructure and reduce complexity

CI/CD Pipeline Updates

  • Replace basic cargo test with coverage-enabled testing in GitHub Actions
  • Add codecov upload step with proper token handling and dependabot-safe error handling
  • Provide detailed coverage feedback in CI output
  • Maintain existing CI reliability while adding coverage metrics

Technical Implementation

Coverage Workflow

# Set LLVM instrumentation flags
export RUSTFLAGS="-Cinstrument-coverage" 
export CARGO_TARGET_DIR="./target/coverage"
export LLVM_PROFILE_FILE="${CARGO_TARGET_DIR}/data/nvim-mcp-%p-%m.profraw"

# Build and run tests with coverage
cargo build --bin nvim-mcp
cargo test

# Generate multi-format reports
grcov ${CARGO_TARGET_DIR}/data \
    --llvm --branch --source-dir . \
    --output-types html,cobertura,markdown

Usage Examples

# Run tests with coverage analysis
nix run .#cov -- --show-output

# Run standard tests  
nix run .#test -- --show-output

# Direct script usage
./scripts/run-cov.sh -- --show-output
./scripts/run-test.sh -- --show-output

Files Added/Modified

New Files

  • scripts/run-cov.sh: Comprehensive coverage execution script
  • scripts/run-test.sh: Standardized test execution script

Modified Files

  • .github/workflows/CI.yaml: Add coverage steps and codecov integration
  • flake.nix: Add grcov, LLVM tools, and new coverage/test apps
  • src/*/integration_tests.rs: Improve reliability and simplify binary management
  • CLAUDE.md, README.md, CHANGELOG.md: Update documentation with coverage instructions

Benefits

  • Visibility: Track code coverage across the entire Rust codebase for the first time
  • Quality Assurance: Identify untested code paths and improve test comprehensiveness
  • Developer Experience: Easy local coverage analysis with HTML reports and terminal summary
  • CI Integration: Automated coverage reporting with PR feedback and historical tracking
  • Reliability: More stable integration tests with improved timeout handling
  • Maintainability: Cleaner test infrastructure with standardized execution scripts

This establishes the foundation for maintaining high code quality and comprehensive test coverage as the project continues to grow.

- Add cargo-tarpaulin to development dependencies in flake.nix
- Add 'cov' app for running tests with coverage using tarpaulin
- Update CI workflow to run tests with coverage and upload to codecov.io
- Modify integration tests to support tarpaulin compilation flags
- Handle CARGO_TARGET_DIR environment variable for coverage builds
- Add proper RUSTFLAGS extraction from tarpaulin for coverage compilation
@linw1995 linw1995 requested a review from Copilot August 22, 2025 14:42
@linw1995 linw1995 self-assigned this Aug 22, 2025
@linw1995 linw1995 added enhancement New feature or request github_actions Pull requests that update GitHub Actions code labels Aug 22, 2025
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds comprehensive test coverage reporting capabilities to the project by integrating codecov.io with cargo-tarpaulin for Rust code coverage analysis. The changes enable automated coverage collection during CI runs and provide developers with a local coverage testing workflow.

Key changes:

  • Integration of cargo-tarpaulin for Rust code coverage collection
  • Addition of a 'cov' flake app for local coverage testing with proper environment setup
  • CI workflow modifications to run tests with coverage and automatically upload results to codecov.io

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
flake.nix Adds cargo-tarpaulin dependency and creates 'cov' app for coverage testing
.github/workflows/CI.yaml Updates CI to run coverage tests and upload results to codecov.io
src/server/integration_tests.rs Modifies integration tests to support tarpaulin compilation with proper flag extraction and target directory handling

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

- Configure tarpaulin to use LLVM engine for more accurate coverage
- Enables consistent coverage reporting across development and CI
- Increase LSP analysis timeout from 5s to 15s for better test reliability
- Increase LSP ready timeout from 5s to 15s for better test reliability
- Extract target directory determination into reusable get_target_dir() function
- Improve code organization in integration test setup
The test_lsp_declaration test fails when running with coverage enabled.
Adding ignore attribute to prevent test failures during coverage runs.
@codecov-commenter
Copy link
Copy Markdown

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

The codecov action can auto-detect coverage files without specifying
the file parameter, simplifying the configuration.
- Replace cargo-tarpaulin with grcov for more reliable coverage reporting
- Add llvm-tools to Rust components for coverage instrumentation
- Simplify integration test binary compilation by removing one-time compilation logic
- Update coverage app script to use LLVM instrumentation and grcov output formats
- Add test app script for basic test execution
- Remove tarpaulin-specific code paths from integration tests
- Remove .tarpaulin.toml configuration file
- Add scripts/run-test.sh and scripts/run-cov.sh for standardized testing
- Update flake.nix to use grcov with LLVM-based coverage reporting
- Support HTML, Cobertura, and Markdown coverage report formats
- Add Nix apps for test and cov commands
- Update CHANGELOG.md, CLAUDE.md, and README.md with new coverage workflow
- Integrate test scripts into development environment packages
Update flake.nix to use correct package names:
- test app now references run-test package
- cov app now references run-cov package
- Simplify CI workflow to use run-cov script directly
- Add console feedback message during coverage report generation
Enable test_lsp_declaration to run with coverage enabled after fixing
coverage compatibility issues.
Remove unnecessary comment and simplify macro implementation
in integration_tests.rs for better code clarity.
@linw1995 linw1995 changed the title Add test coverage reporting with codecov integration Add test coverage reporting with grcov and codecov integration Aug 23, 2025
@linw1995 linw1995 requested a review from Copilot August 23, 2025 09:00
@linw1995 linw1995 merged commit c6ae29e into main Aug 23, 2025
1 check passed
@linw1995 linw1995 deleted the misc/coverage_tests branch August 23, 2025 09:02
@linw1995 linw1995 review requested due to automatic review settings March 23, 2026 22:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request github_actions Pull requests that update GitHub Actions code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants