FunctionalJavascript Tests Failing with WebDriver Connection Timeouts in GitHub Actions

Summary

FunctionalJavascript tests are consistently failing in GitHub Actions due to WebDriver connection timeouts. The tests hang during execution and either timeout or get cancelled, preventing proper completion of the test suite.

Environment

• GitHub Actions: Ubuntu 24.04 runners
• Drupal versions: 10.4.x and 11.0.x
• PHP version: 8.3
• Selenium: standalone-chrome:4.15.0
• PHPUnit: 10.5.29

Error Details

Typical Error Message

Test Skipped (Drupal\Tests\ab_tests\FunctionalJavascript\ControllerTest::testAjaxResponse)
The test wasn't able to connect to your webdriver instance. For more information read core/tests/README.md.
The original message while starting Mink: Could not open connection: Could not start a new session. New session request timed out 
Host info: host: '3712430984d9', ip: '172.18.0.3'
Build info: version: '4.15.0', revision: '1d14b5521b'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.11.0-1018-azure', java.version: '11.0.21'
Driver info: driver.version: unknown

WebDriver Configuration

MINK_DRIVER_ARGS_WEBDRIVER='["chrome", {"browserName":"chrome","goog:chromeOptions":{"args":["--disable-gpu", "--headless", "--no-sandbox", "--disable-dev-shm-usage", "--disable-extensions"]}}, "http://localhost:4444/wd/hub"]'

Affected Tests

The module contains 5 FunctionalJavascript test files that are all affected:

1. ControllerTest.php - Tests AJAX controller functionality
2. NodeDisplayTest.php - Tests node rendering with A/B testing
3. PluginSelectionTest.php - Tests plugin configuration and AJAX behavior
4. TimeoutDeciderTest.php - Tests timeout decider functionality (has incomplete test with TODO)
5. Base class: AbTestsFunctionalJavaScriptTestBase.php

Failure Pattern

1. Connection Phase: Tests fail during WebDriver connection setup
2. Timeout Behavior: GitHub Actions shows runs with status "" (empty) instead of "failed"
3. Process Hanging: Some runs hang indefinitely and require manual cancellation
4. Consistent Failure: Issue affects both Drupal 10.4.x and 11.0.x matrices

GitHub Actions Configuration

Current Service Configuration

services:
  mysql:
    image: mysql:8.0
    ports:
      - 3306:3306
    env:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_DATABASE: drupal
    options: --health-cmd="mysqladmin ping" --health-interval=10s --health-timeout=5s --health-retries=3

  chrome:
    image: selenium/standalone-chrome:4.15.0
    ports:
      - 4444:4444
    options: --health-cmd="/opt/bin/check-grid.sh" --health-interval=10s --health-timeout=5s --health-retries=3

Test Command

SIMPLETEST_BASE_URL=http://127.0.0.1:8080 \
SIMPLETEST_DB=mysql://root:root@127.0.0.1/drupal \
MINK_DRIVER_ARGS_WEBDRIVER='["chrome", {"browserName":"chrome","goog:chromeOptions":{"args":["--disable-gpu", "--headless", "--no-sandbox", "--disable-dev-shm-usage", "--disable-extensions"]}}, "http://localhost:4444/wd/hub"]' \
./vendor/bin/phpunit -c core/phpunit.xml.dist --fail-on-skipped --debug modules/ab_tests/tests/src/FunctionalJavascript

Recent Test Infrastructure History

The module has a history of WebDriver-related issues that have been addressed:

• b21c1e1: Fixed Selenium version compatibility (pinned to 4.15.0)
• 8b635e3: Updated WebDriver configuration for newer Selenium versions
• 3c468c4: Added verbose output to diagnose test skips
• ea89946: Added missing PHP extensions and service readiness checks
• b58c817: Added PHPUnit 12 compatibility with PHP Attributes

Impact

• CI/CD Pipeline: Blocks merge of pull requests due to hanging/failing tests
• Development Workflow: Prevents validation of A/B testing JavaScript functionality
• Test Coverage: Core A/B testing features remain untested in CI
• Module Quality: Cannot verify complex client-side A/B testing behavior

Test Complexity

The A/B Tests module has sophisticated client-side architecture that requires JavaScript testing:

1. Asynchronous Decision Making: Timeout-based variant decisions
2. AJAX Content Replacement: Server-side rendering with client-side updates
3. Context Preservation: Complex serialization/deserialization during Ajax requests
4. Multi-Plugin System: Integration between decider and analytics plugins

Potential Solutions to Investigate

1. Service Coordination Issues

• Add explicit wait conditions for Selenium service readiness
• Implement retry logic for WebDriver connection attempts
• Verify Docker networking between services

2. Chrome Configuration

• Review Chrome options for GitHub Actions environment
• Test alternative Chrome configurations (e.g., different memory settings)
• Consider switching to Chrome binary instead of Selenium Grid

3. Timing and Timeout Adjustments

• Increase WebDriver connection timeout values
• Add delays between service startup and test execution
• Implement health checks with longer intervals

4. Alternative WebDriver Approaches

• Test with different Selenium versions
• Consider ChromeDriver standalone instead of Selenium Grid
• Evaluate alternative WebDriver implementations

5. Test Environment Debugging

• Add comprehensive service health verification
• Implement WebDriver connection testing before test execution
• Add detailed logging of Docker container states and networking

Test Coverage Gap

Missing Tests: The ab_blocks module (block-level A/B testing) has no FunctionalJavascript tests despite being a major feature with complex JavaScript interactions.

Incomplete Tests: TimeoutDeciderTest contains a TODO indicating core A/B testing functionality is not properly tested.

Priority

High - This issue blocks CI/CD pipeline and prevents validation of critical JavaScript functionality for the A/B testing system.

Additional Context

• Tests work locally in development environments
• Issue is specific to GitHub Actions CI environment
• Previous attempts to fix WebDriver configuration have been partially successful but the core connection timeout issue persists