Implement comprehensive E2E testing framework in Go

[bittermandel]

Overview

We need to implement a comprehensive end-to-end testing framework for the Go rewrite of the Neon operator. The Rust implementation has a sophisticated e2e testing system that we need to port and potentially improve.

Reference Implementation

• Rust E2E Framework: https://github.com/molnett/neon-operator/tree/268e35cbbf94f45a743b013f87e5415f1c8dafd7/crates/e2e_tests

Key Features to Port

1. Modern Test Environment (`TestEnv` struct)
2. State Machine Orchestration (test state tracking)
3. Parallel Setup (infrastructure + services)
4. Smart Waiting (with backoff and progress indicators)
5. Comprehensive Observability (metrics, timing, failure tracking)
6. Resource Management (Kind clusters, MinIO, operator deployment)

Technical Components Needed

1. Test Environment (`TestEnv`)

• Kubernetes client setup
• Kind cluster management
• MinIO deployment and configuration
• Operator deployment
• Cleanup coordination

2. State Machine

• Track test execution phases
• Progress reporting (0-100%)
• State transitions with validation
• Timing and duration tracking
• Failure state handling

3. Parallel Setup Orchestrator

• Concurrent infrastructure setup
• Service deployment coordination
• Dependency management
• Configurable parallelism

4. Smart Waiting & Progress

• Exponential backoff
• Resource condition waiting
• Progress indicators
• Timeout handling

5. Observability & Metrics

• Test phase timing
• Resource state observation
• Failure event recording
• Comprehensive test summaries

6. Integration Helpers

• Postgres connectivity validation
• CRD installation/validation
• Resource creation/cleanup
• Log collection

Implementation Strategy

1. Phase 1: Basic TestEnv with Kind cluster management
2. Phase 2: State machine and progress tracking
3. Phase 3: Parallel setup orchestration
4. Phase 4: Advanced observability and metrics
5. Phase 5: Integration helpers and validation functions

Success Criteria

• Create/destroy Kind clusters programmatically
• Deploy operator and dependencies in parallel
• Track test execution with state machine
• Comprehensive resource cleanup
• Postgres connectivity validation
• Full cluster lifecycle testing
• Performance timing and metrics
• Failure debugging and reporting
• CI/CD integration ready

Testing Scope

The e2e tests should cover:

• Complete cluster lifecycle (create → deploy → use → cleanup)
• Multi-component deployments (cluster + project + branch)
• Storage integration (MinIO + pageserver)
• Compute functionality (Postgres connectivity)
• Recovery scenarios (failures, restarts)
• Performance characteristics (setup time, resource usage)

Technical Considerations

• Use `testify` and `ginkgo`/`gomega` for test frameworks
• Implement proper Go context handling for timeouts
• Use structured logging (slog) for observability
• Follow Go testing conventions and patterns
• Consider using `kind` Go SDK instead of CLI calls