FlakyTest TestMTLS_ProxyToBackend

[szuecs]

time="2026-06-29T00:52:44Z" level=info msg="TLS settings not found, defaulting to HTTP"
2026/06/29 00:52:44 http: TLS handshake error from 127.0.0.1:50084: tls: client didn't provide a certificate
time="2026-06-29T00:52:44Z" level=error msg="error while proxying after 4.829958ms, route  with backend network https://127.0.0.1:43009, status code 500: dialing failed false: unexpected error from Go stdlib net/http package during roundtrip: readLoopPeekFailLocked: remote error: tls: certificate required, remote host: ::1, request: \"GET / HTTP/1.1\", host: [::]:35977, user agent: \"Go-http-client/1.1\""
::1 - - [29/Jun/2026:00:52:44 +0000] "GET / HTTP/1.1" 500 22 "-" "Go-http-client/1.1" 5 [::]:35977 - -
time="2026-06-29T00:52:44Z" level=info msg="Got shutdown signal, wait 0s for health check"
time="2026-06-29T00:52:44Z" level=info msg="Start shutdown"
time="2026-06-29T00:52:44Z" level=info msg=done.
time="2026-06-29T00:52:44Z" level=info msg="Stop secrets background refresher"
--- FAIL: TestMTLS_ProxyToBackend (0.05s)
    --- FAIL: TestMTLS_ProxyToBackend/negative:_no_client_cert_causes_backend_to_reject_the_connection (0.02s)
        skipper_mtls_test.go:253: Found address "[::]:35977" to be free
        skipper_mtls_test.go:272: 
            	Error Trace:	/home/runner/work/skipper/skipper/skipper_mtls_test.go:272
            	Error:      	Should be true
            	Test:       	TestMTLS_ProxyToBackend/negative:_no_client_cert_causes_backend_to_reject_the_connection
            	Messages:   	expected 503, got 500
time="2026-06-29T00:52:44Z" level=info msg="Listen on [::]:44995"

[AshrafAhmed9]

Hi @szuecs, I'd like to take a look at this one if it's still available. From the log it looks like the negative subtest (no client cert causes backend to reject the connection) expects a 503 but intermittently gets a 500 when the client cert is missing — so the backend's TLS-handshake rejection is surfacing as a proxy 500 rather than being mapped to a 503, which points to either a race in the test setup or a gap in the error→status mapping.

My plan would be to (1) reproduce the flake locally by running the test in a loop / with -count and -race to confirm whether it's a timing/port-binding race or a deterministic mapping issue, then (2) trace how the mTLS handshake error propagates through the proxy roundtrip to see whether the 500 should be classified as a 503.

I'm a backend engineering student comfortable with Go and looking to contribute — happy to follow the contributing guidelines and open a PR. Could you assign this to me, or let me know if there's additional context I should be aware of?

[szuecs]

@AshrafAhmed9 most flaky tests are having time constraints in tests and in the CI environment we get throttled.
Normally you can run on your desk the test with -count 100 without any failure.

The use of AI does not help for fixing flakytests.

I think if you want to build some interesting things algorithm wise you can check the #557 for different lb algorithms. The issue is basically an ongoing collection of algorithms that "look" interesting. Not all of them are easily done, but some should be straight forward to implement.