Fix & re-enable flaky tests in CI

[opcode81]

Many tests had to be disabled to render CI tests reliable.

The following tests were disabled/marked with xfail with #1041 (resolving #1039) and we should investigate whether we can "fix" something and re-enable them at some point:

• Phpactor tests on Windows
  • Finding references always fails (see Phpactor: Finding references fails on Windows #1022)
  • Other tests sometimes fail
• TestFSharpLanguageServer
  • test_find_referencing_symbols_across_files
  • test_find_referencing_symbols
  • test_hover_information
  • test_go_to_definition
• TestNixLanguageServer::test_hover_information
• TestSwiftLanguageServerBasics::test_request_references_*
• TestSerenaAgent
  • test_find_symbol_fsharp
  • test_find_symbol_rust
  • test_find_symbol_references_fsharp
  • test_find_symbol_references_typescript

[MischaPanch]

The nix error is an actual bug, it always fails. I briefly investigated at some point but didn't find the solution then, and I never prioritized this.

For referencing symbols in F# we could maybe just bump the wait time

[MischaPanch]

Php-tractor failing on windows is also not flaky but rather an actual bug

[opcode81]

The nix error is an actual bug, it always fails.

If that's the case, then why was it written and enabled?

Php-tractor failing on windows is also not flaky but rather an actual bug

No, some of them pass on my computer (and also in CI), but all of them can fail.

For referencing symbols in F# we could maybe just bump the wait time

Try it if you want. I don't have time for this.

Anyway, this ticket is there to re-investigate the respective issues. With #1041, I am just disabling them.

[antigenius0910]

While working on PR #1114 (enabling Rego tests on Windows), we noticed a broader FSharp regression on Windows CI that goes beyond the specific flaky tests tracked here.

All FSharp tests are currently failing on Windows with the following error from FsAutoComplete:

[FsAutoComplete.Lsp.AdaptiveState] Typecheck failed for ...\Calculator.fs with Check aborted (Aborted).
Errors: [|"The type referenced through 'System.Array' is defined in an assembly that is not referenced.
You must add a reference to assembly 'The system type 'System.Array' was required but no referenced
system DLL contained this type'."|]

Tests now failing (beyond those already disabled in #1041):

• TestFSharpLanguageServer::test_find_symbol
• TestFSharpLanguageServer::test_get_document_symbols_program
• TestFSharpLanguageServer::test_get_document_symbols_calculator
• TestFSharpLanguageServer::test_find_referencing_symbols
• TestFSharpLanguageServer::test_nested_module_symbols
• TestFSharpLanguageServer::test_go_to_definition
• TestFSharpLanguageServer::test_completion
• TestSerenaAgent::test_find_symbol_references[fsharp-...]

This was already failing on main before our PR (confirmed via run 22555354392). The root cause appears to be that FsAutoComplete on the Windows CI runner can no longer resolve core .NET types — likely a .NET SDK version or runtime targeting regression in the CI environment. Worth checking whether the .NET SDK version installed in CI has changed recently.

[opcode81]

@antigenius0910 thanks for reporting. I also noticed this and have now fixed the issue with #1115 (just merged).

[antigenius0910]

Follow-up observation from PR #1114 CI run (after the dotnet handling fix from #1115 was merged in):

Ubuntu now has one remaining FSharp failure not currently tracked in this issue:

• TestFSharpLanguageServer::test_go_to_definition[fsharp] — "Could not find declaration. (-32603)" from FsAutoComplete

This is distinct from the two FSharp tests already listed here (test_find_referencing_symbols_across_files, test_hover_information). Looks like another flaky go-to-definition failure on Ubuntu CI.

Windows is now clean — the System.Array regression I noted earlier (comment above) was resolved by #1115.

Also worth noting: the run shows 12 xpassed tests (tests marked xfail that actually passed), which might mean some of the previously-disabled tests have stabilised and could be re-evaluated for re-enabling.