Project Analysis fails on a project using MTP

[dupdob]

When running stryker for our productive repos (we couldn't run it for a while, since we have switched to MTP), we are getting the following error. (I have tested it for 2 different repos, which I cannot share due to data reasons).

This is the command line:

dotnet-stryker --output .stryker/run_1 --project Add.Cli.csproj --test-project test/Add.Cli.Test/Add.Cli.Test.csproj --test-project test/StWriter.Test/StWriter.Test.csproj --test-project test/StWriter.Test.Unit/StWriter.Test.Unit.csproj --test-project test/Cli.Test.Unit/Cli.Test.Unit.csproj --test-project test/Cli.Test/Cli.Test.csproj --test-project test/FileIo.Test/FileIo.Test.csproj --test-project test/FileIo.Test.Unit/FileIo.Test.Unit.csproj --test-project test/Add.Cli.Test.Unit/Add.Cli.Test.Unit.csproj  --testrunner mtp --mutation-level Advanced

This is the output:

Version: 4.12.0
[05:50:37 INF] Analysis starting.
[05:50:37 INF] Stryker will mutate solution SafetyCli.
[05:50:37 INF] Using configuration/platform 'Debug|Any CPU'.
[05:50:37 INF] Identifying projects to mutate in /SafetyCli.sln. This can take a while.
[05:50:45 INF] Could not find an assembly reference to a mutable assembly for project /test/Cli.Test.Unit/Cli.Test.Unit.csproj. Will look into project references.
[05:50:45 INF] Could not find an assembly reference to a mutable assembly for project /test/StWriter.Test/StWriter.Test.csproj. Will look into project references.
[05:50:45 INF] Could not find an assembly reference to a mutable assembly for project /test/Cli.Test/Cli.Test.csproj. Will look into project references.
[05:50:45 INF] Could not find an assembly reference to a mutable assembly for project /test/FileIo.Test/FileIo.Test.csproj. Will look into project references.
[05:50:45 INF] Could not find an assembly reference to a mutable assembly for project /test/StWriter.Test.Unit/StWriter.Test.Unit.csproj. Will look into project references.
[05:50:45 INF] Could not find an assembly reference to a mutable assembly for project /test/Add.Cli.Test/Add.Cli.Test.csproj. Will look into project references.
[05:50:45 INF] Could not find an assembly reference to a mutable assembly for project /test/Cli.Test.Apax/Cli.Test.Apax.csproj. Will look into project references.
[05:50:45 INF] Could not find an assembly reference to a mutable assembly for project /test/FileIo.Test.Unit/FileIo.Test.Unit.csproj. Will look into project references.
[05:50:45 INF] Project /src/Add.Cli/Add.Cli.csproj analysis failed hence can't be mutated.
[05:50:45 INF]   referenced by test project /test/Cli.Test.Unit/Cli.Test.Unit.csproj, analysis failed.
[05:50:45 INF]   referenced by test project /test/Add.Cli.Test/Add.Cli.Test.csproj, analysis failed.
[05:50:45 WRN]   can't be mutated because all referencing test projects' analysis failed.
[05:50:45 INF] Test project /test/StWriter.Test/StWriter.Test.csproj does not appear to test any mutable project, analysis failed.
[05:50:45 INF] Test project /test/Cli.Test/Cli.Test.csproj does not appear to test any mutable project, analysis failed.
[05:50:45 INF] Test project /test/FileIo.Test/FileIo.Test.csproj does not appear to test any mutable project, analysis failed.
[05:50:45 INF] Test project /test/StWriter.Test.Unit/StWriter.Test.Unit.csproj does not appear to test any mutable project, analysis failed.
[05:50:45 INF] Test project /test/Cli.Test.Apax/Cli.Test.Apax.csproj does not appear to test any mutable project, analysis failed.
[05:50:45 INF] Test project /test/FileIo.Test.Unit/FileIo.Test.Unit.csproj does not appear to test any mutable project, analysis failed.
[05:50:45 WRN] Use --diag option to have the analysis logs in the log file.
[05:50:45 INF] Analysis complete.
[05:50:45 INF] Time Elapsed 00:00:08.2930823
Stryker.NET failed to mutate your project. For more information see the logs 
below:
Failed to analyze project builds. Stryker cannot continue.

The diag log can be found as attachment and it took really long ~3:30m. I have redacted all sensitive data.

output.log

Originally posted by @Lorilatschki in #3094

[dupdob]

hello
@Lorilatschki: I created a specific issue to discuss your problem.

[dupdob]

For context, latest Stryker version integrates a redesigned project analysis failed. This redesign was long overdue, as this step grow more complex across the versions. This resulted in an analysis phase that often took a long time and provided no information when it failed. I tried to improve on both aspects, but this kind of endeavor may result in some regressions.

--diag mode should have provide more information regarding the project analysis failed. I understand this may contain sensitive data, but it would be helpful if you could share what was the cause of the failure.

Basically, this is a build log, which is preceded by Project {ProjectFilePath} analysis failed. The MsBuild log is: {Log}.
So If you could provide an excerpt of this part, or share your view on what the cause was, it would be great.

[dupdob]

Stryker may be a bit to strict in its expectations, as it will signal a failure if the analysis fails for any of the projects framework version (for multi targeting project). On the other, a project analysis is flagged as a success it was able to identify the target assembly filename and some source files.

If your project is multi targeting, maybe the analysis fails for one of the framework version. You should then specify with framework version you want to test, it could resolve the issue for you.

[Lorilatschki]

The project is not multitargeting, it's build against dotnet 8.0.
I have removed the solution manually before running in test-project-context, however, it still fails with

Version: 4.12.0
[08:37:12 INF] Analysis starting.
[08:37:12 INF] Analyzing 8 test project(s).
[08:37:20 INF] Could not find an assembly reference to a mutable assembly for project /repo/test/Cli.Test/Cli.Test.csproj. Will look into project references.
[08:37:20 INF] Could not find an assembly reference to a mutable assembly for project /repo/test/StWriter.Test/StWriter.Test.csproj. Will look into project references.
[08:37:20 INF] Could not find an assembly reference to a mutable assembly for project /repo/test/FileIo.Test.Unit/FileIo.Test.Unit.csproj. Will look into project references.
[08:37:20 INF] Could not find an assembly reference to a mutable assembly for project /repo/test/Cli.Test.Unit/Cli.Test.Unit.csproj. Will look into project references.
[08:37:20 INF] Could not find an assembly reference to a mutable assembly for project /repo/test/FileIo.Test/FileIo.Test.csproj. Will look into project references.
[08:37:20 INF] Could not find an assembly reference to a mutable assembly for project /repo/test/StWriter.Test.Unit/StWriter.Test.Unit.csproj. Will look into project references.
[08:37:20 INF] Could not find an assembly reference to a mutable assembly for project /repo/test/Add.Cli.Test.Unit/Add.Cli.Test.Unit.csproj. Will look into project references.
[08:37:20 INF] Could not find an assembly reference to a mutable assembly for project /repo/test/Add.Cli.Test/Add.Cli.Test.csproj. Will look into project references.
[08:37:20 INF] Project /repo/src/Add.Cli/Add.Cli.csproj analysis failed hence can't be mutated.
[08:37:20 INF]   referenced by test project /repo/test/Cli.Test.Unit/Cli.Test.Unit.csproj, analysis failed.
[08:37:20 INF]   referenced by test project /repo/test/Add.Cli.Test.Unit/Add.Cli.Test.Unit.csproj, analysis failed.
[08:37:20 INF]   referenced by test project /repo/test/Add.Cli.Test/Add.Cli.Test.csproj, analysis failed.
[08:37:20 WRN]   can't be mutated because all referencing test projects' analysis failed.
[08:37:20 INF] Test project /repo/test/Cli.Test/Cli.Test.csproj does not appear to test any mutable project, analysis failed.
[08:37:20 INF] Test project /repo/test/StWriter.Test/StWriter.Test.csproj does not appear to test any mutable project, analysis failed.
[08:37:20 INF] Test project /repo/test/FileIo.Test.Unit/FileIo.Test.Unit.csproj does not appear to test any mutable project, analysis failed.
[08:37:20 INF] Test project /repo/test/FileIo.Test/FileIo.Test.csproj does not appear to test any mutable project, analysis failed.
[08:37:20 INF] Test project /repo/test/StWriter.Test.Unit/StWriter.Test.Unit.csproj does not appear to test any mutable project, analysis failed.
[08:37:20 WRN] Use --diag option to have the analysis logs in the log file.
[08:37:20 INF] Analysis complete.
[08:37:20 INF] Time Elapsed 00:00:07.8877012
Stryker.NET failed to mutate your project. For more information see the logs 
below:
Failed to analyze project builds. Stryker cannot continue.

The --diag output is attached.

diag.txt

[dupdob]

I am sorry to ask again, but I am surprised not to see the build logs after the analysis steps. It should start with a line such as
Project xxxx analysis failed. The MsBuild log is: Started process 20798: "dotnet" --info.

I am afraid I can't figure out why it went wrong.
On the bright side of things, the log shows that Stryker has gathered enough information from the test projects to mutate the target project
⇒ I will open a PR with adjusted criteria that should no longer block at this step. Please understand this will not be a fix per se, as I do not actually know what went wrong, but we can hope it will allow to workaround whatever issue is exhibited here.

[Lorilatschki]

I am sorry to ask again, but I am surprised not to see the build logs after the analysis steps. It should start with a line such as
Project xxxx analysis failed. The MsBuild log is: Started process 20798: "dotnet" --info.

Did you expect the Project xxxx analysis failed. The MsBuild log is: Started process 20798: "dotnet" --info to be part of the ouput in both cases, with and without the --diag option? Is there any other way to extend they output, in order to get the root cause, without sharing the project source?

[dupdob]

It should be part of the log with the --diag option.
I think mutation related logs appear afterward, so you could copy paste just the start of the file. Or simply share one of said log.

[Lorilatschki]

The diag.txt in this answer is the full diag log, did you took a look into that already?

[dupdob]

Yes, I did. Thanks for the confirmation. It looks there is small bug here that defeats the purpose of this evolution.
Update: thanks to your remark, I noticed something fishy:

[09:56:14 VRB] **** Buildalyzer result ****
Project: /builds/test/Add.Cli.Test/Add.Cli.Test.csproj
TargetFramework: net8.0
Succeeded: False
...
[09:56:14 DBG] Analysis of project test/Add.Cli.Test/Add.Cli.Test.csproj succeeded

The Succeeded: False signals that the analysis is incomplete. While
Analysis of project test/Add.Cli.Test/Add.Cli.Test.csproj succeeded
signals that Stryker thinks it has enough information. That's what the log was hoping to find was not here.

This findings reveals two things:

• the problem happens down the road in terms of initialization logic
• my proposed fix won't fix anything