Bug: Running the tests does not work on Mac OS out of the box

[lwb4]

Expected behavior

Following the steps in Src/Newtonsoft.Json-for-Unity.Tests/ as-is successfully guides the user to run the tests on all platforms.

(I am assuming that Mac OS is a supported platform since it is required to build iOS.)

Actual behavior

ci/local_build_into_package.ps1 works, but running the tests does not. I tried all three methods.

Running via command line

See output_unity_runtests.txt for the full output. Relevant error sections here:

Assets/Newtonsoft.Json.Tests.Aot/EnsureTypes.cs(4,23): error CS0234: The type or namespace name 'Utilities' does not exist in the namespace 'Newtonsoft.Json' (are you missing an assembly reference?)
Assets/UnityTests/HttpUtilityTests.cs(12,21): error CS0103: The name 'HttpUtility' does not exist in the current context
Assets/UnityTests/HttpUtilityTests.cs(19,21): error CS0103: The name 'HttpUtility' does not exist in the current context

Running via Unity Editor

Same errors as running via command line, but in the editor console.

Running via Docker

(This is not my main issue, as I'd like to be able to run the tests without Docker, but noting here for completeness.)

➜  Newtonsoft.Json-for-Unity git:(master) $ ci/local_test_in_unity_container.ps1 -VolumeSource $(pwd)
Resolve-Path: /Users/lincolnbergeson/Workspace/Newtonsoft.Json-for-Unity/ci/local_test_in_unity_container.ps1:49
Line |
  49 |      Resolve-Path (Join-Path "$PSScriptRoot" "Unity_v$UnityVersion.ulf …
     |      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Cannot find path '/Users/lincolnbergeson/Workspace/Newtonsoft.Json-for-Unity/ci/Unity_v2019.2.11f1.ulf' because it does not
     | exist.

ls ci shows no .ulf files in that directory:

docker_build_images.ps1           local_build_into_test_project.ps1 package-builder.Dockerfile        package-unity-tester.Dockerfile
docker_publish_images.ps1         local_docker_run.ps1              package-deploy-github.Dockerfile  scripts
local_build_into_package.ps1      local_test_in_unity_container.ps1 package-deploy-npm.Dockerfile     version.json

Steps to reproduce

Full repro in a fork at https://github.com/lwb4/Newtonsoft.Json-for-Unity

First I installed the dependencies:

• Install Powershell via Homebrew
• Install Docker Desktop via their website
• Install Unity version 2019.2.11f1 via Unity Hub

Then, to repro the bug:

• Clone this repo
• mkdir Src/Newtonsoft.Json-for-Unity.Tests/Assets/Newtonsoft.Json.Tests
• cp -R Src/Newtonsoft.Json.Tests/* Src/Newtonsoft.Json-for-Unity.Tests/Assets/Newtonsoft.Json.Tests/ (No need to remove bin or obj folders, since none were found)
• ci/local_build_into_package.ps1 (output attached in output_local_build_into_package_ps1.txt)
• /Applications/Unity/Hub/Editor/2019.2.11f1/Unity.app/Contents/MacOS/Unity -runTests -batchmode -projectPath Src/Newtonsoft.Json-for-Unity.Tests -testResults results.xml -testPlatform iOS -logFile output_unity_runtests.txt

output_ci_local_build_into_package_ps1.txt
output_unity_runtests.txt

Details

Host machine OS running Unity Editor: Macbook Pro 2019 running Mac OS Big Sur Version 11.2.3

Unity build target: iOS targeted above, but the default testPlatform parameter (editmode) doesn't work either.

Newtonsoft.Json-for-Unity package version: Latest master from this repo

I was using Unity version 2019.2.11f1

Checklist

• Shutdown Unity, deleted the /Library folder, opened project again in Unity, and problem still remains.
• Checked to be using latest version of the package.

[applejag]

Hi! Thanks for reporting this! Didn't know anyone even inspected those scripts. No doubt that you're having issues with these scripts; they are not well maintained (by me).

I'm really sorry about the lack of documentation and outdated scripts here. I've let those scripts rot because I'm more or less only testing via CircleCI atm.

The ULF file that it needs is intentionally omitted from the repo as it contains personal authentication tokens for Unity.
I did writeup some kind of guide a long time ago here: https://github.com/jilleJr/Newtonsoft.Json-for-Unity.Converters/blob/master/Build/CIRCLECI_SETUP.md#obtain-unity-license-ulf-files No clue how relevant it is anymore. Don't even know why I have that added to my converters repo but not this one.

I will look into this and start cleaning up the scripts and such. I want to be able to run the tests in the editor locally as well, and I remember that I got it working before at some point. It was the only way I could get it to run tests with IL2CPP.

But! I cannot promise a timeframe here. Everytime I touch this project I seem to do a little bit of work and then kind of get burned out by it.

[japhib]

Thanks for the quick response! (I'm working with @lwb4) We were trying to troubleshoot other issues, were digging in and this seemed like it might be related, but then we were able to fix our issues (for now) by following the directions in the wiki for using AotHelper. So ... cool if this gets fixed eventually, but I don't think we're going to be worrying about this much more at the time being.

[applejag]

Good to know! Glad that you got it resolved :)

I'll want to keep this issue open so I don't forget about it.

[lwb4]

Yeah, I found this issue while trying to get a repro for the other issue, so no big deal.

That said, a simple message at the top of that README saying "these scripts are out of date and don't work" would have saved me a lot of time! Just a suggestion. :)

[applejag]

@lwb4 That's a great note. I can really understand your frustration here, even though you're not saying it, as I've been there myself sometimes. :) Will add that note