Review and improve test executions

[donpui]

Review tests executions implementation. This should cover UNIT, INTEGRATION and E2E tests

• improve stability on tests in Github CI
• get rid of retries if possible
• improve speed of tests execution

Issue related with some workarounds applied: #87

[btlogy]

In #87, @donpui said:

Currently when we commit and pull request, two the same tests are running.

I understand we should avoid retrying, but do we need concurrent test at all?
As I'm trying to figure this out for other repositories, I've found this to be pretty common in .github/actions:

on:
  push:
    branches:
    - main
  pull_request:

This should avoid the test to be running twice when pull_request are made from branch created in the same repository (and not from forked one as I understand).
And if any one still want to manually trigger a test in a branch w/o creating a pull_request, we might want to add a workflow_dispatch event.

I've created #101 for this. Could that one be enough to tick the first task of this issue @donpui?

[btlogy]

Now #101 has been merged, it should avoid running test concurrently for the same content.
However, concurrent tests may still occur whenever 2+ branches are updated with different commits around the same time.
Unless we want to ensure our integration test support concurrency, I guess we need to dig into the documentation to avoid it.
After what, we could disable the retry introduced in #87.

[meejah]

I don't understand how two parallel runs can interfere? Is there some global service(s) some of the tests rely on or...?

[btlogy]

I don't understand how two parallel runs can interfere? Is there some global service(s) some of the tests rely on or...?

@donpui will correct me, but the problem I've bumped into in #92 had thrown Exceeded timeout of 5000 ms for a test:

• failed 7 days ago in 6m 26s

Github won't allow me to search the logs, but I could smell that test failing after 6-7 minutes were good candidate:

• failed Nov 4, 2022 in 7m 37s
• failed Nov 1, 2022 in 6m 26s
• failed Nov 1, 2022 in 6m 46s
• failed Oct 6, 2022 in 7m 1s
  ...

@meejah, I suspect Github could be that global service.
Would that not make sense if they were limiting the resources available for the runners per repository?
@wuan: do you know if this could have something about what our 'spending limit'?

[wuan]

@btlogy, the runner time of a public repository does not affect the spending limit.

[meejah]

IME parallel runs are fine (e.g. if two branches are pushed "at the same time") in lots of other repos we work on.

[donpui]

@btlogy you are correct, we are talking about the same timeout error.

We are not using global services (I mean, our public services), however we run up local mailbox, relay and client docker containers in side github runner virtual machine. It can be that something just start earlier, then other servizes are fully operational. For example, when winden docker starts, it takes some time, while all wasm, js recompiles.
Maybe optimisation or refactor on client-ci is needed.

FYI. @JustusFT

[meejah]

If the tests are slow, okay they're slow -- but there's no reason to avoid concurrent runs (or stated the other way: how does avoiding concurrent runs help slow tests?).

[donpui]

Adding reference, there is Draft PR to try alternative way to run tests #105

[btlogy]

While working on #171 and its fix #173, I've noticed the e2e test were randomly failing nearly 2 third of the time!
I had to re-build a couple of time the same commit to pass the check!

I'm not 100% sure the stability part of this issue was already covering these failure (too long ago).
But this is rather annoying when trying to fix the other little issues I've found.

[btlogy]

In the context of #176, I've improved the workflow in #183 with some caching which should speedup the testes...