Consider only running tests for roles that change in a Pull Request

[ehelms]

The more roles we add, the longer the test suite and the more PRs will suffer form intermittent outages. We should see if its possible to detect with Github Actions only that which changed, identify the roles and then reduce the set of tests to just those roles. Ideally, the matrix would reflect this but that seems much harder.

[mdellweg]

I believe you can reference the target branch of the PR. So you might do a git diff --stat and look for role directories.
The question is, can you map them reliably to the tests?

[evgeni]

Given the tests live inside the roles, yes, for this collection we can.

We probably also should disable the FAM-inherited tests as they are empty right now.

[ehelms]

I included a version of tackling this (e8feb48) given this PR keeps failing on katello repositories that we see 50% of the time.

[ehelms]

Just making a note that I tried this in a PR (and reverted the PR):

          roles=$(git diff $GITHUB_BASE_REF --stat | grep roles | cut -d "/" -f 2 | uniq)

This wouldn't work on merge tests, and it doesnt address when one role requires another for testing or as a dependency.

[evgeni]

I think we should rather focus on whether we can run those tests in parallel.

Running tests conditionally has been a source of issues in the past (in other repos) and I'd prefer not to do it here.