Check version compatibility across Spark/Hadoop/Java and warn user when appropriate #322
Description
With the Spark 3.0 transition, there are some version compatibility issues that will likely hit users more often than normal.
- Spark is pre-built either against Hadoop 2.6, 2.7, or 3.2, depending on the Spark version.
- Hadoop only supports Java 8 in Hadoop <= 3.2, whereas Hadoop 3.3+ supports Java 8 or 11.
It's easy to pick a bad combination of Hadoop version, Spark version, and Spark/Hadoop build. Flintrock should check the user's choices and display a warning if it thinks the combination will not work.