Introduce run ordering as a generic feature

Extending the initial implementation for run order randomisation, the
former is not merely a special case of run ordering. We now have
  - DefaultSpecOrderer (basically a no-op)
  - RandomSpecOrderer,
  - AlphabeticalSpecOrderer,
  - AnnotatationBasedSpecOrderer with @order(int).

Relates to #1443.

Author: kriegaex

docs/extensions.adoc

@@ -44,17 +44,17 @@ runner {
 
 See the <<parallel_execution.adoc#parallel-execution,Parallel Execution>> section for a detailed description.
 
-=== Random Test Order Configuration
+=== Test Order Configuration
 
 [source,groovy]
 ----
 runner {
-  randomizeSpecRunOrder true    // randomize specification run order
-  randomizeFeatureRunOrder true // randomize feature run order within each specification
+  orderer new RandomSpecOrderer()  // randomize specification run order
 }
 ----
 
-See the <<#_randomize_run_order>> section for more details.
+Instead of the default run order, you can configure Spock to execute specifications and/or features e.g. in random,
+alphabetical or manually assigned, annotation-based order. See the <<#_run_order>> section for more details.
 
 == Built-In Extensions
 
@@ -721,21 +721,111 @@ runner {
 }
 ----
 
-=== Randomize Run Order
+=== Run Order
 
 Ideally, automated tests in general and Spock specifications in particular should be independent of each other. The same
-applies to feature methods within a specification. One heuristic way to gain confidence that this is indeed the case, is
-to randomize the order in which specifications and/or features within each specification are executed.
+applies to feature methods within a specification. Therefore, you should not rely on any specific order of execution
+(run order).
 
-By default, Spock executes both specifications and features in deterministic order, even though users should not rely on
-any particular run order. You can explicitly activate run order randomization separately for specifications and features
-or combine both randomization modes, using the <<spock-configuration-file,Spock Configuration File>>:
+Nevertheless, you have options to influence the run order, using the <<spock-configuration-file>> and a set of built-in
+orderers derived from super class `SpecOrderer`. Please check the Javadocs for package
+`org.spockframework.runtime.extension.builtin.orderer` for more details. You can also write your own `SpecOrderer`, if
+none of the built-in ones satisfies your needs.
+
+Please note that `@Stepwise` always trumps any run order you might have configured, i.e. `@Stepwise` "wins" against
+`SpecOrderer`.
+
+==== Random Run Order
+
+One helpful way to heuristically increase your confidence that your tests are indeed independent of each other, is to
+explicitly say goodbye to deterministic run order by randomizing it.
+
+[source,groovy]
+----
+import org.spockframework.runtime.extension.builtin.orderer.RandomSpecOrderer
+
+runner {
+  orderer new RandomSpecOrderer(
+    true,                       // Randomize overall specification run order
+    true,                       // Randomize the run order of feature methods within specifications
+    System.currentTimeMillis()  // Set a fixed value, if you want repeatable pseudo-random numbers.
+                                // This might be helpful for reproducing issues when debugging your tests.
+  )
+}
+----
+
+==== Alphabetical Run Order
+
+Less useful than random run order, but available anyway, is a way to execute specifications and/or features
+alphabetically, based on their display names and a simple `String.compareTo(String)` (no fancy locale-based collation).
+The default sorting direction is ascending, optionally you can also sort elements in descending order.
 
 [source,groovy]
 ----
+import org.spockframework.runtime.extension.builtin.orderer.AlphabeticalSpecOrderer
+
 runner {
-  randomizeSpecRunOrder true    // randomize specification run order
-  randomizeFeatureRunOrder true // randomize feature run order within each specification
+  orderer new AlphabeticalSpecOrderer(
+    true,  // Run specifications in alphabetical order by display name
+    true,  // Run feature methods within specifications in alphabetical order by display name
+    false  // Sort in ascending order (use 'true' for descending order)
+   )
+}
+----
+
+==== Annotation-Based Run Order
+
+If you want to basically retain Spock's default run order for most or at least some of your specifications and/or
+feature methods, but modify it for particular specs/features, or take it to the extreme and manually assign run orders
+everywhere, use the `@Order(int)` annotation in combination with the annotation-based orderer:
+
+[source,groovy]
+----
+import org.spockframework.runtime.extension.builtin.orderer.AnnotatationBasedSpecOrderer
+
+runner {
+  orderer new AnnotatationBasedSpecOrderer()
+}
+----
+
+Please note, that `@Order` annotations have no effect whatsoever, if `AnnotatationBasedSpecOrderer` is not configured
+as the active orderer. E.g., you cannot expect to be able to use random ordering in combination with manually assigning
+run orders via annotations for some exceptions. Annotation-based ordering must be explicitly activated and is only
+available as a modification of Spock's default run order.
+
+Using `@Order`, the basic idea is to assume unannotated specifications and features to all carry an implicit `@Order(0)`
+annotation. If you wish to run some specs/features before others, assign them a lower (negative) run order. If you want
+to run them after the default-ordered elements, assign them a higher (positive) order number:
+
+[source,groovy]
+----
+@Order(1)  // Execute after default-ordered specs
+class FirstSpec extends Specification {
+  // Execute features in order 'three', 'one', 'two' in ascending order of assigned @Order values
+  @Order(2) def one() { expect: true }
+  @Order(3) def two() { expect: true }
+  @Order(1) def three() { expect: true }
+}
+
+@Order(-1)  // Execute before default-ordered specs
+class SecondSpec extends Specification {
+  def foo() { expect: true }                          // Default order
+  @Order(99) def bar() { expect: true }               // Execute after default-ordered features
+  @Order(-5) def zot() { expect: true }               // Execute before default-ordered features
+}
+
+// Default order
+class ThirdSpec extends Specification {
+  def "some feature"() { expect: true }               // Default order
+  @Order(1) def "another feature"() { expect: true }  // Execute after default-ordered features
+  def "one more feature"() { expect: true }           // Default order
+}
+
+// Default order
+class FourthSpec extends Specification {
+  def 'feature X'() { expect: true }                  // Default order
+  def 'feature M'() { expect: true }                  // Default order
+  @Order(-1) def 'feature D'() { expect: true }       // Execute before default-ordered features
 }
 ----
 

…ion/builtin/RandomRunOrderExtension.java …me/extension/builtin/OrderExtension.javaspock-core/src/main/java/org/spockframework/runtime/extension/builtin/RandomRunOrderExtension.java renamed to spock-core/src/main/java/org/spockframework/runtime/extension/builtin/OrderExtension.java spock-core/src/main/java/org/spockframework/runtime/extension/builtin/RandomRunOrderExtension.java renamed to spock-core/src/main/java/org/spockframework/runtime/extension/builtin/OrderExtension.java

@@ -15,28 +15,20 @@
 package org.spockframework.runtime.extension.builtin;
 
 import org.spockframework.runtime.extension.IGlobalExtension;
-import org.spockframework.runtime.model.FeatureInfo;
 import org.spockframework.runtime.model.SpecInfo;
 import spock.config.RunnerConfiguration;
 
-import java.util.Random;
-
-public class RandomRunOrderExtension implements IGlobalExtension {
-  private static final Random RANDOM = new Random();
+import java.util.Collection;
 
+public class OrderExtension implements IGlobalExtension {
   private final RunnerConfiguration config;
 
-  public RandomRunOrderExtension(RunnerConfiguration config) {
+  public OrderExtension(RunnerConfiguration config) {
     this.config = config;
   }
 
   @Override
-  public void visitSpec(SpecInfo spec) {
-    if (config.randomizeSpecRunOrder)
-      spec.setExecutionOrder(RANDOM.nextInt());
-    if (config.randomizeFeatureRunOrder) {
-      for (FeatureInfo featureInfo : spec.getAllFeatures())
-        featureInfo.setExecutionOrder(RANDOM.nextInt());
-    }
+  public void initSpecs(Collection<SpecInfo> specs) {
+    config.orderer.process(specs);
   }
 }

spock-core/src/main/java/org/spockframework/runtime/extension/builtin/orderer/AlphabeticalSpecOrderer.java

@@ -0,0 +1,47 @@
+package org.spockframework.runtime.extension.builtin.orderer;
+
+import org.spockframework.runtime.model.SpecInfo;
+
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class AlphabeticalSpecOrderer extends SpecOrderer {
+  private final boolean descending;
+
+  public AlphabeticalSpecOrderer(boolean orderSpecs, boolean orderFeatures, boolean descending) {
+    super(orderSpecs, orderFeatures);
+    this.descending = descending;
+  }
+
+  public AlphabeticalSpecOrderer(boolean orderSpecs, boolean orderFeatures) {
+    this(orderSpecs, orderFeatures, false);
+  }
+
+  @Override
+  protected void orderSpecs(Collection<SpecInfo> specs) {
+    AtomicInteger i = new AtomicInteger();
+    specs.stream()
+      .sorted((o1, o2) -> descending
+        ? o2.getDisplayName().compareTo(o1.getDisplayName())
+        : o1.getDisplayName().compareTo(o2.getDisplayName())
+      )
+      .forEach(specInfo -> specInfo.setExecutionOrder(i.getAndIncrement()));
+  }
+
+  @Override
+  protected void orderFeatures(Collection<SpecInfo> specs) {
+    for (SpecInfo spec : specs) {
+      AtomicInteger i = new AtomicInteger();
+      spec.getAllFeatures().stream()
+        .sorted((o1, o2) -> descending
+          ? o2.getDisplayName().compareTo(o1.getDisplayName())
+          : o1.getDisplayName().compareTo(o2.getDisplayName())
+        )
+        .forEach(featureInfo -> featureInfo.setExecutionOrder(i.getAndIncrement()));
+    }
+  }
+
+  public boolean isDescending() {
+    return descending;
+  }
+}

spock-core/src/main/java/org/spockframework/runtime/extension/builtin/orderer/AnnotatationBasedSpecOrderer.java

@@ -0,0 +1,32 @@
+package org.spockframework.runtime.extension.builtin.orderer;
+
+import org.spockframework.runtime.model.FeatureInfo;
+import org.spockframework.runtime.model.SpecInfo;
+import spock.lang.Order;
+
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class AnnotatationBasedSpecOrderer extends SpecOrderer {
+  public AnnotatationBasedSpecOrderer() {
+    super(true, true);
+  }
+
+  @Override
+  protected void orderSpecs(Collection<SpecInfo> specs) {
+    for (SpecInfo spec : specs) {
+      Order orderAnnotation = spec.getAnnotation(Order.class);
+      spec.setExecutionOrder(orderAnnotation == null ? 0 : orderAnnotation.value());
+    }
+  }
+
+  @Override
+  protected void orderFeatures(Collection<SpecInfo> specs) {
+    for (SpecInfo spec : specs) {
+      for (FeatureInfo feature : spec.getAllFeatures()) {
+        Order orderAnnotation = feature.getFeatureMethod().getAnnotation(Order.class);
+        feature.setExecutionOrder(orderAnnotation == null ? 0 : orderAnnotation.value());
+      }
+    }
+  }
+}

spock-core/src/main/java/org/spockframework/runtime/extension/builtin/orderer/DefaultSpecOrderer.java

@@ -0,0 +1,17 @@
+package org.spockframework.runtime.extension.builtin.orderer;
+
+import org.spockframework.runtime.model.SpecInfo;
+
+import java.util.Collection;
+
+public class DefaultSpecOrderer extends SpecOrderer {
+  public DefaultSpecOrderer() {
+    super(false, false);
+  }
+
+  @Override
+  protected void orderSpecs(Collection<SpecInfo> specs) { }
+
+  @Override
+  protected void orderFeatures(Collection<SpecInfo> specs) { }
+}

spock-core/src/main/java/org/spockframework/runtime/extension/builtin/orderer/RandomSpecOrderer.java

@@ -0,0 +1,39 @@
+package org.spockframework.runtime.extension.builtin.orderer;
+
+import org.spockframework.runtime.model.FeatureInfo;
+import org.spockframework.runtime.model.SpecInfo;
+
+import java.util.Collection;
+import java.util.Random;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class RandomSpecOrderer extends SpecOrderer {
+  private final Random random;
+
+  public RandomSpecOrderer(boolean orderSpecs, boolean orderFeatures, long seed) {
+    super(orderSpecs, orderFeatures);
+    random = new Random(seed);
+  }
+
+  public RandomSpecOrderer(boolean orderSpecs, boolean orderFeatures) {
+    this(orderSpecs, orderFeatures, System.currentTimeMillis());
+  }
+
+  public RandomSpecOrderer() {
+    this(true, true);
+  }
+
+  @Override
+  protected void orderSpecs(Collection<SpecInfo> specs) {
+    for (SpecInfo spec : specs)
+      spec.setExecutionOrder(random.nextInt());
+  }
+
+  @Override
+  protected void orderFeatures(Collection<SpecInfo> specs) {
+    for (SpecInfo spec : specs) {
+      for (FeatureInfo feature : spec.getAllFeatures())
+        feature.setExecutionOrder(random.nextInt());
+    }
+  }
+}

spock-core/src/main/java/spock/config/RunnerConfiguration.java

@@ -14,23 +14,26 @@
 
 package spock.config;
 
+import org.spockframework.runtime.extension.builtin.orderer.DefaultSpecOrderer;
+import org.spockframework.runtime.extension.builtin.orderer.SpecOrderer;
+
 /**
  * Configuration settings for the spec runner.
  *
  * <p>Example:
  * <pre>
+ * import org.spockframework.runtime.extension.builtin.orderer.RandomSpecOrderer
  * import some.pkg.Fast
  * import some.pkg.IntegrationSpec
  *
  * runner {
- *   include Fast // could be either an annotation or a (base) class
+ *   include Fast  // could be either an annotation or a (base) class
  *   exclude {
  *     annotation some.pkg.Slow
  *     baseClass IntegrationSpec
  *   }
- *   filterStackTrace true          // this is the default
- *   randomizeSpecRunOrder false    // this is the default
- *   randomizeFeatureRunOrder false // this is the default
+ *   filterStackTrace true            // this is the default
+ *   orderer new RandomSpecOrderer()  // DefaultSpecOrderer (no-op) is the default
  * }
  * </pre>
  */
@@ -39,8 +42,7 @@ public class RunnerConfiguration {
   public IncludeExcludeCriteria include = new IncludeExcludeCriteria();
   public IncludeExcludeCriteria exclude = new IncludeExcludeCriteria();
   public ParallelConfiguration parallel = new ParallelConfiguration();
+  public SpecOrderer orderer = new DefaultSpecOrderer();
   public boolean filterStackTrace = true;
   public boolean optimizeRunOrder = false;
-  public boolean randomizeSpecRunOrder = false;
-  public boolean randomizeFeatureRunOrder = false;
 }

spock-core/src/main/java/spock/lang/Order.java

@@ -0,0 +1,38 @@
+/*
+ * Copyright 2023 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package spock.lang;
+
+import org.spockframework.runtime.extension.builtin.OrderExtension;
+import org.spockframework.runtime.extension.builtin.orderer.AnnotatationBasedSpecOrderer;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * Assigns an execution order to a specification or feature.
+ * <p>
+ * Annotations of this type are picked up by the {@link OrderExtension}, if and only if the
+ * {@link AnnotatationBasedSpecOrderer} is activated in the Spock configuration file: <pre>
+ * runner {
+ *   orderer new AnnotatationBasedSpecOrderer()
+ * }</pre>
+ */
+@Retention(RetentionPolicy.RUNTIME)
+@Target({ ElementType.TYPE, ElementType.METHOD })
+public @interface Order {
+  int value();
+}