kafka-streams-test-utils dependency missing

[retzkek]

org.apache.kafka/kafka-streams-test-utils is getting included with scope test, so isn't included in the distribution, but is needed by jackdaw.data since 0.9.5 (https://github.com/FundingCircle/jackdaw/blob/master/src/jackdaw/data/producer.clj#L10).

https://repo.clojars.org/fundingcircle/jackdaw/0.9.12/jackdaw-0.9.12.pom:

<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams-test-utils</artifactId>
<version>3.3.2</version>
<scope>test</scope>
</dependency>

Observed behavior:

$ lein repl                          
#error {                                                            
 :cause org.apache.kafka.streams.test.TestRecord    
 :via                                                               
 [{:type clojure.lang.Compiler$CompilerException
   :message Syntax error macroexpanding at (jackdaw/data/producer.clj:7:1).
   :data #:clojure.error{:phase :execution, :line 7, :column 1, :source jackdaw/data/producer.clj}
   :at [clojure.lang.Compiler load Compiler.java 7665]}
  {:type java.lang.ClassNotFoundException
   :message org.apache.kafka.streams.test.TestRecord
   :at [java.net.URLClassLoader findClass URLClassLoader.java 445]}] 
...
jackdaw-test.core=> (require 'jackdaw.admin)
Syntax error compiling at (jackdaw/admin.clj:1:1).
namespace 'jackdaw.data' not found

with only the following dependencies:

  :dependencies [[org.clojure/clojure "1.11.1"]
                 [fundingcircle/jackdaw "0.9.12"]]

But add [org.apache.kafka/kafka-streams-test-utils "3.3.2"] to the project and it works as expected.

I don't know enough about java/maven packaging to suggest the correct fix.

[retzkek]

Oh, maybe it's as simple as removing it from the :dev dependencies, that must be overriding the scope (it's also in the default project dependencies)? https://github.com/FundingCircle/jackdaw/blob/master/project.clj#L80

$ git diff
diff --git a/project.clj b/project.clj
index 550b5b4..9d0862f 100644
--- a/project.clj
+++ b/project.clj
@@ -77,7 +77,6 @@
               :injections [(require 'io.aviso.logging.setup)]
               :dependencies [[io.aviso/logging "1.0"]
                              [aleph/aleph "0.6.1"]
-                             [org.apache.kafka/kafka-streams-test-utils "3.3.2"]
                              [org.apache.kafka/kafka-clients "3.3.2" :classifier "test"]
                              [org.clojure/test.check "1.1.1"]
                              [org.apache.kafka/kafka_2.13 "3.3.2"]
$ lein do pom
Wrote /tmp/jackdaw/pom.xml
$ grep -B2 -A2 test-utils pom.xml
    <dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka-streams-test-utils</artifactId>
      <version>3.3.2</version>
    </dependency>

[invisietch]

In case anyone gets this issue in future, you need to add the dependency for test-utils to your project's deps.edn:

org.apache.kafka/kafka-streams-test-utils {:mvn/version "3.3.2"}