SQL vector support across JDBC/R2DBC with dialect-aware scoring by n0tl3ss · Pull Request #3637 · micronaut-projects/micronaut-data

n0tl3ss · 2025-12-08T16:24:08Z

Adds end-to-end vector capabilities across Oracle/PostgreSQL/MySQL in data-model, data-runtime, data-jdbc, and data-r2dbc, including vector types, converters, binders/mappers, and dialect-specific schema/index generation.
Extends query support with vector scoring/search integration and dialect-aware behavior, plus stricter validation around scoring-function usage and parameter handling.
Expands coverage and usage examples significantly (tests + jdbc-vector doc examples + docs updates) to validate CRUD/search/index flows and sparse/dense vector handling across supported dialects.

data-jdbc/src/main/java/io/micronaut/data/jdbc/config/SchemaGenerator.java

...c/test/groovy/io/micronaut/data/jdbc/oraclexe/vector/OracleJdbcDoubleVectorEntitySpec.groovy

data-model/src/main/java/io/micronaut/data/model/Vector.java

...o/micronaut/data/runtime/operations/internal/query/DefaultBindableParametersStoredQuery.java

dstepanov · 2025-12-09T08:12:55Z

Can you try to implement it without changes in DataType and get/set mapping?

The idea would be to use DataType.OBJECT and use a custom AttributeConverter where the context is JdbcConversionContext/R2dbcConversionContext there we should probably also add the current dialect; by that we can fetch some dialect specific vector helper. (See PostgresEnumsSpec as an example to use it).
The vector data classes should be annotated with @TypeDef. We might want to introduce a new runtime interface that can also be implemented by AttributeConverter to let the schema generator know what type to put.

The idea would be to implement it in almost non-invasive way so other things like geo etc can be supported as well easily.

data-jdbc/src/main/java/io/micronaut/data/jdbc/operations/DefaultJdbcRepositoryOperations.java

data-model/src/main/java/io/micronaut/data/model/runtime/RuntimePersistentProperty.java

data-model/src/main/java/io/micronaut/data/model/runtime/convert/AttributeConverter.java

data-runtime/src/main/java/io/micronaut/data/runtime/mapper/sql/SqlResultEntityTypeMapper.java

data-model/src/main/java/io/micronaut/data/model/vector/DoubleVector.java

data-model/src/main/java/io/micronaut/data/model/vector/FloatVector.java

data-model/src/main/java/io/micronaut/data/model/vector/ByteVector.java

data-model/src/main/java/io/micronaut/data/model/vector/DoubleVector.java

data-model/src/main/java/io/micronaut/data/model/vector/IntVector.java

src/main/docs/guide/dbc/sqlMapping/sqlVectorType.adoc

Copilot

Pull request overview

Copilot reviewed 205 out of 223 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

data-r2dbc/src/main/java/io/micronaut/data/r2dbc/config/R2dbcSchemaGenerator.java:148

generate() logs every CREATE TABLE statement at WARN level (LOG.warn("Create table..." )), which will spam logs during normal application startup/tests even when nothing is wrong. Please downgrade this to DEBUG/TRACE (or remove it) and rely on the existing DataSettings.QUERY_LOG debug logging for visibility.

data-r2dbc/src/main/java/io/micronaut/data/r2dbc/operations/OracleR2dbcVectorBindSupport.java

...o/micronaut/data/runtime/operations/internal/query/DefaultBindableParametersStoredQuery.java

data-jdbc/src/test/groovy/io/micronaut/data/jdbc/h2/many2many/ManyToManyJoinTableSpec.groovy

radovanradic · 2026-03-31T12:24:41Z

.../java/io/micronaut/data/model/runtime/convert/vector/SparseByteVectorAttributeConverter.java

@@ -0,0 +1,31 @@
+/*
+ * Copyright 2017-2021 original authors


Maybe could update year since it may be confusing since it is since 5.0.x (there are other classes in the package to be updated)

Copilot

Pull request overview

Copilot reviewed 205 out of 223 changed files in this pull request and generated 3 comments.

Copilot · 2026-04-01T09:18:09Z

data-model/src/main/java/io/micronaut/data/model/vector/FloatVector.java

+public record FloatVector(float[] data) implements Vector {
+
+
+    /**
+     * Creates a float vector.
+     *
+     * @param data the backing values
+     */
+    public FloatVector {
+        Objects.requireNonNull(data, "FloatVector data must not be null");
+    }


These vector records expose the backing array directly via the record component accessor (data()), and the canonical constructor does not defensively copy the input array. That makes the type effectively mutable (callers can mutate the array after construction, or through data()), breaking the "immutable vector" contract and potentially invalidating equals/hashCode. Consider defensively copying in the canonical constructor and overriding data() to return a copy (or avoid using a record for array-backed immutables).

Copilot · 2026-04-01T09:18:10Z

data-model/src/main/java/io/micronaut/data/model/vector/DoubleVector.java

+public record DoubleVector(double[] data) implements Vector {
+
+
+    /**
+     * Creates a double vector.
+     *
+     * @param data the backing values
+     */
+    public DoubleVector {
+        Objects.requireNonNull(data, "DoubleVector data must not be null");
+    }


DoubleVector has the same mutability issue as FloatVector: the record component accessor returns the internal double[] and the constructor doesn't copy the input array. This allows external mutation and can break the immutability and equals/hashCode expectations. Defensively copy in the constructor and override data() to return a copy (or switch away from records for array-backed values).

Copilot · 2026-04-01T09:18:10Z

data-model/src/main/java/io/micronaut/data/model/vector/ByteVector.java

+public record ByteVector(byte[] data) implements Vector {
+
+
+    /**
+     * Creates a byte vector.
+     *
+     * @param data the backing values
+     */
+    public ByteVector {
+        Objects.requireNonNull(data, "ByteVector data must not be null");
+    }


ByteVector is declared as immutable but, as a record, it exposes its internal byte[] via the generated data() accessor and does not defensively copy the array in the constructor. Callers can mutate the backing storage, which can break equality/hashCode and lead to subtle bugs. Defensively copy the array on construction and override data() to return a copy (or avoid using a record for array-backed types).

Copilot

Pull request overview

Copilot reviewed 205 out of 223 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (7)

data-r2dbc/build.gradle:1

The dependency alias mnSql.mysql.connector.j looks truncated and is very likely not defined in the version catalog (previously mnSql.mysql.connector.java). If the alias doesn't exist, this will break Gradle resolution across multiple modules—please restore the correct alias or add/rename the catalog entry consistently.
data-r2dbc/build.gradle:1
The dependency alias mnSql.mysql.connector.j looks truncated and is very likely not defined in the version catalog (previously mnSql.mysql.connector.java). If the alias doesn't exist, this will break Gradle resolution across multiple modules—please restore the correct alias or add/rename the catalog entry consistently.
doc-examples/r2dbc-example-kotlin/src/main/resources/application.yml:1
The container tag container-registry.oracle.com/mysql/community-server:9.6.0 is likely not a valid MySQL Server version/tag and may fail pulls in CI. Consider using a known published MySQL tag (e.g., an 8.4.x LTS tag) or the existing mysql:<version> image used elsewhere, and avoid registries that require authentication unless that’s explicitly handled.
data-runtime/src/main/java/io/micronaut/data/runtime/operations/internal/sql/SqlJsonColumnMapperProvider.java:1
This changes both the constructor signature and reduces its visibility (from public to package-private). Since the class itself is public, this is a source/binary compatibility break for any external code constructing it. If the intent is DI-only wiring, keep the constructor public (or add a public delegating constructor overload) while adding the new dependency.
data-processor/src/main/java/io/micronaut/data/processor/visitors/finders/FindersUtils.java:1
SearchResults<T> represents a multi-row result container, but this branch routes it through FindOne* interceptors (sync/reactive/async). That is very likely to only read/map a single row and silently drop the rest. A safer approach is to route SearchResults through an interceptor that iterates all rows (similar to FindAll*) and then wraps them into SearchResults.
data-runtime/src/main/java/io/micronaut/data/runtime/operations/internal/query/DefaultBindableParametersStoredQuery.java:1
This uses Java record deconstruction patterns (value instanceof Score(double score) / Similarity(double similarity)), which require Java 21+. If the project/toolchain baseline is below 21, this will not compile; use standard instanceof Score score and score.value() accessors instead (or ensure the build enforces Java 21+).
data-r2dbc/src/main/java/io/micronaut/data/r2dbc/mapper/ColumnNameR2dbcResultReader.java:1
getRequiredValue now returns null when conversion is not possible (convert(...).orElse(null)), which can mask type/codec issues and violates the usual expectation implied by 'Required'. Consider throwing a DataAccessException when conversion returns empty (or using a convertRequired-style path) so mapping failures are surfaced deterministically.

Copilot

Pull request overview

Copilot reviewed 203 out of 221 changed files in this pull request and generated 8 comments.

data-model/src/main/java/io/micronaut/data/model/vector/FloatVector.java

data-model/src/main/java/io/micronaut/data/model/vector/DoubleVector.java

data-model/src/main/java/io/micronaut/data/model/vector/ByteVector.java

data-runtime/src/main/java/io/micronaut/data/runtime/query/internal/DefaultPreparedQuery.java

Copilot · 2026-04-02T10:11:39Z

data-runtime/src/main/java/io/micronaut/data/runtime/convert/DataConversionServiceFactory.java


        // LocalDate
        conversionService.addConverter(LocalDate.class, java.sql.Date.class, java.sql.Date::valueOf);
+        conversionService.addConverter(LocalDate.class, Date.class, localDate -> Date.from(localDate.atStartOfDay(ZoneId.systemDefault()).toInstant()));



There is already a ChronoLocalDate -> java.util.Date converter registered earlier in this factory. Adding a second LocalDate -> Date converter is redundant (LocalDate implements ChronoLocalDate) and can make converter selection harder to reason about. Prefer relying on the existing ChronoLocalDate converter, or document why the more specific converter is necessary.

data-r2dbc/src/main/java/io/micronaut/data/r2dbc/mapper/ColumnNameR2dbcResultReader.java

Copilot · 2026-04-02T10:11:40Z

data-r2dbc/src/test/groovy/io/micronaut/data/r2dbc/postgres/vector/PostgresDbInit.java

+                // Ensure pgvector extension and demo table for vector tests
+                try (CallableStatement st = connection.prepareCall("CREATE EXTENSION IF NOT EXISTS vector;")) {
+                    st.execute();
+                } catch (SQLException e) {
+                    // Ignore if not available or already exists
+                }


This initializer is meant to ensure the vector extension exists for vector specs, but it currently swallows any SQLException from CREATE EXTENSION .... If the extension is missing or permissions prevent creation, tests will fail later with harder-to-diagnose errors. Consider letting the exception propagate (or at least logging) for this vector-specific init path.

data-r2dbc/src/main/java/io/micronaut/data/r2dbc/mapper/ColumnNameR2dbcResultReader.java

sonarqubecloud · 2026-04-03T10:11:41Z

Quality Gate passed

Issues
35 New issues
0 Accepted issues

Measures
0 Security Hotspots
73.7% Coverage on New Code
4.1% Duplication on New Code

See analysis details on SonarQube Cloud

Copilot

Pull request overview

Copilot reviewed 204 out of 222 changed files in this pull request and generated no new comments.

Support Oracle db VECTOR type

cf28195

dstepanov requested changes Dec 9, 2025

View reviewed changes

dstepanov approved these changes Dec 9, 2025

View reviewed changes

...o/micronaut/data/runtime/operations/internal/query/DefaultBindableParametersStoredQuery.java Outdated Show resolved Hide resolved

n0tl3ss added 8 commits December 10, 2025 15:50

refactor Vector type

70359a5

remove gradle properties

efd704e

add support for multiple dialects and adds postgres vectors

cd72557

Merge branch '5.0.x' into support-vector-type-oracledb

d4913f4

fix requires

0098291

refactor for r2dbc

8196182

fix tests

3996937

fix tests

683b1dd

dstepanov requested changes Dec 12, 2025

View reviewed changes

n0tl3ss added 16 commits December 13, 2025 00:41

fix impl

f3268f0

fix postgres

6180793

fix postgres

ab18770

fix test

e59b8f1

fix result reader

775a883

remove extra annotation

ef095f8

remove setup from tests

f700207

improve docs

d094e09

improve docs

1041e83

fix tests

f4eee69

fix tests

2e2a76b

cleanup

5ab85e7

remove mapping

15d3fcb

fix bugs and duplicate code

c3dab64

fix test and duplication

b68fbf1

add more tests

7dc2cb1

graemerocher requested changes Dec 15, 2025

View reviewed changes

Copilot started reviewing on behalf of n0tl3ss March 31, 2026 11:34 View session

graemerocher approved these changes Mar 31, 2026

View reviewed changes

Copilot AI reviewed Mar 31, 2026

View reviewed changes

data-r2dbc/src/main/java/io/micronaut/data/r2dbc/operations/OracleR2dbcVectorBindSupport.java Outdated Show resolved Hide resolved

...o/micronaut/data/runtime/operations/internal/query/DefaultBindableParametersStoredQuery.java Show resolved Hide resolved

radovanradic reviewed Mar 31, 2026

View reviewed changes

data-jdbc/src/test/groovy/io/micronaut/data/jdbc/h2/many2many/ManyToManyJoinTableSpec.groovy Outdated Show resolved Hide resolved

radovanradic reviewed Mar 31, 2026

View reviewed changes

n0tl3ss added 2 commits March 31, 2026 15:51

small improvements

9664a06

remove test

259da60

Copilot AI review requested due to automatic review settings April 1, 2026 09:11

Copilot started reviewing on behalf of n0tl3ss April 1, 2026 09:12 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

n0tl3ss added 2 commits April 1, 2026 13:38

bump micronaut core to M19

5b69598

Merge branch '5.0.x' into support-vector-type-oracledb

2bda935

Copilot AI review requested due to automatic review settings April 1, 2026 11:38

Copilot AI reviewed Apr 1, 2026

View reviewed changes

Copilot started reviewing on behalf of n0tl3ss April 1, 2026 12:48 View session

n0tl3ss added 2 commits April 1, 2026 17:14

bring back old gradle.yml

0c6a977

remove global pgvector image from jdbc

0c70078

Copilot AI review requested due to automatic review settings April 2, 2026 10:00

Copilot started reviewing on behalf of n0tl3ss April 2, 2026 10:04 View session

Copilot AI reviewed Apr 2, 2026

View reviewed changes

n0tl3ss added 2 commits April 2, 2026 15:01

Merge branch '5.0.x' into support-vector-type-oracledb

142d02d

try to cleanup more space

db55340

Copilot AI review requested due to automatic review settings April 3, 2026 09:58

Copilot started reviewing on behalf of n0tl3ss April 3, 2026 10:01 View session

Copilot AI reviewed Apr 3, 2026

View reviewed changes

add one more test

21d869c

graemerocher added this to 5.0.0 Release Apr 3, 2026

graemerocher moved this to Ready for review in 5.0.0 Release Apr 3, 2026

graemerocher added the type: enhancement New feature or request label Apr 3, 2026

Conversation

n0tl3ss commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dstepanov commented Dec 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

radovanradic Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sonarqubecloud bot commented Apr 3, 2026

Quality Gate passed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

n0tl3ss commented Dec 8, 2025 •

edited

Loading