Skip to content

Commit 27594a8

Browse files
pflookypflooky
authored
Bugfix/jdbc UI (#124)
* fix(ui): missing dbtable in connection options for jdbc connection * feat(build): build for different os architectures and fix link for downloads * feat(ui): implement dynamic API configuration and replace fetch calls with apiFetch for improved error handling * doc(changelog): update changelog with hostname resolution fix * fix: add in multiline pipe for build.yml * feat(build): enhance versioning and publishing logic in Docker action and GitHub workflow - Updated docker-action.sh to determine version based on branch and commit hash. - Added conditional logic to publish to Maven only on the main branch. - Modified build.yml to set version dynamically based on the branch, supporting feature and bugfix branches. * fix: change pattern for linux artifact names * fix: change pattern for linux artifact names * fix: add in cors handler * feat(connection): implement connection testing functionality and enhance connection management - Added connection testing feature in the UI, allowing users to test connections before saving. - Updated Connection model to include source information for connections (file or config). - Enhanced ConnectionRepository to manage connections from files and application.conf, ensuring proper handling of deletable and non-deletable connections. - Introduced ConnectionTestService to handle various connection types and provide detailed test results. - Updated UI components to reflect connection source and added test buttons for existing connections. * doc: update changelog for 0.17.3 * feat(ui): implement toast history management and error detail modal - Added functionality to manage toast notifications history, allowing users to view past notifications. - Introduced a modal for displaying detailed error messages, enhancing user experience during error handling. - Updated UI components to include a button for accessing toast history and integrated event listeners for modal interactions. - Refactored existing toast creation logic to support history tracking and badge updates. - Added unit tests for toast history functions to ensure reliability and correctness. * feat(cors): enhance CORS configuration and logging - Introduced configurable CORS settings via environment variables, allowing for stricter security controls. - Updated PlanRoutes to log CORS configuration at startup for better visibility. - Refactored connection testing service to utilize a shared SparkSession, improving performance and resource management. - Fixed foreign key property name handling to align with backend expectations, ensuring proper loading of foreign key links in the UI. * feat(ui): update navigation and add shutdown confirmation modal - Replaced shutdown link with a button that triggers a confirmation modal to prevent accidental server shutdowns. - Moved toast history button to the navigation bar for better accessibility and to avoid overlap with page content. - Removed unused login button from the UI to streamline the interface. * feat(cache): implement cache control for static UI resources - Added Cache-Control headers to responses for static UI resources, setting a max-age of 1 day and requiring revalidation. - Updated routes in PlanRoutes to include cache control for HTML and SVG resources, improving performance and resource management. * fix(ui): improve error handling in login and plan execution - Updated error handling in saveCredentials and executePlan functions to throw errors instead of rejecting promises, enhancing error propagation and debugging capabilities. - This change ensures that errors are properly caught and logged, improving overall reliability of the UI components. * fix: only log record count below expected count when it doesn't match, add in format for jdbc connections from config, use separate config file for integration tests
1 parent 998e146 commit 27594a8

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+11181
-316
lines changed

‎.github/workflows/build.yml‎

Lines changed: 137 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ on:
44
push:
55
branches:
66
- main
7+
- 'feature/**'
8+
- 'bugfix/**'
79

810
jobs:
911
build:
@@ -13,6 +15,21 @@ jobs:
1315
- uses: actions/checkout@v4
1416
with:
1517
fetch-depth: 2
18+
- name: Set version based on branch
19+
id: version
20+
run: |
21+
BASE_VERSION=$(grep version gradle.properties | cut -d= -f2)
22+
COMMIT_HASH=$(git rev-parse --short HEAD)
23+
24+
if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
25+
VERSION="$BASE_VERSION"
26+
else
27+
VERSION="$BASE_VERSION-$COMMIT_HASH"
28+
fi
29+
30+
echo "VERSION=$VERSION" >> $GITHUB_ENV
31+
echo "version=$VERSION" >> $GITHUB_OUTPUT
32+
echo "Building version: $VERSION"
1633
- uses: actions/setup-java@v4
1734
with:
1835
java-version: '17'
@@ -27,6 +44,8 @@ jobs:
2744
- name: Build and push images
2845
run: bash docker-action.sh
2946
env:
47+
VERSION: ${{ env.VERSION }}
48+
GITHUB_REF: ${{ github.ref }}
3049
PACKAGE_TOKEN: ${{ secrets.PACKAGE_TOKEN }}
3150
MAVEN_USERNAME: ${{ secrets.MAVEN_USERNAME }}
3251
MAVEN_PASSWORD: ${{ secrets.MAVEN_PASSWORD }}
@@ -58,7 +77,18 @@ jobs:
5877
with:
5978
fetch-depth: 2
6079
- name: Set version
61-
run: echo "APP_VERSION=$(grep version gradle.properties | cut -d= -f2)" >> $GITHUB_ENV
80+
run: |
81+
BASE_VERSION=$(grep version gradle.properties | cut -d= -f2)
82+
COMMIT_HASH=$(git rev-parse --short HEAD)
83+
84+
if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
85+
APP_VERSION="$BASE_VERSION"
86+
else
87+
APP_VERSION="$BASE_VERSION-$COMMIT_HASH"
88+
fi
89+
90+
echo "APP_VERSION=$APP_VERSION" >> $GITHUB_ENV
91+
echo "Building version: $APP_VERSION"
6292
- uses: actions/setup-java@v4
6393
with:
6494
java-version: '21'
@@ -90,7 +120,18 @@ jobs:
90120
with:
91121
fetch-depth: 2
92122
- name: Set version
93-
run: echo "APP_VERSION=$(grep version gradle.properties | cut -d= -f2)" >> $env:GITHUB_ENV
123+
run: |
124+
$BASE_VERSION = (Get-Content gradle.properties | Select-String '^version=' | ForEach-Object { $_ -replace 'version=','' }).Trim()
125+
$COMMIT_HASH = git rev-parse --short HEAD
126+
127+
if ("${{ github.ref }}" -eq "refs/heads/main") {
128+
$APP_VERSION = $BASE_VERSION
129+
} else {
130+
$APP_VERSION = "$BASE_VERSION-$COMMIT_HASH"
131+
}
132+
133+
echo "APP_VERSION=$APP_VERSION" >> $env:GITHUB_ENV
134+
Write-Output "Building version: $APP_VERSION"
94135
- uses: actions/setup-java@v4
95136
with:
96137
java-version: '21'
@@ -113,25 +154,27 @@ jobs:
113154
path: "DataCaterer-${{ env.APP_VERSION }}-windows-x86_64.exe"
114155
overwrite: true
115156

116-
linux:
157+
linux-amd64:
117158
needs: build
118-
strategy:
119-
matrix:
120-
include:
121-
- runner: ubuntu-latest
122-
arch: x64
123-
arch_name: amd64
124-
- runner: ubuntu-latest
125-
arch: aarch64
126-
arch_name: arm64
127-
runs-on: ${{ matrix.runner }}
159+
runs-on: [ubuntu-latest]
128160

129161
steps:
130162
- uses: actions/checkout@v4
131163
with:
132164
fetch-depth: 2
133165
- name: Set version
134-
run: echo "APP_VERSION=$(grep version gradle.properties | cut -d= -f2)" >> $GITHUB_ENV
166+
run: |
167+
BASE_VERSION=$(grep version gradle.properties | cut -d= -f2)
168+
COMMIT_HASH=$(git rev-parse --short HEAD)
169+
170+
if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
171+
APP_VERSION="$BASE_VERSION"
172+
else
173+
APP_VERSION="$BASE_VERSION-$COMMIT_HASH"
174+
fi
175+
176+
echo "APP_VERSION=$APP_VERSION" >> $GITHUB_ENV
177+
echo "Building version: $APP_VERSION"
135178
- uses: actions/setup-java@v4
136179
with:
137180
java-version: '21'
@@ -143,13 +186,87 @@ jobs:
143186
with:
144187
name: jars
145188
path: app/build/libs/
146-
- name: Package jar as debian package
189+
- name: Package jar as debian package (amd64)
147190
run: 'jpackage --main-jar data-caterer.jar "@misc/jpackage/jpackage.cfg" "@misc/jpackage/jpackage-linux.cfg"'
148-
- name: Rename DEB with architecture
149-
run: mv datacaterer_*_*.deb datacaterer_${{ env.APP_VERSION }}_${{ matrix.arch_name }}.deb
150-
- name: Upload deb
191+
- name: Rename deb with version
192+
run: |
193+
DEB_FILE=$(ls datacaterer_*_amd64.deb 2>/dev/null | head -n 1)
194+
if [ -n "$DEB_FILE" ]; then
195+
echo "Found deb file: $DEB_FILE"
196+
mv "$DEB_FILE" datacaterer_${{ env.APP_VERSION }}_amd64.deb
197+
echo "Renamed to: datacaterer_${{ env.APP_VERSION }}_amd64.deb"
198+
else
199+
echo "No deb file found"
200+
echo "Current directory:"
201+
pwd
202+
echo "Files in current directory:"
203+
ls -lart
204+
exit 1
205+
fi
206+
- name: Upload deb (amd64)
207+
uses: actions/upload-artifact@v4
208+
with:
209+
name: data-caterer-linux-amd64
210+
path: "datacaterer_${{ env.APP_VERSION }}_amd64.deb"
211+
overwrite: true
212+
213+
linux-arm64:
214+
needs: build
215+
runs-on: [ubuntu-latest]
216+
217+
steps:
218+
- uses: actions/checkout@v4
219+
with:
220+
fetch-depth: 2
221+
- name: Set version
222+
run: |
223+
BASE_VERSION=$(grep version gradle.properties | cut -d= -f2)
224+
COMMIT_HASH=$(git rev-parse --short HEAD)
225+
226+
if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
227+
APP_VERSION="$BASE_VERSION"
228+
else
229+
APP_VERSION="$BASE_VERSION-$COMMIT_HASH"
230+
fi
231+
232+
echo "APP_VERSION=$APP_VERSION" >> $GITHUB_ENV
233+
echo "Building version: $APP_VERSION"
234+
- name: Set up QEMU
235+
uses: docker/setup-qemu-action@v3
236+
with:
237+
platforms: arm64
238+
- name: Set up Docker Buildx
239+
uses: docker/setup-buildx-action@v3
240+
- name: Download fat jar
241+
uses: actions/download-artifact@v4
242+
with:
243+
name: jars
244+
path: app/build/libs/
245+
- name: Package jar as debian package (arm64)
246+
run: |
247+
docker run --rm --platform linux/arm64 \
248+
-v $(pwd):/workspace \
249+
-w /workspace \
250+
arm64v8/eclipse-temurin:21-jdk \
251+
bash -c "apt-get update && apt-get install -y fakeroot && jpackage --main-jar data-caterer.jar '@misc/jpackage/jpackage.cfg' '@misc/jpackage/jpackage-linux.cfg'"
252+
- name: Rename output to indicate version and architecture (arm64)
253+
run: |
254+
DEB_FILE=$(ls datacaterer_*_arm64.deb 2>/dev/null | head -n 1)
255+
if [ -n "$DEB_FILE" ]; then
256+
echo "Found deb file: $DEB_FILE"
257+
mv "$DEB_FILE" datacaterer_${{ env.APP_VERSION }}_arm64.deb
258+
echo "Renamed to: datacaterer_${{ env.APP_VERSION }}_arm64.deb"
259+
else
260+
echo "No deb file found"
261+
echo "Current directory:"
262+
pwd
263+
echo "Files in current directory:"
264+
ls -lart
265+
exit 1
266+
fi
267+
- name: Upload deb (arm64)
151268
uses: actions/upload-artifact@v4
152269
with:
153-
name: data-caterer-linux-${{ matrix.arch_name }}
154-
path: "datacaterer_${{ env.APP_VERSION }}_${{ matrix.arch_name }}.deb"
270+
name: data-caterer-linux-arm64
271+
path: "datacaterer_${{ env.APP_VERSION }}_arm64.deb"
155272
overwrite: true

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,8 @@ api_validation_report.txt
4444
# UI
4545
ui/node_modules
4646
ui/coverage
47+
.playwright*
48+
node_modules/
4749

4850
# Docs build caches
4951
.docs

‎app/src/integrationTest/scala/io/github/datacatering/datacaterer/core/ui/plan/ConnectionRepositoryTest.scala‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,41 @@ class ConnectionRepositoryTest extends AnyFunSuiteLike with BeforeAndAfterAll wi
119119
probe.expectMessage(ConnectionRepository.ConnectionRemoved("nonExistentConnection", false))
120120
}
121121

122+
test("saved connections should have source=file when retrieved") {
123+
cleanFolder()
124+
val connection = Connection("testSourceConnection", "csv", Some(CONNECTION_GROUP_DATA_SOURCE), Map("key" -> "value"))
125+
val probe = testKit.createTestProbe[ConnectionRepository.ConnectionResponse]()
126+
127+
connectionRepository ! ConnectionRepository.SaveConnections(SaveConnectionsRequest(List(connection)), Some(probe.ref))
128+
probe.expectMessage(ConnectionRepository.ConnectionsSaved(1))
129+
130+
val connectionSaveFolder = s"$tempTestDirectory/connection"
131+
val retrievedConnection = ConnectionRepository.getConnection("testSourceConnection", connectionSaveFolder)
132+
133+
// Connections loaded from files should have source = "file"
134+
retrievedConnection.source shouldEqual "file"
135+
retrievedConnection.isFromFile shouldBe true
136+
retrievedConnection.isFromConfig shouldBe false
137+
}
138+
139+
test("getAllConnections should return connections with correct source field") {
140+
cleanFolder()
141+
val connection = Connection("testSourceConnection2", "csv", Some(CONNECTION_GROUP_DATA_SOURCE), Map("key" -> "value"))
142+
val saveProbe = testKit.createTestProbe[ConnectionRepository.ConnectionResponse]()
143+
144+
connectionRepository ! ConnectionRepository.SaveConnections(SaveConnectionsRequest(List(connection)), Some(saveProbe.ref))
145+
saveProbe.expectMessage(ConnectionRepository.ConnectionsSaved(1))
146+
147+
val getProbe = testKit.createTestProbe[GetConnectionsResponse]()
148+
connectionRepository ! ConnectionRepository.GetConnections(Some(CONNECTION_GROUP_DATA_SOURCE), getProbe.ref)
149+
150+
val response = getProbe.receiveMessage()
151+
val fileConnections = response.connections.filter(_.name == "testSourceConnection2")
152+
fileConnections should have size 1
153+
fileConnections.head.source shouldEqual "file"
154+
fileConnections.head.isFromFile shouldBe true
155+
}
156+
122157
private def cleanFolder(folder: String = "connection"): Unit = {
123158
val path = Paths.get(s"$tempTestDirectory/$folder").toFile
124159
if (path.exists()) {

‎app/src/integrationTest/scala/io/github/datacatering/datacaterer/core/ui/plan/PlanApiEndToEndTest.scala‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -442,7 +442,7 @@ class PlanApiEndToEndTest extends AnyFunSuite with Matchers with BeforeAndAfterA
442442
val response = postJson("/run", planRunRequest)
443443

444444
response.statusCode() shouldBe 200
445-
response.body() should include("Plan started")
445+
response.body() should include("started")
446446
}
447447

448448
// ============================================================================

‎app/src/main/resources/application.conf‎

Lines changed: 0 additions & 112 deletions
Original file line numberDiff line numberDiff line change
@@ -114,116 +114,4 @@ runtime {
114114
}
115115
}
116116

117-
json {
118-
json {
119-
}
120-
}
121-
122-
csv {
123-
csv {
124-
}
125-
}
126-
127-
delta {
128-
delta {
129-
}
130-
}
131-
132-
iceberg {
133-
iceberg {
134-
}
135-
}
136-
137-
orc {
138-
orc {
139-
}
140-
}
141-
142-
parquet {
143-
parquet {
144-
}
145-
}
146-
147-
jdbc {
148-
postgres {
149-
url = "jdbc:postgresql://localhost:5432/customer"
150-
url = ${?POSTGRES_URL}
151-
user = "postgres"
152-
user = ${?POSTGRES_USER}
153-
password = "postgres"
154-
password = ${?POSTGRES_PASSWORD}
155-
driver = "org.postgresql.Driver"
156-
}
157-
mysql {
158-
url = "jdbc:mysql://localhost:3306/customer"
159-
url = ${?MYSQL_URL}
160-
user = "root"
161-
user = ${?MYSQL_USERNAME}
162-
password = "root"
163-
password = ${?MYSQL_PASSWORD}
164-
driver = "com.mysql.cj.jdbc.Driver"
165-
}
166-
}
167-
168-
169-
org.apache.spark.sql.cassandra {
170-
cassandra {
171-
spark.cassandra.connection.host = "localhost"
172-
spark.cassandra.connection.host = ${?CASSANDRA_HOST}
173-
spark.cassandra.connection.port = "9042"
174-
spark.cassandra.connection.port = ${?CASSANDRA_PORT}
175-
spark.cassandra.auth.username = "cassandra"
176-
spark.cassandra.auth.username = ${?CASSANDRA_USERNAME}
177-
spark.cassandra.auth.password = "cassandra"
178-
spark.cassandra.auth.password = ${?CASSANDRA_PASSWORD}
179-
}
180-
}
181-
182-
bigquery {
183-
bigquery {
184-
}
185-
}
186-
187-
http {
188-
http {
189-
}
190-
}
191-
192-
jms {
193-
solace {
194-
initialContextFactory = "com.solacesystems.jndi.SolJNDIInitialContextFactory"
195-
initialContextFactory = ${?SOLACE_INITIAL_CONTEXT_FACTORY}
196-
connectionFactory = "/jms/cf/default"
197-
connectionFactory = ${?SOLACE_CONNECTION_FACTORY}
198-
url = "smf://localhost:55554"
199-
url = ${?SOLACE_URL}
200-
user = "admin"
201-
user = ${?SOLACE_USER}
202-
password = "admin"
203-
password = ${?SOLACE_PASSWORD}
204-
vpnName = "default"
205-
vpnName = ${?SOLACE_VPN}
206-
}
207-
rabbitmq {
208-
connectionFactory = "com.rabbitmq.jms.admin.RMQConnectionFactory"
209-
connectionFactory = ${?RABBITMQ_CONNECTION_FACTORY}
210-
url = "amqp://localhost:5672"
211-
url = ${?RABBITMQ_URL}
212-
user = "guest"
213-
user = ${?RABBITMQ_USER}
214-
password = "guest"
215-
password = ${?RABBITMQ_PASSWORD}
216-
virtualHost = "/"
217-
virtualHost = ${?RABBITMQ_VIRTUAL_HOST}
218-
}
219-
}
220-
221-
kafka {
222-
kafka {
223-
kafka.bootstrap.servers = "localhost:9092"
224-
kafka.bootstrap.servers = ${?KAFKA_BOOTSTRAP_SERVERS}
225-
}
226-
}
227-
228-
229117
datastax-java-driver.advanced.metadata.schema.refreshed-keyspaces = [ "/.*/" ]

0 commit comments

Comments
 (0)