GardeLogStore

GardeLogStore is a Kotlin/JVM append-only binary log storage library with indexed reads, recovery, repair, JSONL/TXT export, and small foundation layers for import, payload transforms, JDBC helpers, viewing, and schema codecs.

Status: early development. The API and binary format may still change before a stable release, and compatibility is not guaranteed before 1.0.0.

Current version: 0.1.5-dev.

Features

• .glog append-only binary data file
• .gidx index file
• append and read APIs
• range reads
• metadata reads without loading payload bytes
• verification
• repair and index rebuild
• partial last record repair
• text helper APIs
• JSON Lines export and basic JSONL import
• TXT export
• optional payload envelope helper
• opt-in payload compression helper with no-op and gzip codecs
• experimental AES-GCM payload cipher interface and JDK implementation
• JDBC SQL helper without bundled database drivers
• UI-independent log viewer service and a minimal CLI tool
• improved Minecraft schema-only payload codec with constants, helpers, record conversion, and no Minecraft dependency
• automated tests

Foundation Status

• Import is a basic JSONL importer for files produced by GardeLogExporter.exportJsonLines.
• Compression is an opt-in payload helper. GardeLogStore.append does not compress automatically.
• Encryption is experimental, not audited, and key management is the caller's responsibility. It is not enabled automatically.
• SQL support is a JDBC helper only. No SQLite, H2, Postgres, or other runtime driver is bundled.
• UI/log viewer support is minimal. The reusable piece is a UI-independent paging/filtering service plus a small CLI tool.
• Minecraft-specific serialization is improved but remains schema-only. It has no Minecraft, Fabric, Paper, Bukkit, Adventure, or Sponge dependency; live Text handling must stay in the mod/plugin that uses this library.
• Android and JS backends are planned, not implemented.

Still Planned

• full import workflows
• automatic compression policies
• production-grade encryption guidance
• richer SQL integrations
• graphical UI
• log viewer applications
• Minecraft adapter modules outside the core library for live component conversion
• Android backend
• JS backend

Quick Usage

import ru.garde.logstore.GardeLogStore
import ru.garde.logstore.LogRecordTypes
import ru.garde.logstore.export.GardeLogExporter
import ru.garde.logstore.export.LogExportOptions
import ru.garde.logstore.export.LogExportRange
import ru.garde.logstore.importer.GardeLogImporter
import java.nio.file.Path

fun main() {
    GardeLogStore.open(Path.of("logs"), name = "history").use { store ->
        store.appendText(
            type = LogRecordTypes.INFO,
            text = "server started",
        )

        val latest = store.readLast(limit = 1)
        println(latest.firstOrNull()?.payloadAsString())

        val result = GardeLogExporter.exportJsonLines(
            store = store,
            target = Path.of("exports/history.jsonl"),
            range = LogExportRange.All,
            options = LogExportOptions(overwrite = true),
        )

        println("Exported ${result.recordsExported} records")
    }

    GardeLogStore.open(Path.of("logs-imported"), name = "history").use { store ->
        val result = GardeLogImporter.importJsonLines(
            store = store,
            source = Path.of("exports/history.jsonl"),
        )
        println("Imported ${result.recordsImported} records")
    }
}

Storage Format

Each store uses two files with the same base name:

• .glog stores the binary records.
• .gidx stores index metadata for fast lookups and range reads.

Records contain an id, timestamp, type, flags, payload bytes, and CRC32. The index stores ids, timestamps, types, flags, offsets, and record lengths. The optional payload envelope is stored inside the record payload bytes and does not change the outer .glog or .gidx format. See docs/FORMAT.md and docs/PAYLOAD_ENVELOPE.md.

Public API

The main API is GardeLogStore:

• append(type, payload, timestampMillis, flags)
• append(record)
• appendText(type, text, timestampMillis, flags, charset)
• readLast(limit)
• readAll(limit)
• readById(id)
• readRange(startId, limit)
• readAfter(id, limit)
• readBefore(id, limit)
• metadataLast(limit)
• metadataRange(startId, limit)
• metadataAfter(id, limit)
• metadataBefore(id, limit)
• rebuildIndex() / repairIndex()
• repairPartialLastRecord(truncate = false)
• verify()
• flush()
• close()

Additional packages:

• ru.garde.logstore.export for JSONL/TXT export
• ru.garde.logstore.importer for JSONL import
• ru.garde.logstore.payload for payload envelopes
• ru.garde.logstore.payload.compression for opt-in compression helpers
• ru.garde.logstore.payload.encryption for experimental payload ciphers
• ru.garde.logstore.sql for JDBC export/import helpers
• ru.garde.logstore.viewer for UI-independent paging/filtering
• ru.garde.logstore.tools for the minimal CLI entry point
• ru.garde.logstore.minecraft for schema-only Minecraft payload bytes, record type constants, helpers, and view text

Minecraft Schema

The ru.garde.logstore.minecraft package stores and encodes Minecraft-oriented payload schemas only. It does not depend on Minecraft, Fabric, Paper, Bukkit, Adventure, or Sponge APIs, and it does not handle live Text objects. A mod or plugin should convert game components to plain text and JSON strings externally, then pass those values into MinecraftLogPayloads or MinecraftLogPayloadCodec.

Useful entry points:

• MinecraftLogPayload schema 2 model, with schema 1 decode compatibility
• MinecraftLogRecordTypes, MinecraftLogKinds, and MinecraftLogDirections
• MinecraftLogPayloads factory helpers
• MinecraftLogRecords.payloadBytes, appendTo, toRecord, and fromRecord
• MinecraftLogPayloadView.displayLine and searchableText

See docs/MINECRAFT_SCHEMA.md.

CLI Tool

The minimal CLI entry point is ru.garde.logstore.tools.GardeLogStoreCli. It supports:

view <directory> <name> --last 100
export-jsonl <directory> <name> <target>
export-txt <directory> <name> <target>
verify <directory> <name>

Recovery

GardeLogStore validates magic bytes, format version, record length, payload length, and CRC32 checksums. The JVM backend can rebuild .gidx from .glog, verify data and index consistency, and optionally truncate an incomplete trailing record through repairPartialLastRecord(truncate = true).

verify() only inspects files and reports issues. It does not rewrite or truncate files.

Security Note

GardeLogStore is not a security layer. Raw payload bytes are stored as provided. Base64 in JSONL export is only a text encoding of raw bytes. The AES-GCM helper is experimental and not audited; callers must manage keys and operational policy outside this library. See docs/SECURITY_NOTES.md.

Build

On Windows:

.\gradlew.bat build

On Linux/macOS:

./gradlew build

The project uses Java 21 and the Gradle wrapper.

Documentation

• docs/FORMAT.md
• docs/MINECRAFT_SCHEMA.md
• docs/PAYLOAD_ENVELOPE.md
• docs/PLATFORM_BACKENDS.md
• docs/ROADMAP.md
• docs/SECURITY_NOTES.md
• docs/RELEASE_CHECKLIST.md

License

MIT License. See LICENSE.