Skip to content

docs(clp): Rewrite clp methodology docs for better clarity; Bump clp asset version to 0.5.1. #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
quinntaylormitchell wants to merge 9 commits into
base: main
Choose a base branch
from
Open

docs(clp): Rewrite clp methodology docs for better clarity; Bump clp asset version to 0.5.1. #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
3ffeb34
Change reference to user docs to comply with docs PR from clp repo
quinntaylormitchell Aug 4, 2025
1320eb7
Rewrite links and follow line limit
Bill-hbrhbr Sep 24, 2025
7b7e646
Update versions used in dockerfile
Bill-hbrhbr Sep 24, 2025
f328b0e
Address coderabitaai comments and reword
Bill-hbrhbr Sep 24, 2025
cbd2d16
Clarify stage names
Bill-hbrhbr Sep 24, 2025
18bc69a
use v0.5.1 commit hashes
Bill-hbrhbr Sep 24, 2025
ed4c997
minor doc nit
Bill-hbrhbr Sep 24, 2025
a42d244
Merge branch 'main' into docs-restructure-rename-logarchivalbench
Bill-hbrhbr Sep 24, 2025
78e0f66
Apply suggestions from code review
Bill-hbrhbr Sep 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions assets/clp/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# FROM ghcr.io/y-scope/clp/clp-core-x86-ubuntu-jammy:main
# TODO: use properly tagged image version
FROM ghcr.io/y-scope/clp/clp-core-x86-ubuntu-jammy@sha256:e82b958b43289f2e6553057e5649f56740fa7183d299c0fcb9adc55068ae2bf2 \
AS clp-core

FROM ghcr.io/y-scope/clp/clp-core-x86-ubuntu-jammy@sha256:67c34b94fe1bdbf81846a20ebcf3920c5e26f8ba6ae392747a4cd36b47fc05e4
# this is commit bb8244c32b6649bc92901c2cadb6be4e3bf48607

FROM ghcr.io/y-scope/clp/clp-core-dependencies-x86-ubuntu-jammy:main
FROM ghcr.io/y-scope/clp/clp-core-dependencies-x86-ubuntu-jammy@sha256:49c0edf1cdd8e92bae2f46509d7e0032da7ae405f4afc58ae3e3abfe7c448e34 \
AS clp-deps-core

RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y \
Expand All @@ -15,4 +15,4 @@ RUN apt-get update \
tmux \
vim

COPY --from=0 /clp/clp-s /clp/clp-s
COPY --from=clp-core /clp/clp-s /clp/clp-s
22 changes: 16 additions & 6 deletions assets/clp/methodology.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,25 @@
# CLP methodology

**Version:** [0.4.0](https://github.com/y-scope/clp/releases/tag/v0.4.0)
**Version:** [clp-0.5.1]

**File with Formatted Queries:** [Config File](/assets/clp/config.yaml)
**File with Formatted Queries:** [Config File]

## Setup

The benchmark builds on the public [**CLP core Ubuntu-Jammy**](http://ghcr.io/y-scope/clp/clp-core-dependencies-x86-ubuntu-jammy) docker image.
The benchmark builds on the public [CLP core Ubuntu-Jammy] Docker image.

We use the clp-s binary, which is a variant of CLP optimized for semi-structured logs like JSON. We run clp-s with a target encoded size of 256MB, balancing compression ratio with search speed. Increasing the target encoded size can yield even better compression. Additional tuning options are documented in the user guide, [here.](https://docs.yscope.com/clp/v0.4.0/user-guide/core-clp-s)
We use the `clp-s` binary, which is a variant of CLP optimized for semi-structured logs like JSON.
We run `clp-s` with a target encoded size of 256 MB, balancing compression ratio with search speed.
Increasing the target encoded size can yield even better compression results. Additional tuning
options are documented in the [user docs].

There is no need to launch clp-s, once built it can just be called on the command line and will be just shut down when it’s completed its command execution. The only thing we do before calling clp-s is to make a directory for its output files. We provide this as one of the parameters to the ingestion and search commands.
Before calling `clp-s`, create a directory for its output files and pass it as a parameter to the
ingestion and search commands.

CLP doesn’t maintain any internal caches for us to clear, so before each query we flush the file system buffers and clear the filesystem caches.
CLP doesn’t maintain any internal caches, so we only flush the filesystem buffers and clear the
filesystem caches before each query.

[CLP core Ubuntu-Jammy]: https://ghcr.io/y-scope/clp/clp-core-dependencies-x86-ubuntu-jammy
[clp-0.5.1]: https://github.com/y-scope/clp/releases/tag/v0.5.1
[Config File]: /assets/clp/config.yaml
[user docs]: https://docs.yscope.com/clp/v0.5.1/user-docs/core-clp-s.html