Put dataplatform examples in standard docs #12317

ntjohnson1 · 2026-01-05T20:55:01Z

What

Takes our standard set of dataplatform examples and runs them as snippets using the open source server
- Checks in small datasets to compare with
Don't yell about snippets generating empty RRDs. Our snippet comparison is super painful. Will file an issue to clean this up later. For now, add rust/cpp to don't run and add all languages to don't compare, and also add files to backwards check to run but don't verify against rrd 😢

github-actions · 2026-01-05T20:55:29Z

Latest documentation preview deployed successfully.

Result	Commit	Link
✅	`aca1b5f`	https://landing-7sjmmqptn-rerun.vercel.app/docs

^{Note: This comment is updated whenever you push a commit.}

github-actions · 2026-01-05T20:55:33Z

Your changes cannot be automatically cherry-picked to docs-latest.

You should remove the deploy docs label and perform the cherry-pick manually after merging.

Copilot

Pull request overview

This PR integrates dataplatform examples into the standard documentation by adding snippet tests that run against a local open-source server with checked-in test datasets from the DROID robotics dataset.

Key changes:

Adds four new Python example snippets demonstrating dataplatform query operations (dataframe operations, image queries, time alignment, and view operations)
Checks in test RRD files from the DROID dataset to enable testing without external dependencies
Modifies the snippet testing infrastructure to handle examples that don't generate RRD output files

Reviewed changes

Copilot reviewed 27 out of 28 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/assets/rrd/video_sample/simple_frame_data.rrd	Adds Git LFS tracked video sample data for testing
tests/assets/rrd/sample_5/*.rrd	Adds Git LFS tracked DROID dataset episodes for testing
tests/assets/rrd/sample_5/README.md	Documents the DROID dataset source and licensing
tests/assets/rrd/*.rrd	Adds Git LFS tracked DROID dataset files to root test directory
pixi.toml	Adds pandas dependency required for dataplatform examples
docs/snippets/snippets.toml	Configures new dataplatform examples to skip RRD generation checks and exclude from certain language implementations
docs/snippets/compare_snippet_output.py	Removes RRD validation checks to support examples that don't generate output files
docs/snippets/all/howto/*.py	Adds four new Python example files demonstrating dataplatform query operations
docs/content/howto/cloud/*.md	Adds documentation pages for the new dataplatform examples
docs/content/howto/cloud.md	Adds cloud section redirect to documentation
docs/snippets/INDEX.md	Updates snippet index to include new dataplatform examples
crates/build/re_types_builder/src/lib.rs	Fixes orphan file handling to preserve specific redirect files during code generation

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docs/snippets/compare_snippet_output.py

docs/content/howto/cloud.md

github-actions · 2026-01-05T20:56:34Z

Web viewer built successfully.

Result	Commit	Link	Manifest
✅	`aca1b5f`	https://rerun.io/viewer/pr/12317	`+nightly` `+main`

View image diff on kitdiff.

^{Note: This comment is updated whenever you push a commit.}

ntjohnson1 · 2026-01-06T16:09:12Z

Moving this back to draft because we are exploring shuffling our docs a bit. Right now this added cloud which was a bad name. But no reason to figure out a better one if we are about to reorganize

ntjohnson1 · 2026-01-07T14:45:44Z

Ugh preview doesn't load. Missing a link. Back to do not merge but otherwise this should be ok

… test rrd

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

ntjohnson1 · 2026-01-09T12:40:06Z

Moved this back to do not merge since the shuffle got delayed but is soon

…rm_examples

abey79

Great addition overall. I did find some rough edges here and there, plus some proposal to improve the snippets. I'll let you final judge as I dont want perfection get in the way of iterative improvement.

docs/content/howto/query-and-transform/dataframe_operations.md

docs/content/howto/query-and-transform/overview.md

abey79 · 2026-01-12T15:52:24Z

docs/snippets/all/howto/dataframe_operations.py

+CATALOG_URL = server.address()
+client = rr.catalog.CatalogClient(CATALOG_URL)


an reason you are not using client = server.client()?

Yes because this approach is the more general one. For instance if you are running a long running local server or connecting to the cloud the diff is smaller.

abey79 · 2026-01-12T15:55:59Z

docs/snippets/all/howto/dataframe_operations.py

+CATALOG_URL = server.address()
+client = rr.catalog.CatalogClient(CATALOG_URL)
+dataset = client.get_dataset(name="sample_dataset")
+observations = dataset.filter_contents(["/observation/**"]).reader(index="real_time")


This line deserves a comment since there are no explanation around it in the doc page.

Suggested change

observations = dataset.filter_contents(["/observation/**"]).reader(index="real_time")

# Obtain the observations as `datafusion.DataFrame`. This is cheap because DataFusion's dataframes are lazy query plans, aka no data is downloaded or processed yet.

observations = dataset.filter_contents(["/observation/**"]).reader(index="real_time")

Hmm I'd rather NOT have the explanation in the how to. We definitely need to add the explanation but that feels like it fits a different section

docs/content/howto/query-and-transform/query_images.md

docs/snippets/all/howto/query_images.py

abey79 · 2026-01-12T16:25:37Z

docs/content/howto/query-and-transform/view_operations.md

+In order to more narrowly specify relevant content for further dataframe operations you first generate a view.
+This view can filter on episode, time, column name etc.
+This example shows specific instances highlighting these capabilities.
+


Would be nice to have a sentence to help solidify what we mean by "view" exactly. Aka the fact that they apply to dataset before they are "reshaped" into a dataframe-looking thing by the reader() operation.

docs/snippets/all/howto/time_alignment.py

Co-authored-by: Antoine Beyeler <49431240+abey79@users.noreply.github.com>

ntjohnson1 · 2026-01-12T16:43:57Z

docs/content/howto/query-and-transform/dataframe_operations.md

@@ -0,0 +1,44 @@
+---
+title: Common Dataframe Operations


Hmm I tried to split out different things to do with dataframes (joins, aggregations, etc). But without context it feels meaningless? I think Gijs has some follow ups on all this stuff that he is waiting until this lands to add.

ntjohnson1 · 2026-01-12T16:45:57Z

docs/snippets/all/howto/dataframe_operations.py

+CATALOG_URL = server.address()
+client = rr.catalog.CatalogClient(CATALOG_URL)
+dataset = client.get_dataset(name="sample_dataset")
+observations = dataset.filter_contents(["/observation/**"]).reader(index="real_time")


Hmm I'd rather NOT have the explanation in the how to. We definitely need to add the explanation but that feels like it fits a different section

docs/snippets/all/howto/query_images.py

docs/snippets/all/howto/time_alignment.py

Co-authored-by: Antoine Beyeler <49431240+abey79@users.noreply.github.com>

ntjohnson1 · 2026-01-12T16:51:54Z

Annoying fun fact: If you mark a conversation as resolved it will no longer let you apply suggestions from the gui

…rm_examples

…n into nick/dataplatform_examples

### Related RR-3088 Most of this is just moving the files from our private repo where we initially debugged these. https://github.com/rerun-io/dataplatform-examples ### What * Takes our standard set of dataplatform examples and runs them as snippets using the open source server * Checks in small datasets to compare with * Don't yell about snippets generating empty RRDs. Our snippet comparison is super painful. Will file an issue to clean this up later. For now, add rust/cpp to don't run and add all languages to don't compare, and also add files to backwards check to run but don't verify against rrd 😢 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Antoine Beyeler <49431240+abey79@users.noreply.github.com>

ntjohnson1 added 📖 documentation Improvements or additions to documentation exclude from changelog PRs with this won't show up in CHANGELOG.md deploy docs Once this PR is merged to main, the resulting commit will be cherry-picked to docs-latest labels Jan 5, 2026

ntjohnson1 requested a review from Copilot January 5, 2026 20:55

Copilot AI reviewed Jan 5, 2026

View reviewed changes

docs/snippets/compare_snippet_output.py Show resolved Hide resolved

docs/content/howto/cloud.md Outdated Show resolved Hide resolved

ntjohnson1 force-pushed the nick/dataplatform_examples branch from 60f8f75 to 3e0be95 Compare January 6, 2026 13:19

ntjohnson1 removed the deploy docs Once this PR is merged to main, the resulting commit will be cherry-picked to docs-latest label Jan 6, 2026

ntjohnson1 marked this pull request as ready for review January 6, 2026 13:30

ntjohnson1 added the do-not-merge Do not merge this PR label Jan 6, 2026

ntjohnson1 marked this pull request as draft January 6, 2026 16:08

ntjohnson1 removed the do-not-merge Do not merge this PR label Jan 7, 2026

ntjohnson1 marked this pull request as ready for review January 7, 2026 14:44

ntjohnson1 added the do-not-merge Do not merge this PR label Jan 7, 2026

ntjohnson1 removed the do-not-merge Do not merge this PR label Jan 7, 2026

ntjohnson1 force-pushed the nick/dataplatform_examples branch from 651474d to 0ff8b79 Compare January 8, 2026 19:26

ntjohnson1 and others added 10 commits January 8, 2026 16:51

Add skeleton, df ops, and images

70a4762

Add time alignment, views, and images

740fed3

WIP: move to howto

1b0b955

Add test assets

592711a

Move snippets to how to, and figure out how to run snippets but don't…

dd4ac62

… test rrd

Lint rerun

d51520d

Delete improperly copied files

efe3af7

Update docs/content/howto/cloud.md

80b0b3d

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Colon bad, dash good

e54113e

Fix links

f73d54e

ntjohnson1 added 4 commits January 8, 2026 16:51

Don't mangle lock

df269b8

Nest under query data for now to land content ahead of shuffle

34d4ccd

Correct dir name for redirect

ef4bbe1

Minimize pixi lock changes

ddf26f8

ntjohnson1 force-pushed the nick/dataplatform_examples branch from a13e3d1 to ddf26f8 Compare January 8, 2026 21:53

ntjohnson1 added the do-not-merge Do not merge this PR label Jan 9, 2026

ntjohnson1 added 2 commits January 9, 2026 12:42

Merge branch 'main' of github.com:rerun-io/rerun into nick/dataplatfo…

06eddc8

…rm_examples

Remove duplicated files

da0f07e

ntjohnson1 removed the do-not-merge Do not merge this PR label Jan 9, 2026

ntjohnson1 added 3 commits January 9, 2026 13:15

Fix cli manual creation

5032359

Oops use the correct directory for query

6c09cca

Hopefully final merge resolution

639b6de

abey79 approved these changes Jan 12, 2026

View reviewed changes

Apply suggestions from code review

2cb19b9

Co-authored-by: Antoine Beyeler <49431240+abey79@users.noreply.github.com>

ntjohnson1 commented Jan 12, 2026

View reviewed changes

ntjohnson1 and others added 2 commits January 12, 2026 11:50

Update docs/snippets/all/howto/time_alignment.py

a437a34

Apply suggestions from code review

c65a4fb

Co-authored-by: Antoine Beyeler <49431240+abey79@users.noreply.github.com>

ntjohnson1 added 2 commits January 12, 2026 14:09

Merge branch 'main' of github.com:rerun-io/rerun into nick/dataplatfo…

02ff00d

…rm_examples

Merge branch 'nick/dataplatform_examples' of github.com:rerun-io/reru…

aca1b5f

…n into nick/dataplatform_examples

ntjohnson1 merged commit 5aeb0a6 into main Jan 12, 2026
44 checks passed

ntjohnson1 deleted the nick/dataplatform_examples branch January 12, 2026 20:37

ntjohnson1 mentioned this pull request Jan 12, 2026

Demonstrate what doc update looks like #12408

Merged

		CATALOG_URL = server.address()
		client = rr.catalog.CatalogClient(CATALOG_URL)

	observations = dataset.filter_contents(["/observation/**"]).reader(index="real_time")
	# Obtain the observations as `datafusion.DataFrame`. This is cheap because DataFusion's dataframes are lazy query plans, aka no data is downloaded or processed yet.
	observations = dataset.filter_contents(["/observation/**"]).reader(index="real_time")

Put dataplatform examples in standard docs #12317

Put dataplatform examples in standard docs #12317

Uh oh!

Conversation

ntjohnson1 commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related

What

Uh oh!

github-actions bot commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ntjohnson1 commented Jan 6, 2026

Uh oh!

ntjohnson1 commented Jan 7, 2026

Uh oh!

ntjohnson1 commented Jan 9, 2026

Uh oh!

abey79 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

abey79 Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

ntjohnson1 Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

abey79 Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

ntjohnson1 Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

abey79 Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ntjohnson1 Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

ntjohnson1 Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ntjohnson1 commented Jan 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ntjohnson1 commented Jan 5, 2026 •

edited

Loading

github-actions bot commented Jan 5, 2026 •

edited

Loading

github-actions bot commented Jan 5, 2026 •

edited

Loading

github-actions bot commented Jan 5, 2026 •

edited

Loading