Skip to content

Documented inserting multiple files(and transaction support) #88

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
subkanthi wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Documented inserting multiple files(and transaction support) #88

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions examples/scratch/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,69 @@ select * from ice.`flowers.iris_no_copy` limit 10 FORMAT CSVWithNamesAndTypes;

> To clean up simply terminate processes and `rm -rf data/`.

### Inserting Multiple Files

You can insert multiple parquet files at once by piping a list of files to the insert command:

```shell
# insert multiple files from stdin
cat filelist | ice insert flowers.iris -
Copy link

@xieandrew xieandrew Jan 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add -p to the ice command so the table is created if it doesn't exist? (ice insert -p flower.iris -)

```

where `filelist` is the list of filenames
```
iris1.parquet
iris3.parquet
iris2.parquet
```

When inserting multiple files to the same table, all files are processed as a **single transaction**. If any file fails (e.g., doesn't exist or has schema issues), **none of the files are committed** to the table:

```
cat filelist| insert flowers.iris -
2026-01-09 11:28:02 [-5-thread-3] INFO c.a.i.c.internal.cmd.Insert > file://iris2.parquet: processing
2026-01-09 11:28:02 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file://iris.parquet: processing
2026-01-09 11:28:02 [-5-thread-2] INFO c.a.i.c.internal.cmd.Insert > file://iris3.parquet: processing
2026-01-09 11:28:03 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file://iris.parquet: copying to s3://bucket1/flowers/iris/data/...parquet
2026-01-09 11:28:03 [-5-thread-3] INFO c.a.i.c.internal.cmd.Insert > file://iris2.parquet: copying to s3://bucket1/flowers/iris/data/...parquet
2026-01-09 11:28:03 [-5-thread-3] INFO c.a.i.c.internal.cmd.Insert > file://iris2.parquet: adding data file (copy took 0s)
2026-01-09 11:28:03 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file://iris.parquet: adding data file (copy took 0s)
2026-01-09 11:28:03 [main] ERROR com.altinity.ice.cli.Main > Fatal
java.io.IOException: Error processing file(s)
...
Caused by: java.io.FileNotFoundException: iris3.parquet (No such file or directory)
...
```

```sql
-- query shows 0 rows because iris3.parquet failed and the entire transaction was rolled back
select count(*) from `flowers.iris`;
┌─count()─┐
1. │ 0 │
└─────────┘
```

When all files are valid, the transaction commits successfully:

```
cat filelist| insert flowers.iris -
2026-01-09 11:29:03 [-5-thread-2] INFO c.a.i.c.internal.cmd.Insert > file://iris2.parquet: processing
2026-01-09 11:29:03 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file://iris.parquet: processing
2026-01-09 11:29:04 [-5-thread-2] INFO c.a.i.c.internal.cmd.Insert > file://iris2.parquet: copying to s3://bucket1/flowers/iris/data/...parquet
2026-01-09 11:29:04 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file://iris.parquet: copying to s3://bucket1/flowers/iris/data/...parquet
2026-01-09 11:29:04 [-5-thread-2] INFO c.a.i.c.internal.cmd.Insert > file://iris2.parquet: adding data file (copy took 0s)
2026-01-09 11:29:04 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file://iris.parquet: adding data file (copy took 0s)
2026-01-09 11:29:04 [main] INFO o.a.i.SnapshotProducer > Committed snapshot 2260546103981693696 (MergeAppend)
```

```sql
-- query shows 300 rows (150 from each file) after successful commit
select count(*) from `flowers.iris`;
┌─count()─┐
1. │ 300 │
└─────────┘
```

### Troubleshooting

1. `Code: 336. DB::Exception: Unknown database engine: DataLakeCatalog. (UNKNOWN_DATABASE_ENGINE)`
Expand Down