Support multiple Duck Lakes

[ben-pr-p]

I'd like to support multiple DuckLakes since DuckLake supports attaching, and DuckDB itself supports attaching different DuckLakes under different database namespaces. My use case is that my raw data (bronze) is permissioned differently from derivative data sets (silver + gold), and I have different users who can access the different DuckLakes.

Is there any support here for this type of workflow?

[MaxHalford]

Hey there. I didn't have this use case in mind, but it's interesting. Would it help if you could ATTACH to a specific source in a particular query?

Could you maybe share a few queries to detail your use case a bit more? I'm not sure what you want in an ideal world.

[ben-pr-p]

Thanks for your response!

Attaching in a source to a query would be great, but the real ideal would be defining a set of duck lakes up front, which are attached under a given alias for all queries - for me, I basically want a 1-1 relationship between Lea schemas and DuckDB DuckLakes.

For me, it'd be something like this, taken from the README:

scripts/
    schema_1/ # attached to a ducklake called schema_1 in config
        table_1.sql
        table_2.sql
    schema_2/ # attached to a ducklake called schema_2 in config
        table_3.sql # can depend on queries that include schema_1.table_1
        table_4.sql
        sub_schema_2_1/
            table_5.sql
            table_6.sql

Does that seem doable?

Lea would run with access to all DuckLakes, but other systems would provide credentials separately so that certain users could only access derived data sources.

[MaxHalford]

I'm a bit slow but I will work on this eventually, it makes a lot of sense :)