Path forward for new language bindings

[jackye1995]

Recently we have received multiple feature requests and pull requests regarding additional language support for lancedb, including Java, Go and C. This discussion aims at exploring how we can make the integration more maintainable, instead of a one-off contribution that might become obsolete over time.

Problem Statement

The main challenges today for new SDK is that:

1. remote vs local tables: most open source users only care about local tables, and don't have intention to improve remote table use case. Technically, the language should not worry about the difference since the difference is managed at rust layer, but still you have to expose the related configs for remote table, and usually that means some employee in lancedb needs to work on that piece to mark the integration as complete.
2. too many operations to bind: every operation in database and table requires a binding call to the rust layer, that have to be implemented. over time the parameters of each operation also change, resulting in the difficulty to keep things up to date when the attention to those language bindings is not as high as the frequently used ones like python and nodejs.

Proposal

To resolve these problems, I propose the following 2 directions to refactor the codebase:

Remote vs Local Tables

Firstly, there are some details that are surfaced today which should not have been, such as the RemoteDBConnection in python. We should aim at removing those out of date code so that all the language bindings can look at the existing ones as a good example to follow.

Secondly, we have introduced the integration with Lance Namespace. The general goal of Lance Namespace and LanceDB integration has been to make the local mode equivalent to the directory namespace, and the remote mode equivalent to the REST namespace. We have made a lot of progress to converge the 2, and we should continue to do so. With this change, technically for any new language bindings, they should directly start with integration with the namespace-based database as the single necessary implementation, and then push things do to rust layer which will support all 3 cases - local, remote and namespace-backed database.

Too many operations to bind

For this work, we should seek to move towards the experience described in #2696. What that means is that, the minimum requirement of any language binding would become to implement the connection feature + the execute_sql capability so that user can pass a string down to the rust binding layer to be executed.

On top of that, each language could implement additional bindings for any existing APIs (assuming the community can keep up with its development), but those become optional and are mainly for better user experience. Anyone using that language's binding now can access all the rust features through just the SQL API alone.

cc @wjones127 @Xuanwo @westonpace

[westonpace]

This seems reasonable. Exposing SQL will be a simple way to complete bindings and then maintainers for that language can build dataframe-style APIs on top of the SQL should they wish. In fact, in future bindings, that dataframe API could be an independent sub-project, crate, etc.