OrientDB next steps for the next release

[tglman]

Hi,

I wrote a blog post about the work happening and what is going to be in the next release here https://orientdb.dev/news/status-and-backlog/

Please check it and let me know any feedback, also happy to accept any contribution from anyone to make next release happen !

[andrii0lomakin]

Hi @tglman
Thank you for the clarification.
Could you clarify who are "we" in your blog post?
Could you introduce a team that will work on such a huge scope?

I am asking because I have seen only a single committer for years already.

You very often refer "we" in your posts, but I see only a single supporter, and all changes seen now are:

1. Minor bug fixes.
2. Dependency updates.

[tglman]

This project still have a few people with push rights on this repo, including you, also there are external contributors that do PR for fix specific problems that are more than happy to merge, anyone is welcome to contribute on the effort, this is all I can share on this.

[andrii0lomakin]

You wrote:

As today the integration of lucene in the engine is limited, it does not have durability guarantee, and does not have complete backup/restore operations support and as well is not integrated properly in the distributed environment, all of this will be worked on to make support of lucene index at the same level of native indexes

Could you share any design choices that will backup those plans ?

[tglman]

Hi, Yes,

Some refactor work has been already done on the develop branch to make the integration possible, the firs step is to provide a method in the OIndexEngine that receive all the changes and apply to the index, that do more or less the same logic of the OAbstractPaginatedStorage#applyTxChanges. Done that, the implementation of the Lucene IndexEngine will use lucene two phase transactions and than the sequential returned from the prepareCommit will be logged with a specific record in the WAL, this will allow to deterministically detect durning the recover/open process if the lucene indexes are at the same level of transaction of the main storage.

When detected if the lucene index is out of sync a index rebuild can be triggered.
It could also be possible to log the lucene key changes in the WAL for reapply them and that will avoid the rebuild, but more check need to be done on this solution to understand if is possible.

[andrii0lomakin]

Studio has been in low maintenance for a while

That is true. Could you introduce a contributor (and by this, I mean either person or company) who will work on this part to improve its maintenance?

[andrii0lomakin]

Fix major issues in query engine and query performance improvement

This paragraph does not provide any concrete plans.

1. Which optimization algorithms are going to be used?
2. What concrete problems will be fixed ?

From commits are seen there are quite minor improvements in this area.

[tglman]

Hi,

A lot of work is already been done on this, you can check the implementation differences between the query engine in 3.1.x, 3.2.x, and develop, a lot more need to be done, at first the work has been going for a while is migrate all the existing tests that used the old query engine to the new query engine, I started working on this more than 3 years ago with more than 5000 usages in the code, in the last develop we do have less than 300 uses of the old query engine in the test cases, while working on this I fixed also all the bug I found reproduced by those tests, and ported the relative fix to the last stable (3.2.x) so this I guess is the minor improvements you saw in the commits.

The main issues that need to solve are actually this: #9808 #10328

There are some implementations in the code that do the preliminary steps to fix 9808, the main problem is that for detect indexes is used an exponential algorithm (that allocate memory in an exponential way), an algorithm that is solve this problem and is not exponential is already wrote+tested+merged, but is not yet integrated, the integration will be done after the migration of the tests that use the legacy query engine will be complete.

There are additional improvement as well that can be done, especially on query profiling and error reports, but may be done after the release of the next major.

[andrii0lomakin]

Are there any companies that can support and ensure continuous development and the quality of the efforts that will be spent to implement such a big scope?

[tglman]

🤷

[andrii0lomakin]

Could you provide QA measures that you are going to use to ensure quality assurance that is needed to ensure quality characteristics of upcoming and current releases? I am pretty sure that running tests on GitHub actions is insufficient to ensure such quality.

[tglman]

Hi,

I do agree that the tests running on github actions are not complete, I do run before the release mvn clean install falisafe:integration-tests that ran for 5h+ and do additional tests to guarantee the quality, these tests are run periodically as well, mostly are triggered manually, also before the shipment of the release are run some minimal manual test to check the quality of the package.

I'm pretty sure you are pretty familiar with the process as we did it together many times, but is good also to make this information more public.

[andrii0lomakin]

Alas, that is not enough to ensure the user's data safety. :-( IMHO.

[tglman]

More than welcome to integrate and run additional tests, the integration tests are the base for everything, I do know the existence in the code of additional test that are run trough different profiles, like crash tests and secondary tests, feel free to share what would be what you would consider the right set of tests to run to guarantee data safety.

[timw]

Thanks for the update @tglman

The list of planned changes looks really good, and touches on a bunch of things our engineering team would like to see.

I'd be interested in supporting the Lucene durability work, and our team have a good amount of experience in distributed/consensus systems if you want to collaborate on the Hazelcast replacement. The console and query parser (the current grammar is somewhat "suboptimal") would be areas we'd like to see improved as well, so happy to help there.

A few things that sprung to mind reading the list of planned changes:

• I'd really like to see the CE code for backup etc. removed from the codebase and the EA code inlined (Lucene durability being a dependency for this).
• Aside from the large amount of legacy code, a major barrier to contributing to the codebase is the really messy OO abstractions, use poor encapsulation and the generally adhoc lifecycle model in the engine (no disrespect intended there, as I've seen worse in codebases like early Cassandra versions, but it has made understanding, debugging and hacking on OrientDB hard). The lifecycle management in particular makes making distributed operations safe in the presence of errors and lifecycle operations really quite fiddly.
• I'd suggest just jumping to Java 21 as a minimum. The jump from 17 isn't that significant (we in fact have OrientDB 3.1 running on Java 21), and it makes sense to go closer to current for a new major.
• For the HTTP server, something low level like Netty or perhaps Grizzly would be a great option.
• I would suggest replacing the custom comms/serialisation in the remote client and distributed server with gRPC - it would clean up that part of the server significantly, and it'd use Netty under the hood as well, which would be a good technology standardisation if you went with that for the HTTP stack as well.
• I'm not sure what you're planning on the query engine side, but making the query execution engine more "streaming" could be beneficial. i.e. we've had to patch the Lucene index engine to stop it retrieving massive numbers of records, even in a case where the query doesn't require them, since index engine loads retrieve all the index hits all at once. Similarly there's no way in the current engine to support evolution of sort or distinct stages to support streaming modes (i.e. if you can determine that the underlying data is sorted).
• We're considering adding support for multiple repeated elements, and document sub-fields, in indexes (without this we have to manually cross product indexes into distinct vertex instances, which complicates the resulting queries a lot).
• There's no current way to support something like prepared statements for Lucene queries, which means we currently have to do a lot of armouring/escaping in our Lucene query generation to avoid injection issues. An ability for the query parser to support index engine specific DSLs, with parameter injection, would make that a lot easier.
• The official OrientDB containers aren't really suitable for production use, particularly for distributed deployments, and the Docker awareness in the server startup is not really production suitable either (it's out of date for modern cgroups etc. and the automagic calculation is quite fraught). We could collaborate/contribute improvements from our containers if you want to evolve that area.
• A refresh and/or purge of the driver ecosystem would make it clear what is supported/maintained (outside the Java client it's a little unclear).
• Not directly a 4.0 issue, but a major purge of out of date websites/docs and some SEO would improve public access/perception a lot.

Aside from all of these ideas, I think I'd second the suggestion that data integrity testing is paramount for rebuilding confidence in the user community. Starting with deploying 3.1 in production, it was quickly evident that the distributed code had never really been used in anger (simply waiting for an amount of time killed the servers), and we're experienced genuine data file corruption in production systems, which is really un-fun to work around. I know 3.2 has improved that area a lot, but having a public testing strategy covering weak/missing areas, and fuzzing/stress approach, that people in the community could get behind would be a major benefit to the project.

[andrii0lomakin]

Here is one example of problems I am working on now: https://youtrack.jetbrains.com/issue/YTDB-226/Database-is-broken that we found because of dogfooding.

[andrii0lomakin]

major barrier to contributing to the codebase is the really messy OO abstractions

I cannot argue with this. We are also trying to fix this issue. We have a Slack chat with developers for contributors. If you wish to help us streamline the YouTrackDB design, just email me.

Also, @timw, I have been following your contributions for a while, and I value the changes you have proposed enormously. So, if you decide to help us evolve, that would be a great pleasure.

[andrii0lomakin]

@timw In reality we think alike with you. Let me give a bit of proof of that

I'd suggest just jumping to Java 21 as a minimum

https://youtrack.jetbrains.com/issue/YTDB-99/Migrate-to-JDK-21, while remote client will run on JDK 11 https://youtrack.jetbrains.com/issue/YTDB-41/Ensure-that-Java-remote-client-can-work-under-JDK-11.
We are also planning to share the code base between Java and JS drivers using Kotlin Multiplatform: https://youtrack.jetbrains.com/issue/YTDB-221/Align-functionality-of-JavaScript-and-Java-based-drivers.

For the HTTP server, something low level like Netty or perhaps Grizzly would be a great option.

Also already planned https://youtrack.jetbrains.com/issue/YTDB-179/Migrate-from-home-made-TCP-IP-server-to-industry-grade-library

I would suggest replacing the custom comms/serialisation in the remote client and distributed server with gRPC

It will be since the first version https://youtrack.jetbrains.com/issue/YTDB-3/Use-protobuf-serializer-for-commands-of-the-binary-protocol.

The official OrientDB containers aren't really suitable for production use

We ensure that is not the case since first version https://youtrack.jetbrains.com/issue/YTDB-107/Ensure-that-the-docker-build-uses-the-latest-JDK-version by implementation of tests in our build pipeline.

A refresh and/or purge of the driver ecosystem would make it clear what is supported/maintained

Also planned https://youtrack.jetbrains.com/issue/YTDB-58/Write-workflows-for-each-command-in-the-binary-protocol.

So, as you can see, we see the future of DB development alike, I hope to see you among our users/contributors once.

[tglman]

Hi @timw,

Thanks for the detailed points, I'm going to follow up soon of them with a detailed answer (not yet, it take some time to write), but yes very happy to review and accept your contributions, I will open as well discussion on here about the topic you mentioned so we can have a clear state of the art and we can synchronize on the work that can be done.

[tglman]

Hi,

Any help from you and your team is very welcome,and all the patches that you sent so far were very appreciated!

The various points of improvement are shared by me and almost each of the point needs a bit of it's own long answer/discussion, will reply to the point with a short answer here and then open another discussion if needed:

I'd be interested in supporting the Lucene durability work, and our team have a good amount of experience in distributed/consensus systems if you want to collaborate on the Hazelcast replacement. The console and query parser (the current grammar is somewhat "suboptimal") would be areas we'd like to see improved as well, so happy to help there.

For lucene durability I started already some work, but much need to be done on it, here a new discussion about it

I'd really like to see the CE code for backup etc. removed from the codebase and the EA code inlined (Lucene durability being a dependency for this).

The backup is definitely an area of improvement, the plan is to have a single final backup that support, non blocking backup with lucene and all the other features of orientdb, though the legacy backup may remain there for a bit more a think, mostly for compatibility reasons, the Lucene integration in the incremental backup is actually a quite big change that I can't guarantee it will be in the next major, but is a OrientDB target as well.

Aside from the large amount of legacy code, a major barrier to contributing to the codebase is the really messy OO abstractions, use poor encapsulation and the generally adhoc lifecycle model in the engine (no disrespect intended there, as I've seen worse in codebases like early Cassandra versions, but it has made understanding, debugging and hacking on OrientDB hard). The lifecycle management in particular makes making distributed operations safe in the presence of errors and lifecycle operations really quite fiddly

Yes OrientDB is quite messy, I did spend last few years without adding new features on it, but refactoring it, cleaning it up, I did remove recently a bunch of legacy implementations, and more work need to be done on it, as well I've been fighting a lot to reshape the internal abstractions that where in the wrong place (like network and distributed propagation being at storage level, with knowledge of only byte[] and not documents .....) , for the distributed I opened a different discussion with the details on what need to be fixed here

I'd suggest just jumping to Java 21 as a minimum. The jump from 17 isn't that significant (we in fact have OrientDB 3.1 running on Java 21), and it makes sense to go closer to current for a new major.

For now I'm conservatively targeting java 17, this is not a set in stone target, as far as all stakeholders are happy to move forward I will be more than happy to target a newer java version like 21 or 22 as well, staying back for now just because moving forward will be actually quite easy, moving back quite hard/impossible that's why.

For the HTTP server, something low level like Netty or perhaps Grizzly would be a great option.

Yes this is quite needed for modernization, not sure I do have the bandwidth to handle also this for the next major in a decent time, help is really welcome here, in any case I did refactor the current implementation to extract two interfaces: OHttpRequest , OHttpResponse that if implemented with a new server should allow all the specific APIs to work in the same way of before, guaranteeing studio e lagacy http drivers to work.

I would suggest replacing the custom comms/serialisation in the remote client and distributed server with gRPC - it would clean up that part of the server significantly, and it'd use Netty under the hood as well, which would be a good technology standardisation if you went with that for the HTTP stack as well.

Not planning this at all in the short term, what is there works, first stability than when is there a stable http implementation, maybe something else can be built on top, but first I do want to see the current http APIs implemented with a proper http server (I mean refactor the current not yet another similar implementation on the side, took me 4 years to remove a legacy sql implementation end make the one on the side the main, additional module on the side is not sustainable approach).

I'm not sure what you're planning on the query engine side, but making the query execution engine more "streaming" could be beneficial. i.e. we've had to patch the Lucene index engine to stop it retrieving massive numbers of records, even in a case where the query doesn't require them, since index engine loads retrieve all the index hits all at once. Similarly there's no way in the current engine to support evolution of sort or distinct stages to support streaming modes (i.e. if you can determine that the underlying data is sorted).

In the query engine the main scope is reduce is memory usage and improve is performance, as today actually the current develop branch (not in 3.2.x) should already stream the index lookup, so I guess some of the issues you had has already been resolved, yes more work need to be done as well to improve the query execution planning in a way that can leverage indexes in more complex cases, one thing that would be really nice but I do not have the bandwidth for it, is cache on the disc distinct/grouping query that do not fit in memory.

We're considering adding support for multiple repeated elements, and document sub-fields, in indexes (without this we have to manually cross product indexes into distinct vertex instances, which complicates the resulting queries a lot).

Not sure what you mean here, feel free to open a separate issue/discussion

There's no current way to support something like prepared statements for Lucene queries, which means we currently have to do a lot of armouring/escaping in our Lucene query generation to avoid injection issues. An ability for the query parser to support index engine specific DSLs, with parameter injection, would make that a lot easier.

Actually pass forward the query parameters to lucene should not be too hard, feel free to open an issue on this.

The official OrientDB containers aren't really suitable for production use, particularly for distributed deployments, and the Docker awareness in the server startup is not really production suitable either (it's out of date for modern cgroups etc. and the automagic calculation is quite fraught). We could collaborate/contribute improvements from our containers if you want to evolve that area.

The containers are developed here any contribution is more than welcome

A refresh and/or purge of the driver ecosystem would make it clear what is supported/maintained (outside the Java client it's a little unclear).

This may happen at some point, right now is not yet the time, there are some drivers more updated, others left behind, the cleanup will probably be done just about the release of the next major, based on what is supported and what is not.

Not directly a 4.0 issue, but a major purge of out of date websites/docs and some SEO would improve public access/perception a lot.

Right now we are in a quite transitional period so yes the communication is often not uniform or clear, this will probably last for more or less another year or maybe a bit more, what I can definitely guarantee is the constant shipment of patch and minor/major releases, with the related communication, other things (which are as important as releases) are coming as secondary, but will be handled at some point.

Aside from all of these ideas, I think I'd second the suggestion that data integrity testing is paramount for rebuilding confidence in the user community. Starting with deploying 3.1 in production, it was quickly evident that the distributed code had never really been used in anger (simply waiting for an amount of time killed the servers), and we're experienced genuine data file corruption in production systems, which is really un-fun to work around. I know 3.2 has improved that area a lot, but having a public testing strategy covering weak/missing areas, and fuzzing/stress approach, that people in the community could get behind would be a major benefit to the project.

Yes testing is key, as today there is a lot there, but more need to be done I agree, is also true that write an external black box test suite for crash is something anyone can contribute, there is no need of internal knowledge of the system, so help here is very welcome.

In any case if you want I push some specific topic forward in the implementation or need some direct support, you can directly contact me (DM https://matrix.to/#/@tglman:matrix.org email: tglman at tglman.com)