DBD v2.0 format discussion

[Marlamin]

Last month, DBD turned 2 years old. 🎉 It's been quite the ride so far! We're still in the middle of things and will hopefully soon (update: now merged) merge in the pre-wod branch @barncastle has been hard at work on as well as to continue working through the backlog of 6.x/7.x versions we're missing.

Adoption

Adoption still isn't at the point we initially wanted it to be. Why is this? Is it the lack of interest in WoW tool development or is the format too annoying to deal with? Not enough libraries out there to deal with DBD? What can we do to improve this?

Version 2 changes

Throughout this time I'm sure some of you have grown to miss/dislike some things in the current format and with Shadowlands Beta slowly coming to an end changes-wise, I thought this would be a good time to start discussing a possible v2 of the format. Hopefully we can get to a good proposal that we can spend some time on implementing when things calm down in a few months.

A good starting point might be some of the things that were left in the initial format discussion, such as adding enums and flags to the format. I currently deal with these by manually keeping them up to date in the WoW.tools repo, would be cool if we can somehow integrate these into the format if enough people agree. The same goes for flags. No idea how we should handle conditional enums/flags, though.

One thing I'd like us see add (and have discussed previously) is adding a file that combines the tablehash, filedataid and name of DB2s as a way to deal with unnamed DB2s. This idea is based on @MMOSimca's TableHash.cfg, but with the FileDataID added too for if the file is unnamed and to possibly help readers with loading them from CASC.

Also, there are still some open issues such as #40 and #51 that we should tackle here if they need format changes.

Implementation/migration

I'd suggest starting off with a new directory, maybe named "v2" instead of definitions. How to handle DBDv1 going forward is up for debate, but the easiest route when we do switch to primarily v2 will probably be to have some GitHub action run DBDefsConverter which will output v2 -> v1 or something.

Disagree with something? Did I miss something? Post below. Issue will remain open for quite some time until we come to a general consensus like last time.

[barncastle]

My first remark is that I'd like to get a definition for every build we possibly can even if it is just meta dumps before moving on to v2. This will 1. give us a good base for version 2 so we can focus on the spec more than the data and 2. should increase the rate of adoption. This should also be a very achievable considering we have the tools now.

Adoption:
This is quite a hard one to tackle but I'd say the biggest targets that need to be brought across are the server cores themselves. Most tools out there, especially in the modding community, are using Trinity, AzerothCore or (C)Mangos as their main source of reference and rightly so because they are the literal heart of the community. As such most column names, formats and readers are being built in-line with what these projects' implementations. If we could approach these projects and get their input and adoption this would trickle down and increase adoption rates. This also applies to all major/well known projects such as simc and Stoneharry's spell editor.

In regards to the modding community, the biggest issues are favouritism and project longevity. Most modding is happening on WotLK/Vanilla cores as either they are nostalgic favourites and/or their project has spanned several years. At the moment we aren't catering for them however we must do so if we want adoption to increase - ideally we should focus on getting end-of-expansion definitions 100% correct. The unfortunate side-effect is that most modders were raised using abandonware such as MyDBCEditor and WDBX and continue to propagate it to newcomers. As these tools are abandoned we can't target them directly but we could maybe maintain a set of profiles, DBD branded, that these tools support to introduce people into the DBD ecosphere until another tool (like DBMonster or DBClientFiles.Net) that supports DBD comes in to replace them?

In regards to parsers in other languages - this will happen organically once the format is adopted by more people, so I don't feel this is something we need to focus on just yet - something for v3 maybe?

Version 2 changes:

• My biggest OCD gripe at the moment is the lack of standardisation of the layout of the definitions themselves. This is only minor but I'd like to adhere to a standard that orders all definitions by newest to oldest and also uses build ranges as much as possible. These 60+ build lists are really off putting to say the least and merging branches has been a nightmare because everything keeps getting re-ordered. Ideally the automated dump would just increment the final build of the range as we're capturing every build that goes out.

• I'm completely onboard with having an index file for file identification.

• The big topic and focus of v2 (imo) is of course enums and flags. At the moment we're using comments as a quick fix for wow.tools however this isn't ideal. Looking at the original discussion the general consensus was to keep DBD as clean as possible so I'd like to suggest having an additional associated file (even <dbdname>.dat) that stores a range of additional information including this data. How this is formatted is up for debate I'd prefer to have a hex - name pair that we commonly use on the wiki ideally but open for anything else. I'm not sure if Blizz reuse enums or whatnot so versioning might be something we need to look into but we could just reuse the build/build-range/layout spec we've already adopted?

• I'd also like to throw in storing metadata-only data too such as the Flags for WDB3 and the default values for WDB6 common fields and potentially TACT keys (* see the end) since sections aren't always decrypted. The metadata data should really go in main dbd as this is a parsing requirement however TACT keys should probably go into the counterpart data file.

• Something @bloerwald brought up too is adding another keyword for what is (poorly) named secondary indices aka flags & 0x2 - the relationship column for pre-WDC which definitely needs recording!

• For ItemDisplayInfo.db2 wrong cardinality #40: I think we should approach this with symbolic notation as per point 1 here. Namely for the sake of being as true to format as possible and not misleading people with invalid cardinalities when the layouthash doesn't change.

• For Different relationship tables between versions for same columns #51: I think the best choice is to move foreign keys into the definitions themselves like we have for the bitwidths. It does increase the amount of text but it gives much more control over how we handle this data.

Implementation/migration
Personally I'd work across two branches instead of having two folders as this is the point of git. Also, if we went the route of having an additional data counterpart file we could even work on the same repo with minimal fuss.

* I've been thinking for a while that maybe we should create a gist or repo that replicate's the wow.tools tactkey page that doesn't hit the site.

[Shauren]

TrinityCore here

My 2c

Adoption:
I think this is going to be hard when we need to have the format ahead of time in the form of a c++ structure

Naming:
master branch (current client version) has used the renderservice names with some style changes (casing, _lang) since I got my hands on them
3.3.5 has also recently adopted them so it should not be an issue for future projects using it as resource

[Marlamin]

My first remark is that I'd like to get a definition for every build we possibly can even if it is just meta dumps before moving on to v2. This will 1. give us a good base for version 2 so we can focus on the spec more than the data and 2. should increase the rate of adoption. This should also be a very achievable considering we have the tools now.
For 6.0-7.0 we can semi-easily do this by adding more patterns to the dumper. For more versions older than that (though I think pre-wod covers us well there).

Version 2 changes:

• My biggest OCD gripe at the moment is the lack of standardisation of the layout of the definitions themselves. This is only minor but I'd like to adhere to a standard that orders all definitions by newest to oldest and also uses build ranges as much as possible. These 60+ build lists are really off putting to say the least and merging branches has been a nightmare because everything keeps getting re-ordered. Ideally the automated dump would just increment the final build of the range as we're capturing every build that goes out.

I hope to have time to merge the sorting stuff you did as well as update the tools to support sorting (and automatically sort each automatic merge). Update: done.

• The big topic and focus of v2 (imo) is of course enums and flags. At the moment we're using comments as a quick fix for wow.tools however this isn't ideal. Looking at the original discussion the general consensus was to keep DBD as clean as possible so I'd like to suggest having an additional associated file (even <dbdname>.dat) that stores a range of additional information including this data. How this is formatted is up for debate I'd prefer to have a hex - name pair that we commonly use on the wiki ideally but open for anything else. I'm not sure if Blizz reuse enums or whatnot so versioning might be something we need to look into but we could just reuse the build/build-range/layout spec we've already adopted?

• I'd also like to throw in storing metadata-only data too such as the Flags for WDB3 and the default values for WDB6 common fields and potentially TACT keys (* see the end) since sections aren't always decrypted. The metadata data should really go in main dbd as this is a parsing requirement however TACT keys should probably go into the counterpart data file.

Additional metadata file sounds okay to me (maybe we should call it .dbdmeta HEH). Not entirely sure how to deal with versioning, build/build-range/layout could work but we'd have to have the changes for those for v2 pretty cleared out.

• For Different relationship tables between versions for same columns #51: I think the best choice is to move foreign keys into the definitions themselves like we have for the bitwidths. It does increase the amount of text but it gives much more control over how we handle this data.

Sounds good to me.

Implementation/migration
Personally I'd work across two branches instead of having two folders as this is the point of git. Also, if we went the route of having an additional data counterpart file we could even work on the same repo with minimal fuss.

Yeah, that's better.

• I've been thinking for a while that maybe we should create a gist or repo that replicate's the wow.tools tactkey page that doesn't hit the site.

Yeah, it is getting a bit crazy now. 283 new keys were added in the last 5 days alone. Separate repo for this sounds good.
Update: Done. https://github.com/wowdev/TACTKeys

[billy1arm]

Mangos here - although a little late to the party.

I love the approach and agree that this should be as widely supported as possible, i have been working on a suite of DBC/DB2 handling apps which are multiple expansion compatible and I was originally using a format which I devised myself - Which isn't a million miles away from this format.

I'm happy to make the switch to using this (and V2 moving forward). Obviously our primary focus is Classic through to Mop ATM.