Meeting Notes

[maregold]

SIMSSA to Data Lake: all of SIMSSA DB is sample data imported to CSV
- possibly just dump it into JSON-LD

• should do something similar to how we are going to dump the data from other databases we don't have constant access to
  • sent musicbrainz download page
• would be easier for partners to just give us a database dump? vs csv
• relational database
• would need to change from postgres sql to csv (flat)
• every cell in the row is going to be a triple and every column is going to be a predicate in wikidata
  • ld is inefficient but more flexible
  • how much flattening do we want to do
    • flatten until we have enough data
  • a lot of the database will be redundant bc we will use wikidata
    • once we get composer's name you can get the rest of the info from wikidata
• store stuff that is unique to a particular database
  • ex for SIMSSA: the resources such as the midi file, etc.
    • but you need enough metadata to find those files
  • what is the main item of the database that needs to be stored
• storing a link to where you can find the actual object + the triples in the data lake
• draft a specification of how to give us data once we get past the 14 databases we are already partnered with
• sql to csv, into openrefine to add wikidata ids then json-ld
• easy for people to look for things
• what does SOLR like to search through
  • how will it like json-ld files
  • we can use sparql as part of the query (chatgpt will allow to convert natural language into sparql etc)

[maregold]

DATA LAKE 8/1/23

• need to make it as easy as possible for them to push whatever they want
• we will create a pipeline for csv/postgres/mysql respectively so regardless of how data is sent it will be converted to how it needs to be stored in the data lake
  • will need some sort of metadata alignment
• will be quads instead of triples
• may or may not use OpenRefine depending on what we need (which is just the Wikidata ID)
• whatever we can't find we will add on wikidata
• why OpenRefine can't find certain composers during reconciliation with wikidata
• will store anything that is text (a cell)
• how to handle conflicting data
  • RISM: not before and not after (for dates)
• jacky diagram updates
  • eraser.io
• want to go beyond what we can do with SPARQL
• going to build a search engine for information that can't be found on google or wikidata

[maregold]

DATA LAKE 8/8/23

• contributors - those who are going to give us data
• can you go directly from sql dump to json-ld as well as combine the wikidata ids
  • can you go from sql to csv?
• workflow
  • step 1 (for contributors): dump content
    • what happens if a contributor adds/edits information - what is the simplest solution to handle this
  • step 2 (for us): upload to a temporary database to do outer joins
• json-ld and SPARQL used together
• a bunch of json-ld files in a no-sql database (like mongo)
• quads (named graph) will be used to store data provenance
  • every triple has a name
  • subject, predicate, object, the fourth value is some sort of ID
• integrate Solr with SPARQL to have a free text search
• lazy learning/lazy searching - work hard only when trying to find information
  • KNN
• will store anything that is text - such as album reviews

[maregold]

DATA LAKE 8/15/23

• Van & Rebecca: SIMSSA DB Flattening
  • Van talked to Cory - focus on a working "something" and then can focus on the details
  • SIMSSA DB has a lot of layers - like 40 tables
• would be beneficial if the databases could give us entity relationship diagrams
• Utilize WikiData for the metadata mapping we do have to do
  • SIMSSA DB has features that wouldn't be on WikiData
• Using MusicBrainz and Cantus and one more (Dig That Lick? Weimar Jazz?) to see where we would need standardization for the schema
• Part of the submission, for any columns that may not be understood, have them describe what they are
• json to quads to nosql database pipeline research
  • need to know more about quads in general

[maregold]

DATA LAKE 8/22/23

• SIMMSA DB Recap (from Rebecca's discussion post)
  • Wouldn't be terrible to have to manually do this process, if we write a script it may not work for everything
• Van looking into Solr text searching
• Trying to figure out the pipeline between flattening to OpenRefine to Solr
  • How to best use OpenRefine
• Priority is the search
• How to handle specific information (Weimar Jazz Database)
  • multiple performances - how to distinguish them
• LinkedArt
• Large chunks of info (Composer)
  • When you search Bach you should get a compilation of information with links to the databases (this is mentioned in this one this one etc.)
    • If there is differing information, that is specifically pointed out
  • and then for specific renditions of something, that is also its own result
• Storing overlapping information (dates, names)

[annamorphism]

Datalake Meeting notes, 29/08/2023
LinkedMusic intro (for new members): integrated search that points user to partner databases; original plan had schemas but have since turned to Wikidata/datalake idea.

Current problem:
Converting relational databases to CSV, SIMSSADB as test case
Van and Rebecca have a TSV of a full outer join of everything
Another version here
Might be improved with each row as a file/item rather than feature

CSV/TSV format is temporary on the way to making each row a JSON file
Each title will become a Wikidata URI e.g. P2701: “file format”. May need to be reconciled.
Some URIs will need to be created for feature types; will store the value as the object in the triple.
To investigate: is there a standard way to do this JSON-LD conversion? What does it look like?

Question: why JSON-LD and not RDF (which solr works with)
A: not everything works well in RDF (e.g. reviews); if it did we could use SPARQL, but that requires everything to be well organized, which we’re not. JSON-LD will allow text, reviews, etc.

What would this look like for Cantus Ultimus?
Images are the row (where in Cantus it’s the chant id)—each image has a URI in the IIIF file.
How to incorporate the MEI? Volpiano? And use solr to search?
discussion: the problem of combining metadata and content
Cantus Ultimus should have a music search (integrated with the other databases) BUT this is a several-years-out problem
For now just use what is in the database as metadata (so computed features are in SIMSSA but not elsewhere)

[annamorphism]

Datalake Meeting notes, 05/09/2023

Van: discussion of flattening of database with each piece/file type as a row (see https://github.com/malajvan/linkedmusic-datalake/tree/main/simssadb)
Discussion of how this looks in JSON-LD and how this will interact with solr/elastic-search (to be researched)

Features to use in mock-database (SIMSSA+Cantus+??): composers? range? final pitch?

For Cantus portion, will need 50 chants with “composer” (pope Gregory) and the modes
(This is now here: #11)
SIMSSA has no "mode"--could use “pitch of last note” as proxy for (authentic) mode & label with Wikidata property
(Further discussion on mode/tonality as a property here: #8)

Once the files are prepped these @dchiller @jinh0 and @homework36 will look into indexing in solr/elastic search to run some queries across databases (“find pieces that are in Phrygian”). @malajvan to delegate various other issues that need prepping.

Difficulties of dealing with music-specific features discussed. See discussion here: #10

[annamorphism]

Data Lake Meeting Notes, 12/09/2023
-Have a nice JSON-LD dump of CantusDB sample chants (see #11) and SIMSSA; reconciled to WikiData &Schema.org (per #10 (reply in thread))
-Explanation: keys are linked somewhere; every entry has its own context, linking to context file; uses "wdt" for WikiData properties and links to database-specific IDs
-workflow still a bit manual, to be improved.
-still some clean up to do: P1476 is supposed to be a list of names, for example
-this approach is language agnostic (except for e.g. "files," "last pitch class" from the original data).

Some choices made in SIMSSA:
-File: loops over a musical work, each file has own JSON ID (bc of how Simssa is set up wrt feature extraction on files)
-some works have no file, some have several

-P86, composer—this links to Wikidata
-P214(??) is VIAF id: may be useful for automatic reconcialition (finding Wikidata IDs) but not store it afterward

-Note that Q-IDs have the format "wd:Q1234" in order to expand properly using pyld; P-ID are values and not expanded, thus us different format.

Compare to CantusDB JSON-LD:
-inclusion of Volpiano; note this is before "context" (dictionary key, order doesn't matter)
-Properties: P86 is composer, P1922 incipit, P136 Genre Q73178 Mode, Q4484726 as Final
-having Q here could ultimately impact SPARQL; will keep for now
-hard to add everything as a property as Wikidata has a vetting process; will use extant properties if they fit well (how well?), otherwise Q.
-when to include original label e.g. "Finalis"? In this case, have a close match on WD. Encode enough not to have to look back every time. May be able t index some of these with solr.

[Further discussion on properties and context: #13]

Some initial forays into solr, explanation of schemas:
-ID+string+indexed "true" (i.e. can be searched)+stored (returned or calculated)
-Dynamic field=asterisk+underscore; string_i treated as integer and passed as field name; string_ss is multivalued
-note: these suffixes were manually added in and individually decided on. process needs streamlining.

-Started by indexing the unexpanded file; expanded may be more useful in the future
-note: can index/query nested documents; for the SIMSSA files, can create work and files& index relationship
-next steps will be using the context to expand the file and index that

Future datasets:
-Metabrainz: discuss at project meeting
-DIAMM, for potential overlap?
-Session.org, mentioned here: #6
[Note: this discussion developed here: #12 ]

[annamorphism]

Meeting notes, 19/09/2023

Update on CantusDB data dump:
@dchiller commented that some data was poorly handled e.g. “None” in cells and things not found in Wikidata (see malajvan#9).
Was decided to have None be null (i.e. not have the key, delete in JSON.)
Non-reconciled things have more options: could be null, could be pure text, or find another source for it
Decided to prioritize ease of contribution over full-linkedness (given the existence of many short-lived URI providers)
-->keep as text for now

Discussion of IDs vs values and effect on indexing. Decision to try both. (see also #14)

Desired workflow: want things automatic so that when databases update, we don’t have to do anything (provided the metadata scheme is unchanged). Note: there is a service on OpenRefine (Reconcile-CSV)--takes two databases and reconcile with one another; could use this to reconcile old and new?
-keep what an original database meant (“last pitch class”, “tonality”) even though not language agnostic, so that user can query what they want

-SIMSSA flattening update: getting rid of VIAF ID (use to reconcile but not store).
-Question: ARE these IDs always aligned? (Probably, at least for people, but not worrying about it for now.)

-CantusDB: generated a new file with more+improved columns. #11 (comment)
-Question: do the headers (P vs Q) matter for SPARQL? (at the moment no bc it is such a mess…just find every label with X instead of “property matches X”)

Properties discussion:
List of properties made by Van in #13
Note: want to distinguish between “Alban Berg” the composer and “Alban Berg Quartet”;
“Author”: this is a user from The Session who wrote a setting
Note (in general context.md) efforts to follow Wikiproject Music. Might be nice to look over these guidelines
(n.b. looking for adding "properties about musical instruments"!)

Solr update:
#16

-Manual suffix adding: now automated for Cantus and SImssa, but The Session had a breaking use case:
settings vs recordings which each have attributes
(A "tune" can have various recordings, recordings themselves have metadata. see /the_session/jsonld/compact.jsonld)
Do we want to keep them all as single entity (analogous to "chant" on cantus DB)
-->When there are multiple interrelated things being stored (sessions, recording) maybe there should be multiple things? Rows for recordings, rows for settings—would have to be two different tables (all text, so low storage needs). But we don’t want to recreate the relational database…
-Each row could point to a file (but SIMSSA was not done this way)

-Implications for Solr : might want to return tunes, recordings or settings, depending on the query ("all recordings by X artist")
-could have multiple flattenings?
-solr supports indexing nested child documents in the json; sticking with this unless we find it can’t do something.

Future datasets:
-Metabrainz and json-ld output (to discuss at LM meeting)
-TROMPA project took IMSLP, dumped to graph database, produced json-ld from that
-LinkedBrainz: to publish MusicBrains as linked data
-To do: try dumping top 50 from MusicBrainz see what happens
Some results here: #15
-Add databases with export to #9

Elastic search: still work in progress

To do: For project meeting demo, have an ugly box with a query.
[https://github.com//issues/17]

[dchiller]

Agenda for 26 Sept

1. Update on reconciling and exporting databases to JSON-LD

• Non-reconciled values

Was decided to have None be null (i.e. not have the key, delete in JSON.)
Non-reconciled things have more options: could be null, could be pure text, or find another source for it
Decided to prioritize ease of contribution over full-linkedness (given the existence of many short-lived URI providers)
-->keep as text for now
Discussion of IDs vs values and effect on indexing. Decision to try both. (see also #14)

• Automation of this reconciliation and flattening process

Desired workflow: want things automatic so that when databases update, we don’t have to do anything (provided the metadata scheme is unchanged). Note: there is a service on OpenRefine (Reconcile-CSV)--takes two databases and reconcile with one another; could use this to reconcile old and new?
-keep what an original database meant (“last pitch class”, “tonality”) even though not language agnostic, so that user can query what they want

• Updates on exporting CantusDB

-CantusDB: generated a new file with more+improved columns. #11 (comment)
-Question: do the headers (P vs Q) matter for SPARQL? (at the moment no bc it is such a mess…just find every label with X instead of “property matches X”)

• Update on exporting MusicBrainz

2. Update on search (with solr)

-Implications for Solr : might want to return tunes, recordings or settings, depending on the query ("all recordings by X artist")
-could have multiple flattenings?
-solr supports indexing nested child documents in the json; sticking with this unless we find it can’t do something.

3. Update on Elastic Search implementation?
4. Anything else?

[maregold]

9/26

1. Update on reconciling and exporting databases to JSON-LD

• Non-reconciled values

non-reconciled values should just be a string (could be pure text option) - van is resolving everything to text
everything as text that is coming in is going to be replaced by a reliable id and then the text gotten rid of
if a term isnt in wikidata as a property, that term should be kept, or it should be namemapped
just keep as it was in the original database for everything
what is the relationship between schema.org and wikidata? - equivalent property or equivalent class in wikidata
CantusID - either make a property or create a wikipedia page about it and then wikidata will pick it up
can we provide through solr extra linkage - ich would rather have standardization on the query end, not through wikidata

• Updates on exporting CantusDB

cantus ID has been added

• Update on exporting MusicBrainz
  • working on both the json ld and postgres versions

2. Update on search (with solr)

3. Update on Elastic Search implementation?

• on the local server
• compact data stored

[dchiller]

Agenda for 3 October

1. Update on export from Music Brainz

2. Basic search box powered by solr (see Perform basic search on CantusDB, SimssaDB, and the Session #17 for current process)

3. Update on progress with elastic search

4. Review topics for LinkedMusic meeting presentations (Potential talks/discussions at Oct LinkedMusic Project Meeting #19)

• Reminder: Run-through of talks next Friday (13 October, 10am - noon, lab)
• Brainstorm of topics for the LinkedMusic technical discussion (Potential discussion topics for LinkedMusic technical session #21)

6. Anything else?

[maregold]

DATA LAKE 10/3

• Update on export from Music Brainz

  • software that lets you run openrefine in batches (for when you need to update data)
  • thinking about just reconciling the IDs (using the batch software potentially)
    • MB doesn't reconcile with WikiData, they use their own internal linkage instead (linking to their own artist page instead of the WikiData page)

• Basic search box powered by solr (see Perform basic search on CantusDB, SimssaDB, and the Session #17 for current process-

• Update on progress with elastic search

  • server issues (i couldn't hear super well so if anyone has notes please put them here!)

• Review topics for LinkedMusic meeting presentations ( Potential talks/discussions at Oct LinkedMusic Project Meeting #19)
  Reminder: Run-through of talks next Friday (13 October, 10am - noon, lab)

• Brainstorm of topics for the LinkedMusic technical discussion ( Potential discussion topics for LinkedMusic technical session #21)

6. Anything else?

• if there is no secondary information besides the wikidata ID, that means we added it
• all items not have a wikidata ID and then the original stored item
• how to handle someone searching for a name in a different language / different name (Mixolydian vs tone 8) - need to reconcile everything and index the ids

[dchiller]

Agenda for 10 Oct

1. Batch re-reconciliation

Does OpenRefine allow us to do this?
Can we reconcile just with ID's

2. Basic search functionality

In reconciled data, some items just have wikidata ID and some have ID + text
if there is no secondary information besides the wikidata ID, that means we added it in the reconcilia
all items without a wikidata ID have just the original information

3. Elastic search updates

Was going to try to set-up a remote environment but now sticking with local version
Updates on working with Kibana?

4. Anything else?

[maregold]

DATA LAKE 10/10

1. Batch re-reconciliation

Does OpenRefine allow us to do this?
Can we reconcile just with ID's - tried this, was painful with MusicBrainz

OpenRefine (OR) remembers what you've chosen before with Wikidata - export a list of actions and just reapply it (need same column names)

schema.org discussion post
schema.org is all in English
schema.org might have more P's
in the mapping project from WD to schema.org, lots of blanks
we will get to DBpedia later - they work off of Wikipedia
For example, discrepancies between how Wikidata and schema.org handle publication date vs release date

TO DO: make sure to document decisions made with Alastair at upcoming meeting on GitHub

2. Basic search functionality

In reconciled data, some items just have wikidata ID and some have ID + text
if there is no secondary information besides the wikidata ID, that means we added it in the reconciliation
all items without a wikidata ID have just the original information

search update discussion post
do we search at all using WD IDS vs. full text
how to handle different language search
we have to keep all the different spellings in the index of solr
in order to do sparql you need a good ontology, data lake gets around that
should we index wikipedia, grove music, etc. for better full text search

grabbing the multiple languages from the wikidata profile for search
we do not need the QID for search

3. Elastic search updates

Was going to try to set-up a remote environment but now sticking with local version
Updates on working with Kibana?

elastic search/json-ld problem discussion post (#24)
key-values
adding a name value

[dchiller]

Agenda for 17 Oct

1. Continuation of project meeting presentations
2. Elastic Search vs Solr comparison
3. Anything else?

[dchiller]

Agenda for 24 October

1. Reflections and learnings from the project meetings
   • Wonderful presentations everyone!
   • Some have already started a discussion to hold onto data lake-related notes from the weekend at LinkedMusic Project Meeting - Data Processing Note Dump #27 !
2. Next steps: create issues for upcoming development

[maregold]

DATA LAKE 10/24

• Reconciliation Service API - Van will need to test to see if the choices can be re-used like in OpenRefine
• The MusicBrainz JSON is still nested
• Using the API might be more complicated for those who aren't programmers - OpenRefine is easier, so getting rid of OpenRefine may cause problems
• MusicBrainz canonical - project to choose the canonical release
  • 10 releases for the same song, which is the first
  • They utilize string matching for this - which requires a lot of coding
    • Is a type of reconciliation
    • Would only work for us if MusicBrainz had the Wikidata ID
    • We don't need to have everything in Wikidata
      • We are using Wikidata for things that don't have authority somewhere stable (such as our other databases)
• Using the canonical method to find the "original" Wikidata entry for an instrument (for VIM)
• What is the necessity of maintaining nested JSON format
  • We can have content in JSON-LD, but not nested (for SPARQL queries later)
• Go back and re-reconcile Cantus so Mode 1 etc. can show up
• If Wikidata does combine everything together (for anon and unspecified) then we should separate it
• Have the databases make a URI for their niche properties
• https://www.wikidata.org/wiki/Wikidata:Deletion_policy

[dchiller]

31 October Meeting Agenda

1. Reconciliation updates
   • Exploration of reconciliation API
   • Exploration of MusicBrainz
2. Data Processing and Enhancement (Data Processing and Search Enhancement Discussion #30)
3. Search updates
4. SPARQL/graph (SPARQL (GraphDB) and Solr Integrations #4)
   • Virtuoso and Apache Jena

[dchiller]

Query of the week: Find a composer born before 1500 who wrote a piece that contains the text "SOME TEXT"