Handling XML from external namespaces intermingled (with potentially different namespaces)

[ronaldtse]

This is from:

• Spec fixes rng#4

There are XML from external namespaces intermingled in RNG that need round-tripped.

Since these things can happen anywhere, we need to think how this works. Do we allow other elements to occur in XML we parse? As an extension mechanism?

  9) RELAX NG Specification Tests For October 26 version of the spec. For October 26 version of the spec. Section 3 For October 26 version of the spec. Section 3: Tests for foreign element and attribute handling. Section 3 compliance - correct schema round-trip
     Failure/Error: expect(regenerated).to be_equivalent_to_xml(schema_xml)
     
       Expected XML to be equivalent, but it wasn't.
     
       Diff:
        <?xml version="1.0"?>
        <element xmlns="http://relaxng.org/ns/structure/1.0" name="foo">
       -  <empty>
       -    <ext xmlns="http://www.example.com">
       -      <element xmlns="http://relaxng.org/ns/structure/1.0"/>
       -    </ext>
       -  </empty>
       +  <empty/>
        </element>
     # ./spec/rng/spectest_spec.rb:93:in `block (4 levels) in process_test_suite'


  10) RELAX NG Specification Tests For October 26 version of the spec. For October 26 version of the spec. Section 3 For October 26 version of the spec. Section 3: Tests for foreign element and attribute handling. Section 3 compliance - correct schema round-trip
      Failure/Error: expect(regenerated).to be_equivalent_to_xml(schema_xml)
      
        Expected XML to be equivalent, but it wasn't.
      
        Diff:
         <?xml version="1.0"?>
        -<grammar xmlns="http://relaxng.org/ns/structure/1.0" xmlns:eg="http://www.example.com">
        -  <eg:comment/>
        -  <eg:comment/>
        +<grammar xmlns="http://relaxng.org/ns/structure/1.0">
           <start>
        -    <eg:comment/>
             <element>
        -      <eg:comment/>
        -      <eg:comment/>
               <name>foo</name>
        -      <eg:comment/>
               <data type="string"/>
        -      <eg:comment/>
        -      <empty>
        -        <eg:comment/>
        -        <eg:comment/>
        -      </empty>
        -      <eg:comment/>
        -      <eg:comment/>
        +      <empty/>
             </element>
        -    <eg:comment/>
           </start>
        -  <eg:comment/>
         </grammar>
      # ./spec/rng/spectest_spec.rb:93:in `block (4 levels) in process_test_suite'

Originally posted by @ronaldtse in lutaml/rng#4 (comment)

[HassanAkbar]

Do we allow other elements to occur in XML we parse? As an extension mechanism?

@ronaldtse Can you explain this a bit?

There are some namespacing issues in lutaml-model and I'm trying to resolve them. If you can provide me with an example of what you want to achieve here that would help me a lot. Thanks

[ronaldtse]

I don't actually have a good answer for this question, because right now lutaml-model works with XML in a way that either:

• we have the full structure defined
• we don't work with anything else if the full structure isn't defined (unless we are talking about keeping XML mixed elements as raw)

The important task that we really need to handle is:

• Fix XML namespace usage behavior (was: Add a flag preserve_namespaces: true in the xml block) #329

Let's just stash this task for now 😉

[opoudjis]

Please unstash this task. It is now breaking mn-samples-bipm

In the latest release of metanorma-generic, I am parsing the bibdata record of the current document, and exposing it via lutaml-model to make its content available to Liquid, as a future extension to allow docidentifiers (and other content) to be populated by template.

module Metanorma
  module Generic
    class BibdataConfig < ::Lutaml::Model::Serializable
      class Bibdata < ::Lutaml::Model::Serializable
        model ::RelatonBib::BibliographicItem
      end

      attribute :bibdata, Bibdata

      xml do
        root "metanorma"
        map_element "bibdata", to: :bibdata, with: { from: :bibdata_from_xml,
                                                     to: :bibdata_to_xml }
      end

      def bibdata_from_xml(model, node)
        node or return
        model.bibdata = Relaton::Cli.parse_xml(node.adapter_node)
      end

      def bibdata_to_xml(model, parent, doc); end
    end
  end
end

mn-samples-bipm/sources/rapports/2019-04 has a MathML expression in its title, with a different namespace:

<bibdata>
...
<title language="en" format="text/plain" type="title-main">Internal Standard Reference Data for qNMR: 4,4-Dimethyl-4-silapentane-1-sulfonic 
acid-<stem block="false" type="MathML"><math xmlns="http://www.w3.org/1998/Math/MathML">
<mstyle displaystyle="false">
<msub><mi>d</mi><mn>6</mn></msub></mstyle></math><asciimath>d_6</asciimath></stem> 
[ISRD-07]</title>
...

When I run BibdataConfig.from_xml("<metanorma>#{bibdata.to_xml}</metanorma>").bibdata, which is a Lutaml Model from_xml deserialisation of that record (see above), bibdata returns nil.

When I take out the MathML expression, it processes successfully; so it is the MathML that is aborting deserialisation.

To replicate: Go to mn-samples-bipm/sources/rapports/2019-04, bundle update, then bundle exec metanorma document.adoc

bu = "<bibdata type=\"standard\">\n<title language=\"en\" format=\"text/plain\" type=\"title-main\">Internal Standard Reference Data for qNMR: 4,4-Dimethyl-4-silapentane-1-sulfonic acid-<stem block=\"false\" type=\"MathML\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\">\n  <mstyle displaystyle=\"false\">\n    <msub>\n      <mi>d</mi>\n      <mn>6</mn>\n    </msub>\n  </mstyle>\n</math><asciimath>d_6</asciimath></stem> [ISRD-07]</title>\n\n<title language=\"fr\" format=\"text/plain\" type=\"title-main\"/><docidentifier primary=\"true\" type=\"BIPM\"/><docnumber>BIPM-2019/04</docnumber><contributor><role type=\"author\"/><organization>\n<name>Bureau International des Poids et Mesures</name>\n<abbreviation>BIPM</abbreviation></organization></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Steven Westwood</completename></name>\n<affiliation><organization>\n<name>BIPM</name>\n</organization></affiliation></person></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Norbert Stoppacher</completename></name>\n<affiliation><organization>\n<name>BIPM</name>\n</organization></affiliation></person></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Bruno Garrido</completename></name>\n<affiliation><organization>\n<name>INMETRO, Brazil</name>\n</organization></affiliation></person></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Ting Huang</completename></name>\n<affiliation><organization>\n<name>NIM, China</name>\n</organization></affiliation></person></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Takeshi Saito</completename></name>\n<affiliation><organization>\n<name>NMIJ, Japan</name>\n</organization></affiliation></person></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Ilker Un</completename></name>\n<affiliation><organization>\n<name>TUBITAK UME, Turkey</name>\n</organization></affiliation></person></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Taichi Yamazaki</completename></name>\n<affiliation><organization>\n<name>NMIJ, Japan</name>\n</organization></affiliation></person></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Wei Zhang</completename></name>\n<affiliation><organization>\n<name>NIM, China</name>\n</organization></affiliation></person></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Gustavo Martos</completename></name>\n<affiliation><organization>\n<name>BIPM</name>\n</organization></affiliation></person></contributor><contributor><role type=\"publisher\"/><organization>\n<name>Bureau International des Poids et Mesures</name>\n<abbreviation>BIPM</abbreviation></organization></contributor><edition>1</edition><version><revision-date>2019-03-14</revision-date></version><language>en</language><script>Latn</script><status><stage>in-force</stage></status><copyright><from>2019</from><owner><organization>\n<name>Bureau International des Poids et Mesures</name>\n<abbreviation>BIPM</abbreviation></organization></owner></copyright><relation type=\"supersedes\"><bibitem><date type=\"circulated\"/><version><draft/></version></bibitem>\n</relation><ext><doctype>rapport</doctype><flavor>bipm</flavor><editorialgroup><committee acronym=\"\" language=\"en\" script=\"Latn\"/><committee acronym=\"\" language=\"fr\" script=\"Latn\"/></editorialgroup><structuredidentifier><docnumber>BIPM-2019/04</docnumber></structuredidentifier></ext></bibdata>"


BibdataConfig.from_xml("<metanorma>#{bu}</metanorma>") # nil

bu = "<bibdata type=\"standard\">\n<title language=\"en\" format=\"text/plain\" type=\"title-main\">Internal Standard Reference Data for qNMR: 4,4-Dimethyl-4-silapentane-1-sulfonic acid- [ISRD-07]</title>\n\n<title language=\"fr\" format=\"text/plain\" type=\"title-main\"/><docidentifier primary=\"true\" type=\"BIPM\"/><docnumber>BIPM-2019/04</docnumber><contributor><role type=\"author\"/><organization>\n<name>Bureau International des Poids et Mesures</name>\n<abbreviation>BIPM</abbreviation></organization></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Steven Westwood</completename></name>\n<affiliation><organization>\n<name>BIPM</name>\n</organization></affiliation></person></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Norbert Stoppacher</completename></name>\n<affiliation><organization>\n<name>BIPM</name>\n</organization></affiliation></person></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Bruno Garrido</completename></name>\n<affiliation><organization>\n<name>INMETRO, Brazil</name>\n</organization></affiliation></person></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Ting Huang</completename></name>\n<affiliation><organization>\n<name>NIM, China</name>\n</organization></affiliation></person></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Takeshi Saito</completename></name>\n<affiliation><organization>\n<name>NMIJ, Japan</name>\n</organization></affiliation></person></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Ilker Un</completename></name>\n<affiliation><organization>\n<name>TUBITAK UME, Turkey</name>\n</organization></affiliation></person></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Taichi Yamazaki</completename></name>\n<affiliation><organization>\n<name>NMIJ, Japan</name>\n</organization></affiliation></person></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Wei Zhang</completename></name>\n<affiliation><organization>\n<name>NIM, China</name>\n</organization></affiliation></person></contributor><contributor><role type=\"author\"/><person>\n<name><completename>Gustavo Martos</completename></name>\n<affiliation><organization>\n<name>BIPM</name>\n</organization></affiliation></person></contributor><contributor><role type=\"publisher\"/><organization>\n<name>Bureau International des Poids et Mesures</name>\n<abbreviation>BIPM</abbreviation></organization></contributor><edition>1</edition><version><revision-date>2019-03-14</revision-date></version><language>en</language><script>Latn</script><status><stage>in-force</stage></status><copyright><from>2019</from><owner><organization>\n<name>Bureau International des Poids et Mesures</name>\n<abbreviation>BIPM</abbreviation></organization></owner></copyright><relation type=\"supersedes\"><bibitem><date type=\"circulated\"/><version><draft/></version></bibitem>\n</relation><ext><doctype>rapport</doctype><flavor>bipm</flavor><editorialgroup><committee acronym=\"\" language=\"en\" script=\"Latn\"/><committee acronym=\"\" language=\"fr\" script=\"Latn\"/></editorialgroup><structuredidentifier><docnumber>BIPM-2019/04</docnumber></structuredidentifier></ext></bibdata>"

BibdataConfig.from_xml("<metanorma>#{bu}</metanorma>").bibdata # not nil

At the moment, this means that any MathML in the bibliographic descriptions of metanorma-generic-inheriting flavours (BIPM, IHO, CC, CSA, Ribose), and in the bibliographic descriptions of collections, will result in a crash, because their bibliographic data is inaccessible to Lutaml and thus Liquid. I'm doing the release anyway because this won't be that common, but I intend to eventually expand this to all of metanorma, so it does need to be fixed.

[ronaldtse]

@opoudjis the issue here is a bit different. Technically, handling MathML in the text is through allowing MathML inside the object of Stem. Could you help point out the grammar used in the bibdata element so @HassanAkbar or @suleman-uzair can create something that round-trips it?

[opoudjis]

Not seeing that that is that useful:

https://github.com/relaton/relaton-models/blob/bb51846a9639f51a9f665113610a0bcfa263cf78/grammars/biblio.rnc#L601

BibliographicItem =
...
    ## The title(s) of the bibliographic item
    btitle+,
...

## A title of a bibliographic item, associated with a type of title
btitle = element title { TypedTitleString }

TypedTitleString =
  ## Type of title given to a bibliographic item. Indicative values include:
  ## "alternative": alternate title for the item
  ## "original": The original title of the item. Includes the source language title of a translated item
  ## "unofficial": A title that has become prevalent but has never been the official or intended title of the item
  ## "subtitle": Subsidiary title of the item
  ## "main": The default title of the item, privileged in citation
  attribute type { text }?,
  ## (legacy attribute) format; can have value text/plain
  attribute format { text }?,
  ## Content of title
  LocalizedMarkedUpString


LocalizedMarkedUpString =
  LocalizedStringAttributes,
  PureTextElement+

and

https://github.com/metanorma/basicdoc-models/blob/main/grammars/basicdoc.rnc

## Any inline element containing text and associated formatting information.
## Includes inline elements that are identifiers or references to identifiers
TextElement =
  text | em | eref | strong | stem | sub | sup | tt | underline | keyword | ruby |
  strike | smallcap | xref | br | hyperlink | hr | pagebreak | bookmark | image | index | index-xref

## Inline element containing text and associated formatting information,
## but which does not contain any associated identifiers or references to identifiers.
## Restricted recursively to contain only other such inline elements with no identifiers or references to identifiers
PureTextElement =
  text | pure_em | pure_strong | sub | sup | pure_tt | pure_underline | pure_strike | pure_smallcap | br

The grammar has LocalizedMarkedUpString as PureTextElement+, but it should be TextElement+, since stem can be included (or else it should be (PureTextElement | stem)+.

## Mathematically formatted text
stem =
  element stem {
    StemAttributes,
    ## The content of the mathematically formatted text
    ( text | AnyElement )+
}

The grammar has not to this point done anything with namespaces, which is why the MathML in stem has been left in the too-hard basket, and it's just xs:any at the moment.