Trouble reading Harvard Open Metadata MARC files (UTF-8 related?)

[viking2917]

I am trying to use pymarc to read the Harvard Open Metadata MARC files.

Most of the files process ok but some (for example ab.bib.14.20160401.full.mrc) produce errors when processing. The error I am getting is:

Traceback (most recent call last):
  File "domark.py", line 21, in <module>
    for record in reader:
  File "/Library/Python/2.7/site-packages/six.py", line 535, in next
    return type(self).__next__(self)
  File "/Users/markwatkins/Sites/pharvard/pymarc/reader.py", line 97, in __next__
    utf8_handling=self.utf8_handling)
  File "/Users/markwatkins/Sites/pharvard/pymarc/record.py", line 74, in __init__
    utf8_handling=utf8_handling)
  File "/Users/markwatkins/Sites/pharvard/pymarc/record.py", line 307, in decode_marc
    code = subfield[0:1].decode('ascii')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 0: ordinal not in range(128)

The driver code I am using is:

#!/usr/bin/python -tt
# -*- coding: utf-8 -*-
import codecs
import sys
from pymarc import MARCReader

UTF8Writer = codecs.getwriter('utf8')
sys.stdout = UTF8Writer(sys.stdout)

if len(sys.argv) >= 2:
    files = [sys.argv[1]]

for file in files:
    with open(file, 'rb') as fh:
        reader = MARCReader(fh, utf8_handling='ignore')
        for record in reader:
#            print "%s by %s" % (record.title(), record.author())
            print(record.as_json())

Other MARC processing tools (e.g. MarcEdit seem to process the file with no issues so I think the file is legitimate).

Am I doing something wrong? Is there an issue with pymarc, possibly UTF-8 processing related?

[Wooble]

Can you isolate a single record that's displaying this problem?

From the traceback it appears that there's a subfield code that's not ASCII, which is forbidden by the MARC21 spec.

(If such records exist in the wild, though, pymarc should probably have a way to deal with them. This is one area where there's currently no workaround as far as I know...)

[viking2917]

Thank you! Working on isolating the record...it's unfortunately a massive binary file. :(. But from the python debugger, it does look like you are correct that there's unicode chars in a subfield, e.g.

(Pdb) entry_tag
u'040'
(Pdb) subs
[u'  ', '\xc4\x81TRCLS', 'beng', 'erda', 'cTRCLS', 'dOCLCO', 'dHMY']

[viking2917]

Here's what the record looks like dumped to a text file by MarcEdit. It does indeed look like the 040 record has unicode in the subfield code.

If the MARC21 spec indeed forbids this, then this issue should probably be closed, although more tolerant error handling might be helpful.

=LDR  01444cam a2200397Ii 4500
=001  014333604-5
=005  20150806114915.0
=008  150209s2015\\\\ja\a\\\\\\\\\\000\0\jpn\d
=020  \\$a9784480068163
=020  \\$a4480068163
=035  0\$aocn902996729
=040  \\$āTRCLS$beng$erda$cTRCLS$dOCLCO$dHMY
=090  \\$aNA6310$b.A53 2015
=100  1\$6880-01$aAkase, Tatsuzō,$d1946-$eauthor.
=245  10$6880-02$aEki o dezain suru :$bkarā shinsho /$cAkase Tatsuzō.
=264  \1$6880-03$aTōkyō-to Taitō-ku :$bChikuma Shobō,$c2015.
=300  \\$a254 pages :$billustrations ;$c18 cm.
=336  \\$atext$btxt$2rdacontent
=337  \\$aunmediated$bn$2rdamedia
=338  \\$avolume$bnc$2rdacarrier
=490  1\$6880-04$aChikuma shinsho ;$v1112
=650  \0$aRailroad stations$xDesign and construction.
=650  \0$aRailroad stations$vDesigns and plans.
=650  07$6880-05$aEki.$2jlabsh/4
=650  07$6880-06$aShinboru māku.$2jlabsh/4
=880  1\$6100-01$a赤瀬達三,$d1946-$eauthor.
=880  10$6245-02$a駅をデザインする :$bカラー新書 /$c赤瀬達三.
=880  \1$6264-03$a東京都台東区 :$b筑摩書房,$c2015.
=880  1\$6490-04$aちくま新書 ;$v1112
=880  07$6650-05$a駅.$2jlabsh/4
=880  07$6650-06$aシンボルマーク.$2jlabsh/4
=830  \0$6880-07$aChikuma shinsho ;$v1112.
=880  \0$6830-07$aちくま新書 ;$v1112.
=988  \\$a20150327
=049  \\$aHMYY
=906  \\$0MH

[edsu]

It looks like the record is coded as containing Unicode (leader position 9). I forget, why are you using

reader = MARCReader(fh, utf8_handling='ignore')

[viking2917]

I was using 'ignore' because otherwise the processing would stop when it encountered encoding difficulties. (aside: I wonder if better error handling would be to skip offending records and keep going?). Right now when it throws an exception the script halts. (Caveat: I am a python newbie and likely doing something wrong).

[gugek]

@viking2917: right sadly the ignore parameter doesn't ignore everywhere in the MARC field, and also doesn't ignore in areas where UTF-8 isn't permitted.

ALEPH (the ILS Harvard is on) will let you save a unicode character to a subfield.

I have a branch somewhere that does some of this error handling.this for another project

In record.py:

for subfield in subs[1:]:
    if len(subfield) == 0:
        continue
    try:
        code = subfield[0:1].decode('ascii')
    except UnicodeDecodeError:
        if utf8_handling == 'replace':
            code = unidecode(subfield[0:1].decode(encoding,
                                 utf8_handling))
            message = "tag {0}: utf8 - sf code {1}"\
                                   .format(entry_tag, code)
            if self['001']:
                message = "=001 {0}: ".format(self['001'].data)\
                    + message
            logging.error("{0}".format(message))
        else:
            raise

https://github.com/gugek/pymarc/blob/leader-handling/pymarc/record.py

[viking2917]

@gugek Thank you! Will give that a go.

[Wooble]

That sounds like a reasonable patch to me (of course, it wouldn't have helped here, with utf8_handling='ignore'; I can't think of a good way to use ignore here since it implies ending up with the subfield code completely blank.)

Decomposing the "ā" and throwing away diacritics would probably do the right thing in this particular case, since I can't see how that's possibly supposed to be anything but $a... But I don't know that's a good general solution :)

[viking2917]

Yes, 'replace' is probably the right option - I was happy to simply discard records with errors so was using 'ignore', but with 'replace' and this patch, I seem to be able to get pretty much everything.

Thanks everyone!

[viking2917]

I am not sure how aggressively the Harvard Open Metadata project is being maintained, but
I've reported the issue in case someone there is actively maintaining it.
(It is a treasure trove of open source book metadata....)

Thanks.

Mark

On Mon, Apr 25, 2016 at 4:44 PM, Jim Nicholls notifications@github.com
wrote:

Not that this helps anyone usefully progress, but I did want to point out
that this in fact not a valid MARC record.

According to the MARC 21 Specification
https://www.loc.gov/marc/specifications/specrecstruc.html:

subfield code "The two-character combination of a delimiter followed
by a data element identifier. [...]"

delimiter "ASCII control character 1F(hex) [...]"

data element identifier "A one-character code used to identify
individual data elements within a variable field. The data element may
be any ASCII lowercase alphabetic, numeric, or graphic symbol except blank."

And according to the Character Sets and Encoding Options
https://www.loc.gov/marc/specifications/speccharintro.html section:

ASCII "[...] a 7-bit coded character set [...]"

ASCII numerics "ASCII code points 30(hex) through 39(hex)"

ASCII lowercase alphabetics "ASCII code points 61(hex) through 6F(hex)
and 70(hex) through 7A(hex)"

ASCII graphic symbols "The ASCII graphic characters other than
numerics, alphabetics, space, and delete. Code points 21(hex) through
2F(hex), 3A(hex) through 3F(hex), 40(hex), 5B(hex) through 5F(hex),
60(hex), and 7B(hex) through 7E(hex) are included."

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#89 (comment)

Get great book recommendations at The Hawaii Project
http://www.thehawaiiproject.com - my new startup!

mark.watkins1@gmail.com
twitter: @viking2917 https://www.twitter.com/viking2917
linkedin: https://www.linkedin.com/in/markwatkins1

[josepablog]

I know this is an old issue, but I'm having the same problem than @viking2917 ...

I'm trying to parse the Harvard Open Metadata db, and I'm running into exceptions. Is there any patch that I could do? My code breaks and I cannot catch the exception to skip the record.

(I do not understand what @gugek suggested)...

[Wooble]

Can you isolate a record that displays the issue and attach it as MARC21? (I'd prefer to avoid trying to manually create a bad record by hand for testing, and I don't think my ILS can create one nor can I do it programmatically in pymarc :) )

[viking2917]

@josepablog I did finally get around this problem. Here's how I got around it:

First I altered record.py (new file attached), to add some error handling.
record.py.txt

Instead of this driver from the github page:

from pymarc import MARCReader
with open('test/marc.dat', 'rb') as fh:
    reader = MARCReader(fh)
    for record in reader:
        print(record.title())

I changed the driver to this: (basically, importing codecs and sys.). As I recall I had to install

#!/usr/bin/python -tt
# -*- coding: utf-8 -*-
import codecs
import sys
from pymarc import MARCReader

UTF8Writer = codecs.getwriter('utf8')
sys.stdout = UTF8Writer(sys.stdout)

if len(sys.argv) >= 2:
    files = [sys.argv[1]]

for file in files:
    with open(file, 'rb') as fh:
        reader = MARCReader(fh, utf8_handling='replace')
        for record in reader:
            print(record.as_json())

On my mac, I needed to install a few python package:

sudo pip install unidecode
sudo pip install -U six

(I am not 100% sure if I needed to install six or not. Your mileage may vary).

I'm a python n00b and not sure this code is really production-ready, so I did not create a pull request. But it's been working for me. I only half-understand what I did, as I really don't know Python. Good luck!

[josepablog]

@Wooble the file is huge! And my understanding of Marc-21 is extremely limited

I'll give a try to @viking2917 's solution, and hopefully it would work...

Thank you to both!

[viking2917]

(Aside: I traded some emails with the good folks at Harvard and they said something to the effect that there are in fact the occasional invalid record due to the large, distributed nature of their libraries and data. They did correct the issues I brought to their attention but I think it's a good idea to protect against invalid data where possible.)