Openlibrary: Fix book titles with mangled Unicode

Both these records came from archive.org.

I've looked at the _marc.xml files on archive.org. Last modified time of both the marc.xml files are in 2007 and these recrods were created in 2008. It looks like the issue is with the script that parsed those titles.

There are thousands of MARC import records where accented characters have been mangled or handled incorrectly. Another common scenario is that accents or other diacritical marks have been replaced by a space before or after the vowel.

See for example:

http://openlibrary.org/authors/OL4459814A/Heinrich_Schro_der

http://openlibrary.org/works/OL10684450W/Tonbandgera_te-Messpraxis

http://openlibrary.org/show-records/talis_openlibrary_contribution/talis-openlibrary-contribution.mrc:299045317:529

In both the author and title, the umlaut has been changed to a space after the vowel. The linked MARC record shows correctly in the browser.

Should we consider re-importing? And is #149, which also references https://bugs.launchpad.net/openlibrary/+bug/598204, a dependency?

Some I have found recently: https://openlibrary.org/works/OL17670297W
https://openlibrary.org/works/OL17677126W
multiple works by this author:
https://openlibrary.org/authors/OL2450531A/Matthieu_Joseph_Bonaventure_Orfila

https://openlibrary.org/search?q=title%3A+%22©♭%22&mode=everything Still finds well over 17 million matches. This for "é", likely the most common accented letter. Edits like https://openlibrary.org/books/OL26303038M/Anatomie_générale_appliquée_à_la_physiologie_et_à_la_médecine?b=3&a=1&_compare=Compare&m=diff should not be necessarily manual.

@hornc re your May 8 comment, those works were created from editions created from importing
https://openlibrary.org/show-records/ia:b28044277_0001
and
https://openlibrary.org/show-records/ia:b2202010x
Until they're fixed in the ia MARC records, there's no value in reimporting unless the importation gets them passed through normalization

@LeadSongDog interesting, the MARC display you link to shows the characters garbled, but if you click through to the XML representation https://ia800202.us.archive.org/34/items/b28044277_0001/b28044277_0001_marc.xml the accented e's display correctly. There may be an issue with encoding types set incorrectly? I will pick this up shortly, the new openlibrary-client is now in a state where it can be used to make bulk data corrections.

@LeadSongDog I may have figured out how the mangling is happening, in this example marc xml
https://ia600208.us.archive.org/25/items/b2202010x/b2202010x_marc.xml

the a-grave of "Secours à donner" displays correctly in utf-8 encoding

a-grave is U+00E0, which in binary (pythonic notation) is \xC3\xA0

if those bytes were interpreted as MARC8 and "converted", C3 becomes the copyright symbol, and 'A0' becomes a space, which is exactly what we see on the OL pages with "Secours © donner"

I now think these MARC records have utf-8 character encodings, but were imported to OL as if they were MARC8, which explains the mangling.

I did the MARC8 conversion manually from the tables found here https://memory.loc.gov/diglib/codetables/45.html I'll need to use yaz or something to test this out properly, but this will provide a good path to fixing the MARC errors programatically.

I know that there are other unicode mangling errors affecting Amazon imported records, but I think that is from incorrect conversion from Windows or ISO charsets

Thanks for your comment @LeadSongDog, in trying to figure out whether the MARC records were actually wrong or not I think I have stumbled upon the root cause of the issue!

@hornc any updates on MARC mangling and/or if we resolved this issue?

The issue is definitely not resolved. When the import script is fixed, @bfalling suggest of reimporting will most likely be necessary.

From the point of view of triage, it would probably be useful to get an actual count. "Thousands" isn't a very big percentage of 25 million editions.

Has this been resolved with our Python 3 changes or can someone provide steps-to-reproduce on Python 3?

Well https://openlibrary.org/books/OL12903648M/Etudes_Conomiques_De_L'Ocde certainly isn’t fixed, but perhaps we’re done with digging the hole...
There were at least three problem classes:

1. Bad import of good data
2. Literal import of bad data
3. Bad data in place from old cases of 1 or 2 since remedied.
   The move to py3 will at most fix number 1.

Steps-to-reproduce problem class 1?

The earlier examples are better than the most recent one which is an import from crappy Amazon data (which we should not be imported).
https://openlibrary.org/books/OL7165183M/The_Vikramorva%C2%A9%C3%98s%C2%A9%C4%90iyam
https://openlibrary.org/authors/OL4459814A/Heinrich_Schro_der
https://openlibrary.org/books/OL13956174M/Tonbandgera_te-Messpraxis
https://openlibrary.org/books/OL26280693M/Secours_%C2%A9_donner_aux_personnes_empoisonn%C2%A9%E2%99%ADes_ou_asphyxi%C2%A9%E2%99%ADes_suivis_des_moyens_propres_%C2%A9_reconna%C2%A9%CA%BEtre

If the bug has been fixed, reimporting the records should result in the correct encoding. Then the task simply becomes reimporting the millions of corrupted records.

The search which was claimed to return 17+ million records before: https://openlibrary.org/search?q=title%3A+%22%C2%A9%E2%99%AD%22&mode=everything
now returns 23.4M results, but I think that's actually a separate bug and it's just returning all works in the database.

@tfmorris As https://openlibrary.org/search?q=title%3A+%22+%22&mode=everything gets the same result, it seems that yes, it is a simple case of title search for an effectively-blank string.

I created #4223 for the search bug.