Adding books with non-ASCII characters error

When trying to add a book with a title that contains non-ASCII characters from some sources, the non-ASCII characters are garbled or fetched incorrectly.

I haven't tried all sources yet, but it fails with these sources:
Malmö Stadsbibliotek
LIBRIS, svenska forskningsbibliotek
Helsinki Metropolitan Libraries

But it works with these ones:
Göteborg University
Bibliotek.dk

An ISBN to try is 9118540511. The correct title is I samspråk med djuren.

Before realising that several sources are affected, I emailed technical support at Libris, but they have made no changes to their interface recently. I know that it worked on April 10th but not on April 30th.

Known bug, haven't heard recently any comments about this from LT people. I understood that "we the users" should contact each library separately and try to get them to fix the data format back to UTF-8.

So the problem is that LT is most likely expecting UTF-8 data, but some library fields actually contain unicode characters. As you've noticed the problem is made more difficult by LT caching older data even after libraries have fixed their data... Fix for that would be to somehow force refetching the corrupted data from libraries.

That's not quite true. LT has reset certain libraries to other data formats. But it is hard for staff to findout what format is being used because of language difficulties, so it generally gets done when there is a member able to act as a bridge and find out where the difficulty lies.

Thanx for update, that makes sense! Still it's manual process and handled case by case :(

I don't understand. These sources have worked perfectly for the nine years I have been here, up until less than a month ago, and they have changed nothing on their side. What is the bug that I should contact them about?

>1 anglemark: Thanks for the report -- we'll take a look and see where the character encoding issue is coming in, either in the Add Books part of it or within the adding part of it.

>1 anglemark: Malmö Stadsbibliotek should be working now (changed from marc8 to utf8 as record encoding)

>1 anglemark: Looking at LIBRIS.

>1 anglemark: LIBRIS: looking at information about z39.50 here: http://www.kb.se/libris/teknisk-information/libris-via-Z3950/ I've changed the entry for them to latin-1 accessing the libri database. I think that's looking good now?

>1 anglemark: Helsinki Metropolitan: this one has been trickier. Can you take a look again - that ISBN seems to be working, which I hope means everything should work.

Helsinki Metropolitan really seems to be tricky.
I tested some of the same books as here: https://www.librarything.com/topic/182072#4966743

Add books search from HelMet for
"Pekka Töpöhäntä naamiaisissa" finds nothing
"9789512039791" finds nothing

"knutsson, pekka, naamiaisissa" finds "Pekka Töpöhäntä naamiaisissa by Gösta Knutsson (1992). (more) ISBN: 9512039796"

"9512039796" finds nothing

Search for
"Suomen lasten eläinsadut" finds nothing

"9511198181" finds
Suomen lasten elčainsadut by Pirkko-Liisa Surojegin (2004).
(more)
ISBN: 9511198181
Publication: Helsingissča : Otava, 2004.
Subjects:
kaunokirjallisuus
sadut
elčaimet
lastenkirjallisuus
elčainsadut
Suomi

where ča should be ä - eläinsadut, Helsingissä, eläimet.

I had a weird one a few days ago from British Library where Arnaldur Indriðason came through as Arnaldur Indri©ʻason,

OK.

1. Malmö Stadsbibliotek still doesn't work. It works for the ISBN I gave, but I tried another one - 9100572624 - and that has the same problem. It should be Skuggornas förtrogna : om Maria Gripe.

2. Libris seems to work.

3. Helsinki Metropolitan seems to work for 9118540511 but for 9100572624 I get the problem that vivir reported.

>11 vivir: >13 anglemark: Thanks for the testing. I'll take another look and see if I can figure out what's going on.