Thousands of blank (no title, no author) books

This is reposting the bug discussed here but the new system won't allow adding bug status to any thread older than September.

Reposting Tim's message from #37 in that thread where he diagnosed the problem:


Hey. We figured this out. For some reason, Amazon changed it's API in a subtle but very obnoxious way. Previously when there were no results for something, they returned no "results" at all. Now the Amazon API returns one result—a completely empty result (That is, they previously returned XML->Items and now they returned XML->Items->Item but the Item has no title or author). Because we count results this made "no results" turn into "one result." The system didn't have special alarm bells for records that lacked all data because Amazon records always have data. (Live and learn.)

Normal users rarely entered such blank works, but the import queue LOVED them--suddenly everything was found!

This is going to take a while to fix. Basically, the books are all going to be removed. They REALLY don't exist--not on LT, or on Amazon--it's not just some problem moving good data around. Once this is done, the hole will much smaller--confined to books that users somehow combined with a no-title/no-author book, or whatever.

I'm going to be working on whether I should re-run the queue for these.

Bumping this. It would be nice to at least have it acknowledged.

Yep. Only today I detached half a dozen new entries from the black hole again.

The black hole is currently at http://www.librarything.com/work/10400211

(it has changed since the most recent reference in the last thread.)

Hi! I was about to post an issue about this to "Bug Tracking" but could not identify the books:

http://www.librarything.com/work/details/57219864
is one of these books
Entry date: 2010-03-01
Data source: Koninklijke Bibliotheek
also http://www.librarything.com/work/book/57219842
http://www.librarything.com/work/book/57219741

I realised that import is broken at least from that data source and stoped.

For some ureason unknown to me the "black hole" is in series Series: /dev/null.

Because of http://www.librarything.com/topic/100181 one should change the CK and rename the series name to "AAAA - black hole LT error topic (whichever)"

P.S. fixed html (again)
note: If you refer to works you might use a "details" link with the WORK_NUMBER removed. This might be an "exploit" but the syntax is compliant with actual "addbooks".

notes:

a) Just woundered if this also a CK black hole (eating CK) . Today the link to history is : I can not see any suspicious actions there. (It could be a dizaster if CK would disapear because of combining).

b) Maybe one should ask users to delete the books if the records do not contain any valuable informaton. Should tags be deleted first?
I suspect that if the user records get "reused / recycled" they might be combined and lots of subsequent errors could happen.

c) Unfortunatelly "Members" does not show how many "copies" are used by the members. I think that it neither fraud nor mallicious actions are the cause. It just happened that parts of the systems involved changed, where broken ...

>6 gangleri:

d) Looking at the "reviews" you may find:
English (112) Spanish (2) Yiddish (1) Dutch (1) All languages (116)
Just noticed that the one in Yiddish is not of much help ...
Question: Is this a "review eater" as well?

6,7>

Yes and no. It's not "eating" anything, but the CK and reviews that the users enter for their individual copies of no-title and no-author books remain with the overall no-title no-author work.

Maybe one should ask users to delete the books if the records do not contain any valuable informaton. Should tags be deleted first?

Strictly forbidden. We can suggest to the users that blank books may not be what they intended, but we are absolutely not allowed to ask them to do anything.

The (?) black hole work now resides at http://www.librarything.com/work/10532054 and is still accreting new material.

The black hole book really shouldn't have a canonical title entered in it at all, certainly not asking people to fill the details in. It's not what the Canonical Title field is for. Applying Tim's logic from the 'zero copy editions' combination/separation debate, if it wasn't acceptable there to subvert a process/field to carry out a different purpose than it was intended for then subverting the Canonical Title field here isn't right. Creating a fake series is even worse. That's not to say the original problem shouldn't be fixed but misusing LT fields isn't the way to do it. Same issues applied with creating fake SPAM works etc.

How would you recommend we should communicate the information to the plurality of users, including those of us who do not very often use the social networking features?

10>

Frankly, I don't really care. This work is garbage, it is polluting the site, and it is a grotesque travesty that it has been allowed to persist this long with essentially no interest from the staff. Since preventing these blank works from being imported is apparently quite difficult, adding a check afterwards to say "You have imported 20 works with no title and no author, do you want to inspect or delete them?" would seem appropriate, but no go. I'm more than happy to make off-label use of CK to try to alert users and staff that there is a problem. (Before the Canonical Title was set here, there were endless questions about why there was a blank work showing up on the "Popular Books" Zeitgeist. This immediately communicates the nature of the problem.)

I didn't do anything with a series here, so I don't know what that's about, and the only Common Knowledge I did with spam was before we had any way of flagging works. They're not really relevant here.

They're not really relevant here.

They're relevant as proof of principle because they were the two specific examples that Tim used as a proof of principle regarding misuse of fields.

The debate was longer but here's one of the pertinent Tim-quotes. If it's not OK to subvert the fields for these reasons, it's not for this.

"I don't think it makes sense to use title and series for things other than titles and series—for example to change the title to reflect that the work has no copies, or is spam. "

If you're OK with making 'off-label' use then fine. I'm not. It's 'off-label' and shouldn't happen. We must agree to disagree.

I'm not the only person who's grumbled about this misuse of the field so and I don't think people wondering about something is justification for subverting a field. It's an arbitrary decision as to what is 'subversion'. You think this is OK, I don't. Who's to say who is right or wrong. Yes, it immediately communicates the problem but that's not the function of the CT field. It has a specific function that is being misused. I disagree with you enough to post here and I've disagreed enough to delete the erroneous CT on more than one occasion but I had no intention of entering into an edit war with those who were determined to use the field that way. That doesn't mean that it shouldn't be noted on record that it's wrong and that people are misusing a field.

You're right, there should be a check afterwards. That would be great. If there isn't then I still don't think that using the CT field as a messaging service is correct.

I totally agree that it is subverting the purpose of a feature. I am, however, of the opinion that having 27800 items in the databases is not a minor matter, and somebody should do something about it. I am willing to wait a while when I see that something is happening. This fall, for example, Tim resolved at least one problem that kept adding to the hole. After that, I waited (probably too long) before I checked to find out that the numbers are increasing again.

CK is not a messaging service, I agree, again. But when I click the "all members that have this work" on the black hole, I get a blank list. No way to message others but the most recent additions.

Let the record show that I am misusing several features of LibraryThing.

When this came up in regards to zero copy works and spam works, Tim's basic attitude is "I'm against using the system in weird ways. Rather than do that, let me address WHY you're having to use it in weird ways."

http://www.librarything.com/topic/91708#1995365
(and onwards)

And he did address it. Up until that point, he didn't really seem to be aware WHY it was being used in off-label (I really like that term in regards to this, lorax) ways.

I'm not sure how he could possibly not be aware of the black hole work issue. He doesn't seem to be all the interested in addressing the "why" here. As such, I'd say this is the least worst way of addressing it.

14>

I don't think Tim did resolve the issue (assuming the one you're thinking of is Amazon returning an empty result, rather than no results, when it can't find the book.) He diagnosed it, but I don't remember him saying he'd fixed it. (If he did, could you please point me to where?)

15>

I'd say this is the least worst way of addressing it.

I think that's a fair assessment. Is it ideal? No. It's just better than anything else we can do.

16>

Your memory is strong. And his remark only referred to the Amazon entries; I see in recent arrivals several manually entered ones.

Well, to explode ~28k books should not be done without due thought.

Since the bug reporting was added, Tim seems to be addressing bugs with the easiest to fix being first. The easiest bugs seem to have been dealt with and he's on to the harder ones. Several days ago he fixed some problems with tags that took two days. I suspect fixing the black hole work will take longer than that.

18> But since he hasn't actually communicated anything about it, there's no telling if it's even in his plans for the next year.

Do you want me to communicate about things or work on them?

This problem is a problem. It really sucks that a whole bunch of books have no titles and this causes them to glom together. The effect is, frankly, extremely marginal to the use of the site in a way that many other bugs—and features—are not. It need to be fixed, but there's no real fix conceivable here other than deleting all the books and preventing future books from being added.

That's me, by the way. Testing feature.

>18 jjwilson61:

For what it's worth, I've been fixing of various types. Certainly I've done some easier ones, but also some quite difficult one. It's practically all I've been doing since this debuted.

Do you want me to communicate about things or work on them?

Bit of a false choice, isn't it? Doesn't really take long to just say "I'm working on it, but no idea when it will be addressed." A simple note here setting it to Deferred would have even sufficed:
http://www.librarything.com/topic/99386

But if you don't want to communicate, that's fine. The main gist of my post was that if you had a problem with the Canonical Title being set for the black hole work, you would have probably said something about it by now. That's what you did with the zero copy/spam stuff. Since you haven't, I think it's fine for LT users to take that as an okay to address it in this way, the only way they have available to them.

ETA: And yes, we know you've been fixing bugs of various types. And we really appreciate it. Well, I know *I* do. The situation has improved greatly. But you know us users, always wanting more, more, more!

Yeah, I'm torn. People want me to mark things as "seen" or something. Maybe people would want me to say it as a post. Frankly, what the bug reports have allowed me to do is to be "on top" of every bug. I've seen every one, many times. I regularly click through the whole list into tabs, and look for ones that go together or otherwise cry out at the time I do it.

Sorry for the mood :)

23> I think the "seen" status has some merit to it, if only as public relations. Maybe you could get Talk to automatically mark it as seen if it's flagged as having been read by your username. ;)

As far as moods - that's what booze is for! It's like a liquid bugfix for post programming stress disorder.

23> I think if there were a "seen" status, or if you assigned the bugs when you looked at them (you can reassign them later, after all), it would reassure a lot of us, and not take away a lot of time from your working on things (and I completely sympathize with not wanting to spend all your time documenting what you're working on, rather than actually working on it). When a bug has the last status change still as "Reported" more than two weeks ago, and no posts from staff, though, I think it's fair to be unsure whether it's been looked at.

Thanks for stopping by.

Okay. I'll look into doing that.

Actually I'm a +1 for a quick "seen" as well. Or acknowledged or "Yup, thanks" just so it;s known that some one from staff has at least noticed the issue.

I know with most of the bug tracking systems I've seen and/ or used, that or something like it is built in.

Just at least something in passing to acknowledge that someone is paying attention.

And maybe a quick scan to see if additional information is needed, the report makes sense, etc.

I like the idea of making "Assigned" a status, it doesn't have to be as the result of an in-depth investigation, just a quick judgment (as lorax said, you can reassign them later) which would indicate acknowledgment that someone on the staff has seen the report. Tim could make it a goal to completely clear out the Reported bugs every day... except when he's off to Disneyland, of course.

Assigned and acknowledged are two separate things though.

Better?

Didn't mean anything about what you had said. Just that assigned would be the next step after acknowledged. Assigned means someone's responsible for the ticket now. acknowledged just means at least someone's seen it.

Other bug tracking systems are using a "Confirmed" (by other users; not necessarily from stuff). This would help here as well. Then one should wait for the "acknowledged" and "assigned".

... and hope for the "fixed".

I don't think wants to add too many statuses and it seems reasonable to fold Acknowledged into Assigned. Acknowledged could mean that someone has read it over once but not thought about it all, which isn't really very useful to know.

Bump. Any progress on this?

Since we've been told reposting of old and ignored bugs is not only inappropriate but actually spam -- worse, apparently, than authors posting in dozens of groups to try to sell their books, which is merely "enthusiasm" -- I am bumping this. It's data loss, albeit at a very early stage (on import). We were told, once, that LT took data loss seriously. Tim says this is marginal, but I disagree strongly; it's a cataloging bug, which is surely central to the site.

35> Since we've been told reposting of old and ignored bugs is not only inappropriate but actually spam -- worse, apparently, than authors posting in dozens of groups to try to sell their books, which is merely "enthusiasm"

Tim must have REALLY edited his post between the time you read it and I did. I don't remember most of that...

36>

When I read it, he said "This is bug spam". He's said elsewhere that author self-promotion is not spam. I guess he has never explicitly said in so many words that spam is bad; I filled that part in myself.

37> Spam is a term that has a lot of different meanings. Context is king. Would you be less inclined to argue if he'd said "This bug is a misuse of site features"?

38>

Would you be less inclined to argue if he'd said "This bug is a misuse of site features"?

Yes.

I don't really see this as data loss. An API change made certain imports fail, producing blank books. There is absolutely no way to invent data for these books.

The fix here is to spend time finding and deleting them all, and them making sure those deletes cascade through other parts of the site. I think this needs to be done, but it's not at the top of my stack of things to do.

40>

Thanks for the response. Reading between the lines, do I correctly understand that there aren't more books that will slip between the cracks due to this bug? That's the bigger issue than the already-lost books, as far as I'm concerned, and if you've plugged the leak that's great news.

OT - And good riddance to cielacecha1976, you get what you deserve if you're stupid enough to post spam directly after the founder.

Tim, may I direct your attention here? There's lots more where he came from.

/OT

42> You don't need to read between the, post #1 of this thread says that the problem of adding no title, no author books has been fixed (or at least that one cause of it).

44>

That's not how I read it, then or now; I read it as the problem having been diagnosed but not fixed at the time.

So, it depends on the meaning of "problem." The problem by which Amazon changed their API and it caused us to add blank books is fixed. The books, however, are still in the system. That's not been fixed.

46>

Thanks. The books in the system are a side-effect of the bug, as far as I'm concerned; if there aren't more being added, that's fantastic.

Update on why this isn't fixed yet (see http://www.librarything.com/topic/107331).

See elsewhere.

So, should one of them be closed?

Closing. We have more recent exampls.