Very old problem with LCC sort.

While going through my catalog to fix up physical description data and other authors I was reminded of a problem with LCC sorting that dates back to Tim's introduction of true LCC sorting in 2007. I'll simply repost my initial message on the problem as nothing seems to have changed.

---

The first number can have a decimal place (E185.6). Numbers with no decimal place (E185) should come before numbers with a decimal place, instead they are coming at the end.

However, sorting within the decimals is working (E185.8 is before E185.86 is before E185.9).

Assigning to ChrisC.

Still true.

Thanks for confirming its continued existence!

I've swapped some code out: could you take a look and see how it's sorting now?

Looks like you didn't fix the main problem but you managed to break something else.

For the main bug see:
http://www.librarything.com/catalog.php?searchall=1&deepsearch=lcclassificat...

Go to page 2 and scroll to the bottom. The Black Image in the White Mind, E185.F836, is still sorting after Ida: A Sword among Lions, E185.97.W55G53 2008.

The new problem has to do with an old kludge I had used years ago before LCC sort was introduced. It was possible at that time to force a correct sort by adding extra spaces to the LCC numbers. The code was changed to not recognize the spaces, several of us complained and Tim introduced a real LCC sort that worked without the spaces (except for the bug noted in this thread).

Since then, my LCCs with the old extra spaces and newly added books without the kludge both sorted properly together. So I never bothered with the laborious task of taking the spaces out. Now, whatever you've done has made all of the LCCs with the spaces sort before the ones without the spaces.

6> Oh dear. I will have a another look now.

6> Ok, would you please take another look? The code should be stripping out those old spaces now so both call numbers with those spaces and those without should be treated equally for sorting.

9> Thanks. I will investigate further.

OK, so you've undone the new problem and my old kludged LCCs are back to sorting together with the newer unkludged LCCs. Thank you.

But the original bug still exists. If you look at the second page of my link in message 6, all of the E185s without a decimal (beginning with A Documentary History of The Negro People in the United States, E 185 .A58 1951) are still sorting after the last E185 with a decimal (Subversive Southerner, E 185.98 .B73 F67 2002).

11> Thanks, yes, so now no worse than before but no better either. I'll circle back to the original problem.

Alright, I've tweaked the code again, so take a look once more, once?

Hmmm.... nope.

The sort is now all mixed up on the first numerical section. I can't really tell what it's doing. In addition to the links above, take a look at the HQs:
http://www.librarything.com/catalog/eromsted&deepsearch=lcclassification%3AH...

I've got 1064-1090 then 503-504, 801-809, 1101, 1206-1208, 1507, 1904, 470, 770, 1150-1190, 12, 1240, etc.

Not sure what happened, by my LCC sort got out of whack recently.
Simple for instance:
BF637.P4F6
BJ1601.H65 2011
BJ1241.G6 2013
BJ1461.F5
BJ1461.L8

Same issue has occurred throughout my library. Help?

Ok, i'm reverting that change then and let's go back to the way it was before this morning.

Sorry, all, I agree: let's defer this for now. Clearly I need yet more test cases in my test file.

If you all want to throw some examples of correct LCCN sequences in here, I can add them to the test file and come back to this at some later point, with more examples to throw at it.

First, I am not a librarian. My knowledge of LCC comes from use, not training.

The LCC has several sections and they do not all sort the same way. I don't know how the current sort algorithm is handling that, but it seems to work for everything except the first numerical section.

Directly preceding the first numerical section will be one or two letters. Directly following will be a period followed by a letter.

The first numeric section is an ordinary number that can run from the ones to the thousands. It may have no decimal or a decimal of any(?) length, though I don't think I've seen more than 3 or four decimal digits.

The first numeric section, including the decimals, should sort as ordinary numbers. So 1, 3, 7, 12, 12.5, 12.56, 12.7, 30, 67, 90, 100, 101.3, 101.33, 101.332, 101.3321, 101.3328, 101.3341, 101.35, 101.7, 102, 345, 547, 678, 800, 1109, 5498, 9934, etc.

The problem identified in this thread is that it will currently sort 503.4, 503.48, 503.9, 503 instead of 503, 503.4, 503.48, 503.9.

*edited for a bit of clarity*

I'll add, my suspicion is that there is some confusion between the period that acts as a decimal place in the first numeric section when there is a decimal place in that section and the period that separates the first numeric section from the following letter section.

But, of course, I haven't seen the code, so I don't know.

Here are some examples of the sort problems I'm currently experiencing. (I originally posted it on another thread but someone directed me here).

At first it looks like it is being sorted alphabetically thus BL487 occurs before BL51 (which is incorrect):

BL245 .P45 1993
BL31 .C75 1998
BL310 .F73 1951
BL325 .C7 .C64 1999
BL41 .P69 1996
BL48 .S33 1958
BL48 .W34 1998
BL487 .K43 1983
BL51 .H463 2001
BL60 .H265 2001
BL60 .W4313 1963
BL782 .B83 1985
BL795 .S23 D4713 1989
BL80.2 I434 2003
BL80.3 .O92 2010
BL802 .B43 1998
BL802 .F45 1987

But the problem gets worse. In this next set BS2300 and BS2800 books occur between BS191 and BS1555. Plus, the books with decimals occur out of order (e.g., BS1515.2 before BS1515), but this issue I've noticed for awhile. Also notice the volumes numbers for BS2312 are out of order. There are so many issues within this small sample that I'm not sure it's even sorting anything correctly at the moment.

BS191 .A1 2001
BS191 .A1 2006
BS191 .C58 2010
BS191 .K55 1611
BS2302 .M68 1975
BS2312 .A86 2003
BS2312 .I87 2000 V.2
BS2312 .I87 2000 V.1
BS2312 .I87 2000 V.3
BS2312 .I87 2000 V.4
BS2332 .R5313 1988
BS2362 .L44 2009
BS2832 .S3 1965 V.2
BS2832 .S3 1991 V.1
BS1555.2 .H37 1978
BS1555.2 .P61 1965
BS1555 .C65 1993
BS1555 .N49 1991
BS1515.2 .W38 1985 V.1
BS1515.2 .W38 1985 V.2
BS1515 .M29 1968
BS1525 .B75 1965
BS1525 .H65 1986 V.1
BS1525 .H65 1986 V.2
BS1545.3 .A45 1990
BS1545 .Z5613 1979 V.1
BS1545 .Z5613 1979 V.2

>20 brandonw: Do you think this sorting problem is new, i.e., it's cropped up since I started messing with the code in the last couple of days? (In which case we maybe should go back to the way it worked before I started messing with it).

>21 ccatalfo:
I assumed that after your message 17 you were reverting to the way it worked before and I didn't check again. Looking now, the sort appears both different from when I reported in message 14 and still seriously wrong.

So yes, reverting to the status quo ante would be a good idea. The LCC sort used to be almost right. Now it's not right at all.

I suspect this is a solved problem - a quick google turns up code with regexps to normalize LC numbers to string-sortable format. They're in perl and python, but converting them to your language of choice is probably easier than trying to reproduce the same thing from scratch.

Yes, reverting to the code from a few days ago would be definitely better than its current state. The previous code worked with the exception of the decimals (e.g., PA1235.5).

>24 lorax: Indeed it is, we actually have python code that does the same, need to figure out how the php and python versions are differing.

>25 brandonw: Ok, I've reverted to the way the code was at the start of this thread before i so gallantly screwed it up.

I shall slink back to my cave and study the code some more.

Thanks for the testing and patience.

We're working on it today.

Okay, we have it working in theory. It's currently recalcualting a lot of numbers, and will therefore be AMAZINGLY wrong until that's finished.

But we have a question: Should non-validating LCCs be grouped at the front of the list or the end? At present it's going at the front, but I think it should go at the end.

>29 timspalding: I'm not sure what qualifies as non-validating. But if it's what I think it is (i.e. not real LCC format call numbers), then putting them at the front is the way to get folks to notice them. Putting them at the end, they're likely to be missed.

So, part of the concern is that putting all the invalid stuff at the beginning just looks bad. You click to sort by LCC and it starts with stuff that isn't sorting intelligently. Or, indeed, it starts with nothings, then stuff like \No book in LC\, etc.

Fixed. See https://www.librarything.com/topic/252285

Thanks for all the help!