FancyIndex search performance

[jschaffe]

Is there anything that can be done to improve FancyIndex search performance?

I have several lists based on my CDpedia library. Here's my main page: http://web.me.com/jschaffe/Music_Library/Welcome.html.

The largest list is the "Music Library (all)" link, which contains 4,127 albums. Entering something in the search box gets me the "Slow script" message something like 5 or 6 times before the search completes.

The search performance seems to be exponential relative to the size of the list. Another large list, "Freebies" (2,126 albums), which is just over half the size of the largest one, brings up the message just one time.

I'm not sure if there's anything I can do, or even if anything can be done in the template, but I thought I'd ask.

[Conor]

FancyIndex uses a Javascript library called JQuery that is very popular and a number of smart people have worked on it. So although Javascript and HTML was not built for this kind of sorting. Today's fast computers make it possible that we can do cool things such as sort a table in HTML in realtime, but truly large data sets need formats that where designed for sorting and searching (they have an index that reduced the number of comparisons). Most websites out there would use SQL, which DVDpedia uses as well. Even then sorting is an expensive operation and it's the reason websites present you with 10-50 results for searches instead of 1000+. Also the number of compares it has to do grows exponentially with the number of items in a list. When you have 2 items you have to do a minimum of 1 comparisons for a sort, at 3 items 4, with 4 items 5 comparisons, 5 is 7,
6 = 10
7 = 13
8 = 16 //Best possible outcome
...
2000 = 4,000,000 Approximations of comparisons, at 4 million comparisons you receive one warning about the script being slow.
2001 = 4,004,001
...
4000 = 16,000,000 You doubled your items but it take 4 times the number of comparisons and you receive 4 warnings about being a slow script.

Although I got carried away with the explanation for sorting similar math applies to the searches as well. However for searches you can cut down on the columns searched if you know that you will always be searching by album or artist, you could exclude the genre column to save some time. You would need to look at the fi.sharred.js file in the Images folder FancyIndex and on the line below you would make changes to limit the columns searched:

Code: Select all

if ( (this.textContent || this.innerText || "").toLowerCase().indexOf( searchString ) === -1 ) {

[jschaffe]

... Also the number of compares it has to do grows exponentially with the number of items in a list...

Oh my does that take me back!

I was a consultant years ago and was brought in to help a team was taking between 30 and 100 hours to sort a file of a couple thousand items using a program written in Basic. They read through the file remembering the highest valued record, copied the whole list to another file with the highest value last. Then they did it again for the second highest value. And so on. This would have been...1980 or so, so I don't remember the details but I was able to develop some kind of a highly modified bubble or shell sort to accommodate the constraints of Basic and get it down to 30 minutes or so. Those were the days, when a better sort was something fun and new!

But unlike sorting, I expect the computations involved in searching should be fewer, assuming there's an organized index to search against. So what actually has to happen when a user enters a search term in FancyIndex? Does JQuery have to sort before it can search?

I've done some tricky coding in the past, but I really haven't learned much about HTML, JavaScript, or the JQuery library. I have no idea how to change that line of code to exclude the genre column. However, since you indicated that this sort of thing is possible, would it also be possible to have separate search fields for artist, album/track title and Genre?

[Conor]

Those were the days, when a better sort was something fun and new!

Now adays if your not using somebody else's implementation your just being stupid. In DVDpedia we let Apple do our sorting and they let SQLite do it.

But unlike sorting, I expect the computations involved in searching should be fewer.

It would indeed be fewer, I just got carried away with the sorting description when describing why it would get slower. Because it's not a prefix search, but can match any part of a title, there are still quite a number of comparisons to do.

You could indeed separate the search fields into different searches. I have never used JQuery myself, but I know it's meant to help you find items within the HTML. Adding a id to each column in the template would allow you to write different functions that search only a specfied column. I also wish to learn some JQuery as it could be helpful in redesgining the Bruji webpage, but at the moment I am to deep in Cocoa to learn a new framework.

[marumari]

That is quite unusual. When I work on FancyIndex, I test with a database containing over 10000 rows and have acceptable search performance. For some reason, the export you have there uses an old version of the Javascript that is included with the FancyIndex template. It is much, much slower at searching, by at least an order of magnitude. How recent is that export?

If you don't want to do a re-export, you can always grab the two Javascript files from here:
http://www.twoevils.org/html/mycds/Images/fi.shared.js
http://www.twoevils.org/html/mycds/Images/fi.js

And use those. That should speed things up considerably. When I looked at an export from my most recent version of CDpedia, it had the most recent JavaScript file, so perhaps you are using a very old version of CDpedia to do the export?

[marumari]

BTW Conor, I don't check the forum probably as often as I could. If issues come up with FancyIndex in the future and I seem to be absent, feel free to shoot me an email and I'll hop on to take a look at it.

[Conor]

I never thought to check if the JavaScript file was the latest, since I know we did a lot of optimizing and work in the early days of Fancy Index. I didn't want to bother you with any issues, but thank you for the offer to look at an issues that is fabulous support for a free template.

[marumari]

Just trying to be a good citizen! Without your guy's awesome software to begin with, I wouldn't have the ability to create such a cool template.

I'm just happy to see that it is useful enough to become so popular. I occasionally will do Google searches just to look at people's fun movie, music, game, and book collections, and that's pretty satisfying.

[jschaffe]

How recent is that export?

I updated CDpedia in mid January and did all the exports around the end of the month, so they were as recent as I was able to get. I'll take a look at changing the JavaScript files when I next get a chance.

[jschaffe]

I updated CDpedia in mid January and did all the exports around the end of the month, so they were as recent as I was able to get. I'll take a look at changing the JavaScript files when I next get a chance.

I manually replaced the JavaScript files on all my CDpedia FancyIndex pages, and what a difference!

OK, so now the question is, how did the old JavaScript get there? Well, I found the problem, and sorry Conor, but it came from you!

Do you remember the thread where I was having trouble customizing the FancyIndex template? http://www.bruji.com/forum/viewtopic.php?f=7&t=4088&p=16812&hilit=jschaffe#p16812 In order to get me over my fumbled attempts to edit, you modified FancyIndex for me to create FancyArtistIndex. I duplicated the folder I got from you and was eventually able to successfully make additional changes to the template.

Unfortunately the template folder you sent included the old JavaScript files, and they followed along when I duplicated your folder. I checked the custom template folder created directly from FancyIndex and that folder contains the correct version of the files, so the CDpedia update worked correctly.

So I've replaced the older files in my customized templates folder and I should now be able to publish with impunity!

Anyway thanks for the help both on modifying the template and getting to the bottom of the problem.

And thanks for a great product and to marumari for a great template!

[Conor]

Fantastic that Marumari had optimized the original script and speed was never an issue, except for my mistake in sending you the wrong file. I had checked the CDpedia templates to make sure that the scripts where up to date, but I did not check that special customized version I made for you a while back. I must have started from an old copy of the template, I will make sure to remove any copies to avoid confusion in the future.