Autofill rebuild issue

[dagopher]

I'm going through and cleaning up my data in Bookpedia (5.1.0), specifically the genre data. I've gone into my Library, limited which Genre's are actually in use, then I go to Preferences > Autofill > Genre > Rebuild and find that I've got a ton of genre entries still listed that do not exist in my database anywhere I can find. And I can't get rid of them. I've tried closing Bookpedia and deleting autofill.xml but it always gets regenerated the exact same way with the same non-existant genres.

As a test, I did an export of my current data, and then imported it to a clean copy Bookpedia. In this case, the genre data stayed clean! So I believe there's some type of garbage data stored in my old database file. While I can go through and do an export/import and not lose the raw data, so far I haven't found a way to transfer such things as smart folders, of which I have a lot of. I should note that I've gone through multiple upgrades of Bookpedia (I think I started back on 3.x) so there's probably some legacy cruft in my current database file. (As a seasoned Linux Admin, I have a theory that no matter how careful or clean your data is, cruft just grows like fungus in your datastore.... but that's for another discussion ;)

So before I go through do an export/import to a clean DB and then manually re-create all my smart folders, do you have any recommendations here? What I would really like to be able to do is do a complete dump of my database: raw data, smart folder configurations, and what ever else would be appropriate. (I really need to write a SQLite data dumper for myself...)

BTW, I think my issue appears to be related to this post: viewtopic.php?f=5&t=3443

thx
Gopher.

[dagopher]

Well, I figured out at least what's going on. (Finally went searching for a SQLite browser and found one; digging through the raw data is really helpful.)

I finally found those oddball genre entries, they are coming from my Doghouse collections. So it seems as I get data to review/approve for the Doghouse that data is infecting my autofill lists.

As a second thought, I went back to my data export I tried earlier and found that none of the data from Doghouse collections appeared in the output, so it looks like you have some way to filter out the Doghouse data for the exports, can you apply the same logic to the autofill rebuild?

[Conor]

This is the kind of issue I can deal with. When it's broken down and the bug already found that is 90% of the work done. I did the other 10% and the Bookpedia Beta is up with the fix: Doghouse moderator collection no longer included in the autofill rebuild. I apologize I overlooked that feature when adding the new Doghouse collections to the program.

[dagopher]

Well, we're on the right track. I just played with the Beta and it mostly works.

My genre entries cleaned up when I ran 'rebuild' this time, except for two entries which both lived in one of the various Doghouse collections. ...I don't understand it either. So something is sneaking through the filters somewhere. (And I've confirmed via raw SQLite data, that those two records are the only ones for those unwanted genre's and they only live in the DH collections.)

I have not made any changes to my current set of Doghouse entries, so there should have been no reason (as far as I'm aware) for Bookpedia to go fetch new entries from the DH. Based upon the assumption that no DH entries have been added on my end, I don't see any reason why your fix should have over looked these two records.

thx
Gopher.

[Conor]

Should most definitely not include those two entries. Please send me your database file "Database.dvdpd" and let me know the two entries that were included, so I can try it out with the same data. As a fix you could try deleting the moderator collections and bring them back with opening the Doghouse->Settings when you need them.

[dagopher]

Good news, your Beta functions correctly; your devoted user.... no so much. Turns out I was missing some Genre entries hiding in.... the Wish List. Once I found those and fixed them, the autofill rebuild for Genre worked 100%. I even checked with and with out my Doghouse collections; everything still worked correctly.

(I have to admit I'm tossed on this, part of me thinks Wish List items should be in the autofill lists, and part of me doesn't. When you're working under an assumption like I was that all data is available through the Library, you can - and I did - miss data when you search for it. But, on the other hand, items in the wish list will probably end up in the Library one day, so it makes sense to be able to massage that data just like you would an entry in the Library. Hrm...)

So, do I get a dog biscuit or something for doing most of the foot work on this bug? ;)

thx
Gopher.

[Conor]

The wish list and other excluded collections are special in that depending on what approach one takes, they should be include or not in certain features of the program. But in this case it's best to err on the side of being included as it's easier to clean up an included genre than it is to go and have to look up manually for those that do want to check their wish list. I myself get obsessive compulsive and try to fix every single last detail. But through the years of programming I have learned to tolerate the edge cases as there is always going to be one.

As a dog biscuit you get to do it all over again, but now with our data as a moderator in the Doghouse.