Problems loading new Kobo Clara from Calibre

[Marg1780]

I just purchased a second Kobo Clara eReader and am trying to get them both working with Calibre on my Windows 10 laptop. They were purchased about a year apart, so their firmware/software versions are different.

I was running an older version of Calibre, around 5.17, I think. (I only update when a few versions have gone by because of the need to manually re-install some of my plug-ins.) When I tried to load the new Clara, I got an error page that began:

"Your Kobo is running an updated firmware/database version. As calibre does not know about this updated firmware, database editing is disabled, to prevent corruption. You can still send books to your Kobo with calibre, but deleting books and managing collections is disabled...."

I loaded a couple of books, just to see what would happen. They were perfectly readable but the Clara didn't recognize the series name/number as either a series or a collection name.

So, I updated to the current Calibre version (5.33.2) and got the plugins sorted out. Then I reset the new Clara to factory settings, set it up again using the Kobo app, then tried again to load a file to the new Clara. No error this time - a forward step - but the reader still isn't processing the series data. The filename shows as it always did "series name - 01 - title" but according to the Clara, it isn't either a series or a collection.

Is the new Calibre version somehow remembering that this function was turned off? (Am I going to encounter the same issue if I connect my older Clara? Right now I'm rather afraid to do that because that unit has several hundred books on it and I'd really hate to do a factory reset....) I have close and re-opened Calibre and tried again, to the same result.

There is one book on the reader which was downloaded by the Kobo app when I was installing, in case that is somehow screwing things up. (That did not happen when I installed my first Clara. I don't recall doing anything differently, so I expect that the default setting to "auto sync" has changed between then and now. ) It shows up as belonging to a series, so the function is working to some degree within the Clara itself. Just not with the book I'm sideloading from Calibre.

[Marg1780]

Addendum: This now is a problem with both Claras. When I loaded a new book onto my original reader, the book shows up as normal but its series is not displayed in the "Collections" tab. So I've "broken" the feature within Calibre itself, and don't have a clue how to turn it back on. Any and all help vastly appreciated!

[DNSB]

Since Kobo ereaders can display the series as part of the library listing, generating collections for series is pretty much a dead issue. You do need to reconnect your Clara after the books have been imported to have the series information sent for it to show in the library view.

[Marg1780]

Hi David, Thanks for replying, though I'm confused by your comment.

Is this some very recent change in Calibre's behavior? And if so, do I need to do anything additionally to activate it, or is it just sufficient to connect the Clara a second time? (Which would be a minor, but usable pain.)

I've been using Calibre with my older Clara for a year, and haven't seen (or at least haven't noticed) this behavior. I push the books over to the device, eject it, let it sort out the new data and collections/series are available to use. I've created a few special-purpose collections manually and added books to them myself, but newly added books have always found their own ways into an auto-generated collection using their series name as defined in the Calibre plug-board definition for the Kobo driver. I see with the new Clara version they have separate tabs for "collections" and "series" but on the older one, they use the term "collection" for both automatic and manual groupings.

I agree that they aren't an absolute necessity since anything can be found by author, but especially for shared worlds or multi-author series, they're a great convenience. (And a great help to my aging and flaky memory. Who wrote the third book in the "Black Trillium" set?... um... er - let me just go find it through its "collection" page.)

[DNSB]

Originally Posted by Marg1780

Hi David, Thanks for replying, though I'm confused by your comment.

Is this some very recent change in Calibre's behavior? And if so, do I need to do anything additionally to activate it, or is it just sufficient to connect the Clara a second time? (Which would be a minor, but usable pain.)

I've been using Calibre with my older Clara for a year, and haven't seen (or at least haven't noticed) this behavior. I push the books over to the device, eject it, let it sort out the new data and collections/series are available to use. I've created a few special-purpose collections manually and added books to them myself, but newly added books have always found their own ways into an auto-generated collection using their series name as defined in the Calibre plug-board definition for the Kobo driver. I see with the new Clara version they have separate tabs for "collections" and "series" but on the older one, they use the term "collection" for both automatic and manual groupings.

I agree that they aren't an absolute necessity since anything can be found by author, but especially for shared worlds or multi-author series, they're a great convenience. (And a great help to my aging and flaky memory. Who wrote the third book in the "Black Trillium" set?... um... er - let me just go find it through its "collection" page.)

You can also search by series if that helps and there is a Series display tab in Books.

Check that in Preferences => Sending books to devices, that Metadata Management is set to Automatic Management so metadata will be sent each time your Kobo is connected. In the Device configuration, on the Metadata, on device & advanced tab, check that Set series information is checked. I also have Update Metadata on Book Details pages checked. On that page, you can also check Attempt to support newer firmware which will suppress the original error message you saw. If there are going to be any issues with that option, you will likely read about on MobileRead before the firmware hits general availability.

[davidfor]

Originally Posted by Marg1780

Hi David, Thanks for replying, though I'm confused by your comment.

Is this some very recent change in Calibre's behavior? And if so, do I need to do anything additionally to activate it, or is it just sufficient to connect the Clara a second time? (Which would be a minor, but usable pain.)

There has been no change in the behaviour of how collections and series information is handled since setting the series info was added to the KoboTouch driver. Which is at least five years ago and probably something like eight years. Collections can be created when a book is sent to the device, but, the series info can only be set in the database after the book has been imported by the device.

I've been using Calibre with my older Clara for a year, and haven't seen (or at least haven't noticed) this behavior. I push the books over to the device, eject it, let it sort out the new data and collections/series are available to use. I've created a few special-purpose collections manually and added books to them myself, but newly added books have always found their own ways into an auto-generated collection using their series name as defined in the Calibre plug-board definition for the Kobo driver. I see with the new Clara version they have separate tabs for "collections" and "series" but on the older one, they use the term "collection" for both automatic and manual groupings.

I agree that they aren't an absolute necessity since anything can be found by author, but especially for shared worlds or multi-author series, they're a great convenience. (And a great help to my aging and flaky memory. Who wrote the third book in the "Black Trillium" set?... um... er - let me just go find it through its "collection" page.)

The series tab was added at the start if 2020 in firmware 4.20.14601. It is completely separate to the collections. Calibre will populate this when you connect the device again after the books have been imported.

Other than that, @DNSB has probably mentioned all the configuration to check. If it is correct and the collections and series info is still not updating, run calibre in debug mode and post the log for me to look at.

[theducks]

Check your Kobo Utilities configuration
Did you happen to tick Configure options for each device by accident?

[Marg1780]

Originally Posted by DNSB

You can also search by series if that helps and there is a Series display tab in Books.

Check that in Preferences => Sending books to devices, that Metadata Management is set to Automatic Management so metadata will be sent each time your Kobo is connected. In the Device configuration, on the Metadata, on device & advanced tab, check that Set series information is checked. I also have Update Metadata on Book Details pages checked.

The newer firmware has the series tab (along with books, authors and collection), but on my older device, I only see books, authors and collections. That was never a problem since whenever I added books, a collection was created automatically using the name of the series.

Unfortunately, since I didn't need to do any device configuration for my older reader, I'd never been down into the configuration menu and didn't realize it existed. (There are a *lot* of menus in Calibre.) So I have no notes on what settings I had before but it was set to "Manual" when I went in tonight so I changed it to automatic. The "Set series" box was already checked.

Plugging the Kobo in/out a few times tonight, trying to understand those options, I do seem to be experiencing what you mentioned - my books only get added to a "series" category the next time I connect to calibre, not when I'm pushing the ePub over to the device. I'm now wondering if I could have managed not to notice this behavior for an entire year of regular usage... It seems unlikely but of course isn't impossible. Whenever I added something to read immediately, it would have been at the top of the "recents" sort so I wouldn't need to go anywhere else.

Do you know if there is any guidance for eReader configuration in a centralized location somewhere? When I did my first reader, I started with a search of the Calibre documention, of course, couldn't find anything, and eventually tracked down the plugboard information in an article on a different website. But that article had nothing else relating to device set-up. Everything worked after that, so I just heaved a sigh of relief and started reading.

[Marg1780]

Originally Posted by davidfor

There has been no change in the behaviour of how collections and series information is handled since setting the series info was added to the KoboTouch driver. Which is at least five years ago and probably something like eight years.

Okay, so apparently I *could* use the device regularly for a year and not consciously notice that behavior. Thanks for the detailed information.

[davidfor]

Originally Posted by Marg1780

The newer firmware has the series tab (along with books, authors and collection), but on my older device, I only see books, authors and collections. That was never a problem since whenever I added books, a collection was created automatically using the name of the series.

Unfortunately, since I didn't need to do any device configuration for my older reader, I'd never been down into the configuration menu and didn't realize it existed. (There are a *lot* of menus in Calibre.) So I have no notes on what settings I had before but it was set to "Manual" when I went in tonight so I changed it to automatic. The "Set series" box was already checked.

Well, you must have. By default, the collection management is not configured. If calibre is creating any collections on the device, you must have done this at some time in the past. Setting the series info is turned on by default.

Plugging the Kobo in/out a few times tonight, trying to understand those options, I do seem to be experiencing what you mentioned - my books only get added to a "series" category the next time I connect to calibre, not when I'm pushing the ePub over to the device. I'm now wondering if I could have managed not to notice this behavior for an entire year of regular usage... It seems unlikely but of course isn't impossible. Whenever I added something to read immediately, it would have been at the top of the "recents" sort so I wouldn't need to go anywhere else.

Based on comments made about this, most people don't notice this need. Your report of issues after years of use and then surprise at the need to connect again is very common. The main difference is really in how much they swear they never had to do this before. And I can guarantee, that, for the series information, it has always been like that.

Do you know if there is any guidance for eReader configuration in a centralized location somewhere? When I did my first reader, I started with a search of the Calibre documention, of course, couldn't find anything, and eventually tracked down the plugboard information in an article on a different website. But that article had nothing else relating to device set-up. Everything worked after that, so I just heaved a sigh of relief and started reading.

The closest thing to what you want is https://wiki.mobileread.com/wiki/Kob...nd_Collections. There has been a little bit of argument over exactly how to present this, so there are a couple of versions in different styles on that page. And it hasn't been updated perfectly as things in either the driver or firmware have changed.

[Marg1780]

Originally Posted by theducks

Check your Kobo Utilities configuration
Did you happen to tick Configure options for each device by accident?

I'm afraid I can't answer that one, because I've no idea where my "Kobo Utilities" configuration page is. On the eReader or somewhere in Calibre?

[davidfor]

Originally Posted by Marg1780

I'm afraid I can't answer that one, because I've no idea where my "Kobo Utilities" configuration page is. On the eReader or somewhere in Calibre?

The Kobo Utilities plugin is for calibre. It adds a lot of function that people like. Such as storing the reading status of books from the device, updating/cleaning up covers, some annotations related things, database backups and other things. It has some of the same restrictions in that it can only act on books already imported on the device. It is fairly common for people to mix up what is in the plugin and what is in the driver. The plugin has some configuration that can be device specific for people with more than one Kobo. But, in this case, it is all in the driver.

[Marg1780]

Originally Posted by davidfor

The Kobo Utilities plugin is for calibre.

Ah, okay, that explains why I couldn't track it down. Until I started the chain of events that came with buying a second reader, I was totally happy with the functionality I had so I think I'll not add something else new, since it's noncritical.

At this point, I seem to have things going okay on the new reader (now that I understand more about the updating chain of events), but still am not seeing the autogeneration of a collection (ie series) name for new books on the older reader, even if I plug into/unplug from Calibre a second (or third) time... It behaves like there might be a backwards compatibility issue between my updated Calibre and my not-updated Kobo, but the way things are going, I'm probably just hitting another pothole in lack of knowledge.

[davidfor]

Originally Posted by Marg1780

Ah, okay, that explains why I couldn't track it down. Until I started the chain of events that came with buying a second reader, I was totally happy with the functionality I had so I think I'll not add something else new, since it's noncritical.

It isn't necessary and until you see something that it does that you want to use, you can forget about it.

At this point, I seem to have things going okay on the new reader (now that I understand more about the updating chain of events), but still am not seeing the autogeneration of a collection (ie series) name for new books on the older reader, even if I plug into/unplug from Calibre a second (or third) time... It behaves like there might be a backwards compatibility issue between my updated Calibre and my not-updated Kobo, but the way things are going, I'm probably just hitting another pothole in lack of knowledge.

Collections should be created the same for both devices if they have the same books on them. The collection management is done for the books on the device that calibre can match to books in the library. If you are creating collections for series and one of the devices does not have any books in a particular series on it, then the collection will not be created. Similarly, if for some reason, calibre is not matching the books on the device and those in the library properly, then, again, it won't create the collections.

There is one trap with multiple devices and collections. Kobo syncs the collections between devices. But, it doesn't sync anything about sideloaded books. That means if you create a collection on one device with a sideloaded book in it, after syncing both devices, the collection will appear on the other device. But, it will be empty. The trap is that if you have the option "Delete empty bookshelves" on in the driver, the collection will get deleted from the device with the empty collection. When you sync, both devices, it get deleted from the other device, but, then calibre creates it again. And it goes around and around. For this reason, I turn that option off. My collections are fairly generic, so they don't tend to be empty.

[Marg1780]

Hi, and again thanks for your information and help.

Originally Posted by davidfor

Collections should be created the same for both devices if they have the same books on them. The collection management is done for the books on the device that calibre can match to books in the library. If you are creating collections for series and one of the devices does not have any books in a particular series on it, then the collection will not be created.

Any given book should be on only one of the readers. I'll probably load a dupe now and then by mistake, but I can't think of a situation where I'd do it intentionally.

Originally Posted by davidfor

There is one trap with multiple devices and collections. Kobo syncs the collections between devices. But, it doesn't sync anything about sideloaded books. That means if you create a collection on one device with a sideloaded book in it, after syncing both devices, the collection will appear on the other device. But, it will be empty. The trap is that if you have the option "Delete empty bookshelves" on in the driver, the collection will get deleted from the device with the empty collection.

Thanks for the empty bookshelves tip - I was pondering whether to turn that on, since I'd noticed that empties just hang around on the device until I notice and manually delete them.

The older eReader software/firmware only has collections, not series & collections as separate entities. Until a couple of days ago, when I loaded new books that had a series defined in Calibre, a collection was automatically built by Calibre using the series name. I had never noticed that this is a two-step process, as you explained, but it did work seamlessly.

And now... it doesn't, on my older reader at least. (I'll need to double-check the new one, though I think yesterday's advice got that one working - which again makes me wonder if it's a case of backward incompatibility... I don't see anything about a firmware cut-off level in the Calibre features list, but that doesn't preclude the possibility.)

As an experiment, I just added 3 books in a new series (SeriesA for brevity) and 2 additional books for a series which already existed on the eReader (SeriesB).

- When Calibre told me they were on the device, I ejected the eReader from Calibre, let it do its processing, then looked at the entries. As expected, each entry showed, for example

SeriesName - 03 - Title [filename format as set up in plugboard]
[the series/collection data is not yet added]
author name

- I reconnected to Calibre, let it do its job processing (which showed no errors), then ejected the eReader again. This time, the missing middle line was filled in

Series Name - 03 - Title [as before]
series name & number now filled in [as expected]
author name

But if I look in the Collections tab, no entry for SeriesA has been created, and the 2 new books are not showing up in the listing for SeriesB.

If I hook it up into Calibre again, and look at "Show book in the main memory of the device", it shows the same thing. For all 5 of the newly added books, the "collections" column is empty. But if I page down to check the series books which were added using Calibre v5.17 (or earlier), those fields contain the series name.

Believe me, I did not create several hundred series-name collections manually - Calibre did it for me transparently, but at v5.33.2 has ceased to do so. And man, I really hope there's just another unknown-to-me parameter that needs flipping, because I'm not looking forward to needing to do it all manually from now on.... (And there are SOOO many potential rabbitholes attached to the thought of trying to upgrade the reader firmware - starting with an image of it deciding that my many-many books are nothing but sideloads and so can be deleted on a whim. Assuming it's even possible to upgrade the reader's firmware without doing a full factory reset?)