Catalog Conundrum: Splitting up a Large Capture One Pro Catalog
Capture One Pro users are blessed with a fine RAW image processor, excellent color adjustment tools, layered adjustments and perspective correction abilities... and some "challenges." Regardless of those challenges, I have adopted C1P at the center of my workflow, so yes, I'm a fan. However, recently I ran into it's biggest challenge (according to many) and had to address it.
C1P's Catalog, while better than its reputation, suffers and slows as it grows in size. In my case, the need to split it up became mandatory as I found myself often waiting for the catalog to do it's thing.... watching my Mac's dreaded rainbow spinning wheel. Similarly, selecting the "All images" view, bought me time to brew a pot of coffee and catch up on the news.
So the goal here is to split up one large CaptureOne Pro 10 catalog into (in my case) three smaller ones. The expected gains are of course better speed in everyday use and better organization (I needed that!). Note: if you only use Capture One's "Sessions", and not the Catalog, this does not apply to you.
Splitting a catalog should be easy, right? Ummmm, yes and no. I opened a discussion in the Phase One support forum and quickly came to the conclusion that it is kind of a brute force, manual process. No tools exist to simplify this process and the menu options which do exist have limitations. One suggestion given was to simply clone the Master catalog two more times and then delete the unneeded image folders from all three catalogs. Others suggested creating user collections of the images you want to remove and then exporting them as a single catalog.
I ended up with a hybrid approach creating two new catalogs from scratch and trimming down my original. Here's how I did it.
Staring at the rainbow spinning wheel (on Mac) is not my idea of a good time. My C1P catalog had reached 97,000 images (and growing) from paid gigs, family events, sporting events and just messing around. That catalog began life much earlier in Aperture (circa Aperture 1.0 in 2005), prior to which I'd simply used a folder structure with Photomechanic and Nikon Capture NX/NX2. Aperture served me well for quite some time until I saw the writing on the wall. I hung in for another couple years and then jumped on the Lightroom bandwagon for a year or so. I kept wanting to like Lightroom, I really did, but it just rubs me the wrong way, as do subscriptions. (Not trying to offend LR users, it just isn't my cup of tea).
So in 2011 or so , I started experimenting Capture One Pro 6 even though it did not have a catalog. I really loved the RAW conversions and with version 7, in 2012, PhaseOne added a catalog feature and I dove in full time. While certainly no Aperture or LR in terms of asset management, it worked for my then smallish catalog of less than 25,000 images. As my catalog grew in size, PhaseOne also kept incrementally improving their product with versions 8, 9 and 10 and each release sped it up just enough to keep me going in my single, ever-expanding catalog.
Enter early 2017: Even with the latest CaptureOne Pro 10 updates, the speed, while better, just are not adequate for me as I approach 100k images... the rainbow spinning wheel become increasingly present and annoying and I so bit the bullet and decided to split it it up. But how?
Given my, shall we say, "eclectic" collection of images, I quickly decided to divide my Master catalog into three catalogs as follows:
SPORTS (team sports, running events, triathlons)
WORK (portraits, pets, studio product)
PERSONAL (landscape, family, parties, etc.)
I wasn't actually sure if this would split it up appropriately size-wise, but I figured I'd run down that path until and if I saw a need to modify. My goal was to preserve any edits and metadata related to the images being moved. Here's a quick chart of the approach:
IMPORTANT CONCEPT: With Capture One, you can export images to a new Catalog and only a new catalog; you cannot export images directly into an existing catalog.
For this reason, I ended up exporting 93 folders of Sports images resulting in 93 temporary "Sports" catalogs and similarly exported 49 folders of "Work" images creating 49 temporary "Work catalogs". Why? Because the images from each shoot were in their own folder and when exporting a folder in C1P you can only export to one new catalog. Those 142 temporary catalogs then simply became the raw material to be imported into two brand new permanent catalogs called "Sports Vol 1" and "Work Vol 1".
Regarding Temporary Catalog locations
In my situation the internal SSD drive (where my original Master Catalog resides) did not have space to accommodate +/-150GB of temporary catalogs and thus the need to go "external". So on an external drive, I setup folders called "Catalog Assembly Sports" and "Catalog Assembly Work" to hold the 142 temporary catalogs created above. (Note: the name 'Catalog Assembly' is used only because I was not very creative when having to think of a name!). I used a fast Thunderbolt drive with plenty of space on it, but you can use any drive with sufficient room, obviously faster the better.
4 HIGH LEVEL STEPS
I'm obsessed with backups but to be extra careful, I opened my 97k image Master catalog and backed it up to another separate drive one more time.
2) EXPORT temp Catalogs
I went through my Master catalog's folder structure and picked any folder (project) related to Work or Sports, then exported it as a separate catalog to the appropriate temporary folder location.
(note: I named these temp catalogs similar to their original folder name)
3) IMPORT Temp Catalogs
After creating two new, empty Catalogs called "Sports Vol 1" and "Work Vol 1" I executed the rather repetitious chore of importing all the temporary catalogs created in step "2)" into the appropriate new Catalog.
4) DELETE folders in Master
With all the folders exported and imported into their new homes, I went back through the original Master catalog and deleted those image folders to reduce my Master catalog size. The catalogs in produced in 2) served as my check list. Since I named the temp catalogs similar to their folder name back in the master, it was easy to find them and delete them.
The last thing I did was to rename my Master catalog changing it from "Master" to "Personal vol 1" to match my new naming convention. This was done with C1P shut down by renaming the catalog 'package' and within the package, renaming the database itself. This is optional of course.
What about user Collections?
If you have User Collections, you may want to export those and import them into one of your new catalogs. I had some sports albums I'd created and easily exported them end imported them into my new "Sports vol 1" catalog. Collections export just like folders and import the same way. You may, upon import, see messages where C1 gives you an option to add an additional variant (or not).
Why not just export collections instead of all those folders?
My goal was to reduce my catalog size. If I had just exported my user collections, the images would still be maintained in my Master catalog at the folder level and thus I wouldn't achieve any size reduction. At least that's my understanding.
Contributing Details, Warnings, Closing Thoughts
My Folder Structure
Just for reference, my raw images are imported and stored by "project" or shoot. Whenever I download to the computer, I create a separate folder for each shoot. In the example to the right, you can see 8 shoots from November 2015. They fall under a YEAR folder ('2015') and a YEARMO folder ('201511').
The project's RAW images themselves are located in a folder of this convention:
"yyyymmdd - name of project" which you see listed.
How to export a folder to a catalog
Exporting is simple, just highlight a folder - in this example I highlighted "20120714 Julia Shoot, Solar" - right-click to get the menu, then select "Export as Catalog" (in blue).
C1P will prompt you to name the resulting catalog and pick the destination. I named it similar to the source folder (e.g. "2012714 Julia") and set the destination to, in this case, the "Catalog Assembly Work" folder. Similar naming (folder ->temp catalog) helped me delete the folders from my original catalog in the final step.
BEWARE of FILTERS!
BEWARE of FILTERS!!
BEWARE of FILTERS!!!
One thing that bit me is that I didn't religiously clear filters before exporting. As a result, the exported temporary Catalog was incomplete!
CaptureOne will open a folder with the last filter you used so BE SURE you click the "x" here before exporting to remove all filters and thus include all images in the export.
How to import temp catalogs into new catalog
The import process into the new catalogs is easy, simply select
Capture One Catalog
and then select the temporary Catalog you exported in step 2). Repeat this for all your temporary catalogs
This process worked fine for me, resulting in three smaller catalogs based on type of content and they perform more efficiently. Will I keep it this way? Actually, the master catalog only went from 97k images to 56k and I may in fact split it again, we'll see. Also, if PhaseOne improves Media Pro (MP) to work better with C1 , I may move in the MP direction; alternatively if PhaseOne upgrades the C1 catalog significantly, I may rethink it again. But in the meantime, I'm good to go.
There are certainly other approaches to splitting a large catalog and it's not my intention to outline them all. I will say though that if someone at PhaseOne ever happens to read this, that your raw processor is great - I really love the new process preview and multi-step sharpening - but you really need to make some tools to either make it easier to split a catalog (e.g automate what I just did) or make the catalog industrial strength. For instance, all a new tool would have to do is allow the user to export multiple folders at once into a single new catalog. That would have cut the effort in half. Another option would be to better integrate C1 with Media Pro SE (MP) so that users can easily move to MP as their primary catalog. Right now it's messy and MP doesn't even support C1P version 10 edits.
In the end, this probably took me, removing lunch breaks, interruptions, etc, a full day to do this, mostly because of the export process and having to export 142 folders individually. Tedious? yes, but worth it.