Canon DPP Batch Processing
02011-Apr-1, Fri 12:00 in /blah [permalink]
[This information was collected using Digital Photo Professional 3.9.3 on Windows. It may not be applicable to other versions.]

Meet Roadblock

In my current attempt at storing my data for posterity (as if anyone cared), I needed to convert all my photos from RAW-format to JPEG.
Part of the problem was, that I had more than 13.000 RAW files in over 370 directories. Luckily DPP supports batch conversion, but only for files within a single directory. So did I really want to initiate a batch process for 370 directories manually? You sir, second row, wearing a yellow Bikini, are right. The answer to that question is, of course: "No frelling way!"

Have a Sandwich

What were my options?

  • Find another software that works the way I want
    Well, the thing is, that RAW conversion is not a well defined process, so each RAW-processor "develops" pictures in a slightly different way. And I happen to like DPP's way.
    So that's a "no go".
  • Force DPP to work the way I want
    Sounds intriguing to me, so I had found my winner.
    Checking for any command-line parameters for DPPViewer.exe resulted in.. well, nothing. But there were other executables and the most promising was DPPBatch.exe. Starting DPPBatch without any parameters even displays a menu where .vbf files can be loaded. Aha!
    As it turns out DPPViewer communicates with DPPBatch via temporarily created .vbf-files in the output-directory. A rather old-school approach to IPC, but something that can be exploited for my needs. I only needed to figure out how to create my own .vbf-files to use the DPP converter from the command-line.

Recipe for Solution

However, after analysing the file format, it turned out to be a bit more complicated than expected, because of recipes that are required in .vbf-files. Analysing a recipe's format and how it interacts with settings stored within a RAW file itself are a completely different beast altogether and nothing that is accomplished within a few minutes.

So I decided to quickly whip up a Perl script that scans for all RAW files in a directory recursively, stores their original location and moves them to a single directory. Then a single batch process could be started from within the UI of DPP to convert all files at once, after which the script can move the RAW files (and newly created JPEG files) back to their respective directories.

Actually it was not that easy because DPP is still a 32-bit application and ran out of memory when it tried to load that number of files, metadata and thumbnails it. So I had to spread them over a few directories with 2.000 files each. Which allowed me to convert all files by starting only 7 batch processes manually. Compared to the original number of 370 this was highly acceptable however.

.vbf File Format

Nonetheless here is what I've found out about the format of .vbf-files. Perhaps someone else can fill in the blanks or find something useful to do with that information.

Data Types

  • Numbers
    Unsigned 32-bit big endian integers
  • Strings
    The null-terminated string is preceeded by a 32-bit big endian integer specifying the byte-length of the string (including the terminating \0). The characters in the string are 1 byte wide and use the current Windows codepage, not UTF-8 encoding. Way to go!
  • Recipe
    Just like strings, recipes start with the number of bytes for the following recipe-chunk. For DPP 3.9.3 that are 1026 bytes.
SversionThe string "[version 1.10]"
SoutputDirectory where the converted files will be saved
Sfilename IF filename IS "\0xff\0x00"
  • suffix is appended to the source's filename (e.g. IMG_0001.jpg becomes IMG_0001suffix.jpg)
  • counter_init and counter_digits can be ignored
  • *the field suffix does not exist!
  • The resulting filename is generated by appending a number that is counter_digits long to filename, with the number being initialised by the value of counter_init (e.g. filename042.jpg if counter_digits were 3 and counter_init 42)
Sicc[5]Absolute paths to 5 different colour profiles
Nkind_of_fileOutput file format just like in the combo box:
0 = JPEG
1 = TIFF 8-bit
2 = TIFF 16-bit
3 = TIFF 8-bit + JPEG
4 = TIFF 16-bit + JPEG
NcompressionJPEG compression quality ranging from 10 to 100
Njpeg_optimizealways 1
affects only JPEGs by reducing their size. So my guess is: optimisation of huffman tables. And JPEGsnoop shows different distributions within those tables when using one or the other setting.
Njpeg_no_jfifalways 0
When resize is activated it is actually 4, so maybe this is more than just a boolean flag but a bitfield.
NwidthWidth and height are always in pixels. Any settings in the UI regarding the units of measurement are only used in combination with the DPI-setting to calculate the pixel-size
N?set to the same value as 'external' by the UI. However changing this value to an arbitrary value doesn't change anything, by the look of things.
RrecipeSome kind of global recipe?
Nimg_countNumber of image-chunks following:
SfilenamePath of the source file
RrecipeRecipe for this image. How does it interact with the global one? How with the settings stored within the RAW file itself?

Update 2012-01-05

Thanks to commenter DSops for pointing out Canon's free Digital Imaging Developer Programme, which includes an SDK for converting RAW files. However the FAQ states:
Q: What are the criteria for joining the programme?
A. To be eligible to join Canon's Digital Imaging Developer Programme, you should be operating as a legally registered company. Applications from other users will be considered at Canon's discretion. [...]
So it might or might not work out for you.
02011-Sep-24, Sat 21:56
You are star!
Your investigation has helped me a lot. I've just finished implementing an application, that substitutes the original DPPBatch.exe, asks the user how many workers to run at once, then splits the original script file into as many parts and executes one original DPPBatch.exe for each of them. This helps to fully load all the cores of my brand new Intel Core i7 PC and significantly speed up conversion process.
Thanks a lot!
02011-Sep-27, Tue 17:53
Glad to hear that someone found the information useful. :)
02012-Jan-5, Thu 8:43
Another solution is to use the Canon ESSDK (free with registering) for conversions.
02014-Apr-10, Thu 3:02
Recently I have same issue about batch process RAW files in different dirs.
My work flow is to store raw files in aaa/raw and convert to low resolution jpgs to aaa/raw2jpg. Use picasa to tag and upload the jpgs. When I need to collect pictures with specific tag, I use Win7 to search and collect all the pictures with that tag (also get the file list with full path, which makes selecting all original raw files possible). But when I need their high resolution version, I can't get DPP to do it with file list (or one by one with script). Do you have the vbf file for that? Something like the commenter "Just" did.
02014-Apr-10, Thu 22:11
If I understand your problem correctly you could try the following. But I have no idea if this works and I do not have a Windows machine handy at the moment to try it myself, so this is just a wild guess. Best to try it with two files from different directories first and have a back-up at hand, otherwise you might have to move a huge number of files back to their original directory manually.
TL;DR: Danger Will Robinson!
Search for the tagged files in Windows Explorer and MOVE the RAW files to a temporary directory. Do the DPP magic on the files in this directory and when you are done return to Windows Explorer (which needs to remain open, I guess) and Edit/Undo Move.
I do not know how Explorer's undo works, but if it is based on filenames only this should do the trick.
And just to reiterate: This might or might not work. Test it first, back-ups, own risk, &c.
02014-Apr-11, Fri 8:44
Thanks for reply. I do understand your suggestion, but I need the key part of "DPP magic", which is "command line usage of DPP batch converter": Where is the vbf and a easy example of it. Something like:
D:\pic\2014_0130\raw\001.cr2 par1 par2
D:\pic\2014_0210\raw\A01.cr2 par1 par2
Thanks a lot!
02014-Apr-11, Fri 20:06
Oh, then I am sorry but I have to disappoint you. As I pointed out in "Recipe for Solution" I abandoned the idea of generating .vbf files on my own, since that would have meant to reverse engineer the format of recipes.
I had to fall back on DPP to do the batch processing (that is: generating the .vbf file). The same goes for Just's solution, which did not generate a .vbf file on its own, but take the one DPP generated and split it up into multiple parts, so that multiple queues could be converted in parallel.
So in either case DPP itself was still used to generate the original .vbf file.
That is why I have suggested to move the files into one directory. Better to process/batch convert all files in a single directory than multiple files spread across numerous directories.
Maybe DSops' suggestion of Canon's ED-SDK can be of use to you, but I have not looked at it myself, so I do not know what it can do exactly.
My apologies that I cannot be of more help.
02014-Apr-16, Wed 6:02
Thanks for your reply. Seems that duplicating all the selected file to one directory is the only way to solve this problem... Maybe symbolic link can save some time and space?
02014-Apr-16, Wed 14:55
I just tried symbolic way and it works fine!
New comments disabled