Jul 2013

Usual Introductory Blah

About a year ago I wanted to dump Microsoft Visual C++ in favour of MinGW, since I'd have really liked to use 'modern' C++11 features.

However, the promises of out-of-band compiler updates with better C++11 support and the fact that it is easy to create minidumps via dbghelp.dll's functionality and debug them with windbg were enough to delay this move.

Until now.

With the announcement of MSVC 2013 and its - still - lackluster support for C++11 I gave in.
However losing the debugging support for crashdumps really hurts. But every search on the web only turned up other unsuccessful inquiries about a working post-mortem debug facility with MinGW.

The Problem

The only sane way to create crash-dumps under Windows is to use dbghelp.dll's Minidump functionality.
This minidump can then be loaded with windbg (or Visual Studio). However, both applications need a PDB file to show useful information (unless assembly and memory addresses instead of function names is sufficiently informative for you).
A file, gcc does not create, since it stores its debug information in DWARF format. A format neither of those programs can handle.

So how about the other side?
gdb can of course handle DWARF files. But it can not load minidump files.


The Solution

Luckily(?) the ecosystem of the D programming language faced a similar problem.
So that is where the solution can be found: cv2pdb, which can not only convert CodeView files to PDB, but also the DWARFen ones!

Create the Debug Info

  1. Compile/link your project with debug information (gcc -g)
  2. Use cv2pdb to convert the DWARF debug info into PDB format (cv2pdb -C program.exe)
  3. Strip the executable from any debug info (strip -g program.exe)
  4. Distribute the executable and keep the PDB file

The PDB file and the executable cannot be matched automatically based on their checksum, since the strip changed it. So you need an out-of-band way to match the correct files.

The way I do it: on an actual crash I create the minidump and pack it together with the logfile, which contains the application's version number, into a zip or tar file before sending it off. That way I can match the minidump to the correct PDB file.

Actual Debugging

Since the executable was stripped of any link to the PDB file windbg tries to load a generic image00400000.pdb file (or image00000000`00400000.pdb for 64bit executables).
So make sure that is the PDB's file name.
I for one just archive the PDB files named this way and put them in a directory for each version.

  • The directory containing this file needs to be added to windbg's symbol search path. (-y <path>)
  • Further more, since the PDB file and executable cannot be matched .symopt needs to be changed, to include 0x40 (SYMOPT_LOAD_ANYTHING).
    Combined with the default symopts this is 0x80030377. (-sflags 0x80030377)
  • And finally the actual dump file needs to be specified with the -z argument.
  • While I'm at it: I also add -g to skip the initial breakpoint.

All in all I end up with a command line like this to start windbg to do post-mortem debugging with a minidump:

windbg -sflags 0x80030377 -g -y <directory_containing_correct_pdb_file> -z <path_to_dumpfile> <path_of_correct_executable>

02014-Feb-12, Wed 12:29
I tried this, but cv2pdb crashes on my machine, I don't understand why. Does this solution work stable for you?
02014-Feb-12, Wed 15:57
I had cv2pdb crash on me when the PDB file already existed. Deleting the PDB file from a previous run before executing cv2pdb again was a viable workaround for me.
But I just tried to apply the process described in the post again and as it turns out cv2pdb crashes no matter what. I tried the pre-compiled binaries of cv2pdb for version 0.25, 0.27 and 0.28. None of them work.
Since the last time I had to generate a PDB file (a few months) I updated MinGW. So my guess is, that the more current compiler (GCC 4.8.2) outputs some debug information cv2pdb cannot handle.
But that is just a guess. Sorry that I could not help you any further.
Should you stumble across/find a solution, please let me know.
02014-Mar-3, Mon 7:57
A new version of cv2pdb is available, which fixes this bug.
02014-Mar-4, Tue 12:14
Indeed it does. Thanks for the effort and letting me know!
02014-Jun-5, Thu 11:58
Is there any way to debug using VS debugger? When I load the crash dump I get this http://i.imgur.com/6umv0a2.png When I try to load symbols manually I get "A matching symbol file was not found in this folder.".
I tried to use WinDbg as well, but when I execute provided command ("C:\Program Files (x86)\Windows Kits\8.1\Debuggers\x86\windbg.exe" -sflags 0x80030377 -g -y C:\users\xyz\acquisition-bin\ -z C:\users\xyz\acquisition-bin\acquisition_20140605_110858.dmp C:\users\xyz\acquisition-bin\acquisition.exe) I get this: "The command line arguments cannot specify more than one kind of debugging to start". If I remove last argument I get to WinDbg main screen, but now how do I get to source code? Or view where the crash happened?
02014-Aug-8, Fri 22:20
I was able to use this method to debug my gcc-built binary using Visual Studio Express 2013 instead of WinDbg. It was surprisingly simple. Admittedly, my scenario is straightforward (just trying to debug a crash I'm seeing locally, not trying to debug crash dumps submitted from the field). Regardless, my steps to do so were:
1) Build binary using gcc -g [blah] as normal
2) Run cv2pdb -C blah.exe (this generated a .pdb in the same dir as the .exe). I did not strip the .exe.
3) Ran blah.exe until it crashed
4) Ran Task Manager, right-clicked on the process, selected "Create dump file" to create blah.dmp.
5) Ran Visual Studio Express 2013, File > Open, browsed to .dmp file. This shows a "Minidump File Summary" page. On the right, under an "Actions" menu, I clicked the "Debug with Native Only" option. Visual Studio automatically found my .pdb (I'd left it sitting in the same dir as the .exe), and voila - I see all my threads, I can browse the call stack of each, examine locals, see code, etc. with full debug symbols. Magical!
As I was examining different threads, and moving up and down the call stack, VS would occasionally prompt me to help it find a source file, but that was easily handled.
02014-Aug-8, Fri 22:43
Playing around a bit further, I found that I could run my gcc-built app (built using -g, and then use the cv2pdb tool to generate pdb as described above), use Visual Studio to attach to the running process, then break & start stepping my app as if it were built using Visual Studio in the first place. I appear to have full VS debugger functionality: I can set breakpoints, modify the values of locals, see threads that are currently in system calls (something that GDB on Windows can't do - see https://cygwin.com/ml/cygwin/2006-09/msg00410.html), etc. etc.
02014-Oct-5, Sun 9:02
I am trying to do the same thing as "AmazedThatThisWorked", but I cannot get cv2pdb to work at all. The INSTALL file in the cv2pdb zip says:
>If you are using some other program, you'll still need some
>files from one of the distributions. These are mspdb80.dll, mspdbsrv.exe,
>msobj80.dll, mspdbcore.dll and msvcr90.dll from the VS2008 installation or
>mspdb100.dll, mspdbsrv.exe, msobj100.dll, mspdbcore.dll and msvcr100.dll
>from VS2010. They should be accessible through the PATH environment variable.
>(The VS Shell is missing the msobj80.dll/msobj100.dll only).
So I copied what seem to be the necessary files from Visual Studio 2013 into a folder with cv2pdb. When I try to run cv2pdb -C program.exe, it outputs a small pdb file and then crashes; the pdb file seems incomplete and won't load in Visual Studio or whatever. Also, I noticed that it leaves an orphaned mspdbsrv.exe process running which I have to terminate manually.
As the INSTALL file only lists dlls up to Visual Studio 2010, I tried downloading that and using those DLLs instead, but that produced the same crash.
Is there any trick to getting cv2pdb to run? Do I have to have some other files on my PATH? Since I don't usually use Visual Studio as my main IDE I don't have anything of its set on my PATH.
02014-Oct-20, Mon 17:36
I simply unzipped cv2pdb_0.31.zip to some dir, and didn't have to mess around with .dll's or anything. However, I do have VS 10, VS 12, and VS 2013 Express all installed. After reading your post, I read cv2pdb's INSTALL file and got curious -- taking a look at their source package (specifically src\mspdb.cpp), you can see the logic that's used to dynamically find and load mspdbXYZ.dll. Hopefully this can help you figure out why your install isn't working.
If it helps any, I can run cv2pdb without needing to have anything special on my path (I don't have to run vcvars32.bat or vsvars32.bat or anything first). But looking at the code in mspdb.cpp, that makes sense -- I see a getInstallDir() function that's querying the registry to look for my VS install dir, then loading the needed .dll based on that.
New comments disabled