Debugging COM DLL files in AutoCAD x64 and Visual Studio

While attempting to work out some x64 specific kinks in some of the COM code that is being migrated to .NET I hit a pretty annoying wall.  Well, it was actually a series of small walls (probably around knee high)  that kept tripping me…

In this post, I’m working on a problem that is pertaining to a 64 bit version of AutoCAD loading a COM enabled .NET assembly – however this information should apply to any x64 executable calling any x64 COM assembly.   Replace AutoCAD with <YourApp.exe> and you should be good to go =)

The first thing I did was to ensure each of the projects had been set to build for  “ANY CPU”.   This got the ball of fun rolling.   We have a VLX which instantiates the COM class using vlax-get-or-create-object.  When called this would constantly return nil.   Of course this all works just peachy when running an x86 build of AutoCAD.   At this point my breakpoints would appear in Visual Studio as disabled and would happily inform me that “The breakpoint will not currently be hit”.

This usually results when the incorrect DLL is being loaded, or when the PDB file is missing.   Once Visual Studio is running in debug mode you can bring up the Modules window from the Debug/Windows/Modules menu.  This should show you all the assemblies involved in the current session as well a bunch of other information including their paths.  The assembly in question was not listed here – so at this point I’m assuming that Windows cannot find the dll.

Junk in the Registry

I am now thinking that I have orphaned and duplicated types and CLSIDs in the registry, likely pointing to other builds of the dlls.   If the GUIDs for the classes or types changed you end up with a lot of junk in the registry.   I don’t trust those registry “cleaner” apps and in the interest of keeping my OS working – I chose to manually search out and nuke all the registry references manually.  This can be done by searching both for your assembly’s type name, and COM exposed class names in regedit.  A lot of this was done in the HKEY_CLASSES_ROOT root.  After all of that the problem still exists..

Wow6432Node Registry Keys

Registry keys were a two part problem. After dealing with the junk in the registry, I found I also had to deal with the x86 .   DraftLogic has been until this point limited to x86 architecture as a result of  the VB6 code not being x64 friendly.   As a result of this, all of the needed keys had been placed in the WOW6432Node of the appropriate software registry sections.   This was the easiest wall to find and hop over.  Unfortunately, this also didn’t solve the problem.

Improperly Registered Assemblies-thanks for nothingVisual Studio

So the next and fortunately final problem turns out that Visual Studio 2005/2008 (unsure about 2010)   Turns out the last piece of the solution is pretty simple – couple changes to the project.  What’s happening is that Visual Studio is running the x86 build of regasm.exe to register the assemblies on run, probably because Visual Studio is an x86 app itself.  This results in all kinds of x64 hating… To solve this we have to change two things in the COM enabled projects.

First though, I created a new build configuration called x64 so these changes would not affect in my ANY CPU configuration – which will be used to build my shipping assemblies.  It is also important to note that this problem is MOSTLY only a problem when running code from the IDE.  Assemblies installed and registered using an installer should not have this problem if the installer takes into consideration the “bitness” of AutoCAD (or any application calling your COM enabled dll).  ie.  Installshield asks the user on install what “bitness” of AutoCAD they are using, and the assemblies are registered as needed.  It may also be possible to register an assembly using both the 32 and 64 bit versions of regasm – but I have not tried this.

So with the new x64 configuration created I edited the properties of each COM enabled project  I unchecked “Register for COM interop” and added a post-build event command line of:

%Windir%\Microsoft.NET\Framework64\v2.0.50727\regasm $(TargetPath) /register /codebase /tlb

This would ensure that my COM enabled assemblies were properly registered for use by 64 bit  applications.

Page 1 of 2 | Next page