2009
05.28

NXCOMPAT?

Recently, while developing a .NET Winforms desktop application, I needed to integrate a legacy .OCX that did terminal emulation for a serial device.  I had been using XP to develop and had run into no problems until I tried to deploy to a Vista machine.  When running the app on Vista, I was greeted with this exception:

Unable to get the window handle for the ‘myControl’ control. Windowless ActiveX controls are not supported.

Not much help here.  The “inner exception” was equally uninformative:

Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

The location of the exception was at ((System.ComponentModel.ISupportInitialize)(this.myControl)).EndInit();

Pretty worthless information.  After googling around for awhile, I ran across Ed Maurer’s helpful blog (thanks Ed) and he discusses some breaking changes that .NET Framework 2.0 SP1 introduces (also applies to .NET 3.5 Framework).
Apparently in the header of a PE file, there is a flag called IMAGE_DLLCHARACTERISTICS_NX_COMPAT.  This flag tells Vista whether or not to enable something called DEP (Data Execution Prevention) in your process.  If this flag is set, and your legacy .OCX does not have the flag set (which it would not by default), Vista won’t allow your .OCX to load.
Ed’s blog says that Microsoft likes DEP so much that .NET Framework 2.0 SP1 emits binaries with the IMAGE_DLLCHARACTERISTICS_NX_COMPAT on by default (it was off before).  This is going to cause unexpected expenses and time commitments for developers that deploy new builds to customers only to find that suddenly their apps don’t run on Vista anymore.  In my opinion, it is very wrong-headed to release a service pack with these kind of breaking changes.  Bad judgment, Microsoft.

Solutions?
-Disable DEP on the affected machines (run this line at a command prompt:  bcdedit.exe /set {current} nx AlwaysOff ). This is not a very good option for app developers for obvious reasons.
-Update the .OCX to build with a newer compiler (newer than ATL 7.1 and set the /NXCOMPAT linker flag – http://msdn.microsoft.com/en-us/library/ms235442(VS.80).aspx).  This is not an option with 3d party .OCXes, however.
-Run a post-build script in the .NET app that sets NXCOMPAT to NO and redeploy your app.  This script worked for me:

call "$(DevEnvDir)..\..\VC\bin\vcvars32.bat"
call "$(DevEnvDir)..\..\VC\bin\editbin.exe" /NXCOMPAT:NO "$(TargetPath)"

This will let the app run and load the OCX without a problem, but if you are trying to debug, it still has the same exception thrown.

TODOs for Microsoft:
-Don’t introduce this kind of breaking change in a “mandatory” service pack release.
-Provide better error info for the exception thrown.
-Document this somewhere easier to find than the back pages of an msdn blog.

Midniteblogger