Recently, while developing a .NET Winforms desktop application, I needed to integrate a legacy .OCX that did terminal emulation for a serial device.  I had been using XP to develop and had run into no problems until I tried to deploy to a Vista machine.  When running the app on Vista, I was greeted with this exception:

Unable to get the window handle for the ‘myControl’ control. Windowless ActiveX controls are not supported.

Not much help here.  The “inner exception” was equally uninformative:

Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

The location of the exception was at ((System.ComponentModel.ISupportInitialize)(this.myControl)).EndInit();

Pretty worthless information.  After googling around for awhile, I ran across Ed Maurer’s helpful blog (thanks Ed) and he discusses some breaking changes that .NET Framework 2.0 SP1 introduces (also applies to .NET 3.5 Framework).
Apparently in the header of a PE file, there is a flag called IMAGE_DLLCHARACTERISTICS_NX_COMPAT.  This flag tells Vista whether or not to enable something called DEP (Data Execution Prevention) in your process.  If this flag is set, and your legacy .OCX does not have the flag set (which it would not by default), Vista won’t allow your .OCX to load.
Ed’s blog says that Microsoft likes DEP so much that .NET Framework 2.0 SP1 emits binaries with the IMAGE_DLLCHARACTERISTICS_NX_COMPAT on by default (it was off before).  This is going to cause unexpected expenses and time commitments for developers that deploy new builds to customers only to find that suddenly their apps don’t run on Vista anymore.  In my opinion, it is very wrong-headed to release a service pack with these kind of breaking changes.  Bad judgment, Microsoft.

-Disable DEP on the affected machines (run this line at a command prompt:  bcdedit.exe /set {current} nx AlwaysOff ). This is not a very good option for app developers for obvious reasons.
-Update the .OCX to build with a newer compiler (newer than ATL 7.1 and set the /NXCOMPAT linker flag – http://msdn.microsoft.com/en-us/library/ms235442(VS.80).aspx).  This is not an option with 3d party .OCXes, however.
-Run a post-build script in the .NET app that sets NXCOMPAT to NO and redeploy your app.  This script worked for me:

call "$(DevEnvDir)..\..\VC\bin\vcvars32.bat"
call "$(DevEnvDir)..\..\VC\bin\editbin.exe" /NXCOMPAT:NO "$(TargetPath)"

This will let the app run and load the OCX without a problem, but if you are trying to debug, it still has the same exception thrown.

TODOs for Microsoft:
-Don’t introduce this kind of breaking change in a “mandatory” service pack release.
-Provide better error info for the exception thrown.
-Document this somewhere easier to find than the back pages of an msdn blog.


  1. As a followup, apparently even if the NXCOMPAT compiler switch is off, Vista will refuse to load legacy controls while DEP is set to “AlwaysOn”.
    You must disable DEP to get it to work:

    To disable DEP in Vista:

    1. Click Start
    2. Click All Programs
    3. Click Accessories
    4. Right Click on Command Prompt
    5. Left click on Run as Administrator
    6. Click the Allow button if it asks you for permission
    7. Type or copy and paste this into the command prompt window:

    bcdedit.exe /set {current} nx AlwaysOff

    8. Hit enter and you should see the confirmation: Operation Completed Successfully
    9. Reboot and DEP should be off.

    Also, according to Steven Cheng, Microsoft MSDN Online Support Lead:
    As for the DEP setting, I think whether you mark the flag in your exe
    image will not quite concerns to future compatibility(at least from the OS
    DEP configuration’s perspective). Because the DEP behavior is still mainly
    controlled by the OS’s DEP policy. If the system admin always turn on DEP,
    even if you set it off in your exe, the application will still fail.


  2. i always learn something new from your post!

  3. tnx for info!!