VDMDBG Notes

From Open Watcom

Revision as of 12:06, 12 January 2013; view current revision
←Older revision | Newer revision→
Jump to: navigation, search

The purpose of this page is to record information, pointers to information and implementation details related to OW binnt\wd[w].exe. Collected here to help contributors track what is known about the interface (very little) and to document new findings (if any).

Contents

Abstract

Debugging Win16 and DOS programs with binnt\wdw on Windows 2000 and XP does not work in any version of Watcom and Open Watcom tools, at least since Watcom 11.0 up to and including Open Watcom 1.5. VDMDBG.DLL is a shared library available on Windows NT systems. It is supplied to allow debugging of 16-bit Windows and DOS applications hosted on NT. The applications are running under a process named ntvdm.exe (NTVDM = NT Virtual DOS Machine).

Known sources of information

MSDN CD's / DVD's

  • vdmdbg.hlp - The only available "documentation" on the subject.
  • vdmdbg.h - Contains some comments. Corresponding vdmdbg.lib in PSDK.

Articles on the subject

Books

WDW / VDMDBG interaction

Happens in the standard trap file of wdw (binnt\std.dll). The source can be found in bld\trap\lcl\nt\c\. The general idea is that when wdw discovers that the debugee is a Win16 or DOS program it interacts with the ntvdm.exe and for Win16 programs injects wowdeb.exe into its process space. To interact with the debugee, calls/callbacks into/from vdmdbg.dll is used.

State of the WDW speech (OW 1.5)

The current source comments for std.dll indicates that it's WOW (windows on windows) support was mostly written for NT older than NT 3.5. It has comments indicating that it has been modified for NT 4, but that adjustment is not related to WOW. Another comment specifies a WOW adjustment for NT 3.5 (Daytona) so it is likely that debugging Win16 and (possibly) DOS programs worked on that platform. On NT 4 the WIN16 debugging is reported to work (may not be totally stable). DOS program debugging on NT 4 is confused about address information and does not work.

What is failing?

That is a rather complex question which has not been answered yet. A suggestion has been made that the reason why it currently does not work may be that some of the vdmdbg messages are not handled the way they should, so work with that theory is started first. If you know something that is not mentioned about these messages, add it here or post a message on the news server.

Implemented in STATUS_VDM_EVENT switch (accrun.c):

VDMDBG NAME ID TOOLHELP.H Func. in source
DBG_SINGLESTEP 5 INT_1 Single step
DBG_BREAK 6 INT_3 DebugBreak
DBG_GPFAULT 7 INT_GPFAULT General Prot. Fault
DBG_TASKSTART 10 NFY_STARTTASK Debugee started + Record other tasks
DBG_TASKSTOP 11 NFY_EXITTASK Debugee stopped + Record other tasks
DBG_DLLSTART 12 NFY_STARTDLL Recorded
DBG_DLLSTOP 13 (info) Recorded
DBG_ATTACH 14 (info) VDM env. ready (source claim)


Default action for all mentioned below is the same as for the DBG_GPFAULT event, unless VDMProcessException() returns FALSE for the given message, in which case it is ignored.

VDMDBG NAME ID TOOLHELP.H
DBG_SEGLOAD 0 NFY_LOADSEG
DBG_SEGMOVE 1 (info)
DBG_SEGFREE 2 NFY_FREESEG
DBG_MODLOAD 3 (info)
DBG_MODFREE 4 NFY_DELMODULE
DBG_DIVOVERFLOW 8 INT_DIV0
DBG_INSTRFAULT 9 INT_UDINSTR
DBG_TOOLHELP 15 (info)
DBG_STACKFAULT 16 INT_STKFAULT
DBG_WOWINIT 17 (info???)
DBG_TEMPBP 18 (info???)
DBG_MODMOVE 19 (info)
DBG_INIT 20 (break+???)
DBG_GPFAULT2 21 (break)


Guesstimates so far (additional info welcome):
 0 -  4 can safely be ignored with a "just continue" action
 5 -  7 looks like they are handled OK (same as WIN32)
 8 -  9 error
10 - 13 info, register & continue
     14 info?? Source claim it is.
     15 info, register/just continue
     16 error
     17 info? ignore?
     18 info? ignore?
     19 info, just continue
     20 ??? (no doc.)
     21 error? what is normal?


So there are 4 groups:

Type Action
ignore (optionally record info)
info (record)
error (break & tell)
unknown 20, 18, 17, 14 (no doc. impossible to guess)

Notes: Currently 0-4, 8, 9 and 15-21 are treated as GPF's (unless VDMProcessException() returns FALSE for them). That may be wrong.

First real results

By adding advanced instrumentation to std.dll (MessageBox'es) it shows that currently (running on Windows XP), while loading a WIN16 application the following STATUS_VDM_EVENT's (WIN32 debug signal for (VDM) DBG_ messages):

 1 DBG_ATTACH
 1 DBG_INIT
 8 DBG_DLLSTART
 1 DBG_TASKSTART
 3 DBG_DLLSTART
 1 DBG_TASKSTART
 2 DBG_DLLSTART
 1 DBG_TASKSTART
 1 DBG_DLLSTART
 Application shown on screen
 Task manager shows ntvdm.exe with: 
     wowexec.exe
     wowdeb.exe
     myapp.exe
 Chose File | Exit in application
 1 DBG_TASKSTOP

That is it. No more. No DBG_BREAK's or whatever. At least now we know a little about why it does not work. The question now is: have std.dll failed to signal vdmdbg.dll that it is interested (to get additonal message(s)), or is it failing to do what it should on one of the messages it does receive above? I'm leaning towards the first, but the fact that this seems to work on NT 4 indicates otherwise.

Recording test notes from several people

NOTE 1

I also played a little with binnt\wdw. Ignoring the WOW aspect, I wanted to debug DOS apps locally. There is a function pair VDMSetDbgFlags / VDMGetDbgFlags which I tried out. In file accrun.c, "case DBG_INIT" I added calls of these functions to set flags VDMDBG_BREAK_DOSTASK | VDMDBG_BREAK_EXCEPTION. This was just a wild guess because of the intriguing names, I didn't find any documentation about this functions or these flags (If someone can supply some this would be very appreciated!).

After adding these functions I got another notification, this time a break in V86-code, which looked rather promising. But then the VDM crashed. Two things I found out so far:

  • there is code in misc.c which copies the contents of structure CONTEXT to VDMCONTEXT and vice versa. However, the VDMCONTEXT structure's size is smaller than the CONTEXT size, because no extended registers are defined there (currently?).
  • calling VDMGetThreadContext/VDMSetThreadContext both return 0, that means they are failing. In vdmdbg.h there is a note that these functions are obsolete, so I changed it to call VDMGetContext/VDMSetContext. After that VDMGetContext succeeds, but VDMSetContext still fails ... and I have no clue why.


NOTE 2

Q: Do you have any hints on what should normally happen at startup? Exactly when execution control is transferred from debugee to debugger?

A: Normally the trap file starts loading the debuggee until it's sorta there and then gets back to debugger; the debugger will then plant a breakpoint at main() or wherever and run the debuggee. If everything goes well, the breakpoint will be hit and the debugger will be sitting at the beginning of main() waiting for user input.

I am guessing that the trap file is getting confused on post-NT4 systems somehow and isn't mapping the debuggee's address space correctly. As a result, the debugger never stops at main(), probably because the breakpoint ended up somewhere in hyperspace.

I can see that the debugger shows the right information (or at least something resembling it) if the debuggee crashes, but isn't able to correlate the source code with machine code. Since I don't have a NT4 box at hand, I couldn't say what is different.

I stepped through the code a bit and there doesn't seem to be anything obviously wrong, so whatever the difference is might be subtle. I am suspecting ReqMap_addr() - it most likely doesn't work right.

It doesn't. The reason is that VDMGetModuleSelector() is always failing - it is in fact hardcoded to fail. I can also see that the code should be using VDMGetThreadSelectorEntry() instead of GetSelectorEntry(), but that makes no difference because VDMGetModuleSelector() is useless on W2K.

Maybe the debugger is supposed to keep track of the DBG_SEGLOAD etc. events?


NOTE 3

It appears that debugging of DOS apps never worked, with any version of Watcom C/C++ and on any version of Windows NT. Debugging of WoW apps did work to some extent.

As mentioned above, the VDMDBG interface was significantly overhauled on W2K. No documentation whatsoever is available for those changes. On W2K, VDMGetThreadContext is a wrapper around (newly added) VDMGetContext. The odd part is that while VDMGetThreadContext works for WoW threads, it fails for DOS threads. As mentioned above, VDMGetContext works for DOS.

However, neither VDMSetThreadContext nor VDMSetContext works at all. This may be a security/permissions problem, changing the thread context would require different privileges than reading it. Then again, the process/thread handles passed through CREATE_PROCESS_DEBUG_INFO are supposed to have all the necessary rights.


New test notes (Jan. 2013)

1. (quoted text from previous notes): "calling VDMGetThreadContext()/VDMSetThreadContext() both return 0, that means they are failing. ...[snip]... After that VDMGetContext() succeeds, but VDMSetContext() still fails"

Experience (on a XP SP3 system) is that VDMGetThreadContext() returns with != 0 and hence works, while VDMSetThreadContext() always fails and has to be replaced by VDMSetContext() - which works. This is true for both Win16 and DOS.

2. It's guessed that DBG_MODLOAD messages can be ignored.

This is definitely not correct for DOS debugging. This message is very important, because AFAIK handling this message is the only way for the debugger to set a breakpoint at the DOS-debuggee's entry point.

The notes above also mentions the "semi-documented" function pair VDMGetDbgFlags() and VDMSetDbgFlags(). I got a "working" trap file without using them ( both with Win16 and DOS apps ), but DOS debuggees won't report any exceptions besides single-step (01) and breakpoint (03) then. To make the debugger also receive notifications about true exceptions ( access violations, ... ), one has to call VDMSetDbgFlags( VDMDBG_BREAK_DEBUGGER | VDMDBG_BREAK_EXCEPTION ).

Personal tools