Well I don't know anything about vista, but a google search for the error message you received gave this result: (it's for WordPerfect, but worth a try anyway
[quote name='http://www.columbia.edu/~em36/wpdos/vista.html'](1) On your Windows Vista computer, click on the Vista "pearl" where the Start Menu used to be and enter the word "COMMAND" (no quotation marks); a DOS prompt should open. Press Alt-Enter, which normally switches between windowed and full-screen mode. If you get an error message saying "This system does not support fullscreen mode", then proceed to step (2), after entering "exit" to close the window. Otherwise, enter "exit" to close the command prompt, and proceed to step (3).
(2) If you want to be able to use WPDOS in its traditional 25-line full-screen mode, and you got an error message when you tried the experiment in step (1), then you must use Vista's "Standard VGA Graphics Adapter" video driver instead of whatever driver you or Windows installed for use with your video hardware. Right-click on the Windows desktop; choose Personalize from the pop-up menu; choose Display Settings from the Personalize dialog; on the Monitor tab, click Advanced Settings...; on the Adapter tab, click Properties; on the Driver tab, click Update Driver...; click on "Browse My Computer for Driver Software"; click on "Let me pick a list of device drivers on my computer"; in the "Select a device driver" list, scroll down to Standard VGA Graphics Adapter and highlight (select) it; click Next. If your monitor goes blank, wait five minutes, then power down your computer and restart it normally. If your monitor does not go blank, wait while the driver is installed; click Close; then continue to press Close or OK or otherwise close dialog boxes until you are back to the desktop. If you are prompted to restart your computer, do so. You probably should do so anyway. (If you want to reverse this procedure, follow the same instructions, but let Windows automatically install the appropriate software for your system instead of browsing your computer for driver software.)[/quote]
For DOSbox, try running without sound if you haven't tried that, and try the latest CVS..