RW9 uses system memory for rendering, not the memory dedicated to VW. It seems counterintutive, but try decreasing the memory dedicated to VW and make more available to the system, but if you are realy set at 100K (not M) you might want to try increasing the allocation. Have you changed versions of Quicktime recently? VW uses some components from QT to export images, so a new verson of QT may have changed some things.
Go to Vegas, win big at the tables. Stop by the VectorWorks booth and spend some of you winnings on V11. It happened to someone two years ago at LDI.