I'm running interchangeably between two systems:
Mac Pro (Mid 2010) w/ 16 GB RAM running macOS Sierra &
MacBook Pro (Retina, 15-inch, Mid 2014) w/ 16 GB RAM running macOS High Sierra
I have this same issue running Vectorworks 2018 SP2 on both machines.
When rendering (doing anything really, but rendering is the main culprit) I eventually reach a point where the application pauses and I get a message stating that "Your system has run out of application