Jump to content
jweston

Hardware tests and Data

Recommended Posts

So besides the obvious "What hardware can I buy to make it perfect?" type question I want to be more more inclusive of data analytics and actual testing.

 

1) I know that there is the Demo file that Vision comes with, but what are the chance of having a few "user created" scenes that we can send out for testing purposes? I would LOVE to see a data table of the same file, used across a bunch of different hardware and settings.

 

2) What types of setups do you guys test on at Vectorworks HQ? Do you try multiple types of setups from minimal to absolutely insane?

 

3) In a lot of Graphics heavy programs or scenarios there are very specific requirements to get the most out of a system. That includes even down to the exact graphics driver you are using for a given card. Has any testing, data or thought been put into trying to figure that out?

 

4) While yes there are limitations to the way a software can scale too, I would think those limiting factors would be money and time? In my opinion I would like to see improvements made in the ability to get large fixture heavy shows working as fast as possible, and then go back and add in the fancy features. In most cases I end up running my machine at 1024x768, turning off every feature the program has just to get the lights to flash all at one time when I hit the button. Thus all that time put into making those features is a complete waste. Knowing that it is possible for me to buy a 64 core processor, with dual 2080ti, 256gb of ram and then overclocking everything, but not knowing if that actually helps me at all is painful.

 

Lastly, if there is anything we can do to help test this sign me up. 

  • Love 1

Share this post


Link to post
14 hours ago, jweston said:

1) I know that there is the Demo file that Vision comes with, but what are the chance of having a few "user created" scenes that we can send out for testing purposes? I would LOVE to see a data table of the same file, used across a bunch of different hardware and settings.

We do not have any, but this has been discussed in the past. I think one issue we kept running into was users wanting their files kept private. A lot of the files we get from users are on NDA and cannot be shared with other users, but are used for internal testing.

 

14 hours ago, jweston said:

2) What types of setups do you guys test on at Vectorworks HQ? Do you try multiple types of setups from minimal to absolutely insane?

Most of our developers run on fairly minimal machines. Tech Support and Marketing have access to one "beast" machine that we often take to tradeshows. This machine was a beast when we built it, but it is now becoming dated. I'm not sure the CPU/RAM that it has, but we had 2x1080Ti's in SLI in it at one point.

 

14 hours ago, jweston said:

3) In a lot of Graphics heavy programs or scenarios there are very specific requirements to get the most out of a system. That includes even down to the exact graphics driver you are using for a given card. Has any testing, data or thought been put into trying to figure that out?

No, but I like this idea a lot! I'm not sure how we'd go about approaching it, but that's neither here nor there 😛

 

14 hours ago, jweston said:

4) While yes there are limitations to the way a software can scale too, I would think those limiting factors would be money and time? In my opinion I would like to see improvements made in the ability to get large fixture heavy shows working as fast as possible, and then go back and add in the fancy features. In most cases I end up running my machine at 1024x768, turning off every feature the program has just to get the lights to flash all at one time when I hit the button. Thus all that time put into making those features is a complete waste. Knowing that it is possible for me to buy a 64 core processor, with dual 2080ti, 256gb of ram and then overclocking everything, but not knowing if that actually helps me at all is painful.

Those limiting factors are not time and money. I know this is an extreme example, but passwords are very well encrypted. So much so that the fastest computer in the world would take years/lifetimes to crack it. The "issue" here isn't necessarily the hardware. It is the algorithm that is used to crack this password. For example, a dictionary attack may be faster than brute forcing. This is all on the same hardware, but the software imposes an upper limit on "how quickly the password can be cracked".

 

Taking this conversation to Vision, look at Vision 2018 vs Vision 2019. It did not matter how much time/money/hardware you threw at Vision 2018, it just ran like garbage compared to the same machine with Vision 2019. So, while it may seem like the limiting factors of performance are time/money, this is only true to a certain extent. Eventually, you will hit an upper limit of the algorithm itself, and the issue is no longer hardware but the software design.

 

One thing to keep in mind as well as that it was somewhat intended for these features to be shut off for real time renderings. The main reason we keep these features around is for "High Quality Render Movie/Still".

 

Lastly, I do not disagree that Vision performance for real-time renderings needs to be improved. There are a few areas we can look at, but the one I've got my eye on is the physics engine (which is what handles panning/tilting fixtures, moving meshes related to DMX XForms, etc etc).

 

 

14 hours ago, jweston said:

Lastly, if there is anything we can do to help test this sign me up. 

I'd love to get some files from you guys that you wouldn't mind shipping as a demo file for Vision 2021. We usually record some DMX with it so a user without a light board can easily playback a showfile to see what Vision is capable of.

 

The only other thing is finding a way to document hardware performance as well as driver performance. This will need lots of testing.

Perhaps, the best thing to do here is come up with a "workflow" or "benchmark" for testing. For example:

  1. Load the sponza demo file
  2. Ensure all app/doc settings are set to default
  3. Playback recorded DMX
  4. Write down FPS at 0:30, 1:00, 1:30, and 2:00 (or something like that)

The most important thing when doing these kinds of tests is ensuring that everyone is testing the same way.

Having other programs like VW running in the background would affect these tests. So, we'll have to take it on good faith that no shenanigans like that are happening.

Share this post


Link to post
4 minutes ago, bbudzon said:

We do not have any, but this has been discussed in the past. I think one issue we kept running into was users wanting their files kept private. A lot of the files we get from users are on NDA and cannot be shared with other users, but are used for internal testing.

I can absolutely come up with some weird designs to send over. In all honesty they are not as "pretty" as the Sponza demo file. But that is purely by design. At least in our use case, we are trying to get the fastest frame rate, with as much realism as possible. Does this mean we get to use the nice new haze? Sadly no. We just need to have the lights dim, wiggle and show gobos with as much accuracy to real life as possible. Our goal is not photo realistic renderings, but something that closely resembles the timing you would expect in real life on the actual rig.

 

I will try and create a small, medium and large show file to send over your way that can be distributed to anyone.

 

10 minutes ago, bbudzon said:

Most of our developers run on fairly minimal machines. Tech Support and Marketing have access to one "beast" machine that we often take to tradeshows. This machine was a beast when we built it, but it is now becoming dated. I'm not sure the CPU/RAM that it has, but we had 2x1080Ti's in SLI in it at one point.

Where is the tip jar that I can add money too to get you guys good hardware? While I know the current discussion is that the software may not currently scale perfectly with hardware, I would love to push the need to have something that scales better. 

 

12 minutes ago, bbudzon said:

No, but I like this idea a lot! I'm not sure how we'd go about approaching it, but that's neither here nor there 😛

Cannot say I know the best way to test this either, but I do know with other programs this is a much bigger sticking point. And in following this, knowing if a different cpu architecture is better than another, or cores vs clock speed type discussions.

 

Vision is arguably less about data and planning, and purely about speed and looks. So knowing how to get the most out of that is really the only thing that matters to us in Vision (maybe others disagree?)

 

17 minutes ago, bbudzon said:

Those limiting factors are not time and money. I know this is an extreme example, but passwords are very well encrypted. So much so that the fastest computer in the world would take years/lifetimes to crack it. The "issue" here isn't necessarily the hardware. It is the algorithm that is used to crack this password. For example, a dictionary attack may be faster than brute forcing. This is all on the same hardware, but the software imposes an upper limit on "how quickly the password can be cracked".

 

Taking this conversation to Vision, look at Vision 2018 vs Vision 2019. It did not matter how much time/money/hardware you threw at Vision 2018, it just ran like garbage compared to the same machine with Vision 2019. So, while it may seem like the limiting factors of performance are time/money, this is only true to a certain extent. Eventually, you will hit an upper limit of the algorithm itself, and the issue is no longer hardware but the software design.

 

One thing to keep in mind as well as that it was somewhat intended for these features to be shut off for real time renderings. The main reason we keep these features around is for "High Quality Render Movie/Still".

 

Lastly, I do not disagree that Vision performance for real-time renderings needs to be improved. There are a few areas we can look at, but the one I've got my eye on is the physics engine (which is what handles panning/tilting fixtures, moving meshes related to DMX XForms, etc etc).

Fair enough, but one would expect in that extreme case  that the amount of time it takes to crack a password with an old Pentium III CPU vs a new AMD Threadripper 3390x would be significantly lower.

 

Obviously Vision has been getting faster. However lighting, like Moore's law seems to be getting complex just as fast. With all the current multipart fixtures things have gotten big fast.

 

I can get over all of the trials and tribulations (see fixture orientation, dmx transforms and the like...) but allowing the hardware to scale "better" with hardware should be a must.

 

I hear by place my vote to let you guys focus on making the software "solve passwords better" haha. If you guys improve that portion of the software, then I can throw hardware at it. I think it is perfectly acceptable to accept that if I want it to run faster I just throw gear/money at it. Currently I don't feel I can do that with much success.

 

Then once all that settles we can all deal with MVR and GDTF. 🤣

 

(Steps off soapbox)

  • Like 1

Share this post


Link to post
15 minutes ago, jweston said:

I can absolutely come up with some weird designs to send over. In all honesty they are not as "pretty" as the Sponza demo file. But that is purely by design. At least in our use case, we are trying to get the fastest frame rate, with as much realism as possible. Does this mean we get to use the nice new haze? Sadly no. We just need to have the lights dim, wiggle and show gobos with as much accuracy to real life as possible. Our goal is not photo realistic renderings, but something that closely resembles the timing you would expect in real life on the actual rig.

I love this idea! We have always generally focused on prettier looking scenes with lower light counts as these same files were used for marketing material. But focusing on a performance friendly scene with many many lights is a fantastic idea. Let's coordinate on this.

 

20 minutes ago, jweston said:

I will try and create a small, medium and large show file to send over your way that can be distributed to anyone.

Just an FYI, I think the priority here would be large show file then medium then small. I like your, "Go big or go home" idea above 😉 If you can provide all three, that would be great. I'll look into seeing which ones we can get approved.

 

21 minutes ago, jweston said:

Where is the tip jar that I can add money too to get you guys good hardware?

🤣 I think it's somewhat of a personal preference for me. I prefer the mobility of a laptop over a desktop. And don't get me wrong, my laptop is beefy. But a laptop will almost never outperform a decent desktop. And even though I do my primary development on a MBP, I have a Windows PC that is a desktop and it's card is nicer (but could still probably use an upgrade; tbh, I never thought to request one as I use it so infrequently).

 

So, maybe saying minimal hardware wasn't completely accurate. I simply meant that your 2x2080Ti in SLI Thread Ripper is going to demolish my primary machine that I use for development 😂 I think anytime I need more power than my MBP or Windows PC can handle, I just borrow the TechSupport/Marketing machine which is at least 2x1080Ti in SLI. No thread ripper though 😛 

 

28 minutes ago, jweston said:

Fair enough, but one would expect in that extreme case  that the amount of time it takes to crack a password with an old Pentium III CPU vs a new AMD Threadripper 3390x would be significantly lower.

 

Obviously Vision has been getting faster. However lighting, like Moore's law seems to be getting complex just as fast. With all the current multipart fixtures things have gotten big fast.

 

I can get over all of the trials and tribulations (see fixture orientation, dmx transforms and the like...) but allowing the hardware to scale "better" with hardware should be a must.

 

I hear by place my vote to let you guys focus on making the software "solve passwords better" haha. If you guys improve that portion of the software, then I can throw hardware at it. I think it is perfectly acceptable to accept that if I want it to run faster I just throw gear/money at it. Currently I don't feel I can do that with much success.

 

Then once all that settles we can all deal with MVR and GDTF. 🤣

 

(Steps off soapbox)

No you are completely right! My example was poorly formed when talking about passwords as neither of those examples represented how software scales. So getting right to the point as Vision itself is a good example:

(Note: I haven't tested this, strictly speaking. But this is pulling from my general experiences and memory.)

  • If you ran Vision 2018 on an old machine and ran it on a new machine, you may get a 50% increase in performance. I feel this is being generous, but perhaps I'm wrong.
  • If you ran Vision 2019 on an old machine and ran it on a new machine, you may get up to a 200% increase in performance. I expect given the right circumstances it could be much more (given how we leverage the number of textures the GPU supports).

One last thing to point out is that Vision 2019 gave you the ability to control the way performance scales. So, in Vision 2018 you were stuck with the 50% increase when you upgrade your machine. With Vision 2019+, you are given many options to control the quality vs performance. This can be used to offset the poor performance of the old machine to bring it more "inline" with the newer machine.

 

So all this being said, have things improved since 2018? Of course 😄 I don't think anyone is denying that.

 

But to your point, that next leap in performance sure would be nice 🙂 What makes this tough is performing analysis on current code, and then properly executing a new design without breaking existing functionality. I still truly believe from what I've seen, the renderer is running very well. We never truly optimized the physics engine and I believe it is to blame. It will be a major overhaul to get it to be more performant, but I think as we investigate more we will find this is the right course of action.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


 

7150 Riverwood Drive, Columbia, Maryland 21046, USA   |   Contact Us:   410-290-5114

 

© 2018 Vectorworks, Inc. All Rights Reserved. Vectorworks, Inc. is part of the Nemetschek Group.

×
×
  • Create New...