AI / ChatGPT and VectorScript

[Robert Anderson]

I was curious to see what ChatGPT knew about VectorScript. (SInce "VectorScript", unlike "Python", is a pretty unambiguous language name.) I prompted the following: "Write a Vectorworks plug-in using the VectorScript language that creates a parametric pyramid of any size." Here's what it produced (in under a minute):

 

Now, the code is rubbish, of course. It gets its inputs a little mixed up as to types, it seems to confuse the Str2Num() with RealDialog() calls, and it has a little trouble with the sequence of Cartesian coordinates in the Poly() call, among other things. The code won't run, but interestingly, to a casual reader, it kind of looks like a real VectorScript. 

 

When I was a kid, back in the mid 60s (pre-historic,) I read a book called Danny Dunn and the Homework Machine by Abrashkin and Williams about a precocious boy who thinks he can get a computer (mainframe at the time) to do his homework for him. ("Lazy programmer", indeed!) Turned out to be more work to write the code to do the work than just to do the work. I'm not sure why, but looking at the code that ChatGPT produced made me think of that kid-lit.

 

[Sam Jones]

21 hours ago, Robert Anderson said:

Turned out to be more work to write the code to do the work than just to do the work.

Still usually true.  If you don't do it a lot, don't bother to script it.  (Unless you have fun scripting.  See @Pat Stanford 's many script replies.)  As a third party developer, creating the command used occasionally by many is close enough.

 

The ChatGPT is well laid out and clear.  I probably would not thought of ExtrudeAlongPath() function.  I'm curious to see what it produces, after all the errors you mention are corrected.  Although ChatGPT does not seem like it is of any use (yet) for templating (a word?) any mildly complex commands, I will be curious if it can throw up a structure for some of the simple commands that I sometimes throw together.

[Scott C. Parker]

21 hours ago, Robert Anderson said:

the code is rubbish, of course.

Have you tried giving ChatGPT an example script and then ask it to make a new script for a new function based on the sample you feed it?
Something like, Here's an example. Please write me a new one to do X based on the example.

[Sam Jones]

No.  Have never used ChatGPT.  I don't know how to ask it to do anything or where to find it.  That will be more work than anything else.

[Robert Anderson]

32 minutes ago, Scott C. Parker said:

Have you tried giving ChatGPT an example script and then ask it to make a new script for a new function based on the sample you feed it?
Something like, Here's an example. Please write me a new one to do X based on the example.

 

There is nothing wrong with this approach, but I wonder if it would truly save time. You'd pass it an example script, then get a result, then fix and debug. It might work for small changes, but then small changes are easy to do manually—a Catch-22 to be sure. I suspect that the universe of VectorScripts is just too small for ChatGPT to really gain any expertise in the syntax. (Unlike web code and plug-ins for big sites like Wordpress, for which I've read ChatGPT can do competent plug-ins out of the gate.)

[Christiaan]

ChatGPT-4 still spits out rubbish. Funny how it does it so confidently. And it'll just keep trying each time you tell it it's rubbish.

 

Presumably though we can teach it to write it properly within the confines of a single conversation? Perhaps also Marionette. Are there any documents explaining Vectorscript or Marionette that we could simply paste into ChatGPT?

[Robert Anderson]

Here is a good discussion of LLMs (Large Language Models) that suggests indirectly why they won't be successful with, say, Vectorscript. (Hint, they'er not "lossless.")

[Christiaan]

That would explain why it spits out rubbish Vectorscript so confidently. But could we still overcome this by feeding it the lossless version within the confines of a single chat? I suspect not but I don't know.

 

One thing I've experienced is that even if you correct an error it will apologise but can repeat the error later in the same chat, presumably because it's falling back on the snapshot of data it was trained on. As an example it was implying to me that the Code for Sustainable Homes in the UK (withdrawn in 2015) was still relevant. When I corrected it, it apologised but fell back on the error later in the chat. Weirdly if you ask it directly what CfSH is it will correctly tell you that it was withdrawn on in 2015. Only certain lines of discussion would trigger the error.

[Christiaan]

This is what we need apparently:

 

I've also attached an example of priming ChatGPT to write code. In this example it's being taught to write prompts for Midjourney. Not nearly the same level of accuracy required as Vectorscript but it's example of how to paste a series of prompts that will train, provide examples. We'll just need some further prompts to correct errors.

 

gpt4_midjourney_priming.txt

[Jeremy Best]

On 2/18/2023 at 8:45 AM, Robert Anderson said:

I suspect that the universe of VectorScripts is just too small for ChatGPT to really gain any expertise in the syntax.

This was my expectation too and it has proven to be true again and again. When ChatGPT plug-ins are available to all, then perhaps access to the internet will help, but I think it will only read individual pages you point it to. But even then, the relatively small number of examples out there will still be a factor. 

 

@Christiaan, If you have any greater success than you've already shared, I'd been keen to hear from you as I've been trying to use OpenAI's ChatGPT as an assistant for VectorScript quite a lot. Unfortunately, so far I've found @Robert Anderson's parable to be true in my use-cases, but it has been most valuable for getting quick answers to 'beginner' level questions.