Jump to content
Malcolm Woodruff

Lidar data for site design

Recommended Posts

HI. I often use Lidar data for site design. This comes as a set of elevation data arranged in a grid eg. a grid 500m x 500m with elevation points at 2m centres. In order to take these into VW I have to assign an x and y coordinate to each point. It would be preferable to read the array in directly. Can this be done? I am new to VW so have not experience of writing scripts for it. Any suggestions wold be most welcome.

Share this post


Link to post

IDK about your data but I usually have .shp files to work with (or sometimes .dem or .laz/s files  - but VW can't handle  those afaik -I'm on 2012). My .shps come it with no hassles, and none of what you're talking about. Are you able to share a file  or tell us the file extension you have (as there are dozens of possibles).

 

Are you bringing them in to generate contours from?

Share this post


Link to post

I've used Lidar data + you just need to import it into a georeferenced file + there it is: lots of 3D polygons at the correct elevation + grid reference, ready to make a site model from.

700532697_Screenshot2021-04-23at07_11_59.thumb.png.7bdee0952fd82a97fbb5cf438d45f703.png

Or is it something else you're looking to do? 

Share this post


Link to post

I have attached a file showing the data in the normal format I receive it. I have, however, had to change the filetype ending from .asc to.txt so that I can upload it. Once in, I will generate contours or use to generate a site model to calculate volumes etc.

Tom. What file type do you bring your polygons in on?

sz6094_DSM_2M.txt

Share this post


Link to post

I do most of this data munging in a program called LASTools and/or CloudCompare.   .asc files are limited, as it says in the below CloudCompare or sim is needed to convert to something VW finds digestible. I've used CloudCompare, it's a learning curve but immensely useful.

 

IDK what VW version you have but:

"Vectorworks 2020 still cannot directly import point cloud files in the .ASC file format. You’ll have to use CloudCompare (or something similar) to convert the .ASC files to one of the accepted formats prior to importing them into Vw. " 

 

Share this post


Link to post

@Malcolm Woodruff the 3D polygons come in .dwg format.

 

I have also had survey data in .txt format where the surveyor has been unable or unwilling to supply 3D geometry but this is a bit more work because you have to edit the text to clean it up before converting it into 3D data.

 

I opened your .txt file but wasn't really sure what I was looking at:

1707702053_Screenshot2021-04-23at08_47_28.thumb.png.636258b80576da02df0a389d3746777a.png

This is the kind of thing I've had:

1577114661_Screenshot2021-04-23at08_48_13.thumb.png.5de76c3066b081605954bd91197344b0.png

But in this case I had to remove the 'PSL,' text from each line before I could convert it into 3D data which was a pain to say the least.

Share this post


Link to post

Ok. That was possibly a poor example. -9999 is a value where there is no data. Normally it means that it is sea or a lake. I have attached another which shows the elevation data better.  The header lines gives the coordinates of the bottom left corner of the grid , the nrows and ncols indicates it is an array of points  500x500 at cell size = 2m crs. Then the elevations are at each of those points but do not have position data except by where they sit in the array.

Yes, Tom I sometimes get data in the format you have shown. or at least xyz description.

 

I am running Designer 2021 with latest service packs on 2014 5J 27 inch iMac with 4 core I7 and 32GB of RAM.

sz6090_DSM_2M.txt

Share this post


Link to post

Aha I see sorry I'm way out of my depth on this. The Lidar data I had must have been converted into 3D polygons by the company that supplied it: I was ignorant of the .asc format.

 

But the threads that @unearthed pointed towards seem to be discussing exactly the kind of data you're talking about:

 

Share this post


Link to post

@Tom W. Did you remove them manually?!

 

I use the text editor VSC (Visual Studio Code) > select a text string I want to edit  > Ctrl F2 and bingo! it's gone. Worth learning just enough command line tools, I use them now every day, cmd (was dos), some regex, a little Perl, and VSC. Some learning curve but they make things possible that were not.

 

With various heightfiles I also use LasTools as a plugin in QGIS, and Cloud Compare mentioned above.

 

This is complicated / difficult because:

• Many formats predate GUI, and GUI people find cmd  too hard

• GUI's take >>more RAM than cmd so your desktop locks up

• Some vendors (Autodesk, Vectorward and others) either design incompatibility in, or simply don't care

• Some formats are simply very different from each other

• what landscape architects want to do with the data is outside the design/thinking parameters of the data format designers

  • Like 2

Share this post


Link to post

The grid data entry tool is close to what I want. It would have to be modified so you could select the bottom left corner rather than just the top left and the elevations need to be read from a file rather than entered manually. A typical file may have 250000 to 1m points! I have taken the asc file into Mathematica and created a new file of xyz points from it. However, reading this into VW seems incredibly slow. I have also started to look at Cloud Compare. Thank you Unearthed for the reference. It looks very interesting.

Share this post


Link to post
7 hours ago, unearthed said:

@Tom W. Did you remove them manually?!

 

I use the text editor VSC (Visual Studio Code) > select a text string I want to edit  > Ctrl F2 and bingo! it's gone. Worth learning just enough command line tools, I use them now every day, cmd (was dos), some regex, a little Perl, and VSC. Some learning curve but they make things possible that were not.

YES I DID!! Thanks for the VSC tip I did think there must be a clever way to do it sounds complicated but if I ever get foisted with a .txt file again I will look into it - thank you. 

Hopefully though I can avoid surveyors who refuse to give me 3D data from now on + stick within my skill level!

Share this post


Link to post

@Tom W. Not complicated at all for basic VCS, just install it and either do Ctrl + F2 or Ctrl Shift L and presto! Attach shows where I've selected just the two double space segments in a piece of text; it'll do anything like that probably up to 100;s of thousands of lines.

 

Also has a line sorter which is great for species lists.

Case switches, leading and trailing whitespace tools, and multi-cursors which are magic as I build most of my VW functions in (IDK if new VW versions have drag like Excel) and drop them into the worksheet all at once.1150166121_VSCodemultiselect.thumb.JPG.0f831cdfc21feded74f3380ae3673005.JPG

  • Like 1

Share this post


Link to post

It wouldn't be too difficult to create a tool to read in the file's data and place loci at the elevations.

The file's formatting is relatively simple. There are six initial lines of formatting data followed by the lines of elevation data.

Each line begins and ends with a quotation character, and each number in the data lines are separated by a space.

The second line identifies how many rows of data will follow line 6.

The first line identifies how many numbers to expect in each line of data.

Lines three and four give the coordinate location of the lower left corner of the 2d array, (the first elevation number in line seven?) Could this be in meters? (yards or feet?) Line five: I'm not familiar with how cell size is utilized. Maybe it is the marker size, like the diameter of an elevation dot?

Line six tells you the placeholder number used for when no elevation data exists at that location.

A Vectorscript can be written to read in the file and say place a 3d locus at the z location indicated except where the placeholder number occurs. I suppose with a little more effort one could draw connecting lines between adjacent points too.

The script could lay 3d loci out at one metre increments with two REPEAT loops one embedded in the other. Reading in one number at a time from the file.

You would have to determine if each line of data between the quotes represents a row or a column, and confirm that the first digit in line seven represents the elevation data for the lower left corner. It could be the top left for instance, but less likely.

 

  • Like 1

Share this post


Link to post

Larry0 I agree. It should be fairly easy to write a script that reads the file. However, I have no experience of Vectorscript so do not know where to start. Can you point me in the right direction to where I can learn it? The cell size is the distance in metres between points. The cell size normally is either 2m, 1m or 0.25m and the tile size is 1000m x 1000m so the umber of points will vary from 250,000 to 2million. Can VW cope with this number of points?

Share this post


Link to post

VW will not deal well with that many loci. The time lag in drawing all fo the selection handles will kill you. If you can put up with the slow time once and import/create the loci and then make a Group (which only have 4 selection handles to redraw). then you might have something workable for a background image. 

 

If you need to import the loci and then manually work with them, you will probably be unhappy.

 

For a quick test of what I mean, close any other file you are working in. Create a 3D loci and leave it selected. Go to Edit menu and Duplicate Array, Rectangular Array, 1000 columns, 1000 rows, 1 Stack. Tell it yes when it asks if you want to create 1M objects.

 

On my 16" MacBook Pro i9 processor and 32Gig RAM, I gave up after about 20 minutes. I don't know if VW was hung or just taking a very long time, but certainly generating a million loci was Not a Good Thing™.  Which is why I told you to close all the other files first.  ;-)

 

Share this post


Link to post

As Pat says, VW will choke on that many raw points expressed as individual loci.

I was able to generate 250,000 loci in a few minutes but only by creating and grouping them in blocks of 10,000 (100x100 array) and duplicating that block into a 5x5 array of blocks.

I could post a simple layout of a Pascal procedure but it wouldn't be of much use if you have no programming experience. The following is a portion of a Pascal Procedure  and will probably not do anything at best but more likely generate an error because the code is incomplete. Probably an example of poor coding structure too.

 

PROCEDURE Beam_End;

TYPE

BEAMFORMAT=STRUCTURE

Beam: STRING;

d,w,wb,tw,tf,rf: REAL;

{.d = beam depth, .w = beam width, .tw = thickness web,}

{.tw = thickness top flange, .rf = radius flange, .wb = thickness bottom flange}

END;

 

VAR {declarations}

CopeBeamSize,BeamSize: BEAMFORMAT;

PIO_Handle,PIO_Record,PIO_ParentWall,hBeam,CopeBeam_Handle,hDim: HANDLE;

TopET,BttmET,NumOfHoles,unitStyle,dimFormat,angPrec,A3: INTEGER;

prec,dimPrec,A1,A2:LONGINT;

LayerScaleFactor,FileUnits,CopeTop,CopeBttm,DimStrOffset,Y1,Y2,

X_HolesCL,YHoleSpacing,FirstHoleOffset,GussetWidth,A4,rTemp1,rTemp2,HoleOfset: REAL;

FileName,Filepath,PIO_Name,A5,A6: STRING;

BoolResult,showMark,displayFrac: BOOLEAN;

 

BEGIN

{PROGRAMMING CODE GOES HERE}

 

{THE FOLLOWING IS A SLICE OF CODE FOR A PLUGIN I WROTE TO DRAW A COPING PROFILE FOR THE END OF A STEEL BEAM}

{HERE I WAS READING IN THE VW's BEAM DATA FILE LINE BY LINE TO POPULATE MY STRUCTURED VARIABLE WITH THE CHOSEN BEAM'S DIMENSIONS}

 

{Choose Metric or Imperial Beam Data File}

IF unitStyle>=7 THEN BEGIN

FileName:='StructShape_WFlange-Metric.txt';

BoolResult:= FindFileInPluginFolder (filename, filepath);

Units(7); END

ELSE BEGIN

FileName:='StructShape_WFlange-Inch.txt';

BoolResult:= FindFileInPluginFolder (filename, filepath);

PrimaryUnits(2,4,4,3,angPrec,showMark,TRUE); END;

 

IF BoolResult THEN BEGIN   {beam data file found/not found}

 

{Extract Beam Data}

Open(Concat(filepath,filename));

IF NOT EOF(filename) THEN ReadLn(fileunits);

IF fileunits=1 THEN

WHILE BoolResult & NOT EOF(filename) DO BEGIN

ReadLn(BeamSize.beam,BeamSize.d,BeamSize.w,BeamSize.tw,BeamSize.tf,BeamSize.rf,BeamSize.wb);

IF BeamSize.beam=PBeam_SizeI THEN BoolResult:= FALSE; END

ELSE

WHILE BoolResult & NOT EOF(filename) DO BEGIN

ReadLn(BeamSize.beam,BeamSize.d,BeamSize.w,BeamSize.tw,BeamSize.tf,BeamSize.rf,BeamSize.wb);

IF BeamSize.beam=PBeam_SizeM THEN BoolResult:= FALSE; END;

Close(Concat(filepath,filename));

 

END;

RUN(Beam_End);

Share this post


Link to post

Thank you Pat and Larry0. I was afraid that such large arrays may choke VW. I will have to play with the data outside VW to get it down to something more manageable. Thank you Larry for the Pascal clip. It has been many years since I wrote in Pascal but always liked it as a programming language.

Share this post


Link to post

Much better to use something that's designed to do this, or if you must at least thin the dataset first, even autocad dislikes handling such huge datafiles, horses for courses and vw is the wrong horse for munging huge amounts of data.

Share this post


Link to post

@Malcolm Woodruff can you request the data in a different format?

 

If not, Blender with the "blender-addon-photgrammetry-importer" is a nice free tool.

If you use a lot of public data like DEM, Lidar, and Google Earth, Blender should be in your toolset.  Pretty amazing how far it has come in recent years.

 

If you are just chasing this for the love of programming, I am of no assistance other than to cheer you on 🙂

Share this post


Link to post

@Malcolm Woodruff ,

   If you want to see what your data looks like in VW, I've attached a VW 2021 file below with the data imported from your file above. Since I don't know the finer details of the LIDAR file format, I made some assumptions, such as the X values increase from left to right, and the Y values increase from top to bottom as you read your file. If I got the directions wrong, the map may be flipped from what is shown. Also, this file is NOT georeferenced, so you may need to do some more editing.

 

   As to graphic performance, it actually navigates faster in VW 2009, but most people probably don't have the older versions loaded. It is possible that it will perform better on your machine than mine as I'm running a macMINI from 2012 with onboard graphics chips (which is nothing I should be bragging about.) That said, it works well enough for what I currently need. 

 

   Your file only contains 250K Loci, so if you plan to view larger data sets, be warned, it only gets slower.

 

HTH,

Raymond

 

sz6090_DSM_2M v2021.vwx

 

  • Like 2

Share this post


Link to post

Hi Malcolm,

   After looking at the text file you posted, I surmised all the values were Z values and the XY values were implied from the position of the numbers in the data array - 500 values per row, and 500 rows. I then cobbled together a quick script that generated a Locus3D() VectorScript command for each of the values. My assumption was that one row of data from the file had one Y-value and 500 X-values. The next row of data had the next Y-value and the same 500 X-values. Two FOR loops generated all of the intermediate values and assembled the Locus3D() statements. When done I had a text file with 250K lines which I imported into a blank VW file. If my assumptions about the X and Y positions are incorrect, then the Locus array may be flipped. Please inspect the file carefully before using the data.

 

   VW 2021 is a bit slow these days when importing large VS files, so I typically use VW 2009 to do the import, which is very quick in comparison, and then open the file in VW 2021. With a little more finesse the script could be written to directly create the Loci in VW, but speed issues become significant when large amounts of data are generated. This can always change and I haven't tested it recently, but it has been an issue for me in the recent past, which is why I prefer writing VS text files and importing them into older versions of the software. This approach is still faster than working directly in newer versions of VW. 

 

HTH,

Raymond

  • Like 1

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


 

7150 Riverwood Drive, Columbia, Maryland 21046, USA   |   Contact Us:   410-290-5114

 

© 2018 Vectorworks, Inc. All Rights Reserved. Vectorworks, Inc. is part of the Nemetschek Group.

×
×
  • Create New...