Defra Survey Data - LIDAR

[Rebecca R]

Hi,

 

I'd like to create a site model from Defra LIDAR survey Data (https://environment.data.gov.uk/DefraDataDownload/?Mode=survey) but I don't know what 'product' to chose?  Can anyone help please?

 

Alternatively, does VW Landmark 2023 have built in access to 3D terrain data maybe?

 

Thanks!

[Jeff Prince]

3 hours ago, Rebecca R said:

I'd like to create a site model from Defra LIDAR survey Data (https://environment.data.gov.uk/DefraDataDownload/?Mode=survey) but I don't know what 'product' to chose?  Can anyone help please?

 

 

It depends on the resolution and timeliness of the data.  It seems that "National LiDAR Program Point Cloud" at 1M resolution might be okay, depending on what you are doing and how accurate it needs to be.  You would then import that as a point cloud if it's in .laz, .pts, or similar point based formats you can see in the import point cloud command.

 

Keep in mind, government sourced LiDAR and GIS are generally far from "survey grade" data, meaning they are okay for starting a project, but if you need high accuracy... you will eventually need to get a proper survey done. Then, request a point file in .CSV format from your surveyor to get the best data into Vectorworks.

 

Hopefully one of your UK colleagues like @Tom W..  

 

 

3 hours ago, Rebecca R said:

Alternatively, does VW Landmark 2023 have built in access to 3D terrain data maybe?

 

Nope, you have to import it in a suitable format and build a site model with said data.

[Rebecca R]

59 minutes ago, Adios amigos said:

 

It depends on the resolution and timeliness of the data.  It seems that "National LiDAR Program Point Cloud" at 1M resolution might be okay, depending on what you are doing and how accurate it needs to be.  You would then import that as a point cloud if it's in .laz, .pts, or similar point based formats you can see in the import point cloud command.

 

Keep in mind, government sourced LiDAR and GIS are generally far from "survey grade" data, meaning they are okay for starting a project, but if you need high accuracy... you will eventually need to get a proper survey done. Then, request a point file in .CSV format from your surveyor to get the best data into Vectorworks.

 

Hopefully one of your UK colleagues like @Tom W..  

 

 

 

Nope, you have to import it in a suitable format and build a site model with said data.

Thanks @Adios amigos, that's really helpful.  No problem about the accuracy at this stage.  I've been experimenting with point cloud data - I haven't been sure what % to import, how many points to use to make the site model, and how to georeference it.  Any pointers?  Is there a webinar recording on this anywhere that you know of?

Edited August 9, 2023 by Rebecca R

[Jeff Prince]

28 minutes ago, Rebecca R said:

Thanks @Adios amigos, that's really helpful.  No problem about the accuracy at this stage.  I've been experimenting with point cloud data - I haven't been sure what % to import, how many points to use to make the site model, and how to georeference it.  Any pointers?  Is there a webinar recording on this anywhere that you know of?

 

My pleasure.  As you are probably aware, a point cloud is a bunch of 3D points defining a surface.  LiDAR at 1M resolution typically has one point per square meter.  So, for a small site, that isn't too much data, but for a 10km x 10km area, it could be exceptionally point dense and bring your computer to its knees.

 

That is where decimation can be handy during import.  You'll see the total number or points and can reduce it to an acceptable %, based on your computer's horsepower and experience.  I usually try 5% to see how quick a new data source will come in and increase the % on subsequent imports as I see how fast it it.  I recommend not having any files open when bringing in a point cloud for the first time.  Once you know it will behave nicely, you can then reference it into your working file or directly place it.  Just make sure you back up all your work before importing point clouds, you can sometimes run into problems with them that can cause a loss of work.

 

Tutorials... I can't remember if there are any on Vectorworks University or not, but that is the place to look.

I've posted a lot of information about site modeling here on the forum which you can find by searching my name as author.

 

Hopefully some others will jump in here, I'm taking a break from the forum for a while, hence the "adios amigos".

I will still be doing paid tutoring during my break and can be reached by private message here.

[Tom W.]

12 hours ago, Rebecca R said:

I'd like to create a site model from Defra LIDAR survey Data (https://environment.data.gov.uk/DefraDataDownload/?Mode=survey) but I don't know what 'product' to chose?  Can anyone help please?

 

I have never seen this website before but I did acquire some Environment Agency LiDAR data a few years ago from ProMap which I assume came from the same source (DEFRA). It came in two formats - a TIN (mesh) + 3D Polygons - and I think this might be the 'LIDAR Composite DTM 2022 1m Index' option on your website ('DSM' will include trees, buildings, etc so don't choose this one: 'DTM' will just be the terrain data). The 3D Polygons can be used directly as source data for a site model but I believe the TIN data will be more accurate (best to run '3D Polygons to 3D Loci...' to convert to points then 'Validate 3D Data' to remove any duplicates).

 

I use Point Clouds for modelling buildings but have never created a Site Model from one. I assume this would require the use of 'Extract 3D Loci from Point Cloud...' to convert it to source data...?

 

On Georeferencing Point Clouds I personally have found it very problematic. If I import a Point Cloud into a Georeferenced file the data comes in at the correct location but half of it is missing (it has pronounced linear bands where points are missing). The only way I have been able to do it is import it into a non-georeferenced file then manually move it to the correct geographical location (which requires my friendly surveyor to place a datum point in the point cloud with known coordinates). I brought this up with VW but didn't get a response + no one here said anything so it's all a bit of a mystery. But it is definitely the case in my experience that point clouds only import properly if centred on the User Origin. Seems crazy that this should be the case. 

 

Anything else - 3D Loci, 3D contours, TIN - all come into a georeferenced file perfectly however.

[Rebecca R]

Tha

3 hours ago, Tom W. said:

 

I have never seen this website before but I did acquire some Environment Agency LiDAR data a few years ago from ProMap which I assume came from the same source (DEFRA). It came in two formats - a TIN (mesh) + 3D Polygons - and I think this might be the 'LIDAR Composite DTM 2022 1m Index' option on your website ('DSM' will include trees, buildings, etc so don't choose this one: 'DTM' will just be the terrain data). The 3D Polygons can be used directly as source data for a site model but I believe the TIN data will be more accurate (best to run '3D Polygons to 3D Loci...' to convert to points then 'Validate 3D Data' to remove any duplicates).

 

I use Point Clouds for modelling buildings but have never created a Site Model from one. I assume this would require the use of 'Extract 3D Loci from Point Cloud...' to convert it to source data...?

 

On Georeferencing Point Clouds I personally have found it very problematic. If I import a Point Cloud into a Georeferenced file the data comes in at the correct location but half of it is missing (it has pronounced linear bands where points are missing). The only way I have been able to do it is import it into a non-georeferenced file then manually move it to the correct geographical location (which requires my friendly surveyor to place a datum point in the point cloud with known coordinates). I brought this up with VW but didn't get a response + no one here said anything so it's all a bit of a mystery. But it is definitely the case in my experience that point clouds only import properly if centred on the User Origin. Seems crazy that this should be the case. 

 

Anything else - 3D Loci, 3D contours, TIN - all come into a georeferenced file perfectly however.

Hi @Tom W., thanks for responding.

 

I've downloaded the  'LIDAR Composite DTM 2022 1m Index' option but I'm not sure which bits to import or how (see attached images for options).  Are you able to advise please?

 

And yes, I've had similar experiences with Georeferencing Point Clouds so far, although this has been the only Defra data that I've actually successfully made a model of so far!

 

Thanks.

[Rebecca R]

11 hours ago, Adios amigos said:

 

My pleasure.  As you are probably aware, a point cloud is a bunch of 3D points defining a surface.  LiDAR at 1M resolution typically has one point per square meter.  So, for a small site, that isn't too much data, but for a 10km x 10km area, it could be exceptionally point dense and bring your computer to its knees.

 

That is where decimation can be handy during import.  You'll see the total number or points and can reduce it to an acceptable %, based on your computer's horsepower and experience.  I usually try 5% to see how quick a new data source will come in and increase the % on subsequent imports as I see how fast it it.  I recommend not having any files open when bringing in a point cloud for the first time.  Once you know it will behave nicely, you can then reference it into your working file or directly place it.  Just make sure you back up all your work before importing point clouds, you can sometimes run into problems with them that can cause a loss of work.

 

Tutorials... I can't remember if there are any on Vectorworks University or not, but that is the place to look.

I've posted a lot of information about site modeling here on the forum which you can find by searching my name as author.

 

Hopefully some others will jump in here, I'm taking a break from the forum for a while, hence the "adios amigos".

I will still be doing paid tutoring during my break and can be reached by private message here.

Many thanks @Adios amigos, this is all really helpful information.  Thanks for taking the time to help 🙂 .

[Tom W.]

14 minutes ago, Rebecca R said:

Are you able to advise please?

 

Sorry I have no idea! But Jeff/Adios will know I'm sure. I'd probably try the Shapefile (.shp) first... I believe the .shx, .dbf + .prj files are associated shapefiles? My data has always been in .dwg format... or .txt. Tell us what you discover! This is free data right?

[Rebecca R]

40 minutes ago, Tom W. said:

 

Sorry I have no idea! But Jeff/Adios will know I'm sure. I'd probably try the Shapefile (.shp) first... I believe the .shx, .dbf + .prj files are associated shapefiles? My data has always been in .dwg format... or .txt. Tell us what you discover! This is free data right?

Yes, free, so at least it's not costing anything to experiment!

 

I was only able to import the .shp files as shapefiles and they seem to contain regular polygons that don't make much sense on their own. @Adios amigos, are you able to help with this before you adios!?

 

I'm attaching images of the site models I've been able to create with a section of the point cloud data (less than 200m2) - as you can see, it could do with some smoothing and it contains house, hedge and tree levels too which isn't ideal - not sure what to do about any of this!

 

Any advice gratefully received!

[Tom W.]

9 minutes ago, Rebecca R said:

I was only able to import the .shp files as shapefiles and they seem to contain regular polygons that don't make much sense on their own.

 

Can you use Tools > Records > Modify by Record... > Elevate 2D Polys? The geometry should have Record attached with elevation data.

[Rebecca R]

30 minutes ago, Tom W. said:

 

Can you use Tools > Records > Modify by Record... > Elevate 2D Polys? The geometry should have Record attached with elevation data.

Nope, sadly it didn't work.

[Rebecca R]

@Katarina Ollikainen do you use the Defra LIDAR survey data?  If so, I'd be extremely grateful if you can advise please?

[Amanda McDermott]

1 hour ago, Rebecca R said:

re: 'it contains house, hedge and tree levels too which isn't ideal - not sure what to do about any of this!'

 

Is there a DSM download option? Digital Surface Model, should give you just the surface (I think), rather than DTM = Digital Terrain Model

 

 

 

[Rebecca R]

31 minutes ago, Amanda McDermott said:

 

I think it's the other way round though isn't it, and in any event - there's only one point cloud option unfortunately.  Thanks though.

Edited August 10, 2023 by Rebecca R

[Amanda McDermott]

1 minute ago, Rebecca R said:

I think it'd the other way round though isn't it, and in any event - there's only one point cloud option unfortunately.  Thanks though.

Ah sorry, yes I've got mixed up. DTM or DEM should give bare-earth. DSM should include trees and buildings etc. Sounds like they've labelled it wrong!

[Tom W.]

1 hour ago, Rebecca R said:

Nope, sadly it didn't work.

 

Hmm I tried downloading a couple of .shp files but they just came in as blank polygons (a single or pair of rectangles). Perhaps that's because there's no data for the areas I chose...? The only thing I could download were, like you, point clouds but because they include all the surface features they aren't much use.

 

If you go on Promap website you can see they sell the same EA LiDAR data in a variety of different formats. There are sample files you can download:

 

https://www.promap.co.uk/maps-and-data/promap-bim/lidar-height-data/

 

Perhaps the only option is to pay for it if you want something usable...? Although be great if you discover otherwise.

 

[Rebecca R]

25 minutes ago, Tom W. said:

 

Hmm I tried downloading a couple of .shp files but they just came in as blank polygons (a single or pair of rectangles). Perhaps that's because there's no data for the areas I chose...? The only thing I could download were, like you, point clouds but because they include all the surface features they aren't much use.

 

If you go on Promap website you can see they sell the same EA LiDAR data in a variety of different formats. There are sample files you can download:

 

https://www.promap.co.uk/maps-and-data/promap-bim/lidar-height-data/

 

Perhaps the only option is to pay for it if you want something usable...? Although be great if you discover otherwise.

 

Thanks for trying.  Good to know it's not just me.  After a bit of digging, it looks like they (Defra) do supply DWG etc files but only if you're one of their 'associates'.  Maybe Promap is the answer, or I might be able to extract something from point cloud model - bit of a faf though.

Edited August 10, 2023 by Rebecca R

[Jeff Prince]

4 hours ago, Rebecca R said:

Yes, free, so at least it's not costing anything to experiment!

 

I was only able to import the .shp files as shapefiles and they seem to contain regular polygons that don't make much sense on their own. @Adios amigos, are you able to help with this before you adios!?

 

I'm attaching images of the site models I've been able to create with a section of the point cloud data (less than 200m2) - as you can see, it could do with some smoothing and it contains house, hedge and tree levels too which isn't ideal - not sure what to do about any of this!

 

Any advice gratefully received!

 

If you send me links to the specific datasets you want to use, I can tell you how to manipulate them in the next day or so.  It's possible some of the .shp files you downloaded are only polygons representing the rectangular tiles to be downloaded, not very useful for your purposes, but typical with government data to delineate collection areas or tiles completed... not the actual data 🙂

Anyhow, if you make life easy on me for looking at the specific data sets you want to use and project limits, I can give you some pointers.

 

About the images you posted.

The point cloud that has houses and vegetation in it needs to be edited if a ground surface only version of the file is not available as @Amanda McDermott was suggesting.  The easy way to do this is with Agisoft Metashape with offers tools to expedite this kind of thing, especially if the vegetation and structures can be classified and filtered.  However, you can manually delete points from the cloud in Vectorworks.  Using the clip cube to isolate areas for editing and having a copy of the incorrect site mode in the background can make this much easier, but It's a laborious task if you are doing large areas.

 

[Rebecca R]

1 hour ago, Adios amigos said:

 

If you send me links to the specific datasets you want to use, I can tell you how to manipulate them in the next day or so.  It's possible some of the .shp files you downloaded are only polygons representing the rectangular tiles to be downloaded, not very useful for your purposes, but typical with government data to delineate collection areas or tiles completed... not the actual data 🙂

Anyhow, if you make life easy on me for looking at the specific data sets you want to use and project limits, I can give you some pointers.

 

About the images you posted.

The point cloud that has houses and vegetation in it needs to be edited if a ground surface only version of the file is not available as @Amanda McDermott was suggesting.  The easy way to do this is with Agisoft Metashape with offers tools to expedite this kind of thing, especially if the vegetation and structures can be classified and filtered.  However, you can manually delete points from the cloud in Vectorworks.  Using the clip cube to isolate areas for editing and having a copy of the incorrect site mode in the background can make this much easier, but It's a laborious task if you are doing large areas.

 

Hi, thank you so much!  I've send you a message with a zip attached hopefully.  I can work something out with the point cloud if need be, but it would be very helpful to know if anything can be done with these other files for future reference as much as anything.

[Jeff Prince]

4 hours ago, Rebecca R said:

Hi, thank you so much!  I've send you a message with a zip attached hopefully.  I can work something out with the point cloud if need be, but it would be very helpful to know if anything can be done with these other files for future reference as much as anything.

 

I received your message,  but the data was not point cloud data, it was GIS data depicting the area your are interested in, boundaries and descriptors for aerial surveying done to collect lidar, and a geoTIFF DEM.  DEMs are raster heightfield information where the greyscale value of a pixel in the image determines elevation.  Vectorworks does not have a method of using that type of data.  Blender and Rhino do, but you would be better served collecting the actual point cloud files, typical .las, .laz, or .pts.  These point cloud files describe geolocated points in 3D space, which Vectorworks does understand.

 

Here's an example of what you can do with raster data in appropriate software:

 

Landscape Visualizations

 

[Tom W.]

16 hours ago, Rebecca R said:

Maybe Promap is the answer, or I might be able to extract something from point cloud model - bit of a faf though.

 

It's got to be worth finding out how much it would cost given the amount of work cleaning up the point cloud would involve... 

[Rebecca R]

9 hours ago, Adios amigos said:

 

I received your message,  but the data was not point cloud data, it was GIS data depicting the area your are interested in, boundaries and descriptors for aerial surveying done to collect lidar, and a geoTIFF DEM.  DEMs are raster heightfield information where the greyscale value of a pixel in the image determines elevation.  Vectorworks does not have a method of using that type of data.  Blender and Rhino do, but you would be better served collecting the actual point cloud files, typical .las, .laz, or .pts.  These point cloud files describe geolocated points in 3D space, which Vectorworks does understand.

 

Here's an example of what you can do with raster data in appropriate software:

 

Landscape Visualizations

 

Hi Jeff,

 

I've sent the point cloud data now as well.  The info you give about the files I sent is really helpful.  My partner uses Blender so I'll see if he can do anything with it - he doesn't usually work with this sort of data but you never know!

 

If you can give me any pointers on tidying up my point cloud data it would be much appreciated.

 

Again, many thanks for your help, and I'm sorry you'll be leaving the forum!

Edited August 11, 2023 by Rebecca R

[Rebecca R]

7 hours ago, Rebecca R said:

Hi Jeff,

 

I've sent the point cloud data now as well.  The info you give about the files I sent is really helpful.  My partner uses Blender so I'll see if he can do anything with it - he doesn't usually work with this sort of data but you never know!

 

If you can give me any pointers on tidying up my point cloud data it would be much appreciated.

 

Again, many thanks for your help, and I'm sorry you'll be leaving the forum!

UPDATE:  We have had no joy using Blender to convert the files @Adios amigos - I think this is beyond our expertise!

[Jeff Prince]

14 hours ago, Rebecca R said:

Hi Jeff,

 

I've sent the point cloud data now as well.  The info you give about the files I sent is really helpful.  My partner uses Blender so I'll see if he can do anything with it - he doesn't usually work with this sort of data but you never know!

 

If you can give me any pointers on tidying up my point cloud data it would be much appreciated.

 

Again, many thanks for your help, and I'm sorry you'll be leaving the forum!

 

I just read your message.  Unfortunately, there was no lidar, .vwx file, or project coordinates attached or in the link provided, so I'm unable to review the site.

 

I did look at the lidar available in what I suspect is your vicinity of interests.  It looks like the 2019 data is most comprehensive.  I was able to pull that down and generate a site model of a fairly large area using 4 different point clouds.  This is good because it tells you the data is there and it's possible to do something with it.  Here's some of the vicinity brought into Vectorworks.

 

 

 

So, 1st you import the point cloud.  It seems the data either lacks geographic location or Vectorworks is not importing it correctly.  This is where those .shp files of the LiDAR study area are handy, as those do come into their real world position correctly, so import those first.  Then, you can move the point cloud(s) in Top/Plan view to line up with the geometry of .shp files and/or the GIS aerial image built into Vectorworks.  Using top/plan for this move prevents you from snapping to an incorrect elevation and preserving the elevation in the LiDAR.

 

Once you have the LiDAR situated correctly, you can convert it to a site model by selecting it and then issuing the "create site model from data" tool.  You can choose the number of points to import, which is essentially decimates the point cloud for you and takes out a large number of those trees sticking up in the air.

 

Finally, edit this site model by "recreate from source data".  This will give you access to the points.   Using the clip cube to work small sections at a time, you can delete points that you do not wish to use.  After you are done, the site model will be rebuilt.  Repeat the process as required and your model will take on a smoother appearance.

 

In terms of using Blender with geographic point clouds, there is a lot of information on the blender sites and YouTube about Blender GIS and LiDAR Decimation if you want to learn more.  Just be warned, this requires installing add-ons and a higher comfort level with Blender.

 

Hopefully @Katarina Ollikainen or @Tom W. can help you out, I'm going on break for a bit 🙂

 

 

 

[Rebecca R]

5 hours ago, Jeff Prince said:

 

I just read your message.  Unfortunately, there was no lidar, .vwx file, or project coordinates attached or in the link provided, so I'm unable to review the site.

 

I did look at the lidar available in what I suspect is your vicinity of interests.  It looks like the 2019 data is most comprehensive.  I was able to pull that down and generate a site model of a fairly large area using 4 different point clouds.  This is good because it tells you the data is there and it's possible to do something with it.  Here's some of the vicinity brought into Vectorworks.

 

 

 

So, 1st you import the point cloud.  It seems the data either lacks geographic location or Vectorworks is not importing it correctly.  This is where those .shp files of the LiDAR study area are handy, as those do come into their real world position correctly, so import those first.  Then, you can move the point cloud(s) in Top/Plan view to line up with the geometry of .shp files and/or the GIS aerial image built into Vectorworks.  Using top/plan for this move prevents you from snapping to an incorrect elevation and preserving the elevation in the LiDAR.

 

Once you have the LiDAR situated correctly, you can convert it to a site model by selecting it and then issuing the "create site model from data" tool.  You can choose the number of points to import, which is essentially decimates the point cloud for you and takes out a large number of those trees sticking up in the air.

 

Finally, edit this site model by "recreate from source data".  This will give you access to the points.   Using the clip cube to work small sections at a time, you can delete points that you do not wish to use.  After you are done, the site model will be rebuilt.  Repeat the process as required and your model will take on a smoother appearance.

 

In terms of using Blender with geographic point clouds, there is a lot of information on the blender sites and YouTube about Blender GIS and LiDAR Decimation if you want to learn more.  Just be warned, this requires installing add-ons and a higher comfort level with Blender.

 

Hopefully @Katarina Ollikainen or @Tom W. can help you out, I'm going on break for a bit 🙂

 

 

 

Hi Jeff,

 

I replied to your message before reading this.  You've been really clear and I feel like I've got a good enough grasp of what needs doing now.  Thank you!  Great that this is on record on the forum for others now too.

 

Enjoy your break, you will be missed!

 

Rebecca