Importing x,y,z survey data

[Dexie]

I have have downloaded some survey data from the Emap site which comes as an x,y,z text file. The file is 16Mb, so V large for a text file.

There are approximately 650000 lines of data and I am having trouble importing it.

I did a test on the first 1000 lines of data which took 15minutes to complete this import. Which means it will take over 6.5 days to import all that data if it works.

Does anyone know why it takes so long. Vectorworks should be able to import thousands of pieces of data a second.

Looking forward to any comments.

[MullinRJ]

How are you importing the data? What format is it in?

[IanH]

Wow. That's a lot of data. Once its in, its going to be a pain to manage if its simply a set of coordinates with no structure.

When I do a Total Station survey, I normally create about 400-500 sets of coordinates. Importing as comma delimited "ID, Easting, Northing, Elevation, Description" takes no more than a few seconds.

What format do you think the file is in and what command are you importing it in? I wonder if you are trying to import an ascii dxf file as x,y,z coordinates.

Maybe you could post a couple of lines of data?

[Ozzie]

BIM at work?

[Dexie]

I am importing the data using Landmark > Import survey file.

The data covers approx 1.5Km square and data is at 5m intervals.

I normally get the data as dxf. but thought I would try the xyz data as it was a lot cheaper.

The dxf data works well. It's just the conversion that is causing the problem.

[Dexie]

This is how the data comes:-

372592.5,185037.5,93.66

372592.5,185032.5,93.681

372592.5,185027.5,93.698

372592.5,185022.5,93.695

372592.5,185017.5,93.69

372592.5,185012.5,93.685

372592.5,185007.5,93.68

372592.5,185002.5,93.675

372592.5,184997.5,93.669

372592.5,184992.5,93.664

372592.5,184987.5,93.659

372592.5,184982.5,93.639

372592.5,184977.5,93.616

[Kool Aid]

Dexie,

I find it hard to believe that the DTM-module of VW could handle 650000 points?

Looking at your data, they seem to be in a 5 m grid, which for such a large area may be redundantly dense.

The slowness of import is likely to be accentuated by the fact that the import command creates Stake objects, not simple 3D-loci. I should have an old script somewhere that creates just loci. If not, it's be easy to write one.

A script might, depending, also enlarge the grid by skipping lines. (A 10 m grid would have only a quarter of the points.)

[Dexie]

I have just created a terrain model from OS DTM data which are 3D loci containing 220000 points in 1 minute.

I want to create the 3D points from xyz data.

[Dexie]

You are right it is probably the fact that it is trying to create stake objects that is the problem.

I don't think it would have a problem with say 650000 3D loci.

[IanH]

I am importing the data using Landmark > Import survey file.

The data covers approx 1.5Km square and data is at 5m intervals.

I normally get the data as dxf. but thought I would try the xyz data as it was a lot cheaper.

The dxf data works well. It's just the conversion that is causing the problem.

5m intervals is 200 readings per linear KM is it not? So thats 300 x 300 readings, 90000 readings. Where does the 650000 come from?

[Kool Aid]

If it were a square, it'd be 16.25 km2.

SQRT(650000) = 806,225

806 x 5 = 4031

4,031 ^2 = 16,25

End of Practical Math 101 Lesson 1 for Garden Designers.

[Kool Aid]

You are right it is probably the fact that it is trying to create stake objects that is the problem.

I don't think it would have a problem with say 650000 3D loci.

Don't forget that the number of triangles in the TIN/DTM grows faster than the number of definition points.

EDIT

In our grid situation, not faster, but proportionally. Doubling the grid means reducing the 3D-poly count by half.

My original claim may well be incorrect. Have to think about it.

Edited December 1, 2009 by Kool Aid

[Dexie]

My mistake its 4Km x 4Km

[IanH]

My mistake its 4Km x 4Km

Phew. Thought my brain had suddenly forgotten basic maths. Still trying to work out how KA gets 16.25Km2. Thinks he is the one for a basic maths lesson because my calc also gives just over 4Km x 4Km for 650000 points too. Edited December 1, 2009 by IanH

[Dexie]

Got to go.

Daughters Birthday.

[Kool Aid]

My mistake its 4Km x 4Km

Phew. Thought my brain had suddenly forgotten basic maths. Still trying to work out how KA gets 16.25Km2. Thinks he is the one for a basic maths lesson because my calc also gives just over 4Km2 for 650000 points too.

4 x 4 = 16.

km x km = km2

End of lesson 2.

[IanH]

brain fart. Lesson noted. But I was still right in the first place that 1.5km x 1.5km didn't equal 650000 points.

[Kool Aid]

Then we were both right!

(Except that I was wrong in my claim regarding point/3D-poly number relationship: a rectangular grid is the worst case; in other configs, the number of polys increase slower than the number of points.)

[Kool Aid]

You are right it is probably the fact that it is trying to create stake objects that is the problem.

I don't think it would have a problem with say 650000 3D loci.

Well, Dexie, try this:

PROCEDURE ImportDTMData;  { ? Petri Sakkinen 2009 } 
CONST 
comma = ',';

VAR 
data, inputfile : STRING; 
x, y, z : REAL; 
commaPos : INTEGER; 

BEGIN
GETFILE(inputfile); 
IF NOT(DIDCANCEL) THEN BEGIN 
	WHILE NOT EOF(inputfile) DO BEGIN 
		READLN(data); 

		commaPos := POS(comma, data); 
		x := STR2NUM(COPY(data, 1, commaPos-1)); 
		DELETE(data, 1, commaPos); 

		commaPos := POS(comma, data); 
		y := STR2NUM(COPY(data, 1, commaPos-1)); 
		DELETE(data, 1, commaPos); 

		z := STR2NUM(data); 

		LOCUS3D(x, y, z); 
	END;	
	CLOSE(inputfile); 
END; 
END;

RUN(ImportDTMData);

If it is slow, you might use a text editor to replace commas with tabs and then use this:

PROCEDURE ImportDTMData;  { ? Petri Sakkinen 2009 } 

VAR 
inputfile : STRING; 
x, y, z : REAL; 

BEGIN
GETFILE(inputfile); 
IF NOT(DIDCANCEL) THEN BEGIN 
	WHILE NOT EOF(inputfile) DO BEGIN 
		READLN(x, y, z); 
		LOCUS3D(x, y, z); 
	END;	
	CLOSE(inputfile); 
END; 
END;

RUN(ImportDTMData);

Tab-separated data files are preferred.

For filtering/reducing number of points, add a READLN; after LOCUS3D(); so every second line is skipped. (The last point is not created then, though, but doing that manually is not a huge task!)

EDIT

Actually, the extra READLN; could be before the actual READLN(); because then you don't need to go to the end of the file to find the z-value!

Edited December 2, 2009 by Kool Aid

[go2greece]

DEXIE,

i have once imported 96 squarekilometers with a 5 by 5 grid

so dont worry .

first> the comma separated coordinates have to be imported and transfered into 3d points ( if you have difficulties with VW try DL-02 from sierra-soft , which should be free> creates a dxf file of the points).

i would split the VW file into hand-able portions like 100mX100m

> create a master-file and reference the parts

before you do so check the DATUM of the Emap WGS84 .....

it looks like your point are somewhere in africa CHAD???

peter

[go2greece]

petris's proposal is much more elegant

i have to admit

peter

[Dexie]

Thanks for all your efforts guy's. I will try it now.

I have no doubt it will success. I will you know shortly.

Cheers.

[Dexie]

That worked like a dream.

A special thanks to you Petri.

This is a great forum with very helpful contributors.

Thank to you all.

[Kool Aid]

You're welcome, Dexie.

How long did it take?

[Dexie]

It took about 3 minutes to import the coordinates and about the same to create the Site Model. Which is great.

Thanks again.