Dexie Posted December 1, 2009 Share Posted December 1, 2009 I have have downloaded some survey data from the Emap site which comes as an x,y,z text file. The file is 16Mb, so V large for a text file. There are approximately 650000 lines of data and I am having trouble importing it. I did a test on the first 1000 lines of data which took 15minutes to complete this import. Which means it will take over 6.5 days to import all that data if it works. Does anyone know why it takes so long. Vectorworks should be able to import thousands of pieces of data a second. Looking forward to any comments. Quote Link to comment
MullinRJ Posted December 1, 2009 Share Posted December 1, 2009 How are you importing the data? What format is it in? Quote Link to comment
IanH Posted December 1, 2009 Share Posted December 1, 2009 Wow. That's a lot of data. Once its in, its going to be a pain to manage if its simply a set of coordinates with no structure. When I do a Total Station survey, I normally create about 400-500 sets of coordinates. Importing as comma delimited "ID, Easting, Northing, Elevation, Description" takes no more than a few seconds. What format do you think the file is in and what command are you importing it in? I wonder if you are trying to import an ascii dxf file as x,y,z coordinates. Maybe you could post a couple of lines of data? Quote Link to comment
Dexie Posted December 1, 2009 Author Share Posted December 1, 2009 I am importing the data using Landmark > Import survey file. The data covers approx 1.5Km square and data is at 5m intervals. I normally get the data as dxf. but thought I would try the xyz data as it was a lot cheaper. The dxf data works well. It's just the conversion that is causing the problem. Quote Link to comment
Dexie Posted December 1, 2009 Author Share Posted December 1, 2009 This is how the data comes:- 372592.5,185037.5,93.66 372592.5,185032.5,93.681 372592.5,185027.5,93.698 372592.5,185022.5,93.695 372592.5,185017.5,93.69 372592.5,185012.5,93.685 372592.5,185007.5,93.68 372592.5,185002.5,93.675 372592.5,184997.5,93.669 372592.5,184992.5,93.664 372592.5,184987.5,93.659 372592.5,184982.5,93.639 372592.5,184977.5,93.616 Quote Link to comment
Kool Aid Posted December 1, 2009 Share Posted December 1, 2009 Dexie, I find it hard to believe that the DTM-module of VW could handle 650000 points? Looking at your data, they seem to be in a 5 m grid, which for such a large area may be redundantly dense. The slowness of import is likely to be accentuated by the fact that the import command creates Stake objects, not simple 3D-loci. I should have an old script somewhere that creates just loci. If not, it's be easy to write one. A script might, depending, also enlarge the grid by skipping lines. (A 10 m grid would have only a quarter of the points.) Quote Link to comment
Dexie Posted December 1, 2009 Author Share Posted December 1, 2009 I have just created a terrain model from OS DTM data which are 3D loci containing 220000 points in 1 minute. I want to create the 3D points from xyz data. Quote Link to comment
Dexie Posted December 1, 2009 Author Share Posted December 1, 2009 You are right it is probably the fact that it is trying to create stake objects that is the problem. I don't think it would have a problem with say 650000 3D loci. Quote Link to comment
IanH Posted December 1, 2009 Share Posted December 1, 2009 I am importing the data using Landmark > Import survey file. The data covers approx 1.5Km square and data is at 5m intervals. I normally get the data as dxf. but thought I would try the xyz data as it was a lot cheaper. The dxf data works well. It's just the conversion that is causing the problem. 5m intervals is 200 readings per linear KM is it not? So thats 300 x 300 readings, 90000 readings. Where does the 650000 come from? Quote Link to comment
Kool Aid Posted December 1, 2009 Share Posted December 1, 2009 If it were a square, it'd be 16.25 km2. SQRT(650000) = 806,225 806 x 5 = 4031 4,031 ^2 = 16,25 End of Practical Math 101 Lesson 1 for Garden Designers. Quote Link to comment
Kool Aid Posted December 1, 2009 Share Posted December 1, 2009 (edited) You are right it is probably the fact that it is trying to create stake objects that is the problem. I don't think it would have a problem with say 650000 3D loci. Don't forget that the number of triangles in the TIN/DTM grows faster than the number of definition points. EDIT In our grid situation, not faster, but proportionally. Doubling the grid means reducing the 3D-poly count by half. My original claim may well be incorrect. Have to think about it. Edited December 1, 2009 by Kool Aid Quote Link to comment
Dexie Posted December 1, 2009 Author Share Posted December 1, 2009 My mistake its 4Km x 4Km Quote Link to comment
IanH Posted December 1, 2009 Share Posted December 1, 2009 (edited) My mistake its 4Km x 4KmPhew. Thought my brain had suddenly forgotten basic maths. Still trying to work out how KA gets 16.25Km2. Thinks he is the one for a basic maths lesson because my calc also gives just over 4Km x 4Km for 650000 points too. Edited December 1, 2009 by IanH Quote Link to comment
Dexie Posted December 1, 2009 Author Share Posted December 1, 2009 Got to go. Daughters Birthday. Quote Link to comment
Kool Aid Posted December 1, 2009 Share Posted December 1, 2009 My mistake its 4Km x 4KmPhew. Thought my brain had suddenly forgotten basic maths. Still trying to work out how KA gets 16.25Km2. Thinks he is the one for a basic maths lesson because my calc also gives just over 4Km2 for 650000 points too. 4 x 4 = 16. km x km = km2 End of lesson 2. Quote Link to comment
IanH Posted December 1, 2009 Share Posted December 1, 2009 brain fart. Lesson noted. But I was still right in the first place that 1.5km x 1.5km didn't equal 650000 points. Quote Link to comment
Kool Aid Posted December 1, 2009 Share Posted December 1, 2009 Then we were both right! (Except that I was wrong in my claim regarding point/3D-poly number relationship: a rectangular grid is the worst case; in other configs, the number of polys increase slower than the number of points.) Quote Link to comment
Kool Aid Posted December 2, 2009 Share Posted December 2, 2009 (edited) You are right it is probably the fact that it is trying to create stake objects that is the problem. I don't think it would have a problem with say 650000 3D loci. Well, Dexie, try this: PROCEDURE ImportDTMData; { ? Petri Sakkinen 2009 } CONST comma = ','; VAR data, inputfile : STRING; x, y, z : REAL; commaPos : INTEGER; BEGIN GETFILE(inputfile); IF NOT(DIDCANCEL) THEN BEGIN WHILE NOT EOF(inputfile) DO BEGIN READLN(data); commaPos := POS(comma, data); x := STR2NUM(COPY(data, 1, commaPos-1)); DELETE(data, 1, commaPos); commaPos := POS(comma, data); y := STR2NUM(COPY(data, 1, commaPos-1)); DELETE(data, 1, commaPos); z := STR2NUM(data); LOCUS3D(x, y, z); END; CLOSE(inputfile); END; END; RUN(ImportDTMData); If it is slow, you might use a text editor to replace commas with tabs and then use this: PROCEDURE ImportDTMData; { ? Petri Sakkinen 2009 } VAR inputfile : STRING; x, y, z : REAL; BEGIN GETFILE(inputfile); IF NOT(DIDCANCEL) THEN BEGIN WHILE NOT EOF(inputfile) DO BEGIN READLN(x, y, z); LOCUS3D(x, y, z); END; CLOSE(inputfile); END; END; RUN(ImportDTMData); Tab-separated data files are preferred. For filtering/reducing number of points, add a READLN; after LOCUS3D(); so every second line is skipped. (The last point is not created then, though, but doing that manually is not a huge task!) EDIT Actually, the extra READLN; could be before the actual READLN(); because then you don't need to go to the end of the file to find the z-value! Edited December 2, 2009 by Kool Aid Quote Link to comment
go2greece Posted December 2, 2009 Share Posted December 2, 2009 DEXIE, i have once imported 96 squarekilometers with a 5 by 5 grid so dont worry . first> the comma separated coordinates have to be imported and transfered into 3d points ( if you have difficulties with VW try DL-02 from sierra-soft , which should be free> creates a dxf file of the points). i would split the VW file into hand-able portions like 100mX100m > create a master-file and reference the parts before you do so check the DATUM of the Emap WGS84 ..... it looks like your point are somewhere in africa CHAD??? peter Quote Link to comment
go2greece Posted December 2, 2009 Share Posted December 2, 2009 petris's proposal is much more elegant i have to admit peter Quote Link to comment
Dexie Posted December 2, 2009 Author Share Posted December 2, 2009 Thanks for all your efforts guy's. I will try it now. I have no doubt it will success. I will you know shortly. Cheers. Quote Link to comment
Dexie Posted December 2, 2009 Author Share Posted December 2, 2009 That worked like a dream. A special thanks to you Petri. This is a great forum with very helpful contributors. Thank to you all. Quote Link to comment
Kool Aid Posted December 2, 2009 Share Posted December 2, 2009 You're welcome, Dexie. How long did it take? Quote Link to comment
Dexie Posted December 4, 2009 Author Share Posted December 4, 2009 It took about 3 minutes to import the coordinates and about the same to create the Site Model. Which is great. Thanks again. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.