NoGripRacing.com

Go Back   NoGripRacing Forums > Modding > Track Conversion / Creation Help

Reply
Thread Tools
Unread 15 June 17, 15:14   #51
Wee Scot
 
Wee Scot's Avatar
 
Join Date: Jan 2007
Location: Dumfries, Virginia
Age: 62
Default

Thanks to you, both, for this detailed and informative discussion. I'll start with small tests until I learn exactly what steps to take, and in which order, to get the results I want. And rest assured, I'll report on my progress!
Wee Scot is offline   Reply With Quote
Unread 15 June 17, 17:25   #52
Pizzaman
Minifreak & Muscleman
 
Pizzaman's Avatar
 
Join Date: Feb 2007
Location: Groningen Centre of the Universe
Default

Yes the saved projects can always be opened in Evo.
You just lose the mixed textures, and the AIW (pacecar/rolling start) will differ.

Seems no problem to me there.

You can even re-import the saved EVO-project so Rf1 players can enjoy it.

In fact that sounds even better than importing from Google Earth, as that always STAYS an object, and I have to put the road slightly 'over' it (following the landscape) and then connect it all up in a different way (else the cars might get stuck between the road and the landscape if they run off the track). I sometimes have to adjust the landscape object as well, to be able to fit the road (sometimes it has peaks sticking out). Also, no way you can paint the object-landscape in BTB (not even in Evo), change textures or do anything to it except stretch, move, rotate. So I always have to do that in another program.

Your report might be damn handy too - Seems like I'll need to use LIDAR now!
Pizzaman is offline   Reply With Quote
Unread 17 June 17, 00:44   #53
Wee Scot
 
Wee Scot's Avatar
 
Join Date: Jan 2007
Location: Dumfries, Virginia
Age: 62
Default

OK, starting with baby steps...

Ten years ago, when I was working with the early development versions of BTB, I started a new project by placing a JPEG image (which I created in Microsoft Paint by connecting screen captures of several Google Maps satellite images) as my background image, then laying down my "closed loop track" by following the real-life roads in that background image, clicking to add a sufficient number of nodes to accurately trace the exact path those real roads. Then I used the somewhat-accurate elevation data in Google Maps--available by tracing the roads with my mouse cursor--to note the changes in elevation and apply them by hand in BTB.

TODAY, I'm looking to create tracks based on much more accurate elevation data about both the circuit paths and the surrounding terrain.

---------------

Now, in BTB Pro, I'm given the following options:

Under File, I can:

Import... > 'Import GPS', under which there are two tabs:

CSV - "Import GPS data. Must be in a very specific format. See your GPS folder for an example. All values are in meters." If you click the Import button, you are instructed to find *.csv files.

Google Earth - "Use Google Earth to create a path and save it to KML format. Then load here." If you click the Import button, you are instructed to find *.kml files.

Under File, I can also:

Import Cloud... > 'Import Point Cloud', which provides columns for X, Y (Height), and Z values, as well as Intensity and Colour values, Delimiter, and an unexplained Filter slider set by default to 100%. If you click the Select Files & Import button, you are instructed to find *.txt or *.csv data files.

---------------

The LiDAR data I downloaded through the Earth Explorer (USGS EROS) website consists of nine "tiles" which cover the length of the 9-mile Joplin Road "open track" I want to create, each a small (34 KB) CSV file that contains a lot of latitude and longitude data, in both DMS (degrees, minutes, seconds) and decimal form, arranged as such:
LIDAR Entity ID,Project,State/Province/Country,Pulse and NPS Units,Pulse Density,Nominal Pulse Spacing,Beginning Date,Ending Date,Vendor,Vendor Scene ID,Map Projection,Projection Zone,Datum,Vertical Datum,Center Latitude,Center Longitude,NW Corner Lat,NW Corner Long,NE Corner Lat,NE Corner Long,SE Corner Lat,SE Corner Long,SW Corner Lat,SW Corner Long,Center Latitude dec,Center Longitude dec,NW Corner Lat dec,NW Corner Long dec,NE Corner Lat dec,NE Corner Long dec,SE Corner Lat dec,SE Corner Long dec,SW Corner Lat dec,SW Corner Long dec,Display ID,Ordering ID,Browse Link
An example of the data format:
38°32'33.58"N,77°25'03.23"W,38°32'58.58"N,77°25'34 .33"W,38°32'58.00"N,77°24'31.39"W,38°32'08.58"N,77 °24'32.14"W,38°32'09.16"N,77°25'35.06"W,38.542661,-77.4175638,38.5496055,-77.4262027,38.5494444,-77.4087194,38.5357166,-77.4089277,38.5358777,-77.4264054
Now, there doesn't appear to be any height--elevation--data in these files. The only reference I can find that even SUGGESTS height is the 'Vertical Datum' category, which comes just before all the DMS and digital LongLat information. But all the Vertical Datum entries are the same: NAVD88. Does that perhaps refer to a completely DIFFERENT data source, which I have to download separately, but that will be "aligned" with the LongLat information I already have?

The LiDAR data I received through the NOAA Digital Coast Data Team website came to me as a 164 MB zip archive containing four *.laz files, an xml document, and four tiles_index files, with the extensions .dbf, .prj, .shp, and .shx.

EDIT: @Emery-I did as you recommended in our Conversation at Race Department: I used the drawing tool to define a rectangular area that encompassed the two stretches of road--Aden Road and Joplin Road--that I want to use for two short "open" tracks or one longer one. Three "results" were then listed to the right of the map:
NOAA Sea Level Rise DEM
2016 USGS CoNED Topobathymetric DEM Model: Chesapeake Bay (1859-2015)
2012 USGS-FEMA Lidar: Northern Counties (VA-North)
The last result was clearly the one I wanted, but for this one, I was not given the "shopping cart" option to select only the tiles in, or bordering that area I'd defined. My only option was to select Bulk Download.

The email I got in response, however, pointed me to a link to download a 164 MB zip archive, which contained only 4 tiles. As was explained in the email, "This was a large job and was split into 4 tiles for processing."


What do I make of this? What is the procedure for converting LAZ files into a format that can be used by BTB?


I need:

1) Clarification on what to do with the Earth Explorer data, and whether it already has height information in it I'm just overlooking, or whether I need to look elsewhere for that.

OR

2) Step-by-step instructions on what to do with the LAZ files I received from NOAA

OR

2) Information on where ELSE to find a source of terrain data I can use to more easily create real-world terrain in BTB.

Thanks, guys! While I wait for your guidance, I'll be driving my favorite cars on my favorite tracks in GTR2, rFactor, Grand Prix Legends, and GP4!


Last edited by Wee Scot; 18 June 17 at 20:44.
Wee Scot is offline   Reply With Quote
Unread 18 June 17, 08:22   #54
Pizzaman
Minifreak & Muscleman
 
Pizzaman's Avatar
 
Join Date: Feb 2007
Location: Groningen Centre of the Universe
Default

I think nobody has even done this sort of thing before, like that.
At least I haven't. Curious to see what will be the effects.

But the LiDAR data you downloaded through the Earth Explorer (USGS EROS), that is what I would use - just klick the 'Import GPS' button where it asks for CSV input, and then use those 4 files.
Pizzaman is offline   Reply With Quote
Unread 18 June 17, 08:34   #55
gonzaluigiRacer
 
Join Date: Mar 2017
Location: Madrid, Spain
Default

It's LiDAR data?
gonzaluigiRacer is offline   Reply With Quote
Unread 18 June 17, 20:00   #56
Wee Scot
 
Wee Scot's Avatar
 
Join Date: Jan 2007
Location: Dumfries, Virginia
Age: 62
Default

Yikes! Just realized all the data I got from USGS (Earth Explorer) is in degrees,minutes,seconds, but the data set examples in BTB are in decimal degrees! So before I can reformat the data into separate lines arranged LongLatHeight, I'll have to CONVERT all those--one at a time--from DMS to decimal!!!

Forget THAT! If I can't find a "friendlier" set of GPS data I can use, I'll just do all the elevation changes and slopes and cambers by hand.

EDIT: NO, WAIT! LOL I'm a codfish...

The second half of the data for each grouping (tile?) IS decimal! That's what 'dec' stands for! Doh!

Broken out into separate lines:

Center Latitude
Center Longitude
NW Corner Lat
NW Corner Long
NE Corner Lat
NE Corner Long
SE Corner Lat
SE Corner Long
SW Corner Lat
SW Corner Long
Center Latitude dec
Center Longitude dec
NW Corner Lat dec
NW Corner Long dec
NE Corner Lat dec
NE Corner Long dec
SE Corner Lat dec
SE Corner Long dec
SW Corner Lat dec
SW Corner Long dec


38°32'33.58"N
77°25'03.23"W
38°32'58.58"N
77°25'34.33"W
38°32'58.00"N
77°24'31.39"W
38°32'08.58"N
77°24'32.14"W
38°32'09.16"N
77°25'35.06"W
38.542661
-77.4175638
38.5496055
-77.4262027
38.5494444
-77.4087194
38.5357166
-77.4089277
38.5358777
-77.4264054


OK, so all I have to do is reformat the data. But where is the HEIGHT information??? All I see are longitude and latitude numbers. Huh???

Last edited by Wee Scot; 18 June 17 at 20:11.
Wee Scot is offline   Reply With Quote
Unread 19 June 17, 18:01   #57
Emery
 
Join Date: Apr 2008
Location: Fiat Farm, in the wettest corner of Oregon
Age: 54
Default

> The LiDAR data I received through the NOAA Digital Coast Data Team website came
> to me as a 164 MB zip archive containing four *.laz files

Think you meant USGS Earth Explorer gave you four laz files? NOAA should only give you one file.

LAZ or LAS files are what you use and they contain the height info. At this point, it's time to learn & use other free tools:
1) LAStools available at https://rapidlasso.com/lastools/. The comments about licensing apply to commercial use. It is a command-line tool with many capabilities, one of which is to convert LAZ/LAS files to CSV format. You'll still need to geometrically translate the output to a local (0,0,0) reference and move data columns for BTB purposes.
2) CloudCompare available at http://www.danielgm.net/cc/. You can view and manipulate point clouds of many formats. Converting point clouds to CSV and the translating to a local (0,0,0) reference for BTB purposes are some of the functions. You'll likely still need to move data columns for BTB purposes.

Last edited by Emery; 19 June 17 at 18:45.
Emery is offline   Reply With Quote
Unread 19 June 17, 18:28   #58
Emery
 
Join Date: Apr 2008
Location: Fiat Farm, in the wettest corner of Oregon
Age: 54
Default

If you're using the 4 tiles that Earth Explorer found for you, you'll want to use CloudCompare to combine the tiles & trim the area. I much prefer using the NOAA Digital Coast Coast site when I have a choice as they combine tiles and trim the area for you.
Emery is offline   Reply With Quote
Unread 19 June 17, 19:29   #59
Wee Scot
 
Wee Scot's Avatar
 
Join Date: Jan 2007
Location: Dumfries, Virginia
Age: 62
Default

Quote:
Originally Posted by Emery View Post
> The LiDAR data I received through the NOAA Digital Coast Data Team website came
> to me as a 164 MB zip archive containing four *.laz files

Think you meant USGS Earth Explorer gave you four laz files? NOAA should only give you one file.
No, USGS Earth Explorer emailed me links to download nine files (zipped CSV files), each representing one of the "tiles" I'd selected by clicking the footprint icon on their website, each crammed with the information I presented in my 16 June post. It finally occurred to me that they each contained digital latitude and longitude coordinates for the center and NW, NE, SE, and SW corners of 60 subareas (subtiles?), but--even though these files were what Earth Explorer offered in response to my selection of LiDAR from the 'Digital Elevation' data set--I didn't see anything that looked like HEIGHT data.

The four LAZ files were contained in the 164 MB zip file I downloaded through a link in an email from the NOAA Digital Coast Data Team. Also in that zip file were an xml document, and four tiles_index files, with the extensions .dbf, .prj, .shp, and .shx. There were four LAZ files instead of one because, as NOAA explained in one of their emails to me: "This was a large job and was split into 4 tiles for processing."

Now you are advising me to use the LAZ files, after learning how to use more tools! Sigh. Well, I'm not surprised this isn't turning out to be easy, and I'm still up to the challenge! So I'll go away for a while and try to figure out how to use LAStools and CloudCompare to get the NOAA data into the required LongLatHeight format for BTB!

Two clarifications, please:

1) It's clear to me why I need to use LAStools to convert the data in the LAZ files to CSV for use in BTB. Am I also correct that I'll need to use CloudCompare to 'combine the tiles' (the four LAZ files) and 'trim the area' (cut away everything I don't need in order to provide trackside terrain that's sufficiently deep, or dense)?

2) When you write "move data columns for BTB purposes", are you referring to the requirement that the data be arranged 'longitude,latitude,height' in a CSV or TXT file created in Notepad? Or are you talking about using columns in an Excel spreadsheet?

Last edited by Wee Scot; 19 June 17 at 20:24.
Wee Scot is offline   Reply With Quote
Unread 19 June 17, 19:48   #60
Emery
 
Join Date: Apr 2008
Location: Fiat Farm, in the wettest corner of Oregon
Age: 54
Default

The fields to change in NOAA Digital Coast when you're checking out:

Better to use LAZ rather than LAS at this point to save download bandwidth, which I forgot to do in this example.

What a top view of the data looks like in CloudCompare after switching to intensity, setting blue-green-red-yellow as color, and shortening range to match actual values:


Filtering the data for only classification 2 (ground return) eliminates trees, water, etc:


There are still some 20 million points in the cloud. Trimming away most of the non-road area, leaving a buffer of about 100 meters, will get you down to 1 million points.

Exporting that to a CSV file and opening it in a spreadsheet will look like:


You only need the X, Y, Z, & intensity columns for the BTB import. See how big the numbers for X & Y are? They're outside the range that BTB can handle which is why you need to translate them to a local (0,0,0) reference. Even the Z values should be brought down so the lowest point is zero (or within 5 meters). You'll also need to switch the Y & Z columns before importing. Finally, BTB can only reliably import about 250k points at a time, so a million point file needs to be broken into 4 separate files.

Last edited by Emery; 19 June 17 at 20:09.
Emery is offline   Reply With Quote
Unread 19 June 17, 19:57   #61
Emery
 
Join Date: Apr 2008
Location: Fiat Farm, in the wettest corner of Oregon
Age: 54
Default

Quote:
Originally Posted by Wee Scot View Post
Two clarifications, please:

1) It's clear to me why I need to use LAStools to convert the data in the LAZ files to CSV for use in BTB. Am I also correct that I'll need to use CloudCompare to 'combine the tiles' (the four LAZ files) and 'trim the area' (cut away everything I don't need in order to provide trackside terrain that's sufficiently deep, or dense)?

2) When you write "move data columns for BTB purposes", are you referring to the requirement that the data be arranged 'longitude,latitude,height' in a CSV or TXT file created in Notepad? Or are you talking about using columns in an Excel spreadsheet?
1) The /XYZit command in LAStools does that for each LAS/LAZ file you use. Or you can use CloudCompare (File, Save, change the format to .CSV). I prefer using CloudCompare these days, but there are definitely times when LAStools is more convenient, such as hands-off processing a bunch of files.

2) BTB expects CSV import to be the columns arranged as "easting, altitude, northing" whereas the standard /XYZit columns are arranged as "easting, northing, altitude". Longitude & latitude are not used because they are spherical coordinates rather than planar coordinates. That's why step 1 of the download process has you changing the horizontal units to meters and those meters are in reference to the projection plane and horizontal datum.
Emery is offline   Reply With Quote
Unread 19 June 17, 20:07   #62
Emery
 
Join Date: Apr 2008
Location: Fiat Farm, in the wettest corner of Oregon
Age: 54
Default

And just to be clear, point clouds as CSV files are imported using the Import Cloud option in BTB Pro.
Emery is offline   Reply With Quote
Unread 19 June 17, 20:13   #63
Emery
 
Join Date: Apr 2008
Location: Fiat Farm, in the wettest corner of Oregon
Age: 54
Default

You may ask, "Why did I take all classes of points and then filter out only the ground return when I could have downloaded only the ground return to begin with?"

Because you don't know how high tall/thick the trees are. With all classes of points, you can separately import all the non-ground return points and use them for modeling references!
Emery is offline   Reply With Quote
Unread 19 June 17, 20:35   #64
Wee Scot
 
Wee Scot's Avatar
 
Join Date: Jan 2007
Location: Dumfries, Virginia
Age: 62
Default

Quote:
Originally Posted by Emery View Post
The fields to change in NOAA Digital Coast when you're checking out:

Better to use LAZ rather than LAS at this point to save download bandwidth, which I forgot to do in this example.
I carefully followed the instructions you gave me in your June 10 post in our RaceDepartment conversation:

"choose horizontal and vertical units in meters, output product as point, output format as LAZ, data class as All (*) and then click on Next in the lower right and follow directions."

So the four LAZ files I have should contain the data I need.
Wee Scot is offline   Reply With Quote
Unread 19 June 17, 20:45   #65
Wee Scot
 
Wee Scot's Avatar
 
Join Date: Jan 2007
Location: Dumfries, Virginia
Age: 62
Default

Quote:
Originally Posted by Emery View Post
And just to be clear, point clouds as CSV files are imported using the Import Cloud option in BTB Pro.
Ah, OK! Now I understand. I was still thinking I was going to be using File > Import... to pull in a CSV or TXT file I created in Notepad, using the 'longitude,latitude,height' format shown in the examples in the BTB Pro Support\GPS folder!
Wee Scot is offline   Reply With Quote
Unread 19 June 17, 20:57   #66
Wee Scot
 
Wee Scot's Avatar
 
Join Date: Jan 2007
Location: Dumfries, Virginia
Age: 62
Default

OK, I don't have any experience using "command line" instructions, so I just read all the READMEs in the LAStools bin folder, and--unless you want to walk me step-by-step through how to bring up and use the forbiddingly black Command Prompt box--it looks like the tool I want to use is:

las2txt: Converts from binary LAS/LAZ 1.0 - 1.4 to an ASCII text format

But, if so, which of these usage examples do I want to follow?

1) >> las2txt -i lidar.laz -o lidar.txt -parse xyzEit -extra 0.1

converts LAZ file to ASCII and places the x, y, and z coordinate
of each point at the 1st, 2nd, and 3rd entry of each line. the
extra string "0.1" as the 4th entry and the intensity and time
of each point as the 5th and 6th entry.

2) >> las2txt -i lidar.laz -o lidar.txt -parse txyzr -sep comma

converts LAZ file to ASCII and places the gps_time as the first
entry, the x, y, and z coordinates at the 2nd, 3rd, and 4th entry
and the number of the return as the 5th entry of each line. the
entries are separated by a comma.

3) >> las2txt -i lidar.laz -o lidar.txt -parse xyzRGB

converts LAZ file to ASCII and places the x, y, and z coordinates
at the 1st, 2nd, and 3rd entry and the r, g, and b value of the
RGB color as the 4th, 5th, and 6th entry of each line. the entries
are separated by a space. note that lidar.las should be format 1.2
or higher (because 1.0 and 1.1 do not support RGB colors).
Wee Scot is offline   Reply With Quote
Unread 19 June 17, 21:00   #67
Wee Scot
 
Wee Scot's Avatar
 
Join Date: Jan 2007
Location: Dumfries, Virginia
Age: 62
Default

Quote:
Originally Posted by Emery View Post
You may ask, "Why did I take all classes of points and then filter out only the ground return when I could have downloaded only the ground return to begin with?"

Because you don't know how high tall/thick the trees are. With all classes of points, you can separately import all the non-ground return points and use them for modeling references!
Wow, that's really cool! I have digital video of the four tracks that are at the top of my project list, but having the "non-ground return points" for modeling references would enable even greater presentation accuracy!
Wee Scot is offline   Reply With Quote
Unread 19 June 17, 21:43   #68
Emery
 
Join Date: Apr 2008
Location: Fiat Farm, in the wettest corner of Oregon
Age: 54
Default

Quote:
Originally Posted by Wee Scot View Post
OK, I don't have any experience using "command line" instructions, so I just read all the READMEs in the LAStools bin folder, and--unless you want to walk me step-by-step through how to bring up and use the forbiddingly black Command Prompt box--it looks like the tool I want to use is:

las2txt: Converts from binary LAS/LAZ 1.0 - 1.4 to an ASCII text format

But, if so, which of these usage examples do I want to follow?

1) >> las2txt -i lidar.laz -o lidar.txt -parse xyzEit -extra 0.1

converts LAZ file to ASCII and places the x, y, and z coordinate
of each point at the 1st, 2nd, and 3rd entry of each line. the
extra string "0.1" as the 4th entry and the intensity and time
of each point as the 5th and 6th entry.

2) >> las2txt -i lidar.laz -o lidar.txt -parse txyzr -sep comma

converts LAZ file to ASCII and places the gps_time as the first
entry, the x, y, and z coordinates at the 2nd, 3rd, and 4th entry
and the number of the return as the 5th entry of each line. the
entries are separated by a comma.

3) >> las2txt -i lidar.laz -o lidar.txt -parse xyzRGB

converts LAZ file to ASCII and places the x, y, and z coordinates
at the 1st, 2nd, and 3rd entry and the r, g, and b value of the
RGB color as the 4th, 5th, and 6th entry of each line. the entries
are separated by a space. note that lidar.las should be format 1.2
or higher (because 1.0 and 1.1 do not support RGB colors).
Process of elimination...
Do you need gps_time for the import? No.
Does your las/laz file have color information? No.
That leaves one choice from your examples... but if you look more closely, you'll notice that what the examples are really showing is that the -parse arguments can be arranged in various ways. So maybe xyzit would be x,y,z followed by intensity and time. Naturally something I've not noticed before is that you MIGHT be able to rearrange the columns to be xzyi to arrive at the desired format for BTB Pro without taking an intermediate step through a spreadsheet or editor.
Emery is offline   Reply With Quote
Unread 21 June 17, 15:08   #69
Wee Scot
 
Wee Scot's Avatar
 
Join Date: Jan 2007
Location: Dumfries, Virginia
Age: 62
Default

Quote:
Originally Posted by Emery View Post
Filtering the data for only classification 2 (ground return) eliminates trees, water, etc:
OK, so I will only have the option to filter the data so that I only see 'ground return' if I put the data through CloudCompare? Not with LAStools? I definitely want only ground return elevation data depicted for my terrain mesh. Please clarify.

Quote:
Originally Posted by Emery View Post
There are still some 20 million points in the cloud. Trimming away most of the non-road area, leaving a buffer of about 100 meters, will get you down to 1 million points.
How do I 'trim away' what I don't need? Is that also a CloudCompare function?

Quote:
Originally Posted by Emery View Post
Exporting that to a CSV file and opening it in a spreadsheet will look like:


You only need the X, Y, Z, & intensity columns for the BTB import. See how big the numbers for X & Y are? They're outside the range that BTB can handle which is why you need to translate them to a local (0,0,0) reference. Even the Z values should be brought down so the lowest point is zero (or within 5 meters). You'll also need to switch the Y & Z columns before importing. Finally, BTB can only reliably import about 250k points at a time, so a million point file needs to be broken into 4 separate files.
Do you mean that I would click 'Export' in CloudCompare and choose CSV, which would produce a file I would open with Notepad or Excel?

Please remember: I'm doing ALL of this for the very first time, so NONE of the steps are intuitive to me.

Then you wrote (post #61):
BTB expects CSV import to be the columns arranged as "easting, altitude, northing" whereas the standard /XYZit columns are arranged as "easting, northing, altitude".
The example I provided from the LAStools README file was:
1) >> las2txt -i lidar.laz -o lidar.txt -parse xyzEit -extra 0.1

converts LAZ file to ASCII and places the x, y, and z coordinate
of each point at the 1st, 2nd, and 3rd entry of each line. the
extra string "0.1" as the 4th entry and the intensity and time
of each point as the 5th and 6th entry.
And your response to that was:
Process of elimination...
Do you need gps_time for the import? No.
Does your las/laz file have color information? No.
That leaves one choice from your examples... but if you look more closely, you'll notice that what the examples are really showing is that the -parse arguments can be arranged in various ways. So maybe xyzit would be x,y,z followed by intensity and time. Naturally something I've not noticed before is that you MIGHT be able to rearrange the columns to be xzyi to arrive at the desired format for BTB Pro without taking an intermediate step through a spreadsheet or editor.
So, what are you telling me, exactly?

That I need to double-click on the las2txt application in LAStools, which will open up, what, a Command Prompt box in which I need to type the column letters in the order in which I want them to be presented
>> las2txt -i lidar.laz -o lidar.txt -parse xzyi -extra 0.1
...which will produce a simple TXT file, with the data on each line arranged x,z,y,i? The Import Cloud action in BTB Pro is looking for a "TXT or CSV" file, so is that all I need to do?

Please be gentle with me. This is ALL new to me.


Last edited by Wee Scot; 21 June 17 at 15:34.
Wee Scot is offline   Reply With Quote
Unread 21 June 17, 16:34   #70
Wee Scot
 
Wee Scot's Avatar
 
Join Date: Jan 2007
Location: Dumfries, Virginia
Age: 62
Default

OK, let's see just how intuitive the LAStools las2txt program is!

Gulp...

So, I double-click on the las2txt application, and it opens a friendly window!

Ah, yes. There's the 'browse' button.
For some reason, it thinks what I want is likely in 'directory: E:\' Odd...
Well, I'll just change that to 'C:\' and...
scroll down and double-click on \SIMRACING
Yes, and there's \LiDAR data, and below that \NOAA, and Job362566_ncr2014_us, and below THAT: there are the four LAZ files!

NOW

Do I simply double-click on the first one and see what happens?

Or do I first consider the available 'filter' options? Hmmm... Since I don't UNDERSTAND the 'filter' options I'm presented with under 'by coordinates', 'by classification or return', or 'by various criteria', I'll just hold my breath and...

double-click on the first LAZ file!

OK! My computer didn't crash!

I'm staring at a blank blue box, but now I see that, to the right there is another column of choices.

The one at the top reads '1 job on 8 cores'
And below that, the radio button for 'process all files' is selected by default.
Hmmm... I've only selected the first of my LAZ files, so why isn't the radio button for 'selected file only' selected? Let's change that.

Below that, the check boxes for (x), (y), and (z) are checked. Well, the next one down is 'intensity', and I want that too, so...
I put a check mark in that box as well.

NOW, below that is the 'parse string' field! OK, so lets change that from the default (xyzi) to the order I need for BTB (xzyi)

Below THAT is the 'separator' field. Well, that needs to be 'comma' I think.

Now, do I dare click the 'RUN' button? Why not!

OK! The RUN box shows the path as "C:\SIMRACING\LiDAR data\NOAA..." -parse xzyi -sep comma

That seems right, so I click 'START'

Several seconds go by, then the hourglass icon appears, but in the header it tells me "RUN (not responding)"
Then the RUN box goes away.

Huh?

Let's try again.

This time, when I click START, nothing appears to happen. But when I go down to my task bar and check the Command Prompt window, it seems to show that my RUN command was executed both times! So where do I look for my TXT file(s)???

I look through my LAStools folders. Nothing new THERE.
I look in Downloads. Nothing new there, either.

Ah , WAIT. There's a VIEW button in the las2txt window!
But that just brings up the RUN window, with buttons for START, COPY, and CANCEL.

OK, now I'm where I expected to be 30 minutes ago...

LOST!

What next?
Wee Scot is offline   Reply With Quote
Unread 22 June 17, 16:39   #71
Emery
 
Join Date: Apr 2008
Location: Fiat Farm, in the wettest corner of Oregon
Age: 54
Default

Sounds like they updated LAStools since the last time I downloaded it moons ago...

Also sounds like you're VERY close to having it figured out!
Emery is offline   Reply With Quote
Unread 22 June 17, 18:08   #72
Wee Scot
 
Wee Scot's Avatar
 
Join Date: Jan 2007
Location: Dumfries, Virginia
Age: 62
Default

That's it? After all the steps I detailed in the last two posts, that's all you've got for me? LOL

If you really want to help me, download the latest version of LAStools so you can see what I was looking at when I dblclkd on the las2txt application in the bin folder.

Meanwhile, I'm going to see if CloudCompare is any less inscrutable...

So, which version of CloudCompare do I want?
"Latest beta release (2.9.beta, 10-June-2017)"
"Latest stable release (2.8.1 Hogfather + Canupo patched)"
"Last version compatible with old Intel/ATI graphics cards (2.6.3 beta)"
Instinct tells me I should use the "stable" release version instead of the latest & greatest beta, but is my 2010 ATI Radeon HD 5850 graphics card considered "old"? (EDIT: Yes, it's old. Thanks, Pizzaman...)

EDIT: I downloaded the 2.6.3 beta, and opened each of the four LAZ files I'd received from NOAA, accepting the default settings in the 'Open LAS File' dialog box and clicking Apply.

A 'Global shift/scale' dialog box opened telling me "Coordinates are too big (original precision may be lost)! Do you wish to translate/rescale the entity?"
When I opened the last file, for example, the Global shift/scale dialog box showed that the "Point in original coordinate system (on disk)" for x = 3598147.930000, and the corresponding "Point in local coordinate system" would be x = 98147.930.
I chose 'Yes' to translate/rescale each file, with the result that each LAZ files opened (the first one contained 8M points, the second 10M points, the third 4M points, and the last 6.9M points) and displayed in the CloudCompare viewer window as four multicolor rectangles that formed a box, with solid green, solid yellow, solid blue, green/yellow, blue/green, red/yellow, red/green, and--in the last--one solid red band. All these bands of color stretched from NW to SE in orientation.

Here's a screen capture:
http://www.nogripracing.com/gallery/...p?photo=181729

Now, what in the world do I do next? Can you walk me through the steps I need to take to get the results I'm looking for? Or are you going to tell me to spend the next three months reading through all of the CloudCompare wiki documentation?

EDIT 2: OK, I probed a little deeper. I closed and reopened, this time selecting just one of the four LAZ files to open and view. In the 'DB Tree' box, I clicked on the file with the 'cloud' icon beside it, which I judged to be the file created by the rescaling process, and that gave me access to a bunch of Property categories with dropdown menus.

In 'Scalar Fields' I changed the active display (field?) from 'Point Source ID' to 'User Data', which looked like terrain return instead of stripes in a tie! 'Scan Direction' gave the same appearance as 'User Data', but 'Number of Returns' produced an image that looked like the real suburban Northern Virginia terrain, with the suggestion of both wooded and cleared areas. 'Intensity' presented the same pattern as 'Number of Returns', only with more apparent detail.

The property below 'Scalar Fields' is 'Color Scale', and that yielded some attractive results, especially using 'Blue>Green>' with 'Intensity'

When I saved this to a 'Test Save' folder, I changed from the default *.bin to an ASCII (*.txt) file, and a Save ASCII file dialog box came up suggesting the following:
coordinates precision: 8
scalar precision: 6
separator: space
order: [ASC] point, color, SF(s), normal
I don't know enough to know whether I wanted to change the first two values, but I thought I wanted commas instead of spaces for my separators, so I made that one change and clicked OK. Saving the data for those 8 million points took about 100 seconds, resulting in a 1.2GB TXT file(!). Obviously, I've got to do a lot of "reducing" of my terrain before it'll be usable in BTB.

Do I know what I did or what value it might have to me? Of course not! But I'm hoping someone can use this example to advance my education!

Last edited by Wee Scot; 22 June 17 at 21:21.
Wee Scot is offline   Reply With Quote
Unread 22 June 17, 18:58   #73
Pizzaman
Minifreak & Muscleman
 
Pizzaman's Avatar
 
Join Date: Feb 2007
Location: Groningen Centre of the Universe
Default

Yes mate I'm afraid that ATi5850 is considered 'stone age'.

I'm quite curious how you'll fix this.
Pizzaman is offline   Reply With Quote
Unread Yesterday, 14:02   #74
Emery
 
Join Date: Apr 2008
Location: Fiat Farm, in the wettest corner of Oregon
Age: 54
Default

On the loading prompts for CloudCompare, I click "Apply" and then "Yes to All". Switching away from Point Source ID to Intensity in "Active Scalar Field" and changing to a color range instead of grey is my next step. Finally, in the "SF display params" section, I change the saturation down to about 65 (or whatever a suitable range is to match the histogram... the Virginia scans are pretty low intensity compared to others).

Edit, Segment (the scissors tool) allows you to trim away the areas you don't need/want. Use the dropdown in the menu bar for polygonal/rectangular selection, mark out the area to trim, choose whether to include or exclude the selection, then click on the checkmark to apply it. If you're using the polygonal selection, left-clicks add points to your polygon and a right-click ends the selection process.

Let's get your project centered on a local (0,0,0) coordinate. Pick a side view and zoom in on the lowest point. Tools, Point-picking pops up the toolbar for picking points. You only need the first icon in the toolbar to get information about a point. Pick the lowest point and note the z value (the local and global are both displayed; the global is what you want). Go back to the top view and pick a central point, noting the x and y global values. Highlight the cloud you want to center. Edit, Apply Transformation, Axis/Angle tab. Enter the x,y,z values you've noted in the Translation section at the bottom and click the box "Apply inverse transformation", then click OK. File, Save and make sure you change the name so you don't overwrite the original file. Now close the files (In the DB Tree, highlight everything, then right-click and delete) and reopen your centered project.

At this point, it is worthwhile to strip away the vegetation. Change "Active Scalar Field" to Classification instead of Intensity. Classification values go from 0 to 15(*), but not all are used. Edit, Scalar Fields, Filter by Value and select 2 through 14 to strip away the vegetation. Highlight the resulting file and save it as your ground return. Highlight your initial file and save that as your vegetation.

(*) This is just for your project. The Virginia scans have not had much classification done and some of it appears to be haphazard, like putting sample strips of trees into class 15 which is normally a user value. Well-classified scans will have ground, vegetation, buildings, powerlines, pavement, & water separated out.
Emery is offline   Reply With Quote
Unread Yesterday, 15:06   #75
Wee Scot
 
Wee Scot's Avatar
 
Join Date: Jan 2007
Location: Dumfries, Virginia
Age: 62
Default

Thanks for the detailed instructions! I'll give you a progress report after I've had a chance to act on them!

Just to be clear:

I'm going to work with each of my 4 LAZ files, trimming away what I don't need to reduce the size of the ultimate file(s).

Looking ahead, assuming the track I want to build runs through at least two (and maybe three) of those four 'tiles' I received from NOAA, will I next want to 'join' them together to create a single file before splitting that file into 250K files that BTB can handle? Or will I leave the data separated as delivered to me in the LAZ files, and only split the trimmed versions of THOSE SEPARATE files into 250K pieces?

EDIT: I actually found that using the Grey color scale was best for revealing the path of the roads I'll be using for my little Newman Road circuit (3.1 miles). The area I'd identified using the 'Draw Search Area' tool in the NOAA Data Access Viewer was tightly focused on just that small area of Fairfax County just northeast of Clifton, probably no more than 5,000 feet square! And that came to me as four separate LAZ files totaling 168 MB and containing upwards of 30 million points!

I did pull up the same area in Google Earth and put the windows side-by-side just to confirm what I thought were the right road beds in the CloudCompare composite image of the four tiles.

Now, when I 'Import Cloud...' into BTB however many 250K TXT or CSV files result from my editing of the LiDAR data, will the xy coordinates in the data ensure that they are positioned properly relative to each other, or will I have to 'stitch them together' myself to get them positioned properly? I'm assuming that, in the resulting mesh of terrain polygons, the xyz position of all of the vertices will be generated in direct relation to the data in those cloud data TXT files. Correct?

I'm also assuming that this mesh will be transparent, such that I can use the composite of Google Earth images I just put together in Paint as my BTB Background Image for the purpose of correctly positioning Objects around the circuit. Correct?

Now, I want to ask an important question about how I get ONLY THE LiDAR TILES I NEED for the project that is at the TOP of my list: a 9-mile point-to-point track of Joplin Road that cuts through Marine Corps Base Quantico. I TRIED to use the NOAA Elevation 'Draw Search Area' tool in the NOAA Data Access Viewer to get just the tiles I needed for this, but I was not given the 'Shopping Cart' option for that AOI, only the 'Bulk Download' option that I cited when I first asked for your help.

As you pointed out to me two weeks ago in our "What to do with LiDAR data files!?" Conversation at RaceDepartment, the Dewberry Project Report included a map of all of the tiles produced for the project.

Map: http://www.nogripracing.com/gallery/...p?photo=181736

As described in the report:
The LiDAR delivery consists of two thousand one hundred and forty nine (2149) tiles (Figure 1).
Each tile’s extent is 5000 feet by 5000 feet. This conforms to the Orthophotography and highresolution elevation tile grid developed by the state of Virginia Geographic Information
Network.

The projection information is:
Horizontal Datum: NAD83 HARN
Vertical Datum: NAVD88
Projection: State Plane
Zone: Virginia North (FIPS 4501) & South (FIPS 4502)
Units (Horizontal & Vertical): U.S. Survey Feet
Geoid: Geoid09
Now, as I just wrote, when I tried to get LiDAR for just the Joplin Road AOI I defined using the 'Draw Search Area' tool in the NOAA Data Access Viewer, I was pointed only to the 'Bulk Download' page:

https://coast.noaa.gov/htdata/lidar1...12b/data/4990/

So, the question is: Is there a cross-reference MAP anywhere with which I can determine which of these 2,000+ 'Point Cloud Data Files' I need? I don't want to download all 2,000 (62 GB!) and then open each to see if I "recognize" the terrain! Do you see a link to a reference map???


So I'm thinking I'm NOT going to be using LAStools, after all, right? CloudCompare gives me everything I need to prepare my Point Cloud to be imported to BTB? Will I be splitting the file(s) into 250K files using CloudCompare, or will I just be doing that using Notepad, cutting and pasting lines in the CSV (TXT?) files?

Thanks again!


Last edited by Wee Scot; Yesterday at 20:24.
Wee Scot is offline   Reply With Quote
Unread Today, 00:42   #76
Emery
 
Join Date: Apr 2008
Location: Fiat Farm, in the wettest corner of Oregon
Age: 54
Default

You'll want to assemble the desired tiles in CloudCompare, trim, and save for each project. Then do your translation and split out the layers for each project. I usually keep them in LAS until the editing is done and then save as CSV, then split the CSV into multiple files.

For splitting the CSV files, I paid for a CSV editor called "Delimit" since it handles larger CSV files than any other editor I could find and can automatically split those big files; if I wasn't being paid to work on large files, I wouldn't have spent the money. Spreadsheets are limited to a little over a million lines. A few editors can handle 4-10 million lines. Worst case scenario, if you don't want to pay for an editor, is you should split the file up inside CloudCompare since you can easily see how many points are selected before saving the CSV file.

When you import the cloud files in BTB Pro, they are only points. Each file you import can be turned on/off for visibility. You create the mesh the same way you always do in BTB and you can use a background image as a guide.
Emery is offline   Reply With Quote
Unread Today, 01:03   #77
Emery
 
Join Date: Apr 2008
Location: Fiat Farm, in the wettest corner of Oregon
Age: 54
Default

Try making your AOI smaller for Joplin Road and assembling the pieces. 9 miles likely triggers the bulk download. It's okay to have duplicate points because you should be able to filter them out.

And make darned sure your units are meters to save you another step in scaling the cloud.
Emery is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT. The time now is 12:10.
Home - Top

Powered by vBulletin® - Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.

www.nogripracing.com 2003 - 2017
Page generated in 0.05374 seconds with 10 queries