Firstly, the new metadata files are formatted for easy parsing. But more importantly, we now have a pair of parameters titled RADIANCE_MULT_BAND_* and RADIANCE_ADD_BAND_* (one pair for each band). With these parameters we can calculate the Top Of Atmosphere (TOA) reflectance directly, without the need for the intermediary step of radiometric calibration. Read the Using USGS Landsat 8 website for full details. These two parameters are parallel to the well known gain and bias parameters from earlier Landsat missions, however they already take into account Esun and the earth-sun distance. Thus reflectance can be obtained straight-away. We only need to divide by the sun zenith angle to get corrected TOA relectance.

We GRASS users can easily put the above numbers into an `r.mapcalc`

expression to get reflectance for each band. But we also want to take advantage of the scripting capabilities of GRASS to batch process all bands for several tiles. Choosing python for our scripting language we have access to the OS libraries we need as well as the GRASS python library. We first loop through all the Landsat 8 directories in some top level folder where we downloaded the original tiles. We read through the metadata file for each tile, creating a python dictionary of the entries we need. Now we implement an inner loop to import each of the individual bands in the tile as a GRASS raster. Then we run the mapcalc module on each band, creating TOA reflectance for that band. When the inner loop finishes importing and processing all the bands, the outer loop moves to the next Landsat directory and cycles thru the bands in that tile, etc.

For those interested in the nitty-gritty, you’re welcome to clone a small python script I’ve put on github that does the above.

]]>

… the seedling is doing surprisingly well at the end of the summer. No rain yet, but it seems quite green, and all the parameters I have been measuring increased substantially.

located at: Longitude 35d10'18.533"E and Latitude 30d47'33.321"N

Elevation -40 m

Date visited | canopy width (cm.) | height (max) | trunk diameter (cm.) |
---|---|---|---|

2/4/2011 | 58 | 22 | 0.14 |

28/1/2012 | 68 | 30 | 0.8 0.7 0.63 |

23/8/2012 | 72 | 35 | 0.8 0.76 0.57 |

17/11/2012 | 80 | 38 | 0.91 0.75 0.58 |

16/8/2013 | 82 | 51 | 0.96 0.88 0.67 |

18/4/2014 | 93 | 54 | 1.22 0.95 0.69 |

11/10/2014 | 112 | 70 | 1.40 0.98 0.72 |

]]>

I wrote about creating isohyetal contours some years ago. This time I’ll try to improve my interpolation by using the powerful, open source R statistics program.

We begin with the total annual precipitation measurements at discrete locations – rain gauges. Using this point data we interpolate the rainfall everywhere in the region. The interpolation method I choose is kriging, which works with a variance distribution function, the variogram, that we prepare as part of the procedure. Furthermore, the geostatistics procedure called co-kriging allows us to apply a second, independant variable to the analysis to predict even better the interpolated values. Co-kriging is applicable only when we know that there is some correlation between the main, dependant variable and the auxiliary variable. Intuitively, we are aware that in many areas rainfall is stronger in the mountains, and less at lower elevations. So, I will examine elevation of the rain gauges as the independant auxiliary variable.

We begin by starting R, loading some required libraries, and pulling in the rain gauge data.

<br /> library(sp)<br /> library(rgdal)<br /> library(gstat)<br /> # Load precipitation data<br /> rain.data <- "raindata_ttl_2014.csv"<br /> gauges <- read.csv(rain.data, col.names=c("gauge_id","precip","x","y"));<br />

Now I load a border line, just for spatial reference, and make a simple bubble plot of the data.

<br /> # Put data into a Spatial Data Frame<br /> gauges.xy <- gauges[c("x","y")]<br /> coordinates(gauges) <- gauges.xy<br /> border_il <- readOGR('border_il.shp', 'border_il')<br /> # Set the correct CRS for the gauges data<br /> itm_proj4 <- proj4string(border_il)<br /> proj4string(gauges) <- itm_proj4<br /> class(gauges)<br /> summary(gauges)<br /> # Have a look<br /> plot(gauges, pch=16, col='blue', cex=precip/20)<br /> plot(border_il, add=TRUE)<br />

Now we need to determine the variogram. We start by plotting some candidate variogram models. The parameters that we need to set are “cutoff” and “width”. By default, the cutoff (how far from each prediction point to go to look for real points) is chosen as one third the diagonal of the whole analysis region. And the width (the variance between real points within the cutoff) is 1/15 of the cutoff. We can get an idea how these parameters influence the model variograms by plotting some possible combinations. Once we have a good looking variogram model, we use those in the gstat variogram() function to create the model variogram:

<br /> # Create a variogram<br /> # Display some possible models<br /> plot(gstat::variogram(precip ~ 1 , gauges, cutoff=20000, width=2000))<br /> plot(gstat::variogram(precip ~ 1 , gauges, cutoff=30000, width=4000))<br /> plot(gstat::variogram(precip ~ 1 , gauges, cutoff=70000, width=5000))<br /> # Once we have a visually good variogram, use those parameters:<br /> vg <- gstat::variogram(precip ~ 1 , gauges, cutoff=70000)<br />

Now we have a model, and we want to let R to actually fit a variogram to the point pairs that are included in the variogram model to create a fitted curve. The following code prints the final fitted variogram values and gives us the result below:

<br /> vg.fit <- gstat::fit.variogram(vg, vgm(700, "Exp", 20000, 300))<br /> # Here are the fitted variogram parameters:<br /> print(vg.fit)<br /> plot(vg, vg.fit)<br /> model psill range<br /> 1 Nug 355.0346 0.00<br /> 2 Exp 421.1879 12672.45<br />

With that we are now prepared to run the kriging and produce a raster of predicted rainfall throughout the region.

<br /> # create a grid onto which we will interpolate:<br /> # first get the range in data<br /> x.range <- as.integer(range(gauges@coords[,1]))<br /> y.range <- as.integer(range(gauges@coords[,2]))<br /> # now expand to a grid with 100 meter spacing:<br /> grd <- expand.grid(x=seq(from=x.range[1], to=x.range[2], by=100), y=seq(from=y.range[1], to=y.range[2], by=100) )<br /> # convert to SpatialPixel class<br /> coordinates(grd) <- ~ x+y<br /> gridded(grd) <- TRUE<br /> proj4string(grd) <- itm_proj4</p> <p># perform ordinary kriging prediction:<br /> # make gstat object to hold the krige result:<br /> g <- gstat(id="Precipitation", formula=precip ~ 1, data=gauges, model=vg.fit)<br /> precip.krige <- predict(g, model=vg.fit, newdata=grd)<br />

The predict() function chooses ordinary kriging, based on the basic parameters we passed to the function. Now we can go ahead and plot the result. I load the RColorBrewer library to take advantage of the full set of color palettes available, and I use the spplot function from the sp package for plotting, since it easily adds contours (isohyetal lines) to the plot.

<br /> # Some parameters for plotting<br /> par(font.main=2, cex.main=1.5,cex.lab=0.4, cex.sub=1)<br /> # Use the ColorBrewer library for color ramps<br /> library(RColorBrewer)<br /> precip.pal <- colorRampPalette(brewer.pal(7, name="Blues"))<br /> # plot the krige interpolation<br /> spplot(precip.krige, zcol='Precipitation.pred', col.regions=precip.pal, contour=TRUE, col='black', pretty=TRUE, main="Interpolated Precipitation - 2014", sub="Ordinary Kriging", labels=TRUE)<br />

The plot of krige interpolated precipitation looks like this:

Now, as promised, let’s continue on to elevation data. We import a DEM and use the R over() function to attach an elevation value to each gauge. Then we check correlation between the annual precipitation and elevation. These are the steps:

<br /> # Now start with elevation data<br /> elev <- readGDAL('dem_negev.tif')<br /> class(elev)<br /> summary(elev)<br /> # Add elevation values to each gauge<br /> gauges.elev <- over(gauges, elev)<br /> precip.elev <- cbind(gauges, gauges.elev)<br /> coordinates(precip.elev) <- gauges.xy<br /> proj4string(precip.elev) <- itm_proj4<br /> # Check that we have the elevations in a spatial data frame<br /> names(precip.elev)<br /> # the column "band1" contains the elevation values<br /> # Now check for correlation between precipitation and elevation<br /> cor.test(precip.elev$band1, precip.elev$precip)<br />

Checking the output of the cor.test() function we see:

<br /> Pearson's product-moment correlation<br /> data: precip.elev$band1 and precip.elev$precip<br /> t = 0.331, df = 96, p-value = 0.7414<br /> alternative hypothesis: true correlation is not equal to 0<br /> 95 percent confidence interval:<br /> -0.1657725 0.2306346<br /> sample estimates:<br /> cor<br /> 0.03375868<br />

Well, a correlation of 0.033 means essentially

But, we don’t give up so easily. Let’s go back a few years to the winter of 2010. Anyone dealing with rainfall and flooding in the Negev desert remembers that season and the storms of January. We had once in 50 year floods in several wadis throughout the south due to very heavy rains in the Negev mountains. Let’s search thru the archives to pull out the annual rainfall data for that year, and try to correlate with elevation. I rerun all the above steps for the gauge data for 2010. Of course the variogram will be different, and I obtain the ordinary kriging result as below:

Now I load the same elevation dataset and as above I test correlation between the precipitation data and elevation values:

<br /> cor.test(precip.2010.elev$band1, precip.2010.elev$precip)<br /> Pearson's product-moment correlation</p> <p>data: precip.2010.elev$band1 and precip.2010.elev$precip<br /> t = 6.6778, df = 71, p-value = 4.552e-09<br /> alternative hypothesis: true correlation is not equal to 0<br /> 95 percent confidence interval:<br /> 0.4562349 0.7447525<br /> sample estimates:<br /> cor<br /> 0.6211078<br />

Well, a correlation of 0.62 is nothing to write home about, but is does indicate some reasonable level of correlation. Additionally, the p-value is very low, meaning that there is a high level of confidence that the resulting correlation is accurate. So let’s forge ahead and use the elevation values to (hopefully) improve our precipitation estimation.

We first need to redo the gstat object to hold both variables:

<br /> # recreate g, the gstat object with a second variable<br /> rm(g)<br /> g <- gstat(id="Precip", formula=precip ~ 1, data=precip.2010.elev, model=vgm(psill=2000, model="Exp", range=100000, nugget=50))<br /> g <- gstat(g, id="Elevation", formula=band1 ~ 1, data=precip.2010.elev, model=vgm(psill=100000, model="Exp", range=50000, nugget=10))<br /> vg <- gstat::variogram(g)<br /> # Make the multivariable variogram (Linear Model of Coregionalization)<br /> vg.fit <- fit.lmc(vg, g, vgm(psill=2000, model="Exp", range=100000, nugget=50))<br /> # Graph the model and experimental variograms<br /> plot(vg, vg.fit)<br />

Note that we us the fit.lmc() function to create a fitted variogram with multiple variables, each variable with its own set of sill and range parameters. And we must add a non-zero nugget value to insure that the Linear Model of Coregionalization will be chosen.

The plot() shows us three variograms: one for elevation, one for precipitation, and a third for the covariance of the two variables together.

What’s left is to rerun the predict() function, and note that it chooses co-kriging:

<br /> # Now predict() should create a CoKriging interpolation<br /> precip.2010.cokrige <- predict.gstat(vg.fit, newdata=grd)<br /> Linear Model of Coregionalization found. Good.<br /> [using ordinary cokriging]<br />

and here’s our new estimate of distribution of precipitation for 2010. Compared to the ordinary kriging map for 2010 above, we can identify an obvious change of rainfall distribution in the western mountain regions:

Here’s the updated measurements of my acacia seedling:

located at: Longitude 35d10'18.533"E and Latitude 30d47'33.321"N

Elevation -40 m

Date visited | canopy width (cm.) | height (max) | trunk diameter (cm.) |
---|---|---|---|

2/4/2011 | 58 | 22 | 0.14 |

28/1/2012 | 68 | 30 | 0.8 0.7 0.63 |

23/8/2012 | 72 | 35 | 0.8 0.76 0.57 |

17/11/2012 | 80 | 38 | 0.91 0.75 0.58 |

16/8/2013 | 82 | 51 | 0.96 0.88 0.67 |

18/4/2014 | 93 | 54 | 1.22 0.95 0.69 |

11/10/2014 | 112 | 70 | 1.40 0.98 0.72 |

]]>

GRASS numbers the directions in a drainage (flow direction) raster from 1-8 starting eastward and going counter clock-wise. ArcGIS also numbers the directions from the east but clock-wise, and in steps of powers of 2: 1,2,4,8,16, etc. This allows ArcGIS to calculate “in-between” directions when multiple surrounding cells have the same elevation, and the flow direction is not unique. GRASS, on the other hand, handles this situation by “looking further” along the flow direction to find where the overall flow is going, and then determines which of the multiple directions to choose. This is called a least-cost search algorithm, and should produce better results especially in flat and nearly flat regions.

So I want to create the flow_dir raster in GRASS, but the results need to use the ArcGIS numbering scheme. This is accomplished simply with a reclass command. Here’s the reclass rules file I use:

echo "1 -1 = 128

2 -2 = 64

3 -3 = 32

4 -4 = 16

5 -5 = 8

6 -6 = 4

7 -7 = 2

8 -8 = 1

0 = 255" > f_dir_reclass

First, you might ask why the negative numbers? GRASS uses negative direction to indicate that the flow from this cell is going off the map. But the absolute value still indicates the direction. So both 4 and -4 mean west, which in Arc terms is 16. Also, GRASS uses 0 to indicate a sink = no flow at all from this cell. ArcGIS implements a method to insure that single cell sinks are not used by combining values from different flow directions. So I simply set this situation to 255 = the sum of all the other directions.

Now I need only do:

`r.reclass input=f_dir output=f_dir_arc rules=f_dir_reclass`

to get a reclassed raster in the Arc style of numbered directions.

]]>located at: Longitude 35d10'18.533"E and Latitude 30d47'33.321"N

Elevation -40 m

Date visited | canopy width (cm.) | height (max) | trunk diameter (cm.) |
---|---|---|---|

2/4/2011 | 58 | 22 | 0.14 |

28/1/2012 | 68 | 30 | 0.8 0.7 0.63 |

23/8/2012 | 72 | 35 | 0.8 0.76 0.57 |

17/11/2012 | 80 | 38 | 0.91 0.75 0.58 |

16/8/2013 | 82 | 51 | 0.96 0.88 0.67 |

18/4/2014 | 93 | 54 | 1.22 0.95 0.69 |

11/10/2014 | 112 | 70 | 1.40 0.98 0.72 |

I’ll look forward to another visit thru the winter – hopefully after some rainfall – when I expect to see even more growth.

]]>The liblas set of utilities includes lasinfo, which lists all the metadate in a *.las file. Among the details is the “Bounding box” of that file – the minimum and maximum X and Y values of the points. So by grepping for the term “Bounding Box” and filtering out only the values, I could get these coordinates into a comma-separated-values text file. Here’s the command (run in the directory where the las files are located):

<br /> for f in *.las; do echo -n $f, &gt;&gt; bboxes.csv; echo `lasinfo $f | grep "Bounding Box" | awk '{print $3,$4,$5,$6}'` &gt;&gt; bboxes.csv; done

I use a “for” expression to loop thru all the *.las file. The first echo statement puts the file name, followd by a comma into my output file “bboxes.csv”. The “-n” flag causes echo **not** to write a new line. Then the lasinfo command outputs the bounding box row, and pushes it thru the “awk” expression to get the four corner values, then feeds them into the same bboxes.csv file. Here’s what the result looks like:

pt000001.las,215894.56, 489948.39, 216959.19, 491281.06<br /> pt000002.las,215381.44, 490135.64, 215916.49, 491311.03<br /> pt000003.las,218052.39, 498030.01, 220082.14, 499775.40<br /> .....<br />

For convenience, I add a row of column header names: “file”,”min_x”,”min_y”,”max_x”,”max_y” to the CSV file. And I make sure there are no stray spaces between the columns, with a quick search and replace in a text editor. (Beware: spaces will cause spatialite to import the data as text, rather than real numbers!).

Now I start spatialite_gui, and import that csv file into a table “bboxes” in a new database. Next I create a new table “las_tiles” for the polygon tile indices as follows:

CREATE TABLE las_tiles (pk_uid integer primary key autoincrement, file text);<br /> SELECT AddGeometryColumn('las_tiles','geometry', 4326, 'POLYGON',2);

Now I can reach for the spatialite function

to create polygons from each set of the bounding box corners, as follows:**BuildMBR()**

INSERT INTO las_tiles (file, geometry)<br /> SELECT file, BuildMBR(min_x, min_y, max_x, max_y, 4326)<br /> FROM bboxes;<br />

And with that, I now have my polygon layer of the coverage of each of the the *.las files. Now on to some “real” work.

]]>I tried using, for example, GPSEssentials, an Android app with all the bells and whistles. I’m not endorsing any one location app but this one (like some others) stores all its data into sqlite files. So I plugged my phone into a USB socket and copied the “waypoints” file from the gpsessentials folder over to my computer. Then I renamed the file to add the popular *.sqlite extension, and opened it in Spatialite_gui. For our purposes there are three tables of interest: Waypoint, Track, and TrackElement. The Waypoint table contains, of course, the waypoints with Longitude and Latitude coordinates, along with altitude, description, etc. The Track table is a list of tracks, and the TrackElement table is a crumb trail of Longitude/latitude values for each location along each of the tracks.

To transform these tables into true spatial layers, for use in GIS, you must first make the sqlite DB spatial. So:

`SELECT InitSpatialMetaData();`

Now we can use the Longitude/Latitude values in the waypoints table to create actual geometries. First call the AddGeometryColumn function:

`SELECT AddGeometryColumn('Waypoint','geometry',4326,'POINT','XY');`

UPDATE Waypoints SET geometry = MakePoint(longitude, latitude, 4326);

and Waypoint becomes a point layer ready to be opened in QGIS, for example, or exported to a shapefile.

But what about the tracks? Here we need an additional step. We make the Track **and **TrackElement tables spatial, then use the TrackElements to create LINESTRING features in Track geometry column:

`SELECT AddGeometryColumn('Track','geometry',4326,'LINESTRING','XY');`

SELECT AddGeometryColumn('TrackElement','geometry',4326,'POINT','XY');

UPDATE TrackElement SET geometry = MakePoint(longitude, latitude, 4326);

and now do the update on the Track table:

`UPDATE Track SET geometry = (SELECT MakeLine(te.geometry)`

FROM TrackElement as te

WHERE te.trackID = Track._id

GROUP BY te.trackID)

and you’re done. The Track table is now a true line feature, and can be exported as a shapefile, etc. The column names above are as they appear in the sqlite tables created by this particular app, so you might have to replace some of the ingredients. But the recipe should be similar. So get out your spatial mixing bowl, and your phone becomes a handy GIS tool.

]]>

However the move raised a public controversy. All four of the poets are Ashkenazi, from Eastern European decent. Why no Sephardic poets? Silvan Shalom asked where are Yehuda Halevi, Shabazi or Ibn Gvirol? And other MP’s of both Sephardic and Ashkenazi backgrounds voiced similar objections. Of course the debate also brought on a flurry of parodies and satirical designs for the new notes with portraits of anything from pop singers to ninja turtles…

One talk show hosted Professor Roni Reich from the Archaeology department of Hebrew University. He was a leading member of the committee appointed by the Bank of Israel, and responsible for initiating the idea of depicting well know cultural figures rather than politicians. He explained what led him and the rest of the committee to their decision. As to the question of cultural figures from North African decent, he thought that the opinions were legitimate, and in 10 or 12 years time, when the next set of bills will be designed, careful attention should be placed on the backgrounds of the people chosen to decorate the currency. The current designs were made public months ago, and no objections were voiced. Now with the bills ready to go to print, it’s too late for changes.

As soon as I saw Professor Reich’s face, I recognized him from 40 years ago, when he was just beginning his academic career, and heading the excavations at Tel Ashdod. With his bushy hair and sharp features, he would bound from wall to pit, giving us volunteer diggers instructions where to shovel and when to brush. He always had a joke or good word to keep our spirits up through the hot summer days. Most of his professional career he dedicated to digs in and around Jerusalem. Now, after decades of sorting through pottery shards and ancient coins, writing books, and bringing archaeology to the “masses” he found himself helping design our new currency.

]]>Luckily, we had ordered in advance LIDAR coverage of the whole area, so I have a high resolution (1 meter per pixel) DTM of the reservoir and streams which feed into it during flash floods. I first visually examined the elevation values at the spillover point where the reservoir drains when it fills, and the lowest point in the center of the depression. I then ran the GRASS module r.lake feeding it the x,y coordinates of the deepest point as the “seed” and the elevation just below the spillover as the water level parameters. The result gave me the lake’s depths as a raster. Here’s the result:

Now I ran r.univar on the lake raster, to find

- the maximum value = water depth
- the sum of all pixels = total volume of the reservoir
- the number of pixels = surface area covered by water

Since the region settings were 1 meter per pixel, the number of cells = area in sq.m. and the sum of all cells = volume in cubic meters.

Now I wanted to run the same procedure for water levels stepping down from the top spillover level to the bottom of the reservoir when it dries up due to leaching and evaporation. A script suggested itself, and here’s the `depth_volume.sh`

script that I conjured up:

<br /> #!/bin/bash<br /> # Author: micha, 7/11/2012; copyright: GPL &gt;= 2<br /> # Purpose: calculate volume, area and depth of a reservoir<br /> # from a DEM raster and given water level using r.lake<br /> # calculate volume and area with r.univar<br /> # export the results to a text file for use in gnuplot<br /> # Usage: depth_volume.sh {dem raster} {water level} {seed location as x,y}</p> <p>if [ -z "$GISBASE" ] ; then<br /> echo "You must be in GRASS GIS to run this program." 1&gt;&amp;2<br /> exit 1<br /> fi</p> <p>if [ $# -lt 3 ] ; then<br /> echo "Some inputs not defined. Usage: $0 dem_raster water_level seed_x,seed_y"<br /> exit 1<br /> fi</p> <p>DEM=$1<br /> LEVEL=$2<br /> SEED=$3</p> <p>g.message "Flooding lake at water level $LEVEL"<br /> # Get resolution to calculate area and volume based on pixel size<br /> NSRES=`g.region -g | grep nsres | cut -d= -f2`<br /> EWRES=`g.region -g | grep ewres | cut -d= -f2`<br /> PIXEL=`echo $NSRES*$EWRES | bc`<br /> # Run r.lake<br /> r.lake --quiet --o $DEM wl=$LEVEL xy=$SEED lake="$DEM"_tmp</p> <p># Collect needed numbers from r.univar<br /> eval `r.univar -g "$DEM"_tmp`<br /> VOLUME=`echo $sum*PIXEL/1000 | bc`<br /> DEPTH=$max<br /> AREA=`echo $n*$PIXEL | bc`</p> <p># Dump results into a text file<br /> echo "$LEVEL $DEPTH $VOLUME $AREA" &gt;&gt; depth_volume.txt<br />

The script takes three inputs: the DEM, water level and the X-Y coordinates for seeding the lake (as mentioned I chose the lowest spot in the depression). After running r.lake, the r.univar output is parsed to collect the needed numbers, and the results are dropped into a text file.

Now I ran this script in a loop with all the water levels that interested us. We wanted water volumes for each 1/2 meter drop in the water surface, so:

for level in 27 26.5 26 25.5 25 24.5 24 23.5 23 22.5; \<br /> do depth_volume.sh dem $level 194365,397295; done

The depth_volume.txt file that is created has water level, volume and surface area for each of the above elevations. Now, to create a depth-volume curve, I could pull this text file into LibreOffice calc for designing a fancy chart. But for a quick and simple display, I used gnuplot with these commands:

set title "Depth-Volume Curve"<br /> set xlabel "Volume 1000 m3"<br /> set ylabel "Depth m."<br /> set x2label "Area sqm."<br /> set key left box<br /> set x2tics nomirror<br /> set autoscale x2<br /> set grid y<br /> set terminal png enhanced size 800,600<br /> set output "depth_volume.png"<br /> plot "depth_volume.txt" using 3:2 title "Volume 1000 m3" with lines lw 3, \<br /> "depth_volume.txt" using 4:2 title "Area m2" with lines axes x2y1<br />

The graph looks like this:

Once I have setup the GRASS script and gnuplot command file, running the model with other parameters or in some other location is a piece of cake.

]]>