SELECT InitSpatialMetaData();

Now we can use the Longitude/Latitude values in the waypoints table to create actual geometries. First call the AddGeometryColumn function:

SELECT AddGeometryColumn('Waypoint','geometry',4326,'POINT','XY');
UPDATE Waypoints SET geometry = MakePoint(longitude, latitude, 4326);

and Waypoint becomes a point layer ready to be opened in QGIS, for example, or exported to a shapefile.
But what about the tracks? Here we need an additional step. We make the Track and TrackElement tables spatial, then use the TrackElements to create LINESTRING features in Track geometry column:

SELECT AddGeometryColumn('Track','geometry',4326,'LINESTRING','XY');
SELECT AddGeometryColumn('TrackElement','geometry',4326,'POINT','XY');
UPDATE TrackElement SET geometry = MakePoint(longitude, latitude, 4326);

and now do the update on the Track table:

UPDATE Track SET geometry = (SELECT MakeLine(te.geometry)
FROM TrackElement as te
WHERE te.trackID = Track._id
GROUP BY te.trackID)


and you’re done. The Track table is now a true line feature, and can be exported as a shapefile, etc. The column names above are as they appear in the sqlite tables created by this particular app, so you might have to replace some of the ingredients. But the recipe should be similar. So get out your spatial mixing bowl, and your phone becomes a handy GIS tool.

DTM and reservoir created with r.lake

Now I ran r.univar on the lake raster, to find

• the maximum value = water depth
• the sum of all pixels = total volume of the reservoir
• the number of pixels = surface area covered by water

Since the region settings were 1 meter per pixel, the number of cells = area in sq.m. and the sum of all cells = volume in cubic meters.

Now I wanted to run the same procedure for water levels stepping down from the top spillover level to the bottom of the reservoir when it dries up due to leaching and evaporation. A script suggested itself, and here’s the depth_volume.sh script that I conjured up:


#!/bin/bash
# Author: micha, 7/11/2012; copyright: GPL >= 2
# Purpose: calculate volume, area and depth of a reservoir
# from a DEM raster and given water level using r.lake
# calculate volume and area with r.univar
# export the results to a text file for use in gnuplot
# Usage: depth_volume.sh {dem raster} {water level} {seed location as x,y}

# Dump results into a text file
echo "$LEVEL $DEPTH $VOLUME $AREA" >> depth_volume.txt


The script takes three inputs: the DEM, water level and the X-Y coordinates for seeding the lake (as mentioned I chose the lowest spot in the depression). After running r.lake, the r.univar output is parsed to collect the needed numbers, and the results are dropped into a text file.
Now I ran this script in a loop with all the water levels that interested us. We wanted water volumes for each 1/2 meter drop in the water surface, so:

for level in 27 26.5 26 25.5 25 24.5 24 23.5 23 22.5; \
do depth_volume.sh dem $level 194365,397295; done

The depth_volume.txt file that is created has water level, volume and surface area for each of the above elevations. Now, to create a depth-volume curve, I could pull this text file into LibreOffice calc for designing a fancy chart. But for a quick and simple display, I used gnuplot with these commands:

set title "Depth-Volume Curve"
set xlabel "Volume 1000 m3"
set ylabel "Depth m."
set x2label "Area sqm."
set key left box
set x2tics nomirror
set autoscale x2
set grid y
set terminal png enhanced size 800,600
set output "depth_volume.png"
plot "depth_volume.txt" using 3:2 title "Volume 1000 m3" with lines lw 3, \
"depth_volume.txt" using 4:2 title "Area m2" with lines axes x2y1


The graph looks like this:

Depth Volume curve from gnuplot

Once I have setup the GRASS script and gnuplot command file, running the model with other parameters or in some other location is a piece of cake.

In Nathan W’s blog he introduced Expression based labeling. Now we can construct labels using combinations of math functions and string functions to manipulate any attribute column the way we want. Currently, in versions 1.8.x, find and click the “ABC” button to access the “Layer Label settings” window. Stay tuned for the next version of QGIS, when this New Labeling will become the default, and be merged into regular Layer Properties window. In the Layer Label settings window, after checking the checkbox to “Label this layer with” you can click the ellipsis button [...] to open the full expression builder window, and use your imagination to create a variety of informative labels.

Here’s a classic example. If I add an Area column (using the Field calculator, $area function) the values I get, in a projected coordinate system. are square meters. But suppose I want to label a polygon layer with hectares? The label expression will be:
"Area" / 10000
But that’s not enough. This expression returns a real number with a long string of digits after the decimal point. So we instead can enter into the expression calculator:
toint( "Area" / 10000)
to round the number to an integer. And one more improvement: what about adding the “ha” tag as units to the labels?
toint( "Area" / 10000) || ' ha'

We also have the ability now to create conditional labels thanks to Martin Dobias. One example appears in Underdark’s blog. Here’s another straight forward example: I have a point layer of rain gauges. I want to label all those gauges with precipitation greater than 2 mm. No label at all should appear when the gauge shows < 2 mm.
CASE WHEN ("precip" >= 2) THEN "precip" END
Of course several WHEN {condition} THEN {label} pairs can be chained together to get several different categories of labels.

We begin with an elevation raster call ‘dtm’. This could be data from the ASTER GDEM program or SRTM tiles. Zoom in to the general area of interest and (as always) set the GRASS region to match that area.

g.region -p n=30:54N s=30N e=35:30E w=34:36E

First we run two r.stream.* commands to create the stream network, and a database table of stream ordering.

r.stream.extract elev=dtm thresh=25000 stream_rast=streams_25 stream_vect=streams_25 direction=fdir_25 
r.stream.order stream=streams_25 dir=fdir_25 table=stream_order

The r.stream.extract command uses the dtm as input, together with a threshold value which we determine. The threshold represents the number of raster cells which will become a minimim drainage catchment in the resulting stream network. You need to know the size (resolution) of the cells in your dtm raster in order to do the arithmetic for this threshold value. In the example above, the dtm (based on ASTER GDEM data) has a resolution 30 meters, so 25,000 cells is about 20 sq. km.  Then the r.stream.order command uses the newly created streams_25 map of streams to create a table of stream ordering. The command places the table into the default database used by GRASS. In this case sqlite.

Before continuing, we’ll have a look at what we’ve got.

v.db.select streams_25 | head cat|stream_type|type_code 1|start|0 2|start|0 3|start|0 4|intermediate|1 5|start|0

So the vector stream map has only three columns – the cat value, and a name and code indicating if the stream segment is a headwater segment, or further down in the hierarchy. To examine the stream_order table, we need to have a look right in the sqlite database, so:

eval `g.gisenv` 
sqlite3 $GISDBASE/$LOCATION_NAME/$MAPSET/sqlite.db 
sqlite> .header on

A quick SELECT statement on the streams_25 table (attached to the stream_25 vector map) will show the same data as above.

sqlite> SELECT * FROM streams_25 LIMIT 5; 
cat|stream_type|type_code 
1|start|0 
2|start|0 
3|start|0 
4|intermediate|1 
5|start|0

Next we do a SELECT on the stream_order table:

sqlite> SELECT * FROM stream_order LIMIT 10; 
cat         stream      next_str    prev_str01  prev_str02  strahler    horton      shreve      hack        topo_dim    length      cum_length  out_dist     straight    fractal 
----------  ----------  ----------  ----------  ----------  ----------  ----------  ----------  ----------  ----------  ----------  ----------  -----------  ----------  ---------- 
1           1           20          0           0           1           1           1           2           38          929.78593   929.78593   7775.778998  676.39042   1.374629 
2           2           4           0           0           1           3           1           1           29          300.43355   300.43355   8053.78109   233.019313  1.289308 
3           3           4           0           0           1           1           1           2           29          3.828427    3.828427    7757.175967  3.605551    1.061815 
4           4           10          3           2           2           3           2           1           28          667.10974   967.543289  7753.34754   549.315938  1.214437 
sqlite> .quit

So this second table has all the good stuff – next stream cat number, previous streams, the ordering in four different systems, as well as length, cumulative length for each stream reach and so on. We want to attach this table to the streams_25 map. This is accomplished with v.db.connect. The ‘-o’ flag indicates that we are overwriting the original table connection. And the ‘key=cat’ parameter says that the cat values in the geometry match the ‘cat’ column in the sqlite table. Now we quit sqlite, and back in GRASS we issue:

v.db.connect -o streams_25 driver=sqlite table=stream_order key=cat

If we now run a SELECT on the streams_25 map, we find:

v.db.select streams_25 | head 
cat|stream|next_str|prev_str01|prev_str02|strahler|horton|shreve|hack|topo_dim|length|cum_length|out_dist|straight|fractal
1|1|20|0|0|1|1|1|2|38|929.78593|929.78593|7775.778998|676.39042|1.374629
2|2|4|0|0|1|3|1|1|29|300.43355|300.43355|8053.78109|233.019313|1.28930
3|3|4|0|0|1|1|1|2|29|3.828427|3.828427|7757.175967|3.605551|1.061815
4|4|10|3|2|2|3|2|1|28|667.10974|967.543289|7753.34754|549.315938|1.214437
5|5|19|0|0|1|4|1|1|39|1050.680374|1050.680374|8229.62171|817.765247|1.284819
6|6|12|0|0|1|1|1|2|26|215.865007|215.865007|7098.509878|174.103418|1.239867

All the stream ordering and other data is now part of the streams_25 vector map. We next run the r.stream.basins command to create a map of the basins. The ‘-l’ flag is set so that only major basins  (with a outlet at the edge of the current region) are delineated.

r.stream.basins -l dir=fdir_25 stream=streams_25 basin=major_basins

Now we’re ready to make use of the data and maps we’ve created. We create a shaded relief, and overlay the basins with transparency.  In addition, In the GRASS GUI, you can display a line vector at varying widths, taking the width from an attribute column. We’ll use the strahler order values to make stream vector lines at the top slopes of the basin appear thin, and those nearer to the basin outlet will be thicker. Here’s the result.

Faran basin – wadis displayed with strahler order

Often you’ll need to isolate one particular sub-basin and display only that basin and it’s stream network. Here’s a way to do that. We first display the stream_25 raster map, and zoom in very close to the outlet of the sub-basin we’re trying to isolate. We need to get the exact X-Y coordinates at the outlet of that basin. In the GRASS GUI  Map Display window we can use the “Query raster/vector map” button (the gui version of the r.what command) to get the coordinates by clicking on the drainage point. The coordinates appear in the “Command console” of the Layer Manager window. Copy/paste these coordinates to the terminal and run r.stream.basins a second time as follows:

r.stream.basins dir=fdir_25 coors=35.226820,30.632014 basin=basin_1

Display this map in the Map Display to be sure it’s the basin you need. Now it’s a good idea to reset the region parameters to “close-in” on this basin, in order to avoid large raster maps with NULL cells covering large areas around your isolated basin.

g.region n=30:45N s=30:18N e=35:15E w=34:36E -p
r.mask in=basin_1 
r.mapcalc nekarot_basin=basin_1 
r.mapcalc nekarot_dtm=dtm 
r.shaded.relief nekarot_dtm shade=nekarot_shade

We’ve created new rasters for the isolated basin, the dtm, and the shaded relief, all clipped to the area of the isolated basin by using a raster MASK. After completing those steps the mask is no longer necessary, we can remove it, and the temporary map ‘basin_1′ as well. Then use the regular GRASS commands to create an area vector of the isolated basin, and to clip the original streams vector to this basin.

r.mask -r 
g.remove rast=basin_1 
r.to.vect in=nekarot_basin out=nekarot_basin feature=area 
v.select --o ainput=streams_25 atype=line binput=nekarot_basin btype=area output=nekarot_streams operator=overlap

And the result looks like this:

Nekarot basin – wadis displayed with strahler order

[Correction: Spatialite does, of course, have built in functions, and users can create their own functions using the C API. However - unlike PostGIS - there is no support for creating SQL (or other procedural languages) based functions. SQlite does not have the equivalent of the CREATE FUNCTION command in SQL]

The EPSG codes for UTM zones can be calculated as (Longitude+186)/6 + 32600. In the southern hemisphere, it’s (Long+186)/6 + 32700. The last two digits of each EPSG code are the UTM zone. The third digit, 6 or 7 indicates N or S. We can use the Sqlite CASE WHEN…ELSE…END construct to get the correct EPSG code for a table of point locations as follows:

We begin with the freely available geonames cities15000 file. This is a tab delimited text file of some 22,000 cities (with population over 15,000) around the world. I linked to the table, (making a virtual table in Spatialite), and used the data to create a real table with more convenient column names. I then made the table spatial, and used the latitude and longitude columns to set the Geometry for all cities.

CREATE TABLE cities (
id integer primary key autoincrement, geoname_code integer, Name text, Name_en text,
lat double, lon double, Country_ISO text, pop integer, Country text);

INSERT INTO cities

(geoname_code, Name, Name_en, lat, lon, Country_ISO, pop, Country)

SELECT

col001, col002, col003, col005, col006, col009, col015, col018

FROM cities15000;

Now setup the Geometry column

SELECT AddGeometryColumn('cities','Geometry',4326,'POINT','XY');
UPDATE cities SET Geometry=MakePoint(lon, lat, 4326);

And here’s how to construct a query to get the UTM zone for each city:

SELECT Name_en AS "City", Country AS "Continent/Country",
CASE WHEN ST_Y(Geometry)>0 THEN

CASE WHEN ST_X(Geometry)=180 THEN
32660
ELSE
(CAST(ST_X(Geometry) AS INTEGER)+186)/6+32600
END

ELSE

CASE WHEN ST_X(Geometry)=180 THEN
32760
ELSE
(CAST(ST_X(Geometry) AS INTEGER)+186)/6+32700
END

END AS “UTM SRID”
FROM cities
WHERE SRID(Geometry)=4326;

A few comments:

• The final WHERE clause insures that no EPSG codes will be returned if the point layer is not in Lat/Lon WGS84 to begin with.
• The outer CASE statement separates Northern and Southern hemispheres.
• The inner CASE catches the edges where longitude values are 180 degrees.

Here are some results:

City          Continent/Country  UTM SRID  
------------  -----------------  ----------
Sao Jeronimo  America/Sao_Paulo  32722     
Jeremoabo     America/Bahia      32724     
Saint-Jerome  America/Montreal   32618     
Aguada de Pa  America/Havana     32617     
Esbjerg       Europe/Copenhagen  32632     
Jerez de la   Europe/Madrid      32630     
Jeremie       America/Port-au-P  32618     
Jerusalem     Asia/Jerusalem     32636     
West Jerusal  Asia/Jerusalem     32636     
Manjeri       Asia/Kolkata       32643     
Borujerd      Asia/Tehran        32639     
Jerada        Africa/Casablanca  32630     
San Jeronimo  America/Mexico_Ci  32615     
Jerez de Gar  America/Mexico_Ci  32613     
Jerantut      Asia/Kuala_Lumpur  32648     
Ijero-Ekiti   Africa/Lagos       32631     
Oud-Beijerla  Europe/Amsterdam   32631     
Steinkjer     Europe/Oslo        32632     
Jericho       Asia/Hebron        32636     
East Jerusal  Asia/Hebron        32636     
Oum Hadjer    Africa/Ndjamena    32634     
Tajerouine    Africa/Tunis       32632     
Njeru         Africa/Kampala     32636     
Jersey City   America/New_York   32618     
Las Tejerias  America/Caracas    32619

I pulled the data from my GPS using gpsbabel but you can also choose the Quantum GIS GPSTools plugin to download GPS data in GPX format. Now to move these layers straight into a Spatialite database I use the ogr2ogr utility. This is part of the GDAL toolset, the “Swiss Army Knife” of spatial data formats. It allows me to move vector data from one format to another, with several options to filter or alter the original data along the way.
Here’s how I start:

$ ogr2ogr -f SQLite -nln ride -dsco "SPATIALITE=YES" -t_srs "EPSG:4326" bike_rides.sqlite bike_rides.gpx tracks

The first option sets the output format ‘-f‘ as SQLite. Next I use ‘-nln‘ to choose a “New Layer Name” for the table to be created. The ‘-dsco‘ options means “DataSet Create Option” and here we dictate Spatialite. The next flag ‘-t_srs‘ sets the spatial reference system for the data layer. While not strictly necessary (GPS data is always Lon/Lat WGS84) it’s a good policy to always set the SRS explicitly. Now pay attention to the order of the next parameters: the target database is first, and the source (gpx file) is second. This is backwards from many other command line tools. Normally we expect first the “from” followed by the “to”. For ogr2ogr it’s: “destination” first and “source” following. And finally at the end of the above command we choose to bring only GPS tracks into the database.

If our GPX file contains several tracks (perhaps from previous bike rides) we can filter those out with a simple -sql option such as:

$ ogr2ogr -f SQLite bike_rides.sqlite bike_rides.gpx -nln amazia_ride -dsco "SPATIALITE=YES" -sql "SELECT name, number FROM tracks WHERE name='Amazia-2' " tracks


Next, I need to get the stops along the way, which were captured as waypoints. Similar to above:

$ ogr2ogr -f SQLite bike_rides.sqlite waypoints.gpx -nln stops -append waypoints

Now, I don’t user the ‘-dsco’ parameter since the database already exists. What’s more, I must specify ‘-append‘ to instruct ogr2ogr to add this layer as a new table in the existing sqlite database.

OK, how long was the whole ride? The built in Spatialite function GeodesicLength() calculates lengths of Lon/Lat features in meters, along the WGS84 ellipsoid.

spatialite> SELECT GeodesicLength(r.Geometry) AS “Total Length” FROM ride AS r;
Total Length
—————
9696.2803124924

But I want to know how much each section of the ride was. In other words, I want to see the accumulated distance at each of the stops. We have in spatialite a function ST_Line_Locate_Point() which returns the distance along a line feature for points near the line. (It actually finds the nearest point on the line for each of the input points). So we do:

spatialite> SELECT s.name AS "Stop name",
...> ST_Line_Locate_Point(r.Geometry, s.Geometry) AS "Stop Locations"
...> FROM ride AS r, stops AS s
...> GROUP BY s.name ORDER BY s.OGC_FID;
Stop name   Stop Locations
----------  --------------
start       0.0
rest 1      0.124783450768
water stop  0.361703783084
rest 2      0.562004941906
rest 3      0.712163267248
overlook    0.850818111308
end         0.996608235043


The OGC_FID field is automatically created by ogr2ogr as a primary key. So doing “ORDER BY” on that field insures that the stop points are displayed in the correct order. But the numbers above are not distances, but rather the fraction of the total line length where each point falls. Now multiplying these fractions by the total length from the first query above, I get:

spatialite> SELECT s.name AS "Stop name",
...> ST_Line_Locate_Point(r.Geometry, s.Geometry)*GeodesicLength(r.Geometry) AS "Accumlulated Distances"
...> FROM ride AS r, stops AS s
...> GROUP BY s.name ORDER BY s.OGC_FID;
Stop name   Accumlulated Distances
----------  ----------------------
start       0.0
rest 1      1209.93531700926
water stop  3507.18127087581
rest 2      5449.35745373272
rest 3      6905.3346675029
overlook    8249.77090219153
end         9663.3928087156

Nice. One last step, let’s put these distances permanently into the “stops” table.

spatialite> ALTER TABLE stops ADD COLUMN accum_dist float;
spatialite> UPDATE stops SET accum_dist=(SELECT
...> ST_Line_Locate_Point(r.Geometry, s.Geometry)*GeodesicLength(r.Geometry)
...> FROM tracks AS r, stops AS s
...> WHERE s.OGC_FID=stops.OGC_FID);

We use a subquery to UPDATE the stops table. Take note of the WHERE condition: the OGC_FID values from the stops table outside of the subquery must match the values from the same column inside the query.

spatialite> SELECT name AS "Stop name",accum_dist AS "Accumulated Distance" FROM stops;
Stop name Accumulated Distance
---------- --------------------
start       0.0
rest 1      1209.93531700926
water stop  3507.18127087581
rest 2      5449.35745373272
rest 3      6905.3346675029
overlook    8249.77090219153
end         9663.3928087156

There we are: accumulated distances for waypoints along a GPS track.

Download the source code for QGIS and untar in a directory of your choice. Before starting the configure and make steps there’s one simple edit required for EL6 and similar: under the python directory open the file CMakeLists.txt. You’ll find four lines like:
SET(SIP_EXTRA_OPTIONS ${PYQT4_SIP_FLAGS} -o -a ${CMAKE_BINARY_DIR}/python/qgis.core.api)
These are lines 64, 80, 94, 100
You need to remove the “-o” option from each. This is a flag for “sip” which handles bindings between c++ and python. This new flag was introduced in recent versions of sip, and it’s not available in the EL6 sip packages. Removing the extra flag has no ill effects on the compilation (I believe it’s only applicable to the comments in the source code). So each of the four lines should be:
SET(SIP_EXTRA_OPTIONS ${PYQT4_SIP_FLAGS} -a ${CMAKE_BINARY_DIR}/python/qgis.core.api)

If you prefer to make the above changes with ‘patch’ here’s a diff file provided by Volker Froehlich (Fedora GIS maintainer).
Now, following more or less the instructions from that same wiki page above, you create a subdirectory “build’, drop into that directory and run:
$ cmake -L -DGRASS_PREFIX=/usr/grass-6.4.2 -DGDAL_LIBRARY=/usr/lib64/libgdal.so ..
(Don’t skip the two dots .. at the end of the command)

The two specific directives to the cmake command indicate where the GRASS installation and libgdal are. These paths are based on GRASS and GDAL being installed from RPM’s. If you’ve cooked your own GRASS or GDAL, set the above appropriately. Now you’re ready to go:
$ make
$ sudo make install

One more step is required so that QGIS can “find” the GRASS libraries. Add a one line file into /etc/ld.so.conf.d which contains:
$ cat /etc/ld.so.conf.d/grass.conf
/usr/grass-6.4.2/lib

and do:
sudo ldconfig

Now you should be ready to enjoy the new, updated QGIS 1.8.0 ‘Lisboa’

• New functions such as ST_Expand, ST_Reverse, ST_Translate
• Updates to several other functions
• Uses the latest sqlite 3.7.13 with an improvement in rounding double precision numbers
• Uses the recent geos-3.3.2
• An easier format for using a SpatialIndex in a query

To install the suite you will need the ubuntugis-unstable repository. If you haven’t yet added that repo to your software sources, follow the instructions on this OSGeo wiki page
Then it’s just a matter of running:
sudo apt-get install --yes libspatialite spatialite-bin

This will give you the library (probably already installed if you have any GIS software at all on your system), and the CLI binary for working with spatialite data. If you’re a command line type, then you’ve got all you need to start building and querying a spatialite database. You also have the command line tools to import OSM data into a Spatialite database: spatialite_osm_raw, spatialite_osm_filter, etc. And the two utilites shp_doctor to repair shapefiles, and spatialite_tool to import and export data to/from a Spatialite database.

However, the GUI interface is another story. The ready built package is version 1.3 – very old. In fact, if you create a DB using the new CLI program, and add some data layers, it will not open in the ancient ver. 1.3 GUI program. I suggest to take a few minutes to download the sources and manually build your own spatialite_gui. Only a few standard steps in the terminal:

First you’ll need a bunch of development packages:
sudo apt-get install --yes libwxgtk2.8-dev libgeos-dev libgeos++-dev libgeotiff-dev libcairo2-dev libfreexl-dev libspatialite-dev

Now download two source packages, and unzip them in your Downloads directory. First the libgaiagraphics package:

cd ~/Downloads
wget http://www.gaia-gis.it/gaia-sins/libgaiagraphics-0.4b.tar.gz
tar xzf libgaiagraphics-0.4b.tar.gz
cd libgaiagraphics-0.4b
./configure
make
sudo make install-strip
sudo ldconfig
cd ..

Here we download the tarball from the gaia-gis.it website, unzip, then run the usual configure, make and make install commands to compile and put everything in place.
Next the GUI program itself:

wget http://www.gaia-gis.it/gaia-sins/spatialite_gui-1.5.0-stable.tar.gz
tar xzf spatialite_gui-1.5.0-stable.tar.gz
cd spatialite_gui-1.5.0-stable
./configure
make
sudo make install-strip


You might want to add an item to the applications menu, and put the icon into place. While still in the spatialite_gui-1.5.0 directory:


sudo cp gnome_resource/spatialite-gui.desktop /usr/share/applications/
sudo cp gnome_resource/spatialite-gui.png /usr/share/pixmaps/

And off you go collecting and querying spatial data.

Simple UPDATE example
However, when updating the target table with a subquery you must NOT have the target table in the subquery. The WHERE clause in the subquery must refer to the table being updated outside of the subquery. Let’s look at an example. We have a table of store locations, and we receive an excel spreadsheet file with sales data for each store. The excel table has one row for each store with, and a “StoreNum” that matches the StoreID in our stores spatial layer. See the table list in the image below. We want to get the sales values into our spatial layer. We first add a column for the new data:

ALTER TABLE Stores ADD COLUMN Sales double;

Now we can use the support in spatialite for virtual excel sheets, and add the sales data as a virtual table. In Spatialite GUI there’s a VirtualXL button – indicated in green in the image – to add a sheet from an excel file. Once the sales data is available as a virtual table, we want to update the Stores layer with the values from this excel table. So:

Notice – circled in red – that the subquery refers to the Stores table being updated, and this table does not appear in the FROM clause. The FROM clause contains only the sales_summary table, aliased AS s.

UPDATE with Aggregation
On to the next example, we are given an excel spreadsheet with detailed sales data. Rather than a summary for each store, this table contains all the transactions for all the stores. So each store appears many times in the table. We must now aggregate the sales data for each store, and update the sales column in our Stores spatial layer with a sum of sales for each store. This is accomplished easily with the SQL SUM() function. We first use the VirtualXL support to add a virtual table “sales_details”, then:

The UPDATE statement with SUM() to aggregate data for each store

Again, the Stores table, being updated, appears in the WHERE part of the subquery, but not in the FROM clause. The FROM portion has only the sales_details table. And in the SELECT subquery we use SUM(sales) to aggregate sales for each store. Since the WHERE clause matches s.StoreNum to Stores.StoreID, the SUM will be done for each store individually.

UPDATE with a spatial query
For the last example, we do an UPDATE on two spatial tables, using a subquery with a spatial function. Our store locations are spread around many counties (a polygon layer). We need to summarize all sales for each county. But wait a sec, the table being updated is in this case is the Counties, and the source data is from the Stores table. So the Stores table will now drop down to the subquery. We’ll again use the SUM() function, and our WHERE clause utilizes the ST_Within() function to find which stores are inside each county, and total up sales for those stores. In addition we deploy a SpatialIndex on each of the tables to speed up the query. See my previous post for an explanation of using spatial indices. Our spatial query now looks like:

An UPDATE with a spatial query, also using SpatialIndex

The WHERE clause of the spatial query – in green -contains both ST_Within function to find stores in each county, and the search for ROWID in the SpatialIndex limiting the search for stores which fall within each counties bounding box, thus saving lots of time. Again we see that the counties table, being updated, appears in the WHERE clause of the subquery, but not in the FROM part.

spatialite> SELECT * FROM stores LIMIT 10;
pk_uid sales Geometry
---------- ---------------- ----------
1 192617.423921806
2 75489.1390022072
3 163013.639152632
....

Let’s build the query. First, we are using both tables, so the FROM part of our SQL looks like:
FROM councils AS c, stores AS s
and since we want to aggregate total sales for each council, we require a GROUP BY clause:
GROUP BY c.pk_uid
What results do we want? The council name and total sales, so the SELECT part of our query will be:
SELECT c.Name AS Name, SUM(s.sales) AS "Total Sales"
Finally, we formulate the criteria, stores contained in each council, like so:
WHERE ST_Contains(c.Geometry, s.Geometry)

Putting it all together, our “naive” query (no spatial index yet) goes:
SELECT c.Name AS Name, SUM(s.sales) AS "Total Sales"
FROM councils AS c, stores AS s
WHERE ST_Contains(c.Geometry, s.Geometry)
GROUP BY c.pk_uid;

Now this query will run successfully on our two tables as described, but you better go for lunch when you start the query. It might finish on a fast computer by the time you get back… The query loops thru all 20,000 store locations for each of the 280 councils. So that’s over 5,000,000 table scans, and checks for each point in each polygon.

So how do we utilize spatial indexing to speed things up? The R*Tree indexing and its implementation in Spatialite are covered in the Spatialite Cookbook. It’s important to note that Spatialite does NOT make use of a spatial index unless you explicitly incorporate it into the query statement. (BTW, this is unlike PostGIS which automatically uses a spatial index, if it’s available.)
The options to implement a spatial index in a query structure were clearly explained in this thread from the spatialite maillist.

First we create a spatial index for both tables:
spatialite> SELECT CreateSpatialIndex('councils','Geometry');
1
spatialite> SELECT CreateSpatialIndex('stores','Geometry');
1

Now the additional WHERE clause, in it’s new incarnation (requires spatialite >=3.0) looks like this for our query:
s.ROWID IN
(SELECT ROWID FROM SpatialIndex WHERE f_table_name='stores' AND search_frame=c.Geometry)

“SpatialIndex” is a new virtual table, and using the search_frame component of that table we filter out the store locations in the bounding box (AKA Mbr=Minimum Bounding Rectangle). Searching for points in a rectangle is much faster than the full ST_Contains (or equivalent ST_Within) function. After this filtering, the ST_Contains() function runs on only a small subset of the 20,000 store locations.

So the spatial index enabled query will now look like:
SELECT c.Name AS Name, SUM(s.sales) AS "Total Sales"
FROM councils AS c, stores AS s
WHERE ST_Contains(c.Geometry, s.Geometry) AND
s.ROWID IN
(SELECT ROWID FROM SpatialIndex WHERE f_table_name='stores' AND search_frame=c.Geometry)
GROUP BY c.pk_uid;

On my machine, this query, aided by the spatial index, completed in less than 1.5 minutes. The first “naive” query took over 35 minutes. Pretty cool, no?