Sunday, March 9, 2014

More adventures from our remote sensing field trip

Yesterday we were fortunate enough to tour the Decagon Devices facilities where the radiometers I’m basing a lesson plan around are made. All I can say is what a facility! From research and development to a machine shop that would make any shop geek drool, Decagon Devices has my recognition as a GREAT company. They care about their employees and it shows in their products.
Gracious enough to give us his time on this day was Plant Canopy Manager Dr. Steven Garrity. After a full tour of the facility, we took some time to talk more about what we’re doing at MOSS and how we use their products. We also talked about doing a video collaboration with Decagon later this year to show how their devices are being used by middle school and high school students, and that they aren’t just for grad students and other professionals. Steve seemed very excited with what we’re doing and even offered to donate some radiometers to MOSS to help us further our research and lessons. Thank you Steve!

After our meeting with Steve, he invited us to the company lunch that goes on every Wednesday, put on by their onsite chef. Pot roast, pork n beans, corn, salad and tiramisu…. Yum! After lunch we hit the road and cruised over to Seattle. Did it rain on the drive you ask? Of course it did.Well that’s all for now. I’d like to say thanks again to Steve. It was an absolute pleasure and I lookforward to talking to you soon about the video this summer.

Cheers!

Dirk Jr.
Hello everyone out there in blog land!

My name is Dirk Anderson Jr. and as my companions Ross and Janeen have already mentioned, I’m a graduate student with the University of Idaho, doing most of my schooling at the McCall Outdoor Science School (MOSS). As part of our schooling we were assigned various assistantships and I was fortunate enough to be selected to work with some remote sensing (RS) technologies and design a curriculum for middle school and/or high school students. The “toys” I get to play with are called
radiometers, which are a spectral reflectance sensor (SRS) designed and produced by Decagon Devices in Pullman, WA. These radiometers look at several different bands of light with two different types of sensors indices; Normalized Difference Vegetation Index (NDVI) and Photochemical Reflectance Index (PRI). My goal is to produce a lesson plan using these SRSs that will not only introduce the concept of light reflectivity, but also produce a more meaningful and context way to teach students about the visible light spectrum using cutting edge technologies.

As Ross and Janeen have already mentioned we are currently touring the northwest making stops in Lapwaii and Moscow, ID and Pullman and Seattle, WA to visit a range of professionals in the RS,
education, and consulting fields. After our stop in Lapwaii on Monday, we headed up to Moscow to visit with several U of I grad students in the education field. During our talk with Becky, Ryan and Audrey over a cup of coffee at the One World coffee shop, we covered some successes and difficulties with teaching sciences at the K-12 level. They got me really excited when they told me about the work that they have been doing. Previously called the GK12 program, they have been working on pairing graduate students with teachers to help build a stronger learning and teaching environment. If I’m lucky enough, I hope to find myself in this program in the future.

The next day we were able to meet with our friend and former orientation instructor, Troy Magney, who’s a Ph. D student at the U of I in RS. Troy also was a part of the MOSS program a few years back, so his insight was extremely helpful. We spent most of the day with Troy as we set up and ran a scan with a Terrestrial Laser Scanner, talked about his research, and toured a farm where several different SRSs were set up. Troy has already been a great help on my project and idea development and I look forward to continue working with him.

Well that’s all for now. Tomorrow we’re headed to Decagon Devices to see where my radiometers come from.

Cheers!

Dirk Jr.

Thursday, March 6, 2014

3/6/2014--Seattle, Washington

Hello to all reading this blog!
My name is Ross Parsons and I work with Janeen and Dirk at the University of Idaho McCall Outdoor Science School.  As part of my research, I have been working with an Autonomously Operating Terrestrial Laser Scanner (ATLS).  This remote sensing device can be used to quantify morphological changes in our natural world.  Examples of this include, but are certainly not limited to, floodplain changes after dam removal (see earlier post) and measuring snowpack dynamics within a complex forest ecosystem.  It is my goal to develop a lesson plan to introduce young learners (5th and 6th graders, for example) to the world of remote sensing.  Furthermore, I am hoping to convince students that remote sensing is a fun and exciting way to observe our natural world.
Through this field trip, I am excited to gather what people have to say about remote sensing, education, and communicating science to a broader audience.

Our rainy and adventurous day started with meeting Jeff Johnson and Mark Indrebo of Watershed Science and Engineering.  Jeff and Mark helped us to better understand the complexities of communicating to a broad audience about their work.  With respect to flood mitigation, they suggested that listening to all parties involved is the best way to accomplish the task at hand.  The key to success, they argued, is compassion.  This philosophy can be transferred to all walks of science communication.  In their case, "multi-benefit" planning has proved successful. 
Our next meeting was with Steve Warren and Karl Lapo of the University of Washington's Department of Atmospheric Sciences.  Steve is a longtime Professor of the University, with interests in Antarctic climate and black carbon in Arctic snow, just to name a few.  Karl is a graduate student who is studying similar processes.  Along with sharing their impressive research, Steve and Karl stressed the importance of mathematics among young learners.  When asked what skills they would like to see more college freshman have, they simultaneously answered math.
Our final meeting of the day brought us to South Seattle's K-5 STEM at Boren School.  The school is Seattle Public School affiliated, and focuses primarily on the STEM subjects (science, technology, engineering, and mathematics).  We met with Julie Schmick, the school's technology teacher.  Julie told us that in order to get her students interested in STEM, she always asks a "meaty" question--one that they can relate to their community and family.  Julie was very receptive to incorporating remote sensing into her teachings.  As she put it, anything that allows students to have fun will be used in her classroom.

Overall, today's meetings were a great reminder that an interdisciplinary, holistic approach must be taken when communicating and teaching science.  By doing so, we can create innovative young learners who will continue to challenge the way we look at the world.
In Remote Sensing We Trust,

Ross Parsons

Wednesday, March 5, 2014

Hi,
 My name is Janeen. I have been working on an AmericaView project at the University on Idaho which incorporates Landsat data into curriculum designed for kids ages 10-18. As part of my creation of this curriculum I am traveling throughout the Pacific Northwest this week with a small group of my peers. We are all working on ways to bring remote sensing topics to younger audiences. During our travels we have been meeting with specialists in a variety of fields in an effort to better understand the full scope of our topics. You can follow our adventures over the next few days.

As a graduate student at the McCall Outdoor Science School, I regularly teach week long courses for this age group. I have been working on a lesson plan that will use Landsat images to reference a local state park. The kids will then create habitat maps of the area. After that the kids get outside and explore the area, adding information to their maps about vegetation and signs of animal life. Finally, the kids will use their increased understanding to defend decisions they make in how they would manage the park.

My group started our travels on Monday. Our first stop was to meet with Laurie Ames, who works in the GIS department for the Nez Perce Tribe, in Lapwei, ID. From Laurie, I learned a great deal about how agencies use Landsat and other remotely sensed data. Laurie was particularly interested in our plans to help young people to have a sense of how all the pieces of understanding we get from our data fit together to make a picture of how the whole world works.

Our next meeting was with Mark Corrao, a fellow graduate student at the U of Idaho. Mark is working on advanced degrees in natural resources and law, and also works full time at an environmental consulting firm. Mark helped us to better understand the process of turning information about a piece of land into a good plan that benefits the land owner, the ecosystem, the community. Mark hoped our lessons would help kids to increase their critical thinking skills and appreciation for the interdisciplinary nature of decision making about the land.

Next up, my compatriot Ross Parsons will be explaining a bit about his research.

Tuesday, February 25, 2014

Geospatial Tutorial Video Playlist

AmericaView members are some of the most productive people when it comes to generating high-quality geospatial educational videos.  I have complied a playlist of some of the best videos on You Tube.  Most of these come from VirginiaView, but VermontView is not too far behind with a dozen.  You could literally run an introductory remote sensing course with what has been compiled.


Monday, February 10, 2014

AmericaView Seeks Executive Director

AmericaView is seeking an exceptional individual to serve as its Executive Director

AmericaView (AV) is a nationwide non-profit partnership of remote sensing scientists who support the use of remotely sensed satellite data through applied remote sensing research, K-12 and higher STEM education, workforce development, and technology transfer.  Funded by a newly awarded major competitive grant from the U.S. Geological Survey, AV is composed of 39 university-led, state-based consortia working together to sustain a network of state and local remote sensing scientists, educators, analysts, technicians, and stakeholders. AmericaView's networks, facilities, and capabilities are highly leveraged and used for sharing and applying remotely sensed satellite data in a wide range of civilian applications, from formal and informal education, to ecosystem analysis and natural resources management, to disaster response. Our primary goal is to support the many beneficial uses of remote sensing in service to society.

The position calls for a dynamic individual who is prominent in the remote sensing sector.  The Executive Director serves as the voice and face of AmericaView, and will have the opportunity to provide strategic vision and leadership for the organization for years to come.  He/she will build strategic partnerships with sister organizations, and initiate and sustain effective communication with AV’s funders and stakeholders.  The Executive Director represents AmericaView at regional and national meetings, working closely with individuals in the public, private, academic and non-profit sectors.  Management duties include oversight of grant and non-profit fiscal requirements.  The Executive Director reports to the AmericaView Board of Directors and performs duties and discharges those responsibilities in conformance with the Articles of Incorporation and Bylaws of AmericaView and the policies of the Board of Directors.  A highly engaged board and a small team of dedicated and experienced staff will assist the Executive Director in the accomplishment of his/her responsibilities. 
Prospective candidates should have a proven track record of remote sensing research, education, and outreach.  Prior experience in leadership and management positions is highly desirable.  Prospective candidates should have strong communication skills, be capable of exercising sound judgment, and have the desire to engage in fund raising and/or grant writing activities.

The Executive Director position is a 25-35% full-time equivalent (FTE) position.  Salary will be negotiated at the time of hire.  The Executive Director is free to work at the location of his/her choosing.  Annual travel is less than one month per year.


Interested candidates should submit a CV and cover letter to the AmericaView Executive Director search committee chair Jarlath O’Neil-Dunne at joneildu@uvm.edu.  AmericaView is an equal opportunity employer.

Wednesday, August 21, 2013

PostGIS 2.x: Getting Raster Data out of the Database

PostGIS 2.x (latest release, 2.1) enables users to do fairly sophisticated raster processing directly in a database. For many applications, these data can stay in the database; it's the insight into spatial phenomena that comes out. Sometimes, however, you need to get file data (e.g. a GeoTIFF) out of PostGIS. It isn't immediately obvious how to do this efficiently, despite the number of helpful functions that serialize a raster field to Well-Known Binary (WKB) or other "flat" formats.

Background

In particular, I recently needed to create a web service that delivers PostGIS raster outputs as file data. The queries that we needed to support were well suited for PostGIS and sometimes one query would consume another (one or more) as subquer(ies). These and other considerations led me to decide to implement the service layer in Python using either GeoDjango or GeoAlchemy. More on that later. Suffice to say, a robust and stable solution for exporting and attaching file data from PostGIS to an HTTP response was needed. I found at least six (6) different ways of doing this; there may be more:
  • Export an ASCII grid ("AAIGrid")
  • Connect to the database using a desktop client (e.g. QGIS) [1]
  • Use a procedural language (like PLPGSQL or PLPython) [2]
  • Use the COPY declaration to get a hex dump out, then convert to binary
  • Fill a 2D NumPy array with a byte array and serialize it to a binary file using GDAL or psycopg2 [3, 4]
  • Use a raster output function to get a byte array, which can be written to a binary field
It's nice to have options. But what's the most appropriate? If that's a difficult question to answer, what's the easiest option? I'll explore some of them in detail. For better code sample display, refer to this gist.

Export an ASCII Grid

This works great! Because an ASCII grid file (or "ESRI Grid" file, with the *.asc or *.grd extension, typically) is just plain text, you can directly export it from the database. The GDAL driver name is "AAIGrid" which should be the second argument to ST_AsGDALRaster(). Be sure to remove the column header from your export (see image below). However, what you get is a file that has no projection information that you may need to convert to another format. This can present problems for your workflow, especially if you're trying to automate the production of raster files, say, through a web API.

Want an ASCII grid (or "ESRI Grid")? No problem! Just don't export the column names.

Connect Using the QGIS Desktop Client

There is a plug-in for QGIS that promises to allow you to load raster data from PostGIS directly into a QGIS workspace. I used the Plugins Manager ("Plugins" > "Fetch Python Plugins...") in QGIS to get this plug-in package. The first time I selected the "Load PostGIS Raster to QGIS" plug-in and tried to install it, I found that I couldn't write to the plug-ins directory (this with a relatively fresh installation of QGIS). After creating and setting myself as the owner of the python/plugins directory, I was able to install the plug-in without any further trouble. Connecting to the database and viewing the available relations was also no trouble at all. One minor irritation is that you need to enter your password every time the plug-in interfaces with the database, which can be very often, at every time the list of available relations needs to be updated.
You'll be doing this a lot.
There are a few options available to you in displaying raster data from the database: "Read table's vector representation," "Read one table as a raster," "Read one row as a raster," or "Read the dataset as a raster." It's not clear what the second and last choices are, but "Reading the table as a raster" did not work for me where my table has one raster field and a couple of non-raster, non-geometry/geography fields; QGIS hung for a few seconds then said it "Could not load PG..." Reading one row worked, however, you have to select the row by its primary key (or row number in a random selection, not sure which it is returning). This may not be the easiest way for you to find a particular raster image in a table.

Using the COPY Declaration in SQL

My colleague suggested this method, demonstrated in Python, which requires the pygresql module to be installed; easy enough with pip:
pip install psycopg2 pygresql
The basic idea is to use the COPY declaration in SQL to export the raster to a hexadecimal file, then to convert that file to a binary file using xxd:
import os, stat, pg
# See: http://www.pygresql.org/install.html
# pip install psycopg2, pygresql

# Designate path to output file
outfile = '/home/myself/temp.tiff'

# Name of PostgreSQL table to export
pg_table = 'geowepp_soil'

# PostgreSQL connection parameters
pg_server = 'my_server'
pg_database = 'my_database'
pg_user = 'myself'

# Desginate a file to receive the hex data; make sure it exists with the right permissions
pg_hex = '/home/myself/temp.hex'
os.mknod(pg_hex, stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IWGRP)

conn = pg.connect(pg_database, pg_server, 5432, None, None, pg_user)
sql = "COPY (SELECT encode(ST_AsTIFF(ST_Union(" + pg_table + ".rast)), 'hex') FROM " + pg_table + ") TO '" + pg_hex + "'"

# You can check it with: print sql
conn.query(sql)

cmd = 'xxd -p -r ' + pg_hex + ' > ' + outfile
os.system(cmd)
This needs to be done on the file system of the database server, which is where PostgreSQL will write.

Using an Output Function and Serializing from a Byte Array

Despite the seeming complexity of this option (then again, compare it to the above), I think it is the most flexible approach. I'll provide two examples here, with code: using GeoDjango to execute a raw query and using GeoAlchemy2's object-relational model to execute the query. Finally, I'll show an example of writing the output to a file or to a Django HttpResponse() instance.

Using GeoDjango


First, some setup. We'll define a RasterQuery class to help with handling the details. While a new class isn't exactly an idiomatic example, I'm hoping it will succinctly illustrate the considerations involved in using performing raw SQL queries with Django
class RasterQuery:
    '''
    Assumes some global FORMATS dictionary describes the valid file formats, their file extensions and MIME type strings.
    '''
    def __init__(self, qs, params=None, file_format='geotiff'):
        assert file_format in FORMATS.keys(), 'Not a valid file format'

        self.cursor = connection.cursor()
        self.params = params
        self.query_string = qs
        self.file_format = file_format
        self.file_extension = FORMATS[file_format]['file_extension']

    def execute(self, params=None):
        '''Execute the stored query string with the given parameters'''
        self.params = params

        if self.params is not None:
            self.cursor.execute(self.query_string, params)

        else:
            self.cursor.execute(self.query_string)

    def fetch_all(self):
        '''Return all results in a List; a List of buffers is returned'''
        return [
            row[0] for row in self.cursor.fetchall()
        ]

    def write_all(self, path, name=None):
        '''For each raster in the query, writes it to a file on the given path'''
        name = name or 'raster_query'

        i = 0
        results = self.fetch_all()
        for each in results:
            name = name + str(i + 1) + self.file_extension
            with open(os.path.join(path, name), 'wb') as stream:
                stream.write(results[i])

            i += 1
With the RasterQuery class available to us, we can more cleanly execute our raw SQL queries and serialize the response to a file attachment in a Django view:
def clip_one_raster_by_another(request):

    # Our raw SQL query, with parameter strings
    query_string = '''
    SELECT ST_AsGDALRaster(ST_Clip(landcover.rast,
        ST_Buffer(ST_Envelope(burnedarea.rast), %s)), %s) AS rast 
      FROM landcover, burnedarea 
     WHERE ST_Intersects(landcover.rast, burnedarea.rast)
       AND burnedarea.rid = %s'''

    # Create a RasterQuery instance; apply the parameters
    query = RasterQuery(query_string)
    query.execute([1000, 'GTiff', 2])

    filename = 'blah.tiff'

    # Outputs:
    # [(<read-only buffer for 0x2613470, size 110173, offset 0 at 0x26a05b0>),
    #  (<read-only buffer for 0x26134b0, size 142794, offset 0 at 0x26a01f0>)]

    # Return only the first item
    response = HttpResponse(query.fetch_all()[0], content_type=FORMATS[_format]['mime'])
    response['Content-Disposition'] = 'attachment; filename="%s"' % filename
    return response
Seem simple enough? To write to a file instead, see the write_all() method of the RasterQuery class. The query.fetch_all()[0] at the end is contrived. I'll show a better way of getting to a nested buffer in the next example.

Using GeoAlchemy2


GeoAlchemy2's object-relational model (ORM) allows tables to be represented as classes, just like in Django.
class LandCover(DeclarativeBase):
    __tablename__ = 'landcover'
    rid = Column(Integer, primary_key=True)
    rast = Column(ga2.types.Raster)
    filename = Column(String(255))

class BurnedArea(DeclarativeBase):
    __tablename__ = 'burnedarea'
    rid = Column(Integer, primary_key=True)
    rast = Column(ga2.types.Raster)
    filename = Column(String(255))
    burndate = Column(Date)
    burnname = Column(String(255))
Assuming that SESSION and ENGINE global variables are available, the gist of this approach can be seen in this example:
def clip_fccs_by_mtbs_id2(request):

    query = SESSION.query(LandCover.rast\
        .ST_AsGDALRaster(LandCover.rast\
        .ST_Clip(LandCover.rast, BurnedArea.rast\
        .ST_Envelope()\
        .ST_Buffer(1000)), 'GTiff').label('rast'))\
        .filter(LandCover.rast.ST_Intersects(BurnedArea.rast), BurnedArea.rid==2)

    filename = 'blah.tiff'

    # Outputs:
    # [(<read-only buffer for 0x2613470, size 110173, offset 0 at 0x26a05b0>),
    #  (<read-only buffer for 0x26134b0, size 142794, offset 0 at 0x26a01f0>)]

    result = query.all()
    while type(result) != buffer:
        result = result[0] # Unwrap until a buffer is found

    # Consequently, it returns only the first item
    response = HttpResponse(result, content_type=FORMATS[_format]['mime'])
    response['Content-Disposition'] = 'attachment; filename="%s"' % filename 
Here we see a better way of getting at a nested buffer. If we wanted all of the rasters that were returned (all of the buffers), we could call ST_Union on our final raster selection before passing it to ST_AsGDALRaster.

After considering all my (apparent) options, I found this last technique, using the PostGIS raster output function(s) and writing the byte array to a file-attachment in an HTTP response, to be best suited for my application. I'd be interested in hearing about other techniques not described here.