Web Services and Data Ingestion

By Ernest Sin Chit To and Tim Whiteaker, CRWR

 

Table of Contents

 

Introduction

 

The internet explosion has given rise to a plethora of national hydrologic observation repositories such as USGS National Water Information System (NWIS), AmeriFlux, Daymet, etc.   Researchers are constantly developing tools to provide programmatic access to these repositories.  One group of tools is HIS (Hydrologic Information Systems) web services, which is developed by CUAHSI (the Consortium of Universities for the Advancement of Hydrologic Science). 

 

HIS web services are designed to scour a given repository for a desired subset of data and return the data to the user in a format (i.e. xml) that can be ingested into a variety of applications, such as Microsoft Excel, Matlab and ArcGIS.  The figure below illustrates the concept behind HIS web services.  This tutorial demonstrates the use of a custom ArcMap tool called Weather Downloader that invokes these webservices to download meteorological data from Daymet and NAM.

 

Concept of HIS (Hydrologic Information Systems (HIS) web services (from HIS workbook, July 2006, prepared by CUAHSI HIS Project Team).

 

Explanation of Data Sources

Weather Downloader downloads daily historical meteorological data (i.e. anywhere between 1/1/1980 to 12/31/2003) from Daymet and forecasted data (in the next 3.5 days)  in 3 hour intervals from NAM.  The section explains these two data sources.

Daymet

Data providing organization: Numerical Terradynamic Simulation Group (NTSG) at University of Montana

 

Website: http://www.Daymet.org/dataSelection.jsp

 

Daymet is a numerical model that provides daily surfaces of temperature, precipitation, humidity, and radiation over large regions of complex terrain. Daymet was developed to create fine resolution daily meteorological and climatological data necessary for plant growth model inputs. The input to Daymet includes digital elevation model and observations of maximum temperature, minimum temperature and precipitation from ground-based meteorological stations. The data are available as surfaces or as numerical estimates at single points for the contiguous United States at a daily time interval from 1980-1997.

 

NAM

Data providing organization: Unidata program at the University Cooperation for Atmospheric Research (UCAR).

 

Website: http://www.nco.ncep.noaa.gov/pmb/nwprod/analysis/

 

NAM stands for the North American Mesoscale model. NAM model which makes prediction of climate variables is ran four times daily (0:00 UTC, 6:00 UTC, 12:00 UTC and 18:00 UTC), and the predictions are made 84 hours into the future. The spatial extent of the model is limited to North America. The spatial resolution of the model grid is 12.19 km, and the grid dimensions are 614 x 428. The east and west longitudinal extents of the grid (in decimal degrees) are -49.30897 and -133.49621. The north and south latitude extents of the grid (in decimal degrees) are 57.35624 and 12.12367.

 

Weather Downloader downloads historical and forecasted weather data from Daymet and NAM into an Arc Hydro time series table. In its current version, Weather Downloader downloads historical data from Daymet and forecasted data from NAM.

 

Computer and Skill Requirements

To complete this exercise, your computer must meet the following requirements:

 

1.                  Windows 2000 or above

2.                  ArcGIS 9.1 or above

3.                  Internet connection

 

This exercise assumes that you have some familiarity with ArcGIS 9.1 software environment.

Installation

To install the Weather Downloader tool, download the Setup file from the following link:

ftp://ftp.crwr.utexas.edu/pub/outgoing/CUAHSI/WeatherDownloaderSetup.zip. 

 

After download, unzip the file on a local drive (contents shown below), and double-click on Setup.Exe to run the setup file. 

 

 

In the Weather Downloader Setup Wizard, click Next to start the installation process.

 

Use the default installation folder for Weather Downloader or choose your preferred location, specify to install the program for “Everyone” or “Just Me”, and click Next.

 

 

Click Next to get the Confirm Installation Window, and then click Next to start the installation process. You should see the progress bar as shown below:

 

 

 

 

After the WeatherDownloader is successfully installed, click Close to finish the weather downloader installation process.

 

Besides installing the weather downloader, the installation process also adds two files (weather_downloader.mxd and weather_downloader.mdb ) in C:\Program Files\CUAHSI\Weather Downloader (or any other location specified during the set up) that you will use in this exercise.

 

Double-click on weather_downloader.mxd to open the ArcMap document as shown below. 

 

 

This map shows the Sandies and Elm watershed in Texas.  The green polygons are the catchments and the orange points are their outlet points.  In this exercise, we will download weather data for 6 randomly selected outlet points (circled in purple). 

 

Adding the tool into ArcMap

To add the Weather Downloader tool to ArcMap, click on ToolsàCustomize to get the window as shown below. 

 

 

Click on the Commands tab, and then click on Add from file as shown below:

 

Then navigate to the directory: “C:\Program Files\CUAHSI\Weather Downloader” (or wherever you installed Weather Downloader on the local drive).

 

 

Select WeatherDownloader.tlb and click Open. If the installation is successful, the dialog box shown below should appear to confirm that the tool has been successfully added.

-          

 

Click OK, and then drag Weather Downloader tool (see below) from the customize window to anywhere in the ArcMap toolbar to create a new button named Weather Downloader. Then close the customize window.

 

 

The weather downloader tool will appear as a button in the ArcMap toolbar as shown below:

 

 

Downloading weather data

To download data, click on the Weather Downloader button to get the form shown below:

 

 

 

Selecting a point layer

In the top combo box (located below Please select the point feature class), select the feature class that contains the point locations where you want weather data. For this exercise, keep the default DemoPoints. The DemoPoints feature class consists of 6 data points (marked as purple rings) situated in the watersheds shown in the ArcMap document.

 

Selecting identifier field in the point layer

In the second combo box, select the identifier field that will link the feature class to the time series data. Select HydroID as the identifier field.

 

 

 

 

 

Selecting data output location

In the text box for the path and filename of the geodatabase that contains the timeseries table, browse to weather_downloader.mdb (location where the WeatherDownloaderMap.zip file is unzipped). 

 

Selecting variables

Select the variables that you want to download by clicking on the check boxes next to the variable descriptions. For historical data (Daymet), the time series is only available from January 1980 to December 2003. Make sure you provide the Start and End dates between 1980 and 2003 for downloading historical data. If you choose to download forecasted values (UNIDATA), the tool is set to download the most recent forecast results from the NCEP North American Model (12km).  The model is run every six hours at 00:00, 06:00, 12:00, 18:00 Greenwich Mean Time and produces forecasted precipitation at 3 hour-intervals for the next 84 hours (i.e. 3.5 days).

 

For this exercise, check 4 - Precipitation (cm) for historical data and 8 – Total Precipitation (kg m-2 or mm rainfall) for forecasted values. Leave the default start and end date unchanged (start date = 01/01/2000 and end date = 12/31/2000). 

 

Replace/Append to Timeseries table

The last two radio buttons in the form allows you to overwrite the existing data in the TimeSeries table (Replace contents of TimeSeries table), or to keep the existing data in the TimeSeries table (Append to contents of TimeSeries table). For this exercise, select the Replace contents of TimeSeries table option.

 

Running the tool

Once all the inputs are provided, click OK to run the tool. The tool highlights the points for which the data are being downloaded as shown below. Since you are downloading two types of data, each point will be highlighted twice.

 

Besides highlighting the points, the progress of the tool is shown in the status bar in ArcMap at the bottom-left corner of the screen. Once the download is complete, the tool will be display the time taken in a dialog box as shown below. 

 

 

 

Click OK to continue.

 

Inspecting the results

 

The weather downloader tool downloads the data in Arc Hydro time series format. In the ArcMap table to contents (left window), select the source tab and then select the TimeSeries table as shown below.

 

 

Right-click on TimeSeries table, and then click Open to view the contents of the table as shown below:

 

 

The time values are stored in the TSDateTime field, and the corresponding data are stored in the TSValue field. The TSTypeID field stores variable codes which can be looked up in the TSType table for more information such as units, time interval, etc.

 

This concludes the exercise of downloading meteorological data from Daymet and NAM by using HIS web services in ArcGIS.

 

 

Primary Contact:

Ernest Sin Chit To

Center for Research in Water Resources

University of Texas at Austin

e-mail: eto@mail.utexas.edu
 

These materials may be used for study, research, and education, but please credit the authors and the Center for Research in Water Resources, The University of Texas at Austin. All commercial rights reserved. Copyright 2006 Center for Research in Water Resources.