Web Services and Data Ingestion

Ernest Sin Chit To and Tim Whiteaker, CRWR

 

Table of Contents

Web Services and Data Ingestion. 1

Introduction. 1

Explanation of Data Sources. 3

Historical Data - Daymet 3

Forecasted Values - NAM... 4

Streamflow and Groundwater Level Data - USGS National Water Information System (NWIS) 4

Computer and Skill Requirements. 5

Installation. 5

Retrieving Data with Weather Downloader 7

Opening the Map. 7

Adding Weather Downloader to ArcMap. 9

Downloading Weather and Streamflow Data. 11

 

Introduction

 

National hydrologic observation repositories such as the USGS National Water Information System (NWIS), EPA STORET, and Daymet are now available online and can be accessed through a web browser or programmatically.  CUAHSI (the Consortium of Universities for the Advancement of Hydrologic Science) has developed a set of web services called WaterOneFlow, which provides a consistent means of accessing and delivering data from these repositories.  WaterOneFlow eliminates the need for a user to understand the complexities of a given data provider’s website or data retrieval mechanism.  By learning how to use WaterOneFlow services once, the user will have the skills necessary to retrieve data from all data providers for which a WaterOneFlow web service has been created.  And no matter which data provider the user is accessing with WaterOneFlow, the look and feel of data access and the resulting data files is the same.  This is made possible by WaterML, an XML schema that defines how WaterOneFlow web services accept inputs from users, as well as the format of the responses (the data you download) from WaterOneFlow. 

 

A full description of WaterML and the structure of WaterOneFlow web services is beyond the scope of this document.  For more information on WaterOneFlow, visit http://water.sdsc.edu/waterOneFlow/.

 

WaterOneFlow web services are designed to scour a given repository for a desired subset of data and return the data to the user in a format (i.e. xml) that can be ingested into a variety of applications, such as Microsoft Excel, Matlab and ArcGIS.  The figure below illustrates the concept behind WaterOneFlow web services.  The exercise included here demonstrates the use of a custom ArcMap tool called Weather Downloader that invokes these web services to download meteorological data from Daymet and streamflow data from NWIS.

 

 

 

 

Weather Downloader is a very useful tool for downloading time series data needed to describe the hydrology of a given geographical area.  It utilizes WaterOneFlow web services to access meteorological , streamflow and groundwater data stored in public data repositories.  Weather Downloader’s user interface allows the user to extract data by specifying the following inputs:

 

1)      a point feature class in ArcGIS that contains locations of interest;

2)      variables of interest (e.g. precipitation, temperature), and;

3)      time periods of interest. 

 

 

 

When executed, Weather Downloader cycles through each location in the point feature class, downloads the desired data through CUAHSI’s web services and writes them to the TimeSeries table of an Arc Hydro geodatabase specified by the user.

 

Explanation of Data Sources

Weather Downloader downloads daily historical meteorological data (i.e. anywhere between 1/1/1980 to 12/31/2003) from Daymet and forecasted data (in the next 3.5 days) in 3 hour intervals from NAM.  The section explains these two data sources.

Historical Data - Daymet

Data providing organization: Numerical Terradynamic Simulation Group (NTSG) at University of Montana

 

Website: http://www.Daymet.org/dataSelection.jsp

 

Daymet is a numerical model that provides daily surfaces of temperature, precipitation, humidity, and radiation over large regions of complex terrain. Daymet was developed to create fine resolution daily meteorological and climatological data necessary for plant growth model inputs. The input to Daymet includes digital elevation model and observations of maximum temperature, minimum temperature and precipitation from ground-based meteorological stations. The data are available as surfaces or as numerical estimates at single points for the contiguous United States at a daily time interval from 1980-2003.

 

Forecasted Values - NAM

Data providing organization: Unidata program at the University Cooperation for Atmospheric Research (UCAR).

 

Website: http://www.nco.ncep.noaa.gov/pmb/nwprod/analysis/

 

NAM stands for the North American Mesoscale model. NAM model which makes prediction of climate variables is ran four times daily (0:00 UTC, 6:00 UTC, 12:00 UTC and 18:00 UTC), and the predictions are made 84 hours into the future. The spatial extent of the model is limited to North America. The spatial resolution of the model grid is 12.19 km, and the grid dimensions are 614 x 428. The east and west longitudinal extents of the grid (in decimal degrees) are -49.30897 and -133.49621. The north and south latitude extents of the grid (in decimal degrees) are 57.35624 and 12.12367.

 

Weather Downloader downloads historical and forecasted weather data from Daymet and NAM into an Arc Hydro time series table. In its current version, Weather Downloader downloads historical data from Daymet and forecasted data from NAM.

 

 

Streamflow and Groundwater Level Data - USGS National Water Information System (NWIS)

Data providing organization: United States Geological Survey (USGS)

 

Website: http://waterdata.usgs.gov/nwis

 

The USGS NWIS is a comprehensive and distributed program that supports acquisition, processing and storage of water data. Most of the data stored in NWIS is available through NWIS website provided above (NWIS Web).  The data available via NWIS web mainly include information on quantity and quality of surface and ground water. NWIS web serves both historical and real time data. The real time data, however, is not available for all sites.

 

Data provided by NWISWeb are regularly updated from NWIS. Real-time data are generally updated upon receipt at local Water Science Centers. NWISWeb provides access to data by category, such as surface water, ground water, or water quality, and by geographic area. NWIS data are available for all 50 states, plus border and territorial sites, and include data from as early as 1899 (at few stations) to present. Of the over 1.5 million sites with NWIS data, the vast majority (about 800,000) are for groundwater wells, about 25,000 sites are for streamflow data, and about 9,800 of the sites provide real-time data. In addition there are many sites with atmospheric data such as precipitation, and there are nearly 70 million water-quality results from about 4 million water samples collected at hundreds of thousands of sites.

 

 

Computer and Skill Requirements

To complete this exercise, your computer must meet the following requirements:

 

 

This exercise assumes that you have some familiarity with the ArcGIS 9.1 software environment.

Installation

To install the Weather Downloader tool:

 

  1. Download the Setup file from here.
  2. After download, unzip the file on a local drive, and double click on Setup.Exe to run the setup file. 
  3. In the Weather Downloader Setup Wizard, click Next to start the installation process.

 

 

  1. Use the default installation folder for Weather Downloader or choose your preferred location, specify to install the program for “Everyone” or “Just Me”, and click Next.

 

 

  1. Click Next to get the Confirm Installation Window, and then click Next to start the installation process. You should see the progress bar as shown below:

 

 

 

  1. After the WeatherDownloader is successfully installed, click Close to finish the weather downloader installation process.

 

Besides installing the weather downloader, the installation process also adds two files (weather_downloader.mxd and weather_downloader.mdb ) in C:\Program Files\CUAHSI\Weather Downloader (or any other location specified during the set up) that you will use in this exercise.

Retrieving Data with Weather Downloader

In this exercise, you will use Weather Downloader to retrieve climate and streamflow time series data for the San Marcos basin in Texas.

Opening the Map

 

  1. Double click on weather_downloader.mxd to open the ArcMap document as shown below.  (The file is located in the directory where you installed Weather Downloader.)

 

 

 

This map shows the San Marcos watershed which is located between the cities of Austin and San Antonio in Texas.   The green polygons are the catchments and the orange points are their centroids.  The purple and white symbols are USGS stream gages.  The black triangles are USGS groundwater well stations.

 

  1. Open the attribute table for USGS_Gages_SanMarcos.  The table is shown below.

 

 

 

From the table, we see that this feature class contains five points representing USGS stream gages.  The values in the field HydroCode are the unique stream gage IDs assigned to each gage by the USGS.  These are public identifiers by which these features may be identified in any system.  The field HydroID stores the unique ID of these features in the geodatabase.  The HydroID links each stream gage to other features in the geodatabase, such as time series records which you will download in this exercise.

Adding Weather Downloader to ArcMap

  1. From ArcMap’s standard menu bar, click Tools, and then click Customize….
  2. Click on the Commands tab, and then click on Add from file….

 

 

  1. Navigate to the directory: “C:\Program Files\CUAHSI\Weather Downloader” (or wherever you installed Weather Downloader on the local drive).

 

 

 

  1. Select WeatherDownloader.tlb and click Open. If the installation is successful, the dialog box shown below should appear to confirm that the tool has been successfully added.

 

 

(If you see a message saying “No new objects” or something similar, check to see that you have administrator privileges on your computer.  You must be able to register DLLs in order to add Weather Downloader to ArcMap.)

 

  1. Click OK, and then drag Weather Downloader tool (see below) from the customize window to anywhere in the ArcMap toolbar to create a new button named Weather Downloader. Then close the customize window.

 

 

 

The weather downloader tool will appear as a button in the ArcMap toolbar as shown below:

 

 

Downloading Weather and Streamflow Data

  1. Click on the Weather Downloader button to get the form shown below.

 

 

 

Selecting a point layer

 

  1. In the top left combo box (located below “Please select the point feature class”), select the feature class that contains the point locations where you want weather data. For this exercise, select USGS_gages_SanMarcos.

 

Selecting identifier field in the point layer

 

  1. In the second combo box, select the identifier field that will link the feature class to the time series data. By default, the tool selects HydroID in the attribute table of the input layer as the identifier field.  If the HydroID field is unavailable, the combo box defaults to the first integer field in the attribute table.  For this exercise, make sure HydroID is chosen in this combo box.

 

Selecting data output location

 

  1. By default, the data output location is set to look for a geodatabase named
    “weather_downloader.mdb” that resides in the same directory as the ArcMap project.  If a different geodatabase is desired, the user can click on the Browse button and navigate to the location of the desired geodatabase.  For this exercise, leave the default of C:\Program Files\CUAHSI\Weather Downloader\weather_downloader.mdb, or wherever you installed Weather Downloader.

 

Selecting variables

 

  1. Select the variables that you want to download by clicking on the check boxes next to the variable descriptions.  In the frame for Historical Data, check the box for “4 - Precipitation (cm)”. 

 

  1. Now click on the Surface tab.

 

  1. In the frame for Streamflow Data, check the box for “9 – Daily Streamflow Data (cfs)”

 

  1. Please note that for Streamflow data, NWIS uses the USGS gage number (instead of latitude and longitude) to locate the point of interest.  Recall that these gage numbers are stored in the HydroCode field of our USGS_Gages_SanMarcos feature class.  In the combo box next to the 9-Daily Streamflow Data(cfs) checkbox, select HydroCode as the USGS gage number field.

 

 

 

Specifying period of interest

  1. You can specify the start dates and end dates for the Daymet and USGS stations by typing into the Start date and End date boxes. You can type in any date range.  However, the web services will only return data that are available.  For this exercise, leave the default start and end dates unchanged (start date = 01/01/1990 and end date = 12/31/2003) for Daymet and NWIS.

 

 

Note: For Unidata, the tool is set to download the most recent forecast results from the NCEP North American Mesoscale Model (12km).  Because forecasts are generated for a fixed period of time into the future (i.e. 84 hours), the web service is “hard-wired” to download all the data in this period. The North American Mesoscale model is run every six hours at 00:00, 06:00, 12:00, 18:00 Greenwich Mean Time with forecasts being made at 3 hour-intervals into the next 84 hours (i.e. 3.5 days).  We are not downloading forecasted values in this particular exercise.

 

Replace/Append to Timeseries table

 

  1. The last two radio buttons in the form let you choose between overwriting existing data in the TimeSeries table (Replace contents of TimeSeries table), or keeping the existing data in the TimeSeries table (Append to contents of TimeSeries table). For this exercise, select the Replace contents of TimeSeries table option.

 

 

The completed form should look like the following under the Atmospheric tab; and,

 

it should look like the following under the surface tab:

 

 

Running the tool

 

  1. Once all the inputs are provided, click OK to run the tool. The tool highlights the points for which the data are being downloaded as shown below. Since you are downloading two types of data, each point will be highlighted twice.

 

 

Besides highlighting the points, the progress of the tool is shown in the status bar in ArcMap at the bottom-left corner of the ArcMap window.  Please note that the tool may take a few minutes to complete the data download.

 

Once the download is complete, the tool will display the time taken in a dialog box as shown below. 

 

 

  1. Click OK to continue.

 

Inspecting the results

 

  1. The weather downloader tool downloads the data in Arc Hydro time series format.  In the ArcMap table of contents (left window), select the source tab and then select the TimeSeries table as shown below.

 

 

  1. Right click on TimeSeries table, and then click Open to view the contents of the table as shown below:

 

 

The time values are stored in the TSDateTime field, and the corresponding data are stored in the TSValue field. The TSTypeID field stores variable codes which can be looked up in the TSType table for more information such as units, time interval, etc.

 

You can export the attribute table into a .dbf file and do some fancy plots with it.  For instance the following are the hyetograph and hydrograph of USGS gage 08170500 (San Marcos River at San Marcos) that are plotted from the data you have just downloaded.  Notice how well the major storm events in the graphs line-up with each other despite the different sources of data.  Pretty amazing, isn’t it?

 

 

This concludes the exercise.

 

Primary Contacts:

Ernest Sin Chit To

Center for Research in Water Resources

University of Texas at Austin

e-mail: eto@mail.utexas.edu

Tim Whiteaker

Center for Research in Water Resources

University of Texas at Austin

e-mail: twhit@mail.utexas.edu

These materials may be used for study, research, and education, but please credit the authors and the Center for Research in Water Resources, The University of Texas at Austin. All commercial rights reserved. Copyright 2007 Center for Research in Water Resources.