Plotting Time Series of Weather Observations in Emacs

<2017-03-01 Wed>

Sorry, your browser does not support SVG.

Figure 1: Time Series of Weather Observations

Introduction

In this post, we will be using Emacs and org-mode to construct a time series of weather observations. If you want the short version of what we will be doing, change the station-id below and run org-babel-execute-buffer (C-c C-v C-b). For the complete version read on. We will be using the following technologies:

The org source for this file is located at the source link at the bottom of the page. Indeed, to follow along carefully you will have to view that source link as the HTML rendering has removed some of the org source from this web page.

Find a Weather Station

Our first job is to find a weather station near our location. We will employ the SynopticLabs Mesonet API to identify a station ID near my current location. I live in Boulder, Colorado at roughly these coordinates: 39.99, -105.22. The Mesonet API has a service to find weather stations nearest a given latitude and longitude point. You have to supply a US state (easy to forget) and a radius arguments composed of a latitude, longitude center and a radius distance in miles. In addition, there are a couple of other ancillary arguments including a demo token. Consult the API documentation to ensure you are working within the terms of use. The following URL will provide us a station near by with the help of ob-http:

ob-http Request and JSON Response

GET http://api.mesowest.net/v2/stations/metadata?state=co&status=active&radius=39.99,-105.22,1&token=demotoken
{
  "SUMMARY": {
    "METADATA_RESPONSE_TIME": "0.123023986816 ms",
    "RESPONSE_MESSAGE": "OK",
    "RESPONSE_CODE": 1,
    "NUMBER_OF_OBJECTS": 1
  },
  "STATION": [
    {
      "ID": "41017",
      "TIMEZONE": "America\/Denver",
      "LATITUDE": "40.00133",
      "STATE": "CO",
      "LONGITUDE": "-105.23033",
      "STID": "E3608",
      "DISTANCE": 0.96,
      "NAME": "EW3608 Boulder",
      "ELEVATION": "5430",
      "PERIOD_OF_RECORD": {
        "end": "2017-03-07T23:53:00Z",
        "start": "2013-09-25T00:00:00Z"
      },
      "MNET_ID": "65",
      "STATUS": "ACTIVE"
    }
  ]
}

According to this JSON response, there is the "E3608" weather station not far away. We will work with the data from that instrument.

Timeseries API

Next, we will be using the timeseries API to retrieve the station data. For example, the following URL retrieves the air temperature from the KDEN station for the last 30 minutes.

GET http://api.mesowest.net/v2/stations/timeseries?&stid=kden&recent=30&token=demotoken&vars=air_temp&output=csv
# STATION: KDEN
# STATION NAME: Denver, Denver International Airport
# LATITUDE: 39.84658
# LONGITUDE: -104.65622
# ELEVATION [ft]: 5404
# STATE: CO
Station_ID,Date_Time,air_temp_set_1
,,Celsius
KDEN,2017-03-08T01:00:00Z,8.0
KDEN,2017-03-08T01:05:00Z,8.0
KDEN,2017-03-08T01:10:00Z,8.0
KDEN,2017-03-08T01:15:00Z,8.0
KDEN,2017-03-08T01:20:00Z,8.0

There are more examples of API use as well as the API reference on SynopticLabs website.

For the particular example we are shooting for, we will supply the following URL parameters:

  • stid, the station ID that we obtained earlier
  • start, the start time in UTC (e.g., 201702171644)
  • end, the end time in UTC (e.g., 201702241644)
  • units, the desired units (i.e., metrics or english)
  • vars, the sensor variables
  • obtimezone, the timezone of the response data (i.e., UTC or local)
  • output, the output format (e.g., csv)
  • token, the request token, in this case guest

Set up Parameterization With noweb

We will once again be make use of ob-http to retrieve the data, but, this time, we will be employing the Emacs noweb facility to parameterize the ob-http code block. (By the way, "noweb", has always struck me as a terrible name as it seems to imply something about the web browsing, the world wide web, or the Internet, but, in fact, refers to a literate programming style.) Let's start defining our noweb parameters with named org-mode and Emacs Lisp code blocks. We will use the names subsequently via noweb.

Start Time

The start time range will be 24 hours ago. Also set the time zone as UTC since that is what the SynopticLabs API requires.

(setenv "TZ" "UTC0")
(format-time-string "%Y%m%d%H%M" (time-add
                                  (current-time)
                                  (seconds-to-time (- (* 1 24 60 60)))))
201703070128

End Time

The end time will be now.

(format-time-string "%Y%m%d%H%M")
201703080128

Station ID

The station ID we found earlier.

;; "KBJC"
"E3608"
;; "E7829"
E3608

Retrieve the Weather Observations

Retrieve Data

At this point, we can finally retrieve our data with the help our noweb parameters. For example, <<starttime()>> will invoke the starttime code block defined above and insert the result into the http address below. The same is true for station-id and endtime. See the source link at the bottom to see the noweb parameterization.

GET http://api.mesowest.net/v2/stations/timeseries?token=demotoken&stid=E3608&start=201703070135&end=201703080135&obtimezone=local&units=metric&vars=air_temp,dew_point_temperature,wind_speed,wind_direction,sea_level_pressure&output=csv

Post-request cleanup with Emacs Lisp

At this point, we have to write some Emacs Lisp to clean up the response data and make it more org-mode friendly. We will need four helper functions:

  • take-nth is a function to sample data at certain intervals from a list.
  • gnuplot does not like empty org-table cells so the fix-missing function replaces empty cells with the "missing" string.
  • The find-header function locates where the header and actual data are located in the CSV http response.
  • The clean-header function replaces cryptic columns names with ones that are easier to understand.

Armed with these functions, we can clean up our data. Also a note about sampling: large org-tables tend to bog down Emacs so I'm limiting the table size to 30 samples via a Babel header argument, and the take-nth helper function.

(defun take-nth (n list )
  "Returns a list of every nth item in the list."
  (if (<= n 1) list
    (when list
      (cons (car list)
            (take-nth n (nthcdr n list))))))

(defun fix-missing (s)
  "Replace missing CSV values with 'missing' to avoid empty cells in org-tables."
  (if (string-match-p ",\s*," s)
      (fix-missing (replace-regexp-in-string ",\s*," ",missing," s))
    s))

(defun find-header (list)
  "Find row in the CSV data where the column header info starts."
  (when list
    (if (string-prefix-p "Station_ID" (car list) )
        list 
      (find-header (cdr list)))))

(defun clean-header (header)
  "Provide better names for column headers provided by the API."
  (let ((col-map '(("Station_ID" . "Station")
                    ("Date_Time" . "Date Time")
                    ("altimeter_set_1" . "Altimeter")
                    ("air_temp_set_1" . "Temperature")
                    ("relative_humidity_set_1" . "RH")
                    ("wind_speed_set_1" . "Wind Speed")
                    ("wind_direction_set_1" . "Wind Direction")
                    ("dew_point_temperature_set_1d" . "Dew Point")
                    ("pressure_set_1d" . "Pressure")
                    ("sea_level_pressure_set_1d" . "SLP"))))
    (mapcar (lambda (x) (cdr (assoc x col-map)))
            (split-string header ","))))

(let* ((string-data (split-string data))
       (rows (find-header string-data))
       (col-hdrs (clean-header (car rows)))
       (units (split-string (cadr rows) ","))
       (d (take-nth (/ (length rows) samples) (cddr rows))))
  (cons col-hdrs
        (cons units
              (cl-loop for s in d
                       collect
                       (let ((row (fix-missing s)))
                         (split-string row ","))))))
Station Date Time Altimeter Temperature RH Wind Speed Wind Direction Dew Point Pressure SLP
    Pascals Celsius % Meters/second Degrees Celsius Pascals Pascals
E3608 2017-03-06T18:28:00-0700 101659.34 -1.11 34.0 2.68 248.0 -15.11 83224.63 102038.3
E3608 2017-03-06T19:13:00-0700 101727.07 -1.67 39.0 1.79 268.0 -13.94 83280.08 102148.36
E3608 2017-03-06T19:58:00-0700 101828.66 -1.67 36.0 1.79 319.0 -14.92 83363.25 102250.4
E3608 2017-03-06T20:43:00-0700 101896.38 -1.67 37.0 2.68 265.0 -14.58 83418.69 102318.39
E3608 2017-03-06T21:28:00-0700 101964.11 -1.67 33.0 2.23 251.0 -15.97 83474.13 102386.42
E3608 2017-03-06T22:13:00-0700 101997.98 -2.22 42.0 2.68 234.0 -13.52 83501.86 102462.02
E3608 2017-03-06T22:58:00-0700 101997.98 -1.67 34.0 0.45 279.0 -15.61 83501.86 102420.42
E3608 2017-03-06T23:43:00-0700 101964.11 -1.11 26.0 2.23 254.0 -18.31 83474.13 102344.26
E3608 2017-03-07T00:33:00-0700 102031.84 -1.11 28.0 2.23 229.0 -17.44 83529.58 102412.23
E3608 2017-03-07T01:18:00-0700 102065.7 -1.67 29.0 1.79 266.0 -17.51 83557.3 102488.46
E3608 2017-03-07T02:03:00-0700 102065.7 -1.11 28.0 3.13 257.0 -17.44 83557.3 102446.22
E3608 2017-03-07T02:48:00-0700 102065.7 -1.11 29.0 0.9 245.0 -17.02 83557.3 102446.21
E3608 2017-03-07T03:33:00-0700 102065.7 -1.11 26.0 3.13 267.0 -18.31 83557.3 102446.23
E3608 2017-03-07T04:18:00-0700 102099.57 -1.11 28.0 1.79 243.0 -17.44 83585.03 102480.22
E3608 2017-03-07T05:03:00-0700 102099.57 -0.56 27.0 2.23 233.0 -17.39 83585.03 102438.9
E3608 2017-03-07T05:48:00-0700 102133.43 0.0 28.0 4.47 291.0 -16.48 83612.75 102430.96
E3608 2017-03-07T06:33:00-0700 102099.57 0.0 29.0 1.79 13.0 -16.06 83585.03 102396.99
E3608 2017-03-07T07:18:00-0700 102201.16 0.56 30.0 3.58 277.0 -15.16 83668.2 102457.13
E3608 2017-03-07T08:03:00-0700 102268.89 0.56 30.0 3.13 309.0 -15.16 83723.65 102525.03
E3608 2017-03-07T08:48:00-0700 102302.75 2.22 28.0 5.37 271.0 -14.55 83751.37 102436.19
E3608 2017-03-07T09:33:00-0700 102336.62 2.78 25.0 4.92 315.0 -15.44 83779.09 102429.04
E3608 2017-03-07T10:18:00-0700 102370.48 3.33 25.0 4.47 278.0 -14.97 83806.81 102422.75
E3608 2017-03-07T11:03:00-0700 102370.48 4.44 23.0 2.23 275.0 -15.03 83806.81 102342.19
E3608 2017-03-07T11:48:00-0700 102370.48 5.56 22.0 7.15 237.0 -14.62 83806.81 102261.6
E3608 2017-03-07T12:33:00-0700 102302.75 6.67 20.0 6.26 290.0 -14.83 83751.37 102114.83
E3608 2017-03-07T13:18:00-0700 102268.89 7.78 19.0 4.47 291.0 -14.53 83723.65 102002.6
E3608 2017-03-07T14:03:00-0700 102133.43 8.89 17.0 2.23 273.0 -14.95 83612.75 101789.84
E3608 2017-03-07T14:48:00-0700 102065.7 9.44 17.0 4.92 338.0 -14.5 83557.3 101684.12
E3608 2017-03-07T15:33:00-0700 102167.3 10.0 16.0 3.58 355.0 -14.77 83640.48 101746.57
E3608 2017-03-07T16:18:00-0700 102133.43 10.0 17.0 1.79 40.0 -14.04 83612.75 101712.82
E3608 2017-03-07T17:03:00-0700 102167.3 10.0 17.0 0.9 165.0 -14.04 83640.48 101746.56
E3608 2017-03-07T17:48:00-0700 102235.02 8.89 19.0 0.9 329.0 -13.61 83695.92 101891.07

Timeseries with Gnuplot

Gnuplot

At this point, we are ready to plot our data table with gnuplot. The following code took bit of experimentation to get right as well as studying the gnuplot official documentation. But essentially we are plotting in one stacked plot,

  • Sea level pressure
  • Temperature and dew point
  • Relative humidity
  • Wind speed
  • Wind direction

One tricky part is I had to alternate the Y-axis labeling so that the successive plot labels would not run into one another.

reset

set terminal svg size 600, 800 enhanced truecolor font 'Verdana,13'

set multiplot layout 5,1 rowsfirst

set grid
set lmargin 12
set rmargin 8

set style line 1 lc rgb '#000000' lt 1 lw 2 pt 9 ps 1.0
set style line 2 lc rgb '#dd181f' lt 1 lw 2 pt 7 ps 1.0
set style line 3 lc rgb '#0060ad' lt 1 lw 2 pt 7 ps 0.75
set style line 4 lc rgb '#00ff00' lt 1 lw 2 pt 7 ps 1
set style line 5 lc rgb '#00ff00' lt 1 lw 2 pt 5 ps 0.75

set title "Weather for E3608"

set xdata time
set timefmt "%Y-%m-%dT%H:%M:%SZ"
set format x " " 

set yrange [:*]
set ylabel "Pascals" offset 0

set bmargin 0
set offsets graph 0, 0, 500, 10

set size 1,0.24
set origin 0.0,0.76

plot data using " Date Time":"SLP" with linespoints ls 1 title 'Sea Level Pressure'

set title " "

set yrange [:*]
set ytics offset 56,0
set ylabel "Celsius" offset 63

set tmargin 0
set offsets graph 0, 0, 10, 1

set size 1,0.17
set origin 0.0,0.59

plot data using "Date Time":"Temperature" with linespoints ls 2 title 'Temperature',\
     data using "Date Time":"Dew Point" with linespoints ls 3 title 'Dewpoint'

set yrange [0:100]
set ytics offset 0,0
set ylabel "%" offset 0

set offsets graph 0, 0, 0, 0 

set size 1,0.17
set origin 0.0,0.42

plot data using "Date Time":"RH" with linespoints ls 4 title 'RH'

set yrange [0:*]
set ytics offset 55,0
set ylabel "Meters per Second" offset 63

set offsets graph 0, 0, 1, 0

set size 1,0.17
set origin 0.0,0.25

plot data using " Date Time":"Wind Speed" with linespoints ls 5 title 'Wind Speed'

set xtics rotate by -45
set format x "%m/%d %Hh"

set yrange [0:360]
set ylabel "Direction From" offset 0
set ytics ("N" 0,"NE" 45,"E" 90 ,"SE" 135, "S" 180, "SW" 225, "W" 270, "NW" 315) offset 0,0

set bmargin 4

set size 1,0.25
set origin 0.0,0

plot data using " Date Time":"Wind Direction" pt 3 ps 1.0 title 'Wind Direction'

unset multiplot

Sorry, your browser does not support SVG.

Figure 2: Time Series of Weather Observations

Babel

As mentioned in the introduction, to generate the data table and plot, this entire buffer can be run in literate programming fashion with org-babel-execute-buffer (C-c C-v C-b). You can parameterize the start and end time, and the station ID via noweb parameters. I will leave it as an exercise to the reader to supply additional parameterization for the station variables, for example.

Ideas for Improvements

In the future, I would like to incorporate more information in the plot. Specifically,

I'll have to learn more about gnuplot in order to achieve the first two bullet items.

SourceLast updated on Wed Mar 8 01:35:34 2017 • julienchastang.com by Julien Chastang