#+OPTIONS: ':nil *:t -:t ::t <:t H:3 \n:nil ^:t arch:headline author:t c:nil
#+OPTIONS: creator:nil d:(not "LOGBOOK") date:t e:t email:nil f:t inline:t
#+OPTIONS: num:nil p:nil pri:nil prop:nil stat:t tags:t tasks:t tex:t timestamp:t
#+OPTIONS: title:t toc:nil todo:t |:t
#+TITLE: Plotting Time Series of Weather Observations in Emacs
#+DATE: <2017-03-01 Wed>
#+AUTHOR: Julien Chastang
#+EMAIL: julien dot c dot chastang at gmail
#+LANGUAGE: en
#+SELECT_TAGS: export
#+EXCLUDE_TAGS: noexport
#+CREATOR: Emacs 24.5.1 (Org mode 8.3.6)

#+SETUPFILE: ../../templates/level-1.org
#+INCLUDE: ../../templates/level-1.org

#+PROPERTY: header-args :exports both

#+BEGIN_SRC emacs-lisp :results silent :exports none  :tangle no
  (setq org-confirm-babel-evaluate nil)
  (setq org-export-babel-evaluate nil)
#+END_SRC

* <2017-03-01 Wed>

#+CAPTION: Time Series of Weather Observations
file:../../static/weatherobs/timeseries.svg

** Introduction
<<intro>>

In this post, we will be using Emacs and org-mode to construct a time series of weather observations. If you want the short version of what we will be doing, change the ~station-id~ below and run ~org-babel-execute-buffer~ (~C-c C-v C-b~). For the complete version read on. We will be using the following technologies:
  - org-mode
  - ob-http, http requests in org Babel
  - noweb literate programming
  - gnuplot, the venerable plotting utility
  - Mesowest Synoptic Labs API

The org source for this file is located at the source link at the bottom of the page. Indeed, to follow along carefully you will have to view that source link as the HTML rendering has removed some of the org source from this web page.
** Find a Weather Station

Our first job is to find a weather station near our location. We will employ the SynopticLabs Mesonet API to identify a station ID near my current location. I live in Boulder, Colorado at roughly these coordinates: 39.99, -105.22. The Mesonet API has a service to find weather stations nearest a given latitude and longitude point. You have to supply a US ~state~ (easy to forget) and a ~radius~ arguments composed of a latitude, longitude center and a radius distance in miles. In addition, there are a couple of other ancillary arguments including a demo token. Consult the API documentation to ensure you are working within the terms of use. The following URL will provide us a station near by with the help of ob-http:

*** ob-http Request and JSON Response

#+BEGIN_SRC http :pretty
  GET http://api.mesowest.net/v2/stations/metadata?state=co&status=active&radius=39.99,-105.22,1&token=demotoken
#+END_SRC

#+RESULTS:
#+begin_example
{
   "SUMMARY": {
     "METADATA_RESPONSE_TIME": "0.123023986816 ms",
     "RESPONSE_MESSAGE": "OK",
     "RESPONSE_CODE": 1,
     "NUMBER_OF_OBJECTS": 1
   },
  "STATION": [
    {
       "ID": "41017",
       "TIMEZONE": "America\/Denver",
       "LATITUDE": "40.00133",
       "STATE": "CO",
       "LONGITUDE": "-105.23033",
       "STID": "E3608",
       "DISTANCE": 0.96,
       "NAME": "EW3608 Boulder",
       "ELEVATION": "5430",
      "PERIOD_OF_RECORD": {
         "end": "2017-03-07T23:53:00Z",
         "start": "2013-09-25T00:00:00Z"
      },
       "MNET_ID": "65",
       "STATUS": "ACTIVE"
    }
   ]
}
#+end_example


According to this JSON response, there is the "E3608" weather station not far away. We will work with the data from that instrument.

** Timeseries API

Next, we will be using the timeseries API to retrieve the station data. For example, the following URL retrieves the air temperature from the KDEN station for the last 30 minutes.

#+BEGIN_SRC http :pretty
  GET http://api.mesowest.net/v2/stations/timeseries?&stid=kden&recent=30&token=demotoken&vars=air_temp&output=csv
#+END_SRC

#+RESULTS:
#+begin_example
# STATION: KDEN
# STATION NAME: Denver, Denver International Airport
# LATITUDE: 39.84658
# LONGITUDE: -104.65622
# ELEVATION [ft]: 5404
# STATE: CO
Station_ID,Date_Time,air_temp_set_1
,,Celsius
 KDEN,2017-03-08T01:00:00Z,8.0
 KDEN,2017-03-08T01:05:00Z,8.0
 KDEN,2017-03-08T01:10:00Z,8.0
 KDEN,2017-03-08T01:15:00Z,8.0
 KDEN,2017-03-08T01:20:00Z,8.0
#+end_example

There are more examples of API use as well as the API reference on SynopticLabs website. 

For the particular example we are shooting for, we will supply the following URL parameters:

  - ~stid~, the station ID that we obtained earlier
  - ~start~, the start time in UTC (e.g., ~201702171644~)
  - ~end~, the end time  in UTC (e.g., ~201702241644~)
  - ~units~, the desired units (i.e., metrics or english)
  - ~vars~, the sensor variables
  - ~obtimezone~, the timezone of the response data (i.e., UTC or local)
  - ~output~, the output format (e.g., csv) 
  - ~token~, the request token, in this case guest

** Set up Parameterization With noweb
<<params>>

We will once again be make use of ob-http to retrieve the data, but, this time, we will be employing the Emacs noweb facility to parameterize the ob-http code block. (By the way, "noweb", has always struck me as a terrible name as it seems to imply something about the web browsing, the world wide web, or the Internet, but, in fact, refers to a literate programming style.) Let's start defining our noweb parameters with named org-mode and Emacs Lisp code blocks. We will use the names subsequently via noweb.

*** Start Time

The start time range will be 24 hours ago. Also set the time zone as UTC since that is what the SynopticLabs API requires.

#+NAME: starttime
#+BEGIN_SRC emacs-lisp
  (setenv "TZ" "UTC0")
  (format-time-string "%Y%m%d%H%M" (time-add
                                    (current-time)
                                    (seconds-to-time (- (* 1 24 60 60)))))
#+END_SRC

#+RESULTS: starttime
 : 201703070128

*** End Time

The end time will be now.

#+NAME: endtime
#+BEGIN_SRC emacs-lisp
  (format-time-string "%Y%m%d%H%M")
#+END_SRC

#+RESULTS: endtime
 : 201703080128

*** Station ID

The station ID we found earlier.

#+NAME: station-id
#+BEGIN_SRC emacs-lisp
  ;; "KBJC"
  "E3608"
  ;; "E7829"
#+END_SRC

#+RESULTS: station-id
: E3608

** Retrieve the Weather Observations
*** Retrieve Data

At this point, we can finally retrieve our data with the help our noweb parameters. For example, ~<<starttime()>>~ will invoke the ~starttime~ code block defined above and insert the result into the http address below. The same is true for ~station-id~ and ~endtime~. See the source link at the bottom to see the noweb parameterization.

#+NAME: timeseries
#+BEGIN_SRC http :pretty :results silent :noweb yes
GET http://api.mesowest.net/v2/stations/timeseries?token=demotoken&stid=<<station-id()>>&start=<<starttime()>>&end=<<endtime()>>&obtimezone=local&units=metric&vars=air_temp,dew_point_temperature,wind_speed,wind_direction,sea_level_pressure&output=csv
#+END_SRC

*** Post-request cleanup with Emacs Lisp

At this point, we have to write some Emacs Lisp to clean up the response data and make it more org-mode friendly. We will need four helper functions:

  - ~take-nth~ is a function to sample data at certain intervals from a list.
  - gnuplot does not like empty org-table cells so the ~fix-missing~ function replaces empty cells with the "missing" string.
  - The ~find-header~ function locates where the header and actual data are located in the CSV http response.
  - The ~clean-header~ function replaces cryptic columns names with ones that are easier to understand.

Armed with these functions, we can clean up our data. Also a note about sampling: large org-tables tend to bog down Emacs so I'm limiting the table size to 30 samples via a Babel header argument, and the ~take-nth~ helper function.

#+NAME: data-table
#+BEGIN_SRC emacs-lisp :var data=timeseries :var samples=30 :results table drawer
  (defun take-nth (n list )
    "Returns a list of every nth item in the list."
    (if (<= n 1) list
      (when list
        (cons (car list)
              (take-nth n (nthcdr n list))))))

  (defun fix-missing (s)
    "Replace missing CSV values with 'missing' to avoid empty cells in org-tables."
    (if (string-match-p ",\s*," s)
        (fix-missing (replace-regexp-in-string ",\s*," ",missing," s))
      s))

  (defun find-header (list)
    "Find row in the CSV data where the column header info starts."
    (when list
      (if (string-prefix-p "Station_ID" (car list) )
          list 
        (find-header (cdr list)))))

  (defun clean-header (header)
    "Provide better names for column headers provided by the API."
    (let ((col-map '(("Station_ID" . "Station")
                      ("Date_Time" . "Date Time")
                      ("altimeter_set_1" . "Altimeter")
                      ("air_temp_set_1" . "Temperature")
                      ("relative_humidity_set_1" . "RH")
                      ("wind_speed_set_1" . "Wind Speed")
                      ("wind_direction_set_1" . "Wind Direction")
                      ("dew_point_temperature_set_1d" . "Dew Point")
                      ("pressure_set_1d" . "Pressure")
                      ("sea_level_pressure_set_1d" . "SLP"))))
      (mapcar (lambda (x) (cdr (assoc x col-map)))
              (split-string header ","))))

  (let* ((string-data (split-string data))
         (rows (find-header string-data))
         (col-hdrs (clean-header (car rows)))
         (units (split-string (cadr rows) ","))
         (d (take-nth (/ (length rows) samples) (cddr rows))))
    (cons col-hdrs
          (cons units
                (cl-loop for s in d
                         collect
                         (let ((row (fix-missing s)))
                           (split-string row ","))))))
#+END_SRC

#+RESULTS: data-table
:RESULTS:
 | Station | Date Time                | Altimeter | Temperature |   RH |    Wind Speed | Wind Direction | Dew Point | Pressure |       SLP |
 |         |                          |   Pascals |     Celsius |    % | Meters/second |        Degrees |   Celsius |  Pascals |   Pascals |
 | E3608   | 2017-03-06T18:28:00-0700 | 101659.34 |       -1.11 | 34.0 |          2.68 |          248.0 |    -15.11 | 83224.63 |  102038.3 |
 | E3608   | 2017-03-06T19:13:00-0700 | 101727.07 |       -1.67 | 39.0 |          1.79 |          268.0 |    -13.94 | 83280.08 | 102148.36 |
 | E3608   | 2017-03-06T19:58:00-0700 | 101828.66 |       -1.67 | 36.0 |          1.79 |          319.0 |    -14.92 | 83363.25 |  102250.4 |
 | E3608   | 2017-03-06T20:43:00-0700 | 101896.38 |       -1.67 | 37.0 |          2.68 |          265.0 |    -14.58 | 83418.69 | 102318.39 |
 | E3608   | 2017-03-06T21:28:00-0700 | 101964.11 |       -1.67 | 33.0 |          2.23 |          251.0 |    -15.97 | 83474.13 | 102386.42 |
 | E3608   | 2017-03-06T22:13:00-0700 | 101997.98 |       -2.22 | 42.0 |          2.68 |          234.0 |    -13.52 | 83501.86 | 102462.02 |
 | E3608   | 2017-03-06T22:58:00-0700 | 101997.98 |       -1.67 | 34.0 |          0.45 |          279.0 |    -15.61 | 83501.86 | 102420.42 |
 | E3608   | 2017-03-06T23:43:00-0700 | 101964.11 |       -1.11 | 26.0 |          2.23 |          254.0 |    -18.31 | 83474.13 | 102344.26 |
 | E3608   | 2017-03-07T00:33:00-0700 | 102031.84 |       -1.11 | 28.0 |          2.23 |          229.0 |    -17.44 | 83529.58 | 102412.23 |
 | E3608   | 2017-03-07T01:18:00-0700 |  102065.7 |       -1.67 | 29.0 |          1.79 |          266.0 |    -17.51 |  83557.3 | 102488.46 |
 | E3608   | 2017-03-07T02:03:00-0700 |  102065.7 |       -1.11 | 28.0 |          3.13 |          257.0 |    -17.44 |  83557.3 | 102446.22 |
 | E3608   | 2017-03-07T02:48:00-0700 |  102065.7 |       -1.11 | 29.0 |           0.9 |          245.0 |    -17.02 |  83557.3 | 102446.21 |
 | E3608   | 2017-03-07T03:33:00-0700 |  102065.7 |       -1.11 | 26.0 |          3.13 |          267.0 |    -18.31 |  83557.3 | 102446.23 |
 | E3608   | 2017-03-07T04:18:00-0700 | 102099.57 |       -1.11 | 28.0 |          1.79 |          243.0 |    -17.44 | 83585.03 | 102480.22 |
 | E3608   | 2017-03-07T05:03:00-0700 | 102099.57 |       -0.56 | 27.0 |          2.23 |          233.0 |    -17.39 | 83585.03 |  102438.9 |
 | E3608   | 2017-03-07T05:48:00-0700 | 102133.43 |         0.0 | 28.0 |          4.47 |          291.0 |    -16.48 | 83612.75 | 102430.96 |
 | E3608   | 2017-03-07T06:33:00-0700 | 102099.57 |         0.0 | 29.0 |          1.79 |           13.0 |    -16.06 | 83585.03 | 102396.99 |
 | E3608   | 2017-03-07T07:18:00-0700 | 102201.16 |        0.56 | 30.0 |          3.58 |          277.0 |    -15.16 |  83668.2 | 102457.13 |
 | E3608   | 2017-03-07T08:03:00-0700 | 102268.89 |        0.56 | 30.0 |          3.13 |          309.0 |    -15.16 | 83723.65 | 102525.03 |
 | E3608   | 2017-03-07T08:48:00-0700 | 102302.75 |        2.22 | 28.0 |          5.37 |          271.0 |    -14.55 | 83751.37 | 102436.19 |
 | E3608   | 2017-03-07T09:33:00-0700 | 102336.62 |        2.78 | 25.0 |          4.92 |          315.0 |    -15.44 | 83779.09 | 102429.04 |
 | E3608   | 2017-03-07T10:18:00-0700 | 102370.48 |        3.33 | 25.0 |          4.47 |          278.0 |    -14.97 | 83806.81 | 102422.75 |
 | E3608   | 2017-03-07T11:03:00-0700 | 102370.48 |        4.44 | 23.0 |          2.23 |          275.0 |    -15.03 | 83806.81 | 102342.19 |
 | E3608   | 2017-03-07T11:48:00-0700 | 102370.48 |        5.56 | 22.0 |          7.15 |          237.0 |    -14.62 | 83806.81 |  102261.6 |
 | E3608   | 2017-03-07T12:33:00-0700 | 102302.75 |        6.67 | 20.0 |          6.26 |          290.0 |    -14.83 | 83751.37 | 102114.83 |
 | E3608   | 2017-03-07T13:18:00-0700 | 102268.89 |        7.78 | 19.0 |          4.47 |          291.0 |    -14.53 | 83723.65 |  102002.6 |
 | E3608   | 2017-03-07T14:03:00-0700 | 102133.43 |        8.89 | 17.0 |          2.23 |          273.0 |    -14.95 | 83612.75 | 101789.84 |
 | E3608   | 2017-03-07T14:48:00-0700 |  102065.7 |        9.44 | 17.0 |          4.92 |          338.0 |     -14.5 |  83557.3 | 101684.12 |
 | E3608   | 2017-03-07T15:33:00-0700 |  102167.3 |        10.0 | 16.0 |          3.58 |          355.0 |    -14.77 | 83640.48 | 101746.57 |
 | E3608   | 2017-03-07T16:18:00-0700 | 102133.43 |        10.0 | 17.0 |          1.79 |           40.0 |    -14.04 | 83612.75 | 101712.82 |
 | E3608   | 2017-03-07T17:03:00-0700 |  102167.3 |        10.0 | 17.0 |           0.9 |          165.0 |    -14.04 | 83640.48 | 101746.56 |
 | E3608   | 2017-03-07T17:48:00-0700 | 102235.02 |        8.89 | 19.0 |           0.9 |          329.0 |    -13.61 | 83695.92 | 101891.07 |
:END:

** Timeseries with Gnuplot
*** Plot Dimensions :noexport:

A little spreadsheet to help me find the plot dimensions and locations.

| Top Header     |  53 | 0.2 | 0.8 |  53 | 227 | 0.24 | 0.76 |
| Pressure       | 147 |     |     | 174 |     |      |      |
| Temperature    | 200 | 0.2 | 0.6 | 174 | 174 | 0.17 | 0.59 |
| RH             | 200 | 0.2 | 0.4 | 174 | 174 | 0.17 | 0.42 |
| Wind speed     | 200 | 0.2 | 0.2 | 174 | 174 | 0.17 | 0.25 |
| Wind direction | 127 | 0.2 | 0.0 | 174 | 247 | 0.25 |  0.0 |
| Bottom Labels  |  73 |     |     |  73 |     |      |      |

*** Gnuplot

At this point, we are ready to plot our data table with gnuplot. The following code took bit of experimentation to get right as well as studying the gnuplot official documentation. But essentially we are plotting in one stacked plot,

  - Sea level pressure
  - Temperature and dew point
  - Relative humidity
  - Wind speed
  - Wind direction

One tricky part is I had to alternate the Y-axis labeling so that the successive plot labels would not run into one another.

#+BEGIN_SRC gnuplot :var data=data-table :noweb yes :file ../../static/weatherobs/timeseries.svg
  reset

  set terminal svg size 600, 800 enhanced truecolor font 'Verdana,13'

  set multiplot layout 5,1 rowsfirst

  set grid
  set lmargin 12
  set rmargin 8

  set style line 1 lc rgb '#000000' lt 1 lw 2 pt 9 ps 1.0
  set style line 2 lc rgb '#dd181f' lt 1 lw 2 pt 7 ps 1.0
  set style line 3 lc rgb '#0060ad' lt 1 lw 2 pt 7 ps 0.75
  set style line 4 lc rgb '#00ff00' lt 1 lw 2 pt 7 ps 1
  set style line 5 lc rgb '#00ff00' lt 1 lw 2 pt 5 ps 0.75

  set title "Weather for <<station-id()>>"

  set xdata time
  set timefmt "%Y-%m-%dT%H:%M:%SZ"
  set format x " " 

  set yrange [:*]
  set ylabel "Pascals" offset 0

  set bmargin 0
  set offsets graph 0, 0, 500, 10

  set size 1,0.24
  set origin 0.0,0.76

  plot data using " Date Time":"SLP" with linespoints ls 1 title 'Sea Level Pressure'

  set title " "

  set yrange [:*]
  set ytics offset 56,0
  set ylabel "Celsius" offset 63

  set tmargin 0
  set offsets graph 0, 0, 10, 1

  set size 1,0.17
  set origin 0.0,0.59

  plot data using "Date Time":"Temperature" with linespoints ls 2 title 'Temperature',\
       data using "Date Time":"Dew Point" with linespoints ls 3 title 'Dewpoint'

  set yrange [0:100]
  set ytics offset 0,0
  set ylabel "%" offset 0

  set offsets graph 0, 0, 0, 0 

  set size 1,0.17
  set origin 0.0,0.42

  plot data using "Date Time":"RH" with linespoints ls 4 title 'RH'

  set yrange [0:*]
  set ytics offset 55,0
  set ylabel "Meters per Second" offset 63

  set offsets graph 0, 0, 1, 0

  set size 1,0.17
  set origin 0.0,0.25

  plot data using " Date Time":"Wind Speed" with linespoints ls 5 title 'Wind Speed'

  set xtics rotate by -45
  set format x "%m/%d %Hh"

  set yrange [0:360]
  set ylabel "Direction From" offset 0
  set ytics ("N" 0,"NE" 45,"E" 90 ,"SE" 135, "S" 180, "SW" 225, "W" 270, "NW" 315) offset 0,0

  set bmargin 4

  set size 1,0.25
  set origin 0.0,0

  plot data using " Date Time":"Wind Direction" pt 3 ps 1.0 title 'Wind Direction'

  unset multiplot
#+END_SRC

 #+CAPTION: Time Series of Weather Observations
#+RESULTS:
file:../../static/weatherobs/timeseries.svg

** Babel

As mentioned in the introduction, to generate the data table and plot, this entire buffer can be run in literate programming fashion with ~org-babel-execute-buffer~ (~C-c C-v C-b~). You can parameterize the start and end time, and the station ID via noweb parameters. I will leave it as an exercise to the reader to supply additional parameterization for the station variables, for example.

** Ideas for Improvements

In the future, I would like to incorporate more information in the plot. Specifically,

  - Replace the wind and direction time series with wind barbs.
  - Station plot weather symbols 
  - Add precipitation data

I'll have to learn more about gnuplot in order to achieve the first two bullet items.