Search

Recent Posts

Tags


« | Main | »

Using RRDs with Many Data Sources (rrdtool)

By Dale Reagan | August 22, 2011

When reviewing rrdtool I found many examples showing ‘real time’ updates – but few examples for updating RRDs/RRAs (round robin databases/archives) with multiple data sources and/or source data that you might want to ‘migrate’ into an existing RRD.  A simple example:

This post will not explain the structure of RRDs or the details of using rrdtool – the emphasis is on how to get existing data into the RRD.   The example below creates a simple RRD with two ‘data sources‘, temp1 and temp2.  The values will be extracted using lm-sensors on a Linux system.  The command below creates an RRD using the current system time as a starting point.

        rrdtool create /path-to-data/sample_temps.rrd --start N --step 300 \
                DS:temp1:GAUGE:300:U:U \
                DS:temp2:GAUGE:300:U:U \
                  RRA:AVERAGE:0.5:1:12 \
                  RRA:AVERAGE:0.5:1:288 \
                  RRA:AVERAGE:0.5:12:168 \
                  RRA:AVERAGE:0.5:12:720 \
                  RRA:AVERAGE:0.5:288:365

It is easy to get ‘lost’ in and rrdtool update since all updates require the same ‘update pattern‘, i.e. a time stamp, data value, data value…, last data value.  You can use a ‘template’ but it still separates the data_source_name from the update_value.   When you have a simple RRA (only a few data sources) then updates are reasonably simple.  When you have 10+ data sources things can get complicated – but they are still manageable.

An update command would be something like:

rrdtool update -t temp1:temp2  N:var1:var2

where var1 and var2 would be temperature readings taken at time slot N with var1 being the value for temp1 and var2 being the value for temp2.  RRDs are well suited for data sets that do not change (i.e. no new devices/fields will be added) – you simply create the RRD and then add data that meets your required interval (the STEP value used when you created the RRD.)

After creating the simple two-data-source RRD above here is a look at the RRD ‘info’ for the file – the ‘ds’ references are for the data_sources and the ‘rra‘ references are for the round-robin-archives:

rrdtool info sys-temps.rrd
filename = "sys-temps.rrd"
rrd_version = "0003"
step = 300
last_update = 1313192881
ds[temp1].type = "GAUGE"
ds[temp1].minimal_heartbeat = 300
ds[temp1].min = NaN
ds[temp1].max = NaN
ds[temp1].last_ds = "116.6"
ds[temp1].value = 2.1104600000e+04
ds[temp1].unknown_sec = 0
ds[temp2].type = "GAUGE"
ds[temp2].minimal_heartbeat = 300
ds[temp2].min = NaN
ds[temp2].max = NaN
ds[temp2].last_ds = "116.6"
ds[temp2].value = 2.1104600000e+04
ds[temp2].unknown_sec = 0
rra[0].cf = "AVERAGE"
rra[0].rows = 12
rra[0].cur_row = 1
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[0].cdp_prep[1].value = NaN
rra[0].cdp_prep[1].unknown_datapoints = 0
rra[1].cf = "AVERAGE"
rra[1].rows = 288
rra[1].cur_row = 186
rra[1].pdp_per_row = 1
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = NaN
rra[1].cdp_prep[0].unknown_datapoints = 0
rra[1].cdp_prep[1].value = NaN
rra[1].cdp_prep[1].unknown_datapoints = 0
rra[2].cf = "AVERAGE"
rra[2].rows = 168
rra[2].cur_row = 166
rra[2].pdp_per_row = 12
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = 9.8346600000e+02
rra[2].cdp_prep[0].unknown_datapoints = 0
rra[2].cdp_prep[1].value = 1.0429200000e+03
rra[2].cdp_prep[1].unknown_datapoints = 0
rra[3].cf = "AVERAGE"
rra[3].rows = 720
rra[3].cur_row = 650
rra[3].pdp_per_row = 12
rra[3].xff = 5.0000000000e-01
rra[3].cdp_prep[0].value = 9.8346600000e+02
rra[3].cdp_prep[0].unknown_datapoints = 0
rra[3].cdp_prep[1].value = 1.0429200000e+03
rra[3].cdp_prep[1].unknown_datapoints = 0
rra[4].cf = "AVERAGE"
rra[4].rows = 365
rra[4].cur_row = 157
rra[4].pdp_per_row = 288
rra[4].xff = 5.0000000000e-01
rra[4].cdp_prep[0].value = 2.9610746388e+04
rra[4].cdp_prep[0].unknown_datapoints = 27
rra[4].cdp_prep[1].value = 2.9980202388e+04
rra[4].cdp_prep[1].unknown_datapoints = 27

After several days of collecting data a ‘dump’ of the RRD yields something like the below followed by quite a bit of RRA data (the info below is limited to the first ~60 lines of the XML  ‘dump’.)

rrdtool dump sys-temps.rrd | head -some_number_of_lines
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE rrd SYSTEM "http://oss.oetiker.ch/rrdtool/rrdtool.dtd">
<!-- Round Robin Database Dump --><rrd> <version> 0003 </version>
        <step> 300 </step> <!-- Seconds -->
        <lastupdate> 1313197021 </lastupdate> <!-- 2011-08-12 20:57:01 EDT -->

        <ds>
                <name> temp1 </name>
                <type> GAUGE </type>
                <minimal_heartbeat> 300 </minimal_heartbeat>
                <min> NaN </min>
                <max> NaN </max>

                <!-- PDP Status -->
                <last_ds> 116.6 </last_ds>
                <value> 1.4108600000e+04 </value>
                <unknown_sec> 0 </unknown_sec>
        </ds>

        <ds>
                <name> temp2 </name>
                <type> GAUGE </type>
                <minimal_heartbeat> 300 </minimal_heartbeat>
                <min> NaN </min>
                <max> NaN </max>

                <!-- PDP Status -->
                <last_ds> 116.6 </last_ds>
                <value> 1.4108600000e+04 </value>
                <unknown_sec> 0 </unknown_sec>
        </ds>

<!-- Round Robin Archives -->   <rra>
                <cf> AVERAGE </cf>
                <pdp_per_row> 1 </pdp_per_row> <!-- 300 seconds -->

                <params>
                <xff> 5.0000000000e-01 </xff>
                </params>
                <cdp_prep>
                        <ds>
                        <primary_value> 1.1588600000e+02 </primary_value>
                        <secondary_value> NaN </secondary_value>
                        <value> NaN </value>
                        <unknown_datapoints> 0 </unknown_datapoints>
                        </ds>
                        <ds>
                        <primary_value> 1.1660000000e+02 </primary_value>
                        <secondary_value> NaN </secondary_value>
                        <value> NaN </value>
                        <unknown_datapoints> 0 </unknown_datapoints>
                        </ds>
                </cdp_prep>
                <database>
                        <!-- 2011-08-12 20:00:00 EDT / 1313193600 --> <row><v> 1.1732600000e+02 </v><v> 1.1660000000e+02 </v></row>
                        <!-- 2011-08-12 20:05:00 EDT / 1313193900 --> <row><v> 1.1731400000e+02 </v><v> 1.1660000000e+02 </v></row>
                        <!-- 2011-08-12 20:10:00 EDT / 1313194200 --> <row><v> 1.1804600000e+02 </v><v> 1.1660000000e+02 </v></row>
                        <!-- 2011-08-12 20:15:00 EDT / 1313194500 --> <row><v> 1.1767400000e+02 </v><v> 1.1660000000e+02 </v></row>
                        <!-- 2011-08-12 20:20:00 EDT / 1313194800 --> <row><v> 1.1660600000e+02 </v><v> 1.1731400000e+02 </v></row>
                        <!-- 2011-08-12 20:25:00 EDT / 1313195100 --> <row><v> 1.1660000000e+02 </v><v> 1.1696600000e+02 </v></row>
                        <!-- 2011-08-12 20:30:00 EDT / 1313195400 --> <row><v> 1.1552600000e+02 </v><v> 1.1660000000e+02 </v></row>
                        <!-- 2011-08-12 20:35:00 EDT / 1313195700 --> <row><v> 1.1551400000e+02 </v><v> 1.1660000000e+02 </v></row>
                        <!-- 2011-08-12 20:40:00 EDT / 1313196000 --> <row><v> 1.1587400000e+02 </v><v> 1.1660000000e+02 </v></row>
                        <!-- 2011-08-12 20:45:00 EDT / 1313196300 --> <row><v> 1.1732600000e+02 </v><v> 1.1660000000e+02 </v></row>
                        <!-- 2011-08-12 20:50:00 EDT / 1313196600 --> <row><v> 1.1731400000e+02 </v><v> 1.1660000000e+02 </v></row>
                        <!-- 2011-08-12 20:55:00 EDT / 1313196900 --> <row><v> 1.1588600000e+02 </v><v> 1.1660000000e+02 </v></row>
                </database>
        </rra>

Ok – the small sample of ‘dump data’ provides quite a bit of info to ponder (perhaps) but the objective is a small Bash shell script to update the RRD.  I started out using a script-that-created-a-script approach.  A very simple approach would be:

In this case I saved the original lm-sensors data to a simple text file which has the following structure:

        ** Head_/var/logs/headtail.log.tmpdata **
    1. ## Sun Aug  7 08:18:01 EDT 2011 ###
    2. ###
    3. ## Sun Aug  7 08:21:01 EDT 2011 | 1312719661:114.8:116.6 ###
    4. ## Sun Aug  7 08:24:01 EDT 2011 | 1312719841:114.8:116.6 ###
    5. ## Sun Aug  7 08:27:01 EDT 2011 | 1312720021:116.6:116.6 ###
    6. ## Sun Aug  7 08:30:01 EDT 2011 | 1312720201:116.6:116.6 ###
    7. ## Sun Aug  7 08:33:01 EDT 2011 | 1312720381:113.0:116.6 ###
    8. ## Sun Aug  7 08:36:01 EDT 2011 | 1312720561:116.6:116.6 ###
    9. ## Sun Aug  7 08:39:01 EDT 2011 | 1312720741:114.8:116.6 ###
   10. ## Sun Aug  7 08:42:01 EDT 2011 | 1312720921:116.6:116.6 ###
   11. ## Sun Aug  7 08:45:01 EDT 2011 | 1312721101:114.8:116.6 ###
   12. ## Sun Aug  7 08:48:01 EDT 2011 | 1312721281:116.6:116.6 ###
        *************** TAIL_/var/logs/headtail.log.tmpdata ***************
 3854. ## Mon Aug 15 11:00:01 EDT 2011 | 1313420401:116.6:116.6 ###
 3855. ## Mon Aug 15 11:03:01 EDT 2011 | 1313420581:116.6:116.6 ###
 3856. ## Mon Aug 15 11:06:01 EDT 2011 | 1313420761:120.2:116.6 ###
 3857. ## Mon Aug 15 11:09:01 EDT 2011 | 1313420941:116.6:116.6 ###
 3858. ## Mon Aug 15 11:12:01 EDT 2011 | 1313421121:122.0:116.6 ###
 3859. ## Mon Aug 15 11:15:01 EDT 2011 | 1313421301:116.6:116.6 ###
 3860. ## Mon Aug 15 11:18:01 EDT 2011 | 1313421481:118.4:116.6 ###
 3861. ## Mon Aug 15 11:21:01 EDT 2011 | 1313421661:114.8:116.6 ###
 3862. ## Mon Aug 15 11:24:01 EDT 2011 | 1313421841:118.4:116.6 ###
 3863. ## Mon Aug 15 11:27:01 EDT 2011 | 1313422021:114.8:116.6 ###

The head/tail output from the source file provides:

  1. date & time
  2. E_TIME=Unix Epoch time in seconds (used by rrdtool – intentionally included to avoid having to deal with date calculations…)
  3. temp1 (data_source one)
  4. temp2 (data_source two)

To create the RRD for this data set we can use the very first E_TIME value minus 60 seconds so the command becomes something like:

        rrdtool create /path-to-data/new_temps.rrd --start 1312719601 --step 300 \
                DS:temp1:GAUGE:300:10:190 \
                DS:temp2:GAUGE:300:10:190 \
                  RRA:AVERAGE:0.5:1:12 \
                  RRA:AVERAGE:0.5:1:288 \
                  RRA:AVERAGE:0.5:12:168 \
                  RRA:AVERAGE:0.5:12:720 \
                  RRA:AVERAGE:0.5:288:365

Also, for the above RRD, min/max values were used.  Reviewing the new RRD we see:

rrdtool info new_temps.rrd
filename = "new_temps.rrd"
rrd_version = "0003"
step = 300
last_update = 1312719601
ds[temp1].type = "GAUGE"
ds[temp1].minimal_heartbeat = 300
ds[temp1].min = 1.0000000000e+01
ds[temp1].max = 1.9000000000e+02
ds[temp1].last_ds = "U"
ds[temp1].value = 0.0000000000e+00
ds[temp1].unknown_sec = 1
ds[temp2].type = "GAUGE"
ds[temp2].minimal_heartbeat = 300
ds[temp2].min = 1.0000000000e+01
ds[temp2].max = 1.9000000000e+02
ds[temp2].last_ds = "U"
ds[temp2].value = 0.0000000000e+00
ds[temp2].unknown_sec = 1
rra[0].cf = "AVERAGE"
rra[0].rows = 12
rra[0].cur_row = 11
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[0].cdp_prep[1].value = NaN
rra[0].cdp_prep[1].unknown_datapoints = 0
rra[1].cf = "AVERAGE"
rra[1].rows = 288
rra[1].cur_row = 165
rra[1].pdp_per_row = 1
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = NaN
rra[1].cdp_prep[0].unknown_datapoints = 0
rra[1].cdp_prep[1].value = NaN
rra[1].cdp_prep[1].unknown_datapoints = 0
rra[2].cf = "AVERAGE"
rra[2].rows = 168
rra[2].cur_row = 80
rra[2].pdp_per_row = 12
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = NaN
rra[2].cdp_prep[0].unknown_datapoints = 4
rra[2].cdp_prep[1].value = NaN
rra[2].cdp_prep[1].unknown_datapoints = 4
rra[3].cf = "AVERAGE"
rra[3].rows = 720
rra[3].cur_row = 402
rra[3].pdp_per_row = 12
rra[3].xff = 5.0000000000e-01
rra[3].cdp_prep[0].value = NaN
rra[3].cdp_prep[0].unknown_datapoints = 4
rra[3].cdp_prep[1].value = NaN
rra[3].cdp_prep[1].unknown_datapoints = 4
rra[4].cf = "AVERAGE"
rra[4].rows = 365
rra[4].cur_row = 200
rra[4].pdp_per_row = 288
rra[4].xff = 5.0000000000e-01
rra[4].cdp_prep[0].value = NaN
rra[4].cdp_prep[0].unknown_datapoints = 148
rra[4].cdp_prep[1].value = NaN
rra[4].cdp_prep[1].unknown_datapoints = 148

Now we just need to ‘load’ the data – starting with just the first two data points listed above:

rrdtool update new_temps.rrd 1312719661:114.8:116.6 1312719841:114.8:116.6

Now the ‘info’ from the RRD:

rrdtool info new_temps.rrd
filename = "new_temps.rrd"
rrd_version = "0003"
step = 300
last_update = 1312719841
ds[temp1].type = "GAUGE"
ds[temp1].minimal_heartbeat = 300
ds[temp1].min = 1.0000000000e+01
ds[temp1].max = 1.9000000000e+02
ds[temp1].last_ds = "114.8"
ds[temp1].value = 2.7552000000e+04
ds[temp1].unknown_sec = 1
ds[temp2].type = "GAUGE"
ds[temp2].minimal_heartbeat = 300
ds[temp2].min = 1.0000000000e+01
ds[temp2].max = 1.9000000000e+02
ds[temp2].last_ds = "116.6"
ds[temp2].value = 2.7984000000e+04
ds[temp2].unknown_sec = 1
rra[0].cf = "AVERAGE"
rra[0].rows = 12
rra[0].cur_row = 11
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[0].cdp_prep[1].value = NaN
rra[0].cdp_prep[1].unknown_datapoints = 0
rra[1].cf = "AVERAGE"
rra[1].rows = 288
rra[1].cur_row = 165
rra[1].pdp_per_row = 1
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = NaN
rra[1].cdp_prep[0].unknown_datapoints = 0
rra[1].cdp_prep[1].value = NaN
rra[1].cdp_prep[1].unknown_datapoints = 0
rra[2].cf = "AVERAGE"
rra[2].rows = 168
rra[2].cur_row = 80
rra[2].pdp_per_row = 12
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = NaN
rra[2].cdp_prep[0].unknown_datapoints = 4
rra[2].cdp_prep[1].value = NaN
rra[2].cdp_prep[1].unknown_datapoints = 4
rra[3].cf = "AVERAGE"
rra[3].rows = 720
rra[3].cur_row = 402
rra[3].pdp_per_row = 12
rra[3].xff = 5.0000000000e-01
rra[3].cdp_prep[0].value = NaN
rra[3].cdp_prep[0].unknown_datapoints = 4
rra[3].cdp_prep[1].value = NaN
rra[3].cdp_prep[1].unknown_datapoints = 4
rra[4].cf = "AVERAGE"
rra[4].rows = 365
rra[4].cur_row = 200
rra[4].pdp_per_row = 288
rra[4].xff = 5.0000000000e-01
rra[4].cdp_prep[0].value = NaN
rra[4].cdp_prep[0].unknown_datapoints = 148
rra[4].cdp_prep[1].value = NaN
rra[4].cdp_prep[1].unknown_datapoints = 148

Reviewing the above all looks well so we proceed with adding all existing data via a simple script:

#!/bin/bash
##### simple RRD update from log file
COUNT=0
TEMP_LOG="new_temps.rrd.last"
for TEMP in $(grep "|" ${TEMPS_LOG} | awk '/:/ {print $(NF-1)}')
do
        CMD="rrdtool update new_temps.rrd ${TEMP}"
        ((COUNT=COUNT+1))
        printf "%4d.  " ${COUNT}
        echo "${CMD}"
        ${CMD}
        if [[ $? -ne 0 ]] ; then printf "\n\tERROR_Exiting... Check Temp: ${TEMP}\n\n" ; exit 1 ; fi
done
#####
printf "\n\tUpdated ${COUNT} records.\n################\n"

If you re-run the script you will get errors since rrdtool EXPECTS updates to provide data captured SINCE_THE_LAST_Update...

1.  rrdtool update new_temps.rrd 1312719661:114.8:116.6
...
3873.  rrdtool update new_temps.rrd 1313424361:116.6:116.6
3874.  rrdtool update new_temps.rrd 1313424541:116.6:116.6
3875.  rrdtool update new_temps.rrd 1313424721:116.6:116.6
3876.  rrdtool update new_temps.rrd 1313424902:114.8:116.6

bash new.update # re-run simple script above

   1.  rrdtool update new_temps.rrd 1312719661:114.8:116.6
ERROR: new_temps.rrd: illegal attempt to update using time 1312719661 when last update time is 1313424902 (minimum one second step)

        ERROR_Exiting... Check Temp: 1312719661:114.8:116.6

Since the new TEMP time value is in the past we get an error…

To create a simple graph from the start to the end of the data set we use something like:

rrdtool graph All_temps.png \
        --start=08/07/2011 \
        --end=08/10/2011 \
        --imgformat=PNG \
        --width=500 \
        --base=1000 \
        --height=120 \
        --interlaced \
        DEF:a=new_temps.rrd:temp1:AVERAGE \
        DEF:b=new_temps.rrd:temp2:AVERAGE \
        LINE1:a#800000:temp1 \
        LINE1:b#00FF00:temp2 \
        COMMENT:"Sun 2011-08-07 00\:00 - Wed 2011-08-10 00\:00"

Since I am working a project with 40+ temperature sensors which, of course, did not ‘go live’ on the same date I am interested in a simple solution for updating RRDs.  The good news (for me) is that the log data includes Unix Epoch-Time…  The bad news – rrdtool expects data from one time slot for all data_sources being updated in the RRD – so data massage may be needed if/when there is ‘missing data’ in a given time slot; also, in this case, only  one time_value  could be selected for each data_pass (i.e. it takes ~60 seconds to read all of the sensors so the data inputs could be adjusted to the time recorded for the first sensor in the set.)

A simple solution to dealing with  RRDs using data from multiple temperature sensors (data sources) which are/were activated at different times or that have data sources where recorded time-stamps vary only slightly might be to create unique RRDs for each temperature sensor.  As you update simply confirm that the update time is ‘newer’ than the ‘last update’ time…  Using a tool like drraw (last updated in 2009) you can create graphs from multiple sensors/RRDs/data sources.  When you create your graphs the ‘time differences’ (in my case, only seconds) will be ‘averaged out’ – this works for my needs but may not be appropriate in other situations.

Some test results for creating RRDs using the approach described above:

Once the RRD files contain all of the ‘historical’ data, using this approach it takes about 40 seconds to update all of the RRDs with the most recent data with a simple time check using ‘rrdtool last RRD_Name‘; if the current data time stamp is greater than the ‘last’ entry then the new data is added.

Topics: Computer Technology, Problem Solving, Unix-Linux-Os | Comments Off on Using RRDs with Many Data Sources (rrdtool)

Comments are closed.


________________________________________________
YOUR GeoIP Data | Ip: 73.21.121.1
Continent: NA | Country Code: US | Country Name: United States
Region: | State/Region Name: | City:
(US only) Area Code: 0 | Postal code/Zip:
Latitude: 38.000000 | Longitude: -97.000000
Note - if using a mobile device your physical location may NOT be accurate...
________________________________________________

Georgia-USA.Com - Web Hosting for Business
____________________________________