Blah blah!? - IT-Keep track of the network traffic

Small script to keep an eye on server traffic (05.Apr.2010)

Introduction
Script
How it works
How to use it

Introduction

The site you're seeing right now has a maximum amount of monthly traffic included in the basic subscription. If I transmit or receive more than that amount, I have to pay more.
As I didn't have a clue about how far I was from that limit I tried to find a program which keeps track of these numbers.

Unluckily - perhaps because I just didn't search enough - I didn't find anything that matched my requirements. So without almost noticing I started to write my own script.
It was my first PHP-script, and as at the beginning I didn't think it would end up being so complicated I wrote it directly in a tiny terminal, so the end result is a bit messy and probably not well structured.
But in the end it looks like it's working :o)

The requirements of this traffic-usage-tracking-script were:

keep track of all traffic generated by a specific interface - not just e.g. the one generated by Apache or by SSH.
survive server reboots. If I reboot the server or restart the network interface the counters shouldn't reset.
do not be invasive - do not slow down the network interface or use a lot of CPU.
aggregate on a monthly basis.

Here is the script which came out.

Script

<?php

//THIS SCRIPT HAS TO RUN AS OFTEN AS POSSIBLE
//AS DATA BETWEEN THE LAST UPDATE AND THE NEXT RUN IS LOST IF THE SCRIPT IS KILLED

//START-SET THESE VALUES AS YOU WISH
$transfer_db = "/var/log/check_rx_and_tx/myBwidth_db";

//START-Tables
$transfer_tbl = "bwidth_log";
$tbl_monthly = "bwidth_monthly_log";
$tbl_monthly_total = "bwidth_monthly_total";
$tbl_monthly_total_start = "bwidth_monthly_total_start";
//END-Tables

//START-Files
$temp_file1 = "/var/log/check_rx_and_tx/myBwidth_db_temp1fullres.txt";
$temp_file2 = "/var/log/check_rx_and_tx/myBwidth_db_temp2linestart.txt";
$temp_file3 = "/var/log/check_rx_and_tx/myBwidth_db_temp3rxtxline.txt";
//END-Files

//START-Various
$monthlyTrafficTotalMAX = 500 * 1024 * 1024 * 1024; //For max 500GB monthly traffic
$ifconfig_if2search="eth0";
$keep_logs_days=365;
//END-SET THESE VALUES AS YOU WISH

$prct_warning_level = 10;
$curYear = date("Y");
$curMonth = date("m");
$daysInCurMonth = cal_days_in_month(CAL_GREGORIAN, $curMonth, $curYear);
$curDay = date("j");
$curMonthTotalRX = 0;
$curMonthTotalStartRX = 0;
$curMonthTotalTX = 0;
$curMonthTotalStartTX = 0;
$monthlyTrafficTotalNowMIGHT = 0;
$monthlyTrafficTotalNowIS = 0;
$monthlyTrafficTotalStart = 0;
$warning = "N";
//END-Various


//Output the details of all interfaces.
`/sbin/ifconfig > $temp_file1`;


//Get the line nbr where the interface we're searching for outputs its data.
//This is the start of the search.
`grep -n $ifconfig_if2search $temp_file1 > $temp_file2`;
$ifconfig_result_if_start = `awk -F: '{print $1}' $temp_file2`;
settype($ifconfig_result_if_start, "integer"); //I need this otherwise the var is a string with line end.
//echo "Found interface $ifconfig_if2search on line " . $ifconfig_result_if_start . "\n";


//Get for that interface the line which contains the RX and TX data.
`tail -n +$ifconfig_result_if_start $temp_file1 > $temp_file3`;
$ifconfig_result_line = `grep -m $ifconfig_result_if_start "RX bytes" $temp_file3`;
//echo "Line containing data: " . $ifconfig_result_line . "\n";


//Get the RX bytes.
$if_start_RX = strpos($ifconfig_result_line, ":") + 1; //+1 because I don't want the ":".
$if_end_RX = strpos($ifconfig_result_line, " (");
$if_RX = substr($ifconfig_result_line, $if_start_RX, $if_end_RX - $if_start_RX);
//echo "Substring RX: " . $if_RX . "end\n";


//Get the TX files.
$if_start_TX = strpos($ifconfig_result_line, ":", $if_start_RX) + 1;
$if_end_TX = strpos($ifconfig_result_line, " (", $if_end_RX + 1);
$if_TX = substr($ifconfig_result_line, $if_start_TX, $if_end_TX - $if_start_TX);
//echo "Substring TX: " . $if_TX . "end\n";



if ($db = new SQLiteDatabase($transfer_db))
{
        //START-Detailed log table.
        $q = false;
        $q = @$db->query("SELECT 'x' FROM $transfer_tbl");
        if ($q == false)
        {
                $db->queryExec("CREATE TABLE $transfer_tbl (if varchar2(20), dt datetime, in_bytes number, out_bytes number);");
                $db->queryExec("CREATE unique index $transfer_tbl" . "_idx " . "on $transfer_tbl (if, dt);");
        }
        //Insert the new value into the log table.
        $db->queryExec("insert into $transfer_tbl values('$ifconfig_if2search', datetime(strftime('%s','now'), 'unixepoch'), $if_RX, $if_TX);");
        //Delete from the log table all old values.
        $db->queryExec("delete from $transfer_tbl where if = '$ifconfig_if2search' and dt <= datetime(strftime('%s', 'now') - (60*60*24*$keep_logs_days), 'unixepoch', 'localtime')");
        //END-Detailed log table.


        //START-Monthly log table
        //START-Create table.
        $q = @$db->query("SELECT 'x' FROM $tbl_monthly");
        if ($q == false)
        {
                $db->queryExec("CREATE TABLE $tbl_monthly (if varchar2(20), year number, month number, seq number, in_bytes number, out_bytes number);");
                $db->queryExec("CREATE unique index $tbl_monthly" . "_idx " . "on $tbl_monthly (if, year, month, seq);");
        }
        //END-Create table.

        //START-Check if we already have a value for the current month.
        $q = @$db->query("SELECT 'x' FROM $tbl_monthly where if = '$ifconfig_if2search' and year = $curYear and month = $curMonth order by seq desc limit 1;");
        $monthEntryExists = "";
        $monthEntryExists = $q->fetchSingle();
        //If for any reason the values of the beginning of the current month are not there, insert them.
        if ($monthEntryExists != "x")
        {
                $db->queryExec("insert into $tbl_monthly values('$ifconfig_if2search', $curYear, $curMonth, 0, $if_RX, $if_TX);");
        }
        //END-Check if we already have a value for the current month.
        else
        {
                //WHAT DO I DO IF THE SERVER IS RESTARTED IN THE MIDDLE OF THE MONTH AND THE VALUES OF IFCONFIG ARE RESET TO 0?
                //OR IF THE INTERFACE IS RESTARTED WITHOUT THAT THE SCRIPT IS AS WELL RESTARTED?
                //=> In every case, ifconfig will restart from 0.
                //=> This means that I need multiple entries for a single month, one additional entry each time that the iface has a number smaller than
                //the latest entry => to know the total amount I have to sum them up grouping by month minus the first entry of the month.
                $q = @$db->query("SELECT in_bytes FROM $tbl_monthly where if = '$ifconfig_if2search' and year = $curYear and month = $curMonth order by seq desc limit 1;");
                $currMonthLastRX = $q->fetchSingle();
                settype($currMonthLastRX, "float");
                $q = @$db->query("SELECT max(seq) FROM $tbl_monthly where if = '$ifconfig_if2search' and year = $curYear and month = $curMonth;");
                $currMonthLastSeq = $q->fetchSingle();
                settype($currMonthLastSeq, "float");
                //echo "Last RX of the current month: " . $currMonthLastRX . "\n";
                //echo "Last seq of the current month: " . $currMonthLastSeq . "\n";
                //START-If the current RX-bytes are less than the previous ones the iface has been restarted for some reason.
                if ($if_RX < $currMonthLastRX)
                {
                        //If the values of the iface were zeroed for some reason, add a new line with a higher sequence number.
                        $db->queryExec("insert into $tbl_monthly values('$ifconfig_if2search', $curYear, $curMonth, $currMonthLastSeq+1, $if_RX, $if_TX);");
                }
                else
                {
                        //If the iface is still up and running just update the values of the last entry.
                        $db->queryExec("update $tbl_monthly set in_bytes = $if_RX, out_bytes = $if_TX where if = '$ifconfig_if2search' and year = $curYear and month = $curMonth and seq = $currMonthLastSeq;");
                }
                //END-If the current RX-bytes are less than the previous ones the iface has been restarted for some reason.
        }
        //END-Monthly log table

        //START-Monthly total table
        //START-Create table.
        $q = @$db->query("SELECT 'x' FROM $tbl_monthly_total");
        if ($q == false)
        {
                $db->queryExec("CREATE TABLE $tbl_monthly_total (if varchar2(20), year number, month number, in_bytes number, out_bytes number, tot_bytes, tot_bytes_max_now number, tot_gb number, tot_gb_max_now number, warning varchar2(1));");
                $db->queryExec("CREATE unique index $tbl_monthly_total" . "_idx " . "on $tbl_monthly_total (if, year, month);");
        }
        $q = @$db->query("SELECT 'x' FROM $tbl_monthly_total_start");
        if ($q == false)
        {
                $db->queryExec("CREATE TABLE $tbl_monthly_total_start (if varchar2(20), year number, month number, in_bytes number, out_bytes number, tot_bytes);");
                $db->queryExec("CREATE unique index $tbl_monthly_total_start" . "_idx " . "on $tbl_monthly_total (if, year, month);");
        }
        //END-Create table.

        //START-Compute the totals
        $q = @$db->query("select sum(in_bytes) from $tbl_monthly where if = '$ifconfig_if2search' and year = $curYear and month = $curMonth;");
        $curMonthTotalRX = $q->fetchSingle();
        settype($curMonthTotalRX, "float");
        $q = @$db->query("select sum(out_bytes) from $tbl_monthly where if = '$ifconfig_if2search' and year = $curYear and month = $curMonth;");
        $curMonthTotalTX = $q->fetchSingle();
        settype($curMonthTotalTX, "float");
        //END-Compute the totals

        //START-Compute what I reached and the limit I can reach on this day of the month (simple average)
        $monthlyTrafficTotalNowMIGHT = 0.0;
        $monthlyTrafficTotalNowIS = 0.0;
        settype($monthlyTrafficTotalNowMIGHT, "float");
        settype($monthlyTrafficTotalNowIS, "float");
        $monthlyTrafficTotalNowMIGHT = $monthlyTrafficTotalMAX / $daysInCurMonth * $curDay;
        $monthlyTrafficTotalNowIS = $curMonthTotalRX + $curMonthTotalTX;
        //END-Compute what I reached and the limit I can reach on this day of the month (simple average)

        //START-Check if I already have an entry for this month.
        $q = @$db->query("SELECT 'x' FROM $tbl_monthly_total where if = '$ifconfig_if2search' and year = $curYear and month = $curMonth;");
        $monthEntryTotalExists = "";
        $monthEntryTotalExists = $q->fetchSingle();
        //START-Check if I already have an entry for this month.

        //START-Insert or update the informations into the table
        if ($monthEntryTotalExists != "x")
        {
                //If the first entry for the current month does not exist, insert it.
                $db->queryExec("insert into $tbl_monthly_total values('$ifconfig_if2search', $curYear, $curMonth, $curMonthTotalRX, $curMonthTotalTX, $monthlyTrafficTotalNowIS, $monthlyTrafficTotalNowMIGHT, $monthlyTrafficTotalNowIS/1024.0/1024.0/1024.0, $monthlyTrafficTotalNowMIGHT/1024.0/1024.0/1024.0, 'N');");
                //echo "Inserting into tbl_monthly_total first entry RX" . $curMonthTotalRX . " TX " . $curMonthTotalTX;

                //As it is the first entry of the month, I insert this as well into this table - I will have to detract it from the grand total...
                //...to know the status of the month itself
                $db->queryExec("insert into $tbl_monthly_total_start values('$ifconfig_if2search', $curYear, $curMonth, $curMonthTotalRX, $curMonthTotalTX, $monthlyTrafficTotalNowIS);");
                //echo "Inserting into tbl_monthly_total_start first entry RX" . $curMonthTotalRX . " TX " . $curMonthTotalTX;
        }
        else
        {
                //START-Get the 1st value of the month - I have to detract this to know the current status
                $q = @$db->query("select in_bytes from $tbl_monthly_total_start where if = '$ifconfig_if2search' and year = $curYear and month = $curMonth;");
                $curMonthTotalStartRX = $q->fetchSingle();
                settype($curMonthTotalStartRX, "float");
                $q = @$db->query("select out_bytes from $tbl_monthly_total_start where if = '$ifconfig_if2search' and year = $curYear and month = $curMonth;");
                $curMonthTotalStartTX = $q->fetchSingle();
                settype($curMonthTotalStartTX, "float");
                $q = @$db->query("select tot_bytes from $tbl_monthly_total_start where if = '$ifconfig_if2search' and year = $curYear and month = $curMonth;");
                $monthlyTrafficTotalStart = $q->fetchSingle();
                settype($monthlyTrafficTotalStart, "float");
                //START-Get the 1st value of the month - I have to detract this to know the current status

                $monthlyTrafficTotalNowIS = $monthlyTrafficTotalNowIS - $monthlyTrafficTotalStart;
                $curMonthTotalRX = $curMonthTotalRX - $curMonthTotalStartRX;
                $curMonthTotalTX = $curMonthTotalTX - $curMonthTotalStartTX;

                //START-There is a warning if the current transfer total is higher than the one i am allowded to have on this day of the month.
                if ($monthlyTrafficTotalNowIS > $monthlyTrafficTotalNowMIGHT)
                { $warning = "Y"; }
                else
                { $warning = "N"; }
                //END-There is a warning if the current transfer total is higher than the one i am allowded to have on this day of the month.

                $db->queryExec("update $tbl_monthly_total set in_bytes = $curMonthTotalRX, out_bytes = $curMonthTotalTX, tot_bytes = $monthlyTrafficTotalNowIS, tot_bytes_max_now = $monthlyTrafficTotalNowMIGHT, tot_gb = $monthlyTrafficTotalNowIS/1024.0/1024.0/1024.0, tot_gb_max_now = $monthlyTrafficTotalNowMIGHT/1024.0/1024.0/1024.0, warning = '$warning' where if = '$ifconfig_if2search' and year = $curYear and month = $curMonth;");
                //echo "Updating tbl_monthly_total RX" . $curMonthTotalRX . " TX " . $curMonthTotalTX;
        }
        //END-Insert or update the informations into the table

        //END-Monthly total table
}
else
{
        syslog(LOG_INFO, "ALERT-check_rx_and_tx_code failed. Reason 1");
        die($err);
}
?>

You can download the script from here.

How it works

As I said - the script is quite ugly :o).
Some lines are weird as I had some problems executing some external commands and converting strings to numbers.

Anyway, if you want to use it you should create somewhere a directory where the script will put its own files and set the name of that directory in the variable "$transfer_db".
It will create in there 3 intermediate files and a sqlite-database where it will store all intermediate results and the final monthly aggregation.

You should run the script often (e.g. using a cronjob, or just a continuous loop with a "sleep" event in between), as if anything happens between the previous and the new update that data won't be taken into account. In my case I let the script run once every 10 minutes => it's impossible that I have a huge volume increase in just a 10-minutes timespan, therefore losing the volume of that timespan from time to time is still acceptable.

What the script does in the end is to have a look at the figures reported by the command "ifconfig", save them in the database and aggregate their difference.
If it sees that the RX- and TX-numbers reported by "ifconfig" are smaller than the ones it saw previously, it means that the network interface was probably somehow reset (e.g. reboot?) but it will continue to aggregate the difference starting with the new numbers.

I am running this script since ~1 month and until now it seems that the results are correct.

How to use it

Have a look at the top of the script and change the variables according to your needs.
Change especially the values of "$transfer_db" and "$temp_file[123]" setting the directory and filenames you wish, then set for "$monthlyTrafficTotalMAX" your the maximum monthly traffic/bandwidth limit (in bytes) that your provider allows you to generate and finally the name of the network interface you want to monitor ("$ifconfig_if2search").

Once you've done that, run once the script with "/usr/bin/php /<your_directory>/check_rx_and_tx_code".
You should see that the 4 files were created in the directory you set.

Then generate some traffic (transfer some files or whatever) and run the script again.
To check the results go to the directory where the database is located and access it with the usual sqlite commands ("sqlite myBwidth_db").
If you list all tables (".tab") you'll see that there are 4 tables. The one you're interested in is called "bwidth_monthly_total".

Show the column names of the table (".headers on") and issue e.g. the command...

select year, month, in_bytes / 1024 / 1024 "mb_in", out_bytes / 1024 / 1024 "mb_out", tot_gb, tot_gb_max_now, warning from bwidth_monthly_total;

Here an example of what should appear:

year	month	mb_in	mb_out	tot_gb	tot_gb_max_now	warning
2010	03	114.819579124451	111.462564468384	0.220978655852377	500	N
2010	04	482.94416809082	131.63143157959	0.600171484053135	83.3333333333023	N

Today, being April 5th, having started the script for the first time in March, I get the two lines above.
The first one shows me that in the previous month I read 114MB and sent 111MB, which makes a total of 0.22GB and as the end of the month was reached I had as well the potential to reach as well my maximum volume which is 500GB per month.
The second line shows that until today my network interface got 482MB and sent 131MB, which makes a total of 0.60GB and that today's maximum critical amount would have been about 83GB (500GB monthly divided by 30 days multiplied by 5) => if it goes on like this until the end of the month I will therefore be well below the limit of 500GB.

If you want to show all columns and don't know SQL, the command is "select * from bwidth_monthly_total". For more complicated stuff please refer to some SQL-reference.

To exit from "sqlplus" press [CTRL+D].

If you run the script every 10 minutes, the database size will grow by ~200KB per month.
I think I included in the script a line which deletes old stuff (here set to delete anything older than 365 days), but as I didn't test it and as normally stuff that I don't test doesn't work, it's most probably not working :oP

Hope it's useful!!
It's very raw, but it should be easy to customize to meet your needs (e.g. write more "syslog"-lines to alert you automatically if the limit has been reached).