2009-11-10

Bash Script to Web Scrape Forex Data

I like to track the value of the Loonie in US dollars; ie, CAD to USD. I often thought to myself that it would be fantastic to have a software application that can email me foreign exchange rate information on a periodic basis.

Using a script employing bash, grep, awk, and sed, this is possible, especially if you run the script as a cron job. This script does a web scrape of data from fxware.com.

The script fxgrab provides a convenient way to send an email notification of foreign exchange rate data via a bash script in a POSIX-like environment (such as in Cygwin or Linux). Feel free to use the code listed below for your own applications/utilities. The variable FROMCURR represents the currency converted to another currency (TOCURR). Enjoy!


#!/bin/bash


# fxgrab
# Copyright (c) 2009 Mike Quentel

# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation
# files (the "Software"), to deal in the Software without
# restriction, including without limitation the rights to use,
# copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following
# conditions:

# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.

# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.

# CONFIGURATION -- REPLACE THE FOLLOWING VARIABLES TO RUN THIS:
# FROMCURR = THE CURRENCY YOU ARE CONVERTING FROM.
# TOCURR = THE CURRENCY YOU ARE CONVERTING TO.
# FROM = WHO THE EMAIL IS FROM.
# SUBJECT = EMAIL SUBJECT.
# SMPTPSERVER = THE SERVER USED TO SEND THE EMAIL.
# PORT = EMAIL PORT.
# USER = EMAIL USER.
# PASSWD = EMAIL PASSWORD.
# RECIPIENT = EMAIL ADDRESS OF MESSAGE RECIPIENT.


CURRENT_DATE=$(date "+%A %Y%m%d %H:%M:%S")
FROMCURR="CAD"
TOCURR="USD"
FROM="replytothisaccount@someserver.com"
SUBJECT="FOREX NOTIFICATION"
SMTPSERVER="smtp.myisp.com"
PORT="587"
USER="myispuseraccount"
PASSWD="myisppassword"
RECIPIENT="somerecipient@someaddress.com"

cd /var/tmp
wget --post-data "f=$FROMCURR&t=$TOCURR" -o /dev/null -O forex.asp http://www.fxware.com/forex-currency/forex.asp
FX=$(grep -oE toa.*[0-9] forex.asp | awk '{print $3}' | sed -e 's/16px;">//g')
MESSAGE="Foreign exchange data accessed at $CURRENT_DATE on this server. 1 $FROMCURR = $FX $TOCURR"
echo $MESSAGE | email -f "$FROM" -s "$SUBJECT" -r "$SMTPSERVER" -p $PORT -u "$USER" -i "$PASSWD" -m login $RECIPIENT
rm -f forex.asp

2009-11-07

Bash Script Using wget to Update Twitter

The script BWTU (Bash-Wget-Twitter-Updater) provides a convenient way to update Twitter from a command line (such as in Cygwin or Linux). Feel free to use the code below for your own applications/utilities. To put more content in the Twitter updates, you can omit the CURRENT_DATE and UTC_DATE entries which are tagged onto every update by default in this blog post listing. Enjoy!

#!/bin/bash



# BWTU (Bash-Wget-Twitter-Updater)
# Copyright (c) 2009 Mike Quentel

# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation
# files (the "Software"), to deal in the Software without
# restriction, including without limitation the rights to use,
# copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following
# conditions:

# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.

# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.




# DATE AND TIME.
CURRENT_DATE=$(date "+%A %Y%m%d %H:%M:%S")
UTC_DATE=$(date -u "+%A %Y%m%d %H:%M:%S")

# CD TO /var/tmp
cd /var/tmp

# READ IN LOGIN INFO.
echo -n "username: "
read -e USER
echo -n "password: "
stty -echo
read -e PASSWD
stty echo
echo 

# WHILE LOOP -- CTRL+C TO BREAK.
echo "Enter ctrl+c to exit..."

while true; do

  # READ IN STATUS COMMENT.
  echo 
  echo -n "Enter status update to Twitter: "
  read -e STATUS

  # DO wget OF STATUS UPDATE TO TWITTER.
  wget -O /dev/null -o /dev/null --http-user=$USER \
  --http-password=$PASSWD --post-data "status=$STATUS \
  -- $UTC_DATE UTC /  $CURRENT_DATE LOCAL" \
  http://twitter.com:80/statuses/update.xml

done

Using wget to Interact with Websites

wget is an outstanding software for interacting with websites. I've used wget to automate downloads of backup files from an ISP, as well as to make posts to Twitter.

For your convenience, here are some useful examples of posting to a site requiring sign-in and authentication...

Logging into an HTTPS site with user name and password:


#!/bin/bash
CURRENT_DATE=$(date "+%Y%m%d")


# LOGS INTO SERVER.

wget -O /dev/null -o /dev/null --keep-session-cookies --no-check-certificate --save-cookies cookies.txt --post-data "login=myusername&password=mypassword" https://secure.somewebsite.com/Authentication/Account/Index.rails


# NAVIGATES TO A PAGE WITH A LINK TO A DATABASE BACKUP.
wget -O /dev/null -o /dev/null --keep-session-cookies --no-check-certificate --load-cookies cookies.txt --save-cookies cookies.txt https://secure.somewebsite.com/HostingManager/Databases/BackupFiles.rails?databaseId=xxxxxxxx-yyyy-zzzz-1111-222222222222


# DOES A GET REQUEST ON THE PAGE.
wget -O /dev/null -o /dev/null --keep-session-cookies --no-check-certificate --load-cookies cookies.txt --post-data "databaseId=xxxxxxxx-yyyy-zzzz-1111-222222222222&fileName=my_database_name_${CURRENT_DATE}.BAK" https://secure.somewebsite.com/HostingManager/Databases/CopyBackup.rails


# DOES AN FTP REQUEST.
wget ftp://username:password@ftp.someftpsite.com/database/my_database_name_$CURRENT_DATE.BAK*







Now, here is an example of doing an update to Twitter (based on information from an excellent article by Mark Gibbs at http://www.networkworld.com/columnists/2008/052108-gearhead.html):


#!/bin/bash
CURRENT_DATE=$(date "+%A %Y%m%d %H:%M:%S")
UTC_DATE=$(date -u "+%A %Y%m%d %H:%M:%S")


wget -O /dev/null -o /dev/null --http-user=myusername --http-password=mypassword --post-data "status=Hello World using wget! $UTC_DATE UTC / $CURRENT_DATE LOCAL" http://twitter.com:80/statuses/update.xml