Using wget to Interact with Websites

wget is an outstanding software for interacting with websites. I've used wget to automate downloads of backup files from an ISP, as well as to make posts to Twitter.

For your convenience, here are some useful examples of posting to a site requiring sign-in and authentication...

Logging into an HTTPS site with user name and password:


#!/bin/bash
CURRENT_DATE=$(date "+%Y%m%d")

# LOGS INTO SERVER.

wget -O /dev/null -o /dev/null --keep-session-cookies --no-check-certificate --save-cookies cookies.txt --post-data "login=myusername&password=mypassword" https://secure.somewebsite.com/Authentication/Account/Index.rails

# NAVIGATES TO A PAGE WITH A LINK TO A DATABASE BACKUP.
wget -O /dev/null -o /dev/null --keep-session-cookies --no-check-certificate --load-cookies cookies.txt --save-cookies cookies.txt https://secure.somewebsite.com/HostingManager/Databases/BackupFiles.rails?databaseId=xxxxxxxx-yyyy-zzzz-1111-222222222222

# DOES A GET REQUEST ON THE PAGE.
wget -O /dev/null -o /dev/null --keep-session-cookies --no-check-certificate --load-cookies cookies.txt --post-data "databaseId=xxxxxxxx-yyyy-zzzz-1111-222222222222&fileName=my_database_name_${CURRENT_DATE}.BAK" https://secure.somewebsite.com/HostingManager/Databases/CopyBackup.rails

# DOES AN FTP REQUEST.
wget ftp://username:password@ftp.someftpsite.com/database/my_database_name_$CURRENT_DATE.BAK*






Now, here is an example of doing an update to Twitter (based on information from an excellent article by Mark Gibbs at http://www.networkworld.com/columnists/2008/052108-gearhead.html):


#!/bin/bash
CURRENT_DATE=$(date "+%A %Y%m%d %H:%M:%S")
UTC_DATE=$(date -u "+%A %Y%m%d %H:%M:%S")

wget -O /dev/null -o /dev/null --http-user=myusername --http-password=mypassword --post-data "status=Hello World using wget! $UTC_DATE UTC / $CURRENT_DATE LOCAL" http://twitter.com:80/statuses/update.xml