Last year July I wrote about how I connected some internet services to help my dad make it easier to get access to his electronic news paper and magazine subscriptions. Over that last year his memory has been slowly getting worse making it even harder for him to remember making very minor computer related hiccup an unsurpassable hurdle for him.

Over the last 12 months the process detailed in the post ‘Helping out the old man’ has been working fine for most of the time. Occasionally I had a re-authorize IFTTT again for Google Mail and Dropbox, but nothing serious. But for the last couple weeks things started to break down and I haven;t been able to fix it using the services used (IFTT combined with Google Mail and Dropbox). For some reason saving attachments stopped working, by either not saving anything at all or saving and image instead of the attachment.

After a couple of days (not full time of course) trying to fix the issue I just gave it up and wrote a Python script to replace the task handled by IFTTT. The script turned even turned out to work much better than IFTTT recipe ever worked.

In principle the script performs the following tasks;

1. Login to Google Mail and search for recent emails from a specific domain where the mails have at least one attachment using a specific extension
2. Save all of the attachments with the specific extension to a folder
3. In the case an attachment is broken (i.e. 0 bytes) the body email is parsed to find a URL to download that newspaper or magazine
4. Send notification to my iOS device containing the logging information

The notifications are sent using Pushover.net. You can also read about Pushover in some of my other posts on how set it up.

Obviously since I am not using IFTTT anymore I need to make sure that the script is being executed from time to time so his email is being checked and attachments are being saved. I could make sure one of my computers at home is taking care of this task, but since I also have a Synology DiskStation running which is capable of a lot more that just provide storage, I decided to give my DiskStation some additional work.

I have added a job named ‘Anderslezen’ and scheduled it to run daily every 30 minutes. Pay close attention to the settings for ‘First run time’ and ‘Last run time’, otherwise you run the risk that your job still only runs once a day. To keep myself informed about any technical issues I will receive an email when the script exits abnormally.

General

Schedule

Task Settings

The DiskStation (or more precise DSM) comes with a task scheduler aptly named ‘Task Scheduler’ which amongst others allows you to create a job based on a user-defined script, in my case the Python script I have written.

Since the files download by the script need to be sent to my dads Dropbox-folder I also installed the application ‘Cloud Sync’ which allows me to have a synched copy of his Dropbox-folder on my DiskStation. So uploading the files to his Dropbox-folder is just a matter of making sure the Python script is writing to the proper folder on my DiskStation.

The Python script

Of you are in need of something similar feel free to use the code below and adjust it to match your specific needs. The table below notes the line numbers that need to be adjusted for your situation

1

import email
import string
import imaplib
import os
import urllib
import re
import datetime
import locale
import socket

#
# Logging related functions/procedures
#
logTable = []
def addLogging( msg ):
    global logTable

    print( msg )
    logTable.append( msg )

    return

#
# Send pushover notification
#
def sendNotification( msg ):
    import httplib, urllib
    conn = httplib.HTTPSConnection( "api.pushover.net:443" )
    conn.request( 'POST'
                , '/1/messages.json'
                , urllib.urlencode( { 'token'   : '<Your API token>'
                                    , 'user'    : '<Your user key>'
                                    , 'message' : msg
                                    , 'sound'   : 'incoming'
                                    }
                                  )
                , { "Content-type": "application/x-www-form-urlencoded" }
                )
    conn.getresponse()

#
# Use Dutch locale
#
locale.setlocale( locale.LC_TIME, "nl_NL.UTF-8" )

#
# Configuration
#
userName     = '<Your Google Mail account>'
passwd       = '<Your Google Mail password>'
gmail_filter = 'from:passendlezen.nl has:attachment filename:exd newer_than:3d'

#
# Determine directory and folder to be used to store files
#
dir_name = 'anderslezen'
detach_dir = '/volume1/Data_vol/Dropbox/dropbox-user/'

#
# Create container folder if necessary
#
if dir_name not in os.listdir( detach_dir ):
    os.mkdir( dir_name )

#
# Parse the inbox and download attachments
#
try:
    socket.setdefaulttimeout( 10 )
    imapSession = imaplib.IMAP4_SSL( 'imap.gmail.com' )
    typ, accountDetails = imapSession.login( userName
                                           , passwd
                                           )

    if typ != 'OK':
        addLogging( 'Not able to sign in!' )
        raise

    imapSession.select( 'INBOX' )
    typ, data = imapSession.search( None
                                  , 'X-GM-RAW'
                                  , gmail_filter
                                  )

    if typ != 'OK':
        addLogging( 'Error searching Inbox.' )
        raise

    # Iterating over all emails
    for msgId in data[0].split():
        typ, messageParts = imapSession.fetch( msgId
                                             , '(RFC822)'
                                             )

        if typ != 'OK':
            addLogging( 'Error fetching mail.' )
            raise

        emailBody = messageParts[0][1]
        mail = email.message_from_string( emailBody )
        mail_date = mail['Date']
        mail_subject = mail['Subject']

        addLogging( '%s -> %s' % ( mail_date
                                 , mail_subject
                                 )
                  )

        for part in mail.walk():
            # Skip multipart
            if part.get_content_maintype() == 'multipart':
                # print part.as_string()
                continue

            # Skip parts without a content disposition
            if part.get( 'Content-Disposition' ) is None:
                # print part.as_string()
                continue

            #
            # Get filename for the content part and skip it when it does not have the suffix .exd
            #
            fileName = part.get( 'Content-Disposition' )
            fileName = re.findall( 'filename="(.+)"', fileName )[0]

            if not fileName.endswith( '.exd' ):
                continue

            #
            # Replace some invalid filename characters
            #
            dict = string.maketrans( '<>:"/\|?*'
                                   , '  -      '
                                   )
            fileName = fileName.translate( dict )

            # Remove (x) from the filename if pressent
            fileName = re.sub( '\(.*?\)' , '', fileName )

            # Replace underscores (_) by spaces ( )
            fileName = fileName.replace( '_', ' ' )

            # Split filename and extension
            parts = fileName.split( '.' )

            # Split filename into papername and date
            parts = parts[0].split( '-' )

            # Transform date
            datum = datetime.datetime.strptime( parts[1]
                                              , '%y%m%d' ).strftime( '%A %d %B %Y' )

            # Determine the filename to be used
            fileName = parts[0]+', '+datum+'.exd'

            if bool(fileName):
              # Determine filename to be used
                filePath = os.path.join( detach_dir
                                       , dir_name
                                       , fileName
                                       )
 
                # If the file does not already exists on disk, either save attachment or download from url
                if not os.path.isfile( filePath ) :
                    size = len( str( part ) ) / 1024.0

                    if size > 1:
                        # Output information
                        addLogging('   Saving attachment to \'%s\' (%.2f KB)' % ( fileName
                                                                                , size
                                                                                )
                             )

                        # Save attachment to disk
                        fp = open( filePath
                                 , 'wb'
                                 )
                        fp.write( part.get_payload( decode = True ) )
                        fp.close()

                    else:
                        # Get URL from mail body
                        fileURL = re.search( '(?P<url>https?://[^\s]+)'
                                           , emailBody ).group( 'url' )

                        # Download file from URL
                        site = urllib.urlopen( fileURL )
                        data = site.read()
                        meta = site.info()
                        size = int( meta.getheaders( 'Content-Length' )[0] ) / 1024.0

                        # Output information
                        addLogging('   Download from URL to \'%s\' (%.2f KB)' % ( fileName
                                                                                , size
                                                                                )
                             )

                        # Write data to disk
                        fp = open( filePath
                                 , 'wb'
                                 )
                        fp.write( data )
                        fp.close()

    # Send notification
    sendNotification( "\n".join( logTable ) )

    imapSession.close()
    imapSession.logout()

except Exception as e:
    sendNotification( str( e ) )
    addLogging( str( e ) )