«

»

Project Automation – Website Development

When working on a website project, often you’ll find the need to synchronize your local source code to your development server, i.e. after making several changes for a new feature. The straightforward and common method of doing this is with your favorite FTP client (like FileZilla).

Manually uploading changed files gets tedious pretty quickly, especially when you’re making lots of incremental changes. A much more efficient method is to have a shell script that uploads the modified files for you; taking that one step further, it’d be nice to have it automatically execute whenever a file is modified, to provide near real-time updates.

Using bash and python, this is relatively easy to implement.

First things first: you’ll need to install watchdog. On debian-based systems (like Ubuntu), this can be done by running pip install watchdog (More on what this is used for later).

The first problem to tackle is figuring out how to easily retrieve a list of files that have been modified or created recently (for example, in the past 10 minutes).
lsmodifiedmins.sh is a shellscript wrapper for the linux find command, as well as performing some grep operations and stripping off the leading “./” from the filenames (using awk).

lsmodifiedmins.sh - list all files modified within the past N minutes.

#!/bin/bash
#lists files modified within the past N minutes
mins=$1
find . -mmin -$mins -type f -print | awk '{ print substr($1, 3) }'

This script could be more robust, and include checking to make sure the correct amount of parameters are passed, support calling the script with no parameters, etc.


upload-changes.sh - use scp to transfer all files modified in the past N minutes

#!/bin/bash
#by default, show files modified in the past pastMinutes minutes
pastMinutes=30
#remote host to upload to
remoteHost=myhost
#root project path on remote host (note the ending slash)
remoteHostProjectRootPath=/home/patrick/projects/myproject/
#file types regex
fileTypesRegEx="(css|js|php)"

#accept a single argument to modify pastMinutes
if [ $# -gt 0 ]; then
  pastMinutes=$1
fi

filecnt=$(lsmodifiedmins.sh $pastMinutes | grep -v upload-changes | egrep "$fileTypesRegEx" | wc -l)
scpcmds=$(lsmodifiedmins.sh $pastMinutes | grep -v upload-changes | egrep "$fileTypesRegEx" | awk '{ print "scp " $1 " $remoteHost:$remoteHostProjectRootPath" $1 ";\n" }')

echo "Uploading *.css,*.js,*.php files changed in the past $pastMinutes minutes...($filecnt files)"
sh -c "$scpcmds"
echo "Finished."

Don’t forget to manually configure the remoteHost, remoteHostProjectRootPath, and the fileTypesRegEx variables


Now that we have a script that transfers files automatically, we need to make it execute automcaticlly whenever certain file types in the project folder are modified.

watch_for_changes.py - monitor file changes/creations and execute our upload-changes.sh script.

#!/usr/bin/python
import time
import sys
import os
from watchdog.observers import Observer  
from watchdog.events import PatternMatchingEventHandler  

class WebsiteFilesChangeHandler(PatternMatchingEventHandler):
  patterns = ["*.css", "*.js", "*.php"]

  def process(self, event):
      """
      event.event_type 
          'modified' | 'created' | 'moved' | 'deleted'
      event.is_directory
          True | False
      event.src_path
          path/to/observed/file
      """
      # the file will be processed there
      print event.src_path, event.event_type  # print now only for degug
      if event.event_type=='modified':
          time.sleep(2)
          #run `upload-changes 2` for files changed in past 2 minutes only
          #must use spawnv, as execv halts script execution after the first network 
          #connection is made.
          os.spawnv(os.P_WAIT, './upload-changes', ("./upload-changes","2"))
          #initObserver(sys.argv[1:][0])
          
  def on_modified(self, event):
      self.process(event)

  def on_created(self, event):
      self.process(event)

def initObserver(p):
    observer = Observer()
    #set the named-argument "recursive" to True for observer.schedule, to watch in subfolders.
    observer.schedule(WebsiteFilesChangeHandler(), path=p if p else '.', recursive=True)
    observer.start()
    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()

    observer.join()
    
if __name__ == '__main__':
    print "On any file changes, upload-changes is run to synchronize with the webhost."
    print "Monitoring for file changes...\n"

    args = sys.argv[1:]
    initObserver(args[0])

Finally, run watch_for_changes.py myproject1/ and the server will start up, notifying you any time any of the files in myproject1/ or its subdirectories is modified or created.
For the provided script, it monitors changes only to .css, .js, and .php files, but can be modified to suit your needs.


Hopefully, this set of scripts will make your coding process much easier.

Permanent link to this article: http://permafrostcodingstudio.com/articles/project-automation-website-development/

Leave a Reply

Your email address will not be published. Required fields are marked *