Tag Archives: webfaction

Hosting Meteor with MongoDB on Webfaction

Meteor is constantly evolving, and so are the ways to host it. I couldn’t find any single source that explained how to deploy a Meteor site to Webfaction using the latest technology (ie. as of March 2014). So here is how I did it for my site, Signup Zone.

First, follow Webfaction’s instructions to install a MongoDB app and add a data directory. I have called the app mongodb1. Note down its port number, eg. 18000.

The instructions say to add a user with the ["userAdminAnyDatabase"] role. I have also added a second user with the roles ['clusterAdmin', 'userAdminAnyDatabase', 'readAnyDatabase'], based on this Stackoverflow answer.

The instructions then say to run this in an SSH window:

./mongodb-linux-x86_64-2.4.9/bin/mongod --auth --dbpath $HOME/webapps/mongodb1/data/ --port 18000

Use Webfaction’s one-click installer to install a node.js app, and note down its port number too, eg. 17000. Upgrade it to the latest stable version using the “n” package, following Webfaction’s instructions for node.

Following this post, I demeteorized my meteor app, pushed it up to a git repo on Webfaction, and pulled it down into $HOME/webapps/nodeapp/demeteoredapp/.

As mentioned in the Webfaction instructions, in the nodeapp directory, type export PATH=$PWD/bin/:$PATH. Then cd demeteoredapp and type npm install in that directory.

In another SSH shell, enter (the DB name is “admin”):

export MONGO_URL="mongodb://mongo_user:mongo_password@localhost:18000/admin?autoReconnect=true"  # 2
export MAIL_URL='smtp://user:password@smtp.webfaction.com'
export PORT="17000"
export ROOT_URL="http://example.com"

Then type:

./bin/node $HOME/webapps/nodeapp/demeteoredapp/main.js

You can adjust the default bin/start script to do this too.

Having installed the app this way, I have developed a process to update the code. I rename the previously demeteorized app directory, eg. to old; go into my Meteor app directory and type: demeteorize -o ../demeteoredapp, then mv ../old/.git ../demeteoredapp. At that point I can do git add --all and commit and push the new version up to my git repo on Webfaction. I then ssh onto the server and type:

cd webapps/nodeapp/demeteoredapp/appname
git pull; ../bin/stop ; ../bin/start

to pull in the new changes and launch them.

Note there is no need for the forever package when you do it this way. I’m not sure we need the demeteorizer package either, or if the Meteor bundler is sufficient.

I found some comments online that there can be memory problems if you use Meteor and MongoDB on Webfaction. I haven’t experienced this problem. (I’m keeping a close eye on it – I have even built an admin panel which includes a call to a Meteor method that executes the memory_usage.py script provided by Webfaction.)

Ongoing jobs

You will then want to set up cron jobs to start your MongoDB app if it fails for any reason. I have modelled this script off Webfaction’s start script for node.js:

#!/bin/sh
mkdir -p $HOME/webapps/mongoapp/run
mkdir -p $HOME/webapps/mongoapp/log
pid=$(/sbin/pidof $HOME/webapps/mongoapp/mongodb-linux-x86_64-2.4.9/bin/mongod)
if echo "$pid" | grep -q " "; then
  pid=""
fi
if [ -n "$pid" ]; then
  user=$(ps -p $pid -o user | tail -n 1)
  if [ $user = "your-username" ]; then
    exit 0
  fi
fi
nohup $HOME/webapps/mongoapp/mongodb-linux-x86_64-2.4.9/bin/mongod --auth --fork --logpath $HOME/webapps/mongoapp/log/mongo.log --dbpath $HOME/webapps/mongoapp/data/ --port 18000 > /dev/null 2>&1 &
/sbin/pidof $HOME/webapps/mongoapp/mongodb-linux-x86_64-2.4.9/bin/mongod > $HOME/webapps/mongoapp/run/mongo.pid

Put this into your scheduled tasks using crontab -e.

You will also want to back up your database. A simple command to back up the database to a directory called dbbackup is:

$HOME/webapps/mongoapp/mongodb-linux-x86_64-2.4.9/bin/mongodump --port 18000 -o $HOME/dbbackup

You will need to add the admin username and password.

I like to have daily, weekly and monthly backups, so I make three directories $HOME/mongo_backup/daily, weekly, monthly and have a script for each:

The daily script is:

#!/bin/sh
# $HOME/scripts/cron_daily_mongo_backup.sh
# add this to crontab, e.g. to run at 15:45 server time every day:
# 45 15 * * * ~/scripts/cron_daily_mongo_backup.sh

# back up the database
$HOME/webapps/mongoapp/mongodb-linux-x86_64-2.4.9/bin/mongodump --port 18000 -o $HOME/mongo_backup/daily/dump-`date +\%Y\%m\%d` 2>> $HOME/mongo_backup/daily/cron.log

# delete backup directories older than 7 days
find $HOME/mongo_backup/daily -type d -ctime +7 | xargs -r rm -rf  # Be careful using -rf

The weekly script is:

#!/bin/sh
# add this to crontab as, e.g. to run at 16:20 server time every Sunday:
# 20 16 * * 0 ~/scripts/cron_weekly_mongo_backup.sh
cp -R $HOME/mongo_backup/daily/dump-`date +\%Y\%m\%d` $HOME/mongo_backup/weekly
find $HOME/mongo_backup/weekly -type d -ctime +32 | xargs -r rm -rf

And the monthly script is:

#!/bin/sh
# add this to crontab as, e.g. to run at 16:25 server time every 1st of the month:
# 25 16 1 * * ~/scripts/cron_monthly_mongo_backup.sh
#
cp -R $HOME/mongo_backup/daily/dump-`date +\%Y\%m\%d` $HOME/mongo_backup/monthly
find $HOME/mongo_backup/monthly -type d -ctime +370 | xargs -r rm -rf

If you’ve read this far, please check out the finished product at signup.zone!

  

Private media with Django

I have often wanted user-uploaded files and images to be “private” or “secure”, i.e. require some authentication and authorisation to view, but haven’t known how to start. Now that I have a solution that works (e.g. for sites hosted by Webfaction), I’d like to share it with you.

The basic principles are:

  1. Serve your public “static” files (e.g. css and javascript) and any public user/admin-uploaded “media” files from your existing static webapp. This static webapp has no capability to selectively hide some files from view, so we will not use it for the private media.
  2. Serve your private media from a regular Django view:
    • This view will be accessed through a regular Django URL in a urls.py file.
    • The first step in this view is to authenticate and check for authorisation. Be aware that you will only have the request parameters (including the URL path) to do this with. One solution might be to use a directory structure with permissions varying by directory.
    • The second step is to serve the file. You could do this using Django directly, but it would be woeful. Webfaction uses Apache, which has a nice module called mod_xsendfile which lets Apache serve the contents (Django merely specifies the path in the response header). There is a solution if you use nginx too which I have not explored (see django-filer’s secure downloads feature for more). You need to explicitly install and activate this module however – see below for details.
  3. Store your private media somewhere different to your public media. You can do this by providing a custom storage for your private FileFields and ImageFields. If you need instance-specific permissions, you can do this by passing a method as the upload_to directory.

Note – before going down this path, check if the file system you are using already has a protection mechanism (e.g. Amazon’s S3 service), in which case you probably won’t need this.

An implementation

I packaged up an implementation of these principles in django-private-media. You can install it from PyPi with:

pip install django-private-media

This package draws significantly from the secure file download code of Stephan Foulis’s django-filer.  One key difference between the two is that django-filer replaces all file and image fields with a foreign key; in contrast, because my focus is solely on the permissioning, django-private-media just uses the standard Django file and image fields.  As a result, it should be fairly straightforward to convert an existing project to use it.  See the readme for more details.

Code snippets

You’ll find below some code snippets to point you in the right direction. They assume you add a PRIVATE_MEDIA_URL and PRIVATE_MEDIA_ROOT, analogous to Django’s MEDIA_URL and MEDIA_ROOT, to your settings.py file.

urls.py

# urls.py
from django.conf import settings
...
urlpatterns += patterns('appname.views',
    url(r'^{0}(?P.*)$'.format(settings.PRIVATE_MEDIA_URL.lstrip('/')), 'serve_private_file',),
)
...

views.py

# views.py
from django.conf import settings

def has_read_permission(self, request, path):
    "Only show to authenticated users - extend this as desired"
    # Note this could allow access to paths including ../..
    # Don't use this simple check in production!
    return request.user.is_authenticated()

def serve_private_file(request, path):
    "Simple example of a view to serve private files with xsendfile"
    if has_read_permission(request, path):
        fullpath = os.path.join(settings.PRIVATE_MEDIA_ROOT, path)
        response = HttpResponse()
        response['X-Sendfile'] = fullpath
        return response

models.py

# models.py

from django.db import models
from django.conf import settings

private_media = FileSystemStorage(location=settings.PRIVATE_MEDIA_ROOT,
                                  base_url=settings.PRIVATE_MEDIA_URL,
                                  )
# or you could define a custom subclass of FileSystemStorage

class Car(models.Model):
    photo = models.ImageField(storage=private_media)

settings.py

# settings.py
PRIVATE_MEDIA_ROOT = '/home/username/private-media' # for example
PRIVATE_MEDIA_URL = '/private/' # for example

Installing xsendfile

To install and activate xsendfile on Webfaction, follow the advice given by this post.

That’s all!

What have I missed?  Please let me know if you’ve done something similar and have another or better solution.

Otherwise, I hope this helps!

  

Starting a new Django-CMS site with Webfaction

This is a list of steps I followed to start a new Django-CMS site hosted by Webfaction.  Their documentation is excellent, but there are lots of steps across a number of documents, so I hope that bringing them all together in one place helps someone.  At the very least I know it will help me next time.

Start webapps

Once you have a Webfaction account, add two site-specific webapps: a Django application and a static (no .htaccess) application, e.g. newproj_django and newproj_static.

Change the Websites section so your domain points to these two applications.  Give the django app the root URL and the static app the ending “/static“.

Delete the original Django application if you like.

You will also need a git webapp as per http://docs.webfaction.com/software/git.html.  You can just call it git.

Create a database on Webfaction

Create a new database for the app using the Webfaction control panel.  Adjust the settings.  Also change any other settings required.

Create your Django project

I have a cms_starter project on my local computer which I copy for this (largely created by following the steps from the Django-CMS docs).  Make sure the git origin of the new project (in .git/config) is set up like so:

url = user@user.webfactional.com:webapps/git/repos/reponame.git

To rename the project, type:

cp -R cms_starter newproj
cd newproj
mv oldproj newproj

Then open manage.py and change the reference to settings to refer to newproj instead of oldproj.

Repeat with the settings.py file.

Repeat again with the wsgi.py file.

Make sure that when deployed, the Webfaction database is picked up in the settings.py.  (I import a site_settings.py file into settings.py, and put site_settings.py into .gitignore, so that the same code can be used in dev and production.  The only difference is the site_settings.py file.)

Check it works by typing

./manage.py runserver

Finally, commit these changes to git:

git add --all
git commit -a -m "Set up for newproj"

Set up SSH

As described at http://docs.webfaction.com/user-guide/access.html, you can immediately ssh into webfaction in a Mac terminal using

ssh user@user.webfactional.com

However you can make the process smoother by eliminating the need to type in your password every time.  If you already have keys set up in your ~/.ssh directory (if not see the above docs) type:

scp ~/.ssh/id_rsa.pub user@user.webfactional.com:temp_id_rsa_key.pub

Then ssh into your account and type (as per the docs):

cd ~
mkdir .ssh  # only needed if it does not already exist
cat ~/temp_id_rsa_key.pub >> ~/.ssh/authorized_keys
rm ~/temp_id_rsa_key.pub
chmod 600 ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod go-w $HOME

Now – this is not in the docs – you can make a shortcut (e.g. so you can type just ssh quickname) by adding the following to your local computer’s .ssh/config file. The last two lines stop the session from being timed out too soon.

Host quickname

HostName user.webfactional.com
User user
IdentityFile ~/.ssh/id_rsa
ServerAliveCountMax 3
ServerAliveInterval 10

From here it’s programmatic – use fabric

After following the below instructions several times, I have finally wised up and written them into a fabric file, which you’ll find attached at the end of this post (or find the most recent version at github).  With this, all you need to do from here is type: fab prod initialise.

[Update 12 June 2013 - I have now added the above steps for setting up SSH into the fab file as well.]

Set up the git repository on Webfaction

Ssh to the repos directory and initialise a bare repository (which must end in .git) using

git init --bare reponame.git

Following the documentation above, you also need to cd into that repo and turn on “http push” via

cd reponame.git
git config http.receivepack true

Push the local Django project to Webfaction

We are now ready to push your project up to Webfaction. In the newproj directory on your local computer:

git push origin master

This puts it into Webfaction’s git repo. Now we need to bring this into the newproj_django webapp.

cd webapps/newproj_django
rm -rf myproject
git clone ~/webapps/git/repos/reponame.git newproj

Tell Apache that we’ve changed the name of the project directory by opening apache2/conf/httpd.conf and replacing all three occurences of myproject with newproj.

Install Django-CMS

This is covered brilliantly in http://docs.webfaction.com/software/python.html?#installing-python-packages.  Note though you do need to create a tmp directory which is only covered at the very end of the document.  My instructions below include this step.

We will install pip, create a tmp directory, then install Django-CMS into the local webapp newproj.  In any directory type:

easy_install-2.7 pip
mkdir $HOME/tmp
export TEMP=$HOME/tmp
cd $HOME/webapps/newproj_django
pip-2.7 install --install-option="--install-scripts=$PWD/bin" \
    --install-option="--install-lib=$PWD/lib/python2.7" django-cms

Note for the last line you could also type the much simpler:

pip install django-cms ,

but this installs to a user-wide location, so will be available to all Django webapps.  This is a potential version control problem.

This is also a good time to install django-reversion if you use it (for django-cms 2.3.5 you need reversion version 1.6.6; drop the ==1.6.6 if you are using the latest version of django-cms):

cd $HOME/webapps/newproj_django
pip-2.7 install --install-option="--install-scripts=$PWD/bin" \
    --install-option="--install-lib=$PWD/lib/python2.7" django-reversion==1.6.6

Use the right version of python

As described in http://docs.webfaction.com/software/python.html, by default webfaction uses python2.6.6.  If you don’t use python2.7, you will get an error when you first type ./manage.py : “ImportError: No module named django.core.management”.  My solution to this is to add to $HOME/.bash_profile the line:

export PYTHONPATH="$HOME/lib/python2.7:$PYTHONPATH"

Then, manually, every time you ssh into webfaction, you need to type:

export PYTHONPATH="$HOME/webapps/newproj_django/lib/python2.7:$PYTHONPATH"

I used to put this line into $HOME/.bash_profile as well, but the problem is you are hard-wiring into all your ssh sessions which web app to use. An example of why this is a problem: if you upgrade a pip installation in another web app, it will delete your installation from newproj_django and install the upgrade into your other project. Having been stung by this once, I am now taking the laborious approach of modifying PYTHONPATH specifically each time I log in.

And if I want to work on a different project, I log out, ssh back in again and repeat the export PYTHONPATH line with the different project.

Add to $HOME/.bash_profile the lines:

export PYTHONPATH="$HOME/webapps/newproj_django/lib/python2.7:$PYTHONPATH"
export PYTHONPATH="$HOME/lib/python2.7:$PYTHONPATH"

The downside of this approach is that when you are in an ssh shell window, but you are working on another project in your webapps directory, your manage.py script for all your web apps will be accessing the packages you installed for newproj_django.  The website itself however will be using the right packages.

An alternative that I think solves that problem, but only works if you type the command “python” to run manage.py is to add to $HOME/.bash_profile the line:

alias python=python2.7

(This last line will not work if you type  ./manage.py though.)

Set up the database

So far, you have a database, but it does not contain any tables yet.  Assuming you do not have data exported from elsewhere to import, you need to follow these instructions.

For a Django-CMS project using South, you will need to type (see the black screenshot on https://www.django-cms.org/en/documentation/):

python manage.py syncdb --all
python manage.py migrate --fake

You can add data, e.g. via fixtures, at this point if you want.

Set up static content

Make sure the settings point your static and media files something like (this is only one way to do it):

STATIC_ROOT = '/home/user/webapps/newproj_static/static'
STATIC_URL = '/static/static/'
MEDIA_ROOT = '/home/user/webapps/newproj_static/media'
MEDIA_URL = '/static/media/'

And set up static and media subdirectories in the newproj_static webapp:

cd $HOME/webapps/newproj_static/
mkdir media
mkdir static

Then type:

cd $HOME/webapps/newproj_django/newproj/
python manage.py collectstatic

Restart the webapp

Restart the webapp by typing:
../apache2/bin/restart

And you should be able to see the default Django-CMS welcome page on user.webfactional.com .

All done!

If this page has been helpful to you, and you have yet to sign up with Webfaction, please consider signing up as my affiliate using this link. It will not affect your signup or pricing but I will receive a small commission if you do so.

Optional extras

Reversion

If you add django-reversion to your installed apps after you have already set up the database, to fix up the database you will then need to type:

python manage.py syncdb
python manage.py migrate

Either way, once you have your database set up with the initial data you would like (which may be nothing), you will also need:

python manage.py createinitialrevisions

Fabric

I have set up a very simple fabfile.py for deploying changes to the site which looks like this.  You can find the latest version (and even help improve it if you wish!) at github. I prefer to git add --all, git commit and git push manually.

After this file you’ll find a more involved fabfile which implement many of the steps above, so that all you need to type to get going is: fab prod initialise.

from fabric.api import *
from fabric.contrib.console import confirm
import fabric.contrib.files

git_dir = "$HOME/webapps/git/repos"
# replace username in the next path. Can't use $HOME from python.
remote_dir = '/home/username/webapps/newproj_django'
project_name = 'newproj'
code_dir = remote_dir + '/' + project_name
python_dir = remote_dir + '/lib/python2.7'
python_add_str = 'export PYTHONPATH="' + python_dir + ':$PYTHONPATH"; '

def migrate(app=''):
"""Usage:
       fab migrate:app_name.
   If you have added a new app, you need to manually run
       python manage.py schemamigration app_name --init.
"""
    if app:
        local("python manage.py schemamigration %s --auto" % app)
        local("python manage.py migrate %s" % app)
        local("python manage.py createinitialrevisions")

def prod():
    env.hosts = ['user@user.webfactional.com']  # or user@webNNN.webfaction.com

def deploy(app_to_migrate=""):
"""
To save some output text and time,
if you know only one app has changed its database
structure, you can run this with the app's name.

Usage:
    fab prod deploy
    fab prod deploy:myapp
"""
    with cd(code_dir):
        run("git pull")
        run(python_add_str + "python manage.py migrate %s" % app_to_migrate)
        run(python_add_str + "python manage.py createinitialrevisions") # only if using reversion
        run(python_add_str + "python manage.py collectstatic --noinput")
        run("../apache2/bin/restart")

And add this code to automate the initial installation:

install_list = ['django-cms', 'django-reversion']
#
#  Methods for initial installation
#
def install_pip():
    """
    Installs pip itself if needed.

    Usage :
        fab prod install_pip
    """
    with settings(warn_only=True):
        run('mkdir $HOME/lib/python2.7')
        run('easy_install-2.7 pip')

def create_prod_git_repo(git_repo_name):
    """
    Creates a new git repo on the server (do not include the .git ending in git_repo_name)

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod create_prod_git_repo:project

    Requires the git webapp to have been created on the server.
    """
    with cd(git_dir):
        run("git init --bare %s.git && cd %s.git && git config http.receivepack true" %
              (git_repo_name,git_repo_name))

def add_prod_repo_as_origin_and_push(git_repo_name):
    """
    Adds the git repo on the server as the local .git repo's origin, and pushes master to it.
    (do not include the .git ending in git_repo_name)

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod add_prod_repo_as_origin:project

    Requires that the local .git/config has no origin yet (e.g. rename it first if it does).
    """
    local("""echo '[remote "origin"]' >> .git/config""")
    local(r"echo '        fetch = +refs/heads/*:refs/remotes/origin/*' >> .git/config")
    local(r"echo '        url = %s:webapps/git/repos/%s.git' >> .git/config" % (env.hosts[0], git_repo_name))
    local(r"git push origin master")

def update_conf_file():
    """
    Updates the apache httpd.conf file to point to the new project
    instead of the default 'myproject'.

    This is called as part of clone_into_project, or you can call
    separately as:  fab prod update_conf_file
    """
    filepath = remote_dir + "/apache2/conf/httpd.conf"
    fabric.contrib.files.sed(filepath, 'myproject', project_name)

def clone_into_project(git_repo_name):
    """
    Clones the git repo into the new webapp, deleting the default myproject project
    and updating the config file to point to the new project.
    Also adds a site_settings.py file to the project/project folder.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod clone_into_project:project
    """
    repo_dir = git_dir + "/%s.git" % git_repo_name
    with cd(remote_dir):
        run('rm -rf myproject')
        run("git clone %s %s" % (repo_dir, project_name))
        run("echo 'MY_ENV=\"prod\"' > %s/%s/site_settings.py" % (project_name,project_name))
    update_conf_file()

def add_dirs_to_static(static_webapp_name):
    """
    Adds the "/static" and "/media" directories to the static webapp if needed,
    and deletes the default index.html.
    Also adds a project/project/static directory if there isn't one.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod add_dirs_to_static:static_webapp_name
    """
    static_dir = '$HOME/webapps/%s' % static_webapp_name
    with settings(warn_only=True):
        with cd(static_dir):
            run("mkdir static && mkdir media")
            run("rm index.html")
        with cd(code_dir):
            run("mkdir %s/static" % project_name)

def pip_installs():
    """
    Installs the necessary thirdparty apps
    into the local webapp (not globally) using pip.
    Also appends a helpful comment to .bashrc_profile.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod pip_installs
    """
    pip = r'pip-2.7 install --install-option="--install-scripts=$PWD/bin" --install-option="--install-lib=$PWD/lib/python2.7" '
    with settings(warn_only=True):
        run("mkdir $HOME/tmp")
    with cd(remote_dir):
        for installation in install_list:
            run("export TEMP=$HOME/tmp && %s %s" % (pip, installation))
    run("echo '#%s' >> $HOME/.bash_profile" % python_add_str)

def initialise_database():
    """
    Initialises the database to contain the tables required for
    Django-CMS with South.
    Runs syncdb --all and migrate --fake.

    Usage :
        fab prod initialise_database
    """
    with cd(code_dir):
        run(python_add_str + "python manage.py syncdb --all")
        run(python_add_str + "python manage.py migrate --fake")

def initialise(static_webapp_name="myproj_static", git_repo_name="myproj"):
    """
    In brief:
    Creates a git repo on the server, and fills in the django and static webapps with it.
    Initialises the database, and deploys the app.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod install

    Requires a git webapp, the database, the django webapp, the static webapp
    to have been created on the server already.
    Requires that the local .git/config has no origin yet (e.g. rename it if needed first).

    In detail:
      * Installs pip itself on the server, if needed
      * Creates a new git repo on the server (do not include the .git ending in git_repo_name)
      * Add it as the origin to the local git repo
      * Pushes the local code to the new repo
      * Clones it into the new webapp, deleting the default myproject project
      * Modifies the apache httpd.conf file to point to the new project, not myproject
      * Adds a site_settings.py file
      * Adds the "/static" and "/media" directories to the static webapp if needed
      * Adds a project/project/static directory if there isn't one
      * Adds a comment to .bash_profile for the python path (for the user's ref if desired)
      * Installs the necessary thirdparty apps
      * Initialises the database using South
      * Runs the Django-CMS cms check command
      * Deploys the app as normal (the git pull is redundant but harmless)
    """
    install_pip()
    create_prod_git_repo(git_repo_name)
    add_prod_repo_as_origin_and_push(git_repo_name)
    clone_into_project(git_repo_name)
    add_dirs_to_static(static_webapp_name)
    pip_installs()
    initialise_database()

    with cd(code_dir):
        run(python_add_str + "python manage.py cms check")

    deploy()