Tag Archives: git

Starting a new Django-CMS site with Webfaction

This is a list of steps I followed to start a new Django-CMS site hosted by Webfaction.  Their documentation is excellent, but there are lots of steps across a number of documents, so I hope that bringing them all together in one place helps someone.  At the very least I know it will help me next time.

Start webapps

Once you have a Webfaction account, add two site-specific webapps: a Django application and a static (no .htaccess) application, e.g. newproj_django and newproj_static.

Change the Websites section so your domain points to these two applications.  Give the django app the root URL and the static app the ending “/static“.

Delete the original Django application if you like.

You will also need a git webapp as per http://docs.webfaction.com/software/git.html.  You can just call it git.

Create a database on Webfaction

Create a new database for the app using the Webfaction control panel.  Adjust the settings.  Also change any other settings required.

Create your Django project

I have a cms_starter project on my local computer which I copy for this (largely created by following the steps from the Django-CMS docs).  Make sure the git origin of the new project (in .git/config) is set up like so:

url = user@user.webfactional.com:webapps/git/repos/reponame.git

To rename the project, type:

cp -R cms_starter newproj
cd newproj
mv oldproj newproj

Then open manage.py and change the reference to settings to refer to newproj instead of oldproj.

Repeat with the settings.py file.

Repeat again with the wsgi.py file.

Make sure that when deployed, the Webfaction database is picked up in the settings.py.  (I import a site_settings.py file into settings.py, and put site_settings.py into .gitignore, so that the same code can be used in dev and production.  The only difference is the site_settings.py file.)

Check it works by typing

./manage.py runserver

Finally, commit these changes to git:

git add --all
git commit -a -m "Set up for newproj"

Set up SSH

As described at http://docs.webfaction.com/user-guide/access.html, you can immediately ssh into webfaction in a Mac terminal using

ssh user@user.webfactional.com

However you can make the process smoother by eliminating the need to type in your password every time.  If you already have keys set up in your ~/.ssh directory (if not see the above docs) type:

scp ~/.ssh/id_rsa.pub user@user.webfactional.com:temp_id_rsa_key.pub

Then ssh into your account and type (as per the docs):

cd ~
mkdir .ssh  # only needed if it does not already exist
cat ~/temp_id_rsa_key.pub >> ~/.ssh/authorized_keys
rm ~/temp_id_rsa_key.pub
chmod 600 ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod go-w $HOME

Now – this is not in the docs – you can make a shortcut (e.g. so you can type just ssh quickname) by adding the following to your local computer’s .ssh/config file. The last two lines stop the session from being timed out too soon.

Host quickname

HostName user.webfactional.com
User user
IdentityFile ~/.ssh/id_rsa
ServerAliveCountMax 3
ServerAliveInterval 10

From here it’s programmatic – use fabric

After following the below instructions several times, I have finally wised up and written them into a fabric file, which you’ll find attached at the end of this post (or find the most recent version at github).  With this, all you need to do from here is type: fab prod initialise.

[Update 12 June 2013 - I have now added the above steps for setting up SSH into the fab file as well.]

Set up the git repository on Webfaction

Ssh to the repos directory and initialise a bare repository (which must end in .git) using

git init --bare reponame.git

Following the documentation above, you also need to cd into that repo and turn on “http push” via

cd reponame.git
git config http.receivepack true

Push the local Django project to Webfaction

We are now ready to push your project up to Webfaction. In the newproj directory on your local computer:

git push origin master

This puts it into Webfaction’s git repo. Now we need to bring this into the newproj_django webapp.

cd webapps/newproj_django
rm -rf myproject
git clone ~/webapps/git/repos/reponame.git newproj

Tell Apache that we’ve changed the name of the project directory by opening apache2/conf/httpd.conf and replacing all three occurences of myproject with newproj.

Install Django-CMS

This is covered brilliantly in http://docs.webfaction.com/software/python.html?#installing-python-packages.  Note though you do need to create a tmp directory which is only covered at the very end of the document.  My instructions below include this step.

We will install pip, create a tmp directory, then install Django-CMS into the local webapp newproj.  In any directory type:

easy_install-2.7 pip
mkdir $HOME/tmp
export TEMP=$HOME/tmp
cd $HOME/webapps/newproj_django
pip-2.7 install --install-option="--install-scripts=$PWD/bin" \
    --install-option="--install-lib=$PWD/lib/python2.7" django-cms

Note for the last line you could also type the much simpler:

pip install django-cms ,

but this installs to a user-wide location, so will be available to all Django webapps.  This is a potential version control problem.

This is also a good time to install django-reversion if you use it (for django-cms 2.3.5 you need reversion version 1.6.6; drop the ==1.6.6 if you are using the latest version of django-cms):

cd $HOME/webapps/newproj_django
pip-2.7 install --install-option="--install-scripts=$PWD/bin" \
    --install-option="--install-lib=$PWD/lib/python2.7" django-reversion==1.6.6

Use the right version of python

As described in http://docs.webfaction.com/software/python.html, by default webfaction uses python2.6.6.  If you don’t use python2.7, you will get an error when you first type ./manage.py : “ImportError: No module named django.core.management”.  My solution to this is to add to $HOME/.bash_profile the line:

export PYTHONPATH="$HOME/lib/python2.7:$PYTHONPATH"

Then, manually, every time you ssh into webfaction, you need to type:

export PYTHONPATH="$HOME/webapps/newproj_django/lib/python2.7:$PYTHONPATH"

I used to put this line into $HOME/.bash_profile as well, but the problem is you are hard-wiring into all your ssh sessions which web app to use. An example of why this is a problem: if you upgrade a pip installation in another web app, it will delete your installation from newproj_django and install the upgrade into your other project. Having been stung by this once, I am now taking the laborious approach of modifying PYTHONPATH specifically each time I log in.

And if I want to work on a different project, I log out, ssh back in again and repeat the export PYTHONPATH line with the different project.

Add to $HOME/.bash_profile the lines:

export PYTHONPATH="$HOME/webapps/newproj_django/lib/python2.7:$PYTHONPATH"
export PYTHONPATH="$HOME/lib/python2.7:$PYTHONPATH"

The downside of this approach is that when you are in an ssh shell window, but you are working on another project in your webapps directory, your manage.py script for all your web apps will be accessing the packages you installed for newproj_django.  The website itself however will be using the right packages.

An alternative that I think solves that problem, but only works if you type the command “python” to run manage.py is to add to $HOME/.bash_profile the line:

alias python=python2.7

(This last line will not work if you type  ./manage.py though.)

Set up the database

So far, you have a database, but it does not contain any tables yet.  Assuming you do not have data exported from elsewhere to import, you need to follow these instructions.

For a Django-CMS project using South, you will need to type (see the black screenshot on https://www.django-cms.org/en/documentation/):

python manage.py syncdb --all
python manage.py migrate --fake

You can add data, e.g. via fixtures, at this point if you want.

Set up static content

Make sure the settings point your static and media files something like (this is only one way to do it):

STATIC_ROOT = '/home/user/webapps/newproj_static/static'
STATIC_URL = '/static/static/'
MEDIA_ROOT = '/home/user/webapps/newproj_static/media'
MEDIA_URL = '/static/media/'

And set up static and media subdirectories in the newproj_static webapp:

cd $HOME/webapps/newproj_static/
mkdir media
mkdir static

Then type:

cd $HOME/webapps/newproj_django/newproj/
python manage.py collectstatic

Restart the webapp

Restart the webapp by typing:
../apache2/bin/restart

And you should be able to see the default Django-CMS welcome page on user.webfactional.com .

All done!

If this page has been helpful to you, and you have yet to sign up with Webfaction, please consider signing up as my affiliate using this link. It will not affect your signup or pricing but I will receive a small commission if you do so.

Optional extras

Reversion

If you add django-reversion to your installed apps after you have already set up the database, to fix up the database you will then need to type:

python manage.py syncdb
python manage.py migrate

Either way, once you have your database set up with the initial data you would like (which may be nothing), you will also need:

python manage.py createinitialrevisions

Fabric

I have set up a very simple fabfile.py for deploying changes to the site which looks like this.  You can find the latest version (and even help improve it if you wish!) at github. I prefer to git add --all, git commit and git push manually.

After this file you’ll find a more involved fabfile which implement many of the steps above, so that all you need to type to get going is: fab prod initialise.

from fabric.api import *
from fabric.contrib.console import confirm
import fabric.contrib.files

git_dir = "$HOME/webapps/git/repos"
# replace username in the next path. Can't use $HOME from python.
remote_dir = '/home/username/webapps/newproj_django'
project_name = 'newproj'
code_dir = remote_dir + '/' + project_name
python_dir = remote_dir + '/lib/python2.7'
python_add_str = 'export PYTHONPATH="' + python_dir + ':$PYTHONPATH"; '

def migrate(app=''):
"""Usage:
       fab migrate:app_name.
   If you have added a new app, you need to manually run
       python manage.py schemamigration app_name --init.
"""
    if app:
        local("python manage.py schemamigration %s --auto" % app)
        local("python manage.py migrate %s" % app)
        local("python manage.py createinitialrevisions")

def prod():
    env.hosts = ['user@user.webfactional.com']  # or user@webNNN.webfaction.com

def deploy(app_to_migrate=""):
"""
To save some output text and time,
if you know only one app has changed its database
structure, you can run this with the app's name.

Usage:
    fab prod deploy
    fab prod deploy:myapp
"""
    with cd(code_dir):
        run("git pull")
        run(python_add_str + "python manage.py migrate %s" % app_to_migrate)
        run(python_add_str + "python manage.py createinitialrevisions") # only if using reversion
        run(python_add_str + "python manage.py collectstatic --noinput")
        run("../apache2/bin/restart")

And add this code to automate the initial installation:

install_list = ['django-cms', 'django-reversion']
#
#  Methods for initial installation
#
def install_pip():
    """
    Installs pip itself if needed.

    Usage :
        fab prod install_pip
    """
    with settings(warn_only=True):
        run('mkdir $HOME/lib/python2.7')
        run('easy_install-2.7 pip')

def create_prod_git_repo(git_repo_name):
    """
    Creates a new git repo on the server (do not include the .git ending in git_repo_name)

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod create_prod_git_repo:project

    Requires the git webapp to have been created on the server.
    """
    with cd(git_dir):
        run("git init --bare %s.git && cd %s.git && git config http.receivepack true" %
              (git_repo_name,git_repo_name))

def add_prod_repo_as_origin_and_push(git_repo_name):
    """
    Adds the git repo on the server as the local .git repo's origin, and pushes master to it.
    (do not include the .git ending in git_repo_name)

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod add_prod_repo_as_origin:project

    Requires that the local .git/config has no origin yet (e.g. rename it first if it does).
    """
    local("""echo '[remote "origin"]' >> .git/config""")
    local(r"echo '        fetch = +refs/heads/*:refs/remotes/origin/*' >> .git/config")
    local(r"echo '        url = %s:webapps/git/repos/%s.git' >> .git/config" % (env.hosts[0], git_repo_name))
    local(r"git push origin master")

def update_conf_file():
    """
    Updates the apache httpd.conf file to point to the new project
    instead of the default 'myproject'.

    This is called as part of clone_into_project, or you can call
    separately as:  fab prod update_conf_file
    """
    filepath = remote_dir + "/apache2/conf/httpd.conf"
    fabric.contrib.files.sed(filepath, 'myproject', project_name)

def clone_into_project(git_repo_name):
    """
    Clones the git repo into the new webapp, deleting the default myproject project
    and updating the config file to point to the new project.
    Also adds a site_settings.py file to the project/project folder.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod clone_into_project:project
    """
    repo_dir = git_dir + "/%s.git" % git_repo_name
    with cd(remote_dir):
        run('rm -rf myproject')
        run("git clone %s %s" % (repo_dir, project_name))
        run("echo 'MY_ENV=\"prod\"' > %s/%s/site_settings.py" % (project_name,project_name))
    update_conf_file()

def add_dirs_to_static(static_webapp_name):
    """
    Adds the "/static" and "/media" directories to the static webapp if needed,
    and deletes the default index.html.
    Also adds a project/project/static directory if there isn't one.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod add_dirs_to_static:static_webapp_name
    """
    static_dir = '$HOME/webapps/%s' % static_webapp_name
    with settings(warn_only=True):
        with cd(static_dir):
            run("mkdir static && mkdir media")
            run("rm index.html")
        with cd(code_dir):
            run("mkdir %s/static" % project_name)

def pip_installs():
    """
    Installs the necessary thirdparty apps
    into the local webapp (not globally) using pip.
    Also appends a helpful comment to .bashrc_profile.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod pip_installs
    """
    pip = r'pip-2.7 install --install-option="--install-scripts=$PWD/bin" --install-option="--install-lib=$PWD/lib/python2.7" '
    with settings(warn_only=True):
        run("mkdir $HOME/tmp")
    with cd(remote_dir):
        for installation in install_list:
            run("export TEMP=$HOME/tmp && %s %s" % (pip, installation))
    run("echo '#%s' >> $HOME/.bash_profile" % python_add_str)

def initialise_database():
    """
    Initialises the database to contain the tables required for
    Django-CMS with South.
    Runs syncdb --all and migrate --fake.

    Usage :
        fab prod initialise_database
    """
    with cd(code_dir):
        run(python_add_str + "python manage.py syncdb --all")
        run(python_add_str + "python manage.py migrate --fake")

def initialise(static_webapp_name="myproj_static", git_repo_name="myproj"):
    """
    In brief:
    Creates a git repo on the server, and fills in the django and static webapps with it.
    Initialises the database, and deploys the app.

    Usage (in the local project directory, e.g. ~/Python/Projects/project) :
        fab prod install

    Requires a git webapp, the database, the django webapp, the static webapp
    to have been created on the server already.
    Requires that the local .git/config has no origin yet (e.g. rename it if needed first).

    In detail:
      * Installs pip itself on the server, if needed
      * Creates a new git repo on the server (do not include the .git ending in git_repo_name)
      * Add it as the origin to the local git repo
      * Pushes the local code to the new repo
      * Clones it into the new webapp, deleting the default myproject project
      * Modifies the apache httpd.conf file to point to the new project, not myproject
      * Adds a site_settings.py file
      * Adds the "/static" and "/media" directories to the static webapp if needed
      * Adds a project/project/static directory if there isn't one
      * Adds a comment to .bash_profile for the python path (for the user's ref if desired)
      * Installs the necessary thirdparty apps
      * Initialises the database using South
      * Runs the Django-CMS cms check command
      * Deploys the app as normal (the git pull is redundant but harmless)
    """
    install_pip()
    create_prod_git_repo(git_repo_name)
    add_prod_repo_as_origin_and_push(git_repo_name)
    clone_into_project(git_repo_name)
    add_dirs_to_static(static_webapp_name)
    pip_installs()
    initialise_database()

    with cd(code_dir):
        run(python_add_str + "python manage.py cms check")

    deploy()
  

Git

Work.

Today’s task is getting Xcode working with a remote Git repository, so that I can collaborate with others – and use a laptop and a desktop to program.

  • The instructions on github.com are good, and I used them to set up a new username, SSH key, and a repository for a free open-source account.
  • I then followed the instructions at “Xcode 4 and Git for newbies“, which are also good.  The main things to keep in mind seem to be:
    • in Xcode, to set up the repository in the Organizer, you need to press the little plus button at the bottom of the list on the left, and then choose the “Checkout or Clone” drop down – not “add a repository”.  The location is git@github.com:<username>/<repository name>.git .
    • Save the project as “origin”, which sets up an “origin” directory.
    • The password Xcode asks for is the pass phrase used earlier.
  • I then created a new Xcode project, and saved it so that it all went into a subdirectory of the origin directory.
  • Then committing and pushing the files worked perfectly, and they showed up in my github repository on the web.

The next step was to checkout the repository with the same username but onto a different computer.  Trying the naive approach – just opening Xcode and following the same series of steps in the Organizer – led me to the error “Authentication failed because the name or password was incorrect.”  The answer was to copy my private and public SSH keys from the first machine to the second, at ~/.ssh/id_rsa*. Note the terminal command “mount” shows the path to any mounted drives, which is helpful for copying.  Once copied, I tested the connection to github in the terminal with ssh -T git@github.com.  This informed me I had the wrong permissions for my private key, which I fixed with a chmod.  Then I tested the ssh connection as described in the github intro with ssh -vT git@github.com.  This did not work the first time without the “v” flag.  For some reason with the “v” it worked, and I was then able to continue in Xcode and connect to the repository.

One other issue when using multiple machines is making sure you export your developer profile from the Xcode Organizer on the original Mac, and import it into the second Mac’s Organizer.

Off to learn about the functional programming language Haskell this evening, to broaden the mind.