Introduction

The goal is to run web2py with a small memory footprint in a virtual private server (VPS) with shared access to a single processor (CPU).

Issues

In practice there are four main issues at a non application specific level that arise from this goal.

1. Only run one process (or instance) of Python for web2py for all websites
2. Use a web server for web2py with a low memory footprint
3. Only use SQLite databases.
4. Avoid reoccurring tasks (such as cron tasks) that start new instances of Python and other memory hungry processes.

Results

A reasonably complex web2py deployment servicing more than one site with a FastCGI interface to lighttpd has resulted in Python consuming under 35MB resident RAM and lighttpd consuming under 4MB of resident RAM.

The database used was SQLite, which is the web2py default database.

Why is this important?

1. A baseline needs to be established for 'how low web2py can go' and still remain usable for normal applications.
2. The lower the accepted baseline is the more attractive web2py becomes for hosting providers to provide lower cost solutions based on lower memory requirements
3. The lower the cost of deploying web2py the more competitive web2py becomes compared to traditional solutions that lack the productivity advantages of Python and web2py,
4. Resellers who provide turnkey customised VPS solutions need reliable information for costing purposes and to conduct their own tests.
5. Embedded designers considering web interfaces for standalone devices with inexpensive processors, such as ARM9 processors, need reliable information.

Hosting Discussion

Hosting web2py based solutions is expensive compared to traditional web approaches. Web2py runs as part of a Python process that consumes a large amount of memory. This is normal for web frameworks that use Python. Not only that, the Python process needs to remain persistent in memory even when not serving web page. Otherwise there would be too long a delay serving web pages. The requirement for persistence is what makes hosting expensive. FastCGI is the web server technology that allows an interface to any process that remains persistent independent of the web server. WSGI is a variant of FastCGI that applies to Python only and in which the web server is responsible for the persistent Python process.

Web2py sites will never be as cheap to host as PHP sites. However there are enormous productivity benefits compared to PHP that offset the hosting price differences.

Currently the Google App Engine provides an effective free solution for hosting web2py applications that are not CPU intensive and have simple database requirements.

Reoccurring Tasks

Reoccurring cron tasks that launch memory hungry processes are not processes that remain persistent. However they can hit memory limits and cause memory file swapping to occur. This can cause web sites to intermittently 'crawl'. Also a non VPS hosting provider may penalise sites that exceed allocated RAM.

Purpose of web server

The purpose of the web server is simple.

1. Serve static pages (instead of allowing web2py to serve static pages)
2. Map URLs (if desired)
3. Pipe requests through to web2py and accept responses back.

Web Server Interface Discussion

There is a large number of web server choices for web2py. Web2py has an internal web server that is very convenient for development purposes. It is not recommended for use in a production environment. If the internal web server is not run then there are two accepted ways to use an external web server

1. Use the widely accepted FastCGI technology to interface with an existing independent persistent Python process using a UNIX socket or an IP socket
2. Use a WSGI enabled web server that allows the web server to start and control a Python process, interface with this process and then run web2py using a Python function and callbacks

A web server should provide an efficient and fast pipe that does not get in the way. Lightttpd is a web server known for its exceptionally small memory footprint and speed. Lightttpd provided a FastCGI interface.

In fact web2py converts FastCGI socket information is to WSGI information but this is not relevant to the web server. The Fast CGI web server never sees the WSGI information.

Starting up web2py for use with FastCGI

Starting up web2py for use with FastCGI is similar to starting up web2py for use with the internal web server.

In a production environment it is preferable to use a script in

/etc/init.d

For our purposes using a script in

/etc/rc.local

is a simpler alternative.

Instead of placing something like the following in /etc/rc.local

cd /var/www/web2py && sudo -u www-data nohup python web2py.py -p 8000 -a "<recycle>" &

use the following

cd /var/www/web2py && sudo -u www-data nohup python fcgihandler.py  &

Here is hint of you do use /etc/rc.local or a script that is called by rc.local. If you want to restart web2py quickly you can use the command

killall python ; cd /var/www/web2py && sudo -u www-data nohup python fcgihandler.py &

Note the web server does not need to be restarted if you need to restart web2py.

Using sudo allows the Python process to run as user other than root for safety. If necessary, user names in crontab files should be altered appropriately. However, as noted above, cron jobs that launch another Python process in a tight VPS should be avoided. To change web2py files to the www-data user name (and group) the following command can be used at the top level web2py directory.

chown -R www-data.www-data web2py

A standard UNIX socket is used in this example for the point of communication between web2py and lightppd. The last line of fcgihandler.py specifies the socket as

/tmp/fcgi.sock

This can be changed to whatever location and name you want, as long as lightttpd uses the same location and name and as long as the directory is writeable to for user name the web server and the Python process. Where security issues due to shared VPS user accounts may need to be taken into consideration, the /tmp directory is not a good choice to place the socket, also user and group permissions of the socket need to be considered.

Here is a quick way to determine if web2py is running

ps ax | grep python

There should only be one line with 'fcgihandler.py' in it.

Configuring Lighttpd for use with FastCGI

A core principle to remember is that we are setting up communication between two independent processes: Lighttpd and web2py (as a Python process). As such lighttpd does not need to br responsible for starting up web2y and does not nned to know the name of the web2py startup file fcgihandler.py. In the sample file for /etc/lighttpd/lighttpd.conf there is no entry for 'fcgihandler.py'. What is common between lighttpd.conf and fcgihanler.py is the arbitrary socket file name '/tmp/fcgi.sock'.

Following is an example of a configuration of lighttpd that uses lighttpd to serve static pages, map URLs and interface to a single Python process that runs web2py. The next heading shows a complete tested minimalist configuration file.

Lighttpd can be installed and started with

apt-get install lighttpd

Following are edits to

/etc/lighttpd/lighttpd.conf

The less server modules loaded the lower the memory footprint. Server module 'mod fastcgi' is necessary. In the minimalist configuration sample file below, 'mod rewrite' allows lighttpd to serve web2py static files and send the rest of the requests to the UNIX socket with 'mod fastcgi', for web2py to handle.

server.modules              = (
        "mod_access",
        "mod_alias",
        "mod_compress",
        "mod_rewrite",
        "mod_fastcgi",
        "mod_redirect",
        "mod_accesslog",
        "mod_status",
)

Lighttpd needs to know how to specify to use the UNIX socket and where the UNIX socket is. The location and name provided for the socket option, /tmp/fcgi.sock below, must match the location and name in fcgihandler.py:

fastcgi.server = (
   "/handler_web2py.fcgi" => (
      "handler_web2py" => ( #name for logs
        "check-local" => "disable",
        "socket" => "/tmp/fcgi.sock",
      )
   ),
)

The next section tells lighttpd what to do with for all example.com web sites (including example.com).

$HTTP["host"] =~ "(^|\.)example\.com$" {
 server.document-root="/var/www/web2py"
    url.rewrite-once = (
      "^(/.+?/static/.+)$" => "/applications$1",
      "(^|/.*)$" => "/handler_web2py.fcgi$1",
    )
}

1. The option server.document-root is used to tell lighttpd where the server root directory is to find static files, for example.com sites.

2. The line with 'static' in it tell lightppd where to look to serve static files from off the server root directory, for example.com sites.

3. The last line says map all other URL locations to the interface with the Python web2py process. The use of directory '/handler_web2py.fcgi' tells lighttpd to use FastCGI instead of static file serving, as per specified fastcgi.server configuration.

You can check for syntax errors with

lighttpd -t -f /etc/lighttpd/lighttpd.conf

Before you restart lighttpd with a web2py UNIX socket for the first time make sure that you have run 'sudo - u www-data python fcgihandler.py' once. This will create a representative zero length socket file in the file system. After this it does not matter if web2py fcgihandler.py is started before lighttpd or not.

After editing restart the lighttpd web server with

/etc/init.d/lighttpd restart.

Complete Minimalist Lighttpd Configuration File

This is for exclusive use of lighttpd with web2py, web2py static files and optionally SSL.

##/etc/lighttpd/lighttpd.conf file for use with web2py, web2py static files and optiionally SSL
## There are four non default site dependent paths

server.modules              = (
            "mod_compress",
            "mod_rewrite",
            "mod_fastcgi",
)
server.errorlog            = "/var/log/lighttpd/error.log"
index-file.names           = ("index.html", "index.htm", "default.htm", "index.lighttpd.html" )
accesslog.filename         = "/var/log/lighttpd/access.log"
url.access-deny            = ( "~", ".inc" )
static-file.exclude-extensions = ( ".php", ".pl", ".py", ".fcgi" )
server.pid-file            = "/var/run/lighttpd.pid"
dir-listing.encoding        = "utf-8"
server.username            = "www-data"
server.groupname           = "www-data"
compress.cache-dir          = "/var/cache/lighttpd/compress/"
compress.filetype           = ("text/plain", "text/html", "application/x-javascript", "text/css")
include_shell "/usr/share/lighttpd/create-mime.assign.pl"


fastcgi.server = (
   "/handler_web2py.fcgi" => (
      "handler_web2py" => ( #name for logs
        "check-local" => "disable",
        "socket" => "/tmp/fcgi.sock", ##non default site dependent path 1, must match name in fcgihandler.py
      )
   ),
)

server.document-root="/varr/www/web2py" ##non default site dependent path 2

url.rewrite-once = (
   "^(/.+?/static/.+)$" => "/applications$1", # use lighttpd for web2py static files
   "(^|/.*)$" => "/handler_web2py.fcgi$1",
)

# remove comment symbols '#' below to enable SSL use
#$SERVER["socket"] == "0.0.0.0:443" {
#  ssl.engine = "enable"
#  ssl.pemfile = "/etc/lighttpd/ssl/www.example.com.pem" ##non default site dependent path 3
#  ssl.ca-file = "/etc/lighttpd/ssl/www.example.com.ca"  ##non default site dependent path 4
#}

Using Lighttpd on port 80 for Secure Web2py Admin Access without SSL

By default, password protected web2py admin can be accessed either with SSL (port 443) from any IP address, or from port 80 only locally with IP address 127.0.0.1 (localhost).

Because we are limiting ourselves to one instance of Python we cannot start up another instance of Python for admin purposes.

Suppose SSL is not available or you have adjusted web2py to allow only localhost access to web2py admin. To securely use web2py admin with lighttpd on port 80 instead of using the web2py internal server for admin on another port, you need to take two actions.

1. Set up a secure tunnel between an arbitrary port on your PC and between localhost port 80 on the VPS (instead of to default localhost port 8000 on your VPS).
2. Set a password for use with web2py admn on port 80 (when accessed from VPS localhost)

It is not necessary to use SSL with lightttpd for admin access to work securely, since web traffic is tunnelled to securely to the VPS before using port 80.

PuTTY is a common secure client on PCs. Suppose you decide to use port 8001 on your PC for secure tunnelling then in PuTTY, 'Change Settings', 'Connection,' 'SSH, Tunnels' you can set 'Source port' to '8001', 'Destination' to 'localhost:80' and then click on Add.

To set a web2py admin password just run 'sudo -u www-data python web2py.py -p 80'. If lighttpd is running then the command will fail but this not matter as the password will have been set. Kill web2py if it does start to allow lighttpd to start on port 80.

Alternatively you can use the admin password you set up with 'python web2py' by copying file 'parameters_8000.py' to file 'parameters_80.py'.

To then access web2py admin from yout PC use web address http://localhost:8001/admin and enter the admin password.

Adding URL mapping to a lighttpd configuration

This section shows to take advantage of inward URL mapping using lighttpd. Alternatively the 'routes_in' tuple of web2py file 'routes.py' can be used instead to specify some URL mappings.

To take advantage of the facilities provided by lighttpd and lower the load on web2py, static file URL mappings should remain specified in the lighttpd configuration, as specified above.

Assume 'app' is the name of the default web2py application. Short application names that can result in naming ambiguity for URL mapping should be avoided. However the name is good enough for illustration.

$HTTP["host"] =~ "(^|\.)example\.com$" {
 server.document-root="/var/www/web2py"
    url.rewrite-once = (
      "^(/.+?/static/.+)$" => "/applications$1",
      "^(/static/.+)$" => "/applications/app$1",
      "^/$" => "/handler_web2py.fcgi/app/default/index",
      "(^/app.*)$" => "/handler_web2py.fcgi$1",
      "(^/admin.*)$" => "/handler_web2py.fcgi$1",
      "(^/examples.*)$" => "/handler_web2py.fcgi$1",
      "(^/welcome.*)$" => "/handler_web2py.fcgi$1",
      "(^|/.*)$" => "/handler_web2py.fcgi/app/default/$1",
    )
}

1. The option server.document-root is used to tell lighttpd where to find static files.

2. The lines with 'static' in them tell lightppd where to look to serve static files from. The second line with 'static' might look pointless. However it is useful if you have a line in the 'routes_out' tuple of file 'routes.py' like:

   ('/app/static/(?P<any>.*)', '/static/\g<any>'),

3. "^/$" specifies where the default web page for example.com sites is. Note the use of directory '/handler_web2py.fcgi' which tells lighttpd to use FastCGI instead of static file serving.

4. The next four lines map URL locations that start with /admin, /examples, /welcome and /app and tell lighttpd to use FastCGI with no special URL mapping.

5. The last line says map all other URL locations to the default web2py controller /app/default. It is used to remap URLs from which 'app/default' has been removed. To benefit from this either routes.out of routes.py needs to be used or internal techniques need to be used to selectively eliminate '/app/default' from URLs.

Suitable sample 'routes_out' tuple entries are:

('/app/default/index', '/'),
('/app/default/$f', '/$f'),

Alternatively an internal technique might consist of a variable called 'path_prepend' for relevant menu item in response.menu. The variable can be set to "" instead of a value of "/" + request.application + "/default"

Some Application Specific Steps to Increase Responsiveness

While this slice is concerned with non application specific steps to reduce memory use (and CPU use), following are some general steps that can be applied to applications to reduce CPU load and so increase responsiveness.

1. Code in files in the models directory is always executed by an application. If code does not need to be in a file in the models directory, consider putting it elsewhere such as a controller file or modules directory file.

2. In a production environment, allowing the migrate option of the define_table command to remain at a default value of 'True' is an unnecessary penalty that increases CPU load.

3. In a production environment, use your application in its compiled form to reduce load. Also compiling your application may help detect bugs earlier.

Conclusions

This approach achieves our objectives of using a web server for static files, URL mapping and FastCGI with a low memory footprint.

John Heenan