If you benefit from web2py hope you feel encouraged to pay it forward by contributing back to society in whatever form you choose!

web2py trunk ver 1281, modified base on stable Version 1.67.2 (2009-09-28 16:29:33)


beaware, no browser can handle file over 2GB,
related article http://www.motobit.com/help/scptutl/pa98.htm

 

and, this code is just a simple patch that only works when you running web2py standalone,
it just shows the concept about how to implement,
this is really old, so you have to figure it out and implement in your way,
there are alternative to do without hacking inside the server code,
HTML5 have some nice API that make you possible monitor upload info in client site, just google it.


In Short

  1. set GET value X-Progress-ID with yourUUID, for example

    <form action="http://127.0.0.1:8000/exsample/upload/post?X-Progress-ID=4c1dd2cced4f7c961a0919d58120957e">
    
  2. retrieve upload total length from Cache with key "X-Progress-ID:"+yourUUID+":length"

    cache.ram('X-Progress-ID:'+'4c1dd2cced4f7c961a0919d58120957e'+':length', lambda: 0, None)
    
  3. retrieve current uploaded length from Cache with key "X-Progress-ID:"+yourUUID+":uploaded"

    cache.ram('X-Progress-ID:'+'4c1dd2cced4f7c961a0919d58120957e'+':uploaded', lambda: 0, None)
    

REALLY LONG STORY

the implement is base on Django snippts,

http://www.djangosnippets.org/snippets/678/
http://www.djangosnippets.org/snippets/679/

you can take a look for reference


Current Implementation

here is the related part for caching the current upload progress at gluon/main.py

def copystream_progress(request, chunk_size= 10**5):
    """
    copies request.env.wsgi_input into request.body 
    and stores progress upload status in cache.ram
    X-Progress-ID:length and X-Progress-ID:uploaded
    """
    if not request.env.content_length:
        return None
    source = request.env.wsgi_input
    size = int(request.env.content_length)
    dest = tempfile.TemporaryFile()
    if not 'X-Progress-ID' in request.get_vars:
        copystream(source, dest, size, chunk_size)
        return dest
    cache_key = 'X-Progress-ID:'+request.get_vars['X-Progress-ID']
    cache = Cache(request)
    cache.ram(cache_key+':length', lambda: size, 0)
    cache.ram(cache_key+':uploaded', lambda: 0, 0)
    while size > 0:
        if size < chunk_size:
            data = source.read(size)
            cache.ram.increment(cache_key+':uploaded', size)
        else:
            data = source.read(chunk_size)
            cache.ram.increment(cache_key+':uploaded', chunk_size)
        length = len(data)
        if length > size:
            (data, length) = (data[:size], size)
        size -= length
        if length == 0:
            break
        dest.write(data)
        if length < chunk_size:
            break
    dest.seek(0)
    cache.ram(cache_key+':length', None)
    cache.ram(cache_key+':uploaded', None)
    return dest

you can see it retrieve request.get_vars['X-Progress-ID'] as cache key and just store information in Cache,
so, you have to pass a GET with key X-Progress-ID to make this work,
then retrieve what you want from Cache.


Why not use a hidden form field to pass this X-Progress-ID?
don't be silly, you can't get any POST vars before you read all of it lol


EXAMPLE

I am gonna bring in the example, here is the form view post.html

{{extend 'layout.html'}}

<script type="text/javascript">
// Generate 32 char random uuid 
function gen_uuid() {
    var uuid = ""
    for (var i=0; i < 32; i++) {
        uuid += Math.floor(Math.random() * 16).toString(16); 
    }
    return uuid
}

// Add upload progress for multipart forms.
$(function() {
    $('form[enctype=multipart/form-data]').submit(function(){ 
        // Prevent multiple submits
        if ($.data(this, 'submitted')) return false;

        var freq = 1000; // freqency of update in ms
        var uuid = gen_uuid(); // id for this upload so we can fetch progress info.
        var progress_url = '/{{=request.application}}/{{=request.controller}}/{{=request.function}}.json'; // ajax view serving progress info

        // Append X-Progress-ID uuid form action
        this.action += (this.action.indexOf('?') == -1 ? '?' : '&') + 'X-Progress-ID=' + uuid;

        var $progress = $('<div id="upload-progress" class="upload-progress"></div>').
            insertAfter($('input[type=submit]')).append('<div class="progress-container"><span class="progress-info">uploading 0%</span><div class="progress-bar"></div></div>');

        $('input[type=submit]').remove()
        // progress bar position
        /*
        $progress.css({
            position: ($.browser.msie && $.browser.version < 7 )? 'absolute' : 'fixed',  
            left: '50%', marginLeft: 0-($progress.width()/2), bottom: '20%'
        }).show();
        */

        $progress.find('.progress-bar').height('1em').width(0).css("background-color", "red");
        // Update progress bar
        function update_progress_info() {
            $progress.show();
            $.getJSON(progress_url, {'X-Progress-ID': uuid, 'random': Math.random()}, function(data, status){
                if (data) {
                    var progress = parseInt(data.uploaded) / parseInt(data.length);
                    var width = $progress.find('.progress-container').width()
                    var progress_width = width * progress;
                    $progress.find('.progress-bar').width(progress_width);
                    $progress.find('.progress-info').text('uploading ' + progress*100 + '%');
                }
                window.setTimeout(update_progress_info, freq);
            });
        };
        window.setTimeout(update_progress_info, freq);

        $.data(this, 'submitted', true); // mark form as submitted.
    });
});
</script>
{{=BEAUTIFY(response._vars)}}

most AJAX stealed from Django snippets http://www.djangosnippets.org/snippets/679/
the import part is

this.action += (this.action.indexOf('?') == -1 ? '?' : '&') + 'X-Progress-ID=' + uuid;

yap, feed URL with GET X-Progress-ID,
you can do it from your view to generate this or from client side JavaScript,
oops, there is another part also important,

var progress_url = '/{{=request.application}}/{{=request.controller}}/{{=request.function}}.json'; // ajax view serving progress info

yup, I cheat it, I get the form and read my progress from same controller function with JSON, you can make another controller function to feed this.

there is also a trick part,

$.getJSON(progress_url, {'X-Progress-ID': uuid, 'random': Math.random()},

what the heck am I doing? why I need a random value that I never used at all??
well, this is a IE issue
http://robertnyman.com/2007/04/04/weird-xmlhttprequest-error-in-ie-just-one-call-allowed/
IE won't let you continue request same URI, it block the request for a while,
I have tested the behavior on IE6/7/8 and they all have same issue,
so you have to make sure screw the URI to get JSON result, FOR IE.


next is the sample controller upload_progress_examples.py to generate the form, receive the form and read the progress

def post():
    if request.get_vars.has_key('X-Progress-ID'):

        cache_key = 'X-Progress-ID:'+request.get_vars['X-Progress-ID']

        length=cache.ram(cache_key+':length', lambda: 0, None)
        uploaded=cache.ram(cache_key+':uploaded', lambda: 0, None)

        return dict(length=length, uploaded=uploaded)

    form = FORM(TABLE(
        TR('File:', INPUT(_type='file', _name='file',
            requires=IS_NOT_EMPTY())),
        TR('', INPUT(_type='submit', _value='SUBMIT')),
        ))

    return dict(form=form)

because I cheat in my view, I have to deal with request.get_vars['X-Progress-ID'],
I retrieve Cache with None expire time in order to get current value without writing into in, tricky part,


but wait, where is my JSON view? ahh, here is it, post.json

{{
###
# response._vars contains the dictionary returned by the controller action
###
try:
   from gluon.serializers import json
   response.write(json(response._vars),escape=False)
   response.headers['Content-Type']='text/json'
except:
   raise HTTP(405,'no json')
}}

this create JSON output


I know current implementation is not safe, but it works,
it allows you to flush any value in GET, anyone can screw it,
the further improvement maybe a ticket function to issue a UUID that you can use for this specific task.


that's all, any comment is welcome :)

Related slices

Comments (4)


Hosting graciously provided by:
Python Anywhere