This is taken from some of my working code.  

-------------------controller
    ######### enquiry form ##########
    enqform = SQLFORM.factory(
                           Field('name', label='Name *', requires = IS_NOT_EMPTY(),
                                         default='%s %s'%(auth.user.first_name or '', auth.user.last_name or '') if auth.user_id else ''),
                           Field('email', label='Email *', requires = [IS_EMAIL(),IS_NOT_EMPTY()],
                                         default = (auth.user.email or '') if auth.user_id else ''),
                           Field('message', 'text', label='Message'),
                           Field('spam_test', label='Please ignore this'),
                          )
    if enqform.process(dbio=False).accepted:
        response.js="$('#modal_form').dialog('close');"
        if not form.vars.spam_test:
            mail.send(to=['list@of',email@addresses'],
                subject='Enquiry From: '+request.vars.name,
                # If reply_to is omitted, then mail.settings.sender is used
                #reply_to='your_email@gmail.com',
                message='Name: '+request.vars.name
                        +'\nEmail: '+request.vars.email
                        +'\nMessage: '+request.vars.message
                        +'\nRe: Advert: %s [id: %s]'% (rec.business, rec.id)
                )
    elif enqform.errors:
        response.flash = 'Enquiry Form has Errors'
        response.js="$('#modal_form').dialog({autoOpen : true, modal : true, width : 'auto', title : 'Enquiry Form'});"        
    else:
        response.js="$('#modal_form').dialog('open').focus();"        
    return dict(enqform=enqform.....)

Then in your view...

------------- view
<!-- Enquiry Section --> 
<script language="javascript"> 
    $(document).ready(function(){ 
        $("#no_table_spam_test__row").css("display","none");
        $("#enquirybutton").click(function() {
             $('#modal_enqform').dialog({modal : true, width : 'auto', title : 'Enquiry Form'}); 
         });
        {{=XML(response.js)}}    
    }) 
</script> 
<div id="modal_enqform" style='display:none'>{{=enqform}}</div>
<!-- Enquiry Section Ends -->

Include jQuery libraries:

---------layout.html
<!-- **** Take care all libraries are compatible!!! 
I made it work with these added to layout.html (and wasted a lot of time experimenting with others)  **** -->
  <script src="//ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js"></script>
  <link href="http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.23/themes/{{=response.theme or "smoothness"}}/jquery-ui.css"  rel="stylesheet" type="text/css" /> 
  <script src="http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.23/jquery-ui.min.js" type="text/javascript"></script>