[Ur] callbacks from C FFI (again)

[Adam Chlipala]

On 12/15/2013 04:41 PM, Sergey Mironov wrote:
> Hi. I'd like to re-open the topic [1] regarding UrWeb callbacks from C
> code. Imagine the 'anonymous downloader' application, where
> 1. user logs in and enters the link to download
> 2. server starts the download task, task runs for hours
> 3. server sends an email to user when the download finishes
>
> Also, imagine that I have two separate UrWeb,library: one for starting
> downloader threads and one for sending emails. I'd like to glue them
> together but in order to do so I have to provide the thread with a
> callback to call UrWeb handler which eventually calls email sender.
> AFAIK, there is no pre-designed way callback an UrWeb code, so here
> are my ideas and questions:
>    

To recap the previous discussion for anyone just following along in this 
thread, the problem here is that Ur/Web's server-side compilation 
doesn't currently support first-class functions.  Adding first-class 
functions would require nontrivial work in closure conversion or similar 
(no such compiler phase exists now).

> 1)
> A simple solution I see is to provide the downloader with an url to
> trigger completion actions.

I like this solution!  It takes advantage of features that are already 
present to assign simple names to callable functions.  I'll comment more 
below.

> 2)
> A cool solution: Invent some kind of 'detached thread monad'. the new
> code would be like
>
> fun handler (clientId: int) =
>    email<- queryEmail clientId;
>    Emailer.sendTo email "Download completed"
>
> fun download (u:string) = do
>    clientId<- newClient;
>    runInSeparateThread (fn thread_id =>
>        u<- execute ("wget " ^ u);
>        (* other logic in Ur/Web language *)
>        visit (url (handler clientId))
>        )
>    

Sounds like a lot more work, not necessarily less work than adding 
run-time closures in general.

> I just remember that it's not always possible to query server
> port/host because application may use external web server or proxy, so
> it is not always possible to visit callback url. What I actually want
> is to mimic page request from inside the server without actually
> opening HTTP session, parsing headers, etc. I guess it is not that
> simple currently. Correct me if I'm wrong..
>    

It may actually be that simple. :)  The patch that you sent later plugs 
into the request-processing pipeline at a higher level than necessary.

Take a look at urweb/include/urweb/types_cpp.h, in particular at the 
definition of the struct 'uw_app'.  One field is a function pointer 
'uw_handle', which takes as arguments just a context and a request URI.  
It's fine for such a URI to be in exactly the format that [url] will 
generate (or, if it isn't, I'm sure I can make small tweaks to allow 
it).  No new transaction is started by a handler function.

The last ingredient is how to get ahold of a 'uw_app', but uw_get_app() 
does that for you, given a context!

About details of your patch, which are probably also relevant to the 
approach I suggest:

> - the mimic_request function may run out of stack space if recursion
> happens. should I enqueue such requests it in a separate thread?
>    

Well, you have the same issue with any recursive program, so I 
personally wouldn't worry about it too much.  You wouldn't expect an 
infinite loop of successive subrequests, right?

(I still do recommend using separate threads for background tasks, 
though.  Maybe you can even use Ur/Web's "tasks" support, which wouldn't 
require any FFI coding; let me know if you want more information on that 
feature, beyond what's in the manual.)

> - mimic_request starts new database transaction while running in the
> context of a previous one. is it deadlock-free?
>    

One of these transactions could participate in deadlock just as easily 
as any other, and the semantics of nested Ur/Web transactions that the 
database doesn't know are nested could lead to painful management overhead.

> - minimal test works (prints debug message), but server reports
> "Database connection initialized" every time mimic_request is called.
> why is that?
>    

uw_request_new_context() allocates a new context, which includes getting 
it a fresh database connection, which happens to include (a) allocating 
all prepared statements that the app might use and (b) querying the SQL 
catalog to make sure the schema matches what was in your Ur/Web source 
code!  Clearly not something you really want to do on each request, but, 
if it only happens before something slow like sending e-mail, it might 
not be a big enough deal to call for workaround implementation work.  
However, if you just create one background thread to handle all e-mails, 
it's easy to keep one context live there.  There are also other 
"connection pooling" strategies you could implement in C.  (Connection 
pooling has performance advantages independent of the operations 
currently run on each new connection by Ur/Web.)  All this is handled 
for you automagically if you use Ur/Web tasks.