The framework networks an arbitrary number of processes, living on the same or on different nodes, and communicating over TCP. Each process can be a master or a slave. Both masters and slaves can act as TCP servers/clients. Network architecture is up to you; one master client connecting to many slave servers seems to be easy to set up.
Slaves evaluate s-expressions dispatched to them by masters, and report the results. Additionally, each slave has a SRFI-69 hash-table that can be accessed by masters (which are supposed to pick unique keys); this is useful when each slave needs expensive one-time initialization.
This is a cooperative framework; there is no protection against malicious nodes, or against errors yet. Connections are supposed to be permanent -- servers and clients can't just close connections and "walk away".
Dispatches a task (remote s-expression) to an idle slave (or to the specified slave). Spawns a new thread to execute the local procedure (which normally reads and returns the slave's output). Returns a promise that, when forced, joins the thread and returns its result. The slave is marked as busy and is unavailable for other tasks until the local procedure completes.
Both the remote s-expression and the local procedure must take two arguments: an input port and an output port.
;; For masters (define (read-result i o) (read i))
Setting up the network:
;; On node A (require 'dpfw/slave) (import dpfw/slave) (dpfw-slave-server 10000) ;; On node B (require 'dpfw/master) (import dpfw/master) (dpfw-master-client "node-b.example.com" 10000)
(define t (dpfw-make-task `(lambda (i o) (+ 2 3)) read-result)) (force t) ;; ==> 5
(dpfw-parallel `(((lambda (i o) (+ 2 1)) ,read-result) ((lambda (i o) (+ 5 5)) ,read-result))) ;; ==> (3 10)
Saving a generator in a slave's hash-table, then invoking it repeatedly:
(define slave (dpfw-get-slave)) (force (dpfw-make-task `(lambda (i o) (hash-table-update!/default (dpfw-slave-ht) 'uniqID values (let ((n 0)) (lambda () (set! n (+ n 1)) n))) #t) read-result slave)) (force (dpfw-make-task `(lambda (i o) ((hash-table-ref (dpfw-slave-ht) 'uniqID))) read-result slave)) ;; ==> 1 (force (dpfw-make-task `(lambda (i o) ((hash-table-ref (dpfw-slave-ht) 'uniqID))) read-result slave)) ;; ==> 2
Copyright (C) 2006 Dan Muresan (muresan at stanfordalumni dot org) This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License, version 2, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.