Description

Author

Version

• 0.1 Initial implementation

Usage

Download

Documentation

The framework networks an arbitrary number of processes, living on the same or on different nodes, and communicating over TCP. Each process can be a master or a slave. Both masters and slaves can act as TCP servers/clients. Network architecture is up to you; one master client connecting to many slave servers seems to be easy to set up.

Slaves evaluate s-expressions dispatched to them by masters, and report the results. Additionally, each slave has a SRFI-69 hash-table that can be accessed by masters (which are supposed to pick unique keys); this is useful when each slave needs expensive one-time initialization.

This is a cooperative framework; there is no protection against malicious nodes, or against errors yet. Connections are supposed to be permanent -- servers and clients can't just close connections and "walk away".

Master

procedure: (dpfw-master-server port [backlog [host]])

Starts a master TCP server, waiting for slaves to connect. Blocks.

procedure: (dpfw-master-client host port)

Connects to a slave TCP server. Returns immediately after connecting.

procedure: (dpfw-make-task remote-sexp local-procedure [slave])

Dispatches a task (remote s-expression) to an idle slave (or to the specified slave). Spawns a new thread to execute the local procedure (which normally reads and returns the slave's output). Returns a promise that, when forced, joins the thread and returns its result. The slave is marked as busy and is unavailable for other tasks until the local procedure completes.

Both the remote s-expression and the local procedure must take two arguments: an input port and an output port.

procedure: (dpfw-parallel ((remote local) ...))

Executes several tasks in parallel (assuming enough idle slaves are available).Returns a list of results.

procedure: (dpfw-get-slave [wait?]

Returns an idle slave and marks it as busy.

Slave

procedure: (dpfw-slave-server port [backlog [host]])

Starts a slave TCP server, waiting for masters to connect. Blocks.

procedure: (dpfw-slave-client host port)

Connects to a master TCP server and waits for tasks. Blocks.

procedure: (dpfw-slave-ht)

Returns a hash-table that persists across multiple master tasks.

Examples

;; For masters
(define (read-result i o) (read i))

;; On node A
(require 'dpfw/slave) (import dpfw/slave)
(dpfw-slave-server 10000)
;; On node B
(require 'dpfw/master) (import dpfw/master)
(dpfw-master-client "node-b.example.com" 10000)

(define t (dpfw-make-task `(lambda (i o) (+ 2 3)) read-result))
(force t)
;; ==> 5

(dpfw-parallel
 `(((lambda (i o) (+ 2 1)) ,read-result)
   ((lambda (i o) (+ 5 5)) ,read-result)))
;; ==> (3 10)

(define slave (dpfw-get-slave))
(force (dpfw-make-task
        `(lambda (i o)
           (hash-table-update!/default
            (dpfw-slave-ht) 'uniqID values
            (let ((n 0)) (lambda () (set! n (+ n 1)) n)))
           #t)
        read-result slave))
(force (dpfw-make-task `(lambda (i o) ((hash-table-ref (dpfw-slave-ht) 'uniqID)))
                       read-result slave))
;; ==> 1
(force (dpfw-make-task `(lambda (i o) ((hash-table-ref (dpfw-slave-ht) 'uniqID)))
                       read-result slave))
;; ==> 2

License

Copyright (C) 2006 Dan Muresan (muresan at stanfordalumni dot org)

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License, version 2, as
published by the Free Software Foundation.

This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
General Public License for more details.