Description

faster filetransfer over the network

Author

David Krentzlin

Version

Usage

(require-extension sendfile)

Download

sendfile.egg

Documentation

This extension provides a way to do filetransfers with zero-copy-semantics if applicable. It provides an interface to the sendfile syscall on systems that support it. On other systems it emulates it with the help of memory-mapped IO. Sendfile is very configurable and tries to do the best based on the actual file. See below to learn what it provides.

NOTE: theoretically read-write-loop and mmapped-io can be used for file-to-file-transfers, where sendfile can only be used for file-network-transfers.

sending a file. the main interface

procedure: (sendfile source destination #!key force)

Tries to send the file identified by `source` to `destination` as fast as possible. Unless a specific technique is forced it will decide what method to use from the systems capabilities and the filesize.

source ... can be either a port to the inputfile or a filedescriptor of an allready opened file.

destination ... can be either a port to the outputfile (socket) or a filedescriptor (socketdesciptor) of an allready opened file (socket)

force ... is a symbol stating which transmission-technique shall be used. possible values are: 'mmapped,'sendfile,'read-write,'nothing. It defaults to 'nothing

sending a file with the sendfile-syscall

procedure: (sendfile:sendfile source destination len)

If it is available this is the interface to the sendfile-implementation of your operating system

source ... is the filedescriptor of an allready opened file

destination ... is the filedescriptor of an allready opened file (MUST be a socket)

len ... is the size of the file in bytes as e.g. retrieved by (file-size)

This procedure returns the amount of bytes successfully written.

sending a file with memory-mapped-io

procedure: (sendfile:mmapped source destination len)

Sends a file by mapping it into the memory and do repeated writes.

source ... is the filedescriptor of an allready opened file

destination ... is the filedescriptor of an allready opened file (can be a socket)

len ... is the size of the file in bytes as e.g. retrieved by (file-size)

This procedure returns the amount of bytes successfully written.

sending a file with a read-write-loop

procedure: (sendfile:read-write-loop source destination len)

Sends a file by performing repeated reads and writes.

source ... is the filedescriptor of an allready opened file

destination ... is the filedescriptor of an allready opened file (can be a socket)

len ... is the size of the file in bytes as e.g. retrieved by (file-size)

This procedure returns the amount of bytes successfully written.

test if sendfile is nativly available

sendfile:os-dep:sendfile-available?

Is set to #t if the sendfile-syscall is available and #f if not

test if mmap() is available

sendfile:os-dep:mmap-available?

Is set to #t if the mmap() is available and #f if not

Prameters

parameter: sendfile:read-write-buffer-size

The size of the buffer that is used in sendfile:read-write-loop

parameter: sendfile:implementation-selector

A one-argument procedure that gets the size of the file in question passed and is expected to return a procedure to use (either of sendfile:mmapped,sendfile:sendfile,sendfile:read-write-loop)

Examples

(use sendfile)

;;in all the examples
;;we use a generic procedure with-prepared-environment
;;which we assume provides us with the input and outputports
;;needed. Most of the time the output-port will be a socket
;;and the input-port may be connected to a file
;;the size of the input-file was gathered as well


;; use the standard interface and let the system decide what to do
(with-prepared-environment
 (lambda (in out len)
   (sendfile in out)))

;; force a specific method to use: Part I

;;read-write
;;notice the third parameter which forces a specific
;;technique to use for sending the file
;;in this case we use 'read-write
;;you can choose out four options: read-write,sendfile,mmapped,nothing
;;where 'nothing is the default
(with-prepared-environment
 (lambda (in out len)
   (sendfile in out force: 'read-write)))

;;force a specific method to use: Part II

;;sometimes you may want to decide which method to
;;use based on the size of the file.
;;there is an srfi-parameter called sendfile:implementation-selector
;;which does just that. See documentation for details
(with-prepared-environment
 (lambda (in out)
   (paramaterize ((sendfile:implementation-selector) (lambda (len)
                                                       (if (> len 1024)
                                                           sendfile:sendfile
                                                           sendfile:read-write-loop)))
                 (sendfile in out))))