Property

Usually, data in the rows should be accessed by means of properties. Once created, properties are not destroyed even all references to them are gone. To clean up properties information at all, use mk:destroy! with 'props argument.

The attempt to create identically named properties will return the existing property rather than new one.

There are seven types of properties:

• int - 32-bit integer
• string - character string
• float, double - single- or double-precision floating number
• view - nested view (subview in the Metakit terms)
• bytes - memo data (exposed to the Chicken as byte-vectors)
• long - 64-bit integer. Chicken representation depends on the value of the mk:long-prop-type parameter.

(mk:make PROPTYPE PROPNAME)

create a property named PROPNAME. PROPTYPE can be one of the symbols:

int-prop
string-prop
float-prop
double-prop
long-prop
view-prop
bytes-prop

(mk:name PROPERTY)

return the name of the PROPERTY

(PROPERTY VALUE)

create the row, that has PROPERTY set to VALUE. See also Rows and properties example.

(PROPERTY [PROPERTY1 ...])

create view, which initially contains properties PROPERTY, PROPERTY1 ...

(mk:long-prop-type [NEWVALUE])

Parameter. Manage the representation for the long properties. Valid values are:

• double - double-precision floating number (default)
• long - Chicken long foreign type
• bytes - byte-vector of the length 8. The bytes ordering is platform-dependent.

Row

Metakit row is similar to the buffer with data, rowref is a reference to the specific row in the view. Data changes, performed against rowref, will be redirected to the containing view.

(mk:make 'row [ROW/ROWREF1 ...])

create row as an union of rows

(mk:copy! ROW/ROWREF ROW/ROWREF1)

copy the ROW/ROWREF1 to the ROW/ROWREF.

(mk:container ROW/ROWREF)

return container object for the row or rowref (i.e. the view).

(mk:concat! ROW ROW/ROWREF)

concat values from the ROW/ROWREF to the ROW.

Working with row properties

(mk:value PROPERTY ROW/ROWREF)

(set! (mk:value PROPERTY ROW/ROWREF) NEWVALUE)

get/set the value of the PROPERTY in the ROW/ROWREF.

(mk:value BYTES-PROP [OFFSET [LEN]] ROW/ROWREF)

(set! (mk:value BYTES-PROP ROW/ROWREF [OFFSET [DIFF]]) NEWVALUE)

get/set (maybe partial) value of the bytes property and resize it by DIFF bytes.

See also Rows and properties and Storage and view examples.

Main procedures

(mk:make 'view)

create view. See also mk:view and property application forms.

(mk:structure VIEW)

get structure of the VIEW

(mk:row VIEW ROWINDEX)

(set! (mk:row VIEW ROWINDEX) ROW/ROWREF)

get/set the row of the VIEW (view will grow if required).

(mk:item VIEW ROWINDEX COLINDEX)

(set! (mk:item VIEW ROWINDEX COLINDEX) NEWDATA)

get/set the value of the COLINDEX-th property in the ROWINDEX row as a byte-vector.

(mk:item VIEW ROWINDEX PROPERTY)

(set! (mk:item VIEW ROWINDEX PROPERTY) NEWDATA)

get/set the value of the PROPERTY in the ROWINDEX row (actually, it is a shorthand for the mk:value and mk:row combination).

(mk:size VIEW)

(set! (mk:size VIEW) NEWSIZE)

manage the size of the VIEW.

(mk:insert! VIEW ROW/ROWREF)

append a row.

(mk:insert! VIEW POSITION ROW/ROWREF [COUNT = 1])

insert COUNT instances if the row at POSITION.

(mk:insert! VIEW POSITION VIEW1)

insert contents of the VIEW1 into VIEW at POSITION.

(mk:remove! VIEW [POSITION [COUNT = 1]])

remove COUNT rows at POSITION. When no POSITION was given, remove all rows.

(mk:relocate! VIEW FROMPOS COUNT DESTVIEW POS)

move VIEW's rows to the DESTVIEW in same storage.

(mk:num-props VIEW)

return the number of properties present in this view.

(mk:prop VIEW NAME)

index of the property with NAME in the VIEW.

(mk:prop VIEW N)

returns the N-th property (using zero-based indexing).

(mk:add-prop! VIEW PROPERTY)

adds a property column to a view if not already present.

(mk:index-of VIEW ROWREF)

return the index of the specified row in this view.

(mk:compare VIEW VIEW1)

compare two views lexicographically.

(mk:find VIEW ROW/ROWREF [START])

(mk:find VIEW PRED [START])

find index of the the next entry matching the specified key or predicate.

(mk:find VIEW 'restrict POSITION COUNT)

restrict the search range for rows. Return three values: success flag, POSITION and COUNT.

(mk:find VIEW 'sorted ROW/ROWREF)

search for a key, using the native sort order of the view (gives unpredictable results on the unsorted view!).

(mk:find VIEW 'count POS)

return two values: number of matching keys and pos of first one.

(mk:find* VIEW ARGS ...)

the same as mk:find, but return a rowref rather than index. Return #f if the row cannot found.

(mk:transform VIEW OPERATION ARGS)

return view with some combination of the VIEW data (several derived views are able to mirror changes to the base view).

(mk:transform VIEW (list OPERATION1 ...) (list OPARGS1 ...))

combinator to perform several mk:transform operations at once

Valid transformations are:

• duplicate - construct a new view with a copy of the data.

• clone - construct a new view with the same structure but no data.

• sort - create view with all rows in natural (property-wise) order.

• unique - create view with all duplicate rows omitted.

• read-only - create an identity view which only allows reading.

• blocked - create mapped view which blocks its rows in two levels. This view acts like a large flat view, even though the actual rows are stored in blocks, which are rebalanced automatically to maintain a good trade-off between block size and number of blocks. The underlying view must be defined with a single view property named _B, with the structure of the subview being as needed. See also define-blocked-view and Blocked views. [1] An example of a blocked view definition which will act like a single one containing 2 properties:

  (define raw (mk:view storage "people[_B[name:S,age:I]]"))
  (define flat (mk:transform raw 'blocked)) ...
  (display (mk:size flat)) ...
  (mk:insert! flat ...)
  

• sort-on VIEW1 - create view sorted according to the specified properties. E.g. (mk:transform v1 'sort-on (ip)) will create the new view from the v1 data, sorted by ip.

• project VIEW1 - create view with the specified property arrangement. [1] E.g. (mk:transfrom v1 'project (sp ip)) will return the view with rearranged properties (sp will be the firts and ip the second).

• project-without VIEW1 - create derived view with some properties omitted. [1] E.g., (mk:tranform v1 'project-without (ip)) will return the view without the ip property.

• product VIEW1 - create view which is the cartesian product with given view.

• remap-with VIEW - create view which remaps another given view. [1]

• pair VIEW1 - create view which pairs each row with corresponding row. [1]

• concat VIEW1 - create view with rows from another view appended. [1]

• union VIEW1 - create view which is the set union (assumes no duplicate rows).

• intersect VIEW1 - create view with all rows also in the given view (no dups).

• different VIEW1 - create view with all rows not in both views (no dups).

• minus VIEW1 - create view with all rows not in the given view (no dups).

• select ROW/ROWREF - create view with rows matching the specified value.

• sort-on-reverse VIEW1 VIEW2 - create sorted view, with some properties sorted in reverse. E.g., (mk:transform v1 'sort-on-reverse (ip) (sp)) will create the view with the data from v1, but sorted by ip and reverse sorted by sp.

• select-range ROW/ROWREF1 ROW/ROWREF2 - create view with row values within the specified range.

• rename PROP1 PROP2 - create view with one property renamed (must be of same type). [1]

• group-by VIEW1 VIEW-PROP - create view with a subview, grouped by the specified properties. This operation is similar to the SQL GROUP BY, but it takes advantage of the fact that Metakit supports nested views. The view returned from this member has one row per distinct group, with an extra view property holding the remaining properties. If there are N rows in the original view matching key X, then the result is a row for key X, with a subview of N rows. The properties of the subview are all the properties not in the key. E.g., (mk:transform v1 'group-by (ip) vp) will group by ip.

• counts VIEW1 INT-PROP - create view with count of duplicates, when grouped by key. This is similar to group-by, but it determines only the number of rows in each group and does not create a nested view.

• ordered [COUNT] - create mapped view which keeps its rows ordered. This view can be combined with a blocked view, to create a 2-level btree structure. See also define-ordered-view [1]

• hash VIEW1 [COUNT] - create mapped view which adds a hash lookup layer. [1] This view creates and manages a special hash map view, to implement a fast find on the key. The key is defined to consist of the first COUNT properties of the underlying view. The VIEW1 view must be empty the first time this hash view is used, so that Metakit can fill it based on whatever rows are already present in the underlying view. After that, neither the underlying view nor the map view may be modified other than through this hash mapping layer. The defined structure of the map VIEW1 must be _H:I,_R:I. See also define-hashed-view and Hashed views. [1] Example:

  (define raw (mk:view storage "people[name:S,age:I]"))
  (define datah (mk:view storage "people_H[_H:I,_R:I]"))
  (define hash (mk:transform raw 'hash datah 1)) ...
  (display (mk:size hash)) ...
  (mk:insert! hash)
  

• join KEYS-VIEW VIEW2 [OUTER? = #f] - create view which is the relational join on the given keys (as defined by KEYS-VIEW).

• join-prop VIEW-PROP [OUTER? = #f] - create view with a specific subview expanded, like a join.

• slice [FIRST [LIMIT [STEP]]] - create view which is a segment/slice (default is up to end). [1] [2]

• filter PRED - create brand new view, that contains rows from original VIEW, satisfying predicate PRED. [3]

• filter-map PROC - create view from rows, that satisfy predicate PROC (which is a procedure of a ROWREF argument). It must return either #f (don't use this row) or the row to insert into derived view. [3]

• map PROC - create view with rows, that are result of the application PROC procedure to the rows of the original view. [3]

[1]	The resulting view is updatable.

[2]	Slice operation may be useful to apply other transformation only to the some part of the view.

Utilities

Metakit module defines several convenience procedures

(mk:row-rev VIEW ROWINDEX)

return ROWINDEX-th row, counting from the end (0 means the last row).

(mk:fold PROC VIEW SEED [START])

fold over VIEW rows. PROC must accept two arguments: ROWREF and current SEED and return two values: whether to continue enumeration and new the seed value. Last returned seed become the result of mk:fold. [3]

(mk:fold-rev PROC VIEW SEED [START])

fold over VIEW rows in the backward direction. [3]

(mk:for-each PROC VIEW)

apply PROC to the VIEW rows sequentially. [3]

(mk:column PROPERTY VIEW)

(set! (mk:column PROPERTY VIEW [TRUNCATE? = #t]) NEWDATA)

get/set values of the PROPERTY in the all VIEW rows at once. NEWDATA should be either vector or a list. TRUNCATE? flag determines whether to truncate view if NEWDATA is shorter.

[3]	Mimics the usual list operations.

Syntactic forms

Metakit egg provides additional syntactic sugar for working with views and properties.

(R! ROW '(PROP VALUE)... ROWEXPR ... )

allows to set values in the row with ease. It is equivalent to:

(set! (mk:value PROP ROW) VALUE) ...
(mk:concat! ROW ROWEXPR) ...

(R+ '(PROP VALUE) ... ROWEXPR ...)

allows to construct row with complex data. Uses R!.

(define-view (VIEW STORAGE) (PROPNAME PROPTYPE) ...)

allows to work with views and properties in a record-alike manner. Also it service to get rid of the explicit properties definitions and to replace Metakit-style descriptions in the form "view[prop:proptype]" with S-expressions. VIEW is a view name, STORAGE - opened Metakit storage PROPTYPE is either one of the symbols I, S, F, B, L, D or the form (PROPNAME PROPTYPE) ... The latter case represents subview.

E.g.:

(define stg (mk:make 'storage "stg.mk4"))
(define-view (v stg) (ip I) (sp S)
                     (sv (ipp I) (ssv (sspp S)) (spp S))
                     (lp L) (sv2 (ii I)) )

will define:

• properties ip, sp, sv, ipp, ssv, sspp, spp, lp, sv2, ii;
• view v with description v[ip:I,sp:S,sv[ipp:I,ssv[sspp:S],spp:S],lp:L,sv2[ii:I]]
• constructors: make-v, make-sv, make-ssv and make-sv2. They can be invoked with the positional and named arguments. All of them are optional. As usual, all the named arguments must follow the positional ones. E.g. (make-v 100 "sp1val" (lp 200)), (make-sv (ipp 111))
• accessor procedures: v-ip, v-sp, v-sv, sv-ipp etc.
• mutator procedures: v-ip-set!, v-sp-set!, v-sv-set!, sv-ipp-set! etc.

(define-blocked-view (VIEW STORAGE) (PROPNAME PROPTYPE) ...)

open blocked view without dealing on Metakit low-level details about _B[] etc. Behind the scenes this macro transforms description to the form view[_B[props ...]] and defines view as the blocked mapped view using mk:transform procedure with a 'blocked operation. E.g., (define-blocked-view (v stg) (ip I) (sp S)) will allow to work with view v without knowledge it is actually the blocked view.

(define-ordered-view (VIEW STORAGE [NUMKEYS]) PROPS ...)

(define-hashed-view (VIEW STORAGE SUPPLEMENTAL-VIEW [NUMKEYS]) PROPS ...)

Define ordered or hashed view. SUPPLEMENTAL-VIEW can be either:

• #f - temporary view will be created.
• STRING - the view with that name will be created in the storage.
• VIEW1 - use existing view as a supplemental view.

Commit choices

The normal commit requires the exclusive access mode. This call is only possible for files opened in exclusive mode, and therefore can only be done when no other readers exist.

The commit-extend mode requires open mode exclusive or commit-extend , i.e. a mode which allows changes. The main value of this mode, is that it allows concurrent reading and does so without any locking requirement or possible contention. Only one extender can have the file open an any point in time, but with the proper synchronization (outside Metakit), different processes could alternate in obtaining that extender access and then relinquishing it again.

Finally, there is a new commit-aside mechanism, which works in any file mode. This is possible, because the commit saves its changes in a secondary Metakit datafile . That second datafile must be open in mode exclusive or commit-extend, and can itself therefore use either normal commits or commit extends. In fact, the secondary datafile could even be read-only, if it has itself been set up to work in commit-aside mode. In other words, commit-aside can be stacked. See below for more details.

Commit-extend

Commit extend consumes disk space, because every commit will store changes at the end of the datafile. Since changes mean that entire columns are saved, the amount of disk space consumed can grow rapidly when frequent commits are issued, altering large datasets.

In a way, commit extend is like an ordinary commit, but one which does not ever re-use free space inside the datafile. It is relatively fast, because free space is in fact not even tracked in commit-extend mode.

The way to reclaim free space, is to re-open the datafile in exclusive mode , and then commit normally. There are a number of ways to actually bring the file back to its minimal size (this won't happen right away, since space will only become re-usable after the commit completes succesfully).

Commit-aside

Commit aside uses a secondary aside file, which is a regular Metakit datafile, but one of which the contents is managed exclusively by Metakit.

File changes saved with commit-aside require much less spaced than with commit-extend, because only modified sections of columns are saved. This means that commit-aside performance is far more proprtional with the number of changes than with the size of the entire dataset (normal commit speed is highly related to total column sizes).

To use commit aside, both datafiles must be opened. The main datafile can be opened in any mode, since it will not be altered. The secondary aside file must be writable. In commit-aside mode, the secondary file is then associated to the main file (using (set! (mk:aside ...))). From then on, commits on the main file cause differences to be written to the aside file, and a "real" commit on the aside file to be issued.

While in aside mode four different commit/rollback calls can be issued:

• commit fast: this saves changes to the aside file, and commits the aside file
• commit full: this folds all aside changes back into the main file, commits the main file, and clears the aside changes (this mode requires write access to the main file)
• rollback fast: revert the state to the last commit (aside or normal, whichever was last)
• rollback full: this reverts to the last full commit, and has as side effect that the aside mode is turned off (the aside file is no longer special after this)

Commit-aside datafiles are useless without the original datafile. In fact, the moment that original datafile is modified, they become invalid. To this end, Metakit stores a monotonically increasing generation number in each datafile, and stores the current value in the commit-aside file. There is one situation in which changes are actually meant to invalidate the aside file, that is when a full commit is done.

Combined modes

The features described above open a range of options for dealing with multi-user / multi-tasking scenarios, even though they do not yet handle all possible cases.

First of all, commit-extend offers extreme performance in the case when lots of readers require access to a consistent state, but only few changes are made. Readers will see the state at time of opening the datafile (they can close and re-open any time to see new committed changes). This mode is efficient because readers never wait, not even while the extender is committing changes.

The drawback of commit extend is the potentially large disk space consumption, and hence the need to reclaim and clean up periodically. This requires exclusive access.

Commit-aside can be very effective for single-task situations with lots of commits, since commit performance is relatively high, and disk space consumption is limited. But again, at some point, changes will need to be consolidated, and again exclusive access mode is needed to fold all changes back into the main datafile and to clear the aside file.

Finally, an interesting option is to use commit aside with a read-only main datafile, and commit-extend for the secondary aside file. Since only changes are saved, the aside file will not grow that quickly, and since both files support concurrency, multiple readers can get at the latest consistent state without delay. The only contention is that if there are multiple writers, these will have to find a way to take turns in making changes.

Yet more scenarios will become available later, once the main datafile can also be used as the aside file, i.e. modes can be combined.

Hashed views

Hashed views work in combination with a second view to provide O(1) hashed access by key value. The data is - as before - stored in the original view. The second view is just used as a hash index and is completely managed by Metakit. The second view can be either persistent or transient (i.e. not stored on file).

To use a hashed view, you must apply the hashed view mapping, which takes three input arguments: 1) the data view, 2) the secondary hash view, and 3) the number of properties involved in the key. The resulting view looks identical to the data view, but when looking up a key value, Metakit will detect the presence of the hash info and will use it to very quickly locate the row with the given key.

There are a number of rules to make this work:

The secondary view must have a fixed structure, i.e. two int properties called _H and _R, respectively. The size and contents of this view is managed by the hash view mapping. The key is always taken as the first N properties of the data. N is usually 1, but it can be higher in case of compound keys.

For the infinitely curious: the secondary view implements a hash table, with spare slots, so it will have more rows than the actual data view. The size of the secondary view is always a power of two.

Blocked views

Blocked views offer scalability, i.e. they support views which can contain far more rows without becoming slow when making changes. The trade-off is that plain positional access and iteration are somewhat less efficient. Blocked views are implemented as a view of smaller subviews, with all the blocking details fully managed by Metakit. The result looks, walks, and quacks like one huge view.

Unlike hashed views, blocked views need a slightly different definition of the view in which data is being stored. If you wanted a view with say 4 properties, you'd now have to redefine the view as a view of views, i.e. the description names[first:S,last:S,age:I,shoesize:I] needs to be changed into names[_B[first:S,last:S,age:I,shoesize:I]].

To use a blocked view, you pass this new definition to the blocked view mapping, which takes a single argument, being the redefined view. The virtual result will look like the view you actually expect, but internally Metakit will fool around with how it stores things in subviews, moving rows around to keep all subviews reasonably balanced.

For the infinitely curious: blocked views implement a structure which is very similar to B-trees, but fixed to two levels. As a result, storing a million rows virtually will be handled as storing rows in 1,000 subviews of 1,000 rows each (on average).

Rows and properties

(require-extension srfi-17 metakit)

(define ip (mk:make 'int-prop "ip")) ; make integer property with name "ip"
(define sp (mk:make 'string-prop "sp")) ; make string property

(define bp (mk:make 'bytes-prop "bp"))  ; make bytes property
(define rip100 (ip 100)) ; make row with ip = 100
(define r (mk:make 'row)) ; create empty row
  
; create the row with ip = 100 and sp = "qwe"
(define r2 (mk:make 'row rip100 (sp "qwe")))

(set! (mk:value sp r) "ABC") ; set sp of the r to "ABC"
(equal? "qwe" (mk:value sp r2)) ; sp of r2 should be "qwe"
  
(set! (mk:value bp r) (byte-vector 1 2 3)) ; set bp of the r to 1, 2, 3
(equal? (byte-vector 2) (mk:value bp r 1 1)) ; byte at pos. 1 should be 2

Storage and view

See also Convenience syntax example.

(require-extension srfi-17 metakit)

(define stg (mk:make 'storage "stg2.mk4"))
(define name (mk:make 'string-prop "name"))
(define age (mk:make 'int-prop "age"))
(define people (mk:view stg "people[name:S,age:I]")) ; get or define view

(define p1 (mk:make 'row)) ; create empty row
(set! (mk:value name p1) "John") ; fill it with data
(set! (mk:value age p1) 35)
(mk:insert! people p1) ; store it

; create the row with data and insert it at the start of the view
(mk:insert! people 0 (mk:make 'row (name "Helga") (age 27)))

; set value in the row #2 (starting from 0)
; the view will grow if needed
(set! (mk:row people 2) (name "Greg"))
(let ([p2 (mk:row people 2)]) ; get rowref
  ; update data in the rowref, and, accrodingly, in the view
  (set! (mk:value age p2) 31) )

(mk:for-each  ; print the contents of the view people
  (lambda (r)
    (display (mk:value age r))
    (display " ")
    (display (mk:value name r))
    (newline))
  people)

(mk:commit! stg) ; commit changes

; cleanup
(gc #t)
(mk:destroy! people)
(mk:destroy! stg)
(mk:destroy! 'props)

Convenience syntax

See also Storage and view example.

(require-extension srfi-17 metakit)

(define stg (mk:make 'storage "stg3.mk4"))
(define-view (people stg) (name S) (age I)) ; get or define view

(define p1 (make-people)) ; create empty row
(set! (mk:value name p1) "John") ; fill it with data, old approach
(people-age-set! p1 35) ; use the more convenient way
(mk:insert! people p1) ; store it

; create the row with data and insert it at the start of the view
; use positional and named arguments
(mk:insert! people 0 (make-people "Helga" (age 27)))

; set value in the row #2 (starting from 0)
; the view will grow if needed
(set! (mk:row people 2) (name "Greg"))

; accessors and mutators for the top-level view can accept row index
; as a first argument
(people-age-set! 2 31)

;; R+ and R! usage
(let ([r (mk:make 'row)])
  (R! r '(name "Dietrich"))
  (mk:insert! people (R+ '(age 40) r)) )

(mk:for-each  ; print the contents of the view people
  (lambda (r)
    (display (mk:value age r))
    (display " ")
    (display (mk:value name r))
    (newline))
  people)

(mk:commit! stg) ; commit changes

; cleanup
(gc #t)
(mk:destroy! people)
(mk:destroy! stg)
(mk:destroy! 'props)

Blocked, hashed and join views

(require-extension srfi-17 metakit)

(define stg (mk:make 'storage "stg4.mk4"))


; --- blocked
(define-blocked-view (people stg) (name S) (age I))

; alternately we can make the following:
;  (define people-raw (mk:view stg "_B[people[name:S,age:I]]"))
;  (define people (mk:transform people-raw 'blocked))

; now we can work with "people" as if it was "people[name:S,age:I]".


; --- hashed
(define-hashed-view (people2 stg "ppl2hash") (name S) (age I))
; hashed data will be stored in the "ppl2hash" view of the same storage

; alternately, without syntactic sugar:
;  (define ppl2-raw (mk:view stg "people2[name:S,age:I]"))
;  (define ppl2-hash (mk:view stg "ppl2hash[_H:I,_R:I]"))
;  (define people2 (mk:transform ppl2-raw 'hash ppl2-hash))

; now we can work with "people2" through hash access layer
; e.g., to get age of Jack
(display (mk:item nicknames (mk:find nicknames (name "jack")) age))
; or:
(display (mk:value age (mk:find* nicknames (name "jack"))))

; --- join
(define-view (people3 stg) (name S) (age I) (cityid I))
(define-view (city stg) (cityid I) (name S))

(define ppl-city (mk:transform people3 'join (cityid) city))
; join "people3" and "city" on the "cityid" property