File-Like ICN Collection (FLIC)University of Baselchristian.tschudin@unibas.chPARC, Inc.christopher.wood@parc.com
General
ICNRG Working GroupInternet-DraftThis document describes a bare bones “index table”-approach for
organizing a set of ICN data objects into a large, File-Like
ICN Collection (FLIC).At the core of this collection is a so called manifest which acts as
the collection’s root node. The manifest contains an index table with
pointers, each pointer being a hash value pointing to either a final
data block or another index table node.One figureCopy the proven UNIX inode concept:
index tables and memory pointersAdaption to ICN:
hash values instead of block numbers, unique with high probabilityAdvantages (over non-manifest collections):
single root manifest signature covers all elements of the full
collection, including intermediate sub manifestseliminate reference to chunk numbering schemata (hash values only)supports block-level deduplification (can lead to a directed acyclic
graph, or DAG, instead of a tree)Limitations
All data leafs must be present at manifest creation time
(otherwise one cannot compute the pointers)Potential extensions (for study):
Enhance the manifest such that it can serve as a “database
cursor” or as a cursor over a time series, e.g. having entries
for “previous” and “next” collections.We first give the FLIC format in EBN notation:Description:The core of a manifest is the sequence of “hash groups”.A HashGroup (HG) consists of a sequence of “sized” data or manifest pointers.A BlockHashGroup (BHG) consists of a sequence of data or manifest pointers and
a mandatory field that lists the total size of each pointer. These HashGroups
should be used when each pointer (except the last) contains an identical number
of application bytes.Sizes are 64-bit unsigned integers.Data and manifest pointers are cryptographic HashValues encoded according
to the mechanism listed in . Specifically, a HashValue
specifies the cryptographic hash algorithm and the actual digest.A HashGroup can contain a metadata section to help a reader to
optimize content retrieval (block size of leaf nodes, total size,
overall digest etc).None of the ICN objects used in FLIC are allowed to be chunked,
including the (sub-) manifests. The smallest possible complete
manifest contains one HashGroup with one pointer to an ICN object.FLIC’s tree data structure is a generalized index table as it is known
from file systems. The pointers, which in an OS typically are hard disk
block numbers, are replaced by hash values of other ICN objects. These
ICN objects contain either other manifest nodes, or leaf nodes. Leafs
contain the actual data of the collection. Each pointer explicitly indicates
the amount of application data bytes contained by the referred object. For example,
the size of a data pointer (to a leaf) represents the size of the leaf’s
content object payload. Conversely, the size of a manifest pointer represents
the total size of all pointers contained in that manifest.FLIC makes use of “nameless ICN object” where the network is tasked
with fetching an object based on its digest only. The interest for
such an object consists of a routing hint (locator) plus the given
digest value.Starting from the original content, the corresponding byte array is
sliced into chunks. Each chunk is encoded as
a data object, according the ICN suite. For each resulting data
object, the hash value is computed. Groups of consecutive objects are
formed and the corresponding hash values collected in manifests, which
are also encoded. The hash values of the manifest objects replace the
hash values of the covered leaf nodes, thus reducing the number of
hash values. This process of hash value collection and replacement is
repeated until only one (root) manifest is left.Of special interest are “skewed trees” where a pointer to a manifest
may only appear as last pointer of (sub-) manifests. Such a tree
becomes a sequential list of manifests with a maximum of datapointers
per manifest packet. Beside the tree shape we also show this data
structure in form of packet content where D stands for a data pointer
and M is the hash of a manifest packet.A pseudo code description for producing a skewed tree follows below.Obtaining with each manifest a maximum of data pointers is beneficial
for keeping the download pipeline filled. On the other hand, this tree
doesn’t support well random access to arbitrary byte positions: All
data pointers coming before that offset have to be fetched before
locating the block of interest. For random access, binary trees (where
both subtrees of a node cover half of the content bytes) are better
suited. This can be combined with the “skewed tree” approach:
Manifests of intermediate nodes are filled with data pointers except
for the last two slots. The second last slot points to a manifest for
the “first half” of the left content, the last slots then points to a
manifest for the rest.This can be generalized to k-ary trees by allocating k pointers per manifest
instead of 2.To fetch the data associated with a given FLIC (sub-) manifest, the
receiver sequentially works through all entries found in the
HashGroups and issues corresponding hash-based interests. In case of a
data hash pointer, the received content object is appended. In case of
a manifest hash pointer, this procedure is called recursively for the
received manifest. In other words, the collection data is represented
as the concatenation of data leaves from this pre-order depth-first search
(DFS) traversal strategy of the manifest tree. (Currently, pre-order DFS is
the only supported traversal strategy.) This procedure works regardless of
the tree’s shape.A pseudo code description for fetching is below.The above DFS code works for FLIC manifest trees of arbitrary
shape. In case of a skewed tree, no recursion is needed and a single
instance of the DFS procedure suffices (i.e., one uses tail recursion).In FLIC, metadata is linked to HashGroups and permits to inform the
FLIC retriever about properties of the data that is covered by this hash
group. Examples are overall data bytes or the overall hash digest (this is akin
to a Merkle hash). The intent of such metadata is to enable an in-network retriever
to optimize its operation - other attributes linked to the collection as a whole
(author, copyright, etc.) is out of scope.The list of available metadata is below.BlockHashGroups contain a mandatory piece of metadata called the SizePerPtr.
This value indicates the total number of application bytes contained within
each pointer in the hash group except for the last pointer. Normal HashGroups
do not require this piece of metadata; Instead, each pointer includes their
size explicitly.The optional name of a manifest is a mere decoration and has no locator
functionality at all: All objects pointed to by a manifest are
retrieved from the location where the manifest itself was obtained
from (which is not necessarily its name). Example:In this example, the name “/a/b/c” does NOT override
“/the/locator/hint” i.e., after having obtained the manifest,
the retriever will issue requests forUsing the locator metadata entry, this behavior can be changed:The FLIC mechanics has uses cases beyond keeping together a set of
data objects, such as: seeking, block-level de-duplification, re-publishing
under a new name, growing ICN collections, and supporting FLICs with different
block sizes.Fast seeking (without having to sequentially fetch all content)
works by skipping over entries for which we know their size.
The following expression shows how to compute the byte offset of the data
pointed at by pointer P_i, offset_i. In this formula, let P_i represent
the Size value of the i-th pointer.With this offset, seeking is done as follows:Seeking in a BlockHashGroup is different since offsets can be quickly computed.
This is because the size of each pointer P_i except the last is equal to the SizePerPtr
value. For a BlockHashGroup with N pointers, OverallByteCount D, and SizePerPointer L,
the size of P_i is equal to the following:In a BlockHashGroup with k pointers, the size of P_k is equal to:Using these, the seeking algorithm can be thus simplified to the following:Note: In both cases, if the pointer at position i is a manifest pointer, this
algorithm has to be called once more, seeking to seek_pos o inside that manifest.Consider a huge file, e.g. an ISO image of a DVD or program in binary
form, that had previously been FLIC-ed but now needs to be patched.
In this case, all existing encoded ICN chunks can remain in the
repository while only the chunks for the patch itself is added to a
new manifest data structure, as is shown in the picture below. For
example, the
venti archival
file system of Plan9 uses this technique.A log file, for example, grows over time. Instead of having to re-FLIC
the grown file it suffices to construct a new manifest with a manifest
pointer to the old root manifest plus the sequence of data hash
pointers for the new data (or additional sub-manifests if necessary).
Note that this tree will not be skewed (anymore).It can happen that a publisher’s namespace is part of a service
provider’s prefix. When switching provider, the publisher may want to
republish the old data under a new name. This can easily be achieved
with a single nameless root manifest for the large FLIC plus arbitrarily many
per-name manifests (which are signed by whomever wants to publish this
data):Note that the hash computation (of h) only requires reading the
nameless root manifest, not the entire FLIC.This example points out the problem of HashGroups having locator
metadata elements: A retriever would be urged to follow these hints
which are “hardcoded” deep inside the FLIC but might have become
outdated. We therefore recommend to name FLIC manifests only at the
highest level (where these names have no locator function). Child
nodes in a FLIC manifest should not be named as these names serve no
purpose except retrieving a sub-tree’s manifest by name, if would be
required.If chunks do not have regular (block) sizes, the HashGroup can be used
to still convey to a reader the length of the chunks at the manifest
level. (This can be computed based on the size of pointers, but the metadata
field makes this determination simpler.) Example use cases would be chunks each
carrying a single ASCII line as entered by a user or a database with variable
length records mapped to chunks.We express the packet encoding of manifests in a symbolic expression
style in order to show the TLV structure and the chosen type
values. In this notation, a TLV’s type is a combination of
“SymbolicName/Tvalue”, Length is not shown and Values are
sub-expressions. Moreover, we populate the data structure with all
possible entries and omit repetition.Interest: name is locator, use objHashRestriction as selector.The assigned NDN content type value for FLIC manifests is 1024 (0x400).Interest: name is locator, use implicitDigest name component as selector.None.CCNx Messages in TLV Format