Content Distribution Service

The CDS (Content Distribution Service) is used to share binary content between separate Newton processes. The initial use that end-users will see CDS put to is to store OSGi bundles so that they can be installed in any Newton container in the Fabric.

Content is stored in CDS under a specific name and an arbitrary set of attributes and values. For example a bundle foo with version 1.0 and provider codecauldron.org could be stored in cds as:

foo,version=1.0,provider=codecauldron.org

Currently content within CDS is stored in various zones. The default configuration supports the idea of a boot and a remote zone: content stored in the boot zone is accessible from the file system local to a Newton process; content stored in the remote zone is stored in a remote file system and is made available to other Newton processes via a proprietary internal protocol.

Content within the CDS may be accessed either via a programatic interface CDService, or via URL's. The CDS is backed by two sub services IndexService and StorageService. IndexService stores light weight (i.e. small amount of binary data) references to content items stored in CDS. StorageService provides the underlying physical storage of content to which the index items point.

The CDService is published to the local osgi registry of the Newton process. It can be used to programattically access and manipulate content within the repository.

package org.cauldron.newton.cds.content.api;


import java.io.IOException;
import java.util.Map;
import java.util.Set;


import org.cauldron.newton.ldap.model.LDAPExpr;


/**
 * The content distribution service is used to distribute content around a distributed system This may include
 * heavyweight files such as large jars. Lightweight metadata is not propagated using the CDS, but rather the index
 * service. This includes cds entry metadata. This is so that the resource view is updated out of band of large file
 * transfers, improving its consistency.
 * 
 * Content entries are uniquely identified a name, some name-value attibutes and associated content bytes.
 * 
 * Content entries exist in a zone. The range over which updates propagate is determined by the zone. For example, the
 * 'boot' zone includes only a single vm, and includes resources necessary to boot the container, and which must be
 * available before the vm has joined the distributed CDS.
 * 
 * Content entries do not have the content bytes embedded - they just provide access to streams for reading and writing
 * content.
 * 
 */
public interface CDService {
    /**
     * The zone which this cdservice provides content for
     */
    String getZone();


    /**
     * 
     * @param name
     *            content name - null is wild
     * @param query
     *            ldap style query over identity attributes
     * @return set of matching entries
     * @throws IOException
     */
    Set<Content> findContent(String name, LDAPExpr query) throws IOException;


    /**
     * look up a specific piece of content by specifying its name and identity attributes exactly. If there is no match
     * a new Content object with these parameters is created. However it is not persisted the the CDS until some content
     * bytes are written to it.
     * 
     * @param name
     * @param exactMatchAttributes
     * @return the single matching entry.
     * @throws IOException
     */
    Content getContent(String name, Map<String, String> exactMatchAttributes) throws IOException;


    /**
     * 
     * @param url
     * @return A content entry matching this url
     * @throws IOException
     */
    Content urlToContent(String url) throws IOException;


    /**
     * removes all content entries with the given name (null wild) and matching the query
     * 
     * @param name
     * @param query
     * @throws IOException
     */
    void deleteAll(String name, LDAPExpr query) throws IOException;


    /**
     * adds a listener for changes to content matching the supplied name and filter
     * 
     * @param listener
     * @param name
     * @param filter
     */
    void addContentListener(ContentListener listener, String name, LDAPExpr filter);


    /**
     * removes a listener for changes to content matching the supplied name and filter
     * 
     * @param listener
     */
    void removeContentListener(ContentListener listener);
}

Alternatively within a Newton process you can access content from CDS via java.net.URL's of the form:

cds://contentname?zone=zname&attr1=val1&attr2=val2

This allows the developer to read content from cds but not to write or delete it. To write or delete content you must use the CDService API.

The zone concept is a temporary pattern we've used for the time being but in and ideal world we'd like to get rid of it. It's purpose is to specify where content is saved when it is written to cds.

However knowing where the content is stored is actually non beneficial to a range of patterns. For example it would be useful to access content via url in a more “natural” form: cds://name?attr=val vs having the artificial zone= attribute tagged onto the name. This trips us up when we come to load content into cds that has dependent files as all the dependent files need to be updated with the zone on write (not very elegant).

Problems with removing this come when you consider issues of consistency across multiple repositories, i.e. if content is stored in two areas, local file system and remote file system which content is being read/written to for a given set of name attributes?

Anyone with answers or interested in further discussion is encouraged to contact us via our dev mailing list.