Fractal Render Demo

The first thing to note is that though Newton itself is very lightweight in terms of memory usage, this demo is relatively heavyweight due to the queuing nature of the scatter gather worker pattern. Therefore it is advisable to increase the default memory allocation for the Newton JVM. To do this for all Newton JVM's on a single machine you can set the value:

JVM_ARGS="-Xmx512m"

in "$HOME/.newton/container_env.sh" on unix and "%HOMEDRIVE%%HOMEPATH%\.newton\container_env.bat" on windows.

We will start by constructing a local application, run solely from your development machine. To do this we will firstly startup a Newton container and launch two components; reggie and cds-server.

$ cd $NEWTON_HOME
$ bin/container
...
> installer install etc/instances/reggie.composite
> installer install etc/instances/server-cds.composite

Next we load the demo and jini service bundles into CDS. Note we load the content into the remote "zone" of CDS so that later in our demo when we launch other Newton containers they are able to download the demo and jini service bundles from the server-cds component in this container instead of locally adding the bundles to cds in each container.

> cds scan remote examples/fractal/build/lib
> cds publish remote etc/cds/jini-service-bundles.xml

Having initialised the application environment we can then load the first set of components which make up the fractal render demo.

> installer install examples/fractal/build/etc/gui.composite
> installer install examples/fractal/build/etc/local-engine.composite
> installer install examples/fractal/build/etc/local-worker.composite
> installer install etc/instances/local-blitz.composite

You will then see a window that looks something like the following:

Now press the "Draw" button, you should start to see a mandlebrot fractal appear on screen.

If you start a system monitoring tool you should be able to verify that all communications are taking place locally, i.e. no network traffic - all communications are taking place within the Newton container process.

In the figure below the CPU is maxed out but there is no network traffic.

We will now move worker and space components onto a separate process, which may or may not be on a separate machine. To do this type the following into your the current Newton container console:

> installer uninstall examples/fractal/build/etc/local-engine.composite
> installer uninstall examples/fractal/build/etc/local-worker.composite
> installer uninstall etc/instances/local-blitz.composite

This uninstalls the engine, worker and blitz components from this Newton container. You should then install a remote version of the fractal engine in the current Newton container.

> installer install examples/fractal/build/etc/engine.composite

Here it is worth taking a moment to show the difference between the local and remote engine components. Below are the contents of the files examples/fractal/build/etc/engine-template.xml, examples/fractal/build/etc/local-engine.composite and examples/fractal/build/etc/engine.composite.

examples/fractal/build/etc/engine-template.xml

<?xml version="1.0"?>
<composite name="engineTemplate">
        <description>Space based engine for fractal demo</description>

        <reference name="space" multiplicity="1..1">
                <interface.java interface="net.jini.space.JavaSpace05"/>
        </reference>

        <service name="engine-export">
                <interface.java interface="org.cauldron.newton.example.fractal.api.CalculationEngine"/>
                <binding.osgi />
        </service>

        <component name="engine">
                <reference name="space"/>
                <implementation.java.callback impl="org.cauldron.newton.example.fractal.engine.SpaceCalculationEngine"/>
        </component>

        <wire>
                <source.uri>space</source.uri>
                <target.uri>engine/space</target.uri>
        </wire>

        <wire>
                <source.uri>engine</source.uri>
                <target.uri>engine-export</target.uri>
        </wire>
</composite>

You can see that the engine template (upon which each instance is based) imports a reference to a java class that implements the JavaSpace05 interface. However the template does not define the binding from which to import the reference. This is left up to each instance.

examples/fractal/build/etc/local-engine.composite

<?xml version="1.0"?>
<composite name="fractal-engine">
        <bundle.root bundle="fractal-engine-bundle" version="1.0"/>
        <include name="engineTemplate"/>

        <reference name="space" >
                <binding.osgi/>
        </reference>
</composite>

In the local engine case the reference is binding.osgi i.e. bound to the local OSGi registry. Here the Newton framework cause the engine component to connect to an interface published to the osgi registry. If you look inside the components/jini-starter/resources/templates/blitz-local-template.xml file you will see that this is exactly what this component does.

examples/fractal/build/etc/engine.composite

<?xml version="1.0"?>
<composite name="fractal-engine">
        <bundle.root bundle="fractal-engine-bundle" version="1.0"/>
        <include name="engineTemplate"/>

        <reference name="space" >
                <binding.rmi/>
        </reference>
</composite>

In the remote engine case the reference binding is binding.rmi i.e. services are discovered using the remote service registry and communicate using RMI. Here the engine component connects to a service proxy published to the remote registry with that interface. There are similar differences between the local and remote worker components.

This is extremely useful in a number of different situations, for example:

• For developer testing purposes, the developer can test their application purely locally then with a change to their component definition can deploy a tested component across a newtork with no changes to their code
• For administration purposes, the admin can redefine properties of a remote binding.

In future releases of Newton we aim to support other binding mechanisms including WS*, additionally further binding solutions and configuration options are supported by Paremus's Infiniflow products.

Now we will start one or two other Newton containers. You can do this either on the same machine or a remote machine. However if you are using a remote machine bare in mind each container must be able to resolve multicast traffic between them. See Troubleshooting Multicast if you have problems.

Into these containers we will install remote versions of the worker and space components.

$ bin/container
...
> installer install examples/fractal/build/etc/worker.composite

and (possibly in a second container):

$ bin/container -instance=2
...
> installer install etc/instances/blitz.composite

Now when we re draw the mandlebrot equation you should see that the components are communicating over remote interfaces, as shown from the system monitor screenshot below:

The steps so far show you how to modify the deployment of application components in a network. We will now extend this application with new functionality at runtime, i.e. dynamically add in new functionality to the running application.

If you change tabs to the Equation tab at the top of the gui frame and click on the drop down box in the top left hand corner of this tab - you should see that there is only one equation loaded, the mandlebrot equation.

Having verified this we will now add in a Julia equation into our running application. In the console for the Newton container hosting the gui component enter the following:

> installer install examples/fractal/build/etc/equation-addons.composite

Now if you go to the equations drop-down box you should see that a julia equation has been added, select this, return to the render tab and press draw, you should see the application uses the remote worker to render a Julia fractal image:

In remote applications it is important to be able to use a transaction manager to ensure that data is not lost when remote components fail. In this demo transactions are optionally supported by the fractal worker component.

examples/fractal/build/etc/worker-template.xml

<reference name="transaction" multiplicity="0..1" override="no">
    <interface.java interface="net.jini.core.transaction.server.TransactionManager"/>
    <binding.rmi/>
</reference>

This fragment of the worker composite definition we see that the transaction reference is given a multiplicity of 0..1, i.e. the composite can start whether a transaction manager is found or not.

Launch a mahalo component either in one of the existing containers or in a new one:

> installer install etc/instances/mahalo.composite

It is then possible to demonstrate transactions between the worker and space. If a worker component fails whilst processing data (i.e. the worker component is uninstalled or the process of the Newton container hosting the worker component is killed). Then assuming the space and mahalo components keep running then the transaction will be rolled back and the image will still render in full despite the loss.

> installer uninstall examples/fractal/build/etc/worker.composite

The Newton framework contains a number of tools to manage remote component deployments. One of the most useful is the concept of a System. We will now use these components to build a scalable system of workers across any number of Newton containers. The Newton framework will dynamically detect new containers as they come online and provision workers to them as they are started.

Systems are managed by a collection of components themselves defined via provisioner and system manager systems. These initial systems are deployed on a local container, where as the worker system will be deployed across many (remote) containers.

Firstly pick two containers to host the system manager and provisioner components, these may be new or existing containers. Note the provisioner and system manager may also be installed to the same container - if you do this you only need to install the remote-registy component (below) once:

> installer install etc/instances/remote-registry.composite
> system manage etc/systems/remote-manager.system

> installer install etc/instances/remote-registry.composite
> system manage etc/systems/remote-provisioner.system

These components will discovery remote containers and deploy our worker and space components to them.

Now on each container you wish to add to be added to the worker system you should start a further set of components (defined via a system).

> system manage etc/systems/remote-container.system

This seems to be swapping one manual install (the worker for a second manual install, the boot-container). However it is possible to make the boot container automatically startup in your container by modifying the boot-scripts for that container. Further information on boot scripts can be found here.

Create a directory core/boot-scripts/3 and create a file container.boot within it. Into this file put the following text

# launches a boot-container system that allows remote provisioning of components
system manage etc/systems/remote-container.system

Finally to complete the system manager installation, install a cli to access the boot-system manager into one container.

> installer install etc/instances/remote-manager-cli.composite

You may now uninstall the worker and space instances you deployed earlier, these will be replaced by a worker system which consists of a single space and one worker per container.

> installer uninstall examples/fractal/build/etc/worker.composite
> installer uninstall etc/instances/blitz.composite

In the container you installed the remote-manager-cli component can now type:

remote-system manage examples/fractal/build/etc/worker.system

Now when you render the image you should be able to see that many containers are taking part in the job of calculating work. You can see this for yourself by looking at the Discovery and Statistics tabs of the gui.

You can retire all of the worker and space components from the Newton infrastructure using the command:

remote-system retire examples/fractal/build/etc/worker.system

• Try writing your own equations for the fractal gui. Contributions back to the Newton project are appreciated.
• It is also possible to configure Newton to automatically start the provisioner and system manager components. To do this you will need to modify the boot scripts of the Newton container to automatically start these components. This will be described in a follow on demo Newton Bootstrap Demo.