On Screen Handles

Introduction

On screen handles allow you to directly interact with objects in a 3D scene in order to manipulate them in different ways (e.g., to adjust the scale or position of a selected object). This form of user input can provide a more intuitive means of interacting with your visualisation than is offered by more traditional UI controls.

We have developed a simple framework which can be added to an Omegalib script in order to implement support for interactive on-screen handles. In this tutorial we will be taking a look at a simple example which demonstrates how to make use of this framework.

daHandles

The daHandles Omegalib module includes support for the following features and functionality:

• Reusable abstractions for different types of on-screen handle, including handle groups
• Customisable on-screen handle appearance and geometry (e.g., you can create your own handle in a 3D package such as Houdini and export it for use within daHandles)
• Configurable selection manager, which is able to track independent selections for each cursor in the environment

The module includes some example handles (and handle groups), which you may reuse in your Omegalib scripts, or as examples to guide the development of new controls:

• A simple ‘whisker’ handle supporting customisable geometry, which may be used as a building block for the implementation of more complex handle control groups
• A simple ‘tri-axis’ handle group, for manipulating objects in 3D space
• Individual example scale, rotate and translate handle groups
• A more complex transform handle group, which supports multiple operations depending on the selected mode

The example we will look at next demonstrates how to use the majority of these features. As you’ll see, although the script itself is simple and concise, the functionality it provides is very powerful, thanks to the features of the daHandles module.

Example

The example we will look at in this tutorial renders as a simple scene containing two boxes which may be manipulated using on screen handles. The handles are hidden by default. When you use the mouse cursor to hover over and click on one of the boxes, the handles are displayed, in this case, a tri-axis handle control group, as shown in the video below:

In order to manipulate the selected box, press and hold the key corresponding to your desired operation on the keyboard, then, using the cursor to select the desired handle, press the left mouse button and drag. You will need to either drag the mouse from side to side, or top to bottom depending on the axis of the selected handle. To hide the handles once you have finished interacting with the box, click on the background of the scene.

The supported operations, and the corresponding key needed to select each one is as follows:

• t : translate the box along the selected axis
• s : scale the box in both directions along the selected axis
• r : rotate the box around the selected axis

Let’s now take a look at the Omegalib script which was used to implement this example. You can find a copy of this script (manipulator.py) in the examples directory on the DAVM under /local/examples/handles:

import math
import os

from cyclops import *
from daInput import *
from daHandles import *

if __name__ == '__main__':
    """
    This example demonstrates how to setup a scene containing objects which may
    be manipulated using on-screen handles.   It show show to use many of the
    features which are provided by the daHandles library, including:

        - How to use builders to create control geometry, controls and control groups
        - How to specify scene nodes and attach controls to them
        - How to set up a selection manager to process user interactions via controls
    """

    path = os.path.dirname(__file__)
    if not path:
        path = os.getcwd()

    resources = os.path.join(path, 'resources')

    ui_context = UiContext()

    getDefaultCamera().setControllerEnabled(False)

    geometry_builder = CustomControlGeometryBuilder()
    geometry_builder.set_name('handle')
    geometry_builder.set_path(os.path.join(resources, 'handle.obj'))

    control_builder = WhiskerControlBuilder()
    control_builder.set_geometry_builder(geometry_builder)

    transform_builder = TransformControlGroupBuilder()
    transform_builder.set_ui_context(ui_context)
    transform_builder.set_control_builder(control_builder)

    geo1 = BoxShape.create(1, 1, 1)
    geo1.setEffect('colored -d white')
    bbox1 = (geo1.getBoundMinimum(), geo1.getBoundMaximum())

    box1 = ControllableSceneNode('box1', geo1)
    box1.add_control(transform_builder.set_parent(box1).set_bounding_box(bbox1).build())
    box1.node.setPosition(Vector3(-1.5, 1.5, -10))
    box1.node.rotate(Vector3(0, 1, 0), math.radians(-45), Space.Local)

    geo2 = BoxShape.create(1, 1, 1)
    geo2.setEffect('colored -d white')
    bbox2 = (geo2.getBoundMinimum(), geo2.getBoundMaximum())

    box2 = ControllableSceneNode('box2', geo2)
    box2.add_control(transform_builder.set_parent(box2).set_bounding_box(bbox2).build())
    box2.node.setPosition(Vector3(1.5, 1.5, -10))
    box2.node.rotate(Vector3(0, 1, 0), math.radians(-45), Space.Local)

    light = Light.create()
    light.setEnabled(True)
    light.setPosition(Vector3(0, 5, -2))
    light.setColor(Color(1.0, 1.0, 1.0, 1.0))
    light.setAmbient(Color(0.1, 0.1, 0.1, 1.0))

    manager = SelectionManager(ui_context)
    manager.add(box1)
    manager.add(box2)

    def on_event():
        manager.on_event()

    setEventFunction(on_event)

As you can see, the Omegalib script is relatively simple – most of the complexity is hidden away inside the daHandles framework. Lets break this script down and look at each part in turn.

Firstly, we declare a UiContext instance. The context is able to keep track of any custom cursors which may have been added to the scene in addition to the default mouse pointer (we don’t use custom cursor objects in this example). We also disable the navigation controls on the default camera, as we require a fixed camera:

getDefaultCamera().setControllerEnabled(False)

The handles featured in this example use a custom geometry object as their representation in the scene. We load this geometry using a CustomControlGeometryBuilder. This builder allows us to make use of custom handle geometry we have defined in an external 3D editor, such as Houdini, as the graphical representation of the associated handle(s) in our scene. If no custom geometry was available, we could also make use of the CylinderControlGeometryBuilder defined for us within daHandles as an alternative:

geometry_builder = CustomControlGeometryBuilder()
geometry_builder.set_name('handle')
geometry_builder.set_path(os.path.join(resources, 'handle.obj'))

We pass the geometry builder into the control builder we’ve specified to construct the handles in our scene. In this case, we want each of our handles to behave as a ‘whisker’ control, so we use the WhiskerControlBuilder to define them:

control_builder = WhiskerControlBuilder()
control_builder.set_geometry_builder(geometry_builder)

The individual ‘whisker’ controls do not appear in isolation however, rather they will belong to a more powerful control group which provides support for 3D transformations. As before, we pass the ‘whisker’ control builder into the control group builder we’ve defined for the transform controls:

transform_builder = TransformControlGroupBuilder()
transform_builder.set_ui_context(ui_context)
transform_builder.set_control_builder(control_builder)

This pattern (dependency injection) allows us to easily substitute the implementation of different controls into control groups (or geometry into controls) without affecting the underlying implementation of the containing control group (or control).

Now that the control builders have been defined, we are able to start making use of them. The first step is to define some geometry that we wish to interact with, and attach this to a ControllableSceneNode. We use the transform control group builder we defined earlier to generate an instance of the TransformControlGroup and attach it to the scene node. We then specify an initial position and orientation for the node. These steps are repeated for each of the controllable objects in the scene (i.e., the two boxes – we just show the code fragment for the first box below):

geo1 = BoxShape.create(1, 1, 1)
geo1.setEffect('colored -d white')
bbox1 = (geo1.getBoundMinimum(), geo1.getBoundMaximum())

box1 = ControllableSceneNode('box1', geo1)
box1.add_control(transform_builder.set_parent(box1).set_bounding_box(bbox1).build())
box1.node.setPosition(Vector3(-1.5, 1.5, -10))
box1.node.rotate(Vector3(0, 1, 0), math.radians(-45), Space.Local)

Next, we add a light to illuminate the scene, allowing us to see the objects we have just defined (along with their controls, when selected):

light = Light.create()
light.setEnabled(True)
light.setPosition(Vector3(0, 5, -2))
light.setColor(Color(1.0, 1.0, 1.0, 1.0))
light.setAmbient(Color(0.1, 0.1, 0.1, 1.0))

The final step is to define a SelectionManager. The manager keeps track of all of the objects in the scene, the cursors which have been defined, and which object (if any) has been selected by each of the cursors. We need to add each of the controllable scene nodes to the manager so it is aware that they exist, and then register the manager’s on_event method as a callback so that Omegalib will invoke it in response to any input events generated by attached user input devices:

manager = SelectionManager(ui_context)
manager.add(box1)
manager.add(box2)

def on_event():
		manager.on_event()

setEventFunction(on_event)

Where to Next?

If you have a copy of the DAVM installed, you can try the example out for yourself by running the following commands in a terminal window:

$ cd /local/examples/handles
$ orun manipulator.py

Houdini users should also take a look at our Houdini tutorial !!!series, which includes a tutorial showing how to import geometry from a Houdini asset to use as the visual representation for a control, and an example that demonstrates how to connect controls to parameters exposed on a Houdini asset in order to manipulate the underlying asset geometry.

If you would like to dig deeper and learn more about how the daHandles Omegalib python module works, or contribute updates of your own, please refer to the repository on GitHub, where you will be able to find all of the code:

Some possible future enhancements you may like to try to add include:

• Removing reliance on the keyboard for interaction mode selection by adding more handles, so there are separate controls for translate, rotate and scale operations
• Adding more sophisticated handles that behave differently depending on what part of the handle geometry is selected
• Adjusting handle behaviour based on which cursor is currently interacting with the object (e.g., the user would interact with the object using a different mocap cursor, depending on whether the intention is to translate, rotate or scale the object)