Interaction Rectangle
Table of Contents
 | Interaction Rectangle | Rectangular geometry collecting and sending interaction events via UDP or ANC channel |
The Interaction Rectangle node renders a rectangle geometry, collects all interaction (mouse, touch, keyboard) and sends this data in an Open Sound Control based protocol via UDP or the VIO ANC channel for further processing by the Live Video texture provider.
Node Usage
This node was basically introduced to provide the possibility of interaction with a Live Video texture source. Possible scenarios could be:
- A VIO implementation in a game engine providing the render output and giving the possibility to interact with the game engine scene
- Another VIO application could be the interaction with a Virtual Machine like Virtual Box (Remote Desktop from within Ventuz)
- A simpler 'Virtual Machine' application could be accomplished by using a Video Input card to grab the content of another computer; this computer would require a service which receives the interaction data via UDP and injects mouse, keyboard and touch commands
All these applications of course require software development and custom modifications on side of the 'video texture provider'.
Inputs
The Interaction Rectangle provides some input properties to control the behavior of the node.
- Interaction Receiver: You define if the interaction data should be sent via UDP or the ANCillary Channel of the Live Video Input
- Receiver ID: Specifys the receiver of the interaction messages. In ANC mode the ReceiverID must match the Ordinal (index) of the Live Video input.
- IP and Port: Define where to send via UDP
- Screen Aligned: Toggles the geometry size between 1x1 and full layer.
- Interaction: Defines which types of interaction events (Keyboard, Mouse, Touch) should be sent. If you disable the node via the Enabled property, it does not react to any interaction and does not send any data
- Touch Group Filter: property which all Interaction nodes have in common is described here
- Interaction: Mouse and touch events are only sent if they were generated inside the boundaries of the Interaction Rectangle geometry. The position is always mapped to 0-1, 0-1. Keyboard events are always sent if the renderer window has the focus and Keyboard is enabled in the Interaction property
The Mouse and Multi-Touch events can only be interpreted correctly if the Interaction Rectangle is not rotated at all!
The node is only sending if the Cluster Networking is configured to Single or Master!
Tracking Mode:
- All: Tracks all mouse and touch events in the 'air space' of the node even if other interaction nodes are covering parts of it and all keyboard events even if mouse or touch are outside the 'air space'.
- NonCaptured: Tracks all interaction events that are not registered elsewhere by any other interaction nodes (covering the Interaction Rectangle). For remoting other applications or using scripts, it can be interesting to just track touches that are not used by any of Ventuz's interaction nodes.
- NonCapturedWithAllKeys: Like the NonCaptured mode but with the exception that all keyboard events are tracked.
Outputs
Via the Custom Verbs of the Property Editor it is possible to add output property sets which provide information about active Mouse and Multi-Touch events. A description of the fixed output properties and events, that all Interaction nodes have in common, can be found here.
Message Format
The interaction events collected by the node are formatted as OSC bundles. For details about the OSC format see the Remoting 4 Protocol Specification. The message format may look complicated. The reason for this is that it is based on our internal Input Subsystem format that is used for input synchronisation in a multi-machine Cluster.
The message structure looks like follows:
OscBundle Address Arguments Array OscElement /ventuz/<i_ReceiverID>/InputTimecode <d_Timecode> OscElement[n] /ventuz/<i_ReceiverID>/<s_DeviceType>/<i_DeviceID>/<s_ID>/<i_TouchGroup> <i_TouchType>, <s_TouchID>, <f_X>, <f_Y>, <f_Z>, <f_Pressure>, <f_Angle> - <i_Value> <f_X>, <f_Y>, <f_Z>
The OscBundle contains at least two OscElements. The first OscElement carries the timecode message. Its address is /ventuz/<i_ReceiverID>/InputTimecode where <i_ReceiverID> is an integer number representing the receiver ID. The argument carried by this OscElement is a double value representing the timecode of the frame in which the interaction data was recorded. All subsequent OscElements carry interaction data; the address is always /ventuz/<i_ReceiverID>/<s_DeviceType>/<i_DeviceID>/<s_ID>/<i_TouchGroup>.
The address parameters have following meaning and content:
| Address Parameter | Meaning | Possible Values |
| <i_ReceiverID> | identifies the receiver of the OscElement | integer >= 0 |
| <s_DeviceType> | string specifying the type of the interaction data | MultiTouch, Mouse, Keyboard |
| <i_DeviceID> | ID of the interaction device this information was created by | any integer |
| <s_ID> | string ID representing the type of input information | ID_TouchAdd, ID_TouchUpdate, ID_TouchRemove, ID_MouseButtonDown, ID_MousePosition, ID_MouseButtonUp, ID_MouseWheel, ID_MouseHover, ID_MouseDoubleClick, ID_MouseDoubleClick2 |
| <i_TouchGroup> | integer value representing a bitmask for four different touch groups | integer 0-31 |
The different argument types and values are explained here:
| Argument Parameter | Meaning | Possible Values |
| <d_Timecode> | timecode of the interaction data | double >= 0.0 |
| <i_TouchType> | type of touch | 0 = Cursor, 1 = Marker, 2 = VR |
| <s_TouchID> | string representing numeric ID of the touch (e.g. finger) | integer >= 0 |
| <f_X> | X position of touch or mouse | float in range [0.0; 1.0] |
| <f_Y> | Y position of touch or mouse | float in range [0.0; 1.0] |
| <f_Z> | Z position of touch or mouse | float in range [0.0; 1.0] |
| <f_Pressure> | pressure value of the touch | float in range [-1.0; 1.0] |
| <f_Angle> | rotation angle of a touch marker | float in range [0.0; 360.0] |
| <i_Value> | mouse button or key code | integer |
You can use the Ventuz OSC library to receive the interaction data via UDP.
To receive the interaction events in the VIO API use the vAncFromVentuz() method with ITAC as FourCC code.