Navigation

Ventuz Introduction

  • Introduction
  • Getting Started
  • Ventuz Editions
  • Ventuz Products
  • Realtime Rendering
  • Frequently Asked Questions
  • Common Mistakes
  • Deploying a Ventuz Presentation
  • Scene Performance and Tweaks

Quick Guides

  • Quick Guide Index
  • Business Logic
  • 3D Art
  • 2D Art
  • Programming
  • System Engineer

General

  • Index
  • What's New
  • Ventuz System Requirements
  • Communication Protocol Overview
  • Configuration Editor
  • Audio / Video Configuration
  • Machine Configuration
  • Web Configuration Editor and License Manager
  • GPI Configuration for Runtime or Director
  • Supported Formats
  • Supported Hardware
  • Multisampling / Anti-Aliasing
  • Input Subsystem
  • Ventuz Proprietary Files
  • Migrating Content to Ventuz 6
  • Migrating Content to Ventuz 5
  • Summary Shortcuts
  • Terminology
  • Manual Index

Ventuz Designer

  • Designer Indices
Introduction
  • Designer Introduction Index
  • Designer Overview
  • Realtime Rendering
  • Project Browser
  • Designer Interface
  • Designer Options
  • Working with Nodes
  • Hierarchy and Content Editors
  • 2D Workflow
  • 3D Workflow
  • Animation Workflow
  • Best Practices
  • Reading Data in Ventuz
  • Display Images and Movies
  • Scene Performance and Tweaks
  • Deploying a Ventuz Presentation
  • Render to Disk
User Interface
  • Designer User Interface Index
  • Designer Interface
  • Renderer Window
  • Layer Editor
  • Property Editor
  • Property Groups
  • Hierarchy Editor
  • Content Editor
  • Find and Replace
  • Toolbox
  • Animation Editor
  • Shader Editor
  • Text Editor
  • Message View
  • Scene Tree
  • Stage Editor
  • Container Outline
  • Watches Editor
  • Performance Statistics
2D Workflow
  • 2D Workflow Index
  • 2D Workflow
  • Layer Editor
  • Common Layer Properties
  • IPP Effects
  • Color Correction FX
  • Distortion FX
  • Filter FX
  • Hierarchy and Content Editors
  • Display Images and Movies
3D Workflow
  • 3D Workflow Index
  • 3D Workflow
  • Hierarchy and Content Editors
  • Renderer Window
  • Camera Navigation
  • Manipulate Objects with Gizmos
  • Layer Editor
  • Property Editor
  • Hierarchy Editor
  • Working with Nodes
  • Isolated Objects
  • Containers
  • Text Rendering
  • Character Sets
  • Geometry Import
  • Display Images and Movies
  • Particle System
  • Creating Realistic Reflections
  • Unreal Integration
  • Notch Integration
  • E2E Node Overview
Logic Workflow
  • Logic Workflow Index
  • Hierarchy and Content Editors
  • Content Editor
  • Hierarchy Editor
  • Working with Nodes
  • Property Editor
  • Containers
  • Project and Scene Data
  • Reading Data in Ventuz
  • Display Images and Movies
  • Input Subsystem
  • Multitouch
  • TUIO Protocol
  • Open Sound Control
  • Unreal Integration
  • Notch Integration
  • E2E Node Overview
Animation Workflow
  • Animation Workflow Index
  • Animation Workflow
  • Animation Editor
  • Content Editor
  • Hierarchy Editor
  • Property Editor
  • Animation and State Engine
  • Templates
  • Template Engine
  • Unreal Integration
  • Notch Integration
Project Structure
  • Project Structure Index
  • Annotations
  • Projects and Scenes
  • Project Properties
  • Project Maintenance
  • Project and Scene Data
  • Scene Management
  • Scene Statistics
  • Scene Tree
  • Performance Statistics
How Tos
  • Designer How to Index
  • How to Run Ventuz
  • How to Work with Designer
  • Ventuz Designer Drag&Drop Workflow
  • How to work with Shadows
  • How to Build Content for Multiple Screens
  • How to Use Emoijs
  • How to Build a Template
  • How to Use the Color Difference Keyer
  • How To Use the HDR Engine
  • How Create Lens Flares and Bloom
  • How to Create Visuals Loader Node
  • How to Remote Control with a Phone
  • How to use Head Mounted Displays
  • How to work with 3D Reference Layers
  • How to create a Firework Particle System
  • How to use DDS with new Block Compression modes
  • How to use the Substance Integration
  • How To Integrate Unreal
  • How To Integrate Notch
  • How To use the Vertex Integration
  • How To Control and Customize Ventuz
Reference
  • Available Nodes
  • Animation Nodes
  • Material&Color Nodes
  • Data Nodes
  • E2E Nodes
  • Geometry Nodes
  • Interaction Nodes
  • IO Nodes
  • Layers
  • Light Nodes
  • Logic Nodes
  • Render Option Nodes
  • Slides Nodes
  • Sound Nodes
  • Text Nodes
  • Texture Nodes
  • VR Nodes
  • World Nodes
  • Summary Shortcuts
  • Layer Editor Shortcuts
  • Hierarchy Editor Shortcuts
  • Content Editor Shortcuts
  • Animation Editor Shortcuts
  • Director Shortcuts

Ventuz Director

  • Index
  • Introduction
  • Environment
  • Show
  • User Interface
  • Assets
  • Taking Action
  • Property Editor
  • Shot Box
  • Project Data
  • Pages
  • Playlist
  • Timeline
  • Content References
  • Topology
  • Channels
  • Macros
  • Designing Templates
  • Plug-Ins
  • Shortcuts
  • Command Line Options
  • Application Settings
  • Glossary
  • GPI Configuration

Ventuz Runtime & Configuration

  • Runtime Index
  • Configuration Configuration Editor
  • Machine Configuration
  • Video/Audio Configuration
  • Web Configuration Editor and License Manager
  • Render Setup Editor
  • Warping and Soft-Edging Editor
  • Machine Clustering
  • Supported Hardware
  • Director Mode
  • Runtime How Tos Index
  • How to Configure Audio
  • How to Use Live Options
  • How To Play Out On Multiple Screens
  • How To Render on a Machine Cluster
  • How to Use Head Mounted Displays
  • How to Setup Spout with Ventuz
  • How to Use Newtek NDI
  • How to Use a Mixed Frame Rate Cluster
  • How to Use Tracking

How To

Designer
  • Designer How to Index
  • How to Run Ventuz
  • How to Work with Designer
  • Ventuz Designer Drag&Drop Workflow
  • How to work with Shadows
  • How to Build Content for Multiple Screens
  • How to Use Emoijs
  • How to Build a Template
  • How to Use the Color Difference Keyer
  • How To Use the HDR Engine
  • How Create Lens Flares and Bloom
  • How to Create Visuals Loader Node
  • How to Remote Control with a Phone
  • How to use Head Mounted Displays
  • How to work with 3D Reference Layers
  • How to create a Firework Particle System
  • How to use DDS with new Block Compression modes
  • How to use the Substance Integration
  • How To Integrate Unreal
  • How To Integrate Notch
  • How To build and playback Ventuz Content in Vertex
Runtime & Configuration
  • Runtime How Tos Index
  • How to Configure Audio
  • How to Use Live Options
  • How To Play Out On Multiple Screens
  • How To Render on a Machine Cluster
  • How to use Head Mounted Displays
  • How to setup Spout with Ventuz
  • How to use Newtek NDI
  • How to use a Mixed Frame Rate Cluster
  • How to use Tracking
  • How To Integrate Unreal
  • How To Integrate Notch
  • How To build and playback Ventuz Content in Vertex
Director
  • How To Control Multiple Graphics Independently From Each Other
  • How to use the Companion with Director

Ventuz Node Reference

ANIMATION
  • Mover
  • Alternator
  • Simple Control
  • Timeline Control
  • Anmation Rig
  • Keyframe Animation
  • Animation Group
COLOR/MATERIAL
  • Alpha
  • Fog
  • Ground Fog
  • Sky Box
  • Color to RGBA
  • HSLA to Color
  • RGBA to Color
  • Color Transformer
  • HLSL Shader
  • Color
  • Material
  • Color Picker
  • Substance Material
DATA
  • Database
  • Excel
  • JSON
  • RSS Feed
  • Resource Linker
  • Text File
  • XML
E2E
  • E2E Axis
  • E2E Data
  • E2E Control
  • E2E Layer
  • E2E Provider
  • E2E Node Overview
GEOMETRY
  • Rectangle
  • Rounded Rectangle
  • Gradient Rectangle
  • Overlay Rectangle
  • Cube
  • Circle
  • Sphere
  • Cylinder
  • Cone
  • Torus
  • Chart
  • Random Points
  • Mesh Loader
  • Geometry Import (Live)
  • Volume
  • Get Bounding Box
  • Arrow
  • Particle System
  • Path Renderer
  • Geometry Renderer
INTERACTION
  • Interaction Rect
  • Touch Button
  • Touch Excluder
  • Touch Marker
  • Touch Paint
  • Touch Pattern
  • Touch Proxy
  • Touch Ripples
  • Touch Transformations
  • Web Browser
  • Touch Teleport
  • Touch Simulator
INPUT/OUTPUT (I/O)
  • GPI
  • Joystick
  • Keyboard
  • MIDI
  • Mouse
  • Network
  • Open Sound Control
  • Serial
  • Timecode
  • DMX
  • HTTP
  • RamDiskWriter
LAYER
  • 3D Layers
  • 3D Layer Reference
  • 2D Layers
  • PSD Import Layer
  • E2E Layer
  • Others
LIGHT
  • Light Sources
LOGIC
  • Array Processing
  • Convert To Text
  • Cluster Synchronization
  • Counter
  • Date Time
  • Directory
  • Dispatcher
  • Enumeration
  • Expressions
  • Invert
  • Log
  • Loop Breaker
  • Math Effects
  • Matrix Operations
  • Scene Event
  • Script
  • String Operations
  • System ID
  • Text Splitter
  • Timer
  • Toggle
  • URL
  • Value Switch
  • Value Buffer
  • Variables
  • Visual Indexer
RENDER OPTIONS
  • Alpha Blending
  • Color Write
  • Alpha Testing
  • Clip Plane
  • Filter
  • Mask
  • Mirror
  • Effect
  • Render Cube Map
  • Draw Modes
  • Stencil
  • ZTesting
SOUND
  • Audio Clip
  • Sound
  • Volume Control
  • Audio Analysis
SLIDES
  • Slide Manager
  • Slide
  • Slide Port
  • Pivot
TEXT
  • Text Effects
  • Text Layouts
  • Text Rendering
TEXTURE
  • Background
  • Hatch
  • Image
  • Texture
  • SVG Loader
  • Gradient Texture
  • Live Video
  • Movie Stream
  • Movie Frame
  • Movie Clip
  • Texture Loader
  • Snapshot
  • Snapshot Framebuffer
  • Texture Saver
  • Video Source Selector
  • VIO Input
  • Spout Receiver
  • NDI Receiver
  • Substance Loader
  • QR Code
VR/AR
  • Tracked Devices
  • Draw Tracked Devices
WORLD
  • Axis
  • Billboard
  • GetWorld
  • SetWorld
  • Arrange
  • Ticker
  • Layout
  • Group
  • World Z Sort
  • YesNo
  • Switch
  • Spread
  • Filter Pass
  • Set Pass
  • Hierarchy Container
  • Scene Port
  • Content Container
  • Template Port
  • Container Info
  • Camera
  • Paths

Advanced and Development

  • Advanced and Development Index
  • Command Line Options
  • Ventuz IP Ports
  • Ventuz Machine Service
  • TUIO
  • .NET Scripting
  • HLSL Shader Programming
  • Ventuz API and SDK
  • Ventuz Extension API
  • Ventuz VIO API
  • Ventuz File Format (VFF)
  • Ventuz Stream Out API
  • Lens Calibration File for FreeD
  • E2E Node Overview
  • Unreal Integration
  • Notch Integration
Remoting
  • Remoting Index
  • Remoting Overview
  • How To Control and Customize Ventuz
  • Remoting 4
  • Remoting 4 via Websockets
  • Remoting 4 via HTTP
  • Director Remoting
  • Deprecated Remoting
  • Remoting Machine Signature

Misc

  • Presets
« Previous:
» Index «
Next: »

#

Ventuz Video I/O (VIO) API

Table of Contents

  1. Definitions
  2. Using VIO in Ventuz
    1. The Ventuz Configuration
    2. The Ventuz Scene
  3. Using the VIO API
    1. Initialization
    2. Rendering Frames
    3. Ancillary Data
  4. Comments on Common Problems
    1. Converting Matrices from OpenGL to DirectX
      1. Change the handedness of the coordinate system
      2. Fixing the w-range
    2. Using Depth-Buffers with Direct3D9
    3. Correct Blitting to Ventuz with DirectX
    4. Correct Blitting to Ventuz with OpenGL
    5. Upside down
    6. Working with multiple GPU's

The Ventuz Video Input Output API, short VIO, allows Ventuz users to program applications that send video streams towards Ventuz or receive streams from Ventuz. In addition to the color frames, alpha (key) and depth buffer (z buffer) are also supported as well as ancillary data such as Camera or Projection Matrices. On the Output side you can provide the client application with ancillary data from Ventuz.

This allows for the integration of hardware and software that Ventuz usually does not support out of the box. The availability of the depth buffer makes it possible to combine the Ventuz engine with more specialized engines, like real-time ray-tracers, map engines and windows user interface applications. Latency is much better than with SDI or DVI capture since the frames can be generated on the same GPU.

Only VIO To Ventuz does support Depth. VIO From Ventuz therefor does not!

VIO From Ventuz needs an additional License Option. Please contact us via sales@ventuz.com.

The VIO can transfer frames directly with OpenGL, Direct3D9 or Direct3D11 contexts, or via CPU-mapped memory.

Ancillary data can be attached to the stream on both sides. This allows e.g. to pass the view and projection matrices are used to render the frame so that the receiver can continue rendering. Only if the matrices are 100% correct the depth buffer works as expected.

The API is exposed as C *.dll, *.lib and *.h file. It should be no problem to use that in most programming languages. The dll is compiled without dependencies to the C/C++ runtime, so it will link with any compiler. We will assume that Visual Studio 2015 is used.

The header file contains detailed documentation of the functions, their parameters and all structure members and enumerations. This text will explain the general operation of the video engine.

Definitions

Note that we avoid the terms input and output as they are confusing: What is an input for Ventuz is an output for the client.

Client: The application that communicates with Ventuz using the VIO API.

To Ventuz: Transfer from the client to Ventuz.

From Ventuz: Transfer from Ventuz to the client.

Frame: A color buffer with alpha, and possibly a fitting depth buffer.

Ancillary data or Anc: Data that is attached to the frames. We avoid the term meta-data as it is used by the video replay for metadata of the stream, not single frames.

Stream: The connection between Ventuz and the client.

Using VIO in Ventuz

The Ventuz Configuration

To use a VIO input (to Ventuz) a stream has to be created in the Ventuz configuration editor. This is done by dragging the VIO device into the input or output area. See Audio / Video configuration.

The most important option here is synchronous/asynchronous.

For inputs, it is best to configure the resolution to auto detect and let the client define the resolution. The client also chooses many other aspects, like if OpenGL or DirectX is used and what kind of depth buffer (if any) is used.

The Ventuz Scene

The View and Projection Matrices are automatically set to the default camera. Make sure that the Layer3D is set to the default camera. Instead you can also use a Tracked Camera.

To composite the VIO Input into a Layer3D, use the Vio Input Node. See the Node Page for more information on its properties.

To get correct alpha-sorting, think about what to render before the compositing node and what to render afterward.

You can use an Alpha node to set an alpha value for compositing. Other render and material options are ignored.

If you are missing some of the required DLL's, or need additional help getting started, please download the VIO Example project from the Online Demos & Tutorials section of the Project Browser. That will contain everything you need to get started.

Using the VIO API

Following is a basic guide to what to consider when implementing the VIO API in a client application.

To use the VIO API, include the header and link the library. Make sure the DLL is in the executable path of the application

#include "VioApi.h"
#pragma comment(lib,"VioApi")

Initialization

To Initialize the VIO dll, call vInit(). If the client uses DirectX, pass the IDirect3DDevice9Ex interface. The VIO API assumes that the OpenGL context is current when any function is called. When transfer is done via CPU memory, no DirectX or OpenGL context is required as argument for vInit()

When no DirectX device was specified, VioApi will create it's own device. If a windows handle was passed to vInit(), this will be used to create the device. Otherwise an invisible dummy window is created.

To open a stream, call vOpen(). Fill out all fields of the vOpenPara structure. Some of these fields are only required for future extensions, like the color space.

vOpenPara para;

memset(&para,0,sizeof(para));
para.Channel = 0;
para.Mode = VM_ToVentuz;
para.Transfer = VTM_Direct3D9;
para.Color = VCP_BGRA_8;
para.Depth = CONFIG_DEPTH;
para.ColorSpace = VCS_RGB;
para.SizeX = WindowSizeX;
para.SizeY = WindowSizeY;

if(VE_OK==vOpen(&para,&VioHandle))
{
    // success
}

For "FromVentuz" streams, the size can be set to 0,0. On return, the parameter structure will be filled with the stream as defined in the ventuz AV configuration. For "ToVentuz" stream, if they are configured to "autodetect", Ventuz will rezise the stream as specified in the parameter structure. Otherwise, the exact resolution as defined in the AV configuration must be specified.

It does not matter if Ventuz or the Client has started first. Both ends of the connection will try to connect repeatedly until a connection is established, and will try to re-connect if the connection was interrupted.

Multiple streams can be opened by different client processes.

To close a stream call vClose(), to de-initialize the whole dll after all streams have been closed call vExit().

Rendering Frames

The procedure to render a frame is as follows:

vFrame frame;
if(VE_Ok==vLockFrame(VioHandle,&frame))
{
    // work with the frame
    vUnlockFrame(VioHandle);
}

The timing is determined by Ventuz. The client should render as fast as possible and not wait for vertical sync. Ventuz will block the client if it renders faster than Ventuz.

The Ventuz Configuration Editor can configure the stream as either synchronized or not. In a synchronized stream, Ventuz will wait for the client and buffer up a few frames to prevent frame drops. In async mode, Ventuz will always use the latest frame, possibly dropping or duplicating frames.

If the transfer mode is VTM_Direct3D9, the vFrame provides IDirect3DSurface9 interfaces for color and depth buffer.

If the transfer mode is VTM_Direct3D11, the vFrame provides IDirect3DTexture11 interfaces for color and depth buffer, as well as a ID3D11RenderTargetView in case of transfer to Ventuz or a ID3D11ShaderResourceView in case of transfer from Ventuz.

If the transfer mode is VTM_OpenGl, the vFrame provides OpenGL Texture2D names that can be bound either to GL_TEXTURE2D or to GL_FRAMEBUFFER.

If the transfer mode is VTM_Cpu, the vFrame provides pointers to memory. The stride (pitch) of these buffers is aligned to 4 bytes. This is important for 16-bit depth buffers with odd (not even) widths.

The client is expected to render to or read from these resources between (and only between) vLockFrame() and vUnlockFrame().

Ancillary Data

On a stream to Ventuz, the client can add ancillary data using vAncToVentuz(). The data provided will be copied, so the pointer does not need to be valid after the call.

On a stream from Ventuz, the client can extract ancillary data using vAncFromVentuz(). This function provides a pointer that will remain valid only between lock and unlock.

Ancillary data consists of a FourCC code, the data and the size. Multiple such packets can be sent or received, even of the same FourCC code. When sending ancillary to Ventuz, the LiveVideo node can be used to extract the packets for binding with other Ventuz nodes.

The ancillary data that flows in the same direction as the frames is synchronized with the frames. The frame inputs have synchronized vAncToVentuz(). But vAncFromVentuz() and vAncToVentuz() are not synchronized.

Unsynchronized data may have packets merged. This means, for example, if an output calls vAncToVentuz() every frame, timing issues may cause Ventuz to provide no data in one frame and the merged data of two frames in the next frame.

Ancillary data is currently limited to 16 packets and 32KB of data for all packets.

Comments on Common Problems

Converting Matrices from OpenGL to DirectX

OpenGL usually uses a right-handed coordinate system and z-clips from -w to w. DirectX usually uses a left-handed coordinate system and z-clips from 0 to w.

Change the handedness of the coordinate system

You need a mirror-z matrix and then multiply it before and after the camera matrix:

     | 1  0  0  0 |
mz = | 0  1  0  0 | ;   cam' = mz * cam  * mz
     | 0  0 -1  0 |
     | 0  0  0  1 |

This can be done cheaper like this:

cam.k.x *= -1;  cam.i.z *= -1;
cam.k.y *= -1;  cam.j.z *= -1;
cam.k.w *= -1;  cam.l.z *= -1;

This uses a matrix class, that consists of four vectors i, j, k, and l that each have a float of x, y, z and w. Note that cam.k.z is inverted twice, so that goes away too.

Fixing the w-range

The w-range is a bigger problem. The easiest solution is to create a correct DirectX projection matrix along with the OpenGl projection matrix.

Using Depth-Buffers with Direct3D9

Direct3D9 does not support reading the depth buffer as a texture. Various hacks are around, and we use the INTZ FourCC code. For an explanation of how this works see ​Advanced DX9 Capabilities for ATI Radeon Cards. The trick works also with all relevant NVidia and Intel GPUs.

Correct Blitting to Ventuz with DirectX

The color surface provided by vFrame can be used directly as render-target. The depth surface can not, as it is an D3DFMT_R32F or D3DFMT_L16 surface. A real depth surface of type INTZ has to be created and then blitted to the provided surface.

This has to be done by rendering a rectangle, and can not be done with StretchRect(). When doing so, be sure to remember the half-pixel offset required by Direct3D9 rasterization rules.

When rendering with DirectX on a different GPU, the frame has to be transferred to memory. This involves creating an additional texture in D3DPOOL_SYSTEMMEM. To transfer from GPU to CPU memory, use GetRendertargetData(). First copy from an INTZ depth buffer to a D3DFMT_R32F texture in D3DPOOL_DEFAULT by drawing a rectangle, then use GetRendertargetData() to blit that to a D3DFMT_R32F texture in D3DPOOL_SYSTEMMEM, then lock that surface and memcpy() it to the buffer provided by the VIO API. In other words: prefer staying on the same GPU with DirectX.

Correct Blitting to Ventuz with OpenGL

As with DirectX, a real depth buffer has to be created, and then copied into the provided OpenGL texture, by rendering a rectangle. OpenGL rasterization rules do not require any offsets.

When reading from memory, use glReadPixels(). In the OpenGL case, no extra system memory textures are required, glReadPixels() can copy directly into the buffers provided by the VIO API.

Upside down

As a side effect of the blitting, the depth or color buffer may be upside down, or the red and blue color channels may be swapped. This is easily compensated with the Vio Input Node. Otherwise you can also handle this on the client side.

Working with multiple GPU's

If a DirectX device is passed to vInit(), it must use the same GPU as Ventuz. If Ventuz is already running, vGetVentuzAdapter() can be used to find out on which GPU Ventuz is running. If the VioApi creates it's own device, it will automatically use the correct GPU. This will not work correctly if the video configuration has changed between starting Ventuz and the Client. For example: Changing the main display will change the order of DirectX adapters, and the correct GPU can not be found any more.

When rendering on a different GPU than Ventuz, memory transfers (VTM_Cpu) have to be used and no DirectX device may be passed to vInit()

Rendering on a different monitor than Ventuz is not a problem if both monitors are attached to the same GPU.

See also:
  • Vio Input Node
  • Ventuz License Model

« Previous:
» Index «
Next: »
Copyright 2022 Ventuz Technology