The Encoded Stream Out outputs provide a simple pipe based streaming protocol to other processes running on the same machine. This way 3rd party applications can process or distribute the video coming out of Ventuz in real time.
Ventuz makes the encoded streams available to other processes via a named pipe. These processes will need the proper permissions to access the pipe (usually this means running as the same user or at a higher privilege level than the Ventuz runtime).
The pipe's name is "VentuzOutA" for the first output, and B etc. for any additional outputs. Ventuz accepts an arbitrary number of connections to the named pipes, so for example a stream server can choose whether to handle multiple incoming connections itself or to connect to the pipe once per client served.
Although Ventuz allows arbitrarily many connections to each pipe, all connections will share the same output stream; you can't eg. set a different bitrate for each connection.
All data Ventuz sends through the pipe is organized in chunks. All chunks start with an 8 byte chunk header structure that looks like this:
| offset | type | description |
|---|---|---|
| +00 | FourCC | Chunk type |
| +04 | uint32 | Chunk size in bytes |
("uint32" means 32 bit unsigned little endian integer; "FourCC" means Four Character Code, ie. a sequence of 4 character bytes)
The chunk header is directly followed by the chunk data with the given size. For compatibility with future versions please note:
As soon as a client connects to the pipe, Ventuz will first send a Pipe Header chunk with information about the audio and video streams.
| offset | type | description |
|---|---|---|
| +00 | uint32 | Protocol version |
| +04 | FourCC | Video format |
| +08 | uint32 | Video width (pixels) |
| +12 | uint32 | Video height (pixels) |
| +16 | uint32 | Video frame rate numerator |
| +20 | uint32 | Video frame rate denominator |
| +24 | FourCC | Audio format |
| +28 | uint32 | Audio sample rate (Hz) |
| +32 | uint32 | Audio channel count |
Notes:
After sending the pipe header Ventuz will start transmitting video and audio data through the pipe. Every frame begins with a frame header chunk, followed by video and audio frame payload chunks.
| offset | type | description |
|---|---|---|
| +00 | uint32 | Frame index |
| +04 | uint32 | Frame flags |
The frame index is a monotonically increasing number.
Currently the following frame flags are defined:
| value | description |
|---|---|
| 0x00000001 | This video frame is an IDR frame |
This chunk contains exactly one frame worth of encoded video. If the IDR frame flag is set in the frame header it's guaranteed that this frame contains everything that's needed to start or restart a video stream, i.e. Sequence and Picture information, and no reference to past frames.
The first frame sent over a newly opened connection will always be an IDR frame so clients don't need to wait out for it.
The audio chunk contains "one video frame" worth of PCM encoded audio. Which means, the audio chunk corresponds to the exact point and duration in time the video frame is supposed to be displayed. It's therefore guaranteed that audio and video are always in sync with each other.
Even though audio is uncompressed and audio and video always come together, don't rely on audio chunks having a fixed size. When rendering at NTSCish (/1001) frame rates the audio chunk size is going to vary.
While receiving audio and video clients can send commands to Ventuz to set encoding parameters or inject keyboard and touch input. Each command consists of a command byte followed by an optional command payload structure. The following commands are defined:
| byte | followed by | description |
|---|---|---|
| 0x00 | - | No operation |
| 0x01 | - | Request IDR frame |
| 0x10 | Touch parameters | Begin Touch |
| 0x11 | Touch parameters | Move Touch |
| 0x12 | Touch parameters | End Touch |
| 0x13 | Touch parameters | Cancel Touch |
| 0x20 | Key parameters | Key Press |
| 0x30 | Encode parameters | Set encode parameters |
This command instructs the encoder inside Ventuz to emit an IDR frame. Due to the asynchronous nature of the encoding process this may take a frame or two; don't assume the frame received right after sending this command will already be an IDR one.
To inject touch input clients can use the four touch commands. All of them require for following payload:
| offset | type | description |
|---|---|---|
| +00 | uint32 | Touch ID |
| +04 | int32 | X coordinate (pixels from left side of viewport) |
| +08 | int32 | Y coordinate (pixels from upper side of viewport) |
Each "touch" is a sequence of one Begin Touch command, followed by zero or more Move Touch commands, and ended with either an End Touch or Cancel Touch command. For this each touch needs to be assigned a unique ID. Clients can eg. either assign IDs incrementally or randomly whenever a new touch is begun or have a "finger number". The important thing is that it's consistent for one touch and there aren't two touches at the same time with the same ID.
Clients can inject keystrokes (or rather characters) with the Key Press command followed by this structure:
| offset | type | description |
|---|---|---|
| +00 | uint32 | UTF32 codepoint |
Control keys that correspond to control characters such as Backspace or Enter will also work.
This command only works with nodes that accept text input, i.e. Text fields and Web browsers. The single key based nodes won't work.
This command sets certain encode parameters without interrupting the video stream. This way eg. a server can dynamically adjust the bitrate to match client or transport constraints. The parameter structure for this command looks like this:
| offset | type | description |
|---|---|---|
| +00 | uint32 | Rate control mode |
| +04 | uint32 | Bitrate or QP |
There are two values for the rate control mode:
| value | description |
|---|---|
| 0x00000000 | Encode with constant QP (Second field specifies QP (0..51)) |
| 0x00000001 | Encode with constant bitrate (Second field specifies bitrate in kbits/s) |
The encoder will change its behavior as quickly as possible upon receiving this command. Due to the asynchronous nature of the encoding process it may take a few frames unil the new settings are applied.
The following source code opens a connection to Ventuz and dumps the video and audio streams as "test.264" and "test.pcm" into the computer's TEMP directory, and lets you send commands to Ventuz by pressing keys. You're free to copy it partially or in total, or to use it as reference for your own implementation.
using System;
using System.IO;
using System.IO.Pipes;
using System.Runtime.InteropServices;
/// <summary>
///
/// Copyright (C) Ventuz 2020. All rights reserved.
///
/// Example for reading from the Ventuz StreamOut pipe.
/// Opens the pipe and dumps the audio and video streams into files.
///
/// Changelog
/// 2019-08-29 [TH] Initial public version
///
/// </summary>
namespace StreamOutPipeExample
{
// The data coming from the pipe is organized in chunks. every chunk starts with this, followed by the chunk data
[StructLayout(LayoutKind.Sequential)]
public struct ChunkHeader
{
public uint fourCC; // chunk type (four character code)
public int size; // size of chunk data
}
// Pipe header with general information (VVSP chunk)
[StructLayout(LayoutKind.Sequential)]
public struct PipeHeader
{
public static readonly int VERSION = 2;
public uint hdrVersion; // should be VERSION
public uint videoCodecFourCC; // used video codec, currently 'h264' aka h.264
public uint videoWidth; // width of image
public uint videoHeight; // height of image
public uint videoFrameRateNum; // frame rate numerator
public uint videoFrameRateDen; // frame rate denominator
public uint audioCodecFourCC; // used audio codec, currently 'pc16' aka 16 bit signed little endian PCM
public uint audioRate; // audio sample rate, currently fixed at 48000
public uint audioChannels; // audio channel count, currently 2
}
// Frame header, gets sent every frame, followed by video and then audio data (fhdr chunk)
[StructLayout(LayoutKind.Sequential)]
public struct FrameHeader
{
[Flags]
public enum FrameFlags : uint
{
IDR_FRAME = 0x01, // frame is IDR frame (aka key frame / stream restart/sync point)
}
public uint frameIndex; // frame index. If this isn't perfectly contiguous there was a frame drop in Ventuz
public FrameFlags flags; // flags, see above
}
// Commands to send back to the encoder.
enum PipeCommand : byte
{
Nop = 0x00, // Do nothing
RequestIDRFrame = 0x01, // Request an IDR frame. it may take a couple of frames until the IDR frame arrives due to latency reasons.
TouchBegin = 0x10, // Start a touch. Must be followed by a TouchPara structure
TouchMove = 0x11, // Move a touch. Must be followed by a TouchPara structure
TouchEnd = 0x12, // Release a touch. Must be followed by a TouchPara structure
TouchCancel = 0x13, // Cancal a touch if possible. Must be followed by a TouchPara structure
Key = 0x20, // Send a keystroke. Must be followed by a KeyPara structure
SetEncodePara = 0x30, // Send encoder parameters. Must be followed by an EncodePara structure
}
// Parameters for Touch* PipeCommands
[StructLayout(LayoutKind.Sequential)]
public struct TouchPara
{
public uint id; // numerical id. Must be unique per touch (eg. finger # or something incremental)
public int x; // x coordinate in pixels from the left side of the viewport
public int y; // y coordinate in pixels from the upper side of the viewport
};
// Parameters for the Key PipeCommand
[StructLayout(LayoutKind.Sequential)]
public struct KeyPara
{
public uint Code; // UTF32 codepoint for pressed key. Control characters like LF and Backspace work.
};
// Parameters for the SetEnodePara PipeCommand
[StructLayout(LayoutKind.Sequential)]
public struct EncodePara
{
public enum RateControlMode : uint
{
ConstQP = 0, // BitrateOrQP is QP (0..51)
ConstRate = 1, // BitrateOrQP is rate in kBits/s
}
public RateControlMode Mode;
public uint BitrateOrQP;
};
class Program
{
// The following helper functions are only meant as a reference.
// In a full implementation these should be asynchronous or
// at least have some kind of timeout/error handling.
/// <summary>
/// Read a number of bytes from a stream and return as array
/// </summary>
public static byte[] ReadBytes(Stream stream, int length)
{
var bytes = new byte[length];
int done = 0;
while (done < length)
{
int read = stream.Read(bytes, done, length-done);
if (read == 0) throw new IOException("stream ended");
done += read;
}
return bytes;
}
/// <summary>
/// Read a struct from a stream by just blitting the data
/// </summary>
public static T ReadStruct<T>(Stream stream, int size = 0)
{
var tsize = Marshal.SizeOf(typeof(T));
if (size <= 0) size = tsize;
if (size < tsize) throw new ArgumentException("size is too small");
var bytes = ReadBytes(stream, size);
GCHandle handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
T theStructure = (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(T));
handle.Free();
return theStructure;
}
/// <summary>
/// Send a command to the pipe
/// </summary>
/// <param name="cmd"></param>
/// <returns></returns>
public static void SendCommand(Stream stream, PipeCommand cmd)
{
var bytes = new byte[] { (byte)cmd };
stream.Write(bytes, 0, 1);
}
/// <summary>
/// Send a command to the pipe, with data struct
/// </summary>
public static void SendCommand<T>(Stream stream, PipeCommand cmd, T para)
{
int size = Marshal.SizeOf(para);
byte[] arr = new byte[size+1];
arr[0] = (byte)cmd;
var h = GCHandle.Alloc(arr, GCHandleType.Pinned);
Marshal.StructureToPtr(para, h.AddrOfPinnedObject()+1, false);
h.Free();
stream.Write(arr, 0, arr.Length);
}
/// <summary>
/// Make uint FourCC from four characters
/// </summary>
public static uint FourCC(char a, char b, char c, char d) => ((uint)a << 24) | ((uint)b << 16) | ((uint)c << 8) | ((uint)d);
/// <summary>
/// Entry point here
/// </summary>
static void Main(string[] args)
{
Console.WriteLine("Ventuz video stream pipe receiver example\n");
// The pipe name is "VentuzOutA" (or B etc for additional outputs), local only.
// You can connect to the pipe as many times as you wish (eg. once per client you serve),
// or you can connect once and handle distribution yourself.
using (var stream = new NamedPipeClientStream("VentuzOutA"))
{
Console.WriteLine("Connecting...\n");
// connect to the pipe
stream.Connect();
// try to get first chunk
var chunk = ReadStruct<ChunkHeader>(stream);
if (chunk.fourCC != FourCC('V', 'V', 'S', 'P'))
throw new Exception("invalid header from pipe");
// get header
var header = ReadStruct<PipeHeader>(stream, chunk.size);
if (header.hdrVersion != PipeHeader.VERSION)
throw new Exception("wrong protocol version");
// check codecs
if (header.videoCodecFourCC != FourCC('h', '2', '6', '4') || header.audioCodecFourCC != FourCC('p', 'c', '1', '6'))
throw new Exception("unsupported video or audio codec");
Console.WriteLine($"video: {header.videoWidth}x{header.videoHeight} @ {(float)header.videoFrameRateNum / header.videoFrameRateDen}fps");
Console.WriteLine($"audio: {header.audioChannels}ch @ {header.audioRate}Hz");
Console.WriteLine("Press ESC to quit.\n");
// video should always start with a full IDR frame (key frame with decoder reset,
// contains metadata such as picture and sequence information). Let's check if this is true.
bool expectIDRFrame = true;
float bytespersec = 0;
// let's open some files to dump the streams into. You can open the .264 file with
// VLC or Media Player Classic, or mux video and audio using FFMpeg.
FileStream videoFile = null;
FileStream audioFile = null;
try
{
var temppath = Path.GetTempPath();
videoFile = File.Create(temppath + "test.264");
audioFile = File.Create(temppath + "test.pcm");
}
catch (Exception ex)
{
Console.WriteLine($"ERROR: Can't create output file(s): {ex.Message}\n");
}
// now let's receive some stuff! Forever!
bool quit = false;
while (!quit)
{
while (Console.KeyAvailable)
{
var rk = Console.ReadKey();
var key = rk.Key;
// On second thought, quit if esc is pressed.
if ( key == ConsoleKey.Escape )
{
quit = true;
break;
}
else if (key == ConsoleKey.Spacebar)
{
// If the space bar is pressed, request an IDR frame
// (not setting expectIDRFrame here; it might take a frame or two until the IDR frame arrives)
SendCommand(stream, PipeCommand.RequestIDRFrame);
}
else if (key == ConsoleKey.F1)
{
// simulate a singular touch
var para = new TouchPara
{
id = 12345,
x = 100,
y = 100,
};
SendCommand(stream, PipeCommand.TouchBegin, para);
SendCommand(stream, PipeCommand.TouchEnd, para);
}
else if (rk.KeyChar >= '1' && rk.KeyChar <= '9')
{
// numeric keys: switch encoder bitrate to 1 mbit per key :)
SendCommand(stream, PipeCommand.SetEncodePara, new EncodePara
{
Mode = EncodePara.RateControlMode.ConstRate,
BitrateOrQP = ((uint)rk.KeyChar - '0') * 1000,
});
}
else
{
// forward all other keys to Ventuz
SendCommand(stream, PipeCommand.Key, new KeyPara
{
Code = rk.KeyChar
});
}
}
try
{
// frame header first
chunk = ReadStruct<ChunkHeader>(stream);
if (chunk.fourCC != FourCC('f', 'h', 'd', 'r'))
{
// skip unknown chunks
var _ = ReadBytes(stream, chunk.size);
continue;
}
var frameheader = ReadStruct<FrameHeader>(stream, chunk.size);
if (expectIDRFrame)
{
if (!frameheader.flags.HasFlag(FrameHeader.FrameFlags.IDR_FRAME))
throw new Exception("IDR frame expected");
expectIDRFrame = false;
}
// read video data
chunk = ReadStruct<ChunkHeader>(stream);
if (chunk.fourCC != FourCC('f', 'v', 'i', 'd'))
throw new Exception("video frame expected");
byte[] videoFrame = ReadBytes(stream, chunk.size);
// read audio data
chunk = ReadStruct<ChunkHeader>(stream);
if (chunk.fourCC != FourCC('f', 'a', 'u', 'd'))
throw new Exception("audio frame expected");
byte[] audioFrame = ReadBytes(stream, chunk.size);
// calc video bitrate (exponential moving average over packet size times frame rate)
bytespersec += 0.1f * ((float)videoFrame.Length * header.videoFrameRateNum / header.videoFrameRateDen - bytespersec);
// print stuff and dump streams into files
if (frameheader.flags.HasFlag(FrameHeader.FrameFlags.IDR_FRAME))
Console.WriteLine($"Received IDR frame {frameheader.frameIndex} ");
Console.Write($"got frame {frameheader.frameIndex}, {bytespersec * 8 / 1000.0f}kbits/s \r");
videoFile?.Write(videoFrame, 0, videoFrame.Length);
audioFile?.Write(audioFrame, 0, audioFrame.Length);
}
catch (Exception e)
{
Console.WriteLine($"pipe read failed: {e.Message}");
break;
}
}
Console.WriteLine("good bye.");
videoFile?.Close();
audioFile?.Close();
}
}
}
}