Compositor
Since version 1.9.5 Prysm supports a new CSS property: coh-composition-id
. This property allows for completely detaching the element from the default rendering flow and letting the user handle the drawing of the element. This feature is intended for achieving “real 3D” effects with the UI by taking the transformation data that the SDK provides and composing elements at any world position and orientation, not just on the UI plane.
transform-style: preserve-3d;
style on the element’s parent. If that style is not specified, the transform will be flattened to a 2D plane and you won’t get the desired effect.Custom compositor interface
A custom compositor can be set using the cohtml::View::SetCustomSceneCompositor
API which would then receive callbacks when the composition needs to be drawn. You can also specify arbitrary user data that will be passed back through the interface callbacks.
Threading
By default, renoir::ISublayerCompositor
callbacks are invoked on the UI thread and the Render Thread/Layout Thread.
OnCompositionAdded
- UI ThreadOnCompositionRemoved
- UI ThreadOnCompositionVisibility
- UI ThreadOnDrawSubLayer
- Render Thread/Layout Thread
In some cases, it might be required to have them called on the UI Thread, for example, if you want to compose the sublayer with the Material API of an engine that requires all calls to be made from the Main Thread.
You can configure what thread calls the renoir::ISublayerCompositor
callbacks during the initialization of the SDK. To do that, you need to initialize the Cohtml SDK with the following parameters:
- When initializing the Library with
Library::Initialize
:- Set
LibraryParams::UseDedicatedLayoutThread
tofalse
- Set
- When creating a View with
System::CreateView
:- Set
ViewSettings::ExecuteLayoutOnDedicatedThread
tofalse
- Set
ViewSettings::ExecuteCommandProcessingWithLayout
totrue
- Set
LibraryParams::AllowMultithreadedCommandProcessing
to true
.OnCompositionAdded callback
The callback is called on the UI Thread.
During Advance of the view when style for composition-id is matched to the element you’ll receive the OnCompositionAdded
callback with the composition ID and the ID of the view which owns that composition.
You can use that callback to prepare all resources needed for the composition.
void SimpleCompositor::AddDrawData(unsigned viewId, const char* compositionId)
{
std::lock_guard<std::mutex> l(m_DrawDatasMutex);
// Here you can do some work which has to be done on UI Thread.
// For example you can create resources needed for composition rendering.
m_ViewDrawData.emplace(std::string(compositionId), renoir::ISubLayerCompositor::DrawData());
}
OnCompositionRemoved callback
The callback is called on the UI Thread.
When the composition is no longer needed (e.g. the element is deleted from the HTML, you’ll receive the OnCompositionRemoved
callback with the composition ID and the ID of the view which contains the composition.
You can use that callback for cleaning up any resources allocated for the specified composition.
transform
CSS property is meant to be the main influence on their position in the 3D world. However, under some circumstances, elements with coh-composition-id
may end up outside of the viewport due to layout. In these cases, the elements won’t be rendered and no call to the OnDrawSubLayer
will be made. If such element with coh-composition-id
that is outside of the screen and has never been rendered is removed from the DOM, the OnCompositionRemoved
callback will be executed. The rendering library will also produce debug logs informing the user that the textures for these elements won’t be deleted as they have not been created in the first place. This is not necessarily an error but it is something to keep in mind as it may be unexpected that these elements end up outside of the screen.void SimpleCompositor::OnCompositionRemoved(unsigned viewId, const char* compositionId)
{
std::lock_guard<std::mutex> l(m_DrawDatasMutex);
// Here you can do some work which has to be done on UI Thread.
// For example you can destroy resources needed for composition rendering.
m_ViewDrawData.erase(std::string(compositionId));
}
OnCompositionVisibility callback
The callback is called on the UI Thread.
At each frame, you will receive the OnCompositionVisibility
callback with a boolean that is true if the element with the coh-composition-id
is visible and false otherwise. An element is made invisible via the opacity
, display
, and visibility
CSS properties.
You can use this to avoid drawing elements that would not be visible anyway, thus reducing the number of draw calls.
OnDrawSubLayer callback
Depending on ViewSettings::ExecuteCommandProcessingWithLayout option it can be called on different threads
- If it's `true` the callback will come on Layout Thread
- If it's `false` the callback will come on Render Thread
The data available in the OnDrawSubLayer
callback is the renoir::ISublayerCompositor::DrawData
structure:
unsigned ViewId
: The Id of the view which is drawing the composition.const char* SubLayerCompositionId
: Composition ID (coh-composition-id
).void* CustomSceneMetadata
: Custom data set withcohtml::View::SetCustomSceneCompositor
Texture2DObject Texture
: Backend texture ID to be sampled from.Texture2D TextureInfo
: Texture info for the sampled image (e.g. dimensions, format, etc.)float2 UVScale
: UV scaling is needed to sample from the correct location.float2 UVOffset
: UV offset is needed to sample from the correct location. Basically, the sampled point should beinput.Additional.xy * UVScale.xy + UVOffset.xy
float4x4 FinalTransform
: Transformation of the quad to be drawn.Rectangle Untransformed2DTargetRect
: Vertex positions of the quad to be drawn.ColorMixingModes MixingMode
: Color mixing mode with the background.
mix-blend-mode
CSS property to your composited element, the SDK will have no way of applying that. That’s because a composited element is entirely under the client’s control. The SDK supplies only the input data and leaves it at that. This means that you’ll have to apply any custom blending in your pipeline as it will not work automatically.void SimpleCompositor::OnDrawSubLayer(const renoir::ISubLayerCompositor::DrawData& data)
{
std::lock_guard<std::mutex> l(m_DrawDatasMutex);
m_ViewDrawData[std::string(data.SubLayerCompositionId)] = data;
}
UI Thread
might be 2-3 frames ahead from the Render Thread
when multithreaded rendering is used, due to this reason it is possible to receive OnCompositionRemoved
callback before last OnDrawSublayer
. If you have destroyed the resources needed for composition rendering you must guard OnDrawSublayer
calls and don’t use the composition data after OnCompositionRemoved
callback was received. The following snippet can be used as an example:void SimpleCompositor::OnDrawSubLayer(const renoir::ISubLayerCompositor::DrawData& data)
{
std::lock_guard<std::mutex> l(m_DrawDatasMutex);
auto it = m_ViewDrawData.find(std::string(data.SubLayerCompositionId));
if (it != m_ViewDrawData.end())
{
m_ViewDrawData[std::string(data.SubLayerCompositionId)] = data;
}
}
Compositor Input
It’s possible to pass input events that will only trigger for a specific composition and not for regular elements or other compositions in the same view. To do that set the last optional argument of the given input function to the Id of the target composition. Coordinates for the event should be computed by using the Untransformed2DTargetRect
and the UV coordinates from the event hit in the following manner:
CoherentMouseData.X = event->UVcoord.X * Untransformed2DTargetRect.Size.x + Untransformed2DTargetRect.Position.x;
CoherentMouseData.Y = event->UVcoord.Y * Untransformed2DTargetRect.Size.y + Untransformed2DTargetRect.Position.y;
View->MouseEvent(CoherentMouseData, nullptr, UserData, CompositionId);
Known Issues
- Composited elements placed outside of the viewport won’t be updated after their first draw.
- When a composited element is redrawn it also redraws intersecting elements at its original position in the document.
You can workaround that in 2 ways.
- By moving composited elements out of the screen. Such elements will be painted once and won’t be updated further.
- By moving composited elements in a separate View.
- Composited elements that become empty will remain dirty. This can either be caused by removing all children or hiding them via
display
,opacity
, orvisibility
.
You can workaround that in 3 ways.
- By hiding the composited root in the same frame it becomes empty. This can be done via
opacity
,visibility
, anddisplay
. - By forcing the composited root element to be drawn via CSS styles e.g. size not equal to 0 and a background color.
- By leaving a single element with dimensions, background, and near transparent opacity (e.g. 0.0001) inside the composited root.
Sample use case
A common scenario for using the compositor is to achieve a “real 3D” UI. The feature allows you to specify 3D transformations in CSS and then use the information that the SDK passes to the compositor interface to render that element in your 3D world using the transformation that was specified in the CSS.
Usually, you’d create a single rectangle geometry to be stored on the GPU and reuse that for all composited surfaces, only changing the vertex transformation.
It is important to transform the geometry exactly into the Untransformed2DTargetRect
positions, otherwise applying the FinalTransform
matrix may have unexpected results.
transform-origin
of the geometry in the engine. Most engines assume the (0,0,0) point in the local space of an object to be the center of that object. The HTML rectangle and transformation, however, use the top-left corner as transform-origin
.Applying transformations example
Let’s take for example an engine that uses the center of an object as its transform-origin
(as opposed to top-left being the transform-origin
in HTML/CSS, by default). If the rectangle geometry on the GPU has bounds MeshBounds, then the following list of transformations has to be applied:
- Scale the GPU geometry to the size of the rectangle, expected by the SDK
T1 = Scale(Untransformed2DTargetRect.Size / MeshBounds)
- Offset the geometry so that its top-left corner coincides with the
transform-origin
. Any transformation applied will behave as if it was done in HTML/CSS now.
T2 = Translate(Untransformed2DTargetRect.Position + Untransformed2DTargetRect.Size / 2)
- Apply the
FinalTransform
T3 = `FinalTransform`
- The geometry will now be in Cohtml View space, which has its
transform-origin
at the top-left. We have to modify it to the center again, as we’ll be using this geometry in the engine (which uses the object’s center astransform-origin
, as we described in the beginning). Assuming the width/height of the view is known as ViewSize:
T4 = Translate(-ViewSize / 2)
- The geometry is now positioned with the proper
transform-origin
that the engine uses but is still in Cohtml View space units. The last thing left to do is to scale that. Assuming that the parent object that has the in-world Cohtml View attached has bounds ParentBounds, that’s simply:
T5 = Scale(ParentBounds / ViewSize)
Finally, all of these must be combined. M1 = T1 * T2 * T3 * T4
is a simple matrix multiplication. Result = M1 * T5
must be completed as a transform multiplication in order to apply the scaling using the original axes of the element. In general, if a matrix is decomposed into a transform with components Q, S, T, where Q = quaternion, S = scale, and T = translation, the result of multiplying M1 * T5
would be
Result.Q = T5.Q * M1.Q
Result.S = M1.S * T5.S
Result.T = T5.Q * (T5.S * M1.T) + T5.T
Recompose that into a matrix and that’s what the final transform is.