Compositor

Since version 1.9.5 Prysm supports a new CSS property: coh-composition-id. This property allows for completely detaching the element from the default rendering flow and letting the user handle the drawing of the element. This feature is intended for achieving “real 3D” effects with the UI by taking the transformation data that the SDK provides and composing elements at any world position and orientation, not just on the UI plane.

In order to allow 3D transforms on the composited element, you must specify the transform-style: preserve-3d; style on the element’s parent. If that style is not specified, the transform will be flattened to a 2D plane and you won’t get the desired effect.

Custom compositor interface

A custom compositor can be set using the cohtml::View::SetCustomSceneCompositor API which would then receive callbacks when the composition needs to be drawn. You can also specify arbitrary user data that will be passed back through the interface callbacks.

Threading

By default, renoir::ISublayerCompositor callbacks are invoked on the UI thread and the Render Thread/Layout Thread.

OnCompositionAdded- UI Thread
OnCompositionRemoved - UI Thread
OnCompositionVisibility - UI Thread
OnDrawSubLayer - Render Thread/Layout Thread

In some cases, it might be required to have them called on the UI Thread, for example, if you want to compose the sublayer with the Material API of an engine that requires all calls to be made from the Main Thread.

You can configure what thread calls the renoir::ISublayerCompositor callbacks during the initialization of the SDK. To do that, you need to initialize the Cohtml SDK with the following parameters:

When initializing the Library with Library::Initialize:
- Set LibraryParams::UseDedicatedLayoutThread to false
When creating a View with System::CreateView:
- Set ViewSettings::ExecuteLayoutOnDedicatedThread to false
- Set ViewSettings::ExecuteCommandProcessingWithLayout to true

This will effectively move the layout and render command recording to the UI Thread. While it will allow the use of the rendering data in the UI Thread, a large amount of processing will be moved there as well.

When the above settings are enabled, the Cohtml’s rendering library will be used on two different threads – the UI and the render thread. Normally, the library will detect that but if you have a use case where your custom engine can execute “main” (UI) thread and render tasks on the same thread, you’ll have to enable the multithreaded awareness manually by setting LibraryParams::AllowMultithreadedCommandProcessing to true.

OnCompositionAdded callback

The callback is called on the UI Thread.

During Advance of the view when style for composition-id is matched to the element you’ll receive the OnCompositionAdded callback with the composition ID and the ID of the view which owns that composition.

You can use that callback to prepare all resources needed for the composition.

void SimpleCompositor::AddDrawData(unsigned viewId, const char* compositionId)
{
    std::lock_guard<std::mutex> l(m_DrawDatasMutex);
    // Here you can do some work which has to be done on UI Thread.
    // For example you can create resources needed for composition rendering.
    m_ViewDrawData.emplace(std::string(compositionId), renoir::ISubLayerCompositor::DrawData());
}

OnCompositionRemoved callback

The callback is called on the UI Thread.

When the composition is no longer needed (e.g. the element is deleted from the HTML, you’ll receive the OnCompositionRemoved callback with the composition ID and the ID of the view which contains the composition.

You can use that callback for cleaning up any resources allocated for the specified composition.

In general, elements that are to be composed through the compositor by the user are not meant to be affected by the layout that much. The transform CSS property is meant to be the main influence on their position in the 3D world. However, under some circumstances, elements with coh-composition-id may end up outside of the viewport due to layout. In these cases, the elements won’t be rendered and no call to the OnDrawSubLayer will be made. If such element with coh-composition-id that is outside of the screen and has never been rendered is removed from the DOM, the OnCompositionRemoved callback will be executed. The rendering library will also produce debug logs informing the user that the textures for these elements won’t be deleted as they have not been created in the first place. This is not necessarily an error but it is something to keep in mind as it may be unexpected that these elements end up outside of the screen.

void SimpleCompositor::OnCompositionRemoved(unsigned viewId, const char* compositionId)
{
    std::lock_guard<std::mutex> l(m_DrawDatasMutex);
    // Here you can do some work which has to be done on UI Thread.
    // For example you can destroy resources needed for composition rendering.
    m_ViewDrawData.erase(std::string(compositionId));
}

OnCompositionVisibility callback

The callback is called on the UI Thread.

At each frame, you will receive the OnCompositionVisibility callback with a boolean that is true if the element with the coh-composition-id is visible and false otherwise. An element is made invisible via the opacity, display, and visibility CSS properties.

You can use this to avoid drawing elements that would not be visible anyway, thus reducing the number of draw calls.

OnDrawSubLayer callback

Depending on ViewSettings::ExecuteCommandProcessingWithLayout option it can be called on different threads 
- If it's `true` the callback will come on Layout Thread
- If it's `false` the callback will come on Render Thread

The data available in the OnDrawSubLayer callback is the renoir::ISublayerCompositor::DrawData structure:

unsigned ViewId: The Id of the view which is drawing the composition.
const char* SubLayerCompositionId: Composition ID (coh-composition-id).
void* CustomSceneMetadata: Custom data set with cohtml::View::SetCustomSceneCompositor
Texture2DObject Texture: Backend texture ID to be sampled from.
Texture2D TextureInfo: Texture info for the sampled image (e.g. dimensions, format, etc.)
float2 UVScale: UV scaling is needed to sample from the correct location.
float2 UVOffset: UV offset is needed to sample from the correct location. Basically, the sampled point should be input.Additional.xy * UVScale.xy + UVOffset.xy
float4x4 FinalTransform: Transformation of the quad to be drawn.
Rectangle Untransformed2DTargetRect: Vertex positions of the quad to be drawn.
ColorMixingModes MixingMode: Color mixing mode with the background.

If you specified the mix-blend-mode CSS property to your composited element, the SDK will have no way of applying that. That’s because a composited element is entirely under the client’s control. The SDK supplies only the input data and leaves it at that. This means that you’ll have to apply any custom blending in your pipeline as it will not work automatically.

void SimpleCompositor::OnDrawSubLayer(const renoir::ISubLayerCompositor::DrawData& data)
{
    std::lock_guard<std::mutex> l(m_DrawDatasMutex);
    m_ViewDrawData[std::string(data.SubLayerCompositionId)] = data;
}

In some cases the UI Thread might be 2-3 frames ahead from the Render Thread when multithreaded rendering is used, due to this reason it is possible to receive OnCompositionRemoved callback before last OnDrawSublayer. If you have destroyed the resources needed for composition rendering you must guard OnDrawSublayer calls and don’t use the composition data after OnCompositionRemoved callback was received. The following snippet can be used as an example:

void SimpleCompositor::OnDrawSubLayer(const renoir::ISubLayerCompositor::DrawData& data)
{
    std::lock_guard<std::mutex> l(m_DrawDatasMutex);
    auto it = m_ViewDrawData.find(std::string(data.SubLayerCompositionId));
    if (it != m_ViewDrawData.end())
    {
        m_ViewDrawData[std::string(data.SubLayerCompositionId)] = data;
    }
}

Compositor Input

It’s possible to pass input events that will only trigger for a specific composition and not for regular elements or other compositions in the same view. To do that set the last optional argument of the given input function to the Id of the target composition. Coordinates for the event should be computed by using the Untransformed2DTargetRect and the UV coordinates from the event hit in the following manner:

CoherentMouseData.X = event->UVcoord.X * Untransformed2DTargetRect.Size.x + Untransformed2DTargetRect.Position.x;
CoherentMouseData.Y = event->UVcoord.Y * Untransformed2DTargetRect.Size.y + Untransformed2DTargetRect.Position.y;

View->MouseEvent(CoherentMouseData, nullptr, UserData, CompositionId);

Known Issues

Composited elements placed outside of the viewport won’t be updated after their first draw.
When a composited element is redrawn it also redraws intersecting elements at its original position in the document.

You can workaround that in 2 ways.

By moving composited elements out of the screen. Such elements will be painted once and won’t be updated further.
By moving composited elements in a separate View.

Composited elements that become empty will remain dirty. This can either be caused by removing all children or hiding them via display, opacity, or visibility.

You can workaround that in 3 ways.

By hiding the composited root in the same frame it becomes empty. This can be done via opacity, visibility, and display.
By forcing the composited root element to be drawn via CSS styles e.g. size not equal to 0 and a background color.
By leaving a single element with dimensions, background, and near transparent opacity (e.g. 0.0001) inside the composited root.

Sample use case

A common scenario for using the compositor is to achieve a “real 3D” UI. The feature allows you to specify 3D transformations in CSS and then use the information that the SDK passes to the compositor interface to render that element in your 3D world using the transformation that was specified in the CSS.

Usually, you’d create a single rectangle geometry to be stored on the GPU and reuse that for all composited surfaces, only changing the vertex transformation.

It is important to transform the geometry exactly into the Untransformed2DTargetRect positions, otherwise applying the FinalTransform matrix may have unexpected results.

Take care of the transform-origin of the geometry in the engine. Most engines assume the (0,0,0) point in the local space of an object to be the center of that object. The HTML rectangle and transformation, however, use the top-left corner as transform-origin.

Applying transformations example

Let’s take for example an engine that uses the center of an object as its transform-origin (as opposed to top-left being the transform-origin in HTML/CSS, by default). If the rectangle geometry on the GPU has bounds MeshBounds, then the following list of transformations has to be applied:

Scale the GPU geometry to the size of the rectangle, expected by the SDK

T1 = Scale(Untransformed2DTargetRect.Size / MeshBounds)

Offset the geometry so that its top-left corner coincides with the transform-origin. Any transformation applied will behave as if it was done in HTML/CSS now.

T2 = Translate(Untransformed2DTargetRect.Position + Untransformed2DTargetRect.Size / 2)

Apply the FinalTransform

T3 = `FinalTransform`

The geometry will now be in Cohtml View space, which has its transform-origin at the top-left. We have to modify it to the center again, as we’ll be using this geometry in the engine (which uses the object’s center as transform-origin, as we described in the beginning). Assuming the width/height of the view is known as ViewSize:

T4 = Translate(-ViewSize / 2)

The geometry is now positioned with the proper transform-origin that the engine uses but is still in Cohtml View space units. The last thing left to do is to scale that. Assuming that the parent object that has the in-world Cohtml View attached has bounds ParentBounds, that’s simply:

T5 = Scale(ParentBounds / ViewSize)

Finally, all of these must be combined. M1 = T1 * T2 * T3 * T4 is a simple matrix multiplication. Result = M1 * T5 must be completed as a transform multiplication in order to apply the scaling using the original axes of the element. In general, if a matrix is decomposed into a transform with components Q, S, T, where Q = quaternion, S = scale, and T = translation, the result of multiplying M1 * T5 would be

Result.Q = T5.Q * M1.Q
Result.S = M1.S * T5.S
Result.T = T5.Q * (T5.S * M1.T) + T5.T

Recompose that into a matrix and that’s what the final transform is.