UI Surface Partitioning Integration

The “UI Surface Partitioning” Feature #

With version 1.38, Prysm introduces a new feature that aims to help developers integrate Cohtml in a 4K Rendering flow. The main problem that we try to address with this feature is the high memory requirement when Cohtml renders HTML on a 4K texture. Currently, to use Cohtml the user code must always pass a valid texture object to cohtml::ViewRenderer::SetRenderTarget. We call this texture “UI Texture” and generally speaking the size of the texture should be the same as the viewport size of the HTML page. When a game runs in 4K resolution, the size of the UI texture will also be of that same resolution.

There are however cases where the rendered UI does not take the whole 4K texture space in the UI texture. A classic example of this is a HUD where the UI content is only in the corners of the viewport.

If such UI is displayed through Cohtml in 4K resolution, the created UI texture will be large but only parts of it will actually contain meaningful content. The “UI Surface Partitioning” feature tries to address this problem.

Feature Overview #

The idea of the UI Surface Partitioning flow is to let the front-end developers divide the UI into sensible parts that can be rendered individually in separate textures. This way Cohtml will allocate textures with the sizes of the individual UI components, render them there and let the user code draws the textures at the correct positions on the screen.

The general approach to using the UI Surface Partitioning features is as follows:

1. Initialize the cohtml::View in a way that supports UI Surface Partitioning (explained in the next section)
2. Create an HTML page where the top-level elements are marked (explained in the next section)
3. Render the Page as normal through cohtml::ViewRenderer::Paint
4. Receive the textures for the top-level elements and draw them on the screen (explained in the next section)

All of this means that it is up to the front-end developers to author the HTML page in a way where there are clear top-level components for which is sensible to be rendered in separate textures. Not all UIs would be convenient for this in which case the standard rendering flow would probably be better.

Usage #

To make use of the UI Surface Partitioning several points need to be considered.

Integration

First off, the cohtml::View has to be set up in a property way that supports UI Surface Partitioning. Three things need to be done:

1. There is now a new view setting – cohtml::ViewSettings::EnableUISurfacePartitioning – that needs to be set to true when the view will be used in UI Surface Partitioning mode. This will tell the view that it will be used to render some elements (more on the later) in separate textures.

2. The call to cohtml::ViewRenderer::SetRenderTarget can simply pass two nullptrs for the nativeTexture and nativeDepthStencil. It is, however, important to still call the method with the correct width and height or textureDescription with correct renoir::Texture2D::Width and renoir::Texture2D::Height fields. Calling the method with null textures will hint to the Renoir graphics library that there is no UI texture. The given widthtand height will however tell the it the size of the HTML viewport. These dimensions should generally be the size of the cohtml::View as given by ViewSettings::Width and ViewSettings::Height

3. The cohtml::View has to have a valid renoir::ISubLayerCompositor set through the cohtml::View::SetCustomSceneCompositor method. Cohtml will issue callbacks through this compositor to inform the client code that certain elements have been rendered in textures and have to be rendered on the screen. See the page on the Compositor for more details. All of this sets up the cohtml::View to be able to handle the UI Surface Partitioning flow without the need for a UI texture.

Frontend #

There are a couple of things to consider regarding the HTML/CSS of the page that will be rendered with UI Surface Partitioning. Cohtml still needs to know which elements will be rendered in separate textures. This is where the new CSS property coh-partitioned comes into play. Also, Cohtml will be issuing callbacks to client code when these elements are rendered so the coh-composition-id property will also be used. To achieve the UI Surface Partitioning flow, in the HTML/CSS we have to:

1. Mark top-level elements with the new coh-partitioned: on property. This will tell Cohtml that those elements should be rendered in a special way in separate textures.

2. Mark the partitioned elements with coh-composition-id: <some-id> so that Cohtml can generate callbacks to the user code when these elements are created, rendered, or destroyed. See the page on the Compositor for more details.

Quick reference

As a quick reference, here are the most relevant things that are to be done in the C++ integration code to make the UI Surface Partitioning work.

When initializing the cohtml::View:

ViewSettings viewSetting;
viewSetting.EnableUISurfacePartitioning = true;

cohtml::View* View = System->CreateView(viewSetting);

void* userData;
CompositorUIPartitioning* compositor4UIPartitioning = new CompositorUIPartitioning();
View->SetCustomSceneCompositor(compositor4UIPartitioning, userData);

When painting the View:

ViewRenderer->Paint(...);
compositor4UIPartitioning->PaintCompositions();

Compositor class:

class CompositorUIPartitioning : public renoir::ISubLayerCompositor
{
	virtual void OnDrawSubLayer(const renoir::ISubLayerCompositor::DrawData& data) override
	{
		if (Infos.find(data.SubLayerCompositionId) != Infos.end())
		{
			Infos[data.SubLayerCompositionId].Data = data;
		}
	}

	virtual void OnCompositionVisibility(unsigned viewId, const char* compositionId, void*, bool visible) override
	{
		if (Infos.find(data.SubLayerCompositionId) != Infos.end())
		{
			Infos[data.SubLayerCompositionId].Visible = visible;
		}
	}

	virtual void OnCompositionAdded(unsigned viewId, const char* compositionId, void* metadata) override
	{
		Infos[data.SubLayerCompositionId] = {};
	}

	virtual void OnCompositionRemoved(unsigned sceneId, const char* compositionId, void* metadata) override
	{
		Infos.erase(data.SubLayerCompositionId);
	}

	// Paint quads on the screen with the textures and positions given in elements in 'Infos'
	void PaintCompositions()
	{
		for (const auto& drawInfo : Infos)
		{
			// draw the element only if it is visible
			if (!drawInfo.Visible) continue;

			// where to draw the quad and how big
			auto position = drawInfo.Data.Untransformed2DTargetRect.Position;
			auto size = drawInfo.Data.Untransformed2DTargetRect.Size;

			// Texture id for a texture created in the backend
			auto textureId = drawInfo.Texture;

			// how to sample the texture
			float2 uvScale = drawInfo.UVScale;
			float2 uvOffset = drawInfo.UVOffset;

			RenderTextureQuad(position, size, textureId, uvOffset, uvScale);
		}
	}

	struct CompositedElement
	{
		renoir::ISubLayerCompositor::DrawData Data;
		bool Visible;
	};

	std::unordered_map<std::string, CompositedElement> Infos;
};

UI Surface Partitioning Rendering sample

Prysm comes with a sample which demonstrates how the UI Surface Partitioning can be integrated with client code. The sample is implemented in the Samples/Common/SampleUISurfacePartitioning/SampleUISurfacePartitioning.cpp file and it is heavily commented.

The sample shows:

• The changes needed to the cohtml::View initialization (see SampleUISurfacePartitioning::ModifyViewSettings and SampleUISurfacePartitioning::PostInitializeViews())
• How to implement a custom compositor which can handle the drawing of elements marked with coh-partitioned and coh-composition-id (see SampleUIPartitioningCompositor::OnDrawSubLayer, SampleUIPartitioningCompositor::OnCompositionAdded, SampleUIPartitioningCompositor::OnCompositionRemoved, and SampleUIPartitioningCompositor::RenderAllCompositions())

The sample runs the Samples/UIResources/UISurfacePartitioning/index.htmlpage which is a simple demonstration of how 4 elements can be rendered without the use of UI texture.

The sample and the page should illustrate the envisioned rendering flow without UI texture. This can be see from a RenderDoc capture of the running the UI Surface Partitioning Sample: