Pixi.js: Proposal: Improving text rendering

It is should not be in core.
It is heavy package and MUST BE implemented outside main package, core team should not worries about implementation of it.

Pixi SDF is right example.

Plz, don't try make Phaser from PIXI

I'd like to give some more context. I have asked @SukantPal to research alternative Text/BitmapText approaches that have different trade-offs. Particularly, if we can find something that performs better than Text and looks great.

I think there are a number of approaches here. Some might be appropriate as 3rd part plugins, some might be in the repo but not bundled, and some might replace/augment the current API. Let's figure out what the most fruitful Text rendering approach is that optimizes for performance but with fewer trade-off to BitmapText.

There are a few pieces of criteria that I'd use judge a good Text display object:

• Good Runtime Performance - Rendering text and changing text is low-cost
• Low memory - RAM usage is low
• Small Dependencies - dependency footprint file-size is small
• Great Fidelity - works well with different resolutions
• Flexible styling - support for strokes, drop shadows, gradients
• Supports Large Glyph Languages - E.g., Chinese, Japanese, Korean
• Backward compatible - Works with with Context2D

| | Perf | Memory | Dependencies | Fidelity | Styling | CJK | Context2d |
|--|--|--|--|--|--|--|--|
|Text| 👎 | 👎 | 👍 | 👍 | 👍 | 👍 | 👍 | 👍 |
|BitmapText| 👍 | 👎 | 👍 | 👎 | 👎 | 👎 | 👍 |

Ok.
Describes fragment interpolation and all non-native implementation:

1. Needs extra shader and extra data - thats will prevent batching.
2. Needs rebuild geometry when change (p.3).
3. Require a glyph-table for all glyphs existed in language, that require load specific TTF to memory totally, because ot can't support streaming, and then parse it. Looks like as SWF.
   Because we can't load system font as shape - we can't use it.
4. Language specifics rules are moving to lib team from browser team, that implement render rules: ltr/rtl, hindi/arabic glyph concatenation rules, etc... same as move 10% browser to lib. Unity wasn't implement it right still.
5. Heavy shader with extra arguments for supporting styles. That increases render cost, because we should render it in realtime.
   And of course we should be cache it as texture.

@eXponenta

• The text can be batched - although not with other stuff like graphics.

• Current Text implementation caches each glyph multiple times (if a letter is used multiple times across multiple PIXI.Texts). Having one geometry per glyph is better. Also, you don't need to cache at different font sizes.

• Geometry doesn't need to be rebuilt per say - if the letters in the text change, you need to fetch again from the glyph geometry table. If the transform changes, you just need to change one uniform.

• We can use a library for font parsing. The conversion to geometry is trivial.

• I don't know what glyph concatenation rules are. Could you explain?

• Left-to-right vs right-to-left - if you want to reverse the letters, you can. If you want mirror image, then set scale.x=-1 internally.

• Text styling won't add more arguments to the shader. It will change the geometry. Now, storing the glyph as a geometry takes _less_ memory than caching it in a texture.

• Font files fetched from APIs should be cached on the user's machine.

U very naive:

1. It is should be external batcher, because glyphs require extra attribute, such as bezier anchors, shadow values, gradient anchors ..., or all should be passed as uniforms - bye, batching.
2. right-to-left and left-to-right, and their mix is very complex problem.
   Opened issues in PDF rendering lib:
   https://github.com/asciidoctor/asciidoctor-pdf/issues/175
3. Hindi problem (and similar language)
   https://docs.microsoft.com/en-us/typography/opentype/spec/gsub

We can't use external dependences, because will be same problem as isMobile or resource-loader.
Core should be a clean. Minimal external dependencies.

@eXponenta

• Mixed layout (ltr and rtl combined) is less of a priority than high performance. You can always fallback to current algorithm. Similar for Devanagari. Special features like gradient stops will also need a fallback.

• Batching can work here because it will be easier without any textures. Can create a plugin like CanvasGraphicsRenderer.

Ok. U win. Do it. But outside core, then we will looks it and decide what we should do then.
But without heavy dependencies.

@eXponenta please stop being so hostile. We are exploring approaches nothing is solid yet. Please be constructive and do not use insults.

@eXponenta Are you sure we support right-to-left text as of now?

Another approach we are thinking is to render each glyph using canvas 2D (same as of now) separately and cache them into an atlas. The atlas can be reused globally for all PIXI.Text instances. Each glyph + font-size + font-style combination will be cached separately.

What do you think about accepting that into core?

Hey peeps, this is really exciting to hear these new approaches on the table. @eXponenta, echoing @bigtimebuddy, let's keep it constructive please. You clearly have some good insight into the new proposed approach, but lets aim to see if we can mitigate the issues raised and keep it friendly :)

On text, heres my 2 cents (pence?)..

Personally I think the most valuable approach is to gain the speed and performance of bitmap text but without the need to actually create a bitmap font beforehand. Dynamic bitmap texture building if you will!

Dynamic bitmap fonts

pros

• Tools for creating bitmap fonts an are few and far between! Would love to remove that barrier.
• we could create various sizes of text size textures.
• Works with Canvas
• Performance as each glyph would can be batched
• lots off cool effects!
• API would only need slight adjustment:

const textStyle = new TextStyle({
    font:'comic sans',
    fill:'bright green'
});


const bitmapFont = new BitmapFont(textStyle);

cons

• all the other cons of bitmap current bitmap fonts :P
• would dynamically updating the textures atlas cause slow down?
  
  
  
  • Although this would stabilise after a while, and could maybe be cached between sessions?
  
  
  • Current Text also has to upload a texture each time it changes frame, this approach would do it less?
  
  

ideas

• Could cache texture generated in a localstorage? for subsequent visits?
• Only add characters that are used, so textures is as big as it needs to be. Update as we go
• bake in filters to this texture for extra cool effects (outline, shadow etc)
• we can expore reducing memory of bitmap font by not making each glyph a sprite?

SDF
Can the SDF be an extension of the above approach? How complicated would it be to dynamically generate that texture? Would it be super janky? Are we talking increasing the code base by a large amount?

Most people have no idea what that is so it should ideally be hidden. If we need to externally generate the SDF fields or code required to generate is too large or complex then this would be better at home as a plugin for sure:

const textStyle = new TextStyle({
    font:'comic sans',
    fill:'bright green',
    sdf:true, <----- how sweet would this be, maybe rename to more user friendly like 'cleanEdges'
});

const sdfFont = new BitmapFont(textStyle);

Exact bezier curve rendering

This is an interesting approach for sure! I had looked into this in the past and decided that the shader switches introduced into the rendering loop and the extra tessellation required would have a large overhead. @SukantPal , you are closer to this idea than I am right now so would like to get your take on my concerns!

Having said that, maybe it might be worth building a prototype that we can test and bash about a bit to validate our thinking?

Exciting times for text :D

Can the SDF be an extension of the above approach? How complicated would it be to dynamically generate that texture? Would it be super janky? Are we talking increasing the code base by a large amount?

SDF generation is costly , and there are a packing time (when we nded pack different fonts in atlass), but it can be cached on client (as all other variant).
There is a canvas (hehe) implementation of it:
https://github.com/mapbox/tiny-sdf
But SDF compile is not fully correct (glyph change when increase/decrease font size)
Looks like as usiful, but for MSDF we should execute it 3 times (per channnel).

oh wow, thats a tiny bit of code! Looks ripe for a shader to be doing too!

@GoodBoyDigital The tiny-sdf package generates a single channel distance field. It doesn’t produce sharp corners when you cache the text at smaller font sizes. That can be mitigated by caching at the same font size being rendered.

Single channel SDF generation shouldn’t be too bad. The multi channel SDF is superior except that it is way too complicated. I couldn’t understand how Chumskly was encoding corners.

Now, everything we are considering won’t support rtl and Ltr layout mixed (as @eXponenta pointed out correctly). But I don’t think we are officially supporting it right now either.

The exact approach has two benefits - small cache size (you are caching vertices not all the pixels), antialiasing even when the setting “antialiasing” is turned off (text is expected to be antialiased always I think. Current implementation does that by using canvas 2D).

As for the demo, did you look at my quadratic Bézier curve demo? https://codepen.io/sukantpal/pen/GRJawBg?editors=0010

——
Another thing I and @bigtimebuddy discussed as the most simple solution was to keep using canvas 2D and cache the glyphs into a texture. The glyphs can be rendered as a quad. The cache atlas can be reused between texts. Of course, this means large text is being cached at same size and hence large memory usage when a lot of text is being rendered but only a portion of it (like current page in PDF) is visible.

Nice,

Yes, no worries about rtl or ltr that will always be available through current method and is a problem all custom text rendering techniques will need solving at a later date.
These decisions. should ideally be focus on memory consumption and runtime performance.

The Bezier demo is cool, but I do not feel it is enough for understand the true complexities that may be hiding if we render a full sentence.

The stuff I wrote about dynamic bitmapfonts above is pretty much exactly what you mentioned about glyph caching. I think that is definitely a useful route to go down!

In summary:

The Bezier curve solutions: This could be good route, but I feel this requires further R&D to figure how it fits into a real production use cases.

Dynamic Bitmapfonts from normal fonts: This we should definitely explore as we know this will give good perf whilst also keeping API simple. My Favourite :D

Single channel SDF Appreciate its not going to be as good multichannel SDF, but the demo looked pretty good to me! If text can scale nicely with and we can have one texture per font, then I think this route could worth investigating too!

@GoodBoyDigital I think the demo does have pretty good text. But there are visible differences if you try to look for details:

The text on the top is not sharp - the edges have wiggles. Each font+(~12-15px differences in font-size) combinations might need to be cached separately.

I totally fine with going for SDF if the those wiggles are unimportant or if we can go with caching at multiple font sizes.

The simplest solution is also fine for me 💯tbh!

Hey all.

I'm a super laymen on all things Pixi, but am a user of GDevelop which uses Pixi as its backend, and I've been digging into this for a while for GDevelop's implementation of PixiJS and text rendering. It's actually impactful for a game I'm working on that has a lot of text. (You can see my research/examples in the issue I've posted in GDevelop: https://github.com/4ian/GDevelop/issues/1449 )

One of the options I found was focused around outright ignoring the scaling issues with text and eliminating them altogether by always leaving text scale at 100%, but scaling the text font size instead? This is font agnostic, and seems to work at all sizes.

Here's a code example where that's being done using Pixi: https://codepen.io/Tazy/pen/wJVExB
I found it in this thread: https://www.html5gamedevs.com/topic/29576-scalable-text-for-pixijs/

I actually have a bounty on this issue as I was hoping someone could just modify the Pixi Text extension for GDevelop, but (being the layman that I am) I didn't realize that this was a much larger issue with Pixi itself.

No idea if there are potential issues with this method, but it would seem to avoid all of the performance issues of sdf/msdf/etc.

@Silver-Streak If you mean increasing the font-size by the scale applied on the text display-object, then how would that be better in terms of performance? SDF stuff helps performance because you don't re-render at larger font-sizes.

@Silver-Streak Of course, your proposal will improve _quality_ but I don't see how it will improve performance.

@Silver-Streak Of course, your proposal will improve _quality_ but I don't see how it will improve performance.

Unless I'm misunderstanding, wouldn't changing the font size use less resources than scaling the entire object, or loading in another renderer? Again, I'm super layman when it comes to how Pixi's scaling actually works, it just made sense to me that leaving the font at native resolution but just changing the text size would be faster than scaling it. If that's incorrect, apologies for the distraction.

@Silver-Streak Scaling is implemented as a transform. You have a transformation matrix - and changing the scale in that matrix is a trivial operation.

As of now, the text is rendered using Canvas 2D API into a canvas. Then it is copied onto the screen. Changing the scale will just change the screen coordinates to which the text in the canvas is mapped to.

Ahh, that makes sense. That's unfortunate, although to be fair my issue is more with the graphical rendering of fonts becoming blurry after being scaled, but I thought it'd be better performance too.

While I'd be excited for someone to implement it to fix the general text quality issues, I can totally understand wanting to improve performance as well.

Thanks for talking through it with me.

@Silver-Streak No problem. Implementing a quality-first text should not be hard in WebGL either, however. Some applications are implementing text by creating a mesh on their own. You can parse the font file, make a list of vertices, and tessellate it. Those vertices can be rendered through a mesh.

At high scales, this can cause slight "edges" - because ultimately, you are rendering the text as triangles. For even higher quality, you can use the "exact bezier curve" shader I was talking about in this thread - it will render the curves as curves and not triangles.

If you need help with text quality, I can help you guys out :)

@SukantPal I'm definitely not a contributor to GDevelop, (nor would you want me to be. I'm predominantly a Business Systems Analyst/DevOps in life, not a developer 😄 ) although I love that project and am active in the community. I also know there are other members of the community would love to see a solution for scaled text quality.

However, if you want to take a look at the issue in the post, I also have a bounty for it over on Bountysource as well open to all. I don't want to take up any more headspace here, though, as it's obvious there's a much deeper conversation going on around Pixi in general, and my suggestion wasn't as spot on as I thought it was.

@Silver-Streak I might take a look, since I've nothing better to do in corona-season

I know this is a closed thread, but I'm sure everyone is still looking for the best way to improve text rendering. I did noticed that someone had a msdf implementation that worked with Pixi v5. https://github.com/cjsjy123/pixi-msdf-text-v5 Figured I would mention it to those interested.

This thread should definitely be re-opened and kept alive - or at least have some sort of update every so often, as this is one of the most looked for issues.