Three.js: GLTFExporter GLB compatible with Facebook viewer

/ping @zellski

Hey! Yeah, that RGBA check should be gone in two weeks. Don’t work around it. :)

I'm trying to convert WaltHead.obj to glb. I'm loading it in https://threejs.org/editor/ and exporting it to GLB from there (which should have the latest patches already).

Here's WaltHead.glb and this is what I'm getting in Facebook's validator:

Your GLB File has the following errors: The 3D model could not be posted: The JSON portion of this model file is invalid.

🤔

It's syntactically valid JSON, but the nulls in this snippet from WaltHead.gltf don't conform to the type schema:

    {
      "bufferView": 2,
      "componentType": 5126,
      "count": 48480,
      "max": [
        null,
        null
      ],
      "min": [
        null,
        null
      ],
      "type": "VEC2"
    }

The Khronos glTF validator tool also lists about 10,000 instances of some other error in the file, too, all on the order of:

        /accessors/2: Accessor element at index 28922 is NaN or Infinity.

So it seems maybe an accessor is being generated during glTF export, to be filled with indices, but never actually receives any?

UVs are NaN: 🙃

@mrdoob @donmccurdy fixed! https://github.com/mrdoob/three.js/pull/13400
Although we still can't show that example because of

[msg] => Mesh primitives without indices are (temporary) unsupported.

(added to the to-do list)

@zellski do you have any estimation about that feature? ;)

@fernandojsg This one is a little trickier. It'll get fixed, but it might take a little longer. tl;dr Maybe a month?

Longer explanation: the issue is similar to the vertex colour one above, in that it's our client implementations that are lagging behind, and my validator on the backend is just protecting them from models they can't handle yet.

Obviously we have to support non-indexed geometry, the sooner the better. Ideally on the clients, but also by then I should have my backend code up to full Death Star power, where we transform all uploaded .gltf lazily/on-demand, depending on individual clients need. At that point we can do cool things like create indexed geometry on the server for the convenience of our clients.

I assume the error above comes when three.js is trying to export primitives that are naturally non-indexed in its runtime representation?

I assume the error above comes when three.js is trying to export primitives that are naturally non-indexed in its runtime representation?

@zellski that's it! Some primitives or objects loaded on three are non-indexed. The first use case I thought for the facebook glb loader was to include drawings from our A-Painter app (more info: https://blog.mozvr.com/tag/a-painter/) and we use non-indexed geometry there too, so it would be great to have support for that ;)
I just wanted to know if that was on the roadmap, so knowing that is it and we could have it in about a month, is more than reasonable ;) thanks for sharing that info!

We may have to add a dumb index attribute in the meantime (as in 0, 1, 2, 3, 4, 5, 6, 7, 8, ...) 😕

@mrdoob do you mean having some method to convert non-indexed to indexed as you want, or add the hack directly on the exporter?

Yeah, add a temporal hack in the exporter...

I don't know, I just want stuff to work and not have to tell people, "oh? your model doesn't work on facebook? that's because... you know what indexed geometries are? yeah, you shouldn't but..."

Yeah I got it! Ok I'll take a look to see if I could add some non-so-dirty hack :)

@zellski for context...

I've added a Export GLB in http://threejs.org/editor/ that uses this GLTFExporter.

Video: how to export model as glTF on Three.js Editor :D

https://twitter.com/superhoge/status/966689549803053056

Believe me, I understand the reluctance to add hacks. It's quite a struggle to maintain a patient outlook now that we've launched...

Could you make the GLTFExporter hack in a fork that's used in http://threejs.org/editor/ but not elsewhere? I would hope we'll have fixed this flaw by the time r91 comes out, so it seems a bit pointless for you folks to write careful, responsible code for it.

Could you make the GLTFExporter hack in a fork that's used in http://threejs.org/editor/ but not elsewhere?

Yeah, no worries!

I would hope we'll have fixed this flaw by the time r91 comes out

Oh, I'm aiming to release ~March 1st. Changed release cycle to beginning of the month to have proper month releases.

Seems like we still have more features to add before promoting this anyway. I don't think we are exporting roughness, metallic, normal or alpha maps.

Latest test, using 2 diffuse maps, clouds one being a transparent png:
https://www.facebook.com/fakemrdoob/posts/950266411809572

Not sure where the apparent alphaTest: 0.5 is coming from...

Just using the workaround for indices :)
https://www.facebook.com/fernandojsg/posts/10156405595122044

There we go https://github.com/mrdoob/three.js/pull/13410

@mrdoob do you mind adding the missing features you find and that are not related to the facebook requirements on this issue: https://github.com/mrdoob/three.js/issues/11951
Although I need to update the status there 😇

Sounds good. I'll test by exporting and dragging back to the editor.
Should we close this one?

@mrdoob I would leave it open until the RGB vs RGBA for vertex colors issue will get resolved on the facebook side, so if people come here looking for help they could read about it instead of opening another issue.

Btw I've just found this link with some useful info: https://developers.facebook.com/docs/sharing/3d-posts/glb-tutorials

Apparently textures need to be power of 2...
https://twitter.com/drupalati/status/967486854630055936

Doesn't Three.js automatically convert image from non-power-of-two to power-of-two on the fly?

Yes, sorry. The mobile clients don't resize yet, so we have to reject it during validation for a while. We will probably do the resizing on the server, since we have complete control over the delivery pipeline. I'm hoping this restriction too will be lifted in 2-3 weeks.

@takahirox Yes, three.js will resize on the fly. But Facebook's native clients don't use three.js.

The complete list of glTF features which Facebook do not yet support, in rough order of ETA:

• No RGB vertex colours; must be RGBA.
• No NPOT textures for certain clamping/filter combinations
• Only indexed, triangle geometry.
• No animations (silently ignored)
• No sparse accessors (fails validation)
• No morph targets (if any mesh has a 'target' property the model fails validation)
• No cameras (silently ignored) (for now)

Also:

• Maximum file size 3 MB
• No texture dimension larger than 4096
• No extensions other than KHR_material_unlit (for now)

I think that's it.

I made PR #13424 for force POT texture because I think it'd be worth not only for FB.

When using GLTFExporter to export an multi materials mesh (an array of material), I got this error:

GLTFExporter.js:623 Uncaught TypeError: Cannot read property 'toArray' of undefined
    at processMaterial (GLTFExporter.js:623)

@Ben-Mack That's a known issue and I'm working on it now. (But It'd take time a bit).

@zellski any plans for draco support on fb? I can get my meshes from 40MB down to 5mb but that last couple of MB just won't drop off.

@webprofusion-chrisc (man that’s a long @ for a phone keyboard) yes, Draco is most definitely on the roadmap. It needs a fair bit of engineering work so it’s at least a month out, but — in many ways we have built our assumptions around it — as you say, for many models the 3 MB limit is simply untenable. (I’m still not sure what we can do help with textures.)

(I’m still not sure what we can do help with textures.)

@donmccurdy is getting some progress on that front: https://github.com/mrdoob/three.js/pull/12877 😀

We can expect 25–30% smaller textures from GLTFExporter with the #12877.

Longer term, Binomial is working with Khronos to create an extension for cross-platform compressed textures in glTF: https://www.khronos.org/blog/call-for-participation-gltf-creating-a-compressed-texture-extension.

Okay... After #12877 and #13451 a GLB export that used to be 3.3MB is now 480KB 😊

Cool! #13451 meant, the image file size will be huger if we convert jpg to png?

Cool! #13451 meant, the image file size will be huger if we convert jpg to png?

Yes. #13451 is a bit of a workaround due to the fact that the editor doesn't allow changing the format of a texture at the moment. But we do the same thing in the library anyway...

https://github.com/mrdoob/three.js/blob/dev/src/loaders/TextureLoader.js#L34

But yes, GLTFExpoter currently saves as jpg when texture.format === THREE.RGBFormat.

https://github.com/mrdoob/three.js/blob/dev/examples/js/exporters/GLTFExporter.js#L493

Which is not ideal, because we're recompressing a jpg... But better than oversized exports I guess?

I had to add code to FBX2glTF that actually inspects the alpha values of even RGBA images, because people (or, more precisely, people's tools) create them by default, and quite often it's utterly unnecessary to keep them as PNGs. Even after trying the most brutal GPU-enabled, non-linearly-optimising PNG crunchers out there, it seems JPEG really rules the roost... the size difference is pretty incredible!

I am a little worried about inter-channel bleed for things like the ORM texture (occlusion/roughness/metal) where each component carries data that's utterly independent of the other components' data... but in practice it seems to work fine.

The exporter could also make quality level optional when using canvas.toDataURL( mimeType ) - my textures are generated at runtime from composite images, that would help too.

@zellski for fb's pipeline/viewer I'm guessing you could do like sketchfab and serve up a low res texture version, then stream the high/full res textures and replace them in the model as they load. Not a small piece of work but do-able.

@webprofusion-chrisc Yes, we may end up doing that. At the moment we've gone all out on .glb as the transmitted entity, which is hard to combine with selective LOD type streaming (since there's just a single sequential file with no random access). But I expect we'll reevaluate basic assumptions quite often, depending on how things go. :)

GLTFExporter can export BufferGeometryand import to Facebook fine. But any Geometry or BufferGeometry converted using method fromGeometry are not working on Facebook. I always receive this in FB validator:

[msg] => Vertex COLOR_0 attributes of type RGB are (temporarily) not supported. They must be RGBA.

Step to reproduce:

• Using misc_exporter_gltf example latest in dev, Export Sphere or Walthead working good on FB, but export Scene1 result cannot import into FB.

Is this some expected behavior and have to wait FB side?

I expected that BufferGeometry using fromGeometry should working extractly as normal BufferGeometry, please guide me some quick fixes to overcome this issue.

@Ben-Mack this is something that should be fixed in the next weeks according to @zellski -> https://github.com/mrdoob/three.js/issues/13397#issuecomment-367534958

As of build 161 and on (current App Store version is 160) of main FB app, this won’t be a crasher anymore and we’ll remove this validation check. I expect this to happen within the week.

@zellski that's awesome! thanks :)

Makes me wonder though...

The "real" reason THREE.GLTFExporter can't export THREE.Geometry is because when we convert THREE.Geometry to THREE.BufferGeometry we're creating a color attribute which is, in most of the cases, full of zeros.

So, one "solution" (and optimisation) would be to not export the color attribute if material.vertexColors is set to THREE.NoColors?

Ops I didn't know that :D that's for sure a must-do optimization :D

@fernandojsg thanks for the updates you made, much appreciated. There are two more things worth adding:

1. Multi-material meshes support. The ones that have groups in their geometries and array in Mesh.material - they currently cannot be exported properly;
2. Better compatibility between the only supported MeshStandardMaterial and the result we have on Facebook. So far metallic and diffuse surfaces look quite different in three.js and on Facebook. Maybe we'll have a special "Facebook" material one day?

Thank you

@ov I think both of those are likely to be fixed around r91:

1. multi-material export https://github.com/mrdoob/three.js/pull/13536
2. metal/roughness fixes https://github.com/mrdoob/three.js/pull/13501

It's possible that Facebook's materials aren't quite right yet either, but glTF is pretty specific about how things should appear so eventually we should converge.

Oh, we should add ability to export KHR_materials_unlit too...

EDIT: Opened https://github.com/mrdoob/three.js/pull/13566

The "real" reason THREE.GLTFExporter can't export THREE.Geometry is because when we convert THREE.Geometry to THREE.BufferGeometry we're creating a color attribute which is, in most of the cases, full of zeros.

So, one "solution" (and optimisation) would be to not export the color attribute if material.vertexColors is set to THREE.NoColors?

+1 hope this will be fixed soon. Lots of library for Three.js still performing based on THREE.Geometry.

( FB still having error ...must be RGBA with THREE.Geometry )

@Ben-Mack, which libraries are you using that still use Geometry? Perhaps we could work with the owners to get them updated.

@looeee Please take a look at this library: https://github.com/a-jie/threejs-geometry-modifiers

Hey folks -- as of this morning, Facebook no longer rejects RGB (VEC3) vertex colours as "invalid".

The power-of-two textures requirements remains for the moment, but I'm working on that too.

@zellski coool! :) I'm very close to have fully support for a-painter :D Is there any plan to implement the rest of the primitive modes? Right now I believe just TRIANGLE are supported, It could be great to support the rest, eg in A-painter we're using TRIANGLE_STRIP to save some space 👼

It would be crazy not to implement them, right? Ideally all valid glTF should be accepted. I don't know how that work will be prioritised, though. We're a small team with a lot of strong urges. :)

I think this issue can now be closed?