Three.js: HLS on Safari / IOS now not working

It is drawing to canvas and rendering with the canvas renderer albeit dropping frames. Something is required with webgl.

This is a beauty. I have modified a kpano demo a little with the proxied HLS stream.

It is working now on Safari OSX but not on IOS 9. Any ideas what the difference is ? I am not an expert with the webgl flags sadly or what they even do. On IOS9 it's still a black frame.

http://dev.electroteque.org/threejs/webgl.html

this last link is black rect / sound only in safari 9.1.3 / mac os x 10.11.6

What's HLS?

@makc really ? I have OSX 10.10 and Safari 10. Well that explains the recent sabotage change by Apple. Safari 10. So if we're Safari 10 we're all good.

@mrdoob HLS = Apple Streaming. ie for live streams but also VOD. Live streaming is going to become a norm with the Terradeck gear. ie http://dev.electroteque.org/video/360/hls/ultra_light_flight.m3u8

Any ideas what the differences could be there ? I tried changing some things in the first example. The second example is raw webgl functions.

so texImage2D will partially render HLS but not on IOS yet maybe because it's not Safari 10 ?

Please forgive me I have reported on a few webkit bug tickets.

They've gone and sabotaged the CORS proxy hack on an IOS update. macOS also is reported to not have fixed the CORS issue in Safari either.

I have to now figure out how to work around their sabotage. I have to get that working to further test HLS working on IOS. Apple is seriously making me work up a sweat to put out their fires hahah.

If I can get OSX rendering that would be a start. I just need to figure out what is different in three.js ?

I'm still trying to do reduction methods to compare what is different. Once I figure that out I may get IOS working too.

Perhaps the extra shader programs within three.js is causing the rendering problems ? I'm not across it, but trying to replicate in the raw webgl example what three.js is doing in that equirectangular example. Is there example shader code for this ?

I found some raw functions here which hopefully I can replicate fully what three.js is doing. I can't just copy the shader programs directly because some variables are set externally like world position.

https://bl.ocks.org/mbostock/5446416

I may have just run across this or possibly a related issue. I have h264 mp4 files located on an S3 bucket and I am loading them to a video.src as a blob.

If I upload to my S3 bucket using the Amazon web-based uploader, the videos play fine when directly loaded in Mobile Safari (iOS 10.0.1), but if I try to play them using my three.js viewer then I just get a black screen. However, if I upload to the S3 bucket with Cyberduck, they play fine directly in Mobile Safari and in three.js.

It looks like Cyberduck is setting the meta-data 'Content-Type=video/mp4' and the AWS uploader is 'Content-Type=application/octet-stream'. When I manually set this meta-data to video/mp4 everything works correctly.

Strange issue, and one that caused me a lot of head scratching. Let me know if this is the same underlying issue as this ticket, or if I should create a new one.

Thanks,
Dustin

This is for HLS mate as in mpegts fragments or fragmented files not mp4. Mp4 is fine on both Safari and IOS. I checked the mimetype in cyberduck and it says the mpegts is correct.

The same stream is working for me in the raw webgl example just not on IOS of course. Your issue seems similar to mine though as in a black frame. No rendering at all.

@danrossi ok cool. I'll create a new ticket specifically for my issue. Thanks.

I think this is the issue: www.krpano.com/ios/bugs/ios10-webgl-video-texture-crash/
I'm currently trying to implement the same workaround for three.js, but not getting very far.

@phaseOne. It is a safari issue as both a problem on OSX and IOS. Mp4 is working fine for IOS and OSX. I'll tinker with antialias .

Still can't work out the differences between the two. The raw webgl example still won't work on IOS though but getting it working on OSX is a start.

I'm wondering if I can find a working equirectangle shader program that I can drop in.

More info here: http://stackoverflow.com/questions/39123109/rendering-a-video-with-webgl-in-ios-10-beta-7-safari-shows-weird-purplish-co.

I realize that this is a browser bug, but implementing a workaround might be necessary given how long these WebGL bugs take to get fixed.

In the case to use the texture through the canvas, the video was displayed in safari.
However, since there are still CORS problems in iOS, it worked only in the Same Origin.
https://bugs.webkit.org/show_bug.cgi?id=135379

I'm not familiar with the webgl, but please refer to the following.
https://github.com/NishimuraT/videojs-panorama/commit/bd99200d8831c7ad0d10d742e087953da0f44169

@NishimuraT I think I know what you mean use the canavasrender. I did try that to do reduction method.

I think that was working but it causes severe drop frames and playback performance issues due to cpu. Obviously worse on IOS. Ipad 3 drops frames for webgl already.

@danrossi I had a little misunderstanding. I'm sorry. I had sent a misguided thing.

I have confirmed the same problem with the elevr project which uses raw webgl and shaders. Will have to hack that one to work my way back. Something in there is causing rendering issues and would affect three.js the same.

I have confirmed in elevr it is auto clearing binding the texture which is causing it to break ie

webGL.gl.bindTexture(webGL.gl.TEXTURE_2D, null);

If this is commented out it is working.

This was also causing a problem, I'm not sure what it's for.

webGL.gl.pixelStorei(webGL.gl.UNPACK_FLIP_Y_WEBGL, true);

Maybe there is an auto clear property that is doing the same thing in three.js ?

I see many references to

state.bindTexture(_gl.TEXTURE_2D, null);

But don't know where it could possibly be. Is it required to do that ?

code is too crazy to track down what is causing the texture drawing issue. Seems to have something with this binding of texture.

@danrossi is there any page out there that renders it properly?

Sorry about the noise mate. I did more AB testing. It is indeed it seems what I mentioned, as I had to comment it out for elevr demo also.

_gl.pixelStorei( _gl.UNPACK_FLIP_Y_WEBGL, texture.flipY );
https://github.com/mrdoob/three.js/blob/6c7f000734f8579da37fb39e5c2e9e5e2dfb14f8/src/renderers/webgl/WebGLTextures.js#L411

This is causing the problem. However without it the texture is upside down. I'll add another test to show it now working.

http://dev.electroteque.org/threejs/hlsbroken.html
http://dev.electroteque.org/threejs/hls.html

Another raw example I found, commented out FLIP_Y. Possible Webkit bug now ?

http://dev.electroteque.org/threejs/webgl3.html

The only reference to flipY is this but webkit nightly still an issue

https://bugs.webkit.org/show_bug.cgi?id=162491

If I disable flipY which I assume breaks hardware acceleration it is working ?

texture = new THREE.VideoTexture( video );
                texture.minFilter = THREE.LinearFilter;
                texture.format = THREE.RGBFormat;
                texture.magFilter = THREE.LinearFilter;
                texture.flipY = false;

http://dev.electroteque.org/threejs/hlsflipy.html

Trying to see if it's a webkit bug now. How is it possible to rotate the texture around to work around it for now ?

How is it possible to rotate the texture around to work around it for now ?

why not just flip geometry uv or do that in the shader

@makc I have no idea how to go about that.

Just to do heads in further. This FlipY "fix" doesn't help on IOS 9, just tested it.

Something else is a problem there now. It took days just to track this one down.

It may be all webkit's problem not Three.JS from the looks of it. As per usual so sorry about this.

I have filed a report there too https://bugs.webkit.org/show_bug.cgi?id=163866

I have no idea how to go about that.

http://jsfiddle.net/hfj7gm6t/2182/

@makc confirming flipping the uv around helps. How does it inverse it with this ?

uv.setY (i, 1 - uv.getY (i));

http://dev.electroteque.org/threejs/hls.html

Problem still evident on IOS though.

@makc. It's quite possible this shader program is doing the flip, this was in one of the demo sources.

attribute vec2 vx;varying vec2 tx;void main(){gl_Position=vec4(vx.x*2.0-1.0,1.0-vx.y*2.0,0,1);tx=vx;}

It's found in the original demo that seemed to be working because it doesn't use the FlipY flag it seems to be doing it in the shader program.

http://dev.electroteque.org/threejs/webglnoflip.html

No idea how to integrate that into three.js ? Would it be better to use a shader program ?

I'm still debugging IOS now.

This one is interesting. By attempting to replicate that calculation into this example it seems to be flipping it but scales it badly still.

http://dev.electroteque.org/threejs/webglflipped.html

So gl_Position controls it's position.

 gl_Position = vec4(aVertexPosition.x*2.0-1.0,1.0-aVertexPosition.y*2.0,0,1);

Would it be better to use a shader program ?

shader program is executed once every frame, vs flipping uvs in the geometry just once before the whole thing starts.

@makc probably best just set the values at the start then. If I am correct, these uv values are variables in the shader program and used in the calculation ?

I see in the sources a few references to gl_Position. Not sure if this is the right one but there is one like this

gl_Position = projectionMatrix * modelViewMatrix * vec4( position, 1.0 );

Thanks.

I tried all kinds of tricks with webgl on IOS Safari to see if I can get a frame to render and it's just black for now. Mp4 is fine.

Not much resources about this. I believe my discovery people are looking for also. It's for live streaming btw but VOD also.

I had someone test some things out. They believe this raw webgl example with the FLIPY fix works on IOS 10 but the three.js version doesn't in fact it's crashing on them.

I noticed it used the NEAREST filter which makes the texture blocky , I tried to replicate that but same problem for them.

so neither of these work

http://dev.electroteque.org/threejs/hls.html
http://dev.electroteque.org/threejs/hls2.html

But this is for them although IOS 9 for me does not

http://dev.electroteque.org/threejs/webglworking.html

After some very heavy and painful testing I have fined tuned the demos and the issue.

The FlipY work around is required for HLS across both OSX 10.11 and macOS , IOS 9 and 10. The raw webgl example doesn't use FlipY because its doing in in the shader program. With three.js the geometry has to be flipped around.

HLS rendering on IOS 10 is displaying but has severe colour artifact issues. The frames stop working but I believe its an issue with the emulator and frames dropping. I now have no device that can be updated to IOS 10. It doesn't show up at all on IOS 9. Both require the CORS proxy for mp4 and HLS.

I may have to now provide another ticket in regards to the colour rendering problem with HLS on IOS 10 once I confirm it's not the emulator.

This is absolutely painful no wonder nobody wants to support Safari but people need it working.

I've had to provide here different rgb format settings. The default one produces vertical coiourbars , the rgba one the colours are incorrect. But in the emulator. I am now unable to properly test IOS 10 because my Ipad device has become unsupported. I'll try and get a hold of an Iphone with IOS 10 to confirm.

http://dev.electroteque.org/webgl/threejs-hls.html
http://dev.electroteque.org/webgl/threejsrgba-hls.html

http://dev.electroteque.org/webgl/webgl.html
http://dev.electroteque.org/webgl/webglrgba.html

Please concentrate on these two examples. the RGB flags make no difference on an actual device, the simulator produces faulty output but similar output when using the RGBA format.

The FlipY work around is required for IOS 9 / 10/ OSX Safari with HLS.

I have been able to replicate a colour issue with HLS webGL rendering which now needs a second ticket although I'm not sure if a Webkit flakey bug is a three.js issue.

So many massive flaws and bugs causing this to be a show stopper.

So such HLS webgl output is only partially working on IOS 10 not IOS 9. Many devices are stuck on 9 now. I need to find a work around for 9.

http://dev.electroteque.org/webgl/threejs-hls.html
http://dev.electroteque.org/webgl/webgl.html

Hi! I have the same problem. Do you have any solution now ?

yes its in those demos but check the second ticket regarding a colourspace problem with IOS 10. It seems there will be no effort on Webkit / Apple's behalf to ever get it working on IOS 9 which means older devices like the Ipad 3 are left out although it could barely render 5fps anyway. Have to concentrate first getting IOS 10 to function.

It is a work around to this problem though which has been reported to Webkit. It's a problem on all Safari.

http://dev.electroteque.org/webgl/webgl.html seems not working. I tested the page on ios10. But safari crash when the page load.

Use the threejs example.

I tested all examples listed on this issue. It seems that only http://dev.electroteque.org/webgl/webglrgba.html works. But it has a colourspace problem.

@wuyingfengsui this is working on IOS 10 Iphone.

http://dev.electroteque.org/webgl/threejs-hls.html

You are probably checking on a simulator I bet. That is a massive problem. Not only will it not update the frame it displays green colourbars. When using the RGBA flag it will work on a simulator though. There is a wierd graphics issue going on there.

On an actual IDevice you will see the colourspace problem when using the RGB flag like you do with the RGBA flag in the simulator !

I am pushed out testing IOS 10 properly for now because Ipad 3 has been made obsolete but thankfully I had access to an Iphone.

Take note of the CORS proxy url on it too.

Refer to this regarding the colourspace problem now

https://github.com/mrdoob/three.js/issues/10067

and this webkit ticket which I doubt they will ever bother to fix it and get a release pushed.

https://bugs.webkit.org/show_bug.cgi?id=164540

I find it work when I change RGB to RGBA on ios 10 iphone 7.

ok so its different for different devices nice one. I've updated it to RGBA. Hows this ?

http://dev.electroteque.org/webgl/threejs-hls.html

@danrossi It works.

@wuyingfengsui I'll have to update the webkit ticket in regards to that. I see there are other colour flags to mess with and hopefully one is the right one. I tried different HLS streams and they were the same.

You can see in the three.js constants there is a range of values to play with. ie these. There is also byte type flags that are handled. Trial and error and very tiring trial and error.

https://github.com/mrdoob/three.js/blob/dev/src/constants.js#L126

So sadly IOS 9 has been made completely redundant with this it won't work. Older devices will never get IOS 10.

I believe there is some kind of sabotage going on and I believe it was working before the release of Safari 10 ? my testing phase took months because of all the inconsistencies with Apple and sometimes Android. I am fairly certain when I tested HLS it was working.

I'm not familiar with webGL. I think it can't be solved until the webkit fix it.

This is what uploads the texture to webgl. those flags in there are of interest.

gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, video);

@wuyingfengsui if it took them over 5 years to partially "fix" the CORS issue in macOS then don't hold your breath.

That is why somebody has to sit there and figure out work arounds and sadly that is myself hahah.

If you take the raw webgl example , those flags can be messed around with. Or do it via three.js using the constants and the properties "format" and "type" the type is the byte type, the format is the colour format which you've seen RGBA is required to barely make it function.

I'll spend some time tomorrow to go through each flag and hopefully one combination works.

@danrossi How is it going ?

I just tried messing with every possible flag and nothing. It can only take RGB and RGBA flags and the byte type is unsigned. Other options just showed a black canvas. Not sure what else to do ?

@danrossi Thanks for your work. I find Hls.js play the hls video with MSE. But Safari Mobile doesn't support MSE.

@wuyingfengsui this is a native HLS problem with IOS now, my fixes above works around the bug for Safari OSX.

with Dash streaming, older Safari had CORS issues even with the proxy work around. Maybe in macOS they have fixed that too ?

There is possibly nothing that can be done other than provide a work around method in three.js using the code fixes above.

I doubt webkit and Apple will ever fix it. So all eyes on this ticket now. the RGB flag stuff is related to IOS 10 rendering now. IOS 9 and older devices trapped on it is stuffed.

https://github.com/mrdoob/three.js/issues/10067

Any updates here? Does anyone got it working with fixed color rendering on iOS?

Please refer to this ticket.

https://github.com/mrdoob/three.js/issues/10067

I doubt the HLS work around will go into three.js it has to go outside it.

The code for that work around is as follows

texture = new THREE.VideoTexture( video );
texture.minFilter = THREE.LinearFilter;
texture.format = THREE.RGBAFormat;
texture.magFilter = THREE.LinearFilter;
texture.generateMipmaps = false;
texture.flipY = false;

var geometry = new THREE.SphereBufferGeometry( 500, 64, 44 );
geometry.scale( - 1, 1, 1 );

var uv = geometry.getAttribute('uv');
for (var i = 0; i < uv.count; i++) {
     uv.setY (i, 1 - uv.getY (i));
}

If you'd like to mess around with the webgl flags, use the raw webgl example. I tried all possible options and nothing right now. It's well and truly sabotaged and no response from webkit.

http://dev.electroteque.org/webgl/webgl.html

Thanks. Is there any other video formats to stream to iOS video flag besides HLS? Is it possible to stream H.264 without the rendering problems?

@pedrofranceschi mp4 is fine. It's just HLS which stuffs options for live streaming 360 video on IOS.

The flipY fix related to this ticket needs to be applied to HLS streams and on Safari only . This FlipY fix does not work on IOS 9 which shuts out older devices that can't get IOS 10.

My Ipad 3 Ive been using for testing will never get HLS and WebGl working. I had to test on an newer Iphone. The IOS 10 simulator barely worked and had colour artefact issues similar to the other problem.

At least the inline flags for Iphone on IOS 10 worked, inline video and webgl is possible now but CORS is still a problem. CORS is only fixed in Safari on macOS.

I make the point of Safari only as it's another classic webkit bug and HLS.JS streaming on other browsers is ok.

I found it can be fixed by adding an extra canvas. But it only work on iOS 10. Look at this commit: https://github.com/yanwsh/videojs-panorama/commit/0122b1bbd31093b77ca7f09900afa74e2c537037

That was a non event. It's using canvas rendering with drawImage. Not WebGL. So like using the three.js canvas renderer. Canvas renderer really sucks on IOS causes dropped frames, not usable. Is this your fix ?

The quality isn't very good. But it works :)

Dropping frames is more than a quality issue :)

@mrdoob I've tried the canvas renderer as a work around but does this for me. Im not sure what its problem is there, its displaying a mesh. the rotation is not right either. Loading an mp4 file it displays properly but still the mesh lines.

It still drops frames doing this.

http://dev.electroteque.org/webgl/canvas-hls.html

@danrossi

@mrdoob This issue is about HLS on Safari. Its really hard doing all these demos. But I have made some combinations.

I've also just worked out Chrome refuses to play back mp4 now unless its using the Safari CORS proxy version of the mp4 ahh.

As far as the bizarre CanvasRenderer issue goes which may fix the HLS problem temporarily for IOS 10. Here is combinations so show what its doing and some strange rotation value offsets possible. The code is all the same. There is a mesh grid over the texture.

For Safari

http://dev.electroteque.org/webgl/webgl-mp4.html
http://dev.electroteque.org/webgl/canvas-hls.html

For Chrome

http://dev.electroteque.org/webgl/webgl-webm.html
http://dev.electroteque.org/webgl/canvas-webm.html

@danrossi The Safari version also doesn't work on iOS 10…

This should work on IOS 10 but you should see colour artifacts.

http://dev.electroteque.org/webgl/threejs-hls.html

Those other ones are CanvasRenderer tests to try and work around the IOS issue, it should at least be working on desktop browsers and isn't.

This should work on any IOS but Im not too concerned about that one.

http://dev.electroteque.org/webgl/webgl-mp4.html

I don't believe the CanvasRender solution is going to be a good work around for IOS but it's an option. Because not only does it still need the CORS proxy, canvas drawing is only going to cause dropped frames. Safari OSX can use the FlipY work around in threejs-hls.html

I've updated the canvas demos, it doesnt seem it needs drawing externally.

however there is a strange problem with the sphere geometry I can't figure out therefore the picture doesn't render or rotate correctly, it's all skewed into a point.

Click the top left area, and try to drag with orbitcontrols.

The mesh issue is I need to add "overdraw: 0.5" to the MeshBasicMaterial to stop it showing the mesh.

Neither require CORS proxies on safari when using the CanvasRenderer. I will attempt to look at SoftwareRenderer next to see if that will work better as CanvasRenderer drops too many frames.

http://dev.electroteque.org/webgl/canvas-mp4.html
http://dev.electroteque.org/webgl/canvas-hls.html

Considering the CanvasRenderer has been marked depreciated because of the obvious dropping frames. I guess the only last resort is to try video textures with the SoftwareRenderer. That change doesn't seem to function yet and can't find an example setup. It's just a black canvas.

Any workarounds for the color glitch? Everything is blue-ish

@andreabadesso I have two tickets setup maybe I should merge them.

For OSX, doing the FlipY work around will fix that nasty problem. For IOS 10 , something else is required entirely.

Canvas drawing is a massive failure, it's not even rendering properly out of the box , the gemoetry is distorted it seems. It drops frames also.

I am now trying my luck with "SoftwareRenderer" which still relies on canvas drawing. All I get is a black canvas right now so nothing. There is zero documentation how to get video textures working with it.

There is simply no other option, webkit won't respond and I have tried all possible webgl flags. I could go back and try and mess with encodings but the HLS is packaged from the working mp4 file.

Meanwhile, if anyone needs a desperate solution, this will watch for .ts files, convert them to .mp4 and emit a websocket so the client can download the next fragment in mp4 (which is working on ios10):

https://github.com/Lab21k/node-hls-mp4-ws/

I can provide the full solution if anyone is interested.

@andreabadesso Have you the client implementation about your solution ?

That is extremely dodgy. Not only do you still need a CORS proxy, it's attempting to pass through FFMPEG in realtime :O Does IOS even do websockets ? I know it can't do webrtc.

Trying to get a video texture via canvas drawing still via the SoftwareRenderer option is bad also. It's just a black frame. Both options have been a dead end I was hoping at least the SoftwareRenderer one works with video textures only an image texture displays.

Hey everyone, I've been working on the color issue for the past day and I've found a work-around that works for my use-case. This thread has been very useful and I'd like to contribute.
The issue is just that the red and blue colours get swapped around. So I used ffmpeg to pre-switch those colours. As the colours get re-switched on threejs the video goes back to its normal appearance.
It's not an optimal solution but it does work. (tested on IOS 10.2)

@Yralec

I figured it was an encoding issue or mucking around with encoding.

That isn't going to help say for wowza live streaming though. Nobody will likely have control how live encoders work. For VOD perhaps.

Do you have the ffmpeg flag for that to confirm ?

Still amazing. It sounds like this isn't a problem with three.js apart from the FlipY issue which is still needed.

I just cannot fathom why Webkit is like this.

@danrossi

Here's the command I used:
ffmpeg -i in.mp4 -hls_time 10 -hls_list_size 0 -vf "colorchannelmixer=rr=0.0:rb=1.0:bb=0.0:br=1.0,vflip" -pix_fmt yuv420p out.m3u8
As you can see I'm also flipping the output so I don't need to do that in threejs myself.

ok I was going to ask about that.

Wouldn't it be easier to write a shader that swaps blue and green?

@mrdoob I have very little knowledge about shaders so for me it wasn't (I also didn't even think of that), but if you know how to write one that swaps red and blue it's nearly surely a better solution.
Also live streaming might finally work.

@mrdoob I think you are onto something. I'm sorry mate yet again this has become another Webkit problem. The FlipY fix is not avoidable still though.

I did some research and it seems there is a channel order swap like this , I am noway an expert with shaders.

gl_FragColor = texture2D(u_image, v_texCoord).bgra;

@mredoob in my raw webgl example I tried something like this but the program is broken and causes errors. So this is where it obtains colours from the texture which is obviously very bogus and "bgra" swaps the red and blue channels.

No idea yet how to get a working solution from this.

 gl.shaderSource(ps, "precision mediump float;uniform sampler2D sm;varying vec2 tx;void main(){gl_FragColor=texture2D(sm,txt).bgra;}");

There was a program error simply copying and pasting.

Using this code above. I can confirm what is being rendered is what is visible in IOS 10 so it should hopefully invert the red and blur colours that are swapped in the shader ?

http://dev.electroteque.org/webgl/webglbgra.html

@Yralec confirming inverting the colour channel order on the shader gets inverted back in IOS10. In the simulator I had to change the colour format to rgba to get it rendering without strange lines through it.

gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, video);

@mrdoob how is it possible to apply this texture colour change on the built in shader programs ?

gl_FragColor=texture2D(sm,txt).bgra;

@mrdoob. Correction. This might be the one people would be using.

gl_FragColor = texture2D( tEquirect, sampleUV )

var equirect_frag = "uniform sampler2D tEquirect;\nuniform float tFlip;\nvarying vec3 vWorldPosition;\n#include <common>\nvoid main() {\n\tvec3 direction = normalize( vWorldPosition );\n\tvec2 sampleUV;\n\tsampleUV.y = saturate( tFlip * direction.y * -0.5 + 0.5 );\n\tsampleUV.x = atan( direction.z, direction.x ) * RECIPROCAL_PI2 + 0.5;\n\tgl_FragColor = texture2D( tEquirect, sampleUV );\n}\n";

How to get access to it to change it ?

@danrossi You can hack something like this in JS

THREE.ShaderLib[ 'equirect' ].fragmentShader = THREE.ShaderLib[ 'equirect' ].fragmentShader.replace( "texture2D( tEquirect, sampleUV );", "texture2D( tEquirect, sampleUV ).bgra;" );

I believe I need to copy the shaders into a THREE.ShaderMaterial, but no idea how to apply the texture ?

var equirect_frag = "uniform sampler2D tEquirect;\nuniform float tFlip;\nvarying vec3 vWorldPosition;\n#include <common>\nvoid main() {\n\tvec3 direction = normalize( vWorldPosition );\n\tvec2 sampleUV;\n\tsampleUV.y = saturate( tFlip * direction.y * -0.5 + 0.5 );\n\tsampleUV.x = atan( direction.z, direction.x ) * RECIPROCAL_PI2 + 0.5;\n\tgl_FragColor = texture2D( tEquirect, sampleUV );\n}\n";

                var equirect_vert = "varying vec3 vWorldPosition;\n#include <common>\nvoid main() {\n\tvWorldPosition = transformDirection( position, modelMatrix );\n\t#include <begin_vertex>\n\t#include <project_vertex>\n}\n";


                var uniforms =  {
                    tEquirect: { value: null },
                    tFlip: { value: - 1 }
                };

                var material = new THREE.ShaderMaterial( {
                    uniforms: uniforms,
                    vertexShader: equirect_vert,
                    fragmentShader: equirect_frag
                });

That other way is inflexible as it has to be applied just for HLS streams in IOS.

actually this renders in safari but the FlipY fix breaks. It plays upside down. the mind boggles. All this effort because of Apple and Webkit haha.

var uniforms =  {
                    tEquirect: { value: texture },
                    tFlip: { value: - 1 }
                };

                var material = new THREE.ShaderMaterial( {
                    uniforms: uniforms,
                    vertexShader: equirect_vert,
                    fragmentShader: equirect_frag
                });

@WestLangley the patch doesn't seem to work. I should be seeing colours inverted but don't.

gl_FragColor=texture2D(sm,txt).bgra;

this is correct shader to swap [b]lue and [r]ed: default order is rgba, so bgra swaps them.

FlipY fix breaks

txt -> vec2(txt.x, 1.0 - txt.y) for example

no idea how to

http://jsfiddle.net/p2duvg51/14/

the patch doesn't seem to work. I should be seeing colours inverted but don't.

maybe you are applying the patch too late

@makc yes in my raw webgl example its doing the flip in the shader.

I believe if the flip is done on the UV code which is posted in my threejs example it is not flipped frame by frame in the shader. I thought that would remove an unnecessary calculated to process.

in the raw webgl example its flipped on the vertex shader

attribute vec2 vx;varying vec2 tx;void main(){gl_Position=vec4(vx.x*2.0-1.0,1.0-vx.y*2.0,0,1);tx=vx;}

then in the fragment shader this inverts the color channels.

precision mediump float;uniform sampler2D sm;varying vec2 tx;void main(){gl_FragColor=texture2D(sm,tx).bgra;}

trying to replicate that in three.js is the problem now.

Excuse me this seems to work but the FlipY fix doesn't work. I forgot to add bgra.

var equirect_frag = "uniform sampler2D tEquirect;\nuniform float tFlip;\nvarying vec3 vWorldPosition;\n#include <common>\nvoid main() {\n\tvec3 direction = normalize( vWorldPosition );\n\tvec2 sampleUV;\n\tsampleUV.y = saturate( tFlip * direction.y * -0.5 + 0.5 );\n\tsampleUV.x = atan( direction.z, direction.x ) * RECIPROCAL_PI2 + 0.5;\n\tgl_FragColor = texture2D( tEquirect, sampleUV ).bgra;\n}\n";

                var equirect_vert = "varying vec3 vWorldPosition;\n#include <common>\nvoid main() {\n\tvWorldPosition = transformDirection( position, modelMatrix );\n\t#include <begin_vertex>\n\t#include <project_vertex>\n}\n";




                var uniforms =  {
                    tEquirect: { value: texture },
                    tFlip: { value: 1 }
                };

                var material = new THREE.ShaderMaterial( {
                    uniforms: uniforms,
                    vertexShader: equirect_vert,
                    fragmentShader: equirect_frag
                });

If I change the default tFlip value to 1 it's displaying the right way up, so another fix option for the FlipY problem with HLS ?

tFlip: { value: 1 }

the video has decided to stop rendering in the simulator but at least I see a colour change. The patch doesn't work sadly although debugging it it definitely replaces the shader code.

It seems I can do a flip with that value only giving it a positive 1 rather than negative.

I lost access to IOS10 once they deprecated older devices getting it like my Ipad 3. I have to load the simulator on macOS booted on an external drive because I have software that only runs in 10.10. Fun times with all this sabotage.

I have no idea if its updating the frame and rotating properly yet.

Hopefully this example works on IOS 10. Its the best I can do right now. In Safari it should show inverted colors.

http://dev.electroteque.org/webgl/threejs-hls.html

Using the shadermaterial screws up the image scaling, geometry and stops the camera rotation from working like this

camera.position.x = (Math.PI / 2);

Have to try and get this patch to work.

This seemed to work although it still messes with the camera position. I don't think its a viable option. It produces even more bugs just doing it this way. Does not render the same as using MeshBasicMaterial.

THREE.ShaderLib[ 'equirect' ].fragmentShader = THREE.ShaderLib[ 'equirect' ].fragmentShader.replace( "texture2D( tEquirect, sampleUV );", "texture2D( tEquirect, sampleUV ).bgra;" );
                THREE.ShaderLib[ 'equirect'].uniforms.tEquirect = { value: texture };
                THREE.ShaderLib[ 'equirect'].uniforms.tFlip = { value: 1 };

                var material2 = new THREE.ShaderMaterial( {
                    uniforms: THREE.ShaderLib[ 'equirect'].uniforms,
                    vertexShader: THREE.ShaderLib[ 'equirect'].vertexShader,
                    fragmentShader: THREE.ShaderLib[ 'equirect' ].fragmentShader
                });

If I tried to use the basicmaterial it wont work. ie

THREE.ShaderLib[ 'equirect' ].fragmentShader = THREE.ShaderLib[ 'equirect' ].fragmentShader.replace( "texture2D( tEquirect, sampleUV );", "texture2D( tEquirect, sampleUV ).bgra;" );
                var material  = new THREE.MeshBasicMaterial( { map : texture } );

Rather than trying to hack the built-in shader, I would just create a custom shader.

Here you go! 😀

const WIDTH = window.innerWidth;
const HEIGHT = window.innerHeight;

var camera = new THREE.PerspectiveCamera( 75, WIDTH / HEIGHT );

var scene = new THREE.Scene();

// geometry

var geometry = new THREE.SphereGeometry( 1, 32, 16 );

// material

var loader = new THREE.TextureLoader();
var texture = loader.load( 'https://threejs.org/examples/textures/2294472375_24a3b8ef46_o.jpg');

// material

var material = new THREE.ShaderMaterial( {
    uniforms: {
        texture: new THREE.Uniform( texture )
    },
    vertexShader: [
        "varying vec2 vUV;",
        "void main() {",
        "   vUV = vec2( 1.0 - uv.x, uv.y );", // fipY: 1.0 - uv.y
        "   gl_Position = projectionMatrix * modelViewMatrix * vec4( position, 1.0 );",
        "}"
    ].join( "\n" ),
    fragmentShader: [
        "uniform sampler2D texture;",
        "varying vec2 vUV;",
        "void main() {",
        "   gl_FragColor = texture2D( texture, vUV ).bgra;",
        "}"
    ].join( "\n" ),
    side: THREE.BackSide
} );

var mesh = new THREE.Mesh( geometry, material );
scene.add( mesh );

var renderer = new THREE.WebGLRenderer();
renderer.setSize( WIDTH, HEIGHT );
document.body.appendChild( renderer.domElement );

function animate( time ) {
    requestAnimationFrame( animate );
    mesh.rotation.y = time * 0.0001;
    renderer.render( scene, camera );
}

animate();

https://jsfiddle.net/9jy92zxn/

Doesn't render just a black screen.

@danrossi It work for me.

var material = new THREE.ShaderMaterial({
      uniforms: {
        texture: { value: texture }
      },
      vertexShader: [
        "varying vec2 vUV;",
        "void main() {",
        "   vUV = vec2( uv.x, 1.0 - uv.y );",
        "   gl_Position = projectionMatrix * modelViewMatrix * vec4( position, 1.0 );",
        "}"
      ].join("\n"),
      fragmentShader: [
        "uniform sampler2D texture;",
        "varying vec2 vUV;",
        "void main() {",
        " gl_FragColor = texture2D( texture, vUV  ).bgra;",
        "}"
      ].join("\n")
    });

Remember update the color type and variable names

It renders here... (Chrome 55, OSX)

This issue is about Video textures with HLS streams and Safari especially IOS I'm afraid :|

I can't see how the meshbasic_frag works and how a texture is applied to it. That is what it is using.

The equirect_frag program I can't see being used or selected.

gl_FragColor = vec4( outgoingLight, diffuseColor.a );

This is all I see.

I think you should forget about meshbasic_frag and equirect_frag. Try using my code as base and replace this:

var loader = new THREE.TextureLoader();
var texture = loader.load( 'https://threejs.org/examples/textures/2294472375_24a3b8ef46_o.jpg');

With something like this:

var video = document.getElementById( 'video' );
var texture = new THREE.VideoTexture( video );

@mrdoob

the backside was causing the problem. This shader program is certainly much cleaner than the usual one. What is the difference ?

Using the custom shader I lose the original FlipY fix.

for (var i = 0; i < uv.count; i++) {
    uv.setY (i, 1 - uv.getY (i));
}

I had to modify the vertex code a little, x was flipped, and y needed inverting. I'll put that up in a sec.

var uniforms = {
    texture: { value: texture }
};

var vertexShader = [
    "varying vec2 vUV;",
    "void main() {",
    "   vUV = vec2( uv.x, 1.0 - uv.y );", // fipY: 1.0 - uv.y
    "   gl_Position = projectionMatrix * modelViewMatrix * vec4( position, 1.0 );",
    "}"
].join( "\n" );

var fragmentShader = [
    "uniform sampler2D texture;",
    "varying vec2 vUV;",
    "void main() {",
    "   gl_FragColor = texture2D( texture, vUV ).bgra;",
    "}"
].join( "\n" );

var material = new THREE.ShaderMaterial( {
    uniforms: uniforms,
    vertexShader: vertexShader,
    fragmentShader: fragmentShader,
    //side: THREE.BackSide
});

This shader program is certainly much cleaner than the usual one. What is the difference ?

Magic! 😉

@mrdoob thanks champ. sorry about the noise but blame Apple again for this one.

So for equirectangle video, shouldn't the demo be updated to use this perhaps ? So the shader program being used by the MeshBasicMaterial is unnecessary for such video textures ? I'll test it across browsers and different video formats.

Shall we close ?

Here is the updated demo

http://dev.electroteque.org/webgl/threejs-bgra.html

So for equirectangle video, shouldn't the demo be updated to use this perhaps ? So the shader program being used by the MeshBasicMaterial is unnecessary for such video textures ?

Well, MeshBasicMaterial is more flexible and people don't need to write shaders. It's just that, for this specific case, it was simpler to write a custom shader to work around the issue.

Shall we close ?

Yep!

It is working fine on IOS 10. Only that the frames are not updating on the simulator. Will have to wait for hardware. i'll close the other ticket.

var vertexShader = [
    "varying vec2 vUV;",
    "void main() {",
    "   vUV = vec2( uv.x, uv.y );", // fipY: 1.0 - uv.y
    "   gl_Position = projectionMatrix * modelViewMatrix * vec4( position, 1.0 );",
    "}"
].join( "\n" );

var fragmentShader = [
    "uniform sampler2D texture;",
    "varying vec2 vUV;",
    "void main() {",
    "   gl_FragColor = texture2D( texture, vUV );",
    "}"
].join( "\n" );

With slight adjustments to the flipy, the same program is working on chrome with webm.

@mrdoob I believe this shader works on all browsers with video textures. The flip fix, "1.0 - uv.y" is only required for HLS on Safari. The colour channel inversion is only required for IOS10 with HLS.

Pretty crazy layers of work arounds including the CORS proxy.

I believe this performs better than the shader supplied by meshbasicmaterial. Should this be the standard for video textures then ? rotation seems to work with the vertex code.

Hi all
I'm trying to make a 360vr player with three.js that show an HLS live stream (.m3u8 played with hls.js)
It works on Safari osx with webgl enabled but on ipad 10.2 all my try get lost in a black canvas (the sound is working)
If I disable the vr360 mode, the video stream works fine in all devices.
I tried some of the suggested methods but I still have black canvas rendering.
Someone know how to work around?

The FlipY and color channel inversion fixes are in this thread.

The Webkit bugs I tracked are stale and ended up copping some arrogance about it. It's still open but nobody will bother looking at them.

You need the FlipY fix for Safari. So disable it still on the three.js texture.

Here is the formal es6 methods I devised for my player solution. I am still having nightmares months after hunting this crap down haha.

Thanks to mrdoob the champ for coming up the the shadermaterial solution. Im using this for all browsers now. It's a more simpler shader than one that is generated also. It's just for video textures.

 static getVertexShader(unflipY) {
        return [
            "varying vec2 vUV;",
            "void main() {",
            "   vUV = vec2( uv.x, " + (unflipY ? "1.0 - uv.y" : "uv.y") + ");", // fipY: 1.0 - uv.y
            "   gl_Position = projectionMatrix * modelViewMatrix * vec4( position, 1.0 );",
            "}"
        ].join( "\n" );
    }

    static getFragmentShader(invertColour) {
        return  [
            "uniform sampler2D texture;",
            "varying vec2 vUV;",
            "void main() {",
            "   gl_FragColor = texture2D( texture, vUV )" + (invertColour ? ".bgra" : "") + ";",
            "}"
        ].join( "\n" );
    }

    static createShaderMaterial(texture, unflipY, invertColour) {

        texture.flipY = !unflipY;

        if (invertColour) {
            texture.format = THREE.RGBAFormat;
        }

        return new THREE.ShaderMaterial( {
            uniforms: { texture: { value: texture } },
            vertexShader: WebVRUtils.getVertexShader(unflipY),
            fragmentShader: WebVRUtils.getFragmentShader(invertColour)
        });
    }

usage

 let invertColour = false;

        if (WebVRUtils.isSafari) {
            this.hasHLS = hasHls;
            invertColour = hasHls && WebVRUtils.isIpad;
        }

        this.material = WebVRUtils.createShaderMaterial(this.texture, this.hasHLS, invertColour);

ok the comment on the line _gl.pixelStorei(_gl.UNPACK_FLIP_Y_WEBGL, texture.flipY) in three.js makes the video display in Safari (10.0.2, in the previous version it was necessary only enable the webgl in the development tools, this option is not available now) but I still have the problem on iPad and iPhone (that are my target)
About the shaderMaterial solution, I don't know how to exactly apply it in my code...
thanks for your support

FlipY disabling in my tests was required for MacOs Safari 10 also.

Yes also for mine (except for the previous 9.x.x version)
But I need to use the player on mobile devices so that solution is not my solution :-(

You need both the FlipY fix and colour channel inversion on IOS. And only for IOS 10. No good on IOS 9. Which leaves many older devices out. Because Apple have depreciated them like my Ipad 3. IOS 10 supports inline playback so would be preferable anyway.

I'm taking a look at fixing this in iOS now. But I'm a bit confused by the comments.

For HLS videos:

• iOS is providing color channels in the wrong order (bgra instead of rgba). The values are otherwise correct though.
• macOS is producing a texture that requires flipping in the X axis. This can be worked around by using setPixelStorei(UNPACK_FLIP_Y_WEBGL, true)
• iOS is producing a texture that requires flipping in the X axis, but the above workaround does not work.

Is this correct?

The answer is in this ticket above.

Just as a headsup to you guys, apparently the latest iOS 10.3.3 update fixed the bgr issue, which would make the workaround... work the other way around. I'm not using three.js so for any questions / triaging refer to your own testing or the SO question.

As expected. pretty standard. Thankyou for the heads up.

They should do the right thing and update the ticket I made and left open and stale. To update all developers if there is a fix. They have not. So have to check for minor versions now ?

IOS 11 is supposed to have CORS fixes but it has not yet.

I managed to get xcode 9 beta. It doesnt come with 10.3.3 simulator only 10.3.1 and 11. I tested in 11 and the reversed rgba is still required it seems. I should be seeing a green display otherwise or none at all. I'll have to run more tests by disabling the fix.

The IOS11 simulator is a dud. There is no 10.3.3 IOS simulator. IOS11 apparanly has CORS fixed.

Now that iOS 11 is out to the public I have been running some tests and I am not able to use the fix in this thread for iPhone iOS 11 (not tried iPad yet).

I can make it work (using CORS hack, FlipY and color fixes) for iOS 10.3.3 on iPhone and iPad with HLS.
On iPhone iOS 11 I have a black screen with HLS (using CORS hack, FlipY and color fixes). I think there is something else to fix, but so far I have not put my finger on it.

Regarding the CORS issue it appears to have improved in iOS 11 at least for MP4 progressive download but I am not able to confirm for HLS because of the above.

If someone has better luck please update :/

I am trying my best to get a brand new ipad to be able to even get IOS11.

The emulator doesnt have the fixes so was a failure.

I'll have to confirm if I have to do a counter hack there. Or if the rgb colour channels can be switched back around.

Black rendering for HLS isn't a good start and just their usual sabotage that will take years to revert a regression. That is what happens on IOS9.

CORS should have been fixed in IOS11 and doesn't need a proxy

@danrossi I'm also facing this issue now with iOS 11, if you come up with a solution to this problem please let me know. iOS 10 is still working just fine with the CORS fix and Flip Y, but iOS 11 only plays audio..

I need to still get a new ipad with IOS11. HLS is working in the IOS11 beta simulator ? It could be CORS related. The rgb channel flip would display inverted colours. So black might be CORS related ?

I have tried without my CORS proxy and with it, and I still have the same problem. I've also tried without the Flip Y just for the sake of it, and it changes nothing. Only sound, no video :/

There is still the same problem on ios11 device,only sound,no video。 Anyone has solved this problem？

I got it working by drawing the video to a 2D canvas and using that 2D canvas as a texture in webgl.

I can not get 2D canvas texture correctly on ios11， could you poste the code？

That is a very bad idea. Canvas drawing is processor intensive.

Ive updated the webkit ticket in regards to this problem. They have not addressed it at all. I even provided my work arounds with full shader and source code. Which you claim are now not working.

I know someone else is having a problem. But I can't test it properly yet until I get an actual brand new device. The simulator is showing it correctly. have been met with planned obsoleteness where even an Iphone 5 can't get IOS 11.

here is the ticket again. https://bugs.webkit.org/show_bug.cgi?id=163866#c17

@LoveLetterIloveU here is my sample code: https://htmlvideocanvas-sibabi.c9users.io/hello-world.html (Open ONLY on ios)
@danrossi is right, it is not the best solution. It is basically doing memcopy of byte arrays rather than binding the pointer. however, it does the job while apple is maintaining the project. I'm still evaluating this approach and have an mp4 fallback method ready to go as well.

I spent two whole months finding that solution above with the help of everyone fine tuning the shader of course so thankyou. It just means more of our time countering them finding another work around.

I hope there is one.

They have not bothered to try and fix the flipy / HLS / webgl issue at all, you can see in that ticket I made.

They only just fixed the CORS problem after all. Although I'm unsure yet if they managed to get Dash working.

Apple / webkit is the thorn in my side. It might mean I have to go and attempt to fix it myself and I don't do c++. It would be really funny if I did fix it but I'd liked to keep my hair.

Can someone post their code for me to have a look, just to make sure FlipY is enabled / disabled.

See my comment here. I have had an absolute gutfull of this sabotage mess. Disabling flipY on the texture should show a picture but upside down. It's a black frame here. Back to the start. They have caused a regression that was a problem in IOS9.

https://bugs.webkit.org/show_bug.cgi?id=163866#c19

I'm closing the original webkit ticket that was never even looked at or addressed. And starting a new one soon as it will take time to gather evidence.

It has moved beyond the original FlipY problem. It seems they never bother to even check HLS works with WebGL before releasing the OS.

I have closed the original webkit ticket to start a new one specifically HLS video textures with webgl is completely broken with no possible work around yet. The original FlipY work around is not relevant anymore.

I have discovered all of the webkit webgl conformance tests use Mp4.

They have not ever bothered to test with Apple's own streaming format. Hence why it has been broken and still broken with a brand new IOS 11 update.

See here for more

https://bugs.webkit.org/show_bug.cgi?id=163866#c22

@danrossi Do you have a link to the newly opened ticket for webkit? I've just stumbled upon this issue on iOS11 myself and am keen to keep track of any developments.

I'm self funding their work for them. I'm running conformance tests with HLS generated from the Khronos test mp4 files. It claims it passes but the test fails to prove if its displaying anything. The canvas doesn't even display here. Actually I don't even think the test functions properly on IOS at all. There is no logs.

I have to use their own test files to prove that HLS textures don't work. I have built other test files and will upload to make the ticket.

I'm hoping there is some webgl param required or another one that needs to be disabled like last time.

Ok as far as I see with the webkit test files. They are not designed for IOS testing. No user click functionality to play. Mp4 for some reason can autoplay I discovered on IOS 11 although no attribute is specified but HLS can't still.

I don't believe any tests are done specifically for IOS and most definitely no test is made for HLS.

Hence the problems. It's beyond bad.

I have what I need to submit the ticket.

As far as I can see. The webgl webkit tests are flawed. They do not care about the webgl canvas output. They are drawing the video to a canvas as a preview output at a very tiny size.

They have the webgl canvas but its a very tiny size so you won't even see if its rendering or not.

I had to modify the tests for HLS and they don't pass. And now I have increased the size of the webgl canvas it's black.

After I modified the code to add user click the tests actually start running on IOS.

As far as I can see no testing has ever been done for HLS and the tests are flawed. And so it has sabotaged absolutely everyone.

Here you go. I am really losing my patience with Apple and how much personal work is required to fix their problems and run tests for them.

https://bugs.webkit.org/show_bug.cgi?id=179417

Two days and no urgency to look at it. I suspect this will never get addressed. It took years, and multiple versions of IOS and device upgrades for CORS to be fixed ?

I will try and play with webgl flags and shader code to see if anything can be done.

Can people please go there and cause some noise. As it's still not even looked at. Like they don't actually care about testing HLS in their own browser at all.

@jernoble could you point out who could help us in the webkit team on this issue? It would really be appreciated if someone could look into it or give us an update. The issue here on GitHub is quite long but it has been summarized here: https://bugs.webkit.org/show_bug.cgi?id=179417. Thanks in advance.

Hi @radiantmediaplayer. Both @grorg and I are the right people to ping on this issue. I'll CC myself to that WebKit bug and create a Radar tracking it.

Hey guys. Im posting a sample workaround for iOS 11 using temporary canvas as described by @sorob . You can see it working in http://redgetan.cc/demo/ios_11_hls_webgl_canvas_workaround/index.html . I'm not familiar with Three.js though, so its just plain javascript .

CanvasRenderer. that is software rendering. Not production worthy.

I am with @danrossi on this one. Performance issues aside maintaining 2 complete different solutions to display HLS content would be impractical. We need to get to the bottom of the iOS webkit issue. The new webkit ticket is showing activity which is a positive sign. Hopefully we can come up with something even if that means waiting for iOS 11.*.

@danrossi I just noticed that this issue is closed. Would you re-open it?

The brand new issue is here. It's not related to Three.JS so kept for an archive I guess.

The level of tests required was insane. Broke my back like finding the IOS10 work around. have a look there.

Nothing works around it so far. Someone is finally having a look and not in our control. Something to do with a shared reference to the context for the texture seeing HLS is different to Mp4 which might fix it. As far as I gather but they actually have to spend time to do it. I have no means to setup such a build environment as I am on Windows.

But looking at the webkit code explains I think why IOS10 had wierd color issues with HLS. It doesn't use colorspace code. Hence the code to work around both issues there for IOS10 / Safari.

https://bugs.webkit.org/show_bug.cgi?id=179417

Hi there. It seems I myself have to roll back and implement a canvas renderer for IE11 video texture support. It doesnt work with WebGL. Somebody was asking about it.

I reckon at the rate it is taking to be looked at, it might take about 2 years to fix.

IOS 11 will have to suffer with software rendering I think for HLS. In the bin the whole IOS thing is for VR. I'm going to implement this myself.

@redgetan , any ideas how to implement with three.js ? I tried to use the CanvasRenderer and I get a mesh frame visible over the texture and doesn't render a spheregeometry correctly.

The canvas three demo only shows 2D video being rendered.

Sorry you are using a webgl render but using a canvas as the texture image source. Seeing if I can replicate that. It doesn't render and rotate correctly for me when using canvas as the texture.

ok for some reason I had to do this.

Use the webgl renderer as normal. But for the canvas texture and video drawing I had to set the dimensions

imageContext.drawImage( video, 0, 0, image.width, image.height );

It is rendering correctly now. Will run tests on IOS 11.

@danrossi let us know how it goes on iOS 11. I might actually need to go down that road as well at some point.

Ok no worries. I will add a basic test with a fps stats view online soon if it seems to work. It may need anything right now.

Is there a way to check processor performance on IOS ?

I have made a canvas texture test. It is working but a larger resolution video might make it fall over.

This HLS is low quality so is keeping at 40 fps. A larger resolution mp4 but also low quality stayed at 30fps.

A production resolution file might get 10fps out of it. This is for a new Ipad.

http://dev.electroteque.org/webgl/three-canvas.html

Does this issue have to do something with the problem im facing on this page right now: https://digitalejulegaver.firebaseapp.com/#catarina

Open it in your safari browser on a mac, or a macOS device. An ipad or iphone can't handle this one page even if it is in a chrome browser or mozilla.

I get a black page, but in normal chrome on mac, it works

@vongohren You need to use the FlipY work arounds I discovered for HLS with macOS Safari and the shader we worked out in this ticket.

However I have just got back to my webkit tickets and they claimed they fixed the FlipY issue for 10.13. I believe this is High Sierra. There is no chance testing that for me as I am using a cloud based macOS.

This is going to require yet another platform version check to not require FlipY work arounds for 10.13. so anything below version 13 will require it. I have about 4 different platform and browser version checks for macOS and IOS to work around multiple bugs. The IOS 11 check is to turn on software texture rendering !

If you can check 10.13 this might be helpful.

Here is the details of all my tickets.

https://bugs.webkit.org/show_bug.cgi?id=179417#c8
https://bugs.webkit.org/show_bug.cgi?id=176491

https://bugs.webkit.org/show_bug.cgi?id=180863#c3

@danrossi Iv got 10.13, but you got a demo that can be tested?

The three.js version im using now is to new for these fixes, so not sure where to apply them.
I found something called data.flipY, and set it to false. But did nothing for my app.

Im using three.js through this library: https://github.com/iberezansky/flip-book-jquery

CORS problem ?

Does this work ?

http://dev.electroteque.org/webgl/three-hls.html

or this

http://dev.electroteque.org/webgl/three-hls-flipy.html

My massive lists of test files are in those tickets.

FlipY can be disabled on the texture

texture.flipY = false;

Everything is locally so dont see why cors should be effected. But those two links just shows black on both safari and chrome, on Mac 10.13

And the flipY feature, didnt do much, rather than just turn the picture around. But it was still black

It sounds like in 10.13 suffers the same problem as IOS 11 where the FlipY work around is broken. This is good information. and yet another bug to report on. Its so messy its sending me crazy. I will update this in the IOS ticket. in 10.12 the FlipY bug requires to be worked around and made a seperate ticket for it.

So like in my VR feature now . You have to do a platform version check for IOS 11 and 10.13 and provide a software rendering canvas texture mode for hls.

here is some static utils I am using for feature checks especially cors support or use cors proxy sources. Obviously now I have to check for version greater than 13 to temporarily turn on software rendering like IOS 11.

static get supportsCORS() {
        let testVideo = document.createElement("video"),
            hasCORS = false,
            userAgent = navigator.userAgent;

        testVideo.crossOrigin = "anonymous";
        hasCORS = testVideo.hasAttribute("crossOrigin");
        testVideo = null;

        if (hasCORS) {

            //if (_isSafari = WebVRUtils.safariCheck(userAgent)) {
            if (_isSafari) {
                return WebVRUtils.isNewSafari(userAgent);
            }

            //cors support check for IE 11 support. 
            _corsSupported = true;

            return true;
        }

        return false;

    }

static isNewSafari(userAgent) {
        const osxVersion = /Mac OS X (10[\.\_\d]+)/.exec(userAgent);

        if (osxVersion) {

            //check for safari test to fix HLS problems.
           // _olderSafari = _browserVersion.match(/(\d+).(\d+).?(\d+)?/)[1] <= 10;

            const version = osxVersion[1].split("_")[1];
            return +version >= 12;
        }

        return WebVRUtils.isNewIOS(userAgent);
    }

    static isNewIOS(userAgent) {
        const version = navigator.appVersion.match(/OS (\d+)_(\d+)_?(\d+)?/)[1];
        _isLatestIOS = (+version >= 11);
        return _isLatestIOS;
    }

You have done some great work on this, hope you get the tickets sorted out :D

Here is the canvas texture work around. Ive had to activate it for IOS 11. I might need to do that macOS 13 version check and enable it for that too. if you reckon the flipY work around is broken for 10.13 ? I have to report back to them.

Mate you have no idea , I don't get any benefit out of it, self funded, other than making everyone's software including my own feature barely functioning on Apple OS haha.

Here is my full integration right now, it has the works including the apple fixes and work arounds. I now have cubemap video support.

https://flowplayer.electroteque.org/vr360/fp6

This has been ongoing since I first reported it a few years ago which sat there doing nothing !

Does this work for you ?

http://dev.electroteque.org/webgl/three-canvas.html

When you say here is the canvas texture work around, where do you point to?

That flowplayer looks sweet! Has it seen alot of usage?

What should I see in this link: http://dev.electroteque.org/webgl/three-canvas.html?
When I open it in Chrome - Version 63.0.3239.84 (Official Build) (64-bit) - on macOS 10.13.1:
I see black background with an FPS meter in the left corner, and a button saying activate AR in the middle bottom
When i click that, I get a splitscreen, with a settings wheel and a back button. But the canvas itself is just black.

When I open it in Safari Version 11.0.1 (13604.3.5) on macOS 10.13.1:
I see the flying video with an FPS meter in the left corner, and a button saying activate AR in the middle bottom. I can also scroll around for full 360 experience.
When i click VR button, I get a splitscreen, with a settings wheel and a back button. And the video canvas, but cannot scroll, I guess that is for the gyro to controll.

But if I hit back button from the VR mode, the video is stuck in VR mode, and i cannot scroll anymore, and it is stuck on the view the VR mode started in.

That demo is for HLS rendering on Safari. If you think the flipY fix demo doesn't work on Safari

http://dev.electroteque.org/webgl/three-hls-flipy.html

then have to use canvas textures with 10.13 and IOS 11 Safari ?

http://dev.electroteque.org/webgl/three-canvas.html

Let me know between the two. I don't care about rotation controls just that it renders and not a black frame.

It's important about 10.13 as I've added it to the IOS 11 webkit ticket referenced here now. Assuming its a "regression" that completely broke HLS rendering altogether.

@danrossi both works on safari, higher quality with hls version. They are both the right direction if you wondered.

But none worked on chrome.

OK. great I need to undo my reporting.

So this is still a problem correct ?

http://dev.electroteque.org/webgl/three-hls.html

If that is the case use the FlipY work around as standard for macOS then.

The HLS demo on this page should work for both browsers. It will use mediasource streaming for chrome with the hls plugin. It also has the FlipY work arounds for macOS then.

https://flowplayer.electroteque.org/vr360/fp6

You are correct
http://dev.electroteque.org/webgl/three-hls.html
This doesnt work for either chrome or safari

The link flowplayer, works for both browsers

Thanks champ. I had enough info to update the tickets again. The macOS Safari issue is different to the IOS 11 one.

You need to use the canvas texture example for IOS 11.

And for macOS Safari disable FlipY on the texture and reverse the Y position on the shader to turn it the right way around. You need to do the colour channel inversion on the fragment shader for IOS 10 HLS also. The shader code is above.

So I was happy for about 1H this morning when I managed to crack the iOS 11 case with the above workaround (@danrossi I took inspiration from your example at http://dev.electroteque.org/webgl/three-canvas.html) ... but then I put my glasses on and felt like an ice-cube under the desert sun.

This is the result I get on an iPhone 6 Plus with iOS 11.2.1, animation is actually pretty smooth but it is plain ugly, pixelated, aliased ...

The expected behavior is as follows (Nexus 5X with Android 8 and latest Chrome - this does not use the above workaround).

Our use case is a 360 HTML5 player and in both samples above the video is the same HLS single-bitrate feed http://www.rmp-streaming.com:1935/vod/sea360.mp4/playlist.m3u8

If I apply the workaround (see below for my code) to Android I get the same ugly rendering. Is it what I am supposed to get with this new workaround or am I missing something?
I tried to play with some of the settings but I am not seeing any notable improvement.

    // image 
    this.image360 = document.createElement('canvas');
    this.image360.width = this.width;
    this.image360.height = this.height;
    this.imageContext360 = this.image360.getContext('2d');
    this.imageContext360.fillStyle = '#000000';
    this.imageContext360.fillRect(0, 0, this.width, this.height);
    // texture 
    this.texture360 = new THREE.Texture(this.image360);
    this.texture360.format = THREE.RGBFormat;
    this.texture360.minFilter = THREE.LinearFilter;
    this.texture360.magFilter = THREE.LinearFilter;
    this.texture360.generateMipmaps = false;
    this.texture360.wrapS = THREE.ClampToEdgeWrapping;
    this.texture360.wrapT = THREE.ClampToEdgeWrapping;
    this.texture360.flipY = true;
    this.texture360.needsUpdate = true;
    // mesh
    let geometry = new THREE.SphereGeometry(500, 80, 50);
    geometry.scale(-1, 1, 1);
    let material = new THREE.MeshBasicMaterial({ map: this.texture360 });
    this.mesh360 = new THREE.Mesh(geometry, material);
    this.scene360.add(this.mesh360);
    // renderer
    this.renderer360 = new THREE.WebGLRenderer({ antialias: false });
    this.renderer360.setClearColor(0x101010);
    this.renderer360.setPixelRatio(window.devicePixelRatio);
    this.renderer360.setSize(this.width, this.height, true);

I'm doing this. I am fairly certain this should size it correctly.

video.addEventListener("loadeddata", function() {
                      video.width = THREE.Math.ceilPowerOfTwo(video.videoWidth);
                      video.height = THREE.Math.ceilPowerOfTwo(video.videoHeight);

                      image.width = video.width;
                      image.height = video.height;
                      //image.width = video.videoWidth;
                      //image.height = video.videoHeight;
                      //imageContext.fillRect( 0, 0, video.videoWidth, video.videoHeight);
                      imageContext.fillRect( 0, 0, image.width, image.height);


                      //console.log(video.videoWidth);
                });

So I nailed it after a bit a poking. It appears that in our config the off-dom canvas must be of the exact size of the current HLS rendition being played in order to render correctly (hence for ABR it needs to be updated each time the rendition changes). Our testing shows this works ok for iOS 11 (ok is better than nothing). Leaving this here if it helps someone else.

Yeah , hence I do it in the loadedata event to get the video width. I figured this out before and it had your same problem.

However for native HLS there is no event when a bitrate changes ? I'm just setting in the once personally. The amount of work arounds for Apple is already incredible enough.

There is no event, that I am aware of, provided by Safari that can be used to detect a change in rendition for ABR HLS. We just update the off-dom canvas width and height with video.videoWidth and video.videoHeight on each AnimationFrame (which is probably not ideal on a performance point of view but at this stage we have it working and we will settle for it until another option becomes available). Our testing shows that video.videoWidth and video.videoHeight should be updated accordingly when a new rendition is displayed.

Yeah for what it's worth it's hacky enough. So doing that is excessive especially if there is condition statements in a frame callback. Whatever works for now I guess.

They still can't provide a rendition change event. I believe there is a spec for one but was never implemented ?

The CORS proxy to work around older Safari bugs, the HLS flipy fix and now this is enough.

Glad it's all working. I have seen no update yet in my tickets for a good month or so now.

I cannot recall seeing something in the HTML5 video spec about a quality switch event. There is the Apple-maintained HLS spec but it does not describe what should be done by the client when a quality switch happens. Libs like hls.js have this implemented but this is not available for HLS to native HTML5 video in Safari.

I just realize I could "polyfill" our hack to detect change in rendition by querying video.videoWidth at fixed intervals and when a change is detected then call the resize callback. This will be less taxing. I will probably implement this at a later stage in our player.

@radiantmediaplayer You should be able to use the "resize" event to detect changes to the underlying .videoWidth and .videoHeight properties.

Hi guys see my comment here. It's working.

What an ordeal that was making tests, and going through the motions. Only to get no proper feedback or response. I'm seriously done ! No more regressions please !!

Install OSX 11.3 beta and it will work. Previous OSX 11 needs software rendering, OSX 10 needs the flipY and colour fix. A complicated compatibility mess and platform version detection required.

Safari macOS still has the FlipY issue and I got no response there either, it's in a seperate ticket. I can't properly test HLS there anyway until I can try and install macOS on my pc workstation. HLS is blocked from playing in vmware.

https://bugs.webkit.org/show_bug.cgi?id=179417#c43

FlipY work arounds is not needed either for OSX 11.3 but I think I mentioned it before.

Here is my full IOS and Safari compatibility check now starting with CORS.

The mind boggles. This one is insane. I even had to future proof and include anything further than IOS 12.

It is working on integration ! Can't check what is going on with macOS yet.

static get supportsCORS() {
        let testVideo = document.createElement("video"),
            hasCORS = false,
            userAgent = navigator.userAgent;

        testVideo.crossOrigin = "anonymous";
        hasCORS = testVideo.hasAttribute("crossOrigin");
        testVideo = null;

        if (hasCORS) {

            //if (_isSafari = WebVRUtils.safariCheck(userAgent)) {
            //Safari still reports CORS support regardless a bug
            if (_isSafari) {
                return WebVRUtils.isNewSafari(userAgent);
            }

            //cors support check for IE 11 support. 
            _corsSupported = true;

            return true;
        }

        return false;

    }

static isNewSafari(userAgent) {
        const osxVersion = /Mac OS X (10[\.\_\d]+)/.exec(userAgent);

        if (osxVersion) {

            //check for safari test to fix HLS problems.
           // _olderSafari = _browserVersion.match(/(\d+).(\d+).?(\d+)?/)[1] <= 10;

            const version = osxVersion[1].split("_")[1];
            return +version >= 12;
        }

        return WebVRUtils.isNewIOS(userAgent);
    }

static isNewIOS(userAgent) {
        const platform = navigator.appVersion.match(/OS (\d+)_(\d+)_?(\d+)?/),
        version = +platform[1];

        _isLatestIOS = version >= 11;

        //if greater than 11.3 or 12
        _iOSHlsSupported = (_isLatestIOS && +platform[2] >= 3 || version >= 12);

        return _isLatestIOS;
    }

Usage

 set hls(value) {

        //if had a HLS source and the current playlist source isn't clear scene and revert the material.
        if (this.hasHls && this.isSafari && !value) {
            this.clearScene();
            this.initScene(value);
        }

        //flipY fix
        this.hasHls = value && this.isSafari;
        //IOS10 colour inversion fix
        this.invertHLS = this.isIpad;

        //if we have hls and IOS 11 or greater
        if (this.hasHls && WebVRUtils.isLatestIOS) {

          //IOS 11.3 or greater disable flipY and colour inversion fixes
          if (WebVRUtils.iOSHlsSupported) {
              this.hasHls = false;
              this.invertHLS = false;
          } else {
            //anything below IOS 11.3 use software rendering
            this.config.software = true;
          }

        }
    }

Correction one last important point. IOS HLS rendering implementation is running on shoestring and very touchy. The rendering must start after loadedata.

As soon as it hits a teximage2d error , no further rendering happens, just black rendering, it will seem like the fix never happened.

If the rendering starts as soon as there is video data it's ok. It was not a problem before 11.3.

Looks like video is working again (without temporary canvas fix). Verified for latest ios version 11.3 (tested on iphone 6s , 7).

Yes I think I updated on Feb 15. Working again in 11.3. I never got an update. I had to specifically check back and it was working !

We fixed a lot of issues with video -> texture in iOS 11.3 (and the equivalent macOS release). Sorry if we didn't close/update the bugs.

Please let me know if you're still seeing issues. I believe that direct streams (e.g. mp4) and HTTP Live Streams should both work, and flipy should be accurately reflected.

It's all done from February when reported. But I originally reported last year sometime. I randomly updated the IOS beta to 11.3 and it started working. The compatibility check code above is required to deal with multiple platform work arounds.

IOS 10 requires the flipY fix. IOS 11.0 - 11.2 requires software rendering work arounds. 11.3 > can render as normal.

Can anyone share a working sample for this, a jsfiddle or so ? I could still see a black screen in IOS 11 Safari. I could hear audio but no video for HLS streaming

@dhanishapa What is your iOS version? Is it 11.3+ ?

A little while ago (~ May 2018) I too was experiencing the following issues with HLS (? <= 11.2):

• Segments of video sometimes repeat during playback
• Videos sometimes pause and display a black screen
• Videos buffers but audio continues

After having contact with Vimeo (I am using their HLS service) they were able to contact the Apple development team and work together on a fix.

We've just received word from Apple's development team that a new version of iOS (11.3.1) has been released, which includes a fix for the playback issues you reported. The fix should prevent videos from stopping on a black screen, repeating segments of video, or getting caught in an endless loading state.

I noticed that when I supplied H264 encoded MP4 as source during uploading the playback issues went away on iOS versions lower than 11.3.1. I experienced issues when using Quicktime as an input format during uploading.

I spotted some differences between the manifests generated by each video file (the video codecs namely "avc1.64001E,mp4a.40.2” for the working one versus "hvc1.2.4.L63.90,mp4a.40.2” for the non-working one).

Perhaps this information is useful to anyone.

Hi
I am having problem with HLS streaming on IOS safari.
I am using cors proxy. I tried using native hls player and html5 player and even just tried playing m3u8 link directly in IOS but it doesnt work, while the its work in safari (desktop - mac) , chrome, firefox, just not in IOS (iphone)

cors was fixed in IOS 11.0. The other big fix I worked out is in 11.3. 11.3 should be out now. Proxy only needed for IOS 10 and 9.

It's working for me 🙆🏽‍♂️

Here is a Glitch demo of an HLS stream mapped to a sphere (360 video by playhouse-vr).

Tested on macOS v10.13.4 + Safari v11.1 and iOS 11.4.1 + Safari

I am seeing a NEW black screen issue now on iOS 14.0 Safari and iPadOS 14.0 Safari. The exact same player works with iOS 13.7 Safari and iPadOS 13.7 Safari. I tried updating to three.js r120 (from r116) but to no avail. macOS Safari 14 is ok. Is anyone seeing the same thing? Any hint at what is the next step to support iOS 14?

Yeah, check out both https://bugs.webkit.org/show_bug.cgi?id=215908 and https://bugs.webkit.org/show_bug.cgi?id=216250 for iOS 14 regressions. Hope they can both be addressed quickly as many sites are likely impacted.

It's a never ending battle with Safari. I expect complaints again for my own 360 features. It takes months each time to fix them. I found the original work arounds above. So time to find more workarounds !! Webkit don't test HLS, they have no HLS tests for WebGL I had to make some myself.

I removed all the bloaty work arounds for different IOS in my feature. But one of them had to use canvas software rendering for 11.0 ... I still have to do platform checks. With IOS 13 they have broken platform checks now. I had to fix that in the cardboard project. I found HLS doesn't render immediately and takes a while in IOS 13 but have to revisit how I fixed that problem.

They just don't care. No WebXR api, no Wakelock api like Android just implemented.

according to my work around, I have to reconfigure on rendition change. And disable zoom on orbit control. This stops the black rendering in IOS 13. IOS 14 have to check.

            video.addEventListener("resize", () => {
                vrVideo.update();
                vrVideo.fullScreenResize(false);
            });

I've tested 14.2. HLS OK. Fix above will do it ...