HLS Live Streaming Issues with Wowza and Chrome

[theodortruffer]

Describe the bug

We are experiencing some strange behavior when playing a live stream from a Wowza server with the standalone paella player in Chrome: most of the time, the player doesn't load, meaning that the page is stuck with the loading icon. It does work on some tries though (maybe 10% of the tries, it really seems random).

What I found out so far:

• the HLS JS seems to work: there are constantly outgoing requests to fetch the latest .m3u8 and .ts files, even though the player doesn't load
• other live streams (I tried test streams from here: https://ottverse.com/free-hls-m3u8-test-urls/) are working fine, so it seems to be an issue with how the wowza server is delivering the stream
• the problematic live stream works perfectly fine on the hls-js demo site (https://hls-js.netlify.app/demo/), so it shouldn't be an issue with the stream/hls-js
• the problem doesn't occur with the opencast-paella-player
• the problem doesn't occur on other browsers, like firefox
• the browser console shows two (probably unrelated) warnings:
  • "failed to load SourceMap" - which should have no impact on functionality
  • "[Violation] Added non-passive event listener to a scroll-blocking event." - this also appears on successfull tries. I also managed to fix these violations in the code, but the problem remains
• on successful tries, the next console log is 'HLS: quality level changed to 0'. on unsuccessful tries, there are no console logs at all
• i used the standard config.json included in the standalone version (i only disabled timeMarksPlaybackCanvasPlugin to avoid an error)

To Reproduce

Unfortunately, I can't provide you with a working stream to test it, since the customer has to manually start the stream in a classroom. But here are some files that might help:
index.html (streaming url is changed):
index.txt

config.json
config_live.txt

playlist.m3u8 (as returned in the initial request to the wowza server)
playlist.txt

chunklist.m3u8 (example, as return in the successive requests to the wowza server)
chunklist.txt

Workaround (if any)

Environment Information

• Browser: Chrome
• Browser Version: 88.0.4324.96 (Official Build) (64-bit)
• Paella version: 6.4.5

Additional context

Stuck in this state:

Browser console on failed try:

Browser console on successful try:

PS

I know this is not much to reproduce the problem, but I'm hoping that you maybe see the source of the problem from this information, or at least have some idea where to start looking. If you need more information, like some wowza configuration, please let me know.

[marcusm7]

Regarding the PS:
We have set up a system to record and livestream our lectures. I asked theodortruffer for help regarding this issue.
Thank you theodortruffer for analyzing and reporting the issue.

If you need a livestream to reproduce said behaviour or additional information regarding our setup, you can also contact me.

[karendolan]

@theodortruffer add DEBUG log level config into the hls.js config in your Paella config to get more clues. You can also get more general Paella js logs by adding URL param "&logLevel=DEBUG". But, the hls.js log level config is needed to get it from the hls.js library.

Line
"debug": true

To config.json:

...
    "methods":[
      { "factory":"ChromaVideoFactory", "enabled":true },
      { "factory":"WebmVideoFactory", "enabled":true },
      { "factory":"Html5VideoFactory", "enabled":true },
      { "factory":"MpegDashVideoFactory", "enabled":true },
      {
        "factory":"HLSVideoFactory",
        "enabled":true,
        "config": {
          "*** You can add more hls.js settings here": "",
          "https://github.com/video-dev/hls.js/blob/master/docs/API.md": "",
          "maxBufferLength": 30,
          "maxMaxBufferLength": 600,
          "maxBufferSize": 60000000,
          "maxBufferHole": 0.5,
          "lowBufferWatchdogPeriod": 0.5,
          "highBufferWatchdogPeriod": 3,
          "debug": true
        },
...

[theodortruffer]

@karendolan I already tried the debug mode for hls.js, but there were no error messages, only a constant stream of info messages (chosen bandwidth level etc.). Unfortunately I didn't record/screenshot it (because there were no errors). But we could start another debug session and record the messages, if you think that would help?

[turro]

Hi theodor

I just spotted a similar problem with an HLS stream, and we fixed it in 6.4.5 . The problem was that HLS.js system has a function for recovery in wrong HLS streams and due to a race condition paella wa unable to catch the error and then recover.

So what you could do is:

• Install the Chrome extension "Native HLS" player. This is a minimal implementation of the last version of hls.js
• Open within the browser your .m3u8 file
• See if the console says something

In my case, the Native hls.js throw (sometimes, as you) an error, and with that we found why paella wasn't catching it

Could you try?

[marcusm7]

Hi Carlos Turro,

I tried what you asked on my end.
I installed the chrome plugin (this one right?: https://chrome.google.com/webstore/detail/native-hls-playback/emnphkkblegpebimobpbekeedfgemhof).

The said player works fine and I don't see hangups. The player just pauses for a while.

I see bufferStalledErrors:
chrome_bufferstalled_emnphkkblegpebimobpbekeedfgemhof-1612261639146.log

Screenshot with warn and above only:

One (maybe unrelated) addition:
The paella uses HLS.js 0.14.16 and the chrome-plugin uses 0.12.4.
When I use paella with version 0.12.4 the original error (player does not load) occurs less often than with version 0.13+.

[ruthlang1]

Hi,

for me it would be interesting to know if all of you have this problem only with live streaming. Does anyone use HLS streaming also for video on demands ? We tested today several Paella version on different laptops and could not reproduce the problem for a live stream (Extron SMP, Wowza, Opencast, ILIAS) . Are you sure your Wowza server is "strong" enough to generate a HLS stream in one or more resolutions ? Or will the created output stream stick from time to time ?

[ruthlang1]

The Paella verisons are 6.2.7, 6.4.3 and 6.4.5

[turro]

@ruthlang1 At UPV we are using HLS both for live streaming and on demand in several qualities without issues

[turro]

@marcusm7 ; Could you try paella 6.5.0 that we just released today?. We made a fix that maybe solve your problem

[theodortruffer]

@marcusm7 the Opencast plugin's beta branch already has v6.5.0 integrated, could you test this? There will be a few console errors concerning the localization and config, but they shouldn't (considerably) affect the player.
Edit: The plugin doesn't work with the latest player version, I'll make another issue for that.
Edit 2: The other issue: #661

[ruthlang1]

Also the Paella Player v6.5.0 in our test ILIAS system is working fine, for both - video-on-demand and live streaming.

[mliradelc]

Can confirm, version 6.5.0 solved the issue.

[ruthlang1]

There was never an issue :-)