Webpack dev server with rails-erb-loader and spring hangs

[doits]

This was originally posted at webpacker rails/webpacker#785.

Using rails-erb-loader@^5.2.1 and webpacker gem 3.0.1, rails 5.1.4 and spring 2.0.2.

I've the problem that in my current project, webpack-dev-server hangs when first started:

→   ./bin/webpack-dev-server
 11% building modules 10/10 modules 0 active
Project is running at http://localhost:3035/
webpack output is served from /packs/
Content not from webpack is served from/Users/.../public/packs
404s will fallback to /index.html
 66% building modules 468/469 modules 1 active ...xternal_application_form/index.js.erb

It sometimes hangs at some other point, eg.:

 67% building modules 479/480 modules 1 active ..._custom_question_answers_form.vue.erb

so the hang is not consistent, but it always displays some .erb at the end of output.

If I cancel the dev server and restart it, it goes through without a problem:

→   ./bin/webpack-dev-server
 11% building modules 10/10 modules 0 active                                                                                                                                                                    Project is running at http://localhost:3035/
webpack output is served from /packs/
Content not from webpack is served from /Users/.../public/packs                                                                                                                           404s will fallback to /index.html
Hash: 6b8d3991aafe09cef922                                                                                                                                                                                      Version: webpack 3.5.6
Time: 8058ms
[...]

I noticed that it happens only when spring was not started before. If spring is started already (e.g. because of a rails console), the compilation goes through on first try without the need to cancel it first.

Additionally, when I disable spring by commenting out the spring loader from bin/rails, or by using DISABLE_SPRING=1 ./bin/webpack-dev-server it always goes through without a problem, too. So I suspect it to be the combination of spring and the rails erb loader that makes it hang somehow (or is anything else calling bin/rails during compiling?)

I tried to set a timeout for rails-erb-loader by editing node_modules/rails-erb-loader/index.js, suspecting a hang while compiling the erb assets, but this did not change anything (still hangs).

How can we debug this further to see where exactly it hangs and why?

[rhys-vdw]

Yeah, I've been meaning to open an issue for this for a while. We've been hitting this problem ourselves. I think that, as you suggest, that it's a bug in spring. I added a timeout recently in #46 that should kill the ruby process, but it the compile process never ends, nor emits the converted code and the process doesn't end, preventing the loader from completing.

Tbh I'm not sure what to do about this. Definitely open to suggestions. This is a new problem for us, but I'm not sure if it's to do with a spring update or just an increase in the number of files we're transpiling, but it's pretty bad.

[rhys-vdw]

I tried to set a timeout for rails-erb-loader by editing node_modules/rails-erb-loader/index.js, suspecting a hang while compiling the erb assets, but this did not change anything (still hangs).

@doits did you use add a JS timeout, or did you use the existing Ruby-based timeout option?

[doits]

@doits did you use add a JS timeout, or did you use the existing Ruby-based timeout option?

@rhys-vdw I did not realize those where two different timeouts. I did use the ruby-based timeout.

Just now I simply patched node_modules/rails-erb-loader/index.js and added a 10 second js timeout for execFile to use the JS timeout:

function transformSource (runner, config, source, map, callback) {
  var child = execFile(
    runner.file,
    runner.arguments.concat(
      runnerPath,
      ioDelimiter,
      config.engine,
      config.timeout
    ),
    {
      timeout: 10000
    },
    function (error, stdout, stderr) {
    ...

With this, compilation always runs through even with cold spring and it looks like everything is compiled correctly.

A shot in the dark: Might it be that spring is forking a new process when run through execFile (because it was not started before already) and therefore execFile never exits because it thinks the process is still running?

[rhys-vdw]

With this, compilation always runs through even with cold spring and it looks like everything is compiled correctly.

Interesting! Wouldn't the timeout cause an error to be raised, rather than a successful compilation?

Might it be that spring is forking a new process when run through execFile (because it was not started before already) and therefore execFile never exits because it thinks the process is still running?

Ah, interesting. In my experience running after spring stop tends to increase likelihood of success.

Just now I simply patched node_modules/rails-erb-loader/index.js and added a 10 second js timeout for execFile to use the JS timeout:

Would you be interested in opening a PR with these changes, removing the Ruby timeout, and replacing it with the more reliable JS timeout? If so, make sure that the timeout is defined in seconds since I made the previous timeout use seconds as this is the Ruby convention. (I don't want to break backwards compatibility just yet.)

In my experience the existing timeout option doesn't address this at all. I added it to get more feedback on the issue, but it's never triggered in our dev environment.

[rhys-vdw]

Or alternatively we could update to have two arguments timeoutS and timeoutMs so it's more explicit, and deprecate the timeout option.

[rhys-vdw]

Or, if your theory is correct, perhaps switching back to exec instead of execFile would be better. Wish I had some answers here. 😕

[chaffeqa]

I have a similar problem, but I'm suspecting more of the interaction between behavior of webpack caching and the return of the ERB on error.

My suspicion is because when i turn off the webpack caching { cache: false }, it spits out the ERB error rather than hanging

[rhys-vdw]

@chaffeqa okay, that's pretty good to know. Any idea what we could do to fix it?

[chaffeqa]

Don't make errors 😉

It's a bandaid, but I added an execFile timeout (along with extending the buffer because our app sucks):

  var child = execFile(
    runner.file,
    runner.arguments.concat(
      runnerPath,
      ioDelimiter,
      JSON.stringify(config)
    ),
    {
      maxBuffer: 1024 * 1000,
      timeout: 30 * 1000,
    },
    function (error, stdout, stderr) {
      // ...
    }
  )

So if something errors out silently in ERB parsing, eventually the loader will error out.

🤕

[rhys-vdw]

Don't make errors 😉

Hm. In my experience errors raise an error code that fails the build... But perhaps this is misbehaving?

It's a bandaid

@chaffeqa would you open a PR with this fix? We can remove the Ruby timeout handling (#46) which hasn't worked as I'd hoped.

[chaffeqa]

haha no I meant Don't make errors as a joke... because of course the real issue is that as developers we make errors for some darn reason.

Sure thing on the PR!