Local Data processing

[parasjindal-tf]

Hi Redash community, I have a requirement to process data locally before visualizing it.

There is a certain type of logic which is not very straightforward to write in query language. But if I fetch the required data from DB, and have an option to pre process it before rendering using local js scripts, it would be very helpful.

I have explored the option of custom charts, where I can write custom js functions, but confined to use Plotly. If that can be extended to just return the processed data and can use Redash native charts as it is, it should solve the purpose I believe.

Do you folks have this type of requirement, or any other alternatives to solve this kind of problem?

[arikfr]

This is interesting. The main issue is security concerns... need to somehow sandbox the execution.

[parasjindal-tf]

I have created my own fork for this and added a new js script option before rendering. In the sandbox env, I have set various parameters of default js as undefined like document, axios etc.

You can check this PR - parasjindal-tf#1

This is working for me.

[arikfr]

Thank you for the follow up and sharing! From past experience homegrown sandboxing can be bypassed... In your case it can be bypassed with the following:

// This escapes your shadow variables entirely
const actualGlobalThis = (new Function('return this'))();
actualGlobalThis.alert("I escaped!");

Another concern is that it blocks the main UI loop (in case it does some heavy processing, or just a while(true) loop).

Gemini suggested to delegate the processing to a web worker, which will provide isolation and ability to kill it if it doesn't respond in time. The other suggestions were ShadowRealm (but that's in proposal stage and not yet fully supported) or run it in a WASM QuickJS runtime.

[parasjindal-tf]

Got it. Thanks for quick revert. I understand the concern and believe if this functionality is executed in a same way as Plotly, it would open numerous possibilities. I am already using my fork build for our internal dashboarding and this has been a game changer. Single query is powering multiple charts in a same dashboard.