[Enhancement]: Support the function of sorting homework problems

[liangrui198]

Contact Details

liangrui@yy.com

Is there an existing issue for this?

• I have searched all the existing issues

What would you like to be added?

I need a report like this.:

ES sql Only one JSON array can be used as the condition. It is not possible for multiple conditions to be met simultaneously. The data is problematic.

Why is this needed?

From the perspective of an administrator, managing these tasks requires a sorting function. It is necessary to prioritize the top N problematic tasks for optimization, and less important anomalies can be delegated to specific business personnel for optimization. Currently, there is no such sorting function on Compass. For example, if I need to find the "dataSkew top20" jobs, the "compass-detector-*" json in ES is very complex. I have tried many different ES SQL writing methods, but it cannot be counted. It is necessary to perform a re-calculation using Spark first, which is quite troublesome. It would be great if this could be achieved through the Compass function.

Anything else?

No response

[liangrui198]

The ES JSON structure is too complex and Spark cannot derive it. Currently, we can only perform statistics by manually parsing the JSON file. If we could have the sorting function on the Compass interface, it would be much better.

Here is an example of Spark code analysis.
https://liangrui198.github.io/blog/bigdata/ops/compass.html#%E5%BC%82%E5%B8%B8%E6%8E%92%E5%90%8D%E7%BB%9F%E8%AE%A1