When will batch inference support be added?

[leyestd]

Could you please add batch inference support? It would really help speed up large image processing and improve GPU utilization.

[gboeer]

There is an actual PR regarding this:
#1227

[fcakyon]

@leyestd there is no clear ETA yet, but we plan to first introduce optimizations for postprocess, then work on multi-worker inference (which is practically similar to batch inference).

cc: @onuralpszr

[onuralpszr]

@fcakyon thank you, for cc and yes I wrote posprocess changes first to speed up and next will be GPU batch cases very soon.

[leyestd]

Thank you @gboeer, @fcakyon, and @onuralpszr for all the helpful information and the update. Looking forward to it!