The current default method for getRandomContext() is min, however, there is no way for task owners to change this currently.
Ideally this should be controlled from the task config (e.g. https://dynabench.org/task-owner-interface/qa#advanced) similar to aggregation_metric.
We would need to modify: https://github.com/facebookresearch/dynabench/blob/main/frontends/web/src/common/Annotation/CreateInterface.js#L95.
I suspect the steps required to implement are:
- Move the
taskConfig loading slightly earlier
- Process the new
taskConfig.ATTRIBUTE_NAME attribute accordingly with a default value of min (not sure if we need to make any other changes to make it editable)
- Change
.getRandomContext(this.state.task.id, this.state.task.cur_round) to .getRandomContext(this.state.task.id, this.state.task.cur_round, [], taskConfig.ATTRIBUTE_NAME)
The current default method for getRandomContext() is
min, however, there is no way for task owners to change this currently.Ideally this should be controlled from the task config (e.g. https://dynabench.org/task-owner-interface/qa#advanced) similar to
aggregation_metric.We would need to modify: https://github.com/facebookresearch/dynabench/blob/main/frontends/web/src/common/Annotation/CreateInterface.js#L95.
I suspect the steps required to implement are:
taskConfigloading slightly earliertaskConfig.ATTRIBUTE_NAMEattribute accordingly with a default value ofmin(not sure if we need to make any other changes to make it editable).getRandomContext(this.state.task.id, this.state.task.cur_round)to.getRandomContext(this.state.task.id, this.state.task.cur_round, [], taskConfig.ATTRIBUTE_NAME)