Skip to content

Commit 9ca30af

Browse files
wangyukaikew6688
authored andcommitted
[doc] update challenge page
1 parent 3ed96cf commit 9ca30af

7 files changed

Lines changed: 531 additions & 105 deletions

File tree

src/content/docs/challenge.md

Lines changed: 69 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@ Use `gmp online submit` to request a remote evaluation job:
5252
gmp online submit \
5353
--base_url https://internrobotics.shlab.org.cn/eval \
5454
--token "$EBENCH_SUBMIT_TOKEN" \
55+
--task_id "$PREVIOUS_TASK" \ # optional: continue with a previous task
5556
--benchmark_set ebench_generalist \
5657
--model_name internVLA \
5758
--model_type VLA \
@@ -60,7 +61,7 @@ gmp online submit \
6061
--is_public 0
6162
```
6263

63-
### Parameters
64+
#### Parameters
6465

6566
| Parameter | Type | Example | Description |
6667
|-----------|------|---------|-------------|
@@ -75,9 +76,22 @@ gmp online submit \
7576
After the backend task is ready, the command returns fields like:
7677

7778
```json
79+
Waiting for available server (task_id=b5dddc6de60c4aec8236500b8e3dc0e1)...
80+
Still waiting... elapsed 0.1s. Next check in 5.0s.
81+
Still waiting... elapsed 5.3s. Next check in 5.0s.
82+
Ready after 10.4s. endpoint=https://internverse.shlab.org.cn/eval-server/2813aea1/api/predict/embodied_eval.genmanip_eas_1_master_prod
7883
{
79-
"task_id": "9ea5fb6ae980430da626958c4433ea18",
80-
"endpoint": "https://internrobotics.shlab.org.cn/evalserver/9391d9e8/api/predict/embodied_eval.genmanip_eas_1_master"
84+
"task_id": "b5dddc6de60c4aec8236500b8e3dc0e1",
85+
"endpoint": "https://internverse.shlab.org.cn/eval-server/2813aea1/api/predict/embodied_eval.genmanip_eas_1_master_prod",
86+
"response": {
87+
"code": 0,
88+
"msg": "success",
89+
"trace_id": "4a4136c66bdc80922ccc6485c44fa9e5",
90+
"data": {
91+
"ready": true,
92+
"endpoint": "https://internverse.shlab.org.cn/eval-server/2813aea1/api/predict/embodied_eval.genmanip_eas_1_master_prod"
93+
}
94+
}
8195
}
8296
```
8397

@@ -86,15 +100,52 @@ Record both values:
86100
- `task_id`: use this as the `run_id` when running evaluation.
87101
- `endpoint`: use this as the remote evaluation URL.
88102

103+
#### Demo: Auto extract `endpoint` and `task_id`
104+
105+
The following example uses a simplified Python script to run `gmp online submit` and extract `endpoint` and `task_id` from the returned output:
106+
107+
```python
108+
import os
109+
import json
110+
import subprocess
111+
112+
def submit_online_task() -> tuple[str, str]:
113+
cmd = [
114+
'gmp', 'online', 'submit',
115+
'--base_url', 'https://internrobotics.shlab.org.cn/eval',
116+
'--token', os.environ['EBENCH_SUBMIT_TOKEN'],
117+
'--benchmark_set', 'ebench_generalist',
118+
'--model_name', 'internVLA',
119+
'--model_type', 'VLA',
120+
'--submitter_name', 'test',
121+
'--submitter_homepage', 'test',
122+
'--is_public', '0',
123+
]
124+
125+
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
126+
output = result.stdout
127+
json_start = output.find('{')
128+
payload = json.loads(output[json_start:])
129+
endpoint = payload['endpoint']
130+
task_id = payload['task_id']
131+
print('endpoint=' + endpoint)
132+
print('task_id=' + task_id)
133+
return endpoint, task_id
134+
```
135+
136+
After running the script, it will print `endpoint` and `task_id` directly, which you can then use in the subsequent evaluation worker call.
137+
89138
### 4. Start evaluation workers
90139

91140
Run the evaluator against the returned endpoint. This is a test evaluation. Follow the doc to reate your own model evaluation.
92141

93142
```python
143+
endpoint, task_id = submit_online_task()
144+
94145
client = EvalClient(
95-
base_url="https://internrobotics.shlab.org.cn/evalserver/9391d9e8/api/predict/embodied_eval.genmanip_eas_1_master",
96-
token="$EBENCH_SUBMIT_TOKEN"
97-
run_id="9ea5fb6ae980430da626958c4433ea18",
146+
base_url=endpoint,
147+
token=os.environ['EBENCH_SUBMIT_TOKEN'],
148+
run_id=task_id,
98149
worker_ids=["0"]
99150
)
100151
model = ModelClient(...)
@@ -115,9 +166,9 @@ You can start several eval client with different ids. i.e.
115166

116167
```python
117168
client = EvalClient(
118-
base_url="https://internrobotics.shlab.org.cn/evalserver/9391d9e8/api/predict/embodied_eval.genmanip_eas_1_master",
119-
token="$EBENCH_SUBMIT_TOKEN"
120-
run_id="9ea5fb6ae980430da626958c4433ea18",
169+
base_url=endpoint,
170+
token=os.environ['EBENCH_SUBMIT_TOKEN'],
171+
run_id=task_id,
121172
worker_ids=["1"]
122173
)
123174
...
@@ -133,7 +184,7 @@ gmp online submit \
133184
# ...
134185
```
135186

136-
If you encounter connection timeouts, restart the client to recover.
187+
If you encounter connection timeouts, restart the client to restart. The progress will be saved on server.
137188

138189
### 5. Monitor the task
139190

@@ -148,20 +199,17 @@ gmp status \
148199
--run_id "$EBENCH_TASK_ID"
149200
```
150201

151-
## Online Submit URL
152-
153-
Create tasks through the official platform base URL:
154-
155-
```text
156-
https://internrobotics.shlab.org.cn/eval
202+
### 6. Stop the task
203+
Stop an evaluation session by
157204
```
158-
159-
After `gmp online submit`, use the returned per-task endpoint for evaluation:
160-
161-
```text
162-
https://internrobotics.shlab.org.cn/evalserver/<task-endpoint>
205+
gmp online stop \
206+
--url "$EBENCH_ONLINE_ENDPOINT" \
207+
--token "$EBENCH_SUBMIT_TOKEN" \
208+
--run_id "$EBENCH_TASK_ID" \
209+
--user_id "$USER_ID" # get from the website, Your account page
163210
```
164211

212+
165213
## Scoring Rules
166214

167215
- Each evaluated episode produces a task score between `0.0` and `1.0`.

src/content/docs/de/challenge.md

Lines changed: 75 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,8 @@ Verwenden Sie `gmp online submit`, um einen entfernten Bewertungsjob anzufordern
5252
gmp online submit \
5353
--base_url https://internrobotics.shlab.org.cn/eval \
5454
--token "$EBENCH_SUBMIT_TOKEN" \
55-
--benchmark_set EBench \
55+
--task_id "$PREVIOUS_TASK" \ # optional: continue with a previous task
56+
--benchmark_set ebench_generalist \
5657
--model_name internVLA \
5758
--model_type VLA \
5859
--submitter_name test \
@@ -67,17 +68,30 @@ gmp online submit \
6768
| task_id | string | T2025123100001 | Optional, vorherige task_id für Aufgabenwiederholung verwenden |
6869
| model_name | string | internVLA | Modellname |
6970
| model_type | string | VLA | Modelltyp |
70-
| benchmark_set | string | EBench | Benchmark-Set-Typ, derzeit nur EBench erlaubt |
71+
| benchmark_set | string | EBench | Benchmark-Set-Typ, derzeit nur ebench_generalist erlaubt |
7172
| submitter_name | string | SHlab | Organisations-/Entwicklername |
7273
| submitter_homepage | string | http://example.com | Homepage des Einreichers |
7374
| is_public | int | 0 | Ob öffentlich<br>0 Nein<br>1 Ja |
7475

7576
Sobald die Backend-Aufgabe bereit ist, gibt der Befehl Felder wie diese zurueck:
7677

7778
```json
79+
Waiting for available server (task_id=b5dddc6de60c4aec8236500b8e3dc0e1)...
80+
Still waiting... elapsed 0.1s. Next check in 5.0s.
81+
Still waiting... elapsed 5.3s. Next check in 5.0s.
82+
Ready after 10.4s. endpoint=https://internverse.shlab.org.cn/eval-server/2813aea1/api/predict/embodied_eval.genmanip_eas_1_master_prod
7883
{
79-
"task_id": "9ea5fb6ae980430da626958c4433ea18",
80-
"endpoint": "https://internrobotics.shlab.org.cn/evalserver/9391d9e8/api/predict/embodied_eval.genmanip_eas_1_master"
84+
"task_id": "b5dddc6de60c4aec8236500b8e3dc0e1",
85+
"endpoint": "https://internverse.shlab.org.cn/eval-server/2813aea1/api/predict/embodied_eval.genmanip_eas_1_master_prod",
86+
"response": {
87+
"code": 0,
88+
"msg": "success",
89+
"trace_id": "4a4136c66bdc80922ccc6485c44fa9e5",
90+
"data": {
91+
"ready": true,
92+
"endpoint": "https://internverse.shlab.org.cn/eval-server/2813aea1/api/predict/embodied_eval.genmanip_eas_1_master_prod"
93+
}
94+
}
8195
}
8296
```
8397

@@ -86,15 +100,52 @@ Notieren Sie beide Werte:
86100
- `task_id`: Verwenden Sie diesen Wert beim Ausfuehren der Bewertung als `run_id`.
87101
- `endpoint`: Verwenden Sie diesen Wert als entfernte Bewertungs-URL.
88102

103+
#### Demo: `endpoint` und `task_id` automatisch extrahieren
104+
105+
Das folgende Beispiel verwendet ein vereinfachtes Python-Skript, um `gmp online submit` auszufuehren und `endpoint` sowie `task_id` aus der Rueckgabe zu extrahieren:
106+
107+
```python
108+
import os
109+
import json
110+
import subprocess
111+
112+
def submit_online_task() -> tuple[str, str]:
113+
cmd = [
114+
'gmp', 'online', 'submit',
115+
'--base_url', 'https://internrobotics.shlab.org.cn/eval',
116+
'--token', os.environ['EBENCH_SUBMIT_TOKEN'],
117+
'--benchmark_set', 'ebench_generalist',
118+
'--model_name', 'internVLA',
119+
'--model_type', 'VLA',
120+
'--submitter_name', 'test',
121+
'--submitter_homepage', 'test',
122+
'--is_public', '0',
123+
]
124+
125+
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
126+
output = result.stdout
127+
json_start = output.find('{')
128+
payload = json.loads(output[json_start:])
129+
endpoint = payload['endpoint']
130+
task_id = payload['task_id']
131+
print('endpoint=' + endpoint)
132+
print('task_id=' + task_id)
133+
return endpoint, task_id
134+
```
135+
136+
Nach dem Ausfuehren des Skripts werden `endpoint` und `task_id` direkt ausgegeben. Diese Werte koennen Sie anschliessend im Aufruf des Bewertungs-Workers verwenden.
137+
89138
### 4. Bewertungs-Worker starten
90139

91140
Fuehren Sie den Evaluator gegen den zurueckgegebenen Endpoint aus. Dies ist eine Test-Bewertung. Folgen Sie der Dokumentation, um Ihre eigene Modell-Bewertung zu erstellen.
92141

93142
```python
143+
endpoint, task_id = submit_online_task()
144+
94145
client = EvalClient(
95-
base_url="https://internrobotics.shlab.org.cn/evalserver/9391d9e8/api/predict/embodied_eval.genmanip_eas_1_master",
96-
token="$EBENCH_SUBMIT_TOKEN"
97-
run_id="9ea5fb6ae980430da626958c4433ea18",
146+
base_url=endpoint,
147+
token=os.environ['EBENCH_SUBMIT_TOKEN'],
148+
run_id=task_id,
98149
worker_ids=["0"]
99150
)
100151
model = ModelClient(...)
@@ -115,9 +166,9 @@ Sie koennen mehrere Eval-Clients mit verschiedenen IDs starten. Z. B.
115166

116167
```python
117168
client = EvalClient(
118-
base_url="https://internrobotics.shlab.org.cn/evalserver/9391d9e8/api/predict/embodied_eval.genmanip_eas_1_master",
119-
token="$EBENCH_SUBMIT_TOKEN"
120-
run_id="9ea5fb6ae980430da626958c4433ea18",
169+
base_url=endpoint,
170+
token=os.environ['EBENCH_SUBMIT_TOKEN'],
171+
run_id=task_id,
121172
worker_ids=["1"]
122173
)
123174
...
@@ -133,7 +184,7 @@ gmp online submit \
133184
# ...
134185
```
135186

136-
Wenn Sie Verbindungs-Timeouts feststellen, starten Sie den Client neu, um die Verbindung wiederherzustellen.
187+
Wenn Sie Verbindungs-Timeouts feststellen, starten Sie den Client neu, um die Verbindung wiederherzustellen. Der Fortschritt wird auf dem Server gespeichert.
137188

138189
### 5. Aufgabe ueberwachen
139190

@@ -148,6 +199,18 @@ gmp status \
148199
--run_id "$EBENCH_TASK_ID"
149200
```
150201

202+
### 6. Aufgabe stoppen
203+
204+
Stoppen Sie eine Evaluierungssitzung mit:
205+
206+
```
207+
gmp online stop \
208+
--url "$EBENCH_ONLINE_ENDPOINT" \
209+
--token "$EBENCH_SUBMIT_TOKEN" \
210+
--run_id "$EBENCH_TASK_ID" \
211+
--user_id "$USER_ID" # von der Website erhalten, Ihre Kontoseite
212+
```
213+
151214
## URL fuer die Online-Einreichung
152215

153216
Erstellen Sie Aufgaben ueber die offizielle Plattform-Base-URL:
@@ -159,7 +222,7 @@ https://internrobotics.shlab.org.cn/eval
159222
Verwenden Sie nach `gmp online submit` den zurueckgegebenen aufgabenspezifischen Endpoint fuer die Bewertung:
160223

161224
```text
162-
https://internrobotics.shlab.org.cn/evalserver/<task-endpoint>
225+
https://internverse.shlab.org.cn/evalserver/<task-endpoint>
163226
```
164227

165228
## Bewertungsregeln

0 commit comments

Comments
 (0)