doc: add warning

terryyz · web-flow · commit 991e41c06d54 · 2024-10-06T16:34:57.000+08:00
diff --git a/README.md b/README.md
@@ -85,6 +85,9 @@ pip install "git+https://github.com/bigcode-project/bigcodebench.git#egg=bigcode
 ## 🚀 Remote Evaluation
 
 We use the greedy decoding as an example to show how to evaluate the generated code samples via remote API.
+> [!Warning]
+>
+> To ease the generation, we use batch inference by default. However, the batch inference results could vary from *batch sizes to batch sizes* and *versions to versions*, at least for the vLLM backend. If you want to get more deterministic results for greedy decoding, please set `--bs` to `1`. 
 
 > [!Note]
 >

Original file line number	Diff line number	Diff line change
`@@ -85,6 +85,9 @@ pip install "git+https://github.com/bigcode-project/bigcodebench.git#egg=bigcode`
`85`	`85`	`## 🚀 Remote Evaluation`
`86`	`86`
`87`	`87`	`We use the greedy decoding as an example to show how to evaluate the generated code samples via remote API.`
	`88`	`+> [!Warning]`
	`89`	`+>`
	`90`	+> To ease the generation, we use batch inference by default. However, the batch inference results could vary from batch sizes to batch sizes and versions to versions, at least for the vLLM backend. If you want to get more deterministic results for greedy decoding, please set `--bs` to `1`.
`88`	`91`
`89`	`92`	`> [!Note]`
`90`	`93`	`>`