Method

Evaluation protocol

Decoding: deterministic settings throughout — do_sample: false (greedy where the backend supports it; temperature: 0.0 on the API backends), capped at max_new_tokens: 2048. All runs share one prompt template (full hash recorded in each run's config snapshot).
Reasoning settings are the one deliberate variable: most runs use the provider default, some request minimal/low effort, and two disable reasoning. The Results page keeps these groups separate.
Extraction and scoring: strict regular expression 정답은\s*(.*?)\s*입니다; from each match, A–J letters are collected in order. A run is correct only if the extracted string is exactly the gold letter.
Invalid outputs count as wrong. A missing or malformed final-answer phrase scores 0; it is never excluded.

Prompt (exactly as sent)

System message, original Korean:

당신은 문제를 해결하는 전문가입니다.

User message template, original Korean ({question} replaced by the item text):

다음 문제에 대해서 충분히 생각하고 추론하여, 10개의 보기(A, B, C, D, E, F, G, H, I, J) 중 정답을 고르세요.

{question}

답변은 반드시 다음 형식을 엄격히 지켜야 합니다: "정답은 [정답 보기]입니다."로 끝나야하고, [정답 보기]는 A, B, C, D, E, F, G, H, I, J 중 하나여야 합니다.정답: 문제를 풀기 위해, 한 번 천천히 생각해봅시다.

English translation (for reference only; models saw the Korean above): “Think through the following problem and choose the correct answer from the 10 options (A, B, C, D, E, F, G, H, I, J). [question] Your answer must strictly keep the following form: it must end with ‘정답은 [choice]입니다.’ ([The answer is [choice].]), where [choice] is one of A–J. Answer: To solve this, let us think slowly.”

Reproduction

Each run directory keeps a config snapshot and a scored results file (see Runs & evidence). This site is generated from those files; raw model predictions are not published. Regenerate the data file with:

python3 site/build.py

This reads completed run directories under runs/ and writes site/results.json plus evidence copies under site/data/runs/<run>/. Incomplete runs are skipped and listed separately.

Dataset note: the recorded dataset revision is 0.0.0, a placeholder rather than a content hash, so exact dataset identity rests on the benchmark release (see upstream links on the Overview page).