How to Compare Local LLMs for Korean

Summarization, extraction, honorific register, spacing, and structured output are different capabilities. Weight the set according to real use.

The working rule for “Break down the work” is: Create 10–20 Korean prompts with known evidence. Keep the model, quantization, context length, and concurrency fixed so that a device or model comparison has a clear cause.

For verification, save the model and runtime versions, source input, relevant settings, and observed output together. Repeat the step while changing only one factor, and record unexpected results and untested limits as carefully as successes before applying the guidance to private or production data.

List required facts and disqualify unsupported claims. Use public or anonymized documents.

The working rule for “Define expected evidence” is: Fix runtime, prompt, temperature, context, and quantization. Keep the model, quantization, context length, and concurrency fixed so that a device or model comparison has a clear cause.

Preserve the before-and-after state and the time of the check so that another run can reproduce the result. Include at least one failure condition—such as empty input, constrained resources, or a restart—to reveal the boundary of the step rather than documenting only the happy path.

Run each case multiple times and record the full model name, version, runtime, and date.

The working rule for “Repeat under fixed conditions” is: Score fluency and factual accuracy separately. Keep the model, quantization, context length, and concurrency fixed so that a device or model comparison has a clear cause.

Define completion with an observable result instead of a general impression. Repeat the same input, and if the output changes, isolate whether the model, runtime settings, or source data changed before moving to the next stage.

Measure factual match, instruction following, natural Korean, format validity, consistency, and speed.

The working rule for “Score separate dimensions” is: Create 10–20 Korean prompts with known evidence. Keep the model, quantization, context length, and concurrency fixed so that a device or model comparison has a clear cause.

Treat the table as a recording framework, not as a universal answer. Fill it with measurements from the intended device and workload, and mark unavailable values as unknown instead of replacing them with zero or an estimate that could distort the comparison.