AiAWE: An Open-Source LLM Automated Writing Evaluation System Using LoRA-Adapted Instruction-Tuned Models

Most machine learning research assumes that larger models inevitably yield better downstream performance. This study challenges that assumption. The researchers found that a 27B-parameter Gemma model actually outperformed a significantly larger 70B-parameter LLaMA model in essay scoring. This system, named AiAWE, provides a way to automate writing assessment using open-source models. It offers accuracy comparable to or better than private AI services like GPT-3.5. It can provide both a numeric score and helpful written feedback for students.

Moving beyond proprietary black boxes

Automated writing evaluation (AWE) is a critical field in education. It aims to provide scalable and consistent feedback to students. Historically, this has been a labor-intensive process. It is often prone to rater fatigue and human inconsistency. While large language models (LLMs) offer a promising solution, most leading research relies on proprietary APIs (Application Programming Interfaces, or software intermediaries). These include OpenAI’s GPT family.

This reliance creates significant friction for educational institutions. Sending sensitive student essays to commercial APIs raises serious compliance risks. These risks involve data sovereignty and regulations like GDPR or FERPA. Furthermore, closed-source models prevent the kind of auditability that high-stakes assessment demands. Researchers cannot inspect what the model has learned. They also cannot verify if its behavior remains stable over time. The central question remains: can an open-source, locally hosted model match the performance of these powerful, closed-door competitors?

Scaling efficiency with LoRA

The authors of this study tackle this problem using Low-Rank Adaptation (LoRA). This is a fine-tuning technique where the base model weights are frozen. Only a small set of additional "adapter" matrices is trained. You can think of this like adding a specialized lens to a camera. The camera's core mechanics remain untouched. However, the lens allows it to focus specifically on a new subject.

The AiAWE system employs a unique dual-purpose design. As illustrated in, the training process only targets a numeric score.

The assistant target during training was the bare human score. However, the base model's original instruction-following capabilities are preserved. This occurs because the LoRA adapters are kept unmerged from the base weights. At inference time, the system can still generate qualitative, rubric-referenced feedback. It does this by requesting a structured JSON (JavaScript Object Notation, a data format) object. This allows a single, lightweight adapter to transform a general-purpose model into a specialized grader. The model can both assign a number and explain its reasoning.

To make the system practical, the authors utilize quantized inference. Quantization is a compression technique. It reduces the precision of a model's weights. For example, it might move from 16-bit to 4-bit precision. This shrinks the memory footprint. By using the llama.cpp framework and the Q4_K_M quantization scheme, the researchers ensure the models can run on consumer-grade hardware. A single NVIDIA RTX 3090 is sufficient for this task.

Better performance from smaller scales

The study's results challenge the prevailing "bigger is better" intuition. The authors report that the 27B-parameter Gemma-3-27B-it model outperformed the 70B-parameter LLaMA-3.3-70B-Instruct model. This held true across every metric tested. Specifically, the Gemma model achieved a root mean square error (RMSE, a measure of error magnitude) of 0.474. It also achieved a quadratic weighted kappa (QWK, a metric for agreement) of 0.828.

When compared to the industry baseline, the Gemma-based system surpasses the performance of a fine-tuned GPT-3.5 model. That baseline was reported in prior work. The Gemma model achieved an agreement rate of 90.56% within ±0.5 of the human score. This means it matched the human score within a half-point margin nearly 91% of the time. In contrast, the GPT-3.5 baseline sat at 84.72%. This demonstrates that open-weight models can meet or exceed the standards set by proprietary giants.

The authors also highlight a crucial discovery regarding model architecture. They found that LoRA hyperparameters are not model-agnostic. This means settings for one model do not work for all others. They observed that both models used identical LoRA settings. However, the LLaMA model exhibited a tendency to "forget" how to generate qualitative feedback. This happened if the LoRA rank was increased above 64. In contrast, the Gemma model remained robust. It maintained both its scoring accuracy and its ability to provide structured reasoning.

Limits of the current implementation

While the results are compelling, the paper does not claim a universal solution. The study is limited to a single dataset of 480 argumentative TOEFL essays. Consequently, the performance on other genres remains unproven. This includes narrative fiction, research papers, or K-12 creative writing.

There is also a notable class imbalance in the training data. The dataset contains very few essays at the extreme ends of the scoring scale. There are very few essays with scores of 1.0 or 5.0. The authors report that error rates increase at these extremes. This suggests the models struggle to identify exceptionally poor or exceptional writing. Additionally, the study does not include a fairness audit. It is unknown if the scoring behavior varies across different demographic groups or native language backgrounds.

A verdict for local deployment

The AiAWE system proves that high-quality, privacy-compliant writing assessment is possible on a budget. A 27B model, optimized via LoRA and 4-bit quantization, can provide professional-grade scoring on a single consumer GPU. The authors have made the code and the model adapters publicly available. These can be found through their GitHub and Hugging Face repositories.

The research successfully shifts the conversation. It moves from asking "how big is your model?" to "how well is your model adapted?". For developers building educational tools, the takeaway is clear. They should prioritize architectural compatibility and careful hyperparameter tuning. Raw parameter count is not a substitute for proper adaptation.