Dane DeSutter, Ph.D.
Case study

AI grading in Labflow

An AI grading assistant that helps TAs grade rather than replacing them.

The Labflow grading screen with AI: a list of report sections on the left, a student's answer about choosing a solvent in the middle, and on the right a scoring panel with a rubric of 5, 3 and 2 point answers and a feedback box.

AI-supported grading in Labflow uses large language models to assess student responses to lab questions and draft targeted feedback for the TA. I did the foundational research on how accurate the models are at grading, helped define the architecture, and wrote the grading prompts. The engineering team built the interface and the services behind it, and made sure everything worked together and scaled.

Where I started

Before building anything, I wanted to understand where AI could help in university science courses, and what to avoid. I interviewed colleagues and customers about where Labflow could use AI. I also gave a questionnaire to students who had used a competitor’s AI grading tool, and interviewed users of that tool.

What I learned

Large language models give a strong impression of understanding, which shapes what people expect from them. Students who used the competitor’s tool had strong feelings: in the questionnaire, 43% said they were graded fairly, 47% said they were not, and 10% were undecided. But AI wasn’t simply good or bad. Two things drove the negative reactions: it didn’t grade the same answer the same way twice, and its criteria weren’t clear. One thing drove the positive ones: it coached students well on material they were seeing for the first time.

What came of it

I turned those findings into candidate features and scored them against a rubric tied to company goals. The clearest result was an assistant that helps TAs grade, not one that replaces them, and that became the direction for the proof of concept. From there my part was the research on grading accuracy, the architecture, and the prompts. The engineering team built the product around them.