Abstract
This study is part of a larger design study that iteratively improves a robotics programming curriculum as well as a computational thinking (CT) instrument. Its focus was majorly on CT assessment and particularly on an online CT instrument with logging functionality that can store a student's problem-solving process by recording interactions between a test-taker and the items with timestamps. The purposes of this research were to examine the psychometric properties of an online CT instrument, test if significant improvement in CT could be found by 200 5th graders who took a robotics programming course and the CT instrument as pretest and posttest, and explore the use of learning analytics methods, mainly convolutional neural networks (CNNs), to help interpret a student’s application of CT when solving a given problem effectively and efficiently. Rasch testlet model was used to perform item response theory analysis on the CT instrument with six testlets. The results showed good reliability in measuring, adequate discrimination capacity of most of the items, and appropriate difficulty level in measuring CT of 5th graders. No statistically significant results were found regarding improvement in CT from pretest to posttest after the intervention, and possible reasons were listed and discussed. Regarding learning analytics, a CNN model was built, tweaked, and trained by student problem-solving process data from two items in the instrument to predict students' successfulness in solving the problems with good to excellent accuracy. And by inspecting the trained model parameters, specific problem-solving patterns that inform the interpretation of CT use during the problem-solving process were identified and discussed.