Machine Learning
Self-learning algorithm to benefit teachers and students

Mr Rüdian, what is the difference between artificial intelligence, neural networks, and self-learning programs?

We use the term artificial intelligence to describe processes that learn through data and form decisions accordingly. The underlying idea is to use collected data from the real world, such as pictures of cats and dogs for instance. A neural network can then be trained to recognize which images depict cats and which show dogs so that for any future images it will recognize which of the two animals it is ‘seeing’. Structures like neural networks can help us to make predictions. However, there is one pitfall: they only show us an answer they do not reveal how they come to their conclusions. You are researching automated feedback for open writing tasks. What exactly is at the center of the joint project between HU Berlin and Goethe-Institut?

Leo S. Rüdian: During numerous joint workshops with the Goethe-Institut we observed a huge gap between our theoretical research and its practical application. We have been working with many different types of technology which, despite showing great results, still haven’t been practically introduced in the German online courses. One of the areas we have looked at is automated text analysis, where right from the beginning our goal was to transfer our research findings to the online courses of the Goethe-Institut. In the first phase, we are focusing on supporting tutors with providing feedback for open writing tasks.
 
Which problem could an automated text analysis solve in this context?

Leo S. Rüdian: One of the main duties performed by teachers is evaluating and giving feedback for open writing tasks. Teachers must check whether a text is appropriate for the relevant course section, whether students have used the acquired vocabulary and the correct grammatical structures. To do so, teachers require extensive meta-knowledge of a course. We wanted to find out whether we could develop a process, which is capable of giving feedback for open writing tasks, thus, providing a support system for teachers. In fact, there are several objective parameters, which could also be processed by algorithms.
 
Which data do you use to train the program?

Portrait Leo Sylvio Rüdian | Photo (detail): © Leo Sylvio Rüdian Leo S. Rüdian: We work on two levels; using both the course content and already marked open writing tasks submitted in the past. Course content is obviously limited, text submissions, however, can be numerous and their numbers are increasing by the day. One difficulty we face is that for each writing task, we have to retrain the process from scratch. This is very time-consuming as for each exercise we need hundreds, better even thousands, of previously marked text submissions.   
 
According to your research, the self-learning program correctly predicts 70% of feedback given by tutors. Did this result come as a surprise to you?

Leo S. Rüdian: Yes and no. Before we used the collected data, we took a very close look at it so we could develop a method that would predict the adequacy of a submitted text. Some of the features we looked at are obvious; texts that had been highly rated in the past usually contained a higher number of learned words and grammatical structures from the relevant course section than others. What was surprising was that even when using different techniques we were unable to achieve a score higher than 70%.
 
What about the other 30%? How does the program compare to humans in this context?

Leo S. Rüdian: As scientists in the field of Machine Learning, we usually try to achieve much higher values. From our perspective, 70% is a relatively low value, as this means that three out of ten decisions are inaccurate. Consequently, in an additional experiment we reviewed how well our score of 70% performed compared to methods currently in place. We asked a number of teachers to each mark hundreds of the exact same submitted texts in order to measure consistency between their ratings. Fact is, we achieved similar scores of roughly 70%. As we trained the program using data produced by humans, the variance is already encoded. Ultimately, the method we use is based on rules; accordingly, an incorrect text that does not contain the newly acquired vocabulary will not be identified as a good submission – the underlying logic prevents this from happening. However, if we had more data for training, we could most likely make further adjustments to our method. Currently, we are working with several hundred ratings; a greater number could help to achieve better results.
 
How does automated feedback provide technology-based support for tutors?

Leo S. Rüdian: The automated text analyses provides objective feedback based on course content and previous teacher ratings; this can be very helpful for the teacher marking a text. If teachers have to correct texts from various course sections, they also have to know the content of all the sections in detail. The program knows not only all the content of all sections in detail, it also feeds on previous ratings performed by other teachers. Therefore, feedback will be more consistent and thus fairer. This is also an important consideration for students. At the same time, the program also recognizes the adequacy of the vocabulary used in a submission; if the used vocabulary does not match the requirements, a teacher may use the information to ask a student to review their submission. This makes the feedback process much more efficient.
 
The self-learning program is well equipped to detect fake submission. Could you explain more about this specific use case?

Leo S. Rüdian: Sometimes students look to automated translation tools such as Google Translate or DeepL to complete their open writing tasks. Thanks to our objective criteria we are usually able to detect these fake submissions. Those texts often contain linguistic structures and words that are not included in the relevant course. Additionally, they often show very few mistakes making it very unlikely that this would be a text from someone only just starting to learn a language. Our algorithm will flag any combination of very few mistakes and high linguistic complexity in a text and subsequently send an alert to the teacher. The warning prompts the teacher to take a closer look at the relevant submission.
 
What does the future of language learning look like?

Leo S. Rüdian: We all know that learning a foreign language is not just about learning vocabulary. What all language learners are trying to achieve is the ability to interact in a foreign language. However, very few technology-based solutions currently support this important process. The future of language learning will match individual needs based on each student’s previous knowledge and preferences. Where, up until now, a personal tutor was needed for an individual learning experience, technology-based solutions will soon provide personalized, much more efficient approaches to learning a foreign language. These solutions will still not replace teachers any time in the near future instead they will support them in their work.