A pioneering visual system for assessment of student competencies

When a student obtains a university qualification, they are deemed to have acquired with a set of skills, abilities and knowledge learned throughout the courses and years. But what exactly do they know? What competencies lie behind that qualification? To what degree have they acquired them? The Universitat Oberta de Catalunya (UOC) is taking a leap forward in competency-based assessment by launching GRAF, a system that shows students' progress graphically.

This is an innovative project that the UOC started to implement in 2018 with different pilot stages. So far, GRAF has been rolled out in five programs (three master's degrees, one postgraduate program and one specialization program) with a total of 59 courses, 140 members of faculty and 625 students.

"There is a qualitative gap in the final result of the assessment that no one has ever addressed before, which is related to the need to connect more clearly with the employment market," explained Sílvia Sivera, director of the eLearn Center (eLC), the UOC's center for educational innovation and where the system was developed. "GRAF represents a new competency assessment experience at the UOC, allowing us to add value and take a step beyond the usual assessment methods. It has been a cross-disciplinary and interdisciplinary pedagogical engineering challenge," she said. The development of GRAF involved a team of 24 people, including staff from the eLC and the UOC's Academic Services, and directors and support staff from the participating programs.

The system assesses students' progress with their learning and progress with the competencies, and provides them with their own competency graph report. The results are presented graphically for easy inclusion on students' CVs as evidence of the skills acquired during their studies. The teaching staff, for their part, are equipped with a rigorous tool that allows them to assess this progress.

The project is based on the premise that assessing competencies goes beyond marking activities and that there is a need to further develop the results of students' activities beyond the mark usually awarded (A, B, C+, C-, D). This means that students' acquisition of learning must be assessed and they need to be guided in the progressive development of their competencies.

"Thus, the UOC's model is based on a continuous assessment process that lets us monitor students' progress in acquiring the competencies. The activities they carry out let them demonstrate their ability to use knowledge (understood as the combination of information, comprehension, skills, values and attitudes in specific contexts) and their ability to respond to specific demands. These abilities are vital when it comes to boosting their employability," said Carles Sigalés, UOC Vice President for Teaching and Learning.

Assessment tools for the teaching staff

GRAF's competency design tool allows the program director and the coordinating professor to set out the competency map. This competency map is based on two concepts, the dimensions and the rubrics, which help teaching staff to assess students by means of the competency assessment tool.

As Cristina Girona, who is in charge of GRAF's pedagogical aspects and an instructional design consultant for online teaching and learning at the UOC's eLearn Center, explained, "GRAF helps teaching staff to improve the approach to their courses because it requires prior in-depth reflection on the different dimensions of competency work, and a more careful assessment of each activity presented, which means students are better guided for future tasks." Teaching staff receive ongoing training and technical and teaching support at all stages of the process in order to be able to use the tools available to them effectively.

Dimensions are the units into which a competency is broken down—such as the planning and execution of social action projects, the capacity for strategic leadership in companies in the automotive sector, critical thinking for action to transform the planet or the capacity for analysis and problem-solving in the development of computer systems and applications—and are specified in the courses through the activities' learning results. For each competency and dimension an assessment rubric is created. These rubrics have four levels of acquirement for each dimension: not acquired, partially acquired, acquired and excellently acquired.

The GRAF rubrics are the instrument that informs the students of the teaching staff's expectations of what they have to learn and helps them to get an idea of what they know and what they can do.

A step-by-step guide to how GRAF works

The competency map is designed for an entire training program, and then the competency design tool is uploaded to the system and access is provided through the virtual classroom. The students carry out activities to learn and put their skills to test. Once the students hand in their work, the supporting documents are generated and the teaching staff can access the system, assess the level of competency progress using the rubrics and provide feedback. The development of the student's competency learning process is represented graphically, and the student receives their competency graph report (CGR), which they can personalize and export. The CGR is updated with the assessment of each activity, allowing the student to constantly consult their progress.

"Students have an instrument with which they can attest the competencies they have acquired at the University in a way that is understandable to a range of social agents and employers, whether they are competencies linked to the disciplinary field, leadership skills, teamwork in multidisciplinary environments or effective communication," said Núria Domènech, head of the GRAF project and an expert in developing educational innovation processes at the UOC's eLearn Center.

The initiative was a success on the courses and in the faculties where it was pilot tested. Looking to the future, the aim is to address how the model could incorporate evidence from outside the academic world, from students' personal and professional lives, "especially at a university like ours, where more than 90% of the students work," said Sivera. Furthermore, the UOC plans to continue working on the project so that the culture of competency-based assessment becomes a reality at universities.

Competency-based assessment, an official requirement

Designing a competency-based program of study reinforces and strengthens the employability of graduates in the university system, as stated in the Spanish Royal Decree published this September, which establishes how university education is organized and the quality assurance procedure. "Furthermore, with GRAF the UOC aims to go a step further and create something that adds tangible value for both students and companies," explained Sivera. "GRAF is a useful and exportable tool that other universities could adopt and implement."

The UOC continues to work to generate and improve the European Diploma Supplement (DS), the official document that accompanies the university degree certificate. GRAF has been developed along the same lines, as it aims to provide students with a solid and reliable competency accreditation system recognized by a wide range of social agents and employers.

The research was published in Innovations in Learning and Technology for the Workplace and Higher Education.