Preliminary Insights into Student Skills Development

Executive Summary

The objective of this preliminary study was to evaluate the perceived improvement in skills development among student users of the Rflect app during autumn semester 2023. The analysis involved assessing the mean average score changes between pre-and post-surveys across 5 dimensions inspired by the Inner Development Goals¹, as well as examining qualitative responses using a multi-stage qualitative content analysis². The quantitative results exceeded the Rflect team’s initial hypothesis, and the qualitative data confirmed that Rflect has had a positive impact on student skills development and student users’ ability to reflect. This self-study serves as an initial step towards making Rflect more evidence-based, and will be followed by an academic rigor study in spring semester 2024. The results should therefore be interpreted with cautious optimism.

Acknowledgements

Written by Ella Stadler-Stuart³ (Rflect⁴) and Dana Blume⁵ (LeLa⁶, ZHdK Dossier Learning and Teaching - E-Learning⁷), in consultation with Prof. Dr. Anna Jasinenko⁸ (Assistant Prof. Organizational Behavior at the University of St. Gallen⁹) and Monica Barroso PhD¹⁰ (Senior Researcher and Project Manager, University of St. Gallen).

Introduction

In recent years, the emphasis on personal development of competences and reflective practices in education has intensified. Aiming to accelerate this shift in higher education, the Swiss edtech startup Rflect built a web app in 2023 to help universities instill a lifelong habit of inner development in their students. The Rflect app is a unique solution that enables students to personalize and reflect on their learning journeys in real-time, while also enabling students and lecturers to track student progress on competences such as self-awareness, critical thinking, collaboration, and others communicated by the Inner Development Goals (IDGs) framework.

This report presents a preliminary study conducted during the autumn semester of 2023, aimed at evaluating the effectiveness of Rflect in enhancing student skills development, in this case over the course of a semester. By comparing pre- and post-self-assessment survey results and analyzing qualitative feedback, this study seeks to understand the impact of Rflect on various dimensions of personal development. The findings from this study will provide foundational insights for a more comprehensive academic study planned for the spring semester of 2024.

Initial Hypothesis

Rflect’s initial hypothesis posited that students would report improvements in at least 3 out of the 5 tested categories, which were inspired by the IDGs¹¹. In this preliminary version of the self-assessment, the 5 categories were labeled:

1. Self-Awareness and Goal Setting (IDG: Being), 2. Cognitive Skills (IDG: Thinking), 3.Social and Communication Skills (IDG: Collaborating), 4. Emotional Maturity (IDG:Being), and 5. The World Around Me (IDG: Relating).

Specifically, we expected to see notable improvements in categories 1 (Self-Awareness and Goal Setting), 3 (Social and Communication Skills), and 5 (The World Around Me). We based this hypothesis on the conviction that reflective practice enhances self-awareness¹² and because Rflect’s features were conceptualized specifically for student personal development to take place. App features include: student self-assessment, guided reflection, peer-coaching, and insight-to-action challenges (see Image 1 below for a simplified visualization of an average student’s learning journey using Rflect). Across the semester, students are prompted to reflect on their course content as well as their personal learning goals, values, and sense of self. The peer-coaching feature also enables students to interact with their peers and practice essential communication skills.

Image 1: Visualization of student learning journey with Rflect’s core features.

Approach

Student users of Rflect were invited to complete pre- and post-surveys starting with quantitative questions using a 1-10 scale. Each survey took approximately 20 minutes to complete. Between the pre- and post-surveys, students engaged in their regular curricula while using Rflect to reflect on and learn from their experiences¹³. Upon completing the post-survey, they were then asked to write meta-reflections on their comparative results from the two surveys, which are displayed in-app.

A total of 39 users completed both the pre- and post-assessment surveys and these were evaluated for this paper. The results were anonymized and analyzed by the Rflect team together with academic advisor Dana Blume.

Respondents

The survey respondents consisted of 39 students enrolled in 7 different higher education programs in German-speaking Switzerland during the autumn semester of 2023. These programs included various further education certificate programs, such as Certificates of Advanced Studies, as well as modules within Bachelor’s degree programs. In this preliminary study, demographic data was not collected. However, future studies will include demographic data collection to enhance data understanding and to enable more precise clustering of information.

Development of self-assessment survey¹⁴

The tested dimensions and the questions in the self-assessment survey used for this study were driven by the Rflect founding team in consultation with its academic advisors, inspired among others by the IDGs and other best practices on supporting personal development in higher education¹⁵.

Rflect intentionally developed a lean, self-made version of the self-assessment in true startup spirit to iterate and learn. Many improvements have already been made to the spring semester 2024 surveys, such as the introduction of peer-reviewed academic scales (1-7) with items connected to specific competences listed in the IDGs.

Question examples from the autumn semester 2023 surveys:

• How confident are you in your ability to set goals? (from Self-awareness and Goal Setting (IDG: Being)). Students were asked to intuitively give themselves a score from 1 to 10 = 1 Not at all, 5 OK/Solid and 10 Very confident.
• How confident are you in your ability to collaborate with others in a team/group work setting? (from Social and Communication Skills (IDG: Collaborating)).
• Qualitative examples from post-survey, after students have seen comparative results:
  • To what extent do the results fit with where you feel you are in your personal development? Were there any surprises or unexpected insights?
  • In what aspects or specific skills do you personally feel you have grown or changed the most since your pre-assessment? Give 1-3 examples. Is this visible in your results?
  • What aspects or skills are you eager to explore or enhance going forward?

Key findings from quantitative results

The mean differences between the pre- and post-surveys of all 5 measured dimensions were significant according to a paired t-test (p < .01). Image 2 below depicts mean improvement scores for all 5 dimensions.

The quantitative results show that students using Rflect reported a positive mean improvement across all 5 tested dimensions, thus exceeding our initial hypothesis (We expected to see notable improvements in categories 1 (Self-Awareness and Goal Setting / IDG: Being), 3 (Social and Communication Skills / IDG: Collaborating), and 5 (The World Around Me / IDG: Relating)). The most notable changes were observed in dimensions 4: Emotional Maturity (+0.90) and 1: Self-Awareness and Goal Setting (+0.61).

Image 2: Mean Improvement Scores by Competence Dimension by Rflect users 2023.

Other quantitative data collected was the time spent in Rflect. During the autumn semester of 2023, students using Rflect spent between 20.7 and 302.27 minutes writing reflections in the app, and wrote between 4,371 to 53,168 characters in total. The completion rate varied from 52% to 100% of all reflections. Analysis revealed a low to moderate positive correlation (approximately 0.318) between the percentage of completion and the time spent (minutes) writing in Rflect. We believe that several factors likely influenced these outcomes, including the curriculum structure, which determined the number of reflection topics and corresponding questions in the guided reflection plan (ranging from 31 to 88 questions or reflection prompts per program). Additionally, we believe student engagement with Rflect and their metacognitive abilities played a significant role. However, time spent in the app does not necessarily equate to proficiency in metacognition and reflective skills. This observation suggests a need for further investigation to better understand the nuances of student engagement and reflection quality in future studies. In future, we will encourage student users of Rflect to spend at least 10-20 minutes reflecting per topic.

Key findings from qualitative results

“It’s fascinating to watch the growth of my answers over time. Seeing this transition reflects the improvement in my personal development journey.”
Student User, 2023.

Upon completing the quantitative questions in the post-survey, students were able to view their comparative results and asked to write meta-reflections. Questions included:

• To what extent do the results fit with where you feel you are in your personal development? Were there any surprises or unexpected insights?
• In what aspects or specific skills do you personally feel you have grown or changed the most since your pre-assessment? Give 1-3 examples. Is this visible in your results?
• What aspects or skills are you eager to explore or enhance going forward?

Student qualitative responses were examined using a multi-stage qualitative content analysis. In the following section, we shed light on the thematic results as well as our approach to mapping students’ metacognition skills.

Thematic results

During analysis of students’ responses, certain themes surfaced in terms of what the students felt they had improved, what they had learned, and where they saw room for improvement. Below, we highlight a number of competences where significant changes were observed, always accompanied by a quote from a student and learning outcomes for Rflect:

• Self-awareness:¹⁶ “It’s fascinating to watch the growth of my answers over time. Seeing this transition reflects the improvement in my personal development journey… Each answer has evolved and morphed from a brief idea to display a deeper understanding and a more sophisticated perspective. This shift reflects not only the knowledge I’ve received, but also the self-awareness and introspective skills I’ve grown. It’s like seeing a mental and emotional trip unfold, with each step forward a tribute to the time and effort I’ve put into bettering myself.”
  This is where we expected the most positive change, as self-awareness is a key result of regular reflection¹⁷. Both the quantitative and qualitative results seem to confirm this.
• Confidence and self-compassion:¹⁸
  Quote 1: “It applies to all areas: I apparently feel more confident everywhere…It also confirms that I have been through a lot in my life and am moving forward.”¹⁹
  Quote 2: “By reflecting more often, I also realized that I am able to treat myself with more compassion and respect. I am eager to treat myself better in the future.”
  Students were regularly asked to reflect on their progress and learning goals. Tracking and recognising their own progress seems to help build confidence in students.
• Emotional maturity and regulation:²⁰
  Quote 1: “I can handle my emotions better due to the fact that I can reflect on them better and try to understand where they are coming from.”
  Quote 2: “I am eager to enhance my ability to regulate my emotions more when they arise. Because I think it is a very important skill to have and also very beneficial for future group works and projects.”
  For many, this was one of the first times they were asked to reflect on their emotions and emotional responses to key experiences during their learning journey. For Rflect, it will be interesting to explore what further input could help students to grow further in this dimension.
• Importance of values and sustainability (in decision making):²¹
  Quote 1: “…for me personally, values are very important and will always stay very essential. I am excited to inform myself even more with the subject of sustainability.”
  Quote 2: “I believe my values and personal vision should play a larger role in my career decisions.”²²
  Students were asked regularly to reflect on their values and this shows in the results. Rflect plans to integrate a devoted section to personal learning goals and values into the web app, so that they are ever-present.
• Resilience:²³ “These results fit well in my personal development. They’ve emphasized the significance of accepting failure as a stepping stone to achievement, as well as the ongoing nature of personal development. While I’ve improved my critical thinking and self-reflection abilities, I also see opportunities for development, as there is always room for improvement.”
  This is an area that Rflect can build on. We believe helping students train resilience as a muscle could bring value to long-term users of Rflect in particular, for example in a Bachelor’s or Master’s program.
• Holistic understanding of learning:²⁴ “I’ve realized that learning is much more than just academic or visible outcomes. It’s about the combination of various inputs, an integrative process that involves not only classroom education and personal experiences but also, and perhaps most importantly, interactions with others. Learning, I’ve come to understand, is a holistic journey, one that is constantly enriched by the diverse perspectives and knowledge shared by those around us.”
  This was an area in which many students expressed a perspective change. Both the quantitative and qualitative results seem to confirm this. It is worth exploring further and gaining new insights.
• Future learning:²⁵
  Quote 1: “Critical thinking is something I could improve. Questioning my own decisions but also others.”
  Quote 2: “One aspect I am really keen on improving is my public speaking skills.”
  Quote 3: “I have improved my ability to make decisions by being more aware of my thoughts and reflecting more. I aim to make better decisions in the future by continuing my reflection process.”
  Students were asked to reflect on key skills they wanted to improve on as well as reflection as a metacognition skill. Rflect will keep both levels in mind when developing new features.
• Insight into action (transforming knowledge into behavioral change):²⁶ “…I didn’t improve my ability to give feedback and to ask good questions. This is something I need to work on but I don’t really know why it happens.”
  “I already knew all this (about myself) before, now I need to know how to move forward with this and change it.”
  This is an area that we believe Rflect can have more of an impact in future: helping students connect knowledge with effecting change in their lives, for example expanding on the “Actions” feature.

Building metacognition as a skill²⁷

In order to better map and understand the variance in reflection skills across all respondents, academic advisor Dana Blume categorized the responses into evaluation categories visible in Table 1: Evaluation Categories for Reflection, which assesses different levels of reflective ability. Each level represents progress in the ability to reflect, starting with no change, through confirmation of one’s own reflection skills and recognition of the importance of reflection, to improvement in reflection skills and finally to meta-reflection.

Table 1: Evaluation Categories for Reflection: The scale ranges from 0 (‘No change in reflection skills’) to 3 (‘Meta-reflection’), illustrating different levels of reflective ability development. ²⁸

The level “Meta-reflection” identifies reflection at a ‘higher’ level. While reflection means consciously examining and analyzing thoughts, feelings and actions, meta-reflection goes one step further by examining the way in which this reflection takes place. Meta-reflection is about becoming aware of one’s own reflective processes and critically scrutinizing them. In this sense, meta-reflection, i.e. reflection on reflection, shows that the respondents (attempt to) understand and improve their own thought processes, which can lead to deeper and more sustainable learning and personal growth.

Example responses for each category of reflection: “I just found out that there is much more to do. Especially reflecting is something I would like to improve.” (Reflection level 1: Recognition of the importance of reflection)²⁹.

“I think my ability to do reflections has become a bit better and at least made me realize that I lacked, and still do not have, all the abilities to do great ones. I think this is where I grew the most, not that I became better at doing reflections, but the knowledge of the importance of my improvement to understand my behavior, actions, and feelings a bit better. I think it is visible in my result.” (Reflection level 2: Improvement in reflection skills)³⁰.

I would say that the result fits pretty well with how I feel in my personal development. I found it harder to do reflections than I thought I would before the process started, making me question my ability to perceive my goals, visions, and emotions. I think it is hard to know what you are striving for and sometimes goals are not necessary to be motivated. I have always been led by intrinsic motivation to be a great achiever regardless of the situation. Goals work more like a guiding star toward your future and I think it is important that it is possible to change goals during the period as well, making it more adaptable and flexible for environmental changes. (Reflection level 3: Meta-Reflection)³¹.

In this preliminary study, we observed that students’ competence in meta-reflection ranged from self-proclaimed “newbies” to users who reflect often and already show competence in this area. An evaluation of the reflection skills presented here across the entire distribution in a specific group, for example in percentage or in clusters, would also be an interesting outcome. Such a detailed evaluation is currently in progress.

It will be insightful to track further variables to understand this variety better, for example age, gender identity, educational background, type of study program etc. to be able to draw better comparisons and conclusions regarding the qualitative level of reflection skills. It also represents a key area in which Rflect can develop future features.

In summary, these qualitative insights are invaluable for informing Rflect’s product strategy and development, including new features and in-app content, for example making Rflect more interactive, energy tracking, integrating personal learning goals and values and much more.

Conclusion

Overall, the initial results demonstrate promising progress in students’ personal and interpersonal skills across the tested dimensions. These findings underscore the potential of interventions like Rflect in fostering holistic inner development among students, while also highlighting areas for continued exploration and refinement.

Addressing Limitations and Future Outlook

Rflect acknowledges that while these results are promising, there are predicted and accepted limitations to the approach of the preliminary self-study. They will be addressed in the upcoming impact study conducted in spring semester 2024 to ensure a more comprehensive understanding of student skill development:

1. Enhancing Methodological Rigor: In the impact study 2024, we will proceed with a quantitative approach using evidence-based scales³² curated by Dr. Prof. Anna Jasinenko, supported by a qualitative multi-stage qualitative content analysis. Results will be compared with a control group of students who will not use Rflect aside from completing the pre- and post- surveys. This adjustment ensures a higher level of methodological rigor and alignment with established academic standards.
2. Expanding Sample Size: While the data sample in this study was limited to 39 respondents completing both the pre- and post-surveys, efforts are underway to increase the sample size. At the time of writing, 200 students had already completed the pre-survey, and over 100 students had been recruited by our university partners for the control group.
3. Considering Demographic Variables: The current study did not explore variables such as age and gender identity. However, in the upcoming impact study, more comprehensive demographic data will be collected to understand potential correlations between these factors and skills development outcomes.
4. Incorporating Diverse Assessment Methods: Rflect recognizes the inherent limitations of relying solely on student self-assessment for proving true skills development. To address this, mid to long-term iterations will explore the integration of peer, lecturer, and AI-driven assessments.

By addressing these limitations and enhancing the methodology, Rflect aims to strengthen the validity and reliability of its findings, ultimately contributing to a more solid understanding of student skills development.

About Rflect

We started Rflect because we believe that inner development is key to achieving sustainable development, and that students genuinely deserve better. Simply put, we are convinced that more reflection leads to better outcomes for both people and the planet. Our mission is to enable universities to equip their students with a lifelong habit of inner development. We understand this goal is ambitious, and we take pride in that ambition. Our motivation comes from the vision of bringing inner development to 200 million tertiary students worldwide and supporting universities in redefining their pivotal role in society. Rflect is an early-stage edtech startup in Switzerland and is enabled by the Migros Pioneer Fund³³. Learn more: rflect.ch.

Imprint

This report was provided by Rflect in collaboration with academic advisors. Have questions? Get in touch: info@rflect.ch

Bibliography

Blume, D. (personal communication, May 27, 2024).

Gill, G. (2014). The nature of reflective practice and emotional intelligence in tutorial settings. Journal of Education and Learning, 3(1), 86-100. https://doi.org/10.5539/jel.v3n1p86

Livingston, J. A. (2003). Metacognition: An overview. Retrieved from
https://www.researchgate.net/profile/Jennifer-Livingston-5/publication/234755498_Metacognition_An_Overview/links/0fcfd5112609b7ced5000000/Metacognition-An-Overview.pdf (Accessed June 4, 2024)

Mayring, P. (2019). Qualitative Inhaltsanalyse. Abgrenzungen, Spielarten, Weiterentwicklungen. Retrieved from
https://methodenzentrum.ruhr-uni-bochum.de/e-learning/qualitative-auswertungsmethoden/qualitative-inhaltsanalyse/qualitative-inhaltsanalyse-nach-mayring/ (Accessed June 4, 2024)

Inner Development Goals. (2021). Retrieved from
https://innerdevelopmentgoals.org/framework/ (Accessed June 4, 2024)

Weidinger, W., & IPE. (2020). Supporting active learning: Didactic guide to personal development. Retrieved from
https://ipe-textbooks.phzh.ch/globalassets/ipe-textbooks.phzh.ch/english/personal-development-english/pd_teacher_training_en_web.pdf (Accessed June 4, 2024)

Footnotes