DR
InterestMe
Math
The InterestMe Math system provides students with personalized Math Word Problems (MWPs) based on their interests (including familiar names and their career interest) using a simple website design. I developed, designed, and evaluated the system.
This research was independently conducted to fulfill the requirements for the degree of Doctor of Philosophy.
ROLE
UX Research
Web Design
System Evaluation
Quan/Qual Data Analysis
TOOLS
Programming tools (Python, MySQL, JavaScript, PHP, Ngrok, Vagrant, Virtual Box)
MAXQDA
Invision
APPROACH
Expectancy-Value Theory
Design Thinking
_edited_edited.png)
Problem
Many students tend to view math word problems as uninteresting, unimportant, and unnecessary when the problems are perceived to be irrelevant and disconnected from student's real-world experiences. As a result, students may become disengaged and uninterested in the subject altogether. How could I design and develop a math word problem technology that generates comprehensible problems to increase student comprehension, engagement, and interests?
Students perform 10% to 30% worse on word problems that on comparable problems in numeric format
Students struggle with MWPs as early as the 3rd grade
MWP difficulty may be influenced by limited experience, lack of motivation, and the disconnect between everyday experiences and interest
Goals
To address these challenges this work was guided by the following goals:
Provide students with math word problems that ignite their interest:
-
Ignite students' motivation for completing math tasks by presenting word problems infused with their personal interests.
Create a user-friendly system for generating understandable math word problems:
-
Develop a system that not only generates math word problems but ensures they are user-friendly and easily comprehensible.
Approach
In tackling the challenges, I strategically embraced the design thinking approach. This methodology is an non-linear process to understand users, challenge assumptions, redefine problems, and create innovative solutions to prototype and test. I identified opportunities, discovered the unique needs and expectations of learners, and fostered the development of the system through a collaborative and iterative framework. The following sections delve into the specific actions at each step of this transformative process.
Design thinking approach
Emphasize– Understand the users
In the emphasize phase, I focused on gathering information and discovering insights to understand users and their needs. I accomplished the following activities:
-
Literature Review
-
Needs Analysis:
-
User interviews with students
-
Interviews and prototype walkthrough with teachers/Subject Matter Experts (SBEs)
-
The initial step was to conduct a literature review to explore existing research, methodologies, and insights relevant to personalized content for learning technologies.
Lit Review
01.
Culturally-Relevant Pedagogy
+
02.
Expectancy-Value Theory
+
03.
Math Word Problems Generating & Rewriting Systems
Involves the experiences and cultures of students to impact "knowledge, skills, and attitudes"
Explains students' achievement-related performance based on their expectations and values associated with the tasks
Previous research which included manually created MWPs and personalized problems with limited themes (e.g., literary, movie, biology, etc.)
Through the literature review, I discovered that linking learning materials to students' interests can actively engage them and potentially enhance their performance. Drawing on the Expectancy-Value Theory (EVT), it was evident that incorporating valuable themes into problem-solving could positively influence students' math performance and spark situational interest. Consequently, I aimed to develop a system that integrates students' prior knowledge and interests for an enhanced learning experience.
Students
I initially sought to understand students' thoughts and attitudes of a math word problems and how they would use math among their career interests by conducting semi-structured interviews and a participatory drawing session. I chose participatory drawing sessions to allow students to reflect before responding with a more concise depiction and a visual representation of their perspectives on how they anticipate using math in their field. The phase involved exploring the following:
-
General opinions about math and math word problems
-
The utility of math and solving math word problems
-
Concerns and pain points related to math word problems
Exploratory Needs Analysis:
Teachers
Once the needs analysis was completed with students, we wanted to further inform the system's design by obtaining system requirements and feedback from the prototype by talking with SBEs, which in this case, were the teachers. I interviewed 5 teachers to explore their current technology usages and current practices for personalization, and conducted the walkthrough of the low-fidelity prototype. In this phase, we explored:
-
Current technology usages
-
Current practices for personalization
-
System requirements and prototype feedback via Invision
Low-Fidelity Prototype
"I don't like them as much as regular [numerical] problems. You can find keywords in [the problems], but the reading makes it more boring that what it is."
- Student Participant
"You have those teachers that are not as up on change. I think a lot of the problems were trying to fidn the software to align to what they were teaching."
- Teacher Participant
Define – Figure out the problem
The analysis of the information gathered during the emphasize phase helped to define the core problems students and teachers communicated that had with math word problems.
Students
01.
Boring/non-relevant problem content
Many students expressed that their interests are usually not considered, neither are they represented in the MWPs that they normally encounter.
02.
Ownership & Personal Interest
Students desired personal connections with the learning content and sought some form of control regarding the problem topics.
03.
Limited math utility
Most students could not identify how math played a part in their desired career, as their knowledge regarding math-related tasks were limited. They were uncertain of the math utility for careers that do not directly receive customer payment.
Teachers
01.
Teachers desired classroom management support
Several teachers desired easily accessible performance data for students to measure students' success and progress.
02.
Current technology lacked student support and did not align to their lesson plan
A few teachers suggested several adjustments, such as providing aids to the students and grouping lessons based on their state's math standards. The standards should be easily identifiable.
03.
Limited math utility
All teachers expressed that personalized content for students was important. They engaged students using interests obtained via questionnaires and everyday observations. They would appreciate a system that would automate this process.
Here are some of the key pain points and concerns identified by students and teachers that guided our research focus.
Key pain points and design goals
Personas
The coding and organization of the interview data and demographics information from the needs analysis were used to categorized observed behavior. Participants attitudes, activities, motivations, aptitudes and other data-driven variables were considered.
Student personas
Teacher persona
Ideate – Generate Ideas
Before entering this phase, I already knew a lot about the target users and their expectations for a personalized math learning system. Feedback from the low-fidelity prototype assisted with refining the system. However, on the backend, I needed to engage in brainstorming and conduct further research on the technical aspects to approach the generation of these problems.
Several ideas came to mind, including template-based math word problems, rewriting, and full-on generation of problems. I knew the math word problems of a template-based system would be limited to the information students could share with the system. Consequently, these problems would not fully represent career-based tasks and scenarios, which could negatively affect students' engagement.
As the scope excluded generating problems entirely from scratch by training neural networks, I opted for a hybrid approach. This involved providing the system with Math Word Problems (MWPs) that were used to generate new problems based on students' career interests, utilizing natural language processing tools. This technique aimed to ensure that students would encounter similarly structured problems tailored to their interests.
When considering problem rewriting, the option of utilizing a domain ontology was explored. However, this method presented additional steps for teachers ,and the available ontologies were limited. This influenced the selection of the chosen method for rewriting problems.
Prototype – Create Solutions
In this phase, I implemented the system based on feedback from teachers. I also worked on the back-end to develop the math word problem generator and the database for storing student information.
Multiple iterations were conducted to fine-tune the system. Furthermore, more teachers and colleagues actively participated in this stage, suggesting modifications that significantly influenced the system's functionality and user-friendliness.
System Overview
Building upon the design decisions made in the ideation phase, here's a brief overview of the system's functionality:
1. Students input their interest into the system
2. Using web scraping, information about their career interest is gathered from the College Grad website
3. The system utilizes a JSON file to retrieve details about the problem being rewritten, including parts of speech for the words to be replaced.
4. Leveraging the retrieved College Grad content, the JSON file, and Python's natural language processing tools, the system identifies the top words mentioned in the career results. These top words are then integrated into the MWP and tested to formulate a coherent and meaningful question.
Math word problem data from json file
Example college grad career profile information
System Overview
Example problems generated by the system
Test – Evaluate Solutions
The objective of this phase was to assess the system's usability from the users' perspective and evaluate student performance to confirm whether the system adequately fulfills its intended goals.
Method:
For the usability evaluation, 24 students engaged in various tasks within the system, completed the modified System Usability Scale (SUS), and participated in semi-structured interviews via Zoom.
Here are are few highlights from the testing phase:
Usability
The mean SUS score for all participants was rated "excellent" as suggested by an adjective rating scale (82.62)
"I don't think I've ever seen any questions about my friends before."
- Student Participant
More than half stated that the problems were about them.
All but two participants mentioned they would use the system.
Student Performance and System Evaluation
I also assessed InterestMe Math's MWP generating capabilities by measuring the readability and participants' perceived comprehension of the generated MWPs.
Additionally, I explored the system's impact on the math problem-solving accuracy and the triggered situational interest of 5th and 6th-grade students
Method:
Twenty-nine students participated in a cross-sectional study conducted via Zoom. Each student was assigned to one of two assessments, each comprising three numerical (numbers only) problems, three non-personalized math word problems, and three personalized math word problems, totaling nine problems. Participants self-reported their perceptions of each question's interest level and comprehensiveness.
Here are are few highlights from the testing phase :
There were no significant differences between students performance between traditional and personalize problems.
Students perceived the personalized Math Word Problems (MWPs) as more comprehensive than the non-personalized ones. However, they noted that the traditional problems were easier to read, primarily due to the use of simpler words.
There were no significant differences between the self-reported triggered situational interest scores of non-personalized (M = 3.631, SD = 1.239) and personalized (M = 3.821, SD = .9187 ) MWPs (t(27) = -.913, p = .369).
-
Make time to conduct pilot studies. This will help save time and enhance the overall quality and validity of the study.
-
Kids will ask for reassurance while completing task. Encourage them that there is no right or wrong way to answer and knowing their approach is more important than the outcome itself.
-
Flexibility is paramount. Challenges and occasional dead ends are inevitable. It's crucial not to take setbacks personally. Instead, regroup, maintain an open mind, and focus on exploring available options.
-
Whether conducting research with children or adults, simplify in language. Ensure study materials are not only age-appropriate but also easily understandable.