Schoology Item Analysis

Schoology Item Analysis

With the push towards mastery learning in instructional strategy, educators are constantly looking at data to find weaknesses in their instruction, struggling students, or problems with test questions (i.e. “items”).

With item analysis, educators and assessment teams can see if the problem is a badly-written or wrongly scored item by using two crucial metrics: item difficulty and item discrimination.

1. Problem

The existing item analysis tools are often spreadsheets and long tables of numbers that are hard to decipher and/or take action on. Educators need a way to assess quickly which students are struggling and why. Item discrimination and item difficulty are also not well-known terms by teachers. Currently, there are no ways to filter the data by demographics to identify struggling groups (ex: Special Ed or PHLOTE/Primary Home Language Other Than English).

2. Plan

Goals

Create a report within Schoology’s Managed Assessment Reports tool that can be quickly understood by all types of educators (district admins, assessment teams, and instructors) to identify problematic items or struggling students in order to take necessary action. All designs and functionality must be accessible.

Expected impact

Additional assessment scores will increase (as well as item discrimination scores). Users will re-teach areas that all students have missed, re-write questions, give resources to struggling students, and assign advanced material to students that are excelling.

3. Users

While Schoology users range from students to parents to educators, the Managed Assessment Reports is only useful and accessible to educators. Within educators, there are a variety of roles and permissions.

4. Schema & flow

Due to all the user roles and data points, I mapped out all the possibilities. The user flow was created from mobile screen thumbnails.

Item Analysis user flow
Schoology IA

5. Testing with wireframes

We had insightful and useful feedback and overall, the feedback was incredibly positive and excited for the future. When the got to the crucial “Item details” page (metrics about an individual question such as number of students assessed, student responses and percentages, distractor rationale, etc), we had enthused replies of, “THIS is HOT. This is what I wanted. This is why we’re going to buy.”

My product manager and I refined the definition for item discrimination and difficulty to be friendlier and give users an action.

6. Design

Landing page
Assessments
Item analysis
Item