Portrait image
Dr. Grace Hui Yang
Professor of Computer Science Georgetown University
NY Inspires · District Research Partnership
Human–AI Validation of Durable-Skills Assessment
Learning Genie and university research partners are proposing a study to examine how AI can support durable-skills assessment while keeping teachers and trained human raters as the validation standard.
Academic Research Partners
Technology & Implementation Partner
Learning Genie
Meet the Research Team
Portrait image
Portrait image
Dr. Jiao Hong
Professor of Educational Measurement Director, Maryland Assessment Research Center University of Maryland, College Park
Portrait image
Dr. Gene Shi
CEO Learning Genie
Why This Research Matters
NY Inspires is moving districts toward a competency-based system that uses multiple sources of evidence—including student work, performance-based experiences, educator observations, classroom assessments, and statewide assessments—to understand student readiness.
How can schools evaluate critical thinking, problem solving, communication, collaboration, and reflection consistently without creating an unsustainable scoring and calibration burden for teachers?
What We Are Studying
The study compares AI-generated rubric scores with trained K–12 educator ratings to explore how AI can scale high-quality performance assessment without replacing teacher judgment.
- How closely AI scoring agrees with trained human raters.
- Where teachers and AI disagree — and why.
- Which types of student evidence can be evaluated reliably by AI, and which still require human judgment.
- Whether evidence from projects, presentations, portfolios, and career-connected learning can support consistent assessment of durable skills.
- Whether carefully designed AI follow-up questions can surface student thinking not evident in the final product—without leading the student or changing what is being measured.
Safeguards
- No AI score will determine a student's grade, placement, graduation status, or other high-stakes decision during the research study.
- Student information will be handled under applicable district privacy requirements, FERPA, research agreements, and institutional review procedures. Student artifacts will be de-identified where appropriate.
- Teacher and student results will not be used to rank individual teachers, classrooms, or schools.
- Any later phase involving direct student interaction with an AI elicitation agent would require separate district approval and appropriate research and privacy review.
What Participating Districts Receive
Participating districts receive research findings, calibration insights, and practical recommendations for strengthening competency-based assessment.
An aggregate human–AI reliability and calibration analysis.
Findings identifying which competencies and artifact types are most—and least—amenable to AI-assisted evaluation.
Recommendations for improving local competency rubrics and performance-assessment practices.
Early research insight relevant to implementation of Portrait of a Graduate, project-based learning, capstones, CTE, and career-connected learning.
Opportunities for participating educators to contribute practitioner expertise to a rigorous university research study.