Six Acceptance Tests for an AI Tutor Built on Licensed Educational Content
An education group piloting an AI tutor usually asks whether AI tutoring works. That question cannot be signed. A contract needs something narrower: does this assistant, on this licensed corpus, for these learners, pass tests agreed before anyone looks at the results?
The evidence shows why the distinction matters. In a randomized field experiment with nearly 1,000 high school mathematics students in Turkey, an unguarded GPT interface improved practice performance, then left students 17% below the control group on the unassisted exam. A guarded version, built on teacher-provided solutions and giving hints instead of answers, largely avoided that decline.
In short
Licensed content gives an AI tutor a defined corpus and a contractual basis. It does not prove that the assistant retrieves the right passage, teaches at the right level, stays inside the licence, protects student data or leaves the teacher in control.
Six acceptance tests turn those claims into a pass/fail decision: source traceability, pedagogical fit, unaided learning, rights enforcement, data minimisation and teacher control. Two of them, traceability and rights enforcement, also produce the usage records educational publishers need for reporting, renewal and pricing. None of the six should be averaged into a single score.
Licensed content is a starting point, not a pass
Educational publishers are licensing textbooks, exercise banks and teacher guides into AI assistants. Buyers often read the licence as a quality guarantee. It is a scope: which editions, which uses, which users, for how long.
Whether the assistant respects that scope, and whether students learn from it, are separate questions. Each test should name the population, subject, curriculum, language, use case, failure severity and threshold before results are reviewed.
Test 1: Can every material claim be traced to the licensed source?
Build the test set from the exact editions and versions covered by the licence. Include routine questions, ambiguous ones, plausible distractors, conflicts between editions, and questions the corpus cannot answer.
The measurement method already exists. A human audit of four generative search engines found that 51.5% of generated sentences were fully supported by their citations, and that 74.5% of citations supported the sentence they were attached to. That was not an education benchmark, but its citation recall and precision measures transfer directly to a licensed-content assistant.
Subject experts split each answer into claims and map every material claim to its supporting passage. Report citation precision and recall, unsupported claims, retrieval failures and contradictions separately. Automated scoring such as RAGAS can screen the sample; it should not replace expert review, the same discipline required when verifying that a cited source actually supports the claim.
A defensible baseline is zero unsupported claims that could change a learner's understanding, plus a local threshold for minor failures. When the corpus has no answer, the assistant must say so rather than fill the gap from model memory. A passage that was retrieved but never cited fails this test as surely as an invented one.
Test 2: Does the explanation fit the learner and the learning objective?
A correct answer can still be unteachable. It may assume knowledge the learner lacks, use vocabulary above the intended level, or give away the answer when the activity was designed to elicit reasoning.
Teachers should define the rubric before the pilot: prerequisite knowledge, vocabulary, conceptual accuracy, cognitive demand, hint size, handling of common misconceptions and curriculum alignment. A blinded panel then rates responses against teacher-approved examples, and reports agreement between raters instead of hiding it in an average.
Google's LearnLM team combined automatic, quantitative, qualitative and human assessments against learning-science principles. An education group can borrow that structure without borrowing its scores. UNESCO also calls for age-appropriate pedagogical validation, so a small classroom study should follow the rubric review: what looks appropriate to adults may still fail with the students who will use it.
Test 3: Can students reason without the assistant?
Measure learning after the tool is removed. Use a randomized or crossover design where feasible, then give students new, curriculum-aligned problems without AI support after a delay, and score the reasoning steps as well as the final answer.
The Turkish study makes this test concrete, but its scope matters: mathematics, grades 9 to 11, one school. Its 17% relative decline is not a universal forecast. It is evidence that every pilot should measure what students can do once the assistant is gone.
The product passes only if unaided transfer meets the pre-agreed standard, with no sign that apparent progress disappears without the tool.
Test 4: Can the system prove that each content use is permitted?
Turn the licence into a rights matrix tied to each content object and version. Record who may access it, which actions are permitted or prohibited, which attribution or reporting duties apply, and when the permission expires.
This is becoming a product requirement. In 2026, Copyright Clearance Center added internal-use AI reuse rights to its US higher education licence and announced transactional, pay-per-use options for specific AI uses. The type of use now matters alongside access to the work, and a permission defines what may happen, not what did happen.
W3C's ODRL Information Model can express permissions, prohibitions, duties, assets, parties and constraints in machine-readable form. It does not interpret the contract: legal and rights teams still decide how the agreement maps to policy, starting with who may bring which content into which AI system.
Run allowed and denied prompts, expired entitlements, cross-tenant requests, excessive quotation attempts and requests outside the licensed corpus. The assistant should block or transform disallowed uses, preserve required attribution and log which policy it applied. A log that names the policy is an audit trail, not a disclosure; any unauthorised access or output is a failed critical test, not a defect to average away.
Test 5: Can it minimise and protect student data?
Map every data flow before students enter the pilot: account data, prompts, responses, inferred ability level, teacher notes, telemetry, support logs, retention, subprocessors, model-improvement use, deletion and export. Where processing may create a high risk, run a data protection impact assessment with the data protection officer.
Then test the controls. Create and delete accounts, verify retention, inspect role-based access, attempt cross-user retrieval, submit unnecessary personal data and check whether prompts are reused for service improvement. Libraries apply the same logic in a ten-item vendor review on logging, retention and deletion.
The rules depend on jurisdiction and age. In the United States, the FTC's amended COPPA Rule treats disclosure of a child's personal information to a third party for AI training as non-integral to the service, so it requires separate verifiable parental consent, and it prohibits indefinite retention. In France, CNIL guidance for schools and the national education AI framework point the same way: avoid individual student accounts where possible, disable conversation history and reuse for service improvement, and limit tracking. The pass condition is verified behaviour, not a privacy-policy promise.
Test 6: Can the teacher inspect, constrain and override the assistant?
Give teachers realistic tasks rather than a feature tour. Can they restrict the corpus, choose between hints and answers, inspect the supporting passage, correct or withdraw a response, review logs and suspend the tool for a class or a student?
Teacher control has an empirical model. In the Tutor CoPilot randomized trial, students whose human tutors received AI-generated guidance were four percentage points more likely to master mathematics topics, and nine points more for students of lower-rated tutors. The system advised the tutor rather than replacing the tutor's judgment; that is evidence about one human-AI design, not proof that every teacher dashboard improves learning.
Measure task completion, time to intervene, errors and the need for vendor support. The NIST AI Risk Management Framework calls for defined, documented human oversight. Article 14 of the EU AI Act sets oversight requirements for high-risk systems, and Annex III covers specified educational uses such as admission, assignment, assessment and monitoring during tests, not every educational assistant. Regulation (EU) 2026/1744 moved the application of those Annex III requirements to 2 December 2027, which shifts the deadline, not the evidence question the AI Act raises for licensed content.
An override that is hard to find, or impossible to use during a lesson, is not meaningful control.
What the acceptance report gives the publisher
Tests 1 and 4 produce something the other four do not: per-answer records of which passage, from which licensed edition, supported which claim, under which policy. For the educational publisher whose content sits inside the assistant, that is the closest thing to a usage record the deployment will generate.
Publishers should ask for it in the licence, aggregated by title, edition and chapter, including denied requests and out-of-scope attempts. Without it, renewal rests on figures the vendor declares, and the question of who audits the number stays open.
The objection: "This is too heavy for a pilot"
Six tests, expert panels and crossover designs exceed what a mid-sized school group can fund. The objection is fair, but cost scales with severity. Tests 4 and 5 are largely scripted and binary; Test 3 can start as a delayed, unaided quiz on a sample.
What cannot be cut is setting thresholds before results arrive. The objection holds for one narrow case: a teacher-only assistant with no student accounts, where Test 3 and much of Test 5 shrink accordingly.
The procurement decision
Run the six tests against the named use case, not the product in the abstract. Publish the test-set design, content versions, rubrics, thresholds, severe failures, mitigations and retest date in the acceptance report.
Do not collapse the result into one weighted score. Strong citation performance cannot compensate for a rights breach, a privacy failure, or evidence that students stop reasoning when the tool disappears. Some failures block deployment; others justify a narrower use case, a guarded mode or another pilot.
The useful question is not whether AI tutoring works in general. It is whether this assistant, on this content, for these learners and teachers, has passed the tests this deployment requires. Citations Logic works on the record side of that answer: AI usage evidence that shows what licensed content actually did inside an AI system.
Book an AI usage evidence assessment
Continue the evidence chain
AI Reuse Rights Need Usage Signals: The CCC Higher Education Lesson
ALA's AI Vendor Checklist Turns Procurement Into an Audit
Usage Reporting in AI Licensing Deals: Who Audits the Number?
Sources
Bastani et al. — "Generative AI without guardrails can harm learning: Evidence from high school mathematics," PNAS, 2025
Generative AI without guardrails can harm learning
Liu, Zhang and Liang — "Evaluating Verifiability in Generative Search Engines," 2023
Evaluating Verifiability in Generative Search Engines
Es et al. — "RAGAS: Automated Evaluation of Retrieval Augmented Generation," 2023
RAGAS
LearnLM Team, Google — "LearnLM: Improving Gemini for Learning," 2024
LearnLM: Improving Gemini for Learning
UNESCO — "Guidance for generative AI in education and research," 2023
UNESCO guidance for generative AI in education and research
Wang et al. — "Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise," 2024
Tutor CoPilot
Copyright Clearance Center — "CCC Expands Annual Copyright License for Higher Education to Include Internal-Use AI Reuse Rights"
CCC higher education AI reuse rights
Copyright Clearance Center — "CCC Launches New AI Re-Use Rights & Transactional Licensing Capabilities for AI"
CCC AI transactional licensing
W3C — "ODRL Information Model 2.2"
ODRL Information Model
Federal Trade Commission — "FTC Finalizes Changes to Children's Privacy Rule," January 2025
FTC amended COPPA Rule
CNIL — "Deux FAQ sur l'utilisation des systèmes d'IA dans les établissements scolaires"
CNIL FAQs on AI in schools
NIST — "AI Risk Management Framework"
NIST AI RMF
Regulation (EU) 2024/1689 (AI Act), Article 14 and Annex III
EU AI Act on EUR-Lex
Regulation (EU) 2026/1744 (Digital Omnibus on AI), in force 27 July 2026
Regulation (EU) 2026/1744 on EUR-Lex