ANATOMY OF A DISCOTEST®

Introduction

Courtesy of the Annenburg Foundation At first glance, DiscoTests look like nothing more than high quality formative assessmentsThese are assessments that can be embedded in curricula to support learning and provide information about student understanding.. They are. But that’s just what’s on the surface. Underneath is a new approach to assessment—one that integrates insights from the learning sciences, psychometrics, teaching, and educational philosophy. DiscoTests are the elements of a new model for educational assessment that begins with the proposition that the learning interests of individual students should be at the center of all decisions about how to structure education, including educational assessment.

In order to build assessments that benefit every student, we're building a new kind of standardized assessment infrastructure—one in which every assessment event provides a learning experience for students, equips teachers with diagnosticsDiscoTest diagnostics include (1) detailed information about student understanding and how it relates to curriculum targets, and (2) instructional resources that help teachers meet the specific learning needs of their students. for making evidence-based instructional decisions, and provides policy makers with information that can be employed to support teacher and curriculum development. Every DiscoTest includes the assessment itself (called a Teaser) a set of low-inference rubricsDiscoTest's low-inference rubrics are based on extensive ongoing research into how students learn particular concepts over time. All Teasers are associated with a set of rubrics, each of which focuses upon a particular theme or construct. Each selection in a rubric represents a developmental (Lectical®) phase. By making selections on several rubrics, teachers narrow in on the overall Lectical phase of a given performance. , diagnostic reports, and instructional resources that are aligned with assessment outcomes and curricular objectives.

Because they are composed of open-ended questions that require thoughtful written responses, and include evidence-based diagnostics and resources, DiscoTests are ideal for embedding in virtuous cycles of learning—cycles of knowledge seeking, application, and analysis, illustrated below. (View a 20 minute video about our learning model here.)

virtuous cycles of learning

And because all DiscoTests are standardized and calibrated to a single common core developmental metricAll DiscoTests are calibrated to a single learning ruler (common core scale) called the Lectical® scale. This makes it easy to follow student progress, and to compare learning trajectories across multiple subjects. A score on the Lectical scale not only shows what a student understands today and how it compares to what she understood in the past, but it also points to what she is likely to benefit from learning next. , they are also ideal for large-scale evaluations of educational programs. In other words, DiscoTests are built to serve both formative and summative functions, ensuring that every moment devoted to testing is also a moment devoted to learning. Click here to learn more about our mission.

Teasers

The DiscoTest learning cycle begins with a Teaser. At the core of all Teasers are one or more real-world problems or scenarios, focused on a specific set of skills or concepts. These are followed by a series of questions that require students to connect ideas, think through problems, and communicate their reasoning.

poe001 question 3We think of Teasers as “tests teachers want to teach to” because preparing students to do well on a Teaser requires helping them to (a) build a thorough understanding of targeted skills and concepts, (b) develop learning and reasoning skills, and (c) practice written communication skills.

Each Teaser is calibrated to a developmental scale known as the Lectical® scale, which is modeled on Dr. Kurt Fischer's dynamic skill scaleDr. Kurt Fischer of Harvard's Graduate School of Education established a developmental theory called Dynamic Skill Theory during the latter half of the last century. One of the components of his model are developmental levels. He calls these levels dynamic skill levels. We decided to work with this model because we think it is the most useful developmental model in existence.. The scale measures levels of hierarchical complexityEach level in the dynamic skill scale represents a new level of hierarchical complexity, which can be thought of as a new way of thinking. To learn more, see our interactive presentation about DiscoTest. inherent in the structure of concepts. Fischer's work integrates over 100 years of research in developmental psychology. To calibrate a Teaser to the Lectical® scale, we first study how students learn the concepts and skills targeted by the particular Teaser. The research process provides unprecedented insights into the various learning pathways through which students master particular skills and concepts. These important insights are then used to create rubrics, populate reports, and design learning resources.

Teasers are essentially educational tools. Teachers, students, and parents can use them to supplement instruction, to find out what students understand before instruction begins, and to find out how much students’ understanding has changed as a result of instruction.

Formative features of Teasers
  1. They require students to reflect upon and apply what they are learning;
  2. can be supplemented with coding done by teachers and students, using low-inference rubrics that are designed to reinforce learning;
  3. are accompanied by reports that (a) describe how a student performing at a particular level is likely to think about a given topic, and (b) suggest appropriate learning resources and activities; and
  4. are made to be embedded in any standard curricula.
Summative features of Teasers
  1. They are standardized and calibrated to a single common core developmental metric—the Lectical® scale;
  2. they are valid, reliable, and standardized; and
  3. they are aligned to Common Core Standards and National subject area standards.

Coding rubrics

scoring rubric exampleDiscoTests are scored for their Lectical Level with CLAS, our Computer Assisted Lectical Assessment System. Teachers and students can supplement feedback in assessment reports by using mechanics, coherence, and salience scales. For example, the salience scales focus on the degree to which students address the themes targeted in a given teaser. The salience scales for reflective judgment (critical thinking) focus on several themes, including the nature of evidence, evaluating evidence, inquiry, persuasion, conflict resolution, and deliberation. Salience scales are presented as sliders on online coding pages. Each slider is associated with a theme. Teachers (or students) use each slider to estimate the degree to which each of its associated themes was addressed in a student's responses. These selections are analyzed in real time to calculate scores that are used to further customize the student's report. For example, a student who spent considerable time discussing evidence and inquiry without referring to the quality of evidence, would be provided with resources designed to increase their awareness and understanding of approaches to evaluating evidence. Teachers may also enter personal feedback, which is included on student reports.

Coding pages feature:
  1. coding instructions;
  2. student responses displayed by question;
  3. writing mechanics sliders;
  4. logical coherence sliders;
  5. salience sliders;
  6. teacher comment fields (for teachers’ personal feedback to students).

DiscoTest report card

A DiscoTest Report Card shows student growth over time in a range of subjects. Here you can see the power of the common core developmental metric in action. All Teasers are scored on the same scale, making it easy to compare progress across subjects. For example, anyone looking at the report card below can see that this student tends to perform at a higher level in physical science than in social studies, and that growth in social studies is more uneven than growth in physical science.

report card

The DiscoTest Report Card also works to link the big picture with individual performances. Points on the interactive graph represent assessment events. Each point of the graph is a linked button. Users can click on these buttons to view summaries of Teaser results or to jump to a specific report for more detailed information about a particular performance. (Click on the figure above to interact with a DiscoTest report card.)

The interactive DiscoTest Report Card was designed to provide a meaningful way for teachers, parents, and students to engage in learning. Teachers can gain a broader perspective on their students’ academic development. Parents gain new avenues for staying involved with their children’s education. Students gain a more holistic sense of their own development over time, and can use their climb up the Lectical® scale to motivate their future growth as learners.

DiscoTest Report Cards feature:
  1. a graph displaying growth over time for each subject assessed;
  2. linked buttons on the graph that open pop-up windows with summaries of results; and
  3. links to full student reports.

Interactive student reports

Individual student reports are provided for each scored Teaser. Each report explains what a score means in terms of the content covered by a given Teaser. The reports provide personalized feedback on the student’s level of skill in the areas assessed. They also include suggestions for growth and provide Quick-Pick Activities that can be done at home or in the classroom to support understanding. Every report includes student responses, so they can review their work in reference to the feedback given in the report. (Note: The link provided here will take you to an older version of the student report. Several features have been updated.)

student report

Student reports feature:
  1. explanations of students’ performance levels and understanding of key concepts;
  2. analyses of students’ use of targeted concepts;
  3. potential strengths and recommendations for growth;
  4. argumentation scalesMany of our assessments include an optional set of argumentation rating scales that can be used to support the development of writing mechanics and coherence. We suggest that both instructors and test-takers use these scales to evaluate test-takers' written responses, and that instructors use the results to help test-takers develop skills for making accurate self appraisals of their own writing.;
  5. personal comments from teachers;
  6. any comments the student entered at the time of the assessment; and
  7. learning suggestions with links to appropriate Quick-Pick Activities.

 

Quick-pick activities

Quick-Pick Activities are enrichment resources that are tailored for a student’s current level of performance. They are included in Teaser reports, DiscoTest report cards, lesson plans, and resource menus. We choose Quick-Pick Activities with the aim of deepening students’ current understandings and helping them develop significant insights.

Quick-Pick Activity suggestion (example from an individual student report – POE001 Teaser)

-----Quick-Pick-----

Explore the relation between force and energy in the Energy Skate Park. Ask yourself, "What are the factors involved in creating a great skate park?" Then, relate your observations to the principles and math behind skate parks.

Quick-Pick Activity suggestion (example from a DiscoTest report card hot button)

-----Quick-Pick-----

Engineers must use mathematics to predict how the energy systems they design will behave. Now that you have a pretty solid understanding of how force, mass, and energy work together, it’s time to learn to make more precise predictions. Use these formulas to make predictions about the impact velocity of a dropped ball when you change settings for total energy, mass, and “absorbed” energy. If this is too easy, check out these more complex problems.

Quick-Pick Activities feature:
  1. quick experiments or other activities students can do at home with parents or friends;
  2. questions for independent research;
  3. links to interesting and informative video clips or activities related to a given topic;
  4. shortened variations of class activities; and
  5. topic suggestions for relevant discussions.

 

Teacher reports

A DiscoTest is not complete until it is associated with a wealth of resources for teachers. Beyond Teasers, DiscoTest Report Cards, student reports, and Quick-Pick Activities, teachers will have access to class summary reports, general information resources, and customizable lesson planning pages. We are building a rich collection of connected resources that will provide easy-to-use tools aimed at supporting clear links among research, DiscoTest assessment outcomes, and instructional practices.

Teacher reports provide additional details about individual performances, and summaries of group performances on Teasers. For example, they display the distribution of performances for a particular test time or progress over multiple test times, and they are linked to learning resources that can be customized to serve the learning needs of a particular student or classroom.

Teacher reports feature:
  1. descriptions of learning sequences;
  2. graphic representations of the distribution of student performance levels in each class or group of classes;
  3. links to learning materials and lesson planning tools that are appropriate for students performing in different zones;
  4. the option to display data for multiple test times;
  5. the ability to view classroom performance on a heat map (below) that shows the distribution of student performances relative to descriptions of the specific levels of understanding exhibited by students. The figure below shows the distribution of students’ levels of understanding for several constructs. Cells with different colored backgrounds indicate the density of student performances in different phases. Click on the image to view live heat maps. Click here to learn more about teaching with heat maps.

heat map

The process of coding teasers, reading reports, and working with lesson plans supports teachers’ development by enriching their domain and pedagogical knowledge and increasing their understanding of student learning. Because DiscoTests support teacher development so many ways, we will ask schools, districts, and states to provide teachers with continuing education credits every time they code a Teaser.

 

General learning resources

Courtesy of the Annenburg Foundation Our general training and curricular resources use common language to explain what the data show about conceptual development in specific skill areas and to show how that information can be used in relation to current best practices in pedagogy. DiscoTest users will be able to access these resources from several locations on the site, including the DiscoTest Report Card, the student and teacher reports, the lesson planning pages, and the main resources menu.

We also provide free summer internships in which educators can work with us as research coders. Research coders participate in building assessments by using low-inference rubrics to document the content in student performances. Not only do they help us calibrate CLAS (our Computer Assisted Scoring System) and determine how students learn the skills targeted in a new assessment, they build a deep understanding of student learning that most teachers gain only after many years of practice.

General resources feature
  1. the coder training and calibration system;
  2. graphic representations of concept development for each skill area assessed (learning sequences);
  3. general guidelines on how to use information from DiscoTest reports to plan instruction;
  4. explanations of how Common Core Standards and National subject area standards align with DiscoTest learning materials;
  5. anticipated instructional challenges and considerations for students performing at different levels;
  6. suggestions on how particular teaching strategies can be employed at each Lectical® level (e.g., metacognitive strategies);
  7. suggestions for organizing and managing small collaborative group work to support productive discussions;
  8. suggestions for using DiscoTests resources to differentiate instruction; and
  9. video clips of exemplary instructional strategies in action.

 

DiscoTest lesson planning pages

Courtesy of the Annenburg Foundation Our lesson planning pages will allow teachers to tailor classroom-tested core lessons to fit the specific needs and developmental profiles of their students. Based on a teacher’s preferences and classroom profiles, customized lesson plans will be assembled on the fly, with links to additional materials and resources. Teachers will be able to save customized lesson plans for future use.

Lesson planning pages will feature
  1. customizable, ready-to-use core lesson plans;
  2. information on how lessons align with Common Core State Standards and National subject area standards;
  3. links to lesson materials and pertinent general resources; and
  4. links to various supplementary materials (e.g. materials for struggling readers or English language learners, additional homework options, extension activities).

 

DiscoTest Teacher Forum interactive space

Courtesy of the Annenburg Foundation As teachers and schools adopt DiscoTests we will, with their input, begin building a strong, collaborative community of educators. We envision a rich interactive space in which teachers can build knowledge and proficiency by working with others who are engaged in using DiscoTests. Teachers will be able to participate online in a variety of ways as they build mastery for using DiscoTests to inform their daily classroom practices. They will be able to dialogue with other teachers, make suggestions about coding rubrics and existing learning resources, and submit lesson plans that they have developed for consideration. As the Teacher Forum online community grows, we will develop a scaffolded support system, with more experienced users participating as active mentors.

The Teacher Forum will feature
  1. discussion threads;
  2. live topic talks;
  3. user profiles and teaching biographies;
  4. Featured Practitioner;
  5. Quick-Pick activities & lesson plan submission area;
  6. lesson and Teaser feedback;
  7. scheduled Q/A sessions;
  8. video clips of lessons with commentary or interviews; and
  9. a point system that will allow teachers to earn recognition by providing feedback, submitting quality lessons, and participating in various ways.

 

Learn more

Click these links to see Teasers and reports for yourself.