All About oer and Renewable assignments
.png){kind=logo}
Evaluating OER
OVERVIEW OF SECTIONS
Introduction
Once you have created an Open Educational Resource, it is essential that you ensure its effectiveness and quality. You might also be interested in measuring its usage, adoption, and impact. Just like existing OER can be evaluated for their potential adoption and adaptation, newly developed OER can also be assessed for relevant content, production quality, licensing, accessibility, and inclusion.
​
This module explores the various ways you can evaluate your self-created OER. It lists multiple evaluation tools and options to help you devise an evaluation plan suitable for your OER.
Learning Objectives
By the end of the module, you will be able to:
​
-
Assess the technical openness of your OER
-
Identify evaluation tools for OER
-
Create feedback opportunities for your OER
-
Measure usage and adoption of OER
-
Measure OER’s impact
ALMS Framework
Choosing appropriate CC licenses for your OER is critical to enabling your users to engage in the 5Rs (Retain, Revise, Remix, Reuse, Redistribute). However, what about some other technical decisions that could be affecting the openness of your content?
​
The ALMS Framework offers a way of analyzing those technological decisions and how much they enable or hinder a user's capacity to engage in the 5R activities permitted by open licenses. Using the ALMS Framework as a guide, open content publishers such as you can make technical decisions that allow as many users as possible to participate in the 5R activities. Nevertheless, it should be noted that the ALMS framework's utility varies, as it does with any framework or set of standards. The framework helps you reflect on these considerations:
Access to Editing Tools: Can the user edit the OER without the need for specialized or expensive tools?
Level of Expertise Required: Would most users be able to edit the OER at their current skill level?
Meaningfully Editable: Can all parts of the OER be edited?
Self-Sourced: Would the user be able to edit the OER directly, or is a separate editable file needed?
Thus, you can apply the framework to determine the usability and openness of your created OER. To learn more about ALMS framework, engage with the interactive presentation below:
Evaluation Tools
Some popular OER evaluation tools are available that are suitable for use in higher education. These evaluation tools can be used to evaluate a published OER or one that has been created and is ready to be released.
​
Some of these evaluation tools have rubrics while some are based on checklists. You are encouraged to modify the rubrics or checklists for your needs and purpose.
Comprehensive OER Evaluation Tool
The tool comprises eight rubrics. The smallest meaningful unit of your OER should generally be used to apply the rubrics. This could be a single activity, learning experience, or instructional resource, while it could also represent an entire unit of study or collection of resources. These scoring criteria are often applied to evaluate a specific item's potential rather than actual effectiveness in a learning setting.
Click icon to download file
Click icon to download file
Open Textbook Rubric
If you have developed an Open Textbook, you can adapt the criteria set in this rubric in conjunction with the Comprehensive OER Evaluation Tool.
Accessibility Criteria
OER's accessibility is crucial, particularly since most OER are available online. The OER Accessibility checklist will ensure you have met the basic accessibility guidelines.
Additionally, the OER Accessibility Toolkit and WAVE Web Accessibility Evaluation Tool will help ensure that you have created an open and accessible OER.
Click icon to download file
Bonus: Quick checklist for Evaluating Open Educational Resources (OER)
Click icon to download file
Gathering Feedback
As an author and creator, getting feedback on your OER will be beneficial. It will primarily help to ensure the resource's effectiveness and the content's accuracy. Feedback can be gathered from students, peers, colleagues, and experts in the field. Insights can be received not only after OER authoring but also during its development. The nature of feedback can be formal (e.g., a survey) or casual (e.g., verbal opinions).
​
Depending on the length and breadth of the OER, feedback can be sought for the entire work or parts of it, such as a chapter or two of an Open Textbook. Those who provide input can sometimes be compensated with a small honorarium.
​
​
Student feedback on OER may be sought during class or after the course is completed through course evaluations. Some instructors may prefer to share chapters or materials with students as they are being developed. Input can be invited on several aspects of OER such as the potential to achieve the learning objectives, clarity of text, use of multimedia, use of figures, images or data, and the utility of pedagogical aids like glossary terms.
Peer review process would benefit the faculty as well as departments and institutions. Peers can act like Subject Matter Experts (SMEs) who can help ensure that the OER is factually correct and technically sound. The nature of the review could be formal or informal, with some intervention from authorities if it is a formal process.
User reaction to openly published OER may also help determine the quality and effectiveness of the OER. Incorporating a feedback channel through means such as email or embedding a form in the content can be useful. A note could mention that you are open to receiving suggestions to improve your OER.
Tracking the Impact of your OER
Once you have published your OER, you may want to track its usage and adoption to evaluate its impact. There are different ways to do it.
Tracking Usage and Adoptions
​
Usage Metrics
Metrics can be sought from the repositories or databases of the resources hosted on the platform. Alternatively, you can sign up for a third-party metrics provider like Google Analytics if the platform does not offer any metrics of its own. To find out what is feasible, you must check with the platform and service.
Self-reported Adoptions
If you want to know how often your OER is being used, you can ask users to report back when they start using the material. Include the request with an email address or a short form on the website’s homepage and/or in the front matter of the work to help encourage self-reporting of adoptions and usage. While this does necessitate depending on third parties to self-report, the vast majority of the OER community is eager to do so.
Tracking Impact on a Class and Institutional Level
​
COUP Framework
The COUP framework represents cost of education, student success outcomes, patterns of usage of OER, and perceptions of OER. It is an approach the Open Education Group has developed for examining the effects of open pedagogy and open educational resources in higher education. You can use the framework to evaluate the effects and impact of your OER.
OER Adoption Impact Calculator
Using the OER Adoption Impact Calculator, you may discover several potential effects of using open educational resources rather than conventional copyrighted learning materials. On the calculator, users can modify inputs on a sliding scale to calculate their institutional situation’s impact.
Tracking Cost Savings
​
Using a formula
The sum of money that students will not have to pay by switching to your OER from a for-profit textbook is known as cost savings or cost avoidance. Using the formula, it may be calculated:
​
Cost Savings for 1 semester = (number of classes) x (average number of students per class) x (cost of book being replaced)
OR
Cost Savings for 1 semester = (number of students using the OER that semester) x (cost of book being replaced)
​
To calculate overall savings, add together the amount saved each semester.
Summary
The video below provides summary of this sub-module on evaluating your developed OER. Content of this module is also provided in other formats for download and use/reuse.
Click icon to download file
ALMS Framework
Summary
Sub-module content
Knowledge Check: Interactive Quiz
Click icon to download file
Quiz
Note: All the interactive resources on this page including presentations, infographics, diagrams, videos and quizzes are editable and reusable on Genially.
References
Coolidge, A., Doner, S., Robertson, T., & Gray, J. (2018). Accessibility toolkit 2nd edition. BCcampus. https://opentextbc.ca/accessibilitytoolkit/back-matter/appendix-checklist-for-accessibility-toolkit/
​
Concordia University Library. (n.d.). Start evaluating. Concordia University. https://www.concordia.ca/library/oer/evaluate.html
​
Elder, A. (2022, November 21). Open educational resources (OER): Evaluate oer. Iowa State University Library. Retrieved December 7, 2022, from https://instr.iastate.libguides.com/oer/evaluate
​
Meinke, W. (2018). UH OER training. University of Hawai'i Outreach College. https://pressbooks.oer.hawaii.edu/oertraining2018/chapter/determining-technical-openness/
​
OER Services. (n.d.). OER overview & userguide. Accessibility Checklist | OER Overview & Userguide. https://courses.lumenlearning.com/suny-oerguide/chapter/accessibility-checklist/
​
Open Textbook Library. (n.d.). Review rubric. https://open.umn.edu/opentextbooks/reviews/rubric
​
Prince George's Community College. (2022, August 16). Open educational resources (OER): OER evaluation guidelines. PGCC LibGuides. https://pgcc.libguides.com/oer/oerevaluation
​
Utah State University. (n.d.). WAVE web accessibility evaluation tools. WebAIM. https://wave.webaim.org/
​
Virginia Commonwealth University. (2022, April 13). Creating open educational resources: Tracking your OER's impact. Research Guides. https://guides.library.vcu.edu/create-oer/impact
​
Wiley, D. (n.d.). Defining the "Open" in open content and open educational resources. OpenContent. https://opencontent.org/definition/
​
Wiley, D. (2019). OER adoption impact calculator. Luman Learning. http://impact.lumenlearning.com/
​