North West eLearning Presentation

This is the presentation I am doing for the North West eLearning Conference (Thurs. 4:00 PM, Olympia Room). Great conference so far – as usual! All resources are linked. Email me at if you have any questions or would like to continue the conversation.

This presentation was based on a series of postings on quality assurance rubrics for online classes on this blog.

Posted in assessment | Tagged , , , , | Leave a comment

How to Close a Commons with an Open License

There are some interesting paradoxes that arise from alternatives to copyright. I have seen a big push to license articles and textbooks with a CC-by license (free to use if you attribute the author) and to avoid the CC-by-nc (free to use with attribution and for non-commercial purposes). The licenses available at Creative Commons have their use. They are all equally important because each one addresses a particular purpose. I think that there are times to use CC-by and a time to use CC-by-nc. But there are still problems with just CC-by, even though I am a big fan of that and CC-by-sa (Share alike).

The motivation to work with a textbook should be that it is the best one available with an open license, not that it is the one textbook you will be free to one day make a profit from. I would like to see the proliferation of good work because it is good work: not the proliferation of a work because it is licensed in such a way that a private corporation can profit from it. There are too many companies locking down content in ways that make them less accessible to the public.

One company started out by promoting open textbooks and the Creative Commons. Next, they created and charged for a platform to deliver that content. Then they needed to expand their business and make a profit for their shareholders. That is when the problems came in – “sustainabilty.” The open textbooks were Creative Commons licensed as CC-by (you only have to attribute the work to the author). In other words, they could not profit from those books unless they pulled that license and that is exactly what they did. Fortunately, managed to pull down and re-host the textbooks with the truly open license and that other company no longer calls itself an open textbook platform.

I voiced concerns about this when I was on an OER board in California about 7 or 8 years ago, and what I found was that it was very difficult to talk to members of the OER community about the problems with profit – I came across as “anti-business” and was actually called a “Socialist” <gasp!> in a public meeting. I am not anti-business or even anti-profit – but I am vehemently for open access to information and, unfortunately, money can be used to guide, promote and eventually kill a work. This is what happens to commercial textbooks! Have we not learned anything about the limitations of the commercial market?

I understand the importance of the different licenses but it is ironic that the most open license can lead to the shutting down of public access to supposedly open work. David Bollier wrote in his posting, “The Commons, Short and Sweet” that copyright “…privatizes and commodifies resources that belong to a community or to everyone, and dismantles a commons-based culture (egalitarian co-production and co-governance) with a market order (money-based producer/consumer relationships and hierarchies).” And yet this is exactly what can happen with the CC-by license.

There are arguments against the NC license that warn that if I use that license, corporations won’t be able to put my work on a DVD and distribute it with their magazine. Well so what? Maybe I don’t want my work distributed with their magazine. The irony of the argument is that my digital work (which I create for free) will not be available to prop up a dying industry! I can’t think of one magazine that bundles DVDs with their product that wouldn’t sue me if I just started posting their articles and materials on my blog. My argument is not against the magazines or capitalism, but it is against locking my content down in their business model.

Michael W. Carrol makes the point that “Granting readers full reuse rights unleashes the full range of human creativity to translate, combine, analyze, adapt, and preserve the scientific record, whereas traditional copyright arrangements in scientific publishing increasingly are inhibiting scholarly communication.” But the problem with that is that scientific publications are so expensive right now that many libraries can’t afford to subscribe to them. How does giving the work to a commercial corporation for free solve that problem?

Aaron Wolf has an argument against  the non commercial license that goes “I doubt a network TV show, for example, would use a song licensed as CC BY-SA (other than by paying the creator under a regular copyright license), because they wouldn’t want to license the whole show under Creative Commons.” Who says that they would have to license the whole show that way? Where has that been litigated? Are there no other interpretations of the license? Who gets to decide?

Saying that something is not truly open because it is not open to economic exploitation is ludicrous in all the Latinate meanings of the word. If we value education, we need to encourage our governments to support and invest OER in the same way that they support the military or industry. Openness includes access – if we focus our resources on working with corporations for our OER, then we risk seeing the materials that are not as “sustainable” (that is, profitable) withering away. We also risk having the openly licensed materials getting locked up behind “platforms” and “services.” And what happens when those corporations go away? Or get a new board not as hip to openness as the current one? I appreciate the efforts at the California State University system and the community colleges in creating and promoting OER and Open Textbooks but I would like to see more of a national/international effort that values access over profit.

Posted in OER | Tagged , , , , | Leave a comment

Wrapping Up The Rubrics

I started this series with a general overview of rubrics in general with the posting  Evaluating Online Courses: a prelude. I learned a lot reviewing the seven rubrics below for online courses:

  1. CA Community College’s Online Education Initiative Rubric
  2. Quality Matters
  3. Illinois Online Network’s Quality Online Course Initiative
  4. Online Course Evaluation Project
  5. CMU’s Quality Assurance Checklist
  6. OCAT and Peer Assessment Process
  7. CSU’s Quality Online Learning & Teaching Rubric

Looking at these rubrics was a nice walk through the recent history of online learning. Also, each rubric or checklist tells us how each institution defines online learning. I still have a lot of questions that I will need to answer like:

  • How do we define an online course?
  • How does that definition inform the review process?
  • Who should review or assess an online course? In house or out?

These are all questions that institutions should ask as they formulate an implementation process. It is a good time to look at strengths and challenges of existing courses and programs. It is also an opportunity to see what resources and challenges such a process would require or bring.

There is a temptation when creating online course rubrics to attempt to assess everything related to the online class, including things like ADA 508 compliance that should be referred to experts. But this also points out an opportunity: a rubric can be used to help faculty inventory the services and resources on a campus that are there to support teaching and learning.

Photo of hands at the keyboard.

The process should be updated every year or two because of the rapid changes in technology. Concerns that were prevalent ten years ago may not be a concern today. Also, just like choosing (or not choosing) a learning management system or a content management system, the process should reflect the needs of the whole community. If you have a population of students with special needs (i.e. developmental students or first generation college goers), then access to the resources to insure student success should be assessed in online courses.

Why Open Source?
Each institution or community is unique enough, has its own population of faculty, students, and staff; its own access to resources (as rich or limited as that may be); its own challenges to warrant taking the time to work out a process that addresses those needs. This is why it is so important that this work be open source – artifacts created by this process should be openly-licensed. Copyrighting a process is like saying that all institutions are alike and have the same academic and student culture. For a while, all of the Quality Matters training material, research, and copies of the rubric were either not available or harder to find after they chose the copywritten path after fiscal year 2007 marked the end of their grant. This closes door on future development for the sake of “sustainability.” I appreciate the open license on the CSU’s QOLT rubric because if I make changes to solve local problems, then the CSU can get a copy of it and when they encounter similar situations, we will have already done the hard work for them.

A Meta-Rubric Example
Much like the rubrics used to evaluate software – I have been thinking about a meta-rubric for reviewing online course assessment rubrics. My example includes criteria that are common in the education technology world: ease of use, cost, and validity of scoring. And then I have included criteria that would be useful for a few of the community colleges that I have worked for, as well as some of my own values: openness and creating community. My example rubric is meant to start a conversation about what is important for your institution.

This is my “meta-rubric” example. Please feel free to copy, share, or comment.


Posted in assessment | Tagged , , , , , , | Leave a comment

Why Open Matters

Picture of an open door.For myself, “open” covers open pedagogy, open education resources, and open textbooks. I think open education is one of the most significant things to hit education in 500 years – probably since the invention or the wide-spread use of cheap paper. Open education is all about increasing the access to education and overcoming the barriers of geography, culture, and economics.

I currently teach Adult Basic Education. Since the 90s, I have been involved in teaching and education support (tutoring, writing labs, etc.) and have taught developmental English.  Open education or open pedagogy allows me to facilitate my students learning in such a way that they become independent learners. The syllabus becomes a negotiated contract, and the students from the very beginning of class, are brought into the process of mapping out where they are going to go though out the quarter. ABE students are taking courses for a number of reasons: some need to pass the GED, some are looking for HS or college credit to move on to a 2 year degree, some take classes as a step in finding their path. An open pedagogy allows me to help students understand that they have choices and that they are in charge of their learning. They are not just in a “class” but in a network of learners, creators, and professionals who will help them learn far beyond what we are doing in the physical building. Many of these students come from backgrounds and institutions where their own agency has not been respected or fostered. An open pedagogy allows us to teach students how to learn – how to make the critical connections that will allow them to be successful in college, their personal and professional lives.

Open education resources (OER) include open textbooks. ABE students are often the most vulnerable of the student population. Using peer-reviewed, open education resources or open textbooks allows me to lower the cost of education for students who are often at the mercy of their financial aide situation or lack there of. Open textbooks allow me to reuse and remix materials according to the needs of the students rather than hoping that we have found a textbook will actually work for them. I also am using open tools like WordPress.

The Pacific Northwest is growing fast. There are a 1000 people a month moving to Seattle each month. This growth is increasing rents and the cost of living, and so I have a moral obligation to use an open textbook (high quality, peer-reviewed that meets the course outcomes).

For a few years I was not teaching but working in the admin side of education as an instructional designer and director of academic technology. My previous teaching (and instructional design) was heavily influenced by George Siemens and Stephen Downes’ MOOC, Connectivism and Connective Knowledge, and Cormier’s Rhizomatic Education: Curriculum as Community.

Posted in education | Tagged , , | 2 Comments

Introduction to Open Education

I am taking a course this Fall on open education from George Siemens and David Wiley. I am looking forward to it because I have followed George and David’s work pretty closely for a number of years, and it will give me a chance to network with other educators interested in open education.

I find working with other faculty inspirational – I have some experience with open education but I always learn something new, useful, or interesting working with other teachers.

The course starts on Oct. 1st but it is an open course so I don’t think they care about late students 🙂

This posting is a test posting for the Learner Activity Hub.


Posted in Open education | Tagged | Leave a comment

Course Rubrics: CSU’s QOLT

I have used the California State University’s Quality Online Learning and Teaching (QOLT) rubric to evaluate courses as well as its ancestor, the CSU Chico rubric. This is the two-page version of the QOLT rubric. I have seen this rubric used very effectively for course design and for the review of existing courses. It is not, however, without its issues. It is well worth exploring their website to see how quality assurance for online courses is being implemented on a state-wide level.

What Problems Does This Solve?
According to the CSU website: “The Quality Learning and Teaching (QLT) program was developed to assist faculty, faculty developers, and instructional designers to more effectively design and deliver online, blended, and flipped courses. The QLT evaluation instrument, containing 9 sections with 53 objectives, provides guidance and feedback to instructors. In addition, QLT includes an optional section on Mobile Platform Readiness (4 objectives). Each of the sections has a built-in rubric that provides feedback based on the instructor’s formative score.”

This is a good use of these rubrics – faculty new to online learning can use these rubrics to get an idea about how online classes work. Faculty who are new to instructional design can get an idea of the elements required for a successful online course.

How does it work?
The rubric comes with training at all levels much like the Quality Matters rubric. Much of the training centers around making sense of some of the questions which could be streamlined. There is training available for faculty, peer reviewers, and staff. There are classes that the CSU runs that covers:

  • Introduction to Teaching Online Using the QLT Instrument
  • Reviewing Courses Using the QLT Instrument
  • Applying the QM Rubric
  • Improving Your Online Course

This gives you an idea about how the rubric is used in the CSUs – teacher training, reviewing courses, and improving courses.

What does it assess?
The rubric has nine sections with 53 objectives:

1. Course Overview and Introduction (8 objectives)
2. Assessment and Evaluation of Student Learning (6 objectives)
3. Instructional Materials and Resources Utilized (6 objectives)
4. Students Interaction and Community (7 objectives)
5. Facilitation and Instruction (8 objectives)
6. Technology for Teaching and Learning (5 objectives)
7. Learner Support and Resources (4 objectives)
8. Accessibility and Universal Design (7 objectives)
9. Course Summary and Wrap-up (3 objectives)

What are its weaknesses?
I would like to see how the previous research was integrated into this rubric. And if previous research was used, how successful were the courses (or instructors) in using the previous rubrics this work was based on? Did they actually improve teaching and learning? Why are some of the rubrics used in the past not being used now? And as far as I know, little research is being done to compare things like the completion and success rates of courses that have undergone a review and those that have not. There is a report that is supposed to come out this Fall that is called “Quality Assurance Impact Research” – I look forward to seeing that.  But essentially, the weaknesses of this rubric are weaknesses that are common to most.

Some of the items in the rubric are not measurable by an outside or peer reviewer. For instance 4.7 asks “the course learning activities help students understand fundamental concepts, and build skills useful outside of the course.” This would require detailed specialized knowledge on the part of the reviewer to answer in any useful way. Items like “the instructor helped to focus discussion on relevant issues” and 5.1 “the instructor was helpful in identifying areas of agreement and disagreement on course topics” could only be answered AFTER a class has been taught. 6.3 asks “technological tools and resources used in the course enable student engagement and active learning” yet the evidence required is merely the presence of collaborative online tools with no clear way to measure their effective use. Owning a Spanish dictionary does not make me a Spanish speaker.

Some of the items require specialized training or access to experts for instance 8.6 says “all tools used within learning management system or that are third-party are accessible and assistive technology ready.” I have worked for too many colleges that rely on vendors to make those decisions. Real expertise is needed to see if the code in an online tool is accessible or accessible with a screen reader. I am not sure how a checkbox on a rubric solves that problem.

It is long and detailed. It takes time to actually use it. In the end, this is probably a good thing. I see this more as a training tool than a course checklist.

What are its strengths?
One of its strengths is its flexibility. The authors added Mobile Readiness at a later date as they began to find that more and more students access their course materials from their phones. The rubric is openly licensed and documentation aligning it with the Quality Matters rubric is also available. Also, it weaknesses are also its strengths. The length of the rubric points out that it takes a community to create a successful online course. Faculty, instructional designers, librarians, accessibility specialists, and IT departments all have a role to play in the process. The question then becomes: how much of this belongs in an online course development process and how much belongs in an online course rubric?

Posted in assessment | Tagged , , , , , , , | 1 Comment

Course Rubrics: OCAT and Peer Assessment Process

Western Carolina University logoThe Coulter Faculty Center eLearning Faculty Fellows at Western Carolina University developed the “Online Course Assessment Tool and Peer Assessment Process” or “OCAT.” This is essentially a checklist style rubric designed to provide peer feedback on the design of their online courses.

What Problem Did This Solve?
According to the notes on the rubric, they developed this tool and confidential peer assessment process to provide faculty with constructive peer feedback on the design and instruction of online courses. Their expected benefits were:

  • Constructive feedback regarding teaching effectiveness
  • Instructional improvement
  • Faculty development
  • Opportunities for peer support

What Does The Rubric Assess?
The rubric has seven sections. The first five sections assess course design and teaching concerns. The last two sections provide spaces for summary narratives from a peer reviewer and an instructor response.

The first five sections assessing the course include:

  1. Course Overview & Organization
  2. Learner Objectives & Competencies
  3. Resources & Materials
  4. Learner Interaction
  5. Learner Assessment

There are some interesting ideas in this rubric that I think are important to student success in online courses that you usually do not see covered in a rubric. For instance, under “Resources & Materials” they include “provides opportunities for students to contribute to course resources.” This is a powerful learning opportunity for students in online courses. Of course, I would like to see something about open education resources or openly licensed materials here (open textbooks or Creative Commons licenses).

Weaknesses of the Rubric
The implied definition of an online course in this rubric relies too much on content delivery rather than knowledge creation or community building. To be fair, the rubric includes statements like “fosters interaction among constituencies inside and outside the course as appropriate (e.g. student-student, student-instructor, and with external persons or agencies)” but “fostering” is not the same as integrating that interaction into the curriculum. There is a recognition of different learning modalities (“learning styles”) in the rubric but no attention to accessibility. They must address this elsewhere which, given the expertise needed to really address accessibility, is understandable.

Strengths of the Rubric
This is one of those rubrics where the purpose and the process is as important as what it assesses. Some of the issues around using rubrics for assessing online courses include Academic Freedom and possible union issues around evaluating faculty. This rubric is focused on peer assessment. A note in the rubric says “the peer assessment instrument itself will also be available for faculty use as a self-assessment faculty development tool” which, I think, is the most valuable use of an assessment process. The process includes meeting with a trained, peer reviewer. The other strength of this rubric is that each section has a space for writing down “instructional item(s) emerging from peer discussion not included in the list above.” This helps address the individuality of the teacher and different teaching styles. Rubrics should not be used to create cookie-cutter courses. It is obvious that the team that put this rubric together did some research and brought in their local academic community to develop this process.

Posted in assessment | Tagged , , , , , | Leave a comment

Course Rubrics: CMU’s Quality Assurance Checklist

Central Michigan UniversityIn my look at online course rubrics, I want to include Central Michigan University’s Quality Assurance Checklist from their Center for Instructional Design (CID) because it is typical of what most colleges who do have a rubric usually have. There is nothing wrong with keeping it simple.  The CID staff members “research the latest pedagogical and technological information relating to online and hybrid classes in order to provide you with sound instructional technology support. We are a staff of instructional designers and training experts who provide assistance with the development of online and hybrid courses and course elements. We also offer training on the latest instructional technologies and online teaching and course development workshops.”

The rubric implies a basic, simple definition of online learning. Much of the rubric’s concerns are around meeting institutional standards rather than address what does or does not work in an online course. For instance, the first item in the rubric asks if the course “adheres to the Master Course Syllabus.”

What does it assess?
The checklist covers the following five areas:

  1. Course Structure
  2. Syllabus
  3. Content Organization & Usability
  4. Instructor Presence & Learning Community
  5. Assessment

with  a sixth area for “Additional Comments.” There are 42 items in the rubric. I am not sure why issues like “appropriate technologies and methods are used to support course activities/assignments” are in the “Course Structure” area and not in a separate area discussing technology. This is what is unfair about my evaluation of online course rubrics, each item in the rubric has a history either from the checklist the institution borrowed it from or from the institution itself. When you see an item in a teacher’s syllabus that says that poisonous plants are not allowed in the classroom, you know that there-in lies a tale.

What are the weaknesses of the rubric?
Right off the bat, for a rubric that is focused on quality assurance, there is no discussion of accessibility. There is one item that says “transcriptions are provided on PowerPoint narrated lectures and on course intro audio/videos.” But there is no other attention to accessibility. How about alt tags for images? I understand that a rubric cannot solve all of those issues but a rubric is a good place to get that conversation started with faculty. And the rubric does not tell the whole story of the institution. There could be many opportunities elsewhere for faculty to learn how to implement accessible media in their courses. The CID instructional designers appear to be available to consult on accessibility.

What are the strengths of the rubric?
This is one form in a set of forms, if you look at the page, there is a pull-down menu that leads to a “Peer Review Checklist” and one for an “Online Course Revision” form. I am hoping that all of these forms together are part of a larger professional development plan that would include faculty workshops on defining terms in the rubric. I like the attention paid to things like item 15 which shows the students how to get help. Communicating how to get help and what instructor expectations are of students are essential in online classes because those expectations are different than in face-to-face courses.

I appreciate the focus on instructor presence, building community, and the encouragement for having guest speakers. This checklist would be a good starting point to help a department or institution explore problems and issues in online learning. For a rubric to be this simple, there would have to be a lot of agreement (and experiential homogeneity) as to what constituted a quality online experience for students.

Posted in assessment | Tagged , , , , , , , , | Leave a comment

Seeing Ourselves Through Technology

Book cover - Seeing Ourselves Through TechnologyI just finished reading Jill Walker-Rettberg’s 2014 book Seeing Ourselves Through Technology: How We Use Selfies, Blogs and Wearable Devices to See and Shape Ourselves. There are so many things I love about this book, but because I am so passionate about open education resources and Creative Commons licensing, the first to me is that it is freely available under a Creative Commons license from a major publisher.

Walker-Rettberg has been an important part of my thinking on technology for some years now. She brings in to her work scholarship, lucid connections to the humanities, and a personal dimension missing from similar writings on technology. Her work on blogging and hypertext theory are must-reads and because her work is grounded in the humanities, it is always relevant. There is a great intersection in her work of history, literature and philosophy.

The six chapters cover visual self-portraits, filters and filtering, selfies, automated diaries, the quantified self, and a final chapter about privacy and surveillance. Each one of these chapters explores technologies that create a lens through which the self is presented. It is refreshing to read scholarship on selfies that does not just write off an entire generation as self-absorbed – there is so much else going on here. Just as drawing, physical photography and previous art forms have led others to explore and experiment with the presentation of the self, so too with these new technologies. According to an article in the Independent, “For the first time ever there are more gadgets in the world than there are people, including a growing number that only communicate with other machines, according to data from digital analysts at GSMA Intelligence. The number of active mobile devices and human beings crossed over somewhere around the 7.19 billion mark.” To think that we can take the old paradigms and apply them to this brave new world is the height of folly. We need this kind of scholarship to help us sort out where we have been and where we are going.

It is a short book covering a broad subject. In her historical run-up to where we are now, I would have loved to have seen something about the importance of letters and epistolary novels and more about the Twitter-esque writings of Samuel Pepys – who now has a Twitter account. Walker-Rettberg is a good antidote to the classist fear of technology and modern culture found in writers like Sven Birkirts who believes that you are not really reading a book unless you are reading it in the same format and cultural milieu as it was written. It is refreshing to read an examination of technological culture written by someone who is not afraid of it. There are echoes here of Barthes and Merleau-Ponty in its focus on the expressive nature of human beings through writing, image, gesture and lived behavior.

Writing like this is critical. In my own field, education technology, there is a common lack of a sense of history. The entrepreneurial culture is mired in the flash of newness. This is why commercial MOOCs, for instance, have been such a huge failure: folks like Elon Musk and Andrew Ng have no sense of where online teaching and learning has come from so they are stymied by the simplest of issues (e.g. online students need support) that were solved in the 90s.

One of the important points I took away from this book is her acknowledgement that technology does not create the image of ourselves but an image. By looking at the images of ourselves that technology provides, we have to bring into question the image of ourselves that we already hold and why we have them. The book discusses how we see ourselves through the lens of data and the last chapter brings into question how we let corporations track and manage that data. I highly recommend this book and look forward to her future work.

Posted in cyberstudies | Tagged , | Leave a comment

Betsy DeVos Cancels Solar Eclipse

I don’t know if you have gotten this letter yet. Seems a little strange to me but I guess that is where we are now: 

Department of Education LogoAugust 16, 2017

Dear Fellow Educator,

I am writing this letter to announce that the Department of Education is canceling next week’s so-called “solar eclipse.” This is not because others in the administration feel that this eclipse is a holdover from the failures of the Obama administration but because it is completely against the new direction that education is taking in America.

Not all states are getting the total solar eclipse so it is unfair that the Department of Education should support it. This is clearly a states’ rights issue and observation of the eclipse should be decided at the local level on a school district by school district basis.

Also, the eclipse is taking place at different times in different parts of the country. It is a total eclipse in one state, partial in another, and one state is observing at one time and other states at another. In other words, it is totally unpredictable. And why are the STEM disciplines fetishizing the solar eclipse? Why aren’t there any lunar eclipses to provide a balanced view? Experts tell me that during the eclipse the moon will move from right to left across the sun which is an obvious politicizing of what is supposedly an educational event.

There are STEM instructors who are using this eclipse in assignments. This makes the experience of the eclipse a learning artifact assessable in student portfolios which makes the eclipse a clear violation of the Family Educational Rights and Privacy Act of 1974 (FERPA).  In addition, the assignments present ADA concerns as the learning artifact is accessible solely as visual media.

A solar eclipse leaves many of our online students at a disadvantage. I talked to my IT people about the affordances, objectives, and outcomes of the eclipse as a learning object. They said that they were using the Canvas LMS and that plunging the world into darkness and lowering the temperature by 25 degrees was more of a Blackboard thing, but that there were some promising possibilities in Second Life and other simulators.

We are currently setting up a commission to examine this issue to address any eclipses that might occur in the future. This commission will propose a special voucher that school districts will be able to apply for on a case-by-case basis if they wish to participate in any possible subsequent eclipses.


Betsy DeVos

United States Department of Education

Posted in education | Tagged , , | Leave a comment