A course archiving strategy is the white whale of higher education. I can remember being part of discussions a decade ago about how to keep a Moodle instance at a manageable size while preserving information. There were two challenges: come up with a policy that faculty could support, and execute a reasonable technical implementation of that policy. In this post I’ll discuss the tools we built to implement our chosen policy.

Policy

The policy we crafted with faculty input is straightforward:

  • We retain seven years of courses
  • Five of those years are accessible to faculty
  • Two of those years are offline, but may be requested

We settled on seven years because that’s the maximum length of the promotion, tenure, and review process at Lafayette. It’s also long enough to capture infrequently-taught courses. The policy contains some implementation details but we had a pretty free hand to execute it.

Execution

It took us close to five years of thinking, arguing, and false starts before we got this right. We came very close to implementing “yearly” Moodle instances. Each academic year would get a new instance. As instances aged we would make them inaccessible, then delete them. An academic portal would link all the instances together and mask the complexity from faculty.

We eventually rejected this approach on multiple grounds:

  • The engineering involved was significant. We’d have to build the portal and maintain it. We’d have to build a way to reliably provision a new Moodle every year with our desired settings. We’d need a reasonable way to move courses from one instance to another (I actually built that as a proof-of-concept at the 2014 Moodle Moot).
  • We’d lose the benefit of upload deduplication because every year’s worth of courses would have its own moodledata root.
  • User preferences would not persist from year to year, unless we synchronized them somehow.

Keeping everything in one big continuously-upgraded Moodle environment has much to recommend it, assuming we can restrict access in the manner described. It’s also been an article of faith in the CLAMP community until now that mass deletion from the interface is unreliable.

Ad-hoc engineering

The key to solving our problems lay in a little-used feature from Moodle’s Task API, which was introduced in 2.7. The Task API replaced the traditional cron structure and it’s pretty cool—you can define scheduled tasks as you would cron itself, instead of having things run continuously, or at random. The API also introduced the concept of an Adhoc task, something to be run in the background on-demand.

This is a powerful capability. As a developer, I can create an interface for a user to request an expensive change, but they won’t have to keep their web browser open to finish it. That’s the problem with trying to delete a term’s worth of courses in Moodle: 60 categories and 900 courses will often time out before finishing.

We built two tools at Lafayette: Hide courses and Delete courses. Each adds an option in the category administration for mass actions on the courses within that category. In the case of Hide courses, it adds options to either hide or show all the courses. In the case of Delete courses, it deletes all the courses, leaving the categories behind. Both plugins queue up adhoc tasks, one per course. The tasks are then processed in the background by Moodle’s cron, while the user heads off to do other things.

We finish off the offline access by overriding the moodle/course:viewhiddencourses capability for teachers in the categories where we just hid the courses. The courses remain accessible to the managers, and we retain the benefits of data deduplication, but as far as the teachers are concerned the courses are in offline storage. We put up a Qualtrics survey which feeds our issue tracking system for when they need access.

Keep it simple!

The above represents a series of simple, small-scale solutions to a broad problem. There were things we liked about the yearly solution. We still might build an academic portal, and there’s much to recommend it. We came to realize that it was the wrong set of projects for where we were as an organization, and that we needed to solve the archiving problem before the clock started on the policies that the institution had enacted. As a side bonus, we could release both tools to the broader community, and they’re useful even for schools not tackling our specific problems.