Amtrak, B-Movies, Web Development, and other nonsense

Tag: Moodle

Node on RHEL7

In August 2017 we implemented Swarthmore College’s PDF accessibility tool for Moodle. This required us to stand up a Node.js application, which was a new experience for us. Our environment was RHEL7, and our preferred web server Apache.

Application deployment

We followed our usual Capistrano principles for deploying the application. We created a simple project with all the Capistrano configuration and then mounted the Swarthmore project as a git submodule in a top-level directory named public. We configured the Capistrano npm module to use public as its working directory to ensure that the various node modules are installed on deployment.

PM2

PM2 is a Node process manager; its role is to ensure that the application runs at boot. To use it, we first need to install it globally:

sudo npm install -g pm2

Next, we create an ecosystem.json file. This needs to be in the root of the project repository; since we’re using Capistrano we define it in shared/public and symlink it on deploy. This is what ours looked like:

{
"apps": [{
    "name": "{NAME}",
    "script": "./index.js",
    "cwd": "/var/www/{NAME}/current/public",
    "error_file": "/var/www/{NAME}/current/logs/{NAME}.err.log",
    "out_file": "/var/www/{NAME}/current/logs/app.out.log",
    "exec_mode": "fork_mode"
}] }

All straightforward. We create a new user on the unix platform to own this job, have it start the process:

sudo -u {USER} pm2 start ecosystem.json

We can run a second command which generates the necessary syntax for setting up the systemd commands:

sudo -u {USER} pm2 startup

Apache

Having done all that, the node application is happily running on port 8080. We’re not interested in exposing that port in our environment, so we add a proxy pass to our standard Apache configuration for that virtual host:

        ProxyRequests on
        ProxyPass / http://localhost:8080/

We’ll have to revisit this if we ever want to have a second node application on the system, but for now it works.

Featured image by Hermann Luyken [CC BY-SA 3.0 or GFDL], from Wikimedia Commons.

Critical pagination

This is a story about Moodle, PHP, Active Directory, OpenLDAP, and how I stared a problem in the face for two days without realizing what I was looking at.

Pagination

I assumed maintenance of the LDAP syncing scripts plugin (local_ldap) in 2016. One thing I did was to add PHPUnit coverge which I wrote about in Writing LDAP unit tests for a Moodle plugin.

I’ve received reports about a possible bug with the plugin, Active Directory, and large numbers of users. After standing up a test Active Directory server (which is a story for another day), I’ve been extending the unit tests for the local_ldap module to take advantage of pagination.

PHP added support for LDAP pagination in PHP 5.4. Moodle added support soon after in Moodle 2.4 (MDL-36119). Beyond some range queries on the Active Directory side, there wasn’t anything in the plugin using pagination. The queries quickly revealed problems with the Active Directory code:

1) local_ldap_sync_testcase::test_cohort_group_sync
ldap_list(): Partial search results returned: Sizelimit exceeded

/var/www/moodle/htdocs/local/ldap/locallib.php:144
/var/www/moodle/htdocs/local/ldap/locallib.php:626
/var/www/moodle/htdocs/local/ldap/tests/sync_test.php:176
/var/www/moodle/htdocs/lib/phpunit/classes/advanced_testcase.php:80

Adding pagination is fairly straightforward, but you have to take care because not all LDAP implementations support it. Here’s an example cribbed from Moodle’s implementation:

$connection = $this->ldap_connect(); // Connection and bind.
$ldappagedresults = ldap_paged_results_supported($this->config->ldap_version, $connection);
$ldapcookie = '';
do {
	if ($ldappagedresults) {
    	ldap_control_paged_result($connection, $this->config->pagesize, true, $ldapcookie);
    }
    ... // Whatever LDAP task you're doing
    if ($ldappagedresults) {
    	ldap_control_paged_result_response($connection, $ldapresult, $ldapcookie);
    }
} while ($ldappagedresults && $ldapcookie !== null && $ldapcookie != '');

If the LDAP server doesn’t support pagination all the extra code is a no-op and you pass through the do…while loop once.

I wound up adding pagination in three places: the code that gets all the groups from the LDAP server, the code that searches for all the distinct attribute values (e.g. all possible values of eduPersonAffiliation), and the code in the unit test itself which tears down the test container in LDAP. Everything passed in Active Directory, so I made what I figured would be a pro forma push to Travis to test the code against OpenLDAP. I came back from lunch to read an error message I’d never even heard of:

ldap_delete(): Delete: Critical extension is unavailable

Critically unavailable

I’ve spent a fair amount of time debugging bizarre LDAP problems but “Critical extension is unavailable” was a new one on me:

1) local_ldap_sync_testcase::test_cohort_group_sync
ldap_delete(): Delete: Critical extension is unavailable
/home/travis/build/moodle/local/ldap/tests/sync_test.php:405
/home/travis/build/moodle/local/ldap/tests/sync_test.php:67
/home/travis/build/moodle/lib/phpunit/classes/advanced_testcase.php:80

Researching this phrase led me to a discovery: if you’ve run a paginated query against LDAP in PHP, you need to tear down that connection and reconnect to the server afterwards. The code in question was in the PHPUnit code which tore down the environment between tests. It runs two queries; one deletes all the users and groups while the second deletes the organizational units. This was code I’d taken from the Moodle LDAP authentication module (auth_ldap) and extended with pagination when the Active Directory tests failed.

Mocked up, this is what the code did before I modified it:

Establish LDAP connection
Get the top-level information about the test container
Get the users and groups from the test container
Delete those users and groups from the test container
Get the organizational units from the test container
Delete the organizational units from the test container
Delete the test container

After adding pagination and connection closures, the code did this:

Establish LDAP connection
Get the top-level information about the test container
Do
	Get the users and groups from the test container
	Delete those users and groups from the test container
Loop
Close LDAP connection
Establish LDAP connection
Do
	Get the organizational units from the test container
	Delete the organizational units from the test container
Loop
Close LDAP connection
Establish LDAP connection
Delete the test container

And it still didn’t work. I played around with various permutations for hours but didn’t make any progress. It was Friday, I was tired, I went home for a three-day weekend and didn’t think about it at all (maybe a little). When I got back in the office the problem was staring me right in the face.

In the old, unpaginated code, it wasn’t a problem to invoke ldap_delete after retrieving results, using the same LDAP connection. With pagination that logic doesn’t work anymore; we need to ensure we have all the results first, recreate the connection, then separately run the deletes. Thus modified, the logic is like this:

Establish LDAP connection
Get the top-level information about the test container
Do
	Get the users and groups from the test container
Loop
Close LDAP connection
Establish LDAP connection
Delete those users and groups from the test container
Do
	Get the organizational units from the test container
Loop
Close LDAP connection
Establish LDAP connection
Delete the organizational units from the test container
Delete the test container

The tests pass now. It’s interesting to me that this was never an issue with Active Directory, only with OpenLDAP. It’s a valuable lesson about when to refactor your code and check and your assumptions.

Featured image by Nevit Dilmen [GFDL or CC-BY-SA-3.0], from Wikimedia Commons

Running Moodle CI tests in GitLab

I maintain about a dozen plugins in the Moodle plugins repository. I use Moodlerooms’ moodle-plugin-ci project to test all of them on Travis CI. It’s a fantastic tool and rely heavily on it.

This fall I’ve been working on a plugin which, because of various hooks into Lafayette’s internal infrastructure, I’m not releasing publicly. I’d still like to test it in the usual way, so I decided to run the tests on our internal GitLab infrastructure.

Continue reading

Implementing a course archiving strategy in Moodle

A course archiving strategy is the white whale of higher education. I can remember being part of discussions a decade ago about how to keep a Moodle instance at a manageable size while preserving information. There were two challenges: come up with a policy that faculty could support, and execute a reasonable technical implementation of that policy. In this post I’ll discuss the tools we built to implement our chosen policy.

Continue reading

Measuring activity in Moodle

It’s a simple question with a complex answer: in a given academic term, what percentage of our Moodle courses are “active” (used by a faculty member in the teaching of their course). We have to start by figuring out what counts as a “course” in a term, and then come up with an inclusive measurement of activity.

Continue reading

Pick a date, any date

Moodle 3.2 introduced the concept of end dates for courses. Moodle 3.3 added a new Course Overview block which uses end dates to determinate whether a course is in progress, in the past, or in the future. This is pretty great, unless you’re in the following situation:

  • Your school has five years worth of courses
  • Those courses don’t have end dates

Congratulations—you now have five years of courses in progress. Your faculty will have five pages worth of past courses on the Course Overview block! That’s probably undesirable. To avoid it, I’m writing a plugin that lets an administrator set course start and end dates at the category level. While working on it, I ran an interesting edge case with Behat acceptance tests, reminding me that you’d best treat Behat like it’s a real user.

Continue reading

Writing LDAP unit tests for a Moodle plugin

In 2016 Lafayette College began maintaining the LDAP Syncing Scripts (local_ldap) plugin after the tragic death of the previous maintainer, Patrick Pollet.

I didn’t know Patrick but he had a strong reputation in the Moodle community. I’m pleased to say that we made few substantive changes to his code. Most of the changes were simple updates, such as migrating the command-line/cron scripts to Moodle’s task infrastructure, and various nit-picky code standards issues which didn’t affect functionality.

PHPUnit

The biggest lift was implementing PHPUnit test coverage for the plugin. I started out with the following requirements:

  • Fully-scripted setup for OpenLDAP, so that the tests can run inside a continuous integration environment
  • Test coverage for group synchronization
  • Test coverage for attribute synchronization

I started this project by building an OpenLDAP environment inside Moodle Hat, the Vagrant development profile I maintain. Implementing a configuration in Puppet is good practice for wrestling with Travis.

Starting from scratch with OpenLDAP (every time!) presents certain challenges that you don’t encounter in a mature environment. A few I encountered:

  1. When you bootstrap OpenLDAP it has a completely empty schema. PHP’s ldap libraries can’t talk to it in that state. You have to populate it with some data, even if it’s completely arbitrary.
  2. Selection of backend databases matter. LDIF is the quick and easy path, but it doesn’t support pagination and Moodle will break in obscure ways. I chose bdb because it’s available in most repositories and it worked.
  3. When you’re setting a generic testing password in your slapd.conf you can just dump in rootpw SomeArbitraryPlaintextPassword and it’ll work. Don’t run in production! Or, really, anywhere that has state.

Once I’d worked through those issues Christian Weiske’s invaluable blog post provided everything I needed for implementing on Travis.

Travis

LDAP Syncing Scripts leverages Moodlerooms’ excellent moodle-plugin-ci plugin for travis-ci integration, with a few tweaks. The full travis-ci.yml file is visible on the GitHub repository; let me walk through a few things.

We need the slapd and ldap-utils packages installed. To use Moodle’s built-in LDAP PHPUnit testing we need to define the location of the test server in the config file:

define("TEST_AUTH_LDAP_HOST_URL", "ldap://localhost:3389");
define("TEST_AUTH_LDAP_BIND_DN", "cn=admin,dc=example,dc=com");
define("TEST_AUTH_LDAP_BIND_PW", "password");
define("TEST_AUTH_LDAP_DOMAIN", "dc=example,dc=com");

We need to create an INI file to force PHP (in travis) to load the ldap extension, and a slapd.conf file to define how our OpenLDAP enviroment will function. The schema settings need to match what you added to Moodle’s config.php. We start slapd and then, as the final step, import our default data. This data isn’t used, but it gets around the problem of an empty schema. Note that while this data is stored as an ldif file for readability purpose, the backend is bdb.

Tests

The actual tests I derived from the tests for Moodle’s auth_ldap plugin. The code is long but self-documenting. There are no particular gotchas, though I found it helpful to extend auth_ldap_plugin_testcase instead of starting fresh.

Gulp, it’s Code-Checker!

Code-Checker is a tool distributed by Moodle HQ which lets you validate code against core coding standards. You install it as a local plugin in a development environment and run it against specified files. It then spits out all kinds of nit-picky errors:

theme/stellar/layout/default.php

  • #76: ····<?php·//·echo·$OUTPUT->page_heading();·?>
    
  • This comment is 67% valid code; is this commented out code?
  • Inline comments must start with a capital letter, digit or 3-dots sequence
  • Inline comments must end in full-stops, exclamation marks, or question marks

Code-Checker leverages the PHP_CodeSniffer tool; in essence it’s a set of CodeSniffer definitions wrapped in a Moodle plugin. This adds a fair amount of overhead for testing coding standards–you shouldn’t need a functional Moodle environment, nor for that matter a web server.

My preferred integration tool is gulp.js, a task-runner built on node. It’s similar to grunt but without all the front-loaded configuration. There’s a plugin for gulp called gulp-phpcs which integrates PHP_CodeSniffer with gulp and lets you check the files in your project. Happily it was pretty simple to do this with Moodle.

First, you need to have PHP_CodeSniffer available in your development environment. This is how I did it on my Macbook:

cd /usr/local
mkdir scripts
cd scripts
git clone https://github.com/squizlabs/PHP_CodeSniffer.git phpcs

I then added that directory to my PATH:

PATH=/usr/local/scripts/phpcs/scripts:$PATH

Finally, I cloned in the Moodle plugin and added its standards definition to the installed paths for PHP_CodeSniffer:

git clone https://github.com/moodlehq/moodle-local_codechecker.git moodlecs
cd phpcs
./scripts/phpcs --config-set installed_paths ../moodlecs

At this point we’re ready to integrate it into gulp. We need to install the gulp-phpcs module and add it to the project:

npm install gulp-phpcs --save-dev

Now we provide a basic configuration in our gulpfile.js. This example will check all the php files in Lafayette’s Stellar theme:

// List of modules used.
var gulp    = require('gulp'),
    phpcs   = require('gulp-phpcs');    // Moodle standards.

// Moodle coding standards.
gulp.task('standards', function() {
  return gulp.src(['*.php', './layout/**/*.php', './lang/**/*.php'])
    .pipe(phpcs({
      standard: 'moodle'
    })) 
    .pipe(phpcs.reporter('log'));
});

Invoked from the command line, we get the same results as from the web interface, but faster and without the overhead:

[10:24:21] PHP Code Sniffer found a problem in ../theme_stellar/layout/default.php
Message:
 Error: Command failed: 
 
 FILE: STDIN
 --------------------------------------------------------------------------------
 FOUND 0 ERROR(S) AND 3 WARNING(S) AFFECTING 1 LINE(S)
 --------------------------------------------------------------------------------
 76 | WARNING | Inline comments must start with a capital letter, digit or
 | | 3-dots sequence
 76 | WARNING | Inline comments must end in full-stops, exclamation marks, or
 | | question marks
 76 | WARNING | This comment is 67% valid code; is this commented out code?
 --------------------------------------------------------------------------------

This also lets you create watcher tasks to catch errors introduced while developing.