Creating a good scoring rubric and giving detail on how you are assessing features can help protect you in the event of a challenge.
There is a lot to take in: you may wish to make an audio recording of the sessions in case you can't remember the answer to a question or there are differing interpretations of what was said. It can also help to tone down some of the more optimistic sales promises if the supplier knows you have a full record of the discussions! You should, however, make sure you get the agreement of everyone in the room.
The University of Huddersfield is a good example of applying what we know about good assessment practice to selecting a VLE.
The University had four sets of usability tests: basic and advanced academic staff use, student use and use for administrative tasks.
The vendors had to commit to this testing as part of their tender and it involved a lot of preparation on their part. There was a detailed specification of modules to be set up, content to be uploaded and structure for what this had to look like. The University defined the dataset to be used and the testers followed a prescribed script.
Students were involved in the usability testing and were offered Amazon vouchers for participating.
The format of staff and student tests were very similar - they had to conduct basic tasks on each of the VLEs (1 hour per VLE) and then undertake a survey. The basic tasks involved uploading material, participating in discussion boards etc. and there were more advanced tasks for staff. The academic staff had to give up three full days (one day per VLE).
The University was very concerned to remove any bias in usability testing, for example the students were split into three groups and they tested the VLEs in different order so there was no bias as to which system the students saw first or last.
Scottish universities and colleges are part of a procurement consortium known as APUC (Advanced Procurement for Universities and Colleges). The group has set up a framework agreement for VLE supply and Glasgow School of Art acted as the pilot institution for this.When it came to the testing stage there were two testing groups:
Each supplier was provided with a set of resources of different types e.g. quiz, SCORM, assignment, video and asked to build a course that did all of these things and create instances for each of the testing groups.
The tests did not involve use of actual institutional course/student data but they did ask for details about data migration. The testers were only charged with looking at functionality and not other considerations such as cost.