In any procurement exercise there are warning signs that both customers and suppliers should look out for.
On the supplier side there can be indications, sometimes inadvertent, that the customer is not entering into the procurement in as open-minded a way as they might believe themselves to be.
On the customer side you may think you have written a clear and unambiguous statement of requirements that will facilitate suppliers giving a straightforward response but things don't always work out like that.
Many of the contributors to this Toolkit talked about the importance of testing the VLE products rather than simply relying on supplier tender documents and there is a section covering Testing and evaluating.
Even with a comprehensive evaluation, including demonstrations and/or tests, there can sometimes be grey areas as to how well a product will really meet your needs. We have outlined some of the key signs that should prompt you to ask further questions.
This has been compiled from user experiences of testing products. However, suppliers have also warned us to beware of such phrases (from other suppliers).
It always pays to know exactly what version of the product is being demonstrated and what version is quoted for in the proposal. This may be less of an issue in a software as a service (SaaS) environment but you need to be absolutely certain that what you are seeing/or being promised is in the general availability release."That will be in the next version."
If a supplier says definitively what is in the next release they will already have detailed design documentation and confirmed release dates. If, when you ask for this evidence, the follow up is: "That functionality is currently under development; we don't yet have a release date for it" you should treat it as 'vapourware'."You can configure it to do that."
Be clear whether this involves standard end-user configuration, requires a technical specialist or requires bespoke development. Don't go in for bespoke development unless you absolutely have to. It will cost you more, it won't be part of the standard support agreement and it will cause you ongoing maintenance headaches."It is upgrade proof."
Be especially cautious of this claim if it follows the one about configuration because bespoke work is seldom upgrade proof. End user acceptance of screen layout and terminology is a major issue for many universities. Suppliers may tell you how easy it is to rename something that doesn't suit your own terminology or change the layout of a screen but you could be creating a huge maintenance overhead."This is an optional feature."
Double check whether it is actually part of the product as it has been known for suppliers to demonstrate third party add-ons without making this clear. Third party tools mean additional costs and possible contractual and implementation complications. If the optional feature is part of their own suite make sure the proposal includes the full cost of everything they are demonstrating."I'll show you what another customer has done."
If they show you external examples be sure to establish what you are seeing of the core product and what you are seeing of another customer's own development efforts."Yes - it does that."
This warning may sound overly cynical but if the phrase isn't accompanied by a demonstration it isn't worth much!
You need to undertake critical evaluation of claims made by suppliers who may either have misunderstood what you are requesting or give misleading answers. Institutions that use a tick box approach and add up scores can fall victim to this.
Julie Voce, Head of Educational Technology, City,
University of London
You also need to manage your relationship with suppliers who are unsuccessful in the tender process this time round. Give them honest and useful feedback about the shortcomings of their product.
You never know what is around the corner, business needs may change and a solution which is judged to be immature at this stage may take account of your feedback and develop into a desirable solution at a later date.