In 2008, with a widely spread team of assessors signing up new learners, we started to get fed up with the number of our ILRs which were getting lost or delayed in the post. We took the decision to set up the ILR as an electronic document and equip assessors with signature pads so that the signed form could be emailed to the office. This also fitted in well with our use of e-portfolios, as data could be imported directly from the ILR without having to re-key it at the other end.
Naively, I decided to run this past the auditors at the LSC (as it then was). I wrote to the audit managers in all three regions where we had a contract, and waited for the responses – which never came. I then tried Coventry, only to be told flatly that electronic signatures were unacceptable. Apparently, only a “wet” (i.e. ink) signature could provide the necessary evidence of the learner’s existence, and satisfy the LSC that the provider was not claiming funds for a fictitious learner.
I thought about this. All that a signature proves is that someone signed the form. Would the LSC accept ILRs in the names of D. Duck or M. Mouse, merely because they bore a signature? When a bank takes on a new customer, it first undertakes elaborate ‘money laundering’ checks to prove existence. Only then does it ask for a specimen signature (or issue a PIN) which can be used to validate future transactions.
In the world of training, the LSC had no such prior relationship with the learners whom it was being asked to fund. Rather than inspecting signatures which told them nothing, auditors would be better off using the details on the ILR to contact learners and ask them about their training. If there were a significant number of non-responders, or anomalous answers, the boys could be sent in.
But the mindset persisted, not only that signatures amounted to valuable evidence, but that only paper and ink would do. I have heard accounts of funding being clawed back after audit visits, not because a learner had been found to be fictitious, but simply because the form had not been ‘properly’ signed by the learner. In this context, a signature pad was regarded not as a different kind of pen, but as an instrument of the Devil. Like many people who are afraid of the fraud potential of electronic transactions, the LSC failed to grasp that almost every electronic fraud has its exact parallel in the paper world.
Not being someone to take no for an answer – especially when it is downright illogical – I continued to argue the point. This culminated in a 2009 meeting with the CEO and the Director of Audit, where I took along a laptop and a selection of ILRs, both paper and electronic. Some were genuine and others were fabrications, and it rapidly became apparent that, whatever methods might be available to tell them apart, the signature was not one of them.
It was agreed that we had a point. However, the SFA was not yet ready to give carte blanche for the use of electronic records and so, in an attempt to square the circle, we were shoehorned into a pilot project, where our small company of ten staff took its place alongside giants like Burger King and Ford! And so began a comical period of audit visits and huge IT control questionnaires which were completely inappropriate to our set up and which neither we, nor the auditors, had any idea how to adapt to our circumstances. Proprietary systems such as our e-portfolio software – and even the PICS database used by hundreds of providers and colleges to interface with the SFA’s own systems – were investigated at length, even though these same systems are largely ignored by auditors in providers where they exist alongside manual files and paper/ink forms that can be inspected and ticked.
Eventually – probably as a result of sheer exhaustion –we were given permission to dispense entirely with paper systems. Although this was welcome, it was something we had never asked for. But our success took a different turn when, in April, the SFA announced a general policy of accepting electronic evidence, including signatures, from 1 August 2011. Its subsequent briefing note confirmed that, as a result, it would be discontinuing the approach of accrediting individual providers like us. Unfortunately, however, its choice of words “the Agency will not provide certification for paperless systems” led many providers to conclude that the policy of accepting electronic evidence had been abruptly reversed.
According to the SFA, only 10% of providers either use, or have expressed an interest in, electronic records and this is therefore still seen as a minority issue. I suspect the figure would be very much higher if e-portfolios were included, but no matter: the fact is that this is a completely circular argument. If 90% of providers are still using paper, this is not necessarily because they believe it to be easier or better. It is much more likely to be because they are aware of the suspicion which SFA auditors have for ICT and have decided to play safe. In the last week I have heard this view reinforced at a conference by a senior SFA official who advised providers who are planning to scan their ILRs to keep the paper forms in a drawer ‘just in case’ their auditors won’t accept PDF files. Among the providers I have talked to, electronic evidence is one of the hot topics – perhaps ranking just behind guided learning hours, on which I plan to post soon.
Despite the eminently sensible approach now being developed at the top of the SFA – that as long as evidence for the existence, eligibility and achievement of learners is reliable, it doesn’t matter what format it takes – it remains to be seen how this will play at the sharp end. Ideally, SFA will start the audit process by looking at the data from funding claims to identify indicators of risk and by contacting learners on a sample basis to confirm the information submitted. Targeted, follow-up audit visits can then be carried out by fewer, better auditors – people who are prepared to ask challenging questions and assess the answers with an open mind. This is broadly the approach now being taken – albeit with some inconsistency – by Ofsted, as well as by all the large private sector audit firms. However, if decisions on the acceptability of electronic records are going to continue to be left to the discretion of individual auditors, many of whom are wedded to paper documents, we are not out of the woods yet.
I set out below the SFA’s guidance note as I would like to have seen it written.
ALTERNATIVE BRIEFING NOTE AS SFA COULD HAVE WRITTEN IT
Guidance Note 7 advised that from 1 August 2011 providers will be able to hold evidence to support funding claims in an electronic format. This briefing note provides further information and practical guidance, for both providers and auditors, about the application of this policy. It is made in the context of the contractual relationship between the Agency and its providers and does not displace obligations to other agencies or regulators. For example, providers delivering European Social Fund (ESF) funded programmes will need to demonstrate compliance with the relevant co-financing beneficiary guidance and requirements.
The use of electronic records is permitted, not required.
There is a general requirement for providers to establish adequate controls over the completeness, accuracy and security of data. The Agency will seek to be pragmatic and reasonable in its acceptance of evidence in support of funding. That evidence should be capable of demonstrating, to the satisfaction of the Agency’s auditors, the existence, eligibility and achievement of all learners for whom funding is claimed. Provided the evidence is robust and reliable, auditors will not be prescriptive about the form which it takes.
In many cases, electronic evidence has the potential to provide greater assurance than its paper equivalent (for example date and version controls, or verification that the contents of a file have not been altered since creation). In general, however, the same audit considerations apply regardless of the format of evidence, and auditors will not expect electronic evidence to satisfy a more demanding test merely because it is capable of doing so.
Whether learner data is held in paper or electronic format, providers must comply with current data protection legislation.
Because the Agency is now making electronic evidence generally acceptable, there is no longer a need for the approach previously piloted, under which individual providers applied to have their systems certified for this purpose.
Some commonly occurring specific situations are discussed below.
Online data capture
It is acceptable for data concerning learners (e.g. the information on an ILR) to be captured electronically or on paper. Auditors will be concerned with the completeness and accuracy of such information rather than with the means of capture.
It is acceptable for evidence to be retained in the form of scanned documents rather than original paper documents. Providers who wish to do this are responsible for implementing reliable controls over creation, indexing, storage and retrieval, just as they are in the case of paper records.
An e-portfolio is, essentially, a means of storing evidence and controlling the work flow between learners, assessors and verifiers. The Agency takes the view that the method by which portfolio evidence is stored is largely a matter for the provider and the awarding organisation. From an audit standpoint, an e-portfolio may provide a convenient repository for scanned documents and may also provide a source of evidence, in its own right, that learners remain actively engaged in learning.
Auditors will apply the same considerations to the contents of e-portfolios as they do to the contents of paper portfolios. If, for a specific purpose, auditors need to place reliance on an aspect of the e-portfolio software (e.g. access controls or the reliability of date stamping) they may ask the provider to supply or obtain relevant evidence. In the case of proprietary software, in order to minimise duplication and promote consistency, the Agency will maintain, and make available to both auditors and providers, details of products which have been successfully evaluated in this way.
In some circumstances, it is possible for a signature to be compared with a pre-existing, independent signature of the named individual, in order to establish its authenticity (for example, specimen customer signatures held by banks, or a digital signature issued by a competent authority). In these circumstances, the holder of the independent signature will normally have undertaken prior identity checks to confirm the individual’s existence.
Unless these conditions are met, a signature on a document provides no evidence that the named person actually signed the document, or even that the person exists. In addressing learner existence, therefore, the Agency’s auditors will focus attention on the provider’s systems for recruiting, and maintaining contact with, learners. Auditors will also contact learners themselves on a sample basis, in order to verify their existence as well as to confirm details from funding claims.
The evidential value of a signature is chiefly to demonstrate that the person making it accepts responsibility for the accuracy and completeness of the document signed. For that reason the Agency still expects documents such as the ILR, ILP and learner agreement to be ‘signed’ by the learner and, where appropriate, by the provider and/or employer. The format of the signature does not matter – the test is whether it can be shown to have been made by the person named. A graphical signature (however captured, and whether stored in paper or scanned form) can be compared visually with other examples: in the case of an electronic signature there may be software controls, such as the use of secure passwords or PINs, which provide authentication. Auditors will take a pragmatic view of such matters in the light of perceived risk and will not expect different levels of assurance from different forms of signature.