National Pupil Database

In the United Kingdom, the National Pupil Database (NPD) is controlled by the Department for Education, based on multiple data collections from individuals age 2-21 in state funded education in England. Data are matched and linked using pupil names, dates of birth and other personal and school characteristics, to pupils' attainment and exam results over a lifetime school attendance.

In December 2015, the database contained 19,807,973 individual pupil records. It is deemed by its creators to be “one of the richest education datasets in the world".

Schools use Management Information Systems (MIS) to collect and analyse pupil level information at local level. Data from these systems are used to complete the termly school census returns provided to Local Authorities (regional) or directly to the Department for Education (national) three times a year. The National Pupil Database has expanded in its scope of the items collected, and from children of a wider age range over time. Data once stored in the National Pupil Database, are never deleted.

The National Pupil Database referred to here, covers only pupils in state (or partially state-funded) schools in England. However Similar systems operate across the rest of the United Kingdom.

• For Wales, the Welsh Government holds pupil level data back to 2004.
• For Scotland, the Scottish Government holds electronic records for children in Scotland back to 2002 and allows third party access to this data.
• For Northern Ireland, data is available from approximately 1,200 schools, 400 pre-schools and individual level records for over 300,000 pupils each year. The Northern Ireland Schools Census includes data going back to 1990.

Sources of pupil data in the NPD

Details of all data sources contained within the linked set of data which form the National Pupil Database, and the coverage of children within each source.

Data Source	Ages
Census/PLASC1	2-18
PRU Census	2-18
Early Years Census	2-4
Alternative Provision	2-18
Early Years Foundation Stage Profile 2	4
Year 1 Phonics	5
Key Stage 1	6
Key Stage 2	10
Year 7 Progress Tests 3	11
Key Stage 3	13
Key Stage 4 Awarding Body data	14-21
Key Stage 4 Achievement & Attainment Tables data 4	15 (14/16)
Key Stage 5 Awarding Body data	14-21
Key Stage 5 Achievement & Attainment Tables data 5	16-18
Individual Learner recData ord 6	14-21
HESA data	17-21 (18-19 in 04/05)
Children Looked After 7	0-18
Children In Need 7	0-18
PLAMS 8	16-18
NCCIS
Independent Specialist Providers (ISP)

Data types held

The pupil level data are personal confidential data which include sensitive personal data as defined by the Data Protection Act 1998. The National Pupil Database contains:

Identifiers: the pupil, school and local authority identifiers. With effect from 2010/11, the pupil’s UPN (unique pupil number) remained consistent throughout their time in school and remain permanently on the record.

Fixed pupil characteristics which are expected to stay the same, such as date of birth. However some characteristics which may be expected to be static, for example ethnicity, may be inconsistent within an individual record. Ethnicity was found in 2016 to have been ascribed by school staff in some counties, even to the point of overriding parental and pupil choices to refuse to provide the optional information as recommended to schools by Brighton and Hove City Council ahead of the autumn school census. This option was changed in system providers' design to remove fields 'ascribed by' as a result.

Time-varying pupil characteristics: these may change in each sweep of the census because they describe the circumstances of the pupil at that point in time.

Key stage test results and other attainment data: data at the ages of five, Early years foundation stage profile or (FSP) seven (KS1), eleven (KS2) 14 (KS3), 16 (KS4 or GCSE)and 18 (KS5).

School type characteristics: these describe the school the pupil attends at each sweep of the census. School staff data is not included in this census for the National Pupil Database, but through the separate Workforce Census. Workforce personal data includes identifiers such as names and National Insurance numbers and characteristics such as ethnic group, employment contracts and remuneration details, qualifications and absence information.

There are about 400 possible variables to collect on individual pupils. The full national code sets of all the items of data that can be collected on individual children can be downloaded from the Department for Education are listed in the common basic data set (CBDS), including health and SEND (special educational needs and disability).

Use of pupil level data

For uses of Key Stage attainment datasets and School Census dataset see also England: School Census. Raw pupil level personal data are held in the Department for Education National Pupil Database (NPD). The linked datasets contain data which are identifying, and too sensitive or disclosive to be published, although these data are given out to third parties in raw form.

David Cameron announced in 2011, the government would be “opening up access to anonymised data from the National Pupil Database […].” This was an expansion to other third parties, since these data had already been used for many years and extensively by academic public interest researchers.

Since 2012, Secretary of State has had powers to share raw data from National Pupil Database under terms and conditions with named bodies and third parties who for the 'purpose of promoting the education or well-being of children in England are conducting research or analysis, producing statistics, or providing information, advice or guidance', and who meet the Approved Persons criteria of the 2009 Prescribed Persons Act, updated in 2012/13.

The data when released, however are not anonymised, but are sensitive and identifying. "According to centrally held records at the time of writing, from August 2012 to 20 December 2017, 919 data shares containing sensitive, personal or confidential data at pupil level have been approved for release from the National Pupil Database. For the purpose of this answer, we have assumed the term sensitive, personal or confidential uses of information to be data shares classified as either Tier 1 or Tier 2 as set out in the National Pupil Database area on GOV.UK. [In addition] There were 95 data shares approved between March 2012 and this classification system being introduced."

In a presentation to the NPD User group in September 2016, the Director of the DfE Data Modernisation group acknowledged the release of sensitive data: "People are accessing sensitive data, but only to then aggregate. The access to sensitive data is a means to an end to produce the higher level findings.”

The data items for release are classed into four tiers by the Department for Education, as described in the NPD User Guide. Following the change of legislation, releases of the data since 2012 from the Department for Education to third parties have not been anonymous, but have been of identifiable and highly sensitive (Tier 1), identifiable and sensitive (Tier 2), aggregated but may be identifying due to small numbers (Tier 3) and identifying non-sensitive items (Tier 4). Raw, closed data are released on a regular basis to third parties, and the majority of releases are of Tier 1 and 2 data.

A list of completed National Pupil Database Third Party Requests and those in the pipeline, are published on a quarterly retrospective basis.

Government uses of the data are based on a model of data sharing, passing raw data from one location to another, which is viewed by some as 'obsolete'. Intra departmental transfers of data include to the Cabinet Office for preparation of Electoral Registration Transformation work in 2013, to match participant data in the National Citizen Service, and for use in the Troubled Families programme, as well as arms length bodies such as NHS Digital for a survey "What About Youth" mailed home to 300,000 15 year olds in 2014. Not all government uses of the data are recorded in the Third Party Release Register, such as internal use. The volume of Police and Home Office use first made public through Freedom of Information requests in 2016, were first officially published by the Department, in the Third Party Release Register in December 2017, under "External Organisation Data Shares". Police requests were only documented going as far back as July 2015. This omits police access to records before this date, as noted in a ministerial correction (HCWS272) made by Nick Gibb, Minister of State for School Standards, on the numbers of pupils data released to the Home Office and police. “Information supplied by the Data Modernisation Division of the DfE has been identified as containing incorrect facts in the response provided to Parliamentary Questions concerning the volume of children’s records passed onto the police and the Home Office (PQ48634, PQ48635 and PQ52645) and in figures quoted during a House of Lords Debate on the 31 of October 2016 on the Education (Pupil Information) (England) (Miscellaneous Amendments) Regulations 2016. “

Of the documented 887 requests for identifiable data that have been through the DMAP request process in March 2012 – December 2016, only 29 have been for aggregated data, according to analysis by the NGO defenddigitalme. There were 15 rejected applications between March 2012 and September 2016, including a request "by mistake" from the Ministry of Defence to target its messaging for recruitment marketing. Approved uses include identifying and sensitive data released to Fleet Street papers, “to pick interesting cases/groups of students," and about 60% of applications approved (as distinct from volume of data used) for identifying and sensitive, pupil level data, were from think tanks, charities, and commercial companies.

The Telegraph newspaper was granted identifying and sensitive data in 2013, for all pupils in the KS2, KS4 and KS5 cohorts for the years 2008-2012.

Academic uses of school census data make up about 40% of the requests for identifying, pupil level data, processed through and approved by the DMAP process.. The raw data are sent to the requestor's own location. There is no charge made for fulfilling requests. "DfE does not charge for data (and has not since the NPD process began), nor does DfE charge for the processing and delivery of extracts to customers."

There is however no transparency of the volume of how many children’s data have been given away in approved uses either, because,

“the Department does not maintain records of the number of children included in historic data extracts.” (PQ109065)

Public interest research use of pupil level data through other routes of access to the data, include projects linking individual data together with other education and employment data from citizens' interactions with other government departments and public services. For example, the LEO dataset is made up of information from the National Pupil Database (NPD), the Individualised Learner Record (ILR), the Higher Education Statistics Agency (HESA), Her Majesty’s Revenue and Customs data (HMRC), The National Benefit Database, the Labour Market System and Juvos, the unemployment research database. Further work by DfE compares self-reported salaries from the 2008/09 DLHE survey with earnings data from the LEO dataset coming directly from HMRC tax records.

In July 2015, the Department for Education and Home Office Border Removals Team agreed a Memorandum of Understanding to share pupil data including names, date of birth, gender, home address and school address for up to 1,500 children a month, from the last 5 years of their records, for various purposes of direct interventions.

In October 2017, the Department for Education confirmed in an interview with Sky News that, information obtained from the National Pupil Database was used to contact families to "regularise their stay or remove them".

An expansion of the Alternative Provision census starting in January 2018, will add further sensitive data to the National Pupil Database including pregnancy, physical and mental health, and a code for young offender, as reason for transfer out of mainstream education. The AP Guidance 2017-18 indicates that the age group has been lowered. "Within the AP census, pupils should be aged between 2 (as at 31 December 2017) and 18 (at 31 August 2017) - those pupils born between 01/09/1998 and 31/12/2015."

Since legislation changed over time to permit new uses and access to personal data by new third parties, over 15 million people whose data was already in the National Pupil Database and who had already left school pre-2012, have not been informed how their personal data may be used, for what purposes, and by whom.

Data request process

Access is granted through an applications process to the Department for Education Education Division and internal Data Management Advisory Panel (DMAP), and is subject to requesters complying with terms and conditions imposed under contractual licence arrangements. The DMAP Terms of Reference was first published in July 2016 by the Department for Education.

The Department for Education application procedures for handling requests for data from the National Pupil Database, from March 2012, enabled interested parties to request extracts of data from the National Pupil Database (NPD) using forms available on the Department for Education website. Data supply agreements, agreement schedules and individual declarations for researchers and third-party organisations who have received DfE approval for applications for data extracts are completed before users are sent the password protected data.

The sensitive and identifying items that require DMAP approval include name, date of birth, postcode, candidate numbers, Pupil Matching Reference (Non Anonymised), detailed types of disability, indicators of adoption from care, reasons for exclusions (theft, violence, alcohol etc).

There is no ethics committee review for the release of identifying or sensitive data directly from the National Pupil Database by the Data Management Advisory Panel or Education Division.

There has never been any privacy impact assessment of the National Pupil Database, and ongoing expansions continue without.

Legal basis for the release of pupil level data by the Department for Education

Some of the history behind its collection, use and changes to legislation are outlined in a presentation given at an Open Data Institute ODI Friday lunchtime talk: Getting to grips with the National Pupil Database in 2013. (Soundcloud licensed under a Creative Commons License.)

The release of data permitting pupil level release of individuals’ identifiable data to third parties from the National Pupil Database was updated by 2013 changes to legislation. Section 114 of the Education Act 2005, and section 537A of the Education Act 1996, together with the 2009 Prescribed Persons Act, were amended in 2010 and 2013, to allow the release of individual children’s data to third parties. Which data items are involved is based on the 2006 Act around the register data a school must hold, which has subsequently had many amendments.

The Data Protection Act 1998, in particular, Principle 1, sets out a fairness obligation which cannot be set aside merely because of the presence of a legal basis such as a Statutory duty. On October 1, 2015, this latter point was again made explicit for public bodies in the judgment of the Court of Justice of the European Union in the Bara case (C‑201/14) in which it ruled that “[the Directive] must be interpreted as precluding national measures…which allow a public administrative body of a Member State to transfer personal data to another public administrative body and their subsequent processing, without the data subjects having been informed of that transfer or processing,” i.e. individuals must be informed when public bodies share personal data and why.

For sensitive data (Tier 1 and Tier 2 of the National Pupil database include all the data items classified as ‘sensitive’) an additional condition from Schedule 3 of The Data Protection Act 1998 must also be met to justify a legal basis for disclosure. These conditions are a high bar, for example, in the interests of justice.

The Data Protection Act 1998 (s33) gives research exemptions for the purposes of statistical and historic research purposes, most significantly on the principles of indefinite retention and data minimisation, as well as Subject Access rights, for as long as data are processed for the legitimate interests of the Data Controller. To qualify for the research exemption, the research must be able to comply with the following ‘relevant conditions’:

(a) that the data are not processed to support measures or decisions with respect to particular individuals, and

(b) that the data are not processed in such a way that substantial damage or substantial distress is, or is likely to be, caused to any data subject.

Campaigners from the children's privacy NGO defenddigitalme, have questioned whether this legal basis is met for some releases between 2012 and 2017 from the National Pupil Database and whether new uses put the research status of the National Pupil Database at risk.

As observed in 2014 by independent experts, "the central concern is that parents and pupils themselves are not sufficiently aware of the way the data is being shared with third parties." "There appears to have been no concerted effort to bring the consultation or the NPD initiative to the attention of parents or pupils."