Ever since I nervously downloaded a file named FOI2009.zip from a rather dodgy sounding Russian server one November night last year, there has been a nagging question troubling me. When the terms of reference for the Russell Inquiry were announce it seemed certain that, whatever shortcomings this so-called independent inquiry might have, an answer to that question must be forthcoming,
Examine the hacked e-mail exchanges, other relevant e-mail exchanges and any other information held at CRU …..
Review CRU‘s compliance or otherwise with the University‘s policies and practices regarding requests under the Freedom of Information Act (‗the FoIA‘) and the Environmental Information Regulations (‗the EIR‘) for the release of data.
The FOI2009.zip file contained the Climategate files, and the question was, where did it come from? I did not, of course, expect that the Russell Inquiry would reveal the precise circumstances of the leak or hack unless the Police investigation was over. However determining whether a folder with the name FOI2009, or one with the same contents as the file on the Russian server, existed on a University of East Anglia computer must surely be one of the main stepping-stones to unravelling just what the great scientific scandal known as Climategate was all about. If it did exist on a CRU server, then there would be many questions to ask about how and why it was created. No credible investigation of the university’s handing of FOI requests could be carried out otherwise.
From what I can see in the Russell Report, that mystery still persists.
The FOI2009.zip file contains two folders, Mail and Documents, the former containng the notorious emails and the latter consisting of a great deal of research material. As anyone who has made up a .zip file will know, most applications that do this assign a default name to the new file that is the same as the file or folder that is being zipped, but with the .zip file extension. So it would seem likely that the FOI2009.zip file may share it’s name with its parent folder.
Here is what Chapter 4 of the Russell Report, Context of the E-Mails, has to say about the content of the Mail folder in FOI2009, but without any mention of the filename:
2. The information released comprises a very small (less than 0.3%) subset of files which were held on the back-up server at CRU, which include e-mails and other documents - such as text files, Word documents, Excel spreadsheets, PDF documents, and computer code.
3. The focus of this Chapter is e-mails which spanned the period 7th March 1996 to 12th November 2009. The ‘primary’ e-mails number 1073 in total with 166 authors. There are more e-mails and authors if the associated e-mail chains are included. When printed on A4 paper the e-mails run to 3,375 pages and contain many embedded duplicates. Self-evidently each of the primary e-mails was either sent by or received by CRU members, but this is not the case for many of the associated e-mail chains.
And:
1. Recognising that the e-mails improperly released into the public domain represent only a tiny fraction (less than 0.3%) of the e-mails archived by the key individuals in the CRU, the Review team sought to set these in context. The backup server (CRUBACK3) had been taken as evidence by the police as part of their own investigation and was held by police contracted forensic investigators. A full context could only be established by some form of access to the information held on this server.In seeking to gain this access a number of legal issues arose, notably that: [my emphasis]
There can be no doubt that the amount of data that confronted the Russell panel when they began their work was daunting. According to the information available in this section of the report, (and in a report form an ‘independent’ consultant on the inquiry website; see below) there must have been getting on for half a million email files on the server. Anyone who has tried to cope with the 1073 files that were released may feel some sympathy for them, but that’s the way the cookie crumbled and the TOR of the inquiry make it quite clear that they were to investigate whatever information was available on the backup server.
Systems analysts and networking people commenting on blogs seem to think that it is unlikely that a hacker or whistle blower could have spent sufficient time logged on to the UEA server to extract the contents of the mail folder from the vast quantity of files available there without being detected. It seems very much more likely then, that someone quickly snatched a folder, or folders, that looked as if their contents might be compromising. That folder may have had the name FOI2009, or it is possible that two folders were found that had the same content as the sub-folders in FOI2009.zip on the Russian server.
It is reasonable to suppose that an inquiry with the TOR that SIr Muir Russell was given would be able to answer this question. However monumental a detailed examination of the content of all the emails on the backup server might be, discovering whether FOI2009, or its constituent sub-folders, were present on the server - and just there for the taking - would seem fairly straightforward.
Steve McIntyre has an excellent post at Climate Audit entitled, The Botched Examination of the Back-Up Server This is concerned with the failure of the Russell Inquiry to make a proper search of the backup server that the emails came from, but in a rather different context to what I am discussing here. Nevertheless his analysis of what happened is very useful. It points to a certain lack of rigour, or even vigour, on the part of the review panel.
In Chapter 4 of the report, Context of the Emails, we find that:
10. The presumption is that e-mails were selected to support a particular viewpoint. Recognising that they were a tiny fraction of those archived, the Review Team sought to learn more about the full contents of the back-up server. This attempt, summarised in Appendix 6, was largely unsuccessful due to the sheer scale of the task and ongoing police investigation.
To say that this exercise was ‘largely unsuccessful’ exhibits a level of understatement that is breathtaking. The evidence in the Russell Report, and in the ‘independent’ analyst’s report, makes it clear that they failed to learn anything at all from the backup server because they didn’t even find out what was on it, let alone attempt any analysis.
There is no doubt that searching CRUBACK3 posed problems for the inquiry panel, and that these extended beyond the mere volume of data. The server and its contents were evidence in an ongoing police investigation and in the possession of Norfolk Police (Appendix 6, para 1), but as Steve M points out in his post, the Russell Inquiry seem to have been in no hurry to negotiate means of access to the data with the police that would not prejudice a possible prosecution.
The delay does not seem to have been the fault of the police. Although Sir Muir Russell was appointed, and his terms of reference set out, on 3rd December 2009, and the first meeting of his panel was in early January, no decision was taken to contact the police and get to grips with the material on the server until 30thMarch, four months later. The target date for publishing the report was ‘Spring 2010′.
The Russell Inquiry’s terms of reference (see above) make it absolutely clear that examining information on the server was one of the primary tasks with which they were charged. And so was considering the way in which the university had dealt with FOI requests. As we will see, the origins of FOI2009 has an important bearing on this.
Eventually, it was agreed that the police would arrange for the data to be extracted from the server and that it would then be examined by a specialist forensic computer analyst in secure premises. This caution when dealing with evidence in a criminal inquiry is quite understandable, but what happened next was less so.
Instead of the Russell Inquiry appointing and commissioning an analyst, this was done by the UEA, the institution that was being investigated. The universtiy’s choice was Professor Peter Sommer who has connections with the London School of Economics and the Open University. I have no reason to suppose that Professor Sommer is anything other than a first rate expert in his field and a person of unimpeachable integrity. On the other hand, it is rather strange that an ‘independent’ inquiry panel, that might expect their modus operandi exposed to very critical scrutiny by sceptics, would allow the organisation under investigation to appoint a consultant to carry out such a vital task, and that the said consultant should be an academic when it is academics who were being investigated. But then things get even stranger.
On 26th March, prior to the decision to contact the police, Sir Muir Russell and another member of the panel had a meeting with Professor Edward Acton, the Vice Chancellor of the university. This is what the minutes say:
5. We explained that we were considering whether it would be practicable to have some work done to search the CRU server (held by the police) with a view to obtaining a fuller understanding of the basis of the selection of the e-mails that were the subject of the unauthorised release. Professor Acton was strongly supportive. He was concerned that due attention be given to the fact that this could raise Data Protection issues because more personal data, possibly relating to more people, might be involved; and the sheer practical burden of handling the possible outputs of the work had to be considered. We agreed to reflect on those concerns and return to the matter, possibly with a more focused proposition, when Jim Norton and I made our next visit the following week.
The Russell Report gives a little more information about the legal concerns:
In the opinion of UEA‘s legal advisers, unconstrained access to the contents of e-mails on the server by the Review would raise potential privacy and data protection issues.
This flags up another question. The inquiry panel would seem to have felt the need to approach the Vice Chancellor before searching the server although it was in their terms of reference, set out by the university, to do so. Why would they do this? Evidently Acton’s response was laudable enthusiasm swifty followed by dumping a couple of obstacles in the way. Note that the concerns about data protection were not raised by the review panel, but by the university of East Anglia’s solicitors. I’ll come back to this later.
Turning to the analyst’s report mentioned in the quotation from the Russell Report (see above) this makes interesting reading. Here is the first sentence:
In November 2009 a number of emails relating to the work of the Climatic Research Unit (CRU) at the University of East Anglia (UEA) appeared on various websites and were subjected to hostile interpretation on various “blog” sites and in the mainstream media.
Am I alone in thinking that there is something strange in an ‘independent’ forensics expert, who has been brought in solely for the purpose of ferreting out data on a backup server, should feel the need characterise the initial reaction to the Climategate emails as ‘hostile interpretation’, or to enclose the word blog in quotes as thought it was emitting a bad smell? Sorry, that’s a bit of a digression, but it did rather catch my eye.
Here are a few more extracts from Professor Sommer’s report to the panel:
I have at this stage no knowledge of the technical means by which the emails were acquired from the CRU.
…..
I have been supplied by the University with a “thumb drive” said to contain copies of all the emails known to have been published on the websites. I have also been supplied by Norfolk Police with three further “thumb drives” containing the emails extracted from the back-up server associated with the computers of the researchers in question. The extraction was carried out by Qinetiq, as contractors to Norfolk Police.
…..
The material has been given a very high level of security classification which requires that I work at secure facilities and follow particular protocols which, for example, preclude computers being left to run unattended or overnight and at weekends. These procedures, while providing a very high standard of protection to the data, are also very time consuming, particularly in the light of the need for the Review to conclude its work in a timely manner.
…..
The emails as provided to me are in the format of an email program called “Thunderbird” and before they can be searched require indexing. There are large numbers of un-indexed emails and time constraints in preparing this initial report preclude indexing and any form of sophisticated analysis.
…..
I am told there are some 1,073 primary emails and their associated threads in the published material, which itself amounts to 18.7Mb.[1]
From this it appears that Sommer had not been briefed about how the hack or leak was carried out, nor had he formed any opinion on his own initiative. Perhaps this is not surprising in view of the ongoing police investigation. On the other hand his description of the version of the ‘hacked’ emails that was provided on a ‘thumb drive’ by UEA does seem rather cautious, particularly as it is not clear whether the data was in it’s original form as leaked on the net, which was of course a sub-directory of FOI2009.zip. Surely he would want to see this file? The last extract from Somer’s report makes it clear that the university did not give him a copy of FOI2009.zip as, if they had, it would not have been necessary to have tell him that there were 1073 email files; he could have done a file count himself in less than a second.
The working conditions imposed by the police were certainly inconvenient, but understandable. It would have been irresponsible of them to take any risk that might compromise what is apparently a major inquiry that has involved help from specialists in other forces. Indexing and searching large volumes of data - and other parts of Sommer’s report make it clear they are large - is time-consuming even on a fast computer. Not being able to leave machines running unattended while this takes place was certainly a problem in terms of manpower, but by no means an insuperable one if there was a will to solve it.
What really stands out in the analyst’s report is that the data was not extracted by Sommer, but by Qinetic, on someones instructions, but we are not told who. So what were those instructions, and did anyone ask Qinetiq to keep an eye open for a folder with a name like FOI2009, or a couple of folders that contained the same data as the Mail and Data sub-folders in FOI2009.zip of dubious Russian server fame? Neither the Russell report nor Sommer’s report seem to give a clue.
Here is how the ‘independent analyst’ concludes his report:
The processes of analysis to identify (and then review) additional email traffic which might be associated with the issues which are the subject of the allegations which have been levelled against CRU, is likely to take at least several weeks. It would be for the Review Team and the University to determine whether the cost, inevitable time delays and (at this time) uncertain outcomes could be justified. Until the material is subjected to a much downgraded security level, the likely position will be that the University and its appointed team will not be able to carry out any meaningful analysis.
And let’s take another look at what the Russell report has to say:
10. The presumption is that e-mails were selected to support a particular viewpoint. Recognising that they were a tiny fraction of those archived, the Review Team sought to learn more about the full contents of the back-up server. This attempt, summarised in Appendix 6, was largely unsuccessful due to the sheer scale of the task and ongoing police investigation.
So the Russell Inquiry decided to call it a day, in spite of examination of the data on CRUBACK3 being one of the primary requirements in their terms of reference. Evidently it would just take to long and be too much trouble. And then there is the terrible problem of the UEA’s legal adviser’s concerns about data protection legislation. Did the review panel get independent legal advice on this from a source free from the very obvious conflict of interest that the university lawyers would have? There is nothing in the report to suggest that they did.
So the Russell report throws no light whatever on where FOI2009 came from and that matters a very great deal because of the obligation that the TOR put them under to consider the university’s handling of FOI requests. If that folder, or any of its contents was present on the backup server when the hack or leak took place, then it obviously means that the content was compiled by someone at the university. Why would anyone make up such folders containing apparently embarrassing material? Surely that is something that an inquiry charged with reporting on the university’s handling of FOI requests would want to know.
Looking at the problem from another standpoint, if the Russell Inquiry was, as some sceptics suspect, an exercise in clearing the CRU’s name, then if they had established that the information that was leaked or hacked had not already been gathered together on the server, then surely that would be something worth mentioning?
Perhaps the police told the inquiry panel that even to mention FOI2009 would prejudice their investigations, but there is no caveat in the report to the effect that there were aspects of their inquiry that they were unable to mention for legal reasons. There is no such caveat.
So the mystery of FOI2009 not only remains, but it has been extended and deepened. Not only do we not know whether this folder, or its sub-folders, were lifted complete from CRUBACK3, but there is the added conundrum of why the Russell Inquiry seems to have had no great interest in finding out.
One last thought. While the police inquiry continues, and they have possession of CRUBACK3, the integrity of the data on it is beyond dispute and available for inspection. What happens when the inquiry ends?
[1] I’m no propellerhead, but the Mail folder from FOI2009 takes up less than half that space on my HDD. I realise that this may just be the difference between the files being in .txt format rather and the original Thunderbird format, but I would be interested in any views on this.
