1 Thursday, 22 March 2018 2 (9.30 am) 3 CHAIR OF THE INQUIRY: Good morning. As you will be aware, 4 this is the first anniversary of the incident at 5 Westminster where several people lost their lives and 6 others were injured. I ask you all to stand for 7 a minute. 8 Mr Lake. 9 MR LAKE: My Lord, the first witness today is 10 Professor Flyvbjerg. 11 PROFESSOR BENT FLYVBJERG (sworn) 12 Examination by MR LAKE 13 CHAIR OF THE INQUIRY: You are going to be asked some 14 questions by Mr Lake, the Counsel to the Inquiry. If 15 you just listen to the question and answer it as 16 directly as possible, and if you don't understand the 17 question or wish clarification, just say so. 18 Can you speak clearly into the microphone so 19 everyone can hear your evidence, and slowly so that the 20 shorthand writers can record it. 21 A. I will do my best, thank you. 22 CHAIR OF THE INQUIRY: Thank you. 23 MR LAKE: My Lord. 24 Professor Flyvbjerg, could you state your full name, 25 please? 1 1 A. My full name is Bent Flyvbjerg. 2 Q. I think you are the BT Professor and inaugural Chair of 3 Major Programme Management at Oxford University, Said 4 Business School? 5 A. That's correct. 6 Q. And also a professorial fellow of St Ann's College, 7 Oxford? 8 A. That is correct. 9 Q. I think you have been instructed by the Inquiry to 10 provide a report into various issues of risk and 11 optimism bias? 12 A. Yes. 13 Q. I would like you to look at a document, please. It 14 should be in hard copy in front of you, and also shown 15 on the screen to your right. It's reference 16 TRI00000265. 17 Is that the report that you prepared for the 18 purposes of this Inquiry? 19 A. Yes, it is. 20 Q. Are you content that its contents be accepted as your 21 evidence given under oath on this Inquiry? 22 A. Yes. 23 Q. Thank you. 24 I don't intend to go through it page by page, but to 25 consider certain of the issues that arise. Perhaps at 2 1 the outset, we could look at page 41 of this. 2 By way of summary, we can see that your report also 3 includes your CV and details of your experience and 4 expertise? 5 A. Yes. 6 Q. If we go forward to page 53, under the heading "Advisory 7 Roles for DfT and HMT", the Department of Transport and 8 Treasury, we can see the first thing you have noted 9 there is the guidance document on optimism bias 10 procedures in 2004? 11 A. Yes. 12 Q. I think that's something that you consider in the body 13 of your report? 14 A. Yes, it is. 15 Q. If we put the report aside for the moment and take that 16 off the screen, before turning to the questions of 17 optimism bias, I would like to ask you a few questions 18 about risk more generally to inform the Inquiry. 19 First of all, could you explain what is a quantified 20 risk analysis? 21 A. Yes. A quantified risk analysis is basically a register 22 of risk, specific risks that might happen to a project 23 or an investment, and also probabilities of those risks 24 actually occurring and the impacts that the risks might 25 have. So a risk register, a list of probabilities, and 3 1 a list of the impacts that the probabilities might have 2 if they occur. That's basically a quantitative risk 3 assessment. Those are the basics of a quantitative risk 4 assessment. 5 Then on top of that, something that is called Monte 6 Carlo simulations are done based on the elements that 7 I just mentioned. A quantitative analysis called 8 a Monte Carlo simulation is done in order to assess the 9 likelihood at different levels, or the 90 per cent 10 likelihood that something might happen, or the 11 50 per cent likelihood that something might happen. So 12 that's a probability distribution that's generated by 13 this Monte Carlo simulator. 14 Q. If you can just break down some of the elements of that, 15 you started with identification of risks, and inclusion 16 of those in a schedule or a table. 17 How is that done? Who does that in a project? 18 A. The way it's typically done in the quantitative risk 19 assessment is by a panel of experts. So you basically 20 get a panel of experts in a room and they fill out 21 a questionnaire or several questionnaires about what 22 they think are the likely risks to happen to the project 23 or investment at hand, and what the probabilities are, 24 what they think the probabilities are that this would 25 have, what they think the impacts are going to be and so 4 1 on. So you get a range of experts to do this, and then 2 you aggregate the surveys that the experts have done, 3 the numbers that they have put in there, and on that 4 basis you calculate what the risks are according to 5 these experts. 6 CHAIR OF THE INQUIRY: Could I ask you, Professor, to speak 7 a little more slowly for the sake of the -- 8 A. Sorry about that. 9 CHAIR OF THE INQUIRY: Not at all. 10 MR LAKE: Presumably the advantage of having more than one 11 person identifying the risks is to try and minimise the 12 chance that a risk is overlooked? 13 A. Yes, it is. So you want a range of different experts 14 who cover different areas of expertise in relation to 15 the project that you are assessing. And like you said, 16 having a range of experts will give you a broader view 17 of what the risks might be. 18 Q. Although you have a broader view, is that approach in 19 any sense foolproof or is it in practice the case that 20 risks slip through the net? 21 A. Risks might slip through the net, but that actually is 22 not the major problem with QRA, quantitative risk 23 assessment. 24 Q. What is the major problem? 25 A. The major problem is that it has been documented that 5 1 when you assess risk like this, based on the subjective 2 valuation of experts, there will be a bias in the 3 outcome. 4 It turns out that everybody, including experts, are 5 born with biases or develop biases throughout their 6 life, and one basic bias is optimism. So we all tend to 7 be basically optimistic. That's a human thing, and 8 people sometimes think if somebody is an expert in 9 a area, then they are realistic, they are not biased. 10 They don't have the optimism bias. That's actually been 11 documented in research over and over not to be the case, 12 that experts are biased just as laypeople are biased. 13 So what you get is a biased outcome of the 14 quantitative risk assessment when you use this approach. 15 Q. How does that bias you describe there affect the 16 outcome? What's the practical effect? 17 A. So, for instance, if you envision the experts trying to 18 assess what it is going to cost to do a project, if you 19 are optimistic in doing a cost assessment, then you 20 assess that it's going to cost less than it actually 21 turns out to cost. That's optimism. If you are doing 22 a budget and you are optimistic, you will have a low 23 budget, and that will come back as a cost overrun later 24 in the project when it's being built. 25 Q. Looking at that, at the application of that bias in the 6 1 context of the QRA, if you have this panel of experts 2 sitting, how will that bias affect their prediction of 3 the possible financial consequence or time consequence 4 of risk? 5 A. So in each -- I was just using cost as an example, and 6 it can be anything. Like you said, it could also be the 7 timetable, or it can be environmental impact or safety 8 impact, whatever. 9 The way the optimism bias affects things is 10 generally that the costs and risks will be 11 underestimated, because people are optimistic about the 12 cost and the risks, whereas the benefits and the 13 opportunities will be overestimated because that's how 14 optimism plays into benefits and opportunities. We 15 think things are going to go better than they do in 16 general. 17 Q. You said for the quantified risk analysis, as well as 18 identification of the risks, and identification of the 19 possible consequences, there was consideration of the 20 probability of that risk arising. Is that too affected 21 by the optimism bias? 22 A. Yes. So something negative would be -- you would 23 underestimate the risk of that, and something positive, 24 you would overestimate the risk of that, according to 25 the optimism bias. So yes, it does affect the outcome 7 1 of probabilities. 2 Q. We've also heard in the evidence given to the Inquiry so 3 far that when there is a risk analysis, a risk schedule 4 for the project, mitigation measures are proposed and 5 the effectiveness of those is assessed. Is that too 6 something that's affected by optimism bias? 7 A. Yes, it is. Optimism bias is pervasive. It's something 8 that is hardwired into our brains, and it will therefore 9 affect any assessment, including that. 10 Q. Leaving that aside for one moment, you also mentioned 11 Monte Carlo software. Can we be clear that what 12 information is put into the Monte Carlo software, what 13 goes in? 14 A. The input to the Monte Carlo software are the -- are the 15 assessments of the experts in the QRA. So that is the 16 input. Those are the data that the Monte Carlo 17 simulation works on the basis of. 18 Q. What is the output from that software? 19 A. The output is a probabilities -- probability 20 distribution that has been simulated by the software on 21 the basis of that data. 22 Q. Is this software or programme similar to it in general 23 usage? 24 A. Please say again. 25 Q. Is that software used generally, it's accepted that it 8 1 works? 2 A. Yes, it is. It's used worldwide and it's used in the 3 financial sector as well as in project management, which 4 is what we are talking about now. It's a very 5 widespread methodology that is used in a lot of places. 6 Q. So you said it gives you the different probabilities of 7 risk situations arising. We've seen in your report 8 a number of times there's references to P values, P50, 9 P80, P90. What are they? 10 A. They are shorthand for how likely it is to stay within 11 budget or within schedule or within whatever the 12 variable is that you're looking at. So a P90 for the 13 budget means that you have 90 per cent likelihood of 14 staying within budget. And therefore only a 10 per cent 15 risk of going over budget. So that's what the P90. 16 If it's P50, it means that you have a 50 per cent 17 likelihood of staying within budget and a 50 per cent 18 likelihood of going over budget. 19 Q. Just to stay with that P50, just to be clear about what 20 it means, if you're using a P50, it's as likely to be 21 over budget as it is to be under budget? 22 A. Exactly. It's 50/50. That's also called the median. 23 Q. We see also reference to P mean. Is that different to 24 the median? 25 A. Yes. So the P mean is the average, and the average and 9 1 the mean might be the same or they might be different. 2 In this type of project they are typically different. 3 Q. Looking at what the P figure means, taking a higher P 4 number like P80, it's clear from what you said that that 5 still means there's a 20 per cent chance that that 6 budget figure would be exceeded, that risk figure would 7 be exceeded? 8 A. That's correct. 9 Q. Does the quantified risk analysis or the probability 10 software, Monte Carlo analysis, give you any idea by how 11 much it might be exceeded? 12 A. It gives you the probability that it would be exceeded 13 but you need to combine that with the actual size of the 14 budget to find out how much. 15 Q. If we have all these different probability figures, P50, 16 80, 90 or P mean, how does a project choose which is the 17 correct one to use or the appropriate one to use in 18 their circumstances? 19 A. A project decides that by deciding how willing they are 20 to take risks. 21 So if you are willing to take a higher risk, let's 22 say if you are a portfolio manager and you have many 23 projects, you win some, you lose some. Some will come 24 in over budget, some will come in under budget, and they 25 compensate for each other. They even each other out, 10 1 because you have a whole portfolio. 2 Then you would actually want to use a P50 and accept 3 the 50 per cent risk that some projects will go over 4 budget and some projects will go under. But it's up to 5 you to decide, and that's your question really. If 6 you're a very conservative management or owner of the 7 project and you -- maybe you are only doing this one 8 project and it's very important for you not to go over 9 budget, then you would absolutely not be satisfied with 10 a P50 that is a 50 per cent risk of going over budget. 11 You would want to go for P80. So that you have 12 80 per cent chance of staying within budget or even 13 a P09 or even P95. We see that. There are actually 14 projects that are, you know, billion pound projects 15 where it would be a disaster to go substantially over 16 budget; where the owner wants a P95. So 95 per cent 17 assurance to stay within budget. But that's really 18 a decision by the owner or the project team delivering 19 the project to decide what they want, and after they 20 have decided the Monte Carlo simulation, or whatever 21 probability of execution that it is that you refer to, 22 would then tell you what the budget is that you need to 23 have. 24 It's like buying insurance. Most people understand 25 what it means to buy insurance and they understand, if 11 1 you want a higher level of security in your insurance, 2 it costs you more. It's exactly the same with this. If 3 you want a high level of security, that you stay within 4 budget, you need a higher contingency, and therefore 5 your budget will be larger and it will cost you more. 6 Q. Obviously the prediction figure used, the probability 7 figure makes no difference to the actual outturn of the 8 project. It does make a difference to how much 9 confidence you can have that you're not going to exceed 10 your budget amount? 11 A. That is correct, but that in itself might actually also 12 affect the outturn of the project because -- 13 Q. In what way? 14 A. In the way that if you don't stay within budget, you 15 have political scandal, you'll be on the front page of 16 the media, and all of a sudden the project team that is 17 supposed to deliver the project will be preoccupied by 18 putting out fires in the media, and applying for more 19 funding or raising more funding for the project, because 20 they don't have enough money now evidently because 21 there's an overrun. And that will distract them from 22 what they are supposed to do, which is deliver the 23 project, and the quality of delivery would therefore 24 suffer, which in itself will affect the cost and we see 25 this vicious cycle happening with projects once that 12 1 starts. So it actually may affect the outcome of the 2 project in that way. 3 Q. That situation that you are describing there would arise 4 where there has been an insufficient allowance, or the 5 figure has not been set high enough to cope with the 6 probability? 7 A. Yes, exactly. 8 Q. In an earlier answer you gave to me, you contrasted on 9 the one hand an organisation with a portfolio of 10 projects and on the other a particular project promoter 11 who is really only dealing with one project. What's the 12 difference between these two situations in terms of 13 choice of probability? 14 A. So if you have a portfolio, you would want to choose 15 a P50 value basically. So you would accept the 50/50 16 risk of cost overruns because half your projects will go 17 over budget, half your projects will go under, and on 18 balance they will tend to cancel each other out. 19 Whereas if you have only one project, you will 20 typically want a much higher P value, meaning a much 21 higher likelihood of staying within budget because you 22 have one -- only one project. Often a large project 23 that can have negative impacts on your organisation if 24 it goes over budget. 25 So you choose a -- P80 is a typical value that we 13 1 see more conservative clients using, and like I said 2 before, even P90 and P95 for ultra-conservative clients. 3 Q. Presumably, as you move from P80 to P90 and beyond, 4 there are substantial increases in the allowance that 5 has to be made as you get higher and higher? 6 A. Yes. So we call that increasing marginal cost of 7 insurance, and that applies here. So it becomes more 8 and more expensive, the further towards 100 you get, and 9 it becomes very expensive. That's why we haven't seen 10 anybody going beyond P95 yet, because it would be 11 incredibly expensive to get the last 5 per cent. 12 Q. What you are describing there is very much the interests 13 of the person making the allowance for risk, what 14 their -- that body's individual appetite for risk is. 15 Is it likely to be a situation there where, within 16 a particular project, different parties will have 17 different appetites for risk and therefore use different 18 probabilities? 19 A. That's a possibility, and you might even want to do that 20 deliberately, give different actors within the delivery 21 of a project different budgets. 22 So the project owner at the top might want to have 23 reserves available that can be used in an emergency or 24 in a tight situation where there's not enough money or 25 something unexpected happens, so they don't have to go 14 1 back and ask for more money. So they have the 2 contingency. But they might not want to give that 3 contingency to the project director. So the director 4 has actually a lower budget. So we have one budget and 5 one contingency at the owner level, and then the project 6 director has another budget and another contingency that 7 is somewhat lower, and you might even have a third 8 budget which is what you give to the contractors. 9 You want to put pressure on the budget, so to speak, 10 because there is a tendency, if the money is there, it's 11 going to be spent, so you really need to manage the 12 contingency carefully, and this is a way to do that, 13 operating with three. We've even seen some projects 14 with five different contingencies and therefore five 15 different budgets allocated to different levels of the 16 delivery organisation. 17 Q. That brings up a particular issue I was going to ask you 18 about. 19 If a higher probability figure is used, and so 20 a greater allowance is made for risk, is there a danger 21 that once that money is there, it will end up being 22 spent on the project because everybody knows it's there? 23 A. That is a real risk. And you need to have measures in 24 place to avoid that happening. If you're not -- if you 25 don't carefully manage your contingency, the likelihood 15 1 is that it will be spent and you could actually have 2 a worse situation than if you had had no contingency, 3 because the contingency in itself becomes a carrot that 4 will drive spending. And you might end up having 5 a contingency, and then you need to put extra money on 6 top of that in the end, and the overall project becomes 7 more expensive than it would have otherwise been. 8 Q. What sort of measures would you commonly come across to 9 try and restrict the allowance -- avoid the situation 10 where the allowance ends up being spent, simply because 11 it's available? 12 A. So the most efficient way is to create an incentive 13 structure where the people who are delivering the 14 project are actually incentivised not to spend the 15 contingency. 16 You can do that by creating positive incentives that 17 if the -- if the builders are delivering to budget or 18 below budget, they actually get extra payment, they make 19 more money by delivering below budget or on budget. 20 That's a positive incentive. The other is the 21 stick. So you use both sticks and carrots. The first 22 was the carrot and the second is a stick that you punish 23 builders if they go over budget. That's the incentive 24 structure, and it's really important to think carefully 25 about that, and we find that good clients do that, you 16 1 know, mature clients who know how to deliver projects 2 and do it successfully, they have clear incentive 3 structures and partnerships that will incentivise 4 everybody involved to deliver on or below budget or on 5 or below schedule or at or above benefits, revenues, 6 whatever the relevant variables are. 7 So that's one way of doing it. 8 Another way is to make it very difficult to get 9 access to the contingency. So you don't just put it out 10 there. You actually put it with the Treasury, for 11 instance. So this is what the UK Government does for 12 very expensive projects that are funded out of the 13 Treasury. Or ministers. They will have part of the 14 contingency locked up in the Treasury, and you actually 15 have to go there and you have to meet certain criterion, 16 et cetera, in order for the contingency to be released. 17 So that's the second thing. Just to make it difficult 18 to get access to the contingency and to be clear about 19 what the criterion are for the contingency to be 20 released. 21 Q. Turning to the first of the -- returning to the first of 22 the two things there, the idea you said about 23 incentivisation, by sharing the benefit of cost 24 underruns and sharing the burden of cost overruns as 25 well. Would that mean that if the project went over 17 1 budget, the contractor had to bear part of the overrun 2 as well as the employer? 3 A. Yes. 4 Q. If it's under, they both share the benefits of that too? 5 A. Yes. 6 Q. That sort of approach seems slightly at odds with the -- 7 any desire to have a price that's absolutely fixed at 8 the outset, a fixed price contract? 9 A. Yes; correct. 10 Q. So putting that the other way round, is it fair to say 11 that if you have a contract where you insist on an 12 absolutely fixed price, it creates a situation where you 13 lose the incentive? 14 A. Very often with the -- yes, is the answer. And very 15 often with the fixed price you only use the stick. So 16 people get punished if they go over the budget, and so 17 that there might be fines, but also just the fact that 18 if the builder is actually carrying the whole risk, you 19 know, then it's going to hurt the builder that if they 20 go over budget because they actually have to cover that. 21 Q. Or if the way -- if the effect of the contract is it's 22 fixed price, does that in your experience act as 23 an incentive for the contractor to try and claim 24 variations, to drive the price up to get additional 25 money? 18 1 A. Yes, and I would actually say in general, in the vast 2 majority of cases, the notion of a fixed price contract 3 is a theory. It's an illusion that doesn't materialise 4 in reality. 5 It's called a fixed price contract, but it never 6 ends up being fixed price or almost never ends up being 7 fixed price. There are variations, and other things 8 that happen on big and complex projects like what we are 9 talking about. It's impossible to predict everything 10 that can happen, and therefore everything is not in the 11 contract. 12 So situations will arise where things happen where 13 it becomes an open question. Who is going to pay for 14 this? And a builder will say: it's not us; and maybe 15 the owner will say: it's not us; and then you have 16 a conflict, and often arbitration or even going to 17 court. 18 Q. Returning to the question of the appetite for risk from 19 the various parties, and looking in particular at 20 something like the Edinburgh Tram Project, where we have 21 a promoter, a local authority on the one hand, and 22 a funder, which is the Scottish Central Government on 23 the other, would you expect them to have the same or 24 different appetites for risk? 25 A. Probably different appetites for risk with the most 19 1 conservative stance taken the higher you go. 2 Q. If the project, instead of being carried out as 3 a one-off project by local authorities, is carried out, 4 and this would apply in Scotland or England, anywhere, 5 by a central government which is undertaking lots of 6 projects, does that make it easier to manage the risk 7 and to bear the risk of overruns? 8 A. Yes, it does. I mean, for one reason, there is a larger 9 budget to contain it within, but also there's more 10 experience in the sense of doing more projects and 11 having tried this before, and therefore having the 12 skills and the expertise that it takes to deliver 13 a project. 14 Q. Just sticking with the subject of experience, I think 15 you are involved now in a project, the academy of 16 project management, which is training people in the 17 various aspects of delivering these very large projects; 18 is that correct? 19 A. That's correct. It's called the Major Projects 20 Leadership Academy. 21 Q. What does that seek to do? 22 A. That seeks to train UK civil servants in how to deliver 23 major projects. 24 Q. And how many civil servants -- when did that start? 25 A. It started in 2012. 20 1 Q. How many people have been trained so far? 2 A. Approximately 500. 3 Q. Does that include people from Scotland -- 4 A. It does, yes. 5 Q. You referred earlier to the question of when you get to 6 very high probabilities that might be desired, something 7 over 95 per cent, the risk goes up dramatically -- the 8 cost of that goes up dramatically. 9 I think that might lead to something you were also 10 making reference to in your report, a black swan. What 11 is a black swan? 12 A. So a black swan used in this context is a metaphor for 13 a statistical outlier. So it's a very high value of 14 something. So if we talk about a cost black swan, it 15 means a project that had a very large cost overrun. 16 A very high percentage cost overrun. 17 Q. I think you have referred in your report to the fact, 18 you explain it in more detail, that the overrun has to 19 be more than 1.5 times the interquartile range of the 20 project assessments? 21 A. That is correct, and that's just a statistical method, 22 you know. That's how a statistician calculates what 23 a statistical outlier is, and then those are the ones 24 that are called black swans. 25 Q. Because you make the point in your report, after 21 1 analysing the fact, that the Edinburgh Tram Project, 2 over budget as it was, does not qualify for black swan? 3 A. That is correct. So when you use the statistical method 4 that you just referred to, you find that for light rail 5 schemes, given the data that we have, the black swan, so 6 the statistical outliers, are projects that had 7 143 per cent or above in cost overruns in real terms, so 8 no inflation included, and for the Edinburgh tram, the 9 cost overrun is 52 per cent in real terms, and obviously 10 that's much lower than 143 per cent. So the Edinburgh 11 tram is not a statistical outlier in that sense. 12 Q. When you've done that analysis, it's not of all projects 13 generally. That is specifically of light rail projects? 14 A. That is specifically light rail, and 23 per cent of 15 light rail projects have higher cost overrun than the 16 Edinburgh tram. 17 So the median is 25 per cent. So Edinburgh, the 18 Edinburgh tram had twice as big a cost overrun as the 19 median of projects. So the median is the middle value. 20 Q. But the average cost overrun is 25 per cent cost 21 overrun? 22 A. Not the average, the median. So 50 per cent is above 23 the median, and 50 per cent are below the median. 24 Edinburgh tram is clearly above the median. It is 25 twice as high as the median. Median 25 per cent, 22 1 Edinburgh tram 52 per cent. But the outliers don't 2 start until 143 per cent, and 23 per cent of all light 3 rail projects for which we have data are above 143 -- 4 no, are higher than the Edinburgh tram. 5 Q. The question of black swans, you explain there, that's 6 assessed purely by the size of the cost overrun. But 7 does that also -- is part of the definition of a black 8 swan dependent on the probability or the likelihood of 9 that occurring? Or is it purely cost related? 10 A. No. You could define your black swan like that, but 11 that's not how it was defined in the calculation we just 12 talked about. 13 Q. Presumably black swans are sufficiently rare, however, 14 that they are not something that would ever be caught by 15 a risk analysis at P50, P80 or even P90? 16 A. Yes. It's true that the black swans are in the very top 17 of the distribution, or in the right-hand side of the 18 tail, as we also call it sometimes, and they would be 19 above 90 or 95 per cent, that would be where you find 20 them. It depends on the specific distribution. But you 21 would not protect yourself against black swans by a P50 22 or P80, that is correct. You would need something that 23 we call black swan management, where you actually go 24 specifically for eliminating the black swans in order to 25 protect yourselves against them. 23 1 Q. What does that involve? What does black swan management 2 involve? 3 A. That involves identifying what are the specific risks 4 that could spin a project so much out of control that it 5 becomes a black swan, and how do you protect yourself 6 against those risks. 7 Q. If that sort of exercise is undertaken, black swan 8 management, does that have an overall effect on risk? 9 A. It does have -- it has an overall effect in the sense 10 that it also affects the median and the averages, but 11 not so much. A little. It has the -- the biggest 12 effect is actually clipping the tail, as we call it, 13 avoiding the very extreme outcomes, and that's the 14 specific purpose of doing it that way. 15 Q. What I'm really coming to with that question is: had 16 black swan management been employed in the Edinburgh 17 project, which wasn't a black swan, is it the sort of 18 thing that might have made a difference to the cost 19 outcome or would it not -- 20 A. It would have had some effect on the cost outcome, and 21 lowered it somewhat. And it would have protected the 22 Edinburgh tram from becoming an extreme outlier. 23 But with the benefit of hindsight, looking at what 24 happened in Edinburgh, in the Edinburgh tram, it would 25 have been more relevant to protect around the median and 24 1 the average than protecting against the extreme tail, 2 because the extreme tail didn't materialise here. 3 Q. In terms of protecting about the median and the average, 4 does that involve anything other than what might be 5 termed the standard risk management techniques? 6 A. Yes. So there's another technique called optimism bias 7 uplifts that has been developed by the Treasury and the 8 Department for Transport in the UK, and that's an 9 additional technique you use, and that is a specific 10 technique you use to protect yourself, not against the 11 tail, but against the mid-range and sort of upper 12 mid-range risks. 13 Q. I now want to turn to ask you more about optimism bias 14 uplifts or the adjustments that might be made. 15 You refer in your report to the inside view and the 16 outside view. Could you first of all explain what you 17 mean by the inside view? 18 A. So the inside view is understanding a project from the 19 inside. You understand the individual elements that the 20 project is made up of, the different parts that make the 21 project, and how they come together to form the overall 22 project. 23 If you're talking about risk assessment, like we 24 talked about before, the QRA, so the quantitative risk 25 assessment methodology is actually a typical example of 25 1 the inside view. You try to understand what are the 2 specific risks that apply to this project, given that 3 the project looks the way it does, and what are the 4 probabilities and impacts, and you add all this up to 5 get an overall estimate of the risks for the project. 6 That's a typical example of the inside view, so 7 understanding the project on its own terms. That's the 8 inside view. 9 Q. Now, from my understanding of your answers earlier, the 10 difficulty with that is the inside view involves 11 a number of exercises of judgement, and all those 12 exercises of judgements are likely to be tainted slightly 13 by optimism bias. 14 A. Correct. 15 Q. So what's the outside view? 16 A. The outside view is, as the term says, looking at the 17 project from the outside. So you basically say: okay, 18 let's find some other projects that look like this 19 project. So in the case of the Edinburgh tram, let's 20 find other trams. Let's find a bunch of trams that have 21 already been built and where we know what actually 22 happens. And then let's assume that, you know, pretty 23 much the same is going to happen to the Edinburgh tram. 24 We take the average. So what happened to these other 25 projects that have already been completed and we'll 26 1 assume that that has given us the best assessment of 2 what is likely to happen to the Edinburgh tram or 3 whatever project it is that you are planning. 4 So you take a reference group of projects that have 5 already been completed, and for which you therefore have 6 historical data. You don't have to assume things. You 7 actually know what happened. The only assumption you 8 make is that this is a good estimate for what's going to 9 happen to the project that we are now planning. That's 10 the outside view. 11 Q. I was going to ask you about that assumption you just 12 mentioned there. The assumption must be that the cost 13 overrun in any project is in statistical terms likely to 14 be the same as or informed by overruns in projects of 15 a similar nature. 16 A. That is the assumption or even better, the assumption is 17 that the best estimate you can get is what already 18 happened to previous projects historically, and you 19 can't better that by thinking that you are clever enough 20 to figure out what is actually going to happen to this 21 specific project that you are looking at. 22 You will not be better at predicting the future than 23 the historical data for the project that have already 24 happened. That's the -- that's -- a body of research 25 that has been done that has proven that, and that is 27 1 therefore -- that is the result that you take into this 2 methodology. 3 Q. That's what I was going to ask. Although I think both 4 you and I have described that as an assumption, it's not 5 just an assumption, it's been verified by research 6 carried out by many people, including yourself? 7 A. Including myself, but also including a Nobel Prize 8 winner in economics called Daniel Kahneman. So it's 9 very solid research. 10 Q. So using the outside view, optimism bias, how is 11 an allowance made using that approach? 12 A. Basically, you look at those previous projects that have 13 already been completed, and as a minimum we recommend 14 that you look at 20 to 30 projects, but the more you can 15 include, the better, because the more reliable your 16 estimate is. 17 But let's say you are looking at cost and cost 18 overrun, you just say for those 20 to 30 projects, for 19 each project, what was the cost overrun. You add that 20 up and divide by the number of projects you have in 21 average. That is your best estimate of the cost overrun 22 for the project that you are planning, unless you have 23 really good reason to think that your project is 24 different from the other projects. 25 That of course is exactly where you have to be 28 1 careful, because that judgement, whether you're better or 2 worse than the projects in your reference class, is 3 where optimism could re-enter your assessment. 4 Q. That sounds like once again an exercise of judgement by 5 the people reviewing a particular project. 6 A. Correct. 7 Q. Whereas if you're simply looking at an analysis, 8 published figures of what the cost overrun has been on 9 similar tram projects, from what you say, there appears 10 to be little or no judgement involved in that. That's 11 the whole point? 12 A. Correct. 13 CHAIR OF THE INQUIRY: When you're doing the calculation of 14 cost overrun and you achieve the average, would you be 15 including in the data that you are going to average 16 black swans? 17 A. Yes. We would. 18 MR LAKE: That comes to a point I was going to raise now. 19 We have heard that some things might not be included in 20 a QRA, and one of the things you mentioned is a black 21 swan. 22 We've also heard that a QRA wouldn't normally look, 23 for example, at a risk that the contract failed to 24 transfer financial risk to the contractor, or failed to 25 properly administer something that had been intended. 29 1 Would that be caught by reference class forecasting, 2 the extent -- the outside view? 3 A. Yes, it would. 4 Q. Because you just look at the overruns in all these other 5 projects and they would make up the statistical averages 6 that would then be used? 7 A. Basically anything that has happened to the projects in 8 the reference class, the historical projects, will be 9 included in the estimate of cost overrun, including 10 a risk that were not taken into account, and actually 11 including the very famous unknown unknowns. 12 So anything that happens to those historical 13 projects would be included in the numbers. So you don't 14 have to imagine it or visualise it. It's already there, 15 and therefore it's also taken into account for the 16 project that you are planning for. 17 Q. Turning for a moment to imagining a fight back against 18 that by people who use the risk approach, they might 19 say: there's a lot of detail goes into their QRA and 20 they get it checked by external people, therefore, it is 21 reliable, and it's not necessary to use other projects, 22 they are better with their own data. What is your 23 response to that? 24 A. My response is that this is correct regarding the 25 detail. So it is an advantage to have some level of 30 1 detail, but having things checked by outside experts 2 doesn't help because like I mentioned earlier, experts, 3 including outside experts, are subject to the optimism 4 bias. 5 The only way it would help to bring in outside 6 experts, if they use a method that is debiased, like 7 reference class forecasting where the optimism bias 8 doesn't have a chance to re-enter, then it would be 9 a good thing. But you could also do that internally. 10 You don't actually need to bring in outside experts for 11 doing that. 12 Q. Presumably, even if you're using the optimism bias 13 reference class approach, it's still appropriate to have 14 a risk analysis and a schedule of risk -- 15 A. Yes. 16 Q. -- mitigations and all these things, because that 17 controls the underlying costs? 18 A. Of course. You need to know that kind of detail when 19 you're actually hands on managing a project. So what 20 I recommend, and what my team recommends, is 21 a combination of the two, that you do your conventional 22 stuff, because it gives you this disaggregate detailed 23 idea of what the risks are. I think it was 219 24 different risks in the case of the Edinburgh tram, and 25 that's very useful to have that kind of overview. But 31 1 then you actually adjust that by the optimism bias 2 uplift. You know what is the empirical bias and this 3 kind of thing from your historical data, and then you 4 simply adjust what's in the conventional QRA. 5 That's the combination that really works. 6 Q. Would the QRA be of any use to identify which of the 7 risks is best to focus on, the most important risks, the 8 largest risks, or is that again something where it's too 9 affected by bias to be of value? 10 A. It would help you to give an indication of what to look 11 for. So the risks with the highest probabilities and 12 the highest impacts is what you would look for. But you 13 would have to remember that from the QRA, that would 14 still be subjective risk. So you would want to 15 double-check that with empirical data in order to verify 16 whether they actually are the highest probabilities, 17 whether they actually are the highest impacts. 18 Q. We heard -- we were discussing QRA in the Monte Carlo 19 software that it produces a range of probabilities. 20 A. Yes. 21 Q. Is it possible to get different probabilities from 22 reference classes? 23 A. Yes. The outcome of a Monte Carlo simulation and the 24 outcome of a reference class forecast are both 25 probabilities distribution, and actually if you fit in 32 1 the data from the reference class forecast into a Monte 2 Carlo simulation, there should be on principle the same. 3 The problem with Monte Carlo simulation is that they 4 don't do that. They use the subjective data, instead of 5 historical data, typically, and that's why they look 6 different. But the Monte Carlo simulation is typically 7 underestimating risk, whereas reference class 8 forecasting comes out with higher and more realistic 9 risk because they are based on historical data. 10 Q. So just to be sure I'm understanding that, the Monte 11 Carlo software can be used for either the reference 12 class data or the QRA data. The point is that it's the 13 underlying data that's more reliable with reference 14 class forecasting, less affected by bias? 15 A. Yes. So here the same garbage in, garbage out applies 16 as everywhere. So you -- if you put garbage in to 17 a Monte Carlo simulation, you get garbage out. 18 Unfortunately that's what happens in the majority of 19 cases. 20 Q. What I would like to do now is look at your -- return to 21 look at your report and a particular passage in this. 22 The reference is TRI00000265. If we could look at 23 page 12 of that. 24 If we could enlarge the upper half of the screen, 25 please. I want to ask you about the second paragraph, 33 1 first of all. You refer there to research by Kahneman 2 and Tversky. I think that's Kahneman you mentioned 3 earlier was a Nobel Prize winner for his work in these 4 fields: 5 "... argue that the prevalent tendency to 6 underweight or ignore distributional information is 7 perhaps the major source of error in forecasting. 8 Planners should therefore make every effort to frame the 9 forecasting problem so as to facilitate utilising all 10 the distributional information that is available." 11 What do you mean by "distributional information"? 12 A. So distributional information means variation. It means 13 uncertainty basically. So when you plan a project, you 14 are doing something that will happen in the future, and 15 things in the future don't happen with certainty. 16 There's an uncertainty. More things can happen than 17 will actually happen. So there's a range of things that 18 could happen, but only one of them will happen in each 19 instance. 20 That's what uncertainty is, and distribution 21 information is information about that uncertainty, and 22 what Kahneman and Tversky say here is that -- 23 Q. Could I ask you to slow down a little bit, because it's 24 going to become a bit difficult. 25 A. What Kahneman and Tversky say here is that that 34 1 distributional information is often, even typically, not 2 taken into account, and the Monte Carlo simulations that 3 we talked about earlier is such an example where all 4 distributional information is not taken into account, 5 and that's where the results are unreliable or even 6 misleading. 7 Q. How does the reference class forecasting enable that 8 information to be taken into account, or does it take it 9 into account? 10 A. It does take it into account, and this is because it 11 takes into account anything that has happened to the 12 historical projects. So anything that happens is taken 13 into account, and if you have a group of, you know, 60 14 plus light rail projects like we do in our report, then 15 you know what happened to each of these, and if we look 16 at schedule or cost or whatever it is that we want to 17 look at, we know what the variation is. You know, 18 sometimes, the costs come in at this level. Sometimes 19 it comes in at that level, and there's a whole range of 20 values. 21 Instead of us assuming or guessing at what would the 22 value be, we have the whole range of value. 23 So that's what Kahneman and Tversky mean when they 24 talk about all the distributional information available, 25 and that is what reference class forecasting delivers 35 1 you. 2 That's why it's no coincidence that the term 3 "reference class forecasting" was first coined by Daniel 4 Kahneman, actually. 5 Q. That's the point we discussed earlier. It is capturing 6 things which might not or will not be included in the 7 QRA? 8 A. Exactly. 9 Q. It's capturing a broader spectrum of information? 10 A. Yes. 11 CHAIR OF THE INQUIRY: Does it capture the uncertainty that 12 you spoke about in relation to the future, so that right 13 at the start of a project, making certain assumptions 14 about what will happen in the future, there's 15 uncertainty about that, but once the project is 16 concluded, then that -- what actually happened is taken 17 into the calculation of the ultimate cost of the 18 project? 19 A. It gives you the best estimate of the uncertainty that 20 you can get, but you might actually have a situation 21 where the project that you are doing is such an extreme 22 outlier that it's not captured by the distribution in 23 the reference data. So you have even larger uncertainty 24 that materialised. But that still doesn't mean that the 25 estimate that you get from the reference class would not 36 1 still be the most reliable estimate of uncertainty you 2 can get. 3 So this is what has been proven, is that the 4 estimate of uncertainty that you get by using the 5 historical data from the reference class is the most 6 accurate estimate of uncertainty that you can get, as 7 long as you have data from a minimum of 20 to 30 8 projects. 9 CHAIR OF THE INQUIRY: You spoke about an extreme outlier. 10 Is that the black swan or is that something else? 11 A. No, same thing. 12 CHAIR OF THE INQUIRY: Yes. 13 MR LAKE: If I can just pick up on something in your answer 14 to Lord Hardie's question, you said it's the best, most 15 reliable source of predicting, if you've got data from 16 20 to 30 projects. Does the reliability get better as 17 the number of projects increases? 18 A. Yes. So that's simple statistics. The more 19 observations you have, the more reliable an outcome you 20 get. 21 So yes. 22 Q. Of course projects have been done all the time that 23 could be added to your reference class, be it heavy 24 rail, light rail, motorways, hospitals? 25 A. Correct. 37 1 Q. So, for example, every ten years, would it be useful to 2 look again at the reference classes and recalculate 3 them? 4 A. Yes. 5 Q. I think you do in fact, at the Said Business School, 6 have -- compiled data in the form of reference classes? 7 A. Correct. 8 Q. That is a business in which you are involved through the 9 university? 10 A. Yes. That is a business. 11 Q. Just turning now to the foot of page 12 that we see on 12 screen, if we can look to just above the numbered 13 paragraphs indented, you note there that: 14 "Reference class forecasting follows three steps: 15 "1. Identify a sample of past, similar projects - 16 typically a minimum of 20 to 30 projects are enough to 17 get started, but the more projects the better ..." 18 Obviously we see there the figure of 20 to 30 we 19 have just been talking about. 20 What I want to ask you is about the similarity of 21 the projects. The assumption or the research has shown 22 that projects of a similar nature tend to produce 23 a similar outcome. But what is it has to be similar? 24 Is it the nature of the work undertaken? The problems? 25 The funding structures? What similarity do you look for 38 1 in this context? 2 A. So first of all, there is the same type of project. So 3 light rail projects could be compared to each other. 4 That's the first criterion. 5 But also that they are actually delivered under 6 similar regulatory regimes, including the type of law 7 and regulations that the projects have to adhere by. 8 So you probably wouldn't take a light rail scheme 9 from India if we were planning a light rail scheme here 10 in Scotland. You would say most likely that is not 11 comparable. We would be suspicious of that, and we 12 would actually do statistical tests in order to verify 13 whether that should be included or not. But most 14 likely, we would just make a decision that it's not 15 a comparable geography. The laws, the economy, the 16 logistics, the level of corruption and so on are 17 different. And therefore we should not use data from 18 that kind of setting for this kind of setting. 19 So those are the kinds of considerations that you go 20 through when you establish the reference class, and then 21 you do very rigorous statistical analysis. It's 22 actually quite easy by rigorous statistical analysis to 23 decide whether projects are similar or not. So that's 24 how you handle the concept of similarity in establishing 25 the reference class. 39 1 Q. It may seem like an obvious thing to say, but if you do 2 not choose the correct class of projects, you're not 3 going to get the correct optimism bias uplift? 4 A. That is correct. And that -- many people think that 5 that means that you have to be careful about what you 6 include. And that is true, you have to be careful about 7 that. But Daniel Kahneman has pointed out you have to 8 be even more careful about -- you throw out, because we 9 again, humans, have another bias in addition to the 10 optimism bias that is called uniqueness bias, that we 11 tend to think of projects as more unique than they 12 actually are. 13 That means that we are discarding data from other 14 projects that could actually be highly useful for us. 15 We say: no, this is not comparable because our project 16 is unique and therefore we can't compare to these other 17 projects. So Kahneman has argued that we are actually 18 erring on the side of not including enough in the 19 reference class, instead of including too much. 20 So that's the balance you need to strike when 21 you're -- in the considerations that people go through 22 when they are establishing reference classes. 23 CHAIR OF THE INQUIRY: What about the type of project that's 24 in a reference class, I'm thinking of planning or 25 regulations or use classes -- you have a series of 40 1 different uses that are within the same use class. 2 You've mentioned that you would compare a light rail 3 project with a light rail project, assuming the 4 circumstances were similar. Would you look at other 5 projects outside light rail projects, for instance, 6 heavy rail projects, to -- as part of the same class or 7 would they be excluded? 8 A. So we actually tested for that, and it turns out, 9 and I think this is mentioned in the report, that heavy 10 rail projects, as you mentioned, are statistically 11 similar to light rail projects, and therefore, 12 a statistician would tell you that these ought to be 13 included in the reference class to give you more 14 reliable results. So you should actually pool light 15 rail and heavy rail in this case. 16 We even had an example, an extreme example that we 17 had one statistician tell us: look at your data, opera 18 houses are performing the same way as light rail 19 projects, so you should include opera houses. We 20 said: no, we are not going there because we won't be 21 able to justify that. But from a statistician's point 22 of view, if things are similar statistically, they 23 should be proved. 24 CHAIR OF THE INQUIRY: I think you mentioned to Mr Lake that 25 you would advocate or favour a review every ten years or 41 1 perhaps even sooner of the material that you have 2 available. 3 Would that be information that should inform the 4 Treasury about its guidance to other public bodies? 5 A. Yes. 6 CHAIR OF THE INQUIRY: So does that mean that you would 7 envisage the Treasury reviewing its guidance on 8 a regular basis and if so, how often? 9 A. So obviously it's not my -- I'm not in a position to 10 tell the Treasury what to do. But hypothetically, if 11 I was asked, I would say yes, the data needs to be 12 revised from time to time. Preferably continuously 13 actually, because every time a new project has been 14 completed, we have additional useful data that should go 15 into the reference class. But I would say at a minimum 16 every ten years, preferably every five years. 17 CHAIR OF THE INQUIRY: Thank you. 18 MR LAKE: Turning back to your report now, and looking at 19 the second paragraph, the second step in reference class 20 forecasting, you say: 21 "Establish the risk of the variable in question 22 based on these projects - eg identify the cost overruns 23 of these projects." 24 Is that essentially taking the data from the 25 reference class? 42 1 A. It is, yes, establishing -- you basically look at the 2 risk of each project and you plot them in 3 a distribution. That's the second stage. 4 Q. Then the third is: 5 "Adjust the current estimate - through an uplift or by 6 asking whether the project at hand is more or less risky 7 than projects in the reference class, resulting in an 8 adjusted uplift." 9 Now, two parts to that. Firstly, simply applying 10 the uplift, that's essentially arithmetical? 11 A. Yes. 12 Q. And no judgement involved there at all? 13 A. Correct. 14 Q. Now, the second part of that clearly does involve some 15 judgement because someone might ask themselves, is in 16 fact our project less risky than the reference class? 17 A. Correct. 18 Q. Now, that seems to be coming back to what you have 19 described as the uniqueness bias. 20 A. Yes, it could be. That's where the uniqueness bias and 21 optimism could re-enter the whole exercise. 22 Q. I think you go on to deal with that in the following 23 paragraph, where you say: 24 "The final step in the reference class forecasting 25 process considers whether the project at hand is more or 43 1 less risky than the projects in the reference class. It 2 should be stressed that unless this consideration is 3 based on objective evidence, optimism bias might be 4 re-introduced into estimates." 5 If we scroll further down the page: 6 "In addition, planners might consider downward 7 adjustments of optimism bias uplifts based on capability 8 assessment, eg ability to identify and address risks 9 early or commercial structures established with a goal 10 to transfer risk as much as possible. These assessments 11 are all subjective, because they are untested at the 12 planning stage." 13 In that last paragraph, it is not so much assuming 14 that the project is unique, but it is assuming certain 15 qualities or capabilities of either the project 16 structure or the people who are administering the 17 project. 18 A. Correct. 19 Q. Now, that once again seems very much a subjective 20 judgement. People are judging themselves and their 21 project to say: we are better than everything in the 22 class reference. 23 A. Correct. 24 Q. That seems to be very -- to the layman to be very 25 fertile ground for distorted judgement. 44 1 A. Correct. 2 Q. That being so, why is it even in the third step 3 a possibility left open that the actor might ask whether 4 they are more -- that they are less risky than the 5 projects in the reference class? 6 A. You might come across a client or a project team that 7 has a documented track record of being better than 8 average. So they have delivered one success after 9 another, and it's actually documented that that is the 10 case, that they deliver on budget and they have done it 11 now ten times in a row. 12 If you have that kind of evidence, then it actually 13 would be a mistake to apply the average, the expected 14 value from the whole reference class because this team 15 or this client is clearly better than the project in the 16 reference class. I have to say, we haven't come across 17 this. What usually happens when we do reference class 18 forecasting, and enter these considerations with our 19 standard part of the format, is that the people respond: 20 we are no better and no worse than the reference class. 21 That has happened every time, and this is easy to 22 understand because it's basically the same organisations 23 and the same methodologies that are being used in the 24 conventional approach. 25 So it's the same people who have produced the risk 45 1 estimates and the budgets and so on for the project that 2 is being planned, as for the projects in the reference 3 class. That's the explanation that we get this answer. 4 So usually we actually do use the average, the 5 expected value as the uplift. 6 Q. But in the first paragraph after the numbered 7 paragraphs, you refer to whether or not the 8 consideration, that's the consideration of whether or 9 not it's less risky, is based on objective evidence. 10 You gave the example that an entity is able to 11 demonstrate its project team have delivered project 12 after project at an overrun less than the reference 13 class. Is that the sort of thing you are looking for 14 for objective evidence, it must be -- 15 A. Correct. 16 Q. -- concrete data? 17 A. Yes, that is what we are looking for, concrete data. 18 Q. Not a matter of the opinion of anyone involved in the 19 project that they have a structure which will make 20 things better. 21 A. Correct. 22 Q. So if someone were to say: well, we have a novel 23 procurement approach which is going to reduce risk; you 24 would not regard that as a reason to reduce the optimism 25 bias uplift? 46 1 A. Not unless there was a historical track record that that 2 procurement approach actually led to better results. 3 Q. I understand, thank you. 4 I want to now turn to ask you some questions about 5 the guidance that was available at the time the decision 6 to enter the tram project in 2007/2008 was made. 7 It has been considered in your report. But what 8 I would like to do is look at some of the actual 9 underlying guidance documents. If you could firstly 10 look at CEC02084689. 11 I take it you recognise this document of course? 12 A. I do. 13 Q. This is guidance that was provided by Mott MacDonald to 14 the Treasury in July 2002 as a review of large public 15 procurement in the United Kingdom. 16 Do you see that? 17 A. Yes. 18 Q. Now, this made certain recommendations as to the uplifts 19 that might be made for optimism bias and the methodology 20 for making those uplifts? 21 A. Correct. 22 Q. But this was made in 2002. 23 Now, dealing with how, firstly, looking at today's 24 date, some 15, 16 years on, do you consider that this 25 guidance is still valid, or has it lost some of its 47 1 validity? 2 A. It has lost some of its validity in my view. 3 Q. In which particular areas is it of less use now than it 4 once was? 5 A. I would say that the data are dated. So this is now 6 16 years ago or 15/16 years ago, and that's a long time, 7 and we know much more today than we did then. 8 This is an important first document. It was a very 9 important first step in dealing with the types of issues 10 we are discussing here, but a lot of progress has been 11 made since then, and therefore the report is now 12 outdated. 13 Q. In terms of the underlying data, and looking even at the 14 time, 2007/2008, is it the case that there was new data 15 even by that time, 2008, including data that you had 16 provided? 17 A. Yes, in fact, the Mott MacDonald report triggered 18 another report in 2004 that I was involved in, and in 19 2003 it was the basis for the Green Book from the 20 Treasury. So a lot of stuff happened immediately after 21 the Mott MacDonald report, and I would say already in 22 2003 and 2004, better data were available. 23 Q. If we nonetheless go and see what this report has to say 24 about the calculation of optimism bias and look at 25 page 40. 48 1 If we start by looking at the upper half of the 2 page. Under the heading, "Calculation of Optimism 3 Bias", it states: 4 "This section describes how to calculate the 5 optimism bias for the estimated project costs and time. 6 The calculated optimism bias will be used to replace the 7 risk element in the 6 per cent discount rate, formerly 8 recommended by HM Treasury. When calculating optimism 9 bias experienced appraisers should apply a degree of 10 best judgement." 11 I just wanted to ask you really about that last 12 sentence in that first paragraph standing out to suggest 13 that judgement is used in calculating optimism bias. 14 Is that a fair comment, or is the idea of optimism 15 bias really to try and reduce or even eliminate issues 16 of judgement? 17 A. Well, at this stage the data were very rudimentary, and 18 I believe that this is one of the reasons that they 19 formulated it softly like this, that they know that they 20 actually in this report are not providing a very 21 rigorous set of data, and therefore anybody using that 22 data will have to, you know, use a certain level of 23 judgement. 24 For me this is a cautionary sentence, and it's 25 appropriate, given the status of the data, whereas what 49 1 we talked about earlier today are much more rigorous 2 data. 3 So today and already in 2004, much more rigorous 4 data were available where you would do the kind of 5 considerations that we have talked about, where you 6 would say okay, we can actually apply an expected value 7 based on the historical data where we are not using any 8 other judgement than saying that this is the most -- this 9 is the best estimate we can get of uncertainty. 10 So that's how I would explain the difference between 11 this and later reports. 12 Q. What I want to do now, we can see there's a table on the 13 screen in front of us. Those are the figures that were 14 provided by this report, but to put them in context, if 15 we can look at what the paragraph before it says, it 16 notes that: 17 "When carrying out project appraisals, full 18 allowance should be given for any suspected optimism in 19 the costs and time figures originally proposed, giving 20 regard to the outcomes of previous projects of a similar 21 nature. By accounting for optimism more explicitly, 22 project options can be compared more accurately with 23 regard to costs and times. Table 4 ..." 24 Which is the one we can see on screen: 25 "... provides indicative figures for optimism bias. 50 1 It has been prepared by taking into consideration the 2 results of the Mott MacDonald study and reductions in 3 optimism bias levels observed over recent years to 4 provide an upper bound for optimism bias. The lower 5 bound in Table 4 allows for improvements in practice 6 that were evident over the review period and new 7 procurement practices known to have been implemented in 8 the last five years." 9 Now, if we look at the table following that 10 explanation, we can see that the categories that are 11 given, and there's just six of them, are really very 12 broad indeed. 13 We don't, for example, have light rail or heavy rail 14 or roads. We've just got standard civil engineering, 15 non-standard civil engineering. 16 Did that affect the reliability of these estimates? 17 A. When you say these estimates, do you mean the estimates 18 in this table? 19 Q. Yes. 20 A. Okay. Yes, it does. And again, Mott MacDonald didn't 21 have much data at the time they did this report, and 22 they therefore had to use very simple categories and 23 very aggregate categories like they do here, and that 24 will affect the data, and even more will affect the 25 validity and reliability in using the data on specific 51 1 types of projects like light rail. 2 Q. If that's the data, looking underneath the table to see 3 how it's suggested it be used, it says: 4 "The upper bound values recommended for use when 5 calculating optimism bias represent the optimism bias 6 level to expect for current projects without effective 7 risk management and bad scope definition, and are the 8 starting point for calculating optimism bias for 9 projects. These upper bound values reflect the average 10 historic values because the average historic values are 11 similar to the highest values for optimism bias 12 currently being reported for recently completed projects 13 that have experienced high levels of optimism in their 14 project estimates." 15 Now, this seems to be suggesting that you use the 16 upper level when there is no effective risk management 17 and bad scope definition. 18 A. Yes. 19 Q. That seems to suggest that if you have any form of risk 20 management and bad scope definition, you should 21 immediately start reducing from the figures given in the 22 table? 23 A. Correct. 24 Q. Now, that was obviously the understanding at the time. 25 Is that still the understanding now? 52 1 A. I would say today you wouldn't use this approach because 2 there is so much better data available. 3 Q. Right. If we could look at the following page, page 41, 4 it considers the various steps that are involved. 5 Reading through the middle of the page, it notes: 6 "To calculate the optimism bias for project 7 estimates during a project appraisal: 8 "1. Decide which project type is appropriate for 9 the project being appraised." 10 That is simply which one of those six lines on the 11 table it is felt most closely corresponds; is that 12 correct? 13 A. Correct. 14 Q. In the paragraph afterwards it notes: 15 "For the ease of determining a project type for 16 building and civil engineering projects, a project is 17 considered "non-standard" if it satisfies any of the 18 following conditions: (a) it is innovative and/or 19 unique; or (b) construction involves a high degree of 20 complexity and/or difficulty." 21 Do you have any comment on that as being the 22 deciding factor between what is standard and what is 23 non-standard? 24 A. No, I would agree with this as a definition of 25 non-standard. 53 1 Q. If we go over the page to the second step, it then says: 2 "Use the appropriate upper bound value for optimism 3 bias from Table 4 as the starting value ..." 4 That's obviously the very highest one we have seen. 5 The following step, number 3, is: 6 "Reduce this upper bound optimism bias according to 7 the extent to which the project risk areas are managed. 8 The project risks within each project risk area can be 9 managed. If the project risk areas for a project have 10 only been partially mitigated then the contribution to 11 optimism bias can be reduced proportionally to reflect 12 the amount that each project risk area has been 13 mitigated." 14 It goes on to explain that in more detail. 15 This seems to be calling for an exercise of 16 subjective judgement? 17 A. It does. I mean, this is a wide open door for having, 18 you know, a self-defeating result. So supposedly you 19 are trying here to take optimism bias into account, but 20 as part of the process, you have actually steps that 21 would reintroduce optimism bias. 22 Q. The table gives an upper level and a lower level, and 23 essentially invites the project people to exercise 24 a subjective judgement about where in that band they 25 fall? 54 1 A. Correct. 2 Q. Do you consider that accurately gives effect to the 3 purposes of reference class forecasting? 4 A. No. 5 Q. If we could then go to a different document of guidance. 6 It's the Green Book, Treasury Green Book. It's 7 reference CEC02084256. 8 You can see the heading here, "THE GREEN BOOK 9 Appraisal and Evaluation in Central Government". The 10 box underneath it, it's not very clear, but we can just 11 about read: 12 "Note explaining changes made to the Green Book in 13 July 2011. 14 "This is the 2003 edition of the Green Book. 15 However, pages 57 to 58 which deal with the valuation of 16 non-market goods have been updated alongside the release 17 of a Green Book discussion paper on this subject." 18 Now, I'm happy to say that we are not dealing with 19 pages 57 and 58. So is this in essence the guidance 20 that was available in 2003? 21 A. Yes. 22 Q. Could we go and look at page 33 of this document on 23 screen, and look at the lower half of the screen. 24 We have got the big heading, "ADJUSTING FOR BIAS AND 25 RISKS", and the sub-heading, "Optimism bias", under 55 1 which paragraph 5.62 says: 2 "To redress this tendency ..." 3 That's the optimism tendency: 4 "... appraisers should make explicit adjustments for 5 this bias. These will take the form of increasing 6 estimates of the costs and decreasing, and delaying the 7 receipt of, estimated benefits. Sensitivity analysis 8 should be used to test assumptions about operating costs 9 and expected benefits." 10 The following paragraph is: 11 "Adjustments should be empirically based (eg using 12 data from past projects or similar projects elsewhere), 13 and adjusted for the unique characteristics of the 14 project in hand. Cross-departmental guidance for 15 generic project categories is available, and should be 16 used in the absence of more specific evidence. But if 17 departments or agencies have a more robust evidence base 18 for cost overruns and other instances of bias, this 19 evidence should be used in preference. When such 20 information is not available, departments are encouraged 21 to collect data to inform their estimates of optimism, 22 and in the meantime use the available data that best 23 fits the case in hand." 24 I'm just trying to follow through really what has 25 been said in that last paragraph. 56 1 It has to be an adjustment made for optimism bias. 2 And the Green Book, the guidance says it must be 3 empirically based. But invites the reader to use data 4 from past projects or similar projects elsewhere, as 5 adjusted for the unique characteristics of the project 6 in hand. 7 Now, from our discussions that we have had this 8 morning, that once again seems to be an open door to 9 optimism bias coming back in when correcting optimism 10 bias. 11 A. Yes, the very last thing about adjusting for the 12 uniqueness of the project, that's the open door. 13 Q. In terms of basing -- making adjustments empirically, 14 using data from past projects, that in itself wouldn't 15 give rise to bias, provided the class of projects is 16 properly selected? 17 A. Correct. 18 Q. So if you look at something, for example, in published 19 guidance, or something that's provided in the service 20 that you're able to provide projects, that should not be 21 biased? 22 A. Correct. 23 Q. But if a project team decide to start going out and 24 choosing which projects they are going to look at, both 25 inclusion and exclusion, that gives rise to the sort of 57 1 bias again you have discussed this morning? 2 A. Correct. 3 Q. So if there is to be reference class forecasting, it's 4 important that the reference classes have a degree of 5 independence to have validity? 6 A. Correct. 7 Q. Looking at paragraph 5.64, it notes that: 8 "Adjusting for optimism should provide a better 9 estimate, earlier on, of key project parameters. 10 Enforcing these adjustments for optimism bias is 11 designed to complement and encourage, rather than 12 replace, existing good practice, in terms of calculating 13 project specific risk adjustments. They are also 14 designed to encourage more accurate costing. 15 Accordingly, adjustments for optimism may be reduced as 16 more reliable estimates of relevant costs are built up, 17 and project specific risk work is undertaken. Both cost 18 estimates and adjustments for optimism should be 19 independently reviewed before decisions are taken. 20 Annex 4 provides further detail on how to deal with 21 optimism bias." 22 Now, we will come to Annex 4 in a moment, but just 23 looking at what was in that paragraph, it's saying that 24 both cost estimates and adjustments for optimism should 25 be independently reviewed. What degree of safeguard 58 1 does that independent review provide to either estimates 2 of risk or estimates of optimism bias? 3 A. It's always good to get an independent review of things 4 like this. However, if the independent reviewers are 5 using a methodology for the review, that is open to 6 bias, then, you know, it's not going to be a great help. 7 So you actually need an independent review that is using 8 historical data in order to assess whether the 9 assessment that the project team comes up with is 10 realistic. 11 Q. Otherwise in a sense you are just going to make the same 12 mistakes inherent in a QRA all over again? 13 A. Exactly, and to be frank, we often see that with 14 independent reviews, that they don't bring much 15 additional value because they follow the same procedure 16 and they use the same data as the project team, and 17 therefore doesn't come up with different results. 18 Q. Now, accepting that problem with the risk data, even if 19 it's been the subject of external assessment or 20 verification, halfway through that paragraph, it says 21 that the adjustments for optimism may be reduced as more 22 reliable estimates of relevant costs are built up, and 23 project specific risk work is undertaken. 24 Now, that is suggesting as you build up your QRA, 25 you reduce your optimism bias. 59 1 A. Correct. 2 Q. But the QRA is going to contain optimism bias? 3 A. Correct. 4 Q. So is it a valid approach to say, as we build up, as we 5 move to finer detail on our analysis of risk, we should 6 reduce our optimism bias allowance? 7 A. I would say no. 8 Q. Why not? 9 A. Because there's an assumption that you are actually -- 10 you are reducing risk as you do this, and there's 11 an assumption that the more detail you have, the less 12 risk you have, and those are just assumptions, and they 13 need to be verified in order to actually -- for you to 14 justify reducing optimism bias uplifts, and they are not 15 empirically verified in most cases that I have seen. 16 So therefore this combination is really a trap in 17 the sense that it will lead the risk assessment team 18 back into biased -- and underestimated risks. 19 Q. There was reference there to Annex 4. If you could look 20 at that, please. It begins on page 83. 21 This is just to identify that it's Annex 4 we are 22 looking at, headed "Risk and Uncertainty". If we go 23 forward to the optimism bias part on page 89, we can see 24 under the heading "Capital costs", if we go to 25 paragraph 20, it notes that: 60 1 "Appraisers should adjust for optimism bias in the 2 estimates of capital costs in the following way: 3 "Estimate the capital costs of each option; 4 "Apply adjustments to these estimates based on the 5 best available empirical evidence relevant to the stage 6 of the appraisal; and 7 "Subsequently, reduce these adjustments according to 8 the extent of confidence in the capital costs' 9 estimates, the extent of management of generic risks, 10 and the extent of work undertaken to identify and 11 mitigate project specific risks." 12 Just dealing with each of these three in turn, 13 firstly, the first stage, estimating capital costs, 14 I take it there's no objection with that. That's always 15 a necessary first stage? 16 A. Correct. 17 Q. The second one, applying adjustments to these estimates, 18 that's the cost estimates, based on the best available 19 empirical evidence, if that best available empirical 20 evidence is the reference class data that you've 21 discussed with us, do you regard that as part of a good 22 approach also? 23 A. Yes. 24 Q. The third one, however, reducing those adjustments 25 according to confidence in capital cost estimates, is 61 1 that going to cause problems? 2 A. It's likely to cause problems, and what you would need 3 here is a sort of scepticism. Every time it's suggested 4 that you would reduce the optimism bias uplift, somebody 5 needs to play the role of the sceptic and ask for solid 6 evidence why it should be done. Otherwise this is 7 likely to result in optimism bias uplifts that are too 8 low. 9 Q. Just because there's been, to use the wording at the end 10 of that bullet point, work undertaken to identify and 11 mitigate project specific risks, presumably that will 12 happen in every project? 13 A. Yes. 14 Q. And if we start reducing optimism bias uplifts on that 15 basis, will it provide a distorted view? 16 A. In my view, yes. 17 Q. So would it be fair to say on that basis that this 18 guidance is no longer likely to be helpful for the 19 proper management of project costs? 20 A. I would say on this specific point, the guidance is 21 tripping itself up. 22 Q. Looking at the following paragraph, paragraph 21, the 23 guidance states: 24 "Departments or agencies may be able to provide the 25 best empirical evidence to support adjustments for 62 1 optimism. Alternatively, and if applicable, they may be 2 taken from the Green Book home page, which provides the 3 recommended adjustments to be made at the outline 4 Business Case stage for buildings, civil engineering, 5 equipment and development, and outsourcing projects." 6 That guidance, was that the guidance derived from 7 the Mott MacDonald report that was being provided? 8 A. I would have to look at the Green Book home page to see 9 whether that's the case, but I would assume so since 10 nothing else was available at the time. 11 Q. What is noted here is that the intention is that that 12 begins to be applied at the outline Business Case stage. 13 A. Sorry, say again? 14 Q. That uplift would be applied at the stage of the outline 15 Business Case. 16 A. Yes, that's what it says under point 21. 17 Q. Is there any significance to the fact it's specifically 18 to be applied at that stage? 19 A. Well, I guess this is to make sure that optimism bias 20 uplifts are introduced as early in the planning of the 21 project as possible. You can apply uplift and you 22 actually should apply uplifts at each of the different 23 stages of the project. 24 Q. I want to come to ask you a bit more about these stages. 25 We can perhaps do it by reference to the next piece of 63 1 guidance, which is the Department for Transport guidance 2 on procedures for dealing with optimism bias in 3 transport planning. That has reference CEC02084257. 4 We can see the title I gave there on the front page 5 of the report; is that right? 6 A. Correct. 7 Q. The date given is June 2004. 8 A. Yes. 9 Q. I take it not only do you recognise this document, you 10 were one of the authors of it? 11 A. Correct. 12 Q. Could we go first to page 11. If you could enlarge the 13 upper part of the page. 14 Reading from the third paragraph there, the report 15 says: 16 "Taking an outside view requires the following steps 17 for the individual project: 18 "Identification of a relevant reference class of 19 past projects. The key is here that the class is broad 20 enough to be statistically meaningful but narrow enough 21 to be truly comparable with the specific project." 22 Now, introducing the idea of the project being 23 narrow enough is you sought with the additional data you 24 had to narrow down the classes, so you didn't really 25 have standard civil engineering, you had particular 64 1 types of civil engineering project? 2 A. Yes. 3 Q. The second bullet point is: 4 "Establishing a probability distribution for the 5 selected reference class. This requires access to 6 credible data on cost increases (or time schedule delays 7 or benefit shortfalls if these are the key parameter) on 8 a sufficient number of projects within the reference 9 class to make statistically meaningful conclusions 10 (normally at least 10)." 11 Obviously that 10 number is rather less than the 20 12 to 30 you were talking a few minutes ago? 13 A. Correct. So today we know much more about this than we 14 did at the time. We have many more data now, which 15 means that we understand the specific distributions of 16 the data, and the number of observations that you need 17 in order to reliably establish the distribution varies 18 according to which distribution it is. 19 So now that we have the detailed information, we 20 know that because of the many outliers, that the -- we 21 know that the mean is less robust and less reliable in 22 these types of distribution, which is why you need more 23 observations. 24 So that's why we have adjusted it up over time, 25 because we now know that the distributions are such that 65 1 10 is actually not enough to get a reliable estimate. 2 Q. But also the significant thing here is you have -- this 3 guidance seeks to provide the probability distribution 4 for the first time. Because that wasn't provided by 5 Mott MacDonald. But you did seek to provide the various 6 P50, P60, P70 probabilities, so that the appropriate 7 level of certainty could be selected? 8 A. Correct. 9 Q. The third bullet point there is: 10 "Placing the specific project at an appropriate 11 point in the reference class distribution." 12 You note: 13 "This step has an element of intuitive assessment 14 and is therefore liable to optimism bias." 15 I think you go on to explain later how that must be 16 controlled to avoid undermining the whole process? 17 A. Correct. 18 Q. If you go to page 14, I should say, whenever I give page 19 references, I'm referring to the electronic version. 20 You will see the page reference on screen is different, 21 but I will always use the electronic one. The second 22 last paragraph on this page states: 23 "At the same time it is important to recognise that 24 the establishment of budgets which on average are more 25 than adequate (as would be the case if uplifts 66 1 reflecting a higher percentile in the distribution than 2 the 50 per cent percentile is applied) may have an 3 incentive effect which works against tight cost control 4 if the more than adequate budget is available (or 5 perceived as being available) to the project 6 organisation." 7 Is that the problem we discussed earlier that, if 8 the money is there, people will want to feed on it? 9 A. Correct. 10 Q. That's why you have to have the control such as we have 11 already discussed? 12 A. Correct. 13 Q. Or contract structures to incentivise cost control? 14 A. Correct. 15 Q. Turning then to page 19, this is under the heading of 16 "Benchmarking of optimism bias in Britain". If we look 17 at paragraph 3.1.1 with the heading "Sampling and data 18 collection", you say there: 19 "Cost overrun is here defined as the difference 20 between actual and estimated costs in percentage of 21 estimated costs, with all costs calculated in constant 22 prices. Actual costs are defined as real, accounted 23 costs determined at the time of completing a project 24 (outturn costs). Estimated costs are defined as 25 budgeted, or forecasted, costs at the time of approval 67 1 of/decision to build a project, which is typically 2 similar to costs as presented in the Business Case for 3 a project." 4 Now, just looking here, you're choosing the estimate 5 at the time when the decision to build or to progress is 6 made. Was there a particular reason to choose that 7 point? 8 A. Yes. So basically you want to know whether 9 decision-makers were well informed when they made their 10 decision, and you want them to have the right 11 information at that stage. 12 So that's why we're choosing the time of making the 13 decision as our baseline. You could choose other 14 baseline. You can choose any which baseline you want. 15 You can choose a much earlier one like a pre-feasibility 16 study, or you could choose another one, where you had 17 the tendering for the projects, and we had data to do 18 all these different baselines. But this is what we 19 consider, and not only us but the international 20 convention is when you measure these things, that you 21 actually use the date of the decision, the final 22 decision to do a project, because you want to know 23 whether the decision-makers were well informed or not. 24 It's really the most crucial point in a project. 25 When the project is given the green light to go ahead, 68 1 or when it's given the red light not to go ahead. So 2 that's the point we use as the baseline. 3 Q. For your answer there, it presupposed that the decision 4 to go ahead, the green light would be given, and later 5 there would be tendering. 6 A. Yes. That's the assumption. 7 Q. In the tram project the green light was given right at 8 the very last minute, after -- the final green light, it 9 might be said, was given after the tendering had taken 10 place. 11 A. It's not a problem, what I'm saying -- it is the 12 assumption, I should actually say more accurate, that 13 this is the way it usually -- it usually happens. But 14 if the final decision to build it was after the tender, 15 you still use it as the baseline, because in theory the 16 decision-makers could decide not to do the project after 17 the tender. So no final commitment has been made yet. 18 And we are looking for the baseline, where the final 19 commitment is made. 20 Q. When we come, however, to look at what the percentage 21 uplift should be at the time of decision made, do you 22 have to make some allowance for how much information 23 will be available, because if that decision is taking 24 place pre tender, there will be presumably much less 25 information than when that decision is made post tender? 69 1 A. Yes, but that is taken into account. So in the 2 reference class, that would be included in the numbers 3 that you have for the different variables. 4 If the numbers were -- are reflecting projects where 5 the tender came after, and sometimes long after, the 6 decision to build there, then that situation would 7 affect the outcome, and presumably you would have more 8 uncertainty, and therefore more variance and higher 9 delays or cost overruns, or whatever, and that therefore 10 would be built into the reference class. 11 Q. So it's not necessary to make a separate allowance for 12 it? 13 A. I wouldn't say so, but if you had the -- if you have the 14 assumption or if you think that -- if you had the 15 hypothesis that this might be something that affects the 16 result, you would simply go through your data on 17 projects and look for similar projects and see whether 18 it does actually affect the outcomes. If it does, then 19 you would have to build your reference class of that 20 kind of project. 21 So if there's a significant difference between 22 projects with that type of project cycle, sequence like 23 that, and the more conventional sequence, then you would 24 have to look for projects that are similar to the 25 project that you are estimating. 70 1 Q. Just to be sure that I'm understanding that, if your 2 decision to proceed is what we might term a late 3 decision to proceed, post tender, you would try and find 4 a reference class consisting of projects where the 5 decision has been taken late at that stage, because that 6 might generate a different risk profile, optimism bias 7 profile? 8 A. Correct, and all other things being equal, the risk 9 should be lower if you wait with making your decision. 10 Q. In the following paragraph, you note that even if the 11 project planning process varies with project type, place 12 and time, it is typically possible to locate for a given 13 project a specific point in the process that may be 14 identified as the time where formal approval was given 15 to build the project: 16 "Usually a cost estimate is available for this point 17 in time. If not, the closest available estimate was 18 used, typically a later estimate resulting in 19 a conservative bias in the measurement of cost overrun." 20 Is the point there that when you're analysing the 21 reference classes, you need to have some sort of fixed 22 date, so the choosing this one, at least you are 23 choosing a date where there normally is a cost available 24 by way of budget, so you've got your starting point for 25 your comparison? 71 1 A. Correct. 2 Q. If you could go then to page 31. Look at the lower half 3 of the page. It's just for the graphical representation 4 of what we've been talking about. You say: 5 "On the basis of the probability distribution for 6 rail shown above, required uplifts may be calculated, as 7 shown in Figure 9. With a willingness to accept 8 a 50 per cent risk for cost overrun in a rail project, 9 the required uplift would be 40 per cent. If the 10 Department for Transport were willing to accept only 11 a 10 per cent risk for cost overrun, then the required 12 uplift would be 6 per cent." 13 I think those figures are apparent if we look at the 14 graph, whereby the -- we've got the chance of overrun 15 along the X axis along the bottom, and the required 16 uplift on the Y axis to the side. So you see the nearer 17 the left-hand side you are, the lower your appetite for 18 risk, the higher the costs will be? 19 A. Correct. 20 Q. Again, that is a graph that has been produced by you as 21 part of this project from the data that you were 22 analysing? 23 A. Correct. 24 CHAIR OF THE INQUIRY: Mr Lake, will you find 25 a convenient -- 72 1 MR LAKE: This would actually be a convenient time. We will 2 be a bit longer with this guidance, so we won't get to 3 the end of it. 4 CHAIR OF THE INQUIRY: We will have a 15-minute break for 5 the benefit of the shorthand writer. 6 We will resume again at 11.25. 7 (11.06 am) 8 (A short break) 9 (11.26 am) 10 CHAIR OF THE INQUIRY: You are still under oath, Professor. 11 A. Yes. 12 CHAIR OF THE INQUIRY: Yes, Mr Lake. 13 MR LAKE: Thank you very much, my Lord. If we could go, 14 please, in the same document back to page 16. Sorry. 15 I'm using the wrong page reference. Page 18, please. 16 If could you enlarge the table we see there, we can 17 see the categorisation of the transport schemes used for 18 this report and see how much more broken down it is into 19 the various categories; rather than civil engineering, 20 standard, civil engineering, non-standard, you've been 21 able to look at the particular types of construction? 22 A. Correct. 23 Q. Is that because you had access to additional sources of 24 data as opposed to those used by Mott MacDonald in 2002? 25 A. Correct. 73 1 Q. Did that mean that it was able to predict more 2 accurately for each individual type of project, rather 3 than having to take a general approach for civil 4 engineering, standard and non-standard? 5 A. Yes. And it turned out that some of the project types 6 that had been lumped together when we ran statistical 7 tests, we could actually see they should not be lumped 8 together because they performed differently even at 9 statistically significantly critical levels. 10 So that a statistician would say that: they cannot 11 be put together, that would create noise instead of 12 information; so therefore we actually, you know, on 13 a scientific basis, we had to take them apart. 14 So for instance, road and rail are statistically 15 significantly different, and therefore should not be put 16 together, to take one example. 17 Q. So it wasn't just you had -- you did it because you 18 could do it. You did it because it yielded better 19 answers? 20 A. Yes. 21 Q. Previously we saw there was the standard and 22 non-standard distinction, and we saw that explained in 23 the Mott MacDonald guidance. Your categorisation has no 24 distinction between, for example, standard rail and 25 non-standard rail or standard road and non-standard 74 1 road. 2 Was that because you looked at it and decided that 3 wasn't statistically significant? 4 A. Yes. 5 Q. So once again the later work, the work that you have 6 done which had the greater analysis and the more 7 information to analyse, is likely to give a better more 8 accurate picture than the earlier guidance? 9 A. Correct. 10 Q. If we go to page 33. Under heading 4.2 you note that: 11 "The uplifts for optimism bias presented above refer 12 to, and should be applied to, estimated budgets at the 13 time of decision to build. The time of decision to 14 build is typically equivalent to the time of presenting 15 the business case for a project with a view to obtaining 16 the go or no-go for that project. The uplifts refer to 17 cost overrun calculated in constant prices." 18 That's what we've already discussed. You have 19 chosen that figure as an easy fixed point to identify in 20 each project; is that correct? 21 A. Not just easy. As the most relevant fixed point, the 22 most relevant baseline for each project. 23 Q. If you skip the following paragraph, but jump to the one 24 beginning "table 6" you have: 25 "Table 6 shows a simple example of how one would use 75 1 the established optimism bias uplifts in practice." 2 Perhaps if we go over the page to look at table 6, 3 if we can enlarge -- that's fine. I'll read the text 4 from the previous page, but it's useful to look at the 5 table, to have the table in mind when reading it: 6 "If, for instance, a group of planners were preparing 7 the business case for a new motorway, and if they or 8 their client had decided that the risk of cost overrun 9 must be less than 20 per cent, then they would use an 10 uplift of 32 per cent on their estimated capital 11 budget." 12 We can see that you get that easily from looking at 13 the table by looking at the roads row, and then there is 14 a number of different percentiles. They can simply 15 choose the 80 per cent and they know that the uplift 16 must be 32 per cent? 17 A. Correct. 18 Q. So this table gives you more detail both in terms of the 19 type of the project and the probability? 20 A. Yes. 21 Q. So as far as rail is concerned here, if we look at the 22 row beneath the roads, we can see that the 50 per cent 23 percentile, I should say, uplift, is 40 per cent, but if 24 you want to get to 80 per cent, it would be necessary to 25 have an uplift of 57 per cent? 76 1 A. Correct. 2 Q. That would be applied at the time of the decision to 3 build? 4 A. Correct. 5 Q. Although you might have to make some -- take into 6 account for the present project, the Edinburgh Tram 7 Project, the fact that that decision was taken later, 8 when more information was available? 9 A. Mm-hm. Yes. 10 Q. Because the reference class that's been used to provide 11 that data is of ones where the decision to -- has been 12 taken earlier? 13 A. Yes. 14 Q. If the decision was to be taken later, is there 15 generally available data as to the uplifts appropriate 16 for a later decision, or is that something if someone 17 was to come to you or other providers of your service, 18 you would be able to provide that information to them? 19 A. Yes, we would. 20 Q. You would be able to provide it? 21 A. Yes, and I assume others would also. I mean, these are 22 data that are available, and if you collect the data 23 from your projects, you will know this. 24 Q. Right. 25 Now, if we just look there in the rail category, 77 1 just to emphasise again, the 40 per cent figure is given 2 for a 50 percentile confidence level. Do you see that? 3 A. Yes. 4 Q. If we go back just briefly to look at the Mott MacDonald 5 paper, that's CEC02084689. And we go to page 40 within 6 that. 7 We can see there -- enlarge the table -- through 8 standard or non-standard civil engineering projects, 9 we've got the upper limit of respectively 66 per cent 10 and 44 per cent, in relation to capital expenditure? 11 A. Correct, yes. 12 Q. So the figures that seem to be quoted for the 13 Mott MacDonald numbers, does it appear from that that 14 they appear to be quoting around the 50 percentile/60 15 percentile level? 16 A. Well, this is unclear in the Mott MacDonald report, what 17 they do precisely in that respect. 18 Q. Mm-hm. So is it -- do you think it is valid to make any 19 general comparison about the number, the recommended 20 uplift, to compare it with the recommended uplifts in 21 your figure and reach a conclusion as to the probability 22 taken by Mott MacDonald or is that not likely to be 23 reliable? 24 A. So what I would suggest here is that the data in the 25 later report, so the data in the 2004 report, are more 78 1 reliable than the Mott MacDonald report. 2 So as soon as the data in the 2004 report were 3 available, in the areas where they could substitute for 4 the Mott MacDonald data, the data in the 2004 report 5 should be used and you should not use the data in the 6 Mott MacDonald report. 7 Q. Right. That's very clear. Thank you. 8 Go to page 35 within -- go back to the DfT advice 9 that you co-authored. Go to page 35 in that which we 10 now have on screen. Look at the second paragraph on the 11 page there. If you scroll down, it's probably useful to 12 have the chart in sight also. You note there: 13 "Similarly, where a project has not yet reached the 14 approval stage, but is at the strategic outline case or 15 outline business case stage, uplifts should be adjusted 16 to reflect this. Typically budget uncertainty is 17 reduced throughout the project cycle from inception over 18 feasibility studies and approval to construction and 19 start of operations (please refer to Chart 1 below)." 20 Just pausing there, we can see that in 21 a diagrammatic way, you indicate that as one goes 22 forward, the level of uncertainty narrows? 23 A. Correct. 24 Q. Going back to the text in the paragraph: 25 "Therefore, in order to arrive at a valid cost 79 1 estimate for a stage prior to the approval stage, the 2 uplifts listed above should generally be adjusted 3 upwards. Conversely, if a project has moved beyond the 4 approval stage to the stage of detailed design or 5 construction, uplifts should normally be adjusted 6 downwards." 7 A. Correct. 8 Q. Just there, you mention specifically detailed design. 9 Presumably the purpose of that is that once one has 10 a detailed design, if that is -- if the detailed design 11 is costed, there's less scope for bias as to the outturn 12 cost? 13 A. Correct. 14 Q. So it's not just that detailed design has been done. It 15 must be that it is what is costed and contained within 16 the contract? 17 A. Correct. 18 Q. Reading below the chart, you say: 19 "In the latter, or post approval, case, one way of 20 adjusting uplifts downwards would be to follow the 21 simple rule that uplifts are reduced at a specific point 22 in time by the same percentage ... of the total budget, 23 which has been spent up to this point, for instance as 24 follows." 25 If we look at the table, we can see that if when 80 1 you've spent 0 per cent of your budget, you give 2 100 per cent of the uplift, if we read down a few rows, 3 once you have spent, say, 30 per cent, you make 4 an adjustment of 70 per cent of the uplift. You take 5 30 per cent of the uplift away? 6 A. Correct. 7 Q. Now, this is obviously a somewhat arbitrary means of 8 reducing it. 9 A. Correct. 10 Q. The suggestion may be that if we move, as we move 11 forward, it's possible to say: well, we have developed 12 the design further; we have further costings; we can 13 just decide that that the risks inherent in the optimism 14 bias are now dealt with and we can reduce it. Is there 15 a problem with that? 16 A. Yes. Like we discussed earlier, that's opening the door 17 to optimism bias again, and our suggestion here to do 18 this more arbitrary and mechanistic approach is actually 19 to keep optimism at bay, and therefore we prefer this to 20 having subjective assessments reintroduced about how 21 much risk you have actually eliminated. 22 Q. Is this approach that you recommend or you set out here 23 one that in your experience is being applied in 24 practice? 25 A. Yes, it is. 81 1 Q. In large projects, infrastructure projects? 2 A. Correct. 3 Q. If we go over the page, to page 36, underneath the table 4 you say: 5 "Using this simple rule of interpolation has the 6 advantage of avoiding the reintroduction of optimism 7 bias for lack of empirical evidence of how much the 8 uplifts should be reduced at key points in the project 9 cycle. The latter information would ideally be 10 available; in reality, often it is not. If the total 11 budget changes over time, from approval to start of 12 operations, as is often the case, the uplifts would have 13 to be recalculated accordingly, whether or not 14 interpolation or a more empirical method is used to 15 adjust uplifts downwards over time." 16 I think you are aware from the Edinburgh Tram 17 Projects, the uplifts were actually reduced over time as 18 things moved from the interim outline Business Case all 19 the way to the Final Business Case? 20 A. Yes. 21 Q. That wasn't done by your interpolation method. It was 22 done by a judgement on the extent to which risk 23 development meant it was no longer necessary in their 24 view? 25 A. Yes. 82 1 Q. Do you consider that was an appropriate approach? 2 A. No. 3 Q. Why not? 4 A. Again, because it opened the door to optimism, and now 5 we can see with hindsight that that is actually what 6 happened, that the assessment of risk became way too 7 optimistic; and in the way I see it, this happened 8 because the project team assumed that their risk 9 mitigation measures and the identification of risks 10 would actually come through. So the mitigation measures 11 would be effective. So they basically assumed they had 12 realistic assessment of risk, and that the mitigation 13 measures for those risks would be effective. 14 But that's an assumption. That's a theory, and only 15 reality, only the delivery of the project would show 16 whether the assumption was right, and now we know today 17 that the assumption was not right. It was too 18 optimistic. And the way I see it, it was too optimistic 19 because it opened the door again for these subjective 20 assessments of the risk, and risks were adjusted 21 downwards much more than they should have been. There 22 should have been a downward adjustment of risk as the 23 project progressed like we've talked about. That is 24 normal, and that is standard practice. You would 25 actually make an error if you didn't do it, but there 83 1 was way too little scepticism and realism in assessing 2 how much the optimism bias should be reduced over time. 3 Q. Your rule of interpolation as set out in the table is 4 significant in that there will be an optimism bias 5 allowance until 100 per cent of the project costs had 6 been spent? 7 A. Yes. 8 Q. So that would continue right through the construction 9 works? 10 A. Yes. 11 Q. Again, in terms of Edinburgh tram, it was said that they 12 considered that optimism bias would be eliminated at the 13 time of the decision to proceed with the project? 14 A. Yes. 15 Q. Is that realistic? 16 A. No. As long as there's a future, there's optimism about 17 the future. 18 Q. If we look down to paragraph 4.4, you say: 19 "The strength of the uplift for optimism bias 20 established above is that they are firmly grounded in 21 empirical probability distributions of cost overrun for 22 different types of transport projects. Thus the uplifts 23 allow true reference forecasting for specific projects 24 under consideration. It is crucial that uplifts be 25 empirically based in this manner, otherwise the risk is 84 1 high of re-introducing optimism and bias in project 2 preparation and decision making." 3 You're stressing there that the uplifts must be 4 empirically based. That's using the reference classes? 5 A. Correct. 6 Q. But is it equally true that -- perhaps I should go on 7 and look at the next paragraph first. You say: 8 "It may be argued that uplifts should be adjusted 9 downward as risk assessment and management improves over 10 time and risks are thus mitigated. It is however our 11 view that planners and forecasters should carry out such 12 downward adjustments of uplifts only when warranted by 13 firm empirical evidence." 14 Again, the point is there's no point in adjusting 15 empirical uplifts by subjective downgrades? 16 A. Correct. 17 Q. You say: 18 "For 70 years, optimism bias has been high and 19 constant for the types of transport projects considered 20 above, with no indication of coming down. With 21 practices of optimism as deep-rooted as this, hard 22 evidence from post-audits would be required to 23 convincingly argue the case that optimism bias is 24 finally coming down. In general, only at such a time 25 when this evidence is available should uplifts be 85 1 reduced correspondingly." 2 Now, this indication of the importance of empirical 3 evidence, in your experience, is that being applied in 4 practice in large projects? 5 A. Yes, in some projects it is. I would say in the 6 majority of projects it isn't, because it's the -- the 7 balance is changing. So more and more projects are 8 beginning to do this around the world, and in the UK. 9 Q. This guidance has been available generally, I take it, 10 since June 2004, and is -- was published -- prepared for 11 and published by the Department for Transport? 12 A. Mm-hm. 13 Q. You have referred there to post audits being required. 14 Are you involved in any capacity for government in 15 carrying out audits of projects once they are complete? 16 A. From time to time, yes. But we are more involved in 17 preparing projects and in the delivery of projects. 18 There's actually not a great interest in post audits. 19 The national audit office here in the UK and the 20 national audit offices in other countries, for instance, 21 Denmark where I'm from originally, are doing this and we 22 work with both of these auditors in developing their 23 methods and data for how to do this, and other auditors, 24 including in Sweden and in the United States, in more 25 countries. 86 1 Q. Do they have a routine programme of auditing projects of 2 a certain size once they are concluded? 3 A. No. These auditors don't have a routine programme. 4 They are doing it more on an ad hoc basis, plus they 5 have been developing frameworks that they recommend to 6 be used in general for auditing projects. 7 The closest we come to a routine programme is the 8 IPA, the Infrastructure and Project Authority, under the 9 Cabinet Office here in the UK. They have been 10 developing a process and methodology for how to do post 11 audits and are carrying out post audits. 12 Q. And the IPA, the infrastructure projects authority, do 13 they publish the findings of their audits? 14 A. They do. 15 Q. I take it they are available to anybody then who wishes 16 to consider them? 17 A. Yes. 18 Q. Do they only look at projects of a certain size? 19 A. They look at major projects, so the larger end of 20 projects, yes. 21 Q. What is required to qualify as a major project? 22 A. Well, this is not like a scientific definition. But 23 I would say generally projects that cost more than 24 GBP100 million. 25 Q. I think it's correct to say they are not just looking at 87 1 necessarily transport projects. They would also look at 2 computing projects and things such as that; is that 3 correct? 4 A. That's correct. They look at government projects as 5 such, and I should also emphasise that the authority 6 didn't exist, and of course their data therefore were 7 not available at the time that we are talking about now, 8 around 2004. 9 CHAIR OF THE INQUIRY: What's the extent of the jurisdiction 10 of the IPA? Does it cover England and Wales and 11 Northern Ireland or -- 12 A. My understanding is that they cover the UK, but you 13 would have to -- or we would have to verify that, to be 14 sure. 15 CHAIR OF THE INQUIRY: So -- thank you. 16 MR LAKE: If we go over to page 37 within your guidance 17 document, and look at the upper part of the page, you 18 say: 19 "Having stated this general rule ..." 20 Which was the cautionary rule about not making 21 adjustments downwards to the uplift figures, you say: 22 "... it must be observed that individual projects 23 may exist where the claims to improved risk mitigation 24 are so strong that downward adjustment of uplifts is 25 warranted in order to avoid double counting. This may 88 1 be the case if advanced risk analysis (eg risk 2 identification workshop and statistical calculations of 3 volume and cost risks for individual project components) 4 has been applied and their results adequately reflected 5 in the established budget." 6 It might be said that the persons involved in the 7 Edinburgh Tram Project had in fact carried out detailed 8 risk analysis. They had also had it externally checked. 9 Is that enough to warrant a downward adjustment of the 10 recommended uplift? 11 A. It is not, and maybe a word is missing here actually, 12 that it should be realistic statistical calculation, 13 because you can have a statistical calculation of the 14 type garbage in, garbage out. 15 Q. Could I ask you to slow down a little? 16 A. Maybe a word is missing here, and that would be 17 "realistic" in front of statistical calculations, 18 because you can have statistical calculations of the 19 type garbage in, garbage out, that we talked about 20 earlier. And you would have an unrealistic assessment. 21 That would not justify a downward adjustment of the 22 optimism bias uplifts. 23 Only if you'd done a realistic assessment of that 24 type, and it turned out that the results showed you that 25 the risks were lower than what you had originally 89 1 assumed, then you would be justified in down adjusting 2 the uplifts. 3 Q. You make the point at the start of the next paragraph 4 that those cases anyway need to be bolstered by 5 empirical evidence? 6 A. Exactly. We have a whole paragraph explaining what this 7 realism would mean. 8 Q. If you look to the foot of the page, under the heading, 9 "Possible pitfalls": 10 "Another important pitfall in employing the approach 11 described above is that forecasters, when estimating the 12 future costs of a specific transport project, would 13 depart from the basic principles of reference 14 forecasting and would gradually return to the practices 15 of conventional forecasting, with forecasters focusing 16 on the details of the project at hand and attempting to 17 forecast the specific events that would influence the 18 future course of this project. The track record of 19 conventional transport cost forecasting shows that with 20 90 per cent likelihood, this would re-introduce optimism 21 bias in forecasting. This pitfall may be avoided by 22 consistently sticking to the method of reference 23 forecasting described above." 24 That's just emphasising that whenever you start to 25 diverge from the reference classes, making any 90 1 adjustments, that is when errors are going to creep in? 2 A. Yes. 3 Q. That would be equally true of making the downward 4 adjustments because they come later in the project? 5 A. Yes. 6 Q. I have finished with that guidance document for the 7 moment. 8 What I would like to do now is look at the STAG 9 guidance. Reference CEC02084489. 10 Just bear with me one moment. 11 Although we don't see a date in this document, you 12 note that this guidance was in fact issued in 13 September 2003. Is that your understanding? 14 A. Yes. 15 Q. If we go to within this guidance to page 4, and look at 16 paragraph 12.4.1, you can see it's suggested here that: 17 "Reference should be made to the Treasury Guidance 18 for the specific upper and lower bounds for the 19 contributing factors to optimism bias. These will 20 differ depending on the nature of the project. There 21 are six specific project types." 22 And they are narrated. 23 I appreciate this was obviously published the year 24 before your guidance document. And it seems to be 25 harking back to what was contained in the Mott MacDonald 91 1 work and the Green Book; is that correct? 2 A. Correct. 3 Q. So on the basis that the Green Book and the 4 Mott MacDonald are no longer the most up to date and the 5 most reliable, the same would follow this STAG guidance 6 in referring to them? 7 A. Correct. 8 Q. Could we look at 12.4.2 and 12.4.3. 12.4.2 says: 9 "The majority of transport projects will be 10 classified as either standard or non-standard civil 11 engineering projects." 12 Just pausing there, that's because they don't have 13 the benefit of your more refined classes, more accurate 14 classes produced later? 15 A. Yes. 16 Q. Then if we read forward to 12.4.3, the guidance says: 17 "Ideally, rather than use these generic factors, 18 adjustments for bias should be based on empirical 19 evidence from past and/or similar projects and adjusted 20 for the unique characteristics of the projects in hand. 21 It is anticipated that further work will be carried out 22 by the Executive and other bodies in order to refine the 23 figures for optimism bias in transport projects. Advice 24 on applying optimism bias factors should be sought from 25 the Scottish Executive at an early stage of project 92 1 development. Before reaching decisions, both cost 2 estimates and adjustments for optimism should be 3 independently reviewed." 4 Are you aware of whether or not the Scottish 5 Government has issued any further guidance after this? 6 A. No, I'm not aware of this. 7 Q. In the statement at the start of 12.4.3, that rather 8 than use generic factors, which seem to be the standard 9 and non-standard civil engineering projects, it 10 recommends that adjustments for bias should be based on 11 empirical evidence from past and/or similar projects, 12 and adjusted for the unique characteristics of the 13 projects in hand. 14 Now, that seems to be guidance to go away and try 15 and find past or similar projects yourself. 16 A. Yes, yourself or others. So basically, this is the 17 sentence that is taken from the Green Book. 18 Q. In a sense of choosing past or similar projects to make 19 up your own reference class, and then adjusting it for 20 the unique characteristics of the project in hand, just 21 as with the Green Book, is that going to create the very 22 sort of subjectivity and error that this exercise is 23 intended to avoid? 24 A. No, it wouldn't, if you followed the procedure for 25 selecting the reference class that we talked about 93 1 earlier, then it would bring more realism into the 2 assessment. 3 Q. But then if you make an adjustment for the unique 4 characteristics of the project in hand? 5 A. That's where the door is opened again for optimism and 6 other biases. 7 Q. If you could look at the following paragraph, the part 8 of it over the page: 9 "Having adjusted for optimism, the planner should be 10 in a position to provide a better estimate, earlier on, 11 of key parameters. Enforcing these adjustments is 12 designed to complement and encourage, rather than 13 replace existing good practice in terms of calculating 14 project specific risk adjustments and contingency 15 allowances." 16 I take it you wouldn't disagree with any of that, it 17 is important still to analyse risk and seek to reduce 18 it? 19 A. Yes. 20 Q. If we read on: 21 "They are also designed to encourage more accurate 22 costings. Accordingly, adjustments for optimism may be 23 reduced as more reliable estimates of relevant costs are 24 built up, risks are explicitly assessed and quantified, 25 and work to minimise project-specific risk is 94 1 undertaken." 2 Is this the same flaw as we saw in the Green Book 3 guidance, and the Mott MacDonald, in that it assumes as 4 the risk work is developed, it would be appropriate to 5 reduce optimism bias? 6 A. In my view that is the same flaw, yes. 7 Q. That's quite at odds with what was contained in your 8 guidance about the need for an objective basis for 9 reductions, either your interpolation or something 10 evidence based? 11 A. Yes. 12 Q. So once again, if you were asked to consider the 13 validity of this advice for future projects, what is 14 your view as to how valid this advice is? 15 A. I would say that this advice is problematic in the sense 16 that it -- it's counter productive in the sense that it 17 would -- it might defeat the very purpose of what the 18 exercise is about, which is rooting out bias and this is 19 reintroducing. 20 Q. If you look at the following paragraph, 12.5.1, it 21 notes: 22 "As project design and development progresses, it 23 should become possible to explicitly quantify and value 24 risk factors. Ultimately, appraisers should aim to 25 adjust costs and benefits in order to calculate risk 95 1 adjusted 'expected values'. As the previous section 2 explained, in the early stages of an appraisal, these 3 adjustments may be encompassed by a general uplift to 4 a project's net present cost, to offset and adjust for 5 undue optimism. But as the appraisal proceeds, more 6 project specific risks will have been identified ... 7 reducing the need for the application of more general 8 optimism bias factors." 9 Could you comment on that advice? 10 A. In theory it is correct that as you do more and more 11 detailed risk analysis, and as time passes, risks become 12 lower and therefore you should adjust optimism bias 13 uplifts accordingly. But like we talked about earlier, 14 this is where you have the risk of reintroducing 15 optimism bias, and therefore we suggest this more 16 mechanistic approach, where you simply looked at how 17 much of the budget has already been spent, and then you 18 adjust according to that, instead of adjusting to the 19 more subjective assessments of how much risk is taken 20 into account. 21 What we've seen is that if you do it this way, you 22 tend to end up with optimism bias uplifts that are too 23 low. 24 CHAIR OF THE INQUIRY: When you say this way, you mean the 25 way envisioned in this paragraph. 96 1 A. Correct. 2 CHAIR OF THE INQUIRY: What does your more mechanistic 3 approach involve? 4 A. That was the table we looked at earlier in the 2004 5 report, where we said if 10 per cent of the budget has 6 been spent, adjust the optimism bias uplift down by 7 10 per cent. If it 20 per cent of your budget has been 8 spent, downwardly adjust the optimism bias uplift by 9 20 per cent, et cetera. 10 CHAIR OF THE INQUIRY: Thank you. 11 MR LAKE: You've already said this was based on the 12 Mott MacDonald work. If we could go back and look at 13 that for a moment, please, it's document CEC02084689. 14 If we could go back to page 40 that we looked at before, 15 and the lower part of the page. 16 We looked at the second last paragraph which 17 commented on the upper bound values, but if we could 18 look at the very last part of the page, it says: 19 "The lower bound values identified represent the 20 optimism bias level to aim for in current projects with 21 effective risk management by the time of contract 22 award." 23 Now, looking at those lower bound values, for 24 non-standard civil engineering it's 6 per cent for cost, 25 and 3 per cent in respect of standard civil engineering. 97 1 It's suggested that these are the levels of optimism 2 bias you can aim for by reasons of effective risk 3 management. Is that a valid approach? 4 A. In my view, no. So I know enough data now. But we have 5 to be fair to both Mott MacDonald and to the Edinburgh 6 tram project management. They did not have the 7 knowledge that we have today. This is very early 8 stages. This was completely new territory at the time. 9 So the Mott MacDonald report is a very rudimentary 10 attempt to get a grasp on the issues here, and there are 11 lots of biases and errors in it we can see with the 12 knowledge that we have today, and this is one of them. 13 So these lower bound values are way too low, given what 14 we know today about projects, that at this stage that 15 they are talking about here, you actually need 16 substantially higher optimism bias uplifts than the ones 17 that are indicated by the lower bound here. But they 18 didn't know that at the time. They simply didn't have 19 good enough data to establish that. 20 Q. The impression appears to be given by this guidance that 21 the upper bound is when you have no effective risk 22 management, and the lower bound is what you must aim for 23 by effective risk management? 24 A. Yes. 25 Q. Which appear to give the impression that once you get to 98 1 a risk management place, you can just drop it right down 2 to the 6 per cent or the 3 per cent? 3 A. Yes, that's the idea that you get from this. 4 Q. And in a sense, again, you have to be fair to people 5 trying to conduct projects in 2007 and 2008, that that 6 was the guidance they were being given. 7 A. Yes. 8 Q. But it's not very good guidance, is it? 9 A. In my view, no. Also in 2007 and 2008, the 2004 data 10 were available. So there were better data available. 11 But in 2002 and 2003, that was not the case. 12 Q. Had the 2004 data been used, that's your work, the 13 result would have been a larger allowance would have to 14 have been made for optimism bias? 15 A. Yes. 16 Q. Now, that's quite unattractive for a project to have to 17 make that larger allowance, if they're trying to fit 18 within a certain budget? 19 A. That depends on the philosophy of the project management 20 and of the owner. To some project management and 21 owners, that is actually very attractive. They want to 22 be realistic. They know -- want to know what they're 23 getting into, and they also want to have the money if 24 they get into a situation where it is needed, because 25 experienced project managers know that it's a terrible 99 1 situation to get in for a project, that you don't have 2 enough funds, you don't have enough time. 3 It totally disrupts everything, and increases cost 4 and time, in its own way, the vicious circle that we 5 talked about earlier. 6 So from that point of view, it is desirable, 7 attractive to have the contingencies. But from a more 8 conventional point of view, where people are afraid of 9 contingencies, first of all, they are afraid they might 10 not get their project approved and funded if it has 11 a large contingency, because it's more expensive and 12 a more expensive project is less likely to be approved 13 and funded. That's the first thing. 14 The second thing is that some managers are also 15 afraid to have a large contingency, because they know if 16 the contingency is there, the builders, the contractors, 17 are going to go for it, and therefore, just having the 18 contingency might drive up costs. 19 So from that point of view, it might be 20 unattractive, yes, to have a large contingency. But 21 those are really two fundamentally different 22 philosophies that you find out there in reality on real 23 projects. 24 Q. We have just been looking at the Scottish Transport 25 Appraisal Guidance. I'll come to look in a moment at 100 1 the English version, but sticking for a moment with 2 Scotland, we have the Scottish guidance appears to take 3 the approach in the Mott MacDonald work and the initial 4 Green Book? 5 A. Yes. 6 Q. That's one strand of advice? 7 A. Mm-hm. 8 Q. The other strand of advice is your advice from 2004, 9 which, although up to date, has not yet been reflected 10 in the Scottish guidance over the last 15 years or so. 11 A. Yes. 12 Q. In terms of persons trying to implement projects, I take 13 it, it will be a simple thing to say that it would be 14 better if there was not conflicting guidance, there was 15 one set of authoritative guidance? 16 A. I would agree with that. The existing set of guidances 17 are confusing on some points. They are not up to date, 18 and they have these traps that will lead planners back 19 into reintroducing optimism into the planning process. 20 Q. If we could turn now to the Department for Transport, 21 Transport Appraisal Guidance, please. It has reference 22 CEC02084255. 23 We can see the title sheet there. 24 Could we look firstly at page 24. I would like to 25 look at section 3.6. 101 1 This guidance notes: 2 "As a project develops, the Department expects the 3 scheme cost estimate to be refined over time. As it 4 becomes possible to better quantify and value risks, it 5 should be possible to better capture the factors that 6 contribute to appraisal optimism within the risk 7 management process. Therefore, as risk analysis 8 improves as a scheme develops, it is expected that on 9 average the risk-adjusted scheme cost estimate will 10 increase while the applicable level of optimism bias 11 will decrease. 12 "It follows that in general, the Department expects 13 that the allowances for optimism bias should be the 14 largest at the initial stage of the life of a transport 15 project (eg Strategic Outline Business Case). This 16 allowance is expected to be smaller in a more detailed 17 business case (eg Outline Business Case) and smaller in 18 the presence of a fully detailed business case (eg Full 19 Business Case). 20 "As a scheme progresses, there are techniques for 21 reducing optimism bias uplifts through greater certainty 22 over costs and use of risk mitigation measures, and 23 independent reviews of risk and optimism bias. The 24 promoter will be expected to provide reasoning and 25 justification for any reductions in optimism bias 102 1 adjustment, from the recommended optimism bias uplifts." 2 Now, this seems to be suggesting there has to be 3 some reason for reducing it, but it's still not going to 4 your mechanistic interpolation approach, and therefore 5 do you consider that there's still room for subjectivity 6 and error? 7 A. There is. I mean, if reasoning and justification is 8 taken really seriously, then there would be less 9 subjectivity, but this -- the answer is that this is 10 also a door open for reintroducing subjectivity. 11 Q. To that extent do you consider that this guidance would 12 be more appropriately modified? 13 A. Would be more appropriately -- 14 Q. Would we better have this guide modified, so it didn't 15 leave that door open? 16 A. Yes, I believe there's a problem here that the 17 quantitative risk assessment, the QRA that we talked 18 about earlier, whatever method you're using for trying 19 to understand and reduce risk, is at odds with the 20 optimism bias uplift. So you have these two different 21 methodologies together in this exercise, and what 22 happens as you lean more and more on the QRA is that the 23 optimism bias uplifts become reduced and, as we see in 24 the Edinburgh case, completely eliminated in fact. 25 The balance between the two methods is not 103 1 appropriately addressed in the different guidances, 2 including this one. That's the problem. 3 Q. Focusing in particular on what is said in 3.6.3, the 4 last sentence, it says: 5 "The promoter will be expected to provide reasoning 6 and justification for any reductions in optimism bias 7 adjustment." 8 Now, obviously it's better to have reasoning and 9 justification than nothing, but that seems quite 10 different from the empirical evidence that you required? 11 A. Yes, exactly. 12 Q. If we go to paragraph 3.7.4, on the following page, we 13 can see that there are four stages or four steps 14 identified to carrying out the optimism bias assessment, 15 the first being to determine the nature of the project, 16 and the second one, which is a new one, is to identify 17 the stage of scheme development. Do you see that? 18 A. Yes. 19 Q. I think this guidance for the first time introduced 20 expressly different levels of uplift for each stage of 21 the project as it moved through. 22 A. Correct. 23 Q. If we could go to page 26, and 3.7.6, we can see that 24 for the purposes of this guidance: 25 "The Department has identified three main stages in 104 1 the life of a transport project for which default uplift 2 values have been provided, as illustrated in Table 8." 3 If we look at that table, if we scroll down 4 a little, we can see that for railways, for present 5 purposes, you've got three stages, grip stage 1, grip 6 stage 3, and grip stage 5, or to use the descriptive 7 terms, pre-feasibility, option selection and design 8 development. Do you see that? 9 A. Yes. 10 Q. Looking at the following page where we finally come to 11 the table which shows the recommended uplifts, we can 12 see here that there are different categories of roads, 13 rail and fixed link. So to that extent it has followed 14 the approach in your guidance of separating out the 15 different types of project in more detail. Do you see 16 that? 17 A. Yes. 18 Q. But nonetheless we now have the stage guidance, if we 19 look at rail, veers from 66 per cent in the first stage, 20 40 per cent at stage 2, and 6 per cent for stage 3. Do 21 you see that? 22 A. Yes. 23 Q. Now, the figures of 66 per cent at stage 1 and 24 6 per cent at stage 3, I think were respectively the 25 upper bound and the lower bound for non-standard civil 105 1 engineering projects at Mott MacDonald? 2 A. Mm-hm. 3 Q. Do you know, were those figures actually the ones 4 transposed into this document, or was new work done and 5 it's just coincidental that it arrived at exactly the 6 same figures? 7 A. I don't know that. 8 CHAIR OF THE INQUIRY: At the foot of the table, you see 9 beside the figures there's an asterisk, and at the foot 10 of the table it refers to both your report and 11 Mott MacDonald's, 2004 and 2002. 12 A. And that would indicate to me that additional work has 13 not been done since they are building on these two 14 sources, yes. 15 CHAIR OF THE INQUIRY: But if you then look at the note, it 16 refers to the Department for Transport currently 17 undertaking further research into optimism bias in rail 18 schemes. And table 9 will be revised in light of the 19 results of that research. But in the meantime, anyone 20 undertaking a rail appraisal should refer to the 21 forthcoming guidance on rail appraisal. 22 A. Correct. 23 CHAIR OF THE INQUIRY: So does that mean that there was 24 draft guidance or something? It speaks about 25 forthcoming guidance. How would you get access to that, 106 1 or how would a developer get access to that? 2 A. I'm not aware that any draft guidance was available, 3 and I guess the way that developers would get access to 4 this was by asking. 5 MR LAKE: I wonder if it would be possible to have two 6 documents on screen side by side. The page at the 7 moment would be one side of the screen and then on the 8 other side would be the page from the Mott MacDonald 9 report with reference CEC02084689, page 40. 10 Just bear with me, Professor Flyvbjerg. 11 On the right-hand side of the screen we have the 12 Department for Transport TAG Guidance, and on the 13 left-hand side of the screen, we have the Mott MacDonald 14 report. Do you see those? 15 A. Yes. 16 Q. We can see there the figures of 66 per cent and 17 6 per cent appear in the TAG guidance to the right-hand 18 side as the stage 1 and stage 3 respectively for rail 19 projects, whereas on the Mott MacDonald guidance, they 20 are the upper and lower bands for non-standard civil 21 engineering projects? 22 A. Yes. 23 Q. In the Mott MacDonald guidance we were told that the 24 lower bound was a target for which the parties should 25 seek to aim throughout the project? 107 1 A. Yes. 2 Q. They would do that by means of risk management? 3 A. Yes. 4 Q. You've already expressed some concern about whether or 5 not that was a truly valid approach? 6 A. Yes. 7 Q. If we look at the Department for Transport TAG Guidance, 8 it's been elevated from being a target, albeit 9 a questionable one, to being an assumption that the 10 optimism bias will be reduced to 6 per cent at that 11 stage 3? 12 A. Yes. 13 Q. Is that a valid approach? 14 A. I would say no. In my view, no. 15 Q. In your experience, would it be reasonable to assume 16 that by the stage 3, optimism bias has in fact been 17 reduced to merely 6 per cent? 18 A. It is not, but again, I have to say, I have the benefit 19 of hindsight. We know so much more today. So I know 20 that usually optimism bias cannot be reduced to this 21 level of 6 per cent at this stage in the project cycle. 22 If you want to be realistic at this stage, you need 23 a number that is higher. And I would say as a general 24 rule of thumb, whenever you see a single digit number, 25 the red lamps, the warning lamps should go on. That's 108 1 a warning sign, because very rarely are the risks single 2 digit. 3 Q. Are you aware whether there's any published guidance to 4 that effect, saying that the 6 per cent figure is not 5 a realistic one? 6 A. Not that I am aware. 7 Q. If we could change the document on the left-hand side, 8 that is the Mott MacDonald guidance, and substitute it 9 instead with the 2004 work you did, that's reference 10 CEC02084257. It's page 34. 11 We can now see that it's the work that you 12 participated in on the left, and in terms of the rail 13 figures, whereas on the right-hand side, the stage 2 is 14 given as 40 per cent, it's difficult to fit that exactly 15 into your figure, save that 40 per cent was the 16 50 per cent percentile figure for rail? 17 A. At the Final Business Case stage, yes. 18 Q. At the Final Business Case stage. 19 A. Yes. 20 Q. Perhaps the difficulty is that's not the stage that the 21 DfT guidance recommended that it be applied at, because 22 it's being applied at option selection? 23 A. Yes. So it's unclear exactly where option selection is 24 in relation to outline Business Case, and Final Business 25 Case and tendering. So those are just two different 109 1 baselines in the project cycle, and -- but apparently 2 the same number is being applied. So our number is 3 strictly empirical, and that number is then applied in 4 the TAG document for stage 2. 5 Q. Do you consider that's a valid transposition of your 6 number? 7 A. Not really, but I would have to know more about option 8 selection exactly where they see this in relation to the 9 Final Business Case, and the further it is from the 10 Final Business Case, the more inaccurate this assumption 11 would be. 12 Q. In your guidance you provide, as we can see to the 13 left-hand side of the screen, the different percentiles 14 for confidence levels, and we have no information at all 15 in the DfT guidance as to what confidence level they are 16 assuming. Does that affect the usability of this 17 information? 18 A. In my view, yes. 19 Q. Why? 20 A. Because you need to know what the probability is. It 21 relates to this thing we talked about earlier, about 22 buying insurance. If you don't know what level you're 23 insured at, the numbers are not much use, and that's 24 actually the situation with the TAG report, is that you 25 need to know what level of -- what level of security you 110 1 have with the numbers that are given. And that's not 2 immediately evidenced. 3 Q. Bear with me one moment. 4 If we go to page 28 now of the document on the 5 right-hand side, and we just look at the paragraph at 6 the top of the page. It notes: 7 "If the Department is provided with sufficient 8 evidence, it is possible to use uplifts other than those 9 recommended. Uplifts that deviate from the recommended 10 uplifts will reflect both the stage of development of 11 the option, the quality of the risk assessment provided, 12 and the extent to which optimism bias may or may not 13 have been mitigated. The Department does not expect to 14 see uplifts used that are below those given for the next 15 stage of scheme development in table 9." 16 Once again, that suggestion of using evidence to 17 depart from the uplifts, is that, do you consider, 18 a problem, or does the fact it requires evidence mean 19 that it should be problem free? 20 A. It is correct that you need to adjust the optimism bias 21 uplift down over time. So in principle this is correct, 22 and it's also right that evidence is required. The 23 question is what is evidence? And I would say empirical 24 evidence. So if it's empirical evidence it's okay. But 25 if it's evidence that is more arbitrary, stemming from 111 1 the QRA, then it's problematic in my view. 2 Q. Okay. I'm finished with those documents on screen now. 3 CHAIR OF THE INQUIRY: Can I just ask, Professor, the 4 impression I get is that the STAG guidance is out of 5 date, and if you were planning a major project, 6 something of GBP100 million plus, would it be prudent to 7 rely on the STAG guidance to establish the Business Case 8 or to go forward, or what would your advice be about 9 information that should be -- 10 A. Do you mean today or at the time? 11 CHAIR OF THE INQUIRY: Today. 12 A. Today? I would say it needs to be updated. But to be 13 fair to the STAG guidance, it reflects the time that it 14 was made, 2003, and it reflects the information that was 15 available at that time. There are some problematic 16 things, like the relationship between the QRA and the 17 optimism bias uplifts, but otherwise it really reflects 18 its time. 19 But it's not adequate for today. It needs to be 20 updated. 21 CHAIR OF THE INQUIRY: We know that an extension of the 22 tramline is being contemplated, and the likely cost 23 would put it into a major project. Really what I was 24 seeking is: what should be done to try to avoid the 25 difficulties that arose in the line to date? 112 1 A. So I would strongly advise to update the guidance in 2 light of all the information that we have today, which 3 is a lot more than we had when it was first made in 4 2003. 5 CHAIR OF THE INQUIRY: Thank you. 6 MR LAKE: Could I ask you then to look at your report again, 7 which is reference TRI00000265. Can we look at page 22, 8 please. 9 We can see here we've got the heading, "Expert 10 Evaluation", and you start with: 11 "Treatment of Optimism Bias in the Draft Interim 12 Outline Business Case". 13 Now, most of this we can read for ourselves. What 14 I would like to do is look at a passage starting at the 15 top of the following page. 16 We see there it notes that: 17 "... the Draft Interim Outline Business Case argued 18 that the headroom between the then base cost estimate 19 and the funding envelope was 54 per cent and thus 20 covered the project's cost plus 44 per cent of optimism 21 bias uplift suggested by the Mott MacDonald data." 22 First of all, that 44 per cent assumed this was 23 a standard civil engineering project. You said in your 24 view that was bogus because those are all road projects? 25 A. Correct. 113 1 Q. Then it says: 2 "It should be noted that the DfT guidance on 3 optimism bias procedures available at the time shows 4 that at P80 rail projects ought to use an uplift of 5 57 per cent." 6 You also give the figures for P90 and P95, and that 7 was the report that you co-authored? 8 A. Correct. 9 Q. You say then: 10 "At P80 the headroom between the base cost and the 11 funding envelope would have been used up." 12 Just to be clear what that's telling us, if they had 13 used the 2004 guidance that you co-authored, they would 14 have known that for a rail project at P80, if they did 15 want to take a P80 approach, would have required 16 a 57 per cent uplift in the optimism bias figures? 17 A. Correct. 18 Q. There simply wasn't the headroom between the available 19 funding and the anticipated risk adjusted cost to 20 accommodate that? 21 A. Correct. 22 Q. So had the calculation been done appropriately at the 23 time, it would have at least indicated a concern as to 24 how things would be managed? 25 A. Yes. 114 1 Q. You follow that by saying: 2 "In the view of the experts, the Draft Interim Outline 3 Business Case overstates its case with regards to cost 4 risk. The project team argued that it would deliver 5 according to the budget envelope with more than 6 95 per cent certainty while the data in DfT's guidance 7 on optimism bias, which were available to the planners, 8 suggested that a 20 per cent risk of exceeding the 9 funding envelope existed, ie a risk four times higher." 10 Now, 95 per cent we know was the figure that was 11 being advanced by the scheme promoters as being their 12 level of confidence as to their costs. Is that where 13 you get that figure from? 14 A. Yes. 15 Q. When you talk about the 20 per cent risk of exceeding 16 the funding, is that because there wasn't sufficient 17 headroom to achieve the P80 figures? 18 A. Yes. So what we basically do here is that we look at 19 what is available, and then we look at our data from the 20 2004 report, and from that we can read with that budget 21 there's actually a 20 per cent risk that the budget will 22 be higher than what is available. 23 So therefore instead of a 5 per cent risk, so the 24 5 per cent above the 95 per cent, there's a 20 per cent 25 risk, the 20 above the 80 per cent, the P80, and that's 115 1 four times higher than -- the real risk is four times 2 higher than the assumed risk. 3 Q. Or rather the risk that was being reported? 4 A. Yes, exactly. 5 Q. We then see the heading. You go on to look at the 6 treatment of the optimism bias and risk in the Final 7 Business Case. 8 Again, we can read most of this to ourselves, but if 9 I can look at a passage on page 24 first. It's the 10 third paragraph on that page, beginning "In the experts' 11 view". You say: 12 "In the experts' view, there are some doubts about 13 the quality of the quantitative risk analysis which 14 estimated a cost risk of 15 per cent at P90, which seems 15 low given the high level of confidence and the evidence 16 in the official guidance documents available to the 17 project at the time, when the project created the Final 18 Business Case." 19 Just pausing there, when you talk about the 20 documents available to the project, you would be 21 including your own guidance in that? 22 A. Yes. 23 Q. We can look there, we saw the P80 uplift you have 24 already referred to for a rail project would be 25 57 per cent? 116 1 A. Yes. 2 Q. It just goes up and up from there as you get to P90, 3 disproportionately larger? 4 A. Yes. 5 Q. So with that document available, and they do a cost risk 6 analysis which brings out a figure of 15 per cent at 7 P90, what does that tell you? 8 A. Again, that tells me that the team is being optimistic. 9 Again, this is the QRA has had this effect that the team 10 has arrived at a number that is -- that is simply 11 unrealistic. 12 Q. Again, is that because that was generally available 13 information, could that and should that have been 14 ringing alarm bells that the risk figure was being put 15 as low as 15 per cent? 16 A. In my view, it should, and that's not only hindsight. 17 That's actually with the data available at the time, 18 this should have been an indication that something is 19 wrong here, and what is wrong is that optimism has 20 re-entered the planning process. 21 Q. If we could go over to the following page, page 25, and 22 look at the last paragraph, the Inquiry has already 23 heard evidence to the effect that it was the view 24 expressed in the Final Business Case that instead -- 25 A. Sorry, where are we? 117 1 Q. I'm not reading something from here. I'm just setting 2 the scene. 3 A. Sorry. 4 Q. In the Final Business Case, rather than apply even the 5 reduced 6 per cent optimism bias with a risk probability 6 set at P mean, the decision was made to apply a risk 7 figure at a P90 level, on the assumption that that would 8 make a greater allowance than P mean plus the optimism 9 bias. 10 A. Correct. 11 Q. Are you aware of that? 12 A. Yes. 13 Q. Now, the effect of that is, because we're using a P90 14 figure, that's all out of the QRA, and therefore is 15 entirely an inside view? 16 A. Yes. 17 Q. If they had used the other guidance that had been P 18 mean, which is inside view, plus 6 per cent which is an 19 outside view, that would be correct? 20 A. What do you mean by that would be correct? 21 Q. What I mean by that would be correct? Sorry. Not 22 that -- yes. Piling questions upon questions. 23 You say in your report that that there's a switch 24 from a wholly inside view, and I can see that using P90 25 would be a wholly inside view? 118 1 A. Correct. 2 Q. If they had used P mean plus the 6 per cent optimism 3 bias uplift, would that have had any realistic outside 4 view component in it? 5 A. Yes. So that would have had the 6 per cent realistic 6 outside view, but again, like we talked about earlier, 7 that in my view would also be optimistic. 8 Q. Because the 6 per cent is too low? 9 A. Exactly. 10 Q. But I think you have done an analysis, and the claim 11 that is made in that case, that P90 is larger than P 12 mean plus 6 per cent, is in fact incorrect? 13 A. Yes, according to our analysis, it is incorrect. 14 Q. Perhaps rather than work through it, that's what's set 15 out at the foot of page 25. If I could ask us to go and 16 look on to page 26, that's what you're explaining there? 17 A. Correct. 18 Q. If I could ask you to look in page 26 at the paragraph 19 beginning "Moreover". You say: 20 "Moreover, the project's justification for this ..." 21 That is using the 90 per cent probability: 22 "... the project's justification for this seemingly 23 low estimate of the cost risk exposure was centred 24 around two points: (1) that utility diversion (MUDFA) 25 works had already commenced; and (2) that the procurement 119 1 strategy would significantly de-risk the project. The 2 project expressed the view that at contract award, 3 optimism bias is 0 per cent. In other words, the 4 project assumed that due to ongoing refinements of its 5 risk register and its risk analysis and due to its 6 commercial strategy, all risks will be fully known. 7 This indicates to the experts that the risk management 8 team did not fully understand the nature of optimism 9 bias and, because of this, the team and project would 10 have been particularly prone to such bias." 11 Your view that they didn't understand the nature of 12 optimism bias, is that because if they had understood it 13 they would assume that it would never be 0 per cent? 14 A. Exactly. Risks are never fully known in that sense that 15 you can eliminate the optimistic bias uplift. So the 16 fact that this is eliminated or set at zero is the 17 indication that they don't understand the nature of risk 18 and the nature of optimism bias. 19 Q. Turning to the question of recommendations, we see on 20 page 28 of your report -- again, we can read through 21 that ourselves. I don't propose to go through it just 22 now. But just picking out some of the original points 23 that have arisen in our discussion this morning, I think 24 one of the things in response to a question from 25 Lord Hardie is that it would be useful to have updated 120 1 work on the various reference classes and the databases 2 to produce more accurate, more refined figures -- 3 A. Yes. 4 Q. -- for the uplifts? 5 A. Yes. So we actually -- so we have the updated data. So 6 we can test for this. It's quite easy with the database 7 that we have, and like we mentioned in the report, we 8 have already tested for this and we can actually 9 document that the existing data in the guidances are 10 dated. The up-to-date data will give you different 11 results. 12 For instance, for the P80, which is the value most 13 used, the number is now higher than it is in the 14 guidances, just to take a specific example. 15 Q. That's something which you think could usefully be done 16 at intervals of no more than ten years, and perhaps five 17 years would be appropriate? 18 A. That is my view, yes. 19 Q. If that guidance is to be provided, do I take it from 20 our discussions that certainly it would be useful to be 21 broken down into different types of work, as in rail, 22 road and so on and so forth, and that there be 23 probabilities provided for those? 24 A. Yes. 25 Q. In terms of the reduction over time over the course of 121 1 a project in the optimism bias uplifts, is that 2 something where some clarity is needed as to the basis 3 on which it is done and how appropriate that 6 per cent 4 figure is? 5 A. Yes, very much so in my view. 6 Q. Could we look quickly at page 30 of your report. If we 7 look at the paragraph beginning "To provide adequate 8 challenge and control". This is under recommendations 9 to people engaged in projects, rather than government. 10 You note that: 11 "To provide adequate challenge and control, the 12 governance bodies need to receive unbiased and 13 up-to-date information about project performance. In 14 similar projects the experts found that effective 15 governance relies on multiple channels of information to 16 senior decision makers, for example, data-driven reports 17 on project performance and forecasts combined with 18 reports by the management team and independent audits. 19 In the reporting, special emphasis must be placed on 20 detecting early warning signs that cost, schedule and 21 benefit risks may be materialising, as they tend to do, 22 so damage to the project can be prevented. When early- 23 warning signs emerge, projects should revisit their 24 assumptions and re-assess risk and optimism bias 25 forecasts." 122 1 What you are saying there is if it doesn't appear to 2 be going according to plan, that's a sign your risk 3 forecasts may need to be revisited? 4 A. Yes, and it's also a sign that you need to take action 5 immediately. And we found that that is a weakness in 6 many projects, that when the early warning signs appear 7 action is not taken. 8 First of all, sometimes people don't want to report 9 the truth upward in the organisation because that's 10 often not seen as a positive thing, somebody saying that 11 this project is going off-track, we won't be able to 12 meet the schedule or the budgets. Or even if it's 13 reported, it's not clear who is supposed to take action. 14 It's another thing we find, that it's not clear which 15 part of the organisation is supposed to deal with this. 16 This actually hasn't been specified as part of the 17 delivery. 18 So, again, that means that action is delayed and not 19 taken, and that's the worst thing that can happen. The 20 more time that passes between a problem arises and it's 21 dealt with, the more likely it is that it will have 22 a large impact, a negative impact, on schedule or costs 23 or both. 24 Q. The Inquiry has heard evidence that part of the 25 procurement strategy for the tram project was based on 123 1 getting design carried out and completed at an early 2 stage, so it was intended that the detailed design would 3 be 100 per cent complete at contract award, and also 4 that the works to divert the utilities would be complete 5 or largely complete at the time the infrastructure work 6 started. 7 Again, if you can just take it from me that the 8 Inquiry has heard evidence that that wasn't happening, 9 that the design was late and the utilities works weren't 10 being completed, is that the sort of thing that should 11 have immediately raised concern that the risks had not 12 been properly understood? 13 A. Yes, that's typical. Those are two typical early 14 warning signs that not only has risk not been properly 15 understood, but also that risks are going to be 16 increasing from now on and you need to deal with it 17 quickly. 18 Q. That's what I was going to say. What would the 19 appropriate step be to take in response to that 20 situation? 21 A. Two things. First of all, deal with it quickly so that 22 the damage is limited. 23 Second, readjust the risk assessment and risk 24 management accordingly, so it takes into account the 25 change that has happened. 124 1 Q. If there are going to be external checks upon a project 2 from time to time, in your view, is one thing that would 3 be particularly useful to check whether or not there are 4 these divergences or early warning signs from the 5 proposed risk treatment strategies? 6 A. Yes. 7 MR LAKE: Thank you very much, professor. 8 A. You're welcome. 9 CHAIR OF THE INQUIRY: I don't think there's any other ... 10 Thank you very much, professor. You're free to go. 11 We appreciate your assistance. 12 A. Thank you very much. Thank you. 13 (The witness withdrew) 125 1 INDEX 2 PAGE 3 PROFESSOR BENT FLYVBJERG (sworn) .....................1 4 5 Examination by MR LAKE ........................1 6 7 MR STUART FAIR (sworn) .............................126 8 9 Examination by MR MACKENZIE .................126 10 11 Questions by CHAIR OF THE INQUIRY ...........186 12 13 Closing remarks by CHAIR OF THE ....................197 14 INQUIRY 15 16 17 18 19 20 21 22 23 24 25 199