For the prof #225lols

We have been invited by @HallyMk1 to contribute some songs to the open playlist to accompany his eagerly awaited inaugural professorial lecture:

2+2=5: The university and the secular crisis

Well, I added a few and thought I’d share the thoughts that I had on selecting them.  We were asked to consider on the keywords

education; faith; mass intellectuality; 
courage; solidarity; love; University; crisis

My selections don’t necessarily fit those tags perfectly but all have a personal story for me.  They also reflect strongly my current state of mind, working in a University and living in a country at a time when I deeply disagree with the political decisions being taken at all levels.  Alongside and allied to that, I find the state education system that I have fought and stood up for for so long failing one of my children.  These songs help me to describe those feelings, as well as giving me comfort and relating the solidarity and love that I think we need to resist.  I find all these songs (well, maybe bar one!) powerful and beautiful.
Continue reading

Posted in Uncategorized | 4 Comments

Protected: My position on climate change and its modelling

This content is password protected. To view it please enter your password below:

Posted in Uncategorized | Enter your password to view comments.

Statistics, rain, flooding and climate?

Discussing the causes and effects of climate change tends to be emotive.  Even more so on social media than in person, it seems.  A playful prance through the first page of results from Google searching “climate change blog” can convince you of that if necessary.  Although I normally stay out of such discussions – particularly when they resort to hurling polemic and ad hominem comments. However, I have been tempted to offer my opinion on one particular piece and so here goes.

A 5-sigma event (or not)

I happened to notice a comment on twitter which said “how can 2 x ave be a 5-sigma event?” and I dipped my toe in the water (admittedly without context at this point) and responded to the effect that “2 x mean of a random variable could be 5-sigma event, you need to know the variance”.  The exact verbatim tweets, ensuing discussion and my commentary are storified here.  If you need a reference for what sigma refers to here – Wikipedia is sufficient.

I realised that the prompt for the conversation was this blog post.  In it, the author makes the assertion that “The winter of 2013/4 has seen a 5-sigma event in southern Britain.”.  The article doesn’t actually say what that event was, so it is very hard to test the claim, but the context given in the article coupled with recent news stories implies that what it refers to is total rainfall  link it with the Somerset flooding and the met office chief scientist’s statement on the same [1].  The addendum to the blog post implies that the 5-sigma event in question is the total rainfall in January 2014. Before I investigate that claim, I’d like to make clear that I do not doubt whatsoever the motives or integrity of the blog writer (who I do not know).  I imagine the blog to be written from a perspective in which climate change is the largest problem that the world faces and that many perceived instances of “worsening” weather seem to indicate that it is happening apace.

That being said, there is no data referenced or provided to justify the very specific 5-sigma claim.  Much of what is written on the blog about distributions is reasonable – but it is generic and does not cover the application to the specific dataset in question – other than to point out that rainfall cannot be negative and therefore the assumption that the distribution is Normally distributed is just that – an assumption.  However, it’s a reasonable assumption – commonly made in first order analysis of the type that we discuss here.

If we assume that the variable in question is the total January rainfall, The Met Office provides the data by which we can test the claim. I looked at this page and calculated the mean and Standard Deviation for the  monthly total rainfall for January for some weather stations near to Somerset.  I’m aware I could and should consider even more data to test the claim exhaustively [2]. I’m also aware that I should probably not use data series of different lengths.  That’s not my point here – as the following section says.  This is provided only to verify the findings of the twitter debate.

Station Data series dates mean Standard Deviation (sigma) sigma level (2 x mean) sigma level (3 x mean)
Hurn 1957-2013 88.6 42.6 2.1 4.2
Hurn 1957-2013 88.6 42.6 2.1 4.2
Yeovilton 1957-2013 70.5 31.6 2.3 4.5
Oxford 1853-2013 56.0 27.8 2.0 4.0
Southampton 1853-2013 81.5 43.7 1.9 3.7

It seems then, that the 5-sigma claim is wrong – experiencing twice the mean rainfall at these weather stations is roughly a 2-sigma event and observing 3 times the mean a 4-sigma event.  It seems unlikely that it would be right even if the exact months under consideration were slightly different to those I have tested – although I’m happy to rerun the analysis on any data provided. What is remarkable to me is how consistent these distributions are – with sigma being ~= half the mean value. This means that seeing 3x the mean value is about a 4-sigma event. Which is very rare.

The political debate

Even had the maths been correct, though, I think there is a deeper problem here. On the surface, the debate is about whether a statistical claim is correct. I contend that what is really at issue are differing (political) belief-systems and the use of statistics to inform (or at least bolster) their claims.

To be clear, the number being calculated and debated is the probability that a single extreme weather event will happen if no Climate Change is happening. The premise of using these statistics and terms is that the observations are drawn from an underlying “true” stationary random distribution.  The implied argument from the blog post, the Guardian articles and Zoe Williams’ tweets appears to run as follows:

  1. If the observation is very unlikely, then the distribution must have changed.
  2. The distribution changing implies climate change (and often the anthropogenic forcing element thereof).
  3. If the climate is changing – that cannot be coped with using Business-as-usual methods.

These jumps are not dictated by the data, but rather depend on a number of judgements – all of which may be debated.

The first jump requires that observing a very unlikely event means the underlying distribution has changed.  This may or may not be the case. The point of probability is that rare events can happen, even with vanishingly small chance and that is why we put a number to that chance. Frequent occurrence of events that your statistical model says should be rare / unlikely may indicate that your statistical model is wrong. For those interested in when one should change ones model – see the Ludic fallacy from Nicholas Taleb’s Black Swan and the difference between frequentist (what we’re doing here) and Bayesian statistics – you could start here or, more humourously, here.

The second jump depends on more judgements.  Firstly, that climate change, would raise the total January rainfall, or increase the variance in total January rainfall, making the extremely unlikely event significantly more likely. I am not a meteorologist, so can’t judge this.  It’s worth noting that climate science appears to indicate that warming would increase the amount of water that can be carried in the atmosphere and therefore that we can expect “more intense daily and hourly rain events.” [3]. More intense events at short scale, though, does not necessarily imply increased total. Secondly. it also assumes that nothing else could account for a changing distribution.  In the case of the particular blog post under discussion, the article tacitly conflates the flooding in Somerset with increased rainfall (neglected some other causes which could confound the analysis – amongst others the much reported lack of dredging).

The last jump, it seems to me, is rarely debated in the public realm. I condense it as “whether the weather can be weathered”.  Some, no doubt, can: some not. The more subtle debate about what can and can’t be adapted to tends to be lost in hyperbole emitted by those with honestly held beliefs either that we are piling headlong into climate catastrophe, or that we are ploughing money and effort into something that is not a problem. From my point of view, this is the conversation that must be had.

I think that Climate Change probably is happening.  I base this on a judgement about the mechanisms I have seen described in scientific papers.  But it is a judgement and I try to keep it under constant review – in common with any of my scientific judgements and especially in the light of any new evidence and data.  I think that the analysis of weather and determining when it can be said to have changed significantly enough to indicate climate change beyond reasonable doubt is incredibly difficult.  As well as dealing with single measurements as described here, it needs to take account of a complex mix of observations across many measurements and the number and frequency of “out of the ordinary” events over time.  Rigorously.  It seems to me silly to get bogged down in arguments about statistical mistakes, or to vigorously sling mud back and forth.

The best we can say about individual observations is that they may add to the evidence for a changing climate.

We’re having the wrong debate

One observation, however unlikely, cannot prove or disprove a change in underlying distribution.

Fundamentally, the rarity or otherwise of individual weather observations cannot, in my opinion, provide conclusive evidence for or against climate change. Sophisticated analysis of multiple events is necessary – and this is what groups of scientists at the Met Office (and elsewhere) do.  It seems to me a waste of time and effort for us to quibble over stats, or to cite them as evidence for our arguments unless we’re absolutely sure of what they say and the argument that they can support.

Aside: possible further work…

As a nod to some useful further work, I think an interesting, different approach to connecting extreme weather events with climate change might be possible.  I think a Bayesian framework would help. As I’m not a meteorologist, I have no idea whether this is unusual, or done as a matter of course.  This is a useful technique to update a priori probabilities in light of observations.  It should be able to give a probability for whether Climate change is happening given observations if we have knowledge of the probability of making the observations given climate change is (or is not) happening.  I haven’t got an exact formulation of such an approach, but aim to investigate prior research or formulate my own approach (or both) and blog on this in the near future


1. Dame Julia Slingo is careful in her wording – saying

Dame Julia Slingo said the variable UK climate meant there was “no definitive answer” to what caused the storms. “But all the evidence suggests there is a link to climate change,” she added. “There is no evidence to counter the basic premise that a warmer world will lead to more intense daily and hourly rain events.”

2. I am tempted to write some software to take these data and produce distribution plots, mean and variance. Watch this space – I may eventually have time!

3. See footnote 1 above

Posted in Uncategorized | 11 Comments

Lots of agents

Continuing to stress test the simulation a little.  Running with 45,000 households connected to 3 electricity supplier aggregator agents, with a further 50 or so macro level aggregators representing big generators is presenting no problem.  The households are running their physical heating model every tick and deciding on when to place heating demand every day (48 timesteps), whilst supply aggregators are doing an optimisation every day.  Speed is somewhat reduced, but not terribly so, the process is still taking only 20% CPU and fairly quick – executing a simulated year in some 20 minutes or so.  At this rate, 20 years should take around 400 minutes, or just over 6 hours.  A small concern is the memory usage – 2GB+  Leaving it running to see if / when it crashes out.  On that note – time to go to lunch … 🙂

simulation screenshot with 45,000 households

45,000 households connected to 3 supply aggregators with around 50 macro aggregators (represented top right)

UPDATE: Return from lunch – 1 hour 15 minutes sees the simulation well into year 5 without running out of memory, although the process is now reporting some 4GB memory accrued.  Have to investigate what is growing…

UPDATE 2: Simulation easily reached 20 years – in around 4 and three quarter hours.  Memory use has stabilised – suspect that Java just uses all it is allowed before running the Garbage Collector aggressively.

45000 households simulating 20 years' operation

Screenshot of whole simulation, agent probe and task manager for simulation with 45000 households that has run for simulated 20 years.

UPDATE 3: Made it to 40 years in 9 hours 20.  I think the current configuration will basically run almost for ever, with linear performance in simulated time against wall clock time. Called it a day!

Posted in Uncategorized | 1 Comment

Long running simulation

Been doing a bit of stress testing of my Agent Based Modelling (ABM) simulation. This time for length of simulation run. I’m happy to report that our simulation gobbles memory only very slowly, if at all. Its timestep is half-hourly and it has happily run with a thousand households and some 20 or so other agents for a simulated 171 years (3 million ticks) without crashing or hitting any memory limits. Nice to confirm that. The computational load in each agent is fairly low – they each do a fairly simple optimisation every day (48 timesteps) and a bit of data i/o. Still – nice to see it working well.  As our simulation horizon is likely to be 2080, if that, this should be sufficient.

Screenshot of simulation window after 3 million ticks

Simluation window – see top right – 3 million ticks

Posted in Uncategorized | 1 Comment

Adventures with my Pi (part 1 of…)

Bit of a techie post, this. Mainly for my records, although it might help anyone who has a very similar problem

I have had some fun over the last week or so becoming a bit more familiar with the software side of the Raspberry Pi. Having reproduced our hackathon system and added a rather nicer front end (see a future post for nice video etc.), I decided it was time that I exercised the Pi a bit more. So, I thought I would get a web camera connected up to it and allow people to control the house via the web and observe the change via a video stream. This was all very good and, after a bit of tinkering and some really good hints from this post from SirLagz, and this (which seems to augment the former) I had the following.

  1. Compile ffmpeg and ffserver for the Pi (done on the Pi – don’t do that – use a cross-compiler)
  2. Get ffmpeg to grab video from the Camera (a / Microsoft / Lifecam VX-1000 that was lying around)
  3. Get ffserver to stream video out to a nominated port in mjpeg format

This was all running on the Raspbian OS install as provided on the NOOBS SD card (I’d run an apt-get update and an apt-get upgrade previously). This all worked fine and I was able to present a nice interface. Happy in that knowledge, I left it over the weekend and went home.

When I returned on Monday, all was not well. The web page was no longer working. I ssh’d to the Pi no problem and began trying to find the problem. When I couldn’t even use the tab key to autocomplete directory names – I realised that the Pi’s file system was absolutely full. No disk space left at all. A quick look at /var/log/syslog showed it to be rather large (~3GB!), I then discovered that kern.log was almost exactly the same size. I looked through them and discovered that at some point at the weekend, this message had started to be output roughly every 10 micro seconds!

gspca_main: ISOC data error: [11] len=0, status=-71

Ouch. I truncated the logs and got the thing going again, but wanted to diagnose the bug. So, I googled a bit, and the consensus was that this was a bug in some kernel driver code and the way to overcome it was to upgrade the kernel. I checked, and my current kernel version was 3.6.11+

I used rpi-update, this brought the kernel version up to 3.10.19+. Unfortunately, this made the situation worse – the error line appeared immediately as I started the server, rendering last week’s work useless. I decided to downgrade the kernel again and try to debug more methodically. I used rpi-update’s ability to go back to a specific github raspberry pi firmware commit and went back to the latest version of 3.6.11+ before the version bump. Good, I thought, back to square one. But no, the problem was still there as soon as I started the server. Next, I thought I would go back even further, so picked an arbitrary 3.6.11+ commit somewhere around May (unfortunately I didn’t record exactly which one). This was even worse – the Pi wouldn’t boot at all now! Nightmare – I didn’t really want to lose all the work on getting ffmpeg compiled and set up, but as I was prototyping I hadn’t backed it up (d’oh – have now). So, I thought I’d have a go at repairing… Having never worked at low level (i.e. swapping kernels) before – I would have to learn a bit.

A look at the SD card in my laptop revealed that /boot was empty. I tried copying /boot.bak into /boot and over the files in the boot partition and got the Pi booting again – but the problem was still there. I reasoned that the files I had actually copied in, then, were still a different version of the kernel to the one I had originally. I tried
apt-get upgrade
thinking that this should get me back to the latest stable release, unlike rpi-update which is supposed to be somewhat more bleeding edge. I still had the problem.

At this point I put out an appeal on twitter (thanks to @matthewbloch of @bytemark who gave me some advice) and also went back to the logs and realised that I could tell exactly which kernel I had from the boot logs – the output of uname -a is one of the first

From this, I diagnosed that I was now running with

raspberrypi kernel: [ 0.000000] Linux version 3.6.11+ (dc4@dc4-arm-01) (gcc version 4.7.2 20120731 (prerelease) (crosstool-NG linaro-1.13.1+bzr2458 - Linaro GCC 2012.08) ) #
538 PREEMPT Fri Aug 30 20:42:08 BST 2013

whereas originally I was running

raspberrypi kernel: [ 0.000000] Linux version 3.6.11+ (dc4@dc4-arm-01) (gcc version 4.7.2 20120731 (prerelease) (crosstool-NG linaro-1.13.1+bzr2458 - Linaro GCC 2012.08) ) #
474 PREEMPT Thu Jun 13 17:14:42 BST 2013

I’d also been through #600 (3.10.19+), #557 (3.6.11+) and #604 (2.10.21+) on my way through

Using rpi-update, I found the commit hash for the 13th June build (#474 above) and got truly back to where I started with

sudo rpi-update 2681df6c284b3ca0283721af7cd14df5548377ae

I now have a very tight range of firmware changes (between 13th June and 30th August) to investigate for what caused the problem to become a ‘hard’ failure instead of intermittent and hopefully therefore what is causing it in the first place. And I have a working server again (although I think it must still be prone to the original cause of the intermittent issue).

Having got it working again, I put nginx onto the Pi to serve my site, so that I could reverse proxy the different ports serving the video stream and the controls onto a clean web page with nice URLs. That makes it a bit of a confused stack, with creating pages to be served by CherryPy on one port and ffserver serving video stream on another port all fronted by nginx. I’m not really a web server admin – so that’s my next learning experience. For now, though, it looks reasonably good and running for 24 hours so far. Next battle – to get it exposed to the world…

Posted in Uncategorized | 1 Comment

Markov chains, lottery misses and twitter

Yet again,  I’ve been drawn into looking at a little maths problem by an off-hand twitter remark.

Closely followed by

I couldn’t resist. I replied with

This was based on a simple idea – You choose 6 numbers on a ticket. The draw is 6 main and 1 bonus ball drawn from 49 without replacement (Wikipedia – Lotto UK). To match no numbers (even the bonus) in a single draw, all 7 draws must pick numbers that you have not selected. I call this number \(P_{0}\) and it is given as:

P_{0} & =\frac{43}{49}\times\frac{42}{48}\times\frac{41}{47}\times\frac{40}{46}\times\frac{39}{45}\times\frac{38}{44}\times\frac{37}{43} \\
& = \frac{16112057}{42950292} \\
& \approx{0.375}

I then reasoned, to get 5 in a row (\(P_{5}\)), simply combine the probability of the individual events…

P_{5} = (P_{0})^{5}
\approx 0.00743
P_{10} = (P_{0})^{10}
\approx 0.0000552

As always with probability, though, it is essential to think carefully about what is being asked (a recurring theme of my maths blogging and twittering). I was drawn back to the problem by this:

Scott, as usual, is right. I had answered – “What’s the probability that one specific series of 10 draws will not match your numbers?”. The question really is about how likely is the chain of events to occur in a given time period? Of course – there are many, many opportunities for the chain to occur if you play the lottery every week and have done for some time… Still, I couldn’t quite work out how he had arrived at his number. I decided to re-kindle the dim spark of recognition for statistical properties of Markov chains. I couldn’t remember much, except for the lack of dependence on past history for each step in the chain. However, google leapt to my aid and I’ll link a couple of resources here. First, for those with no knowledge whatsoever of the term – the ever useful Wikipedia is reasonable on Markov chains. The resource that helped me refresh a basic knowledge was from the statslab at Cambridge; course and pdf

I decided to set my problem description up as shown below.


Markov process diagram for chains of non-matching draws

The first state is a “base” state where the last draw you matched at least one number somewhere. The next ten states are reached one by one for each consecutive draw in which you match no numbers. The numbers on the arrows are the probabilities that each will be followed. The final state is known as an “absorbing” state, because once the Markov process enters it, it never leaves (probability of remaining in this state for any other draw is 1). This state represents a real world scenario where (at least) one run of 10 or more consecutive “no matches” has been witnessed.

This can be represented as a probability matrix, or a transition matrix, as shown below, where \(P_{i,j}\) represents the probability of moving from state \(i\) to state \(j\)

\[ P = \small{\begin{bmatrix} 0.625 & 0.375 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0.625 & 0 & 0.375 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0.625 & 0 & 0 & 0.375 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0.625 & 0 & 0 & 0 & 0.375 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0.625 & 0 & 0 & 0 & 0 & 0.375 & 0 & 0 & 0 & 0 & 0 \\ 0.625 & 0 & 0 & 0 & 0 & 0 & 0.375 & 0 & 0 & 0 & 0 \\ 0.625 & 0 & 0 & 0 & 0 & 0 & 0 & 0.375 & 0 & 0 & 0 \\ 0.625 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0.375 & 0 & 0 \\ 0.625 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0.375 & 0 \\ 0.625 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0.375\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 \end{bmatrix}}\]

The problem can be rephrased as “what is the absorption probability for state \(C_{10}\) after \(n\) draws?” Or, “What is the probability that the process will be in \(C_{10}\) after \(n\) draws?”.  This can be obtained by multiplying the transition matrix by itself \(n\) times and then reading the value of \(P_{0,10}\) which represents the probability of being in \(C_{10}\) having started in \(B\) (state 0)

So, the probability of seeing 10 consecutive draws with no numbers matched depends (unsurprisingly) on how many times you play the numbers. If we assume 10 years then the period of play began in November 2003. At least one website has archived all the draws since their inception, so we can directly read off that there have been 1044 draws between 7th November 2003 and 7th November 2013 when the 10 consecutive draws were witnessed. By my reckoning – this means the probability of observing this event at least once over that 10 year period was \({(P^{1044})}_{0,10} \approx 0.0350\) – or 3.5%.  This is about half the probability that Scott suggested.  I checked with him and he didn’t include the Wednesday draw – so we could look at \({(P^{520})}_{0,10} \approx 0.0174\) (1.74%) – even less probable! I rather suspect, though that he has forgotten to take account of the 7th draw of the ball (the bonus), which also should be missed. This changes the numbers in \(P\) significantly – replacing each 0.625 with 0.564 and each 0.375 with 0.436. With that modification, I think \({(P^{520})}_{0,10} \approx 0.0692\). Back to where we started….

Interestingly – I don’t think it makes any difference at all whether (or when) you change your numbers. That’s one to ponder fully another day. Or maybe Scott has a view… 🙂

Posted in Uncategorized | 3 Comments

Nesta Dynamic Demand Challenge Hackathon

I have finally got around to writing this up after a holiday and an extremely busy week! On 18th / 19th October, I attended the Nesta Dynamic Demand hackathon with Dr. Peter Boait, my colleague at the Institute of Energy and Sustainable Development. We were joined by Dr. Chris Bailey a senior software developer at the University of Bristol and Peter’s colleague from Exergy Devices. The hackathon was an opportunity for 10 selected semi-finalists to hack away at the ideas that they had entered into the Nesta Dynamic Demand challenge. All the ideas are listed here; our idea is Demand Shaper.  Apologies if some of the links to the NESTA website still require you to register with the site – it’s out of my hands…


The Demand Shaper Team, from L-R: Richard Snape, Peter Boait, Chris Bailey

As described on the hackathon website, the event was hosted by the Mechanical Engineering department at Imperial College, with specialists in Innovation Design to help us on our way.

We arrived on Friday morning (rather late in my case – sorry all) to learn that the hackathon was a pretty tightly organised affair, with an agenda that ran throughout the night and through until 21:30 on the Saturday. We were immediately set to work repeatedly describing our idea in 2 minutes to a series of experts, including representatives of National Grid, the National Physical Laboratory and academics. The 2 minute pitch was pretty quickly honed and we had some interest and useful insight from the experts.

Next, we were told we had to deconstruct some electrical appliances. This, I thought, was right up our street – just what we came for. We managed to get our electrical fan down to individual heat sinks from the power supply and the spindle and coil from a dis-assembled motor. I wonder if we took the deconstruction to unexpected lengths…

Picture of De-constructing an electric fan

De-constructing an electric fan

The idea of this was to help us focus on the function of each component within the product.  It also provided a number of components for us to use in the rest of the hack.

After this fun we got on to the serious business (?!) of hacking our idea around – we were asked to incorporate some of the appliances provided (de-constructed or otherwise) into our concept.

I already had an Agent Based simulation of the ‘Demand Shaper’ concept running in software (on Linux, Mac and Windows laptops) and Peter and Chris had a Raspberry Pi that could do some intelligent control of appliances via a PiFace interface board. A fairly obvious hack then presented itself – putting some hardware in the loop of the simulation.


We busily hacked software (setting up the Pi) …

This we did, dividing labour between the team. Peter got on with physical wiring, Chris setting up the Pi to take requests from the web (hosting a little web server on it) and me setting up our Agent Based Simulation to talk to the Pi.

Wiring the PiFace

… and hardware (wiring appliances through the PiFace)

We decided to have the system control two appliances provided by the organisers – a light and a heat based device. We initially used a fridge to illustrate our concept of “gapping” the cold load without letting the fridge get warm.  Unfortunately, despite being rated at some 200W, it managed to demand a spike of some 2.5kW on switching on which successfully welded our relay contacts together.  Rather disheartening.  Take two, with another PiFace interface board and this time controlling a hair dryer instead of the fridge.  We managed to do get all this running and got to a point where the simulation could control a few appliances just as if it was controlling a real house.

As well as the demo of system capability end to end, our implementation had the nice feature that a web app could be hosted on the Pi and accessed from any smart phone or Wireless connected device –  so we could let anyone turn our appliances on and off with their smart phone (the app is on the left PC on the video). This demonstrated that there is absolutely no reason why appliances couldn’t be controlled remotely from an app if a user so desired (there would be some security concerns, to be sure, which we didn’t address at the hackathon but would before real world deployment…).  All this can be seen in action on the short video below.

Getting our idea working end to end with hardware in the loop was a first for us and seemed to go down well with anyone who came to have a look.  We finally got this going well into the night, but there was no rest for the wicked (although there was pizza… ).  We immediately had to work on a 5 minute presentation to pitch our ideas to a panel of judges the next day. With the aid of some excellent ideas from innovators and artistic design specialists, who bravely arrived to help grumpy participants at about 2am, we got something together.  I managed to get a bit of sleep from 4am until 6, before jumping up for a quick shower at the gym – even South Kensington is pretty quiet at that time…


Imperial in the early morning – it’s not usually this quiet…

One thing that did surprise me about the hack was the heavy emphasis on business models from the organisers – we had to show in no uncertain terms why our idea would be attractive to customers, both domestic and commercial.  Although less comfortable for me as a scruffy academic (shurely some mistake, Ed.), this helped us hone the (strictly) 5 minute presentation of our idea fro delivery to a panel of judges at the end of the event.  It also forced us to consider how our idea would actually help in the real world – a useful reality check!  In the final minutes – we decided to live dangerously and do a live demo in the presentation, rather than the pre-prepared video above.  Always dangerous in a presentation, especially one so short with no opportunity to recover, but everything hung together well – as you can see in the video…

I’m rather pleased to say that we were selected to go through to the final and will be developing our idea until then – so watch this space for further developments. A list of finalists and semi finalists is available on the NESTA website and their blog will no doubt keep updated with progress from the teams. I won’t extend the post with descriptions of all the ideas – but there were some very nice ones – I particularly liked thEnergy and Community substation challenges.  A big thank you to all the organisers – it was a great opportunity to talk to others engaged in the same area as us and to intensively work on our idea alongside the ideas of other teams.

NESTA_UK’s video summary of the event (twitter @nesta_uk, youtube channel uknesta)


Posted in Uncategorized | 4 Comments

PV adoption in response to Feed in Tariff : a video visualisation across time and space

I have decided to post a little visualisation of an element of my research here. As part of parameterising a model on technology adoption in the Smart Grid, I have analysed the uptake of technology in response to the Feed in Tariff in the UK. For those who don’t know, the Feed in Tariff is a scheme which pays those who have installed microgeneration a premium for every kWh that their generator produces. This is paid whether the electricity is consumed locally or exported to the grid. There is a small premium (currently 4.5 p/kWh – originally just over 3p/kWh) for electricity exported to the grid.

I have looked particularly at PV systems adopted in the domestic context (i.e. households deciding to purchase their own solar panels). The pattern of adoption varies both geographically and over time. I produced the little visualisation below to show this. The video shows number of adoptions per postcode district (e.g. LE2, DE4 etc) over time, with a snapshot taken every week and each second on the video corresponding to 5 weeks. The visualisation starts at 1st April 2010 – the date when the Feed in Tariff was introduced.

Things to notice are

  1. The scale is basically logarithmic – so colour changes at the low end are for one or two installations, whereas at the top end a single colour is used for areas with between 500 and 1000 installations.
  2. There has been a huge increase in domestic PV ownership during the time that the Feed in Tariff has been in force.
  3. There are notable “jumps” at certain times in the simulation. These correspond to political events – on which subject more in a future post. They are connected with controversy about the feed in tariff levels.
  4. Big cities tend to remain at very low adoption numbers. This may be a problem if we want microgeneration to offset local demand, as cities also tend to be places of high consumption. On the other hand – microgeneration even at these levels is still well below consumption of even a single house – so as it stands most generation will offset demand locally.

Apologies for the poor quality video – I have a nicer mp4 for anyone interested, but could not embed it locally – resulting in the use of the YouTube that you see above.

Anyway – I think this is a nice visualisation of the data on microgeneration adoption. Unfortunately you can’t embed a video in a thesis – so here it is for the world to share 🙂

I intend to look at capacity and density in the not-too-distant future, as well as posting on any differences found if the geographic scaled is changed (e.g. looking at smaller, or larger, geographic units. As always – comments welcome – especially if you think there are things that I could improve to communicate the data in the video more clearly.

Posted in Analysis, Data | 4 Comments

Scale – the modeller’s recurring dream…

I just had a look at the most recent BP Statistical review of World Energy. As usual it makes interesting reading – condensing the world’s energy appetites into a bite-sized summary. In my work, I am engaged with modelling people’s behaviour with respect to the use of energy in the presence of various contextual factors, such as what other people are doing, government policy, their own beliefs about climate change amongst others. A large part of this is concerned with their adoption of technologies to facilitate less carbon intensive energy use. Examples of this might be adoption of PV panels on their roof (of which more in a future post), adopting heat pumps to heat buildings or adopting smart controllers to use electricity when it is being generated in abundance. All this goes hand in hand with the current drive to incorporate more renewable generation technology onto the grid.

So, reading the summary report, I was struck by the graphs on page 42 – reproduced below.  Figure 1 shows the fraction of the world’s energy consumption generated by each primary fuel. The amount produced by renewable sources is a small fraction. In my area of work, we are used to reading about the positives in the accelerating adoption of renewable technology, but this is a sobering reminder that there is still a huge way to go if we wish to robustly decarbonise our energy diet.  The graph indicates that 13% of our total energy consumption was produced by low carbon means (Renewables, hydroelectric and nuclear), with the rest coming from Coal, Oil and Gas.  The hydro and nuclear fraction seems to be fairly stable over the time period considered (1987 – present) with the renewables fraction increasing, but to only 1.9% of the total in 2012.


Figure 1: Primary energy world consumption (source BP Statistical Review of World Energy 2013)

This brought home to me once again the enormity of the task of decarbonising energy use.  My work focuses on smart use of microgeneration and demand balancing in the domestic context in the UK.  While I try to keep an eye on the large scale context, I am inevitably focusing on a fraction (electricity demand) of a fraction (domestic) of a fraction (UK) of the energy consumption of the world.

Even within this small and relatively focused area, the issue of scale rears its head time and again. A rapid increase in technology takeup might be observed in households, but what is the effect at the town/county/country level? The rate of adoption for a technology might be high, but what are the absolute numbers of adoptees? The increase in generation capacity in households might be threefold, or even tenfold, over a year but still represent a tiny fraction of a percent of total generation. How to effectively model this and present results in a fashion that is as unbiased as possible is a recurring nightmare.  Inevitably, questions of scale occur throughout the modelling exercise – what scale of user should be modelled (household, office, factory, blast furnace…), what scale of generation (domestic PV, small wind farm, medium gas plant, DRAX…), what geographical scale (neighbourhood, district, county, country, continent…) and over what timescale (you get the picture…).  All have interesting characteristics and questions and interact, but not in a simple “just scale that up” way – usually via a complex and hard to predict interaction – with some mechanisms being present only at certain scales. For instance, investment strategy isn’t very relevant at the minute-by-minute timescale, but the ability of plant invested in to satisfy momentary demand is. Similarly, the day-to-day habits of a household might not affect the geographic planning of gas network reinforcement, but a shift en masse will.

When modelling energy scenarios, we will continue to grapple with this.  In debates around energy policy and climate change, arguments at mismatching scales are often pitted against each other.  My own modelling work has shown me that conclusions drawn at one geographical resolution can be unjustified when the analysis is performed at a different resolution.  It can be hard to make the point that the argument being put forward does not hold at the scale under discussion.

Other than to highlight the current situation regarding fossil fuels vs. low carbon generation at the world scale, my purpose in writing this short post is to make a plea for careful statement of the scale and resolution used when analyses are undertaken.  Even better, an analysis of how robust the results are to scale change. In some cases, this will be as important as the results presented.

Acknowledgement: terms and conditions reproduced from report : “Quoting from the review – Publishers are welcome to quote from this review provided that they attribute the source to BP Statistical Review of World Energy 2013”

Posted in Uncategorized | 2 Comments