About me

I work on problems that blend optimisation, data and engineering. I live in Glasgow and married to Sarah.

Thursday, 21 December 2017

Visualizing Power Networks with yEd

Without visual insight into a power network, it is very difficult to identify a problem, let alone solve it. A good graphing tool is therefore very important to help one understand a power network, identify any issues and come up with the solutions.

This post presents a way to graphically represent a power network using a free software called yEd. My first experience of using yEd was back in 2012 when I was trying to build a representative power system for Great Britain. I used a data set from an old security and quality of supply standards (SQSS) report and built a Matpower case file. The data set had around 2000 nodes and 3500 branches. One can imagine the complexity of solving OPF type problems on such a test case. The first challenge with that data set, like any other data set build from scratch (without a load flow solution), was to find a feasible solution. My first attempts at using DC-OPF and AC-OPF models resulted in infeasibility and I was at a loss that what is causing this infeasibility. I tried removing all the constraints on generation and thermal ratings of the lines, but the test case did not yield a feasible solution. After trying all means, I decided to find a way to plot this large network and that's when I stumbled upon yEd. After plotting the network, I found various disconnected components of the graph with very small loads attached to them. One plausible explanation of such disconnected components was that some transformer data linking to such low voltage nodes were missing from the SQSS report. Anyways to cut the long story short, after aggregating these nodes and making sure the network is a one big connected graph, the DC and AC OPF problems on the big Great British transmission system test case yielded feasible solutions (with no bounds on generation and thermal ratings-how I fixed that is another story!)

I would not have built a representative model of Great Britain that yielded plausible results without the help of a visualisation tool. In later years, I used this tool on numerous occasions to plot line flows while trying to understand why the cheapest generation was not being dispatched, and to find out the location of the binding constraints in a network. Power networks are peculiar, sometimes they throw up surprises, things are not very obvious and needs some digging in to get to a plausible reason-that's where tools like yEd comes handy.

yEd is a free and very powerful graphing software. It allows a user to build a graph using a graphic user interface. A more convenient way to visualise large networks is to write a graph file and open it in the editor. yEd can read numerous data formats for a graph including tgf (trivial graph format) and xml (extensible markup language) file formats. tgf is a basic graph file that contains information about number of nodes in a graph, connections and direction of flows. yEd have numerous layout algorithms which are suitable for different types of graphs. Some layout algorithms are as follows:
  • hierarchical (useful for tracing power flows)
  • organic
  • orthogonal
    • Classic (my favourite!)
    • UML style
    • Compact
  • circular
  • tree
  • radial
Hierarchical layout of the 24 bus system
Circular layout of the 24 bus system

Radial layout of the 24 bus system

The network diagrams in yEd can be exported in many different formats including jpeg, pdf, png and svg file formats. 

yEd copes with very large networks reasonably well. The biggest network I have plotted using yEd is a Polish network with 2736 nodes. The jpeg file is too big to attached here. The diagrams can be seen on my github page.

A Python script can be found here that can produce a tgf (Trivial graph format) file from a Matpower test format. A tgf file then can be opened in yEd.

Tuesday, 14 November 2017

Energy data analysis with Pandas

Leo Smith of Gridwatch has done a cracking job of making electricity data available on an online resource. The data available through his website includes total demand, total generation from different generation types, interconnector flows and frequency with a temporal resolution of approximately 5 minutes.

This post is about using a Python package called pandas to perform data analysis on the data downloaded from the gridwatch website. Pandas is a powerful data analysis library for Python.

A small Python script follows that reads gridwatch data into a Pandas data frame, performs data cleaning for erroneous data and plot.

import pandas as pd
import matplotlib.pyplot as plt

The first thing we want to do is to import pandas and matplotlib packages in our Python script. This is achieved by lines 1 and 2.

filename = 'gridwatch.csv'
data = pd.read_csv(filename)

'gridwatch.csv' is a csv file that I have downloaded from the gridwatch website. To keep things simple, I have only downloaded the demand data in this file. However, the same script would work for any other data or a combination of data types. The resulting file is a big csv file (~3.8MB, 104860 lines).

Line 4 reads our csv file as a pandas data frame.

data =  data.rename(columns=lambda x:x.strip())
for key in data:
    data[key] = data[key].map(lambda x: str(x).strip())

The downloaded csv file (and hence the pandas dataframe) contains leading white space. We need to fix this by stripping the white space, as this would cause problems later. This is achieved by lines 5-7.

data['timestamp'] = pd.to_datetime(data['timestamp'],
                                   format="%Y-%m-%d %H:%M:%S")
data['demand'] = pd.to_numeric(data['demand'])
data.set_index('timestamp', inplace=True)
ax = data['demand'].plot()

Line 8 converts the datatype of  our dataframe column 'timestamp' to take a datetime data type. This is useful as I would like to plot the demand against time and for that need pandas to recognise the datetime data type. Line 10 converts the demand to numeric values. Line 11 sets 'timestamp' to index of the data frame. Line 12 and 13 plot the demand against time, and we have the following plot.

The above figure shows some spikes and dips. These spikes and dips corresponds to the erroneous data in the downloaded csv file. The total demand of Great British electricity system can not be higher than 60GW and less that 15GW. Based on this knowledge, we need to clean our data. This can be achieved by using following command.

data = data[(data.demand>15000) & (data.demand<60000)]

See how easy is plotting and making sense of data using Pandas and Python. 

The full code of reading gridwatch data is available on this link.