Jekyll2022-11-11T17:42:40+00:00http://www.trafficflows.org/feed.xmlTraffic flows and other stuffParent, ex-economist, developer/admin - Salesforce, R, Power BIUsing Obsidian2021-02-13T00:00:00+00:002021-02-13T00:00:00+00:00http://www.trafficflows.org/using-obsidian<p>I’ve recently started using <a href="https://obsidian.md/">Obsidian</a> for my notes. I was looking at Notion as a possible replacement for using a combination of OneNote and Google Keep but came across Obsidian and got hooked. Here are my thoughts after a month or so of using it.</p>
<h2 id="why-obsidian">Why Obsidian?</h2>
<p>First off, Obsidian is new, and to me new is always good. It’s under very active development (it’s still in beta but it’s one of the most fully-featured beta products I’ve seen). Although it’s not open source, the developers are very responsive and there is a very engaged community on the <a href="https://forum.obsidian.md/">forum</a>. In fact, that forum is perhaps both the most helpful and most thoughtful forum I’ve ever seen for an application or language. Perhaps that goes with the territory, many people using Obsidian are coming from other applications with different features and/or show the thinking that joined-up note-taking encourages. And the number of third-party plugins is rapidly increasing making for a very productive eco-system. At present Obsidian free and it looks like it will remain free for most uses.</p>
<h2 id="where-am-i-coming-from">Where am I coming from?</h2>
<p>I use OneNote a lot at work to document our systems, because it’s a product that others can use without any special training and there are no IT Support issues about downloading novel software since it’s a core Microsoft product. But it’s clunky and I always fear the sudden loss of information due to accidental vandalism (it is backed up but small deletions are hard to detect). In addition, you’re locked into a proprietary format - I don’t think OneNote is going anywhere but you never know. I also tried using OneNote for small-scale home notes - like for school etc and for meeting notes. But I just didn’t get on with it for those purposes. And don’t forget the awful cut-down version they currently palm you off with (though it seems that is changing).</p>
<p>I’ve used Google Keep a lot over the years and it’s great for small notes and reminders, and anything where you need to quickly record notes particularly on a phone. But it just doesn’t have the linking of notes that good record keeping demands.</p>
<p>I read a lot of technical books and website articles and have never really made proper notes against them, with the result that I’m not properly learning. Using a Kindle, I can highlight parts of the book but then the notes are stuck on the device (or in read.amazon.com). Better if those were integrated with other notes. With Obsidian, I can download those notes and paste them into a note file.</p>
<p>I use Todoist for keeping track of work and household tasks and love the flexibility that it offers, including the easy integration with Alexa, and recurrent tasks.</p>
<h2 id="obsidian">Obsidian</h2>
<p>So where does Obsidian come in? It’s an application that uses plain-text files with <a href="https://www.markdownguide.org/cheat-sheet/">Markdown</a> to determine the formatting and linking. You can use it as a combination of a Wiki and a traditional note-taking application. You can file notes in separate folders if you want, you can use tags and you can link files using Wiki-type links (or the Markdown equivalent). Each file might be a long note on, eg a book, but could equally be a micro-note on a simple concept, or might just be an index that links various other files together. You might have daily journals or files which embed data from external systems, like the plugin which enables you to embed Todoist filters.</p>
<p>Installation is simple, just go to https://obsidian.md/ and follow the instructions there. You will need to create a folder in which to put your “vault”, which is really just a collection of files and folders in which you want to put related material. You can have multiple vaults if you want but in many cases you will want to keep all your material in the same vault.</p>
<p>In the forum there are plenty of examples of how you might structure your folders (my ideas are shown below) but it’s entirely your choice, and it’s super-easy to move files around later as you come to terms with the way you want to work.</p>
<p>As the files exist on your own system, you can back them up anyway you want (and you really must back them up, after all this is your hard work). If you want to, the files can exist on OneDrive or some other cloud service and be automatically backed up. Or you can use Git to back them up and keep a record of each change that you make - a very powerful way of keeping track of how a note develops and back-tracking if necessary. Or any other back-up method you want.</p>
<p>The Obsidian application itself is like a hub to edit and manage these files and to switch between raw Markdown and preview mode (if you want). If you rename or move a file within Obsidian, then all links to that file are changed to point to the new name. You navigate through your links, or just open a file from the file explorer, or from the pop-up quick file opener (which remembers your recently opened files).</p>
<p>You can easily set up templates for different purposes, for example meeting notes or notes on a book, and it’s a few key presses to include them into your current note. These templates can include fields, such as the current date, which get populated when the template is used.</p>
<p>For the hard-code, there is support for the <a href="https://medium.com/@rebeccawilliams9941/the-zettelkasten-method-examples-to-help-you-get-started-8f8a44fa9ae6">Zettelkasten</a> method of note-taking.</p>
<h2 id="markdown">Markdown</h2>
<p>Markdown is easy. In fact, you might have used elements of it elsewhere without thinking about it, particularly bullet points. Some simple concepts:</p>
<p><code class="language-plaintext highlighter-rouge">**bold**</code> becomes <strong>bold</strong></p>
<p><code class="language-plaintext highlighter-rouge">_italics_</code> becomes <em>italics</em></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A simple bullet list:
- line 1
- line 2
- line 3
</code></pre></div></div>
<p>becomes</p>
<p>A simple bullet list:</p>
<ul>
<li>line 1</li>
<li>line 2</li>
<li>line 3</li>
</ul>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Todo items look like this:
- [ ] Uncompleted
- [x] completed
</code></pre></div></div>
<p>Todo items look like this:</p>
<ul class="task-list">
<li class="task-list-item"><input type="checkbox" class="task-list-item-checkbox" disabled="disabled" />Uncompleted</li>
<li class="task-list-item"><input type="checkbox" class="task-list-item-checkbox" disabled="disabled" checked="checked" />completed</li>
</ul>
<p>and they can be ticked off in preview mode.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A markdown link [looks like this](https://www.trafficflows.org)
</code></pre></div></div>
<p>A markdown link <a href="https://www.trafficflows.org">looks like this</a></p>
<p>But for links inside your Obsidian vault, you’ll be better off using Wiki-type links:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A Wiki-type link to another file - [[Visual Studio Code]]
</code></pre></div></div>
<p>A Wiki-type link to another file - [[Visual Studio Code]]</p>
<p>You can <em>transclude</em> (ie embed) other files or parts of files by preceding the link with an exclamation mark.</p>
<p>There’s a lot of excitement about changing the Obsidian editor to be more like <a href="https://typora.io/">Typora</a>, where editing shows the raw Markdown for the line you’re editing but a preview of the output for everything else. I’m neutral on the benefits; looking at raw Markdown becomes very natural to me after a while.</p>
<p>Because Markdown is a plain-text format, you can use any method to edit the files <em>en masse</em> including scripting languages like Perl or Python or external text editors. But some of the new plugins, like Tag Wrangler, reduce the need to do this in any case.</p>
<h2 id="mobile-application">Mobile application</h2>
<p>An Obsidian mobile application is under development and I’m really interested in what they come up with. In the absence of an official application, I’m using <a href="https://gitjournal.io/">Gitjournal</a> on Android - this will connect with GitHub to get at my backup, permit me to edit files, and then commit and push the changes back to GitHub. Next time I’m in the desktop application, I can just pull the changes back down. All very nice, and because Git changes are fully logged, I have an extremely thorough method of tracing changes.</p>
<h2 id="tags-links-folders">Tags, links, folders</h2>
<p>Your vault is your vault so you can do what you want with it. When starting out though, it’s helpful to have a starting position. The forum (and elsewhere on the Internet) has lots of ideas on how best to structure your work. For what it’s worth, this is what I do.</p>
<ul>
<li><strong>folders</strong>
<ul>
<li>rather like GTD, I have an inbox folder for stuff I’ve just created and haven’t yet decided where to file</li>
<li>a daily note folder - I’m trying really hard to do a daily journal</li>
<li>a monthly note folder - I really can’t be bothered to do a weekly one</li>
<li>a todo folder - haven’t really used this</li>
<li>Areas - broken down into (order is random not in importance!)
<ul>
<li>Children</li>
<li>Home</li>
<li>Work</li>
<li>Me</li>
<li>Family</li>
</ul>
</li>
<li>Interests - subfolders are under development
<ul>
<li>Learning</li>
<li>Sport</li>
<li>Wellness</li>
<li>IT</li>
</ul>
</li>
<li>Projects
<ul>
<li>Blog posts - the area where I draft and complete my blog</li>
<li>more to follow</li>
</ul>
</li>
<li>Library
<ul>
<li>Authors - each author page is just a simple index to their books or articles below</li>
<li>Books - at least meta-data on them but usually notes and quotes</li>
<li>Web - ditto</li>
<li>Articles - ditto</li>
</ul>
</li>
<li>Thoughts - those random snippets you want to capture</li>
<li>Attachments - images etc</li>
<li>Archive - older stuff (at the moment I just move older daily journals here)</li>
<li>Templates - the collection of templates I use</li>
</ul>
</li>
<li><strong>links</strong> - links are the prime way I relate different notes together, either directly or via mini-index files (MOCs - map of contents)</li>
<li><strong>tags</strong> - I use tags to denote the type of content or that something needs doing so I have tags like: 0/TODO or TYPE/AUTHOR. Of course, there’s some redundancy here as author notes all ought to be in the author folder.</li>
</ul>
<p>Remember though, it’s easy to change to another way of working. If you wanted to use tags to denote the subject matter then that’s up to you. The Zettelkasten method might lead you to a much flatter folder structure.</p>
<h2 id="plugins">Plugins</h2>
<p>As I mentioned earlier, the plugin eco-system is rapidly developing. It’s easy to browse and install these from within Obsidian and you can go to GitHub to see more information about each one, including tracking bugs and development.</p>
<p>A few of my favourites:</p>
<ul>
<li><strong>Obsidian Git</strong> - provides a nice easy way of pushing and pulling changes to GitHub without using the command line or GitHub desktop</li>
<li><strong>Advanced Tables</strong> - Markdown tables can be a little tricky but this makes editing them very easy</li>
<li><strong>Periodic Notes/Calendar</strong> - these two provide improved functionality for daily, weekly or monthly journalling</li>
<li>**Sliding Panes ** - a bit Marmite this one, sometimes I like it, sometimes I feel it doesn’t work as I expect, but when it does work it makes the interface a bit swish</li>
<li><strong>Templater</strong> - an improved workflow for templates</li>
<li><strong>Find unlinked files/Dangling links</strong> - useful to track down notes that need linking</li>
<li><strong>Todoist Plugin</strong> - integrates Todoist filters so you can transclude these and the tasks just look like native todos. I don’t see Obsidian todos replacing Todoist for me given the latter’s much more complex functionality. But one advantage of this plugin is that I can finally copy the Alexa shopping list into plain text and paste it into the Sainsburys search filter.</li>
</ul>
<p>There are plenty of other plugins I don’t currently use. Of particular interest is Obsidian to Anki, which enables you to extract information from your vault and export it to Anki - a <a href="https://collegeinfogeek.com/spaced-repetition-memory-technique/">spaced-repetition</a> application that I’ve used in the past both for my own purposes and for helping with learning difficulties.</p>
<p>There is so much other customisation you can do. You can remap keys to carry out commands, for example I’ve mapped toggling the left-hand sidebar to CTRL-1 and the right-hand to CTRL-0. There are plenty of themes and you can even add <a href="https://forum.obsidian.md/t/how-to-achieve-css-code-snippets/8474">CSS snippets</a> to fine-tune the display.</p>
<h2 id="graph-view">Graph view</h2>
<p>No mention of Obsidian should be without a graph view, so here’s mine, made nicer because the latest version provides the ability to colour by tag (or by other features):</p>
<p><img src="../assets/2021-02-13-ObsidianGraphView.png" alt="Obsidian graph view" /></p>
<p>It’s fun to see all the links and to zoom around, but for me the most useful part of the graph is to see obviously unlinked files. There’s a local graph view for each note, in which you can just see the links for that file, extending out to different depths if you want.</p>
<h2 id="summary">Summary</h2>
<p>Obsidian enables you to file and link notes quickly and thoroughly but doesn’t tie you into a particular format. If Obsidian folded tomorrow, I’d still have my notes, and given they are in plain-text format, I can still read them with any editor and/or restructure them for another system.</p>
<p>To find out more about Obsidian then download it and play, it comes with a ready-made vault with loads of help. The forum is an amazing resource and extremely friendly to beginners and the expert alike. There are some excellent videos on YouTube including those by Bryan Jenks and “Linking your thinking”, maybe start with the latter’s <a href="https://www.youtube.com/watch?v=QgbLb6QCK88">Obsidian for Beginners: Start HERE — How to Use the Obsidian App for Notes</a>.</p>I’ve recently started using Obsidian for my notes. I was looking at Notion as a possible replacement for using a combination of OneNote and Google Keep but came across Obsidian and got hooked. Here are my thoughts after a month or so of using it.Porting done2021-02-06T00:00:00+00:002021-02-06T00:00:00+00:00http://www.trafficflows.org/porting-done<p>I’ve done the hard work in porting the Wix blog files and images over to GitHub Pages. Please let me know if you see any bad links, poorly formatted code or the wrong image. Getting the code and the images across was by the far the hardest part - I’ll post about the process later.</p>I’ve done the hard work in porting the Wix blog files and images over to GitHub Pages. Please let me know if you see any bad links, poorly formatted code or the wrong image. Getting the code and the images across was by the far the hardest part - I’ll post about the process later.Porting from Wix to Jekyll - how-to2021-02-06T00:00:00+00:002021-02-06T00:00:00+00:00http://www.trafficflows.org/porting-from-wix-to-jekyll-how-to<p>I’ve moved my blog posts from a paid Wix site to GitHub, using GitHub Pages and the <a href="https://jekyllrb.com/">Jekyll</a> website/blog engine. This enables me to write posts in <a href="https://www.ultraedit.com/company/blog/community/what-is-markdown-why-use-it.html">Markdown</a> and integrate with my <a href="https://obsidian.md/">Obsidian</a> vault (I’ll write about that another day).</p>
<p>There were several reasons why I moved this blog:</p>
<ul>
<li>cost - why pay for Wix when GitHub Pages are free?</li>
<li>annoyance with Wix 1 - I don’t like organisations that repeatedly email with (the same) “time-limited” offers</li>
<li>annoyance with Wix 2 - it was really overkill for what I wanted</li>
<li>annoyance with Wix 3 - I didn’t want to get locked into a proprietary system</li>
<li>ease of workflow - drafting posts on the Wix website just doesn’t fit with the way I want to work</li>
<li>challenge - I like changing things around from time to time and to learn new stuff along the way</li>
</ul>
<p>The Markdown format is widely used (in several dialects) and has the great advantage that it’s plain text so easy to copy from one system to another. Having control over the files on my own system (and backing them up!) adds to the feeling I’ve got a much more portable system.</p>
<p>There really wasn’t much challenge involved in this change, partly because I wanted the blog to be very straightforward. I’m using the default theme and no tags but may reconsider this approach.</p>
<h2 id="basic-set-up">Basic set up</h2>
<p>In order to do this you need some basic knowledge of Git/GitHub, a very small amount of HTML and Markdown (but this is really very simple).</p>
<p>I find it easiest to use the GitHub desktop application in Windows but cloning and pushing files to the site can also be done at the command line.</p>
<p>It’s useful but not essential to have your own domain name.</p>
<p>There are various websites that take you through the process in more depth, in particular:</p>
<ul>
<li><a href="https://docs.GitHub.com/en/GitHub/working-with-GitHub-pages/getting-started-with-GitHub-pages">Getting started with GitHub Pages</a></li>
</ul>
<p>The following sites are limited on Medium but if you don’t have a Medium account then you can save them to Pocket:</p>
<ul>
<li><a href="https://towardsdatascience.com/building-a-beautiful-static-webpage-using-GitHub-f0f92c6e1f02">Building a Beautiful Static Webpage Using GitHub</a></li>
<li><a href="https://towardsdatascience.com/launch-a-website-for-free-in-5-simple-steps-with-GitHub-pages-e9680bcd94aa">Launch a Website for Free in 5 simple steps with GitHub Pages</a></li>
</ul>
<p>You can read more about Git and GitHub at</p>
<ul>
<li><a href="https://docs.GitHub.com/en/GitHub/getting-started-with-GitHub">Getting started with GitHub</a></li>
<li><a href="https://techcrunch.com/2012/07/14/what-exactly-is-GitHub-anyway/">What Exactly Is GitHub Anyway?</a></li>
<li><a href="https://towardsdatascience.com/getting-started-with-git-and-GitHub-6fcd0f2d4ac6">Getting started with Git and GitHub the complete beginner’s guide</a></li>
</ul>
<p>A quick run-through of what you need to do to set-up:</p>
<ul>
<li>create an account at GitHub if you don’t already have one</li>
<li>fork the Jekyll Now repository (link)</li>
<li>rename the cloned repository to <em>yourusername</em>.GitHub.io</li>
<li>using the GitHub desktop application, clone the repository to your local system</li>
<li>add a profile picture to the images folder</li>
<li>create a new assets folder</li>
<li>customise the _config.yml file
<ul>
<li>this contains the basic details that is shown on the home page, this is a template so it’s just a case of filling in the blanks, no coding needed</li>
</ul>
</li>
<li>customise the about.md file
<ul>
<li>this is the text of the About page, again this is a simple template</li>
</ul>
</li>
<li>customise the _highlights.scss file
<ul>
<li>renamed the <code class="language-plaintext highlighter-rouge">highlight</code> block to <code class="language-plaintext highlighter-rouge">pre-highlight</code> to get rid of annoying double-boxes around code (as explained at <a href="https://stackoverflow.com/questions/55308142/why-do-i-get-a-double-frame-around-markdown-code-block-on-jekyll-site">Stackoverflow</a>)</li>
</ul>
</li>
<li>customise the index.html file
<ul>
<li>I changed the post list to include the date and to truncate the post excerpt like so:</li>
</ul>
</li>
</ul>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>---
layout: default
---
<div class="posts">
{% for post in site.posts %}
<article class="post">
<h2><a href="{{ site.baseurl }}{{ post.url }}">{{ post.title }} - {{ post.date | date: "%-d %B %Y" }}</a></h2>
<div class="entry">
{{ post.excerpt | strip_html | truncatewords:75 }}
</div>
<a href="{{ site.baseurl }}{{ post.url }}" class="read-more">Read More</a>
</article>
{% endfor %}
</div>
</code></pre></div></div>
<ul>
<li>add posts to the _posts folder and images to the assets folder</li>
<li>commit and push the changes from your system</li>
</ul>
<p>This whole set up took me about 30 minutes, and I’m sure it can be done a lot quicker. New posts are added just with the last two steps, everything else is a one-off.</p>
<p><a href="https://jekyllrb.com/docs/posts/">Jekyll site - how to write posts</a> is a good site to start with in terms of writing posts but it’s really easy.</p>
<h2 id="porting-from-wix">Porting from Wix</h2>
<p>By far the most time I spent on this whole process was copying the posts and images from Wix to the new site. Some issues I faced:</p>
<ul>
<li>I did not use a good file naming system for the screenshots and I found it tedious to match up the actual files with the posts. I’ve improved this by renaming image files to include the date of the related post.</li>
<li>copying the text from Wix introduced backslash quoting of code and file references, particularly to underscores. These were hard to track down.</li>
<li>some of the Power BI M code included double curly braces. Unfortunately these are used by the underlying <a href="https://jekyllrb.com/docs/liquid/">Liquid</a> templating language. I had to surround such code with <a href="https://shopify.GitHub.io/liquid/tags/raw/">raw</a> tags.</li>
</ul>
<p>I used Notepad++ at times to reformat codes since its search and replace features are so powerful. Yes, I used to use Vi(m).</p>
<h2 id="my-new-process">My new process</h2>
<p>Now that Obsidian is now very much part of my daily routine, being in it for much of the day, and being able to see my notes from other sources and my page of ideas, makes the whole writing experience so much better. I found it distracting moving from one tool to another.</p>
<p>I can now draft posts in Obsidian in a separate drafts folder, keeping track of ideas in another file. When they’re cooked, I can transfer posts to the final folder. In the meantime, everything is separately backed up (also to GitHub).</p>
<p>Links to images are fine since the draft and final blog post folders are in the same path relative to the assets folder as they are on GitHub. So the markdown:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>![Figure 1](../assets/yyyy-mm-dd-fig1.png)
</code></pre></div></div>
<p>will work in both. It’s really helpful when drafting to see the images.</p>
<p>Links from post to post don’t work but that’s easily checkable.</p>
<p>I use a short <a href="https://docs.microsoft.com/en-us/powershell/scripting/overview">Powershell</a> script with <a href="https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy">RoboCopy</a> to copy posts from the final posts and assets folders in my Obsidian vault to the cloned Jekyll Now repository in a separate folder structure. Together with my latest toy, <a href="https://keypirinha.com/">Keypirinha</a>, I can copy the files in a few keystrokes. I’m sure if I were still using Linux, I’d have similar bash scripts for this (and committing/pushing changes).</p>
<p>I can then use the GitHub desktop application to commit and push the changes. After a minute or so the changes have been processed and are live on the website.GitHub</p>I’ve moved my blog posts from a paid Wix site to GitHub, using GitHub Pages and the Jekyll website/blog engine. This enables me to write posts in Markdown and integrate with my Obsidian vault (I’ll write about that another day).Porting from Wix to Jekyll2021-02-05T00:00:00+00:002021-02-05T00:00:00+00:00http://www.trafficflows.org/porting-from-wix-to-jekyll<p>I’m in the process of porting my blog from Wix to Jekyll, hosted on GitHub Pages. I’ll post a how-to once I’m done but in the meantime please bear with me.</p>
<p>D</p>I’m in the process of porting my blog from Wix to Jekyll, hosted on GitHub Pages. I’ll post a how-to once I’m done but in the meantime please bear with me. DRoad pricing - bursting the bubble (again)2020-11-29T00:00:00+00:002020-11-29T00:00:00+00:00http://www.trafficflows.org/road-pricing-bursting-the-bubble-again<p><em>This is a revised version of my earlier post, having considered responses to the original and having had a little time to order my thoughts more carefully. I had originally planned a Twitter thread on my thoughts but there is just too much to cover for that format. I am sure there are some logical, factual and theoretical errors, for which I apologise in advance. I suspect this will remain a work in progress, to be edited when I have time.</em></p>
<p>I know many of my friends are in favour of “smart” road user charging/road pricing. As with most things in life I am sceptical. I want to believe, I really do. The theory of road pricing is sound and the potential benefits enormous. But there are many practical issues that will need resolution both before and during the running of any scheme. Given these issues, it is not the panacea people think it is. By holding out hope for such a scheme, we will lose sight of achievable measures that can be implemented in the foreseeable future; waiting for a nirvana that will never come.</p>
<p>Just to be clear, here I am talking about wide-area (probably) national road user charging/pricing not small-scale measures like the London congestion-charge scheme nor micro-schemes that have been proposed for places like Richmond Park. Those schemes have their issues, not in the least the infrastructure needed and the impact that might have on the asset being protected. The sort of scheme I will discuss is one where the user is charged per mile, tracked by satellite, with the cost per mile varying according to the road conditions at the time of travel. This scheme differs from the small-scale ones by an order of magnitude both in the potential impacts and in the difficulty in delivery. I will just use the short-hand “road-pricing” to denote these more complex type schemes.</p>
<p>I will come off the fence here and say a road-pricing scheme will never be delivered in the United Kingdom. Even if it could ever be delivered, by the time it could be then we will have different transport needs and systems, where pricing per mile will likely not be necessary. In the meantime, we should look for smaller-scale solutions that can deliver many of the benefits with far fewer of the problems.</p>
<h2 id="background">Background</h2>
<p>The economic theory behind road-pricing is sound and few would dispute it. In their use of a service, people impact on other people or on the environment. In the case of road transport, such “externalities” may be extra noise, pollution, risk of accidents, or congestion; with the latter adding to the travel time of other users. The problem is that such externalities are not adequately represented in the current pricing structure for road use. In effect, there are times when the cost of travel is too low, and other times when it is too high. Both situations lead to a sub-optimal amount of travel for society as a whole. Society is better off when all the actors face the full price of their travel.</p>
<p>A simple explanation of the theory behind charging drivers for the congestion they cause is at <a href="https://www.economicsonline.co.uk/Market_failures/Road_congestion.html">https://www.economicsonline.co.uk/Market_failures/Road_congestion.html</a>.</p>
<p>Arguably, one of the major externalities, pollution, can be properly captured by charging within the price of fuel, even though imperfectly – assuming a relatively straightforward relationship between fuel used and pollution emitted, one could price an additional component on to the price of fuel. This does not hold for non-fuel sources of pollution, for example, tyre rubber, which is still a problem for electric vehicles.</p>
<p>Congestion has a very different nature, and it is much more difficult to reflect it in a price. It would not be possible to reflect in a uniform tax like fuel duty. In addition, vehicles not subject to fuel duty still contribute to congestion. Travelling in the countryside in the middle of the night will likely have negligible impacts on other users (ignoring other impacts for now); driving in the rush-hour on a busy urban road will have some, potentially major, impact on other drivers. Of course, the marginal impact of one more vehicle will still be quite small – but it is the cumulative effects of many vehicles that causes congestion. Aggregating such cumulative effects is usually the main rationale for motorway widening or village bypasses, ie increasing the supply. But in most urban, and environmentally sensitive, areas this cannot be. Moreover, increasing the supply in one area may be enough to un-cork the demand genie and move the congestion problem elsewhere.</p>
<p>Road-pricing is therefore the answer provided to this – include the externalities in the price road users face so they internalise the full cost. The congestion problem is that some journeys are currently under-priced – road users take the low price and travel more than is optimal. The result is congestion. In theory, if the road user pays the exact amount that they impose on other users; they will make a rational choice on whether to travel, having taken into account the true price of that travel to all of society, not just themselves. Of course, this true price must vary according to the exact road, time of day and other conditions, otherwise it will not work as intended. And therein lies much of the problem.</p>
<p>The reasoning for my pessimism about such schemes is based on five inter-locking concepts:</p>
<p>Equity – would the schemes deliver fairness?</p>
<p>Simplicity – would the schemes be easy to understand</p>
<p>Efficacy – would they work?</p>
<p>Deliverability – how easy will it be to put in place?</p>
<p>Acceptability – would the public accept such schemes?</p>
<p>These concepts can work both together and against each other. A simple scheme may be more acceptable because it is understandable but if it does not deliver the benefits it is at the same time less acceptable. A more complex scheme may have better benefits in theory, but if it cannot be delivered, it is pointless. A scheme that does not track users’ position may be more acceptable but if it does not reduce congestion sufficiently, it may not be worthwhile.</p>
<p>Efficacy - do not believe the economic benefits, small errors in the price will not only diminish the benefits but will also undermine the acceptance which will be precarious in any case.</p>
<p>Let us assume for the sake of argument there is a “true” identifiable economic price for driving at a given time to a given place by a given route, taking account of all the externalities. If this price is levied on each user, the level of traffic is economically optimal. I accept the economic theory of this. My issue is with the practice.</p>
<p>Any error in the price (again putting aside arguments about whether there is a true price) actually offered before travel or levied during travel will impact on whether people get charged too much and decide not to drive at all, leaving the road too empty and the economic benefit of the scheme too low. If there is an error in the opposite direction, people pay too little, and the road is too full. Not only will the scheme benefit be lower than optimal but if people pay any price, they will expect a premium service; if they do not get that service, acceptability falls. Decisions made at the start of journey (eg to go ahead at a certain price) will be affected by temporary occurrences such as accidents. How will a system take these into account?</p>
<p>What would the economically optimal level of congestion look like? Would it be acceptable to road users and wider society?</p>
<p>In order to deliver the maximum benefits, the scheme has to influence travel decisions before they are made. Some kind of slot booking system would be ideal but very difficult to deliver in practice and would be unacceptable to drivers used to driving when they like. How else could road space be charged whilst keeping the theoretical benefits? Consider Waze - an excellent app. But it does not always get the route right, it cannot as road conditions change too quickly. Temporary issues such as accidents will complicate driving times. The best Waze can do once some drivers get hit by poor conditions is to ensure following drivers get diverted if necessary. People will not see benefits all the time and at times Waze will direct them on a route that eventually works out as longer. But Waze is a voluntary app to assist the driver. The stakes are much higher with a money-levying app/device. People will expect a better service.</p>
<p>Let us say a person chooses a cheap but slow route in preference to the expensive but fast route. What happens if the latter is delayed due to an accident? Traffic may divert to the cheaper route. Does the price on that route now go up? In theory it should do so to maximise the scheme benefits - but is it fair? How would the driver be presented with the choice of route and price? Will drivers get price anxiety (“I must get home before the price goes up”, “I’m stuck in traffic, look at the price meter go up”) - would that affect the safety of their driving? (“I’m going to overtake here to reduce my costs”).</p>
<p>So consider what would happen if the price is allowed to vary, a driver chooses to drive at a certain time because it was going to cost £5, the journey gets affected by an accident and the driver has to pay £10 because the congestion increased. Will they be content that the new price is economically optimal given the (revised) traffic conditions? I doubt it. Once acceptance has been lost (if it was ever gained) we have seen with the poll tax what happens.</p>
<p>So, then the decision made at the journey origin might no longer be the best one. Could a system have risk built-in, so the offered price is the expected price? This is arguably better but then the offered price is wrong if nothing occurs and still not necessarily the correct one if an incident occurs. So again, the scheme benefit is lower than the theory would suggest. Perhaps a probability-based range estimate of the cost like Uber use would be deliverable but at that point, we have already moved even further away from the maximum benefit and efficacy. Are the right economic incentives clear enough to users to make the “right” decisions? The evidence is not good that people understand probability so would they understand such a system?</p>
<p>The London congestion charge has had to be turned off for certain incidents (for example, during the aftermath of terrorist attacks or during the 2020 Coronavirus outbreak) – would this be necessary for a wider scheme? Potentially, yes. Again, turning the scheme off will potentially lead to massive disbenefits as drivers take advantage of the lower price and cause congestion, or at least this would be one less day of benefits.</p>
<p>Moving away from incidents, how would the standard range of prices be set? Would local authorities be able to influence them in any way? Would there need to be an “Offroad” regulator to ensure proper pricing decisions? if the demand for road travel increased, would the government have the political strength to increase the price? The current 10 year pause on fuel duty increases suggests not. Would a climate change component be added and therefore vehicle type also affect the price? What about other environmental costs such as pollution or heritage? Do not forget obesity, would the price reflect some encouragement of more active travel? What about the risks of serious accidents at different times of day? Should all of these be in the calculation? Ideally, yes as all of them are externalities of road use. But some do not have good cost estimates. There needs to be transparency but again for maximum benefits the price needs to be economically optimal - can all this be done? Again, move away from the precise optimal price and the scheme benefit will fall. Exclude an externality from the calculation and there may be unintended consequences.</p>
<p>Would there need to be some rounding involved in the pricing? Are there legal implications to charging fractions of a pence per mile? If prices had to be rounded up, would this move away from the optimal price?</p>
<p>There are many other issues that would need addressing. For example, in theory, an exit lane for a motorway should be charged differently to the main carriageway. But we do not have the technology to adequately determine the vehicle position to that accuracy. Even if we did (and with better satellite coverage we might), we might see driver behaviour change - people cutting in late to save money, risking accidents. Would this need infrastructure or legal changes to reduce the risk? Again, the scheme benefits fall.</p>
<p>Equity. We must accept that we are moving from the current position, however imperfect it might be. As with all changes, some people will lose, some will gain - though the idea is that society benefits in aggregate.</p>
<p>We can protect those we think are worthy (however we might come to define that) with transfers, not necessarily money, perhaps travel vouchers? Free miles, bus passes, cycle schemes? But it would not be possible to do this in every area. If society deems it best for those who could not otherwise afford car travel, then we could provide money through some route. If universal basic income existed, that could be one route. But how would we set the level of support? Would it differ for those in cities with adequate public transport versus those in other areas? How would this keep in line with costs? Transparency would be needed. Would people believe the government and trust it to set the appropriate level? Balanced against the equity for drivers, we need to consider the impact on those who live near roads. Is reducing traffic noise and fumes outside (and inside) their homes worth reducing transport choices for other people? How will we trade-off these transfers?</p>
<p>Any system can be cheated, the higher pay-off for success (less cost and better traffic conditions) will undoubtedly spur some attempts. It would have to be widespread to impact on benefits, but any cheating will impact on acceptability. A scheme must be robust enough to minimise the possibility of cheating. This would probably mean any device has to be installed in a vehicle in such a way that it is immediately obvious from outside the vehicle when the device is not present or is not functional.</p>
<p>There are other equity issues that might not be fixable with transfers. Some people might be less capable of understanding, what is to them, a Byzantine charging structure. How much will it be for me to visit my grandchildren on Saturday? If I wait until 8pm how much do I save? These may not be easy calculations for some, even with apps to assist them, and they may just avoid travel instead. This may well be sub-optimal.</p>
<p>What alternatives will people have? Cities are much easier to provide for, though new build is often impossible. If there are no real alternatives, for example in rural areas, then is it fair to price some people out? Arguably yes, in the longer term so that longer term decisions on residential and job location etc are made. But that will mean a lot of short-term pain which is almost certainly politically unacceptable and arguably unfair on those who have to bear the brunt of that pain while planning systems adjust over many years. Few politicians would go down this road.</p>
<p>One issue of a road pricing scheme is that because the cost of driving is now more apparent, so will be the driver benefits of new build – reduced congestion (at least in the short-term) will mean lower prices. Will this apparent benefit distort the provision of new build?</p>
<p>In the long term, is personal motorised transport going to be on the same scale as now? If not, then how long should scheme costs and benefits be valued? We have seen with the current crisis how much work can be done (or not) at home. Will this change in the longer term? If so, will we need to tackle road congestion at all? Conversely, if public transport is a no go for virus reasons, will the road user price need to be so high as unacceptable?</p>
<p>There are potential issues with the way that the difference between work and leisure travel time is presently accounted for in road building schemes that would need to be considered.</p>
<p>Acceptability of technology and of the concept - to me, this is where the early 2000s DfT road pricing study went wrong, there is a massive difference between what people will accept for personal use and for taxation/charging purposes.</p>
<p>People might choose to use Waze and give away their location for such purposes, they will be far less likely to for charge-levying purposes. Some kind of “black box” would be needed. This is ideally also needed for road safety (like the proposed EU box) but then perhaps even harder to get acceptability if the same device were to be a “spy in the car”. Perhaps if it were mandated by insurance companies it might be easier, but then might it be seen a stealth taxation device? What is the win for drivers? Will they really perceive the benefits? Road pricing was not acceptable in 2005; the political environment is even less favourable now.</p>
<p>What transparency would there be on how the money is used? In economics, the net charge should just be taken by the Treasury, but would hypothecation be necessary for acceptance? But at what level – should it go to all transport, just to road users, just to car users, at what geography? What if the money were used to deliver better alternative travel making those modes more attractive? This would impact on the road price. Would the Treasury come to rely on the scheme revenue and not wish to change prices despite changes in need? Would hypothecation strengthen “ownership” of the road?</p>
<p>Deliverability - this would be one of the most complicated projects ever delivered in this country, in terms of both the technology and the charging structure</p>
<p>It is no surprise that the one place a GPS-based road pricing system is likely to be introduced (albeit on a limited scale) is within authoritarian Singapore. The UK government does not have the luxury of discounting the opposition to wider charging.</p>
<p>I’ve not even touched on the likelihood of a road-pricing system being delivered by the public sector, even with most delivered by the private sector - would anyone have any confidence in cost estimates for the delivery of a scheme? How many unknown issues will arise in the delivery stage and in the final product? Some, such as the IEA, will see this as an opportunity for more private sector involvement in the delivery and maintenance of roads.</p>
<p>I have identified quite a few issues here but any scheme with complex technology, public acceptability issues, novel legal concerns, and complex pricing structures will be certain to bring up totally new issues no-one has or could consider beforehand. Furthermore, there would also be entirely unintended consequences of a scheme.</p>
<p>The British political system is based around periodic elections, unfortunately for the deliverability of road pricing, the gaps between such elections are far too small for a scheme to be designed and implemented, let alone, bedded in. There will be far too great an incentive for an opposition party to point to defects in the proposed system and to make populist hay with their alternatives.</p>
<p>Finally, a mantra we hear all the time: “we paid for the use of the road”. Notwithstanding the fact that most such discussion of “road tax” is misinformed (to be polite), that many motorised vehicle users don’t pay Vehicle Excise Duty or pay very little, and that many non-motorised users pay VED and/or the other taxes (like council tax) that actually pay for roads, this refrain will be even stronger in a world with road pricing. “I’m paying extra for this road space at this time, get out of my way!”. Mr Toad will not be happy. Further if space for motorised vehicles were to be reduced, and prices go up to reflect, surely the conflict between “paying” and “non-paying” users will only increase?</p>
<h2 id="what-else">What else?</h2>
<p>As the various benefits of a complex scheme fall by the roadside so the balance shifts to “cruder” but simpler, arguably more acceptable alternatives. At what point does this mean a simpler system would in practice be better (taking into account deliverability)? The government has very limited measures to hand - fuel duty cannot be applied regionally and, in any case, if congestion is the main issue being targeted, EV users don’t pay fuel duty.</p>
<p>So, what are the alternatives? Do nothing is always an option; it is just not a good one for dealing with congestion. It is all too easy to put our hands up and say it is too difficult – after all, that is exactly what I have done above. But there are some measures that can be introduced.</p>
<p>We must accept that some journeys are not in the best interests of everyone, including other drivers, and use non-pricing measures wherever possible. Not everyone can use alternative means of transport – but a lot more can than currently do so. The fewer who drive who do not really need to, the more space there is for those who really do. If we do not accept this simple fact, then there are no solutions that will work.</p>
<p>We could make some journeys less easy by enforcing and re-enforcing existing legislation on parking, idling, speeding, phone use, etc. Little nudges can go a long way. We can ensure that those drivers deemed not fit or proper to drive do not do so, for these people often impose the highest costs on other road users. Safer roads mean less accidents mean less congestion.</p>
<p>We must look for reinforcing behaviour which reduces the need for road use, particularly at peak times – for example, making streets safer for school children to walk or cycle can create a virtuous circle. But such measures can often be divisive and need to be carefully designed to avoid unintended consequences. The collection of good data is key.</p>
<p>We could reduce capacity (inherently increasing the cost but for all drivers so doing it inefficiently) either directly per road (for example, increasing the width of pavements) or indirectly via measures that reduce the supply or increase the cost of parking such as a workplace parking levy. Physical reductions can be done in many ways, whether reducing the physical width of roads, the numbers of roads available, the times that roads can be used, the length of time that traffic signals provide for vehicles, etc. We could make other travel methods more attractive, hence affecting the implicit relative price between modes. Some of the measures to do that may also reduce the physical road space for motorised vehicles. But this is often politically very difficult to implement.</p>
<p>We can work with the providers of navigation devices and applications, either voluntarily or via statutory means, to iron out some of the perceived disbenefits of those systems. Could they have “soft” speed limits or diversions at certain times? We have seen, in effect, an increase in capacity with the use of these applications – roads previously more or less hidden from drivers have now been opened up. Purely from the drivers’ perspective, this is a good thing. But again, because they are not paying for their externalities, the allocation of road space is sub-optimal. Often the side roads now being used are not designed for the speeds being driven nor are the houses designed to reduce the noise from traffic. We need more localised research into why particular drivers are driving and to help find them alternatives where possible.</p>
<p>Some small measures which purport to help increase the flow of traffic may in the long-term actually hinder it. For example, reducing the pedestrian phases at traffic lights will merely attract more vehicles to a route over time, as well as make alternatives to vehicle journeys even less attractive. And that is ignoring the obvious equality impacts on disabled and other road users. Furthermore, incidents will always happen, if more traffic is in a system it becomes much harder to reroute it – some slack in a system is necessary.</p>
<p>We could move to a simpler per mile charging system that has been suggested by the AA chairman, with an equity scheme such as some miles included upfront. However, this would still entail some kind of black box solution that itself might be unacceptable and, by definition, it would not be able to target the congested areas so will have less benefits than a “proper” system. But it might be a useful replacement for fuel duty once enough EVs replace ICE vehicles.</p>
<p>A higher purchase levy or annual duty on vehicles to make up for reduced fuel duty take will merely shift the burden from higher-mileage to lower-mileage users, exactly the opposite impact needed to tackle congestion. Moreover, some users will rationalise their use on a sunk-cost fallacy. Regional pricing is a non-starter since users will use false addresses or find other ways around it and, in any case, it would also be very sub-optimal.</p>
<p>In some areas, it might be possible to increase the cost of or ration the number of residential car parking permits and this may have some marginal impact on congestion. Some users will decide it is no longer worth maintaining a vehicle and that will be one less vehicle on the road. However, without other measures this will just attract some other user from another area who was previously just short of deciding to purchase a vehicle. There are plenty of reasons to reduce parking permits (in particular, road safety) but I do not think congestion is one of them.</p>
<p>The combined effect of those smaller measures that could be introduced might be enough to mean a full road-pricing system is no longer justified.</p>
<p>All such intermediate measures will be seen as a war on drivers. But all will reduce the eventual price that drivers have to pay, either in congestion or in a full-on road-pricing system were it to come to that. The question for drivers is what is acceptable – a range of small measures, a full-on road pricing scheme, or continued congestion?</p>
<p>One less driver on the road is one less in other drivers’ way. But we need to be careful not to just shift the incentives from one set of drivers to another, so that we just have a different set of drivers on the road. That is the inherent beauty of the theory of road pricing – it is totally ambivalent as to who is driving and their reasons for doing so. Of course, that is why equity is such a big issue and why there would almost certainly have to be some kind of monetary or other transfer to those priced out of driving in order for it to be acceptable. It also means it has to be precise in order to deliver the benefits to the right people at the right time. I do not think this will be possible.</p>
<p>Conversely, we could increase road capacity where possible. There will be some short-term gains for some users, but the history of road capacity is that it soon fills up, and that such changes distort planning and housing provision, leading to more and more road journeys. This just creates a traffic bubble; at some point it must burst. But will road pricing be able to burst it? I think not.</p>
<h2 id="references">References</h2>
<p>Booth, Philip (2015) - Pricing roads – why are we waiting <a href="https://iea.org.uk/blog/pricing-roads-%E2%80%93-why-are-we-waiting">https://iea.org.uk/blog/pricing-roads-%E2%80%93-why-are-we-waiting</a></p>
<p>Campaign for Better Transport (2020) - Covid-19 Recovery: Renewing the transport system https://bettertransport.org.uk/sites/default/files/research-files/Covid_19_Recovery_Renewing_the_Transport_System.pdf</p>
<p>Centre for London (2019) – Green Light: Next generation road user charging for a healthier, more liveable, London <a href="https://www.london.gov.uk/sites/default/files/london_stalling_-_reducing_traffic_congestion_in_london.pdf">https://www.london.gov.uk/sites/default/files/london_stalling_-_reducing_traffic_congestion_in_london.pdf</a></p>
<p>Department for Transport (2007) - Public Acceptability of Road Pricing <a href="http://data.parliament.uk/DepositedPapers/Files/DEP2008-1762/DEP2008-1762.pdf">http://data.parliament.uk/DepositedPapers/Files/DEP2008-1762/DEP2008-1762.pdf</a></p>
<p>Department for Transport (2005) - The Government’s Response to the Transport Select Committee’s Report, Road Pricing: The Next Steps <a href="https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/272122/6560.pdf">https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/272122/6560.pdf</a></p>
<p>Deloitte (2012) - Road pricing: Necessity or nirvana? <a href="https://www2.deloitte.com/content/dam/Deloitte/au/Documents/public-sector/deloitte-au-ps-road-pricing-necessity-nirvana-180914.pdf">https://www2.deloitte.com/content/dam/Deloitte/au/Documents/public-sector/deloitte-au-ps-road-pricing-necessity-nirvana-180914.pdf</a></p>
<p>Economics Online (undated) – Market Failure – Road congestion <a href="https://www.economicsonline.co.uk/Market_failures/Road_congestion.html">https://www.economicsonline.co.uk/Market_failures/Road_congestion.html</a></p>
<p>Elliot, Peter, Jennings, Barry (2007) - Road pricing-key challenges <a href="https://www.twobirds.com/en/news/articles/2007/road-pricing-key-challenges">https://www.twobirds.com/en/news/articles/2007/road-pricing-key-challenges</a></p>
<p>Emissions Analytics (2020) – Press Release: Pollution From Tyre Wear 1,000 Times Worse Than Exhaust Emissions <a href="https://www.emissionsanalytics.com/news/pollution-tyre-wear-worse-exhaust-emissions">https://www.emissionsanalytics.com/news/pollution-tyre-wear-worse-exhaust-emissions</a></p>
<p>Guardian (2020) - AA president calls on the government to introduce road miles scheme <a href="https://www.theguardian.com/money/2020/jun/03/aa-president-backs-road-pricing-scheme">https://www.theguardian.com/money/2020/jun/03/aa-president-backs-road-pricing-scheme</a></p>
<p>International Monetary Fund (2010) – What are Externalities <a href="https://www.imf.org/external/pubs/ft/fandd/2010/12/basics.htm">https://www.imf.org/external/pubs/ft/fandd/2010/12/basics.htm</a></p>
<p>Laverty, Anthony A, Vamos, Eszter P, Panter, Jenna, Millett, Christopher (2020) - Road user charging: a policy whose time has finally arrived <a href="https://www.thelancet.com/journals/lanplh/article/PIIS2542-5196(20)30244-8/fulltext">https://www.thelancet.com/journals/lanplh/article/PIIS2542-5196(20)30244-8/fulltext</a></p>
<p>PWC (2019) - Squaring the circle – Reconciling the GDP and welfare impacts of transport interventions <a href="https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/824181/Squaring_the_circle_-_reconciling_the_GDP_and_welfare_impacts_of_transport_interventions.pdf">https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/824181/Squaring_the_circle_-_reconciling_the_GDP_and_welfare_impacts_of_transport_interventions.pdf</a></p>
<p>Walker, John (2011) – The Acceptability of Road Pricing <a href="https://www.racfoundation.org/wp-content/uploads/2017/11/acceptability_of_road_pricing-walker-2011.pdf">https://www.racfoundation.org/wp-content/uploads/2017/11/acceptability_of_road_pricing-walker-2011.pdf</a></p>
<p>Zdnet (2020) - Singapore readies satellite road toll system for 2021 rollout <a href="https://www.zdnet.com/article/singapore-readies-satellite-road-toll-system-for-2021-rollout/">https://www.zdnet.com/article/singapore-readies-satellite-road-toll-system-for-2021-rollout/</a></p>This is a revised version of my earlier post, having considered responses to the original and having had a little time to order my thoughts more carefully. I had originally planned a Twitter thread on my thoughts but there is just too much to cover for that format. I am sure there are some logical, factual and theoretical errors, for which I apologise in advance. I suspect this will remain a work in progress, to be edited when I have time. I know many of my friends are in favour of “smart” road user charging/road pricing. As with most things in life I am sceptical. I want to believe, I really do. The theory of road pricing is sound and the potential benefits enormous. But there are many practical issues that will need resolution both before and during the running of any scheme. Given these issues, it is not the panacea people think it is. By holding out hope for such a scheme, we will lose sight of achievable measures that can be implemented in the foreseeable future; waiting for a nirvana that will never come.New data files added plus some tidying2020-08-17T00:00:00+00:002020-08-17T00:00:00+00:00http://www.trafficflows.org/new-data-files-added-plus-some-tidying<p>I’ve played around with the new Shiny website (<a href="https://trafficflows.shinyapps.io/Traffic/">https://trafficflows.shinyapps.io/Traffic/</a>) and added some new data for Burntwood Lane (two sites from May 2018) and Tranmere Road (two sites from March 2018).</p>
<p>I’ve got a lot more I want to do with the visuals but I keep getting distracted with other things! For now I’ve had to be satisfied with some tidying up of dates and some formatting.</p>I’ve played around with the new Shiny website (https://trafficflows.shinyapps.io/Traffic/) and added some new data for Burntwood Lane (two sites from May 2018) and Tranmere Road (two sites from March 2018).Just what you need to cheer you up - new (Shiny) visuals!2020-07-25T00:00:00+00:002020-07-25T00:00:00+00:00http://www.trafficflows.org/just-what-you-need-to-cheer-you-up-new-shiny-visuals<p>I’ve been thinking of moving away from Power BI for a while for several reasons. First, there’s nothing I like more than some change and learning a new package. Secondly, I felt that R just offers more flexibility when designing charts. So with that in mind, I’ve moved the visuals to <a href="https://rstudio.com/products/shiny">Shiny</a>, which is an app that enables you to publish your R work and make it interactive.</p>
<p><img src="../assets/2020-07-25-fig1.png" alt="Fig1" /></p>
<p>The new visuals are at <a href="https://trafficflows.shinyapps.io/Traffic/">https://trafficflows.shinyapps.io/Traffic/</a>, which in itself is a lot easier to remember than the obscure URL that Power BI used.</p>
<p>You need to select at least one survey and then click on Redraw. If you change any survey selection, you’ll need to click on Redraw again. You can also select if you want to separately show the two directions for each survey or just merge them together.</p>
<p>This is very much work in progress but I think it’s good enough to share at the moment. My to do list includes:</p>
<ul>
<li>better handling of dates so it’s easier to compare surveys</li>
<li>prettifying the speed/flow scatter plots</li>
<li>providing more notes (and bringing the current ones up to date)</li>
</ul>
<p>I also want to write up my experience learning the little I have so far of Shiny. It’s a great tool but with a few twists, so it will probably take a few posts and I don’t have time at the moment.</p>
<p>I won’t be updating the <a href="https://app.powerbi.com/view?r=eyJrIjoiZDc4NTg5MzMtODNjMC00NTFjLThjMDgtNWNjNzViNjY2ODM5IiwidCI6ImMyZGNiMzkwLTQ0YzEtNGVlNy1hMjk4LTQ0N2IwY2I4Mjg4ZiJ9&pageName=ReportSection">old Power BI visuals</a> from now on and will take them down at some point. At least I don’t have to keep switching between my work and personal accounts, which was driving me mad.</p>
<p>I’d love some feedback - my details are at the bottom of this blog.</p>I’ve been thinking of moving away from Power BI for a while for several reasons. First, there’s nothing I like more than some change and learning a new package. Secondly, I felt that R just offers more flexibility when designing charts. So with that in mind, I’ve moved the visuals to Shiny, which is an app that enables you to publish your R work and make it interactive.Choropleth callouts in R2020-06-20T00:00:00+00:002020-06-20T00:00:00+00:00http://www.trafficflows.org/choropleth-callouts-in-r<p>Addendum to my <a href="/uk-choropleth-maps-in-r-merging-with-ons-data">last post on ONS data</a>.</p>
<p>The problem with some maps is that it’s hard to see some areas in a larger map. Ideally we want to see both the big picture but also zoom in on a smaller area either because we’re specifically interested in it or because we couldn’t otherwise see it. This example shows how we can do a callout plot of the London area but the principle works just as well for any other area, you just need to be able to separately identify the area in question so we can pull out the relevant rows.</p>
<p>To be able to identify a wider area we need a lookup table from the smaller area to the larger area. Luckily ONS provides <a href="https://geoportal.statistics.gov.uk/search?collection=Dataset&sort=name&tags=all(LUP_ADM)">a series of lookup tables</a> which we can read into R. I’ve downloaded the all-encompassing file which provides a lookup from ward to local authority to county to region to country and then read it with the following:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lookuptable <-
read_csv("input/Ward_to_Local_Authority_District_to_County_to_Region_to_Country_(December_2019)_Lookup_in_United_Kingdom.csv",
col_types = cols(
FID = col_double(),
WD19CD = col_character(),
WD19NM = col_character(),
LAD19CD = col_character(),
LAD19NM = col_character(),
CTY19CD = col_character(),
CTY19NM = col_character(),
RGN19CD = col_character(),
RGN19NM = col_character(),
CTRY19CD = col_character(),
CTRY19NM = col_character()
)
)
</code></pre></div></div>
<p>Note that I’ve overridden the default parsing to force all the ID columns to be character. Try without this to see what happens.</p>
<p>Now we want to identify just the local authorities in London. Since we’ve got ward level data we need to be careful here otherwise we’ll get a lot of rows for each local authority;</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>df3 <-
lookuptable %>%
filter(CTY19NM %in% c("Inner London", "Outer London")) %>%
distinct(LAD19CD) %>%
left_join(df, by = c("LAD19CD" = "admin-geography")) %>%
filter(`age-groups`=="65+") %>%
left_join(mapdata, by = c("LAD19CD" = "id"))
</code></pre></div></div>
<p>Line by line:</p>
<ol>
<li>Store the result of the entire pipe into df3</li>
<li>Start with the lookup table loaded above</li>
<li>Filter it to just those local authorities in the “counties” (CTY19NM) of Inner and Outer London</li>
<li>Then just keep the distinct values of LAD19CD (the local authority code)</li>
<li>Join that with the main data file</li>
<li>Filter only the 65+ data</li>
<li>Finally, join that with the map data</li>
</ol>
<p>Note that when constructing pipes like this, experiment with each stage to see the intermediate results eg:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>df3 <- lookuptable %>% filter(CTY19NM %in% c("Inner London", "Outer London"))
</code></pre></div></div>
<p>This is very easy - just copy the code back and forward between your script file and the console window. Anyway, go back to the full code. We can now plot this but with a slight difference. Because we want to plot this as an additional component of the big map we need to save it in a special object called a <a href="https://bookdown.org/rdpeng/RProgDA/the-grid-package.html#grobs">grob</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gLon <- ggplot(data = df3, aes( x = long, y = lat, group = group, fill = V4_1 )) + geom_polygon() + scale_fill_gradient(low="#0000ff", high="#ff0000") + coord_equal() + theme_void() + theme(legend.position = "none") +
theme(panel.border = element_rect(
colour = "black",
fill = NA,
size = 1
))
grobLon <- ggplotGrob(gLon)
</code></pre></div></div>
<p>First, we generate the plot using just the London data (removing the legend and adding a plot border) and save it to an object called gLon. We can still plot that object by just typing gLon (and you’ll see another time how we can use this for plotting several graphs neatly). But instead we’ll convert it into a grob named grobLon.</p>
<p>We can then add the grob as an annotation on the main map:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ggplot(data = df2, aes( x = long, y = lat, group = group, fill = V4_1 )) + geom_polygon() + scale_fill_gradient(low="#0000ff",high="#ff0000") + coord_equal() + theme_void() + xlim(0, 1000000) + annotation_custom( grob = grobLon, xmin = 600000, xmax = 900000, ymin = 500000, ymax = 750000 )
</code></pre></div></div>
<p>I’ve changed the x limits to create such extra space and the x/y min and maxes have been determined by trial and error. Note that a different way to expand the limits is to use:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>expand_limits(x = 1000000)
</code></pre></div></div>
<p>That way means you don’t have to specify the other end point.</p>
<p>Anyway, the final map is shown below:</p>
<p><img src="../assets/2020-06-20c-fig1.png" alt="" /></p>
<p>Again, you can prettify this by adding titles and a line from the callout to the London area. Changing the plot code to the following makes it look nicer:</p>
<p><code class="language-plaintext highlighter-rouge">ggplot(data = df2,
aes(
x = long,
y = lat,
group = group,
fill = V4_1
)) +
geom_polygon() +
scale_fill_gradient(low = "#0000ff",
high = "#ff0000",
name = "Femalenpercentagenage 65+") +
coord_equal() +
theme_void() +
expand_limits(x=1000000) +
annotation_custom(
grob = grobLon,
xmin = 600000,
xmax = 900000,
ymin = 500000,
ymax = 750000
) +
annotate(geom = "text",
x = 750000,
y = 780000,
label = "London") +
geom_segment(x = 530000,
xend = 600000,
y = 170000,
yend = 500000) +
labs(title = "Percentage of population aged 65+ who are female",
caption = "Source: https://www.ons.gov.uk/datasets/older-people-sex-ratios/editions/time-series/versions/1") +
theme(plot.caption = element_text(hjust = 0, size = 8))
</code></p>
<p>Notes on the above:</p>
<ol>
<li>I’ve used CTRL SHIFT A in RStudio to put in lots of space to make the code easier to read.</li>
<li>I’ve used the name parameter in scale_fill_gradient to add a legend title (there are lots of different ways of adding a legend title)</li>
<li>The title has the newline character (n) inserted to break it up</li>
<li>I’ve added a text annotation for the callout box</li>
<li>geom_segment adds a line</li>
<li>labs adds a title and a caption</li>
<li>and finally the last theme function changes the caption justification to right and reduces the font size</li>
</ol>
<p>So the final product is:</p>
<p><img src="../assets/2020-06-20c-fig2.png" alt="" /></p>Addendum to my last post on ONS data.UK choropleth maps in R - merging with ONS data2020-06-20T00:00:00+00:002020-06-20T00:00:00+00:00http://www.trafficflows.org/uk-choropleth-maps-in-r-merging-with-ons-data<p>This follows on from the <a href="/creating-choropleth-maps-in-r-with-ggplot2">previous post on UK choropleths</a>, except now we’ll be using real data.</p>
<p>Let’s go and see what’s out there at <a href="https://www.ons.gov.uk/help/localstatistics">https://www.ons.gov.uk/help/localstatistics</a>. Just chose an interesting dataset and download it. Let’s download the data for older people sex ratios. Go to <a href="https://www.ons.gov.uk/datasets/older-people-sex-ratios/editions/time-series/versions/1">https://www.ons.gov.uk/datasets/older-people-sex-ratios/editions/time-series/versions/1</a> and choose Other download options, Complete dataset (CSV format).</p>
<p>What does this dataset look like when downloaded into R?</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>library(tidyverse)
library(rgdal)
library(broom)
df <- read_csv("input//agesexratio.csv")
</code></pre></div></div>
<p>Note that read_csv spits back the column definitions it uses to parse the data and convert to different types. It is essential you always check this as the parsing can get fooled by blanks or other unexpected data in the first 1000 or so rows. You can then copy the column specification and alter it as appropriate into a col_types parameter for read_csv. See <a href="https://readr.tidyverse.org/reference/read_delim.html">https://readr.tidyverse.org/reference/read_delim.html</a>.</p>
<p>The data look like this:</p>
<p><img src="../assets/2020-06-20a-fig1.png" alt="" /></p>
<p>If you scroll down a bit you’ll see that the data are a mix of different levels (England and Wales, Great Britain, local authorities etc). That’s ok as when we merge with an appropriate map table we can exclude any extra rows. The data also contains data for two age-groups - 65+ and 85+ and also some NAs in the main data column V4_1. We’ll need to be aware of that before mapping.</p>
<p>We know from looking at the file that (some of) the data are at UK local authority level and we can see the relevant ID codes in the admin-geography column. It is this column we’ll need to merge with the map data. Let’s get that:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sourcemap <- "Local_Authority_Districts_December_2019_Boundaries_UK_BUC-shp" sourcefile <- str_remove(sourcemap, "-shp") sourcefile <- str_c("maps", sourcemap, "", sourcefile, ".shp") shp <- readOGR(dsn = sourcefile)
</code></pre></div></div>
<p>Before we tidy shp, we need to find out what the ID column is called. Use View(shp) or single-click on shp in the environment list. Remember that map shape files contain several lists - expand the data one thus:</p>
<p><img src="../assets/2020-06-20a-fig2.png" alt="" /></p>
<p>We don’t want the area names, merely the ID codes. These are in the lad19cd column, so we can tidy the shape file keeping that column:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mapdata <- tidy(shp, region = "lad19cd")
</code></pre></div></div>
<p>Mapdata looks like this:</p>
<p><img src="../assets/2020-06-20a-fig3.png" alt="" /></p>
<p>We now want to merge this with the age sex ratio data, using the same left join technique as before. However, we also want to exclude the NAs and only have data for the 65+ ratios. Because we have tidyverse loaded we can do this in one pipe:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>df2 <- df %>%
filter(`age-groups` == "65+" & !is.na(V4_1)) %>%
left_join(mapdata, by = c("admin-geography" = "id")) %>%
filter(!is.na(order))
</code></pre></div></div>
<p>Taking that one line at a time:</p>
<ol>
<li>
<p>save the contents of everything that follows into df2, start with df</p>
</li>
<li>
<p>then filter that so that age-groups is limited to 65+ and all NAs in column V4_1 are excluded (== means equal to, & means and, ! means not)</p>
</li>
<li>
<p>then join the result of that with the map data, matching on the admin-geography and id columns</p>
</li>
<li>
<p>then filter the result of that to exclude any rows where order is NA (ie where there is no corresponding map data - all those non-local authority rows).</p>
</li>
</ol>
<p>The results of the entire pipe are saved into df2.</p>
<p>Note that the age-groups and admin-geography columns have been imported as non-standard R column names (you can’t have a minus sign in a column name for obvious reasons). The tidyverse import functions will refer to such columns with surrounding back-quotes. If you are going to do a lot of analysis with such columns, it’s better to rename them, otherwise you end up with slightly untidy code. I haven’t bothered here. The join doesn’t need the back-quotes since the merge columns are already surrounded by double quotes.</p>
<p>We can then plot the data with a simple map plot as before:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ggplot(data = df2,
aes(
x = long,
y = lat,
group = group,
fill = V4_1
)) +
geom_polygon() +
scale_fill_gradient(low="#0000ff",high="#ff0000") +
coord_equal() +
theme_void()
</code></pre></div></div>
<p><img src="../assets/2020-06-20a-fig4.png" alt="" /></p>
<p>There’s a lot you can do here to tart this up, change the legend title, add a plot title, change the colour scheme, etc.</p>This follows on from the previous post on UK choropleths, except now we’ll be using real data.World choropleth maps in r with ggplot22020-06-20T00:00:00+00:002020-06-20T00:00:00+00:00http://www.trafficflows.org/world-choropleth-maps-in-r-with-ggplot2<p>An appendum to my <a href="/creating-choropleth-maps-in-r-with-ggplot2">previous post</a> in which I discussed UK choropleths in R.</p>
<p>Extending the methodology for UK maps to world maps is easy, There are world maps inbuilt in R but let’s go get them from scratch. <a href="https://ec.europa.eu/eurostat/web/gisco/geodata/reference-data/administrative-units-statistical-units/countries">Eurostat</a> has a collection of country maps. If you download and extract the 2016 country ZIP file you’ll find that it contains different resolutions in separate folders. Here I load the 60 metre resolution file in the same way as I did for the UK maps:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>library(tidyverse)
library(rgdal)
library(broom)
sourcefile <- "..mapsref-countries-2016-60m.shpCNTR_RG_60M_2016_4326.shp"
shp <- readOGR(dsn = sourcefile)
mapdata <- tidy(shp, region = "NAME_ENGL")
</code></pre></div></div>
<p>Note I’ve used the region parameter in tidy so the mapdata table is loaded with the id column containing the English country names:</p>
<p><img src="../assets/2020-06-20b-fig1.png" alt="" /></p>
<p>Let’s do a test plot:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ggplot(data = mapdata, aes(x = long,
y = lat,
group = group)) +
geom_polygon(fill = "#FFFFFF",
colour = "#000000",
size = 0.25) +
coord_equal() +
theme_void()
</code></pre></div></div>
<p>Which gives us:</p>
<p><img src="../assets/2020-06-20b-fig2.png" alt="" /></p>
<p>We merge our data with the mapdata in exactly the same way as we did for the UK maps. Let’s create a dummy dataset:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mydata <- data.frame(id = c("France", "United Kingdom", "Germany", "Spain", "Portugal", "Ireland"), value = c(100, 200, 300, 400, 500, 600))
</code></pre></div></div>
<p>and then merge it (using the alternative version that drops areas for which there are no data):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>df <- left_join(mydata, mapdata, by="id")
</code></pre></div></div>
<p>We can then plot the map:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ggplot(data = df,
aes(
x = long,
y = lat,
group = group,
fill = value
)) +
geom_polygon(color = "#000000", size = .25) +
scale_fill_gradient2(
low = "blue",
mid = "red",
high = "yellow",
na.value = "white"
) +
coord_equal() +
theme_void() +
theme(legend.position = "right")
</code></pre></div></div>
<p>Which gives us an interesting problem:</p>
<p><img src="../assets/2020-06-20b-fig3.png" alt="" /></p>
<p>What is going on? The mapdata table has non-mainland territories coded to the main country. If we want to exclude them, we could either identify them in the mapdata table and specifically exclude them or we could just change the map coordinates. Let’s do the latter, comment out the theme_void() temporarily to show the latitude and longitude scales:</p>
<p><img src="../assets/2020-06-20b-fig4.png" alt="" /></p>
<p>And then experiment to find the correct x and y limits to zoom in on the region we want:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ggplot(data = df,
aes(
x = long,
y = lat,
group = group,
fill = value
)) +
geom_polygon(color = "#000000", size = .25) +
scale_fill_gradient2(
low = "blue",
mid = "red",
high = "yellow",
na.value = "white"
) +
coord_equal() +
theme_void() +
theme(legend.position = "right") +
ylim(36, 61) +
xlim(-12, 15)
</code></pre></div></div>
<p>While not perfect (it still shows non-mainland areas) this will do for now:</p>
<p><img src="../assets/2020-06-20b-fig5.png" alt="" /></p>
<p>Note that any country with data missing is excluded from the map.</p>An appendum to my previous post in which I discussed UK choropleths in R.