LornaJane's Website

Using mitmproxy reverse mode for API testing

2026-06-29T20:06:00+01:00

mitmproxy is my preferred proxy for API testing these days. I'm using it mostly in reverse mode though, so I thought I'd capture my setup in case it's useful for anyone else. My main use case is to inspect the traffic coming into the API server and being returned, mostly when running tests against it. Where mitmproxy is typically used as a proxy to monitor traffic passing into/out of a client application, for this I'm concerned with the server's handling. Reverse mode lets me inspect and adjust the traffic as I go along. I start up the proxy, configure the tests to point to it instead of the actual server, and I'm all set.

Start reverse proxy mode

Mitmproxy has a great web UI as well but as a keyboard-only user, I just use the terminal tools - pick what works best for you, but these instructions are CLI-based.

Start the reverse proxy and tell it where to send traffic onward to; in this example the API I'm calling is on http://localhost:3000

mitmproxy --mode reverse:http://localhost:3000

By default the server listens on port 8080 and forwards everything.

Configure the application

In my use case, I'm adjusting the API tests to point to the proxy instead of the actual server so I can capture, replay, and/or adjust the traffic when I'm working with the tests.

I configure my tests to test an API at http://localhost:8080 (actually I'm running bits of this in docker so it ends up being http://host.docker.internal:8080 but this is more of a note to future-me than general advice!)

That's all. Start using the application and you should start to see the requests and responses showing up in mitmproxy.

Additional notes and tricks

From the logged traffic records, you can replay, duplicate, and edit requests, which is great for testing and inspecting anything that's not trivial to reproduce.

You can also use mitmproxy add-ons to rewrite responses if you need to. I use this feature to throw unexpected responses (that the API should never actually produce) back at the tests to make sure they do fail when they should!

To save a flow, either to share or to come back to later on, press w and provide a filename. Then load it in a future mitmproxy session using a command like mitmproxy -r flows.mitm.

Add a comment if you have additional experiences to share, I'm always excited to learn some new tricks to add to my collection!

Selectively ignore lines in git diff

2026-03-04T07:57:00+00:00

I have a things-as-code project that outputs mostly text-based formats, but a lot of them. To keep an eye on consistency, I rebuild all the outputs and dump them into a local git repository so I can very easily diff to spot any changes - which was fine until we added a build timestamp, so every file looks changed on every run! This post is about ignoring the matching line with git diff -I.

With a task like this where I want a quick check on a LOT of output (a few million lines I think) which probably hasn't changed much, the text-based output and source control tools are a great fit. Also since I'm a confident git user, I can very easily re-run the script in the directory (already set up with the paths for my system) and when it finishes, check there's nothing in git diff.

It worked great - until it didn't! We added a nice piece of metadata showing when the file was last regenerated, which seemed like a good feature and I approved the change myself. But of course it means all the output files look changed each time. I needed a way to ignore only that change and still see anything except the timestamp updates.

Ignore patterns in git diff

The -I (it's a capital "i" and is short for --ignore-matching-lines) accepts a regex and can be used with diff, log, show and probably other git commands. I'm using it with diff, like this:

git diff -I 'x-generated-date'

I only see diff entries that do not match this regex (okay, in this example it's a string, but you can do cleverer things if you need them! I could put the datetime pattern in here to be super certain what I'm matching), which is exactly what I need. I know that this line will always be changed so we can ignore it and only inspect any other changes that are present.

To see just the list of files so I can get an idea of the blast radius on what I just changed, I can add --stat to my diff command.

Working with larger filesets and particularly with generated output where it's useful to quickly sanity check any side effects, tricks like this are so useful - and I thought I would share. If you have additional strategies to offer me in return then I would love to hear about them in the comments, please!

Git renames are not renames

2026-01-28T09:29:00+00:00

I consider myself pretty git-confident, I've worked with it a lot, taught it, been a git consultant, run engineering and various things-as-code teams. This week I had a spectactular git problem where merging one branch into another produced changes that didn't exist on either branch. Turns out, renaming directories in a monorepo with multiple almost-identical boilerplate documentation files comes with surprises...

The short version is: Git actually doesn't track renames at all. It tracks adds and deletes, and presents them to the end user as renames. If the file is 50% the same, it's considered a rename. It's a bit of a blunt instrument and so it goes wrong sometimes. You can adjust whether your client shows you renames, or how similar the files can be - but you don't impact how other users see it.

The situation

I'm in a situation where I need to alter the directory structure of a monorepo with lots of similar-structured but actually different files that appear in every folder. I'm adding an additional folder level, to allow siblings of each project. Did I mention that git doesn't support renames? It doesn't track folders either.

The similar structures of the projects turned out to be even more interesting when I realised that there are two sets of mostly-boilerplate documentation in every folder. The two folders (they both have markup content that gets published as documents) have some files that are named the same, and every folder has both folders, with either no changes, or with very minor (but crucial!) changes.

I renamed every folder, committed the changes, updated ALL the build scripts to handle the changes, and thought I'd done the difficult bit (famous last words). Then when I started applying these updates to the existing work-in-progress branches, I noticed that git was deleting or updating files in other projects within the monorepo - changes that were not in either of the branches I was combining.

Why this happens

Git simply has no idea which adds go with which deletes - and on a big project, with a lot of almost-the-same files which were all renamed? It just fails in a big messy heap.

Things I tried that did not help at all:
- splitting up the restructure into small chunks so I only moved one directory at a time, per commit
- adjusting git's rename settings when merging the branches
- swearing a lot

(I did have it on my list to make all the repeated content into templated files, because it's annoying to maintain as it is - but I had no idea that was on the critical path for the directory restructure!)

This story has no ending

As things stand, I have no solution! I knew this was the way git worked and I knew there was content that could be problematic - but this problem manifested in a way I didn't expect when the changes which had been applied apparently without any issue were then merged into the ongoing feature branches (it's a slow moving project, a lot of contributors, all the reasons I normally advise against doing this sort of repository surgery in the first place). I started seeing phantom changes that simply didn't exist in any of the "before" branches and it took me a moment to understand what happened.

Things I am doing so we can make progress with the changes:
- lots of checks in CI for pull requests to help humans catch any weirdness at review time and before merge
- having a good attitude to if we need to fix minor documentation excitements as a result of these changes, because they are absolutely a positive change for the project and it's the least interesting files that are impacted
- manually applying the same folder restructures to the feature branches (it was a script in the first place so it's repeatable) before allowing users to sync with the main branch seems to help git to know what's happening

My only advice is either not to do this sort of change, or to have a no-pull-requests interval. Neither of those were options for me though so I'm sharing in case you find yourself in a similar situation some day! Be aware that "renames" (and in fact "directories") are not always what they seem ....

Manage Diagrams in AsciiDoc on GitHub

2026-01-25T21:22:00+00:00

I use a lot of asciidoc these days for work documentation (and I love it) and I've been so happy that GitHub renders it when you view a repository in the web browser, just like it does for Markdown and ReStructuredText. BUT what GitHub does not do is render the image types that asciidoc does so even though I'm working with asciidoc and PlantUML, and the asciidoc tools render those diagrams nicely in PDF and HTML output - GitHub's rendering doesn't. So here's a quick overview of how I handle those repositories.

Before I start, I should mention that for our context, the outputs are usually A4-sized PDF documents, so GitHub in the browser is a similar sort of form factor, and none of this stuff needs to be print quality. If you have a use case that's none of the above, you might want to take a different approach.

Setup for GitHub AND Asciidoctor

Asciidoctor has good rendering of PlantUML (and all sorts of other format) diagrams, but GitHub does not, so the tradeoff is to have the diagram source in the repository, have build scripts (that run locally and in CI) to build the SVG version of the diagram and also include that in the repository, and to reference the SVG to include the diagram in the document.

When adding a diagram, the author creates the PlantUML (.puml) file and runs the build script which generates the SVG format in an images/ directory. The author can then put the link to the SVG file into the text.

The build script regenerates images if the user has the PlantUML tooling installed so that any update are included. If the tool isn't there, the images/ directory is included in the repository anyway so that the content is viewable in GitHub, so the existing images can be used locally too.

Image content that works in multiple publish destinations

This version of the diagrams works perfectly well in the generated PDFs too, and since there's a build step for PDFs there was an obvious place to hook the image generation step into.

SVG works well enough in both contexts, and while it's annoying to have the extra build requirement for authors, I think the benefits of having the content readable with images in the repository is worth it. Having to get the published PDF to have all the information available is not great, and this is a document that has a lot more readers than writers!

Tag Kinds in OpenAPI 3.2

2025-12-08T08:32:00+00:00

OpenAPI tags have always been annoying: user-supplied arbitrary data for endpoints should be a fabulous feature - but the documentation tools seem to think that tags are only for them so it becomes more difficult to use tags for other purposes. In fact it is very useful to be able to tag endpoints with lots of different categories of data and so in OpenAPI 3.2, tags were enhanced to include an additional "kind" field so that different kinds of tag could be used.

Being grumpy about documentation tools using tags for themselves is silly (especially as I built some of those tools!), and I'm really pleased there's a better way now. Tags might be useful for:
- tagging endpoints that are only for an exclusive group of users
- tagging endpoints to show something is "new" in an API, or relates to a particular version
- tagging endpoints to indicate their lifecycle status such as being in beta, or being deprecated
- tagging endpoints to say whether they are included in the published documentation or SDKs
- tagging if this endpoint follows a particular pattern or exception to a pattern
- tagging for some other tool to pick up and use the tag for more information later

... you get the idea, tags can be used for anything now.

Declare the kind of tag it is

Tags are declared in their own tags section, and it's here that you can say what sort of tags they are. Here's an example:

tags:
 - name: product
   summary: Products
   description: Product operations
 - name: partner
   summary: Partner
   description: Operations available only to partner organisations, or other privileged accounts.
   kind: audience
 - name: v2
   summary: Version 2
   description: Available in API v2 or later
   kind: version

Tags without a "kind" default to type "nav" - meaning the navigation tags which is the pre-3.2 standard behaviour. This default setting means that you can upgrade an OpenAPI description from 3.1 to 3.2 without needing to make changes, but when you want to introduce new tags you can do so by including the "kind" of tag that they are.

Apply the tag

Applying tags is done the same way as before - again, no changes needed on upgrade and the additional data is applied in the tags entry rather than at the operation level. So it looks something like the example below if I apply a v2 tag:

paths:
  "/v2/products/":
    operationId: getProductsV2
    summary: Get all products
    tags:
      - v2
      - products

Applying multiple different kinds of tag is expected and doesn't need any special setup - the tags have their own properties, and you can apply as many as you like.

Tags Registry

The kind field is an open string - you can define your own kinds of tag and that's intentional so that this feature can be used in different ways in different contexts. That said, there are some common use cases and for those we'll get much better experiences if we collaborate on the tags we use and how we use them. That way, we can all align and have the tools support our use cases and give us seamless interoperability as we send our OpenAPI descriptions through all sorts of different pipelines.

The OpenAPI tag kinds registry has already been set up with some basic entries: nav (default), badge, and audience. Please check out the list for new entries and open pull requests to add tag kinds you're using yourself.

Tags are not just for docs any more

Tell your friends, your colleagues, and your AI assistants: tags are not a docs feature. They're an arbitrary user-supplied data feature, and I can't wait to see what you build!

For more information about tags, read the tag kinds documentation on the OpenAPI Learn site.

Nested tags in OpenAPI 3.2

2025-12-04T20:53:00+00:00

OpenAPI has always had support for simple tags, but the OpenAPI 3.2 release brought in some serious tag upgrades including a summary field, a "kind" field with registry, - and the ability to nest tags which is the focus of today's post. If one level of organisation isn't enough for your API (and on bigger APIs I'd argue it shouldn't be) then the ability to indicate which tag is the parent of this tag will be a good feature to adopt when you upgrade your OpenAPI descriptions.

The basic premise is: each tag has a 'parent' field which contains the name of a tag for this tag to belong to. Let's quickly look at an example:

tags:
  - name: account
    summary: Account Details
    description: Operations relating to the customer's account
  - name: accVerification
    summary: Verification
    description: Operations for verify a new user or user detail change
    parent: account

The parent tag does not have awareness of its children, but the child tag indicates the parent that it belongs to. This works well for APIs that are composed of smaller APIs such as in a miroservices context.

There are no rules in the specification itself about the matching parent must exist - there are also no restrictions on creating structures that clearly won't work such as loops. The tools that implement the tag structures will probably point out the error of your ways though, if you try to use something like that!

APIs that have previously used extensions to group tags, such as x-tagGroups (which I see fairly frequently), will find this approach gives an OpenAPI official way to do the same thing. The major benefit is that all tools now use the same syntax to describe the same thing and your API descriptions should be more portable across different contexts.

For more details on nesting tags in OpenAPI, see the OpenAPI learn site - then come back here and leave me a comment on what your structure looks like!

Quick local API docs with Scalar

2025-09-17T21:05:00+01:00

Today I'm sharing a quick-and-dirty script to take an OpenAPI description, spin up a docs server locally, and copy the URL into your clipboard. I also use a bit of glob expansion in my script to find the right folder, because I have a lot of APIs with long and formulaic directory names (TM Forum members know this story). I'm spinning up Scalar here as it's my current favourite just-works local API docs platform.

The script:

#!/usr/bin/env -S uv run --script

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "argparse",
# ]
# ///

import glob
import subprocess
import random

port = random.randint(0,500) + 3000;

# Add your own fudge/expansion magic here if you have APIs to find
path = f'../api-*/oas/*.oas.yaml'
url = f'http://localhost:{port}'

files =  glob.glob(path)
if len(files) > 0:
  for f in files: # works for one match
    # copy URL to clipboard for convenience
    subprocess.run(["pbcopy"], input=url, text=True)
    subprocess.run(["scalar", "document", "serve", "-p", str(port), f])
else:
  print("API not found");

I'm using Python and I run this script with uv because I run this script from lots of different folders and it takes care of dependencies so neatly (confession: my actual script uses argparse and some inputs to feed the "guess the location of the API file from these three digits" magic but that part probably isn't useful to other people!)

First I generate a random port number and use it to construct a URL - there's no checking that the port is available but given that I rarely have more than 4 or 5 of these up at one time, it rarely collides and I just run the script again if it does!

The subprocess calls then run the commands I would actually run myself on the commandline if I didn't use a script to generate a random port number and do some path fudging logic for me:

`pbcopy http://localhost:3333` (run this command first because the next one blocks)
scalar document serve -p 3333 openapi.yaml

Then I have the URL and can paste it into my browser. You could just open a browser tab instead of copying but I find it always picks the wrong tab/window/app/whatever so in my clipboard is more useful to me. I also use a clipboard history tool so it isn't a problem to have lots of things writing to my copy buffer.

Feel free to adapt the script to fit your own paths and preferred tools; probably some of you do this often enough that it's worth having a wrapper rather than remembering the correct incantation or relying on a hosted platform to download and upload to when the file is already local. And as always, if you do something differently, I'd love to hear about it!

API Specificity with Overlays and Enums

2025-07-21T08:48:00+01:00

The more I work on API standards, the more I realise how few teams understand that they can adopt the standards and, without breaking any contract, adapt them to make a strong interface for their own application. One of my favourite examples is to add enums where a standard interface cannot dictate the exact values, but in your own implementation it is very helpful to do so.

To start with, let's use this library book example snippet from an OpenAPI file - this is just the Components section:

components:
  schemas:
    Book:
      type: object
      properties:
        isbn:
          type: string
          description: ISBN publication identifier
          format: isbn
          example: "9781250236210"
        title:
          type: string
          description: Book title
          example: "A Psalm for the Wild-Built"
        author:
          type: string
          description: Author's name
          example: "Becky Chambers"
        genre:
          type: string
          description: Genre of the book
          example: "Solarpunk"

That genre field at the end of the list is just a string - but in my library, I'm going to only offer a very specific set of genres, so I want to limit the values that can be used there. To that I'll use an OpenAPI Overlay.

Here's the Overlay:

overlay: "1.0.0"
x-speakeasy-jsonpath: rfc9535
info:
    title: "Set the genres list"
    version: "1.0.1"
actions:
    - target: $["components"]["schemas"]["Book"]["properties"]["genre"]
      update:
        enum:
            - Science Fiction
            - Fantasy
            - Speculative
            - Solarpunk
            - Philosophy

There are a few different tools that you can use to apply an Overlay - I've been using Speakeasy CLI a lot lately, mostly because I'm working with other OpenAPI tools and then using this tool's compare command to get an OpenAPI diff in Overlay format that I can reapply elsewhere.

Using Speakeasy CLI, the command to apply the overlay (let's say the file is overlay.yaml) to the OpenAPI description (imaginatively, mine is called library.openapi.yaml) is as follows:

speakeasy overlay apply --overlay=overlay.yaml --schema=library.openapi.yaml --out updated.openapi.yaml

The overlay updates the Book schema so that now the components section looks like this:

omponents:
  schemas:
    Book:
      type: object
      properties:
        isbn:
          type: string
          example: "9781250236210"
          description: ISBN publication identifier
          format: isbn
        title:
          type: string
          description: Book title
          example: "A Psalm for the Wild-Built"
        author:
          type: string
          description: Author's name
          example: "Becky Chambers"
        genre:
          type: string
          description: Genre of the book
          example: "Solarpunk"
          enum:
            - Science Fiction
            - Fantasy
            - Speculative
            - Solarpunk
            - Philosophical

Make your API implementations your own

Adding repeatable OpenAPI description updates with Overlays is a good way to get from a generic or rather vague API description to something that you can implement efficiently in your own projects. Especially with published standards that might be used in many different settings, it can be particularly valuable to adapt to your own needs.

In this example, we showed adding enums which is a great way to give your SDKs and validators clearer instructions on the values that should be expected. Don't neglect the other fields, such as updating descriptions or adding meaningful examples that will help those implementing the API (human or machine) to have the context they need to be successful.

Safe Screensharing Setup

2025-06-10T17:32:00+01:00

In my time I've spent a LOT of time on screenshare, and I'm confident sharing my screen. Part of the confidence comes from maintaining a usage pattern that means I CAN confidently share my screen and rarely embarrass myself. After sharing some tips with a former colleague recently, I have tried to package up the best bits of that conversation and turn it into some tips to share.

Everything about this really hinges on me having two screens, if that's not your setup, then these tips might not work brilliantly for you. Doing anything more than opening a shared document without a second screen to put the video call on is super hard work and probably needs a different approach than I'm using here.

Always share the same screen My setup has my laptop on a stand on the left, my second screen on the right, and the webcam somewhere above the join between the two. I always share the laptop screen, and never share the other one, so my computer is always set up to respect that, even when I'm not screensharing.

Good window location habits Since I know I will sometimes share my laptop screen, I never put unsafe content on that monitor. Ever. ALL chat windows are always on the not-shared screen for example, because you just don't know when a colleague is going to suddenly DM you something.

Popups are a risk I keep my computer permanently in "do not disturb" mode and none of my applications are permitted to pop up toast notifications. I have mobile devices on a stand on my desk for that sort of thing!

Share the whole screen because otherwise you have to keep restarting to share a different tab or application. I often open a new web browser window and put the one I'm actually using onto the "do not share" space, and just open the things I do want to share in the browser window on the laptop screen.

Make the fonts bigger than you'd imagine there is always someone watching on their phone, or who doesn't see as clearly as someone else. This is also another reason for sharing my laptop screen, always - it's smaller than my external monitor and sharing that just makes content unreadably small (or I have to magnify everything a LOT).

Assume that the screenshare is working. To avoid the awkward "can everyone see my screen? [pregnant pause]" situation, take an audio description approach for what's on screen. "I'm sharing this week's meeting agenda so we can look at it together ...." someone will absolutely tell you if they don't see it! Speaking the visual like this can be more inclusive either to those who don't see well or to those who are looking at another window :) Sometimes it's a useful cue that just listening is missing some of the content.

Consider your workspaces setup I use workspaces on my linux machine, essentially four different desktops that I use for different purposes. Most of the time, the desktops span monitors, and I'll have web browser and notes apps on one, development work on another, and so on. If I know I'll be sharing applications from multiple spaces then I sometimes set the second monitor to not change with the workspace changing. The advantage of this is that I can put the video call with other people, the speaker notes, or something else in a location where I can see it throughout the session. I also don't need to learn/remember how to switch between things to demo - because my muscle memory will Just Work (TM) since this is my usual setup.

Your mileage will absolutely vary - and I would love to hear your tips! I so often "lead" a meeting, teach or demo something, or at one time of my life stream on Twitch, that having this setup permanent works well for me and means I don't spend any mental energy on how to present a thing. I just turn on the share, and focus on what I'm actually trying to do. Hopefully there's something there that will be useful in your own setup - let me know?

Publish to GitHub pages with Sphinx

2025-06-05T09:59:00+01:00

I'm a big fan of docs-as-code for more or less any type of content publishing, but I'm less of a fan of Jekyll, the default tool used in GitHub pages. I also prefer ReStructuredText over Markdown as a markup format, so Sphinx is definitely on my shortlist of SSGs (Static Site Generators) for my projects. I recently switched the rst2pdf docs to using Sphinx hosted on GitHub pages, so here are my setup notes.

Basic setup

This project uses Sphinx with a few additions:
- Furo theme - a more modern-looking theme than the default that you'll recognise from older websites!
- External TOC plugin, to make the navigation sane (The toctree is confusing for me, never mind all the fly-by docs contributors this project wants to be ready for)
- Copy button plugin, for easy use of code samples

I also published a list of Sphinx extensions for developer portals which might be interesting if you're setting up your own Sphinx docs project.

Using the basic Furo skeleton as a setup, I moved our existing content (mostly converted with pandoc into the new structure.

Content migration is a whole other blog post so I'll try to avoid that tangent for now.

Build process

Sphinx uses make to build, which is great as it's very widely known, used and supported.
I got everything working locally so that I was sure the content worked and the Sphinx build was all correct and happy.
One thing that I'd recommend is putting all the input content in one folder (not the root - in my case source/) and all the output content in another (I used build/).
The site gets built by a GitHub action, and the new HTML content goes into the build/ folder, then GitHub pages serves the static content from that folder.

The command to build the site is make html - running locally is a good way to work through any errors and fix them.

Sometimes, Sphinx warns about things that aren't important (in this context), so look for the "Build succeeded" message at the end of the output to know if the site did actually build or not.

GitHub workflow

Next, I added the GitHub action to build the site (here's the link to the full workflow, I'm not pasting the whole thing).

The action needs to have write permission, because it's going to build the HTML and commit it to the branch.

Note: this means that if you push a branch, your local branch will be out of date very soon and you should pull down again before you do more work on this branch.

After building the site, you'll also need to deploy it to GitHub pages.

There are a lot of ready-made actions to tie together to make this work, and the GitHub Pages docs are pretty good (just avoid their wizard, it's a quick start with Jekyll).

Overview of how my project workflow works:

actions/checkout@v4
actions/setup-python@v5 Sets up the Python environment.
astral-sh/setup-uv@v3

Sets up the `uv <https://github.com/astral-sh/uv>`__ package manager that we use in this project.
a step that installs the dependencies and actually builds the site.
actions/configure-pages@v5

Gets GitHub Pages ready.
actions/upload-pages-artifact@v3

Uploads the content that was built in an earlier step as an artifact.
actions/deploy-pages@v4

Actually deploys!

Repository settings

All the settings for GitHub pages are under "Pages" on the left hand navigation of the repository settings area.

Use the "GitHub Actions" option to have everything picked up from the workflow you created (this setting is the default but if you're updating an install that used the old branches pattern, this is is how to change it).

Static sites are here to stay

Well-structured and linked content with metadata serves both human and machine audiences, and static sites help us do that with maximum clarity and efficiency, and minimum hassle. GitHub has a whole set of pages starter sets for you to check out if you prefer Hugo, Astro, or another SSG... what are you waiting for? I'd love to see the links to what you build :)

Just Enough Docs

2025-04-28T18:12:00+01:00

Too many open source projects suffer from inadequate documentation, and that hurts their adoption, their communities, and puts more burden on maintainers. Many people who create open source software don't see see themselves as writers, but today I'm sharing tips for documenting open source projects without it needing to take up a huge amount of time to get something published and ready for (docs) contributions.

It seems obvious, but start with your README file - and if you have an existing file, really look at it from the point of view of a new user. Does it still contain the content from the template repository or code generator that this projected was started from? Feel free to throw it away and start again!

Key facts

What would I use this project for?
What does it not do?
Are there platform requirements?

If you can answer these questions in the first paragraph, you're serving even the most casual visitors in the best possible and "hooking" the people who are in the right place. If you have ascii art, those counter/status badges, a video or anything else - put this paragraph first even if you want to keep those other things.

For bonus points: set the topics and description for your project. These are shown at the top of the page and show up in search results when repositories are listed.

Quick start

For users who are already experienced with the tech stack and concepts covered by your tool, they might not need much help to onboard. Add a section for these impatient individuals with quick installation instructions, and a couple of realistic usage commands that will work when copied/pasted.

Installation

For normal users, take a more stepwise approach to explaining the platform requirements, any dependencies, and how to install the tool. Remember that not every user might use your tech stack, so mentioning which dependency management tool to use, and showing the command(s) for installation, can help. Far more people can use your tool than can write the code for it - but the majority of tools should be usable by any technical person. This section makes that actually true.

Usage

The "help" output of your CLI tool can be a useful thing to include, but by itself it is not documentation.

You should also include one or two sentences for each of a handful of common tasks, and then code examples of how to achieve those tasks. You can probably harvest some of this content from your answered issues list. It serves two purposes: one is the usage documentation that is the expected use, but the examples you pick here also illustrate the main purposes of the tool and help users to understand if this thing meets the needs that brought them to your page.

If you have a lot of functionality or complicated usage, move the details to another page or directory and link to it. Similarly if you start a section like this and it just gets really long over time. It's an excellent problem, put everything in a one-section-per-markdown-file format in a folder called docs/ and link to it. Perhaps you'll publish a docs site one day, perhaps you won't, but the simple approaches serve your technical audience well without causing a lot of extra work.

More key information

If your tool has a lot of configuration, requires templates, or has a lot of integrations available - add a section for those main topics here. Again, the issues list (including closed issues) on an established project will give a good sense of what users get tripped up on and where to start. If the information is already available elsewhere then link to it. The same points about putting markdown files in a docs folder to separate topics will help you here.

Contributing file

If your project allows (or encourages!) contributions, say so. From the README file, link to the CONTRIBUTING file. In there, spell out whether contributions are welcome, if there are any requirements around issues, branch naming, license agreements, or anything else. You might already have this information in the README, in which case move it over.

Tell users how to build/run the project from the source code in the respository, how to run the tests, what formatting rules are used - anything else that will help make a pull request closer to acceptable on the first attempt, because those small things save time and effort for both contributors and maintainers.

Call to action

Can users contact you? Open an issue? Join a group? Help in some way? Support the project? Tell them!

Summary: Just enough docs

Start with a good README, clear getting started instructions, and some key usage points, and you won't go far wrong. Encourage users to open pull requests to update the pages in the project if you answer a question for them, and soon you'll have a documentation culture you can be proud of. Documentation is one of the most valuable and transferable skills for developers too, so it's very much worth your time and effort personally as well as for the project.

Right-Size Your RESTful API: More Flexibility Without GraphQL

2025-03-26T10:31:00+00:00

If you publish a RESTful API and there are situations when you need less (or more!) detail in the API responses from your API - then this post is for you. When I run into this with the teams I advise, the initial problem statement usually arrives more like "We need to move to GraphQL", but this is a solution, not a problem to solve. The problem usually turns out to be either or both of:

the API responses are simply too large for some of the clients or uses cases, users need to select just the fields they want
the API doesn't have enough information in the response payload for this client or use case, users need to choose to include nested data

GraphQL is one way to achieve these things, but this post is about the RESTful way to provide the right level of detail in an API response.

A little REST theory

A quick dive into the formal structure of REST will give the foundation we need to understand the options.

REST stands for REpresentational State Transfer; the idea is that we're presenting a way to transfer representations of resources. There are no rules anywhere that say you must or must not include particular data, or that you can only choose one representation (actually there are a whole load of rules that don't exist about REST, you don't have to expose your entire database structure ... I think this is another blog post so I'll save it for another day!).

We can use this concept of representations to provide a choice of formats to an API consumer.

Representations of resources

Let's get away from the theory and look at an (imaginary) example. It's really an API design choice whether I build my potions inventory API 1 to return something like this:

name: Luckyleaf Elixir
effect: Temporarily enhances good fortune

Or if I define an interface that returns a lot more context:

name: Luckyleaf Elixir
effect: Temporarily enhances good fortune
ingredients: [Four-Leaf Clover, Whispering Willow Sap, Stardust]
duration: 3 hours
brewing_instructions: Steep the clover in whispering willow sap under starlight, sprinkle stardust at dawn, and recite a wish.
side_effects: [Mild Overconfidence, Unexpected Coincidences]
origin: First crafted by the renowned herbalist Fortuna Greenfingers to escape a series of unfortunate events.

Either of these representations might be useful, but in different contexts. RESTful services allow us to offer more than one representation of a resource at the same URL, and picking from a short list of options is simpler than figuring out what data structure you want and how to request that.

To implement this option, either keep it simple with a ?format=verbose parameter to switch between response types, or use an Accept header to allow the user to pick the media type they want (a good example of the Accept header approach is how it is used to offer patch/diff/sha/json responses in the GitHub API commits endpoint ).

GraphQL is not (always) the answer

If you have a RESTful-ish or HTTP API that is returning too many fields, or too few, then consider if you can iterate on the solution that is already in place before replacing it with a different approach entirely. GraphQL is good for plenty of use cases, but I usually try to start by looking at using more representations in REST because:
- plain HTTP APIs are more widely understood by both users and tools
- offering a small list of formats is simpler to understand for users to just pick from the menu, and as a provider it is easier to evolve those formats moving forward
- optimising a small number of known options, such as returning just the basics of an item, or returning it and a bunch of related data in the response, is easier to optimise and crucially for APIs running at scale it can be cached (one of the main benefits of REST!)
- it's less disruptive to extend an existing API than to introduce a new technology and support two APIs

GraphQL is a good option where the users' use cases are truly unknown, or data exploration is a feature. But otherwise I would recommend weighing up the options before setting on a whole API replacement programme because one endpoint is too verbose, or not verbose enough. We have established design practices that can help, and I hope the example in this post gave you some ideas.

[1] In the writers room for this post, meaning me and ChatGPT, I asked for some fun API example ideas, and then asked for an OpenAPI file to illustrate the concept. The result made me laugh, and if it would make you laugh too then you can find the Potions Inventory API on GitHub.

Markdown/Mermaid output for OpenAPI Arazzo

2025-03-12T16:01:00+00:00

API reference documentation changed the way we built integrations, and eventually became part of the driving force for OpenAPI adoption and all the good tooling that flowed from it. As a developer experience specialist, I spend a lot of time thinking about how human users can work with the technical assets in a project. HTML-format API reference documentation does a great job of building that bridge when working on OpenAPI projects, but now I'm using Arazzo and it's a very new standard with not nearly as many tools available for that format yet - so I built one.

ApiTapViz is a (very early stage!) Python CLI utility that helps humans to see what's in an Arazzo file without having to read it raw. It picks out the summary data of the workflows and steps in an Arazzo file, and presents them to the user in either Markdown or Mermaid (diagram) format.

The idea is to make it simple to output a representation for users to quickly see how a flow works, or what's covered in a particular workflow. It doesn't have all the details that the Arazzo file has, but it could be useful to capture the output in internal (or external!) documentation, or as an artefact from a CI build.

Please try the tool with your Arazzo files, let me know how you get on, and get involved with the project on its (CodeBerg, there's a GitHub mirror but only for discoverability) page: https://codeberg.org/lornajane/apitapviz

From HTTP to OpenAPI with Optic

2025-02-17T10:07:00+00:00

I've been using Optic's CLI, an OpenAPI tool that does a bunch of things including diffing OpenAPI descriptions and comparing HTTP traffic with OpenAPI. My use case was an established API that didn't have an OpenAPI file yet - using Optic we could create one as a starting point, and then move to a design-first workflow to make the changes that I was there to help with. For this blog post, I've used the example of https://api.joind.in as an excellent representation of an API still in use, but without an OpenAPI file and not built with code that a code generator would recognise.

The basic premise goes something like this:

Configure Optic to set where your API is, and how to run the API tests.
Configure your API tests (or other API traffic source - I used a collection of curl requests which was enough to begin!) to test against the URL in an environment variable.
Run Optic in capture-and-update mode, pointing to an empty/template OpenAPI file.

At the end of the process, you'll have an OpenAPI file with the operations, parameters and data structures that Optic observed from the API traffic. What you won't have is anything fit for users to use, because there are no examples, descriptions, or any other user-helping content. What you will have is a structure to start adding those things into (and I have some suggestions when we get to that point).

Configure Optic

Give Optic the location of your live or sandbox API, and the command to run the test suite for your API.

Here's how my configuration looks:

capture:
openapi.yaml:

# Run with "optic capture openapi.yaml --update interactive"
server:
url: https://api.joind.in/v2.1/
requests:
run:
command: ./test.sh

"API Tests" is a pretty generous term for what I used! Ideally you'd run your end-to-end or integration tests - but this API doesn't have any of that yet. So I hacked together a series of curl commands to run from a shell script; the main aim is to put some API traffic through the Optic proxy so it can see what's going on. You can definitely do better for your own APIs, but I also secretly love that this was enough to get things moving on this project so I'm sharing it :)

#!/bin/bash

curl -s -o /dev/null $OPTIC_PROXY/users
curl -s -o /dev/null $OPTIC_PROXY/users/18888888881
curl -s -o /dev/null $OPTIC_PROXY/users/11

curl -s -o /dev/null $OPTIC_PROXY/events
curl -s -o /dev/null $OPTIC_PROXY/events?filter=hot
curl -s -o /dev/null $OPTIC_PROXY/events?start=10&resultsperpage=20
curl -s -o /dev/null $OPTIC_PROXY/events/8377
curl -s -o /dev/null $OPTIC_PROXY/events/8387/tracks

The main thing to note is that rather than hitting the actual API endpoints, I'm using an environment variable $OPTIC_PROXY. In fact, the API requests all go to Optic, and Optic passes them on to the actual API, checking the request and response data along the way, then returns the response to us.

Proxy and Capture

Let's go! Run Optic, and give it the name of a file to update with its findings:

optic capture openapi.yaml --update interactive

Tada! You have an OpenAPI file! You can't really do anything with it at this point, but you have a great structure in place.

Crucially, you can re-run and re-update the file at any time, which is a great way to catch API drift situations where code changes have been made and not propogated through to the OpenAPI file. In an ideal world, changes would be design-first, and then implemented promptly. I don't live in the ideal world though, so tools like Optic help a lot.

Now draw the rest of the Owl, I mean OpenAPI

The basic structure of an API description would be incredibly tedious to do by hand, so we've saved ourselves a lot of work. But OpenAPI files should be much more than what you can learn from sniffing traffic. They are the API experience you offer to producers and consumers who have to work with your API, so there's more work to do.

It doesn't all have to be done in one go, but take the time to tick off as many of the following as you can:

Fill in the fields in the info section such as contact and license, and definitely title. This stuff is easy to gloss over but as we move more and more to API discovery by machines or through catalogs, the metadata is more important than ever.
For each Operation, add a summary, a description, and as many response examples as you can think of. Include error responses, these are vitally important for tools such as SDK generators to know how to handle the various structures that might be returned by your API.
For each schema and parameter, add a description, any additional format information you can, and an example. These really help users to know how to use the fields available to them.

It might seem like a lot of work! You can consider hiring specialist API documentation writers to help gather together the information you need and work through building out the detail of the OpenAPI description if you don't have the expertise or bandwidth in house. Tech writers understand users and APIs, and can help keep everyone aligned.

An LLM Helping Hand

One thing to note here if you're trying to construct an OpenAPI file is that since everyone publishes their OpenAPI files and they're very formulaic, an LLM can help to speed up the process. Please approach this part of the post with caution: these tools don't "know" anything and will easily mislead you if you don't know either the API or OpenAPI itself very well. The tools are also non-deterministic, so what works one day might do something else completely next time. However I do use LLMs with OpenAPI files when the OpenAPI file is under source control and changes are made one at a time, under my supervision, and reviewed thoroughly by experts.

How did I use an LLM in my own project? Actually the Joind.in API had some hand-written markdown documentation with some good explanations of the various endpoints and parameters. Using the OpenAPI from Optic, and a generated PDF of the API documentation, and telling it not to use any other inputs at all, I did get a reasonable first attempt at an OpenAPI document out of the end of the process! As with all these projects, there's a lot of human work needed (tip: tell your LLM "now suggest and make improvements" a few times, why this works I am unclear but it does! See also my earlier remarks about having it in source control and using small steps, I find whole sections went missing sometimes) before you'd want to give the result to a user, but the machines have done the boring/repetitive parts for us, and now the humans bring their critical thinking skills as expert reviewers.

As more and more organisations embark on API modernisation projects, keeping their tried-and-tested APIs online and updating the processes around them, I expect to see more tools like this in use. If you do a project like this yourself, I'd love to hear how it went and what you'd add to the advice I shared here.

Preview OpenAPI as HTML using Scalar

2025-02-14T14:49:00+00:00

The API DevTools space is alive and well with lots of new and exciting products popping up all the time. I've been especially impressed by the new tools in the docs space, but some of the options are less practical for use as quick human-friendly OpenAPI renderings or previews to use during API development. My current favourite in this space is Scalar because I can get an HTML file to easily include in a build or share with others. It's a bit of a hack though, so I thought I'd share...

What I actually want here is a command that accepts an OpenAPI as an input and outputs a single HTML file as an output. Many of the modern docs tools think they need to start a local server and perhaps that's useful for some use cases but it's not a good developer experience for rendering a preview alongside some API design changes, for example. Other tools expect you to publish your OpenAPI file somewhere publicly, which is always a slow feedback loop and is sometimes not appropriate.

I found that Scalar supports adding OpenAPI into the HTML source and so I made a shell script that does the following:

paste everything before where the HTML goes
paste the OpenAPI file
paste the rest of the HTML content
save the whole lot as a file

Here's the script:

#!/bin/bash

cat template_before.html openapi.yaml template_after.html > output.html

Have fun with your instant-but-hacky HTML preview of OpenAPI files, and thanks Scalar for making it easy!

You can add in configuration (the solarized theme looks SO GOOD in dark mode!) to suit your needs and preferences, and since we're just using cat to put the files together, it's FAST.

Enjoy your OpenAPI-to-HTML build process, and thank you to Scalar for making a great tool.

Presenting with pdfpc

2025-02-11T12:08:00+00:00

My slide deck tool ( rst2pdf ) produces PDFs, and I use pdfpc to present the PDF slides. It shows the current and next slides, my notes, a timer, and it probably does other things too that I don't use! I've used it for years but it was really designed for "in real world" presenting with one or two screens. Recently I discovered it also supports some great options for remote presentations, but I had to look up how to do that so here are my notes (mostly in case I need to look it up 1 minute before I go on stage again!!).

The default modes:
f
* with only your laptop screen, pdfpc gives a full-screen presentation mode.
* with two screens, it assumes you and a projector, with the slides presented on the second screen and the speaker view on the laptop. This is ideal for being at a conference with a computer connected to the projector.

To present with a projector but with the screens mirrored (so you can demo):

use pdfpc -s -S in combination - you can't get the speaker view (because the audience would see it too), but now we can correctly display the presentation only.

There's a new (or I didn't notice it before) windowed mode which is great for streaming, or those fun situations where you are in a room but you connect to a digital meeting room for broadcast instead of to a monitor!

by default, only the presenter mode is windowed
use pdfpc -w both to put both presentation and presenter options into windowed modes - this way you can share the presentation window with your meeting and then move the presenter view around so you can see and present from that.

As a final note, please consider installing a PDF presenter tool (either this one or something else). Every slide deck program can export slides with embedded fonts and speaker notes to PDF, and if someone has a laptop with PDF presentation tools on it then the speaker can still go ahead and do their thing.

Querying the GitHub GraphQL API

2025-01-27T12:48:00+00:00

In a recent project around open source contributors, I wanted to take a look at which projects a particular user (actually a few of them, but I wrote a wrapper to repeat the process for each handle) maintains. GitHub doesn't show this maintainer relationship, so instead I used the v4 GraphQL API and looked at pull request comments on repositories that the user has access to. I'm sharing my query and the Python script I used to make the API calls to.

For the very impatient, here's a gist with the full script.

GraphQL Query

If you're new to GraphQL, the basic idea is that you can fetch data objects, and then use relationships to fetch data related to it. The Introduction to GraphQL from GitHub is a good place to look if you want to learn more.

In my example, I started by fetching a user by their login (their username or handle). Then I used the handy ContributionsCollection field to get information about the user's activity - this is one of my favourite data features of the GitHub v4 GraphQL API and I have used it to build a few different dashboards because it has some cool data in it that would otherwise be laborious to extract from GitHub (although I do really like their v3 REST API too!).

From ContributionsCollection, I can pick the [PullRequestReviewContributions]( as that's all I'm interested in here: I'm looking for review comment activity on repositories where the user has push access. Note that there's no way to tell if the user has previously had push access - for example my own account doesn't show any of the Redocly repositories despite that being a big part of my open source activity in recent years, because I don't have maintainer rights now that I don't work there any more.

Finally I also collect information about the repository that the pull request is in, because my end goal is to collect a list of projects.

Here's the query:

query ($login: String!) {
user(login: $login) {
login
contributionsCollection {
pullRequestReviewContributions(last: 100) {
nodes {
pullRequestReview {
authorCanPushToRepository
}
pullRequest {
repository {
name
owner {
login
}
}
}
}
}
}
}
}

This query uses a $login variable, which is supplied separately when we run the query.

There's a handy explorer tool, but I found it much easier to quickly write a Python wrapper script so I could edit the query and re-run the script using my local development tools. YMMV, but the graphical web-based interfaces don't play that well with my accessibility tools so it was quite literally a painful process for me personally!

Python Script

Why Python? Why not? I don't write Python every day but I wish I did and it's a decent choice for something like this to fetch some data and manipulate it.

I'm using the script as a CLI entry point, including passing in a variable, so the first thing is to grab the input, or prompt the user if there isn't any. That section looks like this:

if len(sys.argv) != 2:
 print("Usage: python3 github-api.py name")
 sys.exit(1)
 name = sys.argv[1]

# variables for query
variables = {
 "login": name
}

I don't need to pass in any other variables, so the variables is ready to pass into the API call when we make it. I also define a query variable that contains the query shown in the first section of the blog post (so I'm not pasting it again here!).

I've got my GitHub access token in an environment variable named GH_KEY, mostly so that I don't accidentally paste when I make the gist!! Using an environment variable is good practice, each user maintains their own and there's no risk of checking something into the repository that shouldn't be there. Remember if you use a tool like dotenv to add those files to your .gitignore file to avoid mishaps.

Calling the API itself is a bit of an anticlimax, here's the Python code:

url = 'https://api.github.com/graphql'
headers = {'Authorization': 'Bearer ' + os.environ['GH_KEY']}
response = requests.post(url, json={'query': query, 'variables': variables}, headers=headers)

Here I'm setting the URL in its own variable: this is super useful in case you ever want to send the request somewhere else, through a debugging or testing tool or something. I'm setting the header to authenticate myself - this step isn't really needed here as I'm only accessing publicly-available data, but if you would need to log in to see something then you need to authenticate. It also gives higher rate limits, which didn't matter for this project but might be useful for you to know.

Finally, the rest of the script checks the data, picks out the projects the user has been working on, puts them in a list, and finally just prints some output.

# be careful, errors are response status 200
if response.status_code == 200:
  data = response.json()
if 'errors' in data:
  print("Errors:")
  print(data['errors'])

elif 'data' in data and 'user' in data['data'] and data['data']['user'] != None:
  login = data['data']['user']['login']
  projects = []

for pr_review in data['data']['user']['contributionsCollection']['pullRequestReviewContributions']['nodes']:
  pr = pr_review['pullRequest']
  repo = pr['repository']['name']
  owner = pr['repository']['owner']['login']

  can_push = pr_review['pullRequestReview']['authorCanPushToRepository']
  project_string = f"{owner}/{repo}"

  if can_push and (project_string not in projects):
    projects.append(project_string)

  print(f"{login} | {projects}")

else:
  print("Error: User data not found in the response.")

else:
  print(f"Error: Request failed with status code {response.status_code}")

The main "gotcha" is that the request can be successful with status 200 but if you did something wrong in your query, there can still be errors in the response body. I'm sure there are other error cases that this script doesn't handle but the only one I'm aware of is that it got VERY confused when someone had deleted their account that I was trying to fetch data about! I also didn't implement pagination because I didn't need it, but that might be a good consideration if you're doing something with a GraphQL API.

REST vs GraphQL

The naming of the REST API as v3 and the GraphQL API as v4 is a bit misleading in my opinion since both have their own strengths and I can't imagine GitHub deprecating their API when it's so widely used! However it is always worth checking what's available in each one, as some things are only in one API or the other; discussions are GraphQL only (or they were last time I implemented something that needed them), so take care.

Good luck with your GraphQL projects, drop me a comment and let me know what you build!

Save edits to OpenAPI as an Overlay

2025-01-09T12:18:00+00:00

For teams that generate OpenAPI from their codebase, there's a tough choice between maintaining rich and extensive content such as Markdown descriptions and examples in codebase annotations, or in making those changes to the generated file - and then losing them when the code changes and the file is regenerated. The new OpenAPI Overlay Specification defines a format for storing updates to an OpenAPI document, and there's a new generation of tools to make it easy to do, so let's take a look.

About Overlays

An Overlay is a set of amendments to be made to an OpenAPI description. It is an ordered series of steps, where each step can add, update or remove elements from the OpenAPI description.

An Overlay is applied to an OpenAPI description - and the output is an (improved) OpenAPI description. OpenAPI is always the interchange format between the tools that you use in your API description pipeline, and so you can add an Overlay, or many of them, at any point in the process.

Common examples might be to add descriptions to a generated OpenAPI file that doesn't have much user-friendly content in it but describes the API and data structures, to remove private endpoints before publishing documentation, or to add extra fields for your SDK generator or API gateway tools.

Edit an OpenAPI and save the edits to re-use

Step 1: edit the OpenAPI file! Make it awesome, add links to descriptions and fill in multiple response examples without using either "foo" or "bar". Generally, go wild :) If you would like something to test with, I'm using the Museum API.

Step 2: use the Speakeasy CLI to generate the overlay. The command looks something like this:

speakeasy overlay compare --before museum.yaml --after museum-updated.yaml --out museum-overlay.yaml

This automated way to create overlays is super convenient, because it means you can use any OpenAPI tool to make changes, save the new OpenAPI, and then get an overlay. Try editing with a live OpenAPI docs preview in your IDE or running locally if you are working on documentation and want to make quick changes and save them!

In my example, I changed the server URL to be a localhost entry because I'm running some local tests against an API, here's the overlay that was generated:

overlay: 1.0.0
info:
title: Overlay museum.yaml => museum-updated.yaml
version: 0.0.0
actions:
- target: $["servers"][0]["url"]
update: http://localhost:8081

Once you make a few edits, you'll get a longer file, but the point I am trying to illustrate is that it pretty tightly mirrors your OpenAPI file. You can edit this overlay file as you wish, or apply the edits, make more edits, and generate a new Overlay - it's absolutely safe to work with and is no more mysterious than the OpenAPI format.

Step 3: Add the new overlay file into your repository alongside the OpenAPI file, and add a step to your CI to apply it before the docs are generated (or otherwise as appropriate). I'd recommend running your API linting after applying the changes, to make sure everything is still as expected.

Tips for Overlays

Overlays make sense in multiple places in the API workflow, and I usually characterise them into two rough groups:

input overlays make your OpenAPI input file ready to be used by the tools in your pipeline. These are steps that add missing error entries, add missing descriptions and licenses, and otherwise patch things up to the standard you want them.
output overlays prepare an OpenAPI file for a specific destination or tool. For example adding some metadata for your SDK generator to know how to group operations and name the functionality. Or adding code samples or formatting information like x-logo before generating documentation.

At each stage you can have one Overlay or many, and apply them as you wish. The same Overlay might be used for multiple API descriptions if it's generic enough (such as adding a license or contact information, or setting some tags that are common for all your APIs).

Edit: between writing this post and publishing it, Speakeasy made a very nice Overlay editor tool!! Go to https://overlay.speakeasy.com and try pasting your OpenAPI in the left hand pane. Remove everything in the right hand pane (to reset the overlay) and then start editing in the middle to get the Overlay generated for your edits as you go along. It uses the same tools as the compare command in this post, so feel free to mix and match these tools while you get familiar with Overlays!

In this post, you saw how you can improve an OpenAPI and then keep those improvements when the API changes and the file gets updated. This approach can help you to get the best out of all the OpenAPI tools you use, and I hope it's helpful! Please leave a comment and let me know how you use Overlays in your own workflow.

Run GitHub Actions on Subdirectories

2025-01-07T13:39:00+00:00

I come across a lot of "greedy" GitHub Actions, where automation is running across a whole project instead of only on the parts that are relevant. Examples might be code linters that report problems with documentation folders, or the inverse of that. It's especially problematic in monorepos where we probably want to use the same tool when we're doing the same task for different subfolders, but that tool might not make sense to run everywhere.

For GitHub Actions (other platforms have equivalents), there are `paths and paths-ignore parameters <https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#example-including-paths>`__ you can add to the pull-request and push configuration. For example, here's the first few lines of a GitHub action that only fires on a push event to one of a specific set of paths:

name: check-links

on:
push:
paths:
- 'project/docs/**'
- 'another-project/docs/**'

In terms of functionality, there's no harm in using the common pattern of having a job that runs, checks if any of the changes need it to do anything, and exiting if not.

In terms of both usability and performance, however, I greatly prefer this approach of filtering with paths. If a change doesn't match a path, the job doesn't need to wait for resources to run and then realise it doesn't need to run. And the humans don't need to scroll past 10 inapplicable jobs that woke up, did nothing, and exited to find the one that related to the project they're actually working on.

Take some time to check the automation on the repositories you work on and if there are exemption checks within the jobs, adjust them so that they don't run at all if the changes don't require it - I can't share the performance improvements I gained on a recent project, but I would love to hear your anecdotes in the comments if you have them :)

OpenAPI Overlays to avoid API oversharing

2024-12-12T11:37:00+00:00

Most APIs aren't perfect. By design they are long-lived and may serve many audiences. I see a lot of organisations maintaining multiple versions of APIs, or simply not documenting some of the endpoints to avoid them being included in the published documentation and confusing users. The problem of API redaction is a real one in most organisations, and it goes unsolved or is solved in a way that causes ongoing friction. I've helped a few API projects to solve this problem with the new OpenAPI Overlay specification so I thought I'd share!

My starting point is always to prefer a "full fat" OpenAPI description, a single moment where all the API surface is available in one place, even if it means building out some API description pipeline or combining files to get there. The big advantage is being able to lint, transform and review changes within the wider context.

From this point, start to filter out the parts that shouldn't be exposed externally. For example, perhaps you have a legacy endpoint that the API gateway needs to honour for historical reasons, but nobody should really know that it is there apart from that! Using the deprecated field on those endpoints makes it easier to filter out.

For example, here's a very short example API description with a single deprecated endpoint:

openapi: "3.1.0"
info:
  version: 1.0.0
  title: Imaginary town
paths:
  '/locations':
    get:
      summary: All locations
      operationId: locationList
      deprecated: true
      responses:
        '200':
          description: Returns all locations
            content:
              application/json:

To remove the operations with deprecated: true, I use an overlay like this:

overlay: 1.0.0
info:
  title: Remove deprecated operations
  version: 1.0.0
actions:
- target: '$..paths..[?(@.deprecated==true)]'
  remove: true

Run it with your chosen overlay tool (I'm using Bump CLI so bump overlay openapi.yaml overlay.yaml > output.yaml works) and you are all set! The OpenAPI description is safe to share.

It's not just the deprecated endpoints that you might not want to share. I also commonly see x-experimental for features that aren't ready for general adoption yet, or other teams use tags to indicate the intended audience such as only pro users, or only partners.

What I like about gathering an "everything" OpenAPI version and then removing it is that it makes easier API governance and maintenance because you're seeing the whole picture. But that you can publish API descriptions to different audiences or different tooling with as much redacted as you like! Generate one set of API documentation to share with your internal teams or early adopters, then use an Overlay to remove the experimental endpoints before publishing to your public developer portal.

The key points here are: API redaction is better than having informal documentation or multiple API descriptions, and Overlays make this easy. Your OpenAPI is not a one-time document, you'll use it as a pipeline, applying many Overlays or other transforms during the build process.

What's your favourite thing to do with an Overlay? Show me an example if you have one!

How to structure and share code examples

2024-11-18T09:28:00+00:00

Writing for technical audiences often means sharing code in one form or another. If the code is only for illustration purposes, then it can be included in the article for observation. But if the code is useful or reusable by your readers, then you need a way to make it available to them. I've seen a few questions on this recently (shout out to the DevRel Collective slack for a brilliant discussion AND nagging me to write it up as a post!), so I'm sharing my experiences on how to publish and maintain code in sane ways!

To help you think more about what you're trying to do, there are some criteria that might help.

Single file or simple code

If the example you need to share is in just one file (or a few files, for an advanced audience), consider sticking to the code-as-exhibit approach and embed it in the article with clear directions for where to copy and paste it to. Doing it this way makes it super simple for both publishers and users, and is very easy to understand and maintain. Since the code quickly ends up in the user's hands, it's also a good choice for code samples that require editing to work, such as "add your API key on line 3" rather than trying to walk through setting an environment variable so that a cloned code sample can run unchanged.

Pro-tip: Make sure the destination publishing platform has an easy "copy" button on code samples.

Use one example from a selection

If you create content for developers, you quickly amass a LOT of shareable code. How much of that code does each user need?

If a user might only need one example, or there are a lot of files that are themselves a project, then one-repo-per-example is a good model. For example, creating template repositories for quickstarts that can be used to scaffold applications.

The downside of having lots of example repos is that now you have lots of example repos to monitor, maintain, and look after! I've had good success using some of the GitHub features (adjust to fit your preferred source control platform) to help manage things:

Add a codeowners file to every repository so that pull requests always have an owner and show up on someone's review requests list! Make this a team name, so you can add more people into the team or easily update who should get these notifications if you have churn.
Automatically set new issues on these repositories to be added to a GitHub project so you can track the maintenance of the projects all in one place. Enable dependabot or an equivalent as well, and then you can tackle updates in batches as needed. You can automatically add the pull requests to a board as well.

Lots of examples, using lots of files

When each code example spans multiple files, and users might want to use a few of them, or dip into them over time - that's a good prompt to build up a library of examples in one repository. Use a clear structure, such as one folder per product with subfolders for each example, so that it's easy to see what goes where, and to add new things in sensible places.

How far you go with making the examples instantly executable in this context, vs just making them available, is really up to you. Two rather divergent examples I've worked on are the Vonage code snippets and the Redocly CLI cookbook. The Vonage code snippets are executable, have automated testing for different versions of the technology they are for, and are included into the main developer documentation. This works really well for snippets that should be ready to use and to just drop into projects that already use the Vonage SDKs to perform simple tasks. The Redocly CLI cookbook is almost the opposite: the main README lists a lot of different examples, and each subfolder has a detailed README for the example and all the files needed to make it work. Users need to copy, paste and almost always edit parts of the provided example, and slot them in to their own project structure. Since each project is different, providing reusable source made sense there.

Code example upgrades

Whichever approach you choose, there are a few things that make every example better:

Use meaningful values: swap foo and bar for product and basket, or illustrate attributes with colour, weight and horsepower. Good examples really are worth a thousand words!
Words always help: Code examples are not self-explanatory. Introduce every example with what it does, and why you'd like to do that. Follow every example with what just happened and how to observe that is, in fact, what happened.
Continue to care: if you publish to GitHub, subscribe to the issues and pull requests of where you've published it. If you publish to a developer portal, make sure you have access to the feedback for those pages. Let users tell you if something gets outdated or needs changing.
There is no such thing as too many examples: Make as many examples as you can with the people/skills/time/energy you have available! People who write code will find the bits they need an mix-and-match to solve their own problems if you give them enough raw material to work with.

Pretty-print JSON with jq

2024-11-01T17:20:00+00:00

Wrangling some document conversion the other day, I ended up in a situation where I had the JSON I needed, but in a completely unreadable format. Luckily, this problem is very easily fixable .... when you know how. So today's post is a quick recap on how I did that using jq, a very handy command-line tool for working with JSON. For the impatient, here's the command:

cat posts.json | jq "." > better.json

In this post we'll look at the data I started with and what the different bits of the command to do help.

My data looked like this (actually I was working with an OpenAPI file, which was huge, so this is some other JSON data that will be easier to show the example with):

[{"title":"Open Standards for APIs","link":"https://beta.lornajane.net/posts/2024/open-standards-for-apis","date":"2024-10-09"},{"title":"Lint APIs with Redocly CLI","link":"https://beta.lornajane.net/posts/2024/lint-apis-with-redocly-cli","date":"2024-08-05"},{"title":"API Description Pipelines","link":"https://beta.lornajane.net/posts/2024/api-description-pipelines","date":"2024-06-18"},{"title":"Checking Links in Docs-As-Code Projects","link":"https://beta.lornajane.net/posts/2024/checking-links-in-docs-as-code-projects","date":"2024-04-27"}]

That's ... not very human friendly! I put this mess into a file called posts.json so I could more easily work with it.

Then I used the command that you saw above. Here is a quick recap of what each part does:

cat is a command to print the contents of the file to stdout. It's a good way to get content into the start of a chain of commands.
The | operator sends the output of the cat command into the next thing in the command.
jq is a JSON tool, and the "." part is technically a filter - but it includes everything, so it's more like a not-filter. The output of jq is the same JSON that we put in, but it's pretty-printed.
If we stopped at this point, we'd get pretty, human-readable JSON content to look at with colour highlights in the terminal. I use this mode a lot when I'm debugging APIs. But in this case, I wanted the cute output in a file.
> sends the output of stdout to a destination that isn't the terminal, and in this case I made it a file better.json.

The data in the new file looks a lot better:

[
  {
    "title": "Open Standards for APIs",
    "link": "https://beta.lornajane.net/posts/2024/open-standards-for-apis",
    "date": "2024-10-09"
  },
  {
    "title": "Lint APIs with Redocly CLI",
    "link": "https://beta.lornajane.net/posts/2024/lint-apis-with-redocly-cli",
    "date": "2024-08-05"
  },
  {
    "title": "API Description Pipelines",
    "link": "https://beta.lornajane.net/posts/2024/api-description-pipelines",
    "date": "2024-06-18"
  },
  {
    "title": "Checking Links in Docs-As-Code Projects",
    "link": "https://beta.lornajane.net/posts/2024/checking-links-in-docs-as-code-projects",
    "date": "2024-04-27"
  }
]

Tools like jq are great to have to hand. What are the tools you can't manage without? I'd love to hear some suggestions!

Open Standards for APIs

2024-10-09T08:21:00+01:00

I work or have worked with APIs of all sorts of standards, sometimes many standards at once - or none. Each standard exists for a purpose and might be a great fit, or a terrible fit for other purposes. Today's post is a recap of some of the common API standards around and what you need to know about each one.

Why do APIs and open standards matter? APIs are the way that machines exchange information. They are the engines that drive connected software. Without APIs, we'd be waiting for a nightly CSV download or parsing data from PDFs. Using open standards gives us established patterns to follow, so that the data can be easily transferred and understood. The familiar formats make APIs easier for humans, and very very much easier for machines.

Of course, there isn't a single standard (and there's an XKCD for that), so let's take a quick look at some of the likely suspects.

OpenAPI

OpenAPI might be the most widely adopted and best known (even if people still think it is called "Swagger" - it isn't) API description formats around. It's an open format; different people from different organisations contribute to and maintain the standard, and all the changes are made through public GitHub activity and open meetings. Disclaimer: I'm one of the maintainers of the OpenAPI specification.

OpenAPI descriptions are written in JSON or YAML format, and they seek to describe the endpoints, requests and responses of an API. This format aims to compromise between API producers, and the consumers of those APIs. In particular it includes user-facing information such as rich text description fields, and supports the inclusion of examples as well. Some organisations generate OpenAPI from their source code, in which case the text content and examples are usually fairly minimal.

OpenAPI is brilliant for RESTful(ish) APIs, and may be useful for other HTTP formats.

AsyncAPI

AsyncAPI is very much from the same school of thought as OpenAPI, but it mostly describes event-driven APIs rather than HTTP ones. Event-driven APIs can use many, many differently protocols such as Apache Kafka, MQTT, websockets, AMQP/RabbitMQ, SQS and many more things including some I hadn't heard of before!

AsyncAPI is therefore more complex than OpenAPI, and any applications that use it tend to have more AsyncAPI specifications in the mix since each producer, broker, and consumer may publish its own API description to describe how to interact with it. AsyncAPI is mostly used by machines and enables us to smoothly transfer many many messages around smoothly and without losing our minds.

AsyncAPI is great for any event-driven applications that use protocols other than HTTP.

gRPC Service Definition

RPC stands for "Remote Procedure Call" and gRPC is a standard service definition format. It's very code-driven, with established ways to declare the service itself and the data payloads that can be used with the various functions. The open standard means that it can be used in multiple technology stacks, and gives us a good option for using open standards with RPC-style APIs.

gRPC is pretty lightweight and it is commonly found in microservices in organisations where REST didn't catch on.

SOAP and WSDL

I always refer to WSDL as the original API description language, it and SOAP have been around for a long time and pre-date most of the other entries in this list. SOAP definitely isn't the new shiny thing, but it's still in active use in many organisations and it has earned its place.

Generally seen as a fairly bulky API format, SOAP services are usually XML-based RPC services. Use of XML gives great support for strict and structured data (in contrast to JSON as used in some modern API formats) and this is one reason for SOAP's staying power. Inertia also counts for a lot, especially in the more established enterprises where these tech stacks are most likely to be spotted.

GraphQL

GraphQL was the new hotness a few years ago but it's not a replacement for all APIs; merely a great solution for some specific use cases. GraphQL is a web-enabled data fetching interface where the client/user defines the fields that should be returned, and can also add related data to make for unexpected and complex queries. For offering a flexible reporting interface, GraphQL is a tool of beauty and satisfaction for both producer and consumer. For replacing a CRUD (create/read/update/delete) API, GraphQL is ..... not that.

The hype is fading, but GraphQL is not. The tools are maturing and awareness and understanding are improving.

Standards matter

Open standards enable us to build applications that talk to one another and achieve more than we could do alone. Choosing and adopting standards in your own work does matter, and being aware of what is out there is very important for tech professionals of all stripes. Hopefully this rather short and irreverent roundup gave you something new or useful (or both!).

Lint APIs with Redocly CLI

2024-08-05T07:58:00+01:00

API governance needs good tooling, and rules that fit - so today's post covers both.

Fun fact: I work at Redocly and had used the tool for years, but recently realised that I have a post about a different tool in the archives of my blog, but no Redocly post. So here's the Lorna-recommended version of OpenAPI linting with Redocly CLI.

Linting won't make your API secure or necessarily delightful, but it can pick out some key aspects and check them programmatically. Typically you'd include linting in your CI (Continuous Integration) setup so that you get the checks run all the way through the API development lifecycle. It's also key to be able to run the tools locally for immediate checks when you're working on your API description.

You can read the docs for Redocly CLI for the details but in summary: an open source CLI tool that helps with every aspect of an OpenAPI description, including configurable linting. Install the tool and lint your API description with the following command:

redocly lint openapi.yaml

If you get a lot of output, DON'T PANIC.

All the linting tools come with default or "recommended" rulesets, but they aren't a perfect fit for every situation so they're not always a great default. My biggest advice around API linting is that every team needs to use a custom ruleset to meet its own needs and context. The recommended is a good starting point when you're totally new to these tools, but they're a one-size-fits-nobody solution, and you'll want to adapt.

Redocly CLI is configured by redocly.yaml, and again, the docs are excellent (yes, I am biased!), but to get you quickly started with a reduced but still relevant ruleset, try this configuration as a starting point:

rules:
spec: error
no-identical-paths: warn
no-server-trailing-slash: warn
no-unresolved-refs: error
operation-operationId-unique: error
operation-operationId-url-safe: warn
spec-components-invalid-map-name: warn
no-invalid-media-type-examples: warn
no-example-value-and-externalValue: error
no-unused-components: warn
operation-operationId: warn

You still need to edit your own configuration and add/remove things that make sense for your use case, but especially if you're retro-fitting API governance to an existing API or APIs, the recommended ruleset can be a bit much!

Beyond the provided rules

In addition to a customised ruleset, I'd expect most teams to need some configurable rules and even some transformation steps of their own. I've written before about OpenAPI Overlays, and Redocly CLI also has some nice custom plugin functionality that makes it easy to add resusable rules, sorting operations, or a variety of other features.

Here's an example for checking that the API description has an info.version field that uses semantic versioning format:

rule/version-is-semver:
subject:
type: Info
property: version
assertions:
pattern: /^[0-9]+\.[0-9]/
severity: error
message: info.version must use semantic versioning.

Creating these custom rules is a great way to pick up on the naming rules and other conventions that you want to have done right for every change to every API.

For more rule and plugin examples, Redocly CLI has a cookbook where you can find lots of examples, and share your secrets with others too - so if you have rules that work for you, I'd love to see them!

API Description Pipelines

2024-06-18T21:54:00+01:00

Working on API tools, I get to see inside lots of different organisations' API projects and processes. Every scenario is different, but a common theme is that many companies use a more complicated API description workflow than you see in conference slide decks! This article shares my typical workflow, steps and chosen tools that might show up in an API description pipeline.

At a high level, the process looks something like this:

(sorry, it's really tall, but really wide didn't work well!)) You can get the mermaidjs source for this diagram and you are welcome to edit/reuse but please credit me.

In this post, I'll dive into these steps and some tools that might be helpful.

Gather the inputs

If you're publishing one API that draws on multiple sources, such as a set of microservices, it helps to combine those OpenAPI descriptions into one unified OpenAPI description. This is different to the bundling/dereferencing step that we'll come to later.

If you're publishing multiple APIs, then you'll run all the steps here for each of the APIs. If you're publishing one API (in various forms) to multiple sources, we'll fan out to the different destinations part way through the pipeline.

Normalise: fill gaps and fix problems

Wave the magic wand and improve the OpenAPI to add in everything that you'll need in later stages. Typically this could include:

finishing the developer experience by adding summaries, descriptions (for operations, tags, parameters, schemas ... everything) and examples.
adding metadata such as servers, info descriptions, licenses, docs platform links, and other items that either weren't included or which need adding to the newly-assembled API description.
fixing things that the generator did wrong (generators make invalid output more often than I realised until I started working a lot with this stuff).

OpenAPI is much richer than any of the possible information sources, or any individual output. For design-first organisations, all the extra goodness is there by design, but for teams working with API descriptions generated from code, or otherwise find themselves working with wish-this-was-better API descriptions, using tools to programmatically patch improvements is a winning game plan. It's important to have a repeatable way of making these changes, since the OpenAPI inputs will update and it's likely that will happen often.

Tools to try: OpenAPI Overlays or Redocly decorators.

Traditional CI checks

Every API needs this step as a minimum! Check that the OpenAPI description is valid, and optionally apply some formatting rules to check that quotes, line wrapping, indentation and that sort of thing are all as you would want them. You could even lint the Markdown in your OpenAPI description or sort entries into alphabetical order.

API governance rules are applied at this point. Governance and linting are a whole separate post (and I write a lot about this in my day job for Redocly since they make excellent tools for these things) so I'm restraining myself from that particular tangent. TL;DR make sure the API is up to scratch before you proceed. If it is not, then stop and try again.

Tools to try: Redocly lint or Spectral.

Fan out from here

This is "half time" in the game of API description pipelines and we have a complete OpenAPI. All the inputs are in, counted, tidied up and verified. Now we can use the API description to feed into all the outputs, adapting a copy of the description for each destination

Filter to fit the use case

Your work-in-progress APIs may include more information than you necessarily want to publish to all users of an API, so the next stage is to weed out things that shouldn't be included. It's common to run this step and all following steps in parallel, once for each deliverable thing or destination of an API description.

For example, you might remove all the endpoints with deprecated: true, or create an API description that only includes the endpoints tagged "partner". This step is important otherwise you can find yourself maintaining a lot of different API descriptions that sort of overlap, or not being able to have API descriptions for things you're actually working on because they can't be published yet.

Tools to try: OpenAPI Overlays or Redocly decorators. And if you end up with tags that have no endpoints, there's a recipe for that.

Prepare for a perfect finish

The API description may be going to a few different destinations, and each one might need some different additions. For example, maybe your documentation renderer doesn't understand multiple tags for one endpoint, so you strip the others at this point. Or the SDK generator you use gives better results if you add some extensions to the OpenAPI description before passing the API description to that tool.

Tools to try: OpenAPI Overlays or Redocly decorators. Yes, again.

Package for delivery

Some of the destination tools in the OpenAPI space (meaning there are no further uses of the OpenAPI description after that tool) such as API Gateways may feature limited capacity to resolve references. Some don't resolve external files and must be supplied a single-file OpenAPI description, in which case you can bundle yours and send that. Others can't follow the OpenAPI $ref syntax at all, so look for a bundler than can "dereference" - meaning it puts everything inline, which can make for very large OpenAPI description files so do take care.

At this stage you may also need to convert back to 3.0 or an even earlier version of OpenAPI, and there are tools available for that too. Just don't let that one tool hold back your whole process!

Tools to try: Redocly bundle.

Designing pipelines

If every API project needed the same tools in the same order, you'd be able to add it with a single click! In fact, every API and organisation are different, and so are their pipelines. I've outlined some common patterns in this article, so start with the "Traditional CI checks" step, and then pick from the other items on the list as you need them. As always, OpenAPI tooling moves quickly and there's some good innovations in this space, so when you're ready to add a new tool to the box it's always worth a visit to https://openapi.tools/ and see what's new or what best fits your requirements.

So, what's missing from my list? Add a comment and let me know!

Reviewdog filter settings with GitHub Actions

2024-03-25T08:10:00+00:00

Reviewdog is a tool to use with GitHub actions for applying review tools in your CI. I use it with Vale, and it's really good. One thing that has tripped me up multiple times is that by default, it only applies the checks to the changes in the pull request, not to the whole project. So when you add something like Vale, which is a prose linter, it only checks ... the workflow file you just added! I always forget that it works this way, and how to change it, so I'm pasting some examples for future reference.

Apply checks to all files

In general, I don't find it helpful to have the review tools reporting errors across a whole project on every build. However when I do want to see GitHub actions running the checks for the whole project, I can never remember the setting. So, future self, note that it is nofilter (the problem is not the option name, it's just that I don't remember things).

At the moment I'm working on a pull request where I'm adding Vale to GitHub actions via Reviewdog. The pull request only changes the workflow file, the Vale config, and the styles/Vocab files that go with Vale, so by default the build goes green and no problems are reported with the existing contents of the docs/ folder. I am pretty sure we didn't write perfect content when we didn't have any tools in place!! In fact if you check the build output, it does find problems in the project, but since they weren't introduced in this pull request, they don't match the filter. You can see the problems listed, but then not reported.

The nofilter setting removes the filter and those errors and warnings get reported to the build. The job configuration looks like this:

vale:
  name: vale action
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - uses: errata-ai/vale-action@reviewdog
      with:
        files: '["README.md", "docs"]'
        filter_mode: nofilter
        fail_on_error: true

Once I'm satisfied that we've introduced the necessary fixes as well as the tool that checks them, I'll switch this back to file before the pull request merges.

Apply checks to changed files

My preferred filter setting for day-to-day use is file. In this mode, the checks run across the whole file for any file that has changed. It works well for something like a prose linter because it means that if something needs updating, we update it everywhere in the file, all at once, while we're changing that file anyway.

Here's the job configuration again:

vale:
  name: vale action
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - uses: errata-ai/vale-action@reviewdog
      with:
        files: '["README.md", "docs"]'
        filter_mode: file
        fail_on_error: true

It can be frustrating if you're changing something tiny and you get a whole file full of annotations - but in general, this approach has worked pretty well for me every time I've remembered what the setting is called! I'd be interested to hear how others manage this sort of thing and if you have any tips to share.

Short tech writing style guide for developers

2024-01-24T09:18:00+00:00

Style guides are vital to successful publishing projects, but they are usually too extensive for casual contributors. After running a number of projects with developers rather than specialist documentarians as the main contributors, I've started using a short-form style guide, short enough to be read and used by people who don't spend as much time thinking about these things as a full time writer does. The main version is published on GitHub, but I'm also sharing the current version here.

Target a single audience and outcome

Know who you are writing for. The new junior developer needs different context and explanation than the Ops person trying to follow disaster recovery documentation. If it helps to think of a specific individual, do that. The key here is to not write down what you know, but write down what the reader needs to reach the desired outcome. An example outcome could be completing a task, learning about a specific topic, or understanding a set of options.

Structure your writing

If your organisation offers templates, always use them; consistency helps readers a lot, even if the content is created by many authors. If not, try the Diátaxis framework as inspiration on how to frame different content types - and don't be afraid to create more than one article from a single topic or assignment. If no structure is provided as a template, create your own by writing the subheadings of the piece before you write the words.

Use subheadings, images or diagrams, lists, and other content types to communicate to your readers. This "page furniture" helps them to navigate potentially long and complex technical articles.

Be positive and inclusive

Focus on things the user can do. Don't explain what can't be done, isn't supported, or hasn't been implmented yet unless these are central to the topic.

Example: Instead of "you cannot back up your data yourself" you could say "data backups can be arranged by contacting the support team".

Inclusive language helps make your writing useful to the widest possible range of people. Avoid gendered language, using "they" and "theirs" rather than "he" or "she". Even when this feels clunky to write, it makes a positive impact on your readers. Other language to find alternatives for includes terms that could seem ableist such as using "OCD" or "crazy" in your writing. It's already standard practice to avoid socially-loaded terminology such as master/slave, blacklist/whitelist, and so on. Keeping a wordlist that is automatically checked, or using a tool like AlexJS can help with spotting words and phrases that can be improved.

Beware of jargon and abbreviations, because they can be difficult for newcomers to understand. Expand your abbreviations the first time you use them, and link to definitions or supporting content for any special terms to give the user the option of reading on , or clicking the link to learn more.

Write commanding titles

Titles should use imperative language to make it clear what the reader is doing in each section. It can feel like an abrupt style, but imagine ordering a small robot to perform each action and you will be pretty close. The side benefits of this approach are that it's often easier to translate, and tends to match user search terms more closely.

Example: Instead of "Configuring the widget" use "Configure the widget"

Use sentence case for titles, and check them automatically using a tool like Vale. Add product names and other proper nouns to Vale to avoid false reports.

Choose examples and screenshots wisely

If a picture is worth a thousand words, then a good example is worth at least twice that amount. In both cases:

Choose content that the user will know or easily understand. Avoid "foo", and instead use "Alice" if it's a name, "Bucket" if it's a product, "paleblue" if it's a colour, and so on. Your reader will internalise that their data should look like your data.
Avoid pop culture references, using these can exclude groups from other demographics than your own.
Every example and screenshot needs text with it, explaining what it shows and why that's useful to the user.

Code examples are always valuable, adding more examples is usually a good thing.

Screenshots are both more difficult to get right and more difficult to maintain over time since small changes to a UI will mean repeating all the screenshots. Use them when you think the user would be likely to struggle with words alone. If you take a series of screenshots, use a small and consistent window size for them all. Always reduce the file size as much as you can, and include alt text on every image (tools such as Markdownlint can check for alt text automatically).

Add hyperlinks

Add links. Link to the tools you mention, the concepts that the user might not know already, and to other articles that are likely to be useful to this reader. Adding links to supporting material can enrich all content types, and help the reader to find their way to success.

Choose words that represent where the link goes to, such as "see the section on hyperlinks". Some tools will offer the links with their link text as a list, so meaningful words or phrases and avoiding duplicate link text within a page is good practice.

Avoid technical documentation missteps

Technical writing has some unwritten norms that exist for mostly good reasons of readability and comprehension. In particular, here are some things you should NOT do:

Use exclamation points. It's not a gossip magazine, it's technical documentation.
Use question marks, especially in titles. It very easily reads like a car advert.
Add emoji, unless the platform you are writing for has it as an established acceptable practice. Emoji are difficult to keep consistent between authors, can be distracting, and can also make the content less accessible.

API Linting Levels

2024-01-11T00:05:00+00:00

I've been thinking about API linting lately, partly because I work on a tool for API linting in my day job, and partly because I get quite a lot of questions from teams wanting to improve their API quality. The "best" ruleset depends entirely on your context, where you are in your API practice journey, and how much you want to invest in the API in question. I typically use a 4-levels model of API linting readiness, and I'm sharing them so that you can find yourself on the map and see where you might go from here.

This post is part of API Futures 2024, a series of posts by different contributors focused on the challenges for this year. My big wish is for everyone to find API tooling and configuration that fits their own context, which is why I'm publishing this linting levels post as part of the event.

Level 0: A valid API description

This sounds basic but a surprising number of public APIs either have no API description, or their OpenAPI description is actually invalid in some way. It may well work with the tools used in the producer organisation, but unless you supply a file that works with all other tools, there are limits to who can use and interact with your API.

Add some unopinionated validation to your CI workflow, and make sure that anyone who works on your API can also run those tools locally while they're making changes - and you can graduate to the next level immediately :) Redocly has a rule for linting against the specification which is a great starting point (other API linting tools are available!). For the next level, there's a great ruleset in the Redocly CLI Cookbook that you can copy and paste into your own projects; it aims to catch the things that match the declared data structure of OpenAPI and yet make absolutely no sense in the real world!

Level 1: Meets basic standards

Every API probably should have its own set of basic standards but I typically ask about the API linting rules that are already in place, such as:
- are there checks that security has been declared on every endpoint?
- is there useful metadata, such as a license and some contact information?
- does the API follow a design, such as RESTful API design, and how clear is that from the OpenAPI description?
- are consistent data formats used, such as cents (or equivalent) for money, ISO format dates, standard scientific units for measures?

Having some sort of API design, with some level of data handling, some supporting metadata, and some security practices in place is a level 1 API. Some APIs only need to be at level 1, such as a service with one or two endpoints that handles something non-critical such as newsletter signups. Sure, this API could be brought to a higher API standard and that could be enforced by tools and linting. But not every API needs the same level of investment or must comply with the same standards. Within an organisation, have a few defined levels, and make clear which APIs meet which standards.

Level 2: Has a consistent interface

There are few things more irritating than an inconsistent API, and yet a deeply consistent API interface is essentially an invisible feature. Users will very quickly feel comfortable and be productive using your API platform, but not being irritating is rarely given as customer feedback! So what are the ingredients for this developer happiness level of API standard? You can think of it as having two main elements: the philosophy, and the mechanics.

The mechanics is the simplest thing to get right. It means using plurals (or not), absolutely everywhere. Always using exactly the same variable name to refer to the same data, wherever it is used, and validating that data exactly the same way in both request and response for every endpoint. Having endpoints that are consistent in their shape and depth, and keeping common parameter patterns for features such as filtering or pagination the same everywhere. Adding linting rules to an existing API for this level of API standard can be quite tough, but add the rules you want to follow, with exceptions for any existing/legacy endpoints that aren't compliant. Enabling the checks on the API, and acknowledging the parts that are known and shouldn't fail the build, means you can automate the checks and add no more violations! The ignores list can also serve as a to-do list for things to work on, either by making backwards-compatible changes or to revisit if a new API version is in the works.

The philosophy part is a bit more difficult to think about but building on prior art in your APIs, for example by following established RESTful API design practices very tightly and adopting other standards that are already known and accepted. This approach means you don't have too many decisions to make since you're following existing and established standards, and the ideas may be familiar/supported by the users/tools that access your API. I particularly recommend looking into Hypertext Application Language (HAL) if you include links in the resources you return from your API endpoints, and RFC 9457: Problem details for APIs (replaces the better-known RFC 7807).

Level 3: Is a joy to use

We don't describe developer experiences as "joy" often enough, but a really great API description definitely qualifies. If your API is your flagship product, or drives a lot of business value that is realised by strong integrations - this is the level you want to aim at. What's funny about these top-level APIs is that their actual API design isn't much different to the earlier levels, however the implementation and supporting materials are the special sauce.

To produce an API at this level, the API governance expands into a more holistic view of the API and what really makes an API experience joyful. This can be achieved with rich OpenAPI descriptions, and supporting documentation as the main pillars. I think there's also an element here of discoverability, and that this aspect will only become more important in the future.

Enriching OpenAPI descriptions can take many forms. Make the most of the description fields; these support Markdown so don't be afraid to add links, lists, or whatever else would help the user. From the very top level, where info.description sets the scene and purpose for the whole API, through the individual endpoint descriptions and down to the individual parameter and schema fields, the descriptions are the magic sauce. Explain not just what the thing is, but also what it does, and why the user would want to do that. Enrich also with meaningful and illustrative examples, using (culturally varied) example data that represents something a user really would be doing, rather than "foo" or "bar". A project name could be "holiday_planning" and an email address "email@example.com", and using values like this does make a more successful and happier end user.

The documentation is not just the reference documentation either. I haven't seen linting rules for this, so perhaps it's a little out of scope, but adding meaningful onboarding documentation or getting started guides, and tutorials for the most common tasks in an API are both excellent additions that make a good API into a great API.

Make your API next-level

Don't be put off by API standards tooling that comes with big opinions of its own. Reflecting on what your API needs, and what your organisation can support, will help you find which level you are on and where you want to be and set your API projects up for success. Tune the API linting rules to get the support you need, but don't be afraid to turn things off that aren't important to you! All the tools come with default rules, but every situation is different so take the time to adjust it for your context rather than gritting your teeth and dealing with someone else's ideals.

Add OpenAI flags to OpenAPI descriptions

2023-11-28T09:48:00+00:00

With OpenAI's new Actions feature allowing you to get a GPT that uses your own API, many new doors are open. However giving an AI an access token the keys to your API and telling it to have fun may lead you to realise that one of the doors available leads to the empty lift shaft of overwritten data, or the bottomless well of cloud bill shock. To reduce the risks, OpenAI supports an extension x-openai-isConsequential that you can add to your OpenAPI description, to indicate which endpoints should not be called without a human confirmation step.

I'm not doing much with OpenAI right now, but I do plenty with OpenAPI and the question "how do I add this field to my existing API description?" is one that I can answer! What's more, you can use the advice in this post to add other extensions or additions to your OpenAPI descriptions using Overlays, this advice isn't OpenAI-specific, but it's used in the examples.

(is anyone else confused by the OpenAI vs OpenAPI similarity yet? I am ....)

OpenAPI Overlays

The OpenAPI Initiative has a group working on a feature called Overlays which solves exactly this problem, where the OpenAPI description exists, but needs some tweaks. Sadly, not very many tools support this yet - the ones I know of are my overlays-js library that I'll use for the examples in this post, and Speakeasy CLI.

Overlay Example

This example shows an Overlay that adds the x-openai-isConsequential field to all POST endpoints:

overlay: 1.0.0
info:
title: Add OpenAPI annotations
version: 1.0.0
actions:
- target: "$.paths.*.post"
update:
x-openai-isConsequential: true

The target field accepts JSONPath, so edit this expression as required to capture the endpoints you want to flag. Note also that you can add as many action steps as you need to, so if it's simpler to add specific targets rather than trying to JSONPath to the right subset, that is an option! (there's a bug with the overlays library that means structured overlays don't work well at this moment, so use targeted overlays).

Use `overlayjs` to update the API description

Check the installation instructions to get the tool installed if you don't have it already.

Run the command and supply the API description as --openapi and the overlay as --overlay. For example, the following command applies overlay.js to openapi.js:

overlayjs --openapi openapi.yaml --overlay overlay.js > new-openapi.yaml

The command writes to stdout, so you can direct the output wherever you would like it to go. In my case, I've directed it to a file called new-openapi.yaml.

Take a look at the updated OpenAPI description, or diff it against the original, and you should see that the post endpoints now have the OpenAI x-openai-isConsequential: true added.

By using OpenAPI Overlays, you get a standard format and a choice of tools to apply it with. You can prepare an OpenAPI file for use with OpenAI, even if you are an API consumer and have no input into the way that the OpenAPI is created - or if you just want to avoid cluttering other stages of your API lifecycle with the metadata specific to the OpenAI use case. Hopefully this helped you get your OpenAPI in shape for OpenAI (yes, still confusing to have such similar words!), I'd be happy to hear how you get on if you want to share in the comments.

When to version bump your OpenAPI description

2023-11-20T13:10:00+00:00

I've heard a variety of bad advice (and some untruths!) regarding how to version your OpenAPI description lately. Rather than dig into those, instead I'll share my opinion on how I like to version API descriptions in my own projects. Every API project is different, but perhaps there's something here that can help your project too.

Version your API Descriptions

Let's start with: please, please do use the version field, and update it from time to time. In OpenAPI, the version of the description is in the info.version field and the official documentation describes it as:

REQUIRED. The version of the OpenAPI document (which is distinct from the OpenAPI Specification version or the API implementation version).

To break this down, the documentation describes what the version field is, and also two things that it is not:

it is the version of the OpenAPI document.
it is not the version of OpenAPI specification being used (this is in the top-level openapi field).
it is not the version of the API that it relates to (many descriptions cover multiple API versions, and may also change without the API changing).

So now we've cleared this up for OpenAPI, and pretty much everything here applies to AsyncAPI as well, let's think about when the version should change.

When to update the API description's version field

TL;DR when the API description changes, update the version field.

The whole point of a version field is so that I can tell if I have the most recent version of your API description, or which version something used. So if the two things aren't the same, then probably their versions shouldn't be the same either.

Some things that should probably cause a version change:

a new endpoint was added to the API, or something was marked as deprecated
the description was updated to include improved summaries, descriptions, tags or examples
the description was restructured

Having a process that publishes a new OpenAPI description with an updated version is something you'll want to be able to do easily and regularly, so make it painless to do.

A note on version formats: OpenAPI only recommends a string. Many organisations do use a SemVer approach, and this is well understood by technical audiences, but choose something that fits you. Take a leaf out of Stripe's book and use dates for versions, for example. Whatever you choose, don't get bogged down in what does or does not require a new release. Make new releases easy and aim for releasing too often rather than too infrequently!

This quick roundup on API versioning was the result of having this conversation several times over the last few weeks/months, with people all over the API ecosystem. We talk a lot about API versioning, but not nearly enough about API description versioning, so I wanted to share some thoughts and most of all encourage everyone to version their API descriptions (at all! With any sort of version!) and to update those version fields from time to time :)

Are there any organisations or APIs you've seen that do this particularly well (or badly, if you can be constructive about why it's not working)? I'd be interested in seeing comments pointing to more examples of what you have spotted in the wild.

API docs previews with Redoc and Netlify

2023-11-08T09:18:00+00:00

I talk to a lot of people about their API lifecycles and processes, and one thing that has value in almost every case is immediate documentation preview for all updates to an OpenAPI description. The time it takes to set it up (about ten minutes now I know what to do) is an investment that makes every future change easier to propose, review, and accept. Here's my recipe for open source OpenAPI documentation using Redoc (disclaimer: I work for Redocly, who publish Redoc) with the files pushed to Netlify for hosting (but it's HTML, so really any hosting will do and you should feel free to adapt the instructions for your preferred platform, and then share them in the comments).

OpenAPI in a GitHub repo

API documentation using OpenAPI assumes a docs-as-code workflow, and so does this post. You should have an OpenAPI description in a GitHub repository, so that when the description changes, the documentation can update. The big win here is that by adding preview builds for all pull requests, incoming changes can easily be shared and reviewed by all stakeholders, without any need to read YAML.

Tip: there's no need to bundle the OpenAPI description to a single file, Redoc will follow references across files as needed so this approach is ideal for the during-development-and-review phase of the process.

Configure the Netlify project

The basic process, for Netlify or any equivalent, is to link the GitHub project to a deployable project so that updates run every time the repository updates. Netlify needs to know how to build the project, which in this case is generating the documentation as HTML, and what to publish. There's a setting for "do this for pull requests as well" which is what makes the whole thing seem like magic!

Get started with the "Add new site" option and choose "Deploy with GitHub". You may need to link your Netlify account to your GitHub account and/or the project you want to use if you haven't used these tools together before or if you grant access on a per-project basis. Once that's done, pick the project to generate API documentation for.

The next screen sets the site configuration. My project has openapi.yaml in the root folder, and my settings are like this:
* The build command is npx @redocly/cli build-docs openapi.yaml -o build/index.html.
* And publish directory is set to build/ since that's where the previous command writes the files to.

Save the settings and the project will (hopefully) deploy successfully - don't look away, it's fast!

I'm not sure when Netlify started enabling preview deploys by default but that's the behaviour in my most recent new site. Test your own setup by making a small but identifiable change to your API description, committing the change to a branch, and opening a pull request. After a few minutes the Netlify bot should comment with a link to the preview of the updated documentation.

All in all, the time investment to get a basic documentation rendering is definitely worth it (and if you need something more then I think I am contractually obliged to suggest you check out Redocly for the rest of the products). Investing in every stage of your API pipeline, from linting, to enrichment, and to docs - delivers rewards. And delivers them every time you make an API description change!

Hopefully you found this useful, I'd love to hear if you set this up yourself, or if you can share equivalent setups for other platforms so that others can learn from your wisdom: add a comment!

Selective prose linting with Vale's glob switch

2023-08-23T11:55:00+01:00

Vale is an open source CLI tool for linting prose (if you're not linting prose yet, why not? I never want to go back). I typically use it for everything, locally and in CI, and it's especially valuable now I'm working for a company that uses US English. You'd think that most of the challenges would be caused by the majority of non-native English language contributors? Nope, most of the chaos comes from the token Brit :)

I ran into one challenge though where I didn't want to enable Vale for everything: a large internal documentation repo. This is the catch-all of things we should share with one another and like most internal company documentation sites, there is a lot going on. I assembled a very minimal set of Vale rules and still the first pass netted me over 10k errors. Luckily, Vale has a --glob option, but it took me a while to find how to exclude multiple files and directories using it, so here's the example for future-me, and anyone else who needs to see it!

Most of Vale's configuration uses a .vale.ini file, but you specify the files and directories to check at the time you run the command, with something like:

vale docs/

In my case, I needed to exclude a mix of folders (there's little value in checking/fixing a meeting notes collection of a past project, and we can't make changes to our legal docs. Also there's a page of good restaurants near the office that is full of proper nouns!). To include just one section or another, I could run Vale multiple times to check each section, but to run everything except some problem areas, I used --glob. I ended up with a command like this:

vale --glob='!{office/where-to-eat.md,legal/*,projects/codename/meetings/*}

Some of the skipped directories we will probably work on improving and then dropping the exception for - but this approach is useful when retrofitting a tool like Vale to an existing repository which can't be made to align all at once. We're also using a pretty reduced ruleset and may incrementally add to this over time.

Glob in GitHub Actions

Probably all the CI integrations have something similar but Vale has a GitHub Action and I could add the whole --glob expression to a vale_flags entry in the with step.

Ideally we'd be running identical commands and configuration both locally and in CI, but since the configuration for this doesn't seem to be able to go in the Vale config file, I compromised on adding it as an npm script as well as a GitHub action. It'll be annoying to maintain but I didn't come up with any better ideas!

Hopefully this post will help me next time I'm wondering how to combine glob expressions for Vale, since I didn't find many other examples. If it helps you do, then that's even better! Share your tips in the comments :)

Visualise OpenAPI with Blockly

2023-06-20T12:57:00+01:00

Giving talks about OpenAPI often means lots and lots of slides of YAML. In fact, one of my talks even contains a "warning, contains yaml" slide! If you aren't a professional yaml developer, it can be difficult to digest and learn from, and so I wanted to do better. The various elements are nested inside one another, and so I had this idea to build a representation of OpenAPI using Blockly (the tooling also used for Scratch, if you've seen that).

The hardest part was (as always!) getting started with an unfamiliar toolchain, knowing what to edit, how to serve the file, and if there was an extra step needed to see changes. The developer documentation for Blockly is very good, and I tried to follow their guidelines as much as I could. I ended up writing a messy web page that had a textarea that I could use to import/export what I'm working on, and writing it all to local browser storage - because every time I changed the toolbox, I lost the data structure I had just built!

It's not a complete library, but it did allow me to create the visuals I wanted for my talk "OpenAPI for Documentarians" at APItheDocs in Amsterdam, and I think it's likely that I'll expand and use this again. I got positive feedback from trusted audience members, and a few people told me they found it "approachable" - which is exactly what I was aiming for.

I've published the whole project on GitHub, and I'd be happy to have forks, contributions, suggestions or feedback on this. There are places where I think there's probably better ways to represent some elements, for example. Also the idea of Blockly is that it's a high level language that can then be compiled to something for the machines when the humans have finished with it. I'm not sure if it's even possible to write the converters for it to output OpenAPI but that would be a really fun project if anyone is up for it :)

If you use it, I'd love to hear about it - drop me a comment!

Tips for better documentation with OpenAPI

2023-06-12T13:30:00+01:00

I love documentation, which is supposed to be rare for software engineers. I am pretty sure that loving documentation is actually very common, but not very cool and so few people admit to it! I love reading good docs, and so I love writing them too. Between hanging out in the Write the Docs community, and recently being at the APItheDocs conference, I've answered a bunch of questions about getting good documentation from an OpenAPI document, so I thought I'd round them up in case it's useful.

Summary vs description

OpenAPI supports both summary and description fields and they're both strings that contain information about the thing they are attached to, such as a parameter, endpoint, schema, or response. So which do you use?

Summary is a plain text field, and should typically be shorter: just one or two sentences. It's often used when listing elements, although different tools render different tools differently, so always check what your tools product and do whatever looks right there.

Description fields support markdown, so this presents an opportunity to link to additional information or documentation. Particularly in the top-level info.description field, this can be an opportunity to write multi-paragraph detailed overview documentation.

Good description fields give context

It's very common to see a description for the user_id field which says something like "the ID of the user". Um, thanks?

Everyone can do better than this - and if you've ever had to read API documentation from someone else, you know how much difference it makes. I advise people to ask themselves WHAT this field/endpoint/whatever does, and WHY it's needed. Think about what the user is doing when they use this endpoint, and give a bit more context. Something like "the ID of the user placing the order. This value can be found in the order detail object or order status update object. Find the current user's ID by going to /user/me`. You get the idea. Context is everything!

Taking pride in API documentation is a Developer Experience multiplier; just make sure that everyone working on your API documentation knows how much you value the extra effort in this area, and that it is something that collaborators should spend time on.

Simplest styleguide

Writing brilliantly is absolutely an art, but writing well enough is attainable for everyone. My advice is always to keep it simple and informative.
* Use short sentences
* Use present tense (the endpoint "does" something, rather than "will do" the thing)
* If you can remove extra phrases, such as "in this case", do so. It keeps things easier to read
* Be kind. Keep things gender-neutral and inclusive, and avoid belittling words such as "simply" and "just"

With these in mind, you won't go far wrong. Engineers can absolutely be brilliant at this without formal training.

Markdown in YAML

Description fields support markdown, which is brilliant, and allows expressive additions if you need them. If you use YAML for OpenAPI documents, there are a couple of additional operators that can make it much easier to work with.

Folded block starts with >. This is great if you just want to wrap your content to make it manageable, or compatible with the linters/formatters you use on the file. I quite like to use a one sentence per line format for markdown, because it makes diffs more readable, and this works nicely with that approach.

description: >
  This is a folded block style in YAML.
  It preserves line breaks but ignores leading
  and trailing white spaces.

Literal block starts with |. Use this approach if you have markdown that uses whitespace, such as titles or bullet points.

description: |
  # Title

  This is some Markdown content.
  It can include multiple paragraphs,
  lists, headers, and other Markdown syntax.

Using the appropriate markdown syntax can make things much easier and maintainable over time.

Link to external documentation from API reference

I don't often see the `externalDocs <https://spec.openapis.org/oas/v3.1.0#external-documentation-object>`__ used but it's a top-level element in OpenAPI that is designed to make the main API documentation (landing pages, quickstarts, tutorials, etc) easily findable from the API reference documentation. Additionally, OpenAPI tags, schemas and operations all also support externalDocs properties, so you can link directly to detailed documentation for any of those elements too. Especially useful when there's a lot of information to include, that can be unwieldy in the description fields.

What are you tips for getting the best documentation from OpenAPI? I'm working full time in this space now (at Redocly, and really enjoying having lots of conversations about docs, APIs, and everything in between!

Raising the Next Generation of DevRel

2023-03-08T20:42:00+00:00

Developer Relations isn't new, but it's also broad, ill-defined, and constantly evolving. It's difficult to keep your own skills up to date, never mind coping with a whole team full of people and constantly creeping industry scope. The required combination of specialist and generalist skills to be successful is any of the Developer Relations and allied roles is extensive, and it's easy to feel overwhelmed and end up coasting being just-good-enough at things but never feeling mastery.

As an individual contributor, manager and general DevRel presence over the years, I've developed some tactics both for myself and my team, and I'd like to share them in case they're useful to others. Also since I'm moving to a less DevRel role, writing this down while I can still think about it!

The Monumental Skillset

When I was in my first ever round of interviews for a Developer Advocate role, I asked the interviewer "What is this role on the way to? I love Software Engineering, but if I become a Developer Advocate next, where does that lead me?". He thought for a while and then said "CTO. I've seen a couple of very good DevRel people do that and the only skillset this weird belongs to a founder".

That phrase stayed with me "skillset this weird"; it's funny because it's true. Developer Relations professionals need to understand or maybe even perform the job of their target audience. We need to communicate in writing as well as the documentarians, operate social accounts like the PR specialists, deliver technical content like a lecturer or keynote speaker - and probably video stream as well. We need to be able to juggle multiple competing projects, and often complex travel arrangements, and be publicly available, polite and positive, online and IRL, at all times. That's three separate career categories right there, so how do you gain all those skills, and develop them in meaningful ways? And when you change jobs and need to switch out to a new tech stack and a different subset of communication and workplace skills, then what?

The first thing to recognise is that many roles need some selection of these skills, but not all of them are equally important in each role. Your team may already do a skills matrix to help you understand what skills are most important in your role, or how the skills are spread across the team members, and this can be useful. If you're doing this with a team for the first time, keep it simple. Mine usually look something like this (I'm resisting the urge to go on a tangent but let me know if a post about this would be useful):

The desired skills and level of those skills will change between organisations, between roles, maybe even within one role over time - that's expected. What you need to do is identify something that you could get better at. Then it's time to make a plan.

Self Help for Developer Relations Professionals

This career isn't well-defined. There aren't many well-recognised certifications or exams to take, and in many ways as an industry we're still working out what would be considered good practice in any particular situation. By picking one skill to work on, you're essentially putting a better tool in the toolbox that you take with you and open when you need to tackle any challenge.

In the absence of a single source of documentation or other truth, we become our own teachers, but luckily there are so many good resources out there that the real challenge might be choosing where to start! Some topics can be studied in an afternoon; if you need to prepare to host the regular webinar on StreamYard then their own docs, a well-presented example webinar and some time spent pressing all the buttons and checking the output is probably all you need.

Other skills or topics may take a little longer, especially where the scope is large; for example public speaking is an apprenticeship that takes a lifetime! Along the way I have: read about ten books and many online articles about various aspects of the craft, given a few hundred talks with a few thousand hours of private rehearsal, watched those talks on video where they were recorded, received a lot of talk feedback, watched a thousand talks by other speakers, helped a few dozen other speakers to prepare and deliver their talks, become quite a connoiseur of talk abstracts for different types of event, spent two years as a regular Toastmaster attendee - and come away with the impression that there really is a lot still to learn.

What's common to both of those learning journeys is the mix of study, activity, and advice that brings improvement in each area. Taking input from different people, in different formats, and crucially practising those skills and reflecting on your progress, is what drives you forward and also consolidates what you learn.

Identifying resources that can give you what you need next can be difficult, the Internet is a big place. Set up a feed reader and follow the writings of people you admire in your area of interest (I started putting some suggestions here, but frankly it got out of hand and should be a separate post). You can also look at the archives for curated resources lists such as DevRel Weekly to find things on specific topics or get to know some individuals to follow.

When it comes to the hands-on element, there's an idea called "deliberate practice" that expands on research showing that the quality of someone's practice is what dictates their skill improvements. Be intentional with learning, with looking for opportunities to use those skills, preparing well for the opportunities, and in reflecting on how things went afterwards.

Multiply the growth

So far, so personal and professional development. What about the next generation? Well, in DevRel, we are all the next generation. There are people just joining, leaving, transitioning in sideways, and others doing the same role with a different title - and all of them are an important part of your story, just as you are of theirs. There is no "still doing it the old way" here, all of us keep on learning, evolving, adapting, improving.

If you have experience in the field, then you can raise the next generation by sharing your skills and knowledge with the individuals behind you. This is super valuable and could have been a blog post by itself - but this is a different post.

The real scaling of the DevRel skill standards comes from us being intentional in the way that we lean on one another to learn. You are probably familiar with all the techniques here, but consider how you might apply them to the next thing you want to learn, with the people you have around you. Here are some ideas and when to use them:

A study group or book club: This is really effective when there's a well-defined knowledge need (rather than a skills one), and you're all at a similiar level to start with, so it's a group with a shared mission. An example of when this could be useful: working through the onboarding guide to a new reporting tool that you're going to be adopting at your organisation.
Shadow an expert: Borrowed from medical teaching practices, where famously they "See one, do one, teach one" to learn a procedure. This approach is ideal when you have an expert who is prepared to work with you to hand over the activity or skill. I've used this before when chairing meetings, and wanting to be able to rotate the chair. First, I explain my chairing process, what I'm looking for, how the mechanics work. Then, I chair a meeting, with my protege in attendance. At the next meeting, they chair, and get some feedback (realistically, we might repeat this step a few times - and I'm sure doctors do too!). Finally, and after some practice, they will take on the responsibility of teaching this to someone else one day.
Feedback loops: Most developer relations teams practice this one already, using the rest of the team or company as talk rehearsal audiences or blog post reviewers is a great way to improve at speaking or writing. Note that the purpose of the feedback loop in this instance isn't to make a great talk/post, but to help the author to reflect themselves on what is already great, and what can be better - and pick that up themselves next time. Getting better at delivering great feedback is also a skill worth working on, try Lara Hogan's Feedback Equation and take your responsibilities as a feedback giver seriously. To be really meta, take a leaf out of the Toastmasters' book and give feedback on how the feedback was for those looking to improve their feedback skills. This approach works really well for improving skills like speaking or giving feedback, and can also work in a team of peers - it doesn't necessarily need an expert to offer feedback that you wouldn't have thought of yourself.
Formal study: Formal study is still pretty effective, actually, especially for individuals with good study skills. Although if you prefer video, then do that too, of course. The hard part about this sort of thing is making time to study solo. An online course with deadlines can help ~force~ encourage you to keep up with your studies and invest the time in this approach. I've written about this before (wow that's an old post!), but it's just as effective today as ever. This approach works best for fairly quantifiable skills, so learning a new tech stack or a particular technique makes this a good option.
Write the docs: Docs-driven study is one of my favourite ways to get to know a tool or technology. Take responsibility for one piece of documentation or a tutorial on a topic that's a little outside of your zone. By working through the details yourself, to a level where another person can come along and use what you produce, you will unavoidably learn something! This is especially good for getting to know the tools that you haven't tried yet, or that cool third party integration you've been reading about.

None of these are difficult to adopt, but it's unlikely any of us will improve if we keep doing the same things, over and over. Go out of your way to pick one thing to improve, focus on, or intentionally practice. It's hard to feel like the other things can wait - but they can. Honestly.

Be Curious, and Generous

The truth is, there's a lot to learn, and there always will be. Good learning habits, and intentional drive to improve are the best things you can teach yourself. There is a lot of research, plenty of good "this worked for me" content, and other resources around us. Go hunting for the best bits, and devour them. Share the links with others in your network, or curate collections so we all keep learning and building.

Crucially, remember that someone is always you on the path, and you have something to teach them. Write what you've learned and how others can use it too. I hope this post helped you to reflect on the resources we have available to teach ourselves, our teams, and our industry - but I would never have written it until one day I looked at the team I had grown, and saw them grow one another, and thought we were on to something that we should share.

Improve Existing OpenAPI Descriptions

2023-02-28T18:04:00+00:00

I'm a firm fan of a design-first approach to building APIs, and advocate for it at every opportunity. Repeatedly. Design-first means that an API change starts in the OpenAPI description, gets reviewed and approved by the stakeholders, and then gets built afterwards.

I also live in the real world, where OpenAPI descriptions get generated from existing codebases, and engineering teams lack either the knowledge or the enthusiasm to modernise their processes. With a generated OpenAPI description, everything else that OpenAPI enables is likely to be disappointing, because the description only contains the information that was in the serverside code, and if you're lucky some comment metadata. Enter OpenAPI Overlays, a way to describe alterations to an existing OpenAPI description that can be reapplied every time that description changes.

OpenAPI Overlay Specification

The OpenAPI Overlay Specification has been around for a while but it didn't have implementations (or I just haven't seen them, which is possible, please correct me if appropriate!). However it solves problems that I see in almost every organisation I talk to - the OpenAPI description is lacking words, examples, links or other additions that would make the documentation output better; the client SDK generators need some metadata to give more detailed hints on how to handle specific fields; not all endpoints in the description file should be fed into another tool (some are private, or restricted).

All these are familiar problems and I hope that the Overlay specification receives the attention and wide adoption that it merits. There are also some other similar solutions in this space, such as the Redocly decorators feature, but having an open standard really appeals to me since it enables switching between tools easily.

A library for Overlays

I've recently had a bit of time for a side project, and decided to start building a NodeJS implementation of this thing that I keep claiming will solve so many of the API problems I see!

Please try out the code, you can find `lornajane/openapi-overlays-js on GitHub <https://github.com/lornajane/openapi-overlays-js>`__. It's early days and I suspect I have a lot more tests to write, but if you're interested in taking it for a spin, please go ahead! I would love to hear what works (and what doesn't).

Manager's Secret Second Calendar

2023-02-26T16:33:00+00:00

Employees expect a lot from their managers, and as managers we do our best to meet those expectations. However most managers are only human (I've met a few that I had doubts about), and so it can be difficult to get absolutely everything right absolutely all the time. I'd like to share a tactic that I learned from another manager around the time that I took my first line manager role, that I think really helped me to at least give the impression that I knew what was going on: a second calendar, visible only to you, with key dates in.

Create a second calendar and set it to be private to you - but have it display in your default calendar view. To this calendar add:

employee start dates, and set them to repeat annually
team member birthdays, again with an annual repeat
milestones such as probation end dates, review dates, or onboarding milestones

With this in hand, you can show that you care - because you do, but it can be hard to keep the details in line. You can also make sure that everyone is getting timely support, for example in my first manager role we did onboarding check-ins at 30, 60 and 90 days - which I really liked and I added those dates to my calendar. When I then onboarded 5 new members of staff in a 6 week period and more than doubled the size of my team in the process, the calendar helped me keep track of all those milestones and not miss anything for any of my valued (but numerous) humans, and ensure I did the best job I could of getting their journey off to a flying start.

(Feel free to add personal reminders to your calendar too if that's a good place for them, I won't tell anyone!)

Do you have a favourite calendar hack that helps you appear clued up at work? Please share, I'm always looking for these extra ideas that can make things run more smoothly for me and for my people.

Generating a nice-looking PDF with Pandoc

2023-01-03T10:33:00+00:00

Pandoc is one of my favourite tools in the world. If you haven't come across it before, it converts between more or less any two document formats. I use it a lot with MarkDown and ReStructuredText so that I can produce content efficiently and then share it in a way that looks nice, but also to get word documents into cleaner formats. However the PDF output has always looked a little bit ... dated? (this is rich from an rst2pdf maintainer I know!) and I was recently delighted to find some tricks that resulted in a better PDF. I'm sharing them here, so I can find them again in the future; you are welcome to use them too, of course!

I'm starting from MarkDown, and in fact since I write blog posts in MarkDown format, I'm using the blog as source material for this (how meta).

For the impatient, here's the punchline:

pandoc pandoc-nicer-pdf.md --template eisvogel -V linkcolor=blue -V header-includes:'\usepackage[export]{adjustbox} \let\includegraphicsbak\includegraphics \renewcommand*{\includegraphics}[2][]{\includegraphicsbak[frame,#1]{#2}}' -o pandoc.pdf

For the curious, the next few sections outline how I got there. But first: here's a screenshot of the blog post you haven't read yet, since I'll need an image for demonstration purposes:

Build the PDF from MarkDown

Start simple! Take a markdown file and make a PDF from it:

pandoc pandoc-nicer-pdf.md -o pandoc.pdf

I mean, it's a great outcome from almost no work at all on my part, but we can do better...

Adopt a template

I looked around for a better template and came across Eisvogel on GitHub which looked really great. The install instructions in the project README are good and will get you set up and the template in the right place for Pandoc to find it.

TL;DR on Ubuntu I needed some additional packages (this is a new laptop so I have hardly anything installed): latex and texlive-fonts-extra to get the ClearSans font that's used.

Apply the template when building the PDF:

pandoc pandoc-nicer-pdf.md --template eisvogel -o pandoc.pdf

Set some variables

It's possible to configure Pandoc with a yaml block but I didn't want to change my source document, and I was also doing a one-off process so I stuck to putting the extra variables on the command line.

PDFs can have clickable links these days, and I expected the document to be mostly used digitally (if printing, you probably want to show link destinations as footnotes), so I set the link colour:

pandoc pandoc-nicer-pdf.md --template eisvogel -V linkcolor=blue -o pandoc.pdf

At this point, the document looks pretty decent, much more "me" than the standard output, but the images were sort of hanging around in the text, looking weird.

Then I found this brilliant question and answer on the TeX StackExchange which pointed me at some additional variables that add a border around images.

pandoc pandoc-nicer-pdf.md --template eisvogel -V linkcolor=blue -V header-includes:'\usepackage[export]{adjustbox} \let\includegraphicsbak\includegraphics \renewcommand*{\includegraphics}[2][]{\includegraphicsbak[frame,#1]{#2}}' -o pandoc.pdf

There are lots of things I like about outputting content to a known layout. PDFs have fonts and images included, so you can always send a document to another person or device, and know that it will arrive intact and looking its best! Since I mostly write for digital outlets, the layout really doesn't matter, but being able to very easily transform it into something tangible is brilliant, and I really like how this turned out.

Here is the PDF of this blog post if you want to see how it turned out.

What are your Pandoc tricks? Please share them in the comments, I would love to add to my box of tricks!

Sphinx extensions for developer portals

2022-11-08T09:27:00+00:00

I've been gushing about docs-as-code, but realised that I haven't shared many hands-on tips for actually building different types of sites in this sort of toolchain. I've been mostly using Sphinx which generates much better-looking documentation than its website suggests. It's a python-based static site generator, with a thriving community and lots of great extensions. Today I thought I'd share the extensions I use the most for developer portals (like the Aiven docs, one of my work projects and the source of the screenshots in this post) and what I like about them. Spoiler: may contain a theme as well as the extensions.

Copy button for code

Short and sweet: The Sphinx Copybutton extension adds a cute and accessible copy button to all the code samples on your site. Especially for developer portals, it's these touches that make the difference.

Use a single-file TOC (Table Of Contents)

The Sphinx External TOC extension allows you to configure a single navigation structure for an entire site. For a developer portal or docs site, where the content isn't going to be reused in another context* , I have found this very manageable. It makes the navigation structure independent of the folder/file structure, and it's all in one place so it makes it easy for contributors to know where to go to make the navigation bar change.

I now always use Sphinx this way, and plan always to do so, that's why this one is top of the list!

* we do reuse content but mostly by including snippets into more than one page, or into external locations; whole pieces of structure aren't reused which is when the toctree stuff is really awesome.

Configurable open graph metadata

Get your metadata and social previews in shape with Sphinx Open Graph extension. It is simple to set up and comes with well-documented configuration for how your site should be presented.

Sprinkle some design elements to support your best content

One thing I love about the docs-as-code approach is that there's no messing about. You want to publish something, you write it, you publish it. You use headings, paragraphs, diagrams, tables, bullet points. However it can all end up looking a bit like a textbook if you're not careful!

By adding a few pre-designed design elements and using them consistently across a site, this can be a nice way to add a touch of something friendlier to the user experience and Sphinx Design makes this pretty straightforward. It can be clunky to write the syntax for it all - but then if it was too easy, we'd overdose our users with accordions, grid layouts, and everything else. Remember: Content is King.

Show last updated date

The Sphinx GitStamp plugin that easily adds to the template the last time the source of the page was changed. This probably isn't desirable on every site but for a developer portal for a very technical audience, this extra metadata is valuable to the reader. It's also pretty valuable to the maintainers to know if things might be a bit stale.

The author, Jared Dillard, makes a bunch of excellent Sphinx things. The next extension on the list is also theirs, and they wrote an great blog post about customizing Sphinx.

Map your site

Having a site map is a boost for humans and robots alike. It can help SEO since the search engines can find their way around and understand your site structure easily. The Sphinx Sitemap extension extension creates this with minimum effort.

Endlessly maintainable diagrams with Mermaid

What's better than diagrams in documentation? Text-based, maintainable, source-controlled diagrams of course! I've been a fan of MermaidJS for a long time, so of course I use Sphinx Mermaid in the documentation portals that I work on.

As a side benefit, we will be able to render the diagrams differently for dark and light themes. And as a user with some accessibility needs, I can also create the diagrams using any text editor and contribute in the same way as everyone else does.

404 pages, just in case

The Sphinx NotFound Page extension adds the 404 page generation that really should be in Sphinx by default. No fuss, just a polite, well-rendered page presented to a user who mistypes or invents a URL they wish existed, with customisable options including the template file to use.

Special mention: Furo Theme

Bending the rules since the final addition here isn't an extension at all, it's a theme. I've found Furo to be modern, robust, and clean. We've adapted and styled it but it gave lots of functionality (secondary navigation, dark theme) out of the box, and we know our users enjoy these features, so it's getting a special mention in this post!

What would you add? I'd love to hear your recommendations for the magic sprinkles that make your developer portal experiences better!

Internal DevRel: Colleague Enablement

2022-09-26T15:09:00+01:00

I work in Developer Relations for a very technical company (Aiven), and I usually describe my job as half explaining my employer's technology to developers, and half explaining developers to my employers. However in the last year or so, I've realised that there is a variation on this theme that is impactful for my internal colleagues: explaining technology and developers to people who are experts in something else. I work with specialists in various aspects of sales and marketing (DevRel reports into Marketing) and my colleagues are genuinely curious to know more about the domain we work in! I thought I'd share more about how I enable my colleagues, and why I think it works for us.

Tech Support for Marketing

This story begins when I ran out of calendar space. As the people in Marketing with the best understanding of the company's technology, DevRel takes lots of questions from the specialists around us who know their own field, but not necessarily the cloud data platforms we offer as our product. In an attempt to remain available but manage my time better, I suggested doing a regular office-hours style thing to try to remain available but at a manageable level - and so it was that Let's Talk Aiven Tech was born.

The Idea

The basic idea is that half the company knows our technology and developer culture really well - but the other half knows a whole raft of other things that we also need but which are not the technology. Asking our deeply technical staff questions involving the technology was often returning overly technical explanations; this is a genuine problem where different departments speak what is essentially different languages. What we need id someone who can meet this not-obviously-technical crowd (spoiler: this lot are geeks, they're just different to the engineers you're used to) where they are, and share an appropriate level of knowledge with them in words that they knew.

Communicating technical concepts appropriately to different audiences is definitely a Developer Relations superpower, so once I understood this problem and how much impact it would make if we could solve it, I knew this was something I wanted to tackle. I was too busy to create the internal curriculum I wished we had, but I had knowledge in my brain and I was prepared to dedicate an hour a week to trying to transfer it out to where others could use it!

The Platform

Engineers are famous for over-engineering solutions but I can honestly say that I under-engineered this one! The whole scheme started as a weekly virtual meeting with four invited guests and an editable calendar invitation. There was a slab page where we kept a list of the upcoming sessions, the past topics with recordings, and another section for topic requests. Some weeks, I remembered to post a reminder in a slack channel the day before to advertise the topic, and ask people to react with a particular emoji (different ones each time) if they wanted to be added to the calendar invite.

The important thing was: it's super casual, people can show up or not, or watch the recording later. It's absolutely opt-in, and I wasn't sure how much use all this tech content was going to be but with a small number of people asking to give it a shot, we just started and were prepared to iterate. In the event, what started as four invitees turned quickly into a large number of people who wanted to be added to the invitation. We've grown from about 300 to 500 people this year, and the calendar invite had between 75 and 100 people on it most of the way through - usually 20-30 people are there for the live session. The hardest part was persuading technical people not to show up and ask their own questions about the technologies; this will silence anyone who can't follow that conversation and it is difficult to recover from that.

Session Format

Since I don't have the bandwidth for any prep at all, I usually show up to these sessions with 5 or 6 bullet points to remind me of key points. However this turned out to be a blessing in disguise, because of course I am not the expert on what other people need in their daily work - they are. We settled into a format where I would give a 5-10 minute overview of a tool or technology, trying to capture at a high level what the thing was FOR, what people used it for and WHY they would use this one rather than something else. These are technical topics, but not especially technical sessions. After the intro, I pause and see what questions are already in the room. Sometimes this is enough to get things moving for the whole hour, and the crowd follows on from one another's questions. Very often, they ask questions that I can't answer but someone else can (other technical staff, someone with a better handle on our data dashboards, someone who attended a meeting about that thing), and I just facilitate.

If conversation dies, I usually demo the thing that is the focus of the session, working through our getting started docs or a typical use case. A few times we have also run this as hands-on labs for things that we can easily do together in the time. Either way, seeing it in action almost always produces more questions, even if it's "why would someone want to do THAT"? I don't think I've had a session that ran more than 5 minutes short in months of running this initiative - my colleagues really are curious about everything.

(As a very well-rehearsed conference speaker, this complete lack of preparation scares me every single week, but it forces me to get out of the way and the audience has to drive the session to the places that they need to visit.)

Speaking the Language

The sillier the story, the easier it is to remember. I am pretty sure that some of my utterly ridiculous analogies were nonsensical but I tried very hard to conjure some mental images to represent the ideas. For something that is a lot bigger than another thing: one is a wheelbarrow, the other is more like a container ship. Logs become an expensive bar tab, and Terraform is a building site supervisor. The other advantage of keeping things fairly silly is that they don't feel too intimidating and jargon-filled. There's still a lot of technical terms and I'm sure there were people that didn't feel they could speak up, but others told me that it did help to understand the concepts.

The hardest part about this whole exercise for me as a technical person was how to answer the questions in a way that the asker could use the information! We could put together a reasonably long outtakes reel of me being completely floored by really excellent questions. Here are a few of my favourites:
* You showed us a different graphs thing last week, why do we need more than one of those? (Relating to Grafana and OpenSearch Dashboards, but we had several variations on this: Postman seems like Terraform when you demo one small thing because they both call the API to make something happen. MySQL and PostgreSQL really ARE similar, and lots of projects could use either)
* This week you said that developers prefer the CLI but last week you said they prefer the API, which do you really use? (this is brilliant because I clearly use both and trying to articulate which task would use which tool made my brain hurt, I might have to write that one as a separate blog post one day)
* So this little dog (the DataDog agent, it was another analogy that got out of hand) is on all our services. Why did we let the dogs in? Aren't there any other options, should we have them all?

Serving an Internal Audience

The key to successful DevRel is always to meet the users needs, in the words they know, in the place that they hang out. I had no idea that so many of my Marketing colleagues were so keen to know this much about the very technical tools on the Aiven platform, but after 9 months (with a few breaks for holidays and conferences), I know that wherever I work and whatever I do, I'll always want to serve the internal audience as well as the external ones. DevRel is about combining empathy with technical skills to enable and empower the audience - and this achieves that goal in spades.

Outline your writing to ease the creative process

2022-04-11T09:39:00+01:00

For most of my career I've been a software developer, but now I'm mostly a communicator. As a manager, I give the right level of detail to many different audiences, and I deliver that in words they can understand. As a Developer Advocate, I explain complex technical concepts in useful, memorable, and occasionally entertaining ways. However the best software developers are lazy and I'm still always looking for ways to get things done with less effort on my part! For writing, the best process I know feels like unnecessary overhead, but it's always worth it in the end so today I'm sharing my secrets: create an outline before you start.

Why outline your writing

With an outline, you can tell before you've started whether you:

understand the topic
can get your points into a coherent order
know what's in scope
need to fill in any gaps before writing

If, like me, you get interrupted while you're working, the outline also means that you don't need to hold a grand plan in your head. The structure is laid out before you start, making it easier to stay on track with your train of thought even if you are jumping in and out of this task among other activities.

If you had the sort of education where you were taught to create an essay plan before writing, you already have the skills you need for outlining. If not, I'm about to tell you how to do it anyway, so keep reading.

How to create an outline

Think about who you are writing for, and what they need to know. Then write a list of 4-6 subheadings that cover the main points. If you need more points than this (do you, really?) then nesting levels of subheadings can also help, especially if it's a longer document.

That's literally it. You created an outline!

Sometimes the outline needs quite a lot of work but this early stage is the best possible time to do it. It's much easier now to run the idea past someone else for their input, or to reflect on whether you could move one section to a separate document and just reference it, than it will be once all the words are there getting in the way.

When to use outlines

Try using outlines on everything that's more than two paragraphs long, and see what sticks! This approach works for me when I'm working on everything from difficult emails to documentation, from blog posts to project updates. I create outlines when I'm writing conference talks too, something about the pre-planned structure gives a nice cadence to the different sections of the talk or written piece when it's finished. Almost as if there had been some sort of plan at the start!

The hardest part of writing outlines is just like the hardest part of writing specifications for software: I'd rather be writing the words/code! The outline always saves time in the end, and makes me really think hard about the overall aims of the thing before I put pen to paper and start wondering if my sentences are too long. For this lazy software developer, the best process is the one that gets the task done soonest, and outlines always win.

Who are you writing that commit message for?

2022-03-31T20:55:00+01:00

I read a lot of commit messages that make me wonder who the committer had in mind when they wrote it. If you don't read commit messages yourself, I think that can make it even more difficult to think about who the audience is, or when someone would be reading those entries. Perhaps you're writing for nobody, or work in a team that doesn't value the metadata that a single sentence written in the moment can deliver.

Next time you write a commit message, try some of these suggestions as your imaginary audience:

Yourself, next week, when you finally get back to working on this thing and can't remember where you were up to
Yourself, when you get a pull request review and can't remember which commit something is in that needs to be removed
Yourself, debugging how this ended up like this, 6 months from now
Your colleague, eyeballing your work to see how you are getting on

Personally, I think of it as a note to myself. Like an alibi, if someone asks you what's already been done, or what this commit that removes one specific line from a long config file. Yes, I worked as a git consultant for a while, the delete-a-single-line with the commit message "Fixed" is always the culprit!

Further reading: https://cbea.ms/git-commit/

Weekly Planning with Obsidian

2022-02-16T13:01:00+00:00

Keeping on top of tasks at work has been a work in progress throughout my career. However now I'm doing a more managerial role, I also need to keep track of all the meetings and preparation/followup work in my meetings. I find my daily log fills so many pages of notebook that I moved my weekly plan to a separate section - and then to my preferred digital notes tool, Obsidian. If you're interested in my setup, read on.

Weekly structure

Meetings is always the longest section, I am essentially a professional meeting attendee at this point!
Shipped reminds me to capture things that actually do get done, in case they get lost and I forget I have achieved things
ToDo collects currently-need-doing items. Usually I don't duplicate things assigned to me on GitHub or any of our ticket tracking software tools.

Meetings

I look at my calendar and add a heading for each meeting that is on my calendar for the week, and I do this before Monday morning. In theory, Friday afternoon and in practice, Sunday night. It's like ironing shirts for work except I don't iron any more :)

For each meeting, I look at whether there is preparation work to do. Am I running the meeting? Is there a pre-read or other work I can do beforehand to make it more useful? I write a list of talking points or other things that I might want to mention in each slot. This helps a lot because my schedule is typically very full, with consecutive meetings that can have VERY different topics. Having the notes ready helps transition from one thing and think clearly about the next one.

For my regular meetings with my manager, and with my reports, I usually start the week with only a few points, but if I want to ask them something I just add it to the section for their session when I think of it. This works very well since I don't need to flip between days, the whole week is in one document.

I don't take notes into the same document, I mostly use a pen in a typical bullet journal daily log style, with some sort of adapted dash-plus notation so that after several hours of meetings, I can roll back through the notes and spot the items that needed further action from me.

Shipped

Almost always the shortest section, which is probably the right thing for someone who is head of a department! My rules here are that it has to be something that is actually finished. If I recorded a video but it isn't on YouTube yet, it's not "shipped" until it's live and all the work has been done.

ToDo

This is things I need to do, I don't copy over things from other systems, so I have Asana, JIRA, and GitHub also open essentially all the time. I add things here if they are not on this list, for example right now the list is reminding me to sort out a thing with my health insurance, email a speaker for an upcoming event, and to follow up on a thing I saw on Twitter.

Obsidian is a markdown tool so it supports lists with checkboxes in - I check the box when I'm done and then when I do the weekly reset, I can see what didn't get done and needs migrating. Every week I grab the stuff that isn't marked "done" from the previous week, decide what just isn't ever going to happen and remove it, and then try to identify from any unprocessed notes the week before which todo items haven't been transferred anywhere useful yet.

Weekly planning cadence

Doing a weekly reset has worked well for me, and the digital format makes it infinitely adjustable so I never run out of space as I would on paper! Doing it this way, the meetings for the day are all next to one another in a document and I just have it open at the edge of my monitor so I can glance over if I need it.

Keeping your finger on the (digital) pulse

2022-02-15T18:25:00+00:00

Whether you are building your own presence, or representing a brand or employer online, it's useful to know what people are saying! Keeping up with all those possible places you should look for news or mentions can be pretty laborious and when things are busy, it's difficult to keep up with this work unless you have some decent tooling in place. There are some great automation offerings around but here's my very basic strategy.

TL;DR a bookmarks folder in my browser to open a bunch of saved searches all at once.

I created a folder of bookmarks in my browser, which is Firefox; it has support to open all the links in a folder at once. These are saved searches for all the things I like to keep an eye on. They are things like:

Search for newest issues across our whole GitHub org so we don't miss something on one of our smaller or less-used repos because it was lost in a big raft of email notifications. The same for pull requests.
Searches on StackOverflow, Reddit and HackerNews to see what people are saying. I wouldn't recommend diving into the middle of all the conversations here, but more to hear the general sentiment. I don't have a search on Twitter since I basically live there full time anyway, but you might find it useful.
Dashboards for whatever the main metrics are for your team, for example I look at both Google Analytics and the GitHub stats for our docs project to understand how people access and read the content, and also how they engage/contribute to it.

Keep all the searches very narrow and specific - I should take my own advice on this since I have a few too many tags in my StackOverflow search and it makes for FAR too many results when I just want to check things quickly. Sort the results so that the newest are first and only the last 30 days are included if that's an option, there is no point seeing the same 2-year-old post every week.

Now set a 30 minute meeting in your calendar every week to check on things, quickly open the bookmarks collection, and then go through seeing what's happening. If there are followup actions then do anything that takes 3 minutes or less immediately, such as sharing it with your team channel, or replying with a comment. For anything more, instead take the time to create a ticket linking to what you found and describing what action should be taken. For me that's often "uh oh, should we have some documentation about this thing?" type tickets, and I really enjoy being able to go back to a question later and offer both some knowledge and also a link. Even if it's too late for the original poster, sites like StackOverflow bring our collective knowledge together in a public place for the next person to use, so it's always worth adding whatever you have!

Just like answering a StackOverflow question, this post came to you when I was giving some advice to someone doing a similar role to mine. A folder of bookmarks isn't rocket science, but it game them a quick start into one tiny part of their day to day job - and sometimes the smallest changes make the biggest difference! What are your "really not rocket science" tips that you'd share? Tell me, I want them all :)

VSCode setup for technical writers

2022-01-05T14:23:00+00:00

I'm working almost entirely with docs-as-code setups for prose at work these days and while I do use vim for most of what I do, VSCode is increasingly in the mix. It's just about accessible enough with keyboard shortcuts and the command palette for me, and I thought I'd share the extensions I'm using and find helpful.

As a former software engineer, I kind of expect certain features from my text editor, but not all writers have had such experience to tools. I encourage everyone to have both good syntax support for the file types they work with, and good previews. VSCode is an ideal writer's tool because the two-panes view shows things as they happen! We're using some RestructuredText and some Markdown so I have Preview which handles both. For syntax hightlighting, I have a RestructuredText syntax highlight extension and also one for YAML highlighting. With the restructured text one, I found it helpful to disable the language server because it wasn't familiar with a lot of the directives our Sphinx site uses (the config setting is restructuredtext.languageServer.disabled - set it to true).

Pro-tip for anyone working with yaml or anything else using indents: Indent-Rainbow gives a lovely visual clue of how indented things are. It's subtle, but it's just enough.

We're using GitHub for collaboration and while I do have the GitHub Pull Requests and Issues plugin and recommend it to everyone else, I almost exclusively use a combination of the web interface and the `gh <https://cli.github.com/>`__. The web interface is almost accessible, usually there's one regression or another (currently: the one-line text box to reply to comments doesn't turn into a full comment box when I focus it, I have to click) but between the CLI and the web tools, I manage. I also heartily recommend the Git Graph extension for teams that use branches. We have what I will politely describe as a mixed set of git skills in my team and I find the visualisation can really help all of us understand what went where.

We're a globally distributed team which means we can't just look over one another's shoulders to fix things. The Live Share extension helps a lot when we need to put our heads or hands together to get something done.

As a final addition, I'd like to give a special mention to the newest addition to my extension collection, Vale extension for VSCode. I'm working on a docs project (it's Aiven Developer in case you are interested, also has a public GitHub repo if you want to dig around the details on any of this stuff) that has a CI build and previews for the documentation. It uses Vale to keep us organised with things like spelling (and capitalising) product names correctly, using consistent uppercase in titles, you get the idea. We can also add rules to it as we go along. We have a lot of occasional contributors and this tool is so complete, it has a CLI, a GitHub action, and a VSCode extension - I mostly use it as a make command, but the VSCode plugin will run on save (or as you type, you choose), and I am looking forward to using it more instead of realising the build failed and then trying to work out what to fix ...

I hope my list gave you some ideas of extensions to improve your experience, and I would love to hear what other tools or extensions you would include in a collection like this! Comments very welcome :)

What Happened to your Blog?

2021-12-12T20:20:00+00:00

What happened to my blog? Nothing happened, I still blog ... except then I looked at my site the other day and realised, it's been a while.

The blog here at lornajane.net is mumble years old. It is full of things I wanted to write down, and tells the story of most of my career to date. As a software engineer, open source project maintainer, author, speaker and teacher, this made perfect sense and I've been thankful many, many times for the thoughtful explanations of past-Lorna that have helped me along my way.

This year, my career changed. I'm a manager now, and the problems I solve don't lend themselves so well to writing or sharing. As a manager of people, I'm both too early in my career growth to have much to say, and also conscious of writing about "my" humans, in a way that might not work for either or both of us. I still work with a lot of technology, and more formally with developer experiences than ever before, but this year has been a professional challenge beyond anything I've tackled for a while (and looking back, quiet patches on the blog have in the past been mostly when I was writing a book, which is also a professional challenge!)

As an engineer, a writer, a developer experience enthusiast and as an open source project maintainer/contributor, I still have a lot to say. Somehow, the combination of a demanding year and also a sizeable side project* doesn't leave me much spare time - and I mostly fill it with management textbooks and Stardew Valley! It has felt like a year of mostly listening and learning, more than sharing.

That's not to say that I didn't write. I really did! Just ... not here (the more Lornajane has the links if you're interested). See also: internal docs, and a whole new developer portal developer.aiven.io.

So what happens now? Nothing big. Just sitting down to write this reminded me why I love having a blog to write on in the first place. It's been a strange year (more than one, for almost everyone), and between two big work transitions - a new employer, then a big promotion once I got there - and half a year of having the kitchen rebuilt, I didn't have words to spare.

I have learned so much this year, and while it would look very different to most of the previous posts here, I would still like to write some of it down. This blog began as a way for me to capture things I wanted to share or reflect on in the future. Sometimes, other people read it too. But the purpose hasn't changed, and it took me time to realise that it makes perfect sense for the content to change as my career does.

* I was preparing a Technical Communications workshop but unfortunately it won't be going ahead. However there are a lot of topics there that I've done a bunch of research and figured out how I want to share, so I'll be putting some of them here when I get chance. Topic requests in the comments, please.

Sphinx front matter and template variables

2021-09-10T21:47:00+01:00

I'm building a docs-as-code site at work at the moment (you can peek if you like https://developer.aiven.io) and working with ReStructuredText in this setup is new to me. One thing I really missed was the concept of front matter, which I'm not sure is even official or documented but seems to be reasonably widely supported (I promise to save the rant about markdown for another day). It took me a while to figure this out for Sphinx/rst but now I'm looking at it working, I had better write it down before I forget!

All this is on Sphinx 4.0 and Python 3.8, in case this information matters to someone reading this post 10 years from now.

Add fields to your content

Sphinx calls this extra information file-wide metadata, variables that apply in the context of the file they appear at the top of. I'm not sure how many Sphinx installations use something other than one-file-per-page but file-wide works perfectly for me.

Here's how I added a description field to a page (I'm not using the meta markup because I'm actually going to use this field in a twitter card implementation):

:description: This page has an excellent description of how to apply cucumber and unicorns to the problem

Heading 1
#########

More words on the page .....

Note that the extra fields have to be the first thing in the file, before the first heading or other content. This adds any field name you care to invent, and a string value (check the docs for more detail on supported types but basically: use a string) to the meta field for this pae.

Using this data in a template

I'm adding basic Twitter Card support by pasting it straight into base.html. Probably the comments on this post will help me to understand better ways to solve this problem, which is why I love blogging. If there's a specific description set, I want the tag to use it - and if not, we'll just grab the first 120 characters of the page itself. Ready?

<!-- Twitter Cards -->
<meta name="twitter:title" content="{{ docstitle|striptags|e }}">
<meta name="twitter:description" content="{{ meta.description or body|striptags|e|truncate(120) }}">

The {{ meta.description }} field in the template holds the value we defined in each page, and you can also see the or syntax there in case we need to fall back to just using a bit of page content (which hopefully does introduce the intent of the page pretty well anyway!).

This was harder to find an example of than I had expected so hopefully this will help someone else running into it next time.

Use Database Connection Strings with Laravel 8

2021-08-18T08:56:00+01:00

I've been doing a lot of database stuff lately, and not much PHP, so when I returned to make my first Laravel project for a while, I had to check the docs to remind myself how some of this works. I noticed that the default approach to database credentials is still to use separate credentials for the host, port, and other variables. I'm using Aiven databases (because I work there and managed databases are great for demo apps as well as real ones!) which supply connection strings, but Laravel supports these too.

For MySQL, the database connection string from Aiven is in the format:

mysql://username:password@hostname:port/dbname

You can remove the configuration for DB_* and instead use the variable DATABASE_URL with the full string.

Pro-tip: Laravel's built in migrations (I was using Laravel Breeze) expect you to have mysql.sql_require_primary_key disabled, I'm not sure why. It's default enabled on a lot of systems, including Aiven and Digital Ocean from what I've seen so far, so you may need change this setting if your migrations don't succeed.

For PostgreSQL, the database connection string looks very similar to MySQL but with a pg-specific prefix:

postgres://username:password@hostname:port/dbname

Again, this connection string goes in the variable called DATABASE_URL. For PostgreSQL, you also need to set DB_CONNECTION to pgsql so that Laravel knows to use this connection.

It's not a big deal to separate the different variables needed for the database connection but especially as a person that spins up a lot of temporary databases to try stuff out and therefore has to update the connections all the time, this was a neat trick that saves me a minute or two every time.

Hold Attention on the Virtual Stage

2021-08-08T22:37:00+01:00

How do I ensure that I keep my attendees' attention when I'm presenting virtually?

I don't.

I have seen a few tactics for holding people's attention during virtual events. One is to require everyone present to perform a particular action such as typing a message or clicking an icon, several times during a session. Another is to require cameras on if the platform allows it. I find this patronising and also unnecessary. Perhaps because my audiences are adults, and my only obligation is to deliver great material - not to spoonfeed anyone.

When I teach a class in person, I tell my attendees that the most important thing I am providing is space. Space for them to learn, to ask, to try things, to really take on something new. Sure, I'm providing valuable knowledge and support but really they learn because there's a space for them to do so. That's why conference videos seem flat if you watch them after the event. But if you schedule your conference video viewing time, take lots of notes, and talk about what you saw afterwards? Then the virtual experience can come close (and I need to publish that draft blog post about watch parties).

When I teach a virtual class, I can only create my end of the space. I invite and welcome my attendees' participation. I strive to make my digital content compelling even though we're all tired of video, and I encourage people to use whatever is on the virtual platform to engage with the topic, each other, and me. That's all I can do.

If you want to listen with half an ear, that's your decision although I feel a little sad on your behalf about the opportunity lost. If you want to empty your hands, to lean in and let me lead you through a journey of knowledge, to take this time that we have together and take everything that you can from it? Also your decision and I love that we can make this happen across space (and time, if you're watching a pre-recorded me).

Presenters can never force attention, all we can do is invite it, encourage it, and hold up our own side of the bargain. The rest is up to each attendee to choose whether they invest in themselves - or not.

Define Navigation Structure in a Sphinx Project

2021-06-13T19:11:00+01:00

As a newcomer to Sphinx, I found many things confusing, but the table of contents or "toc" configuration was the thing that almost made me change my mind and pick a different platform! I now have a setup that works pretty well, using Sphinx External TOC, so I thought I'd share my setup.

What I really like about the external_toc plugin is that all the navigation is defined in one place. I configured my project to look for a _toc.yml file, and that contains all the entries that will be included, and what order they will be included in. I'm using the Furo theme so the table of contents populates the whole of the left hand navigation bar and I find it helpful to have it all in one place rather than looking for toctree directives nested all in different files and being assembled (or not, as I discovered) to create the navigation.

Since each file can only be included in the navigation once, a particular page such as the "getting started" file can be listed specifically, with others from the same directory included using the glob keyword to match the path with a wildcard. This is very helpful and I use it in a few places!

However your navigation is constructed, you can use the .. tableofcontents directive in a page at any level. Just like when using a toctree directive, this displays the navigation entries below the current page, so it's useful for detailing subsections without needing to curate lists or indexes.

If you're building a sphinx project yourself and have tips to add, I'd love to hear them. It's a great tool but I know that I have much to learn as well :)

Considering Developer Relations as a Career Move

2021-05-24T11:02:00+01:00

I think that Developer Relations is a fabulous career, and I'm often asked how I got started, or how someone else could get started. There are a lot of people hiring for Developer Relations roles just now (including me!) so here are my tips.

There are so many different Developer Relations roles (called "DevRel") that I am definitely going to fall short of covering them all in any sort of detail! My own background is in Software Engineering, and I've mostly worked as a Developer Advocate since making the move into DevRel, so that's my context here.

Are you Interested in People?

I am not sure why there's a stereotype that technical people aren't interested in other people. Sure, some people fit that description, but many don't. In DevRel, it's really all about other people. Trying to understand what someone else would find useful that they don't already know - and what they do already know so they don't get bored and wander off before you get to the point!

Signs you may be interested in people include:
* Having a blog where you write stuff down for others to use
* Posting answers on StackOverflow for questions on topics you konw
* Speaking at a usergroup to share your knowledge, or helping to organise one
* Sharing your knowledge at work through presentations or written resources
* Helping out with open source projects, especially getting people onboarded
* Teaching classes to help others learn technical subjects

None of us has unlimited free time, but if you've done a little bit of some of these activities, and you'd like to do more, then I would consider you to be a good candidate for DevRel. My most frequent response to a CV is "why does this person think they want to do DevRel?" - if I can spot any two of: writing, speaking, teaching, or open source? Then that's a great indicator that this could be a good fit.

The other big thing about all these items is that they are all within reach of all of us. So if you're considering a job in DevRel, then writing a blog post, giving a talk, or making open source contributions are all really good ways for you to work out if you like those things enough to have to do them often! They also make excellent portfolio pieces, but that's less important than giving yourself a chance to figure out what works for you, or what you'd like to get better at.

Empathy, and the Other Requirements

Empathy and acting in the interests of (mostly unknown) others is the basic requirement for Developer Relations. However there are some other traits that I see in successful DevRel types (and isn't that the point? To choose a job you'll enjoy and succeed in?).

Proactive: The number one piece of advice I give to all new DevRel types is: we're all just making it up.

There's no magic spell, no perfect approach, no one true strategy. Every product is different, every community is different, and sometimes you'll be far from home with a problem to overcome (or at least hundreds of strangers to make a good impression on). If you're the sort of person that can think of three possible options and pick one, in any scenario, you'll be fine.

Organised: I always talk about how creative DevRel is as a career, and it is, but if you are that rare combination of very creative and fiendishly well-organised, then I think you've found your place! The whole idea that a person can only be logical or creative, and you have to choose, has never made sense to me - and now I have a job that allows both brain-halves to be busy.

On a more serious note, DevRel is often a very large number of small things all in flight at the same time. Even just keeping track of conference deadlines, acceptances, dates, preparation time .... you get the idea! Add in all the demo apps, blog posts, documentation/tutorials and whatever else it is we do (videos? podcasts? digital communities?) and there is a lot to keep track of. If you've ever been camping and not forgotten anything then consider yourself qualified :)

Communicative: Most DevRel roles are remote anyway, since most teams want to have people spread out to get to know more communities. Even if you are office based, the communities that you serve will probably not be very local! So being able to communicate in a friendly manner, in person and in writing and on video, more or less all the time, is key. Keeping people updated and getting conversations going are important skills, but most of all it's about being prepared to put in the time to nurture the connections and build on them.

Taking the First Steps

So now you're convinced, how do you get started? Take a look at the skills that you have already and the job postings that are open. DevRel roles are very much about following your interests - the job is made much easier when your are advocating for something you actually like and use yourself :) Remember that DevRel is typically much more mixed than other technical teams in terms of the obvious, visible diversities - but it's also an excellent choice for career changers or anyone else with an unusual skill combination. You might feel that having started "late" in a tech career is a disadvantage but I have almost always found that the opposite is true.

Are you thinking of moving into a DevRel career? What sorts of things get in your way? And if you're already there, what advice would you give someone following the same path?

Best Things About Developer Advocacy

2021-04-26T20:07:00+01:00

There are some excellent articles around about what a Developer Advocate does, but have you ever wondered WHY we do what we do? I tried to articulate this when chatting with another developer lately, and I'm still thinking about it! So I thought I'd write it down to share a little more widely why I love this strange, misshapen career as much as I do.

It's Alarmingly Varied

Sometimes when I'm doing career development talks or outreach (I'm a STEM Ambassador ) people will ask "Can you describe a typical day?" and the answer is ... not really! Advocacy requires a lot of different skillsets and you can expect in any given week to be planning, writing, speaking, recording, coding, fixing, and attending as many meetings as anyone else. Even when I think my schedule is pretty much under control, there's always someone with a technical question, a new issue or pull request, something that needs my review, or a request from another department.

Honestly? I love it. I cannot at all imagine going back to doing a fairly steady engineering job and reliably delivering applications. Even when I last did that sort of role, I was part-time and doing advocacy/consulting/training for the rest of the time - mostly because I enjoy the variety! Being able to handle interruptions and handle the context-switching are both very key skills for a job like this. Deep work does require blocks of time and I still write and code fairly in-depth things, but it's all in maximum 90 minute blocks! I will always be convinced that the years of doing open source projects with whatever time I had was the best possible training for this working style.

It's Surprisingly Technical

Developer Advocacy has been an incredible way to pick up a broad swathe of new technical skills in a way that I simply didn't get the opportunity to do in a full time engineering role. When I took my first Developer Advocacy job, I knew one programming language very well. I'm now baseline competent in four tech stacks and I can wrangle the dependency managers and code from another few. There is much less gatekeeping of what you're "qualified" to do so if you can produce some content or a plugin that could help a developer: then you just do it. It's also made me absolutely fearless with new technology because I meet a new codebase at least every week and I greatly value this confidence now that I've found it.

Just like tech support, the DevRel roles put you on the spot with any of your employer's products, combined with any tech stack the user might have picked, at more or less any time. This should be horrifying but in fact I find it fascinating and refreshing. The different tech communities have different approaches to so many different things, and finding the familiar reference points for someone to understand your explanation is challenging. It's incredibly rewarding to be able to help many different people with their questions and feel like you're making a positive impact.

It's a Helping Profession

This part might be too sappy but bear with me, if you can. As a Developer Advocate, I help people and I empower them to reach greater achievements. I love being a multiplier - as an engineer I can only build so many things myself. But as a writer and a speaker, I can aid and abet a whole community of developers who are all making awesome things themselves. It's an amazing privilege to choose a profession where I so regularly feel like I'm making a difference to others.

It's Deeply Creative

Developer Advocacy is incredibly creative. Software engineering by itself is already very creative, by definition you're creating things. But Developer Advocacy, to me at least, is like being asked to write at least one very geeky fairy tale every day of the week. Thinking of examples, ways to explain things, concepts to inspire and stories to help humans digest big technical ideas - then putting together a strategy to tie it all together ... Developer Advocates are ideas people because that's the job!

Creating content is all about educating and informing, and helping your audience to take away something of value from the time they spent with you and/or your content. As a day job, this can be slightly stressful "Quick! Be inspiring!" but having the freedom to think about how to present the idea or demonstrate the feature is something that I'll always value.

It's Not Forever

Developer Advocacy improves the not-just-code skills that are so valuable in very senior engineers: the ability to communicate well in written form, the ability to present complex ideas to others, the ability to handle multiple incompatible tasks all at once. I'm generalising for engineers when Developer Advocates are also Data Analysts, Ops people, consultants, and all sorts - but the point still stands.

Doing a role or two in DevRel makes well-rounded technical employees and it's more approachable than people realise. On a very personal level, this has been the career that found me and made me realise I could be good at something, and I don't think we share the joys widely enough! What do you love about your job? Tell me :)

Measuring Repo Community Health with GitHub's API

2021-04-14T09:19:00+01:00

I'm on record saying that GitHub is your Landing Page and when I think about companies having open source profiles, I think about how many developers will have the first contact with them on GitHub. If it's a code example you're looking for then like it or not, GitHub is considered a search engine by many developers.

With that in mind, I wanted to look at GitHub's Community Health measure of the repositories I'm responsible for. You can view each repo's community page separately through the web interface (look under "Insights") but that's not especially scalable if you have a lot of projects to track.

So for a more repeatable approach, I wanted to use the GitHub API. Frustratingly the community health metric isn't in the collection endpoints, you need to fetch data per-repository to get it. Here's the script that I use; it fetches the first 100 public repos (if you have more than 100, you'll need to loop and fetch additional pages of results) of the given org or user. It prints all repos and number of stars, then for repos with more than 10 stars, it also fetches and output the community health measure.

The whole thing outputs some sort of pipe-separated-sort-of format ... one day I will make my hacky code more perfect before I share it, but today is not that day. LibreOffice had no problem ingesting the result when stored in a text file, so I'm calling it "good enough"!

Anyway, here's the script - you should set a GitHub access token as GITHUB_TOKEN and the org or user to use as GITHUB_ORG:

import json
import os
import requests

token = os.getenv("GITHUB_TOKEN")
org = os.getenv("GITHUB_ORG")

headers = {
    "Authorization": "token " + token,
    "Accept": "application/vnd.github.v3+json"
}

# all public repos
base_url = "https://api.github.com/"
repos_url = base_url + "orgs/" + org + "/repos?type=public&per_page=100"

repos_req = requests.get(repos_url, headers=headers)
repos_list = json.loads(repos_req.content)

# print header row
print("Project | Stars | Health")

i = 0
for r in repos_list:
    label = r['full_name'] + " | " + str(r['stargazers_count'])
    if r['stargazers_count'] >= 10:
        url = base_url + "repos/" + r['full_name'] + "/community/profile"
        req = requests.get(url, headers=headers)
        data = json.loads(req.content)

        print(label + "| " + str(data['health_percentage']) + "% health")
    else:
        print(label)
    i = i + 1

# if this is 100, it's time to build pagination
print(str(i) + " public repos in total")

I was surprised when I looked around that I couldn't find an existing script for this, so I thought I had better share mine. That's how the open source community works, after all! Tweaks, suggestions and additions are all welcome via the comments box, I'm happy to hear if this is useful and how you evolved it for your own needs.

From Spreadsheet to Database with ddlgenerator

2021-03-22T16:33:00+00:00

I've been using some fun open data sets recently but I'm more familiar with databases than with traditional data science formats (this one is CSV). Importing CSV into Postgres is fairly well-established, but you need a table structure to import into ... and I was looking at a CSV with 50 columns (for my post about exoplanets on the Aiven blog). Enter this neat utility: ddlgenerator.

I used it for a CSV file and put the data into PostgreSQL but on closer inspection I realised that this tool can ingest a great selection of different formats, including JSON, YAML and even HTML. And it can create the table structure for any of the dialects that SQLAlchemy can speak - I expected PostgreSQL and MySQL support but the selection is pretty extensive, including SQLite, SQL Server and even Oracle databases.

The tool itself is so simple it's almost disappointing! Install it with pip, and it's a CLI tool.

$ pip install ddlgenerator
$ ddlgenerator postgres planets.csv

The output is the SQL statements to run on the platform you specified (e.g. postgres) to create the table structure for the input data (e.g. planets.csv). I used the built in support to import the CSV file to my new PostgreSQL database, but ddlgenerator can also generate all the INSERT statements you'd need to get SQL statements to put the data in the table as well, so that's neat.

If you're also wrangling data, this is one more tool for the box. And now I've written it down, I might remember it next time I need it too :)

Intro to RPi Pico with Mini Memory Game

2021-02-08T21:49:00+00:00

I am the proud owner of a Raspberry Pi Pico which is a dinky microcontroller from the lovely people at Raspberry Pi. I also have the RGB Pico Keypad Base, because my husband was buying something else from Pimoroni (3D printer filament, more on that another day) and we love them and ... here we are :)

I've tried to write down and link to the stuff I found useful along the way with this project, in case it helps you too. Well, no, really so I can come back to it next time I'm using one of these!

Hardware Setup

The picos ship without the legs soldered on, so that was step 1. There's some good info in the Pimoroni getting started tutorial for pico but I would also recommend the How to solder pins to your RPi Pico article from Tom's Hardware, it was a lot like having @biglesp actually here to help me!

Tl;DR put the legs into a breadboard and then solder the board on top, that way everything is aligned correctly.

Firmware

I used the custom firmware from Pimoroni because I needed the library for the rgb keypad. To add the firmware, connect the chip to your computer in bootloader mode (hold down the button when you connect and for another couple of seconds) and it will appear as USB storage. Put the firmware .uf2 file into the USB storage and it will reboot itself.

Tools

A lot of the tutorials suggest something called Thonny but I'm a software developer with accessibility needs, so I steered clear of that. Instead I used a neat tool called rshell which you can install via pip. It is designed for micropython and has a repl and some file handling features; I found it did everything I needed.

One thing to look out for is that the current release 0.0.28 seemed to be a little bit strange about both its default values and it had some problems with file uploads just hanging on particular files. However it's an active project and the pico branch worked like a charm. Clone the repo, check out the branch, and use r.py in place of your usual rshell command. This information will probably be outdated quite quickly though so it's just here in case it helps.

Start rshell with your pico already attached:

rshell -p /dev/ttyAMC0 --buffer-size=512

When it starts, it'll tell you what it thinks your board is called, probably pyboard, in which case the files on the device will appear as being in the path /pyboard/. The RPi Pico runs main.py when it boots, and you can copy it up with a command like:

cp main.py /pyboard/

The Raspberry Pi Pico DataSheet has some excellent information about getting set up and connected, and also has some good examples. It looks like a serious document but the "Say Hello World" and the "Blink an LED" section should work with any repl and any Pico.

It's Ctrl+x to exit the rshell repl. It does helpfully point this out when it starts, but of course that's not the moment when you need the info.

Actual development cycle

I find with a lot of hardware projects that I can get to "hello world" quite quickly but then there's a whole cycle of ejecting the disk or attaching to get debug output that makes the feedback loop pretty tedious. With rshell I am using this one-liner after each time I've saved my mydemo.py file:

rshell cp ../pi-pico/mydemo.py /pyboard/ && rshell repl '~' import mydemo '~'

Take the '~' off the end if you want to keep the repl active and watch the debug output.

Mini Memory Game (The Actual Software)

This bit is the most fun because it's when the lights light up and there are buttons to press :)

I made a game! It works like this:

A pattern of lights is shown for a few seconds
The lights are shuffled around the board
The user presses pairs of buttons to swap lights over until the lights match the first pattern
You win! Press the flashing green button in the corner to get the next level

Posting all the code into a blog post doesn't seem that useful so instead, check out my GitHub repo of fun RPi Pico things.

There are a bunch of other resources that I found useful:
- An excellent demo from Pimoroni but hidden in a micropython directory so look closely!
- Also from Pimoroni A library with docs and examples
- Everything about RPi Pico on Tom's Hardware, that's a link to the roundup post and I had 3 or 4 tabs open at one point, it is all excellent.

This Raspberry Pi Pico is a very different beast from the traditional approach of a tiny linux computer, and is more like the ESPs that I've used before. But they're quite approachable and MicroPython is fairly friendly to code with as well, so I'm looking forward to hearing what everyone is making!

Lab Book Meets Bullet Journal

2021-01-31T21:39:00+00:00

I enjoy a productivity hack as much as the next person, and I'm often asked about my setup. My background is in engineering (I hold an MEng in Electronic Engineering from York) and recognise the legacy of the lab book in much of what I do. If you're new to the lab book concept, check out this excellent article on how to maintain one properly. My practice breaks a LOT of the lab book rules and is also heavily influenced by the Bullet Journal trend (if you have time to spare, the instagram #bulletjournal hashtag gives a prettier introduction), which is a way to combine the benefits of productivity and processing information by writing it. Today's post gives a quick tour of my process.

Lab Book and Daily Log

The lab book is supposed to be a running record of what you did, whether that's a phone appointment, experiment setup, results, things you were thinking about, or anything else. I can't lay my hands on my last "official" lab books but even as a university student I know there were many colours of pen and the occasional glittery sticker!

In bullet journal language, this is a "Daily Log" although I chuck everything in there including meeting notes, plans, whatever. Bullet Journalling has its own rules about the bullets to use but I was already using an adapted (err, probably misremembered!) version of DashPlus so I kind of do my own thing. As long as you have a system and it isn't confusing to you, then it's fine.

My symbols can be summarised as "todo", "done", "info" and "todo but captured somewhere more useful than here", and that seems to cover most things. The main advantage is that I just stick everything on the page in front of me, with the pen I'm holding, and then work the rest out later.

Bullet Journal Additions

First of all, I'm using an A5 filofax, which is all sorts of no-no for a lab book! I love it though, it's everything I liked about an A5 notebook, but with added printables and none of the pain of changing to a new notebook. And yes, I own the special hole punch. I punch scrappy bits of paper, handouts, printed things, leaflets ... it's so good.

I keep a few long-running pages, that sit in sections away from the noise of the daily log. One is a list of content ideas for this blog (suggestions always welcome). Another is a list labelled "people" where I capture the things to say to a particular individual when I next meet with them. It saves interrupting people that you have a sync with anyway, and also saves forgetting things.

At the back of the planner I have recently added a "Projects" section, keeping track of the bigger things that have several ingredients, and will run over several weeks. This is something I've had in mind for a while but it's too soon to say if it's helpful. I'm also experimenting with evergreen notes and I might not need both.

I do use a weekly spread but since each day is basically a list of meetings and/or tasks, and maybe a meal plan, I use a vertical spread. You don't see many of these but it makes sense to my brain! I sometimes make my own layout and then it has some custom weekly checkboxes with "phone parents" and cleaning tasks and things, which I find useful. I use the space to add all my appointments first (from work and personal calendars), then I work out my work priorities for the week and where they can fit in. No plan survives contact with the enemy but I like to try!

The shot above shows both the planner and the week layout so you have some sense of what I'm working with. I'm a dotted paper kind of girl although squares are good too! Sometimes I print other templates (the printer takes A5 paper which is handy) for lists but not consistently.

Weekly Reset

Each week (usually Sunday evening) I create a list for the week. It contains anything from last week's list that is still worth doing (if something gets copied over multiple weeks running, it's a great time to reflect on why, or if it should be here at all), plus all the things I captured in the last week. I leave a bit of space around it, new stuff often shows up here during the week as well and that's OK. I use a highlighter on the items I want to do first/next so I can quickly get started on Monday morning.

The other thing that I do (either Friday afternoon or in the Sunday evening slot) is write down the "shipped" things from the week just finished. Some of the work I do is very tangible and you can count the blog posts or pull requests - but quite a lot of it is planning, talking, connecting, listening, and it's important to remember that this is all important too! I make a quick list of what was done in the week, and for any big achievements, I transfer them to my "scrapbook". The scrapbook is a text file in cloud storage where I keep a month-by-month list of things I'm really proud of. It really helps at annual review time and also when I need to remember when things have gone well (usually in a moment where that isn't the case).

With last week's "shipped" items, a fresh list for the new week, and a weekly spread - I'm ready for Monday!

What about Digital?

I use a digital calendar with reminders for everything that is time-sensitive, such as which night the bin goes out, and my work calendar is also digital. I share a set of Evernote lists with my husband which are mostly for grocery shopping and hardware store shopping, so we can just check the list when we're in either of those places.

In theory I also use todotxt for personal stuff but at the moment a) I'm having problems with android apps and todotxt and Nextcloud and b) we're in lockdown anyway which really seems to have reduced the workload! So a post-it note in my planner works fine for now but it does have "fix todo system" on it :)

I haven't moved to digital note-taking although I'm not morally opposed to it, I just haven't found a system that made me want to jump yet. Pen and paper is so flexible/portable/reliable/accessible and I know I process and retain information better this way. One day, both the input mechanism and the abliity to search what I wrote will win me over I am sure.

What's your setup? I'm always on the lookout for more ideas and improvements, so if you would like you share your story, then I would like to hear it!

Digital Developer Relations: Past and Future

2020-12-23T09:24:00+00:00

I work in Developer Relations, or "DevRel", and 2020 was a strange year with travel restrictions that meant that the entire industry did everything online, or digitally. I see this as a very positive progression of the way we work as DevRel professionals, and I understand that might be controversial, but I'd like to back up my position so hear me out (and then let me know what you think!).

Digital is an obvious way to do DevRel. Where are the developers? On the internet, on StackOverflow and reading dev.to, chatting on Discord and Twitter. To meet all those developers, where they are at, is a developer advocacy superpower, not a second-best option for the year where we couldn't travel.

Digital DevRel is Inclusive

In 2020 I interacted with a whole lot of new people and new types of people than I normally meet in a year. People were joining "local" user groups from all over the world to hear about a specific topic they had an interest in. People were joining their first meetups or conferences because they can't travel in other years either and so this year they can be part of the crowd, just like every one else. My friend in New Zealand gushed to me about how many events were online and how much she'd enjoyed being able to access content that is normally locked up in a one-time-and-place-only offering. And no, recordings from things that are supposed to be in-person are not the same thing.

There is work to do on figuring out the event and content formats that work well, but I don't want to tell all those new voices that I've talked to that we aren't interested in them any more now we've got our "proper" events back. We all belong in this industry, including people who won't ever be at an expensive conference with me. As a side benefit, content delivered digitally often works pretty well asynchronously too. A recorded talk that was played back with a lively text channel alongside it is a decent conference experience AND a decent catch-up experience! Building content this way is inclusive across time as well, because someone new to this topic in two years' time can still access my talk as a first-class citizen at the moment they need that content.

The much more inclusive nature of digital working makes a difference for DevRel as a group of people as well as for the communities we serve. Without the travel component that is "normal" in these types of roles, many more excellent humans could contribute to our efforts to reach and support more developer communities. We wouldn't have to change roles if life brings us reasons to travel less or work more flexibly.

Digital DevRel is Impactful

Without the noise and stress of travel, my personal experience was that I had a very impactful year. I spoke at digital events and the preparation time was the same but there was no jetlag/travel overhead and I was able focus much more and create more thoughtful and meaningful content, code and support. As a side effect I was much more available to my team and my communities, and more effective in what I could produce.

The digital world enabled a new depth to connection. I would interact with someone at an event, who became part of my network on Twitter, joined a Twitch stream, participated in other events I was at ... I genuinely felt that I built more valuable connections this year than I do when I'm standing by a table on the other side of the world giving out t-shirts, because of the echoes of the digital connection and how we can surface ongoing touchpoints on those platforms.

Digital interactions are built for scale - StackOverflow is a great example of one type of interaction; Twitch is a great example of a totally different type of interaction. This year of being forced to think about designing content for new platforms I hope will encourage us all to keep thinking hard about this stuff and keep on learning.

Written content is the gold standard of how to share information. Words need relatively little bandwidth to deliver, and can convey a lot of information; this really matters to a large portion of the world. Words can be read aloud, translated, or simply displayed bigger and with better contrast ... the increased reach because of how versatile words are should never be overlooked. Perhaps most important of all, we all know how to scan written text to find the piece of the puzzle that we need. Developers in particular are very advanced skip-readers and will pick-and-mix bits of three different tutorials to fill the gaps they have in some completely unrelated application they are building. If you want to reach developer communities, all of them, everywhere, now and in the future: begin with the written word. That didn't change this year and isn't changing any time soon.

Digital DevRel is Intentional

There's no substitute for in-person connections, and we can and should still come together on occasion. But let us be more intentional about when we choose to have many people travel to one place. The world is changing and many of us have made some conscious lifestyle choices (when did you last use a disposable plastic carrier bag?), it is time to bring that into our work lives too. A lot of what we do can be delivered asynchronously - the in-depth conference workshops in particular would be better as a distance learning series with some real-time support available.

There is so much that DevRel can learn from established open source project practices. Communication is one - mailing lists, commit history and IRC channels can cover all the knowledge transfer that are needed, and they all scale up to include as many people as they need to. It allows people to contribute to the conversation with a much lower barrier of entry than flying lots of people to one location. Let's make the most of the in-person events, but let's make fewer of them and find ways to connect to one another and our communities beyond the patterns we fell into by default, that we inherited from sales teams who only wanted to reach the top brass. Developers are today's decision makers, and that's why Developer Relations is a rapidly growing profession.

Developer advocacy is about meeting developers where they are, and enabling them to reach their goals. Digital is how we reach the developers hanging out "on the internet" and I am looking forward to more inclusive, impactful and intentional encounters with all of them in the future.

What's New in OpenAPI 3.1

2020-12-14T09:54:00+00:00

With OpenAPI 3.1 coming "soon", I gave a talk at APIDays Paris about what to expect. But I'm a great fan of the written word as reference so here's a written recap of what's in the next OpenAPI release.

Top features:
- Compatible with JSON Schema 2020-12
- Webhook support
- Many other minor improvements

Version Numbering

As of OpenAPI 3.1, the OpenAPI project doesn't follow semantic versioning. That sounds like a totally irresponsible decision but it actually makes some sense for a standard where every API description clearly states the version of OpenAPI it relates to. Also, they don't let me make the rules, sadly! It's not a major release, but some things had to be undone and redone to accommodate JSON Schema.

JSON Schema 2020-12

This is really big news, so much of OpenAPI is used alongside JSON Schema and the 3.0 release was quite a long time ago and didn't quite accommodate everything that makes sense today in JSON Schema. This post won't cover what's new in JSON Schema itself but I'll try to round up the main points from the perspective of OpenAPI users.

First of all: types can now be arrays of types, so something could be of type [string,number]. The available types also include null, so more common will be [string, 'null']. This does impact on existing OpenAPI documents as from 3.1, the nullable keyword is removed - use the array of types and one of the types is null.

parameters:
  - name: friendly-label
    in: query
    schema:
      type:
        - string
        - 'null'

OpenAPI 3.1 is getting support for the examples keyword within a schema, allowing an array of examples. Often the array only contains one element though. The original singular example keyword is still valid, but examples is recommended and if both are present then examples is preferred. Be aware that this array of values is in contrast with the other use of examples in OpenAPI, in a MediaType content object, where the examples field is a map with string keys. Confused? So are we. There is a post all about OpenAPI examples from Phil that explains it all!

A couple of other things are coming in from JSON Schema into the OpenAPI schemas. First of all: arbitrary keys are allowed in a schema; OpenAPI has relaxed its constraints on which fields can appear where to accommodate JSON Schema format objects. Also it's possible to $ref to an object, then put keys alongside it which are considered in addition to what is defined in the component.

content:
  'application/json':
    schema:
      $ref: '#/components/schemas/style'
      required:
        - hue

None of these changes are huge, but having the ability to use JSON Schema within OpenAPI is brilliant for anyone wanting to use the two together so to have this commitment to supporting the ever-improving JSON Schema is excellent news.

Webhooks

I'm totally biased because I proposed this feature. I couldn't believe that OpenAPI didn't already support this common use case and it took time to realise that it wasn't that I didn't understand how to do something - that thing really was outside what OpenAPI 3.0 expected APIs to describe.

OpenAPI 3.0 does have support for callbacks so if the user should make an API call, supply a URL, and stand by for incoming HTTP requests to that URL as a result, that's already supported. This is ideal when an endpoint asynchronously returns data, or for some situations where an API call "subscribes" to an event and gives a URL to send things to.

OpenAPI 3.1 goes a step further and allows webhooks which are incoming HTTP requests that are in response to some external event or stimulus. A great example would be if you've ever linked anything to a GitHub push event, or an incoming order/payment/message (and I've been working for a communications company for a few years, so you can immediately understand how I got entangled in this). The webhooks are described a LOT like the existing callbacks are, and indeed both are much like the existing request descriptions, so I hope this change will be easily adopted by everyone who has a two-way API like this.

The new webhooks keyword is a top-level element, alongside paths. There are also changes to the required fields: OpenAPI 3.0 required openapi and info and paths but in OpenAPI 3.1 only openapi and info are always required, but the document must also contain at least one of paths or webhooks or components. This is brilliant because it allows API descriptions to contain only outgoing API calls, only incoming webhooks, only components that might be referred to by other documents, or any combination or all of these - and still be valid in its own right.

Anyway, back to the webhooks.

webhooks:
  inbound-sms:
    post:
      operationId: inbound-sms
      requestBody:
        content:
          application/json: ...
      responses: ...

Within the webhooks section, each incoming "thing" has a key (such as inbound-sms in the example above) and then it goes on just ... looking like a pathItem, because that is all it is. You don't need to specify the URL path that the webhook will come in on (often the user can nominate that anyway), just explain what will arrive and you are done. Oh and related to this: pathItem is now allowed in the components section and you can $ref to a pathItem from either a path or a callback or a webhook.

Fancy a sneak preview of what the webhooks will look like when the tools get support for it? Redoc already has preview support if you use x-webhooks in your 3.0 OpenAPI documents! I mean, it just looks like really competent API documentation but that's what we need here :)

A note about webhooks and callbacks. Quite a few endpoints could be considered as either a webhook or a callback, and I have already started getting questions about which to use. It probably doesn't matter but if there is no preceeding API call that the callback is a response to, then it's definitely a webhook. Where there is some previous API call with a URL, then it's probably up to you how you want it to work. For example at Vonage the configuration for where to send events for an incoming Message is at the application level, do you do that with the Application API - but I'd rather have the API description with the incoming Message webhook detail shown next to the send Message API call, in the same document, tagged and grouped together in the Message API description. The webhooks keyword gives you the flexbility to approach this as you will.

Small But Perfectly-Formed Upgrades

There are so many small things that have been added in OpenAPI 3.1 but I'm picking my favourites to include! As with every *.1 release, there are things that seemed like a good idea for the *.0 release that can now be tidied up a little now we've all tried them out in anger, it's a good thin.

I'll start with the one I want to implement immediately (or as soon as the tools allow): $ref can now have summary and description as siblings, and they override any existing fields on the referred-to component.

paths:
  /items:
    post:
      parameters:
        - $ref: '#/components/parameters/item'
          description: The specific item in question

There are a few minor things in the info section as well:

Inside info, you can now have summary alongside the description field. Both are optional - title is still required.
In the license object you can use an SPDX code in the new identifier field instead of a url alongside the required name field if you prefer

Finally, paths are no longer required to have responses field for every endpoint. This is useful when an OpenAPI document is under construction because it means that it will validate even when you're still only sketching out the endpoints that the API definition will include.

Bystanders Guide to Repository Maintenance

2020-12-04T20:20:00+00:00

Today's post is about bystanders and open source. We sometimes have anxiety about doing things on "other people's" projects but the thing about open source is that those are not "their" projects, those are "our" projects, and they only exist because people other than their nominated gatekeepers chip in now and then.

Bystanders

A "bystander" is someone who is nearby but not directly involved in something. There is also the "bystander effect" which basically says that as individuals we are more likely to help out with an emergency if we're the only person present. If several others people are also near enough to help, none of us do.

On the open internet it's easy to feel like you're in a crowd. Surely someone else will see this, help here, fix that thing. The thing is, when you're looking at a particular project ... most of the time, there is nobody else around. That one small thing? Is up to you.

Getting Started in Open Source

There are many articles on how to get involved in open source, I've written at least one of them myself, but this is about micrcontributions. The act of giving a moment of attention to every project whose path crosses yours. This post has specific examples that I hope seem approachable - so have a read and then look out for opportunities to lend a little of your skill, time and energy to these open projects that are powered by many small actions by many passing individuals!

Add information to existing discussions

When you search for an error message and find it in an open issue on a project, go ahead and read the discussion thread. Often the answer is here or in a linked pull request (in which case you're about to learn how to build a library from source. Please prepare to open your own pull request if any updates are needed to the installation instructions - most project maintainers aren't installing their projects from fresh on a regular basis so this area of a project can easily get neglected!). If you can, post about the software versions you're using and clarify if any workarounds helped you.

The "me too" posts are tempting but they can really add to maintainer notification fatigue. If you are also experiencing an existing issue, or would also like to see a particular feature implemented, go ahead and use an emoji reaction to indicate this rather than adding an new entry to a thread. Comment if you have new information. Not opinions. Information.

Answer questions

I work in Developer Relations, which requires troubleshooting of any of our products, in any tech stack, with any related dependencies, at any time. This makes me an expert in error messages and suggesting that running the dependency manager command again might help! In fact I think that as developers we can often help one another without any "specialist" knowledge needed. To this end I like to read the first few issues on every project I see and check if there's anything I can help with. Even if I don't know the library at all, "try redirecting stderr to stdout and see if you get any output" can go a long way and "unstick" someone.

It's not your project, and you don't have specific knowledge, but it's likely you have some related knowledge so share it if you do! Even if it's just a "first response" to ask for the error message, what platform they are using ... you have saved the maintainer a little bit of time AND started the clock ticking on waiting for a response. It's more valuable than you can know.

Informal bug report triage

Open source projects grow very unevenly. Usually they start as just one or two people making a thing and sharing it. The early adopters are probably also contributors as well, and that works. Then there's a whole middle size of project which is just ... fine. Quiet, manageable. At the other end of the scale we've got huge, high-profile projects with (hopefully!) a team of maintainers. But before that I see quite a lot of projects that have a LOT of users, of varying levels, and a very small number of maintainers who have other jobs and commitments. At all stages, but especially this last one, a little help goes a really long way. It's very easy for maintainers to be so swamped by issue chatter that they can't ever get out of the backlog of communications to work on something, and without being negative at all, bulk handling of support tickets may not be the best use the universe could make of their time and skills!

To help with this situation, take another look at the issue list, but this time dive a little deeper and look at reported issues that genuinely seem like bugs. You're going to triage by looking at the bug report, trying to reproduce the bug on your system, and then reporting whether you can reproduce or not (including your system information). You get bonus points if you can also write a test for the failure, document a workaround, or even offer a patch :)

For example, I maintain a project but I'm a Linux user and my co-maintainer has a mac. The Windows bugs are going to be there for a long time unless someone else chimes in - just a "looks OK for me on Windows 10" would help us a lot. Many projects also use a tag like "help wanted" for issues where another pair of hands would be especially welcome so if you have a project you'd like to support you can look out for those as an easy way to lend a hand.

Be the myth you want to see in the world

It's a common misconception that open source software is made by noble individuals for the good of the world. In fact, it's cobbled together from a whole raft of passing comments, code copied from another website, and one-line contributions from random and unknown humans. Open source is a movement, and it is strong when we all participate in one way or another. You don't need to become a full-time maintainer to make a difference; make microcontributions part of your normal practice and be part of the strong future of open source. Even the smallest contributions help every project to inch forward, and lets the maintainers know there is someone out there!

Making SDKs a Value-Added Experience

2020-11-16T10:18:00+00:00

I've been working with SDKs quite a bit this year, simple ones that wrap a bunch of API endpoints. The best bits of these SDKs isn't just the delightful developer experience of using the clunky endpoints with the elegant SDK wrapping them. I mean, it is, but I already wrote about consistent and Delightful SDKs for Nordic APIs; today I want to cover the some of the "extra" things that make a big difference in the APIs.

Provide Escape Routes

Don't force your users to be all-or-nothing with your SDK. A great example is with APIs that use access tokens of any kind; I work with a few that use JWTs. As a nice helper function to the user, the SDK will generate the JWT and send it with your request, which is great. The extra value is added when the user can also generate a JWT for their own use; you expose the functionality that the library uses so that developers can use it themselves if they want to.

Save Repetition

If there's a particular piece of functionality that every consumer of the API is going to have to do, however small or well-documented, then add it to your SDK. I spent quite a lot of my life answering questions about how to handle a particular signature check to improve security on our APIs. I updated documentation, improved it, added examples .... once we put the feature into the SDKs, I don't think I've thought at all about this feature since!

Helpers

Don't be afraid to go a little beyond the boundaries of just a wrapper for an API call in what you do. For example, some of our APIs accept quite complicated JSON structures in the body of the requests. Every programming language can produce JSON but some make it easier than others. In PHP or JavaScript, you can pretty much throw some sort of data structure at the JSON function and something will happen. But when I came to create a Golang SDK, I found that things are not quite so simple in strongly-typed languages. Creating defined data structures that could be quickly used and then marshalled to JSON made the library much quicker to get up and running with - for me, as well as for the users I hope will follow and make good use of it.

Delightful but Unobtrusive

Keeping the presence of the SDK as a helper for a particular tech stack, but not a requirement, is a great aim here. SDKs give us tech-stack-aware functionality, and smoother experiences with autocomplete and supplementary dependencies included. But developers may well need to get away from the beaten track in ways that you can't anticipate so I'll always advocate adding small pieces but keeping the responsibility with the developer on choosing to mix in only the pieces that make sense for her project. Hopefully the examples here give you some ideas for your next library, too!

DevContainers, VSCode and GitHub Codespaces

2020-11-15T13:31:00+00:00

I've seen quite a bit of chatter later about the GitHub Codespaces feature lately, and I have beta access to it so I thought I'd share my first impressions and make some notes of what I've tried that might be useful to others.

First up: you can prepare for all this without getting access to Codespaces because it's basically VSCode in the cloud. You can use your local VSCode and be ready to go when GitHub is :) Also since Codespaces is a chargeable feature, I expect it will make sense to run on your local platform if you have one and in the cloud if you're on another device.

DevContainers Setup

Get the Remote - Containers extension for VSCode installed to begin. This uses a standard devcontainer configuration to describe the docker platform that your project runs on. This is useful when you have a lot of projects, a lot of developers, and therefore many platforms to manage. Now I've seen it, I'll be adding it to plenty of my projects!

To add this sort of setup to an existing project, go and rummage in the vscode-dev-containers repo in the containers folder for something that would fit your stack, or at least give you a starting point. You're looking for a .devcontainer folder that you will copy, wholesale, into your own project. This will contain some dockery things, maybe a README with specific instructions (although all the files seem to be really well commented which I always appreciate), and the all-important devcontainer.json file that describes to VSCode how the whole thing is going to fit together. Copy it in (or use the Remote - Containers plugin to do this for you), and configure as desired for your setup.

Once the devcontainer is set up, opening the project in VSCode will cause you to be prompted whether you want to re-open the project in the container.

Pro tip: the vscode-dev-containers repo README also has links to some "try it" projects for common stacks that I found useful.

VSCode Setup

There's another piece of the puzzle but again it's VSCode-specific: the launch configuration for your project. I'm not sure why it's described as "debugging" because I think this is how I make the project run. I'm fairly new to VSCode though so please add advice in the comments!

The launch configuration lives in .vscode/launch.json. This is a great way to share this information but check if .vscode/ is included in the project's .gitignore file - I would not normally advocate including proprietary-tool-specific settings in a repo but here we are! The simplest way to get started seems to be to press the "Debug" button without any configuration in place, and VSCode will hold your hand through the process.

With the launch config in place, you should be able to "Debug" your project and something should ... happen!

GitHub Codespaces

When you get access to the Codespaces feature, you'll probably notice the "Codespaces" item that appears on the top menu between "Pull Requests"/"Issues" and "Marketplace". You will also see new options popping up: The Code download/clone button includes the option to open the project in a codespace, and pull requests will also have a new option under their "Open With" button.

This will provide you with:
* VSCode opens in the browser with the project or pull request in
* The devcontainer in this project gets spun up (be patient!)
* When you "Run" our "Debug" the project, you'll get a notification when the server is ready. You can even Ctrl+Click to open the "localhost" addresses which are then rewritten to your new cloud workspace URLs

Considerations

The convenience makes the whole thing compelling and I imagine I'll have this set up on many of my own projects. However I feel a word of caution is appropriate here. Git is an open standard; GitHub is just one option for a hosted interface to go with git. No project should require use of a specific tool, even a lovely and widely-used one like VSCode. I'm a Linux user with accessibility needs; anything that dictates the tools I use locally is a non-starter (I can't imagine actually working with Codepsaces, it's just a nice way of including other people or quickly testing something from the train on a mobile device).

If you take one thing away, I'd love it to be this: Let's pledge to include these features in our projects, and to include the "standard" usage/install/contributing instructions too. It's a Docker file and some commands to run, no special equipment required.

Ubuntu Command for Mic Mute with Pulsemixer

2020-11-12T17:27:00+00:00

One of the major problems I run into with video conferencing tools is how many of them are not at all accessible. I'm a keyboard-only user so if I need to hover a pointer in a particular area of the screen and then click an icon to mute ... then I can't do that. And if you mute me when I join because there are a lot of people here, then I can't participate at all. To work around this, I need a one-liner to mute (and unmute) my mic ... so here it is.

pulsemixer --id source-5 --toggle-mute

Seems easy, right? There's a catch!

The source-5 identifier changes on boot. Check out your list of sources:

pulsemixer --source-list

Pick your mic from the list, run the toggle command so it's in your command history, and then repeat as needed. You could even wrap it up in a shell script, or as I do, hook it up to your streamdeck or other external input. It would be even better if you could automate finding your source .... I have this dumped in a comment in the code but I haven't dared to use it for real yet:

pulsemixer --list-sources | cut -f3 | grep 'UMC404HD 192k Multichannel' | cut -d ',' -f 1 | cut -c 6-

I will leave it here just in case it's useful to you!

Pro-tip: More and more of the video conference tools seem to be introducing support for muting and unmuting you when your system microphone is muted or unmuted. This is brilliant because people can still see that you actively unmuted to say something!

Keyboard-only Web Browser

2020-10-22T13:53:00+01:00

I haven't used a pointing device in over a decade, and I get a lot of questions about how I navigate the web using only the keyboard. The short answer is: Vimium and funnily enough, it's a productivity tool rather than an accessibility tool. Curious? Let me show you around!

Vimium is available as a plugin/extension for both Chrome and Firefox. Basically, you install it, and then press f.

The little yellow labels pop up on everything you can interact with. If there's anything that "doesn't work", that is almost always an inaccessible web element rather than a fault with Vimium. Yes, the web is a bit rubbish in places!

OK so you know how to click on links or input boxes, what else makes a productive web user? Here are a few tips:

Use j and k to scroll up and down gradually. spacebar goes down a page at a time and is good for low-friction reading longer form content.
Typing F rather than f opens the link in a new tab.
Knowing your browser-level keyboard shortcuts is probably also handy! Try ctrl+t for a new tab, ctrl+w to close a tab and ctrl+pgup/pgdown to move between tabs.

For working one-handed, whether that's a permanent thing or your other hand is currently occupied by either a beverage or a child, tools like this can be a real power up. And for me, two hands but I can't move them a lot, a tool like Vimium is life-changing. Whether you are in it for accessibility or productivity, install it and give it a whirl!

Combine Multiple Field Tags in Go

2020-10-21T09:46:00+01:00

I work a lot with JSON APIs in my Go projects, so almost every struct has some field tags on it to translate AccountID into account_id and that sort of thing. Then one day I needed to load data from config using mapstructure and it took a few attempts with the search engine to find the syntax I needed, so it's here for next time I need it (or in case you need it too).

Combining JSON and Mapstructure

An example truly is worth a thousand words! (This one is from the code that drives the neopixel shelf):

type LEDColour struct {
    Red   uint8 `mapstructure:"red" json:"red"`
    Green uint8 `mapstructure:"green" json:"green"`
    Blue  uint8 `mapstructure:"blue" json:"blue"`
    White uint8 `mapstructure:"white" json:"white"`
}

It turns out that the reason I couldn't find examples of how to combine struct field tags is because you don't! You just add each one required, followed by a space, and then the next, and so on. How simple, how elegant ... how Go!

Adding Tables of Contents to Markdown

2020-10-18T20:34:00+01:00

Once any document gets over a certain length, a table of contents can be useful. I have a favourite markdown table-of-contents tool but it usually takes me a trip to the search engine to remember what it is! (tl;dr: github-markdown-toc )

This tool generates a neat, nested table of contents based on the titles in the file you give it, and outputs it to standard out. My command looks something like:

gh-md-toc /path/to/project/README.md

Capture the output, put it at the top of your file, and you are all set! For example, a Go SDK project I was working on recently. It's such a small thing but it improves user experience and only takes a moment. When it's time to update, just delete what you have and regenerate a new one!

Package Webpages as Apps with Nativefier

2020-09-15T11:22:00+01:00

There are a couple of specific web pages I use in separate windows from my other browser tabs, so I can always find them. However this quickly leads to a too-many-browser-windows problem. Instead, I have been turning them into basic standalone applications with a javascript tool: nativefier and I thought I'd share my recipes.

First: the problem. I'm a keyboard-only user and that is less of a problem than you'd think for day-to-day things. I can drive a terminal, browser, text editor, and slack, which is basically all I need for 95% of my week. However I've recently started streaming on Twitch (you can follow lornajanetv if you'd like this sort of content in real time) and managing the stream as well as showing what I'm doing is .... basically I'm streaming a lot of alt+tab overlays!

What I needed was the ability to focus a window immediately from a command. I have a streamdeck, so I can hook the command up to a button. I had some success with wmctrl which can raise a window by title, but since I use multiple workspaces on my main monitor (workspaces do not span displays, configured with gnome-tweak-tool, so the second monitor holds all the stream-related stuff, and is not really on a workspace), raising one firefox window would switch workspaces as well. I needed a new approach.

Enter: Nativefier. It claims to package webpages as electron apps, and since I have OBS doing what I want it to with the window raising behaviour, I thought this might work well.

Package a Web Page with Nativefier

Honestly, this tool does what it says on the tin! I wanted to put my Stream Manager page into an app, so I installed the tool as a global command and then used this:

nativefier --name StreamManagerLJ --single-instance \
https://dashboard.twitch.tv/u/lornajanetv/stream-manager \
--internal-urls ".*?\.twitch\.tv\.*?" \
--inject inject.js

There's a lot going on here so I'll try to break it down.

Name: Use --name to give your application a name. This is useful as you'll want to refer to it later.

Single instance: By making your app single instance, attempting to run another one will simply focus the existing one. I added this because I was still having some workspace changes when I tried to focus the apps with wmctrl and it works brilliantly. As a side-benefit, I have one streamdeck button that either launches or finds this, regardless of what state it is in.

URL: The page to wrap in an app. Same-domain links will work inside the app, but for everything else you'll need to specify what is an internal URL.

Internal URLs: Add a domain or regex here (you can also send multiple of this switch) to tell the app which URLs should be opened inside it. Everything else will break out into your default browser. If you need to auth, you'll need to add the domains of those pages so that it happens inside the app.

Inject: You can include *.js and *.css files to add to the application. More on this later (even I am not putting a tangent this size into a bullet point!)

The command outputs information about where it put the executable - I get a StreamManagerLJ-linux-64/StreamManagerLJ that I can run. Then I can copy that executable somewhere more useful if I want to although I usually leave it where it is and then create a `.desktop`` file <{filename}/create-desktop-files-for-ubuntu.rst>`__ to make it executable, set its icon, etc.

Injecting Custom JS for Nativefier

One cool thing is that you can also add some scripts that aren't part of the official webpage. I spent some time trying to work out how to package Vimium, which how I use my browser from the keyboard usually. However, Chrome plugins are intentionally not included, presumably because it would get out of hand really fast! Anyway hopefully I'll figure this out at some point but for now ... a little hackery gives me all I need :)

In the stream manager, I basically need to be able to reliably focus the chat window. For everything else (which is rare), I will use my touch screen or pointing device since I can't actually stream completely keyboard-only so I plug in a wacom tablet for this anyway. Here's the inject.js that I created:

document.addEventListener('keydown', doKey);

function doKey(e) {
    switch(e.code) {
        case "KeyC":
            a = document.getElementsByClassName('chat-input__textarea');
            a[0].firstChild.focus()
            break;
    }
}

It's not much, but it's just enough to give me focus where I need it, very easily. I could also expand to add more keys and more selectors or something. Probably this will evolve over time, but if what I have helps you, then I want to share! Helpfully, the apps do come with devtools enabled so that can help with identifying elements and debugging things.

I came to nativefier as a workaround for a window management problem but I see it as a productivity hack as well, the community seems friendly and has lots of people using it for their most-used apps (trello, gmail) that they wish were desktop-native. The tool was easy to use and I can also patch it to extend it if I need to, which is always reassuring. Let me know what you package and how it goes!

Create .desktop files for Ubuntu

2020-09-13T11:21:00+01:00

Having created a couple of .desktop files lately to make launching apps easier, I thought I'd write myself a quick reminder of what I did and how it works. If it's useful to you too, then great! I have one application in a weird path, and another that needs a specific environment variable set for it to work.

Let's start with OBS. On 20.04 and because I'm using a Wacom Tablet, there is a magic incantation required when it runs. By creating a .desktop file for it, I can add that and run it from the usual launcher rather than needed a special terminal script or alias.

The file is ~/.local/share/applications/obs.desktop and looks like this:

[Desktop Entry]
Type=Application
Encoding=UTF-8
Name=OBS Studio
Icon=/path/to/images/obs-logo.png
Exec=env QT_XCB_TABLET_LEGACY_COORDINATES=1 obs

I also have this one pinned as a favourite so it launches on a Super + [number] keyboard combination!

Another example is the webpages-as-apps I've been packaging lately (blog post coming on that too I promise but tl;dr try nativefier!), the executables are in a weird path and since I expect to re-package fairly often, I don't want to have to copy things around all the time. No problem! The desktop file doesn't care, and puts them in my proper launcher regardless:

[Desktop Entry]
Type=Application
Encoding=UTF-8
Name=Twitch Stream Manager
Exec=/wild/long/and/unlikely/path/StreamManagerLJ
Icon=/path/to/images/twitch-logo.png

Hopefully now I've written this down, I'll remember how this works next time without having to come and look up all the instructions ... AGAIN :)

Sharing Screen and Camera from Android to Linux

2020-08-17T09:38:00+01:00

With all the video calls, twitch streaming and product demo work I've been doing recently, I've been building up a library of tricks for using my android devices within the video content I produce from my laptop, so I thought I'd share. I've worked remotely for many years but I think there's more of us not going back to the office any time soon, so strong video communication skills are important!

My setup:
* Lenovo Thinkpad X1 running Ubuntu 20.04
* Android Moto G6 mostly, although I have screenshared my Samsung S2 tablet as well when I wanted to draw something!

(I also have the Behringer UMC404 audio interface, meaning I can route the phone headset audio directly in to my computer but honestly my earlier solution of plugging from 3.5mm headphone socket on the phone into a basic bluetooth speaker and turning that right up worked almost as nicely! It only matters because I work for a communications company and sometimes need to demo phone calls)

Screenshare

I tried `scrcpy <https://github.com/Genymobile/scrcpy>`__ and it's been great so far. It connects over USB and uses adb tools (that I had installed anyway). I have used this approach in a couple of different ways:

First: I wanted to draw something, and a stylus and a tablet is the easiest way for me to do that. Since I only want to use a local application on the tablet, I simply put the device into airplane mode to avoid any notifications, embarrassing or otherwise! That worked well (this picture from a stream about git that I did for work).

The other time I use this approach is when I'm demoing something that is actually on my phone, such as the new WhatsApp Messaging Features from Vonage (I work there, so I am supposed to be able to operate this stuff and show it to others). There are desktop apps for most of these things but I think it's nice to really have the phone in the video.

Do put the phone into "do not disturb" mode before you start - I use a work phone that has hardly anything installed on it so I don't get notifications on that, which helps!

Android Phone as a Webcam.

I've been using DroidCam who seem to be much better at making android apps than they are at making marketing websites! It connects over USB (using adb again) or over wifi. I found that some days only one or the other will connect but once it's stayed connected for 30 seconds then it has usually been rock solid for me - although it does seem to like to be reinstalled if Ubuntu has had updates!

I have been using it as a second camera, in addition to my built in laptop webcam, so that I can include my hardware projects in my Twitch streams. If I work on software, then I only need screensharing but for something that is not on a screen, the second webcam is really handy. At some point I might get an actual webcam, but I started doing this at the start of the pandemic and getting a webcam was pretty difficult - and the phone works fine for the hobby level stuff I'm doing.

Again, this one is my basic work phone, so it doesn't get much in the way of notifications or anything which I think helps.

Adding Android Devices into the Video Mix

It takes time to figure out the tools and set them up in advance of whatever you need them for, but I've found it to be worth it. I use the phone with a gorillapod and basic phone tripod mount and I've found it useful for sharing many different things (including a storybook at bedtime for the little ones that I can't put to bed myself right now!). Let me know what you're doing with your extra devices in your video setup? I would love to hear!

An Intro to Your ESP8266

2020-08-08T20:54:00+01:00

I've been using a microchip on my neopixel shelf project but I kind of set it up, changed one thing, and didn't look at it again until this week. I'd like to have my neopixels do something more interesting than one solid colour or another, so I needed to remind myself of what this thing is and how I do it. These are my notes, for my Ubuntu platform, but you're more than welcome to them!

Start with Firmware

The first thing you need is some firmware to base things on. I used a NodeMCU build; there are a few ways to obtain one of these but I used the charming donation-driven site https://nodemcu-build.com. You pick your options, supply an email address, and wait. The link to your custom build will show up in a few minutes. The chip doesn't have a lot of resources, so the custom build means you can pick and choose what to include. Here's what's in mine:

color_utils
file
gpio
http
mqtt
net
node
sjson
tmr
uart
wifi
ws2812
ws2812_effects

(those last two entries are the neopixel bits!)

While you wait for the build to arrive, connect the ESP8266 to your laptop and install esptool, probably from pip.

Ready? Cool! Upload the build:

esptool --baud 115200 write_flash -fm dio 0x00000 [build file from earlier].bin

Write Some Code

These little chips have a few options for code but I'm using Lua which I hadn't seen before, but which is pretty approachable. I like it, but I am not sure I will need a Lua category on this blog any time soon!

I began with a file named init.lua - by default the built will pick this up when the chip is powered up or reset.

To put the code onto the board, my command looks like this:

nodemcu-tool upload --port=/dev/ttyUSB0 init.lua

The light on the back will show "chatter" while the transfer is happening, then you can press "reset" to power cycle the board and see your code in action!

Debugging Tips

I needed a lot of these. Start with the obvious and keep your code in source control, only change one thing at a time!

You can add print('hello world') commands in your code, and you will them when you attach to the running device with screen, like this:

screen /dev/ttyUSB0 115200

If you're unfamiliar with screen, you may find yourself stuck in this screen! Press Ctrl + a (all screen commands start that way) and then k, followed by y to confirm.

Pro tip: You need to exit screen to be able to use the serial connection to upload a new version of code!

I'll write about my actual application another day, there's enough going on there that it needs a post of its own - but dumping my stuck-in-a-handy-textfile commands here will make it easier for me to find them next time. If you're doing a similar project and refer to these instructions, I would love to hear what you're building!

Creating a Simple OpenAPI Spec

2020-08-06T13:44:00+01:00

Having API descriptions in your workflow is a game-changer, but how do you get from nothing to a functioning spec in the first place? I recently made a very simple spec for a very simple API, and thought it might make a good example for anyone looking to create a spec for the first time. OpenAPI specs are very verbose so this very tiny API makes quite a manageable example!

For the impatient: if you just want to look at an example without any additional context, there's a spec here https://github.com/lornajane/flask-planets-and-webhooks/blob/master/openapi.yaml

About the API

I've been using a very trivial API example project in a few of my recent talks on API-adjacent topics, with just a couple of endpoints. There's a blog post about the Planets API but in short you can get a list of either one planet or many, with requests like:

GET http://localhost:5000/planets
GET http://localhost:5000/planets/3

Now you have a decent grasp of what this thing does, I'll start the description.

Create File, Configure Tools

I'll be using YAML, you can use JSON instead if you prefer. My file is called openapi.yaml which is a reasonably common file name but you can name your file what you like.

I never attempt to create a file like this without some sort of validation tool. Something like Stoplight Studio will do this for you, but I mostly use my usual editor so I'm using Stoplight's standalone open source linting tool spectral combined with watch, something like this:

watch -n 3 spectral lint openapi.yaml

Spectral is fairly Opinionated (with an intentional capital O) but it's a decent starting point.

With this in place, I started adding a few lines to the file:

openapi: 3.0.3
info:
  title: Planets and Webhooks Demo API
  version: 0.0.1
  description: Simple flask-backed API showing some example API endpoints and with webhook debugging features.
  contact:
    name: Lorna Mitchell
    url: https://github.com/lornajane/flask-planets-and-webhooks
    email: github@lornajane.net
servers:
  - url: http://localhost:5000
    description: Sample project running locally
externalDocs:
  description: GitHub project for the code and information about this API
  url: https://github.com/lornajane/flask-planets-and-webhooks

You'll probably still see some validation errors, in this version of OpenAPI 3.0.3 it's required to have a paths: item at the top level, but you get the idea of how the start of the file should look.

Add API Endpoints

This API has two endpoints, so it's a nice, small problem space to try to describe! The /planets route returns JSON, an array of objects where each object describes a planet. The /planets/{id} route returns just one object, but it is identical to the object returned by the collection. OpenAPI supports a special $ref field for reusing definitions across different parts of the spec. You can use it to structure your spec in a more readable way - but its superpower is in making sure that things that are the same are only defined once, making it easier to keep everything consistent and maintainable.

I prefer to start with an endpoint without any reuse, then refactor to components later. As an example, here's the /planets/{id} endpoint in its first iteration:

paths:
  /planets/{planetId}:
    get:
      operationId: onePlanet
      summary: Fetch one planet by position
      description: Get the data relating to one planet
      parameters:
      - name: planetId
        in: path
        required: true
        schema:
          type: number
          example: 4
      responses:
        '200':
          description: Planets in a list
          content:
            application/json:
              schema:
                type: object
                properties:
                  name:
                    type: string
                    description: Name of planet
                    example: "Saturn"
                  position:
                    type: number
                    description: Order in place from the sun
                    example: 6
                  moons:
                    type: number
                    description: Number of moons, according to NASA
                    example: 62

I did say it was verbose! The value is in the detail though - every tool that consumes this description has all this information and examples available.

I know I'll need the actual planet object again though, so here's the same sample again, refactored to put the planet object into components as a reusable element:

paths:
  /planets/{planetId}:
    get:
      operationId: onePlanet
      summary: Fetch one planet by position
      description: Get the data relating to one planet
      parameters:
      - name: planetId
        in: path
        required: true
        schema:
          type: number
          example: 4

      responses:
        '200':
          description: Planets in a list
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/planet"

components:
  schemas:
    planet:
      type: object
      properties:
        name:
          type: string
          description: Name of planet
          example: "Saturn"
        position:
          type: number
          description: Order in place from the sun
          example: 6
        moons:
          type: number
          description: Number of moons, according to NASA
          example: 62

With the reusable component ready, have a look at how the collection looks using it:

paths:
  /planets:
    get:
      operationId: allPlanets
      summary: List all planets
      description: Returns a list of all the planets that are stored in the system.
      responses:
        '200':
          description: Planets in a list
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: "#/components/schemas/planet"

It's a little different as it's an array of objects, but we can reuse that single object as an array item.

Documentation as Output

There are so many things you can do with an OpenAPI spec but a very human-centric one is to generate documentation. There are a few options for this, again Stoplight Studio has it built in, but I mostly use open source tool ReDoc - their 2.0 release isn't official yet but it is great! Importantly, the project was very receptive to, and quick to fix, the accessibility issues I logged.

Generating some documentation as you go along can help show how things are going when you come to describe your own APIs.

Final Touches

If you're playing along at home, you'll notice that spectral is still having Opinions! That's because we haven't added tags yet. Tags aren't that useful on an OpenAPI spec this size, but for larger APIs they are an excellent innovation. The previous yaml example showing the collection endpoint has a tag, and to finish, all I need to show you is the tags: element in the spec:

tags:
  - name: planets
    description: API containing solar system data

For tags, parameters, respond fields, endpoints, and really everything else, the descriptions and examples are at least as important as the field name and data type information! Not all of us are talented writers but the best specifications are those that do include these extra-value items - the improved Developer Experience (or as I like to call it "developer delight factor") is absolutely worth it!

Let me know if you found this post useful or if there's anything more detailed you'd like to see covered. I'm still figuring a bunch of these things out myself but I'm happy to share :)

Using Environments in Postman

2020-07-23T16:55:00+01:00

I'm using Postman more lately and showing off the Environments feature really often. I really like this feature for giving easy ways to handle values I use frequently, but it seems to be a bit of a hidden gem, so I thought I'd write about when I find it useful, in case you do too!

Environments are essentially a collection of variables. Most of mine contain:
* API credentials, either a token or a username/password
* Base URLs so I can change which platform I'm targetting

The controls for Environments are in the top left, no down a little, that's it! A dropdown, an eye, and a settings cog :)

You can set up different Environments for test/live platforms, for different geographical regions, or whatever you need. I use "Lorna Nexmo" for my live Nexmo credentials and API base URLs, and "Prism" for when I'm using prism as a local mock version of an API. Switching between the two becomes very quick and convenient, and I find it a real timesaver!

Once you have variables set up in an Environment, it makes it very easy to use them whenever you need them. This is great because I don't have to keep looking up credentials every time I want to add them to a request. I can just type {{ and see what autocompletes!

Such a simple feature but very handy. Also, see in the screenshot it previews the values? I can use the "Prism" Environment so that I don't display my actual credentials when I'm streaming/speaking/blogging, and then just flip it in the Environments dropdown. Very useful!

Planets and Webhooks: a simple Flask app

2020-07-22T14:50:00+01:00

As a Developer Advocate for an API company, I spend a lot of time talking about APIs and webhooks and HTTP in general. Recently I've been focussing on HTTP tools, but I really wanted a very simple example API that I could use that would return JSON but really let me focus on the tools, not the API. So I created a "Planets and Webhooks" API with a couple of GET endpoints to return JSON data, and another endpoint to receive and log incoming webhook data.

Planets Flask App

The project is on GitHub: https://github.com/lornajane/flask-planets-and-webhooks so you can also grab it yourself to use when testing out any of the HTTP tools or getting to know webhooks that I covered in my talk (this probably makes more sense if you've seen one of my HTTP talks/streams, but hopefully still can be useful by itself).

It very simply returns a little bit of info about a planet. For example:

$ curl http://localhost:5000/planets/5
{"name": "Jupiter", "position": 5, "moons": 79}

You can use it to check out how different HTTP clients (curl, httpie and friends) handle JSON, or how tools like jq or fx can help.

The repo also includes some of the tools I usually mention: https://github.com/lornajane/flask-planets-and-webhooks/#http-tools.

Debugging Webhooks

Sometimes it is useful to have an endpoint that can receive and correctly acknowledge incoming webhooks without doing a whole lot else. This application has a simple GET/POST parameter logger on the /webhook endpoint. You can use Ngrok to allow an external webhook service such as Zapier to access your local running flask app.

Fun With HTTP

Sharing the code I used in my demo is mostly useful for people who saw the demo but if it's useful to you too, then great! I'm not the world's finest python developer (yet) but I had fun building and using this application so let me know if you do too?

Neopixel Shelf

2020-07-18T17:47:00+01:00

In these days of video calling in place of any other social contact, I've seen a lot of excellent video backgrounds. Perhaps I'm old-fashioned but I went the traditional route, and set up my computer close to a wall (so you can't see the mess in the room) and added a nice pinboard behind me. I looked a bit flat against the wall though, so I did what anyone else would do ... and built a handmade wooden shelf with neopixels in it!

According to Instagram, I published pictures there in April, but I realised when streaming (about controlling my LEDs while streaming, obviously!) that I had never written the blog post about this project. So, here goes :)

I started with a leftover bit of aborted piano stool project. This is lockdown, so buying wood is not easy and I love repurposing things anyway.

I pulled out (or cut off) the dowels, then cut the piece into two wider pieces (I needed to join bits to make the length of shelf I needed) and faced them. To join, I drilled into the ends and used dowels and glue to hold things together. Using dowel punches makes this step easier but I still had to re-face the finished product as it didn't perfectly align.

Ours is a rather period home (technically it's Edwardian) so everything has twiddly edges! I routed a pretty edge onto my self, and then also routed a groove near the back of the shelf for the neopixels to sit in so that their light, but not really they, can be seen.

With the last of the remaining top and some shelf supports cut by my handy bystanding CNC owner, I was ready to assemble the main part of the shelf. Using a solid back like this makes everything super sturdy and also gives me an easy way to hang the shelf on the wall with keyhole hangers.

Looking back at these photos reminds me that the workshop was really chaotic earlier this year, we had a lot of new tools delivered literally as lockdown was announced, but we were also storing a bunch of things for family who had just moved. The space is much easier to use now, as you can tell from these shots, things were quite cramped at one point!

Anyway, add the neopixels and a microcontroller (I'll write about that bit in another post, it's an ESP8266 running Lua), add support for sending different colours over MQTT from the streamdeck on your desk and here you have it:

I don't write much about my woodwork or craft projects on my blog these days, it's very software-oriented, but since I'll also be sharing more of the software side of this project I thought I'd start by sharing the shelf itself! Do you have a low-tech background for videos? I would love to know :)

Add a Screenshot Button to Streamdeck with Golang

2020-07-06T10:09:00+01:00

I'm the proud owner of a Streamdeck XL but as an Ubuntu user, the tool support isn't great. There's a Python library that gives a bit of a GUI but I found it hard to use and I'd have needed to put each piece of functionality as a commandline script that this program could call. Instead, I am using go-streamdeck to create a custom application - and I'm having fun! Today's example adds a single button that runs a command to take a screenshot.

Command to Take a Screenshot

I'm on Ubuntu so I can take a screenshot, of the active window, after 2 seconds, with a command like this:

/usr/bin/gnome-screenshot -w -d 2

I wanted to quickly run this from my streamdeck so I could take screenshots while I work (I write quite a lot of technical blog posts, this comes up pretty often). It works a treat!

Pro-tip: Use a script to resize the window you are taking screenshots of (see also: earlier post about resizing windows), this way they will all be the same size, a useful aspect ratio and if you need to replace one, it will match too.

Add the Command to a Streamdeck Button

Here's a simple example script that puts a button labelled "Pic" in the first slot on the streamdeck and sends my screenshot command when it is pressed.

package main

import (
    "fmt"
    "io/ioutil"
    "os/exec"
    "sync"

    "github.com/magicmonkey/go-streamdeck"
    "github.com/magicmonkey/go-streamdeck/actionhandlers"
    "github.com/magicmonkey/go-streamdeck/buttons"
    _ "github.com/magicmonkey/go-streamdeck/devices"
)

func main() {

    fmt.Println("Hello")

    sd, err := streamdeck.New()
    if err != nil {
        panic(err)
    }

    myButton := buttons.NewTextButton("Pic")
    shotaction := &actionhandlers.CustomAction{}
    shotaction.SetHandler(func(btn streamdeck.Button) {
        go takeScreenshot()
    })
    myButton.SetActionHandler(shotaction)
    sd.AddButton(0, myButton)

    var wg sync.WaitGroup
    wg.Add(1)
    wg.Wait()
}

func takeScreenshot() {
    fmt.Println("Taking screenshot with delay...")
    cmd := exec.Command("/usr/bin/gnome-screenshot", "-w", "-d", "2")
    stderr, _ := cmd.StderrPipe()
    stdout, _ := cmd.StdoutPipe()
    if err := cmd.Run(); err != nil {
        panic(err)
    }

    slurp, _ := ioutil.ReadAll(stderr)
    fmt.Printf("%s\n", slurp)
    slurp2, _ := ioutil.ReadAll(stdout)
    fmt.Printf("%s\n", slurp2)

    fmt.Println("Taken screenshot")
}

Using the go routine means that if I have other functionality in this application then I don't need to wait for the command to complete before I can press another button - especially important because this command has a delay in it.

I know I'll be referring back to this post when I want to add commands to buttons in my streamdeck project, so if it's helpful for you as well, then that is even better :) If you want to see my project (under active development just now) then it's on GitHub. Let me know if you use it!

Accessing Nested Config with Viper

2020-06-25T21:28:00+01:00

I'm writing a Go application that glues together a bunch of other things, so it has a whole bunch of basically unrelated config, dumped in a yaml file. I was struggling a little with this non-standard use of Viper but actually, it does everything I needed. And, presumaby, more besides. I thought I would put some examples here to show how to handle this.

A Slice of Structs

Let's start with the configured colours to send to my LED shelf (which I realised I didn't blog about, but I should!) In the config.yaml file it is a series of objects, with properties red, green and blue:

shelf_lights:
  - red: 120
    blue: 200
  - red: 120
    green: 150
  - blue: 200
    green: 40
    red: 60

In my code, I have a struct called LEDColour to represent each of these objects, and it uses field tags to explain how the yaml data fits into the struct - similar to how the JSON field tags work if you've seen those. Viper uses a neat package called mapstructure to handle this part. My struct looks like this:

type LEDColour struct {
    Red   uint8 `mapstructure:"red"`
    Green uint8 `mapstructure:"green"`
    Blue  uint8 `mapstructure:"blue"`
}

To wrangle the yaml data into this structure, I declare the variable it will go into, then Viper has an UnmarshalKey which allows me to grab just the shelf_lights section from config and work on that.

var lights []LEDColour
viper.UnmarshalKey("shelf_lights", &lights)

There was a lot of preamble but the actual moment it happens is neat!

Map of Structs with String Keys

This is pretty similar to above but I think a specific example might help. With a section of the yaml config file like this:

obs_scenes:
  Camera:
    name: Camera
    image: "/camera.png"
  Offline:
    name: Offline
    image: "/offline.png"
  Secrets:
    name: Secrets
    image: "/secrets.png"

I have another struct, also with mapstructure features:

type ObsScene struct {
    Name     string `mapstructure:"name"`
    Image    string `mapstructure:"image"`
    ButtonId int
}

The ButtonId field isn't updated until the scene is assigned to a button dynamically (this is from my streamdeck utility, OBS is a video streaming tool, don't worry if the words don't mean anything! One OBS has many scenes) so we don't need to include it when we're reading from config. Which looks like this:

var buttons_obs map[string]*ObsScene
buttons_obs = make(map[string]*ObsScene)
viper.UnmarshalKey("obs_scenes", &buttons_obs)

As usual, Viper has us covered and while for many applications it makes sense to read in the whole config file and refer to each setting as you need it, this ability to transform sections of config is very handy indeed!

Use a Local Version of a Library in Go

2020-06-15T09:51:00+01:00

I have a couple of projects in Go where I need to work with a branch that isn't the released version of a library that my code depends on. This happens when I'm the developer of the library and an application to use it, or when I'm a contributor to the library so I have my own fork and will want to check out branches to submit or test patches. Go has a pretty need way to allow this: using the replace keyword in the go.mod file.

Set up Git Remotes

Start by cloning the library into the place in your GOPATH that this library would usually live. In this example, I'm using the magicmonkey/go-streamdeck library (lots more to write about this library so watch this space) so it lives at $GOPATH/src/github.com/magicmonkey/go-streamdeck and is a clone of that main GitHub repo at https://github.com/magicmonkey/go-streamdeck.

Since I'm also contributing changes to this library, I forked the GitHub project and have my own copy (that I can commit to branches on) at https://github.com/lornajane/go-streamdeck. Grabbing the clone URL, I added this repo as another remote to the existing go-streamdeck library:

git remote add lornajane [paste clone URL]

Pro-tip: git does not check this remote exists or that you can access it, it just believes you and stores the URL. Run git fetch lornajane (or whatever you named your remote) to actually link with it.

Now the library is all set up, and I need to configure my own project to use the local version of the code, not look for the latest public release.

Configure Go.Mod to use Local Dependency

In the go.mod file for the project that uses this library, the first few lines look like this:

module github.com/lornajane/streamdeck-tricks

go 1.14

replace github.com/magicmonkey/go-streamdeck => /home/lorna/go/src/github.com/magicmonkey/go-streamdeck

... then the usual require block is below.

This setup means that changes I make on disk are reflected immediately in my project, I don't even need to commit. For creating and testing patches to a library, it's excellent and I have to look up the syntax every time so it's here for fast(er) reference!

Edit: Completely failed to link this excellent post on using replace from Pam Selle which covers the same stuff but better.

OBS Studio, Ubuntu 20.04 and a Wacom Tablet

2020-05-31T20:31:00+01:00

When I upgraded my computers to Ubuntu 20.04, I noticed a weird problem with OBS when using an old Wacom Bamboo tablet (because, have you ever tried to use OBS without a pointing device?). I couldn't actually click anything! The pointer seemed to be ... pointing to the wrong place. After some research (and then some help with my research), I came across this post on OBS forums suggesting that this fixes it:

QT_XCB_TABLET_LEGACY_COORDINATES=1 obs

It does, and I never want to do the research for this again, so I'm posting it here and hoping I remember to look here when it happens!

Eye Contact Owl

2020-05-26T20:11:00+01:00

I don't post as much handicraft on this blog as I used to (it's mostly on my instagram account with fewer words now), but I wanted to share my new handmade owl. He solves a genuine problem: how to make eye contact with your webcam.

The Idea

I see a lot of people looking at another screen or some space below the webcam and for a built-in laptop webcam, it's genuinely difficult to look up at the webcam rather than at the video of a person you are talking to, and I don't think anyone minds.

Where it really matters though is when you are presenting slides to a meeting or online conference. There's nothing to look at and so you ... don't. It's weird for the audience and I wanted to make myself somethin gthat could help.

I've seen some excellent hacks (I think this with the post-it notes is my favourite and it's a great article too) and I wanted something. So I thought about it and ended up quite quickly at "owl". They have big eyes to make eye contact with and they're not a complicated shape to create!

Ingredients and method

Using PVA glue, I stuck a piece of leftover fabric from another project to a bit of amazon box (I'd already traced an owl so I knew what size I was aiming for). Apply heavy book, wait for that to dry.

Then I cut out two sizes of felt circles for eyes (they're not very circular, I don't think the owl minds) and also made a beak. I found two matching buttons but I'm sure that more felt would have also looked fine here. More glue! Assemble face, and put cardboard on top (to protect the buttons) and put the heavy book back. Wait for that to dry; luckily it was a long weekend and I had time for waiting for things to dry.

Finally, glue two tufts and an ordinary wooden clothes peg to the back of the cardboard at an angle that amuses you. I was worried about the heavy book bending the cardboard since it no longer had a flat side so I put a couple more pegs (without glue) alongside the glued peg and stood the glue pot on top to give it some weight without crushing it.

And there you have it, the secret to video calls, online presentations, and the obligatory lockdown family zoom calls! Let me know if you make your own? Mine is pretty lonely by himself.

Custom OpenAPI Style Rules with Spectral

2020-05-21T15:54:00+01:00

Edit: There's a newer post on this topic using Redocly CLI.

I work quite a bit with OpenAPI specs and with lots of specs and lots of collaborators, keeping the specs all functional (never mind tidy, consistent, or other dreamwords) is a challenge! We use spectral to check our specs, both when we work on them and in the build process. Spectral is great but it has Opinions(TM)!

For most users, running Spectral out of the box gives quite a lot of output even on an otherwise valid spec. I do think the default ruleset for Spectral is pretty good, but every situation is different so having your own ruleset to use is a good idea. This post shows how to use a ruleset and some examples.

Disabling Spectral Rules

(All these examples are for OpenAPI version 3.0.0 or later)

Spectral (and if you're using Stoplight Studio this as well) will pick up a file called .spectral.yml in the root of the project. In this file we can pick which style rules to follow, enable/disable rules and add our own rules. For a very simple starting point, try this:

extends: spectral:oas
rules:
  openapi-tags: false
  operation-tags: false

Spectral is very keen that every endpoint should have a tag but in fact this isn't required by the spec. I work with a large number of small API specs and most don't have tags (since we'd just tag them all the same thing) so by using the spectral:oas ruleset as a starting point, then disabling the two tag-related rules, I can run Spectral to get the benefit of all the other checks without getting error output from these tag rules.

To know which rules to disable, check your output. Spectral gives very detailed error information including which rule was included and which line of the spec it happened on. It's your choice whether to fix the spec or silence the rule :)

Custom Functions

What happens when you need to add rules that don't already exist? You make your own!

Spectral has some good built in functions that you can use and re-use as needed. For example, here is a rule I'm using to check that the version field in the info block is actually a version number (after we discovered one with version set to "beta" that broke some of our automated tagging stuff!)

rules:
  # Require 3-part Semantic Versions as the spec versions
  semver:
    severity: error
    recommended: true
    message: Specs should follow semantic versioning. {{value}} is not a valid version.
    given: $.info.version
    then:
      function: pattern
      functionOptions:
        match: "^([0-9]+.[0-9]+.[0-9]+)$"

Add this to the spectral.yml file and it will pick up any spec where the version field in the info block isn't a number in the format x.x.x - very handy!

One thing that tripped me up was the JSON Path expressions needed to refer to different parts of the spec so here is a more deeply nested example: checking that all the response fields (that we always put in components/schemas) are in snake_case.

response-property-names:
  severity: warn
  message: "Invalid response property name: {{property}} (expected: lowercase with underscores)"
  recommended: true
  given: "$..components.schemas.*.properties[*]~"
  then:
    function: pattern
    functionOptions:
      match: "^[a-z_]+$"

By turning off the rules we don't need (or aren't ready to meet yet!), the style checking features of Spectral become much more useful. With the ability to use the tools locally and in our build chain (and of course included in Stoplight Studio), that makes it very easy for all our contributors to do the "right" thing rather than make changes and then have either me or the pull request build criticise their hard work!! Tools like this really improve the quality of the OpenAPI specs - but the detail can be complicated. Hopefully sharing this post helps a bit (and I know I'll be using it for reference in the future too).

The python-is-python2 package on Ubuntu Focal Fossa

2020-04-24T19:15:00+01:00

I did a fresh install of Ubuntu 20.04 Focal Fossa on my laptop, and was surprised at what happened when I went to install python:

$ sudo apt install python
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'python-is-python2' instead of 'python'

Um, what?

It turns out that since the python package has historically been Python 2, and there's a separate python3 package (and the commands match the package names, also pip3 etc), that default has remained in an attempt to break things for fewer people. Which is nice, but how do I get current, stable python?

sudo apt install python-is-python3

There's a sister package called python-is-python3 and installing that makes my python command use version 3 as I expected! It's a fairly good solution to the problem but it took me a moment to work out how to install it so I thought I'd write it down for next time! Hope it helps you too :)

A First Netlify Function in Golang

2020-04-15T11:20:00+01:00

I love all things serverless, and Netlify Functions is a neat and convenient way to access AWS Lambda. I do also love AWS Lambda but it's so powerful and flexible that creating something like a webhook receiver can be hard going by the time you have all the permissions and API Gateway setup sorted out - for a simple webhook receiver, Netlify functions is a nice and easy way to get going. Best of all: it supports Golang but the docs are very JS-heavy so I am writing my notes here in case I need to refer back to them some day.

Write a function

The Netlify docs for golang are a great starting point. I am deploying this project manually, so I need to build the thing myself rather than letting the Netlify CI servers do that.

Here's a very simple example with a handler declared in main() and then a function that will greet you by name if you supply a name parameter in the query (this code lives in src/first.go):

package main

import (
    "context"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
)

func handler(ctx context.Context, request events.APIGatewayProxyRequest) (*events.APIGatewayProxyResponse, error) {
    params := request.QueryStringParameters

    name := "World"
    if params["name"] != "" {
        name = params["name"]
    }

    return &events.APIGatewayProxyResponse{
        StatusCode: 200,
        Body:       "Hi " + name,
    }, nil

}

func main() {
    lambda.Start(handler)
}

Great, we wrote a function! Now what? Time to build the code before deploying it.

Build the binary

You can push code to GitHub and instruct the Netlify CI on how to use the source code to build what you want ... in this case I'm just pushing one function as an example so I'll do the building myself. I like to use a Makefile for this:

build:
    mkdir -p functions
    GOOS=linux GOARCH=amd64 go build -o functions/first ./src/first.go

Note that for go build setting the appropriate config for Lambda is important here - on my Ubuntu machine, probably this would build correctly, but by being specific in my Makefile I'm less likely to screw this up in future :)

When I run this by typing make, it puts the compiled first binary into the functions/ directory. This is important as we work up to deploying in the next step.

Deploy to Netlify

I'm doing a manual deploy because I'm too lazy to set up a pipeline for a one-off project, so I am using the `netlify-cli <https://docs.netlify.com/cli/get-started/#installation>`__ command line tool here. Before I do though, I'll configure the settings in a netlify.toml file in the root of my project. Mine looks like this:

[build]
  command = "make build"
  functions = "functions"
  publish = "./"

The important part here is the functions setting - my build step above put the binaries into the functions/ directory, and that's what Netlify will deploy. There's more information in the `.netlify.toml docs <https://docs.netlify.com/configure-builds/file-based-configuration/>`__ too.

Now the deploy command! By default this has a really neat feature where you can deploy a draft and check it before promoting to production. For this project though, I'm just going to go straight for the live platform. You only live once, after all.

netlify deploy --prod

This will upload whatever I put in the functions/ folder and then come back to me with a "Live URL" that it deployed to.

Call the function

Now I can call the function! The URL is a bit funky: [Live URL]/.netlify/functions/first - but Netlify make this easy to find through the web interface, look in the Site you are working on, choose functions from the top bar and when you click on the function name, you get detailed information including its "Endpoint" and also some logs (the output of fmt.Print* goes to the logs here, very useful for debugging!).

I can append ?name=Jane to see my function greet Jane rather than World, which is rather simple but neat. You can access a load of the request info AND the serverless function context from within the function so many more things are possible. I tend to think that the hard part is usually the moving parts and not the code, so hopefully this will help me next time I want to do this. If it helped you too, feel free to share more tips in the comments box!

Wget Direct to S3 with Golang Streams

2020-04-08T16:30:00+01:00

One thing I find very elegant about working with golang is that it is very open minded about where data is flowing in from or going out to. I recently built an application that downloaded an audio file from one location and pushed it to s3 - and golang makes it very easy to do that without needing to write an intermediate file and then upload that. Here's my code, in case I need to do this again some day :)

Downloading a File

By creating an HTTP request, I get a basic request object that I can change other parameters of if I need to. This example sets an Authorization header too, to give you an idea of how the other configuration works.

req, _ := http.NewRequest("GET", RecordingURL, nil)
req.Header.Set("Authorization", "Bearer "+jwt)
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
    panic(err)
}
defer resp.Body.Close()

We haven't actually done anything with the request yet, but by deferring the close we can access the resp.Body as a stream when we come to send the file over to S3.

Uploading to S3

It turns out that AWS already has a very nice SDK for working with Golang - it's not as popular as some of the other SDKs so I didn't find lots of code examples for working with it, but it was a very easy starting point! The SDK has a few ways to supply credentials but I like to use environment variables:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

With these credentials set, and the resp from the section above to read from, I can upload the file to S3. This is a .mp3 file so the ContentType is set accordingly - change this for other file types in your own applications.

format := "audio/mpeg"
sess, err := session.NewSession(&aws.Config{
    Region: aws.String("us-east-1")},
)
if err != nil {
    panic(err)
}

uploader := s3manager.NewUploader(sess)
_, err = uploader.Upload(&s3manager.UploadInput{
    Bucket:      aws.String("my-bucket"),
    Key:         aws.String("uploaded-file.mp3"),
    Body:        resp.Body,
    ContentType: &format,
})

I don't need to do anything with the response here, but if there's an error I do want to catch it and respond, so I've assigned err in the example above.

Just a very quick and simple example but it took me a little while to line up all the pieces I needed so I thought I'd share what I did!

HTTPMock for Testing a Golang API Client

2020-03-30T20:08:00+01:00

I'm working on an API client for some Nexmo APIs and having tests to work through the various responses that the API can return and check that it does what I expect has been very handy so I thought I'd share my thoughts. I've got a bit of a head start here too since the OpenAPI specs have example responses in them that I can grab and feed to the mocking tool I'm using in my tests!

For Golang, I'm using HTTPMock. It allows me to configure a particular response to send when a particular request is received. In fact, you can queue up a few responses but I'm writing a really thin API client so nothing needs more than one call so far.

Inside each test, I configure the request to respond to and the response to send, something like this:

httpmock.Activate()
defer httpmock.DeactivateAndReset()

httpmock.RegisterResponder("GET", "https://api.nexmo.com/verify/json",
    func(req *http.Request) (*http.Response, error) {
        resp := httpmock.NewStringResponse(200, `
{
    "request_id": "abcdef0123456789abcdef0123456789",
    "status": "0"
}
`,
        )

        resp.Header.Add("Content-Type", "application/json")
        return resp, nil
    },
)

What's great about this is that the rest of my test looks "normal" - this tool hijacks the web requests my code would have made, and returns its own responses instead. So I go ahead and put in the test basically the code I'd put in an application that used the library I'm working on.

auth := CreateAuthFromKeySecret("12345678", "456")
client := NewVerifyClient(auth)
// the next line calls the code that actually makes the HTTP request that HTTPMock answers
response, _, _ := client.Request("44777000777", "NexmoGoTest", VerifyOpts{})

message := "Request ID: " + response.RequestId
if message != "Request ID: abcdef0123456789abcdef0123456789" {
    t.Errorf("Verify request failed")
}

In contrast to the other languages I've used with XUnit-type testing, golang doesn't really use the "assert" approach for testing, the idea is that you write code and if it doesn't work, there's your error. There are assertion libraries around but since I'm writing all new tests for basically new code, I thought I'd try "the go way" and see how it worked out. So far it's definitely a new way of thinking but I feel equally confident in the tests I'm creating.

The magic really comes when I start adding failure cases to the tests, that way I can check that things fail the way I'd expect - and of course I can invent responses that the API should never be sending and check that the code also copes with the unexpected!

Remove Accidental Content from Git Commit

2020-03-15T20:24:00+00:00

When I teach git I try to show loads of good practice about how to inspect what's staged before commit, etc etc. Good practice is good of course, but knowing how to undo a mess you created is better - and mistakes will happen. For example, accidentally including a node_modules directory in your otherwise excellent and useful commit. This post will walk you through how to fix the problem without losing any of your work.

Oops, I committed node_modules to my repo

This is a real scenario (and it wasn't actually my mistake this time but honestly, I'm confident fixing this thing because I've done it more than once!!), a developer unintentionally included node_modules/ in the work he was doing. At this point, the project looks like this:

$ git log --oneline
7a6528c (HEAD -> awesome-feature) Try to fix the problem
2144ece Make a brilliant change
70dbb37 (master) Project going well

It's pretty common to try to remove things that shouldn't have been included in a repo by adding a commit to remove it but in lots of cases that doesn't solve the problem. If you accidentally committed your API credentials for example, they're still present in git history. Equally, if you included dependencies as this example does, and the problem is that they are too large to be accepted by GitHub: you still won't be able to push because the intermediate commit has the large files in.

So what to do? First of all, keep calm!

I also like to create an extra branch to point to the current HEAD commit so I don't lose anything before I perform git surgery of any kind. Try this:

git branch safety

Get back to the original problem

Here, it's the second commit in the list that's a problem - but I already tried to fix it by adding another commit. That's not the right solution, so I'm going to start by throwing away the last commit (that tries to fix the problem) completely.

Here's git log again so we can check what's going on:

$ git log --oneline
7a6528c (HEAD -> awesome-feature, safety) Try to fix the problem
2144ece Make a brilliant change
70dbb37 (master) Project going well

Pro-tip: Check which branch you're on before you proceed. I have the branch name in my prompt but git status will also include the current branch name.

To undo the "fix" commit (that didn't fix things at all), make your current branch point to the previous commit (if you run ``git reset --hard`` with uncommitted changes, they will be lost forever. Commit, make the safety branch, and breathe deeply before you rush in to this):

git reset --hard HEAD~1

Where I used HEAD~1, you can also supply the git commit reference. Using HEAD~1 just means "the commit before this one that I'm on now".

Undo your commit but not your changes

For your next trick, we're going to use another reset command but this time without the --hard. The git reset command can affect either the local disk state (what's in the real files on your computer), or the git metadata such as what commit HEAD points to, or both. You already did git reset --hard which does both and puts your working directory exactly to the state of a specific commit (and brings the HEAD pointer to the same place).

Now you're going to use git reset without the --hard to move the HEAD pointer but not change any files at all.

git reset HEAD~1

Run git log --oneline again and we're pointing to the earliest commit, before all the changes (good and unintentional) were made. But check git status next and you should see all your changes there and ready for you to stage only the ones you intended to include! Some tips:
* if you only need to add parts of a file, try git add -p
* use .gitignore files to avoid committing dependencies, config files with secrets in, build directories ... whatever makes sense
* run git status one more time before you commit (this is advice for every day for me!)

When you're happy that only the good changes are included, go ahead and commit.

Pushing changes

If you already pushed changes, and then undid and redid them following this post, your push will be rejected. That's because git never allows us to rewrite history without leaving evidence that we did that!

To push a branch that has changed history, you need to force push - and before you do that, you need to understand the consequences:

If anyone has pushed to the branch since you did, you will lose their changes. Do git pull --rebase to make sure you have everything
If anyone has started a new branch from your branch, their work will be affected and while it's not unrecoverable, it's a tricky situation

This is why we work on feature branches, kids. It's also why most master branches will not allow force pushes.

If you're sure that force pushing is the right thing, for example if you are the only collaborator on the branch, then go ahead:

git push -f

And you're done! Well done for rescuing what could have been a tricky situation without having to do all that work again :) Go ahead and delete that safety branch while you're thinking about it.

Teach Your API Test Platform to Send Callbacks

2020-02-18T15:50:00+00:00

I already wrote about using Prism as an API test platform but I didn't include an example with callbacks so this post is to fill that gap! If you didn't read the previous post, the tl;dr is that Prism is a nodejs open source tool that can take your OpenAPI specification and then do an impression of your live API, validating the API calls sent to it and responding with the example data described in the spec. It can also follow up the API response and send an HTTP request of its own to mock the callbacks described in your OpenAPI spec.

Describing Callbacks in OpenAPI

I'm working with the Nexmo APIs quite a bit so here's an easy example from one of their specs. It's from the "Number Insight" API - you provide a phone number and the API returns you information about that number. There's a few different levels of information (and different levels of cost for each), but the "Advanced" level is pretty unreliable if you call it synchronously - so instead Nexmo will respond to your API call, then send a followup request to the URL you specify with a payload of all the data in it.

Here's how the OpenAPI spec looks:

"/advanced/async/{format}":
  parameters:
    - $ref: "#/components/parameters/format"
  get:
    operationId: getNumberInsightAsync
    summary: Advanced Number Insight (async)
    parameters:
      - $ref: "#/components/parameters/callback"
      - $ref: "#/components/parameters/number"
    responses:
      "200":
        description: OK
        content:
          application/json:
            schema:
              $ref: "#/components/schemas/niResponseAsync"
    callbacks:
      onData:
        "{$request.query.callback}":
          post:
            operationId: asyncCallback
            summary: Asynchronous response
            description: Contains the response to your Number Insight Advanced API request.
            requestBody:
              content:
                application/json:
                  schema:
                    $ref: "#/components/schemas/niResponseJsonAdvanced"
            responses:
              "200":
                description: OK

(if you want to see the whole thing, it's on GitHub )

The callbacks section details the incoming requests to expect as a followup to the client's API call, and what response to send. In the initial request, the callback parameter specifies where the callback will be directed to - and Prism can do that too.

Mocking The API and Callbacks

I can start up the "pretend" API with Prism like this:

prism mock number-insight.yml

Prism will output the endpoints that it got from the spec and then report its location - for me that is usually http://localhost:4010.

In order to receive the callback, I'll need to pass a URL that Prism can reach. This is one scenario where you won't need a public URL to test a webhook - because the webhook is coming from another local tool. In this case, I've got a simple webserver running on port 8080 that will simply output the data it receives so I can inspect it.

Ready? Here's my curl command to request the async endpoint from the local mock API, and gives the callback to send the data payload to:

curl "http://localhost:4010/advanced/async/json?api_key=123&api_secret=456&number=44777000777&callback=http://localhost:8080"

I get a response to my API call, containing a request_id and some other information, then (but seemingly instantly) Prism follows up by sending more example data to the callback URL I specified before.

Testing API Callbacks Locally

This setup is brilliant for a situation like this where I want to develop something to handle this callback - but I don't want to keep on hitting the live API (not to mention that I will run out of credit if I keep hitting this one!). I can work locally, regardless of whether my connectivity is great or even whether the API is up (or even exists yet).

In the future, Prism is expected (by me at least) to also have support for the new Webhooks feature that is slated for OpenAPI 3.1 - enabling us to use a local mock server to send incoming requests as well as handling our outgoing ones. I, for one, can't wait!

Make Thumbnails of PDF Pages with ImageMagick

2020-02-16T15:06:00+00:00

All my talk slides are PDF - I use rst2pdf to transform text-based ReStructuredText content into presentation slides. With all these PDFs hanging around, it can be very handy to have them as thumbnails. I use the images both in the printable speaker notes that I produce (and I should blog that too now I've mentioned it), and to share on twitter - especially the resources slide that everyone photographs! My image file is much more readable than your cameraphone picture in terrible lighting :) So here's my script for thumbnails in case you want to do the same; most presentation tools will export to PDF if you're not already working in that format.

ImageMagick

I'm a huge ImageMagick fan and it's one of the open source tools that I rely on hugely. There are probably graphical frontends that use it too, but I'm a command-line sort of a girl and I also automatically regenerate the thumbnails when I update my slides.

The imagemagick command to work with images is convert so to get the first slide as an image, I can do something like:

convert presentation.pdf[0] slide1.png

This outputs an image, which is nice. But I want all the slides as thumbnails so I need to go a step further.

Building ImageMagick Commands with Python

There is probably a neater way to do this (answers in the comments please!) but I already use PyPDF2 for a bunch of PDF manipulation such as grabbing out my speaker notes, so I stuck with that approach and simply asked python to generate and then run a convert command for each of the slides it finds in my PDF.

Here's the code; it lives in a file called thumbnails.py:

import sys
import PyPDF2, traceback
from subprocess import call

try :
    src = sys.argv[1]
except :
    src = r'/path/to/my/file.pdf'

input1 = PyPDF2.PdfFileReader(open(src, "rb"))
nPages = input1.getNumPages()

for i in range(nPages) :
    # generate the thumbnail
    cmd = 'convert ' + src + '[' + str(i) + '] thumbnails/thumbnail' + str(i) + '.png'
    call(cmd, shell=True)

Run it with python3 thumbnails.py presentation.pdf and it will write a series of thumbnails to the thumbnails/ directory. Which you should create before you run the script, I just didn't mention it until now, sorry.

Hope it's useful!

Instant Test API Platforms with Prism

2020-01-23T11:17:00+00:00

I've been writing a bit about OpenAPI lately and I realised that I didn't share how I'm creating local, test versions of production APIs to work with while I develop new client code or test SDK features. The recipe is simple: you will need an API spec, the excellent open source tool Prism and an HTTP client. Mix it up and be ready to change your API development workflows forever.

Start with an OpenAPI Spec

If you're already working with OpenAPI, try your own. Otherwise, check if your favourite API provider has one. I'm working at Nexmo and we have quite a selection so pick one of these and look for the "Download" button on the API reference.

Use Prism to Pretend to be Your API

Prism runs as a local server, and it "pretends" to be your API. When it starts up, it announces which URLs it knows about, and when you make requests to those URLs, it will attempt to respond appropriately, using the default or example values from the OpenAPI spec. If you provided complete responses, it will use those.

Prism is a NodeJS tool, so install it like this:

npm install -g @stoplight/prism-cli

Then if your OpenAPI spec is openapi.yml, start the server with a command like:

prism mock openapi.yml

Give it a few seconds and it'll announce itself (usually on https://localhost:4010).

Use the Mock Server

Now you can use any HTTP client to make requests to your new API platform and see how it performs! For a quick way to get started, try importing your OpenAPI spec into Postman to get a collection of requests that are ready to go!

Remember that Prism only has the information in the OpenAPI spec file, it can't check your real credentials or do any logic on the server side.

Take Prism to the Next Level

By specifying multiple responses, you can teach Prism all the various responses that might come back from your "real" API. This is great for testing success cases, but it becomes really really powerful when it comes to testing failure cases. I'd really prefer customers use this rather than flatten our servers with traffic trying to generate the rate limit failure message, for example :)

In OpenAPI, you can specify as many responses as you like, and give them names. Here's a snippet from a Nexmo OpenAPI spec:

content:
  application/json:
    examples:
      throttled:
        summary: Request limit exceeded
        value:
          status: "1"
          error_text: Throttled

      missing-param:
        summary: Missing a parameter
        value:
          status: "2"
          error_text: Missing api_key

Here you can see a couple of examples of error responses. I've added the examples in full to the spec and named them. Using the names like throttled, I can then get Prism to return exactly this response (by default it returns the first applicable one it finds) by using the __example query parameter when sending the request.

So if I usually send a request like GET /verify/search?request_id=123456 then I can instead send GET /verify/search?request_id=123456&__example=throttled to get a specific response. I use this extensively to make sure that the code I write (and sometimes that's library code so it can easily end up in other people's projects!) responds predictably and in a sane manner to all the possible responses that might come back from the API.

Having tools like this is moving OpenAPI beyond being "just" a documentation tool, and already I can't imagine working with APIs without it.

Grab Annotations from a PDF with pypdf2

2020-01-15T20:37:00+00:00

If you've noticed a lot of PDF content around here lately, that's because I work with PDF a lot! Most of all, all my slide decks are in PDF and in the last year or so I've started using speaker notes in my presentations. Yes, this means that if you saw me speak in the first ten years of my speaking career, that was without speaker notes.

There are some situations where I don't have access to my speaker notes. Usually this is a good reason, such as I have mirrored my displays so I can demo or play a video without fiddling with my display settings in the middle of a talk. Sometimes, it's because something bad happened and I'm presenting from someone else's machine or a laptop that's completely off stage and I only have the comfort monitor. For those situations I use a printed set of backup speaker notes so I thought I'd share the script that creates these.

First, a complete aside

If you'd like to be ready to support a conference speaker with tech fail, read the post about presenting from PDF and install the tools on your laptop. A bunch of my colleagues past and present have done this and I've hugely appreciated that support! Almost every presentation tool can export to PDF so if the borrowed laptop has presentation tools you at least get the view that has the timer and the next slide.

Script to drag titles and annotations out of a PDF

Speaker notes are usually applied to PDF slides as annotations that are not in the visible space (they're like off the top left or something). You can use this approach for other PDF annotations too.

With the following python code in notes.py:

import sys
import PyPDF2, traceback
import pprint
from subprocess import call

try :
src = sys.argv[1]
except :
src = r'/path/to/my/file.pdf'

# put the role into the rst file
print('.. role:: slide-title')
print('')

input1 = PyPDF2.PdfFileReader(open(src, "rb"))
nPages = input1.getNumPages()

for i in range(nPages) :
# get the data from this PDF page (first line of text, plus annotations)
page = input1.getPage(i)
text = page.extractText()

print(':slide-title:`' + text.splitlines()[0] + '`')
print('')

try :
for annot in page['/Annots'] :
# Other subtypes, such as /Link, cause errors
subtype = annot.getObject()['/Subtype']
if subtype == "/Text":
print(annot.getObject()['/Contents'])
print('')
except :
# there are no annotations on this page
pass

print('')

To use this, run python notes.py [pdf file name].

The script will output rst content (that I then use with rst2pdf) with the first line/title of each slide and the annotations associated with it. Even if there are no annotations, the title is still added which can be useful for just keeping track of which slides are coming up when you don't have any other support information.

It's a simple script but I found it handy (and will probably find it here another day and use it for something else, which is exactly the point of a blog). I hope it's useful to you too!

Raspberry Pi SenseHat Night Clock

2019-12-28T09:36:00+00:00

I recently upgraded my smartwatch and the new model needs charging daily (whether that's really an upgrade is a separate discussion!) so I drop it on the charger by my bed overnight. As a result, I can't tell the time if I wake up in the night which is only annoying because I don't know if it's midnight or if the alarm is going to go off in ten minutes! So I made myself a colourful fuzzy time clock with a raspberry pi and a SenseHat that were lying around in my office.

(note: it's quite hard to photograph neopixels, I tried)

The basic idea is that the colour in the middle changes with the hour, and the white lights around the outside indicate sort of approximate areas on the clock. This way, I avoid staring at the clock reading 03:32 but can still tell that it is time to go back to sleep!

The code is python; run the script python3 clock.py and the script will check the time on the pi, set the lights and exit. To keep the clock running, try setting up a cron job that does this every minute:

* * * * * /usr/bin/python ~/night-clock/clock.py

It's much simpler than trying to keep a long-running process going.

Pro-tip: Neopixels are really bright - much too bright to have anywhere near you in the dark! Look for a variable called on that serves as the basic light level (everything else works off that setting). The max is 255 but I'm working off about 80 and even that is plenty bright enough :)

The code is on GitHub: https://github.com/lornajane/sensehat-night-clock and you're more than welcome to try it, amend it, let me know how you go!

Counting Duplicate Commit Messages

2019-12-16T16:24:00+00:00

When chatting about source control good practice the other day, I got a question about repeated git commit messages. In general, I would always advise that the same commit messages appearing multiple times in a project's history is a definite red flag - and if I'm responsible for that repository I will probably make fun of you for doing it. Workplace harrassment aside, if you can repeat your commit messages, then they are probably not descriptive enough. There are always projects that break this rule for some reason, but in all the years I've worked in software, I have never worked on a project where I'd make an exception for something like that.

To support my point, I also checked one of the larger repos at work for duplicate commit messages. It was simple to do but I thought I'd share my script in case anyone else wants to use it on their own repos and offer constructive feedback to their own colleagues!

git log --oneline | cut -c 10- | sort | uniq -c | sort -n

This shows every commit message in the history of the project, with a count of how many times it appears - and it sorts them by that count (increasing, so that the most repeated messages appear immediately above your cursor when the command completes). Typically I do see quite a few "Merge branch master into ...." type messages and we also have some automation that produces some very similar messages - all that is fair enough. When I find the person who thinks that "Update [filename]" is an acceptable commit message though, I will be taking some time to ~point and laugh~ offer some constructive advice.

Ruby for non-Rubyists with Snap

2019-10-07T09:52:00+01:00

I'm an Ubuntu user and I enjoy most server-side scripting languages but I haven't done much Ruby and I never get the feeling that Ruby is pleased to see me when I do. Something is always out of date or has an incompatible dependency or ... and since I don't use it regularly, I don't have the familiarity that enables me to easily debug these things with other platforms.

I've been using the Ruby Snap on Ubuntu for a while now, and I think it's a bit better? I've referred to my notes every time I've done it though so I thought I had better put them somewhere I'll find them even when I'm using a new notebook :)

What versions are available?

snap info ruby will show your choices of versions to install. I usually just need a major.minor version combination so I set the version with:

sudo snap switch ruby --channel=2.5/stable
sudo snap refresh

If you see a lot of "extension not built" anger then try this: ruby.gem pristine --extensions

Using Ruby and Gem and Bundle

The snap actually ships with all these commands available as separate commands:

ruby (just check this is the right one! Should be the same as /snap/bin/ruby - if it's not check what order things are in your $PATH)
ruby.gem is the gem executable for this snap ruby
ruby.bundle is the bundler gem for this snap ruby (I use this one the most since my ruby usage is basically bundle exec jekyll serve most of the time!)

Beware that the 2.5 channel has bundler v2 and the 2.6 channel has v1, I have no idea why but it's tripped me up at least four times now

Sniglet is a Font for Avoiding Font Loading Mistakes

2019-09-16T20:45:00+01:00

I am not a designer, nor will I ever be. Things look perfectly fine to me a very long time before they look OK to anyone else! Along with that, I don't always find it easy to tell fonts apart. I can do serif-ish and sans-ish but if the fixed-width font has serifs on it I sometimes miss that detail if I'm not looking for it - which also means that I have no idea if I've loaded the fonts I was thinking of or not.

To get around this, I test everything by switching the font to Sniglet from the League of Moveable Type. It's ... umistakable!

Take this simple presentation slide:

Maybe you can tell that this has correctly loaded my preferred font (Gentium, in case you're interested) rather than fallen back to a Times New Roman or something. I am never really sure about it (or at this point I'm completely convinced that I don't know what I'm doing, either way) so I test by loading a different font by the same mechanism: Sniglet.

So NOW I can tell if the font has loaded because even I can tell Sniglet apart from a default/fallback font!! I also love it, it's cute and fun, and makes me smile :) What top tips do you have for catching errors that you might otherwise overlook? I love little tricks like this so leave me a comment if you have something to share!

OpenAPI Description using API key and secret

2019-08-15T18:10:00+01:00

I've been working on a few OpenAPI descriptions of APIs lately, and really enjoying the benefits they bring. In particular the ability to import into Postman as a collection I think is a "killer feature" for APIs looking to get developers up and running quickly. The catch? I found that an API that needs an API key and secret, or a username and password/token, is supposed to be described with multiple security schemes - but this does not play nicely with tools like Postman.

To work around this problem, I changed the security schemes to describe a single API key field (it's a query parameter). This is well understood by all tools, and also most of them will just pick the first one when multiple are specified. Then I added the API secret field as a required parameter to all the endpoints.

This isn't an ideal situation, because the secret really is part of the auth, not a parameter. However I did find that it helped not only my Postman import, but also use of Prism as a mock server, and that some documentation tools rendered the required fields for each endpoint more clearly.

As things stand, I have a single security scheme listed in the components: section:

securitySchemes:
  apiKey:
    type: apiKey
    name: api_key
    in: query
    description: >-
      You can find your API key in your Nexmo account [developer
      dashboard](https://dashboard.nexmo.com/)

Then for each endpoint, I added the additional parameter (described in the parameters: section of components: and using a $ref). This is perhaps less formally correct, but more working, than what I had before - so I'm living with it for now.

If you want to see the spec in action, it's here: https://github.com/Nexmo/api-specification/blob/master/definitions/verify.yml and if you have any better ideas, I'd love to hear them!

The Laravel Synchronous Queue

2019-05-20T15:43:00+01:00

Using queues for asynchronous processing is one of my favourite tricks for offloading hard work from web servers. When working with Laravel recently I was pleased to find that it supports beanstalkd out of the box. I've got opinions about frameworks with Opinions but I did find one thing I really liked in the way Laravel uses queues: the sync queue option that runs your queue synchronously on your development platform so you can develop and debug your work, then switch the queue platform you use later.

It's quite tricky to debug Laravel "Jobs" as they call the worker code - if errors occur, they don't show up in the general output. You can use error_log() and see that output, but if there's a PHP error, you won't see that error. However you can configure the QUEUE_CONNECTION in .env to be "sync". The sync queue type is not asynchronous at all - instead of putting the job onto your chosen queue backend, it goes right ahead and processes the job in the current thread of execution. This completely defeats the object of using a queue but is a brilliant approach for debugging how the data passes from the dispatching context (where the data is put into the job and onto a queue) to the worker context (where the job comes back off the queue and actually gets processed) and how the job is processed.

For development use, it's a nice feature, but do take care to enable your "real" queue configuration when deploying to a production environment so that you get the advantage of the queues. There are some good queueing services available for your cloud applications - specifically for Laravel, look at Laravel Horizon which is Redis with nicer dashboards attached :)

The First Thousand Blog Posts

2019-04-30T09:51:00+01:00

This is the 1000th post on https://beta.lornajane.net. People ask "how do you even do that?" and the short answer is "start in 2006", but the long answer isn't all that long.

It all began because I couldn't hold down a job. After graduating, I had five jobs in five years, which was fabulous because I learned a lot. What was not great was losing all my useful scripts every time because I didn't know what I might need again in the future and BYOD hadn't been invented yet. So, I began to write things down in an internet-enabled location.

Storing my notes publicly is incredibly useful. I can send links to other people who ask me questions I once knew the answer to (people who know me well ask the blog for help before they ask me, it's a more reliable witness). I also discovered after a few years that my favourite search engine has a much better idea what I've already written than I have - I regularly find results on https://beta.lornajane.net appearing in my search results and quite often it's exactly what I needed!

At first, it didn't matter what I wrote, nobody read it and I was just throwing things out for future reference. I'd like to write now about how much my approach has improved but I would be lying!! The truth is that I'm still just throwing things out for my future reference. All the bystanders of the internet are very welcome to read it, but almost all of it is written for me in case I need to come back to it. Many bloggers or aspiring bloggers are stuck on not being good enough, not having enough ideas, not having finished building their blog. Perhaps I should have worried about some of those things but it seems a bit late now!

Unintended Consequences

Keeping a blog made me a better writer, mostly because it made me actually write from time to time. Also there was a time where people read blogs and wrote comments and I got some excellent feedback through that (there is some irony that this is the first blog post I've cross-posted to Medium, times do change!). The other thing about blogging is that you write how to do something and then four people chime in and show you at least three better ways to do it!! This has been the best thing ever, I have learned so much from those commenters who took the time to explain very politely that I was giving terrible advice. The moral of that story is that you should always read the comments on blog posts :)

Another side effect has been that my career has included some writing in a professional capacity. From writing from my own blog, to work blogs, to industry publications, to eventually writing a couple of books ... now I'm doing a job that includes writing documentation and tutorials. Combining the written word with solid software engineering skills is pretty rare, and without the blog I don't know if I'd have realised I could write.

Blog As A Brain

My blog is basically an outboard brain, it's full of things I once knew but have completely forgotten. If there's a technology I've worked on, there's probably a post about it on my blog. It's a rare week where I don't come here looking for something or other, whether it's how to set up my IRC client, or the syntax for a particular coding library. Some years I've blogged more than others, and I'm still terrible at metrics, marketing, engagement, and all those other things - it's a wordpress blog with a CDN in front, I just type my thoughts in markdown into the textarea while I'm drinking my coffee in the morning.

As the saying goes "content is king" and I do really believe that. By taking the time to write up my experience into blog posts, I've helped myself to understand countless technical topics. I've given myself an incredible reference library that I can use whenever I need it. And if anyone else finds it useful too, then that just makes a good thing better.

Here's to the next thousand posts!

Handy Beanstalkd Admin Console

2019-04-08T09:13:00+01:00

I've been building apps with queues lately and mostly using beanstalkd as my queue because it is very simple, very fast and on my platform it is [apt install beanstalkd] -easy to install. I have also been using a handy web interface for beanstalkd which I like so much that I felt I ought to share! It's beanstalk-console, which is a PHP-based web interface to one (or many) beanstalkd servers.

Beanstalkd is a very simple queueing system and I've enjoyed using it from a few different tech stacks. It uses "tubes" rather than channels, supports priorities, and uses a simple string for the body of the "job" on the queue.

The console has a very simple Composer-based setup and is also ready-to-run in a docker container so should be accessible to most users, with or without PHP skills/installation handy. It allows you to configure servers and then the web interface shows these servers. You get a list of tubes with some information about each one:

That's pretty cool and I use it both when I'm developing applications and when demo-ing code in presentations. The real magic happens when you drill further in though; get down to the individual tube level and you can inspect, create and "kick" (delete) jobs from the queues from the web interface.

Having a good admin tool can give insight into the contents of a queue which would be bewildering for a production application under any sort of load but which is ideal for tracing problems or showing off smaller pieces of a larger application during demo or development phases. I've really enjoyed this particular tool so I wanted to share in case it's useful for anyone else too!

HTTP Toolbox

2019-02-27T09:53:00+00:00

As Web Developers, we need to know how to work with HTTP from every angle. I gave a 2-hour tutorial at PHP UK that included some of my most trusted tools - but it was sold out and a bunch of people asked me if there was video (there wasn't, tutorials make little sense when videoed). Instead, I thought I'd try to set out a self-study version of the workshop (I rarely teach these days so I'm unlikely to deliver it anywhere else).

There's a slide deck, some exercises and a sample repo on GitHub ... let's dive in!

Step 0: Get the Materials

You're more than welcome to grab the materials I used for the workshop, they are all in a GitHub repo here: https://github.com/lornajane/fun-with-http - look in the workshop/ folder for the slide deck and all the exercises. The slides are probably hard to follow by themselves so I'll try to put some basic outline and links into this post.

Step 1: Understand HTTP

You think you understand HTTP already, I know :) And I'm not going to write an essay here - I already wrote the book so if you need words, they're available.

You should set up the sample app (check the README in the project repo) at this point.

Then, try an HTTP client of your choice (maybe try something new?) and make simple requests to the app. These are the suggestions I offered in the workshop:

curl https://curl.haxx.se/
Postman https://www.getpostman.com/
HTTPie https://httpie.org/
http-console https://github.com/cloudhead/http-console
Paw (Mac only) https://paw.cloud/client
Insomnia https://insomnia.rest/
probably your IDE?

I've also blogged about this before and there are some excellent recommendations on that post so check that out also.

Step 2: Get comfortable with data formats

I kept it pretty simple and focussed on form-encoded and JSON formats in the workshop but you may also come across XML and there's a trend for binary formats too so you may need more tools than I'm covering here, depending what you are doing.

Recommended tools for JSON:
- jq https://stedolan.github.io/jq/
- fx https://github.com/antonmedv/fx

Step 3: Making simple API calls with PHP - and debugging them

For this section I showed two options for making simple POST requests from PHP: firstly using streams, which is very powerful and does not need dependencies - and secondly using Guzzle which is fabulous, also very powerful, and a bit more user-friendly. Here's the two sets of example code so you can compare and contrast:

$url = "https://httpbin.org/post";
$data = ["name" => "lornajane", "message" => "Hi there"];

$context = stream_context_create(
    ["http" => [
        "method" => "POST",
        "content" => http_build_query($data)
    ]
]);

$response = file_get_contents($url, false, $context);

And with Guzzle:

require "vendor/autoload.php";

$url = "https://httpbin.org/post";
$data = ["name" => "lornajane", "message" => "Hi there"];

$client = new GuzzleHttpClient();
$response = $client->request("POST", $url, [
    "form_params" => $data
]);

At this point it would be good to mention a couple of tools that I use when I need to diagnose if my PHP application is making the HTTP request I hope it is! Sometimes when integrating multiple applications it isn't clear at which end the problem is, and these tools can help.

First up: HTTPBin is a hosted service which offers a series of endpoints that return known data, headers, status codes, whatever you need for testing. This is very useful for testing error cases that are hard to replicate on the real API you are using.

My preferred tool in this space is RequestBin - please note that this link points to my own GitHub fork since the project itself seems to have been abandoned and there were a couple of patches in the pull request queue that are needed to run this on Heroku. It's a brilliant tool for testing your outgoing web requests and can run in docker on your laptop or on the cloud very easily.

I challenged the workshop attendees to complete a half-made route in the sample application for this part of the workshop. We also stopped for coffee, so you might like to do that too :)

Step 4: Ngrok for development platform access

If you're not already familiar with ngrok, you might like to be! It is an application that runs on your development platform, and makes your local webserver accessible from the open internet via a secure tunnel. I use it a LOT, mostly for testing incoming webhooks but also sometimes to see how something will work on mobile, or to let someone in another location take a look at something I'm working on.

Ngrok also allows you to inspect the full request and response of the traffic going over the tunnel, which is fantastic especially at the development stage. And that isn't even the coolest part! The best bit is the "Replay" button available alongside the details of each incoming request: if your code doesn't quite do the right thing, you can edit it and then run the request again, see what happens the next time! This is such a gift especially when the event that triggers an incoming requests is expensive or annoying to reproduce - again, I use this mostly for webhooks and you can probably see why!

Take the time to try the feature you made in the previous step via Ngrok.

Step 5: Proxies for Debugging

OK so one of these tools is not a proxy (Wireshark actually copies your traffic rather than it passing through the appplication) but I use them all for the same sorts of problems! Knowing how to use proxy tools gives you a very easy way to inspect all the requests and responses and this can be really valuable for allowing detailed debugging of your application without having to make a lot of diagnostic additions to your actual project.

My recommended tools in this category:

Charles Proxy (paid product, free trial available) https://www.charlesproxy.com/
Mitmproxy (open source) https://mitmproxy.org/
Wireshark (slightly lower level tool but brilliant for buggy HTTP) https://www.wireshark.org/

I have used all three of these and especially Charles Proxy has been a huge boost. It enables you to play tricks such as rewriting URLs (because the images are hardcoded to the wrong path and don't work on test, perhaps) and simulating slower network speeds so you get a sense of what the experience will be for a real user. Another big feature with Charles is that you can replay requests, and also store and export them - very handy for attaching to an open issue in a bug tracker that shows exactly what happens, and will allow someone else to replay that request again later!

You had to be there

Hopefully some of the links and suggestions here are useful, the content wasn't designed for a blog post but if you're interested and motivated to work through the exercises for each step as a "self study" exercise then you're really welcome to do so. I'm also happy to take questions if you have them - just leave me a comment!

Using Tags in your OpenAPI Spec

2019-02-26T10:15:00+00:00

Working with OpenAPI is bringing so many possibilities to the way that developers work with APIs, it's exciting! The spec is very comprehensive though and I've found myself answering questions on individual aspects of it recently, so I thought I'd capture one or two of those things here. Today: tags.

Tags in OpenAPI are optional but very useful! If you use Speccy then it has Opinions(TM) and will suggest that you need tags on every endpoint. The formal definition doesn't require it however and I tend to turn off these rules for small APIs.

When tags really come into their own is on larger APIs, perhaps an application only has one API and everything goes there - or there is one "utility" API that's a catchall for anything not important or big enough to need its own API. Using tags on these larger APIs can really help users to find their way around.

Adding Tags to an OpenAPI Spec

To add a tag to your OpenAPI spec, you need to do two things:

Define the tags in a top-level tags element
Add a tag to each operation in the spec

The tags element looks something like this:

tags:
  - name: Account
    description: Relating to an account (the user must have access to the account)
  - name: User
    description: Manage a user account (either your own, or one you have permissions to access)
  - name: Admin
    description: Only available to superusers

I'd recommend adding descriptions here, it helps to make it clearer both to developers and to future maintainers of the API and its description what the tag was intended for.

Then to add the tag to a path, the tags array goes at the same level as operationID, like this (remember you can add as many tags as make sense):

/accounts/{api_key}/secrets:
  get:
    summary: Retrieve API Secrets
    operationId: retrieveAPISecrets
    tags:
      - Account

Quite a few of the documentation tools (including ReDoc which is my personal favourite) will render the endpoints grouped by tag which is very valuable. Speccy recommends having the tags alphabetically but I tend to turn this rule off where alphabetical doesn't make sense (for example above where I have a few endpoints tagged "Admin" but most users can't access those so I have it last in the list).

Using tags can be a valuable addition to your OpenAPI spec so I hope that sharing this example is helpful!

Windows removed my grub menu

2019-01-28T10:45:00+00:00

I know this will happen to me again at a bad moment so I'm putting it here on my blog and hoping I can find it in my future hour of need ...

My work machine is a Windows PC, with dual boot to the Ubuntu partition that I actually use. Sometimes, when booting Windows, it "repairs" its disks and removes my grub menu, booting straight into Windows without showing me the grub menu.

To fix this: interrupt the startup, choose the boot device and pick the Ubuntu partition from the list.

Once booted, you can boot-repair to sort out grub - I also travel with a boot disk, just in case!

Ready-to-wear API Collections with OpenAPI and Postman

2019-01-08T12:50:00+00:00

Have you noticed that API providers are starting to offer "descriptions" of their APIs? These are usually OpenAPI specifications, computer-readable documents that describe every aspect of an API's endpoints. They can be used to generate documentation, SDKs, but now they can also be used for my favourite API-related activity: poking around! Postman just added a feature to allow importing OpenAPIs and I love it.

The premise is simple: choose Import, and supply an OpenAPI specification. At Nexmo, we hold ours on GitHub so I just click the "Raw" button and paste the URL as a link to import, e.g. https://raw.githubusercontent.com/Nexmo/api-specification/master/definitions/number-insight.yml

You get a collection with all the described endpoints and when you select one, the request is already crafted such that all you need to do is fill in the values you want to send (Pro tip for Nexmo APIs: you'll usually need to add the api_secret field yourself).

What I love about this is how quickly I can start to poke around in unfamiliar APIs, with very easy access to request formats, descriptions of what should be in each field, and so on. The world of OpenAPI is wide and growing all the time, and as more and more API providers offer API descriptions, more and more developers will have the ability to play in the sand using familiar tools and a new API.