Getting acquainted with a legacy codebase

Over the last few days I’ve been getting to grips with a codebase that has been neglected. The original product team have disappeared, the product has had sporadic updates over the years meaning tests have gone awry and product knowledge is held by no one.

For the last six months, a lone developer has spent 2 days a week stabilising the codebase, slowly fixing tests, meeting users and delivering the occasional requirement.

Now it’s my turn to get under the bonnet.

The first thing I’ve spent time on is optimising the Dockerfile to leverage the build cache. The CI build was taking about 7 to 8 minutes, although a good proportion of this was pushing the image (1.4GB) to the registry. Spending time on the Dockerfile has re-acquainted with Docker and given me a greater understanding of how the codebase is put together.

The next thing to do is get the end to end tests working in CI. Protractor wrapped Selenium Web Driver, which is configured to run the tests locally in PhantomJS. For some reason, these tests don’t work, so I’ve simplified the Protractor configuration file and set it to run Puppeteer in headless mode. This works fine on my local machine but it’s going to take more effort to get Puppeteer, which is a wrapper around Chromium, to work in a Docker container. Chromium depends on a bunch of packages being installed, but I don’t want to add to the already bloated Docker image if I can help it.

I’ll write back once I figured this out.

Developer tools and modules I’ve used recently

I wanted to list a few of the tools I’ve found useful over the past 6 months.

CSV Kit

I had to analyse a bunch of spreadsheets. CSVKit proved great for peeping quickly into a sheet within an XLSX file, ordering the data and displaying the columns I was interested in. Here’s a sample of a command I used:

in2csv birmingham2017.xls --sheet 'Claim Form' --skip-lines 6 > birmingham2017.csv
csvcut -x -C 1,2,3,4,5,6,23,25,26,27,29,31,32 birmingham2017.csv | csvlook | less -S

The first line converted the sheet named ‘Claim Form’ into a CSV, skipping the first 6 rows of crap in the spreadsheet. The second line deleted the empty rows, filtered the columns specified, formatted the data in markdown-compatible, fixed-width format.

JSON Server

This was great for quickly creating a fake back-end server that responded with JSON.

// JSONServer
const jsonServer = require("json-server")
const server = jsonServer.create()
const router = jsonServer.router(`${__dirname}/db.json`)
const middlewares = jsonServer.defaults()

// Set default middlewares (logger, static, cors and no-cache)
server.use(middlewares)

// Add custom routes before JSON Server router
server.get("/echo", (req, res) => {
  res.jsonp(req.query)
})

// To handle POST, PUT and PATCH you need to use a body-parser
// You can use the one used by JSON Server
server.use(jsonServer.bodyParser)
server.use((req, res, next) => {
  if (req.method === "POST") {
    req.body.createdAt = Date.now()
  }
  // Continue to JSON Server router
  next()
})

// Use default router
// server.use('/api', router)
server.use(router)

module.exports = server

HMRC Screens

I re-purposed HMRC Screens to browse screenshots of the various product prototypes. The main benefit was time-saving. Getting access to and navigating the prototypes often would be problematic for the team. Storing screenshots of the prototypes provided a handy overview, plus saved time when a team member might need a screenshot to embed in a document.

JS-XSLX

I needed to read a spreadsheet that was uploaded via a web interface. This npm module proved handy. Here’s an example of how I used it to read a spreadsheet that followed a particular convention:

const xlsx = require("xlsx")
const _ = require("lodash")

const HEADER_RANGE = "A7:AG7"
const DATA_STARTING_ROW = 7

let workbook

const claimForm = filepath => {
  if (typeof filepath !== "string") {
    throw new TypeError(`Expected string, got ${typeof filepath}`)
  }
  try {
    workbook = xlsx.readFile(filepath, { cellText: false, cellDates: true })
    return data()
  } catch (err) {
    throw new Error(err)
  }
}

const headers = () => {
  return xlsx.utils.sheet_to_json(workbook.Sheets["Claim Form"], {
    range: HEADER_RANGE,
    header: 1
  })[0]
}

const data = () => {
  const camelCaseHeaders = camelCase(headers())
  const rows = xlsx.utils.sheet_to_json(workbook.Sheets["Claim Form"], {
    range: DATA_STARTING_ROW,
    header: camelCaseHeaders,
    dateNF: "yyyymmdd"
  })
  return removeEmptyRows(rows)
}

const camelCase = arr => arr.map(item => _.camelCase(item))

const removeEmptyRows = rows => {
  return _.reject(rows, row => row.accountNumber === "0")
}

module.exports = claimForm

The code reads headers and data rows from a sheet called “Claim Form” and removes empty rows.

Serverless framework

I started playing with the AWS stack of services, in particular AWS Lambda. I discovered Serverless, which is a framework for provisioning infrastructure and deploying lambda functions with ease. Here’s a YAML file that runs a function called save in a file called handler.js. It fetches data from a URL and stores it in S3 bucket and is scheduled to run each weekday at 10:15.

service: fetch-file-and-store-in-s3

frameworkVersion: ">=1.1.0"

custom:
  bucket: courtfinderdata

provider:
  name: aws
  runtime: nodejs8.10
  stage: dev
  region: eu-west-2
  iamRoleStatements:
    - Effect: Allow
      Action:
        - s3:PutObject
      Resource: "arn:aws:s3:::${self:custom.bucket}/*"

functions:
  save:
    handler: handler.save
    environment:
      BUCKET: ${self:custom.bucket}
    events:
      - schedule:
          rate: cron(15 10 ? * MON-FRI *)
          input:
            data_url: "https://courttribunalfinder.service.gov.uk/courts.json"
            key_suffix: "courts.json"

Farnham Street’s most popular articles

I stumbled across Farnham Street. I want to read everything on there! I signed up for the newsletter and the first email makes a suggestion about where to start. To increase the likelihood of me not forgetting to read everything, I’ve copied the suggestions here as a reminder.

Decision Making

Decisions Under Uncertainty — One way to realize how ignorant we are is to look back, read some old newspapers, and see how often the world did something that wasn’t even imagined.

Avoiding Stupidity is Easier than Seeking Brilliance — “The amateur duffer seldom beats his opponent, but he beats himself all the time.”

Daniel Kahneman’s Favorite Approach For Making Better Decisions — “We build on Nobel winner Daniel Kahneman’s favorite approach for making better decisions. This may sound weird, but it’s a form of imaginary time travel.”

Thinking About Thinking

The Best Introduction to Mental Models — Acquiring knowledge may seem like a daunting task. There is so much to know and time is precious. Luckily, we don’t have to master everything.

How To Think — Thinking is not IQ. When people talk about thinking they make the mistake of thinking that people with high IQs think better.

The Difference Between Knowing the Name of Something and Knowing Something — Physicist Richard Feynman explains why knowing the name of something doesn’t mean you understand it.

Changing How We Think — You need to learn how to integrate.

Adding Mental Models to Your Mind’s Toolbox — The world is always changing so which models should we prioritize learning?

Creativity

Steve Jobs on Creativity — Creativity is connecting things.

The Remarkable Ways We Gain Insights — “Intuition is the use of patterns they’ve already learned, whereas insight is the discovery of new patterns.”

The Meaning of Life

Finding Your Purpose and Living a Meaningful Life — A famous letter from Hunter S. Thompson to his friend Hume Logan. A must read.

Richard Feynman’s Love Letter to His Wife Sixteen Months After Her Death — I tear up a little each time I read this.

Tiny Beautiful Things — “Inexplicable sorrows await all of us. … Life isn’t some narcissistic game you play online. It all matters — every sin, every regret, every affliction.”

Miscellaneous

How to Read a Book — I’m serious. You don’t know how to read and this small fact is costing you time and money.

A list of What We’re Reading — everything from the past few years with mini-summaries.

The Buffett Formula — How to get smarter using a simple but not easy approach.

The Knowledge Project — A podcast where I interview people from across the globe to deconstruct how they think, live, and connect ideas.

100 words 005

I spent a couple of hours in the Supreme Court today. The basement has a cafe and an exhibition.

There was an interactive touchscreen application. You chose a case, read through the evidence and made a judgement. I selected a case about an uninsured campervan driver who was involved in a fatal crash. He was was prosecuted for causing the death of the other driver. My judgement, based on common-sense, was that the law was wrongly interpreted. The supreme court made the same judgement.

I recommend the guided tour too.

The day before was the historic brexit ruling in court 1.

100 words 004

Democracy reigns supreme. Parliamentary democracy. Here’s a cartoon.

Eight out of eleven judges from the highest court in the UK voted that parliament must approve a decision. Supreme Court President Lord Neuberger said:

The UK’s constitutional arrangements require such changes to be clearly authorised by Parliament.

Tomorrow, I’ll be on a tour of the Supreme Court with a few of my work colleagues. A few facts:

  • Started work on 1 October 2009
  • It cannot overturn any primary legislation made by Parliament
  • Sir Peter Blake designed an abstract depiction of the four floral elements of the court’s badge, in use in the carpets

100 words 003

I reviewed a pull request today. A method was called with a local variable, e.g.:

def log(msg)
  msg << ' ' + Time.now.to_s
end
message = 'message'
log(message)
puts message # prints "message 2017-01-23 22:15:41 +0000"

The log() method mutates the caller. The object that the local variable message points to has changed to the value set by the log() method. This is because Ruby passes-by-value, where the value is a reference.

Tom Dalling says:

Avoiding mutability by default is, in my opinion, a rule worth applying.

If a method is going to mutate the caller, indicate this by using an exclamation-mark/bang in the method name, e.g. log!(msg).

100 words 002

We spent a few hours walking in Happy Valley. We ate our picnic, sat on a wooden bench, inspecting the clear, blue sky above Devilsden Wood, for birds and helicopters. The winter sun made us all squint.

Every dog is called Rufus at the moment, thanks to We’re Going On A Bear Hunt. So many sniffs, wags and happy barks in Happy Valley.

The Fox was full. We drove to BoxPark. Claire devoured the tastiest vegan meal she’s had since turning. Leigh and Chris spotted us, happy to be child-free for an afternoon. I shared Lebanese food with the boy.

100 words 001

I woke early this morning, soaked porridge oats in milk, turning on the hob when I heard the sound of giggles upstairs. Rise and shine.

Minutes later, the seated boy demanded his breakfast. I served it with strawberry jam. It was gone in a flash, football fuel in the tank.

By 9am, my son and an eager bunch of other small people burnt energy off running around a sports hall, occasionally kicking a ball.

Afterwards, an italian cafe’s customers endured three rowdy boys for half an hour (too long) before we descended upon Westow Park, looking for sticks and slides.

Weeknotes: 2016 Week 1

I’m writing again. Not sure how long it’s gonna last.

Lots of stuff happened between now and then. Maybe I’ll attempt a retrospective of kinds in a later post.

The last week and a bit

I started at the Ministry of Justice as a developer a week or so ago. I’m working on Out Of Court Pathway stuff, with a focus on family law. To get to grips with this domain I’ve read quite a few reports and user research findings and spoken to some very knowledgable team members and was a member of winning pub quiz team (answering just one question)!

One thing I’ve been impressed with is the focus on the discovery of user needs. This is encouraged and practiced by everyone, no matter what a person’s role is.

I’ve discovered the gov_uk_prototyping kit which I’m sure will come in handy. I’ve run brew install countless times – mongodb; mysql; start postgresql; heroku; bash-completion (for docker completion); nvm (don’t bother, just use the one-liner curl command to install nvm) and slowly stepped towards setting up my machine for development. I’m using Slack as a personal note taker, Visual Studio Code as an editor and iTerm3. A lot of the projects have been dockerised so I downloaded Docker For Mac and it just worked.

I’ve caught up on past editions of MoJ Frontend Weekly, perused some of the codebases, signed up for accounts on HipChat, Confluence, G Suite (formerly known as Google Apps), Slack and various internal systems.

The project I’m working on is becoming clearer day by day. Lots to look forward to!