Developer tools and modules I’ve used recently

I wanted to list a few of the tools I’ve found useful over the past 6 months.

CSV Kit

I had to analyse a bunch of spreadsheets. CSVKit proved great for peeping quickly into a sheet within an XLSX file, ordering the data and displaying the columns I was interested in. Here’s a sample of a command I used:

in2csv birmingham2017.xls --sheet 'Claim Form' --skip-lines 6 > birmingham2017.csv
csvcut -x -C 1,2,3,4,5,6,23,25,26,27,29,31,32 birmingham2017.csv | csvlook | less -S

The first line converted the sheet named ‘Claim Form’ into a CSV, skipping the first 6 rows of crap in the spreadsheet. The second line deleted the empty rows, filtered the columns specified, formatted the data in markdown-compatible, fixed-width format.

JSON Server

This was great for quickly creating a fake back-end server that responded with JSON.

// JSONServer
const jsonServer = require("json-server")
const server = jsonServer.create()
const router = jsonServer.router(`${__dirname}/db.json`)
const middlewares = jsonServer.defaults()

// Set default middlewares (logger, static, cors and no-cache)
server.use(middlewares)

// Add custom routes before JSON Server router
server.get("/echo", (req, res) => {
  res.jsonp(req.query)
})

// To handle POST, PUT and PATCH you need to use a body-parser
// You can use the one used by JSON Server
server.use(jsonServer.bodyParser)
server.use((req, res, next) => {
  if (req.method === "POST") {
    req.body.createdAt = Date.now()
  }
  // Continue to JSON Server router
  next()
})

// Use default router
// server.use('/api', router)
server.use(router)

module.exports = server

HMRC Screens

I re-purposed HMRC Screens to browse screenshots of the various product prototypes. The main benefit was time-saving. Getting access to and navigating the prototypes often would be problematic for the team. Storing screenshots of the prototypes provided a handy overview, plus saved time when a team member might need a screenshot to embed in a document.

JS-XSLX

I needed to read a spreadsheet that was uploaded via a web interface. This npm module proved handy. Here’s an example of how I used it to read a spreadsheet that followed a particular convention:

const xlsx = require("xlsx")
const _ = require("lodash")

const HEADER_RANGE = "A7:AG7"
const DATA_STARTING_ROW = 7

let workbook

const claimForm = filepath => {
  if (typeof filepath !== "string") {
    throw new TypeError(`Expected string, got ${typeof filepath}`)
  }
  try {
    workbook = xlsx.readFile(filepath, { cellText: false, cellDates: true })
    return data()
  } catch (err) {
    throw new Error(err)
  }
}

const headers = () => {
  return xlsx.utils.sheet_to_json(workbook.Sheets["Claim Form"], {
    range: HEADER_RANGE,
    header: 1
  })[0]
}

const data = () => {
  const camelCaseHeaders = camelCase(headers())
  const rows = xlsx.utils.sheet_to_json(workbook.Sheets["Claim Form"], {
    range: DATA_STARTING_ROW,
    header: camelCaseHeaders,
    dateNF: "yyyymmdd"
  })
  return removeEmptyRows(rows)
}

const camelCase = arr => arr.map(item => _.camelCase(item))

const removeEmptyRows = rows => {
  return _.reject(rows, row => row.accountNumber === "0")
}

module.exports = claimForm

The code reads headers and data rows from a sheet called “Claim Form” and removes empty rows.

Serverless framework

I started playing with the AWS stack of services, in particular AWS Lambda. I discovered Serverless, which is a framework for provisioning infrastructure and deploying lambda functions with ease. Here’s a YAML file that runs a function called save in a file called handler.js. It fetches data from a URL and stores it in S3 bucket and is scheduled to run each weekday at 10:15.

service: fetch-file-and-store-in-s3

frameworkVersion: ">=1.1.0"

custom:
  bucket: courtfinderdata

provider:
  name: aws
  runtime: nodejs8.10
  stage: dev
  region: eu-west-2
  iamRoleStatements:
    - Effect: Allow
      Action:
        - s3:PutObject
      Resource: "arn:aws:s3:::${self:custom.bucket}/*"

functions:
  save:
    handler: handler.save
    environment:
      BUCKET: ${self:custom.bucket}
    events:
      - schedule:
          rate: cron(15 10 ? * MON-FRI *)
          input:
            data_url: "https://courttribunalfinder.service.gov.uk/courts.json"
            key_suffix: "courts.json"