Data Standards + Who's Got Dirt? API

Looking at the possibilities for standard to increase data interoperability between our datasets and tools, including the idea of a common API for investigative entity mapping tools: “Who’s Got Dirt?” API. Presented by James McKinney.

Frame of the presentation

Collaboration models: How the people work together

Interoperability APIs and formats: How the software works together

Workflow definitions: What should the software do

  • There are a lot of different databases out there and the usual way to know if there’s data matching your query is finally made by e-mail


  • First we work with what we have: LittleSis, OpenCorporates, OpenOil, Poderopedia, Quién Manda, Open Duka, CrocTail, Grano, etc.
  • The first problem around this is that you have to work around distributed queries
  • End up looking at databases that might help
  • So you have a big issue around discoverability: finding the information you need

So, the current common process is:

  1. Consult the list of websites / APIs
  2. Visit each website / API
  3. read each website’s documentation
  4. / read each API reference
  5. perform searches on each website
  6. / send requests to each API
  7. Read the data and interpret its schema
  8. (whether it’s CSV, JSON, XML, HTML)

The proposed process is:

  1. Consult one website / API
  2. Visit one website / API
  3. read one website’s documentation
  4. / read one API reference
  5. perform searches on one website
  6. / send requests to one API
  7. Read the data and interpret its schema
  8. (learn only one response format)

(Shows Demo)

One request format

GET /entities?queries=<queries>

  "q0": {
    "query": {
      "type": "Entity",
      "name~=": "John Smith",
      "jurisdiction_code|=": ["gb", "ie"],
      "memberships": [{
        "role": "director",
        "inactive": false

One request format, one response format

  • Metaweb Query Language (MQL) from Freebase
  • OpenRefine Reconciliation Service API
  • Popolo terms for field names


  • /entities
  • /relations
  • /lists


  • A library that accepts MQL parameters and returns the URL to send to each API
require 'whos_got_dirt'
require 'faraday'

input = {
  'name~=' => 'John Smith',
  'jurisdiction_code|=' => ['gb', 'ie'],
  'memberships' => [{
    'role' => 'director',
    'inactive' => false,

url =
#=> ""
  • A library that accepts an API response and returns results in Popolo format
response = Faraday.get(url)

results =
#=> [{"@type"=>"Person",
#  "name"=>"JOHN SMITH",
#  "updated_at"=>"2014-10-25T00:34:16+00:00",
#  "identifiers"=>[{"identifier"=>"46065070", "scheme"=>"OpenCorporates"}],
#  "links"=>[{"url"=>"", "note"=>"…"}],
#  "memberships"=>
#   [{"role"=>"director",
#     "organization"=>
#      {"name"=>"EVOLUTION (GB) LIMITED",
#       "identifiers"=>[{"identifier"=>"05997209", "scheme"=>"Company Registry"}],
#       "links"=>[{"url"=>"", "note"=>"…"}],
#       "jurisdiction_code"=>"gb"}}],
#  "current_status"=>"CURRENT",
#  "jurisdiction_code"=>"gb",
#  "occupation"=>"MANAGER",
#  "sources"=>
#   [{"url"=>"", “note"=>"OpenCorporates"}]}]
  • A server that handles all the ugly parts (error handling, request queueing, response caching, etc.)

Project scope

  • At least two APIs must agree on something for it to even be a candidate for inclusion


  • Add /relations and /lists endpoints
  • Support more APIs
  • Update Popolo

Issue trackers