extract - Tool for periodic mirroring of full data-set from GraphQL endpoint?

Question

I'm looking for a tool that can make a clone of data exposed on a GraphQL API.

Basically something that can run periodically and recurively copy the raw data reponse to disk, making use of connection based pagination & cursors to ensure consistency of progress of the mirrored content.

Assuming this would be a runner that extracts data 24/7, it will either have to rewrite/transform already copied data, or even better apply updates in a more event sourced way to make it easier to provide diff-sets of changes in the source API data.

question from:https://stackoverflow.com/questions/66055462/tool-for-periodic-mirroring-of-full-data-set-from-graphql-endpoint

Answer 1

I'm not aware of any such tool. I'm not sure such a tool will exist, because

1. retrieving data from GraphQL requires only the thinnest of layers over the existing GraphQL libraries, which are quite feature-rich
2. the transformation/writing will likely be part of a different tool. I'm sure several tools for this already exist. The simplest example I could think of is Git. Getting a diff is as simple as running git diff after overwriting an existing version-controlled file.

A simple example of retrieving the data is adapted from the graphql-request Documentation Quickstart

import { request, gql } from 'graphql-request'
import { writeFile } from 'fs/promises'

const query = gql`
  {
    Movie(title: "Inception") {
      releaseDate
      actors {
        name
      }
    }
  }
`

request('https://api.graph.cool/simple/v1/movies', query)
  .then((data) => writeFile('data.json', data))