shell - How to retrieve list of 1000 plus repoistories in a GitHub org using GitHub API

Question

I am using the below script to generate a list of repositories in one of my GitHub Enterprise orgs and it works fine; however, by default it only fetches 100 repos at a time.

How can I modify it to generate the entire list? I have some 2000 repos in my GitHub org.

curl --silent --user "myusername:mypassword" "https://github.***.com/api/v3/orgs/myorg/repos?page=1&per_page=2000" | npx jq '.[].clone_url' | while read repo
do
    repo="${repo%"}"
    repo="${repo#"}"
    echo "$repo"
done > repolist.txt

I am unable to tweak the page=*&per_page=* here and no matter what number combinations I use, when I execute the above shell script, a file called repolist.txt is generated with the list of first 100 repos in the GitHub org.

question from:https://stackoverflow.com/questions/65834527/how-to-retrieve-list-of-1000-plus-repoistories-in-a-github-org-using-github-api

Answer 1

From docs

You can specify how many items to receive (up to a maximum of 100);

You can go to /orgs/{org} and read public_repos

With this value, you can make total_pages=$(($public_repos / 100 + 1)) and iterate in total_pages incrementing your page prop.

Below is a small code, just add your credentials and Org Name:

#!/bin/bash

user=""
password=""
org=""
public_repos=$(curl -s -u "${user}:${password}" "https://api.github.com/orgs/${org}" | jq .public_repos)
per_page=100
total_pages=$(($public_repos / $per_page + 1))

for page in $(seq 1 $total_pages); do
  curl -s -u "${user}:${password}"
    "https://api.github.com/orgs/${org}/repos?page=${page}&per_page=${per_page}" | 
    jq -r '.[].clone_url'
done