Add charity navigator data import script

[slifty]

We want to pull in data from the Charity Navigator API.

For now this will use a few "skunk works" api endpoints in the service which essentially just store raw responses for use in the data viewer.

[bickelj]

Some progress, expand for gory detail.

First, the following link is not the GraphQL API: https://charitynavigator.github.io/api/CharityNavigatorAPI/generated/RZenHtmlDocs/CharityNavigator_doc.html

Second, the specs do not appear in the API products pages, it looks like this:

Third, the playground page is also sparse, it looks like this:

However, the https://github.com/CharityNavigator/cn-examples repository is extremely useful in that it provides an example of working calls.

The example software uses Apollo Client.

Adding the following fetch arg to cnapi.js's createHttpLink args from cn-examples repo allows us to see the actual HTTP request body on the console (special thanks to apollographql/apollo-client#4017 (comment)):

  const PRINT_RAW_REQUEST_BODIES = true

// ...

    fetch: (...pl) => {
      if (!PRINT_RAW_REQUEST_BODIES) return fetch(...pl);
      const [_, options] = pl;
      console.log(`🍣 ${options.body}`);
      return fetch(...pl);
    },

At first I didn't think this looked right because it looks like this:

🍣 {"operationName":"PublicSearchFaceted","variables":{"term":"","states":["AL","AK","FL","OH","WY","WI"],"sizes":[],"causes":["Medical education","Nursing education","Health"],"ratings":["1","2","3","4"],"c3":false,"from":1200,"result_size":200,"beacons":[],"advisories":[],"orderBy":""},"query":"query PublicSearchFaceted($term: String!, $states: [String!]!, $sizes: [String!]!, $causes: [String!]!, $ratings: [String!]!, $c3: Boolean!, $result_size: Int!, $from: Int!, $beacons: [String!]!, $advisories: [String!]!, $orderBy: String!) {\n publicSearchFaceted(\n term: $term\n states: $states\n sizes: $sizes\n causes: $causes\n ratings: $ratings\n c3: $c3\n result_size: $result_size\n from: $from\n beacons: $beacons\n advisories: $advisories\n order_by: $orderBy\n ) {\n size\n from\n term\n result_count\n results {\n ein\n name\n mission\n organization_url\n charity_navigator_url\n encompass_score\n encompass_star_rating\n encompass_publication_date\n cause\n street\n street2\n city\n state\n zip\n country\n highest_level_alert\n __typename\n }\n __typename\n }\n}"}

But confirmation comes from wireshark session when decrypting TLS and seeing this, it's the same body:

POST /graphql HTTP/1.1
host: api.charitynavigator.org
connection: keep-alive
accept: */*
content-type: application/json
authorization: [api_key_here]
accept-language: *
sec-fetch-mode: cors
user-agent: node
accept-encoding: br, gzip, deflate
content-length: 1216

{"operationName":"PublicSearchFaceted","variables":{"term":"","states":["AL","AK","FL","OH","WY","WI"],"sizes":[],"causes":["Medical education","Nursing education","Health"],"ratings":["1","2","3","4"],"c3":false,"from":1200,"result_size":200,"beacons":[],"advisories":[],"orderBy":""},"query":"query PublicSearchFaceted($term: String!, $states: [String!]!, $sizes: [String!]!, $causes: [String!]!, $ratings: [String!]!, $c3: Boolean!, $result_size: Int!, $from: Int!, $beacons: [String!]!, $advisories: [String!]!, $orderBy: String!) {\n publicSearchFaceted(\n term: $term\n states: $states\n sizes: $sizes\n causes: $causes\n ratings: $ratings\n c3: $c3\n result_size: $result_size\n from: $from\n beacons: $beacons\n advisories: $advisories\n order_by: $orderBy\n ) {\n size\n from\n term\n result_count\n results {\n ein\n name\n mission\n organization_url\n charity_navigator_url\n encompass_score\n encompass_star_rating\n encompass_publication_date\n cause\n street\n street2\n city\n state\n zip\n country\n highest_level_alert\n __typename\n }\n __typename\n }\n}"}


So the bodies are not easy to construct manually, they appear to require software.

This still doesn't answer the question as to what's available.

I was able to query the API with an introspection request as described here and got a monster JSON response. Hurray.

Overview of the response body:

cn-examples$ jq 'keys' graphql_introspection_response.json 
[
  "data"
]
cn-examples$ jq '.data | keys' graphql_introspection_response.json 
[
  "__schema"
]
cn-examples$ jq '.data.__schema | keys' graphql_introspection_response.json 
[
  "directives",
  "mutationType",
  "queryType",
  "subscriptionType",
  "types"
]
cn-examples$ jq '.data.__schema.directives | keys' graphql_introspection_response.json
[
  0,
  1,
  2
]
cn-examples$ jq '.data.__schema.mutationType | keys' graphql_introspection_response.json
[
  "name"
]
cn-examples$ jq '.data.__schema.querytype | keys' graphql_introspection_response.json
jq: error (at graphql_introspection_response.json:1): null (null) has no keys
cn-examples$ jq '.data.__schema.subscriptionType | keys' graphql_introspection_response.json
[
  "name"
]
cn-examples$ jq '.data.__schema.types | keys' graphql_introspection_response.json
[
  0,
  1,
  2,
  3,

[bickelj]

Some more progress, expand for gory detail.

This tool helped generate an SVG of the possibilities. A nice friendly diagram! https://github.com/NathanRSmith/graphql-visualizer

There is a nonprofitsPublic Query available, apparently, that takes a filter of NonprofitFilters. I'll see if I can call that one. I guess there's also the bulkNonprofits already written into the example, so if I can filter on that, that's a good one too.

[bickelj]

More progress, expand for gory detail

Hurray, finally have a gql declaration that queries for a given EIN!

const queryNonprofitsPublic = gql`
  query NonprofitsPublic(
    $perPage: Int!
    $filter: NonprofitFilters
  ) {
    nonprofitsPublic(
      filter: $filter
    ) {
        edges {
          ein
          name
        }
        pageInfo {
          totalPages
          totalItems
          currentPage
        }
    }
  }
`;

curl -X POST   -H "Content-Type: application/json" -H "Authorization: Bearer [token]"   -d '{"operationName":"NonprofitsPublic","variables":{"filter":{"ein":{"in":["043091431"]}}},"query":"query NonprofitsPublic($perPage: Int!, $filter: NonprofitFilters) {\n  nonprofitsPublic(filter: $filter) {\n    edges {\n      ein\n      name\n      __typename\n    }\n    pageInfo {\n      totalPages\n      totalItems\n      currentPage\n      __typename\n    }\n    __typename\n  }\n}"}' https://api.charitynavigator.org/graphql

Response:

{"data":{"nonprofitsPublic":{"edges":[{"ein":"043091431","name":"Electronic Frontier Foundation (EFF)","__typename":"NonprofitPublic"}],"pageInfo":{"totalPages":1,"totalItems":1,"currentPage":1,"__typename":"PageInfoNew"},"__typename":"PaginatedNonprofitPublic"}}}

[slifty]

Note that we're almost finished with PhilanthropyDataCommons/service#1321 and I suspect that will be an important endpoint for this script!

[bickelj]

From this morning's meeting, for the moment, we'll (manually) pass the IDs of the changemakers as an argument to the script.

[bickelj]

Progress...

I have a rudimentary integration into the data-scripts of the basic thing I got working before.

data-scripts$ npm run start -- charityNavigator lookup --charity-navigator-api-key=the_key eins=[043091431]

> data-scripts@1.0.0 start
> ts-node src/index charityNavigator lookup --charity-navigator-api-key=the_key eins=[043091431]

[2025-12-17 3:18:57 PM] INFO: CharityNavigator result
    result: {
      "data": {
        "nonprofitsPublic": {
          "__typename": "PaginatedNonprofitPublic",
          "edges": [],
          "pageInfo": {
            "__typename": "PageInfoNew",
            "totalPages": 0,
            "totalItems": 0,
            "currentPage": 1
          }
        }
      }
    }

Though apparently the eins are not being properly passed.

[bickelj]

More progress

Yargs has a special manner in which to pass command-line arguments. First, the option was wrong, it's --ein not ein, Second, don't include JSON such as brackets and quotes in the value, so 012345 not ["012345"]. Third, multiple values can be specified either by --ein=value1 --ein=value2 or --ein value1 value2, no commas, only spaces or separate options.

Successfully querying two EINs simultaneously:
npm run start -- charityNavigator lookup --charity-navigator-api-key=the_key --eins 043091431 042888848

[2025-12-18 11:13:53 AM] INFO: CharityNavigator result
    result: {
      "data": {
        "nonprofitsPublic": {
          "__typename": "PaginatedNonprofitPublic",
          "edges": [
            {
              "__typename": "NonprofitPublic",
              "ein": "042888848",
              "name": "Free Software Foundation"
            },
            {
              "__typename": "NonprofitPublic",
              "ein": "043091431",
              "name": "Electronic Frontier Foundation (EFF)"
            }
          ],
          "pageInfo": {
            "__typename": "PageInfoNew",
            "totalPages": 1,
            "totalItems": 2,
            "currentPage": 1
          }
        }
      }
    }

[bickelj]

I have the script querying PDC for changemakers and then using a list of valid EINs from that to query Charity Navigator. At the moment the next thing to get working is the TypeScript types of the Apollo response object so that I can type-safely access various objects from the response body.