API Catalog Blog Post

[flagbrad]

A high-level overview of major USGS water-data APIs, conveyed in a logical ordering that encourages uptake of newer APIs preferentially over older legacy ones.

[flagbrad]

Ready for USGS internal peer review, link available on internal Teams chat (or view content/api_catalog.md)

[padilla410]

Hi Brad,

FIrst, great job. It's clear a lot of work went into this and I like the overall tone. I was only able to make it through the "Water Quality Portal" section. Here are my comments:

• Intro

  • the link for smaller group of users results in a broken link for me
  • API is not defined anywhere in a way that maps to the letter "API". Here is the line from the blog:

    "...our water-data web services (APIs)"

• OGC sensorthings API

  • "OGC" doesn't appear to be defined anywhere (I did a quick search for the term)
  • Missing a period in the end of the first paragraph for "OGC SensorThings API"
  • The sentence "We are developing a Python Notebook" feels incomplete. Maybe "We are developing a Python Notebook so you can test it yourself." <-- (adjust tone to match whatever you're trying to hit for the article). Also is there a timeline for when and where it will be available?
  • Are most folks familiar with the concept of "things" in relation to IoT? If not, is there a reasonable link that could be put in here?
  • Item 1 - I would probably italicize the "thing" in the "For example" sentence or something to denote that it isn't just the word "thing", but I'm not sure what best practice is here.
  • A few things in Item 2
    • "pursued" --> perhaps "used" instead?
    • Is the parenthetical statement needed? Perhaps the link should be on the word "this" instead?
    • I clicked the link under for the line "which should point here" and then did a search for the term Datastreams@iot.navigationLink and did not find a result. Is this the expected outcome? I did find Observations@iot.navigationLink in the link.
  • Item 3
    • I like that you've included the translation for two popular parameter codes. Perhaps a chance to link to other USGS docs?
  • Overall I these items in this list are very close. It just needs to be re-worked a bit for clarity.

• OGC API

  • no comments. Seems good.

• Water quality portal

  • "Then the flagship Water Quality Portal." <- this sentence seems incomplete
  • It feels like a number of the topics mentioned in the second paragraph here could be links to doc (e.g., "Water Quality eXchange (WQX) open standard")
  • I would change the second sentence in the second paragraph to this:

    " Some of our users have expressed express interest in a wide format which is in a simplified gridded output where constituents define columns, laboratory samples define rows, and numeric results and qualifiers are in each grid cell."

  • Folks may not know the difference between "narrow" and "wide" and I think these should be described somewhere in here. Perhaps the sentence after this one could describe what "narrow" means.
  • I think maybe one more sentence is needed after "it simply sacrifices too much essential detail". It may not be clear to the user what is being given up because wide is great for many of them. Perhaps something like "we use a narrow format because XYZ".

[flagbrad]

Brad's Catalog Blog

[wdwatkins]

Looks good to me, the main thing I noticed was this link was a 404: https://webvadevvs04.er.usgs.gov/updates/user_operationalized_pull/. But might just the dev tier?

[flagbrad]

Comments from @wdwatkins and @padilla410 have all been incorporated and will be in a forthcoming pull request. Thank you!

[BlakeDraper]

Great work on this. Overall a great post with a wealth of info. I learned a lot! I actually would love to see this formed into a permanent reference page for USGS water data services since blogs are somewhat ephemeral. This is the clearest round-up of all the available data services I have ever seen.

Now for my few critiques:

You introduce "JSON structure" without explaining what JSON is. I know it's outside the scope of this post to explain JSON, but I think you need to at least define it as a common web data structure and link to an explainer. I would maybe add a footnote about it, or move up the one you have (#3) to the first mention of JSON and expand on it.

---

This part from the OGCSensorThinngs API section:

Try iterating through this array searching for the 5-digit USGS parameter code you are interested in (often, it’s 00060 for Discharge or 00065 for Gage Height).

Is a bit confusing because the URL the reader gets from Step 2 results in array like this:

In which each of those objects in the array is fairly complex object with more objects nested, and the parameter code is not even at the root of that object. I get what you mean when you say iterate through it, but will your audience get it? If web developers is the audience, then probably fine. If not, maybe clarify this part.

---

or for the more stalwart reader there is the official SensorThings specification.

Be careful here. It implies if the reader is intimidated by the full spec, they are not stalwart (loyal, reliable, hardworking). I may use something more like 'adventurous' or just 'interested'.

---

For most large rivers and non-fractured groundwater systems, one daily value per day is adequately if not wholly representative of that water resource–less data but just as informative as dozens of instantaneous values per day. But for small rivers and sundry hydrological phenomena, a daily value is an expedient trade-off that discards useful information

I appreciate the writing / voice here, but it may be a little on the verbose side for a public-friendly plain language blog. Consider rephrasing.

[flagbrad]

All comments from @BlakeDraper have been incorporated into a forthcoming pull request. Thank you!

[flagbrad]

Addressed comments in issue #615. Also addressed comments from lstanish as follows:

• Adjusted language to call all URL arguments “input parameters” to align with Water Quality Portal documentation
• Adjusted conventions of API and web service to align with WQP documentation.
• Added footnote about challenge of data discovery using 5-digit parameters codes. Unfortunately there is no concise publicly-available list of the codes in use only for instantaneous-values datastreams (though there could be someday if need be)
• Clarified what plans are for providing OGC API resources.
• Clarified that WQP offers embeddable code only in the advanced form.
• Clarified that our OGC API tutorials and documentation are still rudimentary, asking readers to check back. A February API webinar hopefully will spur the generation of some of these materials.
• Reworded NOTE in WQP section as suggested. Thank you! Much clearer.
• Reworded the NLDI second paragraph as suggested, using “relationships” as key word of the paragraph.
• Clarified that NLDI example is just an example and not an exhaustive examination of the NLDI API.
• Reworded legacy instantaneous-values language. One cannot simply call these “sensor measurements,” because (for example) streamflow is a derived measurement, not a direct sensor measurement. But I think I was able to improve phrasing in any case.
• Revised daily-values language as suggested to reference flagship API alternatives.
• Simplified deprecated section’s language as suggested.

And answers to some questions from lstanish:

• No, discrete surface-water field measurements are not available through water-quality web services. What you MIGHT be think about is the fact that, for convenience of the data analyst, discrete water-quality data often provide the closest 15-minute sensor reading/calculation of gage height and discharge. These are not actual manual measurements by field staff during the sampling event, but are just redundant copies of instantaneous-value data injected into the water-quality dataset. It’s convenient for end users, but certainly has been confusing over the years. In a modernized system, my recommendation would be these gage height and discharge values be obtained ON THE FLY from web services, instead of being stored as duplicate copies in the discrete water-quality information system.
• Yes, SensorThings presently is for recent (past 120 days) time-series data from environmental sensors or computationally derived from data from those sensors.
• For SensorThings, as far as I know WaterML is NOT part of its output formats. If I am in error @jkreft-usgs please let me know; is WaterML folded in as any kind of subschema in USGS SensorThings implementation?

[flagbrad]

Revisions completed based on management review. After pull request is merged, api_catalog.md is ready for publication to the production tier.