Changes to NWIS water quality services

USGS discrete water samples data are undergoing modernization, and NWIS services will no longer be updated with the latest data starting mid-March 2024, with a full decommission expected 6 months later.

For the latest news on USGS water quality data, see: https://doi-usgs.github.io/dataRetrieval/articles/Status.html. Learn more about the changes and where to find the new samples data in the WDFN blog.

What does this mean for dataRetrieval users? Eventually water quality data will ONLY be available from the Water Quality Portal rather than the NWIS services. There are 3 major dataRetrieval functions that will be affected: readNWISqw, whatNWISdata, and readNWISdata. This vignette will describe the most common workflows conversions needed to update existing scripts.

How to find more help

This vignette is being provided in advance of any breaking changes, and more information and guidance will be provided. These changes are big, and initially sound overwhelming. But in the end, they are thoughtful changes that will make understanding USGS water data more intuitive. Please reach out for questions and comments to: [email protected]

WQX2 -> WQX3

If you have already converted your workflow from NWIS to WQP, much of the hard work has been done! The final step of the process is to make sure workflows are using the most modern WQX3 formats. WQX stands for Water Quality Exchange. WQX2 is the format that has been historically available on the Water Quality Portal, WQX3 is the more modern format that all the data is being converted to.

This is available as of dataRetrieval version 2.7.16. Starting with 2.7.16, all WQP functions default to the newer “WQX3” profiles if available. See WQX Conversions below to get a table of column names in WQX3 vs WQX2.

NWIS -> WQX3

wqpData <- readWQPqw(paste0("USGS-", site_ids), parameterCd)

nrow(nwisData)

## [1] 208

nrow(wqpData)

## [1] 208

ncol(nwisData)

## [1] 36

ncol(wqpData)

## [1] 67

names(attributes(nwisData))

##  [1] "names"        "class"        "row.names"   
##  [4] "queryTime"    "url"          "headerInfo"  
##  [7] "comment"      "siteInfo"     "variableInfo"
## [10] "header"

names(attributes(wqpData))

## [1] "names"      "row.names"  "class"      "headerInfo"
## [5] "legacy"     "siteInfo"   "queryTime"  "url"

site_NWIS <- attr(nwisData, "siteInfo")
site_WQP <- attr(wqpData, "siteInfo")

library(dplyr)

nwisData_relevant <- nwisData |> 
  select(
    site_no, startDateTime, parm_cd,
    remark_cd, result_va
  ) |> 
  arrange(startDateTime, parm_cd)

knitr::kable(head(nwisData_relevant))

wqpData_relevant <- wqpData |> 
  select(
    site_no = Location_Identifier,
    startDateTime = Activity_StartDateTime,
    parm_cd = USGSpcode,
    remark_cd = Result_ResultDetectionCondition,
    result_va = Result_Measure,
    detection_level = DetectionLimit_MeasureA
  ) |> 
  arrange(startDateTime, parm_cd)
knitr::kable(head(wqpData_relevant))

censored_text <- c(
  "Not Detected",
  "Non-Detect",
  "Non Detect",
  "Detected Not Quantified",
  "Below Quantification Limit"
)

wqpData_relevant <- wqpData |> 
  mutate(left_censored = grepl(paste(censored_text, collapse = "|"),
    Result_ResultDetectionCondition,
    ignore.case = TRUE
  )) |> 
  select(
    site_no = Location_Identifier,
    startDateTime = Activity_StartDateTime,
    parm_cd = USGSpcode,
    left_censored,
    result_va = Result_Measure,
    detection_level = DetectionLimit_MeasureA,
    dl_units = DetectionLimit_MeasureUnitA
  ) |> 
  arrange(startDateTime, parm_cd)

knitr::kable(head(wqpData_relevant))

whatNWISdata

This function will continue to work for any service EXCEPT “qw” (water quality discrete data). “qw” results will eventually no longer be returned, and are currently showing values that were frozen in March 2024.

The function to replace this functionality for discrete water quality data is currently whatWQPdata:

whatNWIS <- whatNWISdata(
  siteNumber = site_ids,
  service = "qw"
)

WARNING: NWIS does not deliver
new discrete water quality data or updates to existing data. 
For additional details, see:
https://doi-usgs.github.io/dataRetrieval/articles/Status.html

whatWQP <- whatWQPdata(siteNumber = paste0("USGS-", site_ids))

There are some major differences in the output. The NWIS services offers back one row per site/parameter code to learn how many samples are available. This is not currently available from the Water Quality Portal, however there are new summary services being developed. When those become available, we will include new documentation on how to get this information.

readNWISdata

If you get your water quality data from the readNWISdata function, no new data will be available and the function will generate a warning message. The other services are working as before. This is not an especially common dataRetrieval workflow, so there are not a lot of details here. Please reach out if more information is needed to update your workflows.

See ?readWQPdata to see all the ways to query data in the Water Quality Portal. Use the suggestions above to convert the output of the readWQPdata function to convert the WQP output to what is important for your workflow.

WQX Conversion

A table is provided on the EPA website that shows the conversions of WQX3 to the WQX2 mappings. This table may change periodically while WQP services are under active development, which is expected to last through Fall 2024.

Here is an example for pulling the EPA table and comparing column names.

schema <- readr::read_csv("https://www.epa.gov/system/files/other-files/2024-07/schema_outbound_wqx3.0.csv")

Within that schema table, the column “FieldName3.0” shows the column names for the new WQX3 profile. There are several “FieldName2.0.XXXX” columns that show how the older 2.0 profiles align with the newer columns.

For example, the 2.0 “narrow” dataProfile match up with these new WQP3 columns:

sub_schema <- schema |> 
  select(WQX3 = FieldName3.0,
         WQX2 = FieldName2.0.Narrow) |> 
  filter(!is.na(WQX2))

knitr::kable(sub_schema)

Disclaimer

This information is preliminary and is subject to revision. It is being provided to meet the need for timely best science. The information is provided on the condition that neither the U.S. Geological Survey nor the U.S. Government may be held liable for any damages resulting from the authorized or unauthorized use of the information.