# R script to retrieve historical data

**URL:** <https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046>\
**Category:** API\
**Created:** [December 22, 2022, 9:38pm UTC](https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046 "2022-12-22T21:38:08Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![willianflores](https://sea2.discourse-cdn.com/flex020/user_avatar/community.purpleair.com/willianflores/32/2123_2.png) [@willianflores](https://community.purpleair.com/u/willianflores)\
**Post date:** [December 22, 2022, 9:38pm UTC](https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046/1 "2022-12-22T21:38:08Z")

</div>

Hi everyone! I’m still new to this group, so I apologize in advance if this has been answered before. Does anyone know of an existing R script that uses the new API to create a data file of all historical data for a given list of sensors? I tried to access the data but got the following error while processing the data:

Error: lexical error: invalid char in json text.  
\<!doctype html\> \<html lang=“en”  
(right here) ------^

I coordinate an air quality monitoring network composed of 30 sensors and I need long-term historical data for several sensors. Any comments will be most welcome. Thank you very much.

Willian Flores

---

<div class="post-metadata">

**Author:** ![ltravis](https://avatars.discourse-cdn.com/v4/letter/l/ea5d25/32.png) [@ltravis](https://community.purpleair.com/u/ltravis)\
**Post date:** [December 23, 2022, 5:06pm UTC](https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046/2 "2022-12-23T17:06:53Z")

</div>

Hi Willian, we’ve developed a script to pull historical data. It may not be the most succinct or pretty, but it’s worked for us. This is for one sensor so you may have to adjust to make it a group of sensors.

#Code -----

library(tidyverse)  
library(httr) # authenticate API  
library(jsonlite) # read JSON  
library(lubridate) # work with dates

#Get current time - 2 days (Pull historical only allows 2 days of data at a time)  
current.epoch ← as.character(as.integer(  
as.POSIXct(Sys.time()) - days(2)  
))

#List of variables you’d like to pull, formatted to paste into web address  
fields=paste0(“?average=0&start\_timestamp=”, current.epoch, “&fields=latitude%2C%20longitude%2C%20altitude%2C%20rssi%2C%20humidity%2C%20temperature%2C%20pressure%2C%20pm2.5\_atm%2C%20pm2.5\_atm\_a%2C%20pm2.5\_atm\_b%2C%20pm2.5\_alt%2C%20pm2.5\_alt\_a%2C%20pm2.5\_alt\_b%2C%20pm2.5\_cf\_1%2C%20pm2.5\_cf\_1\_a%2C%20pm2.5\_cf\_1\_b%2C%20pm10.0\_atm%2C%20pm10.0\_atm\_a%2C%20pm10.0\_atm\_b%2C%20pm10.0\_cf\_1%2C%20pm10.0\_cf\_1\_a%2C%20pm10.0\_cf\_1\_b”)

#Pull historical data from API  
raw\_data\_hist ← httr::GET(url = paste(“[https://api.purpleair.com/v1/sensors/YY\_ADD\_SENSOR\_ID/history",fields,sep="](https://api.purpleair.com/v1/sensors/YY_ADD_SENSOR_ID/history%22,fields,sep=%22)”),  
config = add\_headers(‘X-API-Key’ = “YY\_ADD\_YOUR\_KEY”))

#Structurized data in form of R vectors and lists  
jsonParsed ← fromJSON(content(raw\_data\_hist, as=“text”))

#Dataframe from JSON data  
modJson ← as.data.frame(jsonParsed$data)  
names(modJson) ← jsonParsed$fields

---

<div class="post-metadata">

**Author:** ![willianflores](https://sea2.discourse-cdn.com/flex020/user_avatar/community.purpleair.com/willianflores/32/2123_2.png) [@willianflores](https://community.purpleair.com/u/willianflores)\
**Post date:** [December 23, 2022, 8:08pm UTC](https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046/3 "2022-12-23T20:08:49Z")

</div>

Hi Laura, thanks for your script, it worked perfectly… It will help a lot in the management of the air quality network in the State of Acre, Brazil.  
Thank you very much!

---

<div class="post-metadata">

**Author:** ![Lance](https://sea2.discourse-cdn.com/flex020/user_avatar/community.purpleair.com/lance/32/422_2.png) [@Lance](https://community.purpleair.com/u/Lance)\
**Post date:** [December 24, 2022, 9:18pm UTC](https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046/4 "2022-12-24T21:18:15Z")

</div>

Hello Laura Travis–

I was pleased to see that you are including the PM2.5\_alt algorithm along with the CF1 and CF\_ATM algorithms provided by Plantower in your R script. Based on my experience with all three algorithms, I would expect that the CF1 and CF\_ATM algorithms will give results 30-90% higher than the PM2.5\_alt algorithm. The latter has been compared to FEM/FRM regulatory monitors to obtain an accurate calibration factor for both the PMS1003 and PMS 5003 sensors used in the PA-I and PA-II monitors [1-3]. Recently a very large study was carried out to estimate long-term indoor exposures using indoor (4000) and outdoor (10,000) PurpleAir sites in the three West Coast states [4].

Are you making any indoor studies? If so, I can predict that there will be a large number of zeros returned by the CF1 and CF\_ATM algorithms. The PM2.5\_alt algorithm returns no zeros, since it is based on particle number and the particle number in the 0.3-0.5 um size category never falls to zero.

1. Wallace, L. Zhao, T. and Klepeis, N.E… Calibration of PurpleAir PA-I and PA-II monitors using daily mean PM2.5 concentrations measured in California, Washington, and Oregon from 2017 to 2021. _Sensors_ **2022, 22** , 4741. [https://doi.org/10.3390/](https://doi.org/10.3390/) s22134741 [Calibration of PurpleAir PA-I and PA-II Monitors Using Daily Mean PM2.5 Concentrations Measured in California, Washington, and Oregon from 2017 to 2021 - PubMed](https://pubmed.ncbi.nlm.nih.gov/35808235/)

2. Wallace, L. Intercomparison of PurpleAir Sensor Performance over Three Years Indoors and Outdoors at a Home: Bias, Precision, and Limit of Detection Using an Improved Algorithm for Calculating PM2.5. _Sensors_ **2022** , _22_, 2755. [Sensors | Free Full-Text | Intercomparison of PurpleAir Sensor Performance over Three Years Indoors and Outdoors at a Home: Bias, Precision, and Limit of Detection Using an Improved Algorithm for Calculating PM2.5](https://doi.org/10.3390/s22072755)

3.Wallace, L., Bi, J., Ott, W.R., Sarnat, J.A. and Liu, Y. (2021) Calibration of low-cost PurpleAir outdoor monitors using an improved method of calculating PM2.5. _Atmospheric Environment_, 256 (2021) 118432. https//doi.org/10.1016/j.atmosenviron.2021.118432 [Calibration of low-cost PurpleAir outdoor monitors using an improved method of calculating PM2.5 - ScienceDirect](https://www.sciencedirect.com/science/article/abs/pii/S135223102100251X)

1. Wallace, L.A., Zhao, T., Klepeis, N.R. 2022 Indoor contribution to PM2.5 exposure using all PurpleAir sites in Washington, Oregon, and California.Indoor Air 32: (9) 13105. [https://onlinelibrary.wiley.com/doi/abs/10.1111/ina.13105](https://onlinelibrary.wiley.com/doi/abs/10.1111/ina.13105).

---

<div class="post-metadata">

**Author:** ![Sicheng\_Dai](https://sea2.discourse-cdn.com/flex020/user_avatar/community.purpleair.com/sicheng_dai/32/1813_2.png) [@Sicheng\_Dai](https://community.purpleair.com/u/Sicheng_Dai)\
**Post date:** [January 5, 2023, 6:18pm UTC](https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046/5 "2023-01-05T18:18:42Z")

</div>

Hi Laura, I am trying to use your code to pull historical data. However, I got the same lexical error as Willian. I apologize that I am really new to coding and R, but I really need to figure this out for my thesis.

Here is my code

```auto
library(tidyverse)
library(httr) # authenticate API
library(jsonlite) # read JSON
library(lubridate) # work with dates

#Get current time - 2 days (Pull historical only allows 2 days of data at a time)
current.epoch <- as.character(as.integer(
as.POSIXct(Sys.time()) - days(2)
))

#List of variables you’d like to pull, formatted to paste into web address
fields=paste0("?average=0&start_timestamp=", current.epoch, "&fields=latitude%2C%20longitude%2C%20altitude%2C%20rssi%2C%20humidity%2C%20temperature%2C%20pressure%2C%20pm2.5_atm%2C%20pm2.5_atm_a%2C%20pm2.5_atm_b%2C%20pm2.5_alt%2C%20pm2.5_alt_a%2C%20pm2.5_alt_b%2C%20pm2.5_cf_1%2C%20pm2.5_cf_1_a%2C%20pm2.5_cf_1_b%2C%20pm10.0_atm%2C%20pm10.0_atm_a%2C%20pm10.0_atm_b%2C%20pm10.0_cf_1%2C%20pm10.0_cf_1_a%2C%20pm10.0_cf_1_b")

#Pull historical data from API
raw_data_hist <- httr::GET(url = paste("https://map.purpleair.com/1/b/i/mAQI/a10/p0/cC0?select=101555#12.75/43.65301/-79.3985", fields, sep=" "),
config = add_headers("X-API-Key" = "DB0239FE- ****-11ED-**** - ***********"))

#Structurized data in form of R vectors and lists
jsonParsed <- fromJSON(content(raw_data_hist, as='text'))

#Dataframe from JSON data
#modJson <- as.data.frame(jsonParsed$data)
#names(modJson)<- jsonParsed$fields

```

This is the error  
Error: lexical error: invalid char in json text. \<!doctype html\> \<html lang=“en” (right here) ------^

1. parse\_string(txt, bigint\_as\_char)

2. parseJSON(txt, bigint\_as\_char)

3. parse\_and\_simplify(txt = txt, simplifyVector = simplifyVector, simplifyDataFrame = simplifyDataFrame, simplifyMatrix = simplifyMatrix, flatten = flatten, …)

4. fromJSON(content(raw\_data\_hist, as = “text”))

Thank you very much!

Sicheng

---

<div class="post-metadata">

**Author:** ![willianflores](https://sea2.discourse-cdn.com/flex020/user_avatar/community.purpleair.com/willianflores/32/2123_2.png) [@willianflores](https://community.purpleair.com/u/willianflores)\
**Post date:** [January 6, 2023, 7:21am UTC](https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046/6 "2023-01-06T07:21:04Z")

</div>

Hi Sicheng, I have developed an R function to download historical data from PurpleAir sensors and it is available on my GitHub page [https://github.com/willianflores/getPurpleairApiHistory](https://github.com/willianflores/getPurpleairApiHistory). The function is still in the testing phase, if you use it and find any errors, please let me know.

Willian Flores

---

<div class="post-metadata">

**Author:** ![Shyra\_LaGarde](https://sea2.discourse-cdn.com/flex020/user_avatar/community.purpleair.com/shyra_lagarde/32/1868_2.png) [@Shyra\_LaGarde](https://community.purpleair.com/u/Shyra_LaGarde)\
**Post date:** [January 12, 2023, 3:04pm UTC](https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046/8 "2023-01-12T15:04:33Z")

</div>

Hi Willian I tried using your code from GitHub im running into an error Error in structure(.External(.C\_dotTclObjv, objv), class = “tclObj”) :  
[tcl] invalid command name “toplevel”.  
Is it possible to do execute without the tcltk package? do you have any suggestions. I have a Mac I install XQuarts the required dependency.

---

<div class="post-metadata">

**Author:** ![willianflores](https://sea2.discourse-cdn.com/flex020/user_avatar/community.purpleair.com/willianflores/32/2123_2.png) [@willianflores](https://community.purpleair.com/u/willianflores)\
**Post date:** [January 13, 2023, 7:13pm UTC](https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046/9 "2023-01-13T19:13:41Z")

</div>

Hi Shyra, this function was developed in a Windows environment and I have not tested it in a Mac environment.

---

<div class="post-metadata">

**Author:** ![apeer](https://avatars.discourse-cdn.com/v4/letter/a/b77776/32.png) [@apeer](https://community.purpleair.com/u/apeer)\
**Post date:** [January 24, 2023, 5:01pm UTC](https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046/10 "2023-01-24T17:01:13Z")

</div>

Hi Willian, I found no errors running the function on my mac on RStudio. However, I think I found a bug with the 6-hour average pull. It appears that the 6 pm average reading every other day is missing. For reference, I tried to get 6-hour averages for multiple sensors for December 2022. I have not tried it with other averages yet. Otherwise, thank you so much for putting this together! It has been very helpful.

---

<div class="post-metadata">

**Author:** ![Aaron\_Lamplugh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.purpleair.com/aaron_lamplugh/32/651_2.png) [@Aaron\_Lamplugh](https://community.purpleair.com/u/Aaron_Lamplugh)\
**Post date:** [March 1, 2023, 5:50pm UTC](https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046/11 "2023-03-01T17:50:22Z")

</div>

If people are still interested in this topic, I have recently posted a python script to download group data from a historical period through the API:

> [@Python script for downloading and organizing historical API data](https://community.purpleair.com/t/python-script-for-downloading-and-organizing-historical-api-data/3726):
>
> Hey all, I just finished this python script to access historical data through the API. It pulls data for all monitors in a registered group. It then creates a file structure where it dumps the data and compiles it into a single CSV for each monitor. It’s currently setup to download the files in a very similar format to the way the thingspeak download page used to work (primary and secondary A and B channels). I also have it making a files similar to the format that comes off the SD cards. This …

---

<div class="post-metadata">

**Author:** ![Aaron\_Lamplugh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.purpleair.com/aaron_lamplugh/32/651_2.png) [@Aaron\_Lamplugh](https://community.purpleair.com/u/Aaron_Lamplugh)\
**Post date:** [March 1, 2023, 5:53pm UTC](https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046/12 "2023-03-01T17:53:22Z")

</div>

I realize this is a thread about R, which is also my native language, but I found that python was a much better solution for this type of problem. Once it’s downloaded, I still use R for all of my processing and analyses.

---

<div class="post-metadata">

**Author:** ![willianflores](https://sea2.discourse-cdn.com/flex020/user_avatar/community.purpleair.com/willianflores/32/2123_2.png) [@willianflores](https://community.purpleair.com/u/willianflores)\
**Post date:** [March 1, 2023, 6:50pm UTC](https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046/13 "2023-03-01T18:50:53Z")

</div>

Hello Alyssa, I have identified this problem, apparently it is not related to the R code, but to the processing of the API itself. What I have been doing is downloading the data with an average of 10 minutes and doing a post-processing to obtain the other time averages, if necessary.

---

<div class="post-metadata">

**Author:** ![willianflores](https://sea2.discourse-cdn.com/flex020/user_avatar/community.purpleair.com/willianflores/32/2123_2.png) [@willianflores](https://community.purpleair.com/u/willianflores)\
**Post date:** [March 1, 2023, 6:55pm UTC](https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046/14 "2023-03-01T18:55:06Z")

</div>

Hi Aaron, thank you very much for pointing out new possibilities for accessing PurpleAir data.

---

<div class="post-metadata">

**Author:** ![TyM](https://avatars.discourse-cdn.com/v4/letter/t/90ced4/32.png) [@TyM](https://community.purpleair.com/u/TyM)\
**Post date:** [March 7, 2023, 2:43am UTC](https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046/15 "2023-03-07T02:43:21Z")

</div>

Hi Willian,

I tried using your code that you posted earlier, however, I am met with the error: 'Could not find function “getPurpleairApiHistory”

I am wondering what package you used that had this function.

Thank you!

---

<div class="post-metadata">

**Author:** ![Jpern](https://avatars.discourse-cdn.com/v4/letter/j/a88e57/32.png) [@Jpern](https://community.purpleair.com/u/Jpern)\
**Post date:** [March 7, 2023, 4:52am UTC](https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046/16 "2023-03-07T04:52:05Z")

</div>

That I could help with, you have to go to where the function is saved also on the github and copy and paste the whole thing into R and run it, so the function for R is actually loaded into memory.

I have a question for William though, I ran it, and got this error,

Error in sprintf(“web service error %s from:\n %s\n\n%s”, status\_code, :  
object ‘webserviceUrl’ not found

Which seems to suggest that a URL is bad in the function? Or am I entering my information incorrectly?

---

<div class="post-metadata">

**Author:** ![apeer](https://avatars.discourse-cdn.com/v4/letter/a/b77776/32.png) [@apeer](https://community.purpleair.com/u/apeer)\
**Post date:** [March 22, 2023, 6:44am UTC](https://community.purpleair.com/t/r-script-to-retrieve-historical-data/3046/17 "2023-03-22T06:44:40Z")

</div>

Hi John, I have run into the same error and believe this happens when the data for a specific sensor does not exist at the specified time frame or when a sensor is private. I have bypassed this error by inserting the try() function at line 209. This will keep the code running through API calls that don’t return anything. I have found this to be helpful when querying multiple sensors over time. The code would look like how it does below. You could also implement the trycatch() function if you want to keep the error message but ensure the loop continues.

> r\_temp ← try(httr::GET( # added try so that loop will continue regardless of error  
> URLbase,  
> query = queryList,  
> config = add\_headers(“X-API-Key” = apiReadKey)  
> ))