Events, stories, historical visualizations

Data
3

I took a try at this using the below python script which will take a group of csv files in a given folder and use the time and last column value in the air quality file to create a corresponding kml file, but forgot how poorly google earth handles a large number of points(say > 15k points). I downloaded all the map points at the below San Francisco map location from September 1 to September 20, 2020 when the ‘orange sky’ news event occured, it was around 13,000 files and took several hours, probably could have run this download part faster/better using the official API than the map download page. Using the map download page, would have been nice to be able to exclude empty files(where data does not exist historically at a platform) and filter say for only ‘A’ sensors (not redundant ‘B’) and ‘outside’. The kml file generated was around 100+ megabytes unzipped, several megabytes zipped as a kmz file and google earth became unusable trying to load/show it.

So the next visualization step I might try would be to reformat the output into a single csv file(time,lat,long,values) that I could load into QGIS and have QGIS run through the data and create an animated gif of the changing color point values for a region.

https://www.qgistutorials.com/en/docs/3/animating_time_series.html

San Francisco region target map

from glob import glob
import sys

import datetime

import simplekml
kml = simplekml.Kml()

import csv

file = sys.argv[1]

file_cnt = 0

allFiles = glob(sys.argv[1]+"/*.csv")
for file in allFiles:
    file_cnt = file_cnt+1

    print (file_cnt)
    print (file)

    if ' B ' in file:
        continue

    if 'inside' in file:
        continue

    file_fmt = file.replace('(Outside)','')
    file_fmt = file_fmt.replace('(exterior)','')
    file_fmt = file_fmt.replace('(Evelyn)','')
    file_fmt = file_fmt.replace('(1 of 5)','')
    file_fmt = file_fmt.replace('(2 of 5)','')
    file_fmt = file_fmt.replace('(3 of 5)','')
    file_fmt = file_fmt.replace('(4 of 5)','')
    file_fmt = file_fmt.replace('(5 of 5)','')
    file_fmt = file_fmt.replace('(outdoor)','')
    file_fmt = file_fmt.replace('(ish)','')

    latlon = file_fmt.split('(')
    print (latlon[2])

    latlon = latlon[2].split(')')
    #print (latlon)
    (lat,lon) = latlon[0].split(' ')

    with open(file) as csv_file:
        csv_reader = csv.reader(csv_file, delimiter=',')
        line_count = 0
        
        for row in csv_reader:
            if line_count == 0:
                #print(f'Column names are {", ".join(row)}')
                line_count += 1
            else:
                #print(f'\t{row[0]} val {row[8]}')
                line_count += 1

                aq = row[8]

                if aq == '':
                    continue

                pnt = kml.newpoint()

                pnt.coords = [(lon, lat)]
                
                this_time = row[0].replace(' UTC','')
                this_time = this_time.replace(' ','T')
                pnt.timestamp.when = this_time

                pnt.name = aq

                #pnt.style.iconstyle.icon.href = 'http://maps.google.com/mapfiles/kml/shapes/road_shield3.png'

                if float(aq) <= 30:
                    pnt.style.iconstyle.color = 'ff00ff00'  # Green
                if float(aq) > 30:
                    pnt.style.iconstyle.color = 'ff9999ff'  # Red                 
                if float(aq) > 80:
                    pnt.style.iconstyle.color = 'ff4d4dff'  # Red
                if float(aq) > 180:
                    pnt.style.iconstyle.color = 'ff0000ff'  # Red

kml.save("purple_air_sf_2020.kml")
1 Like