Data can be downloaded using a list of links or in bulk
Bulk Download
Tools
software referenced is for example only, not as an endorsement or recommendation
- AWS CLI (the command line interface to access AWS S3 buckets
- wget
- PowerShell
- Python3
A geopackage datadase that is an index to the entire CAP uploaded collection is generated every night into this location
http://s3.amazonaws.com/fema-cap-imagery/Support/cap_index.gpkg
AWS CLI
Once installed, the AWS CLI can be used like this to BULK download a snapshot of the an entire CAP Event or just one Team within the Event
For an entire Event
aws s3 sync s3://fema-cap-imagery/Images/<event id>>/ ./
or for an entire work of a Mission (sortie) within an Event
aws s3 sync s3://fema-cap-imagery/Images/<event id>// ./
In the above examples:
replace <event id> with the Numeric Event Id
replace <mission id> with the Numeric Mission Id
Event Id 2024 - LA - March Flooding 16 2016-03-01
Mission Id 100538 - A0013
replacing the text the command to type in becomes
aws s3 sync s3://fema-cap-imagery/Images/2024/100538/ ./
Searching the Image Index
The Image Index is a geopackage file. It can be ready by R, Python3, ArcGIS Pro, QGIS, the sqlite3 command and other tools
The core tables are
events - a list of Event Ids, Event Names, and Event Dates
missions - a list of Event Mission Ids, and Mission Dates
images - metadata on each photo available in the collection
The images table can be used to extract the individual links to the photos
SELECT imageURL FROM images WHERE ImageEventId = 2024 AND ImageMissionId = 100538;
How to use the links file
The image links button creates a CSV file of the links in the content display.
The file can be used by a number of tools to bulk download the images.
- WGet
- PowerShell
- Fetch
- Firefox - Extension (Simple Mass Downloader)
- Firefox - Extension (DownloadThemAll)
WGet
wget -i <urls.csv>
PowerShell
ps < "gc urls.csv | % { iwr $_ -outf $(split-path $_ -leaf)} "
Using BitsTransfer
Fetch
h2>
fetch