- Reflector -- what does it do?
This program is meant to mirror satellite data from various government
sources. A local copy is maintained, generally an image file or short
movie.
For each configured source, the reflector is able to predict and
mirror upstream data. By default, the system seems to ensure the more
recent loop period -- where the loop period is configured by the
source -- has populated data.
Currently, the reflector supports FTP and HTTP protocols.
- Future expansion
The following tools are planned:
A period script, possibly run by timer, which captures upstream data
as needed. Optional argument specifies how far back to go, default is
24 hours.
Various media producers which generate visualizations based on
captured data.
- Application Design
Config has SourceConfigs
a SourceConfig can be used to construct a Mirror -- if all looks good!
Capture represents a single mirrored entity.
Mirror methods:
GetLatest -> Result
grab the latest, if necessary, or else return from local cache
EnsureSince(Duration) -> Result
ensure our local mirror is up to date, going back from now - Duration
TimeRange is (StartSystemTime, EndSystemtime)
Mirrors know how to convert from time ranges into file names on the
remote site we're mirroring. For that we need:
- a period, which defines the time between upstream files
- a PathMaker, which can go from time to filename and back
- a seed time -- often midnight
We need a number of supporting fuctions. Start by considering EnsureLatest,
consider a naive implementation that going from more recent to least:
- p is the period of the Mirror, e.g., 24 hours for a daily mirror
- t starts as now
- loop:
- consider TimeRange (t-p, t)
- if we have a Capture within that period, collect it
- take away p from t
- keep looping while the span from t to now is less than Duration
Capture constructors, these are probably methods on the Mirror:
CapturesInRange(TimeRange) -> Result
CapturesSince(Duration) -> Result
- implied now
LatestCapture() -> Result
Maybe this is blue sky but it would also be nice to have a way for the
mirror to qualify the regularity of the source:
Deviation(Since)
Retrieval has its own sets of issues. I wonder.... Since we need to
support ftp and http both, I wonder if we can boil out a simple common
API, define that as a trait, and put up some wrapper classes for them.
The trait might be defined by:
- SizeOf(url) -> Result
- List(TimeRange) -> returns list of URLs and timestamps within range
- Get(url) -> Result
or
Grab(url, writer) -> Result
We also need a URL factory that the mirrors can use to find what to
fetch for a given TimeRange. In fact, we need a bunch of these
factories. Use a trait.
- First pass
We want to capture
https://sdo.gsfc.nasa.gov/assets/img/dailymov/2023/02/20/20230220_1024_0094.ogv
Once per day.
A possible TOML file:
#+begin_example
[[source]]
name = "Solar Data Observatory"
remote = "https://sdo.gsfc.nasa.gov/assets/img/dailymov"
local = "/home/adam/tmp/sat/sdo"
pathmaker = "SDO"
freq = "Daily"
flatten = true
#+end_example
- Oscillation of the mirror
Esssentially each SourceConfig describes an oscillation, described by:
period
loop_period
offset
So, for each day, our first element is:
midnight + offset
and then continues, once per period, until the day ends.
- URLs of note
ftp://ftp.nnvl.noaa.gov/GOES/ABI_GeoColor/
-- 5424x5424, all year, reliability issues
- Other interesting sources
https://cdn.star.nesdis.noaa.gov/GOES16/ABI/FD/GEOCOLOR/
-- 10848x10848 and 5424x5424 and other resolutions, 72 hours only
https://www.star.nesdis.noaa.gov/GOES/fulldisk.php?sat=G16
-- overview of what's available
** SSEC
https://realearth.ssec.wisc.edu/
- tasks / issues / problems
See BUGS.org