1. Echogram and EchoStats: Plotting echogram and statistics from echo data generated by a moored echosounder#
1.1. Introduction#
1.1.1. Goals#
Illustrate how to connect work between
echopype
andechoshader
.Illustrate examples for plotting echogram using
HoloViz
ecosystem.Illustrate a common workflow for plotting echogram using
echoshader
.Illustrate a common workflow for plotting statistics from echogram interactively using
echoshader
1.1.2. Description#
This notebook uses EK60 echosounder data from the U.S. Ocean Observatories Initiative (OOI). The raw data is converted, combined, calibrated using echopype
. Calibrated MVBS data from echopype procession is stored in form of .nc
files locally or cloudly. After loading Calibrated data, echoshader can load these files to plot echogram and statistics. This lazy load mechanism can avoid memory overflow resulted by processing all raw data at the same time.There are some examples in this jupyterbook to teach users how to get a echogram with panel control widgets. Echoshader leverages two of APIs: Reactive functions
and Parameterized class
. Users can also add their own widgets.
Data are from the OOI Oregon Offshore Cabled Shallow Profiler Mooring collected on August 1-31, 2017.This was the day before and of a solar eclipse, during which the reduced sunlight affected the regular diel vertical migration (DVM) patterns of marine life. This change was directly observed using the upward-looking echosounder mounted on this mooring platform that happened to be within the totality zone.
1.1.3. Outline#
Establish file system connection and Process raw files with
echopype
Load converted MVBS files from
Google Drive
, and plot echogram simply withHoloViz
ecosystem.Plot echogram and control widgets with
echoshader
.Plot statistics from echogram interactively with
echoshader
1.1.4. Warning#
When Image
dimension ping_time is not evenly sampled to relative tolerance of 0.001. There will be an error.
Use echogram.erase_error()
to get rid of them.
1.1.5. Note#
We encourage importing echopype
as ep
for consistency.
from pathlib import Path
import itertools as it
import datetime as dt
from dateutil import parser as dtparser
import fsspec
import gc
from urllib import request
import xarray as xr
import echopype as ep
from echopype import qc
import panel
import pandas
import param
import hvplot.xarray # noqa
import holoviews as hv
from echoshader.echogram import Echogram
from echoshader.echostats import EchoStats, simple_hist
hv.extension("bokeh")
1.2. Establish file system connection and Process raw files with echopype
#
base_dpath = Path('./exports')
base_dpath.mkdir(exist_ok=True)
calibrated_dpath = (base_dpath / 'OOI_30_Days')
calibrated_dpath.mkdir(exist_ok=True)
fs = fsspec.filesystem('https')
ooi_raw_url = (
"https://rawdata.oceanobservatories.org/files/"
"CE04OSPS/PC01B/ZPLSCB102_10.33.10.143/2017/08"
)
start_datetime = dt.datetime(2017, 8, 1, 0, 0)
end_datetime = dt.datetime(2017, 8, 31, 23, 59)
desired_day_urls = [f"{ooi_raw_url}/{str(day).zfill(2)}" for day in range(start_datetime.day, end_datetime.day + 1)]
all_raw_file_urls = it.chain.from_iterable([fs.glob(f"{day_url}/*.raw") for day_url in desired_day_urls])
def in_range(raw_file: str, start: dt.datetime, end: dt.datetime) -> bool:
"""Check if file url is in datetime range"""
file_name = Path(raw_file).name
file_datetime = dtparser.parse(file_name, fuzzy=True)
return file_datetime >= start and file_datetime <= end
desired_raw_file_urls = list(filter(
lambda raw_file: in_range(
raw_file,
start_datetime,
end_datetime
),
all_raw_file_urls
))
print(f"There are {len(desired_raw_file_urls)} raw files within the specified datetime range.")
for desired_raw_file_url in desired_raw_file_urls:
raw_fpath = Path(desired_raw_file_url)
try_times = 0
while True:
try_times += 1
# if try_times > 2:
# print(f"Failed to process raw file {raw_fpath.name}: {e}.")
# break
try:
# Access file directly from S3 to create a converted EchoData object in memory
ed = ep.open_raw(f"{desired_raw_file_url}", sonar_model='ek60')
# Use the EchoData object "ed" to generate calibrate Sv
ds_Sv = ep.calibrate.compute_Sv(ed)
# Check and correct reversed ping_time to allow compute_MVBS,
# which does not allow non-monotonic ping_time
qc.coerce_increasing_time(ds_Sv)
# Computed MVBS and save to netcdf
ds_MVBS = ep.preprocess.compute_MVBS(
ds_Sv,
range_meter_bin=1, # in meters
ping_time_bin='1800s' # in seconds
)
ds_MVBS.to_netcdf(calibrated_dpath / f"MVBS_{raw_fpath.stem}.nc")
# release to avoid memory overflow
del ed
del ds_Sv
del ds_MVBS
gc.collect()
except Exception as e:
print(f"Failed to process raw file {raw_fpath.name}: {e}. Execute again")
if try_times > 2:
continue
else:
break
break
1.3 Load converted MVBS files from Google Drive, and plot echogram simply with HoloViz ecosystem#
# Calibrated data is stored in Google Drive
url = 'https://drive.google.com/uc?export=download&id=1zHSleP3Ox_NjKrZfZ_sVvkc5g9r4K7tc'
def urllib_download():
request.urlretrieve(url, 'test_Echogram_EchoStats.nc')
urllib_download()
MVBS_ds = xr.open_mfdataset(
paths = 'test_Echogram_EchoStats.nc',
data_vars = 'minimal', coords='minimal',
combine = 'by_coords'
)
To visualize, invert the echo_range
axis since the echosounder is upward-looking from a platform at approximately 200 m water depth.
MVBS_ds["echo_range"] = MVBS_ds["echo_range"].values[::-1]
HvPlot provides one API to explore data of many different types including pandas
and xarray
.
We can use hvplot.image to plot xarray type data (gridded data) directly.
MVBS_ds["Sv"].hvplot.image(
x = 'ping_time',
y = 'echo_range',
color = 'Sv',
rasterize = True,
cmap = 'jet',
clim=(-90, -30),
xlabel = 'Time (UTC)',
ylabel = 'Depth (m)',
clabel = 'Sv(dB)'
).options(width = 800, invert_yaxis = True)
WARNING:param.Image00855: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image00855: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
A simple hist is provided in echostats for holoview-made echogram.
But user should turn off rasterize
mode.
gram = MVBS_ds["Sv"].sel(channel = 'GPT 38 kHz 00907208dd13 5-1 OOI.38|200').hvplot.image(
x = 'ping_time',
y = 'echo_range',
color = 'Sv',
cmap = 'jet',
clim=(-90, -30),
xlabel = 'Time (UTC)',
ylabel = 'Depth (m)',
clabel = 'Sv(dB)'
).options(width = 800, invert_yaxis = True)
simple_hist(gram)
WARNING:param.Image01160: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image01160: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image01160: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image01160: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
The image can automatically apply a groupby
along a particular dimension(‘channel’ in this example), allowing us to explore the data as images along that dimension with a slider.
We also can use panel
lib to create widgets to control the echogram.
We can supply both regular values, widgets and parameters as arguments to methods on the .interactive
accessor in hvplot. Here, we can use widgets from the Panel library. The repr of the resulting object will contain a layout of the widget and a view of the resulting output.
MVBS_ds.Sv.interactive.sel(channel = panel.widgets.Select).hvplot(
kind = 'image',
x = 'ping_time',
y = 'echo_range',
color = 'Sv',
cmap = 'jet',
clim=(-90, -30),
xlabel = 'Time (UTC)',
ylabel = 'Depth (m)',
clabel = 'Sv(dB)'
).options(width=800, invert_yaxis=True)
WARNING:param.Image01450: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image01450: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image01525: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image01525: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image01525: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image01613: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image01613: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image01613: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
Also, we can use four APIs
provided from panel
to link widgets and echogram.Each of these APIs has its own benefits and drawbacks. I recommend using Reactive functions
and Parameterized class
to accomplish our goals. See more details about them in APIs.
Here, we use panel.bind
to declare that function’s input values should come from the widgets’ value parameters..It is a form of Reactive functions
and requires the programmer to select and configure widgets explicitly and to lay out components explicitly, without relying on inference of widget types and ranges and without any default layouts.Once widgets have been bound to a reactive function, you can lay out the bound function and the widgets in any order or combination you like, including across Jupyter notebook cells if desired.
If users are writing code specifically for building an app, and do not wish to keep domain and GUI code separate, the functionality of pn.bind is also available as a decorator @pn.depends
.
channel_select = panel.widgets.Select(
name = 'Channel Select',
options = MVBS_ds.channel.values.tolist())
lower_time = pandas.to_datetime(MVBS_ds.ping_time.data[0])
upper_time = pandas.to_datetime(MVBS_ds.ping_time.data[-1])
datetime_range_input = panel.widgets.DatetimeRangeInput(
name = 'Datetime Range Input',
start = lower_time,
end = upper_time,
value = (lower_time, upper_time))
def create_echogram(channel_select, datetime_range_input):
lower_time = datetime_range_input[0]
upper_time = datetime_range_input[-1]
image = MVBS_ds.Sv.sel(
channel = channel_select,
ping_time=slice(lower_time,upper_time)).hvplot(
kind = 'image',
x='ping_time',
y='echo_range',
color='Sv',
cmap='jet',
clim=(-90, -30),
xlabel='Time (UTC)',
ylabel='Depth (m)'
).options(invert_yaxis=True)
return image
bind_gram = panel.bind(create_echogram, channel_select = channel_select, datetime_range_input = datetime_range_input)
panel.Column(channel_select,datetime_range_input, bind_gram)
WARNING:param.Image01717: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image01717: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image01717: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
1.4 Plot echogram and control widgets with echoshader
#
In echoshader
, we combine Parameterized Classes
and Reactive Functions
APIs.
The Param library allows expressing the parameters of a class (or a hierarchy of classes) completely independently of a GUI implementation. Panel and other libraries can then take those parameter declarations and turn them into a GUI to control the parameters. This approach allows the parameters controlling some computation to be captured specifically and explicitly (but as abstract parameters, not as widgets).
Although it is a aim for Parameterized Classes
that declaring way of expressing parameters and dependencies between parameters and computation and the resulting code is not tied to any particular GUI framework and can be used in other contexts as well. In echoshader
, we don’t use param
to get a widget, but still use panel.widgets
. This allows us to get more powerful and flexible widgets with Parameterized Classes
class structure. Values of panel.widgets rather than param
are as input for @param.depends
decorator.
echogram = Echogram(MVBS_ds)
Below shows how to get control widgets and echoram.
Notice: No add parentheses after ‘view_gram’ to accomplish links, although it is a function
panel.Row(echogram.widgets , echogram.view_gram)
WARNING:param.Image01844: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image01844: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
Below shows how to get MVBS data in select box.
echogram.get_box_data()
<xarray.Dataset> Dimensions: (echo_range: 205, ping_time: 1642) Coordinates: * echo_range (echo_range) float64 204.0 203.0 202.0 ... 2.0 1.0 0.0 * ping_time (ping_time) datetime64[ns] 2017-08-01 ... 2017-08-31T2... channel <U39 'GPT 38 kHz 00907208dd13 5-1 OOI.38|200' Data variables: Sv (ping_time, echo_range) float64 dask.array<chunksize=(1642, 205), meta=np.ndarray> frequency_nominal float64 dask.array<chunksize=(), meta=np.ndarray> Attributes: processing_software_name: echopype processing_software_version: 0.6.0 processing_time: 2022-08-19T22:36:57Z processing_function: preprocess.compute_MVBS
Below shows how to get echograms with all frequencies at the same time.
panel.Row(echogram.widgets , echogram.view_all_gram)
WARNING:param.Image02349: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image02349: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image02413: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image02413: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image02479: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image02479: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
Below shows how to get MVBS data with all frequencies in select box.
echogram.get_all_box_data()
<xarray.Dataset> Dimensions: (echo_range: 205, ping_time: 1642, channel: 3) Coordinates: * echo_range (echo_range) float64 204.0 203.0 202.0 ... 2.0 1.0 0.0 * ping_time (ping_time) datetime64[ns] 2017-08-01 ... 2017-08-31T2... * channel (channel) object 'GPT 38 kHz 00907208dd13 5-1 OOI.38|... Data variables: Sv (channel, ping_time, echo_range) float64 dask.array<chunksize=(3, 1642, 205), meta=np.ndarray> frequency_nominal (channel) float64 dask.array<chunksize=(3,), meta=np.ndarray> Attributes: processing_software_name: echopype processing_software_version: 0.6.0 processing_time: 2022-08-19T22:36:57Z processing_function: preprocess.compute_MVBS
To do more operations about layouts and customization,Use echogram.Echogram.__doc__
to look for relevant attributes
1.5 Plot statistics from echogram interactively with echoshader#
Users can just process data from select box.
Also, plotting statistics directly is provided in echoshader
.
There using EchoStats
, wchich is a subclass of Echogram
.
echostats = EchoStats(MVBS_ds)
Below shows how to show a table describing stats info about selsect box Sv data with a specific frequency.
panel.Row(echostats.widgets , panel.Column(echostats.view_gram, echostats.view_table))
WARNING:param.Image04012: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image04012: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
Below shows how to show a histogram describing stats info about selsect box Sv data with a specific frequency.
panel.Row(echostats.widgets , panel.Column(echostats.view_gram, echostats.view_hist))
WARNING:param.Image04556: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image04556: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
Below shows how to show a table describing stats info about selsect box Sv data with all frequencies.
panel.Row(echostats.widgets , panel.Column(echostats.view_all_gram, echostats.view_all_table))
WARNING:param.Image05190: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image05190: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image05254: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image05254: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image05320: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image05320: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
Below shows how to show a overlay hist stats info about selsect box Sv data with all frequencies.
A Overlay is A collection of HoloViews objects to be displayed overlaid on one another with the same axes.
panel.Row(echostats.widgets , panel.Column(echostats.view_all_gram, echostats.view_overlay_hist))
WARNING:param.Image06857: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image06857: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image06921: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image06921: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image06987: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image06987: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
Below shows how to show a layout hist stats info about selsect box Sv data with all frequencies.
A Layout is a collection of any HoloViews objects to be displayed side by side.
panel.Row(echostats.widgets , panel.Column(echostats.view_all_gram, echostats.view_layout_hist))
WARNING:param.Image08706: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image08706: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image08770: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image08770: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image08836: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.
WARNING:param.Image08836: Image dimension ping_time is not evenly sampled to relative tolerance of 0.001. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the Image constructor.