Computation & data access

Data required to compute diagnostics are not read and diagnostics themselves not computed until one of the computation methods is called.

All the methods presented below are available to both NadirData and SwathData data containers.

Sequential computation

The compute() method allows to read data and compute diagnostics one after each other.

Read data and compute statistics. Limited to the provided list of
data if specified.

Parameters
----------
stats
    Name of the data to limit the computation to.

Diagnostics can be computed individually

ad.compute(stats=["Sigma 0"])

Or all at once

ad.compute()

Parallel computation

Parallel computation allows to speed-up computation time and solve memory issues when working with large amount of data.

This kind of computation is done using the compute_dask() and a Dask cluster’s client.

Read data and compute statistics. Limited to the provided list of
data if specified.

Parameters
----------
stats
    Name of the data to limit the computation to.
freq
    Minimal split frequency (day, pass, cycle or any pandas offset aliases [1]_)
    to respect.
jobs_number
    Number of jobs to create (Default to the maximum possible number of periods
    the data can be split into according to the provided frequency).
bar
    [Does not work on xarray datasets] Whether to display a progress bar or not.
    If None, will display if logging level <= INFO.
dask_client
    Dask client on which to submit jobs.

        - if a client, use this client
        - if a scheduler_file, connect a client to it
        - try to get an existing dask client in the environment
        - create a local cluster and connect to it
kwargs
    Additional parameters determine the time splitting characteristics.
    This parameter works with pandas.date_range [2]_ frequencies.

References
----------
.. [1] https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html        #timeseries-offset-aliases
.. [2] https://pandas.pydata.org/pandas-docs/stable/reference/api/        pandas.date_range.html

In order to parallelize, the data is split along the main dimension (usually time).

freq parameter

• CLS Table

  • This parameter is required

• xarray Datasets

  • This parameter is optional and the parallelization is done using the current chunking of the dataset (each chunk usually correspond to a single file)

  • Setting this parameter triggers a re-chunking of the data.

jobs_number parameter

• is optional and default to the maximum possible number of periods the data can be split into according to the given frequency (or the existing number of chunks for xarray Datasets).

• will split the data in a multiple of the provided minimal frequency if less than the possible maximum number of jobs.

These parameters and the cluster’s parameters depend on each other and must be set according to each other. There are no right or wrong parameters but some general guidelines can be applied to find suitable parameters

• Creating a cluster with more workers than jobs_number will waste resources

• Creating a cluster with more per core memory than the memory used by a single of your job will waste resources (and potentially delay the cluster creation)

• Increasing jobs_number will speed-up CPU bounds computations

• Increasing jobs_number will slow-down I/O bounds computations

• Increasing jobs_number implies more I/O so more stress on the storage

• Increasing jobs_number will increase parallelization overhead

• Decreasing jobs_number will increase the memory usage of each job

Limitations

Using this computation mode introduce some constraints and limitations you have to take into consideration.

• For now, some diagnostics are not computed when using this method:

  • Nadir crossovers statistics

  • Editings

  • Section analysis

• Median statistics cannot be computed for the following diagnostics:

  • Geographical box statistics

  • Scatters

  • Binned statistics

• Median can be computed for the along time diagnostics if the compute_dask() freq parameter is a multiple of the add_time_stat() freq parameter. Some examples of compute_dask() and add_time_stat() frequency multiples:

  • 1 day is a multiple of 1 day

  • 1 day is a multiple of 1 hour

  • 1 day is a multiple of 6 hours

  • 1 day is not a multiple of 7 hours

  • 1 cycle is a multiple of 1 pass

  • 1 pass is not a multiple of 3 cycles

Dask clusters

Warning

CLS Tables database are not thread safe.

Multi-threaded workers should be avoided when working with this kind of data source.

A single threaded cluster is a cluster having the same number of cores and processes.

Dask clusters can be directly created directly using the dask’s distributed or Dask-Jobqueue libraries or through the DaskCluster utility class which will automatically set some parameters specific to the CLS and CNES environments and offer an homogeneous interface between local and distributed clusters.

Used to create a dask cluster (mostly) independently of environment.

Parameters
----------
cluster_type
    Type of cluster wanted.
start_timeout
    Timeout (in seconds) used by the wait_for_workers method (default: 120).
    Setting it to None will wait indefinitely.
jobs
    Number of job to start (min, max).
walltime
    Maximum execution time for each worker (Unit is minutes, default to 30).
cores
    total number of cores per job
processes
    Cut the job into this many processes. Good for GIL workloads or
    for nodes with many cores.
memory
    total amount of memory per job. Unit is fractional GiB.
name
    name of dask workers
queue
    queue name
project
    project name
resource_spec
    resources needed, format may depend on batch system
silence_logs:
    Level of logs to print out to stdout. logging.CRITICAL by default.
    Use a falsey value like False or None for no change.
interface
    network interface for worker communications (like 'eth0' ou 'ib0')
death_timeout
    Seconds to wait for a scheduler before closing workers
job_extra_directives
    specification of batch system options. Depends on batch system.

    If it is a list, it is is used "as is" and passed to jobqueue.
    If it is a dictionary, the keys are the batch system name and
    the values, the list of options to pass.
worker_extra_args
    additional arguments to pass to dask-worker
job_script_prologue
    other commands to add to script before launching worker
env_vars
    Environment variables to pass to workers. If this is a list,
    this names of environment variables to take from os.environ
    (current process environment variables).

    If this is a dictionary, the key is the variable name, the
    value is the value to pass. If the value is None, the actual
    value is taken from os.environ.

    .. note::
       `env_vars` is used to simplify passing environment variables,
       in fact it enriches the `job_script_prologue` parameter.
env_vars_treatment
    What to do with environment variables given with `env_vars`
local_directory
    dask worker local directory for file spilling (used by LOCAL clusters)
log_directory
    directory to use for job scheduler logs
python
    python executable used to launch Dask workers.
shebang
    path to desired interpreter for your batch submission script.
jupyterhub
    Set this cluster to be usable in a jupyterhub context.
expand_environment_variables
    List of parameters for which to expand environment variables.
    If None, ['log_directory', 'worker_extra_args', 'job_script_prologue'] is taken.
    If Empty no variable is expanded.
kwargs
    Additional cluster parameters.

Dask is a powerful tool allowing you to book and access large amount of CPU and memory resources. These resources are shared among users and need to be freed when you’re done using them.

Freeing cluster’s resources is done by calling the .close() method.

Local cluster

Local clusters allow to use all or a subpart of the CPUs of the computer on which your program is running.

In the following example we create and use a local cluster with 4 single threaded workers each using 3 GB of memory.

Example

DaskClusterdask’s distributed

from octantng.core.dask import DaskCluster

dask_cluster = DaskCluster(
    cluster_type="local",
    jobs=4,
    memory=3,
    local_directory="./dask_stuff",
)
client = dask_cluster.client

ad.compute_dask(dask_client=client, jobs_number=4, freq="pass")

dask_cluster.close()

Slurm cluster

Note

If used at CLS, the account=XYZ parameter is optional.

If used at CNES, set the interface="ib0" parameter to improve network performances.

If used through CNES jupyterhub, you can provide the jupyterhub=True parameter in order to make the dashboard accessible.

Example 1

In the following example we create and use an adaptive Slurm cluster scaling between 5 and 10 single threaded workers each using 3 GB of memory and having a 10 minutes lifetime.

Using CLS resources

DaskClusterdask_jobqueue

from octantng.core.dask import DaskCluster

dask_cluster = DaskCluster(
    cluster_type="slurm",
    jobs=(5, 10),
    memory=3,
    walltime=10,
    log_directory="./dask_stuff",
)
client = dask_cluster.client

ad.compute_dask(dask_client=client, jobs_number=100, freq="pass")

# Remember to close your cluster and free its resources!
dask_cluster.close()

Using CNES jupyterhub resources

DaskClusterdask_jobqueue

from octantng.core.dask import DaskCluster

dask_cluster = DaskCluster(
    cluster_type="slurm",
    account="XYZ",
    jobs=(5, 10),
    memory=3,
    walltime=10,
    interface="ib0",
    log_directory="./dask_stuff",
    # Do not set this if outside jupyterhub!
    jupyterhub=True,
)
client = dask_cluster.client

ad.compute_dask(dask_client=client, jobs_number=100, freq="pass")

# Remember to close your cluster and free its resources!
dask_cluster.close()

Example 2

In the following example we create and use a Slurm cluster with 12 workers each using 3 GB of memory (grouped in jobs of 4 cores and 12 GB of memory).

These workers are not single threaded.

Using CLS resources

DaskClusterdask_jobqueue

from octantng.core.dask import DaskCluster

dask_cluster = DaskCluster(
    cluster_type="slurm",
    jobs=(3, 3),
    cores=4,
    processes=2,
    memory=12,
    walltime=10,
    log_directory="./dask_stuff",
)
client = dask_cluster.client

ad.compute_dask(dask_client=client, jobs_number=100, freq="pass")

# Remember to close your cluster and free its resources!
dask_cluster.close()

Using CNES jupyterhub resources

DaskClusterdask_jobqueue

from octantng.core.dask import DaskCluster

dask_cluster = DaskCluster(
    cluster_type="slurm",
    account="XYZ",
    jobs=(3, 3),
    cores=4,
    processes=2,
    memory=12,
    walltime=10,
    interface="ib0",
    log_directory="./dask_stuff",
    # Do not set this if outside jupyterhub!
    jupyterhub=True,
)
client = dask_cluster.client

ad.compute_dask(dask_client=client, jobs_number=100, freq="pass")

# Remember to close your cluster and free its resources!
dask_cluster.close()

Computed data access

Existing diagnostics can be listed using the list_diagnostics() method.

List the subset of diagnostics' respecting provided criteria.

Parameters
----------
dtype
    Limit diagnostics list to provided type.
    Data types are:

     - RAW
     - RAW_COMPARISON
     - EDITING
     - GEOBOX
     - TEMPORAL
     - BINNED
     - BINNED_2D
     - HISTOGRAM
     - SCATTER
     - RATIO
     - CROSSOVER
     - MISSING_POINTS
     - SECTION_ANALYSES
containing
    Limit to diagnostic names containing this element.
computed
    Limited diagnostics list based on their computation status:

     - [Default] None: All diagnostics
     - False: Non-computed diagnostics
     - True: Computed diagnostics
freq
    Limit to diagnostics compatible with a dask computation at the provided
    frequency.

Returns
-------
:
    List of diagnostic names.

ad.list_diagnostics(dtype="temporal", containing="sla")

['SLA pass', 'SLA (6h)']

Diagnostics are internally stored in class inheriting from Diagnostic containing its specification and results.

This objects can be accessed using the get_diagnostic() and get_diagnostics() methods.

Get the requested diagnostic data container.

If name is not an existing diagnostic, a new raw or raw_comparison diagnostic
will be generated with the provided parameters.

Parameters
----------
name
    Name of the diagnostic.
field
    Field for which to create a raw_data diagnostic.
x
    Field to use as x-axis (raw comparison diagnostic).
y
    Field to use as y-axis (raw comparison diagnostic).
z
    Field to use as z-axis (raw comparison 3d diagnostic).
kwargs
    Additional read_data parameters.

Returns
-------
:
    Diagnostic data container.

Computed data are contained in xarray datasets.

Data are internally stored as raw data or groups of compatible diagnostics.

For example, diagnostics of 2 time statistics diagnostics using the same frequency will be grouped, computed together and can be accessed in the same dataset.

Once computed, diagnostics data can be accessed in different manner depending on their nature:

• using the data attribute for raw data

• using the get_data() method for individual diagnostics data

• using the get_data_group() method for a whole group of diagnostics.

Raw data access.

ad.data

<xarray.Dataset> Size: 33MB
Dimensions:         (time: 582789)
Coordinates:
  * time            (time) datetime64[ns] 5MB 2019-06-01T05:30:29.807497 ... ...
Data variables:
    LONGITUDE       (time) float64 5MB 17.01 17.14 17.27 ... -123.5 -123.3
    LATITUDE        (time) float64 5MB -66.15 -66.15 -66.15 ... 66.15 66.15
    wind            (time) float64 5MB nan nan nan nan nan ... nan nan nan nan
    SLA             (time) float64 5MB nan nan nan nan nan ... nan nan nan nan
    SIGMA0.ALTI     (time) float64 5MB nan nan nan nan nan ... nan nan nan nan
    EDITING.CUSTOM  (time) int64 5MB 1 1 1 1 1 1 2 1 1 1 ... 1 1 1 1 1 1 1 1 1 1

xarray.Dataset

• Dimensions:
  • time: 582789
• Coordinates: (1)
  • time
    (time)
    datetime64[ns]
    2019-06-01T05:30:29.807497 ... 2...

    standard_name :

    time

    array(['2019-06-01T05:30:29.807497000', '2019-06-01T05:30:30.826208000',
           '2019-06-01T05:30:31.844918000', ..., '2019-06-08T01:28:01.950456000',
           '2019-06-08T01:28:02.969165000', '2019-06-08T01:28:03.987875000'],
          dtype='datetime64[ns]')

• Data variables: (6)
  • LONGITUDE
    (time)
    float64
    17.01 17.14 17.27 ... -123.5 -123.3

    array([  17.006113,   17.135617,   17.265119, ..., -123.587257,
           -123.457709, -123.328159])

  • LATITUDE
    (time)
    float64
    -66.15 -66.15 ... 66.15 66.15

    array([-66.149034, -66.148977, -66.148805, ...,  66.149439,  66.14963 ,
            66.149706])

  • wind
    (time)
    float64
    nan nan nan nan ... nan nan nan nan

    array([nan, nan, nan, ..., nan, nan, nan])

  • SLA
    (time)
    float64
    nan nan nan nan ... nan nan nan nan

    array([nan, nan, nan, ..., nan, nan, nan])

  • SIGMA0.ALTI
    (time)
    float64
    nan nan nan nan ... nan nan nan nan

    array([nan, nan, nan, ..., nan, nan, nan])

  • EDITING.CUSTOM
    (time)
    int64
    1 1 1 1 1 1 2 1 ... 1 1 1 1 1 1 1 1

    array([1, 1, 1, ..., 1, 1, 1])

• Indexes: (1)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2019-06-01 05:30:29.807497', '2019-06-01 05:30:30.826208',
                   '2019-06-01 05:30:31.844918', '2019-06-01 05:30:32.863628',
                   '2019-06-01 05:30:33.882337', '2019-06-01 05:30:34.901048',
                   '2019-06-01 05:30:35.919757', '2019-06-01 05:30:36.938466',
                   '2019-06-01 05:30:37.957177', '2019-06-01 05:30:38.975886',
                   ...
                   '2019-06-08 01:27:54.819487', '2019-06-08 01:27:55.838196',
                   '2019-06-08 01:27:56.856907', '2019-06-08 01:27:57.875616',
                   '2019-06-08 01:27:58.894327', '2019-06-08 01:27:59.913036',
                   '2019-06-08 01:28:00.931746', '2019-06-08 01:28:01.950456',
                   '2019-06-08 01:28:02.969165', '2019-06-08 01:28:03.987875'],
                  dtype='datetime64[ns]', name='time', length=582789, freq=None))

• Attributes: (0)

Individual diagnostic access.

ad.get_data(name="SLA pass", stat="mean")

<xarray.Dataset> Size: 3kB
Dimensions:  (time: 175)
Coordinates:
  * time     (time) datetime64[ns] 1kB 2019-06-01T05:58:35.692381 ... 2019-06...
Data variables:
    SLA      (time) float64 1kB -2.283 -0.2079 -2.382 ... 1.91 -2.641 -2.712

xarray.Dataset

• Dimensions:
  • time: 175
• Coordinates: (1)
  • time
    (time)
    datetime64[ns]
    2019-06-01T05:58:35.692381 ... 2...

    long_name :

    Box center

    array(['2019-06-01T05:58:35.692381000', '2019-06-01T06:54:48.840566000',
           '2019-06-01T07:51:01.899977000', '2019-06-01T08:47:14.784509000',
           '2019-06-01T09:43:27.743745000', '2019-06-01T10:39:40.610329000',
           '2019-06-01T11:35:53.156019000', '2019-06-01T12:32:06.214703000',
           '2019-06-01T13:28:18.770337000', '2019-06-01T14:24:31.892810000',
           '2019-06-01T15:20:44.741688000', '2019-06-01T16:16:57.722028000',
           '2019-06-01T17:13:10.610198000', '2019-06-01T18:09:23.435305000',
           '2019-06-01T19:05:36.336331000', '2019-06-01T20:01:49.013241000',
           '2019-06-01T20:58:02.208237000', '2019-06-01T21:54:15.033586000',
           '2019-06-01T22:50:27.827404000', '2019-06-01T23:46:40.769905000',
           '2019-06-02T00:42:53.611749000', '2019-06-02T01:39:06.382526000',
           '2019-06-02T02:35:19.309748000', '2019-06-02T03:31:32.307793000',
           '2019-06-02T04:27:45.279884000', '2019-06-02T05:23:57.959704000',
           '2019-06-02T06:20:10.589317000', '2019-06-02T07:16:23.645090000',
           '2019-06-02T08:12:36.793758000', '2019-06-02T09:08:49.348665000',
           '2019-06-02T10:05:02.670515000', '2019-06-02T11:01:15.390840000',
           '2019-06-02T11:57:28.372635000', '2019-06-02T12:53:41.013405000',
           '2019-06-02T13:49:53.855491000', '2019-06-02T14:46:06.672350000',
           '2019-06-02T15:42:19.787062000', '2019-06-02T16:38:32.431471000',
           '2019-06-02T17:34:45.257063000', '2019-06-02T18:30:58.496446000',
           '2019-06-02T19:27:11.333197000', '2019-06-02T20:23:23.986336000',
           '2019-06-02T21:19:37.181816000', '2019-06-02T22:15:50.252628000',
           '2019-06-02T23:12:03.043293000', '2019-06-03T00:08:15.454609000',
           '2019-06-03T01:04:28.322161000', '2019-06-03T02:00:41.515461000',
           '2019-06-03T02:56:54.074976000', '2019-06-03T03:53:07.243047000',
           '2019-06-03T04:49:20.226054000', '2019-06-03T05:45:32.827045000',
           '2019-06-03T06:41:45.999242000', '2019-06-03T07:37:59.099643000',
           '2019-06-03T08:34:11.485977000', '2019-06-03T09:30:24.416837000',
           '2019-06-03T10:26:37.207503000', '2019-06-03T11:22:50.392795000',
           '2019-06-03T12:19:03.406849000', '2019-06-03T13:15:16.136634000',
           '2019-06-03T14:11:28.680141000', '2019-06-03T15:07:41.762594000',
           '2019-06-03T16:03:54.409427000', '2019-06-03T17:00:07.715267000',
           '2019-06-03T17:56:20.363557000', '2019-06-03T18:52:33.180903000',
           '2019-06-03T19:48:46.494505000', '2019-06-03T20:44:59.355509000',
           '2019-06-03T21:41:12.208025000', '2019-06-03T22:37:24.988018000',
           '2019-06-03T23:33:37.858239000', '2019-06-04T00:29:50.434732000',
           '2019-06-04T01:26:03.604260000', '2019-06-04T02:22:16.517898000',
           '2019-06-04T03:18:29.392972000', '2019-06-04T04:14:42.084191000',
           '2019-06-04T05:10:55.101639000', '2019-06-04T06:07:07.779760000',
           '2019-06-04T07:03:20.906844000', '2019-06-04T07:59:33.916532000',
           '2019-06-04T08:55:46.668387000', '2019-06-04T09:51:59.634173000',
           '2019-06-04T10:48:12.559696000', '2019-06-04T11:44:25.497347000',
           '2019-06-04T12:40:37.922732000', '2019-06-04T13:36:50.874936000',
           '2019-06-04T14:33:03.689371000', '2019-06-04T15:29:16.658794000',
           '2019-06-04T16:25:29.572675000', '2019-06-04T17:21:42.324532000',
           '2019-06-04T18:17:55.214401000', '2019-06-04T19:14:08.423222000',
           '2019-06-04T20:10:21.349958000', '2019-06-04T21:06:34.131405000',
           '2019-06-04T22:02:47.261639000', '2019-06-04T22:58:59.886398000',
           '2019-06-04T23:55:12.506308000', '2019-06-05T00:51:25.729198000',
           '2019-06-05T01:47:38.332130000', '2019-06-05T02:43:51.566661000',
           '2019-06-05T03:40:04.406806000', '2019-06-05T04:36:17.290126000',
           '2019-06-05T05:32:29.997112000', '2019-06-05T06:28:43.015529000',
           '2019-06-05T07:24:55.974767000', '2019-06-05T08:21:08.665745000',
           '2019-06-05T09:17:21.622070000', '2019-06-05T10:13:34.263081000',
           '2019-06-05T11:09:47.118748000', '2019-06-05T12:06:00.430169000',
           '2019-06-05T13:02:12.927590000', '2019-06-05T13:58:25.857479000',
           '2019-06-05T14:54:38.828116000', '2019-06-05T15:50:51.536800000',
           '2019-06-05T16:47:04.515684000', '2019-06-05T17:43:17.251774000',
           '2019-06-05T18:39:30.413299000', '2019-06-05T19:35:43.493326000',
           '2019-06-05T20:31:56.178966000', '2019-06-05T21:28:09.338063000',
           '2019-06-05T22:24:22.097197000', '2019-06-05T23:20:34.807094000',
           '2019-06-06T00:16:47.479394000', '2019-06-06T01:13:00.649892000',
           '2019-06-06T02:09:13.264222000', '2019-06-06T03:05:26.449758000',
           '2019-06-06T04:01:39.097560000', '2019-06-06T04:57:51.955170000',
           '2019-06-06T05:54:04.748260000', '2019-06-06T06:50:17.941558000',
           '2019-06-06T07:46:30.758175000', '2019-06-06T08:42:43.742393000',
           '2019-06-06T09:38:56.391895000', '2019-06-06T10:35:09.167037000',
           '2019-06-06T11:31:22.491312000', '2019-06-06T12:27:34.865034000',
           '2019-06-06T13:23:47.867444000', '2019-06-06T14:20:00.700313000',
           '2019-06-06T15:16:13.915198000', '2019-06-06T16:12:26.640131000',
           '2019-06-06T17:08:39.492644000', '2019-06-06T18:04:52.382513000',
           '2019-06-06T19:01:05.536033000', '2019-06-06T19:57:18.053101000',
           '2019-06-06T20:53:31.393868000', '2019-06-06T21:49:43.907056000',
           '2019-06-06T22:45:56.760298000', '2019-06-06T23:42:09.801274000',
           '2019-06-07T00:38:22.360547000', '2019-06-07T01:34:35.427961000',
           '2019-06-07T02:30:48.450503000', '2019-06-07T03:27:01.348618000',
           '2019-06-07T04:23:14.042504000', '2019-06-07T05:19:27.144846000',
           '2019-06-07T06:15:39.983293000', '2019-06-07T07:11:53.179743000',
           '2019-06-07T08:08:05.643451000', '2019-06-07T09:04:18.786055000',
           '2019-06-07T10:00:31.215563000', '2019-06-07T10:56:44.553178000',
           '2019-06-07T11:52:57.095229000', '2019-06-07T12:49:09.862124000',
           '2019-06-07T13:45:22.783279000', '2019-06-07T14:41:35.991373000',
           '2019-06-07T15:37:48.580721000', '2019-06-07T16:34:01.439786000',
           '2019-06-07T17:30:14.271441000', '2019-06-07T18:26:27.474440000',
           '2019-06-07T19:22:40.389048000', '2019-06-07T20:18:53.235742000',
           '2019-06-07T21:15:06.353364000', '2019-06-07T22:11:19.003590000',
           '2019-06-07T23:07:32.004303000', '2019-06-08T00:03:44.668356000',
           '2019-06-08T00:59:57.689929000'], dtype='datetime64[ns]')

• Data variables: (1)
  • SLA
    (time)
    float64
    -2.283 -0.2079 ... -2.641 -2.712

    array([-2.28265003, -0.20786509, -2.38207967, -2.4357797 ,  0.36038206,
           -2.43042596,  3.13104819, -2.47987539, -2.35612418, -2.46214445,
           -1.49596477, -2.47016595, -2.40872963, -2.68646305,  1.69717311,
           -0.92712757, -1.86601747, 11.53248762, -2.42629378,  3.0517613 ,
           -2.15873684, -2.29983024, -2.26444189, -2.40594512, -2.26874344,
           -2.25632891, -2.28625965, -1.70241223, -2.19127771, -1.62412854,
           -1.80477225, -2.40321494,  0.14149508, -2.47905742, -1.62539129,
           -1.4457048 , -1.93003269, -2.02842294,  2.21621684, -2.44807525,
            0.57223089, -1.66480165, -1.93214742, 13.22346901, -1.56323938,
            0.74400713, -2.38514998,  0.6733592 , -2.52003381, -2.40878648,
           -2.11569893, -2.09177238, -2.3802208 , -2.32684167, -2.28674604,
           -1.54644611,  0.7865388 , -2.42238235, -2.26884705, -2.43004058,
           -2.42703556, -1.017817  , -2.25802773, -2.36993501, -1.85945871,
           -2.12932351,  2.28219953,  0.61019099, -2.39318156, 15.29427523,
           -2.36586862, -1.48976121, -2.53792184, -2.49810958, -0.17081063,
           -2.4658483 , -1.4290227 , -1.27045528, -2.2242183 , -2.41798989,
            0.22618447, -2.40292642,  2.63386054, -2.50659474, -2.54519294,
           -2.31966599, -2.19002952, -2.47174382, -2.31145004, -2.46072778,
            2.66908924, -2.23733858, -1.80862302,  1.27893813, -2.36331261,
           -2.3739704 , -0.4361207 , -2.08626764, -2.40064252, -2.52369213,
           -1.70233223, -2.60275554, -2.68047876, -1.45585019, -2.42141436,
           -2.55740698, -1.45696537, -2.08267331, -0.64021039, -2.60531214,
           -1.42669645, -2.15776924, -2.46828729, -2.08303448, -2.44813111,
           -2.44877774,  2.48867935, -1.986869  , -2.62975832, 12.93051913,
           -2.40396722,  2.46770495, -2.22710797, -1.62898845, -2.6873578 ,
           -2.31915065, -1.92787013, -2.83645009, -2.68622283, -2.30588463,
           -2.47084067, -1.29865439, -1.55082829, -2.43126549, -2.39862907,
           -2.596386  , -2.50377108, -2.00614099, -2.49610628, -1.66828898,
           -2.01571561, -2.64342225,  8.40959582, -1.73074253, -2.57257363,
           13.12375499,  1.44513437, -2.31153341, -2.74582698, -2.48325975,
            1.12089127, -2.12980987, -2.07879826, -2.85259267, -2.74376773,
           -2.48193629, -2.384709  , -1.28591951,  0.0604219 , -2.43744726,
           -2.53157269, -2.50747428, -2.5556773 , -2.30839129, -2.53054813,
           -2.14549156, -2.49359734, -2.01550753,  4.33199303,  2.74495777,
           -2.54990982,  2.49772126,  1.90997036, -2.64124176, -2.71195672])

• Indexes: (1)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2019-06-01 05:58:35.692381', '2019-06-01 06:54:48.840566',
                   '2019-06-01 07:51:01.899977', '2019-06-01 08:47:14.784509',
                   '2019-06-01 09:43:27.743745', '2019-06-01 10:39:40.610329',
                   '2019-06-01 11:35:53.156019', '2019-06-01 12:32:06.214703',
                   '2019-06-01 13:28:18.770337', '2019-06-01 14:24:31.892810',
                   ...
                   '2019-06-07 16:34:01.439786', '2019-06-07 17:30:14.271441',
                   '2019-06-07 18:26:27.474440', '2019-06-07 19:22:40.389048',
                   '2019-06-07 20:18:53.235742', '2019-06-07 21:15:06.353364',
                   '2019-06-07 22:11:19.003590', '2019-06-07 23:07:32.004303',
                   '2019-06-08 00:03:44.668356', '2019-06-08 00:59:57.689929'],
                  dtype='datetime64[ns]', name='time', length=175, freq=None))

• Attributes: (0)

Some diagnostics (section analysis here) might have custom outputs.

ad.get_data(name="Analysis 1")

			First position			Last position
#	Cycle	Pass	Date	Longitude	Latitude	Date	Longitude	Latitude	Length	False %
1	122	22 - 23	2019-06-02T02:04:53	47.98	-65.09	2019-06-02T02:11:56	97.61	-62.04	416	1.92%
2	122	42 - 43	2019-06-02T20:47:21	112.56	-62.90	2019-06-02T20:54:08	161.09	-64.80	401	1.50%
3	122	98 - 99	2019-06-05T01:17:23	52.07	-65.25	2019-06-05T01:24:09	99.87	-62.20	400	0.00%
4	122	170 - 171	2019-06-07T20:41:41	91.96	-61.05	2019-06-07T20:50:53	155.09	-63.29	543	1.84%
5	122	174 - 175	2019-06-08T00:29:23	52.61	-64.96	2019-06-08T00:36:33	103.19	-62.07	424	0.00%

Group of diagnostics access.

ad.get_data_group(name="SLA pass", stat="mean")

<xarray.Dataset> Size: 4kB
Dimensions:      (x: 175)
Coordinates:
  * x            (x) datetime64[ns] 1kB 2019-06-01T05:58:35.692381 ... 2019-0...
Data variables:
    SIGMA0.ALTI  (x) float64 1kB 13.99 13.52 13.77 13.81 ... 13.93 13.69 14.82
    SLA          (x) float64 1kB -2.283 -0.2079 -2.382 ... 1.91 -2.641 -2.712

xarray.Dataset

• Dimensions:
  • x: 175
• Coordinates: (1)
  • x
    (x)
    datetime64[ns]
    2019-06-01T05:58:35.692381 ... 2...

    long_name :

    Box center

    array(['2019-06-01T05:58:35.692381000', '2019-06-01T06:54:48.840566000',
           '2019-06-01T07:51:01.899977000', '2019-06-01T08:47:14.784509000',
           '2019-06-01T09:43:27.743745000', '2019-06-01T10:39:40.610329000',
           '2019-06-01T11:35:53.156019000', '2019-06-01T12:32:06.214703000',
           '2019-06-01T13:28:18.770337000', '2019-06-01T14:24:31.892810000',
           '2019-06-01T15:20:44.741688000', '2019-06-01T16:16:57.722028000',
           '2019-06-01T17:13:10.610198000', '2019-06-01T18:09:23.435305000',
           '2019-06-01T19:05:36.336331000', '2019-06-01T20:01:49.013241000',
           '2019-06-01T20:58:02.208237000', '2019-06-01T21:54:15.033586000',
           '2019-06-01T22:50:27.827404000', '2019-06-01T23:46:40.769905000',
           '2019-06-02T00:42:53.611749000', '2019-06-02T01:39:06.382526000',
           '2019-06-02T02:35:19.309748000', '2019-06-02T03:31:32.307793000',
           '2019-06-02T04:27:45.279884000', '2019-06-02T05:23:57.959704000',
           '2019-06-02T06:20:10.589317000', '2019-06-02T07:16:23.645090000',
           '2019-06-02T08:12:36.793758000', '2019-06-02T09:08:49.348665000',
           '2019-06-02T10:05:02.670515000', '2019-06-02T11:01:15.390840000',
           '2019-06-02T11:57:28.372635000', '2019-06-02T12:53:41.013405000',
           '2019-06-02T13:49:53.855491000', '2019-06-02T14:46:06.672350000',
           '2019-06-02T15:42:19.787062000', '2019-06-02T16:38:32.431471000',
           '2019-06-02T17:34:45.257063000', '2019-06-02T18:30:58.496446000',
           '2019-06-02T19:27:11.333197000', '2019-06-02T20:23:23.986336000',
           '2019-06-02T21:19:37.181816000', '2019-06-02T22:15:50.252628000',
           '2019-06-02T23:12:03.043293000', '2019-06-03T00:08:15.454609000',
           '2019-06-03T01:04:28.322161000', '2019-06-03T02:00:41.515461000',
           '2019-06-03T02:56:54.074976000', '2019-06-03T03:53:07.243047000',
           '2019-06-03T04:49:20.226054000', '2019-06-03T05:45:32.827045000',
           '2019-06-03T06:41:45.999242000', '2019-06-03T07:37:59.099643000',
           '2019-06-03T08:34:11.485977000', '2019-06-03T09:30:24.416837000',
           '2019-06-03T10:26:37.207503000', '2019-06-03T11:22:50.392795000',
           '2019-06-03T12:19:03.406849000', '2019-06-03T13:15:16.136634000',
           '2019-06-03T14:11:28.680141000', '2019-06-03T15:07:41.762594000',
           '2019-06-03T16:03:54.409427000', '2019-06-03T17:00:07.715267000',
           '2019-06-03T17:56:20.363557000', '2019-06-03T18:52:33.180903000',
           '2019-06-03T19:48:46.494505000', '2019-06-03T20:44:59.355509000',
           '2019-06-03T21:41:12.208025000', '2019-06-03T22:37:24.988018000',
           '2019-06-03T23:33:37.858239000', '2019-06-04T00:29:50.434732000',
           '2019-06-04T01:26:03.604260000', '2019-06-04T02:22:16.517898000',
           '2019-06-04T03:18:29.392972000', '2019-06-04T04:14:42.084191000',
           '2019-06-04T05:10:55.101639000', '2019-06-04T06:07:07.779760000',
           '2019-06-04T07:03:20.906844000', '2019-06-04T07:59:33.916532000',
           '2019-06-04T08:55:46.668387000', '2019-06-04T09:51:59.634173000',
           '2019-06-04T10:48:12.559696000', '2019-06-04T11:44:25.497347000',
           '2019-06-04T12:40:37.922732000', '2019-06-04T13:36:50.874936000',
           '2019-06-04T14:33:03.689371000', '2019-06-04T15:29:16.658794000',
           '2019-06-04T16:25:29.572675000', '2019-06-04T17:21:42.324532000',
           '2019-06-04T18:17:55.214401000', '2019-06-04T19:14:08.423222000',
           '2019-06-04T20:10:21.349958000', '2019-06-04T21:06:34.131405000',
           '2019-06-04T22:02:47.261639000', '2019-06-04T22:58:59.886398000',
           '2019-06-04T23:55:12.506308000', '2019-06-05T00:51:25.729198000',
           '2019-06-05T01:47:38.332130000', '2019-06-05T02:43:51.566661000',
           '2019-06-05T03:40:04.406806000', '2019-06-05T04:36:17.290126000',
           '2019-06-05T05:32:29.997112000', '2019-06-05T06:28:43.015529000',
           '2019-06-05T07:24:55.974767000', '2019-06-05T08:21:08.665745000',
           '2019-06-05T09:17:21.622070000', '2019-06-05T10:13:34.263081000',
           '2019-06-05T11:09:47.118748000', '2019-06-05T12:06:00.430169000',
           '2019-06-05T13:02:12.927590000', '2019-06-05T13:58:25.857479000',
           '2019-06-05T14:54:38.828116000', '2019-06-05T15:50:51.536800000',
           '2019-06-05T16:47:04.515684000', '2019-06-05T17:43:17.251774000',
           '2019-06-05T18:39:30.413299000', '2019-06-05T19:35:43.493326000',
           '2019-06-05T20:31:56.178966000', '2019-06-05T21:28:09.338063000',
           '2019-06-05T22:24:22.097197000', '2019-06-05T23:20:34.807094000',
           '2019-06-06T00:16:47.479394000', '2019-06-06T01:13:00.649892000',
           '2019-06-06T02:09:13.264222000', '2019-06-06T03:05:26.449758000',
           '2019-06-06T04:01:39.097560000', '2019-06-06T04:57:51.955170000',
           '2019-06-06T05:54:04.748260000', '2019-06-06T06:50:17.941558000',
           '2019-06-06T07:46:30.758175000', '2019-06-06T08:42:43.742393000',
           '2019-06-06T09:38:56.391895000', '2019-06-06T10:35:09.167037000',
           '2019-06-06T11:31:22.491312000', '2019-06-06T12:27:34.865034000',
           '2019-06-06T13:23:47.867444000', '2019-06-06T14:20:00.700313000',
           '2019-06-06T15:16:13.915198000', '2019-06-06T16:12:26.640131000',
           '2019-06-06T17:08:39.492644000', '2019-06-06T18:04:52.382513000',
           '2019-06-06T19:01:05.536033000', '2019-06-06T19:57:18.053101000',
           '2019-06-06T20:53:31.393868000', '2019-06-06T21:49:43.907056000',
           '2019-06-06T22:45:56.760298000', '2019-06-06T23:42:09.801274000',
           '2019-06-07T00:38:22.360547000', '2019-06-07T01:34:35.427961000',
           '2019-06-07T02:30:48.450503000', '2019-06-07T03:27:01.348618000',
           '2019-06-07T04:23:14.042504000', '2019-06-07T05:19:27.144846000',
           '2019-06-07T06:15:39.983293000', '2019-06-07T07:11:53.179743000',
           '2019-06-07T08:08:05.643451000', '2019-06-07T09:04:18.786055000',
           '2019-06-07T10:00:31.215563000', '2019-06-07T10:56:44.553178000',
           '2019-06-07T11:52:57.095229000', '2019-06-07T12:49:09.862124000',
           '2019-06-07T13:45:22.783279000', '2019-06-07T14:41:35.991373000',
           '2019-06-07T15:37:48.580721000', '2019-06-07T16:34:01.439786000',
           '2019-06-07T17:30:14.271441000', '2019-06-07T18:26:27.474440000',
           '2019-06-07T19:22:40.389048000', '2019-06-07T20:18:53.235742000',
           '2019-06-07T21:15:06.353364000', '2019-06-07T22:11:19.003590000',
           '2019-06-07T23:07:32.004303000', '2019-06-08T00:03:44.668356000',
           '2019-06-08T00:59:57.689929000'], dtype='datetime64[ns]')

• Data variables: (2)
  • SIGMA0.ALTI
    (x)
    float64
    13.99 13.52 13.77 ... 13.69 14.82

    array([13.98551652, 13.52472868, 13.76544872, 13.81065417, 12.84655729,
           13.94048553, 13.93910725, 14.71910793, 14.06445236, 14.02599372,
           13.62702996, 14.77228155, 13.78054668, 13.68293764, 14.30784578,
           13.87495134, 14.37133842, 13.79951728, 13.87965426, 12.77671525,
           13.93437238, 13.64756616, 14.31633177, 13.91608771, 14.37140647,
           14.05201761, 13.43365275, 14.37998725, 13.82309503, 13.89143289,
           12.73969196, 14.08867771, 13.54454084, 14.23411744, 14.61984736,
           14.83143419, 13.60090789, 14.04149687, 13.58281316, 13.49320298,
           14.87220746, 13.49988036, 13.9091692 , 13.60578436, 13.82389053,
           13.67954317, 13.96935051, 14.03325863, 14.51963997, 13.58131636,
           15.06956659, 14.04475474, 13.3419021 , 14.09672708, 12.79690533,
           13.55258162, 12.65208215, 14.24501549, 14.23899336, 13.98445236,
           14.32205391, 14.09704051, 13.40888801, 14.89978311, 14.02466485,
           13.93904   , 14.46796272, 13.44641527, 13.56788815, 13.50909229,
           14.30348921, 13.9808267 , 14.41220613, 13.97101307, 14.42650658,
           14.46311152, 13.91624318, 13.62138778, 13.81940935, 14.11027941,
           13.18272321, 13.93680322, 13.20754895, 13.90279758, 14.00109756,
           14.29053396, 13.67314769, 14.43027855, 13.50774517, 13.87128267,
           13.97310271, 14.20197516, 13.8018541 , 14.16604785, 14.0937989 ,
           12.60227754, 13.80190375, 13.48431849, 14.30041949, 13.91689996,
           14.04425645, 13.66693009, 13.44581892, 14.84016667, 13.39600279,
           13.7176897 , 11.73644041, 13.54972757, 14.66875881, 13.69151167,
           13.96909291, 14.22171369, 13.62383843, 13.55014663, 13.77030747,
           13.76566346, 14.3066223 , 14.11517256, 14.27043894, 14.25415486,
           13.56340626, 13.42620082, 13.50379864, 13.98660765, 13.72536413,
           13.7710791 , 14.03860396, 13.16921829, 12.89126173, 13.97917614,
           13.22764105, 13.66878351, 12.04158004, 13.86179567, 13.85924686,
           13.83741959, 14.02147391, 13.84342872, 13.81234823, 12.90030581,
           13.87493397, 13.01142159, 13.85621842, 13.40960183, 13.20452502,
           13.05664787, 13.58315534, 13.21215752, 13.87122172, 14.03754425,
           14.67198686, 14.0942734 , 13.93011586, 14.11834753, 12.68397615,
           14.23032451, 13.18907183, 13.63108775, 13.19750168, 14.33359703,
           14.04378873, 13.84559904, 13.88131409, 14.17196244, 13.6389079 ,
           13.58054082, 14.11001975, 13.16359349, 14.03079526, 14.01150025,
           13.48937401, 12.28901437, 13.9311953 , 13.68738981, 14.81811983])

  • SLA
    (x)
    float64
    -2.283 -0.2079 ... -2.641 -2.712

    array([-2.28265003, -0.20786509, -2.38207967, -2.4357797 ,  0.36038206,
           -2.43042596,  3.13104819, -2.47987539, -2.35612418, -2.46214445,
           -1.49596477, -2.47016595, -2.40872963, -2.68646305,  1.69717311,
           -0.92712757, -1.86601747, 11.53248762, -2.42629378,  3.0517613 ,
           -2.15873684, -2.29983024, -2.26444189, -2.40594512, -2.26874344,
           -2.25632891, -2.28625965, -1.70241223, -2.19127771, -1.62412854,
           -1.80477225, -2.40321494,  0.14149508, -2.47905742, -1.62539129,
           -1.4457048 , -1.93003269, -2.02842294,  2.21621684, -2.44807525,
            0.57223089, -1.66480165, -1.93214742, 13.22346901, -1.56323938,
            0.74400713, -2.38514998,  0.6733592 , -2.52003381, -2.40878648,
           -2.11569893, -2.09177238, -2.3802208 , -2.32684167, -2.28674604,
           -1.54644611,  0.7865388 , -2.42238235, -2.26884705, -2.43004058,
           -2.42703556, -1.017817  , -2.25802773, -2.36993501, -1.85945871,
           -2.12932351,  2.28219953,  0.61019099, -2.39318156, 15.29427523,
           -2.36586862, -1.48976121, -2.53792184, -2.49810958, -0.17081063,
           -2.4658483 , -1.4290227 , -1.27045528, -2.2242183 , -2.41798989,
            0.22618447, -2.40292642,  2.63386054, -2.50659474, -2.54519294,
           -2.31966599, -2.19002952, -2.47174382, -2.31145004, -2.46072778,
            2.66908924, -2.23733858, -1.80862302,  1.27893813, -2.36331261,
           -2.3739704 , -0.4361207 , -2.08626764, -2.40064252, -2.52369213,
           -1.70233223, -2.60275554, -2.68047876, -1.45585019, -2.42141436,
           -2.55740698, -1.45696537, -2.08267331, -0.64021039, -2.60531214,
           -1.42669645, -2.15776924, -2.46828729, -2.08303448, -2.44813111,
           -2.44877774,  2.48867935, -1.986869  , -2.62975832, 12.93051913,
           -2.40396722,  2.46770495, -2.22710797, -1.62898845, -2.6873578 ,
           -2.31915065, -1.92787013, -2.83645009, -2.68622283, -2.30588463,
           -2.47084067, -1.29865439, -1.55082829, -2.43126549, -2.39862907,
           -2.596386  , -2.50377108, -2.00614099, -2.49610628, -1.66828898,
           -2.01571561, -2.64342225,  8.40959582, -1.73074253, -2.57257363,
           13.12375499,  1.44513437, -2.31153341, -2.74582698, -2.48325975,
            1.12089127, -2.12980987, -2.07879826, -2.85259267, -2.74376773,
           -2.48193629, -2.384709  , -1.28591951,  0.0604219 , -2.43744726,
           -2.53157269, -2.50747428, -2.5556773 , -2.30839129, -2.53054813,
           -2.14549156, -2.49359734, -2.01550753,  4.33199303,  2.74495777,
           -2.54990982,  2.49772126,  1.90997036, -2.64124176, -2.71195672])

• Indexes: (1)
  • x
    PandasIndex

    PandasIndex(DatetimeIndex(['2019-06-01 05:58:35.692381', '2019-06-01 06:54:48.840566',
                   '2019-06-01 07:51:01.899977', '2019-06-01 08:47:14.784509',
                   '2019-06-01 09:43:27.743745', '2019-06-01 10:39:40.610329',
                   '2019-06-01 11:35:53.156019', '2019-06-01 12:32:06.214703',
                   '2019-06-01 13:28:18.770337', '2019-06-01 14:24:31.892810',
                   ...
                   '2019-06-07 16:34:01.439786', '2019-06-07 17:30:14.271441',
                   '2019-06-07 18:26:27.474440', '2019-06-07 19:22:40.389048',
                   '2019-06-07 20:18:53.235742', '2019-06-07 21:15:06.353364',
                   '2019-06-07 22:11:19.003590', '2019-06-07 23:07:32.004303',
                   '2019-06-08 00:03:44.668356', '2019-06-08 00:59:57.689929'],
                  dtype='datetime64[ns]', name='x', length=175, freq=None))

• Attributes: (0)

ad.get_data_group(name="Missing points 1", stat="value")

{MISSING_POINTS: <xarray.Dataset> Size: 19MB
 Dimensions:    (time: 586885)
 Coordinates:
   * time       (time) datetime64[ns] 5MB 2019-06-01T05:30:29.807497 ... 2019-...
 Data variables:
     LONGITUDE  (time) float64 5MB 17.01 17.14 17.27 17.39 ... 236.3 236.4 236.5
     LATITUDE   (time) float64 5MB -66.15 -66.15 -66.15 ... 66.15 66.15 66.15
     missing    (time) bool 587kB False False False False ... False False False
     mp_groups  (time) int64 5MB 1 1 1 1 1 1 1 1 1 1 1 ... 0 0 0 0 0 0 0 0 0 0 0,
 GEOBOX: {'GLOBAL': None, 'LAND': None, 'OCEAN': None},
 TEMPORAL: {'GLOBAL': {'PASS': None},
  'LAND': {'PASS': None},
  'OCEAN': {'PASS': None}},
 SECTION_ANALYSES: {}}

Editing computed data and adding new ones

Raw data can be accessed and edited, through the data container data property.

Accessing this property will return a copy of the data container raw data but assigning a new modified dataset to this property will make everything in it usable for the computation of new diagnostics.

More information can be found here about how to assign new variables to xarray datasets.

A complete example on how to manually edit data is available in this notebook.