Report Manager

The ReportManager is a generic tool allowing to generate and execute notebooks from a template notebook and a configuration.

These notebooks can, then, be exported to PDF, HTML or LaTeX documents.

Notebook’s template are regular notebooks that might contain Jinja annotation referencing predefined data from the notebook itself or data included in the provided configuration.

This template can use markdown, HTML, LaTeX or any of the possibilities offered by the notebook format to create the structure of a report.

Configurations can be provided as a simple dictionary (build from a YAML or JSON file for example) whose keys and values are accessible through Jinja annotations or using a Report object allowing powerful features like code generation and chapter/section iterations.

Quick overview

There are a few simple steps to perform in order to generate a report:

Create a template notebook and a configuration

Initialize a ReportManager with its template and configuration

Expand the template notebook

Execute the expanded notebook

Export the executed notebook

The following code illustrate these different steps.

from octantng.data.report import ReportManager
from octantng.parameters.configuration import Config

# Initializing
rm = ReportManager(
    template="template.ipynb",
    configuration=Config.from_file("simple_conf.yaml")
)

# Expanding (might be saved)
rm.expand_template()
# rm.store_notebook(path="notebook_expanded.ipynb")

# Executing (might be saved too)
rm.execute_notebook()
# rm.store_notebook_executed(path="notebook_executed.ipynb")

# Exporting to PDF, HTML or LaTeX
rm.to_pdf(report_name="report.pdf")
# rm.to_html(report_name="report.html")
# rm.to_latex(report_name="report.tex")

Notebook template

A template is regular notebook (created using JupyterLab or Jupyter Notebook) which might contain additional Jinja templates elements:

{{ ... }} for expressions referencing data from configuration, report and notebook

{% ... %} for statements

Note

Statements cannot be split over several cells. They only have effect in a single cell.

Jinja expressions and statements can contain variables declared inside the template itself or in the configuration.

An additional loop statement have been designed in order to be able to perform loops containing any number of cells including nested loops.

The following loop will iterate over vars_list while putting the current value into a_var:

__loop {% set variable, values = "a_var", {{ vars_list }} %}

… any number of cells …

… references to {{ a_var }} …

__end_loop

Note

The __loop and __end_loop statements have to be contained in Markdown cells.

set, variable and values are keywords that cannot be changed when using loops.

Example: Notebook template containing Jinja expressions and loops.

Configuration

A configuration dictionary might contain any kind of data or sub-dictionaries and can be accessed by their keys using Jinja expressions: {{ ... }}.

configuration_example.yaml

title: This is a title

fields:
- name: SLA
  text: Some text related to SLA
- name: SIG0
  text: Some text related to Sigma0

misc:
  a_text: >
          Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's
          standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make
          a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining
          essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages,
          and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum

Data from this configuration file, once loaded into a dictionary, can be accessed in the following manner.

title content: {{ title }}

a_text content: {{ misc.a_text }}

Looping over fields names: __loop {% set variable, values = “field”, {{ fields }} %}

Example: Notebook template containing expressions and loops with variables declared in the configuration.

Report

Using a Report class instead of or in addition to a configuration dictionary provides another way to inject data into templates.

The Report class can be inherited from to deeply customized and validate the type of data a configuration might contain. ReportCasys is an example of this kind of usage. Additional information can be found on the casys report page.

Class representing a report.

Parameters
----------
metadata
    Report metadata (used in PDF exports).
data
    Dictionary of sections containing data required in chapters.
chapters
    Dictionary of chapters.
misc
    Dictionary of miscellaneous data.
level
    Report's level.
    If the report level is 100 (default), only chapters and sections with a level
    less or equal to 100 will be included.

The Report object can be accessed in templates using the _report keyword followed by the attribute name: {{ _report.chapters["chapter 1"] }}.

Using a Report gives access to the following features:

Chapters iteration using the chapters_list attribute:

__loop {% set variable, values = "chapter", {{ _report.chapters_list }} %}

Automatic definition of the _chapter variable containing the current ReportChapter

Sections iteration using the sections_list attribute

__loop {% set variable, values = "section", {{ _chapter.sections_list }} %}

Automatic definition of the _section variable containing the current ReportSection

Subsections iteration using the sections_list attribute

__loop {% set variable, values = "subsection", {{ _section.sections_list }} %}

Automatic definition of the _subsection variable containing the current ReportSection

Executable code generation for sections and subsections using the code attribute

{{ _section.code }}

{{ _subsection.code }}

report_example.yaml

metadata: <ReportMetadata_representation>

data:
    <data1>:
        <Data_representation>

    <data2>:
        <Data__representation>
    ...

chapters:
    <chapter1>:
        <ReportChapter_representation>
    <chapter2>:
        <ReportChapter_representation>
    ...

misc:
    <dictionary>

level: <report_level>

Loading report

Report can be instantiated from dictionary using the load() method.

Instantiate a report using the provided configuration dictionary.

Parameters
----------
config
    Report's configuration.

Returns
-------
:
    Instantiated Report.

from octantng.data.report import Report, ReportManager
from octantng.parameters.configuration import Config

conf = Config.from_file("report_conf.yaml")
r = Report.load(conf)

rm = ReportManager(template="generic_template.ipynb", report=r)

Report’s metadata

ReportMetadata are used to customize PDF reports.

Metadata used in report to design the first page of the pdf report.

Parameters
----------
title
    Report's title.
date
    Report's date.
authors
    Authors' list.
    Each author is represented as a dictionary: {"name": "author_name"}.
logos
    List of lists of logos path.
    Each sublist of logos will be displayed on the same line.

metadata_example.yaml

title: <report_title>

date: <report_date>

authors:
    - <author_1>
    - <author_2>
    ...

logos:
    - [<Path/to/logo_1>, <Path/to/logo_2>]
    - [<Path/to/logo_3>]
    ...

Report’s data

Report data is a dictionary of ReportSection representing data used as inputs for the template.

These data are automatically injected in the beginning of the expanded notebook in order to be accessible in chapter’s sections.

Report’s chapter

ReportChapter is the highest level report’s content structure allowing to group and give a name to a set of ReportSection.

Representation of a report chapter.

Parameters
----------
title
    Chapter's title.
text
    Chapter's text.
sections
    Chapter's sections.
misc
    Miscellaneous chapter's data.
level
    Chapter's level.
    Only chapters with a level less or equal to the report's level will be included.

It is possible to iterate over ReportChapter using the report’s chapters_list attribute.

Doing so will set a _chapter variable usable in the current loop.

_loop {% set variable, values = "chapter", {{ _report.chapters_list }} %}
...
_end_loop

This chapter’s ReportSection can themselves be iterated over using the sections_list attribute which will set a _section variable usable in the current loop.

_loop {% set variable, values = "section", {{ _chapter.sections_list }} %}
...
_end_loop

chapter_example.yaml

title: <chapter_title>

text: <chapter_text>

sections:
    - <ReportSection_representation>
    - <ReportSection_representation>
    ...

misc:
    <dictionary>

level: <chapter_level>

Report’s section

ReportSection is the lowest content structure of a report. It represents a single content with its attributes (title, text, misc).

Representation of a report section.

Parameters
----------
title
    Section's title.
text
    Section's text.
content
    Section's content.
misc
    Miscellaneous section's data.
reference
    Reference name for this section.
sections
    List of subsections.
level
    Section's level.
    Only sections with a level less or equal to the report's level will be included.
context
    Context information used to instantiate this section.

It might contain subsections which can be iterated over using the sections_list attribute, which will set a _subsection variable usable in the current loop.

__loop {% set variable, values = "subsection", {{ _section.sections_list }} %}
...
__end_loop

When implemented, a section content can be transformed into executable code by using its code attribute instead of the content one.

_section.code

If this code represents an object, it might be named using the reference attribute. This reference’s name can be used in the following notebook’s cells.

section_example.yaml

title: <section_title>

text: <section_text>

content: <section_content>

misc:
    <dictionary>

reference: <section_reference>

sections:
- <ReportSection_representation>
- <ReportSection_representation>
...

level: <section_level>

Visibility level

Report, ReportChapter and ReportSection can be assigned a level, in order to set the visibility level of these elements and easily control which parts of the report will be present in the expanded notebook (and the report).

If the report level is 100 (default), only chapters and sections with a level less or equal to 100 will be included.

When not set, elements levels are inherited from their parent.

Example: Only section 1 and 2 will be included. The whole chapter 2 is excluded (event section 5) due to its own level.

level_example.yaml

level: 50

chapters:
  chapter_1:
    level: 25

    title: Chapter 1

    sections:
      - title: Section 1
        content: ...
      - title: Section 2
        level: 10
        content: ...
      - title: Section 3
        level: 75
        content: ...

  chapter_2:
    level: 75

    title: Chapter 2

    sections:
      - title: Section 4
        content: ...
      - title: Section 5
        level: 10
        content: ...

YAML aliases

Report, ReportChapter and ReportSection configurations files (and dictionaries) accept an alias parameter.

It might be used to declare YAML anchors that can be referred to using aliases in the configuration.

This parameter will not be parsed during the instantiation and can be used to put any kind of data.

More information about the YAML anchors mechanism can be found here.

alias_example.yaml

alias:

  params:
    params_1: &params_1
      v1: value 1_1
      v2: value 2_1

    params_2: &params_2
      v1: value 1_2
      v2: value 2_2

  operations:
    operations_1: &op_1
      - type: operation_1
      - type: operation_2

  plots:
    plot_swh_day_ts: &plot_swh_day_ts
      data: data1
      data_name: swh_day_ts
      stat: mean
      text_params:
        << : *params_1
        title: Day Mean SWH
      plot_params: *params_2
      operations: *op_1

chapters:
  chapter_1:
    title: Chapter 1
    sections:
      - title: Section 1
        content:
          << : *params_1
          v3: value_3

      - title: Section 2
        content:
          << : *params_2
          v2: value_2x

      - title: Section 3
        content: *params_1

Processing steps

Generating a report is done by exporting an executed notebook into a specific format.

The ReportManager integrates all the processing steps required to get from a template to a report.

These processing steps include:

ReportManager initialization

Template expansion

Template execution

Report generation

Note

Generating a report does not require to manually execute each previous step: required steps will be automatically executed.

Initialization

This step requires to instantiate a ReportManager with optional configuration and/or report parameters.

Class allowing to build a notebook report from a template, a Report
and/or a configuration file.

The template is rendered using the provided configuration data and report.

Using a report object allows more advanced feature and notations.
The report object is available through the '_report' name.

Parameters
----------
template
    Notebook to use as template.
report
    Report provided through the '_report' key word to the template.
report_metadata
    Report's metadata used to design generated pdf.
configuration
    Dictionary from which keys are provided to the template.

Example

from octantng.data.report import ReportManager
from octantng.parameters.configuration import Config

rm = ReportManager(
    template="template.ipynb",
    configuration=Config.from_file("simple_conf.yaml")
)

Template expansion

Expanding a notebook means expanding loops and replacing all references (Jinja expressions) to configuration data in the template.

This is done using the expand_template() method. The result is an executable, but not yet executed, notebook.

Note

Notebooks containing jinja expressions referencing variables from the notebook itself will only be expanded during the execution step.

Example

rm.expand_template()

Expanded notebook can be optionally stored using the store_notebook() method.

rm.store_notebook(path="notebook_expanded.ipynb")

Example: Expanded notebook.

Expanded notebook execution

The last step before exporting a notebook is to execute it. This is done using the execute_notebook() method.

Execute the content of the expanded notebook.

Parameters
----------
kernel_name
    Name of the kernel to use.
timeout
    Timeout in seconds.
allow_errors
    Whether to allow cell execution errors or not.
kwargs
    Additional preprocessor parameters.

Tip

This function accepts named and kwargs parameters passed to nbconvert Preprocessors

rm.execute_notebook(timeout=20, allow_error=False)

Example

rm.execute_notebook()

Executed notebook can be optionally stored using the store_notebook_executed() method.

rm.store_notebook_executed(path="notebook_executed.ipynb")

Example: Executed notebook.

Report generation

The report generation allows to transform the executed notebook into one of the following formats:

PDF

HTML

LaTeX

Warning

Raw cells are not correctly rendered when exporting to one of these formats.

Avoid using this kind of cells!

Generating a PDF report

rm.to_pdf(report_name="report", output_dir="generated")
Tip

The first page of the generated PDF design can be configured using the ReportMetadata class, provided into the Report parameter.

Generating a HTML report

rm.to_html(report_name="report", output_dir="generated")

Generating a LaTeX report

rm.to_latex(report_name="report", output_dir="generated")

Resources extraction

Outputs of a notebook, such as images, can be extracted and stored into a specific directory using the extract_resources() method.

Extract notebook outputs (images) and save them in the provided
directory.

Parameters
----------
output_dir
    Absolute or relative path where to store notebook resources.

Export functions

Export functions are available as standalone functions to export any executed notebook to PDF, HTML or LaTeX.

Note

Notebook cells can be tagged with the no_output tag in order to be excluded from the report.

More information about cell tags can be found on this page.

Exporting to PDF: export_notebook_to_pdf()

Export the provided notebook to pdf.

Parameters
----------
notebook
    Notebook to export
report_name
    Name of the pdf report.
output_dir
    Output directory where to save report and resources.
template_file
    Template file path to format output pdf
use_pdflatex
    Whether to use pdflatex instead of xelatex or not.

Exporting to HTML: export_notebook_to_html()

Export the provided notebook to html.

Parameters
----------
notebook
    Notebook to export
report_name
    Name of the html report (without extension).
output_dir
    Output directory where to save report and resources.

Exporting to LaTeX: export_notebook_to_latex()

Export the provided notebook to latex.

Parameters
----------
notebook
    Notebook to export
report_name
    Name of the latex report (without extension).
output_dir
    Output directory where to save report and resources.
template_file
    Template file path to format output latex

Notebook examples

List of notebook examples linked in this documentation:

Template showing loops and internal variables usage.

Template showing dictionary configuration usage (with nested loops).

Expanded notebook.

Executed notebook.