Results visualization

The simulation results from Nexus-e can be interactively viewed through a visualization tool (also called the “webviewer”), developed with Python Dash.

This documentation only introduces the general workflow of developing the visualization tool. Detailed instructions can be found in the README.md files in the associated repositories.

The visualization of the simulation results is achieved in the workflow shown in the figure. The key processes will be explained in the sub-sections below.
_images/visualization-workflow.png

Postprocess

After a simulation finishes, selected results are postprocessed and uploaded to the output database. This postprocess procedure is carried out in the following two ways. The two ways are designed for different purposes and both are needed.

  1. Embedded in the run_Nexuse.m script

    • Purpose: Automatically process the results when finishing a simulation, so that the reseacher could promtly see the results of this simulation from the webviewer.

    • How it work: After the simulation finishes, run_Nexuse.m calls the MATLAB script Shared/resultPostProcess/postProcess.m. This scripts calls other MATLAB/Python scripts to postprocess the results.

  2. Stand-alone

    • Purpose: Batch-process multiple simulations, fine-tune/modify/test/debug the post-process scripts.

    • How it works: Run the MATLAB script postProcess_batch.m locally. This script loops through multiple simulations and runs only the required post-process scripts (e.g., we don’t have to run the complete post-process for quick debugging). Which simulations to loop through and which post-processes to run can be configured at the “To be edited” section in the script. Details can be found in the comments in the script.

    Note

    To properly run the postProcess_batch.m script, you need to set up the paths for the required software (e.g., Python, GAMS). To do this, follow the instructions in Run_Nexuse/user_example_conf.m and the online documentation.

Output database

In the output database (host: itet-psl-s02, port: 3307), each simulation’s results is stored in one schema named with the simulation submission date and the scenario name (e.g., the scenShortName variable in run_Nexuse.m), which looks like “01-dec-2020_baseline”.

The output database can be viewed with MySQL Workbench in the same way as introduced here.

Retrieve & visualize data with a URL

After the postprocessed data has been uploaded to the output database, the data can be retrieved and visualized by provideing a URL through the webviewer. Three types of URL are accepted.

  1. https://nexus-e.org/results/JOB-SUBMISSION-DATE_SCENARIO_NAME, e.g., https://nexus-e.org/results/01-dec-2020_baseline. This will retrieve and visualize only the data from the exact MySQL schema named as “JOB-SUBMISSION-DATE_SCENARIO_NAME”.

  2. https://nexus-e.org/results/JOB-SUBMISSION-DATE, e.g. https://nexus-e.org/results/01-dec-2020. This will retrieve and visualize the data from all the MySQL schemas whose name start with “JOB-SUBMISSION-DATE”. In other words, results from all simulations submitted on the same day will be shown on the website.

  3. https://nexus-e.org/results/MANUALLY_DEFINED_URL, e.g. https://nexus-e.org/results/project. This will retrieve and visualize the results from all simulations associated with the MANUALLY_DEFINED_URL. This is particularly useful if we run multiple scenarios for one project. The mapping of URL and simulations is specified in the constants.py in the repository for the webviewer.

Note

Synchronizing data between the web server and the database

Not all changes in the output database will be automatically sychronized to the webviewer. The reason lies in how the data is downloaded from the database to the web server: based on the given URL, the webviewer checks the output database and decides which simulation(s) will be shown, i.e., which schema(s) should be downloaded to the web server; the webviewer then checks (by comparing the existing folder names and the schema name) whether the needed schema has already been downloaded; if the viewer finds a folder with the same name as the needed schema, it won’t re-download the schema.

Therefore, to show a changed data in the output database (e.g., after changing the postprocess script, added new data), you need to first delete the corresponding data folder on the web server so that the webviewer will re-download the data from the database.

Embed the webviewer into the homepage

The webviewer itself is plainly a Python Dash application without much Nexus-e branding. To change this, it is embedded into the homepage with iframe - a html element to display a nested webpage within another webpage. By doing so, we can easily have the same header, footer and menubar for the visualization tool as in other Nexus-e webpages.

The embedding can only be done by the administrators of the homepage. If you are an administrator, you could login to the homepage and check the backend of this page to learn how it is done.