- Jupyter Notebook 62.7%
- Python 37.3%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
Updated README to include repository overview, data modalities, processing scripts, data access instructions, and licensing information. |
||
| data | ||
| scripts | ||
| src | ||
| CITATION.cff | ||
| environment.yml | ||
| LICENSE | ||
| README.md | ||
What is this Repo?
This repository accompanies the paper and the Zenodo dataset “Batch Distillation Data for Developing Machine Learning Anomaly Detection Methods” (DOI: https://doi.org/10.5281/zenodo.17395543). It provides the code and scripts used for data preprocessing, visualization, and reproducible anomaly-detection workflows described in the manuscript.
Data modalities and generation scripts
Each data modality included in this repository is generated by a dedicated script with the exact same name as the modality folder:
00_Timeseries_Label_Anomaly_Metadata01_Timeseries_Sensors02_Timeseries_Actuators03_Timeseries_Sensor_Uncertainties04_Tabular_Ambient_Conditions05_Tabular_Mixture_Information06_Tabular_Plant_Information07_Tabular_Operation_Logs08_NMR_Composition09_NMR_Composition_Equipment10_GC_Composition11_Audio12_Image
For each modality, the corresponding script loads and processes an example dataset located at:
data/{MODALITY}/raw/example
and writes the processed outputs to the respective modality directory (e.g., data/{MODALITY}/processed/ or an equivalent output folder defined in the script).
Recommended order
The recommended order of running the processing notebooks is:
Process_01_Timeseries_Sensors.ipynb(Processing steps 1 and 2)Process_03_Timeseries_Sensor_Uncertainties.ipynbProcess_01_Timeseries_Sensors.ipynb(Processing step 3)Process_02_Timeseries_Actuators.ipynbProcess_10_GC_Composition.ipynbProcess_09_NMR_Composition_Equipment.ipynbProcess_08_NMR_Composition.ipynbProcess_04_Tabular_Ambient_Conditions.ipynbProcess_05_Tabular_Mixture_Information.ipynbProcess_06_Tabular_Plant_Information.ipynbProcess_07_Tabular_Operation_Logs.ipynbProcess_11_Audio.ipynbProcess_12_Image.ipynb
Data access via eLabFTW (as used in the paper)
In the workflow described in the paper, most data are not read from local files but are retrieved on demand from our laboratory organization system eLabFTW via its API. In practice, the relevant records (e.g., experiments, resources, attachments, metadata) are accessed using their respective eLabFTW identifiers (IDs) to ensure unambiguous linkage between measurements, annotations, and contextual information.
This repository therefore provides scripts and utilities that can either:
- operate on the example files included under
data/{MODALITY}/raw/example, or - pull the corresponding data directly from eLabFTW using the appropriate IDs and API credentials (see the configuration instructions in this repository).
Using eLabFTW as the central system for laboratory organization helps to maintain a consistent provenance trail (data ↔ experiment ↔ metadata ↔ attachments) and enables reproducible data retrieval aligned with the procedures reported in the paper.
During the processing pipeline, data modalities are handled internally using their eLabFTW IDs and are processed in that ID-based form while the scripts run. Only in a later step are the generated outputs reorganized into the Zenodo dataset folder structure. This final re-structuring is performed by the dedicated SORTING script, which maps each experiment ID to its corresponding directory and moves the processed files accordingly.
Process control system script (hardware- and software-specific)
This repository includes a script that was used as the process control system (PCS) during the experiments. It is provided for transparency and documentation, but it cannot be used out of the box.
In its current form, the PCS script is specific to an iLudest LM 2/S batch distillation plant and requires:
- the vendor control software IluLab to be running, and
- OPC UA communication enabled for data exchange and control.
In addition, the script depends on the IDE/program environment itom (version ≥ 4.2.2):
- itom project page: https://itom-project.github.io/start.html
The implementation is further tailored to the experimental setup used in this work, including specific camera and audio devices, a cooling-water thermostat, and additional peripheral equipment (e.g., device-specific drivers, OPC UA endpoints, tag mappings, and local instrument configuration). Adapting the PCS script to other hardware configurations will require corresponding changes to these interfaces and settings.
Due to performance limitations observed during operation, audio recording was handled in a separate script rather than within the PCS script itself.
Distillation recipes and eLabFTW prerequisites
Distillation recipes are loaded from eLabFTW into the process control system (PCS). Therefore, the corresponding experiment entry must be created in eLabFTW before conducting a run. In addition, all required references/links (e.g., to the associated mixture, plant, and operating_point entities) must be predefined in eLabFTW so the PCS can resolve them via the API at runtime.
Live monitoring recommendation (IluLab live plots)
It is strongly recommended to keep the IluLab live plots running during experiments, as the current PCS implementation does not provide live visualization of sensor time-series data. IluLab’s live plots therefore serve as the primary interface for real-time monitoring and sanity checks while the PCS is operating.
Technical validation workflow (notebook)
A notebook describing the workflow used for the paper’s Technical Validation section is available under /scripts/. It outlines the general processing and evaluation steps applied in this work to generate the results presented in the Sectio "Technical Validation" of the main paper.
Please note that the notebook is not runnable out of the box, because the original workflow relied on the TimeSeAD package to perform the hyperparameter grid search for the anomaly detection methods. As a result, executing the workflow end-to-end requires an appropriate TimeSeAD installation and the corresponding experiment configuration used in this study.
We also refrain from uploading the trained models generated during this workflow, as storing and distributing them would require substantial disk space.
License
This dataset is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.
You are free to share and adapt the material for any purpose (including commercial use), provided that you give appropriate credit, provide a link to the license, and indicate if changes were made.
- License text: https://creativecommons.org/licenses/by/4.0/
- Zenodo record (DOI): https://doi.org/10.5281/zenodo.17395543
How to cite
If you use this dataset in your work, please cite the Zenodo record:
Arweiler, J., Jungjohann, I., Muraleedharan, A., Leitte, H., Burger, J., Münnemann, K., Jirasek, F., & Hasse, H.
Batch Distillation Data for Developing Machine Learning Anomaly Detection Methods (Version 1.0.2) [Data set]. Zenodo (2026). https://doi.org/10.5281/zenodo.17395543
BibTeX
@misc{MainDataset,
title = {Batch Distillation Data for Developing Machine Learning Anomaly Detection Methods},
author = {Arweiler, J. and Jungjohann, I. and Muraleedharan, A. and Leitte, H. and Burger, J. and Münnemann, K. and Jirasek, F. and Hasse, H.},
doi = {10.5281/zenodo.17395543},
url = {https://zenodo.org/records/17395543},
note = {Version 1.0.2},
year = {2026},
publisher = {Zenodo}
}