Note: This environment variable is required for fully deterministic CuBLAS ops on CUDA >= 10.2 when reproducible=True is set below. Without it, PyTorch raises a RuntimeError instead of training deterministically. It must be set before torch is imported. See the README FAQ for details.

In [1]:

%env CUBLAS_WORKSPACE_CONFIG=:16:8

%env CUBLAS_WORKSPACE_CONFIG=:16:8

env: CUBLAS_WORKSPACE_CONFIG=:16:8

How To Use Imagix¶

Imagix is our implementation of a variational autoencoder for image data.
This tutorial follows the structure of Getting Started - Vanillix, but is much less extensive because our pipeline works similarly across different architectures. Here we focus only on Imagix specifics.

AUTOENCODIX supports far more functionality than shown here, so we’ll also point to advanced tutorials where relevant.

IMPORTANT

This tutorial only shows the specifics of the Imagix pipeline. If you're unfamiliar with general concepts,
we recommend following the Getting Started - Vanillix Tutorial first.

What You'll Learn¶

You’ll learn how to:

1. Initialize the pipeline and run it.
   
   
2. Understand the Imagix specific pipeline steps.
   
   
3. Access the Imagix specific results (mus, sigma, kl/mmd losses).
   
   
4. Visualize outputs effectively.
   
   
5. Apply custom parameters. or Architecture
   
   
6. Save, load, and reuse a trained pipeline.
   
   

Setting the Correct Path

In [2]:

import os

p = os.getcwd()
d = "autoencodix_package"
if d not in p:
    raise FileNotFoundError(f"'{d}' not found in path: {p}")
os.chdir(os.sep.join(p.split(os.sep)[: p.split(os.sep).index(d) + 1]))
print(f"Changed to: {os.getcwd()}")

import os p = os.getcwd() d = "autoencodix_package" if d not in p: raise FileNotFoundError(f"'{d}' not found in path: {p}") os.chdir(os.sep.join(p.split(os.sep)[: p.split(os.sep).index(d) + 1])) print(f"Changed to: {os.getcwd()}")

Changed to: /home/alicia/dev/biomarker_autoencoder/autoencodix_package

1) Initialize the Imagix Pipeline¶

Imagix is a standard VAE implementation for image data. We don't allow different data modalities for Imagix.
To run the pipeline we need to prepare two things:

1. A directory with image files. We allow the following extensions: ".jpg", ".jpeg", ".png", ".tif", ".tiff" (NOT case sensitive).
   
   
2. An annotation file with metadata, where we map sample_ids to image paths. For this, we need to provide the name of the column where the image path information is stored. This is done via the img_paths_col config parameter.

❗❗Requirements ❗❗¶

The data for this tutorial is hosted on Hugging Face Hub (autoencodix/mnist) and is downloaded (and extracted) automatically in the cell below on first run.

1.1 The Dataset¶

Here we use a balanced subsample of the classic MNIST handwritten digits dataset: 2000 images, 200 per digit (0-9), 28x28 grayscale. The annotation column label gives the digit class for each image.

A Look Inside the Annotation File:

In [3]:

import os
import zipfile
import pandas as pd
import autoencodix as acx
from huggingface_hub import hf_hub_download
from autoencodix.configs.default_config import (
    DefaultConfig,
    DataConfig,
    DataCase,
    DataInfo,
)

IMGMAPPING = hf_hub_download(repo_id="autoencodix/mnist", repo_type="dataset", filename="mnist_mappings.txt")
images_zip = hf_hub_download(repo_id="autoencodix/mnist", repo_type="dataset", filename="mnist_images.zip")

IMGROOT = os.path.join("data/images/mnist_images/")
if not os.path.isdir(IMGROOT):
    with zipfile.ZipFile(images_zip) as zf:
        zf.extractall("data/images/")

anno_df = pd.read_csv(IMGMAPPING, sep="\t", index_col=0)
anno_df.head()

import os import zipfile import pandas as pd import autoencodix as acx from huggingface_hub import hf_hub_download from autoencodix.configs.default_config import ( DefaultConfig, DataConfig, DataCase, DataInfo, ) IMGMAPPING = hf_hub_download(repo_id="autoencodix/mnist", repo_type="dataset", filename="mnist_mappings.txt") images_zip = hf_hub_download(repo_id="autoencodix/mnist", repo_type="dataset", filename="mnist_images.zip") IMGROOT = os.path.join("data/images/mnist_images/") if not os.path.isdir(IMGROOT): with zipfile.ZipFile(images_zip) as zf: zf.extractall("data/images/") anno_df = pd.read_csv(IMGMAPPING, sep="\t", index_col=0) anno_df.head()

Out[3]:

	img_paths	label
sample_ids
mnist_0_0000	mnist_0_0000.png	'0'
mnist_0_0001	mnist_0_0001.png	'0'
mnist_0_0002	mnist_0_0002.png	'0'
mnist_0_0003	mnist_0_0003.png	'0'
mnist_0_0004	mnist_0_0004.png	'0'

Define Config and Run Pipeline

In [4]:

imgconfig = DefaultConfig(
    data_case=DataCase.IMG_TO_IMG,
    checkpoint_interval=25,
    epochs=250,
    reconstruction_loss="bce",
    beta=0.005,
    scaling="MINMAX",
    anneal_function="logistic-late",
    data_config=DataConfig(
        data_info={
            "IMG": DataInfo(
                file_path=IMGROOT,
                scaling="MINMAX",
                data_type="IMG",
            ),
            "ANNO": DataInfo(
                file_path=IMGMAPPING,
                data_type="ANNOTATION",
            ),
        },
    ),
)

imagix = acx.Imagix(config=imgconfig)
result = imagix.run()
backup_datset = result.datasets

imgconfig = DefaultConfig( data_case=DataCase.IMG_TO_IMG, checkpoint_interval=25, epochs=250, reconstruction_loss="bce", beta=0.005, scaling="MINMAX", anneal_function="logistic-late", data_config=DataConfig( data_info={ "IMG": DataInfo( file_path=IMGROOT, scaling="MINMAX", data_type="IMG", ), "ANNO": DataInfo( file_path=IMGMAPPING, data_type="ANNOTATION", ), }, ), ) imagix = acx.Imagix(config=imgconfig) result = imagix.run() backup_datset = result.datasets

Given image size is possible, rescaling images to: 64x64
Successfully loaded 2000 images for IMG
calling normalize image in _process_ing_to_img_case
anno key: IMG
Converting 1400 images to torch.float32 tensors...
Converting 400 images to torch.float32 tensors...
Converting 200 images to torch.float32 tensors...

/home/alicia/dev/biomarker_autoencoder/autoencodix_package/src/autoencodix/trainers/_general_trainer.py:233: UserWarning: Gradient clipping was first applied in epoch 0. Total norm of gradients (adjusted for number of features): 15.0112 exceeded max norm of 5.
  warnings.warn(

Epoch 25 - Train Loss: 446.9194
Sub-losses: recon_loss: 446.9194, var_loss: 0.0000, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 25 - Valid Loss: 502.1609
Sub-losses: recon_loss: 502.1609, var_loss: 0.0000, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 50 - Train Loss: 393.6598
Sub-losses: recon_loss: 393.6594, var_loss: 0.0004, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 50 - Valid Loss: 494.4483
Sub-losses: recon_loss: 494.4479, var_loss: 0.0004, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 75 - Train Loss: 368.1934
Sub-losses: recon_loss: 368.1889, var_loss: 0.0045, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 75 - Valid Loss: 502.7059
Sub-losses: recon_loss: 502.7012, var_loss: 0.0047, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 100 - Train Loss: 356.6522
Sub-losses: recon_loss: 356.6066, var_loss: 0.0456, anneal_factor: 0.0008, effective_beta_factor: 0.0000
Epoch 100 - Valid Loss: 509.2719
Sub-losses: recon_loss: 509.2250, var_loss: 0.0470, anneal_factor: 0.0008, effective_beta_factor: 0.0000
Epoch 125 - Train Loss: 349.5668
Sub-losses: recon_loss: 349.1469, var_loss: 0.4199, anneal_factor: 0.0062, effective_beta_factor: 0.0000
Epoch 125 - Valid Loss: 513.8068
Sub-losses: recon_loss: 513.3625, var_loss: 0.4443, anneal_factor: 0.0062, effective_beta_factor: 0.0000
Epoch 150 - Train Loss: 347.2119
Sub-losses: recon_loss: 344.2315, var_loss: 2.9805, anneal_factor: 0.0439, effective_beta_factor: 0.0002
Epoch 150 - Valid Loss: 517.9274
Sub-losses: recon_loss: 514.8240, var_loss: 3.1034, anneal_factor: 0.0439, effective_beta_factor: 0.0002
Epoch 175 - Train Loss: 353.0752
Sub-losses: recon_loss: 344.8779, var_loss: 8.1973, anneal_factor: 0.2535, effective_beta_factor: 0.0013
Epoch 175 - Valid Loss: 537.1290
Sub-losses: recon_loss: 528.8809, var_loss: 8.2481, anneal_factor: 0.2535, effective_beta_factor: 0.0013
Epoch 200 - Train Loss: 358.6517
Sub-losses: recon_loss: 348.4656, var_loss: 10.1861, anneal_factor: 0.7150, effective_beta_factor: 0.0036
Epoch 200 - Valid Loss: 536.7677
Sub-losses: recon_loss: 525.9826, var_loss: 10.7851, anneal_factor: 0.7150, effective_beta_factor: 0.0036
Epoch 225 - Train Loss: 357.3189
Sub-losses: recon_loss: 347.0379, var_loss: 10.2810, anneal_factor: 0.9488, effective_beta_factor: 0.0047
Epoch 225 - Valid Loss: 552.0830
Sub-losses: recon_loss: 541.7670, var_loss: 10.3160, anneal_factor: 0.9488, effective_beta_factor: 0.0047
Epoch 250 - Train Loss: 357.5438
Sub-losses: recon_loss: 347.5162, var_loss: 10.0277, anneal_factor: 0.9928, effective_beta_factor: 0.0050
Epoch 250 - Valid Loss: 551.6637
Sub-losses: recon_loss: 541.0791, var_loss: 10.5847, anneal_factor: 0.9928, effective_beta_factor: 0.0050
Processed 400 / 400 samples

2) Understand Imagix Specific Steps¶

Since Imagix is jus a Varix for images, there are no extra steps for this pipeline.

3) Access Imagix Specific Results¶

The result object follows our standard interface. Refer to [1] for more details.
We don't have any Imagix specific results, but you can directly visualize the reconstructions as images, as shown in the code below:

[1] Tutorials/DeepDives/PipelineOutputTutorial.ipynb

In [5]:

import matplotlib.pyplot as plt

sample_img = result.reconstructions.get(split="test", epoch=-1)
sample_img = sample_img[0, :, :, :]
sample_img = sample_img.squeeze()
sample_img.shape

plt.imshow(sample_img, cmap="grey")

import matplotlib.pyplot as plt sample_img = result.reconstructions.get(split="test", epoch=-1) sample_img = sample_img[0, :, :, :] sample_img = sample_img.squeeze() sample_img.shape plt.imshow(sample_img, cmap="grey")

Out[5]:

<matplotlib.image.AxesImage at 0x7fd9f62a78e0>

4) Visualize Imagix Results¶

Since the results of Imagix are visually interpretable, we add an additional visualization:
We show a grid of original images and reconstructed images with label information. See the code below for how to obtain this visualization.

In [6]:

imagix.visualizer.show_image_recon_grid(result=imagix.result, n_samples=5)

imagix.visualizer.show_image_recon_grid(result=imagix.result, n_samples=5)

Standard Visualizations
We also offer the standard visualizations as for all other pipelines. You can pass one or more column names from the annotation data. These columns will be used to color the visualizations accordingly. In this example, we use the label parameter.

In [7]:

imagix.show_result(params=["label"])

imagix.show_result(params=["label"])

Creating plots ...

5) Customize Imagix¶

Via the Config object we can apply customize the pipeline by chaning the loss function or the number of epoch or preprocessing. To get an full overview of the adjustable parameters, please refere to [2].
In Addition, there is a way to use another architecture for the image autoencoder. We currently offer two architectures:ImageVAEArchitecture and ImageVAEFastArchitecture. The fast architecture does not has BatchNorm and inplace activation functions, which can speed up the training process. The ImageVAEArchitecture is the default architecture and based on Yang & Uhler.

We can simply pass the architecture as type to the model_type parameter of the Imagix pipeline.

[2] Tutorials/DeepDives/ConfigTutorial.ipynb.

In [8]:

from autoencodix.modeling._imgfast_architecture import ImageVAEFastArchitecture
from autoencodix.modeling._imagevae_architecture import ImageVAEArchitecture


imgconfig = DefaultConfig(
    data_case=DataCase.IMG_TO_IMG,
    checkpoint_interval=25,
    epochs=250,
    reconstruction_loss="bce",
    beta=0.005,
    scaling="MINMAX",
    anneal_function="logistic-late",
    data_config=DataConfig(
        data_info={
            "IMG": DataInfo(
                file_path=IMGROOT,
                scaling="MINMAX",
                data_type="IMG",
            ),
            "ANNO": DataInfo(
                file_path=IMGMAPPING,
                data_type="ANNOTATION",
            ),
        },
    ),
)

imagix = acx.Imagix(config=imgconfig, model_type=ImageVAEFastArchitecture)
result = imagix.run()
backup_datset = result.datasets

from autoencodix.modeling._imgfast_architecture import ImageVAEFastArchitecture from autoencodix.modeling._imagevae_architecture import ImageVAEArchitecture imgconfig = DefaultConfig( data_case=DataCase.IMG_TO_IMG, checkpoint_interval=25, epochs=250, reconstruction_loss="bce", beta=0.005, scaling="MINMAX", anneal_function="logistic-late", data_config=DataConfig( data_info={ "IMG": DataInfo( file_path=IMGROOT, scaling="MINMAX", data_type="IMG", ), "ANNO": DataInfo( file_path=IMGMAPPING, data_type="ANNOTATION", ), }, ), ) imagix = acx.Imagix(config=imgconfig, model_type=ImageVAEFastArchitecture) result = imagix.run() backup_datset = result.datasets

Given image size is possible, rescaling images to: 64x64
Successfully loaded 2000 images for IMG
calling normalize image in _process_ing_to_img_case
anno key: IMG
Converting 1400 images to torch.float32 tensors...
Converting 400 images to torch.float32 tensors...
Converting 200 images to torch.float32 tensors...
Epoch 25 - Train Loss: 467.5948
Sub-losses: recon_loss: 467.5943, var_loss: 0.0005, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 25 - Valid Loss: 525.0574
Sub-losses: recon_loss: 525.0569, var_loss: 0.0005, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 50 - Train Loss: 402.6618
Sub-losses: recon_loss: 402.6540, var_loss: 0.0078, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 50 - Valid Loss: 524.4652
Sub-losses: recon_loss: 524.4579, var_loss: 0.0073, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 75 - Train Loss: 372.4660
Sub-losses: recon_loss: 372.3845, var_loss: 0.0815, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 75 - Valid Loss: 541.9507
Sub-losses: recon_loss: 541.8746, var_loss: 0.0760, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 100 - Train Loss: 355.8949
Sub-losses: recon_loss: 355.3382, var_loss: 0.5567, anneal_factor: 0.0008, effective_beta_factor: 0.0000
Epoch 100 - Valid Loss: 552.9639
Sub-losses: recon_loss: 552.4393, var_loss: 0.5247, anneal_factor: 0.0008, effective_beta_factor: 0.0000
Epoch 125 - Train Loss: 350.4519
Sub-losses: recon_loss: 348.7685, var_loss: 1.6834, anneal_factor: 0.0062, effective_beta_factor: 0.0000
Epoch 125 - Valid Loss: 562.0247
Sub-losses: recon_loss: 560.3930, var_loss: 1.6316, anneal_factor: 0.0062, effective_beta_factor: 0.0000
Epoch 150 - Train Loss: 349.7719
Sub-losses: recon_loss: 346.1480, var_loss: 3.6239, anneal_factor: 0.0439, effective_beta_factor: 0.0002
Epoch 150 - Valid Loss: 568.0187
Sub-losses: recon_loss: 564.5588, var_loss: 3.4599, anneal_factor: 0.0439, effective_beta_factor: 0.0002
Epoch 175 - Train Loss: 358.1825
Sub-losses: recon_loss: 350.8648, var_loss: 7.3176, anneal_factor: 0.2535, effective_beta_factor: 0.0013
Epoch 175 - Valid Loss: 583.4551
Sub-losses: recon_loss: 576.0541, var_loss: 7.4010, anneal_factor: 0.2535, effective_beta_factor: 0.0013
Epoch 200 - Train Loss: 365.6818
Sub-losses: recon_loss: 354.7713, var_loss: 10.9105, anneal_factor: 0.7150, effective_beta_factor: 0.0036
Epoch 200 - Valid Loss: 591.0481
Sub-losses: recon_loss: 580.3883, var_loss: 10.6598, anneal_factor: 0.7150, effective_beta_factor: 0.0036
Epoch 225 - Train Loss: 365.7800
Sub-losses: recon_loss: 353.9301, var_loss: 11.8499, anneal_factor: 0.9488, effective_beta_factor: 0.0047
Epoch 225 - Valid Loss: 599.0756
Sub-losses: recon_loss: 587.8735, var_loss: 11.2021, anneal_factor: 0.9488, effective_beta_factor: 0.0047
Epoch 250 - Train Loss: 361.9047
Sub-losses: recon_loss: 350.5236, var_loss: 11.3811, anneal_factor: 0.9928, effective_beta_factor: 0.0050
Epoch 250 - Valid Loss: 603.1658
Sub-losses: recon_loss: 591.9307, var_loss: 11.2351, anneal_factor: 0.9928, effective_beta_factor: 0.0050
Processed 400 / 400 samples

6) Load Save and Reuse Imagix¶

There are not Imagix specific steps here. See the Getting Started - Vanillix for details. Below is a basic save/load usecase.

In [9]:

import os
import glob

outpath = os.path.join("tutorial_res", "imagix.pkl")

imagix.save(file_path=outpath)

folder = os.path.dirname(outpath)
pkl_files = glob.glob(os.path.join(folder, "*imagix.pkl"))
model_files = glob.glob(os.path.join(folder, "*imagix.pth"))

print("PKL files:", pkl_files)
print("Model files:", model_files)

# the load functionality automatically will build the pipeline object out of the three saved files
imagix_loaded = acx.Imagix.load(outpath)

import os import glob outpath = os.path.join("tutorial_res", "imagix.pkl") imagix.save(file_path=outpath) folder = os.path.dirname(outpath) pkl_files = glob.glob(os.path.join(folder, "*imagix.pkl")) model_files = glob.glob(os.path.join(folder, "*imagix.pth")) print("PKL files:", pkl_files) print("Model files:", model_files) # the load functionality automatically will build the pipeline object out of the three saved files imagix_loaded = acx.Imagix.load(outpath)

Preprocessor saved successfully.
saving memory efficient
Pipeline object saved successfully.
PKL files: ['tutorial_res/imagix.pkl']
Model files: []
Attempting to load a pipeline from tutorial_res/imagix.pkl...
Pipeline object loaded successfully. Actual type: Imagix
Preprocessor loaded successfully.

In [10]:

result_predict = imagix_loaded.predict(data=backup_datset)

result_predict = imagix_loaded.predict(data=backup_datset)

Processed 400 / 400 samples

Visualize on the new predictions.

In [11]:

imagix_loaded.show_result(params=["label"])

imagix_loaded.show_result(params=["label"])

Creating plots ...
Absolute loss plot not found in the plots dictionary
This happens, when you did not run visualize() or if you saved and loaded the model with `save_all=False`

/home/alicia/dev/biomarker_autoencoder/autoencodix_package/src/autoencodix/utils/_result.py:348: UserWarning: Could not retrieve latent representations for epoch 249 and split 'train'. Returning empty DataFrame. This may be due to missing data in the Result object or incorrect keys.


  warnings.warn(
/home/alicia/dev/biomarker_autoencoder/autoencodix_package/src/autoencodix/utils/_result.py:348: UserWarning: Could not retrieve latent representations for epoch 249 and split 'valid'. Returning empty DataFrame. This may be due to missing data in the Result object or incorrect keys.


  warnings.warn(

/home/alicia/dev/biomarker_autoencoder/autoencodix_package/src/autoencodix/utils/_result.py:348: UserWarning: Could not retrieve latent representations for epoch 249 and split 'train'. Returning empty DataFrame. This may be due to missing data in the Result object or incorrect keys.


  warnings.warn(
/home/alicia/dev/biomarker_autoencoder/autoencodix_package/src/autoencodix/utils/_result.py:348: UserWarning: Could not retrieve latent representations for epoch 249 and split 'valid'. Returning empty DataFrame. This may be due to missing data in the Result object or incorrect keys.


  warnings.warn(

In [12]:

imagix_loaded.visualizer.show_image_recon_grid(result=imagix_loaded.result, n_samples=5)

imagix_loaded.visualizer.show_image_recon_grid(result=imagix_loaded.result, n_samples=5)