Note: This environment variable is required for fully deterministic CuBLAS ops on CUDA >= 10.2 when reproducible=True is set below. Without it, PyTorch raises a RuntimeError instead of training deterministically. It must be set before torch is imported. See the README FAQ for details.

In [1]:

Copied!

%env CUBLAS_WORKSPACE_CONFIG=:16:8
%env CUBLAS_WORKSPACE_CONFIG=:16:8

env: CUBLAS_WORKSPACE_CONFIG=:16:8

How To Use Varix¶

Varix is our implementation of a variational autoencoder.
This tutorial follows the structure of our Getting Started - Vanillix, but is much less extensive, because
our pipeline works similarly for different architectures, so here we focus only on Varix specifics.

AUTOENCODIX supports far more functionality than shown here, so we’ll also point to advanced tutorials where relevant.

IMPORTANT

This tutorial only shows the specifics of the Varix pipeline. If you're unfamiliar with general concepts,
we recommend following the Getting Started - Vanillix tutorial first.

What You'll Learn¶

You’ll learn how to:

Initialize the pipeline and run the pipeline.
Understand the Varix-specific pipeline steps.
Access the Varix-specific results (mus, sigma, KL/MMD losses).
Visualize outputs effectively.
Apply custom parameters.
Save, load, and reuse a trained pipeline.

Let’s get started! 🚀

Requirements 1: Be in the correct directory (execute below)¶

In [2]:

Copied!





import os

p = os.getcwd()
d = "autoencodix_package"
if d not in p:
    raise FileNotFoundError(f"'{d}' not found in path: {p}")
os.chdir(os.sep.join(p.split(os.sep)[: p.split(os.sep).index(d) + 1]))
print(f"Changed to: {os.getcwd()}")
import os

p = os.getcwd()
d = "autoencodix_package"
if d not in p:
    raise FileNotFoundError(f"'{d}' not found in path: {p}")
os.chdir(os.sep.join(p.split(os.sep)[: p.split(os.sep).index(d) + 1]))
print(f"Changed to: {os.getcwd()}")

Changed to: /home/alicia/dev/biomarker_autoencoder/autoencodix_package

Requirements 2: Obtain tutorial data or use own data¶

We use real human lung single-cell RNA-seq data assembled from the CZ CELLxGENE Census. It is hosted on Hugging Face Hub (autoencodix/census-lung) and is downloaded automatically in the cells below on first run — no manual download or placement into data/raw needed. The full dataset contains ~89,000 cells spanning four clearly distinguishable lung cell types (malignant cells, pulmonary alveolar type 2 cells, respiratory basal cells, alveolar macrophages). To keep this tutorial fast to run, we draw a balanced subsample of 2,000 cells per cell type below. Alternatively you can use your own single-cell dataset (Varix works with other datatypes, but in this example we expect single-cell data: h5ad).

In [3]:

Copied!





import numpy as np
import anndata as ad
import mudata

from huggingface_hub import hf_hub_download
from autoencodix.data.datapackage import DataPackage

sc_path = hf_hub_download(
    repo_id="autoencodix/census-lung", repo_type="dataset", filename="census_lung.h5ad"
)
full_adata = ad.read_h5ad(sc_path)

# Balanced subsample: 2,000 cells per cell type keeps the tutorial fast to run
# while keeping all four (well-separated) cell types equally represented.
N_PER_TYPE = 2000
rng = np.random.RandomState(42)
subsample_idx = []
for _, group in full_adata.obs.groupby("cell_type", observed=True):
    n = min(N_PER_TYPE, len(group))
    subsample_idx.extend(rng.choice(group.index.values, size=n, replace=False))

adata = full_adata[subsample_idx].copy()
print(adata.obs["cell_type"].value_counts())

# Instead of passing a pandas DataFrame, we wrap the AnnData in a MuData object
# inside our custom DataPackage structure, this time with single-cell data.
sc_data = DataPackage()
sc_data.multi_sc = {"multi_sc": mudata.MuData({"rna": adata})}
import numpy as np
import anndata as ad
import mudata

from huggingface_hub import hf_hub_download
from autoencodix.data.datapackage import DataPackage

sc_path = hf_hub_download(
    repo_id="autoencodix/census-lung", repo_type="dataset", filename="census_lung.h5ad"
)
full_adata = ad.read_h5ad(sc_path)

# Balanced subsample: 2,000 cells per cell type keeps the tutorial fast to run
# while keeping all four (well-separated) cell types equally represented.
N_PER_TYPE = 2000
rng = np.random.RandomState(42)
subsample_idx = []
for _, group in full_adata.obs.groupby("cell_type", observed=True):
    n = min(N_PER_TYPE, len(group))
    subsample_idx.extend(rng.choice(group.index.values, size=n, replace=False))

adata = full_adata[subsample_idx].copy()
print(adata.obs["cell_type"].value_counts())

# Instead of passing a pandas DataFrame, we wrap the AnnData in a MuData object
# inside our custom DataPackage structure, this time with single-cell data.
sc_data = DataPackage()
sc_data.multi_sc = {"multi_sc": mudata.MuData({"rna": adata})}

cell_type
alveolar macrophage               2000
malignant cell                    2000
pulmonary alveolar type 2 cell    2000
respiratory basal cell            2000
Name: count, dtype: int64

1) Initialize and Run Varix¶

We set a few custom parameters of the config file. For a deep dive into the config object, see:

Tutorials/DeepDives/ConfigTutorial.ipynb

1.1 The Dataset¶

We use the sc_data DataPackage prepared above (the census-lung dataset wrapped in an AnnData/MuData object) and color the downstream plots by the cell_type metadata column.

In [4]:

Copied!





from autoencodix.configs.varix_config import VarixConfig
from autoencodix.configs.default_config import DataCase, DataConfig, DataInfo
import autoencodix as acx

# Varix has its own config class
# with an additional loss term: either Kullback-Leibler (KL) or Maximum Mean Discrepancy (MMD)
my_config = VarixConfig(
    learning_rate=0.001,
    epochs=30,
    checkpoint_interval=5,
    default_vae_loss="kl",  # 'kl' or 'mmd' possible
    anneal_function="logistic-mid",
    k_filter=2000,
    data_case=DataCase.MULTI_SINGLE_CELL,
    annotation_columns=["cell_type"],
    data_config=DataConfig(
        data_info={"multi_sc": DataInfo(is_single_cell=True, data_type="NUMERIC")},
    ),
)

print("\nStarting Pipeline")
print("-" * 50)
varix = acx.Varix(data=sc_data, config=my_config)
result = varix.run()
from autoencodix.configs.varix_config import VarixConfig
from autoencodix.configs.default_config import DataCase, DataConfig, DataInfo
import autoencodix as acx

# Varix has its own config class
# with an additional loss term: either Kullback-Leibler (KL) or Maximum Mean Discrepancy (MMD)
my_config = VarixConfig(
    learning_rate=0.001,
    epochs=30,
    checkpoint_interval=5,
    default_vae_loss="kl",  # 'kl' or 'mmd' possible
    anneal_function="logistic-mid",
    k_filter=2000,
    data_case=DataCase.MULTI_SINGLE_CELL,
    annotation_columns=["cell_type"],
    data_config=DataConfig(
        data_info={"multi_sc": DataInfo(is_single_cell=True, data_type="NUMERIC")},
    ),
)

print("\nStarting Pipeline")
print("-" * 50)
varix = acx.Varix(data=sc_data, config=my_config)
result = varix.run()

Starting Pipeline
--------------------------------------------------
in handle_direct_user_data with data: <class 'autoencodix.data.datapackage.DataPackage'>
mudata: View of MuData object with n_obs × n_vars = 8000 × 8189
  var:	'soma_joinid', 'feature_id', 'feature_name', 'feature_type', 'feature_length', 'nnz', 'n_measured_obs'
  1 modality
    rna:	8000 x 8189
      obs:	'soma_joinid', 'dataset_id', 'assay', 'assay_ontology_term_id', 'cell_type', 'cell_type_ontology_term_id', 'development_stage', 'development_stage_ontology_term_id', 'disease', 'disease_ontology_term_id', 'donor_id', 'is_primary_data', 'observation_joinid', 'self_reported_ethnicity', 'self_reported_ethnicity_ontology_term_id', 'sex', 'sex_ontology_term_id', 'suspension_type', 'tissue', 'tissue_ontology_term_id', 'tissue_type', 'tissue_general', 'tissue_general_ontology_term_id', 'raw_sum', 'nnz', 'raw_mean_nnz', 'raw_variance_nnz', 'n_measured_vars'
      var:	'soma_joinid', 'feature_id', 'feature_name', 'feature_type', 'feature_length', 'nnz', 'n_measured_obs'
Processing 1 MuData objects: ['multi_sc']
Processing train modality: multi_sc
Processing valid split
Processing valid modality: multi_sc
Processing test split
Processing test modality: multi_sc

/home/alicia/dev/biomarker_autoencoder/autoencodix_package/src/autoencodix/trainers/_general_trainer.py:233: UserWarning: Gradient clipping was first applied in epoch 0. Total norm of gradients (adjusted for number of features): 13.2772 exceeded max norm of 5.
  warnings.warn(

Epoch 5 - Train Loss: 1772.1442
Sub-losses: recon_loss: 1772.1319, var_loss: 0.0123, anneal_factor: 0.0007, effective_beta_factor: 0.0001
Epoch 5 - Valid Loss: 1632.8667
Sub-losses: recon_loss: 1632.8576, var_loss: 0.0092, anneal_factor: 0.0007, effective_beta_factor: 0.0001
Epoch 10 - Train Loss: 1720.7331
Sub-losses: recon_loss: 1720.1487, var_loss: 0.5844, anneal_factor: 0.0180, effective_beta_factor: 0.0018
Epoch 10 - Valid Loss: 1593.5738
Sub-losses: recon_loss: 1593.0346, var_loss: 0.5393, anneal_factor: 0.0180, effective_beta_factor: 0.0018
Epoch 15 - Train Loss: 1710.1009
Sub-losses: recon_loss: 1701.1942, var_loss: 8.9067, anneal_factor: 0.3392, effective_beta_factor: 0.0339
Epoch 15 - Valid Loss: 1585.0462
Sub-losses: recon_loss: 1577.8735, var_loss: 7.1727, anneal_factor: 0.3392, effective_beta_factor: 0.0339
Epoch 20 - Train Loss: 1702.0190
Sub-losses: recon_loss: 1691.3949, var_loss: 10.6241, anneal_factor: 0.9350, effective_beta_factor: 0.0935
Epoch 20 - Valid Loss: 1583.2787
Sub-losses: recon_loss: 1574.7341, var_loss: 8.5445, anneal_factor: 0.9350, effective_beta_factor: 0.0935
Epoch 25 - Train Loss: 1692.5758
Sub-losses: recon_loss: 1682.4728, var_loss: 10.1030, anneal_factor: 0.9975, effective_beta_factor: 0.0998
Epoch 25 - Valid Loss: 1578.0161
Sub-losses: recon_loss: 1569.7830, var_loss: 8.2330, anneal_factor: 0.9975, effective_beta_factor: 0.0998
Epoch 30 - Train Loss: 1684.9201
Sub-losses: recon_loss: 1675.1868, var_loss: 9.7333, anneal_factor: 0.9999, effective_beta_factor: 0.1000
Epoch 30 - Valid Loss: 1573.3768
Sub-losses: recon_loss: 1565.8223, var_loss: 7.5545, anneal_factor: 0.9999, effective_beta_factor: 0.1000
Processed 1600 / 1600 samples

2) Specific Varix Steps (with Beta Annealing Explanation)¶

The pipeline (run) does not have Varix-specific steps.
It calls:

preprocess
fit
predict
visualize

However, the fit step works slightly differently because we have a second loss term for the distribution loss.
This loss term can be weighted with a hyperparameter beta (set via config), and the weighting can be changed during training depending on the epoch via beta annealing.

Beta annealing gradually increases the weight of the VAE distribution loss during training.
For example, you might start with beta=0 to let the reconstruction loss dominate early, and then gradually increase beta to 1 as training progresses.
Different annealing strategies are supported, including logistic schedules, multi-phase linear or logarithmic schedules, and constant or no annealing.
This allows more stable training and better latent space learning, especially for early epochs.

Since we're working with a variational autoencoder, we can call an additional pipeline step, sample_latent_space.
The latent space inside the pipeline is sampled once from the fitted normal distribution, with one set of mu and sigma per latent dimension.

By default, we sample from the trained model and the test data, but you can specify the split and epoch.

ATTENTION

If you want to sample from a different split (uses data from that split as input) or epoch (uses the model at this epoch), you need to ensure you select an epoch that was checkpointed (see checkpoint_interval config parameter).

In [5]:

Copied!





print(f"Annealing Strategy for training was: {my_config.anneal_function}")
print(f"beta hyperparam was set to: {my_config.beta}")
print("\n")
print("Starting sampling different latent spaces")
print("-"*50)
latent_spaces_sampled = varix.sample_latent_space(n_samples=100)
print(f"sampled latent space shape: {latent_spaces_sampled.shape}")

# or
sampled = varix.sample_latent_space(split="train", epoch=4,n_samples=50)
sampled
print(f"Annealing Strategy for training was: {my_config.anneal_function}")
print(f"beta hyperparam was set to: {my_config.beta}")
print("\n")
print("Starting sampling different latent spaces")
print("-"*50)
latent_spaces_sampled = varix.sample_latent_space(n_samples=100)
print(f"sampled latent space shape: {latent_spaces_sampled.shape}")

# or
sampled = varix.sample_latent_space(split="train", epoch=4,n_samples=50)
sampled

Annealing Strategy for training was: logistic-mid
beta hyperparam was set to: 0.1


Starting sampling different latent spaces
--------------------------------------------------
sampled latent space shape: torch.Size([100, 16])

Out[5]:

tensor([[ 3.9897,  4.1779,  1.8247,  2.3360,  1.1621,  6.5704,  4.2918,  0.8106,
          1.2946, -1.0189,  3.0731,  3.7557,  1.0169,  5.1850,  3.3394,  2.4349],
        [ 5.4976,  3.6496,  2.7947,  1.3314,  2.6984,  5.9357,  3.2230, -1.7498,
          0.4365,  0.9875, -0.4079,  5.1207,  3.0713,  3.0493,  3.3609,  2.9824],
        [ 4.2046,  2.2885,  4.5450,  0.0513,  1.7615,  7.2449,  4.5604,  0.9969,
          2.8329,  3.4808,  2.3766,  3.0640,  1.7307,  6.3812,  3.9349,  1.3993],
        [ 4.6709,  3.0498,  1.0624,  1.7734,  0.0657,  5.5635,  3.4210, -2.4210,
         -0.4838,  1.5256, -0.5709,  5.5177,  3.0401,  4.6961,  2.1426,  2.6823],
        [ 4.4480,  2.9986,  3.1832,  1.2259,  3.6005,  5.0852,  3.9627, -0.7497,
          1.2437,  2.3503,  0.5664,  6.6973,  3.5103,  4.7849,  2.5734,  1.6953],
        [ 3.1932,  3.2885,  1.4465,  1.5176,  0.4998,  4.5944,  5.1866,  1.7368,
          2.2024,  1.1406,  1.8937,  2.6345,  1.8833,  5.0871,  1.4922,  1.6122],
        [ 3.8947,  3.4285,  3.8185,  1.0312,  0.0963,  5.1008,  4.0787, -0.4031,
          1.1214,  1.1419, -0.0764,  3.0710,  2.7482,  3.0395,  1.8291,  1.7256],
        [ 4.8788,  3.4333,  3.9663, -1.0568,  0.6186,  3.5102,  4.3233, -0.1575,
          1.1605,  1.1413,  1.7532,  2.3862,  4.2504,  4.5982,  3.1169,  0.7457],
        [ 4.8506,  3.2920,  2.4493,  2.7486,  1.2036,  5.7954,  3.6908, -1.0056,
          3.0519,  2.2471,  1.5073,  3.3352,  2.4925,  4.7927,  2.7642,  3.3198],
        [ 2.6878,  4.2589,  3.0489,  2.2954,  0.6193,  3.6618,  1.8470, -0.1019,
          1.0595,  0.8465,  2.1162,  5.1094,  2.1355,  4.2325,  3.4177,  2.4656],
        [ 2.8378,  2.7538,  3.1237,  2.0716,  2.0786,  7.0835,  3.1613, -1.1583,
          0.3080, -0.0282,  0.8765,  4.6263,  3.1052,  4.9754,  4.2008,  0.8487],
        [ 2.4476,  2.7313,  1.2384,  0.6757,  2.9385,  5.7029,  2.3004,  0.9140,
         -0.5391,  3.2165,  0.5776,  3.5659,  3.1089,  3.9679,  4.8885,  4.7642],
        [ 4.2116,  1.6357,  1.6338,  0.4075,  1.6522,  4.8240,  1.9459, -0.4264,
          1.4115,  2.0392,  2.5556,  3.0936,  3.6552,  4.9145,  2.5359,  2.8324],
        [ 3.2254,  2.5174,  2.0800,  2.1039,  0.8280,  5.1618,  2.9418,  1.9556,
          1.0148,  1.8160,  1.7916,  4.7589,  1.2774,  4.1057,  2.6448,  3.8139],
        [ 1.5846,  2.3309,  4.4815,  0.3691,  2.7857,  5.8614,  4.3167, -0.0357,
          2.3315, -0.0218,  2.1312,  5.6027,  3.4802,  4.0986,  2.0490,  1.9036],
        [ 4.1307,  2.0066,  1.2703,  1.2824,  1.6188,  3.6514,  5.3790, -0.1230,
         -0.0797, -0.5275,  0.7606,  3.4950,  2.3617,  4.2383,  4.2128,  3.7861],
        [ 4.3599,  4.1792,  1.1445,  1.8330,  0.5930,  4.1228,  2.5574, -0.6411,
          2.7593,  0.1467,  0.2999,  3.9019,  1.3348,  1.9823,  0.9904,  5.4124],
        [ 6.2960,  0.9289,  2.8897,  1.3115,  0.9947,  5.2014,  4.0889, -0.4651,
          1.0783, -0.8659,  1.6770,  2.7691,  3.4997,  3.6158,  3.7402,  1.8536],
        [ 3.7664,  3.9914,  1.2181,  0.0515,  0.9840,  6.2559,  5.3040,  0.1689,
          2.3972,  1.9133,  0.1996,  5.1297,  2.0039,  5.3169,  2.6769,  1.4529],
        [ 3.8404,  2.8456,  2.4552,  2.3359,  1.6488,  5.3061,  4.1186, -0.9893,
          2.1649,  2.5745,  2.5995,  2.2949,  2.3875,  3.1978,  1.4837,  1.9929],
        [ 4.2926,  3.3016,  2.2517, -0.6866,  1.0894,  4.3353,  3.2855,  0.4257,
          2.4056,  1.4264,  2.3332,  4.8085,  2.1467,  3.7170,  1.6814,  2.1618],
        [ 4.2677,  4.7061,  1.9314,  1.6386, -0.6365,  6.0619,  3.8623, -1.0573,
          0.7680,  0.3162,  2.5458,  4.6985,  2.8703,  5.0664,  2.2438,  0.8451],
        [ 3.3825,  3.1412,  2.3874,  1.4558,  2.8128,  4.0695,  3.5587,  0.9427,
          1.3877,  1.0520,  1.3150,  3.9352,  3.0074,  5.0639,  4.4199,  1.5905],
        [ 3.2504,  4.1532,  1.5998,  1.2945, -0.5937,  4.6828,  3.7978, -1.6958,
          2.5848,  1.3192,  2.3662,  4.5234,  1.9827,  5.7141,  1.9336,  2.3728],
        [ 2.0694,  1.5284,  1.9963,  1.1892,  2.9973,  5.0614,  3.2551, -1.5891,
          1.1728,  1.1425,  0.3544,  3.2171,  1.4586,  5.2386,  4.1895,  2.3967],
        [ 4.4011,  3.4322,  2.6520,  2.0830,  0.8431,  5.6731,  2.9842,  1.8272,
          1.3224,  0.4511,  2.0769,  3.4533,  2.0077,  3.2562,  1.8612,  2.8596],
        [ 3.8049,  5.2333,  3.4705,  1.4310,  0.1047,  4.7920,  1.7061, -0.0416,
          1.7465, -0.0357,  1.2870,  4.8265,  2.5176,  5.2567, -0.1049,  2.0154],
        [ 4.4466,  3.7065,  1.6124,  2.2614,  1.8870,  4.4186,  4.0371,  2.2588,
          1.9454,  0.9202,  0.9742,  4.1197,  2.2989,  4.1240,  3.5764,  0.5222],
        [ 2.6849,  2.9684,  0.2723,  0.2081, -0.0146,  4.4049,  3.4078,  2.5937,
          1.8721,  1.3141,  0.7986,  3.2091,  4.1286,  3.8560,  1.8486,  2.7847],
        [ 4.8564,  2.7846,  1.4665, -0.6019,  1.6682,  4.1358,  3.6395, -2.0272,
          2.9377,  1.0671,  1.6877,  3.5498,  1.7075,  3.5182,  4.3156,  2.3967],
        [ 2.4480,  1.8565,  2.2424,  2.6267,  1.8144,  4.3647,  4.2695, -0.7121,
          1.3851, -0.0531,  2.4627,  4.6171,  0.9041,  5.1812,  2.4905,  2.9776],
        [ 4.4109,  4.2225,  4.7380,  0.4170,  2.7323,  4.7937,  2.4964,  0.3149,
          1.3168,  1.5230,  0.2027,  1.2152,  1.2538,  4.0088,  2.0650,  3.5317],
        [ 3.9265,  3.2160,  3.1083,  0.6176,  1.1427,  5.5795,  4.2378,  0.8136,
          2.1128,  2.1580, -0.2427,  3.3830,  2.8504,  2.4779,  4.1177,  3.5762],
        [ 2.1202,  3.4093,  2.2411,  1.3045,  1.0675,  4.6012,  3.9094, -1.4338,
         -0.0822,  0.6938,  2.5667,  3.6814,  1.7400,  5.0690,  1.7663,  2.9761],
        [ 3.3126,  2.4914,  3.5674,  0.4738,  1.9337,  4.3273,  3.9449,  1.2722,
          0.9200,  0.6026,  2.0357,  3.3515,  1.3852,  4.8255,  4.0067,  2.3287],
        [ 4.3111,  3.1039,  1.7090,  0.5363,  1.8466,  5.5181,  3.0293, -0.4970,
          1.0195,  0.9938, -0.1135,  2.4220,  1.1604,  4.6212,  4.3118,  2.9605],
        [ 3.2749,  3.5971,  2.1631,  0.8320,  2.4171,  4.0603,  3.8554, -1.7744,
          1.9234,  1.1360,  0.8604,  4.4355,  1.0436,  3.1726,  3.0614,  4.8307],
        [ 1.8397,  2.4236,  2.5782,  0.9802,  1.2471,  7.2168,  3.3061,  0.2531,
          0.6532,  0.6556,  1.8536,  3.8381,  1.8564,  4.2432,  1.6487,  3.4495],
        [ 3.6000,  2.0780,  1.5972,  0.9568,  1.4195,  2.9402,  2.6527, -0.5432,
          1.7087,  1.0849,  1.2738,  4.3350,  2.3058,  4.5502,  2.7937,  2.4609],
        [ 4.0757,  3.2504,  3.7570,  0.7621,  1.0736,  5.6269,  2.5974,  2.5999,
          1.7650,  1.4346,  2.4470,  4.3854,  3.7922,  3.8308,  1.9880,  1.2160],
        [ 3.3738,  1.9633,  2.5027, -1.0388,  1.9913,  6.1064,  2.8293, -2.9994,
          1.6348,  0.4685,  2.1090,  2.5562,  2.8689,  7.1168,  2.1063,  2.5277],
        [ 4.4658,  1.9552,  3.9799,  1.3656,  3.2007,  3.2079,  2.5630, -0.1730,
          1.6410,  0.7156,  2.2678,  4.2639,  3.0226,  6.1909,  5.4368,  2.4841],
        [ 4.0822,  4.3164,  2.2534,  3.3088,  0.8083,  3.8113,  5.0359, -1.2380,
          1.6543, -0.4035,  2.1328,  4.4057,  3.0668,  4.2439,  2.5502,  1.5507],
        [ 4.8267,  2.9794,  2.3273,  1.3404,  1.3077,  3.4276,  4.1624, -0.7620,
          2.9742,  1.0375,  1.1422,  4.1053,  2.8869,  4.3778,  0.8896,  0.4536],
        [ 3.9119,  2.4069,  1.8083,  1.2932,  0.3658,  5.7046,  5.2096,  2.2183,
          2.1208, -0.5397,  3.0277,  3.8026,  2.8541,  5.7916,  1.7351,  2.4798],
        [ 3.9301,  2.4724,  3.2285,  0.4204,  1.6635,  6.9525,  4.0364,  1.0757,
          0.4539,  1.1663,  2.5490,  3.8018, -0.0837,  5.6462,  2.7957,  2.7732],
        [ 3.2946,  3.8439,  3.2163,  0.5983,  0.5594,  4.7550,  3.5663,  0.7506,
          0.7128,  2.4360,  1.7824,  3.2272,  4.2420,  4.5259,  1.9242,  1.7268],
        [ 2.7769,  1.3834,  3.5355,  1.5109,  0.6312,  5.9893,  2.7953, -0.8159,
          0.3054, -0.3478,  5.0515,  3.9981,  1.5742,  3.7268,  4.3139,  1.5894],
        [ 3.0773,  1.7113, -0.0818,  0.7988,  2.1839,  4.3691,  3.7347, -0.8462,
          1.6362,  1.1404,  1.7167,  3.8530,  1.4562,  3.4271,  2.7185,  0.7911],
        [ 4.6082,  0.8077,  0.7850,  2.0872,  1.9254,  3.9792,  4.5952, -0.1775,
          0.8533,  0.6686,  0.6368,  2.9203,  3.1537,  4.5880,  1.4741,  2.4437]])

3) Inspect Varix-Specific Results¶

In addition to the results that the Vanillix pipeline provided, we can access:

Fitted distribution parameters mu and sigma
total, reconstruction, and vae losses
anneal_factor

A note on the different loss types:
For our variational autoencoder, the total loss consists of a reconstruction loss and a distribution loss (i.e., KL-divergence).
To investigate these losses, the result object has the attribute sub_losses.
This is a LossRegistry with the name of the loss as the key, and the value is a TrainingDynamics object, which can be accessed in the same way as for the Vanillix results.

For more details, check Tutorials/DeepDives/PipelineOutputTutorial.ipynb.

In [6]:

Copied!





sub_losses = result.sub_losses
print("Sub Losses:")
print(f"keys: {sub_losses.keys()}")
print("\n")
recon_dyn = sub_losses.get(key="recon_loss")
print("Value of reconstruction loss in epoch 4 for train split")
print(recon_dyn.get(split="train", epoch=4))
sub_losses = result.sub_losses
print("Sub Losses:")
print(f"keys: {sub_losses.keys()}")
print("\n")
recon_dyn = sub_losses.get(key="recon_loss")
print("Value of reconstruction loss in epoch 4 for train split")
print(recon_dyn.get(split="train", epoch=4))

Sub Losses:
keys: dict_keys(['recon_loss', 'var_loss', 'anneal_factor', 'effective_beta_factor'])


Value of reconstruction loss in epoch 4 for train split
1772.131879185268

4) Show Visualizations¶

This follows the standard pipeline process and can be done by calling show_result().
We can also add the keyword argument params to show_result(), which colors the plots according to a metadata column.

In [7]:

Copied!

varix.result.datasets.test.metadata.head()

varix.show_result(params=["cell_type"])
varix.result.datasets.test.metadata.head()

varix.show_result(params=["cell_type"])

Creating plots ...

No description has been provided for this image

In [8]:

Copied!

varix.visualizer.show_loss(plot_type="relative")
varix.visualizer.show_loss(plot_type="relative")

In [9]:

Copied!

# A quick look at the real metadata available for coloring plots:
adata.obs[["cell_type", "disease", "sex"]].value_counts()
# A quick look at the real metadata available for coloring plots:
adata.obs[["cell_type", "disease", "sex"]].value_counts()

Out[9]:

cell_type                       disease              sex    
malignant cell                  lung adenocarcinoma  male       1196
alveolar macrophage             normal               male       1012
respiratory basal cell          normal               unknown     894
pulmonary alveolar type 2 cell  normal               male        792
                                                     female      765
malignant cell                  lung adenocarcinoma  female      730
respiratory basal cell          cystic fibrosis      unknown     542
alveolar macrophage             normal               female      539
                                                     unknown     449
pulmonary alveolar type 2 cell  normal               unknown     443
respiratory basal cell          normal               male        381
                                                     female      183
malignant cell                  lung adenocarcinoma  unknown      74
Name: count, dtype: int64

5) Customize Varix¶

To customize the behavior of our pipeline, you adjust the configuration.
There are two ways to work with the config:

Create a customized instance of the config class.
Provide a yaml file and use the config class to read it.

We will focus on option 1 and show a few examples. For a deeper dive into configurations, please refer to Tutorials/DeepDives/ConfigTutorial.ipynb

In this section, we demonstrate some Varix-specific parameters:

Loss term
Beta
Annealing strategy
Retrieve information about all config parameters

In [10]:

Copied!





from autoencodix.configs import VarixConfig

custom_config = VarixConfig(
    default_vae_loss="mmd",
    beta=1.5,
    anneal_function="3phase-linear",
    data_case=DataCase.MULTI_SINGLE_CELL,
    epochs=50,
    k_filter=2000,
    annotation_columns=["cell_type"],
    data_config=DataConfig(
        data_info={"multi_sc": DataInfo(is_single_cell=True, data_type="NUMERIC")},
    ),
)

custom_varix = acx.Varix(config=custom_config, data=sc_data)
result = custom_varix.run()
from autoencodix.configs import VarixConfig

custom_config = VarixConfig(
    default_vae_loss="mmd",
    beta=1.5,
    anneal_function="3phase-linear",
    data_case=DataCase.MULTI_SINGLE_CELL,
    epochs=50,
    k_filter=2000,
    annotation_columns=["cell_type"],
    data_config=DataConfig(
        data_info={"multi_sc": DataInfo(is_single_cell=True, data_type="NUMERIC")},
    ),
)

custom_varix = acx.Varix(config=custom_config, data=sc_data)
result = custom_varix.run()

in handle_direct_user_data with data: <class 'autoencodix.data.datapackage.DataPackage'>
mudata: View of MuData object with n_obs × n_vars = 8000 × 8189
  var:	'soma_joinid', 'feature_id', 'feature_name', 'feature_type', 'feature_length', 'nnz', 'n_measured_obs'
  1 modality
    rna:	8000 x 8189
      obs:	'soma_joinid', 'dataset_id', 'assay', 'assay_ontology_term_id', 'cell_type', 'cell_type_ontology_term_id', 'development_stage', 'development_stage_ontology_term_id', 'disease', 'disease_ontology_term_id', 'donor_id', 'is_primary_data', 'observation_joinid', 'self_reported_ethnicity', 'self_reported_ethnicity_ontology_term_id', 'sex', 'sex_ontology_term_id', 'suspension_type', 'tissue', 'tissue_ontology_term_id', 'tissue_type', 'tissue_general', 'tissue_general_ontology_term_id', 'raw_sum', 'nnz', 'raw_mean_nnz', 'raw_variance_nnz', 'n_measured_vars', 'n_genes'
      var:	'soma_joinid', 'feature_id', 'feature_name', 'feature_type', 'feature_length', 'nnz', 'n_measured_obs'
Processing 1 MuData objects: ['multi_sc']
Processing train modality: multi_sc
Processing valid split
Processing valid modality: multi_sc
Processing test split
Processing test modality: multi_sc

/home/alicia/dev/biomarker_autoencoder/autoencodix_package/src/autoencodix/trainers/_general_trainer.py:233: UserWarning: Gradient clipping was first applied in epoch 0. Total norm of gradients (adjusted for number of features): 11.8769 exceeded max norm of 5.
  warnings.warn(

Epoch 10 - Train Loss: 1716.3193
Sub-losses: recon_loss: 1716.3193, var_loss: 0.0000, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 10 - Valid Loss: 1702.9074
Sub-losses: recon_loss: 1702.9074, var_loss: 0.0000, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 20 - Train Loss: 1682.3635
Sub-losses: recon_loss: 1677.4623, var_loss: 4.9012, anneal_factor: 0.1400, effective_beta_factor: 0.2100
Epoch 20 - Valid Loss: 1681.6059
Sub-losses: recon_loss: 1676.8131, var_loss: 4.7928, anneal_factor: 0.1400, effective_beta_factor: 0.2100
Epoch 30 - Train Loss: 1678.1424
Sub-losses: recon_loss: 1661.8913, var_loss: 16.2512, anneal_factor: 0.7400, effective_beta_factor: 1.1100
Epoch 30 - Valid Loss: 1680.7923
Sub-losses: recon_loss: 1666.1517, var_loss: 14.6406, anneal_factor: 0.7400, effective_beta_factor: 1.1100
Epoch 40 - Train Loss: 1672.8351
Sub-losses: recon_loss: 1655.7218, var_loss: 17.1133, anneal_factor: 1.0000, effective_beta_factor: 1.5000
Epoch 40 - Valid Loss: 1674.4009
Sub-losses: recon_loss: 1659.7907, var_loss: 14.6102, anneal_factor: 1.0000, effective_beta_factor: 1.5000
Epoch 50 - Train Loss: 1661.5501
Sub-losses: recon_loss: 1644.9416, var_loss: 16.6085, anneal_factor: 1.0000, effective_beta_factor: 1.5000
Epoch 50 - Valid Loss: 1669.5920
Sub-losses: recon_loss: 1656.0150, var_loss: 13.5769, anneal_factor: 1.0000, effective_beta_factor: 1.5000
Processed 1600 / 1600 samples

In [11]:

Copied!

custom_varix.visualizer.show_loss(plot_type="absolute")
custom_varix.visualizer.show_loss(plot_type="absolute")

In [12]:

Copied!

custom_varix.visualizer.show_loss(plot_type="relative")
custom_varix.visualizer.show_loss(plot_type="relative")

6) Re-use, Save, Load¶

There are not Varix specific steps here. See the Getting Started - Vanillix for details. Below is a basic save/load usecase:

In [13]:

Copied!





import os
import glob
# use a filename without extension, we handle this internally
outpath = os.path.join("tutorial_res", "varix")
varix.save(file_path=outpath, save_all=False)

folder = os.path.dirname(outpath)
pkl_files = glob.glob(os.path.join(folder, "*.pkl"))
model_files = glob.glob(os.path.join(folder, "*.pth"))

print("PKL files:", pkl_files)
print("Model files:", model_files)

# the load functionality automatically will build the pipeline object out of the three saved files
varix_loaded = acx.Varix.load(outpath)
varix_loaded.predict(data=sc_data)
varix_loaded.visualize()
varix_loaded.show_result(params=["cell_type"])
import os
import glob
# use a filename without extension, we handle this internally
outpath = os.path.join("tutorial_res", "varix")
varix.save(file_path=outpath, save_all=False)

folder = os.path.dirname(outpath)
pkl_files = glob.glob(os.path.join(folder, "*.pkl"))
model_files = glob.glob(os.path.join(folder, "*.pth"))

print("PKL files:", pkl_files)
print("Model files:", model_files)

# the load functionality automatically will build the pipeline object out of the three saved files
varix_loaded = acx.Varix.load(outpath)
varix_loaded.predict(data=sc_data)
varix_loaded.visualize()
varix_loaded.show_result(params=["cell_type"])

Preprocessor saved successfully.
saving memory efficient
Pipeline object saved successfully.
PKL files: ['tutorial_res/varix_preprocessor.pkl', 'tutorial_res/ontix_reactome_preprocessor.pkl', 'tutorial_res/ontix.pkl', 'tutorial_res/maskix_preprocessor.pkl', 'tutorial_res/imagix.pkl', 'tutorial_res/ontix_preprocessor.pkl', 'tutorial_res/disent.pkl', 'tutorial_res/imagix_preprocessor.pkl', 'tutorial_res/ontix_reactome.pkl', 'tutorial_res/disent_preprocessor.pkl']
Model files: ['tutorial_res/varix_model.pth', 'tutorial_res/ontix_reactome_model.pth', 'tutorial_res/disent_model.pth', 'tutorial_res/imagix_model.pth', 'tutorial_res/ontix_model.pth', 'tutorial_res/maskix_model.pth']
Attempting to load a pipeline from tutorial_res/varix...
Pipeline object loaded successfully. Actual type: Varix
Preprocessor loaded successfully.
in handle_direct_user_data with data: <class 'autoencodix.data.datapackage.DataPackage'>
mudata: View of MuData object with n_obs × n_vars = 8000 × 8189
  var:	'soma_joinid', 'feature_id', 'feature_name', 'feature_type', 'feature_length', 'nnz', 'n_measured_obs'
  1 modality
    rna:	8000 x 8189
      obs:	'soma_joinid', 'dataset_id', 'assay', 'assay_ontology_term_id', 'cell_type', 'cell_type_ontology_term_id', 'development_stage', 'development_stage_ontology_term_id', 'disease', 'disease_ontology_term_id', 'donor_id', 'is_primary_data', 'observation_joinid', 'self_reported_ethnicity', 'self_reported_ethnicity_ontology_term_id', 'sex', 'sex_ontology_term_id', 'suspension_type', 'tissue', 'tissue_ontology_term_id', 'tissue_type', 'tissue_general', 'tissue_general_ontology_term_id', 'raw_sum', 'nnz', 'raw_mean_nnz', 'raw_variance_nnz', 'n_measured_vars', 'n_genes'
      var:	'soma_joinid', 'feature_id', 'feature_name', 'feature_type', 'feature_length', 'nnz', 'n_measured_obs'
Processing 1 MuData objects: ['multi_sc']
Processing test split
Processing test modality: multi_sc
n_samples in format recon: 8000
train
n_samples from datatpackge: {'paired_count': 8000}
Creating plots ...

/home/alicia/dev/biomarker_autoencoder/autoencodix_package/src/autoencodix/utils/_result.py:348: UserWarning: Could not retrieve latent representations for epoch 29 and split 'train'. Returning empty DataFrame. This may be due to missing data in the Result object or incorrect keys.


  warnings.warn(
/home/alicia/dev/biomarker_autoencoder/autoencodix_package/src/autoencodix/utils/_result.py:348: UserWarning: Could not retrieve latent representations for epoch 29 and split 'valid'. Returning empty DataFrame. This may be due to missing data in the Result object or incorrect keys.


  warnings.warn(

/home/alicia/dev/biomarker_autoencoder/autoencodix_package/src/autoencodix/utils/_result.py:348: UserWarning: Could not retrieve latent representations for epoch 29 and split 'train'. Returning empty DataFrame. This may be due to missing data in the Result object or incorrect keys.


  warnings.warn(
/home/alicia/dev/biomarker_autoencoder/autoencodix_package/src/autoencodix/utils/_result.py:348: UserWarning: Could not retrieve latent representations for epoch 29 and split 'valid'. Returning empty DataFrame. This may be due to missing data in the Result object or incorrect keys.


  warnings.warn(

In [14]:

Copied!





varix_loaded.evaluate(
	params=["cell_type"],
	split_type="CV-5",
	)
varix_loaded.evaluate(
	params=["cell_type"],
	split_type="CV-5",
	)

Perform ML task with feature df: Latent
Latent
Perform ML task for target parameter: cell_type

/home/alicia/dev/biomarker_autoencoder/autoencodix_package/src/autoencodix/utils/_result.py:348: UserWarning: Could not retrieve latent representations for epoch 29 and split 'train'. Returning empty DataFrame. This may be due to missing data in the Result object or incorrect keys.


  warnings.warn(
/home/alicia/dev/biomarker_autoencoder/autoencodix_package/src/autoencodix/utils/_result.py:348: UserWarning: Could not retrieve latent representations for epoch 29 and split 'valid'. Returning empty DataFrame. This may be due to missing data in the Result object or incorrect keys.


  warnings.warn(
/home/alicia/dev/biomarker_autoencoder/autoencodix_package/.venv/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:465: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. OF ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(
/home/alicia/dev/biomarker_autoencoder/autoencodix_package/.venv/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:465: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. OF ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(
/home/alicia/dev/biomarker_autoencoder/autoencodix_package/.venv/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:465: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. OF ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(
/home/alicia/dev/biomarker_autoencoder/autoencodix_package/.venv/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:465: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. OF ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(
/home/alicia/dev/biomarker_autoencoder/autoencodix_package/.venv/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:465: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. OF ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(

Out[14]:

Result Object Public Attributes:
------------------------------
latentspaces: TrainingDynamics object
sample_ids: TrainingDynamics object
reconstructions: TrainingDynamics object
mus: TrainingDynamics object
sigmas: TrainingDynamics object
losses: TrainingDynamics object
sub_losses: LossRegistry(_losses={'recon_loss': TrainingDynamics(), 'var_loss': TrainingDynamics(), 'anneal_factor': TrainingDynamics(), 'effective_beta_factor': TrainingDynamics()})
preprocessed_data: Tensor of shape (0,)
model: VarixArchitecture
model_checkpoints: TrainingDynamics object
datasets: DatasetContainer(train=None, valid=None, test=None)
new_datasets: DatasetContainer(train=<autoencodix.data._numeric_dataset.NumericDataset object at 0x725f6510a5c0>, valid=None, test=<autoencodix.data._numeric_dataset.NumericDataset object at 0x725f944f0850>)
adata_latent: AnnData object with n_obs × n_vars = 8000 × 16
    uns: 'var_names'
final_reconstruction: MuData object with n_obs × n_vars = 8000 × 2000
  1 modality
    rna:	8000 x 2000
      obs:	'soma_joinid', 'dataset_id', 'assay', 'assay_ontology_term_id', 'cell_type', 'cell_type_ontology_term_id', 'development_stage', 'development_stage_ontology_term_id', 'disease', 'disease_ontology_term_id', 'donor_id', 'is_primary_data', 'observation_joinid', 'self_reported_ethnicity', 'self_reported_ethnicity_ontology_term_id', 'sex', 'sex_ontology_term_id', 'suspension_type', 'tissue', 'tissue_ontology_term_id', 'tissue_type', 'tissue_general', 'tissue_general_ontology_term_id', 'raw_sum', 'nnz', 'raw_mean_nnz', 'raw_variance_nnz', 'n_measured_vars', 'n_genes'
sub_results: None
sub_reconstructions: None
embedding_evaluation:   cv_run score_split CLINIC_PARAM       metric     value              ML_ALG  \
0   CV_1        test    cell_type  roc_auc_ovo  0.991365  LogisticRegression   
1   CV_2        test    cell_type  roc_auc_ovo  0.991712  LogisticRegression   
2   CV_3        test    cell_type  roc_auc_ovo  0.992083  LogisticRegression   
3   CV_4        test    cell_type  roc_auc_ovo  0.991306  LogisticRegression   
4   CV_5        test    cell_type  roc_auc_ovo  0.992496  LogisticRegression   
5   CV_1       train    cell_type  roc_auc_ovo  0.992446  LogisticRegression   
6   CV_2       train    cell_type  roc_auc_ovo  0.992335  LogisticRegression   
7   CV_3       train    cell_type  roc_auc_ovo  0.992311  LogisticRegression   
8   CV_4       train    cell_type  roc_auc_ovo  0.992504  LogisticRegression   
9   CV_5       train    cell_type  roc_auc_ovo  0.992172  LogisticRegression   

          ML_TYPE ML_TASK ML_SUBTASK  
0  classification  Latent     Latent  
1  classification  Latent     Latent  
2  classification  Latent     Latent  
3  classification  Latent     Latent  
4  classification  Latent     Latent  
5  classification  Latent     Latent  
6  classification  Latent     Latent  
7  classification  Latent     Latent  
8  classification  Latent     Latent  
9  classification  Latent     Latent  
embedding_attributions: Empty DataFrame
Columns: []
Index: []
embedding_explanations: Dict with 0 items

In [15]:

Copied!





varix_loaded.visualizer.show_evaluation(
	param="cell_type",
	metric="roc_auc_ovo",
)
varix_loaded.visualizer.show_evaluation(
	param="cell_type",
	metric="roc_auc_ovo",
)

Showing plot for ML algorithm: LogisticRegression