Note: This environment variable is required for fully deterministic CuBLAS ops on CUDA >= 10.2 when reproducible=True is set below. Without it, PyTorch raises a RuntimeError instead of training deterministically. It must be set before torch is imported. See the README FAQ for details.

In [ ]:

%env CUBLAS_WORKSPACE_CONFIG=:16:8

%env CUBLAS_WORKSPACE_CONFIG=:16:8

How to Prepare Your Data for AUTOENCODIX¶

In this tutorial we'll show how AUTOENCODIX deals with different types of data and the different ways to pass data into the pipeline.

IMPORTANT

This tutorial mainly explains the types of data we allow and how to format the data so that AUTOENCODIX can work with it. If you're unfamiliar with general concepts,
we recommend following the Getting Started - Vanillix Tutorial first.

What You'll Learn¶

We'll cover:

• Our general data logic
• How to pass data from files for different data modalities
• How to pass data directly into our pipeline
• Differences between paired and unpaired data
• How our preprocessing works and how to skip it

Data Model Theory¶

There are two main ways to pass data to AUTOENCODIX and we support three main data modalities. For these main modalities, we offer seven main use cases. Besides that, you can choose whether we should preprocess the data or you want to start with training the models directly.

The two main ways to pass data are:

• Providing a file path in the config
• Passing the data directly to our pipeline

The three main data modalities are:

• Tabular numeric data
• Sparse numeric data (Single Cell)
• Image data

The seven main use cases are:

• Combining multi-omics data from single cell sequencing
• Combining multi-omics data from bulk sequencing (e.g., mRNA and methylation)
• "Translating" between bulk sequencing data
• "Translating" bulk and image data, and vice versa
• "Translating" between single cell data
• Training an image autoencoder

We encode these cases in our DataCase class, and you need to specify in the config which DataCase you have. This is shown in the code in the next section.

The general way to feed data to our pipeline is independent of the specific model (Varix, XModalix, etc.). There might be minor pipeline-specific requirements; for example, for Ontix we won't cover these here—please refer to the tutorials of the specific pipeline. Here we use XModalix and Varix as examples.

1) Passing Data from File¶

Depending on which pipeline you use, you will be interested in one of our seven main use cases as described above.

Regardless of the use case, the principle of passing data is always the same; only your file types will differ, and you need to specify the use case in the config via the DataCase:

from enum import Enum

class DataCase(str, Enum):
    MULTI_SINGLE_CELL = "Multi Single Cell"
    MULTI_BULK = "Multi Bulk"
    BULK_TO_BULK = "Bulk<->Bulk"
    IMG_TO_BULK = "IMG<->Bulk"
    SINGLE_CELL_TO_SINGLE_CELL = "Single Cell<->Single Cell"
    SINGLE_CELL_TO_IMG = "Single Cell<->IMG"
    IMG_TO_IMG = "IMG<->IMG"  # standard image autoencoder

1.1 Combining Multi-Omics Data from Bulk-Sequencing¶

First, we need to prepare our config object. We can either (a) directly provide a Python object, or (b) provide a YAML file. Here, we show option (a). For option (b), refer to [1].

ATTENTION:
If you use .txt or .csv files, it is best practice to add the sep parameter. If none is given, the reader will try to auto-detect the separator, which is error-prone. This would look like:

IMPORTANT:

For all your bulk data files, we expect the first column to be some kind of unique sample ID. Please prepare the data accordingly.

[1] Tutorials/DeepDives/ConfigTutorial.ipynb

First make sure we are in the root of our package:

In [1]:

import os

p = os.getcwd()
d = "autoencodix_package"
if d not in p:
    raise FileNotFoundError(f"'{d}' not found in path: {p}")
os.chdir(os.sep.join(p.split(os.sep)[: p.split(os.sep).index(d) + 1]))
print(f"Changed to: {os.getcwd()}")

import os p = os.getcwd() d = "autoencodix_package" if d not in p: raise FileNotFoundError(f"'{d}' not found in path: {p}") os.chdir(os.sep.join(p.split(os.sep)[: p.split(os.sep).index(d) + 1])) print(f"Changed to: {os.getcwd()}")

Changed to: /Users/maximilianjoas/development/autoencodix_package

Now we can prepare the config:

Besides the actual data files, we expect one annotation file with metadata. We add the parameter data_type and set it to ANNOTATION, as shown in the code below.
If you have columns inside the ANNOTATION data that you want to use later in downstream visualization or evaluation tasks, please specify them with the annotation_columns parameter of the DataConfig. This will prevent errors later, as NaNs will be handled automatically during preprocessing for these columns.

❗❗ Requirements: Getting Tutorial Data ❗❗¶

The data for this example is hosted on Hugging Face Hub (autoencodix/tcga) and is downloaded automatically in the cell below.

In [ ]:

import autoencodix as acx
from autoencodix.configs.default_config import DataConfig, DataInfo, DataCase
from autoencodix.configs import VarixConfig
from huggingface_hub import hf_hub_download

# Data is hosted on Hugging Face Hub and downloaded (and locally cached) automatically
HF_REPO_ID = "autoencodix/tcga"
mrna_file = hf_hub_download(repo_id=HF_REPO_ID, repo_type="dataset", filename="rna.parquet")
meth_file = hf_hub_download(repo_id=HF_REPO_ID, repo_type="dataset", filename="methylation.parquet")
clin_file = hf_hub_download(repo_id=HF_REPO_ID, repo_type="dataset", filename="clinical.parquet")

"""
Our config class has the attribute `data_config`.
Here, we fill the `data_info` Dict, by defining names for our data modalities ("RNA" for example).
The value of this Dict is a instance of the `DataInfo` class. All we need to pass here is a `file_path` argument.
"""
data_config = DataConfig(
    annotation_columns=["CANCER_TYPE"],
    data_info={
        "RNA": DataInfo(file_path=mrna_file),
        "METHYLATION": DataInfo(file_path=meth_file),
        "CLINICAL": DataInfo(
            file_path=clin_file, data_type="ANNOTATION"
        ),
    },
)

"""
Lastly, we pass our data_config to the VarixConfig. Remember to pass the appropriate `DataCase`.
"""
bulk_config = VarixConfig(data_config=data_config, data_case=DataCase.MULTI_BULK, epochs=30)
varix = acx.Varix(config=bulk_config)

result_varix = varix.run()
varix.show_result()

import autoencodix as acx from autoencodix.configs.default_config import DataConfig, DataInfo, DataCase from autoencodix.configs import VarixConfig from huggingface_hub import hf_hub_download # Data is hosted on Hugging Face Hub and downloaded (and locally cached) automatically HF_REPO_ID = "autoencodix/tcga" mrna_file = hf_hub_download(repo_id=HF_REPO_ID, repo_type="dataset", filename="rna.parquet") meth_file = hf_hub_download(repo_id=HF_REPO_ID, repo_type="dataset", filename="methylation.parquet") clin_file = hf_hub_download(repo_id=HF_REPO_ID, repo_type="dataset", filename="clinical.parquet") """ Our config class has the attribute `data_config`. Here, we fill the `data_info` Dict, by defining names for our data modalities ("RNA" for example). The value of this Dict is a instance of the `DataInfo` class. All we need to pass here is a `file_path` argument. """ data_config = DataConfig( annotation_columns=["CANCER_TYPE"], data_info={ "RNA": DataInfo(file_path=mrna_file), "METHYLATION": DataInfo(file_path=meth_file), "CLINICAL": DataInfo( file_path=clin_file, data_type="ANNOTATION" ), }, ) """ Lastly, we pass our data_config to the VarixConfig. Remember to pass the appropriate `DataCase`. """ bulk_config = VarixConfig(data_config=data_config, data_case=DataCase.MULTI_BULK, epochs=30) varix = acx.Varix(config=bulk_config) result_varix = varix.run() varix.show_result()

1.1.1 All Options for Data Specifc Parameters¶

Inside the DataInfo object we set the file_path, but there are way more custom data parameters that we can set. Here's a complete list of all parameters you can set in the DataInfo class:

DataInfo General¶

Parameter	Type	Description
file_path	str	Path to the raw data file
data_type	Literal['NUMERIC', 'CATEGORICAL', 'IMG', 'ANNOTATION']	Type of data modality
scaling	Literal['STANDARD', 'MINMAX', 'ROBUST', 'MAXABS', 'NONE', 'NOTSET']	Overrides the globally set scaling method for this modality
filtering	Literal['VAR', 'MAD', 'CORR', 'VARCORR', 'NOFILT', 'NONZEROVAR']	Feature filtering method
sep	Optional[str]	Delimiter for CSV/TSV input files (passed to pandas.read_csv)
extra_anno_file	Optional[str]	Path to an additional annotation file

---

Single-Cell Specific Parameters¶

Parameter	Type	Description
is_single_cell	bool	Whether the dataset represents single-cell data
min_cells	float	Minimum fraction of cells in which a gene must be expressed to be kept (filters rare genes)
min_genes	float	Minimum fraction of genes a cell must express to be kept (filters low-quality cells)
selected_layers	List[str]	Layers to include from the single-cell dataset; must always include "X"
is_X	bool	Whether the data originates from the "X" matrix only
normalize_counts	bool	Whether to normalize single-cell counts by total expression per cell
log_transform	bool	Whether to apply log1p transformation after normalization
k_filter	Optional[int]	Automatically set based on global config; do not override manually

---

Image-Specific Parameters¶

Parameter	Type	Description
img_width_resize	Optional[int]	Target width for image resizing (must equal height)
img_height_resize	Optional[int]	Target height for image resizing (must equal width)

---

XModalix & Translation Parameters¶

Parameter	Type	Description
translate_direction	Optional[Literal['from', 'to']]	Defines translation direction in cross-modal (XModalix) training
pretrain_epochs	int	Number of pretraining epochs specific to this modality (overrides global pretraining setting)

1.2 Use Single Cell Data¶

This works analogously to bulk/tabular data. You will most likely pass h5ad files instead of parquet or CSV files.
For single cell data, you typically use only one data modality, so the term MULTI_SC in our DataCase may be a bit misleading; think of it as supporting 1–n modalities. Also, you most likely won't have a separate annotation file, because this information is already encoded in the h5ad file.

❗❗ Requirements: Getting Tutorial Data ❗❗¶

Download the data here, or use your own h5ad file: https://cloud.scadsai.uni-leipzig.de/index.php/s/qkMY2J5jjofGBZY/download/Sc-2-mini.h5ad Note:

It is best practice to add the is_single_cell parameter to the DataInfo of the modality.

In [4]:

filepath = "data/raw/Sc-2-mini.h5ad"
data_config = DataConfig(
    annotation_columns=["cell_type"],
    data_info={"SC": DataInfo(file_path=filepath, is_single_cell=True)},
)
sc_config = VarixConfig(data_config=data_config, data_case=DataCase.MULTI_SINGLE_CELL)
varix_sc = acx.Varix(config=sc_config)
result_sc = varix_sc.run()

filepath = "data/raw/Sc-2-mini.h5ad" data_config = DataConfig( annotation_columns=["cell_type"], data_info={"SC": DataInfo(file_path=filepath, is_single_cell=True)}, ) sc_config = VarixConfig(data_config=data_config, data_case=DataCase.MULTI_SINGLE_CELL) varix_sc = acx.Varix(config=sc_config) result_sc = varix_sc.run()

Number of common cells: 4554
mudata: View of MuData object with n_obs × n_vars = 4554 × 9009
  obs:	'author_cell_type', 'age_group', 'donor_id', 'nCount_RNA', 'nFeature_RNA', 'nCount_ATAC', 'nFeature_ATAC', 'TSS_percentile', 'nucleosome_signal', 'percent_mt', 'assay_ontology_term_id', 'cell_type_ontology_term_id', 'development_stage_ontology_term_id', 'disease_ontology_term_id', 'self_reported_ethnicity_ontology_term_id', 'organism_ontology_term_id', 'sex_ontology_term_id', 'tissue_ontology_term_id', 'suspension_type', 'is_primary_data', 'batch', 'tissue_type', 'cell_type', 'assay', 'disease', 'organism', 'sex', 'tissue', 'self_reported_ethnicity', 'development_stage', 'observation_joinid'
  var:	'feature_is_filtered', 'feature_name', 'feature_reference', 'feature_biotype', 'feature_length', 'feature_type'
  1 modality
    SC:	4554 x 9009
      obs:	'author_cell_type', 'age_group', 'donor_id', 'nCount_RNA', 'nFeature_RNA', 'nCount_ATAC', 'nFeature_ATAC', 'TSS_percentile', 'nucleosome_signal', 'percent_mt', 'assay_ontology_term_id', 'cell_type_ontology_term_id', 'development_stage_ontology_term_id', 'disease_ontology_term_id', 'self_reported_ethnicity_ontology_term_id', 'organism_ontology_term_id', 'sex_ontology_term_id', 'tissue_ontology_term_id', 'suspension_type', 'is_primary_data', 'batch', 'tissue_type', 'cell_type', 'assay', 'disease', 'organism', 'sex', 'tissue', 'self_reported_ethnicity', 'development_stage', 'observation_joinid'
      var:	'feature_is_filtered', 'feature_name', 'feature_reference', 'feature_biotype', 'feature_length', 'feature_type'
      uns:	'batch_condition', 'citation', 'schema_reference', 'schema_version', 'title'
      obsm:	'X_joint_wnn_umap', 'X_umap'
      layers:	'log2_X', 'log_X'
Processing 1 MuData objects: ['multi_sc']
Processing train modality: multi_sc
Processing valid split
Processing valid modality: multi_sc
Processing test split
Processing test modality: multi_sc
Epoch 1 - Train Loss: 8789.4698
Sub-losses: recon_loss: 8779.0058, var_loss: 10.4640, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 1 - Valid Loss: 10710.5063
Sub-losses: recon_loss: 10660.8622, var_loss: 49.6440, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 2 - Train Loss: 8726.5404
Sub-losses: recon_loss: 8672.9203, var_loss: 53.6201, anneal_factor: 0.0344, effective_beta_factor: 0.0034
Epoch 2 - Valid Loss: 9948.2770
Sub-losses: recon_loss: 9947.1120, var_loss: 1.1651, anneal_factor: 0.0344, effective_beta_factor: 0.0034
Epoch 3 - Train Loss: 8494.0153
Sub-losses: recon_loss: 8471.1358, var_loss: 22.8795, anneal_factor: 0.9656, effective_beta_factor: 0.0966
Epoch 3 - Valid Loss: 9945.7579
Sub-losses: recon_loss: 9938.5164, var_loss: 7.2416, anneal_factor: 0.9656, effective_beta_factor: 0.0966

Special Case for Visualization
As explained before, we can define one or more columns to use for visualization. In our example, we chose the cell_type column. Internally, the column gets prefixed with the name of the data modality defined in the DataInfo dict. Therefore, to visualize this column, we need to add the prefix to the column name, e.g., SC:cell_type.

This should be fixed now, allowing you to pass the column name without the prefix.

In [5]:

varix_sc.show_result(params=["cell_type"])

varix_sc.show_result(params=["cell_type"])

Creating plots ...

1.3 Working with Images¶

Working with images is similar to the previous cases. The main differences are:

• You need to provide a directory path where the image files are located, instead of a path to a single file.
• You need to match each image path to a sample_id, so that we know which metadata belongs to which image.
  • This is done via the annotation file. The annotation file should have one column containing only the image file name (e.g., image.jpg, NOT mydir/image.jpg). The name of this column must be passed to the img_path_col parameter in the config.

We support the following image file extensions (case-insensitive):
".jpg", ".jpeg", ".png", ".tif", ".tiff"
Ensure the directory you specify contains files with these supported extensions.

Optionally, you can specify an image size (only square dimensions are allowed).

❗❗ Requirements: Getting Tutorial Data 2❗❗¶

The data for this example is hosted on Hugging Face Hub (autoencodix/tcga) and is downloaded automatically in the cell below.

In [ ]:

from autoencodix.configs import XModalixConfig
import os
import autoencodix as acx
from autoencodix.configs.default_config import DataConfig, DataInfo, DataCase

from autoencodix.configs import VarixConfig


# ---------------------------------------------------------------------
# Data is hosted on Hugging Face Hub and downloaded (and locally cached)
# automatically, then placed under the paths used below
# ---------------------------------------------------------------------
import shutil
import zipfile
import pandas as pd
from huggingface_hub import hf_hub_download

HF_REPO_ID = "autoencodix/tcga"
xmod_data_root = "data/XModalix-Tut-data"
os.makedirs(xmod_data_root, exist_ok=True)

rna_path = hf_hub_download(repo_id=HF_REPO_ID, repo_type="dataset", filename="rna.parquet")
shutil.copyfile(rna_path, os.path.join(xmod_data_root, "combined_rnaseq_formatted.parquet"))

clin_path = hf_hub_download(repo_id=HF_REPO_ID, repo_type="dataset", filename="clinical.parquet")
mapping_path = hf_hub_download(
    repo_id=HF_REPO_ID, repo_type="dataset", filename="xmodalix/tcga_image_mappings.txt"
)
# The base clinical file has no image column; the XModalix image mapping is a separate,
# per-architecture extension that we join in here (left join: not every sample has an image).
clin_df = pd.read_parquet(clin_path)
mapping_df = pd.read_csv(mapping_path, sep="\t", index_col="sample_ids")
clin_df["img_paths"] = mapping_df["img_paths"]
clin_df.to_parquet(os.path.join(xmod_data_root, "combined_clin_formatted.parquet"))

images_zip = hf_hub_download(
    repo_id=HF_REPO_ID, repo_type="dataset", filename="xmodalix/tcga_fake_images.zip"
)
images_extract_dir = os.path.join(xmod_data_root, "images")
if not os.path.isdir(os.path.join(images_extract_dir, "tcga_fake")):
    with zipfile.ZipFile(images_zip) as zf:
        zf.extractall(images_extract_dir)

clin_file = os.path.join("data/XModalix-Tut-data/combined_clin_formatted.parquet")
rna_file = os.path.join("data/XModalix-Tut-data/combined_rnaseq_formatted.parquet")
img_root = os.path.join("data/XModalix-Tut-data/images/tcga_fake")

xmodalix_config = XModalixConfig(
    img_path_col="img_paths", # col in your annnotation file
    checkpoint_interval=5,
    class_param="CANCER_TYPE_ACRONYM",
    epochs=30,
    requires_paired=False,
    pretrain_epochs=10,
    skip_preprocessing=True,
    data_case=DataCase.IMG_TO_BULK,
    data_config=DataConfig(
        data_info={
            "img": DataInfo(
                file_path=img_root,
                data_type="IMG",
                scaling="MINMAX",
                translate_direction="to",  # For XModalix you need to specify "to" or "from"
                img_width_resize=32,
                img_height_resize=32,
                # extra_anno_file=imganno_file,
            ),
            "rna": DataInfo(
                file_path=rna_file,
                data_type="NUMERIC",
                scaling="MINMAX",
                translate_direction="from",
            ),
            "anno": DataInfo(file_path=clin_file, data_type="ANNOTATION", sep="\t"),
        },
    ),
)

xmodalix = acx.XModalix(config=xmodalix_config)
result = xmodalix.run()

from autoencodix.configs import XModalixConfig import os import autoencodix as acx from autoencodix.configs.default_config import DataConfig, DataInfo, DataCase from autoencodix.configs import VarixConfig # --------------------------------------------------------------------- # Data is hosted on Hugging Face Hub and downloaded (and locally cached) # automatically, then placed under the paths used below # --------------------------------------------------------------------- import shutil import zipfile import pandas as pd from huggingface_hub import hf_hub_download HF_REPO_ID = "autoencodix/tcga" xmod_data_root = "data/XModalix-Tut-data" os.makedirs(xmod_data_root, exist_ok=True) rna_path = hf_hub_download(repo_id=HF_REPO_ID, repo_type="dataset", filename="rna.parquet") shutil.copyfile(rna_path, os.path.join(xmod_data_root, "combined_rnaseq_formatted.parquet")) clin_path = hf_hub_download(repo_id=HF_REPO_ID, repo_type="dataset", filename="clinical.parquet") mapping_path = hf_hub_download( repo_id=HF_REPO_ID, repo_type="dataset", filename="xmodalix/tcga_image_mappings.txt" ) # The base clinical file has no image column; the XModalix image mapping is a separate, # per-architecture extension that we join in here (left join: not every sample has an image). clin_df = pd.read_parquet(clin_path) mapping_df = pd.read_csv(mapping_path, sep="\t", index_col="sample_ids") clin_df["img_paths"] = mapping_df["img_paths"] clin_df.to_parquet(os.path.join(xmod_data_root, "combined_clin_formatted.parquet")) images_zip = hf_hub_download( repo_id=HF_REPO_ID, repo_type="dataset", filename="xmodalix/tcga_fake_images.zip" ) images_extract_dir = os.path.join(xmod_data_root, "images") if not os.path.isdir(os.path.join(images_extract_dir, "tcga_fake")): with zipfile.ZipFile(images_zip) as zf: zf.extractall(images_extract_dir) clin_file = os.path.join("data/XModalix-Tut-data/combined_clin_formatted.parquet") rna_file = os.path.join("data/XModalix-Tut-data/combined_rnaseq_formatted.parquet") img_root = os.path.join("data/XModalix-Tut-data/images/tcga_fake") xmodalix_config = XModalixConfig( img_path_col="img_paths", # col in your annnotation file checkpoint_interval=5, class_param="CANCER_TYPE_ACRONYM", epochs=30, requires_paired=False, pretrain_epochs=10, skip_preprocessing=True, data_case=DataCase.IMG_TO_BULK, data_config=DataConfig( data_info={ "img": DataInfo( file_path=img_root, data_type="IMG", scaling="MINMAX", translate_direction="to", # For XModalix you need to specify "to" or "from" img_width_resize=32, img_height_resize=32, # extra_anno_file=imganno_file, ), "rna": DataInfo( file_path=rna_file, data_type="NUMERIC", scaling="MINMAX", translate_direction="from", ), "anno": DataInfo(file_path=clin_file, data_type="ANNOTATION", sep="\t"), }, ), ) xmodalix = acx.XModalix(config=xmodalix_config) result = xmodalix.run()

In [7]:

fig = xmodalix.visualizer.show_2D_translation(
    result=result,
    translated_modality="img.img",
    split="test",
    param="CANCER_TYPE_ACRONYM",
    reducer="UMAP",
)

fig = xmodalix.visualizer.show_2D_translation( result=result, translated_modality="img.img", split="test", param="CANCER_TYPE_ACRONYM", reducer="UMAP", )

len of tensor-list: 711
len of tensor_ids: 711

2) Passing Data Directly¶

If you want to provide the data yourself, or the data comes from another script directly as a Python object, we show how to prepare it for AUTOENCODIX.
We will go over our three main data modalities as in the previous section.

The main class for this is the DataPackage class. This serves as a container for pandas DataFrames and AnnData/MuData.
Let's see in code how this works to make things clearer:

2.1) Passing Numeric Tabular Data¶

Let's assume we have two data modalities as pd.DataFrames and one annotation file with metadata, also as a pd.DataFrame.
The only requirement for the data files is that they are numeric. For the annotation file, categorical variables are also allowed.

In [8]:

from autoencodix.utils.example_data import raw_protein, raw_rna, annotation
from autoencodix.data import DataPackage

print(f"Type of rna: {type(raw_rna)}")
raw_protein.head()

from autoencodix.utils.example_data import raw_protein, raw_rna, annotation from autoencodix.data import DataPackage print(f"Type of rna: {type(raw_rna)}") raw_protein.head()

Type of rna: <class 'pandas.core.frame.DataFrame'>

Out[8]:

	protein_0	protein_1	protein_2	protein_3	protein_4	protein_5	protein_6	protein_7	protein_8	protein_9	...	protein_70	protein_71	protein_72	protein_73	protein_74	protein_75	protein_76	protein_77	protein_78	protein_79
sample_0	-12.042401	0.087191	-2.569039	1.194324	3.232265	-0.849021	-2.456986	-4.714146	-0.330510	1.791241	...	1.298304	-0.888675	2.211052	0.704696	2.647467	3.053971	1.389405	-1.531050	-0.931632	-8.202654
sample_1	-5.774403	0.189626	-4.880973	0.945787	1.178465	-5.730645	-2.277935	-1.755148	-4.730100	0.394876	...	-4.046358	-3.408130	3.022748	-0.314424	2.378629	0.222959	-0.101802	4.570407	0.801779	-5.779545
sample_2	2.853659	-1.509067	-0.102335	-0.900886	-1.202590	1.023140	-0.556068	-0.431105	-3.262940	-0.705130	...	2.127268	2.228164	0.949771	0.055224	-5.031155	-2.850230	-3.012782	1.672382	0.657073	-0.844778
sample_3	3.409531	-0.221087	1.766617	3.460660	-1.476767	-0.344091	-3.543586	3.578511	-0.855364	2.385952	...	-2.032749	-1.866859	-0.718189	0.603248	2.518338	-0.300170	-3.655031	1.102813	0.516071	1.866424
sample_4	2.056824	-7.165520	1.986310	-2.968593	2.047759	-6.310128	2.237384	1.204833	-3.900744	0.986285	...	-2.756762	-2.793976	4.341621	-4.084905	-2.004152	7.571476	5.221252	1.209981	1.946205	-3.417621

5 rows × 80 columns

In our DataPackage class we have four attributes:

• multi_bulk: any tabular numeric data goes here
• multi_sc: single-cell data
• annotation: metadata for samples
• img: image data

Each attribute holds a Dict, where you can pass your DataFrames (or AnnData/MuData) and assign a data modality key, like:

In [9]:

my_dp = DataPackage(
    multi_bulk={"rna": raw_rna, "protein": raw_protein}, annotation={"anno": annotation}
)

my_dp = DataPackage( multi_bulk={"rna": raw_rna, "protein": raw_protein}, annotation={"anno": annotation} )

The instance of DataPackage can be directly passed to our pipeline object via the data keyword argument.

In [10]:

import os
import autoencodix as acx
from autoencodix.configs.default_config import DataConfig, DataInfo, DataCase
from autoencodix.configs import VarixConfig

config = VarixConfig(data_case=DataCase.MULTI_BULK, epochs=30)
varix = acx.Varix(data=my_dp, config=config)
result = varix.run()

import os import autoencodix as acx from autoencodix.configs.default_config import DataConfig, DataInfo, DataCase from autoencodix.configs import VarixConfig config = VarixConfig(data_case=DataCase.MULTI_BULK, epochs=30) varix = acx.Varix(data=my_dp, config=config) result = varix.run()

in handle_direct_user_data with data: <class 'autoencodix.data.datapackage.DataPackage'>
anno key: anno
Epoch 1 - Train Loss: 213.0875
Sub-losses: recon_loss: 213.0875, var_loss: 0.0000, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 1 - Valid Loss: 181.1046
Sub-losses: recon_loss: 181.1046, var_loss: 0.0000, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 2 - Train Loss: 203.6570
Sub-losses: recon_loss: 203.6569, var_loss: 0.0000, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 2 - Valid Loss: 182.7873
Sub-losses: recon_loss: 182.7873, var_loss: 0.0000, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 3 - Train Loss: 197.0060
Sub-losses: recon_loss: 197.0060, var_loss: 0.0001, anneal_factor: 0.0002, effective_beta_factor: 0.0000
Epoch 3 - Valid Loss: 181.2981
Sub-losses: recon_loss: 181.2981, var_loss: 0.0000, anneal_factor: 0.0002, effective_beta_factor: 0.0000
Epoch 4 - Train Loss: 195.3388
Sub-losses: recon_loss: 195.3387, var_loss: 0.0001, anneal_factor: 0.0003, effective_beta_factor: 0.0000
Epoch 4 - Valid Loss: 182.7998
Sub-losses: recon_loss: 182.7997, var_loss: 0.0001, anneal_factor: 0.0003, effective_beta_factor: 0.0000
Epoch 5 - Train Loss: 193.6254
Sub-losses: recon_loss: 193.6251, var_loss: 0.0002, anneal_factor: 0.0007, effective_beta_factor: 0.0001
Epoch 5 - Valid Loss: 181.9047
Sub-losses: recon_loss: 181.9045, var_loss: 0.0002, anneal_factor: 0.0007, effective_beta_factor: 0.0001
Epoch 6 - Train Loss: 190.1059
Sub-losses: recon_loss: 190.1054, var_loss: 0.0005, anneal_factor: 0.0013, effective_beta_factor: 0.0001
Epoch 6 - Valid Loss: 181.2319
Sub-losses: recon_loss: 181.2315, var_loss: 0.0004, anneal_factor: 0.0013, effective_beta_factor: 0.0001
Epoch 7 - Train Loss: 188.5050
Sub-losses: recon_loss: 188.5040, var_loss: 0.0010, anneal_factor: 0.0025, effective_beta_factor: 0.0002
Epoch 7 - Valid Loss: 178.8502
Sub-losses: recon_loss: 178.8494, var_loss: 0.0007, anneal_factor: 0.0025, effective_beta_factor: 0.0002
Epoch 8 - Train Loss: 187.1796
Sub-losses: recon_loss: 187.1778, var_loss: 0.0018, anneal_factor: 0.0048, effective_beta_factor: 0.0005
Epoch 8 - Valid Loss: 177.7683
Sub-losses: recon_loss: 177.7668, var_loss: 0.0015, anneal_factor: 0.0048, effective_beta_factor: 0.0005
Epoch 9 - Train Loss: 186.2260
Sub-losses: recon_loss: 186.2216, var_loss: 0.0043, anneal_factor: 0.0093, effective_beta_factor: 0.0009
Epoch 9 - Valid Loss: 178.5588
Sub-losses: recon_loss: 178.5560, var_loss: 0.0029, anneal_factor: 0.0093, effective_beta_factor: 0.0009
Epoch 10 - Train Loss: 184.8700
Sub-losses: recon_loss: 184.8601, var_loss: 0.0099, anneal_factor: 0.0180, effective_beta_factor: 0.0018
Epoch 10 - Valid Loss: 176.6685
Sub-losses: recon_loss: 176.6625, var_loss: 0.0060, anneal_factor: 0.0180, effective_beta_factor: 0.0018
Epoch 11 - Train Loss: 183.3115
Sub-losses: recon_loss: 183.2924, var_loss: 0.0191, anneal_factor: 0.0344, effective_beta_factor: 0.0034
Epoch 11 - Valid Loss: 175.8696
Sub-losses: recon_loss: 175.8575, var_loss: 0.0121, anneal_factor: 0.0344, effective_beta_factor: 0.0034
Epoch 12 - Train Loss: 181.8502
Sub-losses: recon_loss: 181.8104, var_loss: 0.0398, anneal_factor: 0.0650, effective_beta_factor: 0.0065
Epoch 12 - Valid Loss: 174.2435
Sub-losses: recon_loss: 174.2198, var_loss: 0.0236, anneal_factor: 0.0650, effective_beta_factor: 0.0065
Epoch 13 - Train Loss: 180.3452
Sub-losses: recon_loss: 180.2692, var_loss: 0.0760, anneal_factor: 0.1192, effective_beta_factor: 0.0119
Epoch 13 - Valid Loss: 173.8356
Sub-losses: recon_loss: 173.7906, var_loss: 0.0450, anneal_factor: 0.1192, effective_beta_factor: 0.0119
Epoch 14 - Train Loss: 179.0371
Sub-losses: recon_loss: 178.8860, var_loss: 0.1510, anneal_factor: 0.2086, effective_beta_factor: 0.0209
Epoch 14 - Valid Loss: 176.6850
Sub-losses: recon_loss: 176.6072, var_loss: 0.0778, anneal_factor: 0.2086, effective_beta_factor: 0.0209
Epoch 15 - Train Loss: 177.7575
Sub-losses: recon_loss: 177.4681, var_loss: 0.2894, anneal_factor: 0.3392, effective_beta_factor: 0.0339
Epoch 15 - Valid Loss: 174.4373
Sub-losses: recon_loss: 174.3145, var_loss: 0.1228, anneal_factor: 0.3392, effective_beta_factor: 0.0339
Epoch 16 - Train Loss: 176.9952
Sub-losses: recon_loss: 176.5270, var_loss: 0.4682, anneal_factor: 0.5000, effective_beta_factor: 0.0500
Epoch 16 - Valid Loss: 171.4875
Sub-losses: recon_loss: 171.2920, var_loss: 0.1955, anneal_factor: 0.5000, effective_beta_factor: 0.0500
Epoch 17 - Train Loss: 176.3160
Sub-losses: recon_loss: 175.7257, var_loss: 0.5903, anneal_factor: 0.6608, effective_beta_factor: 0.0661
Epoch 17 - Valid Loss: 172.6819
Sub-losses: recon_loss: 172.4107, var_loss: 0.2712, anneal_factor: 0.6608, effective_beta_factor: 0.0661
Epoch 18 - Train Loss: 174.9301
Sub-losses: recon_loss: 174.2704, var_loss: 0.6597, anneal_factor: 0.7914, effective_beta_factor: 0.0791
Epoch 18 - Valid Loss: 173.1308
Sub-losses: recon_loss: 172.7752, var_loss: 0.3555, anneal_factor: 0.7914, effective_beta_factor: 0.0791
Epoch 19 - Train Loss: 174.6540
Sub-losses: recon_loss: 173.8625, var_loss: 0.7915, anneal_factor: 0.8808, effective_beta_factor: 0.0881
Epoch 19 - Valid Loss: 171.5805
Sub-losses: recon_loss: 171.2483, var_loss: 0.3321, anneal_factor: 0.8808, effective_beta_factor: 0.0881
Epoch 20 - Train Loss: 173.4465
Sub-losses: recon_loss: 172.6194, var_loss: 0.8272, anneal_factor: 0.9350, effective_beta_factor: 0.0935
Epoch 20 - Valid Loss: 173.5106
Sub-losses: recon_loss: 173.1158, var_loss: 0.3948, anneal_factor: 0.9350, effective_beta_factor: 0.0935
Epoch 21 - Train Loss: 171.5685
Sub-losses: recon_loss: 170.6473, var_loss: 0.9213, anneal_factor: 0.9656, effective_beta_factor: 0.0966
Epoch 21 - Valid Loss: 170.0671
Sub-losses: recon_loss: 169.6319, var_loss: 0.4352, anneal_factor: 0.9656, effective_beta_factor: 0.0966
Epoch 22 - Train Loss: 171.4459
Sub-losses: recon_loss: 170.5522, var_loss: 0.8937, anneal_factor: 0.9820, effective_beta_factor: 0.0982
Epoch 22 - Valid Loss: 170.3481
Sub-losses: recon_loss: 169.9050, var_loss: 0.4431, anneal_factor: 0.9820, effective_beta_factor: 0.0982
Epoch 23 - Train Loss: 170.4136
Sub-losses: recon_loss: 169.5085, var_loss: 0.9051, anneal_factor: 0.9907, effective_beta_factor: 0.0991
Epoch 23 - Valid Loss: 166.4105
Sub-losses: recon_loss: 165.8741, var_loss: 0.5364, anneal_factor: 0.9907, effective_beta_factor: 0.0991
Epoch 24 - Train Loss: 169.3760
Sub-losses: recon_loss: 168.4489, var_loss: 0.9271, anneal_factor: 0.9952, effective_beta_factor: 0.0995
Epoch 24 - Valid Loss: 168.8194
Sub-losses: recon_loss: 168.2755, var_loss: 0.5439, anneal_factor: 0.9952, effective_beta_factor: 0.0995
Epoch 25 - Train Loss: 167.9767
Sub-losses: recon_loss: 167.0011, var_loss: 0.9756, anneal_factor: 0.9975, effective_beta_factor: 0.0998
Epoch 25 - Valid Loss: 166.1262
Sub-losses: recon_loss: 165.5105, var_loss: 0.6158, anneal_factor: 0.9975, effective_beta_factor: 0.0998
Epoch 26 - Train Loss: 168.2633
Sub-losses: recon_loss: 167.2798, var_loss: 0.9835, anneal_factor: 0.9987, effective_beta_factor: 0.0999
Epoch 26 - Valid Loss: 166.0647
Sub-losses: recon_loss: 165.4552, var_loss: 0.6096, anneal_factor: 0.9987, effective_beta_factor: 0.0999
Epoch 27 - Train Loss: 166.0185
Sub-losses: recon_loss: 164.9996, var_loss: 1.0188, anneal_factor: 0.9993, effective_beta_factor: 0.0999
Epoch 27 - Valid Loss: 162.5063
Sub-losses: recon_loss: 161.8390, var_loss: 0.6673, anneal_factor: 0.9993, effective_beta_factor: 0.0999
Epoch 28 - Train Loss: 165.6559
Sub-losses: recon_loss: 164.6413, var_loss: 1.0146, anneal_factor: 0.9997, effective_beta_factor: 0.1000
Epoch 28 - Valid Loss: 159.9763
Sub-losses: recon_loss: 159.3159, var_loss: 0.6603, anneal_factor: 0.9997, effective_beta_factor: 0.1000
Epoch 29 - Train Loss: 164.1951
Sub-losses: recon_loss: 163.0985, var_loss: 1.0966, anneal_factor: 0.9998, effective_beta_factor: 0.1000
Epoch 29 - Valid Loss: 159.6291
Sub-losses: recon_loss: 158.8910, var_loss: 0.7381, anneal_factor: 0.9998, effective_beta_factor: 0.1000
Epoch 30 - Train Loss: 164.5758
Sub-losses: recon_loss: 163.4312, var_loss: 1.1446, anneal_factor: 0.9999, effective_beta_factor: 0.1000
Epoch 30 - Valid Loss: 158.6753
Sub-losses: recon_loss: 157.8998, var_loss: 0.7755, anneal_factor: 0.9999, effective_beta_factor: 0.1000

You still can customize your config by providing a DataConfig with DataInfo as shown in section 1.

In [11]:

my_dp = DataPackage(
    multi_bulk={"rna": raw_rna, "protein": raw_protein}, annotation={"anno": annotation}
)
data_config = DataConfig(
    data_info={
        "rna": DataInfo(scaling="MINMAX"),
        "protein": DataInfo(scaling="MINMAX"),
    },
)

bulk_config = VarixConfig(data_config=data_config, data_case=DataCase.MULTI_BULK, epochs=30)
varix = acx.Varix(config=bulk_config, data=my_dp)
r = varix.run()

my_dp = DataPackage( multi_bulk={"rna": raw_rna, "protein": raw_protein}, annotation={"anno": annotation} ) data_config = DataConfig( data_info={ "rna": DataInfo(scaling="MINMAX"), "protein": DataInfo(scaling="MINMAX"), }, ) bulk_config = VarixConfig(data_config=data_config, data_case=DataCase.MULTI_BULK, epochs=30) varix = acx.Varix(config=bulk_config, data=my_dp) r = varix.run()

in handle_direct_user_data with data: <class 'autoencodix.data.datapackage.DataPackage'>
anno key: anno
Epoch 1 - Train Loss: 70.5554
Sub-losses: recon_loss: 70.5554, var_loss: 0.0000, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 1 - Valid Loss: 41.6647
Sub-losses: recon_loss: 41.6647, var_loss: 0.0000, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 2 - Train Loss: 47.3735
Sub-losses: recon_loss: 47.3734, var_loss: 0.0001, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 2 - Valid Loss: 31.0064
Sub-losses: recon_loss: 31.0064, var_loss: 0.0000, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 3 - Train Loss: 34.0758
Sub-losses: recon_loss: 34.0757, var_loss: 0.0001, anneal_factor: 0.0002, effective_beta_factor: 0.0000
Epoch 3 - Valid Loss: 22.1221
Sub-losses: recon_loss: 22.1221, var_loss: 0.0000, anneal_factor: 0.0002, effective_beta_factor: 0.0000
Epoch 4 - Train Loss: 25.4825
Sub-losses: recon_loss: 25.4823, var_loss: 0.0002, anneal_factor: 0.0003, effective_beta_factor: 0.0000
Epoch 4 - Valid Loss: 17.1424
Sub-losses: recon_loss: 17.1423, var_loss: 0.0001, anneal_factor: 0.0003, effective_beta_factor: 0.0000
Epoch 5 - Train Loss: 21.8146
Sub-losses: recon_loss: 21.8142, var_loss: 0.0004, anneal_factor: 0.0007, effective_beta_factor: 0.0001
Epoch 5 - Valid Loss: 15.0417
Sub-losses: recon_loss: 15.0414, var_loss: 0.0003, anneal_factor: 0.0007, effective_beta_factor: 0.0001
Epoch 6 - Train Loss: 18.5185
Sub-losses: recon_loss: 18.5175, var_loss: 0.0010, anneal_factor: 0.0013, effective_beta_factor: 0.0001
Epoch 6 - Valid Loss: 12.8975
Sub-losses: recon_loss: 12.8969, var_loss: 0.0006, anneal_factor: 0.0013, effective_beta_factor: 0.0001
Epoch 7 - Train Loss: 16.4023
Sub-losses: recon_loss: 16.4005, var_loss: 0.0017, anneal_factor: 0.0025, effective_beta_factor: 0.0002
Epoch 7 - Valid Loss: 11.2239
Sub-losses: recon_loss: 11.2227, var_loss: 0.0011, anneal_factor: 0.0025, effective_beta_factor: 0.0002
Epoch 8 - Train Loss: 15.2823
Sub-losses: recon_loss: 15.2787, var_loss: 0.0036, anneal_factor: 0.0048, effective_beta_factor: 0.0005
Epoch 8 - Valid Loss: 11.2603
Sub-losses: recon_loss: 11.2578, var_loss: 0.0024, anneal_factor: 0.0048, effective_beta_factor: 0.0005
Epoch 9 - Train Loss: 14.2300
Sub-losses: recon_loss: 14.2231, var_loss: 0.0069, anneal_factor: 0.0093, effective_beta_factor: 0.0009
Epoch 9 - Valid Loss: 9.5415
Sub-losses: recon_loss: 9.5367, var_loss: 0.0047, anneal_factor: 0.0093, effective_beta_factor: 0.0009
Epoch 10 - Train Loss: 13.1526
Sub-losses: recon_loss: 13.1392, var_loss: 0.0134, anneal_factor: 0.0180, effective_beta_factor: 0.0018
Epoch 10 - Valid Loss: 8.9612
Sub-losses: recon_loss: 8.9526, var_loss: 0.0086, anneal_factor: 0.0180, effective_beta_factor: 0.0018
Epoch 11 - Train Loss: 12.3679
Sub-losses: recon_loss: 12.3416, var_loss: 0.0263, anneal_factor: 0.0344, effective_beta_factor: 0.0034
Epoch 11 - Valid Loss: 8.5276
Sub-losses: recon_loss: 8.5107, var_loss: 0.0170, anneal_factor: 0.0344, effective_beta_factor: 0.0034
Epoch 12 - Train Loss: 11.8942
Sub-losses: recon_loss: 11.8449, var_loss: 0.0493, anneal_factor: 0.0650, effective_beta_factor: 0.0065
Epoch 12 - Valid Loss: 8.5881
Sub-losses: recon_loss: 8.5546, var_loss: 0.0335, anneal_factor: 0.0650, effective_beta_factor: 0.0065
Epoch 13 - Train Loss: 11.5795
Sub-losses: recon_loss: 11.4895, var_loss: 0.0899, anneal_factor: 0.1192, effective_beta_factor: 0.0119
Epoch 13 - Valid Loss: 7.7008
Sub-losses: recon_loss: 7.6399, var_loss: 0.0609, anneal_factor: 0.1192, effective_beta_factor: 0.0119
Epoch 14 - Train Loss: 11.0137
Sub-losses: recon_loss: 10.8602, var_loss: 0.1535, anneal_factor: 0.2086, effective_beta_factor: 0.0209
Epoch 14 - Valid Loss: 8.2267
Sub-losses: recon_loss: 8.1196, var_loss: 0.1070, anneal_factor: 0.2086, effective_beta_factor: 0.0209
Epoch 15 - Train Loss: 10.6267
Sub-losses: recon_loss: 10.3694, var_loss: 0.2573, anneal_factor: 0.3392, effective_beta_factor: 0.0339
Epoch 15 - Valid Loss: 7.3922
Sub-losses: recon_loss: 7.2290, var_loss: 0.1633, anneal_factor: 0.3392, effective_beta_factor: 0.0339
Epoch 16 - Train Loss: 10.3686
Sub-losses: recon_loss: 9.9804, var_loss: 0.3882, anneal_factor: 0.5000, effective_beta_factor: 0.0500
Epoch 16 - Valid Loss: 7.6217
Sub-losses: recon_loss: 7.3875, var_loss: 0.2342, anneal_factor: 0.5000, effective_beta_factor: 0.0500
Epoch 17 - Train Loss: 10.6520
Sub-losses: recon_loss: 10.1693, var_loss: 0.4826, anneal_factor: 0.6608, effective_beta_factor: 0.0661
Epoch 17 - Valid Loss: 7.4087
Sub-losses: recon_loss: 7.1050, var_loss: 0.3036, anneal_factor: 0.6608, effective_beta_factor: 0.0661
Epoch 18 - Train Loss: 10.4505
Sub-losses: recon_loss: 9.8999, var_loss: 0.5506, anneal_factor: 0.7914, effective_beta_factor: 0.0791
Epoch 18 - Valid Loss: 7.4526
Sub-losses: recon_loss: 7.0974, var_loss: 0.3552, anneal_factor: 0.7914, effective_beta_factor: 0.0791
Epoch 19 - Train Loss: 9.9851
Sub-losses: recon_loss: 9.3365, var_loss: 0.6485, anneal_factor: 0.8808, effective_beta_factor: 0.0881
Epoch 19 - Valid Loss: 7.6091
Sub-losses: recon_loss: 7.2028, var_loss: 0.4063, anneal_factor: 0.8808, effective_beta_factor: 0.0881
Epoch 20 - Train Loss: 9.7975
Sub-losses: recon_loss: 9.1787, var_loss: 0.6187, anneal_factor: 0.9350, effective_beta_factor: 0.0935
Epoch 20 - Valid Loss: 7.9724
Sub-losses: recon_loss: 7.5623, var_loss: 0.4101, anneal_factor: 0.9350, effective_beta_factor: 0.0935
Epoch 21 - Train Loss: 9.8118
Sub-losses: recon_loss: 9.2231, var_loss: 0.5887, anneal_factor: 0.9656, effective_beta_factor: 0.0966
Epoch 21 - Valid Loss: 7.0572
Sub-losses: recon_loss: 6.6666, var_loss: 0.3906, anneal_factor: 0.9656, effective_beta_factor: 0.0966
Epoch 22 - Train Loss: 9.2632
Sub-losses: recon_loss: 8.6515, var_loss: 0.6117, anneal_factor: 0.9820, effective_beta_factor: 0.0982
Epoch 22 - Valid Loss: 7.1510
Sub-losses: recon_loss: 6.7570, var_loss: 0.3940, anneal_factor: 0.9820, effective_beta_factor: 0.0982
Epoch 23 - Train Loss: 9.4137
Sub-losses: recon_loss: 8.8684, var_loss: 0.5453, anneal_factor: 0.9907, effective_beta_factor: 0.0991
Epoch 23 - Valid Loss: 6.8356
Sub-losses: recon_loss: 6.4421, var_loss: 0.3936, anneal_factor: 0.9907, effective_beta_factor: 0.0991
Epoch 24 - Train Loss: 9.0168
Sub-losses: recon_loss: 8.5040, var_loss: 0.5128, anneal_factor: 0.9952, effective_beta_factor: 0.0995
Epoch 24 - Valid Loss: 7.3864
Sub-losses: recon_loss: 7.0261, var_loss: 0.3603, anneal_factor: 0.9952, effective_beta_factor: 0.0995
Epoch 25 - Train Loss: 8.8792
Sub-losses: recon_loss: 8.3994, var_loss: 0.4799, anneal_factor: 0.9975, effective_beta_factor: 0.0998
Epoch 25 - Valid Loss: 6.8562
Sub-losses: recon_loss: 6.5303, var_loss: 0.3259, anneal_factor: 0.9975, effective_beta_factor: 0.0998
Epoch 26 - Train Loss: 8.6873
Sub-losses: recon_loss: 8.2279, var_loss: 0.4594, anneal_factor: 0.9987, effective_beta_factor: 0.0999
Epoch 26 - Valid Loss: 6.8721
Sub-losses: recon_loss: 6.5580, var_loss: 0.3142, anneal_factor: 0.9987, effective_beta_factor: 0.0999
Epoch 27 - Train Loss: 8.2207
Sub-losses: recon_loss: 7.7731, var_loss: 0.4476, anneal_factor: 0.9993, effective_beta_factor: 0.0999
Epoch 27 - Valid Loss: 6.7171
Sub-losses: recon_loss: 6.4000, var_loss: 0.3171, anneal_factor: 0.9993, effective_beta_factor: 0.0999
Epoch 28 - Train Loss: 8.3027
Sub-losses: recon_loss: 7.8728, var_loss: 0.4299, anneal_factor: 0.9997, effective_beta_factor: 0.1000
Epoch 28 - Valid Loss: 6.4943
Sub-losses: recon_loss: 6.1929, var_loss: 0.3014, anneal_factor: 0.9997, effective_beta_factor: 0.1000
Epoch 29 - Train Loss: 8.1898
Sub-losses: recon_loss: 7.7680, var_loss: 0.4218, anneal_factor: 0.9998, effective_beta_factor: 0.1000
Epoch 29 - Valid Loss: 6.2955
Sub-losses: recon_loss: 6.0105, var_loss: 0.2851, anneal_factor: 0.9998, effective_beta_factor: 0.1000
Epoch 30 - Train Loss: 8.3483
Sub-losses: recon_loss: 7.9515, var_loss: 0.3968, anneal_factor: 0.9999, effective_beta_factor: 0.1000
Epoch 30 - Valid Loss: 6.2954
Sub-losses: recon_loss: 6.0223, var_loss: 0.2731, anneal_factor: 0.9999, effective_beta_factor: 0.1000

2.2) Passing Single Cell Data¶

This works analogously to the numeric tabular case above. Instead of pd.DataFrames, we pass AnnData or MuData objects (from scverse) to our DataPackage.

It is also possible to pass AnnData or MuData directly to the pipeline.

Working with AnnData

In [12]:

from autoencodix.utils.example_data import sample_adata, sample_mudata

sample_adata

from autoencodix.utils.example_data import sample_adata, sample_mudata sample_adata

Out[12]:

AnnData object with n_obs × n_vars = 1000 × 500
    obs: 'cell_type', 'batch', 'donor', 'cell_cycle'

For AnnData, this is now actually super simple: you can just pass the AnnData object directly to the pipeline, without specifying the DataCase for Varix, because we can infer it from the pipeline type and the input data type.

In this example, we did not pass a config, which means the pipeline will use default parameters.

Of course, you can also define and pass a custom config if needed.

In [13]:

varix = acx.Varix(data=sample_adata)
result = varix.run()

varix = acx.Varix(data=sample_adata) result = varix.run()

in handle_direct_user_data with data: <class 'anndata._core.anndata.AnnData'>
mudata: View of MuData object with n_obs × n_vars = 1000 × 500
  1 modality
    user-data:	1000 x 500
      obs:	'cell_type', 'batch', 'donor', 'cell_cycle'
Processing 1 MuData objects: ['multi_sc']
Processing train modality: multi_sc
Processing valid split
Processing valid modality: multi_sc
Processing test split
Processing test modality: multi_sc
Epoch 1 - Train Loss: 565.4079
Sub-losses: recon_loss: 565.4079, var_loss: 0.0000, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 1 - Valid Loss: 569.3834
Sub-losses: recon_loss: 569.3833, var_loss: 0.0001, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 2 - Train Loss: 524.0325
Sub-losses: recon_loss: 523.9518, var_loss: 0.0807, anneal_factor: 0.0344, effective_beta_factor: 0.0034
Epoch 2 - Valid Loss: 545.3499
Sub-losses: recon_loss: 545.1603, var_loss: 0.1896, anneal_factor: 0.0344, effective_beta_factor: 0.0034
Epoch 3 - Train Loss: 500.4935
Sub-losses: recon_loss: 497.7722, var_loss: 2.7214, anneal_factor: 0.9656, effective_beta_factor: 0.0966
Epoch 3 - Valid Loss: 530.9623
Sub-losses: recon_loss: 529.8343, var_loss: 1.1280, anneal_factor: 0.9656, effective_beta_factor: 0.0966

Working with MuData
In some cases, you may want to combine multiple single-cell modalities. Here, we expect a MuData object. This can be passed directly to the pipeline, as shown for AnnData, or via our DataPackage, as shown in the tabular numeric case.

In [14]:

sample_mudata

sample_mudata

Out[14]:

MuData object with n_obs × n_vars = 1000 × 700
  2 modalities
    rna:	1000 x 500
      obs:	'cell_type', 'batch', 'donor', 'cell_cycle'
    protein:	1000 x 200
      obs:	'cell_type', 'batch', 'donor', 'cell_cycle'

In [15]:

varix = acx.Varix(data=sample_mudata)
result = varix.run()

varix = acx.Varix(data=sample_mudata) result = varix.run()

in handle_direct_user_data with data: <class 'mudata._core.mudata.MuData'>
mudata: View of MuData object with n_obs × n_vars = 1000 × 700
  2 modalities
    rna:	1000 x 500
      obs:	'cell_type', 'batch', 'donor', 'cell_cycle'
    protein:	1000 x 200
      obs:	'cell_type', 'batch', 'donor', 'cell_cycle'
Processing 1 MuData objects: ['multi_sc']
Processing train modality: multi_sc
Processing valid split
Processing valid modality: multi_sc
Processing test split
Processing test modality: multi_sc
Epoch 1 - Train Loss: 775.5065
Sub-losses: recon_loss: 775.5065, var_loss: 0.0001, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 1 - Valid Loss: 841.4985
Sub-losses: recon_loss: 841.4978, var_loss: 0.0007, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 2 - Train Loss: 692.7548
Sub-losses: recon_loss: 692.6353, var_loss: 0.1195, anneal_factor: 0.0344, effective_beta_factor: 0.0034
Epoch 2 - Valid Loss: 726.0012
Sub-losses: recon_loss: 725.8485, var_loss: 0.1527, anneal_factor: 0.0344, effective_beta_factor: 0.0034
Epoch 3 - Train Loss: 660.3665
Sub-losses: recon_loss: 655.4157, var_loss: 4.9508, anneal_factor: 0.9656, effective_beta_factor: 0.0966
Epoch 3 - Valid Loss: 699.4251
Sub-losses: recon_loss: 697.4528, var_loss: 1.9723, anneal_factor: 0.9656, effective_beta_factor: 0.0966

Single Cell Specific Processing TODO

2.3) Passing Image Data¶

We recommend passing image data via a folder path in the config, as shown in Section 1.3.
For completeness, we show how one could pass images directly to the pipeline, but this is not recommended.

We store image data as a list, where each item represents one image and is of our custom class ImgData.

In [16]:

from autoencodix.data import ImgData
from autoencodix.data.datapackage import DataPackage
import numpy as np
import pandas as pd

imgdata = [ImgData(img=np.array(1), sample_id="myImg", annotation=pd.DataFrame())]
dp = DataPackage({"IMG": imgdata})
# in real applications this will be a longer list of images

from autoencodix.data import ImgData from autoencodix.data.datapackage import DataPackage import numpy as np import pandas as pd imgdata = [ImgData(img=np.array(1), sample_id="myImg", annotation=pd.DataFrame())] dp = DataPackage({"IMG": imgdata}) # in real applications this will be a longer list of images

3) Note on Paired and Unpaired Data¶

When working with mutli-modal data it is common that not all data modalities are present for all samples. For most the models Vanillix, Varix, and Ontix, we require paired data due to inherent architecture constraints and remove samples that are not present in all data modalities.

For the models XModalix and Stackix don't have the same arechitectural constraints. Thus, we can allow also unpaired data. This needs to be specified in the config via the param requires_paired=False.

4) Notes on Data Modality Specific Preprocessing and Skipping Preprocessing¶

Depending on the data modality, our pipeline applies different preprocessing steps. We provide an overview of applied preprocessing techniques for our three main data modalities.

4.1 Preprocessing for Tabular Numeric Data¶

The following steps are applied in order:

• Remove missing and NaN values from tabular numeric data.
• Consolidate missing and NaN values to one "missing" class for the annotation columns (as specified with the annotation_columns parameter in config) of the annotation dataframe.
• Split data into train/validation/test sets according to the ratios defined in the config.
• Filter features based on the method defined in the config. The statistical operations to filter the data are performed on the train set to avoid data leakage. The learned filtering from the train set is then applied to the validation and test sets.
• Fit a sklearn scaler on the train data, depending on the defined scaling algorithm (MinMax, Standard, etc.). The fitted scaler is then applied to the validation and test sets.
  • Filtering and scaling are also applied if you run the predict step with new unseen data.
• Transform the data into a PyTorch dataset to be ready for training with nn.Module.

4.2 Preprocessing for Sparse Numeric Data (Single Cell)¶

The following steps are performed in order:

• Replace missing values with zeros.
• Apply two single-cell-specific processing steps if defined in the config:
  • Filter cells based on a gene count, defined in min_genes in the config via sc.pp.filter_cells.
  • Log-transform, if specified in config under log_transform via sc.pp.log1p.
• After this, follow the preprocessing for tabular numeric data (see above).
• Keep the data sparse in our PyTorch dataset until batch level. This prevents out-of-memory errors for large datasets when transforming from sparse to dense.

4.3 Preprocessing for Image Data¶

For image data, preprocessing is as follows:

• Split data into train/validation/test sets as defined in the config.
• Resize images to a quadratic image size that fits our encoder depth or to the specified width and height in the config.
• Scale pixels depending on the defined scaling algorithm (MinMax, Standard, etc.) specific to images, not using sklearn.
• Transform data into a PyTorch dataset specialized for image data.

4.4 Skipping Preprocessing¶

Preprocessing can be skipped via the config parameter skip_preprocessing. However, we cannot guarantee that there won't be RuntimeErrors.
Errors should not occur if your data does not contain NaNs and your loss function matches the scaling. For example, if you set reconstruction_loss to bce, ensure your input data is between 0 and 1, because the last layer of our architecture is a Sigmoid.

5) Adding Custom Splits¶

Our standard preprocessing splits the data randomly into train/validation/test sets. The custom splits feature is currently deprecated. We may add this feature in the future, likely working as follows:

import numpy as np
from autoencodix.configs.default_config import DataCase

sample_data = np.random.rand(100, 10)
custom_train_indices = np.arange(75)  # we won't allow overlap between splits
custom_valid_indices = np.arange(75, 80)
custom_test_indices = np.arange(80, 100)

# the custom split needs to be a dictionary with keys "train", "valid", and "test" and indices of the samples to be included in each split as numpy arrays
custom_split = {
    "train": custom_train_indices,
    "valid": custom_valid_indices,
    "test": custom_test_indices,
}
config = DefaultConfig(data_case=DataCase.MULTI_BULK)
van = acx.Vanillix(data=raw_bulk, custom_splits=custom_split, config=config)
van.preprocess()
van.fit()