Note: This environment variable is required for fully deterministic CuBLAS ops on CUDA >= 10.2 when reproducible=True is set below. Without it, PyTorch raises a RuntimeError instead of training deterministically. It must be set before torch is imported. See the README FAQ for details.

In [1]:

# Setting CUBLAS for reproducibility if you train on GPU
%env CUBLAS_WORKSPACE_CONFIG=:16:8

# Setting CUBLAS for reproducibility if you train on GPU %env CUBLAS_WORKSPACE_CONFIG=:16:8

env: CUBLAS_WORKSPACE_CONFIG=:16:8

Explain Step Tutorial¶

What you'll learn¶

1. Theory Primer
   
2. How to run explain step
   
3. How to customize the explain step
   
4. Using LLMs to get biological insights of the latent dimensions
   

1) Theory Primer¶

TODO

2) How To Perform the Explain Step¶

First, we need to run an AUTOENCODIX pipeline, Ontix is a very good choice here.

❗❗ Requirements: Getting Tutorial Data ❗❗¶

The data for this tutorial is hosted on Hugging Face Hub (autoencodix/tcga) and is downloaded automatically in the cell below on first run.

Extra 2: Get correct path¶

We assume you are in the root of the package. The following code ensures that the correct paths are used. [1] Tutorials/DeepDives/ConfigTutorial.ipynb

In [ ]:

import os

p = os.getcwd()
d = "autoencodix_package"
if d not in p:
    raise FileNotFoundError(f"'{d}' not found in path: {p}")
os.chdir(os.sep.join(p.split(os.sep)[: p.split(os.sep).index(d) + 1]))
print(f"Changed to: {os.getcwd()}")

# ---------------------------------------------------------------------
# Data is hosted on Hugging Face Hub and downloaded (and locally cached)
# automatically, then placed under the paths used below
# ---------------------------------------------------------------------
import shutil
from huggingface_hub import hf_hub_download

HF_REPO_ID = "autoencodix/tcga"
os.makedirs("data/raw", exist_ok=True)
for hf_filename, local_name in [
    ("rna.parquet", "combined_rnaseq_formatted.parquet"),
    ("methylation.parquet", "combined_meth_formatted.parquet"),
    ("clinical.parquet", "combined_clin_formatted.parquet"),
]:
    downloaded_path = hf_hub_download(
        repo_id=HF_REPO_ID, repo_type="dataset", filename=hf_filename
    )
    shutil.copyfile(downloaded_path, os.path.join("data/raw", local_name))

import os p = os.getcwd() d = "autoencodix_package" if d not in p: raise FileNotFoundError(f"'{d}' not found in path: {p}") os.chdir(os.sep.join(p.split(os.sep)[: p.split(os.sep).index(d) + 1])) print(f"Changed to: {os.getcwd()}") # --------------------------------------------------------------------- # Data is hosted on Hugging Face Hub and downloaded (and locally cached) # automatically, then placed under the paths used below # --------------------------------------------------------------------- import shutil from huggingface_hub import hf_hub_download HF_REPO_ID = "autoencodix/tcga" os.makedirs("data/raw", exist_ok=True) for hf_filename, local_name in [ ("rna.parquet", "combined_rnaseq_formatted.parquet"), ("methylation.parquet", "combined_meth_formatted.parquet"), ("clinical.parquet", "combined_clin_formatted.parquet"), ]: downloaded_path = hf_hub_download( repo_id=HF_REPO_ID, repo_type="dataset", filename=hf_filename ) shutil.copyfile(downloaded_path, os.path.join("data/raw", local_name))

2a) Creating a synthetic ground truth¶

Feature importance by attribution scores is hard to quantify for its performance and reliability of outcome. Hence, we will introduce in this tutorial a little synthetic signal on selected genes of a specific chromosome only for a group of sample to check if they will be picked up by our xAI method. Hence, we will alter the gene expression and methylation intensities of some genes of the 21 Chromosome only for 'male' samples (compare also Ontix tutorial).

In [3]:

import numpy as np
import pandas as pd
### Test with synthetic signal ###

def synth_signal(df, feature_list, sample_list):
	max_q90 = df.stack().quantile(0.9) # Our signal expression
	for sample in sample_list:
			df.loc[sample, feature_list] = max_q90*4 + np.random.normal(loc=0, scale=max_q90 * 0.25) # Add some noise to the signal
	return df

# Some genes on chromosome 21 which we will alter
# be aware that some will be filtered out during preprocessing by variance filtering
feature_synth = [
	"102724219",	# 21:p12
	"105379499",	# 21:p12
	"246312",	# 21:q21.1
	"105372749",	# 21:q21.2
	"266917",	# 21:q21.2
	"5651",	# 21:q21.1
	"105369292",	# 21:q21.1
	"102724951",	# 21:p11.2
	"1525",	# 21:q21.1
	"101927843",	# 21:q21.1
	"378828",	# 21:q21.1
	"118421",	# 21:q21.3
	"284825",	# 21:q21.3
	"101927869",	# 21:q21.2
	"9875",	# 21:q22.11
	"100874059",	# 13:q12.3
	"100131902",	# 21:q22.11
	"58494",	# 21:q21.3
	"105372746",	# 21:q21.1
]

df_rna = pd.read_parquet("data/raw/"+"combined_rnaseq_formatted.parquet")
df_meth = pd.read_parquet("data/raw/"+"combined_meth_formatted.parquet")
df_anno = pd.read_parquet("data/raw/"+"combined_clin_formatted.parquet")

sample__intersection = df_rna.index.intersection(df_meth.index).intersection(df_anno.index)
sample_group = "Male"
sample_synth = df_anno.loc[df_anno["SEX"] == sample_group].index.intersection(sample__intersection).to_list()

df_rna = synth_signal(df_rna, feature_list=feature_synth, sample_list=sample_synth)
df_meth = synth_signal(df_meth, feature_list=feature_synth, sample_list=sample_synth)

# Save the modified dataframes
df_rna.to_parquet("data/raw/"+"combined_rnaseq_formatted_synth.parquet")
df_meth.to_parquet("data/raw/"+"combined_meth_formatted_synth.parquet")

import numpy as np import pandas as pd ### Test with synthetic signal ### def synth_signal(df, feature_list, sample_list): max_q90 = df.stack().quantile(0.9) # Our signal expression for sample in sample_list: df.loc[sample, feature_list] = max_q90*4 + np.random.normal(loc=0, scale=max_q90 * 0.25) # Add some noise to the signal return df # Some genes on chromosome 21 which we will alter # be aware that some will be filtered out during preprocessing by variance filtering feature_synth = [ "102724219", # 21:p12 "105379499", # 21:p12 "246312", # 21:q21.1 "105372749", # 21:q21.2 "266917", # 21:q21.2 "5651", # 21:q21.1 "105369292", # 21:q21.1 "102724951", # 21:p11.2 "1525", # 21:q21.1 "101927843", # 21:q21.1 "378828", # 21:q21.1 "118421", # 21:q21.3 "284825", # 21:q21.3 "101927869", # 21:q21.2 "9875", # 21:q22.11 "100874059", # 13:q12.3 "100131902", # 21:q22.11 "58494", # 21:q21.3 "105372746", # 21:q21.1 ] df_rna = pd.read_parquet("data/raw/"+"combined_rnaseq_formatted.parquet") df_meth = pd.read_parquet("data/raw/"+"combined_meth_formatted.parquet") df_anno = pd.read_parquet("data/raw/"+"combined_clin_formatted.parquet") sample__intersection = df_rna.index.intersection(df_meth.index).intersection(df_anno.index) sample_group = "Male" sample_synth = df_anno.loc[df_anno["SEX"] == sample_group].index.intersection(sample__intersection).to_list() df_rna = synth_signal(df_rna, feature_list=feature_synth, sample_list=sample_synth) df_meth = synth_signal(df_meth, feature_list=feature_synth, sample_list=sample_synth) # Save the modified dataframes df_rna.to_parquet("data/raw/"+"combined_rnaseq_formatted_synth.parquet") df_meth.to_parquet("data/raw/"+"combined_meth_formatted_synth.parquet")

We can check the result by plotting the gene expression distribution for the altered features (genes):

In [ ]:

# Plot the distribution of the modified features compared to all features
import matplotlib.pyplot as plt

plt.figure(figsize=(10, 6))
plt.hist(df_rna.loc[sample_synth,feature_synth].stack(), bins=100, alpha=0.5, label="Male", color="blue")
plt.hist(df_rna.loc[~df_rna.index.isin(sample_synth), feature_synth].stack(), bins=100, alpha=0.5, label="Female/unknown", color="orange")

plt.axvline(
	df_rna.stack().quantile(0.9),
	color="red",
	linestyle="dashed",
	linewidth=2,
	label="Injected Signal Threshold",
)
plt.title(f"Distribution of all injected features with Injected Signal")
plt.xlabel("Value")
plt.ylabel("Frequency")
plt.legend()
plt.show()

# Plot the distribution of the modified features compared to all features import matplotlib.pyplot as plt plt.figure(figsize=(10, 6)) plt.hist(df_rna.loc[sample_synth,feature_synth].stack(), bins=100, alpha=0.5, label="Male", color="blue") plt.hist(df_rna.loc[~df_rna.index.isin(sample_synth), feature_synth].stack(), bins=100, alpha=0.5, label="Female/unknown", color="orange") plt.axvline( df_rna.stack().quantile(0.9), color="red", linestyle="dashed", linewidth=2, label="Injected Signal Threshold", ) plt.title(f"Distribution of all injected features with Injected Signal") plt.xlabel("Value") plt.ylabel("Frequency") plt.legend() plt.show()

2b) Train an Ontix on the data set with synthetic signal.¶

In [5]:

import os
import autoencodix as acx
from autoencodix.configs.default_config import DataConfig, DataInfo, DataCase
from autoencodix.configs import OntixConfig

# ---------------------------------------------------------------------
# Paths
# ---------------------------------------------------------------------
data_root = "data/raw"
# rna_file = "combined_rnaseq_formatted.parquet"
# meth_file = "combined_meth_formatted.parquet"
# clin_file = "combined_clin_formatted.parquet"
rna_file = "combined_rnaseq_formatted_synth.parquet"
meth_file = "combined_meth_formatted_synth.parquet"
clin_file = "combined_clin_formatted.parquet"
ont_genelevel = "chromosome_ont_genelevel_ncbi.txt"
ont_hiddenlevel = "chromosome_ont_hiddenlevel.txt"
# ---------------------------------------------------------------------
# Define individual data modalities
# ---------------------------------------------------------------------
rna_info = DataInfo(
    file_path=os.path.join(data_root, rna_file),
    data_type="NUMERIC",
    filtering="VAR",
)

meth_info = DataInfo(
    file_path=os.path.join(data_root, meth_file),
    data_type="NUMERIC",
    filtering="VAR",
)

anno_info = DataInfo(
    file_path=os.path.join(data_root, clin_file),
    data_type="ANNOTATION",
)
# ---------------------------------------------------------------------
# Combine into DataConfig
# ---------------------------------------------------------------------
data_config = DataConfig(
    data_info={
        "RNA": rna_info,
        "METH": meth_info,
        "ANNO": anno_info,
    },
    annotation_columns=[
        # "CANCER_TYPE",
        "CANCER_TYPE_ACRONYM",
        # "TMB_NONSYNONYMOUS",
        # "AGE",
        # "OS_STATUS",
        # "GRADE",
        "SEX",
    ],
)

# ---------------------------------------------------------------------
# Define the full DefaultConfig (roughly equivalent to old cfg)
# ---------------------------------------------------------------------
ontix_config = OntixConfig(
    data_config=data_config,
    reproducible=True,
    global_seed=42,
    epochs=100,
    learning_rate=0.0005,
    batch_size=128,
    drop_p=0.3,
    k_filter=2000,
    latent_dim=6,
    # device="cpu",
    device="cuda",
    reconstruction_loss="mse",
    default_vae_loss="kl",
    beta=0.001,
    save_memory=False,
    scaling="MINMAX",
    train_ratio=0.7,
    test_ratio=0.2,
    valid_ratio=0.1,
)

# ---------------------------------------------------------------------
# Now pass into your Ontix object
# ---------------------------------------------------------------------

ont_files = [ont_hiddenlevel, ont_genelevel]
ont_files = [os.path.join(data_root, f) for f in ont_files]
ontix = acx.Ontix(
    ontologies=ont_files,
    config=ontix_config,
)

import os import autoencodix as acx from autoencodix.configs.default_config import DataConfig, DataInfo, DataCase from autoencodix.configs import OntixConfig # --------------------------------------------------------------------- # Paths # --------------------------------------------------------------------- data_root = "data/raw" # rna_file = "combined_rnaseq_formatted.parquet" # meth_file = "combined_meth_formatted.parquet" # clin_file = "combined_clin_formatted.parquet" rna_file = "combined_rnaseq_formatted_synth.parquet" meth_file = "combined_meth_formatted_synth.parquet" clin_file = "combined_clin_formatted.parquet" ont_genelevel = "chromosome_ont_genelevel_ncbi.txt" ont_hiddenlevel = "chromosome_ont_hiddenlevel.txt" # --------------------------------------------------------------------- # Define individual data modalities # --------------------------------------------------------------------- rna_info = DataInfo( file_path=os.path.join(data_root, rna_file), data_type="NUMERIC", filtering="VAR", ) meth_info = DataInfo( file_path=os.path.join(data_root, meth_file), data_type="NUMERIC", filtering="VAR", ) anno_info = DataInfo( file_path=os.path.join(data_root, clin_file), data_type="ANNOTATION", ) # --------------------------------------------------------------------- # Combine into DataConfig # --------------------------------------------------------------------- data_config = DataConfig( data_info={ "RNA": rna_info, "METH": meth_info, "ANNO": anno_info, }, annotation_columns=[ # "CANCER_TYPE", "CANCER_TYPE_ACRONYM", # "TMB_NONSYNONYMOUS", # "AGE", # "OS_STATUS", # "GRADE", "SEX", ], ) # --------------------------------------------------------------------- # Define the full DefaultConfig (roughly equivalent to old cfg) # --------------------------------------------------------------------- ontix_config = OntixConfig( data_config=data_config, reproducible=True, global_seed=42, epochs=100, learning_rate=0.0005, batch_size=128, drop_p=0.3, k_filter=2000, latent_dim=6, # device="cpu", device="cuda", reconstruction_loss="mse", default_vae_loss="kl", beta=0.001, save_memory=False, scaling="MINMAX", train_ratio=0.7, test_ratio=0.2, valid_ratio=0.1, ) # --------------------------------------------------------------------- # Now pass into your Ontix object # --------------------------------------------------------------------- ont_files = [ont_hiddenlevel, ont_genelevel] ont_files = [os.path.join(data_root, f) for f in ont_files] ontix = acx.Ontix( ontologies=ont_files, config=ontix_config, )

/home/ewald/Github/acx_main_releases/autoencodix_package/src/autoencodix/configs/default_config.py:433: UserWarning: annotation_columns in DataConfig is deprecated. Please set it directly in DefaultConfig instead.
  warnings.warn(

In [6]:

result = ontix.run()

result = ontix.run()

reading parquet: data/raw/combined_rnaseq_formatted_synth.parquet
reading parquet: data/raw/combined_meth_formatted_synth.parquet
reading parquet: data/raw/combined_clin_formatted.parquet
anno key: paired
Features in feature_order not found in all_feature_names: ['100133144', '10357', '10431', '155060', '390284', '57714', '645851', '653553', '729884', '246182', '119385', '653268', '728404', '200810', '138649', '441425', '728747', '729171', '23520', '303', '304', '305', '244', '375719', '441432', '503640', '641522', '432369', '92270', '6791', '85319', '606', '23629', '286076', '414235', '170393', '255352', '283422', '283416', '374467', '84837', '400223', '650662', '280655', '283651', '80035', '283687', '196968', '284185', '147429', '147525', '494514', '574036', '284573', '149469', '84791', '253868', '284836', '54094', '54067', '282566', '114041', '114043', '149992', '55267', '29798', '348738', '339942', '93556', '646450', '317648', '92070', '79614', '153571', '116349', '441108', '285679', '85411', '653483', '387097', '79992', '129790', '113763', '158228', '445577', '158314', '157983', '120329', '348254', '221016', '348249', '643253', '1038', '94158', '100130418', '1057', '729338', '386593', '152302', '440508', '1197', '8418', '646300', '285464', '440359', '161635', '646243', '440224', '3580', '170063', '645090', '653687', '1564', '440081', '729582', '222161', '374387', '79469', '285987', '100128285', '100132911', '503645', '554236', '574029', '441032', '196549', '100131454', '326342', '347918', '8475', '284729', '145165', '387071', '728262', '100132403', '728882', '55747', '548321', '100132948', '439965', '414241', '55855', '339521', '100133172', '440078', '100132923', '692224', '9103', '619190', '80307', '2282', '79667', '80094', '401237', '200058', '255031', '649446', '730971', '283011', '202020', '388685', '388182', '645644', '392490', '402483', '399844', '283102', '100036519', '284802', '2498', '388965', '389523', '492303', '2679', '645367', '374650', '222611', '728932', '442245', '2952', '653238', '401375', '375513', '653188', '767811', '768096', '594842', '352961', '80867', '54435', '80868', '400322', '440362', '100128124', '3136', '3137', '10151', '378465', '391634', '664618', '343477', '3311', '84099', '91353', '644619', '3653', '389293', '387628', '654466', '100132341', '10748', '22973', '11026', '100271835', '100101266', '100124692', '100126784', '100128288', '100128292', '100128542', '100128573', '100128822', '100128842', '100129034', '100129387', '100129637', '100130238', '100130557', '100130581', '100130872', '100130932', '100130987', '100131551', '100132111', '100132287', '100132707', '100132724', '100132832', '100133161', '100133331', '100133669', '100133991', '100134868', '100144604', '100170939', '100188947', '100190939', '100190940', '100190986', '100216545', '100270710', '100270746', '100271722', '100271836', '100272146', '100272216', '100272217', '100272228', '100286793', '100287227', '100302640', '100302650', '115110', '143188', '143666', '144438', '146880', '148696', '149134', '150381', '151009', '151162', '162632', '168474', '202181', '219347', '220429', '220594', '222699', '253724', '255167', '256880', '282997', '283050', '283070', '283922', '284009', '284232', '284440', '284441', '284578', '285033', '285359', '285548', '286002', '286367', '338758', '339047', '341056', '347376', '349114', '349196', '374491', '387646', '387647', '388152', '388242', '388955', '389458', '389705', '391322', '399744', '399815', '400027', '400657', '400927', '401010', '401052', '401127', '401588', '402377', '407835', '440354', '440461', '440905', '441089', '441204', '441208', '441454', '441455', '442308', '442454', '442459', '493754', '550112', '606724', '641298', '642826', '642846', '643387', '643719', '644165', '644172', '644936', '645431', '646214', '646471', '646762', '646999', '647288', '647859', '648740', '650368', '650623', '651250', '653113', '653501', '653566', '727896', '728024', '728264', '728323', '728554', '728613', '728640', '728723', '728758', '728855', '728875', '728989', '729176', '729234', '729375', '729603', '729799', '731789', '80154', '90110', '90246', '90784', '90834', '91450', '96610', '29931', '55073', '147172', '767558', '79136', '653639', '378938', '442229', '4213', '84815', '85009', '84848', '84849', '114130', '113691', '81854', '90768', '401884', '389538', '4276', '84953', '10934', '359821', '729633', '401827', '11209', '11223', '100129405', '55545', '326343', '4500', '339483', '84176', '83955', '51471', '728936', '285622', '343505', '100188954', '80161', '283981', '79854', '55389', '440072', '400508', '114915', '64493', '283458', '100129354', '119369', '10896', '4950', '441295', '26636', '647033', '728773', '8123', '347746', '54661', '84054', '171423', '728939', '267004', '375133', '206426', '266971', '440456', '441194', '5379', '5383', '646074', '25812', '29797', '5505', '55370', '100169750', '28997', '29053', '377047', '100130889', '26255', '5820', '83956', '401331', '10740', '379013', '286140', '283345', '100129424', '402176', '644128', '649946', '284942', '222901', '132241', '85495', '376693', '728963', '441502', '256355', '653162', '730092', '653390', '84127', '440352', '54581', '677769', '727956', '728609', '641977', '201175', '440044', '65012', '9906', '81893', '387254', '342615', '751867', '387066', '641638', '100093630', '735301', '677821', '6044', '6043', '652966', '100033431', '100033820', '100033416', '677850', '64173', '441251', '389517', '10638', '23145', '400410', '54441', '442582', '442578', '7955', '441394', '474338', '440423', '767557', '654341', '27004', '7012', '374500', '286102', '7117', '7151', '7152', '11257', '728402', '646405', '117852', '154754', '128854', '56604', '643224', '606551', '652995', '374666', '653635', '171022', '339005', '440253', '51352', '150244', '7696', '7754', '387328', '651302', '388507', '401303', '664701', '401898', '221584']
Features in feature_order not found in all_feature_names: ['100033413', '100130426', '100133144', '1038', '10431', '10748', '10934', '114038', '136542', '145165', '155060', '170393', '200058', '221016', '222611', '255352', '25812', '2679', '283416', '285759', '317712', '348738', '387328', '388685', '389649', '391343', '4276', '4950', '51352', '553137', '57714', '5820', '645851', '65012', '652919', '7012', '728603', '729884', '7696', '79614', '8123', '84127', '90011', '9906']
Reproducibility settings for device cuda are not implemented or necessary i.e. for cpu.
Ontix checks:
All possible feature names length: 27248
Feature order length: 2000
Feature names without filtering: 2000
Mask layer 0 with shape torch.Size([817, 25]) and 817.0 connections
Mask layer 1 with shape torch.Size([2000, 817]) and 1949.0 connections
Latent Dim: 25
Epoch 1 - Train Loss: 278.7943
Sub-losses: recon_loss: 278.7943, var_loss: 0.0000, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 1 - Valid Loss: 263.9739
Sub-losses: recon_loss: 263.9739, var_loss: 0.0000, anneal_factor: 0.0000, effective_beta_factor: 0.0000
Epoch 2 - Train Loss: 258.0811
Sub-losses: recon_loss: 258.0811, var_loss: 0.0000, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 2 - Valid Loss: 241.9943
Sub-losses: recon_loss: 241.9943, var_loss: 0.0000, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 3 - Train Loss: 234.7858
Sub-losses: recon_loss: 234.7858, var_loss: 0.0000, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 3 - Valid Loss: 221.4323
Sub-losses: recon_loss: 221.4323, var_loss: 0.0000, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 4 - Train Loss: 210.3714
Sub-losses: recon_loss: 210.3714, var_loss: 0.0000, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 4 - Valid Loss: 195.7710
Sub-losses: recon_loss: 195.7710, var_loss: 0.0000, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 5 - Train Loss: 186.3277
Sub-losses: recon_loss: 186.3277, var_loss: 0.0000, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 5 - Valid Loss: 174.5568
Sub-losses: recon_loss: 174.5568, var_loss: 0.0000, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 6 - Train Loss: 162.8373
Sub-losses: recon_loss: 162.8373, var_loss: 0.0000, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 6 - Valid Loss: 153.4373
Sub-losses: recon_loss: 153.4373, var_loss: 0.0000, anneal_factor: 0.0001, effective_beta_factor: 0.0000
Epoch 7 - Train Loss: 141.4351
Sub-losses: recon_loss: 141.4350, var_loss: 0.0000, anneal_factor: 0.0002, effective_beta_factor: 0.0000
Epoch 7 - Valid Loss: 130.9373
Sub-losses: recon_loss: 130.9373, var_loss: 0.0000, anneal_factor: 0.0002, effective_beta_factor: 0.0000
Epoch 8 - Train Loss: 122.7023
Sub-losses: recon_loss: 122.7022, var_loss: 0.0000, anneal_factor: 0.0002, effective_beta_factor: 0.0000
Epoch 8 - Valid Loss: 114.6288
Sub-losses: recon_loss: 114.6288, var_loss: 0.0000, anneal_factor: 0.0002, effective_beta_factor: 0.0000
Epoch 9 - Train Loss: 107.9473
Sub-losses: recon_loss: 107.9473, var_loss: 0.0000, anneal_factor: 0.0002, effective_beta_factor: 0.0000
Epoch 9 - Valid Loss: 104.2357
Sub-losses: recon_loss: 104.2357, var_loss: 0.0000, anneal_factor: 0.0002, effective_beta_factor: 0.0000
Epoch 10 - Train Loss: 96.3132
Sub-losses: recon_loss: 96.3132, var_loss: 0.0000, anneal_factor: 0.0003, effective_beta_factor: 0.0000
Epoch 10 - Valid Loss: 93.2880
Sub-losses: recon_loss: 93.2880, var_loss: 0.0000, anneal_factor: 0.0003, effective_beta_factor: 0.0000
Epoch 11 - Train Loss: 87.7343
Sub-losses: recon_loss: 87.7342, var_loss: 0.0000, anneal_factor: 0.0003, effective_beta_factor: 0.0000
Epoch 11 - Valid Loss: 87.6504
Sub-losses: recon_loss: 87.6504, var_loss: 0.0000, anneal_factor: 0.0003, effective_beta_factor: 0.0000
Epoch 12 - Train Loss: 81.6043
Sub-losses: recon_loss: 81.6042, var_loss: 0.0001, anneal_factor: 0.0004, effective_beta_factor: 0.0000
Epoch 12 - Valid Loss: 80.8453
Sub-losses: recon_loss: 80.8452, var_loss: 0.0000, anneal_factor: 0.0004, effective_beta_factor: 0.0000
Epoch 13 - Train Loss: 76.9594
Sub-losses: recon_loss: 76.9593, var_loss: 0.0001, anneal_factor: 0.0005, effective_beta_factor: 0.0000
Epoch 13 - Valid Loss: 75.6597
Sub-losses: recon_loss: 75.6596, var_loss: 0.0001, anneal_factor: 0.0005, effective_beta_factor: 0.0000
Epoch 14 - Train Loss: 73.3621
Sub-losses: recon_loss: 73.3620, var_loss: 0.0001, anneal_factor: 0.0006, effective_beta_factor: 0.0000
Epoch 14 - Valid Loss: 73.0839
Sub-losses: recon_loss: 73.0838, var_loss: 0.0001, anneal_factor: 0.0006, effective_beta_factor: 0.0000
Epoch 15 - Train Loss: 71.0181
Sub-losses: recon_loss: 71.0179, var_loss: 0.0001, anneal_factor: 0.0007, effective_beta_factor: 0.0000
Epoch 15 - Valid Loss: 70.6346
Sub-losses: recon_loss: 70.6345, var_loss: 0.0001, anneal_factor: 0.0007, effective_beta_factor: 0.0000
Epoch 16 - Train Loss: 68.8574
Sub-losses: recon_loss: 68.8573, var_loss: 0.0002, anneal_factor: 0.0009, effective_beta_factor: 0.0000
Epoch 16 - Valid Loss: 68.2907
Sub-losses: recon_loss: 68.2906, var_loss: 0.0001, anneal_factor: 0.0009, effective_beta_factor: 0.0000
Epoch 17 - Train Loss: 67.7698
Sub-losses: recon_loss: 67.7696, var_loss: 0.0002, anneal_factor: 0.0011, effective_beta_factor: 0.0000
Epoch 17 - Valid Loss: 65.8907
Sub-losses: recon_loss: 65.8905, var_loss: 0.0002, anneal_factor: 0.0011, effective_beta_factor: 0.0000
Epoch 18 - Train Loss: 66.8111
Sub-losses: recon_loss: 66.8108, var_loss: 0.0003, anneal_factor: 0.0014, effective_beta_factor: 0.0000
Epoch 18 - Valid Loss: 64.7314
Sub-losses: recon_loss: 64.7312, var_loss: 0.0003, anneal_factor: 0.0014, effective_beta_factor: 0.0000
Epoch 19 - Train Loss: 66.0533
Sub-losses: recon_loss: 66.0529, var_loss: 0.0004, anneal_factor: 0.0017, effective_beta_factor: 0.0000
Epoch 19 - Valid Loss: 62.7016
Sub-losses: recon_loss: 62.7012, var_loss: 0.0003, anneal_factor: 0.0017, effective_beta_factor: 0.0000
Epoch 20 - Train Loss: 65.7814
Sub-losses: recon_loss: 65.7810, var_loss: 0.0005, anneal_factor: 0.0020, effective_beta_factor: 0.0000
Epoch 20 - Valid Loss: 64.0714
Sub-losses: recon_loss: 64.0710, var_loss: 0.0004, anneal_factor: 0.0020, effective_beta_factor: 0.0000
Epoch 21 - Train Loss: 64.8267
Sub-losses: recon_loss: 64.8261, var_loss: 0.0006, anneal_factor: 0.0025, effective_beta_factor: 0.0000
Epoch 21 - Valid Loss: 62.9184
Sub-losses: recon_loss: 62.9179, var_loss: 0.0005, anneal_factor: 0.0025, effective_beta_factor: 0.0000
Epoch 22 - Train Loss: 64.5225
Sub-losses: recon_loss: 64.5218, var_loss: 0.0007, anneal_factor: 0.0030, effective_beta_factor: 0.0000
Epoch 22 - Valid Loss: 61.9442
Sub-losses: recon_loss: 61.9435, var_loss: 0.0007, anneal_factor: 0.0030, effective_beta_factor: 0.0000
Epoch 23 - Train Loss: 63.7980
Sub-losses: recon_loss: 63.7971, var_loss: 0.0009, anneal_factor: 0.0037, effective_beta_factor: 0.0000
Epoch 23 - Valid Loss: 62.0157
Sub-losses: recon_loss: 62.0148, var_loss: 0.0009, anneal_factor: 0.0037, effective_beta_factor: 0.0000
Epoch 24 - Train Loss: 63.8265
Sub-losses: recon_loss: 63.8253, var_loss: 0.0012, anneal_factor: 0.0045, effective_beta_factor: 0.0000
Epoch 24 - Valid Loss: 60.6831
Sub-losses: recon_loss: 60.6820, var_loss: 0.0011, anneal_factor: 0.0045, effective_beta_factor: 0.0000
Epoch 25 - Train Loss: 63.0792
Sub-losses: recon_loss: 63.0777, var_loss: 0.0015, anneal_factor: 0.0055, effective_beta_factor: 0.0000
Epoch 25 - Valid Loss: 61.3274
Sub-losses: recon_loss: 61.3260, var_loss: 0.0014, anneal_factor: 0.0055, effective_beta_factor: 0.0000
Epoch 26 - Train Loss: 62.6624
Sub-losses: recon_loss: 62.6606, var_loss: 0.0018, anneal_factor: 0.0067, effective_beta_factor: 0.0000
Epoch 26 - Valid Loss: 59.6285
Sub-losses: recon_loss: 59.6268, var_loss: 0.0017, anneal_factor: 0.0067, effective_beta_factor: 0.0000
Epoch 27 - Train Loss: 62.8538
Sub-losses: recon_loss: 62.8515, var_loss: 0.0023, anneal_factor: 0.0082, effective_beta_factor: 0.0000
Epoch 27 - Valid Loss: 59.9669
Sub-losses: recon_loss: 59.9648, var_loss: 0.0021, anneal_factor: 0.0082, effective_beta_factor: 0.0000
Epoch 28 - Train Loss: 61.9454
Sub-losses: recon_loss: 61.9426, var_loss: 0.0028, anneal_factor: 0.0100, effective_beta_factor: 0.0000
Epoch 28 - Valid Loss: 58.5268
Sub-losses: recon_loss: 58.5242, var_loss: 0.0027, anneal_factor: 0.0100, effective_beta_factor: 0.0000
Epoch 29 - Train Loss: 61.8808
Sub-losses: recon_loss: 61.8773, var_loss: 0.0035, anneal_factor: 0.0121, effective_beta_factor: 0.0000
Epoch 29 - Valid Loss: 58.1750
Sub-losses: recon_loss: 58.1717, var_loss: 0.0033, anneal_factor: 0.0121, effective_beta_factor: 0.0000
Epoch 30 - Train Loss: 61.4542
Sub-losses: recon_loss: 61.4498, var_loss: 0.0044, anneal_factor: 0.0148, effective_beta_factor: 0.0000
Epoch 30 - Valid Loss: 57.9234
Sub-losses: recon_loss: 57.9193, var_loss: 0.0041, anneal_factor: 0.0148, effective_beta_factor: 0.0000
Epoch 31 - Train Loss: 61.2463
Sub-losses: recon_loss: 61.2408, var_loss: 0.0055, anneal_factor: 0.0180, effective_beta_factor: 0.0000
Epoch 31 - Valid Loss: 58.2266
Sub-losses: recon_loss: 58.2215, var_loss: 0.0051, anneal_factor: 0.0180, effective_beta_factor: 0.0000
Epoch 32 - Train Loss: 60.8242
Sub-losses: recon_loss: 60.8174, var_loss: 0.0068, anneal_factor: 0.0219, effective_beta_factor: 0.0000
Epoch 32 - Valid Loss: 57.4518
Sub-losses: recon_loss: 57.4456, var_loss: 0.0062, anneal_factor: 0.0219, effective_beta_factor: 0.0000
Epoch 33 - Train Loss: 60.1056
Sub-losses: recon_loss: 60.0972, var_loss: 0.0084, anneal_factor: 0.0266, effective_beta_factor: 0.0000
Epoch 33 - Valid Loss: 56.9922
Sub-losses: recon_loss: 56.9844, var_loss: 0.0078, anneal_factor: 0.0266, effective_beta_factor: 0.0000
Epoch 34 - Train Loss: 60.1895
Sub-losses: recon_loss: 60.1793, var_loss: 0.0102, anneal_factor: 0.0323, effective_beta_factor: 0.0000
Epoch 34 - Valid Loss: 56.3890
Sub-losses: recon_loss: 56.3795, var_loss: 0.0095, anneal_factor: 0.0323, effective_beta_factor: 0.0000
Epoch 35 - Train Loss: 59.3472
Sub-losses: recon_loss: 59.3345, var_loss: 0.0127, anneal_factor: 0.0392, effective_beta_factor: 0.0000
Epoch 35 - Valid Loss: 55.9338
Sub-losses: recon_loss: 55.9219, var_loss: 0.0119, anneal_factor: 0.0392, effective_beta_factor: 0.0000
Epoch 36 - Train Loss: 59.1634
Sub-losses: recon_loss: 59.1477, var_loss: 0.0156, anneal_factor: 0.0474, effective_beta_factor: 0.0000
Epoch 36 - Valid Loss: 56.0696
Sub-losses: recon_loss: 56.0555, var_loss: 0.0141, anneal_factor: 0.0474, effective_beta_factor: 0.0000
Epoch 37 - Train Loss: 58.7254
Sub-losses: recon_loss: 58.7065, var_loss: 0.0189, anneal_factor: 0.0573, effective_beta_factor: 0.0001
Epoch 37 - Valid Loss: 55.5548
Sub-losses: recon_loss: 55.5372, var_loss: 0.0176, anneal_factor: 0.0573, effective_beta_factor: 0.0001
Epoch 38 - Train Loss: 58.9289
Sub-losses: recon_loss: 58.9056, var_loss: 0.0233, anneal_factor: 0.0691, effective_beta_factor: 0.0001
Epoch 38 - Valid Loss: 54.9897
Sub-losses: recon_loss: 54.9686, var_loss: 0.0211, anneal_factor: 0.0691, effective_beta_factor: 0.0001
Epoch 39 - Train Loss: 58.7652
Sub-losses: recon_loss: 58.7368, var_loss: 0.0284, anneal_factor: 0.0832, effective_beta_factor: 0.0001
Epoch 39 - Valid Loss: 54.7179
Sub-losses: recon_loss: 54.6915, var_loss: 0.0264, anneal_factor: 0.0832, effective_beta_factor: 0.0001
Epoch 40 - Train Loss: 57.5489
Sub-losses: recon_loss: 57.5144, var_loss: 0.0345, anneal_factor: 0.0998, effective_beta_factor: 0.0001
Epoch 40 - Valid Loss: 54.0976
Sub-losses: recon_loss: 54.0650, var_loss: 0.0326, anneal_factor: 0.0998, effective_beta_factor: 0.0001
Epoch 41 - Train Loss: 57.7241
Sub-losses: recon_loss: 57.6822, var_loss: 0.0418, anneal_factor: 0.1192, effective_beta_factor: 0.0001
Epoch 41 - Valid Loss: 54.1641
Sub-losses: recon_loss: 54.1246, var_loss: 0.0395, anneal_factor: 0.1192, effective_beta_factor: 0.0001
Epoch 42 - Train Loss: 57.4466
Sub-losses: recon_loss: 57.3968, var_loss: 0.0498, anneal_factor: 0.1419, effective_beta_factor: 0.0001
Epoch 42 - Valid Loss: 52.9623
Sub-losses: recon_loss: 52.9152, var_loss: 0.0470, anneal_factor: 0.1419, effective_beta_factor: 0.0001
Epoch 43 - Train Loss: 57.8669
Sub-losses: recon_loss: 57.8066, var_loss: 0.0603, anneal_factor: 0.1680, effective_beta_factor: 0.0002
Epoch 43 - Valid Loss: 53.8612
Sub-losses: recon_loss: 53.8056, var_loss: 0.0556, anneal_factor: 0.1680, effective_beta_factor: 0.0002
Epoch 44 - Train Loss: 56.8796
Sub-losses: recon_loss: 56.8087, var_loss: 0.0709, anneal_factor: 0.1978, effective_beta_factor: 0.0002
Epoch 44 - Valid Loss: 52.7708
Sub-losses: recon_loss: 52.7044, var_loss: 0.0663, anneal_factor: 0.1978, effective_beta_factor: 0.0002
Epoch 45 - Train Loss: 57.0952
Sub-losses: recon_loss: 57.0105, var_loss: 0.0848, anneal_factor: 0.2315, effective_beta_factor: 0.0002
Epoch 45 - Valid Loss: 54.6888
Sub-losses: recon_loss: 54.6086, var_loss: 0.0802, anneal_factor: 0.2315, effective_beta_factor: 0.0002
Epoch 46 - Train Loss: 56.6543
Sub-losses: recon_loss: 56.5555, var_loss: 0.0987, anneal_factor: 0.2689, effective_beta_factor: 0.0003
Epoch 46 - Valid Loss: 52.1609
Sub-losses: recon_loss: 52.0680, var_loss: 0.0929, anneal_factor: 0.2689, effective_beta_factor: 0.0003
Epoch 47 - Train Loss: 56.3723
Sub-losses: recon_loss: 56.2566, var_loss: 0.1157, anneal_factor: 0.3100, effective_beta_factor: 0.0003
Epoch 47 - Valid Loss: 51.8487
Sub-losses: recon_loss: 51.7389, var_loss: 0.1098, anneal_factor: 0.3100, effective_beta_factor: 0.0003
Epoch 48 - Train Loss: 55.5826
Sub-losses: recon_loss: 55.4487, var_loss: 0.1339, anneal_factor: 0.3543, effective_beta_factor: 0.0004
Epoch 48 - Valid Loss: 52.2896
Sub-losses: recon_loss: 52.1644, var_loss: 0.1252, anneal_factor: 0.3543, effective_beta_factor: 0.0004
Epoch 49 - Train Loss: 56.5611
Sub-losses: recon_loss: 56.4074, var_loss: 0.1537, anneal_factor: 0.4013, effective_beta_factor: 0.0004
Epoch 49 - Valid Loss: 52.0288
Sub-losses: recon_loss: 51.8849, var_loss: 0.1438, anneal_factor: 0.4013, effective_beta_factor: 0.0004
Epoch 50 - Train Loss: 55.6162
Sub-losses: recon_loss: 55.4424, var_loss: 0.1738, anneal_factor: 0.4502, effective_beta_factor: 0.0005
Epoch 50 - Valid Loss: 52.1462
Sub-losses: recon_loss: 51.9821, var_loss: 0.1641, anneal_factor: 0.4502, effective_beta_factor: 0.0005
Epoch 51 - Train Loss: 56.2650
Sub-losses: recon_loss: 56.0700, var_loss: 0.1950, anneal_factor: 0.5000, effective_beta_factor: 0.0005
Epoch 51 - Valid Loss: 51.2220
Sub-losses: recon_loss: 51.0397, var_loss: 0.1823, anneal_factor: 0.5000, effective_beta_factor: 0.0005
Epoch 52 - Train Loss: 55.3489
Sub-losses: recon_loss: 55.1351, var_loss: 0.2137, anneal_factor: 0.5498, effective_beta_factor: 0.0005
Epoch 52 - Valid Loss: 51.1573
Sub-losses: recon_loss: 50.9511, var_loss: 0.2062, anneal_factor: 0.5498, effective_beta_factor: 0.0005
Epoch 53 - Train Loss: 55.0340
Sub-losses: recon_loss: 54.7970, var_loss: 0.2370, anneal_factor: 0.5987, effective_beta_factor: 0.0006
Epoch 53 - Valid Loss: 50.8726
Sub-losses: recon_loss: 50.6502, var_loss: 0.2224, anneal_factor: 0.5987, effective_beta_factor: 0.0006
Epoch 54 - Train Loss: 54.8366
Sub-losses: recon_loss: 54.5801, var_loss: 0.2565, anneal_factor: 0.6457, effective_beta_factor: 0.0006
Epoch 54 - Valid Loss: 51.0262
Sub-losses: recon_loss: 50.7822, var_loss: 0.2440, anneal_factor: 0.6457, effective_beta_factor: 0.0006
Epoch 55 - Train Loss: 55.5876
Sub-losses: recon_loss: 55.3126, var_loss: 0.2749, anneal_factor: 0.6900, effective_beta_factor: 0.0007
Epoch 55 - Valid Loss: 50.6547
Sub-losses: recon_loss: 50.3894, var_loss: 0.2652, anneal_factor: 0.6900, effective_beta_factor: 0.0007
Epoch 56 - Train Loss: 54.5468
Sub-losses: recon_loss: 54.2512, var_loss: 0.2956, anneal_factor: 0.7311, effective_beta_factor: 0.0007
Epoch 56 - Valid Loss: 50.3729
Sub-losses: recon_loss: 50.0919, var_loss: 0.2810, anneal_factor: 0.7311, effective_beta_factor: 0.0007
Epoch 57 - Train Loss: 54.3138
Sub-losses: recon_loss: 54.0004, var_loss: 0.3134, anneal_factor: 0.7685, effective_beta_factor: 0.0008
Epoch 57 - Valid Loss: 50.7423
Sub-losses: recon_loss: 50.4447, var_loss: 0.2975, anneal_factor: 0.7685, effective_beta_factor: 0.0008
Epoch 58 - Train Loss: 54.8566
Sub-losses: recon_loss: 54.5272, var_loss: 0.3294, anneal_factor: 0.8022, effective_beta_factor: 0.0008
Epoch 58 - Valid Loss: 50.7776
Sub-losses: recon_loss: 50.4669, var_loss: 0.3108, anneal_factor: 0.8022, effective_beta_factor: 0.0008
Epoch 59 - Train Loss: 54.4016
Sub-losses: recon_loss: 54.0516, var_loss: 0.3500, anneal_factor: 0.8320, effective_beta_factor: 0.0008
Epoch 59 - Valid Loss: 52.2446
Sub-losses: recon_loss: 51.9173, var_loss: 0.3272, anneal_factor: 0.8320, effective_beta_factor: 0.0008
Epoch 60 - Train Loss: 53.9935
Sub-losses: recon_loss: 53.6356, var_loss: 0.3579, anneal_factor: 0.8581, effective_beta_factor: 0.0009
Epoch 60 - Valid Loss: 49.6073
Sub-losses: recon_loss: 49.2704, var_loss: 0.3369, anneal_factor: 0.8581, effective_beta_factor: 0.0009
Epoch 61 - Train Loss: 53.6060
Sub-losses: recon_loss: 53.2382, var_loss: 0.3678, anneal_factor: 0.8808, effective_beta_factor: 0.0009
Epoch 61 - Valid Loss: 49.8455
Sub-losses: recon_loss: 49.4872, var_loss: 0.3583, anneal_factor: 0.8808, effective_beta_factor: 0.0009
Epoch 62 - Train Loss: 54.2903
Sub-losses: recon_loss: 53.9064, var_loss: 0.3839, anneal_factor: 0.9002, effective_beta_factor: 0.0009
Epoch 62 - Valid Loss: 48.9368
Sub-losses: recon_loss: 48.5830, var_loss: 0.3538, anneal_factor: 0.9002, effective_beta_factor: 0.0009
Epoch 63 - Train Loss: 53.3513
Sub-losses: recon_loss: 52.9630, var_loss: 0.3883, anneal_factor: 0.9168, effective_beta_factor: 0.0009
Epoch 63 - Valid Loss: 49.2949
Sub-losses: recon_loss: 48.9299, var_loss: 0.3650, anneal_factor: 0.9168, effective_beta_factor: 0.0009
Epoch 64 - Train Loss: 54.1783
Sub-losses: recon_loss: 53.7820, var_loss: 0.3963, anneal_factor: 0.9309, effective_beta_factor: 0.0009
Epoch 64 - Valid Loss: 49.1743
Sub-losses: recon_loss: 48.8010, var_loss: 0.3733, anneal_factor: 0.9309, effective_beta_factor: 0.0009
Epoch 65 - Train Loss: 53.3164
Sub-losses: recon_loss: 52.9105, var_loss: 0.4059, anneal_factor: 0.9427, effective_beta_factor: 0.0009
Epoch 65 - Valid Loss: 48.5832
Sub-losses: recon_loss: 48.1970, var_loss: 0.3862, anneal_factor: 0.9427, effective_beta_factor: 0.0009
Epoch 66 - Train Loss: 52.4245
Sub-losses: recon_loss: 52.0154, var_loss: 0.4091, anneal_factor: 0.9526, effective_beta_factor: 0.0010
Epoch 66 - Valid Loss: 49.1806
Sub-losses: recon_loss: 48.7807, var_loss: 0.3998, anneal_factor: 0.9526, effective_beta_factor: 0.0010
Epoch 67 - Train Loss: 54.1743
Sub-losses: recon_loss: 53.7539, var_loss: 0.4205, anneal_factor: 0.9608, effective_beta_factor: 0.0010
Epoch 67 - Valid Loss: 48.5802
Sub-losses: recon_loss: 48.1872, var_loss: 0.3930, anneal_factor: 0.9608, effective_beta_factor: 0.0010
Epoch 68 - Train Loss: 52.8245
Sub-losses: recon_loss: 52.4013, var_loss: 0.4233, anneal_factor: 0.9677, effective_beta_factor: 0.0010
Epoch 68 - Valid Loss: 48.4582
Sub-losses: recon_loss: 48.0631, var_loss: 0.3951, anneal_factor: 0.9677, effective_beta_factor: 0.0010
Epoch 69 - Train Loss: 52.5422
Sub-losses: recon_loss: 52.1122, var_loss: 0.4300, anneal_factor: 0.9734, effective_beta_factor: 0.0010
Epoch 69 - Valid Loss: 47.9845
Sub-losses: recon_loss: 47.5862, var_loss: 0.3983, anneal_factor: 0.9734, effective_beta_factor: 0.0010
Epoch 70 - Train Loss: 52.9821
Sub-losses: recon_loss: 52.5511, var_loss: 0.4310, anneal_factor: 0.9781, effective_beta_factor: 0.0010
Epoch 70 - Valid Loss: 48.8698
Sub-losses: recon_loss: 48.4592, var_loss: 0.4107, anneal_factor: 0.9781, effective_beta_factor: 0.0010
Epoch 71 - Train Loss: 52.4099
Sub-losses: recon_loss: 51.9716, var_loss: 0.4383, anneal_factor: 0.9820, effective_beta_factor: 0.0010
Epoch 71 - Valid Loss: 47.9132
Sub-losses: recon_loss: 47.4986, var_loss: 0.4147, anneal_factor: 0.9820, effective_beta_factor: 0.0010
Epoch 72 - Train Loss: 52.9619
Sub-losses: recon_loss: 52.5193, var_loss: 0.4426, anneal_factor: 0.9852, effective_beta_factor: 0.0010
Epoch 72 - Valid Loss: 48.6012
Sub-losses: recon_loss: 48.1829, var_loss: 0.4183, anneal_factor: 0.9852, effective_beta_factor: 0.0010
Epoch 73 - Train Loss: 52.2639
Sub-losses: recon_loss: 51.8241, var_loss: 0.4398, anneal_factor: 0.9879, effective_beta_factor: 0.0010
Epoch 73 - Valid Loss: 47.9614
Sub-losses: recon_loss: 47.5463, var_loss: 0.4152, anneal_factor: 0.9879, effective_beta_factor: 0.0010
Epoch 74 - Train Loss: 52.3250
Sub-losses: recon_loss: 51.8811, var_loss: 0.4439, anneal_factor: 0.9900, effective_beta_factor: 0.0010
Epoch 74 - Valid Loss: 48.0939
Sub-losses: recon_loss: 47.6821, var_loss: 0.4118, anneal_factor: 0.9900, effective_beta_factor: 0.0010
Epoch 75 - Train Loss: 51.4451
Sub-losses: recon_loss: 50.9968, var_loss: 0.4483, anneal_factor: 0.9918, effective_beta_factor: 0.0010
Epoch 75 - Valid Loss: 47.5213
Sub-losses: recon_loss: 47.0946, var_loss: 0.4268, anneal_factor: 0.9918, effective_beta_factor: 0.0010
Epoch 76 - Train Loss: 51.4035
Sub-losses: recon_loss: 50.9543, var_loss: 0.4492, anneal_factor: 0.9933, effective_beta_factor: 0.0010
Epoch 76 - Valid Loss: 47.7936
Sub-losses: recon_loss: 47.3603, var_loss: 0.4333, anneal_factor: 0.9933, effective_beta_factor: 0.0010
Epoch 77 - Train Loss: 52.8587
Sub-losses: recon_loss: 52.4010, var_loss: 0.4576, anneal_factor: 0.9945, effective_beta_factor: 0.0010
Epoch 77 - Valid Loss: 47.7301
Sub-losses: recon_loss: 47.3061, var_loss: 0.4240, anneal_factor: 0.9945, effective_beta_factor: 0.0010
Epoch 78 - Train Loss: 51.9347
Sub-losses: recon_loss: 51.4696, var_loss: 0.4651, anneal_factor: 0.9955, effective_beta_factor: 0.0010
Epoch 78 - Valid Loss: 50.0970
Sub-losses: recon_loss: 49.6718, var_loss: 0.4252, anneal_factor: 0.9955, effective_beta_factor: 0.0010
Epoch 79 - Train Loss: 51.0861
Sub-losses: recon_loss: 50.6327, var_loss: 0.4534, anneal_factor: 0.9963, effective_beta_factor: 0.0010
Epoch 79 - Valid Loss: 47.1443
Sub-losses: recon_loss: 46.7183, var_loss: 0.4261, anneal_factor: 0.9963, effective_beta_factor: 0.0010
Epoch 80 - Train Loss: 51.4871
Sub-losses: recon_loss: 51.0241, var_loss: 0.4630, anneal_factor: 0.9970, effective_beta_factor: 0.0010
Epoch 80 - Valid Loss: 46.7771
Sub-losses: recon_loss: 46.3438, var_loss: 0.4332, anneal_factor: 0.9970, effective_beta_factor: 0.0010
Epoch 81 - Train Loss: 51.0046
Sub-losses: recon_loss: 50.5471, var_loss: 0.4574, anneal_factor: 0.9975, effective_beta_factor: 0.0010
Epoch 81 - Valid Loss: 46.3823
Sub-losses: recon_loss: 45.9480, var_loss: 0.4343, anneal_factor: 0.9975, effective_beta_factor: 0.0010
Epoch 82 - Train Loss: 50.6255
Sub-losses: recon_loss: 50.1609, var_loss: 0.4646, anneal_factor: 0.9980, effective_beta_factor: 0.0010
Epoch 82 - Valid Loss: 46.8554
Sub-losses: recon_loss: 46.4171, var_loss: 0.4382, anneal_factor: 0.9980, effective_beta_factor: 0.0010
Epoch 83 - Train Loss: 51.0016
Sub-losses: recon_loss: 50.5369, var_loss: 0.4647, anneal_factor: 0.9983, effective_beta_factor: 0.0010
Epoch 83 - Valid Loss: 47.2359
Sub-losses: recon_loss: 46.8030, var_loss: 0.4329, anneal_factor: 0.9983, effective_beta_factor: 0.0010
Epoch 84 - Train Loss: 51.3971
Sub-losses: recon_loss: 50.9282, var_loss: 0.4689, anneal_factor: 0.9986, effective_beta_factor: 0.0010
Epoch 84 - Valid Loss: 48.4162
Sub-losses: recon_loss: 47.9736, var_loss: 0.4427, anneal_factor: 0.9986, effective_beta_factor: 0.0010
Epoch 85 - Train Loss: 50.5306
Sub-losses: recon_loss: 50.0625, var_loss: 0.4681, anneal_factor: 0.9989, effective_beta_factor: 0.0010
Epoch 85 - Valid Loss: 47.4384
Sub-losses: recon_loss: 46.9907, var_loss: 0.4478, anneal_factor: 0.9989, effective_beta_factor: 0.0010
Epoch 86 - Train Loss: 51.1431
Sub-losses: recon_loss: 50.6683, var_loss: 0.4747, anneal_factor: 0.9991, effective_beta_factor: 0.0010
Epoch 86 - Valid Loss: 46.0038
Sub-losses: recon_loss: 45.5667, var_loss: 0.4371, anneal_factor: 0.9991, effective_beta_factor: 0.0010
Epoch 87 - Train Loss: 50.6118
Sub-losses: recon_loss: 50.1381, var_loss: 0.4737, anneal_factor: 0.9993, effective_beta_factor: 0.0010
Epoch 87 - Valid Loss: 46.3734
Sub-losses: recon_loss: 45.9328, var_loss: 0.4406, anneal_factor: 0.9993, effective_beta_factor: 0.0010
Epoch 88 - Train Loss: 50.2289
Sub-losses: recon_loss: 49.7587, var_loss: 0.4702, anneal_factor: 0.9994, effective_beta_factor: 0.0010
Epoch 88 - Valid Loss: 46.5857
Sub-losses: recon_loss: 46.1322, var_loss: 0.4535, anneal_factor: 0.9994, effective_beta_factor: 0.0010
Epoch 89 - Train Loss: 50.9356
Sub-losses: recon_loss: 50.4594, var_loss: 0.4762, anneal_factor: 0.9995, effective_beta_factor: 0.0010
Epoch 89 - Valid Loss: 46.1687
Sub-losses: recon_loss: 45.7217, var_loss: 0.4470, anneal_factor: 0.9995, effective_beta_factor: 0.0010
Epoch 90 - Train Loss: 50.0814
Sub-losses: recon_loss: 49.6065, var_loss: 0.4749, anneal_factor: 0.9996, effective_beta_factor: 0.0010
Epoch 90 - Valid Loss: 45.7531
Sub-losses: recon_loss: 45.3050, var_loss: 0.4480, anneal_factor: 0.9996, effective_beta_factor: 0.0010
Epoch 91 - Train Loss: 50.5753
Sub-losses: recon_loss: 50.1029, var_loss: 0.4724, anneal_factor: 0.9997, effective_beta_factor: 0.0010
Epoch 91 - Valid Loss: 46.2869
Sub-losses: recon_loss: 45.8395, var_loss: 0.4474, anneal_factor: 0.9997, effective_beta_factor: 0.0010
Epoch 92 - Train Loss: 50.9353
Sub-losses: recon_loss: 50.4572, var_loss: 0.4781, anneal_factor: 0.9997, effective_beta_factor: 0.0010
Epoch 92 - Valid Loss: 46.0028
Sub-losses: recon_loss: 45.5560, var_loss: 0.4468, anneal_factor: 0.9997, effective_beta_factor: 0.0010
Epoch 93 - Train Loss: 50.2209
Sub-losses: recon_loss: 49.7460, var_loss: 0.4749, anneal_factor: 0.9998, effective_beta_factor: 0.0010
Epoch 93 - Valid Loss: 45.7390
Sub-losses: recon_loss: 45.2780, var_loss: 0.4610, anneal_factor: 0.9998, effective_beta_factor: 0.0010
Epoch 94 - Train Loss: 49.7338
Sub-losses: recon_loss: 49.2576, var_loss: 0.4763, anneal_factor: 0.9998, effective_beta_factor: 0.0010
Epoch 94 - Valid Loss: 45.8681
Sub-losses: recon_loss: 45.4080, var_loss: 0.4601, anneal_factor: 0.9998, effective_beta_factor: 0.0010
Epoch 95 - Train Loss: 49.4561
Sub-losses: recon_loss: 48.9776, var_loss: 0.4785, anneal_factor: 0.9998, effective_beta_factor: 0.0010
Epoch 95 - Valid Loss: 45.4386
Sub-losses: recon_loss: 44.9910, var_loss: 0.4476, anneal_factor: 0.9998, effective_beta_factor: 0.0010
Epoch 96 - Train Loss: 49.8873
Sub-losses: recon_loss: 49.4097, var_loss: 0.4776, anneal_factor: 0.9999, effective_beta_factor: 0.0010
Epoch 96 - Valid Loss: 44.9739
Sub-losses: recon_loss: 44.5244, var_loss: 0.4495, anneal_factor: 0.9999, effective_beta_factor: 0.0010
Epoch 97 - Train Loss: 49.1776
Sub-losses: recon_loss: 48.6954, var_loss: 0.4823, anneal_factor: 0.9999, effective_beta_factor: 0.0010
Epoch 97 - Valid Loss: 45.3063
Sub-losses: recon_loss: 44.8620, var_loss: 0.4442, anneal_factor: 0.9999, effective_beta_factor: 0.0010
Epoch 98 - Train Loss: 49.4944
Sub-losses: recon_loss: 49.0110, var_loss: 0.4833, anneal_factor: 0.9999, effective_beta_factor: 0.0010
Epoch 98 - Valid Loss: 46.1903
Sub-losses: recon_loss: 45.7328, var_loss: 0.4575, anneal_factor: 0.9999, effective_beta_factor: 0.0010
Epoch 99 - Train Loss: 49.3520
Sub-losses: recon_loss: 48.8653, var_loss: 0.4867, anneal_factor: 0.9999, effective_beta_factor: 0.0010
Epoch 99 - Valid Loss: 45.5614
Sub-losses: recon_loss: 45.1071, var_loss: 0.4543, anneal_factor: 0.9999, effective_beta_factor: 0.0010
Epoch 100 - Train Loss: 49.5983
Sub-losses: recon_loss: 49.1044, var_loss: 0.4939, anneal_factor: 0.9999, effective_beta_factor: 0.0010
Epoch 100 - Valid Loss: 47.1719
Sub-losses: recon_loss: 46.7135, var_loss: 0.4584, anneal_factor: 0.9999, effective_beta_factor: 0.0010
Reproducibility settings for device cuda are not implemented or necessary i.e. for cpu.
Processed 707 / 707 samples

Check if we have a little signal on Chromosome 21 (male/femal seperation)¶

In [7]:

# ontix.show_result(params=["SEX"])
ontix.visualizer.show_latent_space(result=ontix.result, plot_type="Ridgeline", param=["SEX"])

# ontix.show_result(params=["SEX"]) ontix.visualizer.show_latent_space(result=ontix.result, plot_type="Ridgeline", param=["SEX"])

Now we can call ontix.explain()¶

This will calculate gene-by-latent-dimension attribution scores. The explain step returns these as pd.DataFrame and also saves this DataFrame to result.embedding_attributions

In [9]:

# latent_contributions = ontix.explain()
latent_contributions = ontix.explain(
    method="DeepLiftShap", # "DeepLiftShap" or "IntegratedGradients"
    # sel_latent_dim=None, # To calculate contribution for all latent dimensions
    sel_latent_dim='21', # Or specify a single latent dimension to explain
	input_type="grouped", # We will test group 'Male' (input) vs. 'Female' (baseline)
	input_group="Male",
	baseline_type="grouped",
	baseline_group="Female",
	anno_col="SEX",
)

# latent_contributions = ontix.explain() latent_contributions = ontix.explain( method="DeepLiftShap", # "DeepLiftShap" or "IntegratedGradients" # sel_latent_dim=None, # To calculate contribution for all latent dimensions sel_latent_dim='21', # Or specify a single latent dimension to explain input_type="grouped", # We will test group 'Male' (input) vs. 'Female' (baseline) input_group="Male", baseline_type="grouped", baseline_group="Female", anno_col="SEX", )

/home/ewald/Github/acx_main_releases/autoencodix_package/.venv/lib/python3.10/site-packages/anndata/_core/anndata.py:1820: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")

Start feature attribution calculation
Calculating attributions for latent dimension: 2

/home/ewald/Github/acx_main_releases/autoencodix_package/.venv/lib/python3.10/site-packages/captum/attr/_core/deep_lift.py:304: UserWarning: Setting forward, backward hooks and attributes on non-linear
               activations. The hooks and attributes will be removed
            after the attribution is finished
  warnings.warn(

In [10]:

print(latent_contributions)

print(latent_contributions)

            21
8755  0.000410
6439  0.000139
1277  0.000194
2335  0.000635
1278  0.000314
...        ...
3229  0.001345
2322  0.004136
7539  0.003321
962   0.015129
8354  0.002853

[2000 rows x 1 columns]

In [11]:

latent_contributions.loc[:, ["21"]].sort_values(by="21", ascending=False)

latent_contributions.loc[:, ["21"]].sort_values(by="21", ascending=False)

Out[11]:

	21
9185	4.276336e-02
1525	4.158988e-02
58494	4.089054e-02
26658	4.071428e-02
128646	4.040800e-02
...	...
135656	1.418938e-06
118430	1.338562e-06
70	1.145638e-06
27121	8.217619e-07
213	1.419851e-07

2000 rows × 1 columns

Now we can plot top 10 highest contribution scores of chromosome 21 and highlight features with synthetic signal¶

We see indeed that our groundtruth signal genes show the highest attribution scores

In [12]:

chrom_21_contributions = latent_contributions.loc[:, ["21"]].sort_values(by="21", ascending=False)[:10]

color_map = ["red" if feature in feature_synth else "blue" for feature in chrom_21_contributions.index]

plt.figure(figsize=(10, 6))
plt.bar(chrom_21_contributions.index, chrom_21_contributions.values.flatten(), color=color_map, label="Chromosome 21 Features")
plt.title("Top Chromosome 21 Feature Contributions to Latent Dimension 21")
plt.xlabel("Feature")
plt.ylabel("Contribution Score")
plt.xticks(rotation=90)
plt.legend()
plt.tight_layout()
plt.show()

chrom_21_contributions = latent_contributions.loc[:, ["21"]].sort_values(by="21", ascending=False)[:10] color_map = ["red" if feature in feature_synth else "blue" for feature in chrom_21_contributions.index] plt.figure(figsize=(10, 6)) plt.bar(chrom_21_contributions.index, chrom_21_contributions.values.flatten(), color=color_map, label="Chromosome 21 Features") plt.title("Top Chromosome 21 Feature Contributions to Latent Dimension 21") plt.xlabel("Feature") plt.ylabel("Contribution Score") plt.xticks(rotation=90) plt.legend() plt.tight_layout() plt.show()

3) How to Customize the Explain Step¶

You have the follwong options:

        method: Literal["DeepLiftShap", "IntegratedGradients"] = "DeepLiftShap",
        sel_latent_dim: Union[list, int, str, None] = None,
        input_type: Literal["random", "grouped"] = "grouped",
        input_group: Optional[str] = None,
        baseline_type: Literal["mean", "random", "grouped"] = "mean",
        baseline_group: str = "all",
        anno_col: Optional[str] = None,
        n_subset: int = 100,
        seed_int: int = 12,
        split: Literal["train", "test", "valid"] = "train",
        llm_explain: bool = False,
        llm_client: Literal["ollama", "mistral", "openrouter", "scads-llm"] = "mistral",
        llm_model: str = "mistral-large-latest",
        top_n_genes: int = 40,
        prompt: str = PROMPT,

More Details on the Options:

        """Runs the feature-importance explainer and returns gene-by-latent-dimension attribution scores.

        Args:
            method:  Specifies which attribution algorithm to use for explaining the model.
            sel_latent_dim: Specifies which latent dimension(s) to explain. If None, all dimensions will be explained.
            input_type: Specifies whether the feature-importance algorithm should use 'random' samples or 'grouped' samples (see input_group) as input for the attribution computation.
            input_group: If input_type is 'grouped', this specifies which subset in anno_col to filter for to use as input for the attribution computation.
            baseline_type: Specifies whether the feature-importance algorithm should use the 'mean' baseline, a 'random'-sample baseline, or samples from a 'grouped' baseline (see baseline_group).
            baseline_group: Specifies whether the baseline is computed using all data (default) or which subset in anno_col to filter for.
            anno_col: If baseline_group is not 'all', this specifies the annotation column used to filter the baseline data.
            n_subset: Specifies the number of cells to use when subsampling for the attribution computation.
            seed_int: Defines the random seed used for reproducible subsampling and attribution calculations.
            split: The split to use for feature importance calculation (train, valid, test), default is train.
            llm_explain: Whether to use LLM explainers for feature importance calculation.
            llm_client: The LLM client to use for feature importance calculation.
            llm_model: The LLM model to use for feature importance calculation.
            top_n_genes: How many top (contribution to embedding) genes to consider for LLM explanation.

        Returns:
            pd.DataFrame: The generated samples in the input space.
                A DataFrame of attribution scores with genes as the index and latent dimensions as columns.
                Each entry represents the contribution of a given gene to a specific latent dimension.
        """
        ```

4) How To Use LLM to Get Insights of Gene Contributions¶

We can send the gene attribution score directly to an LLM and get biological insights for each latent dimension, i.e. latent dim 1 is associated with cell cylce .. The LLM will give you and explanation for each latent dim and will save this in an extra markdown file. Additionally, we get parsable JSON that is stored in result.embedding_explanations.

Before, we can obtain this, we need to set up our LLM provider (in the future this might be provided by use) See below how to do this:

🧬 Gene Expression Explanation – README¶

You can get LLM explanations from our .explain step by setting llm_explain=True. Therefore you need to setup LLM Clients

You can use either:

• Mistral API (cloud-based)
• Ollama (local models running on your machine)
• OpenRouter (cloud-based)
• ScaDS-LLM (internal-LLM-server)

Requirements¶

Depending on if you want to use Mistral or Ollama you need to have:

• a Mistral API key
• Ollam installed and at least one model served

Environment Setup¶

1. Using Mistral API¶

Set an API key in your .env file in the root of the repository

MISTRAL_API_KEY=your_api_key_here

Models

You may use any model served by Mistral, for example:

• mistral-small-latest
• mistral-medium-latest
• mistral-large-latest

Make sure the model_name you pass to .explain matches an available Mistral model.

2. Using Ollama¶

Ollama runs models locally. You must first install Ollama and pull the model you want:

ollama pull <model-name>

Your model_name must match exactly the name of the model served by Ollama:

• qwen2.5:0.5b
• deepseek-r1:8b

3. Using Openrouter¶

To use a widespread options of models you can use Openrouter. For this simply set a environment variable in your terminal with:

export OPENROUTER_PREMIUM_API_KEY="sk-my-key"

4. Using ScaDS-LLM server¶

If you are a ScaDS.AI member or have a ZIH TU Dresden account, you can use our self-hosted LLM server (https://llm.scads.ai/docs/).

To enable this, set an environment variable like this:

export SCADS_LLM_API_KEY="sk-my-key"

🧬 Using .explain() for Gene Expression Interpretation¶

The .explain() method can generate a short biological explanation and hypothesis about what is happening in disease vs healthy samples given a list of altered genes.

Example¶

In [14]:

ontix.explain(
	method="IntegratedGradients", # "DeepLiftShap" or "IntegratedGradients"
    # sel_latent_dim=None, # To calculate contribution for all latent dimensions
    sel_latent_dim=['21', 'X'], # Or specify a single latent dimension to explain
	input_type="grouped", # We will test group 'Male' (input) vs. 'Female' (baseline)
	input_group="Male",
	baseline_type="grouped",
	baseline_group="Female",
	anno_col="SEX",
    llm_explain=True,
    llm_client="openrouter",
    llm_model="gemma-4-31b-it",
    top_n_genes=10)

ontix.explain( method="IntegratedGradients", # "DeepLiftShap" or "IntegratedGradients" # sel_latent_dim=None, # To calculate contribution for all latent dimensions sel_latent_dim=['21', 'X'], # Or specify a single latent dimension to explain input_type="grouped", # We will test group 'Male' (input) vs. 'Female' (baseline) input_group="Male", baseline_type="grouped", baseline_group="Female", anno_col="SEX", llm_explain=True, llm_client="openrouter", llm_model="gemma-4-31b-it", top_n_genes=10)

/home/ewald/Github/acx_main_releases/autoencodix_package/.venv/lib/python3.10/site-packages/anndata/_core/anndata.py:1820: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")

Start feature attribution calculation
Calculating attributions for latent dimension: 2
Calculating attributions for latent dimension: 7
Start LLM explanation generation
Explaining latent dimension X with LLM...
Explaining latent dimension 21 with LLM...
Saved explanations to: /home/ewald/Github/acx_main_releases/autoencodix_package/latent_explanations.md

Out[14]:

	21	X
8755	0.000434	0.000608
6439	0.000218	0.000231
1277	0.000247	0.000566
2335	0.000747	0.000988
1278	0.000391	0.000766
...	...	...
3229	0.001973	0.005991
2322	0.004055	0.008223
7539	0.003087	0.006265
962	0.015750	0.005568
8354	0.004040	0.002510

2000 rows × 2 columns

This creates a verbal descriptions which is stored in the file latent_explanations.md as markdown document with an output like this:

Latent Dimension X¶

Most influential genes¶

3597, 399668, 9949, 9185, 58494, 2010, 55613, 56180, 9968, 9130

TLDR¶

This latent dimension likely represents the coordination of RNA processing, splicing, and nuclear transport mechanisms essential for cellular gene expression regulation.

Details¶

Dominant Biological Themes¶

The provided gene set is heavily enriched for components of the spliceosome, RNA-binding proteins, and nuclear pore complex components, indicating a strong theme of mRNA maturation and nucleocytoplasmic transport.

Mechanistic Hypotheses¶

• The dimension captures a regulatory program governing the efficiency of pre-mRNA splicing and export to the cytoplasm.

• This dimension may represent a cellular state characterized by high transcriptional activity requiring robust RNA processing machinery.

Summary of Pathways/Processes¶

Key implicated processes include the Spliceosome (KEGG), RNA transport (GO), and mRNA processing (GO), focusing on the transition of genetic information from the nucleus to the ribosome.

---

Latent Dimension 21¶

Most influential genes¶

9185, 58494, 1525, 1525, 399668, 9130, 128646, 2010, 9949, 56180

TLDR¶

This latent dimension likely represents a cellular state focused on protein synthesis, ribosomal biogenesis, and basic metabolic maintenance.

Details¶

Dominant Biological Themes¶

The gene set is dominated by components of the translational machinery, specifically ribosomal proteins and factors involved in protein folding and synthesis.

Mechanistic Hypotheses¶

• The dimension captures the global translational capacity or protein synthesis rate of the cells.

• The dimension represents a metabolic shift toward anabolic growth and ribosomal biogenesis.

Summary of Pathways/Processes¶

Implicated processes include Ribosome biogenesis (GO:0000154), Translation (GO:0006412), and Protein folding (GO:0006457).

---