Note: This environment variable is required for fully deterministic CuBLAS ops on CUDA >= 10.2 when reproducible=True is set below. Without it, PyTorch raises a RuntimeError instead of training deterministically. It must be set before torch is imported. See the README FAQ for details.

In [1]:

%env CUBLAS_WORKSPACE_CONFIG=:16:8

%env CUBLAS_WORKSPACE_CONFIG=:16:8

env: CUBLAS_WORKSPACE_CONFIG=:16:8

How to train an ontology-based variational autoencoder (Ontix)¶

In normal autoencoders, latent dimensions are not explainable by design. To gain explainability and incorporate biological information, a popular approach is to restrict the decoder of the autoencoder to match feature connectivity, such as an ontology.

IMPORTANT

This tutorial only shows the specifics of the Ontix pipeline. If you're unfamiliar with general concepts,
we recommend following the Getting Started - Vanillix Tutorial first.

What You'll Learn¶

In this notebook we will show two types of ontologies and how they can be used to train an explainable variational autoencoder, Ontix. The first is based on biological pathways from the Reactome database (left), and the second uses chromosomal location of genes (right) as a showcase.

You’ll learn how to:

1. Initialize the pipeline, ontologies, and run Ontix with chromosomal ontologies.
   
   
2. Initialize the pipeline, ontologies, and run Ontix with Reactome pathways.
   
   
3. Evaluate Ontix on downstream tasks.
   
   

By working with these examples, we will also cover on the go:

• Understand the Ontix-specific pipeline steps.
  
  
• Access the Ontix-specific results (mu, sigma, KL/MMD losses).
  
  
• Visualize outputs effectively.
  
  
• Apply custom parameters.
  
  
• Save, load, and reuse a trained pipeline.
  
  

❗❗ Requirements: Getting Tutorial Data ❗❗¶

The data for this tutorial is hosted on Hugging Face Hub (autoencodix/tcga) and is downloaded automatically in the cells below on first run.

Extra 2: Get correct path¶

We assume you are in the root of the package. The following code ensures that the correct paths are used. [1] Tutorials/DeepDives/ConfigTutorial.ipynb

In [2]:

import os

p = os.getcwd()
d = "autoencodix_package"
if d not in p:
    raise FileNotFoundError(f"'{d}' not found in path: {p}")
os.chdir(os.sep.join(p.split(os.sep)[: p.split(os.sep).index(d) + 1]))
print(f"Changed to: {os.getcwd()}")

import os p = os.getcwd() d = "autoencodix_package" if d not in p: raise FileNotFoundError(f"'{d}' not found in path: {p}") os.chdir(os.sep.join(p.split(os.sep)[: p.split(os.sep).index(d) + 1])) print(f"Changed to: {os.getcwd()}")

Changed to: /home/alicia/dev/biomarker_autoencoder/autoencodix_package

1) Set-up Ontology and Initialize Pipeline with Chromosomal Ontology¶

The only thing you need to do to train an Ontix model is to provide information about your ontologies. You can do this via .txt files or directly as Python dictionaries. In this example, we use text files. We plan to extend this tutorial in the future with an example using Python dictionaries (TODO JE).

You can provide up to n ontology files and pass them as a list to the Ontix pipeline object. The list should be ordered so that the last ontology level corresponds to the mapping of your features (e.g., Gene ID) to an ontology level like subpathways or cytobands of chromosomes. More levels are optional but recommended, as they map gene names to higher-level categories like top-level pathways or chromosome regions.

The mapping of the last level should have the format:
Gene 1 separator Pathway1
Gene 2 separator Pathway1
Gene 3 separator Pathway2

1.1 Prepare Ontology Data¶

We quickly show how to prepare a downloaded ontology into a text file that serves as input for Ontix.

Example 1: Set-up chromosomal ontology:

From Ensembl via Biomart or any other sequence database, you can get cytoband (karyotype) and chromosomal information for human genes like this:

In [3]:

import pandas as pd

# Download from https://github.com/jan-forest/autoencodix/blob/main/Tutorials/genes_chromosomes.txt
df_genes = pd.read_csv("genes_chromosomes.txt", sep="\t")
df_genes.head()

import pandas as pd # Download from https://github.com/jan-forest/autoencodix/blob/main/Tutorials/genes_chromosomes.txt df_genes = pd.read_csv("genes_chromosomes.txt", sep="\t") df_genes.head()

Out[3]:

	Gene stable ID	Gene stable ID version	Karyotype band	Chromosome/scaffold name	Gene start (bp)	Gene end (bp)	HGNC symbol	NCBI gene (formerly Entrezgene) ID
0	ENSG00000198888	ENSG00000198888.2	NaN	MT	3307	4262	MT-ND1	4535
1	ENSG00000198763	ENSG00000198763.3	NaN	MT	4470	5511	MT-ND2	4536
2	ENSG00000198804	ENSG00000198804.2	NaN	MT	5904	7445	MT-CO1	4512
3	ENSG00000210151	ENSG00000210151.2	NaN	MT	7446	7514	MT-TS1	113219467
4	ENSG00000198712	ENSG00000198712.1	NaN	MT	7586	8269	MT-CO2	4513

We have to solve some issues before we can use this as ontology.
(1) We only want chromosomes and not scaffolds
(2) Karyotype/cytoband should have identification of the chromosome in their name as identifier

In [4]:

df_genes = df_genes.loc[
    df_genes["Chromosome/scaffold name"].str.len() < 3
]  ## get rid of scaffolds and keep only chromosomes
df_genes.loc[df_genes["Chromosome/scaffold name"] == "MT", "Karyotype band"] = (
    "MT"  ## create missing karyotype for mito genes
)
print("This will be our chromosomes and latent dimensions in Ontix:")
print(df_genes["Chromosome/scaffold name"].unique())
print(f"Latent dimension: {len(df_genes['Chromosome/scaffold name'].unique())}")

df_genes = df_genes.loc[ df_genes["Chromosome/scaffold name"].str.len() < 3 ] ## get rid of scaffolds and keep only chromosomes df_genes.loc[df_genes["Chromosome/scaffold name"] == "MT", "Karyotype band"] = ( "MT" ## create missing karyotype for mito genes ) print("This will be our chromosomes and latent dimensions in Ontix:") print(df_genes["Chromosome/scaffold name"].unique()) print(f"Latent dimension: {len(df_genes['Chromosome/scaffold name'].unique())}")

This will be our chromosomes and latent dimensions in Ontix:
['MT' 'Y' '21' '13' '18' '22' '20' 'X' '15' '14' '10' '9' '8' '16' '4' '5'
 '7' '6' '19' '12' '11' '3' '17' '2' '1']
Latent dimension: 25

In [5]:

# Combine Chromosome name and cytoband
df_genes = df_genes.copy()
df_genes.loc[:, "Chr_and_karyotype"] = df_genes.loc[
    :, ["Chromosome/scaffold name", "Karyotype band"]
].apply(lambda x: ":".join(x.values.tolist()), axis=1)
print("This will be our hidden layer in the sparse decoder:")
print(df_genes["Chr_and_karyotype"].unique()[0:20])
print(f"Hidden layer dim: {len(df_genes['Chr_and_karyotype'].unique())}")

# Combine Chromosome name and cytoband df_genes = df_genes.copy() df_genes.loc[:, "Chr_and_karyotype"] = df_genes.loc[ :, ["Chromosome/scaffold name", "Karyotype band"] ].apply(lambda x: ":".join(x.values.tolist()), axis=1) print("This will be our hidden layer in the sparse decoder:") print(df_genes["Chr_and_karyotype"].unique()[0:20]) print(f"Hidden layer dim: {len(df_genes['Chr_and_karyotype'].unique())}")

This will be our hidden layer in the sparse decoder:
['MT:MT' 'Y:p11.2' 'Y:q11.223' 'Y:q11.221' 'Y:q11.222' 'Y:q11.23'
 'Y:p11.31' 'Y:p11.32' 'Y:q12' '21:p12' '21:q21.1' '21:q21.2' '21:p11.2'
 '13:q12.12' '21:q21.3' '21:q22.11' '13:q12.3' '13:q14.12' '13:q14.2'
 '21:q22.12']
Hidden layer dim: 817

Now we can save this as files in the correct format for the two levels:

In [6]:

import os

p = os.getcwd()
d = "autoencodix_package"
if d not in p:
    raise FileNotFoundError(f"'{d}' not found in path: {p}")
os.chdir(os.sep.join(p.split(os.sep)[: p.split(os.sep).index(d) + 1]))
print(f"Changed to: {os.getcwd()}")
# ---------------------------------------------------------------------
# Paths
# ---------------------------------------------------------------------
data_root = "data/raw"
os.makedirs(data_root, exist_ok=True)
ont_genelevel = "chromosome_ont_genelevel_ncbi.txt"
ont_hiddenlevel = "chromosome_ont_hiddenlevel.txt"

# FEATURE TO TO ONTOLOGY LEVEL
df_genes[
    [
        "NCBI gene (formerly Entrezgene) ID",
        "Chr_and_karyotype",
    ]  # Level Feature: Feature (gene) to hidden layer (cytoband)
].drop_duplicates(  # Chromosomal ontology must be unique
).to_csv(
    os.path.join(data_root, ont_genelevel),
    sep="\t",
    header=False,
    index=False,
)

# ONTOLOGY TO LATENT DIMENSION LEVEL
df_genes[
    [
        "Chr_and_karyotype",
        "Chromosome/scaffold name",
    ]  # Level Hidden: hidden layer (cytoband) to latent dimension (chromosome)
].drop_duplicates(  # Chromosomal ontology must be unique
).to_csv(
    os.path.join(data_root, ont_hiddenlevel),
    sep="\t",
    header=False,
    index=False,
)

import os p = os.getcwd() d = "autoencodix_package" if d not in p: raise FileNotFoundError(f"'{d}' not found in path: {p}") os.chdir(os.sep.join(p.split(os.sep)[: p.split(os.sep).index(d) + 1])) print(f"Changed to: {os.getcwd()}") # --------------------------------------------------------------------- # Paths # --------------------------------------------------------------------- data_root = "data/raw" os.makedirs(data_root, exist_ok=True) ont_genelevel = "chromosome_ont_genelevel_ncbi.txt" ont_hiddenlevel = "chromosome_ont_hiddenlevel.txt" # FEATURE TO TO ONTOLOGY LEVEL df_genes[ [ "NCBI gene (formerly Entrezgene) ID", "Chr_and_karyotype", ] # Level Feature: Feature (gene) to hidden layer (cytoband) ].drop_duplicates( # Chromosomal ontology must be unique ).to_csv( os.path.join(data_root, ont_genelevel), sep="\t", header=False, index=False, ) # ONTOLOGY TO LATENT DIMENSION LEVEL df_genes[ [ "Chr_and_karyotype", "Chromosome/scaffold name", ] # Level Hidden: hidden layer (cytoband) to latent dimension (chromosome) ].drop_duplicates( # Chromosomal ontology must be unique ).to_csv( os.path.join(data_root, ont_hiddenlevel), sep="\t", header=False, index=False, )

Changed to: /home/alicia/dev/biomarker_autoencoder/autoencodix_package

1.2 Create Your Config and Run Pipeline¶

The initialization of the config does not work differently as for other pipelines, but with the constraint that we only allow MINMAX scaling as scaling method. The ontix pipeline itself takes an additional ontologies keyword argument. Here you pass either the list of ontology files (with the pathway-to-gene-id mapping as last entry).

In [7]:

import os
import autoencodix as acx
from autoencodix.configs.default_config import DataConfig, DataInfo, DataCase
from autoencodix.configs import OntixConfig

# ---------------------------------------------------------------------
# Data is hosted on Hugging Face Hub and downloaded (and locally cached) automatically
# ---------------------------------------------------------------------
from huggingface_hub import hf_hub_download

HF_REPO_ID = "autoencodix/tcga"
rna_path = hf_hub_download(repo_id=HF_REPO_ID, repo_type="dataset", filename="rna.parquet")
meth_path = hf_hub_download(repo_id=HF_REPO_ID, repo_type="dataset", filename="methylation.parquet")
clin_path = hf_hub_download(repo_id=HF_REPO_ID, repo_type="dataset", filename="clinical.parquet")

# ---------------------------------------------------------------------
# Define individual data modalities
# ---------------------------------------------------------------------
rna_info = DataInfo(
    file_path=rna_path,
    data_type="NUMERIC",
    filtering="VAR",
)

meth_info = DataInfo(
    file_path=meth_path,
    data_type="NUMERIC",
    filtering="VAR",
)

anno_info = DataInfo(
    file_path=clin_path,
    data_type="ANNOTATION",
)
# ---------------------------------------------------------------------
# Combine into DataConfig
# ---------------------------------------------------------------------
data_config = DataConfig(
    data_info={
        "RNA": rna_info,
        "METH": meth_info,
        "ANNO": anno_info,
    },
    annotation_columns=[
        # "CANCER_TYPE",
        "CANCER_TYPE_ACRONYM",
        # "TMB_NONSYNONYMOUS",
        # "AGE",
        # "OS_STATUS",
        # "GRADE",
        "SEX",
    ],
)

# ---------------------------------------------------------------------
# Define the full DefaultConfig (roughly equivalent to old cfg)
# ---------------------------------------------------------------------
ontix_config = OntixConfig(
    data_config=data_config,
    reproducible=True,
    global_seed=42,
    epochs=100,
    learning_rate=0.0005,
    batch_size=128,
    drop_p=0.3,
    k_filter=2000,
    latent_dim=6,
    device="cpu",
    reconstruction_loss="mse",
    default_vae_loss="kl",
    beta=0.001,
    save_memory=False,
    scaling="MINMAX",
    train_ratio=0.7,
    test_ratio=0.2,
    valid_ratio=0.1,
)

# ---------------------------------------------------------------------
# Now pass into your Ontix object
# ---------------------------------------------------------------------

ont_files = [ont_hiddenlevel, ont_genelevel]
ont_files = [os.path.join(data_root, f) for f in ont_files]
ontix = acx.Ontix(
    ontologies=ont_files,
    config=ontix_config,
)

import os import autoencodix as acx from autoencodix.configs.default_config import DataConfig, DataInfo, DataCase from autoencodix.configs import OntixConfig # --------------------------------------------------------------------- # Data is hosted on Hugging Face Hub and downloaded (and locally cached) automatically # --------------------------------------------------------------------- from huggingface_hub import hf_hub_download HF_REPO_ID = "autoencodix/tcga" rna_path = hf_hub_download(repo_id=HF_REPO_ID, repo_type="dataset", filename="rna.parquet") meth_path = hf_hub_download(repo_id=HF_REPO_ID, repo_type="dataset", filename="methylation.parquet") clin_path = hf_hub_download(repo_id=HF_REPO_ID, repo_type="dataset", filename="clinical.parquet") # --------------------------------------------------------------------- # Define individual data modalities # --------------------------------------------------------------------- rna_info = DataInfo( file_path=rna_path, data_type="NUMERIC", filtering="VAR", ) meth_info = DataInfo( file_path=meth_path, data_type="NUMERIC", filtering="VAR", ) anno_info = DataInfo( file_path=clin_path, data_type="ANNOTATION", ) # --------------------------------------------------------------------- # Combine into DataConfig # --------------------------------------------------------------------- data_config = DataConfig( data_info={ "RNA": rna_info, "METH": meth_info, "ANNO": anno_info, }, annotation_columns=[ # "CANCER_TYPE", "CANCER_TYPE_ACRONYM", # "TMB_NONSYNONYMOUS", # "AGE", # "OS_STATUS", # "GRADE", "SEX", ], ) # --------------------------------------------------------------------- # Define the full DefaultConfig (roughly equivalent to old cfg) # --------------------------------------------------------------------- ontix_config = OntixConfig( data_config=data_config, reproducible=True, global_seed=42, epochs=100, learning_rate=0.0005, batch_size=128, drop_p=0.3, k_filter=2000, latent_dim=6, device="cpu", reconstruction_loss="mse", default_vae_loss="kl", beta=0.001, save_memory=False, scaling="MINMAX", train_ratio=0.7, test_ratio=0.2, valid_ratio=0.1, ) # --------------------------------------------------------------------- # Now pass into your Ontix object # --------------------------------------------------------------------- ont_files = [ont_hiddenlevel, ont_genelevel] ont_files = [os.path.join(data_root, f) for f in ont_files] ontix = acx.Ontix( ontologies=ont_files, config=ontix_config, )

/home/alicia/dev/biomarker_autoencoder/autoencodix_package/src/autoencodix/configs/default_config.py:439: UserWarning: annotation_columns in DataConfig is deprecated. Please set it directly in DefaultConfig instead.
  warnings.warn(

In [8]:

r = ontix.run()

r = ontix.run()

reading parquet: /home/alicia/.cache/huggingface/hub/datasets--autoencodix--tcga/snapshots/80fb52f9814c7524de64860ff6e0c7496adc2375/rna.parquet
reading parquet: /home/alicia/.cache/huggingface/hub/datasets--autoencodix--tcga/snapshots/80fb52f9814c7524de64860ff6e0c7496adc2375/methylation.parquet
reading parquet: /home/alicia/.cache/huggingface/hub/datasets--autoencodix--tcga/snapshots/80fb52f9814c7524de64860ff6e0c7496adc2375/clinical.parquet
anno key: paired
Features in feature_order not found in all_feature_names: ['100133144', '10357', '10431', '155060', '390284', '57714', '645851', '653553', '729884', '246182', '119385', '653268', '728404', '200810', '138649', '441425', '728747', '729171', '23520', '303', '304', '305', '244', '375719', '441432', '503640', '641522', '432369', '92270', '6791', '85319', '606', '23629', '286076', '414235', '170393', '255352', '283422', '283416', '374467', '84837', '400223', '650662', '280655', '283651', '80035', '283687', '196968', '284185', '147429', '147525', '494514', '574036', '284573', '149469', '84791', '253868', '284836', '54094', '54067', '282566', '114041', '114043', '149992', '55267', '29798', '348738', '339942', '93556', '646450', '317648', '92070', '79614', '153571', '116349', '441108', '285679', '85411', '653483', '387097', '79992', '129790', '113763', '158228', '445577', '158314', '157983', '120329', '348254', '221016', '348249', '643253', '1038', '94158', '100130418', '1057', '729338', '386593', '152302', '440508', '1197', '8418', '646300', '285464', '440359', '161635', '646243', '440224', '3580', '170063', '645090', '653687', '1564', '440081', '729582', '222161', '374387', '79469', '285987', '100128285', '100132911', '503645', '554236', '574029', '441032', '196549', '100131454', '326342', '347918', '8475', '284729', '145165', '387071', '728262', '100132403', '728882', '55747', '548321', '100132948', '439965', '414241', '55855', '339521', '100133172', '440078', '100132923', '692224', '9103', '619190', '80307', '2282', '79667', '80094', '401237', '200058', '255031', '649446', '730971', '283011', '202020', '388685', '388182', '645644', '392490', '402483', '399844', '283102', '100036519', '284802', '2498', '388965', '389523', '492303', '2679', '645367', '374650', '222611', '728932', '442245', '2952', '653238', '401375', '375513', '653188', '767811', '768096', '594842', '352961', '80867', '54435', '80868', '400322', '440362', '100128124', '3136', '3137', '10151', '378465', '391634', '664618', '343477', '3311', '84099', '91353', '644619', '3653', '389293', '387628', '654466', '100132341', '10748', '22973', '11026', '100271835', '100101266', '100124692', '100126784', '100128288', '100128292', '100128542', '100128573', '100128822', '100128842', '100129034', '100129387', '100129637', '100130238', '100130557', '100130581', '100130872', '100130932', '100130987', '100131551', '100132111', '100132287', '100132707', '100132724', '100132832', '100133161', '100133331', '100133669', '100133991', '100134868', '100144604', '100170939', '100188947', '100190939', '100190940', '100190986', '100216545', '100270710', '100270746', '100271722', '100271836', '100272146', '100272216', '100272217', '100272228', '100286793', '100287227', '100302640', '100302650', '115110', '143188', '143666', '144438', '146880', '148696', '149134', '150381', '151009', '151162', '162632', '168474', '202181', '219347', '220429', '220594', '222699', '253724', '255167', '256880', '282997', '283050', '283070', '283922', '284009', '284232', '284440', '284441', '284578', '285033', '285359', '285548', '286002', '286367', '338758', '339047', '341056', '347376', '349114', '349196', '374491', '387646', '387647', '388152', '388242', '388955', '389458', '389705', '391322', '399744', '399815', '400027', '400657', '400927', '401010', '401052', '401127', '401588', '402377', '407835', '440354', '440461', '440905', '441089', '441204', '441208', '441454', '441455', '442308', '442454', '442459', '493754', '550112', '606724', '641298', '642826', '642846', '643387', '643719', '644165', '644172', '644936', '645431', '646214', '646471', '646762', '646999', '647288', '647859', '648740', '650368', '650623', '651250', '653113', '653501', '653566', '727896', '728024', '728264', '728323', '728554', '728613', '728640', '728723', '728758', '728855', '728875', '728989', '729176', '729234', '729375', '729603', '729799', '731789', '80154', '90110', '90246', '90784', '90834', '91450', '96610', '29931', '55073', '147172', '767558', '79136', '653639', '378938', '442229', '4213', '84815', '85009', '84848', '84849', '114130', '113691', '81854', '90768', '401884', '389538', '4276', '84953', '10934', '359821', '729633', '401827', '11209', '11223', '100129405', '55545', '326343', '4500', '339483', '84176', '83955', '51471', '728936', '285622', '343505', '100188954', '80161', '283981', '79854', '55389', '440072', '400508', '114915', '64493', '283458', '100129354', '119369', '10896', '4950', '441295', '26636', '647033', '728773', '8123', '347746', '54661', '84054', '171423', '728939', '267004', '375133', '206426', '266971', '440456', '441194', '5379', '5383', '646074', '25812', '29797', '5505', '55370', '100169750', '28997', '29053', '377047', '100130889', '26255', '5820', '83956', '401331', '10740', '379013', '286140', '283345', '100129424', '402176', '644128', '649946', '284942', '222901', '132241', '85495', '376693', '728963', '441502', '256355', '653162', '730092', '653390', '84127', '440352', '54581', '677769', '727956', '728609', '641977', '201175', '440044', '65012', '9906', '81893', '387254', '342615', '751867', '387066', '641638', '100093630', '735301', '677821', '6044', '6043', '652966', '100033431', '100033820', '100033416', '677850', '64173', '441251', '389517', '10638', '23145', '400410', '54441', '442582', '442578', '7955', '441394', '474338', '440423', '767557', '654341', '27004', '7012', '374500', '286102', '7117', '7151', '7152', '11257', '728402', '646405', '117852', '154754', '128854', '56604', '643224', '606551', '652995', '374666', '653635', '171022', '339005', '440253', '51352', '150244', '7696', '7754', '387328', '651302', '388507', '401303', '664701', '401898', '221584']
Features in feature_order not found in all_feature_names: ['100033413', '100130426', '100133144', '1038', '10431', '10748', '10934', '114038', '136542', '145165', '155060', '170393', '200058', '221016', '222611', '255352', '25812', '2679', '283416', '285759', '317712', '348738', '387328', '388685', '389649', '391343', '4276', '4950', '51352', '553137', '57714', '5820', '645851', '65012', '652919', '7012', '728603', '729884', '7696', '79614', '8123', '84127', '90011', '9906']
Reproducibility settings for device cpu are not implemented or necessary i.e. for cpu.
Ontix checks:
All possible feature names length: 27248
Feature order length: 2000
Feature names without filtering: 2000
Mask layer 0 with shape torch.Size([817, 25]) and 817.0 connections
Mask layer 1 with shape torch.Size([2000, 817]) and 1950.0 connections
Latent Dim: 25
Epoch 10 - Train Loss: 129.8296
Sub-losses: recon_loss: 129.8296, var_loss: 0.0000, anneal_factor: 0.0003, effective_beta_factor: 0.0000
Epoch 10 - Valid Loss: 122.9064
Sub-losses: recon_loss: 122.9064, var_loss: 0.0000, anneal_factor: 0.0003, effective_beta_factor: 0.0000
Epoch 20 - Train Loss: 65.5959
Sub-losses: recon_loss: 65.5954, var_loss: 0.0005, anneal_factor: 0.0020, effective_beta_factor: 0.0000
Epoch 20 - Valid Loss: 64.0073
Sub-losses: recon_loss: 64.0069, var_loss: 0.0004, anneal_factor: 0.0020, effective_beta_factor: 0.0000
Epoch 30 - Train Loss: 61.1895
Sub-losses: recon_loss: 61.1847, var_loss: 0.0048, anneal_factor: 0.0148, effective_beta_factor: 0.0000
Epoch 30 - Valid Loss: 58.6947
Sub-losses: recon_loss: 58.6902, var_loss: 0.0045, anneal_factor: 0.0148, effective_beta_factor: 0.0000
Epoch 40 - Train Loss: 59.2094
Sub-losses: recon_loss: 59.1703, var_loss: 0.0392, anneal_factor: 0.0998, effective_beta_factor: 0.0001
Epoch 40 - Valid Loss: 55.7500
Sub-losses: recon_loss: 55.7134, var_loss: 0.0366, anneal_factor: 0.0998, effective_beta_factor: 0.0001
Epoch 50 - Train Loss: 57.6535
Sub-losses: recon_loss: 57.4522, var_loss: 0.2012, anneal_factor: 0.4502, effective_beta_factor: 0.0005
Epoch 50 - Valid Loss: 55.0650
Sub-losses: recon_loss: 54.8756, var_loss: 0.1894, anneal_factor: 0.4502, effective_beta_factor: 0.0005
Epoch 60 - Train Loss: 56.3597
Sub-losses: recon_loss: 55.9302, var_loss: 0.4295, anneal_factor: 0.8581, effective_beta_factor: 0.0009
Epoch 60 - Valid Loss: 54.1814
Sub-losses: recon_loss: 53.7886, var_loss: 0.3928, anneal_factor: 0.8581, effective_beta_factor: 0.0009
Epoch 70 - Train Loss: 55.3732
Sub-losses: recon_loss: 54.8487, var_loss: 0.5244, anneal_factor: 0.9781, effective_beta_factor: 0.0010
Epoch 70 - Valid Loss: 53.1894
Sub-losses: recon_loss: 52.7172, var_loss: 0.4722, anneal_factor: 0.9781, effective_beta_factor: 0.0010
Epoch 80 - Train Loss: 54.8069
Sub-losses: recon_loss: 54.2369, var_loss: 0.5700, anneal_factor: 0.9970, effective_beta_factor: 0.0010
Epoch 80 - Valid Loss: 54.0036
Sub-losses: recon_loss: 53.4958, var_loss: 0.5077, anneal_factor: 0.9970, effective_beta_factor: 0.0010
Epoch 90 - Train Loss: 53.8781
Sub-losses: recon_loss: 53.2895, var_loss: 0.5886, anneal_factor: 0.9996, effective_beta_factor: 0.0010
Epoch 90 - Valid Loss: 51.7410
Sub-losses: recon_loss: 51.2137, var_loss: 0.5273, anneal_factor: 0.9996, effective_beta_factor: 0.0010
Epoch 100 - Train Loss: 53.1618
Sub-losses: recon_loss: 52.5523, var_loss: 0.6095, anneal_factor: 0.9999, effective_beta_factor: 0.0010
Epoch 100 - Valid Loss: 49.6905
Sub-losses: recon_loss: 49.1378, var_loss: 0.5527, anneal_factor: 0.9999, effective_beta_factor: 0.0010
Reproducibility settings for device cpu are not implemented or necessary i.e. for cpu.
Processed 707 / 707 samples

Gaining insights by explainable latent dimensions and visualizations¶

The idea behind using the chromosomal location of features (genes) as an ontology for Ontix is that we can expect male and female patients to be separated along the Y and X chromosomes.
Let’s check if this is indeed the case:

In [9]:

ontix.show_result(params=["SEX"])
# ontix.show_result()

ontix.show_result(params=["SEX"]) # ontix.show_result()

Creating plots ...

In [10]:

ontix.evaluate()

ontix.evaluate()

Perform ML task with feature df: Latent
Latent
Perform ML task for target parameter: CANCER_TYPE_ACRONYM

/home/alicia/dev/biomarker_autoencoder/autoencodix_package/.venv/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:465: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. OF ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(
/home/alicia/dev/biomarker_autoencoder/autoencodix_package/.venv/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:465: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. OF ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(

Perform ML task for target parameter: SEX

Out[10]:

Result Object Public Attributes:
------------------------------
latentspaces: TrainingDynamics object
sample_ids: TrainingDynamics object
reconstructions: TrainingDynamics object
mus: TrainingDynamics object
sigmas: TrainingDynamics object
losses: TrainingDynamics object
sub_losses: LossRegistry(_losses={'recon_loss': TrainingDynamics(), 'var_loss': TrainingDynamics(), 'anneal_factor': TrainingDynamics(), 'effective_beta_factor': TrainingDynamics()})
preprocessed_data: Tensor of shape (0,)
model: OntixArchitecture
model_checkpoints: TrainingDynamics object
datasets: DatasetContainer(train=<autoencodix.data._numeric_dataset.NumericDataset object at 0x71df3bbcc9d0>, valid=<autoencodix.data._numeric_dataset.NumericDataset object at 0x71df3bbcc9a0>, test=<autoencodix.data._numeric_dataset.NumericDataset object at 0x71dfde7f0160>)
new_datasets: DatasetContainer(train=None, valid=None, test=None)
adata_latent: AnnData object with n_obs × n_vars = 707 × 25
    uns: 'var_names'
final_reconstruction: None
sub_results: None
sub_reconstructions: None
embedding_evaluation:   score_split         CLINIC_PARAM       metric     value              ML_ALG  \
0       train  CANCER_TYPE_ACRONYM  roc_auc_ovo  0.932074  LogisticRegression   
1       valid  CANCER_TYPE_ACRONYM  roc_auc_ovo  0.946326  LogisticRegression   
2        test  CANCER_TYPE_ACRONYM  roc_auc_ovo  0.932994  LogisticRegression   
0       train                  SEX  roc_auc_ovo  0.881106  LogisticRegression   
1       valid                  SEX  roc_auc_ovo  0.934637  LogisticRegression   
2        test                  SEX  roc_auc_ovo  0.882318  LogisticRegression   

          ML_TYPE ML_TASK ML_SUBTASK  
0  classification  Latent     Latent  
1  classification  Latent     Latent  
2  classification  Latent     Latent  
0  classification  Latent     Latent  
1  classification  Latent     Latent  
2  classification  Latent     Latent  
embedding_attributions: Empty DataFrame
Columns: []
Index: []
embedding_explanations: Dict with 0 items

2) Set-up Ontology and Initialize Pipeline with Reactome Pathways¶

Chromosomal location as ontology is mostly a proof of concept as shown above how to gain explainability of latent dimensions.
In practice, something like biological pathways or gene ontology is most commonly used to gain biological insights.

In [11]:

from huggingface_hub import hf_hub_download

reactome_hiddenlevel = hf_hub_download(
    repo_id="autoencodix/tcga", repo_type="dataset", filename="ontology/reactome_ont_lvl2.txt"
)
reactome_genelevel = hf_hub_download(
    repo_id="autoencodix/tcga", repo_type="dataset", filename="ontology/reactome_ont_lvl1.txt"
)
ont_files = [reactome_hiddenlevel, reactome_genelevel]
ontix_react = acx.Ontix(ontologies=ont_files, config=ontix_config)

from huggingface_hub import hf_hub_download reactome_hiddenlevel = hf_hub_download( repo_id="autoencodix/tcga", repo_type="dataset", filename="ontology/reactome_ont_lvl2.txt" ) reactome_genelevel = hf_hub_download( repo_id="autoencodix/tcga", repo_type="dataset", filename="ontology/reactome_ont_lvl1.txt" ) ont_files = [reactome_hiddenlevel, reactome_genelevel] ontix_react = acx.Ontix(ontologies=ont_files, config=ontix_config)

In [12]:

result = ontix_react.run()

result = ontix_react.run()

reading parquet: /home/alicia/.cache/huggingface/hub/datasets--autoencodix--tcga/snapshots/80fb52f9814c7524de64860ff6e0c7496adc2375/rna.parquet
reading parquet: /home/alicia/.cache/huggingface/hub/datasets--autoencodix--tcga/snapshots/80fb52f9814c7524de64860ff6e0c7496adc2375/methylation.parquet
reading parquet: /home/alicia/.cache/huggingface/hub/datasets--autoencodix--tcga/snapshots/80fb52f9814c7524de64860ff6e0c7496adc2375/clinical.parquet
anno key: paired
Features in feature_order not found in all_feature_names: ['100133144', '100134869', '10357', '10431', '155060', '388795', '390284', '57714', '645851', '653553', '729884', '8225', '90288', '87769', '144568', '729522', '79719', '80755', '132949', '9625', '79963', '650655', '150000', '653190', '730013', '55324', '83451', '145447', '84945', '25864', '116236', '84696', '11057', '79575', '51225', '25890', '29777', '80325', '25841', '9744', '23527', '116983', '84680', '57001', '52', '53', '92370', '84519', '93953', '56', '81569', '57180', '653857', '64431', '90', '97', '98', '161931', '10863', '203102', '8755', '27299', '55803', '105', '57143', '90956', '79934', '203054', '22850', '23394', '141', '113622', '165', '64782', '60312', '134265', '84632', '246182', '4299', '2334', '3899', '172', '54812', '387763', '119385', '116987', '116988', '119016', '729092', '414189', '653268', '728404', '3268', '10551', '155465', '27245', '113146', '79026', '10598', '130872', '64853', '199', '83543', '9131', '150209', '51390', '202', '55057', '326', '55966', '11215', '158798', '445815', '10566', '9472', '9465', '10270', '79647', '55122', '80709', '254268', '83592', '64400', '126133', '222', '200810', '644974', '8846', '54784', '84964', '84266', '200420', '245', '115701', '57538', '151254', '130540', '65062', '79800', '257', '83650', '57463', '386724', '9949', '83607', '196394', '155185', '51321', '201283', '51434', '23357', '90806', '9068', '23452', '83854', '10218', '150709', '348094', '162282', '54882', '404734', '54467', '255239', '126549', '51281', '57037', '57763', '55608', '29123', '23253', '88455', '124930', '81573', '338692', '54522', '26057', '138649', '441425', '728747', '729171', '118932', '26287', '200539', '170961', '147463', '84250', '341405', '284615', '340120', '148741', '375248', '57730', '84832', '353322', '51239', '91369', '338699', '134548', '91526', '339416', '157567', '54851', '57182', '283373', '79998', '129138', '79722', '345079', '63926', '347454', '22881', '23294', '56899', '124401', '257629', '203286', '55139', '10541', '23520', '81611', '93550', '311', '312', '303', '304', '305', '306', '307', '310', '653145', '244', '8416', '313', '55435', '8120', '8943', '10947', '10239', '323', '10307', '10297', '147495', '164284', '27301', '8539', '200558', '333', '140564', '200316', '342', '23780', '80833', '80832', '80830', '81575', '10513', '55198', '54840', '653437', '375719', '441432', '206338', '10565', '27236', '503640', '55082', '79822', '10620', '138715', '10865', '285598', '115761', '80117', '54622', '339231', '51326', '641522', '10124', '379', '26225', '221079', '23204', '51329', '151188', '55207', '132946', '83787', '55156', '84071', '219681', '55130', '79798', '93436', '79637', '80210', '51309', '9823', '64860', '54470', '81873', '84517', '92714', '27106', '57561', '91947', '645432', '116969', '421', '170302', '653308', '79827', '8853', '55616', '430', '55723', '434', '8623', '54529', '554235', '253982', '57168', '259266', '151516', '28990', '431705', '460', '23245', '80816', '54454', '55210', '219293', '79915', '85300', '11101', '11016', '80063', '89849', '23130', '55102', '80162', '51062', '64225', '25923', '23300', '474', '220202', '84913', '79572', '432369', '267020', '55101', '92270', '158381', '64756', '91647', '93974', '8455', '26033', '25814', '6310', '342371', '6311', '11273', '222255', '127002', '56970', '552889', '550', '6791', '6795', '10677', '23080', '60370', '64343', '283358', '338707', '79870', '25825', '60468', '85319', '57597', '22893', '575', '576', '577', '8938', '55024', '56033', '8538', '7916', '84726', '23215', '7918', '7920', '116071', '55509', '11177', '29994', '56987', '4059', '55973', '8412', '8537', '54828', '55653', '56647', '64919', '10017', '83596', '23786', '440603', '599', '602', '255877', '605', '9275', '9274', '606', '9774', '54880', '63035', '497258', '146227', '57673', '389206', '79656', '221336', '222389', '55859', '84707', '56271', '340542', '51283', '631', '8419', '168620', '80823', '80114', '638', '55909', '79444', '54841', '10904', '282991', '55589', '651', '653', '654', '353500', '656', '168667', '729096', '399761', '54796', '664', '149428', '666', '51027', '388962', '2186', '8019', '23629', '65980', '286076', '140707', '55299', '84312', '84446', '9024', '254065', '26580', '55108', '54836', '9044', '84280', '121551', '118663', '643376', '149478', '55643', '22903', '55727', '284697', '114781', '689', '91408', '10950', '54718', '10917', '11149', '9689', '28969', '11067', '414152', '219621', '414235', '55853', '221060', '399726', '83938', '55088', '79892', '84458', '170371', '387707', '26098', '387640', '54906', '220979', '54838', '27291', '119032', '219738', '100131213', '118924', '143384', '254427', '64115', '414236', '80195', '84293', '256815', '79741', '196740', '90271', '79591', '119392', '80217', '79949', '143379', '63877', '80007', '170393', '255352', '79946', '387695', '746', '64776', '56673', '56672', '25858', '29125', '53838', '56946', '280636', '256329', '25758', '160298', '219833', '120534', '79081', '79096', '91894', '341032', '28970', '55216', '10944', '79684', '79864', '160140', '220004', '28971', '83638', '85016', '54494', '119710', '56935', '79703', '220042', '790955', '144097', '254439', '399947', '399949', '387804', '399948', '120376', '65998', '745', '60314', '90488', '29902', '84190', '283460', '91298', '84915', '55196', '283422', '374470', '64897', '121053', '57102', '51275', '55010', '79794', '160419', '283450', '84934', '115749', '113246', '120939', '144608', '283416', '374467', '91574', '387856', '440087', '341346', '728568', '440107', '387882', '400073', '57213', '121793', '80183', '80209', '93081', '283487', '144809', '144811', '84935', '400120', '728591', '54916', '55051', '55172', '55195', '26175', '11161', '55237', '55668', '55017', '112487', '84837', '51527', '56967', '64430', '161176', '84520', '90141', '145508', '122945', '112849', '84334', '80017', '55449', '161394', '643382', '283579', '400223', '283551', '650662', '280655', '161424', '122525', '9556', '100129075', '645687', '145407', '91748', '80127', '64207', '145376', '388011', '283600', '145200', '91828', '122616', '283643', '60686', '57184', '283651', '90417', '56851', '161502', '123591', '80035', '79768', '196951', '80072', '283687', '348110', '56905', '123207', '84529', '90381', '84419', '196968', '388115', '256646', '644809', '90416', '390637', '388135', '145853', '643338', '25764', '84326', '750', '89927', '123775', '84080', '730094', '29965', '283897', '124045', '79650', '64755', '80178', '123811', '80262', '146562', '29035', '404550', '283870', '9605', '29105', '388284', '388272', '400506', '146556', '283951', '90835', '388327', '79701', '256302', '284071', '100130311', '201229', '283987', '84299', '79018', '79736', '284029', '124783', '124944', '146853', '339263', '78995', '284185', '146705', '124989', '284018', '54785', '284021', '254863', '79415', '55731', '339201', '256957', '339210', '147081', '92340', '55018', '64149', '388341', '146723', '284099', '55028', '388407', '55421', '654434', '388325', '284184', '339229', '84981', '360205', '100170841', '400566', '100141515', '147429', '753', '147525', '125228', '83608', '79863', '147339', '497661', '374864', '85019', '162681', '494514', '29919', '83636', '147685', '126353', '91300', '148046', '55009', '148223', '255057', '126321', '8725', '64073', '374872', '255809', '126074', '79086', '79002', '84167', '374877', '163183', '126526', '84798', '79036', '352909', '90580', '28974', '284325', '148137', '51398', '79173', '55049', '284361', '55337', '100170765', '91304', '100128569', '55150', '199800', '284422', '257044', '55791', '284618', '55765', '54955', '55732', '79729', '57821', '79762', '79098', '127687', '54987', '126859', '200197', '148345', '57095', '400746', '128061', '574036', '79000', '26099', '767846', '92342', '284573', '54991', '126868', '128346', '65260', '199920', '126695', '127254', '339448', '374977', '163747', '439927', '128229', '55924', '440712', '374946', '388701', '541468', '257177', '127003', '84886', '90529', '84852', '284677', '149466', '113444', '148898', '127703', '81563', '400798', '374973', '400793', '339541', '388759', '284485', '81627', '54823', '54953', '9473', '25912', '79078', '148523', '148423', '388722', '79630', '54964', '84284', '148362', '10485', '57035', '149563', '51093', '200205', '339453', '148304', '127428', '149469', '112770', '128344', '79363', '56063', '51430', '375057', '126731', '84791', '10882', '389941', '338761', '114897', '114898', '114899', '114900', '114904', '114905', '338872', '387911', '51279', '24141', '200232', '388799', '116151', '51526', '140688', '54994', '140710', '140711', '55184', '140699', '170487', '140701', '140893', '140894', '140706', '128497', '63939', '25943', '79025', '149840', '284756', '441951', '253868', '198437', '400831', '284805', '55969', '26074', '54976', '55317', '29058', '57136', '25980', '51507', '55321', '140828', '92667', '128602', '128710', '140680', '84996', '150142', '728039', '284836', '150147', '150135', '284835', '54094', '755', '54084', '8209', '388815', '54067', '84221', '54059', '54058', '56683', '56245', '59271', '84536', '85395', '282566', '56911', '391267', '114036', '114041', '114043', '54149', '149992', '83606', '150248', '84645', '25775', '128989', '55267', '150291', '79680', '253143', '25770', '348645', '388886', '128977', '150383', '51233', '646023', '79640', '23313', '25966', '9854', '145741', '388125', '126567', '100191040', '150590', '84226', '54978', '27013', '29798', '51374', '92749', '84417', '54980', '60526', '348738', '130813', '151477', '79919', '343990', '285154', '375307', '130162', '493753', '130951', '401027', '151050', '388969', '205327', '339778', '257407', '339804', '130355', '129881', '84279', '391356', '388963', '389084', '653140', '51057', '84281', '129293', '57415', '89876', '25871', '51161', '51244', '152002', '285343', '84319', '132001', '51066', '285315', '84984', '339883', '80111', '56941', '285237', '84657', '132228', '339942', '132200', '93556', '79669', '389119', '152078', '205428', '151963', '375341', '23272', '285203', '646600', '677779', '200844', '285382', '646450', '401089', '317648', '404201', '84273', '55286', '55345', '54969', '80167', '79625', '132720', '132321', '401152', '201895', '132989', '285555', '152641', '152756', '92070', '55319', '345222', '201725', '441054', '401115', '84709', '389203', '260436', '9315', '56951', '140947', '55322', '79614', '134553', '375484', '202299', '64417', '90355', '375444', '285600', '153571', '389289', '10826', '65250', '643155', '51149', '389336', '134121', '285636', '100190949', '492311', '63920', '116349', '441108', '133874', '285679', '85027', '79747', '84830', '64771', '85411', '58527', '168090', '221491', '401288', '26238', '653483', '84300', '389383', '154467', '221443', '647024', '221545', '442213', '135398', '90523', '221749', '387097', '29113', '221424', '79940', '57150', '206412', '63914', '154313', '253714', '84553', '221322', '387104', '90632', '285753', '728464', '116843', '404220', '51250', '387119', '80069', '79624', '100131814', '389384', '221416', '619208', '441150', '401253', '401251', '80737', '729515', '100133205', '57827', '50854', '347744', '135154', '79992', '81688', '55776', '55780', '116254', '221481', '221477', '80129', '79783', '136647', '129790', '79161', '79020', '79034', '221927', '113763', '115416', '136895', '57002', '285958', '222166', '55069', '55262', '340277', '221908', '78996', '84310', '222950', '375607', '286006', '27099', '154791', '136288', '79974', '402573', '79846', '84060', '84792', '51236', '286122', '65265', '116328', '157657', '55472', '114926', '80185', '157695', '56260', '157777', '56892', '254778', '203111', '157773', '78998', '51337', '541565', '401466', '642475', '619351', '84933', '286103', '389643', '286144', '441376', '340393', '375748', '414328', '286333', '100128385', '51490', '138162', '286207', '375757', '158228', '90871', '84302', '392307', '445577', '100128782', '138724', '401563', '89958', '286257', '286343', '401546', '51531', '158055', '79095', '375791', '389799', '389813', '441476', '195827', '138716', '84688', '203259', '91283', '85026', '55071', '257169', '158314', '81571', '55848', '375759', '23731', '84267', '54942', '157983', '55064', '90120', '84850', '11094', '169693', '203228', '58493', '79886', '158401', '138241', '55684', '84270', '11092', '203197', '203238', '169436', '56934', '770', '340591', '767', '56997', '57010', '164633', '26256', '57685', '778', '93589', '27101', '199731', '93664', '8618', '285782', '794', '57658', '10241', '51063', '810', '91860', '163688', '83698', '50632', '57118', '57172', '55450', '94032', '79012', '157922', '23271', '23261', '23125', '23066', '822', '4076', '65981', '84698', '828', '133690', '29775', '79092', '114769', '84674', '23589', '55259', '255082', '113201', '64921', '57524', '57513', '120329', '57091', '54897', '440278', '863', '23624', '869', '147381', '643866', '140689', '55871', '150472', '445571', '644019', '23492', '54862', '147372', '92922', '79839', '388389', '112942', '29903', '203260', '255101', '256309', '201973', '153733', '220136', '29070', '93233', '164592', '150275', '90060', '79635', '160857', '115098', '202243', '90693', '133957', '223075', '81576', '152206', '55610', '79879', '84229', '64753', '339230', '165055', '285025', '84865', '9720', '284047', '348254', '64770', '57639', '159686', '130940', '91050', '284992', '115948', '100129792', '283152', '645811', '147872', '550631', '80071', '339965', '126075', '347475', '126661', '149483', '343099', '25790', '64793', '149473', '55246', '25901', '79140', '728621', '80125', '91057', '339834', '348807', '83643', '339829', '55036', '387885', '124808', '90799', '201134', '57003', '79825', '152137', '79714', '152185', '51019', '28958', '284001', '131076', '160777', '729440', '84660', '92558', '146439', '85478', '285331', '159989', '8030', '80323', '26112', '64925', '221016', '51372', '493860', '90557', '91409', '253635', '84318', '124093', '151887', '60494', '79780', '338657', '114800', '11007', '317762', '79080', '55231', '283234', '220388', '63933', '60492', '55297', '26093', '80212', '54520', '257236', '90324', '54535', '881', '6358', '348249', '6359', '6362', '6369', '10344', '6355', '83605', '57820', '85417', '23582', '10983', '645121', '54619', '79616', '57018', '81669', '10309', '219771', '151195', '9236', '643253', '283316', '8763', '388611', '913', '57124', '80381', '10421', '9936', '951', '921', '923', '969', '924', '3732', '9308', '81602', '146059', '8872', '168448', '166979', '55561', '56882', '157313', '83461', '55038', '83879', '55536', '64866', '200008', '1014', '28316', '64405', '60437', '92211', '54825', '222256', '389118', '53841', '8558', '5218', '5127', '5128', '5129', '23552', '8099', '10263', '1018', '51654', '80279', '8814', '8999', '51265', '344387', '6792', '55602', '91368', '100048912', '1033', '441549', '1038', '1039', '30850', '374286', '146822', '94158', '284040', '55573', '1044', '124359', '9425', '388551', '56971', '90273', '388550', '1089', '10153', '27443', '100130717', '27440', '27439', '100130418', '1056', '23436', '10658', '10659', '11189', '56853', '60680', '60677', '1057', '9620', '1952', '1951', '752014', '51286', '1059', '92806', '201161', '153241', '9859', '645455', '9857', '55165', '23177', '51148', '375298', '51716', '283848', '1070', '729338', '10428', '1073', '142913', '116828', '114336', '94027', '8545', '84952', '10669', '10668', '1101', '150356', '10036', '8208', '1106', '26038', '84181', '55636', '55743', '1117', '53344', '26511', '386593', '57132', '140578', '26973', '63928', '54108', '8646', '25884', '1140', '91612', '10519', '23152', '27141', '152302', '148113', '51550', '9541', '1153', '55847', '493856', '284106', '25792', '26586', '150468', '51192', '1164', '9629', '23155', '56650', '125875', '6320', '161198', '23274', '388512', '348174', '497190', '283971', '51267', '154790', '165530', '440508', '283420', '160365', '1047', '1192', '9022', '25932', '54102', '7461', '79745', '1195', '1196', '1197', '1198', '57396', '574028', '574016', '79789', '1203', '54982', '2055', '116449', '81570', '1209', '81037', '119467', '22883', '64084', '9746', '27098', '171425', '8418', '152100', '80790', '129607', '113540', '146225', '123920', '146223', '112616', '152189', '202333', '150384', '84735', '84518', '1261', '29097', '154043', '1264', '1266', '26507', '54805', '26505', '26504', '57472', '1267', '10330', '245812', '25927', '163882', '124817', '79935', '54875', '26047', '79937', '116840', '23242', '22837', '1690', '8161', '646300', '8292', '118881', '51117', '57175', '23603', '7464', '10391', '84940', '79585', '84701', '10328', '125965', '1346', '1347', '1358', '51200', '93979', '57094', '27151', '64506', '132864', '22849', '80315', '10814', '594855', '221184', '131034', '57699', '151835', '130749', '642843', '126129', '54504', '56265', '119587', '8532', '1379', '57585', '23418', '286204', '9586', '1389', '58487', '200407', '78987', '79174', '1390', '51232', '1396', '1397', '401262', '285464', '9419', '83690', '51379', '9696', '84809', '114819', '55118', '1407', '1408', '1412', '1413', '1414', '1415', '1417', '131544', '155051', '1427', '1429', '9946', '389903', '8531', '440359', '27254', '7812', '1434', '64478', '114784', '114788', '122011', '161635', '53944', '1456', '283106', '79848', '64651', '81566', '80034', '1466', '1469', '1470', '1472', '1474', '8530', '128817', '196993', '64693', '100128553', '340307', '643854', '1486', '140690', '58190', '10217', '51496', '115908', '29119', '8727', '1501', '1506', '83992', '55917', '404217', '404093', '79004', '51596', '23316', '50624', '55280', '80157', '646243', '440224', '9547', '284340', '3580', '80231', '170063', '256643', '51260', '645090', '79742', '159013', '91966', '541578', '158801', '54967', '100132994', '653687', '55086', '254158', '286464', '158830', '80319', '1534', '284613', '11068', '124637', '124936', '220002', '50626', '57404', '339761', '1556', '1564', '260293', '199974', '9595', '54360', '23384', '92521', '116159', '27065', '23500', '1602', '168002', '147906', '55152', '1611', '92196', '26528', '9802', '1620', '80174', '79007', '55861', '51163', '25853', '340578', '90379', '285429', '285761', '131566', '341019', '51473', '149069', '8642', '54798', '9201', '166614', '85443', '149095', '127579', '23564', '84301', '115265', '23109', '1652', '100037417', '1662', '84771', '440081', '8886', '55308', '11269', '57062', '203522', '55661', '55794', '64794', '55510', '79009', '317781', '79039', '57696', '54606', '83479', '55601', '91351', '10522', '162989', '9191', '54849', '64789', '8562', '55635', '120863', '64798', '91614', '28955', '25786', '8220', '9993', '8214', '85359', '27294', '115817', '79154', '79758', '147015', '10202', '728635', '317749', '51635', '207063', '54505', '22907', '55760', '56919', '9704', '79665', '653645', '90957', '64150', '23181', '57609', '22982', '148252', '54769', '9077', '84925', '729582', '115752', '27185', '84976', '85458', '93429', '222161', '401232', '91056', '374387', '27122', '27120', '9940', '10301', '8847', '79469', '220107', '9228', '65989', '1745', '1746', '1747', '1748', '285987', '127343', '93099', '10655', '58524', '63951', '63950', '728656', '9988', '1762', '1657', '23312', '196385', '8701', '201625', '25981', '127602', '8632', '146754', '55567', '1767', '1768', '56171', '1769', '1770', '27019', '64446', '9093', '54788', '374407', '79982', '3300', '11080', '25822', '150353', '54431', '56521', '85406', '29103', '23341', '64215', '55192', '202052', '134218', '79962', '548645', '51277', '54943', '84277', '3338', '85479', '23234', '83544', '7802', '1773', '1775', '1776', '58511', '373863', '144132', '728489', '100128285', '23549', '116092', '30836', '8448', '8447', '285489', '1797', '29980', '23033', '9980', '25911', '135656', '8193', '5977', '8110', '100132911', '57628', '1804', '54878', '91039', '503645', '1805', '23333', '283417', '554236', '349152', '442523', '147991', '286148', '165545', '55332', '128338', '10311', '53820', '257203', '11034', '25778', '92675', '401124', '1837', '1838', '56986', '285605', '196403', '90527', '405753', '64118', '56931', '11062', '8446', '11266', '11072', '128853', '150290', '142679', '56940', '54935', '78986', '92235', '285193', '574029', '344875', '84332', '54808', '6990', '9149', '8444', '8798', '116729', '161582', '199221', '1877', '55837', '64641', '253738', '57593', '84650', '11319', '9427', '55862', '55268', '79746', '1842', '641700', '345930', '8721', '10085', '441032', '196549', '100130771', '79645', '84288', '146779', '283229', '374786', '64800', '151651', '114327', '80258', '80303', '23167', '22979', '10278', '79631', '25975', '51162', '80864', '133584', '100126791', '23301', '254102', '30844', '26298', '9538', '23741', '163126', '126272', '10209', '84285', '10289', '83939', '728689', '442720', '1979', '8637', '56478', '143244', '3692', '55520', '1993', '1995', '1996', '2000', '2001', '392617', '114794', '2004', '2005', '79767', '55531', '255520', '84173', '84337', '100131454', '64123', '51705', '129080', '2009', '24139', '256364', '161436', '400954', '2012', '2013', '2014', '326342', '2016', '196047', '2020', '8507', '2021', '8909', '2022', '375704', '219670', '55556', '55068', '10495', '59084', '150350', '347918', '64097', '54566', '26122', '54749', '79852', '253152', '9852', '55040', '83481', '8475', '54869', '79574', '1833', '3266', '23085', '26059', '2078', '57222', '51290', '51614', '2079', '112479', '79033', '157697', '114625', '57471', '79956', '10595', '121506', '10961', '54206', '405754', '51575', '11082', '284729', '54465', '2115', '2116', '2117', '440695', '2119', '51513', '2123', '2124', '7813', '115704', '645027', '2128', '2130', '161829', '55218', '54932', '283849', '90332', '23233', '9941', '23086', '346007', '2145', '8263', '220832', '3995', '283985', '11124', '51011', '151313', '23017', '9214', '55179', '124402', '283991', '144347', '359845', '399665', '284611', '83640', '84923', '90736', '54491', '80039', '100129396', '11170', '83641', '144717', '150368', '145165', '83541', '90362', '642273', '63901', '374393', '64773', '91523', '92689', '9747', '285966', '81558', '150864', '55007', '79607', '25895', '23196', '158293', '54954', '116224', '159090', '159091', '205147', '220108', '79843', '84668', '285172', '8933', '26071', '441518', '116496', '64855', '199786', '131408', '348487', '388581', '257415', '79137', '54463', '162427', '51059', '84908', '387071', '285512', '220965', '25854', '317662', '338094', '167555', '285596', '283726', '728215', '27112', '29057', '728262', '100132403', '51016', '348378', '729830', '84067', '57700', '64760', '84140', '145483', '26355', '221303', '148753', '51101', '79696', '54065', '401565', '730112', '83648', '84734', '23201', '130074', '283777', '221061', '284069', '165215', '83989', '65990', '134145', '345757', '400451', '55194', '283635', '400823', '55719', '51252', '165186', '23116', '389558', '220382', '284800', '728882', '440585', '79632', '27146', '222234', '121006', '84070', '80013', '84182', '23359', '9413', '10712', '780776', '201158', '51030', '401145', '54462', '80011', '8603', '54540', '131831', '84331', '348262', '642938', '100131897', '729085', '51313', '139231', '407738', '338811', '284467', '25817', '221786', '285550', '9917', '387680', '55747', '253725', '728118', '728130', '54754', '441457', '653567', '196792', '254228', '441168', '548321', '100132948', '54537', '439965', '414241', '9780', '63895', '60343', '54097', '131177', '85369', '57464', '284593', '131583', '163933', '404636', '55855', '55603', '115572', '54855', '100129583', '81553', '26240', '152877', '9679', '51307', '113115', '56181', '120400', '120406', '91775', '54827', '79850', '83723', '92002', '339521', '64762', '57795', '339479', '58516', '55793', '54629', '54478', '79567', '140876', '100133172', '440078', '100132923', '100132103', '388650', '138311', '125704', '55026', '348013', '161142', '112703', '84691', '346653', '729533', '653820', '728833', '441452', '199870', '143684', '286336', '149297', '100288380', '89837', '145773', '153643', '151393', '55177', '51115', '128876', '54854', '113828', '644815', '286077', '151354', '157638', '85002', '653333', '55199', '692099', '375061', '23625', '51439', '55138', '137392', '339145', '100133036', '84191', '25940', '147965', '92565', '2191', '79675', '79072', '2195', '2196', '120114', '79633', '89885', '345630', '129804', '64319', '57666', '25827', '26233', '157574', '84893', '692224', '26261', '26260', '23219', '26273', '254170', '55030', '130888', '81545', '162517', '54455', '286151', '200933', '23403', '554251', '26269', '8857', '2210', '100132417', '9103', '2217', '89848', '9873', '115350', '79368', '115352', '83416', '343413', '84824', '127943', '619190', '91893', '80307', '90342', '654463', '55612', '9638', '9637', '389549', '55785', '55277', '2267', '114827', '85462', '2272', '2273', '2275', '9457', '29109', '80206', '84929', '387758', '9158', '11153', '55137', '63979', '401720', '11259', '84922', '24147', '60681', '51303', '23307', '2282', '2286', '2287', '51661', '360132', '79147', '2218', '2314', '55056', '55096', '286042', '54508', '440101', '79667', '79857', '80094', '100192386', '401237', '200058', '400710', '401172', '644873', '401491', '284131', '255031', '649446', '168455', '730971', '283011', '283521', '729614', '439931', '202020', '388685', '645784', '285962', '388182', '401105', '645644', '441094', '440556', '378805', '401207', '392490', '400242', '402483', '399844', '283102', '440465', '55640', '84256', '114984', '342184', '56776', '2329', '2330', '2332', '23360', '84624', '22862', '64778', '64838', '252995', '54752', '219595', '2355', '2297', '2306', '2298', '200350', '100036519', '286380', '653404', '2301', '2295', '2299', '2302', '55810', '22887', '2300', '8456', '3344', '1112', '121643', '2310', '93986', '116113', '94234', '55572', '80020', '2307', '80144', '158326', '341640', '2483', '284802', '79981', '257019', '55691', '23150', '84978', '122786', '83786', '22844', '143162', '391059', '10129', '285527', '2487', '29999', '79187', '83856', '161835', '23105', '2498', '55783', '23070', '8880', '8939', '65991', '388965', '9513', '53827', '79443', '55632', '23766', '126626', '2563', '2564', '2568', '4616', '10912', '64090', '79690', '2585', '130589', '2595', '202309', '84253', '10634', '246176', '283431', '60674', '8522', '57798', '283459', '352954', '389523', '2630', '26301', '257144', '94104', '8521', '2651', '51301', '145781', '78997', '54834', '2662', '10220', '9518', '9573', '392255', '151449', '2661', '54857', '492303', '8328', '81577', '199720', '79893', '2679', '645367', '2681', '91227', '27069', '100126793', '64599', '26058', '170575', '26157', '55303', '55340', '474344', '168537', '155038', '54826', '54810', '126326', '349149', '2713', '256356', '80318', '113263', '342035', '2738', '144321', '152007', '148979', '84662', '169792', '51031', '392465', '51022', '144423', '55830', '83468', '29998', '29997', '84656', '64395', '64396', '10691', '26205', '2764', '54584', '2794', '29889', '54552', '79158', '84572', '114814', '55592', '55889', '647042', '283767', '374650', '727832', '440295', '401647', '23015', '440270', '729786', '100132979', '283768', '283796', '64083', '55204', '127845', '51026', '54856', '92344', '10223', '55105', '63906', '54865', '23131', '65056', '60313', '2823', '2824', '11321', '54707', '51184', '57720', '56927', '266977', '222611', '165082', '221188', '221393', '25960', '166647', '2825', '84873', '283383', '64582', '56834', '7107', '283554', '353345', '115330', '390212', '387509', '151556', '165829', '80045', '57512', '26996', '27239', '29909', '54328', '84636', '440435', '160897', '11318', '2842', '2844', '2845', '2827', '2857', '9289', '118442', '81491', '10149', '10936', '27197', '54329', '53836', '54112', '653519', '51463', '728932', '9737', '114928', '9052', '51704', '55890', '55507', '9721', '285513', '57655', '57476', '54762', '196996', '65983', '23151', '400581', '160622', '80000', '79977', '57822', '2894', '392862', '2907', '81488', '56850', '79927', '79774', '83743', '284110', '55876', '56169', '146395', '83903', '79807', '442245', '79712', '11036', '653238', '728340', '2969', '2970', '9569', '84163', '389524', '401375', '85865', '9567', '23560', '26164', '29083', '121355', '2978', '2974', '60558', '51454', '387036', '375513', '728411', '653188', '387751', '283464', '727936', '2996', '64412', '3001', '3003', '283120', '8971', '55506', '9555', '767811', '440093', '9563', '3026', '84264', '57817', '9001', '60484', '145864', '404037', '768096', '768097', '283254', '594842', '10456', '26959', '3049', '3052', '54985', '493812', '414777', '352961', '253018', '80867', '54435', '80868', '3059', '10866', '51020', '374659', '84717', '50810', '84064', '81932', '139324', '54919', '55027', '399671', '25938', '54497', '63897', '727957', '51696', '57493', '92797', '3070', '113510', '9931', '51409', '253012', '341208', '400322', '440362', '64224', '54626', '57801', '55502', '790952', '8820', '284004', '10614', '124790', '164045', '100128124', '3087', '84439', '79802', '10086', '11148', '11147', '64645', '84641', '84278', '23119', '51751', '192286', '123346', '25988', '3094', '135114', '10114', '147746', '8479', '337875', '3096', '3097', '3108', '3116', '3128', '3136', '3137', '3139', '3131', '3142', '79618', '83872', '10363', '3149', '3151', '9324', '10473', '79366', '22993', '10042', '51617', '3167', '51155', '90861', '10949', '144983', '10151', '3182', '3189', '11100', '221092', '92906', '57594', '51361', '29911', '84376', '84525', '84072', '100124700', '3206', '221883', '3209', '3202', '3204', '3205', '10481', '3215', '3216', '3217', '3218', '3219', '3226', '3222', '3223', '3225', '3236', '3239', '3234', '3235', '50809', '3208', '3241', '51440', '84842', '84343', '11234', '79803', '55806', '646962', '64342', '440498', '374875', '158160', '83693', '84263', '3298', '11077', '3299', '100130086', '84941', '378465', '391634', '664618', '343477', '3311', '3316', '8988', '126393', '27129', '94086', '79663', '23640', '29092', '29094', '10553', '93164', '94031', '203100', '30811', '135458', '54768', '81888', '219844', '57061', '285148', '25998', '3382', '130026', '22858', '84099', '9592', '51124', '51278', '389792', '25900', '126917', '122509', '83982', '10561', '10964', '338376', '3475', '7866', '3476', '9543', '57722', '51214', '347252', '374918', '147920', '444882', '91156', '3508', '91353', '402665', '285313', '152404', '3547', '84966', '283284', '3321', '150084', '10261', '93185', '57549', '22997', '22807', '22806', '64375', '64376', '27190', '53342', '400935', '286676', '387597', '80895', '10994', '196294', '83943', '3617', '50939', '9118', '10207', '388324', '8552', '8551', '64423', '3621', '27160', '387755', '723961', '3645', '285905', '644619', '27130', '26034', '51194', '9670', '79711', '10527', '55705', '3652', '3653', '79781', '55721', '115811', '132141', '84223', '64799', '124152', '9922', '23096', '154865', '134728', '26145', '359948', '126298', '79191', '10265', '79190', '389293', '64843', '57611', '140862', '145501', '51015', '79763', '729920', '91464', '81533', '83986', '9270', '26548', '3697', '80760', '347365', '142683', '9452', '81618', '85450', '150771', '162073', '10625', '84522', '152789', '9832', '282973', '221895', '122953', '51528', '339123', '56704', '57158', '57338', '84502', '8629', '8690', '126306', '10899', '353219', '25959', '256949', '163782', '11104', '84056', '83473', '10300', '81621', '9920', '166348', '25948', '143879', '55709', '3769', '10984', '283518', '57582', '343450', '375616', '83892', '147040', '115207', '65987', '57528', '79734', '130535', '146212', '222658', '283219', '23510', '54442', '54793', '79070', '143888', '387628', '654466', '80759', '23351', '9933', '9674', '9808', '23065', '9703', '57291', '9834', '23199', '9772', '9897', '9711', '9778', '23506', '9766', '283638', '79932', '23392', '9675', '9907', '9811', '9665', '9813', '57212', '9764', '23247', '9731', '23078', '9786', '9858', '23277', '100132341', '9840', '9851', '23499', '23255', '23366', '653319', '22889', '23240', '23251', '23325', '23349', '84162', '57456', '57189', '57462', '57179', '57481', '57482', '56243', '57495', '57221', '57501', '27143', '26128', '57535', '222223', '57536', '56204', '57562', '54627', '57577', '25962', '57587', '57608', '57613', '57614', '57624', '57648', '57650', '57653', '80256', '57662', '57691', '57707', '57710', '57724', '85352', '85379', '80726', '55425', '80817', '80856', '85459', '85457', '85452', '85449', '54914', '84451', '84437', '84542', '340390', '114796', '170954', '158405', '84960', '90231', '205717', '340533', '158358', '100144748', '7071', '10661', '8462', '11278', '51621', '136259', '83855', '128209', '10365', '51274', '1316', '8609', '11279', '687', '23008', '122773', '23588', '54758', '127707', '113730', '55220', '200942', '126823', '317719', '57565', '80311', '339451', '23276', '151230', '54800', '55295', '54813', '114818', '377007', '401265', '114792', '257240', '283212', '79786', '340359', '56062', '89857', '55975', '57563', '5655', '11012', '55554', '9622', '5653', '10748', '129285', '3823', '346689', '85442', '124751', '51315', '65095', '889', '125113', '200185', '200634', '112970', '339855', '54596', '84456', '91133', '246269', '114294', '51110', '3898', '22973', '22798', '27074', '10314', '55915', '347404', '51056', '9741', '55353', '7805', '55132', '113251', '23185', '55323', '51574', '3927', '54900', '143903', '81606', '85474', '390598', '167691', '150082', '414332', '138307', '401562', '254251', '197021', '9079', '11155', '388633', '401944', '143458', '23641', '84247', '389170', '93273', '114823', '94059', '54741', '23484', '137994', '25875', '85329', '3957', '3960', '3963', '653499', '3964', '284194', '654346', '51557', '1939', '10186', '340596', '375612', '375323', '26468', '79168', '11025', '79166', '51474', '22998', '80774', '54923', '96626', '100271835', '158038', '339398', '55180', '9516', '167410', '128077', '3996', '3993', '84298', '64327', '55716', '92255', '89782', '84823', '55885', '8543', '56203', '22853', '114783', '4009', '4010', '348801', '222484', '100009676', '100101266', '100124692', '100125556', '100126784', '100127888', '100128164', '100128191', '100128239', '100128288', '100128292', '100128542', '100128573', '100128640', '100128675', '100128788', '100128822', '100128842', '100129034', '100129066', '100129387', '100129534', '100129550', '100129637', '100129716', '100129726', '100130015', '116841', '100130238', '100130331', '100130522', '100130557', '100130581', '100130691', '100130776', '100130872', '100130932', '100130933', '100130987', '100131193', '100131434', '100131496', '100131551', '100131691', '100131726', '100132111', '100132215', '100132247', '100132287', '100132288', '100132354', '100132707', '100132724', '100132832', '100133161', '100133331', '100133545', '100133612', '100133669', '100133957', '100133985', '100133991', '100134229', '100134259', '100134368', '100134713', '100134868', '100144603', '100144604', '100170939', '100188947', '100188949', '100189589', '100190938', '100190939', '100190940', '100190986', '100216001', '100216545', '100233209', '100240735', '100268168', '100270710', '100270746', '100270804', '100271722', '100271836', '100272146', '100272216', '100272217', '100272228', '100286793', '100286844', '100287227', '100288778', '100289341', '100302401', '100302640', '100302650', '100303728', '113230', '115110', '121838', '121952', '127841', '134466', '143188', '143666', '144438', '144486', '144571', '145783', '145820', '145837', '146336', '146880', '147727', '147804', '148189', '148413', '148696', '148709', '149134', '149837', '150197', '150381', '150776', '150786', '151009', '151162', '151174', '151534', '152217', '153328', '153684', '154761', '154822', '157381', '158376', '158572', '162632', '168474', '200030', '201651', '202181', '202781', '219347', '220429', '220594', '220729', '220930', '221442', '221710', '222699', '253039', '253724', '254559', '255167', '256880', '257358', '25845', '282997', '283050', '283070', '283267', '283314', '283663', '283731', '283867', '283922', '284009', '284023', '284100', '284232', '284233', '284276', '284440', '284441', '284578', '284749', '284837', '284900', '285033', '285074', '285359', '285419', '285456', '285548', '285593', '285629', '285768', '285830', '285954', '286002', '286367', '286467', '338651', '338758', '338799', '339047', '339290', '339524', '339535', '339674', '341056', '342346', '344595', '344967', '347376', '348926', '349114', '349196', '374443', '374491', '375190', '387646', '387647', '388152', '388242', '388588', '388692', '388789', '388796', '388955', '389332', '389333', '389458', '389493', '389634', '389705', '389791', '390595', '391322', '399744', '399815', '399959', '400027', '400043', '400657', '400696', '400752', '400759', '400794', '400891', '400927', '400931', '401010', '401052', '401093', '401127', '401387', '401397', '401431', '401588', '402377', '407835', '440173', '440354', '440356', '440461', '440563', '440896', '440905', '440925', '440944', '440957', '441046', '441089', '441204', '441208', '441294', '441454', '441455', '441666', '441869', '442308', '442421', '442454', '442459', '493754', '494141', '541471', '541473', '550112', '550643', '554202', '572558', '595101', '606724', '613037', '619207', '641298', '641367', '642826', '642846', '642852', '643008', '643387', '643677', '643719', '286183', '643837', '644165', '644172', '644538', '644936', '645166', '645323', '645332', '645431', '645676', '646214', '646471', '646762', '646851', '646982', '646999', '647121', '647288', '647859', '647946', '647979', '648740', '649330', '650368', '650623', '651250', '652276', '653113', '653501', '653566', '653653', '653786', '654342', '654433', '678655', '723809', '723972', '727896', '728024', '728190', '728264', '728323', '728392', '728554', '728606', '728613', '728640', '728643', '728723', '728743', '728758', '728819', '728855', '728875', '728989', '729082', '729156', '729176', '729234', '729375', '729603', '729678', '729799', '729991', '730101', '730668', '731789', '80054', '80154', '81691', '84740', '84856', '84931', '84989', '90110', '90246', '90586', '90784', '90834', '91316', '91450', '91948', '92249', '92659', '51754', '93622', '96610', '118426', '503693', '29931', '164832', '79836', '125336', '80350', '9170', '22859', '23266', '23284', '4026', '9404', '642946', '987', '23143', '57631', '84859', '4034', '145581', '136332', '9860', '121227', '340745', '345193', '84918', '53353', '55805', '4037', '91355', '4043', '390205', '389257', '131578', '90668', '10234', '474354', '64922', '55222', '10233', '441381', '126364', '389816', '80313', '79442', '123355', '26231', '79782', '2615', '375387', '151827', '55282', '474170', '374819', '55073', '9884', '114659', '147172', '81543', '127495', '116135', '55631', '115353', '254050', '201255', '64101', '90506', '57470', '83450', '57689', '123872', '219527', '115399', '255252', '116064', '55379', '65999', '23639', '339977', '100130742', '100130733', '57554', '85444', '9209', '84125', '127255', '344657', '79705', '57633', '10446', '54674', '164312', '221091', '654429', '222229', '55341', '124801', '26065', '149986', '4046', '7940', '4058', '51631', '55692', '7798', '338645', '767558', '56925', '58496', '80741', '79136', '4065', '4063', '55646', '129530', '254773', '66004', '130574', '653639', '127018', '57149', '57226', '144363', '90624', '388695', '256586', '116068', '145748', '1130', '84328', '8216', '11178', '84445', '10586', '346389', '28992', '140733', '9587', '10296', '4110', '4101', '4102', '4105', '728239', '81557', '57692', '64110', '28986', '54551', '9223', '260425', '79917', '84549', '4117', '114569', '4118', '378938', '7851', '256691', '158056', '284358', '63905', '149175', '54682', '4130', '254042', '643246', '55201', '4133', '4294', '7786', '9175', '389840', '10746', '4216', '9064', '4293', '4134', '11184', '5871', '8491', '11183', '4135', '79929', '9053', '55700', '256714', '79649', '79884', '225689', '93487', '9479', '23542', '23162', '23005', '10982', '22924', '162333', '55016', '51257', '115123', '57574', '54708', '64844', '220972', '92979', '65108', '4139', '2011', '83742', '153562', '91862', '22983', '23139', '23031', '375449', '4147', '9782', '4150', '8512', '255374', '153364', '4154', '10150', '55796', '54799', '92014', '147407', '442229', '401612', '4163', '23101', '8888', '114044', '254394', '79772', '55784', '28985', '4188', '29969', '130752', '56890', '23195', '145553', '259283', '4201', '116931', '55384', '84466', '84465', '1953', '1954', '1955', '150365', '644890', '4212', '56917', '4213', '9833', '4222', '4223', '4224', '4225', '56257', '59274', '23184', '4232', '317751', '79006', '284207', '79066', '64745', '196074', '28989', '751071', '51603', '339175', '55798', '64863', '29081', '131965', '196410', '79828', '51108', '92312', '84206', '399664', '9848', '56947', '9258', '83552', '10227', '79157', '64747', '113655', '148808', '84975', '54842', '162387', '84179', '256471', '84804', '146664', '84815', '84793', '84983', '85009', '84848', '84849', '85001', '114130', '113691', '389741', '196872', '197187', '65996', '51237', '81854', '167359', '79100', '90768', '401884', '414919', '389538', '256227', '10724', '4256', '8190', '440823', '4276', '9645', '57553', '84953', '85377', '79778', '11043', '90007', '57708', '54531', '166968', '57409', '60672', '4285', '145282', '114614', '407975', '129531', '4288', '84365', '57496', '4289', '2872', '23609', '7681', '283078', '2315', '23209', '4291', '8079', '55904', '8028', '10962', '4302', '79083', '22877', '23531', '79258', '8510', '79148', '79812', '4330', '55329', '4335', '3110', '64112', '81532', '126308', '79817', '148932', '25843', '23041', '23515', '79710', '10934', '79906', '729967', '283385', '118812', '254956', '56180', '64598', '26002', '8777', '219972', '54737', '10198', '84954', '4355', '4356', '58538', '51678', '65258', '744', '255027', '84769', '9019', '10205', '196264', '3140', '112609', '55686', '93621', '114932', '116535', '83876', '359821', '84311', '729633', '51154', '84689', '219995', '931', '51338', '64231', '245802', '58475', '83661', '9242', '151507', '401827', '692094', '11209', '11223', '55154', '100129405', '55545', '326343', '4500', '27085', '23787', '23788', '4515', '92140', '80298', '9650', '92170', '64779', '9633', '10903', '339483', '51537', '55149', '9617', '9788', '92154', '57509', '23281', '345778', '55745', '84939', '139221', '347273', '389125', '63915', '4084', '83463', '4601', '25878', '439921', '91663', '255275', '4608', '343263', '23077', '26292', '84073', '4610', '10408', '80177', '50804', '150678', '26579', '22989', '84176', '4625', '57644', '85366', '91807', '340156', '55892', '80022', '23026', '84700', '4640', '4430', '4642', '4643', '4542', '64005', '283446', '55930', '4648', '4653', '26509', '8736', '9172', '127294', '9499', '58529', '91977', '84665', '339344', '4661', '7593', '55728', '90634', '10443', '23138', '8260', '80155', '79612', '51126', '80018', '79829', '80218', '10004', '254827', '342538', '4666', '23148', '83955', '112939', '138151', '92345', '4668', '51172', '203245', '4671', '340719', '4673', '4674', '4675', '4676', '266812', '256236', '26502', '4678', '57106', '79903', '24142', '51471', '26151', '89796', '89797', '89795', '65065', '4681', '79804', '100132406', '25832', '284565', '55672', '728936', '285622', '84224', '148545', '653149', '343505', '400818', '10230', '4685', '342897', '23154', '654816', '654817', '344148', '57701', '56926', '8031', '57727', '135112', '147650', '399668', '644596', '100188953', '100188954', '266655', '283932', '80161', '283981', '79854', '205251', '55389', '112597', '440072', '400508', '80862', '100287569', '285908', '378832', '284739', '503538', '100302692', '554203', '125144', '387644', '319085', '286967', '114915', '64493', '23413', '80762', '54602', '4693', '57447', '57446', '65009', '56901', '283131', '64168', '54550', '63941', '4739', '4744', '4741', '152110', '79858', '4750', '4752', '6787', '341676', '284086', '26012', '4745', '29937', '10763', '81831', '93082', '84461', '91624', '10725', '84901', '4776', '4779', '9603', '4783', '84807', '4795', '4796', '64332', '27247', '4799', '152518', '25983', '51335', '4807', '283948', '57224', '340527', '60491', '51199', '4814', '4815', '8508', '25934', '55335', '4817', '79570', '154215', '128414', '222698', '85407', '85409', '4818', '284353', '55922', '4820', '159296', '4824', '54475', '204801', '55655', '171389', '199713', '338321', '51068', '283458', '8382', '10201', '4826', '64318', '79954', '8996', '8715', '51406', '55035', '9221', '64434', '23420', '283820', '408050', '4841', '51491', '9722', '388677', '4856', '4857', '4858', '4861', '64067', '4863', '79716', '9284', '23117', '10360', '4883', '4884', '4885', '23467', '100129354', '4892', '441478', '340371', '4898', '83714', '56675', '203447', '4901', '11270', '140767', '80023', '10412', '63899', '55695', '155400', '260294', '79730', '221294', '64943', '51559', '123803', '126147', '81788', '4925', '64710', '84955', '134492', '23386', '84309', '152195', '200035', '84304', '11162', '119369', '26747', '57532', '9253', '91181', '54830', '26471', '51203', '10204', '4931', '284434', '56000', '64359', '158046', '30010', '11248', '11247', '55916', '57523', '220323', '29989', '79629', '54940', '132299', '10896', '4950', '654231', '57489', '440836', '161753', '284451', '10178', '57451', '55714', '26011', '55753', '79676', '11054', '79627', '170392', '10439', '93145', '283298', '169611', '25903', '56944', '116448', '4975', '9480', '80207', '611', '441295', '26636', '79315', '93129', '220064', '23762', '127700', '64172', '29948', '734', '116039', '100128731', '58505', '202459', '54726', '51633', '5013', '124641', '5017', '58495', '92106', '55074', '9943', '286530', '54681', '641455', '647033', '80227', '80336', '728773', '5042', '8761', '132430', '140886', '135138', '133015', '23241', '5047', '51247', '400961', '55003', '23022', '342979', '5064', '54873', '5066', '25891', '167153', '64282', '390928', '89932', '64895', '124222', '54852', '79957', '164091', '85315', '8123', '117583', '25849', '57097', '64761', '165631', '10039', '347746', '25859', '64098', '23178', '197135', '23598', '5074', '5083', '55872', '64081', '5089', '5090', '80714', '57326', '84105', '54039', '57575', '27328', '83259', '51294', '5097', '27253', '54510', '64881', '5101', '56139', '56138', '56137', '56136', '56147', '56146', '56145', '56144', '56143', '56142', '56141', '56140', '9752', '56135', '56134', '56126', '56125', '56124', '56123', '56122', '56121', '57717', '54661', '54660', '84054', '56133', '56132', '56131', '26167', '56130', '56129', '56128', '56127', '56106', '56105', '26025', '56114', '56113', '56112', '56111', '56110', '56109', '56108', '9708', '56107', '56104', '56103', '56102', '8641', '56101', '56100', '56099', '56120', '5098', '56098', '56097', '200373', '84759', '10336', '55795', '63935', '115294', '55251', '57092', '22990', '80003', '399909', '542767', '5121', '654790', '27344', '54760', '9159', '51449', '11235', '5134', '84306', '9141', '10016', '79031', '347862', '9659', '64146', '5157', '64714', '171423', '9601', '149420', '9124', '64236', '27295', '8572', '81572', '23042', '283970', '57026', '23037', '57595', '79955', '118987', '5174', '10158', '728939', '23024', '29951', '375033', '157310', '553115', '23089', '5178', '53918', '5184', '5187', '8864', '8863', '5188', '92960', '375189', '441531', '27315', '93210', '84547', '267002', '267004', '161779', '79605', '5225', '10404', '5229', '5239', '595135', '54858', '221692', '116154', '65979', '5251', '55274', '51131', '57649', '148479', '9678', '112885', '79142', '23469', '84295', '51533', '7262', '23612', '23187', '90102', '653583', '493911', '29085', '57661', '10745', '57157', '254295', '9796', '84457', '51050', '221476', '728233', '375133', '10464', '55022', '55011', '120379', '113791', '11040', '415116', '5301', '5303', '54984', '54103', '206426', '138429', '266971', '644139', '26207', '5307', '168507', '114780', '342372', '9033', '91461', '10343', '5314', '93035', '5569', '5570', '11142', '63876', '647087', '84647', '9373', '10761', '257000', '191585', '153770', '219348', '5324', '5326', '196463', '5334', '23228', '55344', '257068', '345557', '26499', '144100', '51054', '58473', '55041', '79156', '79666', '57475', '130271', '79990', '55111', '9842', '440456', '23207', '389072', '84069', '51177', '5342', '729359', '440503', '55857', '126520', '51090', '5354', '5355', '5358', '5359', '57047', '57048', '57088', '83483', '57125', '84898', '23654', '135293', '83449', '5378', '441194', '441263', '5379', '5380', '5387', '5382', '5383', '139728', '25953', '154197', '9240', '10687', '29944', '114824', '84968', '55228', '57469', '285848', '10957', '25886', '282809', '134359', '127435', '79883', '5420', '79983', '57645', '23126', '26073', '5428', '246721', '548644', '84820', '100101267', '646074', '25812', '29797', '29774', '51371', '22932', '64091', '64208', '10631', '728378', '5450', '25833', '5459', '5462', '5463', '11281', '100130449', '56342', '692312', '84814', '79144', '51535', '653598', '645142', '644591', '5480', '728448', '10105', '285755', '5496', '57460', '333926', '132160', '147699', '5514', '6992', '54776', '26472', '81706', '54866', '84919', '84988', '26051', '5502', '151242', '5504', '5505', '153743', '79660', '5509', '90673', '89801', '648791', '5510', '55607', '84687', '5521', '5522', '5523', '55012', '5524', '9989', '55370', '57718', '51029', '27351', '160760', '80148', '130814', '11230', '84106', '23532', '118471', '5544', '768206', '56980', '56981', '59335', '59336', '63977', '7799', '11107', '93166', '344405', '153768', '5550', '9581', '10216', '5554', '5555', '166336', '4007', '29964', '145270', '100169750', '112464', '5612', '5616', '90826', '3275', '56341', '28997', '29053', '147011', '8842', '150696', '9762', '11212', '5629', '283571', '55119', '55015', '25766', '5961', '5630', '5635', '5636', '55771', '57479', '54458', '78994', '222171', '79170', '51334', '285800', '284338', '163154', '255783', '80742', '11272', '553158', '79899', '80758', '133619', '5638', '5639', '79056', '80863', '112476', '285368', '401399', '5396', '51450', '8492', '10279', '64063', '83886', '124221', '260429', '167681', '146547', '377047', '29122', '339105', '56952', '283659', '158471', '58497', '84249', '23362', '23550', '5662', '100101490', '387590', '5681', '8624', '56984', '84262', '389362', '170679', '170680', '100130889', '55269', '84722', '9050', '375743', '58155', '26024', '79810', '139411', '57540', '11191', '9317', '5738', '5754', '5757', '5763', '53635', '7803', '11156', '138639', '11099', '5790', '5791', '5793', '5794', '5797', '5798', '5801', '11122', '148713', '138428', '754', '10744', '26255', '390667', '5813', '5814', '29942', '150962', '83448', '126789', '79037', '5820', '11137', '114825', '170394', '137902', '54899', '149628', '79912', '84795', '23615', '5858', '54814', '54870', '84074', '55278', '79832', '169714', '203069', '23518', '22864', '140902', '80223', '84440', '26056', '9364', '282808', '84084', '10567', '79874', '9910', '5877', '11159', '11158', '285282', '83956', '729475', '25788', '23132', '8438', '55698', '646024', '5891', '26064', '10742', '9649', '55103', '22913', '64901', '8498', '26953', '5911', '9771', '51735', '51195', '65059', '5913', '5918', '401331', '51655', '23551', '158158', '153020', '255426', '54922', '10633', '91608', '387496', '65997', '51285', '644943', '11186', '9770', '283349', '83937', '83593', '166824', '8045', '11228', '9182', '125950', '55225', '57786', '10741', '54033', '10137', '389677', '64783', '29890', '22828', '92400', '9904', '282996', '55147', '221662', '64062', '54439', '155435', '23029', '5935', '55544', '55285', '375287', '375316', '129831', '5936', '54502', '83759', '10180', '5937', '5939', '27303', '494115', '83758', '116362', '11030', '348093', '149041', '54542', '1827', '10231', '55213', '1102', '5955', '57333', '283248', '55758', '92241', '343035', '201299', '9401', '9400', '5965', '51308', '221035', '80346', '7905', '92840', '5967', '83998', '768211', '285613', '84957', '28954', '161253', '387849', '29803', '11079', '473', '85004', '79785', '84666', '57455', '57109', '317671', '5988', '10740', '10739', '10737', '342931', '23180', '130132', '55159', '5989', '5990', '5991', '5992', '5993', '64864', '731220', '8625', '5994', '158234', '57529', '340526', '266747', '9104', '400966', '653489', '285190', '84220', '729540', '401190', '84236', '57414', '25807', '64285', '9028', '54933', '162494', '121268', '158800', '158787', '26150', '79608', '60626', '55188', '353116', '196383', '23504', '85376', '150221', '9699', '9783', '140730', '54101', '92129', '6018', '6013', '146206', '55005', '64795', '64777', '6023', '338879', '6038', '246243', '10535', '79621', '84153', '440400', '9921', '26994', '7732', '55298', '79845', '376412', '11342', '379013', '50862', '153830', '378925', '284996', '57484', '114804', '494470', '115992', '26001', '254225', '81790', '285533', '285671', '138065', '54546', '149603', '388591', '727800', '285498', '257160', '200312', '441191', '79596', '643904', '11237', '79102', '140545', '152006', '80352', '22838', '286140', '51136', '84900', '6050', '6051', '57140', '54538', '9991', '79641', '6094', '140823', '152015', '83853', '6096', '6098', '6101', '94137', '6100', '441212', '84268', '26015', '79657', '80135', '84154', '6103', '57096', '22895', '9501', '387841', '645683', '644511', '283345', '100129424', '402176', '644128', '649946', '118433', '284942', '222901', '118432', '641311', '132241', '285855', '113157', '85495', '56475', '376693', '728963', '441502', '256355', '26750', '83694', '204010', '388524', '653162', '113000', '27079', '285367', '84881', '6239', '730092', '653390', '100131998', '23223', '51018', '23076', '91695', '23212', '6247', '55316', '54665', '222194', '26156', '51187', '728194', '89765', '83861', '345895', '221421', '89970', '51319', '65117', '83546', '8634', '27156', '219790', '6252', '6253', '84816', '64108', '25914', '55680', '22902', '285180', '146923', '84127', '440352', '10900', '154661', '862', '23623', '9853', '112611', '10069', '201965', '6284', '57402', '140576', '6273', '6274', '6275', '6276', '6277', '64766', '170591', '6289', '6291', '113174', '29901', '26278', '9667', '6297', '140700', '148398', '401474', '148418', '201191', '90378', '154075', '23034', '55095', '389432', '54809', '219285', '64092', '9701', '56681', '9733', '23328', '54440', '163786', '112483', '340562', '51119', '155370', '100133234', '388228', '55206', '22904', '374897', '10609', '58506', '10066', '10067', '113178', '192683', '51282', '54581', '114821', '49855', '51435', '91179', '692148', '677777', '677781', '677769', '677766', '677775', '677772', '677767', '619383', '100158262', '8796', '152579', '6447', '4246', '92304', '29970', '85477', '6322', '10389', '256380', '79005', '59342', '11341', '9805', '90507', '79634', '642658', '57410', '55681', '57147', '55153', '27111', '9147', '10807', '6388', '23753', '51150', '644096', '54949', '255812', '727956', '728609', '8436', '56948', '93517', '6397', '23541', '284904', '9717', '653677', '10483', '25956', '7095', '11231', '9728', '6398', '23231', '8991', '58515', '140606', '83642', '51714', '7869', '10512', '223117', '6405', '56920', '10509', '54910', '57715', '10501', '10500', '26054', '57337', '9403', '22929', '57190', '151011', '55752', '1731', '124404', '55964', '5413', '23157', '285961', '641977', '23176', '10801', '6415', '84947', '26135', '8293', '10169', '26297', '253190', '94009', '387923', '256394', '5268', '8710', '5272', '5176', '5274', '29950', '9792', '29946', '56256', '91404', '26040', '23067', '54093', '55209', '6419', '23064', '124925', '26470', '23544', '51460', '57713', '6424', '6425', '140890', '285672', '10147', '57466', '11129', '8227', '25957', '9169', '6433', '113402', '375035', '84826', '389376', '94081', '118980', '81855', '119559', '94097', '6442', '6443', '6444', '8910', '6445', '124923', '129049', '9905', '27352', '54557', '10045', '10044', '63898', '387694', '400745', '284948', '646892', '6450', '83699', '83442', '6451', '6452', '9467', '80851', '201175', '51100', '56904', '285590', '153769', '344558', '54436', '79628', '26751', '85358', '6462', '399694', '79801', '56961', '126669', '90525', '387914', '152573', '149345', '388336', '729956', '729993', '6474', '134549', '357', '57619', '57477', '54414', '283514', '54847', '51092', '400709', '284367', '150094', '23235', '23387', '64374', '6490', '6493', '57568', '23094', '284759', '51547', '27240', '10572', '6496', '51804', '147912', '221150', '8631', '122060', '57606', '6504', '56833', '89886', '8273', '201780', '347051', '84068', '84561', '56996', '162515', '387700', '201232', '151473', '9122', '9121', '9120', '220963', '63910', '9389', '5003', '440044', '63027', '151295', '79085', '29957', '114789', '253512', '84275', '284723', '5250', '399512', '55186', '54977', '51629', '55972', '284427', '203427', '283130', '91137', '65012', '116369', '56731', '7782', '55676', '148867', '10463', '1318', '113829', '55032', '10237', '51006', '340146', '79939', '728661', '9906', '55508', '339665', '222553', '54733', '148641', '80255', '84255', '124565', '151258', '145389', '55238', '146167', '201266', '91252', '55334', '54946', '29015', '50651', '57210', '57864', '283537', '55652', '83959', '22950', '6526', '386757', '28968', '388662', '6545', '81893', '387254', '84138', '9368', '9351', '81796', '643036', '91607', '55106', '342615', '146857', '162394', '200172', '7871', '51012', '79811', '9597', '57228', '60682', '64744', '6594', '56916', '50485', '23347', '140771', '125170', '54471', '140775', '55671', '57223', '64093', '64094', '10924', '27293', '23676', '6525', '219537', '342527', '114826', '10322', '333929', '6623', '25992', '283596', '128439', '23642', '85028', '8420', '751867', '724102', '387066', '641638', '84973', '100093630', '735301', '8303', '677805', '677821', '677832', '692158', '6044', '6043', '574040', '26821', '677797', '677847', '654320', '652966', '100033431', '100033820', '100033416', '114599', '692086', '677850', '692223', '9751', '54861', '6640', '6641', '6645', '54221', '132203', '29887', '29916', '29934', '23161', '57231', '29907', '64089', '6642', '9784', '399979', '124460', '90203', '79856', '28966', '83891', '81609', '92017', '401548', '169166', '254122', '257364', '8723', '58533', '51375', '29886', '55084', '30837', '54937', '6651', '8470', '114815', '57537', '25928', '6664', '6666', '8403', '6665', '54345', '347689', '11063', '6660', '30812', '3431', '11262', '93349', '6668', '6671', '389058', '80320', '221833', '53340', '124912', '6674', '79582', '200162', '6676', '10615', '9552', '26206', '353324', '64173', '128153', '132671', '64847', '202051', '124044', '166378', '79029', '54558', '55812', '83890', '375686', '221409', '65244', '26010', '25803', '245711', '387778', '285955', '441273', '441272', '442590', '729597', '441251', '389517', '25876', '79925', '10290', '220082', '246777', '80208', '23111', '51324', '10638', '6689', '10927', '54466', '474343', '169981', '139886', '6690', '6691', '27290', '56907', '84501', '83985', '201305', '90853', '6695', '9806', '121665', '10251', '81848', '84926', '283377', '144108', '10011', '55133', '10847', '80725', '136853', '153443', '6345', '26576', '27286', '8406', '402055', '222183', '100170229', '6760', '26039', '51188', '23635', '23648', '170463', '6744', '54434', '85464', '54961', '8082', '23145', '10534', '117178', '6759', '9705', '400410', '7982', '54879', '93653', '93655', '338069', '342667', '246329', '6769', '54441', '442582', '442578', '64940', '26228', '6780', '27067', '6781', '26872', '79689', '6491', '57620', '8576', '9263', '9262', '8859', '10494', '56164', '202374', '55351', '282974', '65975', '140901', '27148', '23012', '27347', '83931', '7955', '50861', '9399', '219736', '56977', '55342', '29966', '29888', '201595', '8676', '23673', '415117', '2054', '8417', '9482', '252983', '134957', '9515', '29091', '55359', '6815', '51657', '10923', '283507', '441394', '23213', '55959', '474338', '387082', '6835', '6838', '64420', '56241', '203328', '55061', '26032', '440423', '79987', '6840', '55530', '136306', '94056', '55638', '100130958', '221711', '81493', '10492', '9144', '9143', '23546', '55333', '23336', '11346', '171024', '79933', '11276', '6856', '284612', '767557', '57586', '255928', '83849', '51760', '84258', '148281', '54843', '94120', '94122', '255061', '6867', '10579', '4070', '29114', '6876', '6887', '85461', '26115', '9344', '55080', '202018', '6894', '23435', '445347', '123283', '83940', '128387', '23232', '55296', '25771', '55633', '55773', '23102', '414059', '654341', '729877', '9779', '11138', '23158', '23061', '55171', '219899', '93627', '9755', '26608', '9519', '84897', '347853', '6913', '9096', '9095', '30009', '6909', '6926', '9496', '123036', '146771', '6919', '6920', '9338', '140597', '85012', '79921', '340543', '158931', '56849', '90843', '170082', '6939', '6941', '6942', '6943', '22980', '10732', '84260', '8115', '27004', '6949', '6953', '140290', '6954', '55346', '255394', '6988', '202500', '343521', '6998', '126668', '81550', '163589', '23424', '25851', '9895', '7008', '27285', '64518', '150483', '146279', '9894', '23371', '7011', '374739', '7012', '26136', '54997', '10420', '56159', '56155', '400629', '113419', '374618', '7023', '7024', '29842', '7030', '7942', '22797', '7032', '7980', '7038', '23483', '7041', '9392', '7052', '7053', '7047', '56906', '57215', '55145', '83591', '90326', '51078', '168451', '152815', '80764', '199745', '79725', '387357', '79896', '55258', '374500', '80745', '25917', '29087', '7075', '497189', '200765', '166815', '220359', '201798', '84948', '81789', '91151', '201633', '91937', '7079', '27283', '64129', '25976', '261726', '93643', '27134', '8277', '116238', '79816', '9874', '11011', '83660', '3195', '3196', '83941', '83877', '80213', '4071', '116441', '116211', '79853', '7104', '9032', '53346', '53345', '51768', '10548', '9375', '56889', '9777', '51643', '7009', '342125', '147798', '79838', '79905', '147138', '23023', '9911', '57458', '54499', '55002', '255104', '55374', '79613', '286102', '11018', '222068', '146456', '283578', '8577', '55273', '84336', '284114', '54868', '284186', '113277', '54664', '79022', '84314', '66000', '79073', '375346', '89894', '84216', '338773', '83862', '144404', '80757', '114908', '128218', '84233', '55654', '85013', '222865', '23505', '114795', '92293', '124842', '80194', '65084', '219902', '51524', '135932', '55281', '85014', '55260', '55314', '284339', '10430', '79713', '28978', '81853', '51522', '129303', '284417', '441027', '256472', '441151', '201799', '132332', '80008', '25907', '57146', '54958', '54929', '153396', '81615', '84187', '55858', '153339', '56900', '64418', '92691', '124491', '100113407', '134285', '200728', '84286', '55365', '28959', '80775', '130733', '79847', '129787', '57583', '130827', '92703', '202915', '25829', '55751', '84548', '79134', '8269', '387521', '147744', '84222', '201931', '23306', '100131211', '55266', '130612', '147007', '114801', '399474', '645369', '199953', '159371', '94107', '79652', '374882', '55248', '29100', '84928', '255349', '155006', '54867', '401498', '221468', '219854', '388335', '84065', '23670', '79064', '80723', '161145', '79583', '642987', '387890', '84866', '219623', '161291', '203562', '55161', '59353', '140738', '79041', '55151', '55254', '55116', '55287', '90407', '440026', '131616', '79188', '93109', '55076', '120224', '83604', '81671', '23585', '757', '10329', '55092', '339456', '79639', '113452', '55529', '148534', '25789', '85025', '199964', '80021', '55362', '57156', '169200', '157378', '51669', '137695', '51249', '54968', '137835', '643236', '157753', '84283', '283232', '388730', '388595', '283673', '51234', '144110', '84910', '92162', '58986', '646658', '79953', '641649', '162461', '252839', '83460', '27346', '26022', '147184', '56674', '126259', '643853', '84000', '64699', '56649', '80975', '344805', '360200', '9168', '11013', '286527', '7117', '83857', '160335', '160418', '84899', '83590', '79089', '81542', '51075', '56255', '7127', '55504', '8711', '10188', '64102', '51086', '30000', '23534', '84629', '7145', '10140', '10766', '114034', '10040', '146691', '387990', '10953', '84134', '116447', '7151', '7152', '8940', '163590', '27433', '64222', '84969', '27324', '9878', '9760', '9537', '90313', '58476', '11257', '729355', '7162', '7165', '286016', '728402', '11076', '51673', '131601', '285386', '127262', '348825', '64499', '23430', '25823', '7178', '646405', '29896', '80305', '10758', '80342', '9618', '7188', '10906', '133022', '9697', '9881', '10597', '55809', '11219', '29953', '28951', '9865', '89870', '10626', '147166', '373', '135644', '54765', '91107', '84851', '57159', '84675', '25893', '286827', '391712', '201292', '9866', '440730', '378108', '81786', '117852', '9319', '9325', '7205', '27037', '79979', '54952', '388610', '7216', '10024', '6738', '7221', '7227', '83707', '142940', '26995', '154754', '8848', '9819', '81628', '80705', '254187', '10194', '128553', '57616', '9383', '60385', '25987', '203062', '55815', '83882', '10103', '441631', '23554', '27075', '26262', '90139', '10100', '10099', '6302', '10077', '7106', '7105', '7103', '10867', '222642', '7259', '128854', '23270', '85453', '90121', '7260', '10078', '83942', '81629', '283629', '83983', '158427', '84630', '54970', '79573', '151613', '158248', '7265', '55761', '118491', '54902', '199223', '55001', '64927', '153657', '164118', '83538', '55622', '23331', '64427', '130502', '23548', '9694', '143941', '7267', '55020', '22996', '158219', '125488', '7268', '57217', '145567', '23508', '148014', '283237', '8458', '7272', '440307', '79183', '7275', '80086', '56604', '643224', '51174', '51175', '7286', '55000', '7287', '7288', '286319', '11334', '64852', '57045', '167838', '55787', '51061', '51060', '79770', '57544', '84817', '51314', '347736', '10190', '54957', '114112', '7301', '441250', '91373', '337867', '55833', '9898', '53347', '84959', '606551', '389898', '388165', '134111', '10277', '56061', '5412', '84993', '134510', '254048', '7342', '56893', '143630', '130507', '51366', '55148', '80019', '92181', '127733', '165324', '137886', '23190', '80700', '7993', '652995', '100113386', '55293', '51506', '51569', '402682', '55325', '54887', '9706', '54986', '89766', '9094', '23025', '440279', '55898', '146862', '25972', '222643', '54346', '85451', '400224', '11045', '7348', '7379', '7380', '105375355', '100134938', '139596', '55245', '440567', '9875', '9816', '55665', '7399', '83878', '84833', '8237', '389856', '57478', '84669', '57558', '57602', '84640', '55230', '64854', '158880', '54532', '159195', '9098', '10208', '84294', '80146', '10791', '114990', '57687', '25806', '26609', '51481', '79674', '7716', '55591', '51442', '389136', '9686', '7429', '50853', '63894', '5212', '203547', '400673', '284013', '57191', '55350', '81552', '23230', '157680', '54832', '55187', '112936', '23355', '54621', '147645', '340547', '23584', '11326', '391123', '7447', '284415', '128434', '79679', '340706', '146177', '200403', '4013', '127731', '90113', '375567', '220001', '221806', '375260', '374666', '375690', '653635', '441818', '23559', '23558', '164684', '51186', '81554', '171022', '155368', '135886', '57590', '115825', '23001', '57705', '11169', '64743', '146845', '116966', '79446', '80232', '253769', '114987', '22884', '401551', '55255', '54521', '22911', '151790', '55779', '348793', '84058', '54853', '54554', '284403', '126820', '149465', '144406', '93594', '256764', '23335', '84942', '54663', '79968', '79819', '49856', '349136', '83889', '126248', '112840', '197335', '116143', '56964', '151525', '55093', '494551', '280664', '58189', '140686', '117166', '124857', '339005', '440253', '84305', '8840', '8839', '8838', '51352', '58525', '56897', '23302', '9671', '80014', '55841', '7499', '165904', '7503', '114786', '389610', '286046', '55113', '389668', '402415', '7511', '63929', '64328', '23214', '23039', '9213', '91419', '143570', '51087', '150223', '90522', '54432', '78992', '25844', '84272', '81555', '374887', '56252', '29799', '388403', '83719', '219539', '79693', '80122', '91746', '64848', '54915', '51441', '253943', '55249', '404281', '353174', '284273', '51776', '79740', '79413', '84327', '9889', '58486', '65986', '27107', '221527', '22890', '26137', '9278', '9841', '7597', '57621', '57684', '27033', '10009', '403341', '84614', '79842', '253461', '9880', '9923', '360023', '100128927', '23099', '29068', '57659', '84878', '140685', '92999', '3104', '166793', '9925', '10773', '51341', '51043', '201501', '653121', '728116', '221504', '84872', '340554', '85463', '340152', '23091', '79882', '124245', '23144', '23211', '376940', '29066', '23264', '92092', '55906', '54819', '170261', '23174', '51538', '644353', '219654', '54877', '85364', '29063', '84186', '55596', '84240', '55063', '152098', '57683', '84885', '29800', '54503', '79683', '158866', '84287', '23390', '84243', '131540', '254887', '254359', '55146', '64429', '150244', '10444', '79752', '90637', '130617', '60685', '7763', '57623', '196441', '79776', '64397', '7541', '124961', '286128', '346171', '643836', '55734', '284406', '80829', '139735', '7542', '23217', '51663', '7543', '53349', '84936', '79038', '23503', '118813', '57732', '653808', '124220', '100125288', '84619', '11244', '22882', '23051', '85416', '342357', '84460', '64393', '79698', '57178', '83637', '10269', '79830', '9203', '9202', '9205', '9204', '51364', '10771', '84217', '84225', '118490', '116225', '23613', '51427', '51351', '7675', '7691', '7693', '7696', '7701', '7705', '7707', '7718', '7564', '7727', '7730', '339318', '7569', '7739', '7741', '7746', '7754', '7756', '51222', '7570', '7772', '7775', '7776', '10472', '8187', '7572', '90987', '286101', '8882', '339324', '10778', '10838', '92822', '11179', '129025', '140883', '55609', '54816', '23528', '284349', '342909', '729288', '23036', '49854', '162979', '24149', '57567', '399669', '79692', '387328', '64288', '284695', '7580', '79673', '27309', '84905', '25850', '23567', '7584', '140467', '149076', '22891', '167465', '195828', '100129482', '171017', '151126', '79750', '651302', '346157', '55893', '252884', '84307', '100101467', '342908', '55628', '79797', '57862', '84330', '23090', '126299', '220929', '55311', '51710', '203523', '26036', '58499', '84627', '133923', '197407', '642819', '118738', '126069', '162968', '115560', '91392', '84858', '22847', '118472', '84450', '130557', '9658', '9849', '85460', '147807', '170958', '116115', '84503', '55205', '147658', '9745', '84215', '147947', '126295', '137209', '64763', '284346', '79177', '147660', '163033', '51157', '51545', '9640', '51042', '84622', '90850', '57507', '23060', '114991', '90441', '9831', '89887', '23361', '57232', '51193', '84146', '9726', '127665', '22834', '115950', '55279', '79894', '641339', '339500', '257101', '57592', '57116', '7620', '79986', '148203', '284390', '91661', '90321', '7629', '79970', '79724', '7553', '284323', '163131', '163115', '147808', '126208', '388507', '285989', '168850', '91752', '390980', '7634', '388558', '126017', '347344', '730051', '401303', '125893', '55565', '55552', '664701', '152485', '283489', '374899', '128611', '401898', '55769', '90485', '162962', '116412', '284371', '283933', '284391', '91664', '162993', '7637', '54753', '7639', '643641', '7554', '642280', '729747', '345462', '400713', '169834', '7643', '7644', '81931', '148198', '57169', '10467', '741', '9326', '54680', '9406', '84083', '284312', '9753', '221584', '80345', '65982', '7579', '7589', '342945', '222696', '54993', '146050', '79149', '342933', '90204', '140831', '65249', '57643', '57688', '125150', '221302', '7789', '158586', '79364', '440590', '79699', '7791', '23140']
Features in feature_order not found in all_feature_names: ['100033413', '10009', '100130426', '100133144', '10016', '10017', '10024', '100287569', '10036', '10040', '10044', '10045', '10047', '10066', '10067', '10069', '10085', '10099', '10100', '10105', '10136', '1014', '10140', '10143', '10169', '10180', '10186', '10188', '10194', '10198', '10201', '10205', '10207', '10208', '10214', '10215', '10217', '10220', '10223', '10231', '10233', '10237', '10241', '10255', '10263', '10265', '10269', '10278', '10279', '10289', '10290', '10296', '10297', '10300', '10309', '10322', '10329', '1033', '10336', '10343', '10344', '10360', '10363', '10365', '1038', '10389', '1039', '10391', '10404', '10420', '10421', '10428', '10431', '10439', '1044', '10443', '10444', '10446', '10456', '1046', '10463', '1047', '10472', '10473', '10481', '10483', '10485', '10492', '10494', '10495', '105', '10500', '10512', '10513', '10527', '10529', '10534', '10535', '105375355', '10548', '10551', '10553', '1056', '10565', '10566', '10572', '1059', '10615', '10620', '10625', '10626', '10634', '10647', '10648', '10655', '10661', '10668', '10669', '10677', '10691', '1070', '10712', '10716', '10724', '10725', '1073', '10732', '10738', '10741', '10742', '10745', '10748', '10758', '10761', '10763', '10766', '10771', '10773', '10801', '10814', '10826', '10847', '10865', '10866', '10867', '10880', '10881', '10882', '1089', '10899', '10900', '10903', '10904', '10906', '10912', '10924', '10927', '10934', '10944', '10947', '10949', '10950', '10953', '10957', '10964', '10982', '10983', '10994', '11007', '11016', '11018', '1102', '11030', '11040', '11043', '11045', '11054', '11055', '11057', '1106', '11061', '11063', '11072', '11077', '11082', '11086', '11094', '11099', '11100', '11107', '1112', '11122', '11137', '11138', '11142', '11153', '11155', '11166', '1117', '11170', '11172', '11177', '11178', '11186', '11191', '11212', '11215', '11228', '11230', '11231', '11244', '112464', '11247', '112476', '11248', '112487', '112609', '11262', '11266', '11269', '11270', '11272', '11273', '11276', '11278', '11279', '11281', '112840', '112885', '112939', '112942', '113115', '11318', '11326', '113277', '11330', '11334', '11341', '113419', '11342', '113444', '113622', '113655', '113746', '113791', '113828', '1140', '114038', '114294', '114327', '114571', '114801', '114815', '114824', '114897', '114898', '114899', '114904', '114905', '114928', '114932', '114960', '114984', '115123', '115207', '115294', '115399', '115416', '115560', '115572', '115653', '115701', '115704', '115752', '115761', '115795', '115825', '115861', '115908', '115950', '116071', '116113', '116115', '116154', '116173', '116211', '116225', '116238', '116254', '116337', '116369', '1164', '116441', '116447', '116496', '116511', '116535', '116843', '116966', '116969', '116983', '116987', '117154', '117166', '117178', '117196', '117286', '1178', '118442', '118471', '118490', '118491', '118738', '118812', '118856', '118881', '118924', '118987', '119032', '119180', '119467', '1195', '119559', '119587', '1196', '1198', '120224', '120534', '1209', '120939', '121227', '121268', '121273', '121355', '121506', '121551', '121599', '121643', '121665', '121793', '122011', '122046', '122060', '122258', '122773', '122786', '122953', '123036', '123283', '123591', '123775', '123811', '123920', '124044', '124220', '124245', '124402', '124460', '124491', '124565', '124751', '124801', '124857', '124912', '124930', '124961', '124989', '125170', '126069', '126074', '126133', '126206', '126248', '126259', '126272', '126299', '126306', '126308', '126353', '126364', '1264', '126402', '126549', '1266', '126626', '1267', '126731', '126820', '127018', '127262', '127343', '127435', '127700', '127703', '127707', '127731', '127733', '127845', '127943', '128218', '128344', '128434', '128488', '128497', '128646', '128821', '128861', '128876', '128989', '129025', '129138', '129285', '129303', '130026', '130271', '130540', '130557', '130560', '130574', '130589', '130617', '130749', '130813', '130814', '130827', '130872', '131034', '131177', '131377', '131408', '131540', '131566', '131578', '1316', '131616', '1318', '131831', '132001', '132160', '132228', '132299', '132625', '132671', '132720', '132724', '132851', '132864', '132949', '132989', '133015', '133022', '133584', '133619', '133690', '1339', '134218', '134285', '134510', '134548', '134553', '134957', '135114', '135154', '135398', '135644', '135656', '1357', '1358', '135927', '135932', '136259', '136306', '136332', '136541', '136542', '136895', '137209', '137735', '137835', '137994', '138009', '138151', '138311', '138429', '138639', '138716', '1390', '139081', '139105', '139170', '139562', '139596', '139599', '1396', '140290', '140576', '140597', '140625', '140688', '140689', '1407', '140700', '140701', '140706', '140707', '140710', '140733', '140767', '1408', '140836', '140856', '140870', '140880', '140883', '140886', '140893', '1409', '140901', '140902', '140947', '141', '1411', '1412', '1414', '1415', '1417', '1418', '1419', '1420', '1421', '142679', '142683', '142940', '143162', '143379', '1434', '144110', '144132', '144321', '144347', '1444', '144423', '144568', '144717', '144809', '145165', '145264', '145270', '145282', '145376', '145447', '145497', '145501', '145553', '1456', '145645', '145748', '145781', '145858', '145864', '145942', '145946', '146223', '146225', '146279', '146395', '146456', '1466', '146664', '146779', '146862', '1469', '1470', '147011', '147015', '1472', '1473', '147381', '1474', '147463', '147495', '147685', '147746', '147798', '147808', '147906', '147912', '147947', '147991', '148109', '148113', '148137', '148252', '148281', '148304', '148345', '148398', '148423', '148479', '148523', '148534', '148741', '148753', '148808', '148823', '148870', '148930', '148932', '148979', '149076', '149175', '149345', '149420', '149428', '149466', '149473', '149483', '149563', '149685', '149699', '149708', '149830', '150000', '150094', '1501', '150142', '150160', '150209', '150275', '150290', '150291', '150350', '150468', '150483', '1506', '150677', '150678', '150696', '151195', '151230', '151254', '151258', '151354', '151449', '151473', '151516', '151525', '151556', '151613', '151647', '151835', '151887', '151963', '152006', '152007', '152065', '152110', '152138', '152189', '152789', '152816', '153396', '1534', '153769', '154043', '154197', '154215', '154791', '155051', '155060', '155368', '157378', '157567', '157574', '157638', '157680', '157697', '157753', '157848', '157922', '158046', '158056', '158062', '158158', '158160', '158219', '158297', '158471', '158521', '158798', '158800', '158801', '158809', '158866', '158880', '159013', '159091', '159195', '159371', '160065', '160140', '160335', '160492', '160622', '160760', '160762', '160777', '160897', '1611', '161145', '161176', '161198', '161582', '161753', '1618', '161835', '161931', '162333', '162461', '162515', '162681', '162968', '162979', '162989', '163033', '163115', '163126', '163154', '163183', '163479', '163589', '163590', '163778', '163782', '163933', '164091', '164127', '164153', '164284', '164592', '164633', '164684', '164781', '165', '165055', '165100', '165215', '165324', '165530', '165545', '1657', '165721', '165829', '1662', '166336', '166647', '166655', '166815', '166824', '166863', '166968', '167', '167153', '167359', '167410', '167555', '167681', '167691', '167826', '168090', '168537', '168620', '168667', '168850', '1690', '169166', '169611', '169792', '170062', '170261', '170302', '170392', '170393', '170394', '170463', '170626', '170627', '170680', '170825', '171017', '171024', '171389', '171483', '171484', '1745', '1747', '1748', '1762', '1767', '1768', '1769', '1770', '1773', '1775', '1791', '1804', '1827', '1837', '1838', '1877', '1951', '1952', '1954', '195828', '196264', '196385', '196410', '196463', '196475', '196477', '196740', '196792', '197021', '197407', '1979', '199', '199221', '1995', '1996', '199713', '199731', '199786', '199870', '199964', '2000', '200008', '200058', '2001', '200150', '200172', '200350', '2004', '2005', '200523', '200539', '200634', '200728', '2009', '200942', '2011', '201191', '2012', '201232', '201266', '201299', '2013', '201305', '2014', '201595', '2016', '201633', '201780', '2019', '202', '202052', '2021', '202151', '2022', '202459', '202500', '202865', '202915', '203100', '203228', '203238', '203245', '203260', '203427', '203430', '2038', '205147', '2054', '205428', '2078', '2115', '2117', '2119', '2128', '2145', '2186', '2191', '219293', '219409', '219654', '219681', '219738', '219771', '219790', '219793', '219844', '219902', '219995', '220001', '220004', '220047', '220064', '220082', '220108', '220202', '220296', '220323', '220388', '220929', '220965', '220972', '221016', '221061', '221079', '221091', '221184', '221223', '221241', '221294', '221302', '221409', '221421', '221477', '221491', '221504', '221527', '221545', '221662', '2217', '2218', '221833', '221895', '221908', '221927', '222008', '222068', '222166', '222171', '222183', '222389', '222484', '222611', '222894', '222950', '225689', '2272', '2275', '22834', '22838', '22844', '22847', '22853', '22858', '22859', '2286', '22877', '22883', '22884', '22887', '22889', '22891', '22895', '22902', '22903', '22904', '22907', '22911', '22913', '22924', '22929', '2295', '2297', '22980', '22982', '22983', '22986', '2299', '22990', '22998', '23008', '2301', '2302', '23023', '23024', '23026', '23029', '23034', '23039', '2304', '23051', '23061', '23064', '2307', '23070', '23077', '23085', '23086', '23089', '23091', '23099', '23101', '23111', '23125', '23126', '2314', '23144', '2315', '23152', '23157', '23158', '23177', '23180', '23185', '23190', '23195', '23204', '23209', '23212', '23213', '23215', '23219', '23223', '23231', '23235', '23240', '23245', '23255', '23264', '23270', '23271', '23277', '23300', '23301', '23312', '23313', '23316', '23328', '23335', '23336', '2334', '23357', '23362', '23366', '23371', '23386', '23394', '23424', '23430', '23434', '23435', '23440', '23483', '23484', '23492', '23499', '23500', '23503', '23504', '23506', '23515', '23519', '23527', '23528', '23531', '23541', '23542', '23548', '23549', '2355', '23551', '23552', '23554', '23558', '23559', '23564', '23567', '23582', '23584', '23588', '23591', '23598', '23603', '23609', '23612', '23613', '23615', '23624', '23635', '23639', '23641', '23648', '23670', '23671', '23676', '23742', '23746', '23753', '23786', '23787', '23788', '24139', '24141', '24149', '245711', '245806', '246176', '246243', '246269', '246329', '246705', '246721', '2483', '2487', '252839', '252884', '252995', '253012', '253143', '253152', '253190', '253738', '253982', '254042', '254065', '254170', '254240', '254251', '254268', '254295', '254394', '254427', '254528', '254778', '254863', '255057', '255101', '255104', '255119', '255352', '255394', '255520', '2563', '256309', '256356', '256369', '256380', '2564', '256472', '256536', '256586', '256643', '256646', '256691', '256710', '256714', '256764', '2568', '256949', '257', '257000', '257044', '257068', '257101', '257144', '257240', '257364', '25758', '25766', '25770', '25771', '25778', '25788', '25789', '25790', '25803', '25806', '25812', '25814', '25825', '25829', '25833', '25841', '25849', '25850', '25853', '25876', '25878', '25884', '25891', '25893', '25895', '25900', '25901', '25903', '25907', '25917', '259234', '259240', '259249', '259266', '25927', '25928', '25932', '25940', '25956', '25957', '25960', '25975', '25980', '25987', '25988', '26002', '26015', '26022', '260293', '26033', '26038', '26040', '260425', '260436', '26047', '26051', '26056', '26058', '26059', '26064', '26065', '26093', '26098', '26103', '26112', '26128', '26135', '26136', '26137', '26145', '26156', '26164', '26167', '261726', '26206', '26225', '26228', '26240', '26256', '26257', '26278', '26287', '26292', '26297', '26298', '26301', '26470', '26499', '26504', '26505', '26507', '26509', '2651', '26511', '26526', '26528', '26576', '26586', '26608', '2661', '26750', '2679', '26953', '26973', '26994', '26996', '26998', '27023', '27033', '27065', '27067', '27074', '27075', '27091', '27098', '27101', '27106', '27122', '27136', '27156', '27164', '27190', '27201', '27239', '27240', '27253', '27286', '27288', '27293', '27295', '27299', '27301', '27303', '27309', '27315', '27328', '27346', '27347', '27352', '2738', '27439', '2764', '280658', '2823', '2824', '2827', '282973', '282991', '2830', '283078', '283120', '28316', '283197', '283212', '283234', '283248', '283298', '283316', '283358', '283377', '283383', '283416', '283420', '283431', '283446', '283450', '283461', '283487', '283489', '2835', '283537', '283600', '283629', '283635', '283643', '283820', '283847', '283849', '283870', '283897', '283991', '284001', '284013', '284029', '284114', '284186', '2842', '284273', '284312', '284339', '284340', '284346', '284358', '284382', '284406', '284415', '284451', '284467', '284565', '284613', '284697', '284723', '284739', '2849', '284904', '284996', '285025', '285126', '285154', '285203', '285237', '285313', '285343', '285367', '285386', '285489', '285598', '285641', '285759', '285855', '286046', '286077', '286257', '286336', '286343', '286514', '286676', '286827', '287015', '2895', '28951', '28954', '28955', '28966', '28974', '28978', '28985', '28986', '28992', '29015', '29035', '29058', '29085', '29091', '29094', '29097', '29103', '29109', '29113', '29114', '29123', '2969', '2974', '29775', '29777', '2978', '29799', '29842', '29889', '29890', '29896', '29901', '29903', '29907', '29916', '29919', '29930', '29934', '29937', '29944', '29946', '29950', '29951', '29957', '29980', '29997', '29998', '30009', '3001', '30012', '3003', '30062', '3010', '3026', '3049', '3050', '3059', '307', '3070', '30811', '30812', '30813', '30836', '30844', '30850', '3087', '3094', '3096', '3097', '310', '3104', '3108', '311', '3110', '312', '3131', '3140', '3142', '3149', '3151', '317671', '317712', '317749', '317754', '3182', '3202', '3204', '3205', '3208', '3209', '3217', '3218', '3219', '3225', '3226', '3228', '3229', '323', '3234', '3235', '3236', '3238', '3239', '3241', '326', '326340', '3266', '3268', '3275', '3298', '3299', '3300', '3316', '333', '333926', '3344', '337867', '3382', '338321', '338322', '338657', '338699', '338707', '338773', '338811', '338872', '338917', '339416', '339479', '339500', '339501', '339665', '339669', '339779', '339976', '340277', '340596', '340719', '341116', '341405', '341567', '342035', '342667', '342897', '342945', '343035', '3431', '343450', '343472', '343641', '344561', '344838', '345757', '346007', '346389', '346673', '3475', '3476', '347736', '347741', '347853', '348094', '348738', '348807', '349136', '350383', '352909', '353322', '353500', '359948', '360132', '360200', '3621', '3641', '3692', '3697', '373863', '374393', '374407', '374618', '374739', '374864', '374872', '374897', '374955', '374977', '375035', '375061', '375287', '375307', '375341', '375346', '375387', '375444', '375484', '375567', '375607', '379', '386617', '387119', '387328', '387496', '387509', '387597', '387694', '387700', '387758', '387836', '387837', '387882', '387990', '388021', '388272', '388394', '388407', '388533', '388591', '388611', '388685', '388730', '388759', '389084', '389119', '389197', '389208', '389320', '389376', '389383', '389558', '389649', '389799', '3898', '389874', '3899', '390212', '391343', '3927', '3950', '3957', '3960', '3963', '3982', '3993', '3995', '399512', '3996', '399668', '399671', '399694', '399823', '399979', '400120', '400359', '400696', '4007', '400823', '4009', '401', '4010', '401253', '401262', '4013', '401474', '401498', '4026', '4037', '404037', '404093', '4043', '4058', '4059', '4065', '407', '4070', '4071', '4076', '408029', '4084', '4107', '4109', '4113', '4115', '4118', '4133', '4134', '4142', '4147', '4150', '4163', '4184', '4188', '421', '4212', '4223', '4232', '4250', '4256', '4276', '4288', '4291', '4293', '4294', '4299', '430', '4302', '4330', '434', '4340', '4342', '4355', '439921', '4430', '4542', '460', '4601', '4608', '4610', '4622', '4640', '4643', '4661', '4666', '4668', '4673', '4676', '4681', '4693', '473', '4739', '474', '4741', '4744', '4745', '4752', '4761', '4762', '4776', '4779', '4783', '4796', '4807', '4814', '4815', '4818', '4820', '4824', '4858', '4861', '4879', '4883', '4885', '4892', '4925', '4931', '4950', '4951', '4956', '49854', '49856', '49860', '5013', '5017', '5042', '5047', '50485', '50613', '50632', '5066', '5074', '5078', '50804', '50805', '50853', '50861', '50862', '5089', '5090', '50945', '5097', '5100', '51006', '5101', '51016', '51018', '51022', '51027', '51029', '51032', '51043', '51050', '51056', '51059', '51063', '51066', '51068', '51087', '51100', '51101', '51108', '51115', '51124', '51126', '51136', '51148', '51157', '51177', '51187', '51192', '51195', '51207', '5121', '51225', '51237', '51239', '51244', '51247', '51257', '51260', '51265', '51267', '5127', '51270', '51277', '51278', '5128', '51282', '51283', '51285', '51286', '5129', '51294', '51303', '51307', '51308', '51309', '51313', '51314', '51315', '51319', '51321', '51324', '51327', '51334', '51335', '51337', '5134', '51352', '51361', '51364', '51371', '51375', '51379', '51390', '51406', '51409', '51427', '51435', '51438', '51439', '51440', '51450', '51454', '5146', '51460', '51463', '51474', '5149', '51490', '51496', '51506', '51513', '51522', '51526', '51531', '51547', '51555', '51559', '51569', '5157', '51596', '51614', '51629', '51633', '51635', '51655', '51669', '51673', '51678', '51696', '51704', '51751', '5176', '51760', '51768', '51776', '51804', '5184', '5187', '5188', '5218', '5225', '5229', '5250', '5251', '5268', '5272', '5274', '5303', '5307', '5309', '5314', '5334', '53342', '53346', '53347', '53353', '5354', '5355', '5358', '5359', '53635', '5375', '53820', '53838', '5387', '53940', '53944', '5396', '54039', '54072', '54084', '54097', '54101', '54102', '54108', '54112', '5412', '5413', '54149', '5420', '54212', '54328', '54345', '54360', '54432', '54434', '54436', '54440', '54442', '54463', '54465', '54470', '54475', '54491', '54499', '5450', '54502', '54504', '54521', '54529', '54538', '54544', '54550', '54551', '54552', '54554', '54558', '5456', '54566', '5459', '54596', '54606', '54619', '5463', '54664', '54665', '54680', '54681', '54682', '5470', '54707', '54714', '54718', '54726', '54733', '54737', '54738', '54749', '54752', '54758', '54768', '54769', '54776', '54785', '54796', '54798', '54799', '5480', '54800', '54805', '54808', '54810', '54812', '54819', '54828', '54832', '54838', '54840', '54849', '54852', '54855', '54856', '54857', '54858', '54861', '54866', '54870', '54878', '54880', '54897', '54899', '54900', '54910', '54915', '54922', '54929', '54935', '54937', '54940', '54943', '54954', '54955', '54958', '5496', '54961', '54964', '54968', '54976', '54977', '54978', '54980', '54982', '54984', '54987', '54991', '54993', '54997', '55000', '55001', '55007', '55009', '55015', '5502', '55020', '55022', '55024', '55026', '55027', '55038', '5504', '55040', '55041', '55051', '5506', '55064', '55068', '55071', '55074', '55076', '55079', '55086', '55092', '55093', '55103', '55108', '55113', '55119', '55130', '55132', '55133', '55137', '55146', '55149', '55151', '55154', '55156', '55161', '55165', '55171', '55172', '55177', '55179', '55186', '55188', '55195', '55198', '55204', '55205', '55206', '55207', '5521', '55210', '55213', '55218', '55222', '55228', '55237', '55238', '55246', '55248', '55255', '55258', '55262', '55273', '55274', '55280', '55281', '55285', '55286', '55293', '55295', '55296', '55299', '55303', '55308', '55311', '553137', '55314', '55316', '55317', '55323', '55324', '55325', '55329', '55335', '55340', '55342', '55345', '55346', '55351', '55353', '55359', '55374', '55379', '55384', '55435', '5545', '55450', '5550', '55502', '55504', '55506', '55507', '55509', '55510', '55520', '55529', '55536', '55539', '55544', '55554', '55556', '55589', '55601', '55602', '55607', '55608', '55609', '55610', '55612', '55616', '55632', '55635', '55640', '55643', '55652', '55653', '55661', '55668', '55671', '55676', '55686', '5569', '55691', '55692', '55695', '55705', '55716', '55719', '55721', '55723', '55734', '55743', '55751', '55760', '55769', '55773', '55776', '55779', '55783', '55793', '55796', '55798', '55803', '55805', '55806', '55809', '55810', '55833', '55837', '55841', '55847', '55848', '55857', '55858', '55859', '55885', '55890', '55891', '55892', '55893', '55898', '55906', '55917', '55922', '55959', '55966', '55969', '55973', '55975', '56', '56000', '56062', '5612', '56121', '56122', '56123', '56124', '56126', '56130', '56132', '56133', '56156', '56163', '56164', '56169', '56180', '56255', '56257', '5626', '56269', '5629', '5630', '56311', '56341', '5635', '5636', '5639', '56475', '56478', '5653', '5655', '56650', '56673', '56674', '56683', '56704', '56731', '5681', '56833', '56884', '56889', '56890', '56891', '56893', '56897', '56901', '56904', '56905', '56914', '56916', '56917', '56920', '56936', '56940', '56941', '56944', '56946', '56947', '56952', '56961', '56967', '56971', '56981', '56987', '56996', '56997', '57001', '57002', '57003', '57010', '57026', '57035', '57045', '57047', '57088', '57092', '57094', '57097', '57106', '57118', '57124', '57125', '57140', '57143', '57147', '57151', '57152', '57158', '57159', '57168', '57172', '57175', '57178', '57179', '57180', '57190', '57191', '57213', '57223', '57226', '57228', '57231', '57333', '57337', '57338', '5738', '57396', '57402', '57409', '57410', '57415', '57446', '57447', '57467', '57470', '57472', '57478', '57509', '57513', '57538', '5754', '57585', '57592', '57595', '576', '57616', '57620', '57621', '5763', '57645', '57649', '57658', '57663', '57684', '57685', '57691', '57692', '57696', '57699', '577', '57714', '57715', '57718', '57720', '57724', '57727', '57795', '57801', '57817', '57821', '57822', '57827', '57862', '57864', '5791', '5793', '5794', '5797', '5798', '5801', '5813', '5814', '58155', '58190', '5820', '58473', '58476', '58486', '58487', '58493', '58495', '58516', '58524', '58533', '5858', '5871', '5877', '5891', '59084', '5911', '5913', '5918', '59271', '59274', '59335', '5935', '59353', '5936', '5937', '5939', '5955', '5961', '5967', '5968', '5989', '599', '5990', '5992', '5993', '5994', '6013', '6018', '60312', '60313', '60343', '60370', '60385', '60437', '60467', '60468', '60484', '60492', '60494', '605', '6050', '60506', '6051', '60526', '60529', '60626', '60672', '60681', '60682', '60686', '6096', '6098', '6103', '611', '6239', '6252', '6253', '6273', '6274', '6275', '6276', '6277', '6289', '6302', '63035', '631', '6310', '6311', '6320', '6322', '6355', '638', '63877', '63895', '63897', '63898', '63906', '63915', '63926', '63928', '63933', '63935', '63948', '6397', '63974', '63977', '6398', '6405', '64063', '64066', '64067', '6407', '64083', '64084', '64089', '64094', '64098', '64100', '64110', '64111', '64129', '6415', '6419', '64207', '64208', '64211', '64215', '64222', '64224', '64225', '64231', '64236', '6424', '6425', '64318', '64319', '64327', '6433', '64342', '64343', '64359', '64374', '64393', '64395', '64396', '64400', '64405', '64412', '64417', '64418', '64420', '64423', '6443', '6444', '64446', '6445', '6447', '64478', '64499', '6450', '64518', '6452', '64582', '645851', '64598', '646', '6462', '64645', '64651', '64693', '64699', '6474', '64743', '64744', '64745', '64747', '64760', '64761', '64762', '64770', '64771', '64777', '64778', '64782', '64783', '64789', '64793', '64798', '64800', '64838', '64843', '64844', '64847', '64848', '64852', '64860', '64866', '64881', '64895', '64897', '6491', '64919', '6492', '64921', '64925', '6493', '6496', '65009', '65012', '6504', '65061', '65095', '651', '65108', '65121', '65122', '652276', '6525', '65258', '652919', '653583', '654', '6594', '65975', '65981', '65982', '65983', '65987', '65990', '65991', '66004', '66037', '664', '6640', '6641', '6642', '6645', '6660', '6664', '6665', '6666', '6668', '6671', '6674', '6676', '6689', '6691', '6695', '6744', '6756', '6758', '676', '6760', '6779', '6780', '6787', '6792', '6815', '6838', '6840', '6856', '6860', '6867', '687', '6876', '689', '6894', '6906', '6909', '6919', '6926', '6939', '6941', '6942', '6943', '6949', '6954', '6990', '6992', '7008', '7009', '7011', '7012', '7024', '7030', '7032', '7041', '7047', '7071', '7080', '7104', '7105', '7107', '7127', '7142', '7145', '7162', '7165', '7180', '7188', '7216', '7227', '7260', '7262', '7265', '7272', '7275', '728116', '7286', '728603', '7287', '7288', '729884', '734', '7348', '7379', '7380', '7399', '741', '7429', '744', '7447', '7461', '7511', '753', '754', '7541', '7543', '755', '7553', '7554', '7564', '757', '7580', '7584', '7589', '7629', '7637', '7639', '7644', '7681', '7691', '7693', '7696', '7707', '7716', '7718', '7730', '7739', '7741', '7746', '7756', '7763', '7775', '7776', '7791', '7799', '7802', '7803', '7805', '7812', '7851', '786', '7866', '7869', '7871', '78994', '78996', '78997', '79000', '79002', '79006', '79007', '79009', '79020', '79022', '79025', '79026', '79029', '79036', '79041', '79056', '79073', '79080', '79083', '79092', '79095', '79098', '79102', '79137', '79140', '79144', '79149', '79154', '79158', '7916', '79170', '79183', '79187', '7920', '79366', '79368', '794', '79415', '7942', '79443', '79444', '79567', '79570', '79574', '79575', '79582', '79583', '79585', '79605', '79608', '79612', '79614', '79616', '79621', '79628', '79629', '79637', '79641', '79647', '79650', '79660', '79665', '79666', '79669', '79673', '79674', '79679', '79683', '79684', '79689', '79692', '79711', '79712', '79719', '79724', '79725', '79730', '79734', '79740', '79745', '79750', '79752', '79762', '79767', '79768', '79770', '79774', '79778', '79785', '79786', '79789', '7980', '79801', '79811', '79812', '79816', '79817', '79825', '79827', '79828', '79829', '79836', '79838', '79839', '79842', '79843', '79845', '79846', '79847', '79850', '79856', '79863', '79882', '79884', '79892', '79893', '79905', '79915', '79925', '79927', '79929', '7993', '79933', '79935', '79939', '79940', '79946', '79949', '79953', '79954', '79957', '79962', '79968', '79970', '79979', '79981', '79982', '79983', '79986', '79990', '79998', '80013', '80018', '80023', '80039', '80045', '80111', '80117', '80135', '80148', '80155', '80157', '80174', '80178', '80185', '8019', '80206', '80207', '80208', '80210', '80213', '80218', '80231', '80255', '80256', '80262', '8030', '80303', '80309', '8031', '80315', '80319', '80320', '80325', '80345', '80346', '80352', '80381', '8045', '8048', '80712', '80714', '80723', '80745', '80757', '80758', '80762', '80774', '80775', '8079', '8082', '80823', '80830', '80864', '80895', '8092', '8099', '810', '81037', '8120', '8123', '81491', '81492', '81493', '81542', '81543', '81545', '81550', '81551', '81552', '81554', '81558', '81563', '81566', '81570', '81571', '81572', '81575', '81577', '81602', '81609', '81611', '81615', '81618', '81621', '81628', '81688', '81706', '81786', '81788', '81831', '81832', '81848', '81853', '8187', '81873', '8190', '8208', '8216', '8220', '8237', '8269', '8273', '8303', '83259', '8328', '83442', '83445', '83447', '83451', '83461', '83463', '83468', '83479', '83482', '83541', '83543', '83546', '83550', '83590', '83593', '83597', '83604', '83607', '83608', '83636', '83640', '83641', '83643', '83648', '83659', '83660', '83661', '8369', '83690', '83694', '83699', '83714', '83723', '83742', '83743', '83758', '83759', '83786', '8382', '83853', '83854', '83856', '83857', '83873', '83876', '83878', '83879', '83886', '83890', '83893', '83931', '83937', '83938', '83943', '83959', '83982', '83985', '83986', '83989', '83992', '8403', '84058', '84065', '84066', '84067', '84068', '84069', '84070', '84071', '84076', '84103', '8412', '84127', '84134', '84146', '84153', '84154', '8416', '8417', '84179', '84181', '84186', '84187', '8419', '84191', '84203', '84206', '84215', '84216', '84224', '84229', '84233', '84236', '84249', '84253', '84255', '84260', '84266', '84267', '84270', '84272', '84273', '84275', '84280', '84284', '84293', '84294', '84295', '84298', '84299', '84300', '84302', '84306', '84309', '84311', '84314', '8433', '84330', '84331', '84336', '84337', '84343', '8436', '84365', '8438', '84419', '84439', '8444', '84440', '84450', '84451', '84457', '84458', '8446', '84465', '84466', '8448', '84502', '84504', '84518', '84519', '84522', '84525', '84547', '84548', '84549', '8455', '8458', '84636', '84640', '84641', '84647', '84650', '84651', '84660', '84662', '84665', '84666', '84668', '84669', '84680', '84684', '84688', '84690', '84696', '8470', '84700', '84701', '84707', '84722', '84759', '84798', '84804', '84814', '84823', '84826', '84830', '84842', '84851', '84859', '84872', '84873', '84878', '84885', '84886', '84897', '84898', '84901', '84905', '84908', '8491', '84913', '84918', '84919', '8492', '84922', '84925', '84926', '84935', '84940', '84952', '84957', '84960', '84964', '84969', '84975', '84976', '84978', '84988', '84993', '85004', '85012', '85014', '85016', '85019', '8507', '8521', '8531', '8532', '85329', '85359', '85366', '8537', '85377', '8538', '8539', '85406', '85407', '85409', '85416', '8543', '85438', '85442', '8545', '85453', '85457', '85478', '85479', '85481', '85508', '85509', '85569', '8558', '8572', '8577', '85865', '8603', '8609', '8618', '8624', '8629', '863', '8631', '8632', '8634', '8642', '8676', '8711', '8715', '8723', '8727', '8736', '8749', '8763', '8777', '8798', '881', '8814', '8838', '8842', '88455', '8848', '8853', '8857', '8863', '8864', '8880', '8882', '8886', '8929', '8933', '8938', '8940', '8943', '89765', '89795', '89796', '89857', '89870', '89876', '8988', '8991', '89958', '8996', '8999', '90', '9001', '90011', '90060', '90070', '90102', '90121', '90139', '9019', '90203', '90204', '9022', '9024', '9028', '90324', '90326', '9033', '90332', '90355', '90381', '90407', '90416', '9044', '90441', '9050', '90507', '9052', '90522', '90523', '90525', '90529', '9053', '90557', '90624', '90637', '9064', '90693', '90737', '9077', '9079', '90806', '90826', '90853', '90861', '9093', '9094', '9095', '90956', '9098', '9104', '91057', '91074', '91133', '91137', '91151', '91156', '91179', '9118', '9121', '91252', '91283', '913', '91300', '91304', '9131', '91373', '91392', '91409', '9141', '9143', '9149', '91574', '9158', '91607', '91608', '91663', '9168', '9170', '91748', '9175', '91752', '91775', '91807', '91860', '91862', '91894', '9191', '91937', '91947', '91977', '9201', '92014', '9202', '9204', '9205', '92086', '92092', '921', '9213', '9214', '92140', '92162', '92181', '922', '9221', '9223', '923', '92344', '92345', '9236', '9240', '9242', '9247', '9253', '92558', '9262', '9263', '92675', '92689', '92714', '92749', '9275', '9278', '92797', '92840', '9289', '92906', '92922', '92960', '9308', '93081', '931', '93129', '9315', '9317', '93185', '93233', '9324', '9325', '9326', '9344', '93487', '93550', '93594', '93621', '9364', '93643', '9368', '9375', '9389', '9392', '93974', '94009', '94031', '9404', '94056', '94059', '9406', '94081', '94086', '94097', '94122', '94160', '94234', '9425', '9452', '9457', '9473', '9479', '9480', '9496', '9499', '9506', '951', '9518', '9519', '9537', '9538', '9543', '9547', '9555', '9556', '9563', '9567', '9569', '9573', '9576', '9581', '9586', '9595', '9601', '9603', '9607', '9617', '9618', '9620', '9622', '9623', '9625', '9633', '9638', '9640', '9645', '9649', '9650', '9659', '9670', '9679', '9686', '9689', '969', '9694', '9697', '97', '9701', '9703', '9704', '9705', '9706', '9722', '9733', '9737', '9741', '9744', '9745', '9746', '9747', '9751', '9753', '9758', '9764', '9770', '9772', '9774', '9777', '9778', '9779', '9780', '9783', '9788', '9792', '9796', '98', '9802', '9805', '9806', '9813', '9819', '9823', '9831', '9833', '9834', '9853', '9854', '9857', '9858', '9859', '9860', '987', '9873', '9874', '9889', '9897', '9904', '9906', '9910', '9911', '9917', '9922', '9923', '9925', '9931', '9933', '9940', '9943', '9947', '9949', '9988', '9989', '9991', '9993']
Reproducibility settings for device cpu are not implemented or necessary i.e. for cpu.
Ontix checks:
All possible feature names length: 11308
Feature order length: 2000
Feature names without filtering: 2000
Mask layer 0 with shape torch.Size([638, 29]) and 641.0 connections
Mask layer 1 with shape torch.Size([2000, 638]) and 12083.0 connections
Latent Dim: 29
Epoch 10 - Train Loss: 113.4932
Sub-losses: recon_loss: 113.4932, var_loss: 0.0000, anneal_factor: 0.0003, effective_beta_factor: 0.0000
Epoch 10 - Valid Loss: 103.5205
Sub-losses: recon_loss: 103.5205, var_loss: 0.0000, anneal_factor: 0.0003, effective_beta_factor: 0.0000
Epoch 20 - Train Loss: 61.4592
Sub-losses: recon_loss: 61.4588, var_loss: 0.0004, anneal_factor: 0.0020, effective_beta_factor: 0.0000
Epoch 20 - Valid Loss: 57.3201
Sub-losses: recon_loss: 57.3197, var_loss: 0.0004, anneal_factor: 0.0020, effective_beta_factor: 0.0000
Epoch 30 - Train Loss: 55.9777
Sub-losses: recon_loss: 55.9731, var_loss: 0.0046, anneal_factor: 0.0148, effective_beta_factor: 0.0000
Epoch 30 - Valid Loss: 52.5384
Sub-losses: recon_loss: 52.5340, var_loss: 0.0044, anneal_factor: 0.0148, effective_beta_factor: 0.0000
Epoch 40 - Train Loss: 53.2601
Sub-losses: recon_loss: 53.2204, var_loss: 0.0396, anneal_factor: 0.0998, effective_beta_factor: 0.0001
Epoch 40 - Valid Loss: 50.5189
Sub-losses: recon_loss: 50.4806, var_loss: 0.0383, anneal_factor: 0.0998, effective_beta_factor: 0.0001
Epoch 50 - Train Loss: 51.4811
Sub-losses: recon_loss: 51.2689, var_loss: 0.2122, anneal_factor: 0.4502, effective_beta_factor: 0.0005
Epoch 50 - Valid Loss: 48.8328
Sub-losses: recon_loss: 48.6314, var_loss: 0.2013, anneal_factor: 0.4502, effective_beta_factor: 0.0005
Epoch 60 - Train Loss: 50.6716
Sub-losses: recon_loss: 50.2284, var_loss: 0.4432, anneal_factor: 0.8581, effective_beta_factor: 0.0009
Epoch 60 - Valid Loss: 49.9662
Sub-losses: recon_loss: 49.5406, var_loss: 0.4256, anneal_factor: 0.8581, effective_beta_factor: 0.0009
Epoch 70 - Train Loss: 49.5703
Sub-losses: recon_loss: 49.0193, var_loss: 0.5510, anneal_factor: 0.9781, effective_beta_factor: 0.0010
Epoch 70 - Valid Loss: 47.8898
Sub-losses: recon_loss: 47.3738, var_loss: 0.5159, anneal_factor: 0.9781, effective_beta_factor: 0.0010
Epoch 80 - Train Loss: 48.7276
Sub-losses: recon_loss: 48.1328, var_loss: 0.5948, anneal_factor: 0.9970, effective_beta_factor: 0.0010
Epoch 80 - Valid Loss: 46.0186
Sub-losses: recon_loss: 45.4700, var_loss: 0.5486, anneal_factor: 0.9970, effective_beta_factor: 0.0010
Epoch 90 - Train Loss: 47.7396
Sub-losses: recon_loss: 47.1260, var_loss: 0.6136, anneal_factor: 0.9996, effective_beta_factor: 0.0010
Epoch 90 - Valid Loss: 46.2176
Sub-losses: recon_loss: 45.6503, var_loss: 0.5673, anneal_factor: 0.9996, effective_beta_factor: 0.0010
Epoch 100 - Train Loss: 46.8402
Sub-losses: recon_loss: 46.2012, var_loss: 0.6390, anneal_factor: 0.9999, effective_beta_factor: 0.0010
Epoch 100 - Valid Loss: 45.1629
Sub-losses: recon_loss: 44.5749, var_loss: 0.5880, anneal_factor: 0.9999, effective_beta_factor: 0.0010
Reproducibility settings for device cpu are not implemented or necessary i.e. for cpu.
Processed 707 / 707 samples

Gaining Insights by Visualizing Latent Dimensions¶

In comparison to other pipelines, Ontix has interpretable latent dimensions as shown in the plots below:

In [13]:

ontix_react.show_result(params=["SEX"])

ontix_react.show_result(params=["SEX"])

Creating plots ...

Obtain Results¶

If we're interested in working with latent spaces, reconstructions, or losses, we can access these for Ontix as for any other pipeline. For more details see Tutorials/DeepDives/PipelineOutputTutorial.ipynb.

In [14]:

print(f"The following results are saved: {list(r.__dict__.keys())}")
train_loss = r.losses.get(split="train", epoch=20)

print(f"Loss at epoch 20 for split train was: {train_loss}")
ls = r.latentspaces.get(split="test", epoch=-1)
ls

print(f"The following results are saved: {list(r.__dict__.keys())}") train_loss = r.losses.get(split="train", epoch=20) print(f"Loss at epoch 20 for split train was: {train_loss}") ls = r.latentspaces.get(split="test", epoch=-1) ls

The following results are saved: ['latentspaces', 'sample_ids', 'reconstructions', 'mus', 'sigmas', 'losses', 'sub_losses', 'preprocessed_data', 'model', 'model_checkpoints', 'datasets', 'new_datasets', 'adata_latent', 'final_reconstruction', 'sub_results', 'sub_reconstructions', 'embedding_evaluation', 'embedding_attributions', 'embedding_explanations']
Loss at epoch 20 for split train was: 64.98011489435729

Out[14]:

array([[ 1.4387159 ,  0.33241153,  6.9562674 , ...,  1.8334532 ,
         0.9250494 ,  5.724798  ],
       [ 2.0699162 ,  0.3377459 ,  4.3744817 , ...,  1.351126  ,
         0.07841497,  3.9260871 ],
       [ 0.6098666 ,  4.088594  ,  7.725185  , ...,  1.9639463 ,
         2.3745835 ,  7.642152  ],
       ...,
       [ 0.87403136,  5.399689  ,  7.207836  , ...,  7.074959  ,
         6.9044776 ,  7.544634  ],
       [-0.5679002 ,  5.166001  , 10.70929   , ...,  8.351236  ,
         9.499353  ,  6.655398  ],
       [ 2.2208037 ,  2.6046386 ,  7.954392  , ...,  3.9101214 ,
         3.159804  ,  5.492159  ]], shape=(707, 25), dtype=float32)

Save Ontix¶

Like all other pipelines we can save, load and re-use Ontix as shown below:

In [15]:

import os
import glob

outpath = os.path.join("tutorial_res", "ontix.pkl")
ontix.save(file_path=outpath, save_all=True)

folder = os.path.dirname(outpath)
pkl_files = glob.glob(os.path.join(folder, "*.pkl"))
model_files = glob.glob(os.path.join(folder, "*.pth"))

print("PKL files:", pkl_files)
print("Model files:", model_files)

# the load functionality automatically will build the pipeline object out of the three saved files
ontix_loaded = acx.Ontix.load(outpath)

import os import glob outpath = os.path.join("tutorial_res", "ontix.pkl") ontix.save(file_path=outpath, save_all=True) folder = os.path.dirname(outpath) pkl_files = glob.glob(os.path.join(folder, "*.pkl")) model_files = glob.glob(os.path.join(folder, "*.pth")) print("PKL files:", pkl_files) print("Model files:", model_files) # the load functionality automatically will build the pipeline object out of the three saved files ontix_loaded = acx.Ontix.load(outpath)

Preprocessor saved successfully.
Pipeline object saved successfully.
PKL files: ['tutorial_res/varix_preprocessor.pkl', 'tutorial_res/ontix_reactome_preprocessor.pkl', 'tutorial_res/ontix.pkl', 'tutorial_res/maskix_preprocessor.pkl', 'tutorial_res/imagix.pkl', 'tutorial_res/ontix_preprocessor.pkl', 'tutorial_res/disent.pkl', 'tutorial_res/imagix_preprocessor.pkl', 'tutorial_res/ontix_reactome.pkl', 'tutorial_res/disent_preprocessor.pkl']
Model files: ['tutorial_res/varix_model.pth', 'tutorial_res/ontix_reactome_model.pth', 'tutorial_res/disent_model.pth', 'tutorial_res/imagix_model.pth', 'tutorial_res/ontix_model.pth', 'tutorial_res/maskix_model.pth']
Attempting to load a pipeline from tutorial_res/ontix.pkl...
Pipeline object loaded successfully. Actual type: Ontix
Preprocessor loaded successfully.

In [16]:

ontix_loaded.show_result()

ontix_loaded.show_result()

Creating plots ...

In [17]:

test = ontix_react.result.datasets
ontix_react.save(file_path=os.path.join("tutorial_res", "ontix_reactome.pkl"))
ontix_react_loaded = acx.Ontix.load(os.path.join("tutorial_res", "ontix_reactome.pkl"))
ontix_react_loaded.predict(data=test)

test = ontix_react.result.datasets ontix_react.save(file_path=os.path.join("tutorial_res", "ontix_reactome.pkl")) ontix_react_loaded = acx.Ontix.load(os.path.join("tutorial_res", "ontix_reactome.pkl")) ontix_react_loaded.predict(data=test)

Preprocessor saved successfully.
saving memory efficient
Ontix checks:
All possible feature names length: 11308
Feature order length: 2000
Feature names without filtering: 2000
Mask layer 0 with shape torch.Size([638, 29]) and 641.0 connections
Mask layer 1 with shape torch.Size([2000, 638]) and 12083.0 connections
Latent Dim: 29
Pipeline object saved successfully.
Attempting to load a pipeline from tutorial_res/ontix_reactome.pkl...
Pipeline object loaded successfully. Actual type: Ontix
Preprocessor loaded successfully.
Reproducibility settings for device cpu are not implemented or necessary i.e. for cpu.
Processed 707 / 707 samples

Out[17]:

Result Object Public Attributes:
------------------------------
latentspaces: TrainingDynamics object
sample_ids: TrainingDynamics object
reconstructions: TrainingDynamics object
mus: TrainingDynamics object
sigmas: TrainingDynamics object
losses: TrainingDynamics object
sub_losses: LossRegistry(_losses={'recon_loss': TrainingDynamics(), 'var_loss': TrainingDynamics(), 'anneal_factor': TrainingDynamics(), 'effective_beta_factor': TrainingDynamics()})
preprocessed_data: Tensor of shape (0,)
model: OntixArchitecture
model_checkpoints: TrainingDynamics object
datasets: DatasetContainer(train=None, valid=None, test=None)
new_datasets: DatasetContainer(train=<autoencodix.data._numeric_dataset.NumericDataset object at 0x71de8a4ebdc0>, valid=<autoencodix.data._numeric_dataset.NumericDataset object at 0x71def455ff10>, test=<autoencodix.data._numeric_dataset.NumericDataset object at 0x71de8a383ac0>)
adata_latent: AnnData object with n_obs × n_vars = 707 × 29
    uns: 'var_names'
final_reconstruction: None
sub_results: None
sub_reconstructions: None
embedding_evaluation: Empty DataFrame
Columns: []
Index: []
embedding_attributions: Empty DataFrame
Columns: []
Index: []
embedding_explanations: Dict with 0 items

Generate New Data¶

For a variational autoencoder, the generate or sample_latent_space step draws new latent vectors from the model’s learned latent distribution.

Latent Sampling: The model first aggregates the posterior over all encoded latent vectors in the chosen split and epoch by computing the mean (global_mu) and log-variance (global_logvar). It then samples new latent points from a diagonal Gaussian defined by these aggregate statistics, using the reparameterization trick to inject Gaussian noise.

Number of Samples (n_samples): Users can specify how many latent points to generate. The method expands the aggregated mean and log-variance to match the requested number of samples before sampling.

Custom Latent Prior: Optionally, a custom latent_prior can be provided (a tensor or NumPy array with shape (n_samples, latent_dim)), which will be used directly instead of the aggregated posterior. This is basically the decode step.

In [18]:

generated_reconstructions = ontix_react_loaded.generate(n_samples=5)
print(generated_reconstructions)

generated_reconstructions = ontix_react_loaded.generate(n_samples=5) print(generated_reconstructions)

Reproducibility settings for device cpu are not implemented or necessary i.e. for cpu.
tensor([[0.0037, 0.0719, 0.0144,  ..., 0.3478, 0.7191, 0.6222],
        [0.0033, 0.0685, 0.0133,  ..., 0.2980, 0.7082, 0.5671],
        [0.0084, 0.0705, 0.0136,  ..., 0.3001, 0.5611, 0.5723],
        [0.0057, 0.0616, 0.0139,  ..., 0.3159, 0.6229, 0.5725],
        [0.0068, 0.0671, 0.0142,  ..., 0.3098, 0.4982, 0.5968]])

3) Evaluate Ontix on Downstream Tasks¶

Coming Soon

In [ ]: