Skip to content

Commit

Permalink
Update the model structure and goals
Browse files Browse the repository at this point in the history 
I just generate a new method for the evaluation and loading of the data, a new method to process and the variational method, just trying to simplify the understanding and looking for a more confidence and stable method for the results

Co-Authored-By: Dong Han <dong.han@uconn.edu>
  • Loading branch information
@lrm22005 @doh16101
lrm22005 and doh16101 committed Nov 27, 2023
1 parent b93972d commit b3b4b4d
Show file tree
Hide file tree
Showing 3 changed files with 2,188 additions and 8 deletions.
28 changes: 20 additions & 8 deletions project_1.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -278,7 +278,9 @@ def multivariate_normal_log_pdf_MFVI(x, mu, sigma_sq):

return log_p

def perform_mfvi(data, K, n_optimization_iterations, convergence_threshold=1e-5, run_until_convergence=True):
import gc # Import the garbage collection module

def perform_mfvi(data, K, n_optimization_iterations, convergence_threshold=1e-5, run_until_convergence=False):
N, D = data.shape[0], data.shape[1] * data.shape[2] # Calculate feature dimension D
# Define the variational parameters for the GMM
miu_variational = torch.randn(K, D, requires_grad=True)
Expand Down Expand Up @@ -326,7 +328,11 @@ def perform_mfvi(data, K, n_optimization_iterations, convergence_threshold=1e-5,

prev_elbo = elbo
iteration += 1


# Clean up CPU memory using garbage collection
gc.collect()

print(f"Iteration {iteration}/{n_optimization_iterations}")
# Extract the learned parameters
miu = miu_variational.detach().numpy()
pi = torch.softmax(alpha_variational, dim=0).detach().numpy()
Expand Down Expand Up @@ -425,7 +431,7 @@ def perform_dimensionality_reduction(data, data_type, saving_path):

def perform_clustering(data, labels, pca_reduced_data, pca_labels, tsne_reduced_data, tsne_labels, mfvi_labels, data_type, saving_path):
log_progress(f"Performing clustering for {data_type} data")

print('starting clustering')
if data_type == "labeled":
method ='PCA on Labeled Data'
title_tsne = 't-SNE on Labeled Data'
Expand All @@ -448,38 +454,44 @@ def perform_clustering(data, labels, pca_reduced_data, pca_labels, tsne_reduced_
plot_pca(pca_reduced_data, labels, method, save_path=saving_path)
except Exception as e:
logger.error(f"An error occurred while plotting PCA: {str(e)}")
print('Performed PCA on the data')

try:
# Evaluate clustering for PCA results
ari_pca, ami_pca, silhouette_pca, davies_bouldin_pca = evaluate_clustering(data.view(data.size(0), -1).numpy(), labels, pca_labels)
except Exception as e:
logger.error(f"An error occurred while clustering PCA results: {str(e)}")
print('Evaluated clustering for PCA results')

# Perform t-SNE on the data
try:
# Plot t-SNE for the data
plot_clusters(tsne_reduced_data, tsne_labels, title_tsne, save_path=saving_path)
except Exception as e:
logger.error(f"An error occurred while plotting t-SNE: {str(e)}")

print('Performed t-SNE on the data')

try:
# Evaluate clustering for t-SNE results
ari_tsne, ami_tsne, silhouette_tsne, davies_bouldin_tsne = evaluate_clustering(tsne_reduced_data, labels, tsne_labels)
except Exception as e:
logger.error(f"An error occurred while clustering t-SNE results: {str(e)}")

print('Evaluated clustering for t-SNE results')

try:
# Plot MFVI for data
plot_clusters(data.view(data.size(0), -1).numpy(), mfvi_labels, title_mfvi, save_path=saving_path)
except Exception as e:
logger.error(f"An error occurred while plotting Mean-Field Variational Inference in labeled data: {str(e)}")
print('Plotted MFVI for data')

try:
# For MFVI on data
ari_mfvi, ami_mfvi, silhouette_mfvi, davies_bouldin_mfvi = evaluate_clustering(data.view(data.size(0), -1).numpy(), labels, mfvi_labels)
except Exception as e:
logger.error(f"An error occurred while clustering Mean-Field Variational Inference results in labeled data: {str(e)}")

print('Evaluated MFVI on data')

return ari_pca, ami_pca, silhouette_pca, davies_bouldin_pca, ari_tsne, ami_tsne, silhouette_tsne, davies_bouldin_tsne, ari_mfvi, ami_mfvi, silhouette_mfvi, davies_bouldin_mfvi

def visualize_and_analyze_data(data, original_data, segments, labels, data_type, saving_path, data_format):
Expand All @@ -497,7 +509,7 @@ def visualize_and_analyze_data(data, original_data, segments, labels, data_type,

try:
# Perform MFVI on data
miu, pi, resp = perform_mfvi(data, K=4, n_optimization_iterations=300, convergence_threshold=1e-5, run_until_convergence=False)
miu, pi, resp = perform_mfvi(data, K=4, n_optimization_iterations=10, convergence_threshold=1e-5, run_until_convergence=False)

# Extract cluster assignments from MFVI
mfvi_labels = torch.argmax(resp, dim=1).numpy()
Expand Down Expand Up @@ -633,5 +645,5 @@ def handle_error(task, error):

if __name__ == "__main__":
# Specify the case you want to run: 'labeled', 'unlabeled', or 'all_data'
case_to_run = "labeled"
case_to_run = "all_data"
main(case_to_run)

2 comments on commit b3b4b4d

@lrm22005
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This decoder implementation builds a convolutional neural network to map a latent representation to an image, by gradually upsampling and increasing the number of channels.

Some key points:
It first maps the latent code to a hidden representation using a fully-connected layer. This acts as the "bottleneck" that compresses information.
It then reshapes the hidden representation into a 3D tensor and uses transpose convolutions (similar to upsampling) to increase spatial dimensions. Each transpose conv gradually increases the (width, height) while reducing number of channels.
Final layers upsample to reach desired (width, height) output, and event_dims handles the channel dimension as probabilistic event dimensions.
This overall structure is commonly used in variational autoencoders (VAEs) to map a compressed latent representation back to the original input distribution. Some relevant papers that utilize similar architectures:

Auto-Encoding Variational Bayes (https://arxiv.org/abs/1312.6114) - Original VAE paper
Neural Discrete Representation Learning (https://arxiv.org/abs/1711.00937) - Uses transposed convolutions in decoder
Understanding disentangling in β-VAE (https://arxiv.org/abs/1804.03599)

ReLU Activations

The ReLU activation function is applied after each transposed convolution, for example:

nn.ConvTranspose2d(128, 64, 5, 2),
nn.ReLU(),
The ReLU simply thresholds the activations from the previous layer at 0, so:
ReLU(x) = max(0, x)

This introduces non-linearities which helps the model learn more complex mappings from the latent representation to the image. Without the ReLU (or other activation function), the model would be limited to only linear transformations, reducing its representational capacity.

The non-linearity helps the network warp and reconstruct more accurate images, as well as propagate useful gradients during training.

Transposed Convolutions

The transposed convolution layers upsample the spatial dimensions using learned convolution filters. Some key points:

They allow upsampling and expanding image spatial size in an learnable way through optimized convolution kernels.
Each transpose conv gradually increases the (width, height) while reducing number of channels, to reach final image size.
Parameters like kernel size, stride and padding determine the exact upsampling factor.
So the transposed convolutions provide an efficient way to learn an upsampling pathway for the compressed latent representation, increasing dimensions until we generate the final image.

NOTE: I don't use residual layers and I know that those can be deeper but we have a signal that is not too much complex or that requires more layers. Maybe I am doing something wrong but I'll note it.

@doh16101
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I did not know the importance of ReLU layer and forgot to add it after each BatchNorm layer. I will ask Darren to try it tomorrow @dac20022 .

Dear Darren @dac20022 ,

Please save the training loss of the models with ReLU everywhere right after the BN layer and compare the training loss. For each of the six training loss line, we should have a ReLU version of it, and please plot the training loss curves if possible.
https://github.uconn.edu/dac20022/Pulsewatch_autoencoder/blob/Cassey/Time_Frequency/plot_Losslists.m

ResNet V2 model structure with ReLU layer everywhere:
https://arxiv.org/abs/1603.05027
https://github.com/pytorch/vision/blob/c1e2095c3a16fbe7db25b9e2f206025488c2c203/torchvision/models/resnet.py#L94C11-L94C11

Please sign in to comment.