Daesoo Lee's Blog

french wash / lime wash painting

DS-Lee — Sun, 16 Feb 2025 05:46:47 +0900

https://www.youtube.com/watch?v=NWmJBvL5z4A&t=226s&ab_channel=HomeByMon

2. https://www.youtube.com/watch?v=4EC1GHQS4Tk&ab_channel=TheMerrythought

Remarks
- when painting, it's might not be obvious that you're getting the effect. When dried, it becomes obvious.
- damping a brush on water like in the first video should smooth the painting process.
- never deep-dip the brush, only on the tip, otherwise you end up putting a chunk.

[SFTP] transfer files from a server to a local device

DS-Lee — Wed, 20 Nov 2024 15:14:12 +0900

To connect to a server using SFTP and transfer a file from the server to your local PC, follow these steps:

Step 1: Prepare the Required Information

Server details:
- Username (e.g., root or daesoo)
- Server IP address or hostname (e.g., 213.173.108.222)
- Port (default is 22, but it may differ if configured differently on your server).
Authentication:
- Password or an SSH private key file (e.g., ~/.ssh/id_ed25519).

Step 2: Using the Command Line

Most Linux and macOS systems come with SFTP pre-installed. On Windows, you can use tools like PowerShell or sftp in an SSH-compatible terminal.

Steps:

Open a terminal on your local machine.

Use the following command to connect to the server:

sftp -i /path/to/private/key username@server_ip

Example using your server details:

sftp -i ~/.ssh/id_ed25519 daesoo@213.173.108.222

Once connected, navigate to the directory on the server where your file is located:
```
cd /path/to/remote/directory
```
Use the get command to download the file to your local machine:
```
get remote_filename local_filename
```
Example:
```
get data.txt ~/Downloads/data.txt
```
Exit the SFTP session:
```
exit
```

slurm job script example

DS-Lee — Fri, 8 Nov 2024 21:26:22 +0900

job_script.sh

#!/bin/bash

# Job name:

#SBATCH --job-name=hilcodec_training

# Project:

#SBATCH --account=nn11068k

# Wall time limit:

#SBATCH --time=00-00:05:00

# Other parameters:

#SBATCH --partition=accel

#SBATCH --gres=gpu:1

#SBATCH --mem=50G

#SBATCH --cpus-per-task=16

# Set up job environment:

set -o errexit # Exit the script on any error

# Initialize Conda without sourcing .bashrc

MINIFORGE_INSTALL_PATH="/cluster/projects/nn11068k/miniforge3"

eval "$($MINIFORGE_INSTALL_PATH/bin/conda shell.bash hook)"

# Load conda environment

ENV_NAME="hilcodec_inductive_bias"

conda activate "$ENV_NAME"

echo "Current Conda environment: $(conda info --envs | grep '*' | awk '{print $1}')"

# Check if PyTorch and CUDA can be loaded

echo "Checking if PyTorch and CUDA are available..."

python -c "

import torch

print(f'PyTorch loaded successfully. Version: {torch.__version__}')

cuda_available = torch.cuda.is_available()

print(f'CUDA available: {cuda_available}')

if cuda_available:

print(f'CUDA device count: {torch.cuda.device_count()}')

print(f'CUDA device name: {torch.cuda.get_device_name(0)}')

# log in wandb

wandb login MY_API_KEY

# Run Python script

echo "Run the script..."

SCRIPT_FNAME="/cluster/projects/nn11068k/daesoo/hilcodec_inductive_bias/stage1.py"

CKPT_PATH="/cluster/projects/nn11068k/daesoo/hilcodec_inductive_bias/ckpts/epoch=2-step=23320.ckpt"

python "$SCRIPT_FNAME" --ckpt_path "$CKPT_PATH"

The job can be sumibtted by running "$ sbatch job_script.sh"

NB! replace MY_API_KEY

Lipschitz continuity in GAN

DS-Lee — Fri, 8 Nov 2024 14:25:02 +0900

Lipschitz continuity plays a crucial role in Generative Adversarial Networks (GANs), especially in the context of Wasserstein GANs (WGANs). Understanding its importance requires delving into how GANs function and why enforcing Lipschitz continuity leads to more stable and effective training.

Background: Generative Adversarial Networks (GANs)

A GAN consists of two neural networks:

Generator (G): Attempts to produce data that resembles real data.
Discriminator (D): Tries to distinguish between real data and data generated by G.

The training involves a minimax game where the generator aims to fool the discriminator, and the discriminator aims to accurately classify real and generated data.

Challenges with Traditional GANs

Traditional GANs often suffer from issues like:

Mode collapse: The generator produces limited varieties of outputs.
Vanishing gradients: The generator's updates become negligible, hindering learning.
Training instability: The minimax game can be difficult to balance, leading to oscillations.

These problems arise partly because of the loss functions used, such as the Jensen-Shannon (JS) divergence, which can provide poor gradients when the generator and real data distributions do not overlap significantly.

Introduction of Wasserstein GAN (WGAN)

The Wasserstein GAN addresses these issues by:

Using the Wasserstein-1 distance (Earth Mover's Distance): Measures the minimum cost of transporting mass to transform one distribution into another.
Providing better gradients: The Wasserstein distance offers meaningful gradients even when distributions have non-overlapping supports.

Role of Lipschitz Continuity in WGAN

The critical aspect of WGAN is that it requires the discriminator (often called the "critic" in this context) to be 1-Lipschitz continuous. This requirement arises from the Kantorovich-Rubinstein duality, which expresses the Wasserstein-1 distance as:

\[ W(P_r, P_g) = \sup_{\|f\|_L \leq 1} \left( \mathbb{E}_{x \sim P_r}[f(x)] - \mathbb{E}_{x \sim P_g}[f(x)] \right) \]

Here, $ P_r $ and $ P_g $ are the real and generated data distributions, respectively, and the supremum is over all functions $ f $ that are 1-Lipschitz continuous.

Why Enforce Lipschitz Continuity?

Valid Wasserstein Distance Computation:
- The dual form of the Wasserstein distance relies on the function being 1-Lipschitz.
- Without Lipschitz continuity, the discriminator might not correctly approximate the Wasserstein distance, leading to incorrect gradients.
Stable Training:
- Enforcing Lipschitz continuity prevents the discriminator from becoming too "sharp" or "steep," which can cause instability.
- It ensures that small changes in input lead to small changes in output, providing smoother gradients for the generator to learn from.
Avoiding Exploding Gradients:
- Without Lipschitz constraints, the discriminator can produce large gradients that destabilize training.
- Lipschitz continuity bounds the gradients, preventing them from becoming excessively large.

Methods to Enforce Lipschitz Continuity

Several techniques are used to ensure the discriminator is 1-Lipschitz continuous:

Weight Clipping (Original WGAN):
- Constrains the weights of the discriminator within a specific range.
- Simple but can lead to optimization issues and capacity limitations.
Gradient Penalty (WGAN-GP):
- Adds a penalty term to the loss function that penalizes deviations from a gradient norm of 1.
- Encourages the discriminator's gradients with respect to its inputs to have a norm of 1, promoting Lipschitz continuity.
Spectral Normalization:
- Normalizes the spectral norm of each layer's weight matrix.
- Provides a more direct and effective way to enforce Lipschitz constraints without the drawbacks of weight clipping.

Conclusion

Lipschitz continuity is essential in GANs, particularly WGANs, because it ensures the discriminator (critic) can effectively approximate the Wasserstein distance between the real and generated data distributions. By enforcing Lipschitz continuity, GANs benefit from:

Improved training stability: Prevents erratic updates and promotes convergence.
Meaningful gradients: Provides useful feedback for the generator even when distributions are disjoint.
Enhanced performance: Leads to better quality generated data by accurately guiding the generator's learning process.

In summary, Lipschitz continuity is crucial in GANs to ensure that the discriminator provides reliable and stable gradients to the generator, facilitating effective learning and overcoming common challenges associated with GAN training.

Certainly! I'd be happy to clarify what the notation $ \sup_{\|f\|_L \leq 1} $ means in the context of the Wasserstein distance and GANs.

Breaking Down the Notation

$ \sup $: This stands for supremum, which is the least upper bound of a set. In simpler terms, it's the highest value that a function can approach (but not necessarily reach) within a given set.
$ \|f\|_L \leq 1 $: This denotes all functions $ f $ whose Lipschitz constant is less than or equal to 1. The Lipschitz constant $ \|f\|_L $ measures how steep a function can be—it’s the maximum rate at which the function's output can change with respect to changes in its input.
- Lipschitz Constant Definition: \[ \|f\|_L = \sup_{x \neq y} \frac{|f(x) - f(y)|}{|x - y|} \] This means that for any two points $ x $ and $ y $, the change in $ f $ between these points is bounded by $ \|f\|_L \times |x - y| $.

Putting It All Together

In the expression:

\[ W(P_r, P_g) = \sup_{\|f\|_L \leq 1} \left( \mathbb{E}_{x \sim P_r}[f(x)] - \mathbb{E}_{x \sim P_g}[f(x)] \right) \]

$ W(P_r, P_g) $: Represents the Wasserstein-1 distance between the real data distribution $ P_r $ and the generated data distribution $ P_g $.
$ \mathbb{E}_{x \sim P_r}[f(x)] $: The expected value of $ f(x) $ when $ x $ is sampled from the real data distribution.
$ \mathbb{E}_{x \sim P_g}[f(x)] $: The expected value of $ f(x) $ when $ x $ is sampled from the generated data distribution.
$ \sup_{\|f\|_L \leq 1} $: We are considering the maximum possible value of the expression $ \mathbb{E}_{x \sim P_r}[f(x)] - \mathbb{E}_{x \sim P_g}[f(x)] $ over all functions $ f $ that are 1-Lipschitz continuous.

Why Only Functions with $ \|f\|_L \leq 1 $?

The restriction to functions with $ \|f\|_L \leq 1 $ (1-Lipschitz functions) is crucial because:

Lipschitz Continuity Ensures Smoothness:
- A function being 1-Lipschitz means it doesn't change too abruptly; its rate of change is bounded by 1.
- This property is essential for the mathematical foundations of the Wasserstein distance, ensuring that the measure is finite and well-defined.
Kantorovich-Rubinstein Duality:
- This duality theorem relates the Wasserstein distance to the supremum over Lipschitz functions.
- It states that the Wasserstein-1 distance can be expressed as the maximum difference in expectations over all 1-Lipschitz functions.
- Mathematically: \[ W(P_r, P_g) = \sup_{\|f\|_L \leq 1} \left( \mathbb{E}_{x \sim P_r}[f(x)] - \mathbb{E}_{x \sim P_g}[f(x)] \right) \]

Intuitive Explanation

Imagine you're trying to find the biggest difference between the expectations of two distributions using a "smooth" function $ f $ that doesn't spike or drop too sharply (since it's 1-Lipschitz). The "sup" (supremum) tells us to find the function $ f $ within this smooth class that maximizes this difference.

Why Maximize This Difference?
- By finding the function that maximizes the difference in expectations, we're effectively measuring how distinguishable the two distributions are using smooth functions.
- The greater the Wasserstein distance, the more different the distributions are.

Example to Illustrate

Suppose we have two distributions $ P_r $ and $ P_g $ on the real line, and we want to compute their Wasserstein distance.

Choose All Possible 1-Lipschitz Functions:
- Consider every function $ f $ that doesn't change faster than at a rate of 1 (its slope anywhere is at most 1).
Compute the Difference in Expectations:
- For each such function $ f $, calculate $ \mathbb{E}_{x \sim P_r}[f(x)] - \mathbb{E}_{x \sim P_g}[f(x)] $.
Find the Maximum Difference:
- The supremum $ \sup_{\|f\|_L \leq 1} $ tells us to pick the function $ f $ that gives the largest possible difference in expectations.

Visual Interpretation

Imagine plotting all possible 1-Lipschitz functions on a graph.
For each function, you measure how differently it "sees" $ P_r $ and $ P_g $.
The supremum picks the function that accentuates the difference between $ P_r $ and $ P_g $ the most, but still within the constraint of being 1-Lipschitz.

Summary

$ \sup_{\|f\|_L \leq 1} $: Represents taking the maximum over all functions $ f $ that are 1-Lipschitz continuous.
Purpose in Wasserstein Distance:
- Ensures that the distance measure captures the "best" way to distinguish between $ P_r $ and $ P_g $ using smooth functions.
- Maintains mathematical properties that make the Wasserstein distance a meaningful and robust metric.

In the Context of GANs

In Wasserstein GANs, the discriminator (critic) aims to approximate this supremum by finding a function $ f $ (parameterized by a neural network) that maximizes the difference in expectations.
By enforcing that $ f $ is 1-Lipschitz (through techniques like gradient penalty), we ensure that the critic stays within the set of functions over which the supremum is defined.

Conclusion

The notation $ \sup_{\|f\|_L \leq 1} $ is a compact way of expressing that we're looking for the maximum value of the expression over all functions $ f $ that are 1-Lipschitz continuous. It's fundamental to defining the Wasserstein distance in a way that is both mathematically rigorous and practically useful in training GANs.

installing miniforge

DS-Lee — Wed, 6 Nov 2024 22:42:07 +0900

Example of installing miniforce (kind like mini conda) on the sigma2 server machine.

#!/bin/bash

# Variables
MINIFORGE_URL="https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh"
MINIFORGE_INSTALL_PATH="/cluster/projects/nn11068k/miniforge3"

# Silently download and install Miniforge
echo "Downloading and installing Miniforge..."
wget -q "$MINIFORGE_URL" -O /tmp/Miniforge3.sh
bash /tmp/Miniforge3.sh -b -p "$MINIFORGE_INSTALL_PATH"

# Initialize Conda
echo "Initializing Conda..."
eval "$($MINIFORGE_INSTALL_PATH/bin/conda shell.bash hook)"
conda init

# Ensure Conda is available in the current shell
source ~/.bashrc

creating a custom module in slurm

DS-Lee — Wed, 6 Nov 2024 21:47:59 +0900

The following shows the example of creating a custom module for PyTorch in Slurm.

Creating a custom module for PyTorch version 2.5.1 in a Slurm-managed high-performance computing (HPC) environment involves several steps:

Create a virtual environment

mkdir -p modules/pytorch/2.5.1
python -m venv modules/pytorch/2.5.1
source activate modules/pytorch/2.5.1/bin/activate

Install PyTorch and Other Libraries
```
 python modules/pytorch/2.5.1/bin/pip torch ... 
```
"modules/pytorch/2.5.1/bin/pip" specifies the exact pip to use.

Create the Modulefile

Modulefiles are scripts that configure the environment variables needed to use the software. Here's how to create one:

Create the Directory Structure:
```
mkdir -p modulefiles/pytorch
```

Create the Modulefile:

Create a file named 2.5.1 (it's called a TCL file) inside /usr/local/modulefiles/pytorch/ with the following content:

nano modulefiles/pytorch/2.5.1

copy and paste the following

 #%Module1.0#####################################################################
 ##
 ## PyTorch 2.5.1 modulefile
 ##
 proc ModulesHelp { } {
     puts stderr "This module loads your personal PyTorch 2.5.1 virtual environment."
 }
 module-whatis "Loads your personal PyTorch 2.5.1 virtual environment"

 # Set the root of your virtual environment
 set root /cluster/projects/nn11068k/modules/pytorch/2.5.1

 # Set the VIRTUAL_ENV environment variable
 setenv VIRTUAL_ENV $root

 # Unset PYTHONHOME to avoid conflicts
 unsetenv PYTHONHOME

 # Prepend the virtual environment's bin directory to PATH
 prepend-path PATH $root/bin

 # Prepend the virtual environment's library directories to LD_LIBRARY_PATH
 prepend-path LD_LIBRARY_PATH $root/lib
 prepend-path LD_LIBRARY_PATH $root/lib64

 # Prepend the site-packages to PYTHONPATH
 prepend-path PYTHONPATH $root/lib/python3.11/site-packages

 # If using CUDA, you might need to set CUDA paths (optional)
 # prepend-path LD_LIBRARY_PATH /usr/local/cuda/lib64

Test the Module

 module load Python/3.11.5-GCCcore-13.2.0
 module use modulefiles  # register the directory for our custom modulefiles
 module load pytorch/2.5.1

test it by running python and try

import torch
torch.rand(4)
torch.__version__

intuitive understanding of magnitude and phase from STFT

DS-Lee — Tue, 22 Oct 2024 18:51:40 +0900

In the context of audio signals, the magnitude and phase obtained from the Short-Time Fourier Transform (STFT) have intuitive meanings related to the perception and structure of sound:

Magnitude:

The magnitude represents the strength (or amplitude) of each frequency component in the signal at a given time. Intuitively, it tells you:

How loud each frequency is: Higher magnitude values correspond to louder sounds at that particular frequency.
Timbre and spectral content: The combination of different magnitudes across frequencies shapes the timbre of the sound, giving it its characteristic tone. For instance, a violin and a piano playing the same note will have different magnitudes in their harmonic overtones, leading to distinct sounds.

In essence:

Magnitude controls volume and timbre. For example, a loud bass drum hit will have high magnitude values in low frequencies, whereas a violin playing a high-pitched note will have higher magnitudes at higher frequencies.

Phase:

The phase represents the timing or alignment of the frequency components in relation to one another. While it might be less directly perceptible to the human ear than magnitude, it is still important for:

Waveform reconstruction: The phase ensures that the different frequency components combine at the correct points in time when reconstructing the signal. Incorrect phases can cause destructive interference, leading to a distorted or incorrect audio signal.
Localization of sound: Phase differences between left and right ear signals help the brain localize where sound is coming from (e.g., stereo sound). This is especially true for low frequencies, where phase differences play a significant role in directional perception.

In essence:

Phase controls how frequencies align to shape the final waveform. If two audio signals have the same magnitude spectra but different phase information, they can sound quite different when reconstructed.

Example:

Magnitude affects how "big" a note or sound feels (loudness, timbre).
Phase affects the fineness of timing and how frequencies combine to form the final sound. Without correct phase alignment, a note or sound could become muddled or hollow.

So, while magnitude gives you the energy of the sound at different frequencies, phase ensures that this energy is combined correctly to reconstruct the original sound accurately.

intuitive understanding of imaginary number

DS-Lee — Tue, 22 Oct 2024 17:57:44 +0900

REF: https://www.youtube.com/watch?v=sZrOxm5Gszk

rsync, parallel

DS-Lee — Tue, 22 Oct 2024 13:19:08 +0900

Basic Form of rsync

rsync -rv --progress local_dir destination/

-r: recursive
-v: verbose

Transfer a directory from local PC to an EBS attached to an ec2 instance

rsync -rv --progress -e "ssh -i ~/.ssh/your-ssh-key" local_dir destination/

-e is for inputting credential

(example)
rsync -rv --progress -e "ssh -i ~/.ssh/daesoo-linux-pc.pem" SoundlySpeech_24000hz ec2-user@ec2-13-60-224-245.eu-north-1.compute.amazonaws.com:/workspace/

Basic Form of parallel

parallel echo "First: {1} Second: {2}" ::: A ::: X Y Z
output:

First: A Second: X
First: A Second: Y
First: A Second: Z

rsync with parallel

parallel -j 12 --eta rsync -rv {1} {2} ::: local_dir ::: destination/

-j: number of jobs
--eta: estimated time of arrival

(example)
parallel -j 12 --eta rsync -rv -e {1} {2} {3} ::: "ssh -i ~/.ssh/daesoo-linux-pc.pem" ::: SoundlySpeech_24000hz ::: ec2-user@ec2-13-60-224-245.eu-north-1.compute.amazonaws.com:/workspace/

study materials for diffusion models

DS-Lee — Mon, 5 Jun 2023 09:38:24 +0900

https://towardsdatascience.com/understanding-the-denoising-diffusion-probabilistic-model-the-socratic-way-445c1bdc5756

Understanding the Denoising Diffusion Probabilistic Model, the Socratic Way

A deep dive into the motivation behind the denoising diffusion model and detailed derivations for the loss function

towardsdatascience.com

with the background in ELBO from https://process-mining.tistory.com/161

VAE 설명 (Variational autoencoder란? VAE ELBO 증명)

Variational autoencoder, 줄여서 VAE는 GAN, diffusion model과 같이 generative model의 한 종류로, input과 output을 같게 만드는 것을 통해 의미 있는 latent space를 만드는 autoencoder와 비슷하게 encoder와 decoder를 활용

process-mining.tistory.com

Daesoo Lee's Blog

french wash / lime wash painting

[SFTP] transfer files from a server to a local device

Step 1: Prepare the Required Information

Step 2: Using the Command Line

Steps:

slurm job script example

Lipschitz continuity in GAN

Background: Generative Adversarial Networks (GANs)

Challenges with Traditional GANs

Introduction of Wasserstein GAN (WGAN)

Role of Lipschitz Continuity in WGAN

Why Enforce Lipschitz Continuity?

Methods to Enforce Lipschitz Continuity

Conclusion

Breaking Down the Notation

Putting It All Together

Why Only Functions with \( \|f\|_L \leq 1 \)?

Intuitive Explanation

Example to Illustrate

Visual Interpretation

Summary

In the Context of GANs

Conclusion

installing miniforge

creating a custom module in slurm

intuitive understanding of magnitude and phase from STFT

Magnitude:

Phase:

Example:

intuitive understanding of imaginary number

rsync, parallel

study materials for diffusion models