Idun (HPC) at NTNU

1. Working Directories & Quotas (link)

Every user is the owner of a home and work directory. The home directory is located at /cluster/home/<username>, and is the current directory after login. The work directory is located at /cluster/work/<username>, and should be used as the main directory for the output of computations. In order to avoid the over-utilization of storage by a single user, both directories are limited in the size of usable disk space and number of files by a disk quota. The quota can be checked as follows:

1. Go to /cluster/work/<username> on your terminal.
2. Run lfs quota -u <username>

2. Transferring Data (link)

Reference: Original NTNU website regarding Idun - Transferring Data (link)

On the original website, several approaches are presented:

1. Network Drives – Mounting Idun’s home and work directory to your local machine.
2. Smbclient – Accessing NTNU’s network drives directly from Idun
3. Secure Copy (scp)
4. Secure File Transfer Protocol (sftp)

The 1st approach is very likely to have slow data transfer speed, and the 2nd and 4th approaches are not familiar to me. The 3rd approach, 'scp' is the data transfer approach that's often used in AWS (amazon web service).

2.1 Transfer a File from Local Computer to HPC (Idun) Using SCP (Windows10)

Make sure your VPN is turned on; Let's assume that you have a text file: C:/Users/Daesoo/local.txt in your local computer, and you want to send it to the HPC's home directory: clusters/home/daesool/ (= ~/) or work directory: /clusters/work/daesool/.

1. Open a PowerShell on your local computer.
2. Run scp <filename in local> <username>@<DNS>:<dirname in server> For example, scp C:/Users/Daesoo/local.txt daesool@idun-login1.hpc.ntnu.no:~/. If your working directory is already in 'C:/Users/Daesoo', you can run scp local.txt daesool@idun-login1.hpc.ntnu.no:~/

* Make sure that there is no word space after the colon.

2.2. Transfer a File from HPC to Local Computer Using SCP

Let's assume that you have a text file: /cluster/home/daesool/server.txt in HPC, and you want to send it to the local computer: C:/Users/Daesoo/:

1. Open a PowerShell on your local computer.
2. Run scp <username>@<DNS>:<fname in server> <dirname in local>. For example, scp daesool@idun-login1.hpc.ntnu.no:server.txt C:/Users/Daesoo/

2.3. Transfer a Directory from Local Computer to HPC Using SCP

Let's assume that you have a directory: C:/Users/Daesoo/local_dir in your local computer, and you want to send it to the HPC: /cluster/home/daesool.

1. Open a PowerShell on your local computer.
2. Run scp -r <dirname in local> <username>@<DNS>:<dirname in server>. For example, scp -r local_dir/ daesool@idun-login1.hpc.ntnu.no:~/.

2.4. Transfer a Directory from HPC to Local Computer Using SCP

Let's assume that you have a directory: /cluster/home/daesool/server_dir in the server, and you want to send it to your local computer: C:/Users/Daesoo.

1. Open a PowerShell on your local computer.
2. Run scp -r <username>@<DNS>:<dirname in server> <dirname in local>. For example, scp -r daesool@idun-login1.hpc.ntnu.no:~/server_dir/ C:/Users/Daesoo/.

2.5. Do all the above with WinSCP (highly recommended)

[1] YouTube, "How to Use WinSCP Tutorial - downloading, installing and understanding WinSCP" (link)
[2] YouTube, "WinSCP Tutorial - Connecting with FTP, FTPS, SFTP, uploading and downloading" (link)

References

[1] Hayden James, 2020, "SCP Linux – Securely Copy Files Using SCP examples" (link)
[2] Anthony James, 2012, "SSH and SCP: Howto, tips & tricks" (link)
[3] Lee D., 2020, "[AWS] how to transfer data using scp (secure copy) from a local computer"(link)

3. Install/Load PyTorch

1. Check PyTorch's version and its dependency (e.g., fosscuda-2020b) by module available pytorch
2. Check the dependency by module available fosscuda and load it (e.g., module load fosscuda/2020b).
3. Check what GCCcore your dependency uses by module list: (e.g., GCCcore/10.2.0), and load Python accordinly (e.g., module load Python/3.8.6-GCCcore=10.2.0).
4. Load PyTorch accordinly (e.g., PyTorch/1.7.1-fosscuda-2020b).

Then, you can import torch right away (e.g., in IPython).

Note that your default mounted user space is assigned 2 GPUS (Tesla P100), and 48 logic cores.