Seneca Cluster

General Information

Seneca cluster is an HPC cluster featuring CPUs, Nvidia H100 GPUs, a BeeGFS parallel file system and 400 Gb NDR infiniband interconnect. The cluster runs on Rocky Linux 9 and uses slurm 25.05.3 for job scheduling.

Seneca cluster is currently in early user beta. See the Request Access section for how to get access to the cluster.

Hardware and Networking

Servers

Name CPU Threads RAM GPUs Public IP Private IP IPoIB IP
Login Node
seneca-login1 64 1 TB N/A 128.84.3.171 192.168.16.253 192.168.17.253
Compute Nodes
c0001 128 1 TB 4 x Nvidia H100 80 GB HBM3 N/A 192.168.16.1 192.168.17.1
c0002 128 1 TB 4 x Nvidia H100 80 GB HBM3 N/A 192.168.16.2 192.168.17.2
c0003 128 1 TB 4 x Nvidia H100 80 GB HBM3 N/A 192.168.16.3 192.168.17.3
c0004 128 1 TB 4 x Nvidia H100 80 GB HBM3 N/A 192.168.16.4 192.168.17.4
c0005 128 512 GB 2 x Nvidia H100 80 GB PCIe N/A 192.168.16.5 192.168.17.5
c0006 64 128 GB 2 x Nvidia A100 40 GB PCIe N/A 192.168.16.6 192.168.17.6
c0101 256 1 TB N/A N/A 192.168.16.101 N/A
c0102 256 1 TB N/A N/A 192.168.16.102 N/A
c0103 256 1 TB N/A N/A 192.168.16.103 N/A
c0104 256 1 TB N/A N/A 192.168.16.104 N/A
c0105 256 1 TB N/A N/A 192.168.16.105 N/A
c0106 256 1 TB N/A N/A 192.168.16.106 N/A
c0107 256 1 TB N/A N/A 192.168.16.107 N/A
c0108 256 1 TB N/A N/A 192.168.16.108 N/A
c0109 256 512 GB N/A N/A 192.168.16.109 N/A
c0110 256 512 GB N/A N/A 192.168.16.110 N/A
c0111 256 512 GB N/A N/A 192.168.16.111 N/A
c0112 256 512 GB N/A N/A 192.168.16.112 N/A

See the --gres option in the Submit a Job section on how to request GPU access.

Storage

The following storage is available. Note: storage on Seneca cluster are not backed up.

  • User Home Directory: Each user has a home directory with a 100 GB default quota. It can be access by the path ~.

  • BeeGFS Parallel File System: Each project has a directory on the BeeGFS parallel file system: /mnt/beegfs/<institution>/<project name>. The directory is readable and writable to all users belonging to the project. <institution> in the path is one of:

    • ithaca: for projects from Ithaca campus
    • qatar: for projects from Weill Cornell Medicine Qatar campus
    • weill: for projects from Weill Cornell Medicine NYC campus

  • Local Scratch: Local scratch is mounted on /tmp on all nodes.

In addition to scp/sftp, the storage on seneca cluster is available via the Seneca Cluster Globus collection. See this page for more information on Globus transfer.

Networking

The cluster nodes are connected by the following networks:

  • Public network: The cluster head node and login node are connected to the public Internet via 25 Gb ethernet.
  • Private network: All cluster nodes are connected to a private 128.84.16.0/24 network via 25 Gb ethernet.
  • IPoIB network: All cluster nodes are connected to a private 128.84.17.0/24 network via IPoIB network for BeeGFS file system.

In addition, login and GPU nodes are connected by a NDR Infiniband interconnect (400 Gbit NDR).

Request Access

To get access to Seneca cluster:

  1. Cornell faculty and staff can create a CAC project with Seneca cluster access,
  2. Current CAC project PI and proxy can add seneca cluster to their project. Note: During early user beta test, please email CAC Help to request Seneca cluster access. Please include your project name in the request.
  3. Other users can request to join a project with Seneca cluster access

Create a CAC Project with Seneca Cluster Access

Note: During early user beta test, please create a new CAC project and email CAC Help to request Seneca cluster access. Please include your project name in the request.

See Create a New CAC Project for instructions on creating a new CAC project with Seneca cluster access on the CAC Portal

Billing and cost information

See CAC Rates for detailed billing and cost information. Note: during early user beta test, the project will not be charged. The project PI and proxy will be notified before the cluster goes into production and usage charges begin.

Join a Seneca project

See How to Join an Existing Project for steps to request access to an existing Seneca project.

Access the Cluster

SSH

You must use an ssh key to log into the seneca cluster. Password logins are disabled. See the Initial Login via SSH Key section on how to generate an ssh keypair on CAC Portal to log into the cluster for the first time.

Initial Login via SSH Key

After being added to a project with access to Seneca cluster, you can:

  1. Log into the CAC Portal by clicking on the Login button in the upper right corner.

  2. Choose Cornell University or Weill Cornell Medical College as your organizational login and complete the login using your Cornell NetID or Weill CWID and password (not your CAC account password).

  3. On the Portal dashboard, click on the Generate SSH key pair link in the Manage your CAC login credentials section in the upper right corner.

  4. Click on the Generate a new SSH key button to generate a new ssh key pair.

  5. Click on the Download your SSH private key button to download the private key to your computer.

  6. On your computer, make sure the private key file is readable and writable to you only: chmod 600 <private key file>

  7. SSH to seneca cluster using the private key you just downloaded: ssh -i <path to the private key file> <NetID>@seneca-login1.cac.cornell.edu

  8. On seneca-login1, you can add additional public ssh keys to your ~/.ssh/authorized_keys file, 1 key per line. The key generated by CAC Portal is marked by the CAC_SSO comment in the authorized_keys file. Keys that do not end in CAC_SSO will be left alone by CAC Portal.

OpenOnDemand

You can access seneca cluster using a web browser via the cluster's OpenOnDemand interface. Before using OpenOnDemand, you must first log into the login node via ssh seneca-login1.cac.cornell.edu at least once.

Scheduler

Users gain access to compute nodes using the slurm scheduler. The Slurm Quick Start Guide is a great place to start. For more detailed explanations, see CAC's Slurm page.

Partitions

Currently the seneca cluster has 2 partitions:

Partition Nodes Resources
gpu_only c000[1-4] Each node has 128 CPU threads, 1 TB RAM, and Nvidia GPUs
cac_cpu c0[101-112] Each node has 256 CPU threads and 1 TB RAM

Submit a Job

  • Interactive Job: srun --pty <options> /bin/bash

After slurm allocates a node as specified by your options, you will be given a login prompt on the compute node to run jobs interactively.

  • Batch Job: sbatch <options> <job script>

Slurm will run the job script on the allocated node(s) as specified by your options.

The following options are relevant in seneca cluster. Required options are denoted by *.

  • -P, or --partition*: Partition. Currently gpu_only is the only partition. More partitions will be added in the future.

  • -A, or --account*: Slurm account/CAC project against which this job will be billed.

If you do not specify --account, your Slurm DefaultAccount will be used automatically.
You can view your DefaultAccount with:

  sacctmgr show user $USER format=User,DefaultAccount

Qatar projects are always allowed to submit to the gpu_only partition. All other projects must have a positive balance at job submission time or the request will be rejected.

  • --gres*: Requested GPU resource

For the gpu_only partition, at least 1 GPU must be requested like this:

--gres=gpu:<type>:<count>

The following GPU types are available:

GPU --gres Option
Nvidia H100 80 GB HBM3 --gres=gpu:h100:<number of GPUs>
Nvidia H100 80 GB PCIe --gres=gpu:h100pcie:<number of GPUs>
Nvidia A100 40 GB PCIe --gres=gpu:a100:<number of GPUs>

For example, to request 2 NVIDIA H100 GPUs: --gres=gpu:h100:2

  • --time*: Time limit in the HH:MM:SS format

Maximum time limit for the gpu_only and cac_cpu partition is 24 hours (1 day).

If you need to run more than 24 hours in the gpu_only and cac_cpu partition, email your request to help@cac.cornell.edu.

  • --qos: longrun for long running (time limit >24 hours) jobs

If approved to run more than 24 hours, use the --qos=longrun option to request time limit longer than 24 hours. For example:

  # OK: 1 day (24h) or less
  --time=24:00:00

  # OK: >24h with longrun QOS if your project is approved to run longer than 24 hours
  --time=72:00:00 --qos=longrun

  # FAIL: >24h without longrun QOS
  --time=72:00:00

Here are some minimal examples:

  • An interactive job with 4 CPU threads/2 physical CPU cores and 1 Nvidia H100 GPU with time limit of 2 hours:

srun -p gpu_only --account=abc123_0002 --gres=gpu:h100:1 --time=02:00:00 -c 4 /bin/bash

  • The following job script (gpu_job.sh) can be submitted using the sbatch gpu_job.sh command to run on 4 CPU threads/2 Physical CPU cores and 1 Nvidia H100 GPU with time limit of 2 hours. Loading the anaconda3 module brings Python 3.13 into the environment, along with a number of useful packages:
  #!/bin/bash
  #SBATCH --job-name=my_gpu_job
  #SBATCH --account=abc123_0002
  #SBATCH --partition=gpu_only
  #SBATCH --gres=gpu:h100:1
  #SBATCH --time=02:00:00
  #SBATCH -c 4

  module load anaconda3/2025.06
  python my_gpu_script.py

Software

General access software is installed via the module system. Current software includes (but is not limited to):

  • R
  • alphafold3
  • apptainer
  • anaconda3 (python3 and assorted packages - matplotlib, numpy, scipy, torch, etc.)
  • cuda/12.9
  • nanoplot
  • rstudio
  • tensorrt

To view a list of installed modules:

module avail

To load a software module (anaconda3, for example):

module load anaconda3

  • Users are eligible to compile and install software within their home directories, but space will count against your home directory quota.

  • Please do not compile anaconda or conda in your home directory as it is very large. Python packages are available by loading the anaconda3 module. Installed python packages are show by doing:

module load anaconda3
pip list

If you require additional python packages, please send a request to help@cac.cornell.edu.