Rivanna
About Rivanna
Rivanna is the University of Virginia's main High-Performance Computing (HPC) systems. Users can access the HPC system in a variety of ways to submit jobs that require large-scale compute power. Rivanna also comes with pre-installed software packages available for computational research across many disciplines. Rivanna can be accessed online through Open OnDemand where users can directly develop and run Jupyter notebooks, MATLAB instances, and more using HPC compute resources.
Accessing Rivanna
To be able to use Rivanna, you'll need to fill out two forms to get access to compute resources: one to create a group, and one to assign resources to that group. Please see the sections "Creating a Group" and "Requesting an Allocation" to complete these steps.
Once you have access, there are a few options available for accessing Rivanna.
The simplest way to access the system is through the OpenOnDemand site which provides an online virtual machine that connects you to Rivanna. Please read the section "Using Rivanna via OOD" to learn more.
If you'd like to use your own desktop environment instead, you can also access Rivanna via SSH. Please read the section "Using Rivanna via SSH" to learn more.
Creating a Group
Navigate to the "Create new Grouper group(s) form" located on the UVA IT website here.
Example of the first half of a filled out form
Under Nickname, Alternate Contact, and Watchlist Options you can add additional points of contact for your ticket request. This nickname field does NOT correlate with the name of your group. If you're going to be working on a team, we recommend you add your teammates as watchlist users.
In the next section, select Add to begin filling out the details of your group.
Under Memebership Management, we recommend you select Personal as this gives you the most control over your group's details and is ideal for smaller cases.
Under Will this group be used for a Microsoft 365 shared mailbox or resource calendar? select No (unless of course you have additional intentions for the group).
Under Group Name, we recommend you choose a name that is descriptive of the team or project you'll be working with. If you are only creating this group for compute resources, it could be useful to explain this in the group name or description as well.
Under Group Description you may leave a blurb that will be visible to you and any members you select.
Under Primary and Secondary Owner, select the individuals who you wish to maintain and manage the group. You may add more people under Additional Owners (Individuals) or, if you'd like another group to have full access to this one, you can designate access under Additional Owners (Groups).
Example of the second half of a filled out form
Under Responsible Department, select which department is associated with the group or project. If all owners of the group are no longer available, the department you choose will be referenced instead.
Under Allow group members to view the group select Yes so you are able to reference this group when allocating resources.
Under Additional Notes, the Rivanna team recommends that you specify “This group will be used for Rivanna/Afton access” in order to expedite the process of creating a group.
Hit Save at the bottom. If you would like to create additional groups, you can repeat these steps up to 50 times. When you're done, click on the acknowledgement at the bottom of the form and hit Submit. Please note that the group creation process may take up to several days.
Next, let's look at how to request Rivanna compute resources.
Requesting a Compute Allocation
The form to request a resource allocation can be found on the Rivanna website at https://www.rc.virginia.edu/form/allocation-standard/
In the top section, fill out your Name, Email, and User ID.
Under Department/Group, select IT-Research Computing, unless you are part of a more appropriate group.
Under Classification, select which role best suits you among Faculty, Student, etc.
Under Academic Discipline, select the discipline that best suits your research project.
Below this section you are prompted to input Name of Grouper/MyGroup Account. This is the same group you just created. Enter the name in the field.
Under New or Renewal, you should select New, unless you have already been allocated resources in the past.
Under Description of Research Project leave a summary of what you will be doing with your allocated resources.
Once you have filled out the form, hit Submit at the bottom of the page. You should receive a confirmation email when your submission is received.
Using Rivanna via OOD
Open OnDemand (OOD) provides an easy-to-use graphical user interface through which you can access the HPC system. You can find the web client at https://ood.hpc.virginia.edu. This section will provide a brief overview of OOD's features and their locations. For more information regarding Open OnDemand, read the official guide here.
We recommend using Open OnDemand during the development stage of a project as the GUI and Interactive Sessions make it easy to plan and write code quickly. For executing a large-scale project using large data, running your code via Slurm scripts on the commandline is more ideal. For more information on Slurm, see "Creating Slurm Jobs."
File System
From the OOD navbar, clicking on Files gives you a few options, all of which take you to the file system.
Each user is given a small amount of permanent storage, accessible via the Home Directory button in the navbar or at address /home/<User ID>.
Users are also given substantial temporary storage (called "scratch" storage), located at /scratch/<User ID>. It's recommended to only use this folder for staging tables and similar non-permanent files as scratch folders are periodically cleared and your data will be lost.
The file system GUI allows a few basic operations such as file upload/download, file creation/renaming, and the option to open the terminal at your current location.
For large file transfers (>2 gb), UVA IT recommends transferring via an alternate method such as scp/sftp from an SSH environment or from the built-in Globus transfer tool. For more information on official recommendations please read the File Explorer documentation on the IT website here.
Interactive Apps
From the OOD navbar, select Interactive Apps and hit any of the options.
This will bring you to the Interactive Session configuration page. Each of the apps on the left column are associated with a specific service that you can launch.
In the center of the page you can configure app-specific options as well as assign specific compute resources to the launch.
Number of Hours determines how long the Interactive Session will stay available before deactivating.
Number of cores determines the cores that will be allocated to your session. Having more cores allows you to write code in parallel, which can speed up certain processes.
Memory Request in GB determines how many gigabytes of RAM will be allocated to your session. Having more RAM is useful for reading, writing, and manipulating large datasets (or many datasets at the same time).
Allocation determines which group you'd like the session to be associated with. Certain groups may not be able to request the maximum amount of resources.
When you have completed filling out the form, hit Launch at the bottom of the page to enter the queue. Depending on the resources requested, it could take some time for the session to spin up. It may be advantageous to check the box labeled "I would like to receive an email when the session starts" before submitting to the queue.
Development
The most useful Interactive Apps available for development are under Server: CodeServer, JupyterLab, and RStudio Server.
CodeServer creates a VSCode instance in your browser.
JupyterLab allows for the creation of code notebooks and consoles, namely in Python and PySpark
RStudio Server creates an R development environment
Each of these apps work with the Rivanna file system and have the resources you request available to them
File Transfer Strategies
Globus
Globus is a useful file transfer service that you can use to move otherwise cumbersomely large data from Rivanna's remote system to other locations. This tutorial will show you how to transfer to and from your local machine.
Local Setup
To transfer data from a remote Globus collection to your personal machine, you will need to install Globus Personal, available here for Mac, Windows, or Linux.
After installing, select Login. This will open up your browser for authentication. If this is your first time using Globus, select the University of Virginia as your organization. You may be asked to complete some brief account setup questions.
After logging in, Globus will ask for your consent and for you to provide a name for your machine for future reference. Enter a device name you will recognize and select Allow.
Next, you will be asked to provide a collection name. This is the name you will search for when transfering files from a remote location. Enter a recognizable name and hit Save.
Transfer
To access Globus from Rivanna OOD, first navigate to the file system. From any file location, select the button Open in Globus.
Once you have logged in, you should see the Rivanna file system from the Globus website. Navigate the file system to the data you want to download or transfer.
From here, go to the menu on the right side and select "Transfer or Sync data from one collection to another".
On the left half of your screen you should see the Rivanna file system. On top on the right half of your screen you can select the storage you would like to transfer to by hitting Search.
Under Your Collections, select the collection associated with your local machine.
On the left side of the screen, select the file(s) you'd like to download. On the right side, select a destination. When you're ready, hit Start on the left side. Your transfer will be submitted to the queue. Larger file transfers will take more time to complete.
SFTP with MobaXterm (Windows)
This section will show how to setup and use MobaXterm as an SSH and SFTP client for file transfer. The Rivanna team recommends using MobaXterm as your SSH client for Windows, but ultimately you can use any method you like.
To install MobaXterm, visit their website here. Select Installer edition.
After clicking through the setup wizard and opening the application, select Session in the top left corner.
Now, choose the session type SSH, again in the top left corner to open the connection menu
For remote host enter login.hpc.virginia.edu
For the username enter your UVA computing ID.
You can leave the port number as is. Select OK to connect. You should be prompted to enter your password in the terminal prompt that appears. If you do not see this, ensure that you are connected to UVA Anywhere.
Now you are connected to the Rivanna system! The MobaXterm client provides some handy features, such as a file system on the left hand side that tracks your terminal's location.
SFTP in MobaXterm
Using SFTP in MobaXterm is very similar to connecting via SSH. First, select Session in the top left corner.
Next, select SFTP on the screen that appears.
From here, fill out the form in the same way you filled out the SSH connection form.
Now you are connected to Rivanna via SFTP. The SFTP session contains a file browser where you can upload and download data from/to your local machine.
For large data, this method of upload and download is recommended over using Open OnDemand.
Using JupyterLab
From JupyterLab you can read and write data using Jupyter notebooks.
To get started, navigate to the Interactive Apps page. On the lefthand side, select JupyterLab under Servers.
Now, fill out the form to allocate resources to your Jupyter session.
Allocating more cores allows you to write code in parallel that can make your program vastly more efficient. If you are not planning on writing specifically for multiple cores, simply allocate 1 core to your session. This is suitable for most small and medium operations.
Allocating more memory (under "Memory Request in Gb") can be useful if you are reading or writing large files. Additionally, if your program will be performing many operations on relatively big datasets you may want to request more memory as the system might be storing extra temporary copies.
Once you have launched and connected to the JupyterLab session, you will see the Launcher. From here you can select whether to open a notebook, console, or other kind of environment.
Under Notebook, select Pyspark 4.0.0 to create a PySpark Notebook.
Now you are in a typical notebook environment where Python is the default code language.
Cells can be created either by selecting the + sign at the top or by hovering beneath the lowest cell. Cells can contain code, markdown text with light styling features, or raw text. Cells can also be rearranged via dragging and dropping.
To run a cell, hit the run button at the top menu or hit shift + enter. Cells can be run out of order, although if a cell references a variable or object from another cell, they will likely need to be run in order to function properly.
To actually use PySpark in your notebook, you'll first need to create a SparkSession like so:
Now we can use this session to read data we have uploaded to Rivanna, like so:
This loads the content of "movies.csv" into a Spark dataframe. You can also write data back to your file system like so:
This puts the contents of the dataframe into csv files located in the newly created "output" folder, along with a file named _SUCCESS that gets created upon the operation's successful completion:
For more information on PySpark, we recommend you take a look at the official documentation here.
Creating Slurm Jobs
Guide to create Slurm jobs, showing first how to construct the file and then how to submit it to the scheduler (and for OOD, show how to get to the terminal)
In Rivanna, Slurm jobs are used to run programs you have created on the HPC system. To create a Slurm job you'll first need a Slurm script that specifies which resources and modules you would like to run your job using.
This guide will show how to create a Slurm script and job using the Rivanna Job Composer. We will be running a simple Python file.
First, open up the Job Composer under Jobs in the Rivanna Open OnDemand menu. At the top of the screen, select New Job -> From Default Template.
Now, on the right hand side, you should see the job details. Scroll to the bottom, below the preview of the default Slurm script that was created for us. Select Open Editor to begin customizing the Slurm script.
The first few rows of our file are for requesting resources.
#SBATCH --ntasks=1 tells the system we only want 1 task to happen at a time. You might change this if you are running an advanced job with tasks running in parallel.
#SBATCH --mem=6000 requests 6 GB (technically 6000 MB) of RAM to work with
#SBATCH --partition=standard tells the system to run on the standard partition. Programs running on different partitions can request different amounts of resources.
#SBATCH --account=your_allocation is where you specify which group you'd like the job to run under.
#SBATCH --time=00:10:00 gives the job 10 hours to complete before being automatically stopped
Below these #SBATCH settings is where we write the actual commands we want the Rivanna computers to run. We will replace the echo command with the following:
This code first gets rid of all existing modules with "module purge". Then we load a specific version of Python, and then we run the program itself.
Hit Save on the left menu and return to the Job Composer. You should see the updated Slurm script on the right. Before we can run this job, however, there are a couple more steps we need to take. Firstly, we need to put the Python program we referenced in the same directory as the Slurm script so the system knows what we are referencing.
Select Edit files in the Job Composer to bring up the file system where our script is located. By whichever method you prefer, move a Python program named "hello-world.py" to the same directory as your Slurm script. Now, in the Job Composer, you should see the Python file listed on the right under "Folder Contents."
Finally, we need to choose a cluster to submit our job to. To do this, select Job Options in the Job Composer. Under Cluster, select HPC. Leave everything else as-is and hit Save.
Now our job is ready to be submitted. Simply hit Submit to put the job in the queue.
Any number of jobs might be in the HPC queue. It's not always guaranteed that your job will run right away. To check the status of your job either view it in the Job Composer or navigate to the HPC console and run squeue -u <Computing ID> to bring up a list of your queued and active jobs.
When the job completes, you should see a Slurm output file in the folder where the Slurm script lives. If all went well, it should read "Hello, world!"
Using Rivanna via SSH
If you are located off-grounds, you will need the UVA Anywhere VPN to connect to Rivanna resources through SSH. A guide to connecting to UVA Anywhere on both Windows and Mac can be found here. Ensure that you select "UVA Anywhere" once you have the VPN installed.
To connect to the Rivanna system using SSH, open your preferred terminal environment and enter the following:
You will be prompted for your UVA password. If you see nothing after entering the SSH command, ensure that you are connected to UVA Anywhere via the Cisco Secure Client.
After entering your password, you are connected! Type ls to show the contents of your current folder, and you should see that you are in your Rivanna home directory. For a general briefing on command line navigation commands, we recommend this Codecademy tutorial. If you are new to using the commandline, feel free to bookmark the page as a reference.
VSCode as an SSH client
VSCode has a few handy features that make development in Rivanna convenient and simple.
First, VSCode has an optional remote extension pack that allows you to connect quick and see your file system in the full VSCode environment. The extensions can be found here.
Once added to your client, navigate to the Remote Explorer tab on the left hand menu.
Hover over SSH at the top and select + New Remote. You will be prompted for a command at the top of your screen. Simply enter the same ssh command you would from a terminal and hit Enter. You will be asked to update a specific SSH config file. Unless you are planning for more advanced usage, simply click the first option.
Now you should see login.hpc.virginia.edu in the remote explorer menu. Select Connect in New Window. When prompted for the platform, select Linux. Then, enter your password.
Now you are connected! Select Open folder on the left to choose where to view first. You may have to reenter your password, but then you will be able to see the full Rivanna file system from your VSCode environment.
This approach allows for far more customization when it comes to development. If you already have a VSCode workflow in place, you can now use it to develop on Rivanna.
Rivanna AI
This section will cover two opportunities to use AI with Rivanna. The first method is with Codex, a built-in assistant for your IDE that, when using SSH with Rivanna, can analyze your code and data as you write. The second is the newly available UVA RC GenAI that you can query up to 60 times per minute for no cost from a browser or with the API. First we will look at using Codex.
Codex in VSCode
Codex is an AI coding partner from OpenAI that sits right in your IDE, integrating seamlessly into any workflow. To get started using Codex, open VSCode, and search Codex in the Extensions tab. Install the following extension:
Next, navigate to any file in your system. You should see the OpenAI logo appear in the top right of your screen, alongside your file tabs. Click this to open Codex.
You will be asked to connect, do so using the API key provided by Darden. Ensure that you do not share institutional data outside of the Darden plan. Upon your first time connecting, you will be shown many of the development features that make Codex a handy tool to have. Now you should be connected! Please keep in mind Codex will use the rate limits of the account it's connected to.
One of the main reasons Codex and similar IDE AI tools are so powerful is because they have the context of the environment you're currently working in. You can analyze multiple files at a time and ask the tool to build entire systems for you, if you'd like. In the following example you can see how Codex can easily scan an summarize a table for you, even providing additional tips.
For additional questions or concerns regarding Codex, please refer to the documentation here.
RC GenAI
The UVA RC GenAI is a new AI server availble to UVA researchers. The benefit of this AI server is that all queries and data are processed securely in-house at UVA. The server is currently running Kimi K2.5 and can be accessed either from a browser or using the API.
The server can be accessed from a browser here. Please note this model is intended for research only, and you will need a Research Computing account to access it.
From the browser chat you can directly type questions, upload files, and save responses. To use the model within a program, you'll need an API key.
Click on your profile in the top right, navigate to Settings -> Account -> API Keys (hit show) -> Create new secret key.
Never share a secret key outside of the organization. For best practices and Python code examples using the model, please check out the RC website.