High Performance Storage: Store Data for Short Term Processing Needs
High Performance Storage (HPS) is primarily for running data analytics and other highly-transactional workloads against data sets without being stored on a permanent basis. This service is only accessible via the NFS protocol and is best utilized from a server or desktop that has high-speed network connectivity. HPS is not a place to store data that needs to be kept long-term or data that needs to be backed up or replicated.
In this article:
Requesting an account
- Prepare what’s needed for the storage request form.
- Ensure your department has been on-boarded to Customer Experience Standardization for Infrastructure (CESI) as a storage consumer.
- Ensure that you have the ability to access a share over the NFS protocol.
- Determine where in your process or workflow you can take advantage of a high i/o storage space (/swap, /temp, /cache, /scratch, etc.).
- Implement a workflow to copy data off of HPS when the task is complete.
- Estimate the amount of space your data will likely consume.
- Identify systems that will need access to the export.
- Fill out the storage request form.
Note: Once you have submitted the form, a share will be configured for you. Approval typically takes 2 business days. Once you gain access to a share, you will get a notification when your share is ready.
Moving files into HPS
Important: Command line utilities like Rsync and Robocopy are examples of processes available for copying data to your share. With any command line utility, there is some risk of data loss. Make sure you understand the syntax to use these tools, so you don’t inadvertently overwrite existing data.
- Use your preferred method (e.g. Rsync, etc.) to make a copy of the data that you will be processing.
- Run your processes against it.
- When you are done processing your data, copy your results and important files back to your long-term storage location.
Files are only accessible via the NFS protocol. Files may only live on HPS up to a maximum of 60 days and will be automatically deleted after that time. In the event that you need more time, you can request an extension to the 60 days in the storage request form. Data will not be backed up or replicated.
High Performance Storage is one of seven new storage service offerings for the University community rolling out in 2020. For more information about the storage services provided by the Office of Information Technology, please see the Storage Portfolio Restructuring Initiative.