When a Storage Project ends

What happens to data in a project directory when the project ends?

A Project Directory is where your storage resides. When the storage project that a project storage directory belongs to ends, the project no longer has the right to continue using the NAISS storage resource to store new data and should start the process of removing all stored data.

If you anticipate that your deletion process will finish within 30 days of the project end date, you can stop reading now.

Why must data be removed at all?

As stated in the NAISS storage policy, the project PI should save project data at the latest 90 days after the project ends. The remaining data will then be deleted by the Centre where the resource is located.

The easiest way to implement this policy would be to automatically delete all remaining data after 90 days. However, we believe that would often lead to the loss of valuable research data.

Instead, we have decided on the method described below. In short:

• Send repeated email reminders to the PI and project members telling them to cater for their data.
• Make the directory more and more inaccessible to ensure that users notice that something is about to happen.
• Make a reasonable effort to get confirmation from the PI to delete the data.
• Finally: delete any remaining data.

Phase 1: “30 days of normal use + email warnings”

When a project ends, the project directory remains writable for 30 days.

This period allows running jobs to finish (jobs are allowed to start up until the project ends) and lets you do limited post-processing (e.g. package files into zip or tar archives) and to transfer data away from the storage resource.

Please plan to finish all post-processing and remove your data no later than 30 days after your project ends! The intent with any time beyond that is that it should be an extra margin in the case of illness, technical problems, etcetera.

During this period, automated emails are sent weekly to the project PI and all members who have data stored in the project directory.

Please note that we cannot guarantee that you will receive those emails. Common problems are:

• The email address you have registered in SUPR is no longer valid, or you don’t read it very often.
• The automated emails are classified as spam and end up in your spam folder.

If you know that a project you are a member of has ended more than a week ago and you have not received these emails, look for them in your spam folder or contact NAISS Support. The emails sent from the resource owner have the subject “Important information regarding … on … Storage”.

If you don’t know what to do with your data after the project ends, talk to the project PI, your local Data Office, or contact NAISS User Support. In some cases, Swestore can be a viable alternative.

Phase 2: “another 30 days, the data is delete-only + email warnings”

From 30 to 60 days after the end of the project, the data remain on disk. However, you cannot write new data, only read and delete old data. Due to the technical implementation, the behaviour for Swestore may be slightly different.

By making the data “delete-only”, any jobs that use the project directory for output will fail, and that user will notice something is about to happen.

During this period, automated email reminders will continue.

Phase 3: “another 30 days, the data is unavailable but still restorable”

From 60 to 90 days after the project ends, the data remains on disk but stored hidden from view.

The data is hidden to make sure any users or applications that use the data (e.g. as a read-only data set) notice that it is gone and can act before the actual data deletion.

Instead of sending automated email reminders during this period, the resource owner will make a reasonable effort¹ to reach the project PI to obtain an acknowledgement to delete the remaining data and that the project has saved any valuable data.

If the resource owner cannot reach the PI with a reasonable effort, we reserve the right to delete the remaining data in the project directory 90 days after the project ends (or later), or to keep it on disk for up to a year, depending on the circumstances.²

Phase 4: deletion

Once the PI has acknowledged that the remaining data is up for deletion, the project directory the deletion may take place even if less than 90 days have passed since the project ended. The automated emails will stop upon the acknowledgement.

If we cannot reach the PI, we reserve the right to delete the data anyhow but will not do so until at least 90 days after the project has ended.

Once the deletion of the data has commenced, there is no way to get it back. There are no backup copies, e.g. on tape, or similar.

If you cannot move the data in time

If you are a project member: talk to the project PI, or if it’s getting close to the 90-day limit, contact NAISS Support directly.

If you are the project PI: talk to the NAISS Support, which will usually give projects a reasonable extension to allow you enough time to move your data. But there has to be a reason for it, and extensions are always limited in time.

Adopted storage

Sometimes a project will continue under another name (e.g. when a 12-month compute project is replaced by another one). To avoid the movement of all data and the editing of job scripts to use the new location, the project PI can decide to adopt the directory of a previous project. That is done in SUPR when applying for the new project.

That can happen anytime, even during the 90-day process described here. When an active project takes over an old project directory, the directory is immediately automatically restored to normal writable status, and the email reminders cease.

What happens when a user is removed from the project?

Data belonging to a user who is no longer a member of the project stays in the project directory but will not be accessible by that user. It is then up to the project PI what to do with the data (delete, keep, change owner, …). If the project PI cannot access or delete the data, it is possible to request help from NAISS Support.

Data in a user’s home directory is not affected by project membership.

1. This “reasonable effort” usually consists of attempts to email the project PI and proxy at the email addresses registered in SUPR. If we receive no reply, we will typically try one more time using some other method, e.g. SMS text message to the phone number the PI has registered in SUPR or checking for updated email addresses in the directory of the PI’s home university. ↩︎
2. Examples: If the storage space is needed for other projects, we will delete the data to make room for them. ↩︎