How to manage your files
A typical imaging session can create anywhere from a couple of dozens to several hundreds of files. Keeping track of them can be quite a challenge. And if you process several months later, the challenge only increases. Therefore a consistent and informative file management and naming convention can prevent a lot of headaches during processing.
There is no absolute truth on this topic, and a lot of what is being described comes down to personal taste. Whatever works for one, may not work for someone else. So this blog is intended just to share how I have setup my workflow and why. Perhaps the most important message here is to think about it, and design your own. As some level of standardisation does absolutely help to keep track of the files during processing.
Step 1: Blinking the images
The first thing to do for me after a night of imaging is to put all images together in one directory, with subdirectories for each filter. That allows me to quickly check the quality of individual subs in PixInsight using the tool Process>ImageInspection/Blink. Passing clouds, focusing errors, tracking errors etc. can be easily identified and bad frames are immediately deleted. Blinking has the option to show the images as a kind of time-lapse automatically, which speeds up the process a lot. It offers the option to have the same auto-stretch applied to all images (so you can recognise dusk and dawn conditions better), or to each image individually (for example to compare images of different exposure in one go).
Blinking is a great way in PixInsight to quickly get an impression of the quality of large sets of images.
Step 2: Renaming the files
Most capture software will allow metadata (e.g. date, filter, etc) to be added to the filenames. So perhaps you’re fine with the file naming that the frames come out with. But in my experience, those choices are limited and customisation often not possible. So in my workflow, all frames are renamed according to a standardised naming convention. When the set of frames is final after the blinking, that would be a good time to do it.
The tool I use is Better Rename 10, which is available on the App Store. It offers an enormous amount of fine-tuning and flexibility and easily handles a lot of files at once. In general, this is a good tool to have around. Oh, and if you think you’ve made a mistake during renaming, you can always check the FITS-header. All critical capture information is embedded there, so you don’t have to be concerned about loosing any information.
The naming convention is designed to include all information needed during pre-processing. That information for me is:
Object-name
Date
Gain
Exposure
Sensor Temperature
Filter
With this information, the proper calibration frames can be applied and the proper light frames can be stacked. The next thing is to apply this information in a consistent way. The format I am using looks like this:
Object_Date_GxxxTxxxCxxF_###.fits
Note that there are no spaces used. Instead, underscores are applied as alternatives for spaces. PixInsight and some file servers don’t like spaces that much. For myself I refer to this as the GTCF-format. For a particular object, this would look like:
M031_20201015_G139T180C25L_012.fits
This is the 12th frame captured on 15 October 2020 of the Andromeda galaxy. It is a 180s exposure at Gain 139, with a sensor temperature of -25ºC. A counter is added for ease of identifying individual frames. Using this convention, regular sorting in the Finder groups frames together in a logical order as well, so you’re not too dependent on a directory structure to provide grouping.
Slight variations of this convention are used for naming other files than the regular light frames:
Dark_20201015_G139T180C25_001.fits (a dark-frame with same camera settings)
Bias_20201015_G138T32uC25_001.fits (a bias-frame of 32 µs with the same camera settings)
Flat_20201015_G139T001C25L_001.fits (a flat-frame of 1s to correct the above M31 frame)
MDark_20201015_G139T180C25_#50.fits (a master dark frame based on 50 images)
M031_20201015_G139T180C25L_#63.fits (a stacked image based on 63 images)
Better Rename 10 is a great tool that would fit well on any Mac
Step 3: Directory structure and Back-up
Next the files need to be put in a logical place. I keep images grouped based on target, with the name of that target being the main folder. The default subfolders in that main folder are ‘Capture’, ‘Process’, and ‘Output’. At this stage only the Capture folder contains files. Intermediate files, calibrated and registered subs etc would end up in the process folder. The output folder is the place for final TIFF or JPG images. Within the main folder, the PixInsight project file for that target is kept. Within each project intermediate result files, masks, etc. together with their history are saved. This makes it easy to reconstruct the processing at a later time.
The main folder will move, depending on the progress for that target. It can have either of the states ‘Capturing’, ‘Editing’, or ‘Finished’.
Directory structure with a hierarchy of Processing Status > Target > file types
Anyone who has ever had a harddrive fail on them will understand the importance of backing up. Having had a 5-disk RAID unit more or less melt away, rendering completely useless, I’ve gone a bit paranoid on backup strategies. Of all my files, I will have at least two local copies and at least one cloud-based copy. Whatever your strategy is, now is a good time to back-up your images.
As for backing up, everything that happens on the main computer is constantly backed up to a cloud-based backup service, called Backblaze. For a modest price it offers unlimited backup of the computer plus any connected harddrive, a very compelling offer. And it always stays in the background, quietly doing its work without you ever noticing it. Bear in mind, this is a back-up solution, not a cloud-based fileserver like iCloud or Dropbox. So if you need your files back because of some sort of disaster, you can do this by them preparing a zip-file for you with your files for download (smaller volumes, free), or them preparing a USB-stick or harddrive that will be shipped to you (large volumes, at a cost).
The second backup is a local backup to a second set of harddrives that is kept in a separate place from the main computer. Every night, an incremental backup of the main computer is made automatically. The software used for this is CarbonCopyCloner.
The images have been renamed in a way that even months or years later you can easily see what they are, and they are safely stored, with multiple copies in different places. It is like good wine in a barrel, perfectly safe to store for long periods of time, and always available when you’re ready to process/consume them.
When processing is complete, all files will be moved to a cloud-based server. In my case that is Box. Since I have unlimited space available there, also all intermediate files, such as calibrated, registered, drizzled files etc go there. If you’re more space-restricted, you can decide to only put the capture files and a limited set of processing files (e.g. stacked image per filter, PixInsight project file, final output, etc.) in the cloud.
While it is tempting to start processing right away after capturing, I’ve found it worthwhile to spend this bit of extra time to file management and (re)naming. It helps a lot in finding your way through all the captures, and provides peace of mind that however long it’s been, you can quickly find back what you’re looking for.