Handy script for keeping Fractal-Bot backups organized

iaresee

Administrator
Moderator
I point my Fractal-Bot(s) at the same drive for all backups. Now that Mr. Bot prefixes the backups with the unit they came from it's pretty trivial to run little scripts to sort and organize the backups into sub-directories automatically.

I'm on a Mac so ruby is there by default. And Ruby makes this kind of stuff nice and easy.

Here's the script:



I actually have this setup as a LaunchAgent entry and the OS fires it every time there's a change to the folder for me. I don't even have to think about it that way. I just always have a nice, neatly organized, backup area:

Screenshot 2023-02-05 at 1.01.39 PM.png

Add your methods for staying organized to this thread? What do Windows users do for this sort of thing? A PowerShell script?
 
I have one I wrote using Ruby or Python (two separate scripts that do the same thing, just ’cause) that:
  1. grab all the *.syx files in the directory.
  2. group them by the device and time stamp.
  3. determines if they constitute a full or partial backup.
  4. pass the group iteratively to zip to archive them, and add “partial” to the name if it’s a partial backup. zip uses verbose output so I can see the progress.
  5. touch them with the archive’s datetime stamp so the OS can sort and group them appropriately.
  • Unzipped files are much bigger than a zipped archive: ~23 MB uncompressed vs. ~2.0 MB zipped.
  • Various command-line flags are available.
I’ll add them in a few minutes.


There are some constants to adjust for people's individual systems, such as BACKUP_DIR in both the Ruby and Python scripts, or use the command-line flag, and ZIP and TOUCH in the Python version. And, of course -h or --help are your friends.

Ruby:
#!/usr/bin/env ruby

#
# This works for me. It probably won't work for you, but if it does, cool.
#

require 'debug' if $DEBUG
require 'optparse'

BACKUP_DIR = '/Volumes/8SI/Dropbox/Modelers/Fractal Audio/Backups'.freeze

VERBOSE_OR_NOT = false

# The array elements should be in UPPERCASE…
IIIMk2_FILES = %w[
  BANK-A BANK-B BANK-C BANK-D BANK-E BANK-F BANK-G BANK-H
  SYSTEM+GB+FC
  USER-CAB-BANK-1 USER-CAB-BANK-2
  USER-CAB-BANK-FR
].freeze

FM9_FILES = %w[
  BANK-A BANK-B BANK-C BANK-D
  SYSTEM+GB+FC
  USER-CAB-BANK-1
].freeze

FM3_FILES = %w[
  BANK-A BANK-B BANK-C BANK-D
  SYSTEM+GB+FC
  USER-CAB-BANK-1
].freeze

# Strip the extension from filenames. Fold to UPPERCASE…
def get_backup_filetypes(files)
  files.map { |s|
    File.basename(s, File.extname(s)).split('-', 4).last.upcase   #=> "USER-CAB-BANK-1"
  }
end

# Is the list of files being backed up a full or partial backup?
# Compare using UPPERCASE.
def is_partial_backup?(files)
  device = files.first.split('-').first

  # Test against uppercase names to be more foolproof.
  # Report the actual case for clarity.
  required_files = case device.upcase
                   when 'FM3'
                     FM3_FILES
                   when 'FM9'
                     FM9_FILES
                   when 'IIIMK2'
                     IIIMk2_FILES
                   else
                     abort %Q["#{ device }" is an unknown device type.]
                   end

  backup_filetypes = get_backup_filetypes(files)
  ((required_files - backup_filetypes) + (backup_filetypes - required_files)).any?
end

# -----------------
# FIRE IN THE HOLE!
# -----------------

# Grab any command-line options…
options = {
  dir: BACKUP_DIR,
  verbose: VERBOSE_OR_NOT
}
OptionParser.new do |parser|
  parser.banner = "Usage: #{ File.basename($0) } [options]"

  parser.on('-d', '--dir BACKUP_DIRECTORY', 'The path to the directory containing backup .syx files.') do |v|
    options[:dir] = v
  end

  parser.on('-t', '--[no-]testrun', 'Testrun only.') do |v|
    options[:testrun] = v
  end

  parser.on('-v', '--[no-]verbose', 'Run verbosely.') do |v|
    options[:verbose] = v
  end

  parser.on('-h', '--help', 'Prints this help.') do
    puts parser
    exit
  end
end.parse!

working_dir = Dir.getwd
puts "chdir '#{ options[:dir] }'" if options[:verbose]
Dir.chdir(options[:dir])

# Get the *.syx files
syx_files = Dir['*.syx'].sort

if syx_files.empty?
  puts 'No .syx files exist. Back something up!'
else
 
  # Get the unique device_yyyymmdd_hhmmss stamps as a hash of the device+datestamp+timestamp => [files that match]
  uniq_backups = syx_files.group_by { |f| f[f.split('-')[0..2].join('-')] }

binding.break if $DEBUG

  uniq_backups.each do |k, v|
 
    # We know what files to expect from a full backup. If we don't have them
    # all then label it with "partial".
    zipfile = k
    zipfile += '-partial' if is_partial_backup?(v)
    zipfile += '.zip'

    yymmdd, hhmmss = k.split('-')[1..2]
    hhmm = hhmmss[0..3]
    ss = hhmmss[-2..-1]

    # Do things…
    [
      "zip -m #{ zipfile } #{ k }*.syx",
      %Q[touch -ct "#{ yymmdd }#{ hhmm }.#{ ss }" #{ zipfile }]
    ].each do |cmd|
      puts cmd if options.values_at(:verbose, :testrun).any?
      system(cmd) unless options[:testrun]
    end

  end
 
end

puts "chdir '#{ working_dir }'" if options[:verbose]
Dir.chdir(working_dir)
Python:
#!/usr/bin/env python

#
# This works for me. It probably won't work for you, but if it does, cool.
#

import argparse
import glob
import os
from pathlib import Path
import subprocess
import sys

#
# set up the defaults…
#
BACKUP_DIR = '/Volumes/8SI/Dropbox/Modelers/Fractal Audio/Backups'
ZIP = '/usr/bin/zip'
TOUCH = '/usr/bin/touch'

VERBOSE_OR_NOT = False

#
# The array elements should be in UPPERCASE…
#
IIIMk2_FILES = sorted([
    'BANK-A', 'BANK-B', 'BANK-C', 'BANK-D', 'BANK-E', 'BANK-F', 'BANK-G', 'BANK-H',
    'SYSTEM+GB+FC',
    'USER-CAB-BANK-1', 'USER-CAB-BANK-2',
    'USER-CAB-BANK-FR'
])

FM9_FILES = sorted([
    'BANK-A', 'BANK-B', 'BANK-C', 'BANK-D',
    'SYSTEM+GB+FC',
    'USER-CAB-BANK-1'
])

FM3_FILES = sorted([
    'BANK-A', 'BANK-B', 'BANK-C', 'BANK-D',
    'SYSTEM+GB+FC',
    'USER-CAB-BANK-1'
])


#
# Strip the extension from filenames. Fold to UPPERCASE…
#
def get_backup_filetypes(files):
    return [Path(f).stem.split('-', 3)[-1].upper() for f in files]


#
# Is the list of files being backed up a full or partial backup?
# Compare using UPPERCASE.
#
def is_partial_backup(files):
    device = files[0].split('-')[0]

    match device.upper():
        case 'FM3':
            required_files = FM3_FILES
        case 'FM9':
            required_files = FM9_FILES
        case 'IIIMK2':
            required_files = IIIMk2_FILES
        case _:
            sys.exit(f"{device} is an unknown device type.")

    return sorted(get_backup_filetypes(files)) != required_files


# -----------------
# FIRE IN THE HOLE!
# -----------------

def main() -> int:
    #
    # Grab any command-line options…
    #
    parser = argparse.ArgumentParser()
    parser.add_argument('-d', '--dir', help='the path to the directory containing backup .syx files',
                        default=BACKUP_DIR)
    parser.add_argument('-t', '--testrun',
                        help='testrun only', action='store_true')
    parser.add_argument('-v', '--verbose', help='run verbosely',
                        action='store_true', default=VERBOSE_OR_NOT)
    args = parser.parse_args()

    working_dir = os.getcwd()
    if args.verbose:
        print(f"chdir {args.dir}")

    os.chdir(args.dir)

    #
    # Get the *.syx files
    #
    syx_files = sorted(glob.glob('*.syx'))

    if len(syx_files) == 0:
        if args.verbose:
            print(f'chdir {working_dir}')

        os.chdir(working_dir)
        sys.exit('No .syx files exist. Back something up!')

    #
    # Get the unique device_yyyymmdd_hhmmss stamps as a hash of
    # the device+datestamp+timestamp => [files that match]
    #
    uniq_backups = {}
    for sf in syx_files:
        backup_device = '-'.join(sf.split('-')[0:3])

        if backup_device not in uniq_backups:
            uniq_backups[backup_device] = []

        uniq_backups[backup_device].append(sf)

    #
    # We know what files to expect from a full backup. If we don't have them
    # all then label it with "partial".
    #
    for k, v in uniq_backups.items():

        zipfile = k

        if is_partial_backup(v):
            zipfile += '-partial'

        zipfile += '.zip'

        yymmdd, hhmmss = k.split('-')[1:3]
        hhmm = hhmmss[0:4]
        ss = hhmmss[-2:]

        # Do things…
        for cmd in [
            f"{ZIP} -m {zipfile} {' '.join(v)}",
            f"{TOUCH} -ct \"{yymmdd}{hhmm}.{ss}\" {zipfile}"
        ]:
            if args.verbose or args.testrun:
                print(cmd)

            if not args.testrun:
                subprocess.run(cmd, shell=True, check=True)

    if args.verbose:
        print(f"chdir {working_dir}")

    os.chdir(working_dir)
    return 0


if __name__ == '__main__':
    sys.exit(main())

On macOS, when a directory is set to display in columns, grouping by "Date Created" you'll get a listing like…
Screenshot 2023-02-05 at 1.55.31 PM.png
The entries that are locked and tagged are how I usually denote an archive that occurred immediately prior to a firmware update in case I need to jump back to it quickly. I lock them so I don't easily delete them.
 
Last edited:
Long time programmer here, but I never felt a need for something like this, maybe never I only have an Axe III.

My brackups are structured in folders by firmware version then date, like I think most people do. When I'm going to back up, I just create one or both levels of folders as needed, and point FractalBot there, done.

No automation needed.
 
Backups?


Macaulay Culkin What GIF by filmeditor
 
Long time programmer here, but I never felt a need for something like this, maybe never I only have an Axe III.

My brackups are structured in folders by firmware version then date, like I think most people do. When I'm going to back up, I just create one or both levels of folders as needed, and point FractalBot there, done.

No automation needed.
I think there's automation occurring, it's just happening on the biological side of the process.
 
I'd like to see the option to add the firmware version to the emitted files and/or be able to create the directories automatically based on the device ID and firmware version.
I back up to hierarchical directories by version then date. It would save some time and potential for error if that happened automatically.

+1.
 
Back
Top Bottom