SLUB SIP Builder

Description

A CLI tool to create valid BagIt based submission information packages (SIPs) for ingest into the SLUBArchiv, the digital long-term preservation archive of the Saxon State and University Library Dresden (SLUB).

As a user, you have one of three options to create your SIP: copy, move or replace.
NOTE: The use of option copy is strongly recommended. The remaining options modify your source directory. Please use these with caution!

Support

Supported Operating Systems

• Debian 10
• Windows 10

Supported Versions

• official release builds only

Installation (Releases)

###Debian 10

1. Update system packages (optional but recommended):
   sudo apt update && sudo apt upgrade -y
2. Install required tools and libraries:
   sudo apt install -y unzip perl libperl-dev liblwp-protocol-https-perl libfile-copy-recursive-perl libmarc-record-perl libxml-libxml-perl libxml-xpath-perl libdatetime-format-iso8601-perl cpanminus && sudo cpanm Archive::BagIt
3. Download the latest Release:
   → ZIP file like SLUB_SIP_Builder.version-XYZ-master.zip
4. Extract the ZIP archive and change into the project directory:
   unzip SLUB_SIP_Builder.version-XYZ-master.zip
cd SLUB_SIP_Builder
5. Run SLUB SIP Builder:
   → Try an example command from below
   perl bin/slubsipbuilderbagit.pl...

###Windows 10

1. Check UTF-8 flag at [ENG] Systemadministration/TimeAndRegion/Region/Administration/AreaPattern or [DE] Systemsteuerung/Zeit und Region/Region/Verwaltung/Gebietsschema ändern/, for using such characters as äöüÄÖÜß. Computer will be restarted!
2. Download strawberry perl environment strawberry-perl-5.30.3.1-64bit-portable
3. Extract files from archive
4. Download Archive::BagIt library (Example: Archive-BagIt-0.074.tar.gz)
5. Start strawberry perl console portableshell.bat
   1. Unpack archive tar zxf Archive-BagIt-0.074.tar.gz (alternative: 7-zip, if tar is not available)
   2. Change dir cd Archive-BagIt-0.074
   3. Generate Makefile perl Makefile.pl
   4. Run gmake, than gmake install
   5. Check if module installed cpanm Archive::BagIt. At this point Archive::BagIt module installed.
   6. Change dir back cd ..
6. Install additional libraries:
   cpanm MARC::Record XML::XPath DateTime::Format::ISO8601 Win32::Console::ANSI
7. Download the latest Release:
   → ZIP file like SLUB_SIP_Builder.version-XYZ-master.zip
8. Extract the ZIP archive and change into the project directory:
   cd SLUB_SIP_Builder
9. Run SLUB SIP Builder:
   → Try an example command from below
   perl bin\slubsipbuilderbagit.pl...

Development

This guide will help you run (unreleased) development versions provided in this repository.
Please note, only release builds are supported and will have version or release date information.

###Debian 10

1. Update system packages (optional but recommended):
   sudo apt update && sudo apt upgrade -y
2. Install required tools and libraries:
   sudo apt install -y build-essential git perl libperl-dev liblwp-protocol-https-perl libfile-copy-recursive-perl libmarc-record-perl libxml-libxml-perl libxml-xpath-perl libdatetime-format-iso8601-perl cpanminus
sudo cpanm Archive::BagIt
3. Clone SLUB SIP Builder repository:
   git clone https://git.slub-dresden.de/digital-preservation/SLUB_SIP_Builder.git
cd SLUB_SIP_Builder
   → Hint: For simple updates at a later date, just use git pull in directory SLUB_SIP_Builder to get the latest version.
4. Checkout the desired branch/tag/commit:
   → for example most current master
   git checkout master
5. Run unit tests (optional):
   perl t/slubsipbuilderbagit.t
6. Run SLUB SIP Builder:
   → Try an example command from below
   perl bin/slubsipbuilderbagit.pl...

Windows 10

1. Check UTF-8 flag at [ENG] Systemadministration/TimeAndRegion/Region/Administration/AreaPattern or [DE] Systemsteuerung/Zeit und Region/Region/Verwaltung/Gebietsschema ändern/, for using such characters as äöüÄÖÜß. Computer will be restarted!
2. Download strawberry perl environment strawberry-perl-5.30.3.1-64bit-portable
3. Extract files from archive
4. Download Archive::BagIt module (Example: Archive-BagIt-0.074.tar.gz)
5. Start strawberry perl console portableshell.bat
   1. Unpack archive tar zxf Archive-BagIt-0.074.tar.gz for example in the same dir where strawberryperl extracted
   2. Change dir cd Archive-BagIt-0.074
   3. Generate Makefile perl Makefile.pl
   4. Run gmake, than gmake install
   5. Check if module installed cpanm Archive::BagIt. At this point Archive::BagIt module installed.
   6. Change dir back cd ..
6. Install additional libraries:
   cpanm MARC::Record XML::XPath DateTime::Format::ISO8601 Win32::Console::ANSI
7. Clone SLUB SIP Builder repository:
   git clone https://git.slub-dresden.de/digital-preservation/SLUB_SIP_Builder.git
cd SLUB_SIP_Builder
   → Hint: For simple updates at a later date, just use git pull in directory SLUB_SIP_Builder to get the latest version.
8. Run unit tests (optional):
   perl t\slubsipbuilderbagit.t
9. Run SLUB SIP Builder:
   → Try an example command from below
   perl bin\slubsipbuilderbagit.pl...

Building Releases

Debian 10

1. Check external dependencies:
   → i.e. Appropriate rights schema version included?
2. Build ZIP archive with release version:
   cd SLUB_SIP_Builder
bash build_release.sh
   → creates a ZIP file likeSLUB_SIP_Builder.version-XYZ-master.zip
3. Create gitlab release
4. Add zip archive to gitlab release

Documentation

• The Project is based on the BagIt 1.0 specification, visit RFC8493 for more information.
• For valid SIP examples based on operating systems, please have a look at export_dir_kitodo/bagit/lin_* or export_dir_kitodo/bagit/win_*
• Specification for creating a SLUBArchiv.digital SIP SLUBArchiv_Produzenten_SIP_Spezifikation.pdf
• For valid SLUBArchiv.digital BagIt examples and faulty take a look at submission application project.
• SLUBArchiv.digital rights coding specification SLUBArchiv_Produzenten_Rechteauszeichnung.pdf

Recommendation

• Do not name your directories data
• marc21.xml and rights.xml file names are reserved in directory mets. All same file names will be renamed to 1.xml, 2.xml and so on
• Use absolute file and directory paths
• If you try to execute the program for the first time, than use save_option copy

Explanation

Usage

perl bin/slubsipbuilderbagit.pl -help

Result

Usage:
    slubsipbuilderbagit.pl [options]

    Options: -help brief help message -man full documentation

            -save_option=<option>           payload files save as <copy>, <move>, <replace>, the <copy> option should be prefered
            -IE_directory=<IE dir>          existing IE directory (absolute path!)
            -SIP_output_path=<target dir>   where to put the SIP dir (absolute path!)
            -ppn=<ppn>|-noppn=<noppn>       SWB-PPN or any identifier (uses minimalistic marc21)

            -SIP_version=<version>          mandatory, needs for identification of SIP format, supported <v2020.1>
            -external_workflow=<workflow>   mandatory, should be uniqe workflow name
            -external_id=<id>               mandatory, should be uniqe ID (i.e. a catalog ID), reuse an ID only when updating existing AIP
            -external_ISIL=<isil>           optional, ISIL number of library
            -external_conservation_flag     optional, if set no other "original" still exists
            -external_value_descr=<text>    mandatory, the reason why to archive
            -rights_version=<version>       mandatory, SLUB law managment specification, supported <1.0>

            -rights_xml=<file>              mandatory, path to SLUB rights xml file(absolute path!)
            -add_meta_file=<file>           optional, can be repeated, additional meta files(absolute path!)
            -add_key_value=<"key:value">    optional, can be repeated, additional key value pairs for bag-info.txt

Options:
    -help   Print a brief help message and exits.

Full documentation with examples

perl -I lib/ bin/slubsipbuilderbagit.pl -man

Result

• MUST: Write command in one line.

EXAMPLES
  Copy    (minimalistic)
    perl bin/slubsipbuilderbagit.pl --save_option=copy
    --IE_directory=/IE/git/SLUB_SIP_Builder/export_dir_kitodo/bagit/test2
    --SIP_output_path=/output_sips --ppn=457035137 --SIP_version=v2020.1
    --external_id=10008 --external_workflow=kitodo --external_ISIL=DE-14
    --external_value_descr="Gesetzlicher Auftrag" --rights_version=1.0
    --rights_xml=/metadata/rights/Fallbeispiel-02.xml

  Copy
    perl bin/slubsipbuilderbagit.pl --save_option=copy
    --IE_directory=/IE/git/SLUB_SIP_Builder/export_dir_kitodo/bagit/test2
    --SIP_output_path=/output_sips --ppn=457035137 --SIP_version=v2020.1
    --external_id=10008 --external_workflow=kitodo --external_ISIL=DE-14
    --external_value_descr="Gesetzlicher Auftrag" --rights_version=1.0
    --rights_xml=/metadata/rights/Fallbeispiel-01.xml
    --add_meta_file=/metadata/lido.xml --add_meta_file=/metadata/marc21.xml
    --add_key_value="Author:Lew Nikolajewitsch Tolstoi"
    --add_key_value="Titel:Krieg und Frieden" --add_key_value="Genre:Roman"

  Replace (minimalistic)
    NOT RECOMMENDED! IF FAILS, "IE" MUST BE MANUAL RESTORED TO THE PREVIOUS
    STATE!!!

    perl bin/slubsipbuilderbagit.pl --save_option=replace
    --IE_directory=/IE/SLUB_SIP_Builder/export_dir_kitodo/bagit/test2
    --ppn=457035137 --SIP_version=v2020.1 --external_id=10008
    --external_workflow=kitodo --external_ISIL=DE-14
    --external_value_descr="Gesetzlicher Auftrag" --rights_version=1.0
    --rights_xml=/metadata/rights/Fallbeispiel-01.xml

  Move    (minimalistic)
    NOT RECOMMENDED! IF FAILS, "IE" MUST BE MANUAL RESTORED TO THE PREVIOUS
    STATE!!!

    perl bin/slubsipbuilderbagit.pl --save_option=move
    --IE_directory=/IE/git/SLUB_SIP_Builder/export_dir_kitodo/bagit/test2
    --SIP_output_path=/output_sips --ppn=457035137 --SIP_version=v2020.1
    --external_id=10008 --external_workflow=kitodo --external_ISIL=DE-14
    --external_value_descr="Gesetzlicher Auftrag" --rights_version=1.0
    --rights_xml=/metadata/rights/Fallbeispiel-03.xml

Run tests

• Start script in project directory to run all tests properly.
• NOTE: while running tests path tmp/bagit will be created.

perl t/slubsipbuilderbagit.t

or

prove t