bzip2 not found during "Compress install space"

[scpeters]

The Gazebo buildfarm has a few jenkins jobs configured from ros_buildfarm (documented here), and a recent build failed to execute the tar command during the Compress install space phase with an error stating that bzip2 is not found:

• https://build.osrfoundation.org/job/ci__colcon_any-manual_ubuntu_noble_amd64/60/

14:21:32 # BEGIN SECTION: Compress install space
14:21:32 + cd /home/jenkins/workspace/ci__colcon_any-manual_ubuntu_noble_amd64
14:21:32 + tar -cjf ros2-global-linux-noble-amd64-ci.tar.bz2 -C ws --transform s/^install_isolated/ros2-linux/ install_isolated
14:21:32 /bin/sh: 1: bzip2: not found
14:21:32 tar: ros2-global-linux-noble-amd64-ci.tar.bz2: Cannot write: Broken pipe

I don't know enough about ros_buildfarm to know where this dependency should be specified.

[tfoote]

This is the snippet that's failing:

ros_buildfarm/ros_buildfarm/templates/ci/ci_job.xml.em

Line 376 in 9cc156f

'tar -cjf ros%d-%s-linux-%s-%s-ci.tar.bz2' % (ros_version, rosdistro_name or 'global', os_code_name, arch) +

That's running on the host, which must have a different bootstrap image such that for the ROS installation has bzip preinstalled. I might suggest that this be something covered in https://github.com/ros-infrastructure/chef-ros-buildfarm I don't find it explicitly in a quick search. So maybe it's an implicit dependency of something on the ROS farm but not OSRF.

[scpeters]

suggestion by @cottsay on zulip:

It might be a small chunk of work, but we could also switch from using the tar and bz2 executables to using Python to create the archive. Both tar and bz2 are in the Python standard library.

This is the snippet that's failing:

note the whole command in that snippet:

ros_buildfarm/ros_buildfarm/templates/ci/ci_job.xml.em

Lines 376 to 379 in 9cc156f

	'tar -cjf ros%d-%s-linux-%s-%s-ci.tar.bz2' % (ros_version, rosdistro_name or 'global', os_code_name, arch) +
	' -C ws' +
	' --transform "s/^install_isolated/ros%d-linux/"' % (ros_version) +
	' install_isolated',

while it is pretty straightforward to replace tar -cjf output.tar.bz2 input_files with python3 -m tarfile -c output.tar.bz2 input_files, that snippet also uses the -C parameter to change directories before processing files and also the GNU tar option --transform. So yes, I think it would be a little bit of work to reproduce the -C and --transform functionality

[tfoote]

I would suggest that trying to do it on the command line exclusively will be a bit of a challenge, but if we turn it into a script it will likely be moderately easy.

Gemini gives me this which shows how to use the filter argument and arcname to do the -C and transform equivalents. I doubt this can drop in as is but with those two key elements this should be relatively easy to reproduce the command in a script.

import tarfile
import os

# Setup variables (placeholders for your variables)
ros_version = 2
rosdistro_name = "humble"
os_code_name = "jammy"
arch = "x86_64"

# 1. Format the archive name
archive_name = "ros%d-%s-linux-%s-%s-ci.tar.bz2" % (
    ros_version, 
    rosdistro_name or 'global', 
    os_code_name, 
    arch
)

# 2. Define the transformation logic
def rename_filter(tarinfo):
    # This mimics: --transform "s/^install_isolated/ros%d-linux/"
    old_prefix = "install_isolated"
    new_prefix = "ros%d-linux" % ros_version
    
    if tarinfo.name.startswith(old_prefix):
        tarinfo.name = tarinfo.name.replace(old_prefix, new_prefix, 1)
    
    return tarinfo

# 3. Create the archive
# "w:bz2" handles the -j (bzip2 compression)
with tarfile.open(archive_name, "w:bz2") as tar:
    # Change context to 'ws' and add 'install_isolated'
    # We use arcname to ensure the interior path starts with 'install_isolated'
    source_path = os.path.join("ws", "install_isolated")
    
    tar.add(
        source_path, 
        arcname="install_isolated", 
        filter=rename_filter
    )

print(f"Created: {archive_name}")

Feature	Shell Command (tar)	Python (tarfile)
Compression	-j	Open mode "w:bz2"
Working Dir	-C ws	Handled by os.path.join and arcname
Transformation	--transform "s/old/new/"	The filter callback function
Recursion	Automatic	Automatic (default behavior of .add())

[cottsay]

If we do transition to doing this in Python, we should try to compute the SHA256 as we write the compressed file out to generate the -CHECKSUM file to avoid re-reading the file later.

This isn't something I'm going to bite off right now, but I'm happy to review a PR.

[j-rivero]

Made an experiment here #1119 . No fully sure about if the code cover all the use cases but at least the result tar.bz2 looked fine to my eyes.