SquashFS PART 2

The SquashFS tools exposed

Using mksquashfs

mksquashfs is the tool for creating new squashed file systems, and for appending new data to existing squashed file systems. The general command-line format for mksquashfs is:

bash# mksquashfs source1 source2 ... destination [options]

• source1, source2, etc.: files and directories to be added to the resulting file system, given with relative and/or absolute paths
  
• destination: a regular file (filesystem image file), or a block device (such as /dev/fd0 or /dev/hda3) where you want to have your squashed file system
  

Notes for default mksquashfs behavior:

• When the new files are added to the new file system or appended to an existing one, mksquashfs will automatically rename files with duplicate names: if two or more files named text will appear in the same resulting directory, the second file will be renamed to text_1, third one to text_2 and so on.
  
• Duplicate files will be removed, so there will be only one physical instance (By the SquashFS 2.x, you can disable the detection/removal of the duplicates with the -no-duplicates option).
• If destination has a pre-existing SquashFS file system on it, by default, the new source items will be appended to the existing root directory. Examine the options table below to force mksquashfs to overwrite the whole destination and/or change the way new source items are added.
  
• If a single source file or directory is given, it becomes the root in a newly created file system. If two or more source files and/or directories are given, they will all become sub-items in the root of the new file system.
  
• The resulting filesystem will be padded to a multiple of 4 Kb: this is required for filesystems to be used on block devices. If you are very sure you don't ned this, use the -nopad option to disable this operation.
  

See the next section for more details about all possible options.

Command-line options

All possible options for mksquashfs are shown in the table below.

Table 1. Command-line options of the mksquashfs tool

Option	Description
-2.0	force mksquashfs version 2.1 to create a version 2.0 filesystem
-all-root or -root-owned	make all files in the target file system owned by root (UID=0, GID=0)
-always-use-fragments	divide all files greater than block size into fragments (by the version 2.x). It will result in greater compression ratios
-b [block size]	use [block size] filesystem block size (32 Kbytes default for 2.x, 128 kbytes for 3.x) - this can be either 4096, 8192, 16384, 32768, 65536 or 131072
-be or -le	force a big or little endian file system, respectively
-check-data	enable additional file system checks
-e [file1] ( [file2] ... )	specify which files and/or directories to omit from the new file system that is to be created
-ef [file]	specify a file which contains the list of files/directories to exclude
-force-gid [GID]	set all group IDs in target file system to [GID] (can be specified as a name or a number)
-force-uid [UID]	set all user IDs in target file system to [UID] (can be specified as a name or a number)
-info	print files, their original size and compression ratio, as they are added to the file system
-keep-as-directory	if the source is a single directory, force this directory to be a subdirectory of the root in the created file system
-noappend	if the destination file/device already contains a squashed file system, overwrite it, rather than append the new data to an existing file system
-no-duplicates	do not detect/remove duplicate file names
-noD or -noDataCompression	do not compress the data
-noF or -noFragmentCompression	do not compress the fragments (avaliable by 2.x)
-no-fragments	do not generate fragment blocks (avaliable by 2.x, this will produce almost the same filesystem as 1.x did)
-noI or -noInodeCompression	do not compress the inode table
-nopad	do not pad the resulting file system to a multiple of 4 KBytes
-root-becomes [name]	can be used while appending to a pre-existing squashed file system: it will make a new root, and [name] directory will contain all pre-existing files/directories
-version	print the version, copyright and license message
-recover [name]	recover filesystem data using recovery file [name] (3.3)
-no-recovery	don't create a recovery file (3.3).
-no-exports	don't make avaliable file system to export via NFS (3.x)
-no-sparse	don't check for sparse files (3.x)
-processors [number]	set the number of CPU to create file system. By default it will be used all avaliable processors (3.x)
--read-queue [size]	set input queue to [size] Mb. (Default is 64 Mb)(3.x)
-write-queue [size]	set output queue to [size] Mb (3.x)
-sort [sort_file]	sort files relating to priorities in [sort_file] (3.x)
-wildcards	enable the extended shell wildcards to exclude directories/files (to be used with -e)
-regex	enable to use POSIX regular expressions (3.3)

In most cases, you should leave all compression/block options by default, as they allow mksquashfs to achieve the best possible compression ratios.

Using unsquashfs

unsquashfs is the tool for extracting data from squashed file systems. The general command-line format for unsquashfs is:

unsquashfs [options] target [files/directories to extract]

• target is the squashed file system to extract.

Notes for unsquashfs behavior:

• By not specifying any destination path, unsquashfs extracts the compressed file system in the ./squashfs-root directory.
• The tool does not extract a squashed file system on already exsisting directory unless the -f option is specified.
• You can specify on the command line, a multiple number of files/directories to extract and the items to be extracted can be also be given in a file with -e [file] option.

All possible options for unsquashfs are shown in the table below.

Table 2. Command-line options of the unsquashfs tool

Option	Description
-v[ersion]	print the version, licence and copyright message
-i[nfo]	print the files as they are extracted from the file system
-l[ist]	list the squashed file system without extracting files
-li	list files with theyr attributes as they are unsquashed (3.3)
-ll	list the squashed file system files with attributes without any extraction (3.3)
-d[estination] path	specify a destination path for unsquashed items
-f[orce]	if files exist overwrite them
-s[tat]	display file system's superblock informations (it can discover the file system version and the options used to compress it) (3.3)
-e[f] [extract file]	list of directories or files to extract (entries given one per line) (3.3)
-r[egex]	treat extract names as POSIX regular expressions (3.3)

Note that by 3.x release you can extract 1.x and 2.x squashed file system too.

Creating and using squashed file systems

Basic steps

In order to create a squashed file system out of a single directory (say, /some/dir), and output it to a regular file (thus, producing a file system image), you need to say only one magic phrase:

bash# mksquashfs /some/dir dir.sqsh

mksquashfs will perform the squashing and print the resulting number of inodes and size of data written, as well as the average compression ratio. Now you have your /some/dir directory image in the dir.sqsh file. You can now use the mountcommand to mount it using a loopback device:

bash# mkdir /mnt/dir bash# mount dir.sqsh /mnt/dir -t squashfs -o loop

To check if you have what's expected:

bash# ls /mnt/dir

If you want to output the file system directly into a device (say, your floppy at /dev/fd0):

bash# mksquashfs /some/dir /dev/fd0

Then just mount the device:

bash# mount /dev/fd0 /mnt/floppy -t squashfs

And check if it's okay:

bash# ls /mnt/floppy

 Squashing file systems

Operations described here correspond to most cases where a read-only compressed file system can be used, whether you want it to be on a block device or in a file. This could be anything from large FTP/HTTP-served archives that don't change often, to having a squashed /usr partition and anything alike with these.

Example 1

Let's suppose you have a /var/arch directory with lots of files and that you want to turn it into a squashed file system and keep it on your root partition as a file (it will be a file system image that you will mount via a loopback device). The operations needed to perform are as follows.
Squash the directory, then mount it via loopback to test it:

bash# mksquashfs /var/arch /var/arch.sqsh bash# mkdir /mnt/tmp   bash# mount /var/arch.sqsh /mnt/tmp -t squashfs -o loop bash# ls /mnt/tmp

If everything is as expected, make this file system mount automatically at boot time by adding this line to your /etc/fstab:

/var/arch.sqsh /var/arch squashfs ro,defaults 0 0

Unmount the file system from the temporary mount point, and mount using it's fstab entry:

bash# umount /mnt/tmp bash# mount /var/arch

Now just ensure that everything works fine:

bash# ls /var/arch

Example 2

Say you have two hard disk partitions, /dev/hda6 (which is empty) and /dev/hda7 (which is bigger than /dev/hda6, mounted at /var/arch, contains some data and is full). Now, say you want to squash the /dev/hda7 file system and move it to/dev/hda6, then use /dev/hda7 for some other purposes. We will suppose you have the following line in /etc/fstab (reiserfs is just an example file system used on /dev/hda7):

/dev/hda7 /var/arch reiserfs defaults 0 0

In the same fashion as with the previous example:

bash# mksquashfs /var/arch /var/arch.sqsh bash# mkdir /mnt/tmp bash# mount /var/arch.sqsh /mnt/tmp -t squashfs -o loop bash# ls /mnt/tmp

If everything went fine, unmount /dev/hda7 (if needed) and use dd to copy /var/arch.sqsh to /dev/hda6:

bash# umount /dev/hda7 bash# dd if=/var/arch.sqsh of=/dev/hda6

Now change the line in /etc/fstab for /dev/hda7 to:

/dev/hda6 /var/arch squashfs ro,defaults 0 0

Mount the new file system and check to see if all went fine:

bash# mount /var/arch bash# ls /var/arch

Don't forget to erase the unneeded file system image:

bash# rm /var/arch.sqsh

Creating tiny/embedded systems

By saying "tiny/embedded", I mean Linux systems that are being built for booting from floppy disks, IDE/USB flash disks, iso9660 CD-ROMs, small-sized hard drives and the like. Whether you want to have your whole root file system on a single media (a single partition, a single floppy), or have a modular system (several floppies or disk partitions), the procedure is almost identical. Creating such Linux systems themselves is out of scope of this HOWTO - there are dedicated HOWTOs and guides for this (like the Bootdisk HOWTO and Linux From Scratch - visit www.tldp.org to retrieve these documents).

Squashed file systems on floppy/flash/hard disks

In order to use SquashFS for creating Linux systems on small disks, you just have to follow the usual steps for creating a minimal system, performing the following operations at respective points:

1. When developing a kernel for your system, make sure you enable SquashFS support so it can mount squashed file systems
2. Use mksquashfs for creating read-only initial ram disks and/or root and/or other file systems
3. Don't forget to set file system types to squashfs in /etc/fstab and/or the startup scripts of your system for mounting squashed file systems

Floppy example. Let's say you have your floppy system tree at /home/user/floppylinux and you want to place the root file system on one floppy and /usr on another. What you should do is:

bash# cd /home/user bash# mksquashfs floppylinux root.sqsh -e usr bash# mksquashfs floppylinux/usr usr.sqsh

Note 1: you can see here how we use the -e option to exclude the /usr directory for root file system's image.
Note 2: don't forget to specify squashfs in your root disk's /etc/fstab or startup scripts when mounting the /usr file system.
Insert a root disk in your 3.5" floppy drive (I assume you have a lilo or grub on it, and, thus, a file system exists on this floppy, and the root file system will reside under the /boot directory of this file system):

bash# mount /mnt/floppy bash# cp root.sqsh /mnt/floppy/boot

When done, unmount the root floppy, change the floppy to a /usr disk and use dd to transfer the usr file system:

bash# dd if=usr.sqsh of=/dev/fd0

Squashed file systems on CD-ROMs

With SquashFS, you can compress large file systems that will be used in live CDs (just as an example). For this purpose SquashFS is also used with UnionFS.

1. Enable SquashFS in the linux kernel of the target system
2. Create a squashed root file system
3. Modify the /etc/fstab or startup scripts of the target system to mount the squashd file system when you need it

If you create a root file system out of a running Linux system, use the -e option for mksquashfs to exclude all pseudo-filesystems such as /proc, /sys (on linux kernels after 2.5.x) and /dev (when using DevFS). Also, don't forget to add the file system image itself that is being created with mksquashfs (I think you know the reasons for these exclusions).

Making it writeble

As mentioned, another interesting use for SquashFS is with Unionfs filesystem, which provides copy-on-write semantics for the read-only file systems, enahancing the possibilities. (For unionfs you can look at http://www.filesystems.org/project-unionfs.html)
Just to make an example, you may want to make your /home/user squashed, to compress and backup your files without losing the possibility to apply changes or writing new files.
Create the ro.fs squashed file system and the rw.fs dir.

bash# mksquashfs /home/user1 ro.fs bash# mkdir /home/rw.fs

Mount the squashed ro.fs file system using the loopback device

bash# mount -t squashfs ro.fs /mnt -o loop

mount the unionfs file system, that makes /mnt and /home/rw.fs apparently merged under /home/user1 location.

bash# cd /home bash# mount -t unionfs -o dirs=rw.fs=rw:/mnt=ro unionfs user1

As you can see, now you can create new files in /home/user1.

bash# cd /home/user1 bash# touch file1 bash# ls

umount the unionfs and the squashfs file systems and list the content of /home/user1 and /home/rw.fs dir

bash# cd ..   bash# umount /home/user1   bash# umount /mnt   bash# ls /home/user1  bash# ls /home/rw.fs

You can see that the new file1 was created in /home/rw.fs
When you want to add the new created files to the stable and compressed squashed file system, you have to add them to the exsisting one.

bash# mksquashfs /home/rw.fs /home/ro.fs

Now, to mount your squashed user home directory at system startup, you can do as follow:
Make squashfs and unionfs modules loaded at boot time.

bash# echo squashfs >> /etc/modules bash# echo unionfs >> /etc/modules

Change the owner of the writeble branch to match user1.

chown user1 /home/rw.fs

Add these lines to /etc/fstab file to mount squashfs and unionfs at boot time.

... /home/ro.fs /mnt squashfs loop 0 0 unionfs /home/user1 unionfs dirs=/home/rw.fs=rw:/mnt=ro 0 0

How to create and extract zip, tar, tar.gz and tar.bz2 files in Linux

Data compression has been extremely useful to us over the years. Whether its a zip file containing images to be sent in a mail or a compressed data backup stored on a server, we use data compression to save valuable hard drive space or to make the downloading of files easier. There are compression formats out there which allow us to sometimes compress our data by 60% or more. I’ll run you through using some of these formats to compress and decompress files and directories on a Linux machine. We’ll cover the basic usage of zip, tar, tar.gz and the tar.bz2 formats. These are some of the most popular formats for compression used on Linux machines.

Before we delve into the usage of the formats I’d like to share some of my experience using the various formats of archiving. I’m talking about only a few data compression formats here, and there are many more out there. I’ve realized that I need two or three formats of compression that I’m comfortable using, and stick to them. The zip format is definitely one of them. This is because zip has become the de-facto standard choice for data compression, and it works on Windows as well. I use the zip format for files that I might need to share with Windows users. I like to use the tar.gz format for files that I would only use on my Mac and Linux machines.

ZIP

Zip is probably the most commonly used archiving format out there today. Its biggest advantage is the fact that it is available on all operating system platforms such as Linux, Windows, and Mac OS, and generally supported out of the box. The downside of the zip format is that it does not offer the best level of compression. Tar.gz and tar.bz2 are far superior in that respect. Let’s move on to usage now.

To compress a directory with zip do the following:

# zip -r archive_name.zip directory_to_compress

Here’s how you extract a zip archive:

# unzip archive_name.zip

TAR

Tar is a very commonly used archiving format on Linux systems. The advantage with tar is that it consumes very little time and CPU to compress files, but the compression isn’t very much either. Tar is probably the Linux/UNIX version of zip – quick and dirty. Here’s how you compress a directory:

# tar -cvf archive_name.tar directory_to_compress

And to extract the archive:

# tar -xvf archive_name.tar.gz

This will extract the files in the archive_name.tar archive in the current directory. Like with the tar format you can optionally extract the files to a different directory:

# tar -xvf archive_name.tar -C /tmp/extract_here/

TAR.GZ

This format is my weapon of choice for most compression. It gives very good compression while not utilizing too much of the CPU while it is compressing the data. To compress a directory use the following syntax:

# tar -zcvf archive_name.tar.gz directory_to_compress

To decompress an archive use the following syntax:

# tar -zxvf archive_name.tar.gz

This will extract the files in the archive_name.tar.gz archive in the current directory. Like with the tar format you can optionally extract the files to a different directory:

# tar -zxvf archive_name.tar.gz -C /tmp/extract_here/

TAR.BZ2

This format has the best level of compression among all of the formats I’ve mentioned here. But this comes at a cost – in time and in CPU. Here’s how you compress a directory using tar.bz2:

# tar -jcvf archive_name.tar.bz2 directory_to_compress

This will extract the files in the archive_name.tar.bz2 archive in the current directory. To extract the files to a different directory use:

# tar -jxvf archive_name.tar.bz2 -C /tmp/extract_here/

Data compression is very handy particularly for backups. So if you have a shell script that takes a backup of your files on a regular basis you should think about using one of the compression formats you learned about here to shrink your backup size.

Over time you will realize that there is a trade-off between the level of compression and the the time and CPU taken to compress. You will learn to judge where you need a quick but less effective compression, and when you need the compression to be of a high level and you can afford to wait a little while longer.

what is squashfs?

squashfs is a heavy-compression based read-only filesystem that is capable of compressing
2 to 3GB of data onto a 700 MB file. Have you ever thought of how Linux Live CDs work?
When a Live CD is booted it loads a complete Linux environment. Linux Live CDs make use
of a read-only compressed filesystem called squashfs. It keeps the root filesystem on a
compressed filesystem file. It can be loopback mounted and files can be accessed. Thus when
some files are required by processes, they are decompressed and loaded onto the RAM and
used. Knowledge of squashfs can be useful when building a custom live OS or when required
to keep files heavily compressed and to access them without entirely extracting the files.
For extracting a large compressed file, it will take a long time. However, if a file is loopback
mounted, it will be very fast since the required portion of the compressed files are only
decompressed when the request for files appear. In regular decompression, all the data is
decompressed first.

Show time

Some think in my mind about LAMP