Advanced dCore SCE Strategies

Although default SCE management behaviours are adequate for most use cases, dCore is feature rich and provides advanced strategies and options to fine tune extensions. The dCore SCE import system is flexible and provides significant control over the software utilized. The following strategies are not listed in any particular order or preference. A system may be set up to utilize none or most of these strategies, in any desired combination.

• Minimize redundant dependencies
• Improve extension load and boot time
• Reduce required drive space
• Manage default SCE actions
• Manage locales and languages
• Import and utilize man pages

As outlined in the dCore SCE Overview, SCE extensions (self contained extensions, *.sce) reside on a storage location, each comprised of a SquashFS that contain all the required packages and dependencies needed to run a desired application. Loading an SCE mounts and symlinks it into the root file system for use.

The size and complexity of SCEs varies significantly dependending on the underlying application and how the applicaton is imported. A larger SCE means a larger squash file and typically more dependencies. Loading a larger SCE, either at boot or OnDemand, requires more time and system resources to check integrity (md5sum), mount and symlink to the file system. In addition, larger and more complex SCEs require more bandwidth to download Debian files and take longer to update check and re-import. As loaded SCEs run from RAM, larger SCEs use more RAM. Limited RAM systems may, therefore, need to access a swap partition or swap file, hindering performance.

Many strategies minimize the drive space requirement of a dCore installation. As storage media is now plentiful and relatively inexpensive, these strategies primarily benefit limited/older hardware, embedded systems or where a small footprint and peak performance are priority. Some strategies also reduce the number of dependencies and duplicate dependencies between SCEs, improving overall SCE processing performance.

> Return to the dCore Wiki Welcome page

Lightweight software results in a smaller and more efficient SCE with less dependencies:

• Debian/Ubuntu repositories provide lighter alternatives for most any application.
• A CLI only system may be adequate for a server or command line music player.
• Xvesa can be trialed instead of Xorg, see dCore Graphic Servers and Drivers.
• A specific graphic driver can be used instead of xorg-all, see dCore Xorg-<specific> Driver Installation.
• Some Window Managers require few dependencies (FLWM, FLWM_topside, dwm), see dCore X Window System.
• The dCore pre-built repository contains light applications (Beaver, EmelFM, Fluff, XMMS), see the dCore sce-searchprebuilt Command.
• Choose software with shared dependencies, such as only CLI or ncurses, FLTK or legacy GTK applications.

Lightweight software example:

> Return to the dCore Wiki Welcome page

Numerous applications can be imported together via a list file to produce an SCE mega-extension, review the dCore sce-import Command for more information. Although self-contained extensions are robust and modular, one of Linux strengths is the ability to use shared common dependencies.

Even though importing several applications as a list file results in a larger overall extension, there will be no duplicate dependencies amongst the list file imported packages. This will reduce required drivespace but may also reduce overall import, load and update check time depending on the grouped applications.

The primary drawbacks of importing with a list file is that the entire list needs to be re-imported to add or remove applications and the entire mega-extension needs to be loaded even if the user only wants to access one software package. Results will vary depending on the packages imported and the number of shared dependencies between the applications.

For example, create a file arbitrarily named /tmp/office containing the following line separated entries:

efax-gtk
evince
galculator
libreoffice-gtk

Import the list to create office.sce and make the applications available in OnDemand:

sce-import -lo /tmp/office

When it's time for office work, select office from the OnDemand menu to load the applications.

From the above example:

Many dCore users prefer to import large base SCEs of frequently used packages via list files. As outlined above, this may significantly reduce dependency duplication on the system. Future software imports can then be made dependent on these large base SCE(s) to further leverage the shared dependencies, see the dependency SCE section below. Since all /etc/sysconfig/tcedir/sceboot.lst applications are loaded at boot, re-importing these applicatons as a large list file SCE rather than individual applications would likely be beneficial. Then simply replace all sceboot.lst items with the name of the newly created base SCE.

• Base > Desktop > Multimedia by netnomad
• Base by jls_legalize

When dealing with extensions that are defined by lists of packages, one may want to change the defining list. Changes can only take effect after re-importing the respective extension by issuing sce-import. That command exhibits a peculiar behavior with respect to lists that is well suited to this case.

If a file name is specified as argument to the -l option then it is used as list, naturally. There is a subtle change in behavior of the command when re-importing an extension that had been defined by means of a list. Originally, the user had supplied the list in form of a text file which got copied to the '.sce.lst'-file of the extension. Re-importing an extension with existing .sce.lst'-file considers that file as list, even if the -l option is not specified.

Thus there is no real need in keeping the original text files for defining lists. If need be, one can alter the definition in the corresponding '.sce.lst'-file and re-import the extension by name.

There is a potential catch if one had defined an extension through a list that bears the same name as an individual extension from the repositories. For instance, defining an extension named wireless through a list containing the “real” extensions wireless-<KERNEL>-tinycore and wireless makes for an ambiguous situation when issuing sce-import wireless. This ambiguity is resolved in favor of the user-defined list 'wireless.sce.lst'.

Dependency chains can also be added as desired, see the Dependency SCE(s) section below. For example:

• An independent SCE for suspending.
• A desktop environment SCE depending on all X11-related extensions, in turn dependent on one for locale settings.
• All desktop SCEs depend on the previous one, notably one for all things related to wireless networking.
• A web browser SCE dependent on the wireless networking SCE.

The first two sets may be loaded at boot-time through 'sceboot.lst'.

> Return to the dCore Wiki Welcome page

The sce-import -d (dependency) command can be used to select existing SCE(s) to provide dependencies for the new SCE. This may make the new SCE significantly smaller. Review the dCore sce-import Command for more information.

An example from a system using a base SCE imported via a list file, arbitrarily named bootlist, and loaded at boot. This list file SCE (bootlist.sce) consists of numerous applications, including xorg-intel, fluxbox, alsa, conky, dillo and emelfm. Obviously some of these applications will share common dependencies, which are already loaded during boot. If a user wanted to add a PDF viewer, mupdf for example, importing without using a dependency (eg. sce-import mupdf) results in an 11 MB SCE comprised of 16 packages with a total of 546 files. Importing mupdf with a dependency on the base SCE (bootlist.sce) via sce-import -d mupdf results in a 5 MB SCE comprised of only 3 packages and 18 files. The result is a much smaller SCE to store, md5sum check, mount, symlink and load, update check and re-import.

A couple of other SCE import examples using the dependency option:

> Return to the dCore Wiki Welcome page

There are two seemingly similar approaches for combining extensions. Extensions defined by a list (option -l of sce-import) bundle different packages into one single '.sce'-file. By defining dependencies among extensions (option -d of sce-import), on the other hand, one can enforce the loading of extensions beforehand. From a practical point of view, both techniques allow for loading multiple extensions at once.

Their different nature lies in the way that compilation dependencies are included. Combining different packages into one single list will most likely allow for some redundancy among imported packages. Nevertheless their “path” of compilation dependencies will usually reach back to libc; all of which will be included into the '.sce'-file. Technically speaking, the intersection of all compilation dependencies is included.

Making one extension depend on some other one as its base allows for trimming the path of compilation dependencies by the ones already included in the base extension. Only the difference of the package lists will be included.

Both techniques allow for reducing the number of packages included thereby helping to avoid multiple packages being present in various '.sce'-files at once. They serve different purposes, however, as the following typical examples show.

• The libreoffice package may be bundled into a list together with its associated packages for help and spell checking.
• One might naturally want to make any desktop application depend on a working desktop environment.

There is no real need to use either technique. As a rule of thumb, extension lists should be used for providing exteneded functionality by combining different packages into one single extension. The enchaining of dependencies, however, serves for hierarchically structuring extensions. – Simply put, extension lists allow for “horizontal” aggregation whereas extension chains allow for “vertical” ordering.

Self-created extensions, like for instance one for locales, should not be made dependencies of any other regular extension. Indeed, the sce-update command will choke upon such dependencies and fail. Furthermore there is no advantage in the sense that no packages can be saved from being imported multiple times.

> Return to the dCore Wiki Welcome page

Working in a desktop environment allows one to define different “profiles” of available extensions. For instance, one may want to combine LibreOffice with WiFi and a cloud synchronisatioon tool like OwnCloud, or one may prefer to work offline and have LibreOffice available on demand on its own. Since OnDemand menus entries are defined by files under 'tce/ondemand/' alone, it is easy to adapt them to one's wishes.

This technique is complementary to that of dependencies among extensions presented above. In fact, the extensions remain independent but their loading is done in a specific order automatically.

Find a preferred balance between sceboot.lst and OnDemand SCEs.

> Return to the dCore Wiki Welcome page

SCE(s) can be automatically purged during import prior to the squash process by utilizing custom data entered into an /etc/sysconfig/sce.purge file. The /etc/sysconfig/sce.purge-example file explains usage and provides a warning. Depending on the purge file utilized, the resulting SCE may be considerably smaller and contain significantly less files and directories. This allows the SCE to load faster during boot or runtime. The disadvantage is that it may cause extension breakage.

Contents of /etc/sysconfig/sce.purge-example:

# Enter files or directories to purge before squashing SCE during sce-import.
# Ensure there are no spaces before or after entries, leading slash not needed,
# wildcards accepted.
# 
# To activate move this file to /etc/sysconfig/sce.purge or create a new one.
#
# Add /opt/.filetool.lst entry to save any changes:
#   etc/sysconfig/sce.purge
#
# Warning: Purging files may result in smaller SCEs but break the extension.
#          If problems arise modify the sce.purge file, re-import and retest
#          the SCE. Although the sce-import -z command will ignore this purge
#          file when importing SCEs, running sce-update without the -z option
#          will by default use this purge file again during SCE re-imports.
#
# Example entries:
#
#usr/lib/i386-linux-gnu/libLLVM-3.5.so.1
#usr/lib/i386-linux-gnu/libLLVM*
#usr/share/locale
#usr/include

An example of a fairly aggressive /etc/sysconfig/sce.purge file:

# Enter files or directories to purge before squashing SCE during sce-import.
# Ensure there are no spaces before or after entries, leading slash not needed,
# wildcards accepted.
# 
# To activate move this file to /etc/sysconfig/sce.purge or create a new one.
#
# Add /opt/.filetool.lst entry to save any changes:
#   etc/sysconfig/sce.purge
#
# Warning: Purging files may result in smaller SCEs but break the extension.
#          If problems arise modify the sce.purge file, re-import and retest
#          the SCE. Although the sce-import -z command will ignore this purge
#          file when importing SCEs, running sce-update without the -z option
#          will by default use this purge file again during SCE re-imports.
#
# Example entries:
#
#usr/lib/i386-linux-gnu/libLLVM-3.5.so.1
#usr/lib/i386-linux-gnu/libLLVM*
#usr/share/locale
#usr/include
etc/alternatives
etc/apt
etc/cron.daily
etc/dpkg
etc/ld.so.conf.d
etc/logrotate.d
etc/rc*
lib/systemd
usr/bin/dpkg*
usr/bin/update-alternatives
usr/include
usr/lib/debug
usr/lib/gcc
usr/lib/i386-linux-gnu/libLLVM*
usr/lib/i386-linux-gnu/audit
usr/lib/i386-linux-gnu/gbm
usr/lib/i386-linux-gnu/gconv
usr/lib/i386-linux-gnu/perl
usr/lib/mime
usr/lib/pkgconfig
usr/lib/python*
usr/lib/xorg/modules/drivers/vesa*
usr/lib/xorg/modules/input/synaptics*
usr/lib/xorg/modules/input/vmmouse*
usr/lib/xorg/modules/multimedia
usr/sbin/dpkg*
usr/sbin/update-alternatives
### DO NOT PURGE xorg.conf.d DIRECTORY EVEN IF DON"T USE IT, WITH xorg-intel LOCKS-UP SYSTEM:
#usr/share/X11/xorg.conf.d
usr/share/X11/xorg.conf.d/50-synaptics.conf
usr/share/X11/xorg.conf.d/50-vmmouse.conf
### DO NOT PURGE, BREAKS ATERM CUT/PASTE:
#usr/share/X11/locale
### DO NOT PURGE, CONTAINS DESKTOP FILES AND BREAKS THINGS:
#usr/share/applications
usr/share/bug
usr/share/doc
usr/share/doc-base
usr/share/dpkg
usr/share/gcc*
usr/share/gdb
usr/share/info
usr/share/locale
usr/share/man
usr/share/perl
usr/share/perl5
usr/share/pkgconfig
usr/share/zoneinfo
var/cache
var/lib

In two use cases a similar sce.purge file resulted in an ~25-40% reduction in boot time:

http://forum.tinycorelinux.net/index.php/topic,19543.msg121157.html#msg121157

http://forum.tinycorelinux.net/index.php/topic,20413.msg127208.html#msg127208

> Return to the dCore Wiki Welcome page

The nomd5 boot code skips md5sum checking of boot time SCEs, slightly improving boot speed. Loading SCEs during runtime, using sce-load and OnDemand, will also be slightly faster. In a test system, using the nomd5 boot code resulted in a 6% faster boot time (measured from boot loader to boot screen). To determine approximate boot time improvement, run time /bb/md5sum SCE.sce against all sceboot.lst SCEs. The combined total of these timed md5sum tests will give the approximate boot time improvement.

By default SCE md5sum checks are performed during loading to ensure SCE integrity. As such, the nomd5 boot code should not be used if there are system security concerns, such as potential tampering within the SCE directory. For systems with high security priority, the SCE *.md5.txt files could be saved securely and periodically checked against the actual extension. The process could be automated using a relatively simple script. This backup and check process would need to be repeated whenever SCEs are updated.

> Return to the dCore Wiki Welcome page

* cf. relevant wiki pages

> Return to the dCore Wiki Welcome page

By default dCore does not keep man (manual) and documentation pages during the SCE import process. Although these documents only marginally increases SCE size, most of the time they are not required, alternatives:

• Most commands provide usage help, enter <command_name> –help (two dashes).
• Use an online man page resource, for example linux.die.net.
• Directly visit the softwares website or help forum.

To activate global man page imports for SCEs during the import process, change the /etc/sysconfig/sceonfig 'KEEPDOC=FALSE' entry to 'TRUE'. For SCE specific man pages, simply use the '-k' (keep) option during the import, example sce-import -k PKG. Run sce-import –help for more information.

> Return to the dCore Wiki Welcome page

The /etc/sysconfig/sceconfig file can be used to configure default behaviours when managing SCEs. Usage information is included in the file, default preferences outlined below. Regarding SCE management performance, the RAM option may import SCEs faster (quicker in/out access and reduce disk read/writes), provided adequate system RAM available. If PRESERVEDEBINXMODE is set to true, reduced bandwidth and time will be required to import extensions although the DEBINX (Debian Index) data may become stale. Setting a lower NICE value will make the sce-import process more aggressive, at the expense of simultaneous running processes.

# SCE configuration file
# Default actions for managing SCEs

# Add /opt/.filetool.lst entry to save changes:
#   etc/sysconfig/sceconfig

# For more information run:
#   sce-debpurge --help
#   sce-import --help
#   sce-remove --help
#   sce-update --help

# Change FALSE to TRUE for any option desired.

# Use KEEPDOC=TRUE to keep /usr/share/doc and /usr/share/man data
# in SCEs. Useful for the man command from the man-db package.
KEEPDOC=FALSE

# Use RAM=TRUE to unpack package contents in RAM when making
# an SCE. Particularly useful for flash devices.
RAM=FALSE

# Use SIZE=TRUE to display estimated RAM, HD space
# and download sizes when importing.
SIZE=FALSE

# Use KEEPLIST=TRUE to keep .lst and .dep files of SCEs removed
# by sce-remove. Useful for future re-importing.
KEEPLIST=FALSE

# Use NONINTERACTIVE=TRUE for non-interactive use of sce-import. If package
# name exists no further user interaction is needed for a complete import.
NONINTERACTIVE=FALSE

# Use CHECKALL=TRUE for a complete list of Debian packages
# that have part of the target name.
CHECKALL=FALSE

# USE VIEWPKGS=TRUE to show a list of packages that will be imported.
VIEWPKGS=FALSE

# Use PRESERVEDEBINXMODE=TRUE to not update the DEBINX package files.
PRESERVEDEBINXMODE=FALSE

# UPDATEDEBINXMODE is default and not needed to be specified here.

# Use SAFEBACKUP=TRUE to backup SCE(s) in /etc/sysconfig/tcedir/sce/backup/
# during re-import. Ensure adequate freespace, manually purge backups.
SAFEBACKUP=FALSE

# Use NICE=19 for the most cpu friendly use of sce-import and sce-update.  
# Use NICE=0 to give sce-import and sce-update high system priority.
NICE=19

# Use UIMMODULE=<plugin> to enable Universal Input Method plugins for uim-fep.
# The following example enables the SKK plugin:
# UIMMODULE=skk

> Return to the dCore Wiki Welcome page

As SCEs are self-contained the potential for dependency version conflicts between loaded SCEs exist. If different SCEs have a common dependency, a frequent occurrence, but were not imported or updated at the same time, one SCE may still be using an older dependency version while another has the updated version.

Although rare, this may cause extension breakage if two or more SCEs with out of sync dependency versions are loaded at the same time. More concerning is a potential security issue, as there is no guarantee loaded SCEs that use common dependencies are all using the most up to date security releases.

Strategies to avoid dependency version conflicts:

• Import all SCEs using list file(s) and/or dependency chains to minimize dependency duplication (discussed above).
• Periodically run sce-update -a (see dCore sce-update Command) to ensure all SCEs are updated at the same time.
• Periodically manually re-import all SCEs. The sce-update check process is CPU intensive and most SCEs will receive security updates during a longer time frame. If the SCEs were not update checked for some time, it may be quickest to periodically re-import all system SCEs manually, for example monthly. The re-import will ensure updated dependencies are utilized.

SCEs imported with advanced techniques, particularly using /etc/sysconfig/sce.purge, may break extensions. Prior to reporting extension problems on the forum, the problematic SCE and any dependency SCEs should be re-imported using the -z option (ignore locale.nopurge, sce.purge, sceconfig files), reboot and re-tested.

As an abundance of software is available from Debian/Ubuntu repositories, it is rarely necessary to create a personal extension. It is, however, occasionally desirable, possible reasons include:

• Developed own application.
• Using an application not from Debian/Ubuntu repositories.
• Prefer to custom compile software instead of using repository's version.
• Avoid cluttering the home directory with personal data, keep backup size small.

Creating a personal data or software SCE extension is straightforward. The dCore sce-import and dCore sce-deb2sce commands use mksquashfs to create the final squashfs (squash filesystem) SCE extension. This mksquashfs command is built-in and does not need to be imported separately, like Tiny Core's squashfs-tools.

Useful resources for creating extensions:

• Into the Core book in .pdf format (chapters 14 and 15): http://tinycorelinux.net/book.html
• Tiny Core wiki → Creating Extensions: http://wiki.tinycorelinux.net/wiki:creating_extensions

Regarding the Creating Extensions wiki link, most steps are not required as the extension will not be submitted to a repository. Instead of a Tiny Core '*.tcz' extension it's a dCore '*.sce' extension. An existing '*.tcz' extension can simply be renamed as an '*.sce'. The extension requires an *.md5.txt file in order for it to be processed using the dCore sce-load Command. To be fully utilized by the system, personal extensions can be placed in /etc/sysconfig/tcedir/sce/. Ensure the choosen SCE name does not match an existing Debian/Ubuntu package to avoid conflicts when updating the system. When creating personal software extensions, ensure required dependencies are also loaded.

Basic commands:

mksquashfs /tmp/my_application/ /tmp/my_application.sce
md5sum /tmp/my_application.sce > /tmp/my_application.sce.md5.txt
mv my_application.* /etc/sysconfig/tcedir/sce/
sce-load my_application 

Note there is no need to submit extensions to a repository. Recommend backing up personal extensions on other media for safekeeping. If the purpose of a personal extension is for a small executable, it can alternatively be placed in ~/.local/bin. If there is some essential software that may be useful for other dCore users, not available in the Debian/Ubuntu repositories, bring this to the attention of the dCore developer, forum member Jason W, for consideration of adding it to dCore's pre-built repository (see dCore sce-searchprebuilt Command).

> Return to the dCore Wiki Welcome page