GitLab

* NEWS: Record release date.

* build-aux/gen-single-binary.sh: Avoid bash only constructs.
Reported by Assaf Gordon

* tests/misc/env.sh: Skip if we can't execute the generated shebang,
which would be the case if there are spaces in the directory hierarchy.
This is triggered by `make distcheck`

This issue was identified by the manifest comparisons
done by `make distcheck`

* src/local.mk (noinst_HEADERS): Remove coreutils.h from this always
distributed list.
(nodist_src_coreutils_SOURCES): Add coreutils.h as its contents
are determined at configure time, so pointless to distribute.
(src_coreutils_SOURCES): Define explicitly so that the corresponding
nodist_ variable is honored.
(DISTCLEANFILES): Add coreutils.h to this rather than CLEANFILES,
as its contents are determined at configure time.

With libselinux-2.2.1-6.fc20.x86_64, kernel-3.12.6-300.fc20.x86_64
`cp --preserve=context src dst` was seen to succeed when src and
dst where on the same fixed context file system, as lsetfilecon()
returned success in this case when the context wasn't being changed.

* tests/cp/cp-a-selinux.sh: Copy from a different file system to
most likely have a different context that will test context
setting logic correctly.

* cfg.mk: Avoid warning for @AMDEP_TRUE@ usage in man/local.mk

This fixes a problem with native Solaris 'make', which does not
grok '-include' lines (a GNU extension to POSIX 'make').
* configure.ac (man/dynamic-deps.mk): Create it, with an old
time stamp, if doing dynamic dependency tracking.
* man/local.mk (DISTCLEANFILES): Put man/dynamic-deps.mk here,
rather than in CLEANFILES.
(man/dynamic-deps.mk): Don't create it read-only, so that we
can easily touch it later.
Include it with '@AMDEP_TRUE@@am__include@', not with '-include',
as '-include' does not work with native Solaris 'make'.

* src/numfmt.c (simple_strtod_int): Replace isdigit() with c_isdigit()
to avoid locale concerns and -Wchar-subscripts warnings on cygwin.
Remove the now redundant locale guard.
(simple_strtod_human): Cast characters to unsigned so that the promoted
int value passed to isblank() is positive, allowing it to work correctly
for all characters in unibyte locales.  Previously character 0xA0,
i.e. non-breaking space, would be misclassified for example.
(process_suffixed_number): Likewise.
(skip_fields): Likewise.
Both issues were triggered by the -Wchar-subscripts warning on GCC 4.8.3
on cygwin, due to the is*() implementations used there, but the issue
is present on all platforms defaulting to signed chars.
* NEWS: Mention the bug fix.

Reported by Eric Blake

* tests/misc/od-endian.sh: '\n' is not generally supported
in the replacement, so use the more portable "\\$NL".

* tests/dd/ascii.sh: Quote so that '\\' is passed
to printf rather than a single '\', as that's not portable.
bash, dash, zsh and external solaris printf were seen to support
  printf '\%03o' 1 2
while ksh, freebsh shell and external GNU printf need the more portable
  printf '\\%03o' 1 2
Note we don't use env here to call the coreutils printf implementation,
as there are many printf calls, so relying on portable shell
implementations will be faster.

* tests/split/b-chunk.sh: Never show the rm prompt which would
hang the test suite.  This was seen when split erroneously
created files with no persmissions, which was triggered by
this compiler bug in clang 3.4:
http://llvm.org/bugs/show_bug.cgi?id=18346

* src/chroot.c: Explicitly cast int to pointer type.

* gnulib: Sync recent cleanups and the fix for
missing df entries in the presence of bind mounts:
http://bugs.gnu.org/17833
* NEWS: Detail the df bug fix.

This didn't seem to cause any invalid operation on GNU/Linux at least,
but depending on the implementation, mutex deadlocks could occur.
For example this might be the cause of lockups seen on Solaris:
http://lists.gnu.org/archive/html/coreutils/2013-03/msg00048.html

This was identified with valgrind 3.9.0 with this setup:

  seq 200000 > file.sort
  valgrind --tool=drd src/sort file.sort -o file.sort

With that, valgrind would _intermittently_ report the following:

 Destroying locked mutex: mutex 0x5419548, recursion count 1, owner 2.
    at 0x4C2E3F0: pthread_mutex_destroy(in vgpreload_drd-amd64-linux.so)
    by 0x409FA2: sortlines (sort.c:3649)
    by 0x409E26: sortlines (sort.c:3621)
    by 0x40AA9E: sort (sort.c:3955)
    by 0x40C5D9: main (sort.c:4739)
 mutex 0x5419548 was first observed at:
    at 0x4C2DE82: pthread_mutex_init(in vgpreload_drd-amd64-linux.so)
    by 0x409266: init_node (sort.c:3276)
    by 0x4092F4: init_node (sort.c:3286)
    by 0x4090DD: merge_tree_init (sort.c:3234)
    by 0x40AA5A: sort (sort.c:3951)
    by 0x40C5D9: main (sort.c:4739)

 Thread 2:
 The object at address 0x5419548 is not a mutex.
    at 0x4C2F4A4: pthread_mutex_unlock(in vgpreload_drd-amd64-linux.so)
    by 0x4093CA: unlock_node (sort.c:3323)
    by 0x409C85: merge_loop (sort.c:3531)
    by 0x409F8F: sortlines (sort.c:3644)
    by 0x409CE3: sortlines_thread (sort.c:3574)
    by 0x4E44F32: start_thread (in /usr/lib64/libpthread-2.18.so)
    by 0x514EEAC: clone (in /usr/lib64/libc-2.18.so)

* src/sort.c (sortlines): Move pthread_mutex_destroy() out to
merge_tree_destroy(), so that we don't overlap mutex destruction
with threads still operating on the nodes.
(sort): Call the destructors only with "lint" defined, as the
memory used will be deallocated implicitly at process end.
* NEWS: Mention the bug fix.

Neither issue impacts on the correct operation of sort.
The issues were detected by both valgrind 3.8.1 and 3.9.0 using:

  seq 200000 > file.sort
  valgrind --tool=drd src/sort file.sort -o file.sort

For tool usage and error details see:
 http://valgrind.org/docs/manual/drd-manual.html

* src/sort.c (queue_insert): Unlock mutex _after_ signalling the
associated condition variable.  Valgrind flags this with:
  "Probably a race condition: condition variable 0xffeffffb0 has been
   signaled but the associated mutex 0xffeffff88 is not locked by the
   signalling thread."
The explanation at the above URL is:
  "Sending a signal to a condition variable while no lock is held on
   the mutex associated with the condition variable.  This is a common
   programming error which can cause subtle race conditions and
   unpredictable behavior."
This should at least give more defined scheduling behavior.

(merge_tree_destroy): Make symmetrical with merge_tree_init() thus
destroying the correct mutex.  Valgrind flags this with:
  "The object at address 0x5476cf8 is not a mutex."

* src/df.c (get_disk): Avoid an inconsequential mem leak
spotted by coverity.  Also s/duplicities/duplicates/.

* man/local.mk (man/dynamic-deps.mk): Use the same code to
derive FOO from man/FOO.1 as in the .x.1 rule below.
Using the more concise "name=$${man:4: -2}" is not portable enough.

Add the --enable-single-binary option to the configure file.
When enabled, this option builds a single binary file containing
the selected tools.  Which tool gets executed depends on the value
of argv[0] which can be set implicitly through symlinks to the
single program.

This setup reduces significantly the size of a complete coreutils
install, since code from lib/libcoreutils.a is not duplicated in
every one of the more than 100 binaries.  Runtime overhead is
increased due to more dynamic libraries being loaded, and extra
initialization being performed for all utils.  Also initially
a larger binary is loaded from storage, though this is usually
alleviated due to caching and lazy mmaping of unused blocks,
and in fact the single binary should have better caching
characteristics.

Comparing the size of the individual versus single binary on x86_64:
  $ cd src
  $ size coreutils
  $ size -t $(../build-aux/gen-lists-of-programs.sh --list-progs |
              grep -Ev '(coreutils|libstdbuf)') | tail -n1
     text    data     bss     dec     hex filename
  1097416    5388   88432 1191236  122d44 src/coreutils
  4901010  124964  163768 5189742  4f306e (TOTALS)

Storage requirements are reduced similarly:
  $ cd src
  $ du -h coreutils
  $ du -ch $(../build-aux/gen-lists-of-programs.sh --list-progs |
             grep -Ev '(coreutils|libstdbuf)') | tail -n1
  1.2M    coreutils
  5.3M    total

When installing, the makefile will create either symlinks or
shebangs based on the --enable-single-binary setting, for
each configured tool. In this way, all the tools are still
callable individually, but they are all implemented by the same
"coreutils" binary installed on the same directory.

* .gitignore: Add new generated files.
* Makefile.am: New rules to generate build-aux/gen-single-binary.sh
  and install symlinks.
* NEWS: Mention the new feature.
* README: Add "coreutils" to the list of utils.
* bootstrap.conf: Regenerate src/single-binary.mk
* build-aux/gen-lists-of-programs.sh: New --list-progs option.
* build-aux/gen-single-binary.sh: Regenerate
* configure.ac: New --enable-single-binary option and other variables.
Disallow --enable-single-binary=symlinks with --program-prefix et. al.
* man/coreutils.x: Manpage hook.
* man/local.mk: Add manpage hook and fix dependencies.
* src/coreutils.c: Multicall implementation.
* src/local.mk: New rules for the single binary option.
* tests/local.mk: Add $single_binary_progs to support
require_built_() from init.cfg
* tests/misc/env.sh: Avoid the use of symlink to echo.
* tests/misc/help-version.sh: Add exception for coreutils.
* tests/install/basic-1.sh: Really avoid using ginstall strip
functionality if there is an issue with the independent strip command.
* src/kill.c: Changes to call exit() in main.
* src/readlink.c: Likewise.
* src/shuf.c: Likewise.
* src/timeout.c: Likewise.
* src/truncate.c: Likewise.

Avoid complicated and error-prone parsing of df's output via
sed(1), cut(1), etc., and instead use df's more modern --output
option.

* src/ioblksize.h (in a comment): Simplify the extraction of the
device name of the mounted file system from df's output.
* tests/dd/skip-seek-past-dev.sh: Likewise.
* tests/du/2g.sh: Likewise for the 'avail' column here.
Also avoid the deprecated use of "tail -NUM".
* tests/misc/stat-mount.sh: While at it, remove the determination
of the mount point of "." via df(1) plus sed(1) as it is unused
since commit v8.5-159-gf57cb371 anyway.  Instead, improve this test
by verifying that the output of "stat -c%m ." at least starts with
a slash '/'.

Revert commit v8.22-131-g3e89d5bb as even though POSIX
states that the default mode should be -L,
common practice for stand-alone pwd implementations
is to default to -P.

* src/pwd.c (usage): Retain mention of the default mode of operation.

Suggested by Bob Proulx

* tests/ls/readdir-mountpoint-inode.sh: Quote appropriately
to process mount points with spaces in the path.
Previously items like these would usually be skipped,
though if the path also contained a '-' for example,
that would stat stdin, thus producing a wrong inode
and a false failure.
Fixes http://bugs.gnu.org/17863

* tests/df/no-mtab-status.sh: Skip if getmntent() not available.
* tests/df/skip-duplicates.sh: Likewise.
Fixes http://bugs.gnu.org/17863

This works for any program that might also be a shell built-in.
It is more concise.  In addition, it makes output more reproducible:
some diagnostics include argv[0], which will now be just the program
name, rather than the full absolute name of the executable.
* tests/misc/sort-compress.sh: Use env kill, rather than absolute name.
* tests/install/trap.sh: Likewise.
* tests/misc/timeout.sh: Likewise.
* tests/touch/no-dereference.sh: Do the same for two uses of "test".
* tests/touch/no-create-missing.sh: Likewise.

On systems without libselinux, context_t is defined as an int type,
but defined as a pointer type by SELinux.

* src/chcon.c: Init with 0 rather than NULL to avoid warning.

* tests/misc/pwd-option.sh (base): Initialize with -P,
now that -L is the default, to accommodate an initial
working directory with a symlink component.
* tests/misc/readlink-fp-loop.sh: Use $(env pwd -P) to get the
absolute working directory.  Using "env" ensures we do not invoke
any shell built-in, and PATH ensures we invoke the one from coreutils.
* tests/readlink/can-e.sh: Likewise.
* tests/readlink/can-f.sh: Likewise.
* tests/readlink/can-m.sh: Likewise.

The security_context_t type was always an artificial separation
from a standard char* string, and various libselinux using code
assumed both were synonymous.  In addition, prior to libselinux 2.3
function declarations were incorrect wrt constness of this type.
Here we replace security_context_t with char*, and also
provide a wrapper function to cater for the const issue on
older libselinux.

* src/system.h (se_const): A new function to avoid and identify
the const issue on older libselinux.
* src/copy.c: s/security_context_t/char */.
* src/cp.c: Likewise.
* src/id.c: Likewise.
* src/install.c: Likewise.
* src/ls.c: Likewise.
* src/mkdir.c: Likewise.
* src/mkfifo.c: Likewise.
* src/mknod.c: Likewise.
* src/runcon.c: Likewise.
* src/selinux.c: Likewise.
* tests/cp/no-ctx.sh: Likewise.
* src/chcon.c: Likesize.

Since context is verified by security_check_context() it can be used in
change_file_context() without converting to context_t every time.

* src/chcon.c (change_file_context): Use specified_context directly.

context_new() and _free() are used for checking validity of a
specified context.  libselinux provides security_check_context
for this purpose so use it.

Note that context_new() can fail for a valid context - e.g. ENOMEM.

* src/chcon.c (main): Use security_check_context().

* src/pwd.c (main): Adjust default mode to be "logical"
and independent of the POSIXLY_CORRECT env var.
(usage): Mention the default mode of operation.
* doc/coreutils.texi (pwd invocation): Adjust accordingly.
* tests/misc/pwd-option.sh: Likewise.
* NEWS: Mention the change in behavior.

* HACKING: GID is more useful in tests than group name, so rename
input param from NON_ROOT_GROUP to NON_ROOT_GID to make it obvious
that only a group ID is now acceptable, thus allowing GID lookups
to be avoided throughout the tests.
* init.cfg (require_root_): Likewise.
* tests/misc/truncate-owned-by-other.sh: Avoid looking up the GID.
* tests/touch/now-owned-by-other.sh: Likewise.
* tests/misc/chroot-credentials.sh: Likewise.  Also fix an instance
of comparison against NON_ROOT_GROUP which would have given a false
failure if a non numeric value was passed in.
* tests/id/setgid.sh: Use previously looked up gid as a more
accurate base for the subsequent adjustment, and move
the uid lookup within chroot, rather than having the overhead
of a separate `id` invocation.

* NEWS: Mention bug introduced in v8.1 (with commit v8.0-88-g8ba5d1a7).

* src/id.c (print_full_info): When no user is specified,
output the effective group for the _process_, rather than
the default group from the system database, which may be different.
* tests/id/setgid.sh: Add a case for `id` as well as `id -G`.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/7320
Reported at http://bugzilla.redhat.com/1016163

* src/df.c (last_device_for_mount): A new function to identify
the last device mounted for a mount point.
(get_disk): Use the above to discard mount entries for a device,
where a later mount entry uses a different device name than
that of the user specified device.
* tests/df/over-mount-device.sh: A new root test.
* tests/local.mk: Reference the new test.
* NEWS: Reword for all these related recent fixes.
Discussed at: http://bugs.gnu.org/16539#69

* src/df.c (get_disk): Include whether we can access the mount dir,
in the mount entry selection criteria.  This handles the case where
a device is (bind) mounted multiple times with the shortest mount path
not being accessible, while some of the other mount points are.
Discussed at: http://bugs.gnu.org/16539#63

A system provided mount entry may be unavailable due to TOCTOU race,
or if another device has been over-mounted at that position, or due to
access permissions.  In all these cases output "-" placeholder values
rather than either producing an error, or in the over-mount case
outputting values for the wrong device.

* src/df.c (device_list): A new global list now updated by
filter_mount_list().
(filter_mount_list): Adjust to take a parameter as to whether
update the global mount list, or only the mount <-> device ID mapping.
(get_dev): Use the device ID mapping to ensure we're not outputting
stats for the wrong device.  Also output placeholder values when we
can't access a system specified mount point.
(get_all_entries): Set the DEVICE_ONLY param for filter_mount_list().
(devname_for_dev): A new function to search the mount <-> dev mapping.
* test/df/skip-duplicates.sh: Adjust accordingly.
* NEWS: Mention the bug fixes.

Discussed at: http://bugs.gnu.org/16539

* src/du.c (process_file): Treat cycles due to bind mounts
like cycles due to following symlinks.
* tests/du/bind-mount-dir-cycle.sh: Adjust accordingly.
* NEWS: Mention the change in behavior.
Reported at http://bugzilla.redhat.com/836557

* src/ls.c (usage): State that --human changes --size format.
Fixes http://bugs.gnu.org/17838

* doc/coreutils.texi (chown invocation): Mention the system dependent
restrictions on setting groups.
(chgrp invocation): Likewise. Reference the 'chown' superset.
* man/chgrp.x: Cross reference chown(1) which is the superset interface,
and also chown(2) which gives details of the platform restrictions.
Fixes http://bugs.gnu.org/17495

* src/df.c (filter_mount_list): Recent commit v8.22-108-g25a2c948
failed to copy file system type along with the updated device name.
Therefore simply replace the existing mount entry with the
current one with all the latest device details.  Note the name,
even if not shorter in this entry, will be replaced with a shorter
name in a subsequent mount entry.
* tests/df/skip-duplicates.sh: Add a test case.

* src/seq.c (main): Avoid seq_fast() with a start or end of -0.
* tests/misc/seq.pl: Add test cases.
* NEWS: Mention the fix.
Fixes http://bugs.gnu.org/17800