GitLab

* NEWS: Record release date.

* tests/misc/ptx-overrun.sh: Quote appropriately to avoid this
recently added issue, noticed by `make taint-distcheck`.

* Makefile.am (my-distcheck): Add the -r option to xargs so that
rmdir doesn't return an error when there are no extraneous .deps dirs.

* tests/misc/ls-time.sh: Skip the test rather than ERROR
when `touch -m -d ...` fails (Hurd).
* tests/tail-2/follow-stdin.sh: Avoid false FAILs by ignoring
the variances in sterror output.
* tests/rm/rm-readdir-fail.sh: Likewise.  Also avoid ERRORs
on systems that don't define _D_EXACT_NAMELEN.

* src/head.c (elide_tail_bytes_file): Ensure we don't use
st_size unless we've previously used seek() to determine
the CURRENT_POS in the seekable file.
This was seen to cause issue on FreeBSD 11 when the pipe
buffer was filled with `yes | head --lines=-0`, in which
case st_size was 64KiB while ST_BLKSIZE() was 4KiB.
Reported by Assaf Gordon.

... in the case where two or more directories nested in each other are
created and each of them defaults to a different SELinux context.

* src/install.c (make_ancestor): When calling defaultcon(), give it the
same path that is given to mkdir().  The other path is not always valid
wrt. current working directory.
* src/mkdir.c (make_ancestor): Likewise.
* NEWS: Mention the bug fix.

Reported at https://bugzilla.redhat.com/1398913

This was detected with ASAN, but can also be seen without ASAN with:
  $ tac - - <&-
  tac: standard input: read error: Bad file descriptor
  *** Error in `tac': malloc(): memory corruption: 0x...

* src/tac.c (copy_to_temp): Don't close our output stream on
(possibly transient) output error, or on input error.
(temp_stream): clearerr() on the stream about to be reused,
to ensure future stream use is not impacted by transient errors.
* tests/misc/tac-2-nonseekable.sh: Add a test case.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/25041

Reproduced under UBSAN with `tail -f <&-` giving:
  tail.c:2220:18: runtime error: load of value 190,
  which is not a valid value for type ‘_Bool'

* src/tail.c (tail_file): Ensure f->ignore is initialized
in all cases where we can't tail the specified file.
* tests/tail-2/follow-stdin.sh: Add a test case which
checks stderr has no UBSAN warnings.
Fixes http://bugs.gnu.org/25041

* NEWS: Mention in improvements about the workaround for
the glibc issue with closed stdin, and the new supported file systems.
* tests/misc/b2sum.sh: Spelling fix.

Update with the results from:

  kgit='https://git.kernel.org/cgit/linux/kernel/git'
  wget -q $kgit/torvalds/linux.git/plain/include/uapi/linux/magic.h \
    -O src/fs-latest-magic.h
  make src/fs-magic-compare

* src/stat.c (human_fstype): Add entries for:
BALLOON_KVM, CGROUP2, DAXFS, ZSMALLOC.

* src/longlong.h: Sync from gmp repo incorporating:
Protect umul_ppmm with do ... while (0)
Replace obsolete ARC asm 'J' constraints with 'Cal'
Provide umul_ppmm for riscv64

glibc has changed the public define
from _STRING_ARCH_unaligned to _STRING_INLINE_unaligned as per
https://sourceware.org/bugzilla/show_bug.cgi?id=19462

* gl/lib/rand-isaac.c: Cater for both defines.
* gl/lib/randread.c: Likewise.
* src/system.h: Update commented out code.

Problem reported by Alex Ryan (Bug#25029).
* tests/misc/shuf.sh: Test for shuffling with stdin closed.

* src/numfmt.c (main): Don't implicitly coerce pointer in bool
initializer.  Although it's portable C99 code, it's confusing.

* tests/ls/quote-align.sh: Remove "total" line
which can vary per file system depending on allocation.
Reported by Assaf Gordon on OpenSolaris (5.11/5.10).

This was detected with:
  echo a > a; pr "-S$(printf "\t\t\t")" a -m a > /dev/null
Resulting in ASAN triggering:
  ====================================================
  ERROR: AddressSanitizer: global-buffer-overflow
  READ of size 1 at 0x00000041b622 thread T0
    #0 0x40506a in print_sep_string ../src/pr.c:2241
    #1 0x407ec4 in read_line ../src/pr.c:2493
    #2 0x40985c in print_page ../src/pr.c:1802
    #3 0x40985c in print_files ../src/pr.c:1618
    #4 0x4036e0 in main ../src/pr.c:1136

* src/pr.c (init_parameters): Ensure we only override the
specified separator when it's a single tab, thus matching
the calculated separator length.
* tests/pr/pr-tests.pl: Add a test case.
* NEWS: Mention the fix.

* src/ptx.c (fix_output_parameters): Ensure line_width doesn't
go negative, which can happen when the --width is less
than the --gap-size.
* tests/misc/ptx-overrun.sh: Add a test case that triggers
with ASAN.  (Note the longer filename is needed to trigger).
Fixes http://bugs.gnu.org/25011

* src/readlink.c (usage): Mark the --quiet/--silent option as active
by default.
* doc/coreutils.texi (readlink invocation): Likewise.

* tests/rm/rm-readdir-fail.sh: ASAN correctly indicated
that fts was writing to freed memory.  This was because
we reused a single dirent in our readdir() test wrapper.
Since fts was deallocating those dirents, we now get
a new dirent for each call to our readdir wrapper.

ASAN reported this error for: split -n2/3 /dev/null
  ERROR: AddressSanitizer: negative-size-param: (size=-1)
  #0 0x7f0d4c36951d in __asan_memmove (/lib64/libasan.so.2+0x8d51d)
  #1 0x404e06 in memmove /usr/include/bits/string3.h:59
  #2 0x404e06 in bytes_chunk_extract src/split.c:988
  #3 0x404e06 in main src/split.c:1626

Specifically there would be invalid memory access
and subsequent processing if the chunk to be extracted
was beyond the initial amount read from file (which is
currently capped at 128KiB).  This issue is not in a
released version, only being introduced in commit v8.25-4-g62e7af03

* src/split.c (bytes_chunk_extract): The initial_read != SIZE_MAX
should have been combined with && rather than ||, but also this
condition is always true in this function so remove entirely.
* tests/split/b-chunk.sh: Add a test case.

Fixes http://bugs.gnu.org/25003

In recent commit v8.25-93-g7fc7206b we used the f->remote flag
which wasn't set in all cases.  This was detected with
ASAN giving this error when reading f->remote;
  runtime error: load of value 190,
  which is not a valid value for type '_Bool'

* src/tail.c (fremote): Query the system even without inotify.
(recheck): Always set f->fremote for valid files.

Installing with -j2 or greater could result in a failure like
  /bin/sh: line 29: /P/bin/install: Permission denied
when /P/bin/install specifies your PATH-selected install program.
This would arise because we're using "install" to install all
man/*.1 files, and that command would run concurrently with the one
that installs "/P/bin/install" itself.  We would run this command:
"src/ginstall src/ginstall /P/bin/install", and it would result
in intervals during which the destination file does not exist,
is empty or incomplete and not executable.  We addressed this
problem long ago for installation of actual binaries by telling the
installation rules to use our just-built bin/ginstall (only when not
cross-compiling) rather than the PATH-resolved "install" program.
This change is to do the same for those .1 files.
* src/local.mk (INSTALL): Override automake's default of something
like "INSTALL = /P/bin/install -c".
(INSTALL_PROGRAM): Now that we set INSTALL, there is no longer any
need to set this derived variable.  Its default definition,
"INSTALL_PROGRAM = ${INSTALL}" does what we require.
Improved by Eric Blake.

* cfg.mk (sc_gitignore_missing): Add $(srcdir) to .gitignore filename.
(sc_gitignore_redundant): Likewise.

* .gitignore: Add entry for the above file which is created
by  './bootstrap'.

Problem reported by Marcel Böhme (Bug#24996).
* configure.ac (WERROR_CFLAGS): Avoid -Wtype-limits.
* src/pr.c (col_sep_string): Now a const pointer.  All uses changed.
(integer_overflow): New function.
(separator_string, main, init_parameters, init_store_cols):
Check for integer overflow.
(align_column, read_line, print_stored): Avoid integer overflow.

* src/comm.c (total_option): Add bool variable for the new option.
(TOTAL_OPTION): Add enum value.
(long_options): Add array element for the new option.
(usage): Document the new option here.
(compare_files): Count the lines in total[3], and output the summary at
the end.
(main): Accept the new option.
* doc/coreutils.texi (comm invocation): Document it.
* tests/misc/comm.pl: Test it.  While at it, improve the test data
to have 1 unique line in the first file, 2 unique lines in the second
file, and 3 common lines.
* NEWS (New Features): Mention the new option.

Fixes http://bugs.gnu.org/24929

Also sync these copies with gnulib:
* bootstrap: Sync missed part of AIX support.
* tests/init.sh: Support compare_ on AIX.

* tests/misc/printf-quote.sh: FreeBSD 11 was seen to treat
\u0378 as a printable character.  Therefore change to
using the \u0081 C1 control character.  We use the UTF-8
representation because our printf implementation explicitly
disallows \u0081 as input.
Reported by Assaf Gordon

On BSD /bin/sh it was seen that unexported env vars passed to
returns_() would not be propagated to the wrapped command.

* cfg.mk (sc_prohibit_env_returns): Add a syntax check to disallow.
* tests/misc/csplit-io-err.sh: Rearrange to export vars in a subshell.
* tests/rm/rm-readdir-fail.sh: Likewise.
* tests/misc/nohup.sh: Export and unset vars around returns_.
* tests/misc/printenv.sh: Likewise.
Reported by Assaf Gordon

* tests/tail-2/pipe-f.sh: Avoid issue with readable
directories on BSD systems.
* tests/tail-2/retry.sh: Likewise.
Reported by Assaf Gordon

This provides better alignment when some names are quoted,
which also provides better indication that quotes are not
part of the name.

* src/ls.c (align_variable_outer_quotes): A new variable
set when ls is aligning columns (not using -m, non-zero -w),
and has a variable quoting style (shell, shell-escape, c-maybe).
(quote_name_buf): Writes to buffer rather than FILE,
taking care to avoid data copying if possible.  Refactored from...
(quote_name): ...here.  This now manages the buffer passed
to quote_name_buf() and outputs the padding, colors and name
in the appropriate order, while managing the --dired offsets.
(get_color_indicator): A new function to return the color sequence,
refactored from...
(print_color_indicator): ...here.  This now simply outputs.
(print_dir): Refactor common parts to quote_name().
(clear_files): Reset the flag indicating at least one
file is quoted in the current directory.
(needs_quoting): A new function to indicate at the scan stage
whether a name needs quoting.  Called from...
(gobble_file): ...here, until we find the first quoted file.
(print_name_with_quoting): Mostly refactored to quote_name().
* tests/ls/quote-align.sh: A new test for various output formats.
* tests/local.mk: Reference the new test.
* NEWS: Mention the improvement.

Note we don't support the --algorithm option of the
b2sum command in the external BLAKE2 project, as that
was deemed too confusing for users.
"BLAKE2b" was chosen as the default algorithm to use,
which is single threaded but performs well on 64 bit.

* src/blake2: CC0 source copied from external project.
* cfg.mk[VC_LIST_ALWAYS_EXCLUDE_REGEX]: Exclude blake2/
from syntax checks, make update-copyright, etc.
* src/local.mk: Reference the sources for b2sum,
and set the compilation flags.
* doc/coreutils.texi (b2sum invocation): Reference the
md5sum invocation node, and add descriptions of -l.
* tests/misc/b2sum.sh: Add new test.
* tests/local.mk: Reference new test.
* AUTHORS: Add new binary.
* README: Likewise.
* build-aux/gen-lists-of-programs.sh: Likewise.
* man/.gitignore: Likewise.
* scripts/git-hooks/commit-msg: Likewise.
* man/b2sum.x: New man page template.
* man/local.mk: Reference new template.
* src/.gitignore: Ignore new binaries.
* src/blake2/.gitignore: Ignore new build atrifacts.
* src/md5sum.c (usage): Describe the new -l option.
* NEWS: Mention the new program.

* Makefile.am (my-distcheck): Remove all .deps directories before the
recursive diff that searches for left-behind files.  Otherwise, with
automake master (some time after v1.15), "make distcheck" would fail
due to those directories being left behind after "make distclean".

* src/copy.h (cp_options): Add a new flag for install(1).
* src/copy.c (copy_internal): For cp, also output remediation
advice which also indicates why directories aren't copied by default.
The message is unchanged for install(1).
* src/cp.c (cp_option_init): Init install_mode to false.
* src/mv.c (cp_option_init): Likewise.
* src/install.c (cp_option_init): Init install_mode to true.
* tests/install/basic-1.sh: Add a test case.
* tests/cp/link-deref.sh: Adjust test case.
Fixes http://bugs.gnu.org/24958

* tests/misc/date.pl: Years < 1900 fail on solaris at least.
Reported by Assaf Gordon

* tests/misc/sort-h-thousands-sep.sh: The sv_SE locale
may be available while sv_SE.utf8 is not.  Therefore
test and use the same locale, now definfed in a variable.
Reported by Jim Meyering.

* src/dircolors.hin: Add .mjpg and .mjpeg multimedia files.
This resolves http://bugs.gnu.org/24436.

The previous "returns_"-using change failed to convert many
uses of "$?".  Convert all but two of the remaining ones.
* tests/ls/stat-vs-dirent.sh: Likewise.
* tests/misc/head-write-error.sh: Likewise.
* tests/misc/nice.sh: Likewise.
* tests/misc/nohup.sh: Likewise.
* tests/misc/stdbuf.sh: Likewise.
* tests/misc/sync.sh: Likewise.
* tests/tail-2/pid.sh: Likewise.
* tests/tail-2/wait.sh: Likewise.
Thanks to Bernhard Volker for spotting this.

This reduces a standard coreutils install size by about 160K.

* src/cat.c: Change to proper_name() which removes about 18K text.
* src/cp.c: Likewise.
* src/df.c: Likewise.
* src/du.c: Likewise.
* src/getlimits.c: Likewise.
* src/realpath.c: Likewise.
* src/split.c: Likewise.
* src/stdbuf.c: Likewise.
* src/timeout.c: Likewise.
* src/truncate.c: Likewise.
* src/local.mk: Remove -llibiconv from the above programs.
* cfg.mk (sc_check-AUTHORS): Adjust to use factor(1).
* AUTHORS: Adjust to use ASCII to satisfy sc_check-AUTHORS.