GitLab

* NEWS: Record release date.

* Makefile.am: Remove our dependence on src/sort which
induces awkward dependencies for `make dist` since
THANKS will be rebuilt once src/sort is newer.
Instead we remove the problematic -f option to sort
which actually doesn't change the order given
our current input.

* cfk.mk (sc_tests_executable): Restrict the check to git files,
so we don't flag any gnulib files added to test/ during
`make distcheck`.

* gnulib: Update to latest, with fixes to tests edge cases.
* tests/init.sh: Update from gnulib.

Since non interactive shells don't generally set $SHELL,
its value is propagated through the tests and may cause issues;
for example if $SHELL implicitly adjusts $PATH when run.
Instead we set $SHELL to that determined by the posix-shell module,
and use that consistently for all test sub scripts,
including those created thorugh the `split --filter` command.

* tests/local.mk: Explicitly set $SHELL to $(PREFERABLY_POSIX_SHELL)
which defaults to $CONFIG_SHELL and thus usually /bin/sh.
* tests/envvar-check: Remove bash environment variables with
side effects, in case /bin/bash was selected for $SHELL.
* tests/misc/help-version.sh: Remove redundant initialization of $SHELL.
* tests/install/strip-program.sh: Use $SHELL for sub script.
* tests/misc/sort-compress-hang.sh: Likewise.
* tests/misc/sort-compress-proc.sh: Likewise.
* tests/misc/sort-compress.sh: Likewise.
* tests/misc/timeout-group.sh: Likewise.
* tests/rm/fail-eperm.xpl: Remove redundant elision of bash env vars.
* tests/misc/pwd-long.sh: Likewise.

* tests/du/threshold.sh: Homogenize getopt error messages.
* tests/misc/numfmt.pl: Likewise.
* tests/mv/i-3.sh: Skip on *BSD not just FreeBSD.

tests/misc/wc-proc.sh fails when the page size is 64K

* src/wc.c (wc): The lseek adjustment should be based on st_blksize,
rather than on the internal buffer size.  This is significant on
aarch64 where st_blksize in /proc is the 64K (the page size) and
thus larger than the internal buffer.
* src/split.c (main): Even though the similar processing is done
on the internal buffer size, that's based on st_blksize and
so fine in this regard.  Add an assert to enforce this.
Avoid this path for the undocumented ---io-blksize option.

Mainly with build fixes for FreeBSD and OS X.

* m4/jm-macros.m4 (HAVE_FPSETPREC): Define if needed.
* src/numfmt.c (main): Call fpsetprec() if needed.
Fixes large-15 and large-16 test failures on 32 bit FreeBSD.

* tests/misc/stty.sh: FreeBSD returns ENOTTY for
the TIOCEXT ioctl, so just avoid this option for now.

The new tests/misc/factor-parallel.sh test was
seen to fail on FreeBSD (derived) systems, which was
due to split(1) --filter reading partial lines
through pipes, as factor(1) was writing a little
over PIPE_BUF each time.

* src/factor.c (lbuf): A new structure to internally buffer lines.
(lbuf_alloc): A new function to allocate enough at program start.
(lbuf_putint): A new function to buffer a uintmax_t.
(lbuf_flush): A new function to write directly to standard output.
(lbuf_putc): A new function to buffer a character and if enough
lines are buffered, then output complete lines <= PIPE_BUF,
and continue to buffer the rest.
(main): Call the internal buffer allocator, and register
the final flush from the internal buffer at program exit.

* tests/dd/stats.sh: Wait 20s for dd to write 250MB through a fifo,
rather than 10s for 500MB.  The failure was seen often on
a lightly loaded SPARC-Enterprise-T5220 running Solaris 10.

* bootstrap.conf (gnulib_modules): Add setenv, to make this
module dependency explicit; setenv is also used by split.
* src/stdbuf.c (set_LD_PRELOAD) [__APPLE__]: Use the OS X setenv
function, rather than putenv, per that documentation:
https://developer.apple.com/\
library/mac/documentation/Darwin/Reference/ManPages/man3/putenv.3.html

* src/numfmt.c (parse_field_arg): Rename parameter s/optarg/arg/,
to avoid shadowing getopt's global variable.
Otherwise, building on OS X, with --enable-gcc-warnings, I saw this:

  In file included from src/numfmt.c:19:0:
  src/numfmt.c: In function 'parse_field_arg':
  ./lib/config.h:3109:25: error: declaration of 'rpl_optarg' shadows\
    a global declaration [-Werror=shadow]

* src/numfmt.c (double_to_human): Fix the argument order
passed to snprintf, which happened to work on amd64 with
its separate va_arg storage area for floats¹,
but would fail tests for example on i686.

¹ https://blog.nelhage.com/2010/10/amd64-and-va_arg/

* src/df.c (filter_mount_list): Clarify why we still stat even
though devices IDs may already be available.  Note using
/proc/self/mountinfo is still an advantage to get filtered items
with accurate device patchs in chroots and with bind mounts.
I.E. on older setups with static /etc/mtab, df will now
bypass that to get the more accuracte and dynamic info.

* src/chroot.c (main): Quote the passed argument,
to avoid confusing error messages.

* src/factor.c (print_uintmaxes): Comment that the
value of n_out doesn't matter on error, and add an
explicit cast to avoid any future warnings.
Suggested by Jim Meyering RE commit v8.23-229-g4d2d6c5b

* tests/misc/sync.sh: Ensure dir is unreadable before
including the permission check.

The LD_PRELOAD checks by -fsanitize=address are overly strict:
https://groups.google.com/forum/#!topic/address-sanitizer/jEvOJgkDqQk
A workaround is to first export LD_PRELOAD=libasan.so.2
The tests below are adjusted so that workaround is not discarded.

* tests/cp/no-ctx.sh: Append to $LD_PRELOAD.
* tests/df/no-mtab-status.sh: Likewise.
* tests/df/skip-duplicates.sh: Likewise.
* tests/ls/getxattr-speedup.sh: Likewise.
* tests/rm/r-root.sh: Likewise.
* tests/cp/nfs-removal-race.sh: Likewise.  Also check that
LD_PRELOAD is effective to aid future maintainability
and avoid false failure if libasan.so.2 is not preloaded.

Mainly for -fsanitize=address and -fsanitize=undefined fixes

* src/make-prime-list.c (main): When building with
the above option, avoid this build stopping error:
"LeakSanitizer: detected memory leaks"

GCC 5.1.1 -fsanitize=undefined with glibc 2.21 is returning:
"runtime error: null pointer passed as argument 1,
 which is declared to never be null"
* src/ptx.c (sort_found_occurs): Avoid the call with no entries.

* src/factor.c (n_out): A new global variable to track
how much data has been written to stdout.
(print_factors_single): Use n_out to determine whether
to flush the current (and previous) lines.
* tests/misc/factor-parallel.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug fix.

src/seq.c (scan_arg): Set precision and width _after_ exponentiation.
For example, this will make '1.1e1 12' and '11 1.2e1' equivalent.
One can still set the precision by specifying extra precision on
the start value, or more naturally with a precision on a step value.
* tests/misc/seq-precision.sh: Add new cases.

* src/seq.c (scan_arg): Set precision to 0 for hex constants
(while avoiding hex floats).  This will use then use the
fast path for these arguments.  Note we also set the precision
of inf to 0 here, which ensures we use consistent precision
on output where possible.
* tests/misc/seq-precision.sh: Add corresponding test cases.

* src/seq.c (main): Call seq_fast for infinite last value.
This implicitly avoids format conversion on the
999999 -> 1000000 transition.
* src/seq.c (seq_fast): Generalize the buffer handling,
and adjust to handle the "inf" last value specifics.
* tests/misc/seq-precision.sh: A new test.
* tests/local.mk: Reference the new test.

* doc/coreutils.texi (main menu): Add numfmt.

* src/numfmt.c (MAX_UNSCALED_DIGITS): Set this to LDBL_DIG
rather than hardcoding at 18 for better portability.
* tests/misc/numfmt.pl: Restrict limit tests to supported platforms.

* src/numfmt.c (simple_strtod_int): Don't count leading zeros
as significant digits.  Also have leading zeros as optional
for floating point numbers.
* tests/misc/numfmt.pl: Add test cases.
* NEWS: Mention the fix.

Due to existing limits this is usually triggered
with an increased precision.  We also add further
restrictions to the output of increased precision numbers.

* src/numfmt.c (simple_round): Avoid intmax_t overflow.
(simple_strtod_int): Count digits consistently
for precision loss and overflow detection.
(prepare_padded_number): Include the precision
when excluding numbers to output, since the precision
determines the ultimate values used in the rounding scheme
in double_to_human().
* tests/misc/numfmt.pl: Add previously failing test cases.
* NEWS: Mention the fix.

* src/numfmt.c (usage): Update the --format description
to indicate precision is allowed.
(parse_format_string): Parse a precision specification
like the standard printf does.
(double_to_human): Honor the precision in --to mode.
* tests/misc/numfmt.pl: New tests.
* doc/coreutils.texi (numfmt invocation): Mention the new feature.
* NEWS: Likewise.

* src/numfmt.c: Replace field handling code with logic that understands
field range specifiers.  Instead of processing a single field and
printing line prefix/suffix around it, process each field in the line
checking whether it has been included for conversion.  If so convert and
print, otherwise just print the unaltered field.
(extract_fields): Removed.
(skip_fields): Removed.
(process_line): Gutted and heavily reworked.
(process_suffixed_number): FIELD is now passed as an arg instead of
using a global.
(parse_field_arg): New function that parses field range specifiers.
(next_field): New function that returns pointers to the next field in
a line.
(process_field): New function that wraps the field conversion logic
(include_field): New function that checks whether a field should be
converted
(compare_field): New function used for field value comparisons in a
gl_list.
(free_field): New function used for freeing field values in a gl_list.
Global variable FIELD removed.
New global variable all_fields indicates whether all fields should be
processed.
New global variable all_fields_after stores the first field of a N-
style range.
New global variable all_fields_before stores the last field of a -M
style range.
New global variable field_list stores explicitly specified fields to
process (N N,M or N-M style specifiers).
(usage): Document newly supported field range specifiers.
* bootstrap.conf: Include xlist and linked-list modules.  numfmt now
uses the gl_linked_list implementation to store the field ranges.
* tests/misc/numfmt.pl: Add tests for 'cut style' field ranges.
Adjust existing tests as partial output can occur before an error
Remove test for the 'invalid' field -5.. this is now a valid range.
* gnulib: update to avoid compiler warnings in linked-list.
* NEWS: Mention the new feature.

* src/numfmt.c (usage): Don't scale output from df
so that numfmt outputs the correct values.

* src/numfmt.c (unit_to_umax): Support SI (power of 10) suffixes
with the --from-unit and --to-unit options.  Treat suffixes like
is done with --from=auto, which for example will change the meaning
of --to-unit=G to that of --to-unit=Gi.  The suffix support was
previously undocumented and it's better to avoid the traditional
coreutils suffix handling in numfmt by default.
* doc/coreutils.texi: Document the new behavior.  Also fix a typo
mentioning {from,to}=units=.
* tests/misc/numfmt.pl: Adjust accordingly.
* NEWS: Mention the change in behavior.

* gnulib: Update to get the new gnu-web-doc-update with --mirror option.
* README-release: Use the --mirror option in the instructions.
Also clarify and update various release steps.

* src/tail.c (recheck): Display diagnostices for replaced files
even with reused inodes which is a common case.
* tests/tail-2/F-vs-missing.sh: Use correct diagnostic in comment.
* tests/tail-2/F-vs-rename.sh: Likewise.

* src/tail.c (tail_forever_inotify): Use the fspec pointer to
distinguish previously output files, rather than a descriptor
from the inotify event.  That event descriptor was that of
the parent directory when files were created or renamed etc.
(check_fspec): Adjust for the new comparison.  Also show the
header when the file is truncated, since we show data
in this case also.
* tests/tail-2/F-headers.sh: A new test case.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug fix.

* .gitignore: Add entries for potentially generated headers.
Also remove a couple of items already present in lib/.gitignore.
* cfg.mk (sc_gitignore_missing): A new syntax check rule to
identify missing .gitignore entries.
(sc_gitignore_redundant): A new syntax check rule to
identify redundant .gitignore entries.
Reported by Tomas Nordin.

* configure.ac: Comment on why we link rather than run the test,
and remove the moot __ELF__ check since we never ran it anyway,
and the new CFLAGS and LDFLAGS are a more direct test of support.
* tests/misc/wc-parallel.sh: Fix a syntax error in the previous change.
* tests/misc/md5sum-parallel.sh: Use better error checking, consistent
with that used in wc-parallel.sh.