Tuesday, 3 March 2015

Lrzip 0.620

I finally found some time to give lrzip some much needed love again. Fortunately the last release, 0.616, proved very stable for the vast majority of workloads so there was never any great need to give it attention, though little things slowly cropped up, so I've accumulated all the bugfixes till now to release a new stable version, 0.620.

Freecode has long since died so here's the link to the (ghetto) home and download page:


and the git source code page:


Summary of changes:
- It would previously crash if trying to decompress from STDIN with a file that was too large to fit in ram which has now been fixed.
- It would previously fail to decompress files that were too large to fit in ram decompressed, now fixed.
- There were some scenarios lrzip would run out of ram when there was plenty to allocate, now fixed.
- Some other unix platforms would consider locking a mutex in one thread and releasing it in another a bug, so I've converted the use of those mutexes to anonymous semaphores.
- In order to maintain compatibility with platforms that don't properly support anonymous semaphores (OSX I'm looking at you), I've added the use of my custom fake semaphores as discussed here: unnamed-semaphores-and-pososx
- Some files would have their size reported wrongly with -i, now fixed.
- Added the ability to limit the use of ram with -m since lrzip happily uses all of it normally.
- Other minor changes and fixes for rare corner cases.

The changelog:
* Increase maxram when we abandon use of temporary input/output buffers
* Don't delete the tmpinfile when decompressing from stdin before allowing seek
to end to succeed in checking md5
* Use temporary file from read_seekto when STDIN will not fit in the ram input
* Remove unused read_i64 function
* Add message about issue tracker in BUGS
* Use a common exit path iin lrzip_compress/decompress and fix lr leak on
successful return
* Fix parenthesis placement inside of unlikely().
* Clear sa_mask and sa_handler before calling sigaction().
* Fix for lrzip -i. Decompressed size wrong
* added '-m' command line option
* Fix wrong README file being included in Makefile
* Pass strict sizes to decompress length, rounding up only the amount we're
allocating to not confuse decompression libraries
* Convert the thread locking to use cksems
* Add cksems to util.h
* Fix 'Failed to malloc ckbuf in hash_search2' with very large files.
* Round up compression and decompression buffers to page size since malloc will
allocate them that large anyway.
* Increase the compressed buffer size given to libzpaq in case of incompressible
data since it does not check if it's trying to write beyond the end of the
* Provide a helper function to round a value up to the nearest page size for
malloc optimisations.

Friday, 27 February 2015

BFS 461, linux-3.19-ck1

Announcing a resync and update of BFS for linux-3.19

BFS by itself:


-ck branded linux-3.19-ck1 patches:

3.19-ck1 patches

Apart from a resync with mainline and merging of the pending patches that were around for BFS460, there are no new changes. Apologies if I've been unable to address any new issues posted here - as per usual lack of time is the reason. There are some pending changes to the scheduler for mainline (as pointed out by kernelOfTruth here: link) but they're not finalised so I won't be delaying this release to wait for them.


Thursday, 11 December 2014

BFS 460, linux-3.18-ck1

Announcing a resync and update of BFS for linux-3.18

BFS by itself:


-ck branded linux-3.18-ck1 patches:

3.18-ck1 patches

Uncharacteristically I found time to resync up quickly for this latest stable linux release. There are no new BFS features, but there have been a number of changes to stay in sync with mainline. Apart from keeping up with the usual churn in new releases, of which there was a modest amount this time, a number of other low level changes were committed making this much less of a trivial resync so some caution is warranted before blindly updating.

Hilf Danton pointed out a bug in the yield_to code (thanks!) which is now fixed. Since almost nothing uses this code you probably won't notice anything. He also pointed out some other now outdated components in BFS which are also updated. The above_background_load function has also been removed since the VM tweaks in older -cks no longer exist to use it. 

More substantially, I've reworked the plugged I/O code to match mainline now, which I had been reluctant to touch previously because of the deadlocks the unlocking and relocking in the scheduler code path introduced when the the first plugged I/O code made its way into BFS needing iterations of fixes - watch for any I/O misbehaviour/stalls. There are some changes to how mainline responds to idle CPUs so watch for any unusual behaviour there.

Having said that I've been using it for a while and not noticed anything out of the ordinary, but please report back if there are any issues.


Tuesday, 18 November 2014

BFS 458, linux-3.17-ck2

This is a bugfix release for the power usage regression as reported here with BFS 457.

BFS by itself:

CK branded BFS separate and combined patches:

Incremental change from BFS 457-458: