Software Update: HTCondor 8.6.3

The HTCondor Team at the University of Wisconsin-Madison has released a new stable version of its workload management system HTCondor. The version number has ended up at 8.6.3. HTCondor focuses on the management of compute-intensive tasks and can distribute them over several connected nodes. The user sends his task to HTCondor, after which it handles the process based on set policies and the availability of connected resources, and finally sends the results back to the user. HTCondor can, for example, control a dedicated Beowulf cluster, but also regular desktops that have nothing to do for a while. During the day SC16 Google, Fermilab and the HTCondor Team have a 160k-core cloud-based elastic compute cluster demonstrated. The list of changes in this release is as follows:

Version 8.6.3

Bugs Fixed:

• Fixed a bug that rarely corrupts the condor_schedd’s job queue log file when the input sandbox of a job with an X.509 proxy file is spooled. (Ticket #6240).
• Fixed a memory leak in the Python bindings related to logging. (Ticket #6227).

New Features:

• Allow Python modules to be used as condor_collector plugin. This undocumented feature is to be used by expert developers only. (Ticket #6213).

Version 8.6.2

New Features:

• Added metaknobs for defining map files for use with the ClassAd usermap function in the condor_schedd, and a metaknob for automatically assigning an accounting group to a job based on a mapping of the owner name of the job. (Ticket #6179).
• When the condor_credd is polling for credentials, the timeout is now configurable using CREDD_POLLING_TIMEOUT.
• The reverse option for condor_q was changed to reverse-analyze, and it now implies better-analyze. Formerly, the reverse option was ignored unless -better-analyze was also specified. (Ticket #6167).

Bugs Fixed:

• Fixed a bug that could cause condor_store_cred to fail on Windows due to a case-sensitive check of the user’s account name. (Ticket #6200).
• Updated Open MPI helper script to catch and handle SIGTERM and to use bash explicitly. (Ticket #6194).
• Docker Universe jobs now update the RemoteSysCpu attributes for job and in the job log. Previously, this field was always 0. (Ticket #6173).
• Docker universe detection is now more robust in the face of extraneous output to standard error on docker startup. This was preventing Condor from detecting that docker was properly working on hosts. (Ticket #6185).
• Fixed a bug that prevented SUBMIT_REQUIREMENT and JOB_TRANSFORM expressions from referencing job attributes describing the job’s X.509 proxy credential. (Ticket #6188).
• The Linux kernel tuning script no longer adjusts some kernel parameters unless a condor_schedd will be started by the master. (Ticket #6208).
• Fixed a bug that caused all but the first in a list of metaknobs to be ignored unless there were commas separating the list items. So use ROLE : Execute CentralManager would incorrectly add only the Execute role. Previously, use ROLE : Execute, CentralManager would correctly add both roles. (Ticket #6171).
• Worked around a problem with FORTRAN programs built with condor_compile and recent versions of gfortran (4.7.2 was OK, 4.8.5 was not), where those executables would not write to standard out if started in the standard universe. Also, updated the checkpointing library to permit condor_compile to successfully link FORTRAN (and other) programs calling certain math functions and built against up-to-date versions of glibc. (Ticket #6026).
• The default values ​​for HAD_SOCKET_NAME and REPLICATION_SOCKET_NAME have changed to enable the documented configuration for using these services with shared port to work. (Ticket #6186).
• Fixed a bug that caused condor_dagman to sometimes (rarely, but repeatably) crash when parsing DAGs containing splices. (Ticket #6170).
• The configuration parameters that control when job policy expressions are evaluated now work as documented. Previously, the default value for PERIODIC_EXPR_INTERVAL was 300, not 60 as intended. Also, the parameters MAX_PERIODIC_EXPR_INTERVAL and PERIODIC_EXPR_TIMESLICE were ignored for grid universe jobs. (Ticket #6199).
• Fixed a bug that could cause the Job Router to crash if the job_queue.log contained invalid or incomplete records. (Ticket #6195).
• Fixed a bug that caused updates of the job attribute x509UserProxyExpiration to be ignored for job policy evaluation when the job was managed by the Job Router. (Ticket #6209).
• Changed the default value of configuration parameters CREAM_GAHP_WORKER_THREADS to the value of GRIDMANAGER_MAX_PENDING_REQUESTS. This should prevent a back-log of commands in the CREAM GAHP observed by some users. (Ticket #6071).
• Fixed modification of PYTHONPATH environment variable that could fail in bash if set -u is enabled. (Ticket #6211).
• bosco_quickstart no longer assumes that submitting to a Slurm cluster requires the PBS emulation module. (Ticket #6211).
• Fixed a bug that caused condor_submit -dump to crash when the submit file had an attribute to enable the use of an x509 user proxy. (Ticket #6197).
• Updated the supported platform list in the Bosco installer script to include Ubuntu 16 and Mac OSX 10.12. Also, dropped Ubuntu 12 and Mac OSX 10.6 through 10.9. (Ticket #6178).
• Fixed a bug which in some obscure configurations caused a spurious PERMISSION DENIED error was printed in the StartLog when activating a claim. (Ticket #6172)..
• Fixed a bug which forced the administrator to restart (rather than reconfigure) running daemons after adding an entry to a DENY_* authorization list. (Ticket #6172)..

Version 8.6.1

New Features:

• condor_q now checks to see if authentication and security negotiation are enabled before attempting to request only the current users jobs from the condor_schedd. Prior to this change, configurations that disabled security or authentication would also need to set CONDOR_Q_ONLY_MY_JOBS to false. (Ticket #6125).
• The CLAIMTOBE authentication method is now in the list of methods for READ access if no list of authentication methods for READ or DEFAULT is specified in the configuration. This change allows sites that use the default host based security model to use condor_q -global with the only-my-jobs feature without making changes to their security configuration. (Ticket #6125).
• The collector now records the authentication method used to determine the authenticated identity. (Ticket #6122).

Bugs Fixed:

• Update Docker interface to be able to retrieve usage information from running containers and to remove containers when certain errors occurred when using Docker version 1.13. (Ticket #6088).
• In Docker universe, all writes to files in /tmp and /var/tmp by default write inside the container. There is a limit on the file size within the container, and jobs that write a lot to /tmp may hit that. If a docker universe job now runs on a system with MOUNT_UNDER_SCRATCH defined, HTCondor now adds those mounts as volume mounts, so file writes do not go to the container, but to the host file system. (Ticket #6080).
• Fixed a bug in condor_status -format and condor_q -format that caused the tools to truncate output to the width specified in the format specifier. The most likely manifestation of this bug was that punctuation after the format would not be printed when the format had an explicit width. (Ticket #6120).
• Fixed a bug that caused spurious shared port-related error messages to appear in the dagman.out file (by adding the new DAGMAN_USE_SHARED_PORT configuration macro). (Ticket #6156).
• Fixed a bug that caused VM universe jobs to fail if the vm_disk submit command contained spaces after a comma. (Ticket #6132).
• Fixed a bug that can cause the Job Router and condor_c-gahp to crash if they fail to submit a job due to submit transforms or submit requirements. (Ticket #6152).
• Fixed a bug that caused the Job Router to not route any jobs if the JOB_ROUTER_DEFAULTS configuration parameter value started with white space. (Ticket #6128).
• Fixed several bugs in how the Job Router writes to job event logs. (Ticket #6092).
• Removed Bosco’s attempt to configure a default value for grid_resource in the submit description file, as condor_submit no longer supports this ability. Also, Bosco now works with Slurm clusters. (Ticket #6106).
• Changed Bosco’s configuration of the condor_ft-gahp to eliminate worrying error messages in the condor_ft-gahp’s log file. (Ticket #6107).
• Fixed a bug that could cause a grid batch job submitted to PBS or Slurm to go on hold when the job’s X.509 proxy is refreshed. (Ticket #6136).
• Fixed a bug where the condor_gridmanager fails to put a job on hold due to the desired hold reason containing invalid characters. (Ticket #6142).
• Improved the hold reason when submission of a grid-type batch job fails. (Ticket #3377).
• Update helper scripts to work with current versions of Open MPI and MPICH2. (Ticket #6024).
• Fixes a bug that could cause events for local universe jobs to not be written to the global event log. (Ticket #6100).
• Fixed a bug on execute machines that enable PID namespaces that would generate a spurious error message in the daemon log when condor_off -fast was issued. (Ticket #6137).
• Fixed a bug that could corrupt the job queue log file such that the condor_schedd cannot restart. The bug is mostly likely to occur if the disk becomes full. (Ticket #6153).
• Incremented the ClassAd library version number, since the deprecated iostream interface has been removed. (Ticket #6050). (Ticket #6115).

Version 8.6.0

New Features:

• Added two new job ClassAd attributes, CumulativeRemoteSysCpu and CumulativeRemoteUserCpu, which keep a running total of system and user CPU usage, respectively, across all job restarts. Also, immediately clear attributes RemoteSysCpu and RemoveUserCpu on job start, instead of on first update. (Ticket #6022).
• Added a new configuration knob, ALWAYS_REUSEADDR, which defaults to True. When True, it tells HTCondor to set the SO_REUSEADDR socket option, so that the schedd can run large numbers of very short jobs without exhausting the number of local ports needed for shadows. (Ticket #6040).
• Changed the default value of IGNORE_LEAF_OOM to True. (Ticket #5775).

Bugs Fixed:

• Fixed a bug causing unnecessarily slow updates from the condor_startd. If you depend on the old behavior, set UPDATE_SPREAD_TIME to 8. A value of 0 enables the fix. (Ticket #6062).
• Fixed a race condition when running multiple competitor jobs on the same claim. When the starter exits, it notifies the shadow, which tells the started to kill the starter. Immediately after the shadows tells the started, it fetches the next job, and tries to start it. If the starter hasn’t completely exited yet (perhaps it needs to clean up a large sandbox), it will notice the shadow has closed the command socket, and the starter will go into disconnected mode, and get confused. This has been fixed. (Ticket #6049).
• Fixed an infelicity with condor_submit -i and docker universe, where it would start an interactive shell without a container. Added error message expressing that this combination is not currently supported. (Ticket #6083).
• When a job claimed by the Job Router is held or removed, it is no longer considered a failure of the job route chosen for that job. (Ticket #5968).
• Fixed a bug in recovering a Google Compute Engine (GCE) job if the condor_gridmanager restarts during submission of the instance request. (Ticket #6078).
• Fixed a bug that could cause re-installation of a remote cluster to fail in Bosco. (Ticket #6042).
• Fixed a bug with handling the proxy files of grid-type batch jobs when the proxy’s file name is a relative path. (Ticket #6053).
• Fixed a bug that caused the batch_gahp to crash when a job’s X.509 proxy is refreshed and the batch_gahp is configured to not create a limited copy of the proxy. (Ticket #6051).
• Fixed a bug in the virtual machine universe where RequestMemory and RequestCPUs were not changing the resources assigned to the VM created by HTCondor. Now, VM_Memory defaults to RequestMemory, and the number of CPUs defaults to RequestCPUs. (Ticket #5998).