Take this job:
It gets stuck half-way and stays like this for >23h.
Isn't there a global job timeout ?
Why does the job lock-up. My hinch is a packet drop on the zmq pipe to the master that causes the connection to reset somehome. Then the master cannot re-aquire the correct job state / log (or vice versa, the slave possibly terminates correctly but it is not captured by the master).
Result is that a long queue builds-up in jenkins for test-jobs waiting for these to finish.