DO Agent failing with `error code 100`

Start with /opt/digitalocean/droplet-agent/scripts/update.sh and the /var/log/droplet-agent.update.log output shown in the issue. Reproduce or inspect the update path around the apt lock failure across affected hosts. Done means the cause of the locks and the expected update behavior are established.

This is a tracking issue, some logs:

coffee@buildbot:~$ cat /var/log/droplet-agent.update.log
Waiting 799 seconds
Checking architecture support...OK
Preparing to check for update
Package Host: https://repos-droplet.digitalocean.com
Package Path: apt/droplet-agent/pool/main/main/d/droplet-agent/droplet-agent_
Keyring Path: apt/droplet-agent/pool/main/main/d/droplet-agent-keyring/droplet-agent-keyring_
Local Keyring:2.0.0
Local Version:1.2.11
Checking Latest Version...
Latest keyring package:2.0.0
Latest agent package:1.2.1
Updating droplet-agent deb package
E: Could not get lock /var/lib/apt/lists/lock. It is held by process 1708639 (apt-get)
E: Unable to lock directory /var/lib/apt/lists/

the logs do exist, are empty, but eventaully go away:

coffee@buildbot:~$ sudo /opt/digitalocean/droplet-agent/scripts/update.sh
Waiting 465 seconds
Checking architecture support...OK
Preparing to check for update
Package Host: https://repos-droplet.digitalocean.com
Package Path: apt/droplet-agent/pool/main/main/d/droplet-agent/droplet-agent_
Keyring Path: apt/droplet-agent/pool/main/main/d/droplet-agent-keyring/droplet-agent-keyring_
Local Keyring:2.0.0
Local Version:1.2.11
Checking Latest Version...
Latest keyring package:2.0.0
Latest agent package:1.2.1
Updating droplet-agent deb package
coffee@buildbot:~$ echo $?
0
coffee@buildbot:~$ cat /var/log/droplet-agent.update.log
Waiting 493 seconds
Checking architecture support...OK
Preparing to check for update
Package Host: https://repos-droplet.digitalocean.com
Package Path: apt/droplet-agent/pool/main/main/d/droplet-agent/droplet-agent_
Keyring Path: apt/droplet-agent/pool/main/main/d/droplet-agent-keyring/droplet-agent-keyring_
Local Keyring:2.0.0
Local Version:1.2.11
Checking Latest Version...
Latest keyring package:2.0.0
Latest agent package:1.2.1
Updating droplet-agent deb package

so i'm not really sure yet what's causing the locks across many hosts 🙃

https://costinstefan.ro/post/2020-02-16-how-to-fix-do-agent-exited/
https://www.digitalocean.com/community/questions/do-agent-exited-with-a-return-code-100