ValueError: list.remove(x): x not in list

/shared/letsencrypt/hook_script.py の clean_challenge と 62 行目から始め、両方の challenge token が関係するログの全シーケンスを再現します。ACME response によって記録された token と、challenge iRule に配置された token を比較し、各 cleanup の前に virtual server のルールを調べます。challenge response が成功し、list.remove エラーなしで cleanup が完了すれば完了です。

I figure we'll start with what I believe is the main error and everything I'm using is what you have as current from March 21st :)

File "/shared/letsencrypt/hook_script.py", line 62, in clean_challenge
vip.properties['rules'].remove('/Common/le_challenge_rule')
ValueError: list.remove(x): x not in list
ERROR: clean_challenge hook returned with non-zero exit code

I see the files created on my Apache server using "watch ls" since it creates then removes them quickly. They did happen so that's working.
The iRule fails to delete per the error and it remains on the box. I've deleted it manually and re-run various times trying various things and still no luck.
I've commented out a few sections of the hook script and it still fails but it gives some insight. I'm not going to explain that now unless ya'll ask.

The other thing I'm seeing but this might be the issue w/ the iRule deletion failure.
The tokens seem to mismatch from what's in the log in the CLI vs. the token stored in the challenge iRule.
I can share most anything since this is my personal web server/F5.
Thoughts?

Here's a full log example below:

INFO: Using main config file /shared/letsencrypt/config
Processing projectbaiu.org with alternative names: www.projectbaiu.org

Signing domains...
Generating private key...
Generating signing request...
Requesting new certificate order from CA...
Received 2 authorizations URLs from the CA
Handling authorization for www.projectbaiu.org
Handling authorization for projectbaiu.org
2 pending challenge(s)
Deploying challenge tokens...
(hook) Deploying Challenge
(hook) Challenge rule added to virtual.
(hook) Deploying Challenge
(hook) Challenge rule added to virtual.
Responding to challenge for www.projectbaiu.org authorization...
(hook) Invalid Challenge
(hook) Invalid Challenge Args: ['www.projectbaiu.org', '["type"]\t"http-01"\n["status"]\t"invalid"\n["error","type"]\t"urn:ietf:params:acme:error:unauthorized"\n["error","detail"]\t"63.226.21.50: Invalid response from http://www.projectbaiu.org/.well-known/acme-challenge/oQL3Lr39uV-G-Bp4xq54VG4OQD7sexwc1uV2DmWsXHQ: 404"\n["error","status"]\t403\n["error"]\t{"type":"urn:ietf:params:acme:error:unauthorized","detail":"63.226.21.50: Invalid response from http://www.projectbaiu.org/.well-known/acme-challenge/oQL3Lr39uV-G-Bp4xq54VG4OQD7sexwc1uV2DmWsXHQ: 404","status":403}\n["url"]\t"https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2306706054/wc9l2A"\n["token"]\t"oQL3Lr39uV-G-Bp4xq54VG4OQD7sexwc1uV2DmWsXHQ"\n["validationRecord",0,"url"]\t"http://www.projectbaiu.org/.well-known/acme-challenge/oQL3Lr39uV-G-Bp4xq54VG4OQD7sexwc1uV2DmWsXHQ"\n["validationRecord",0,"hostname"]\t"www.projectbaiu.org"\n["validationRecord",0,"port"]\t"80"\n["validationRecord",0,"addressesResolved",0]\t"63.226.21.50"\n["validationRecord",0,"addressesResolved"]\t["63.226.21.50"]\n["validationRecord",0,"addressUsed"]\t"63.226.21.50"\n["validationRecord",0]\t{"url":"http://www.projectbaiu.org/.well-known/acme-challenge/oQL3Lr39uV-G-Bp4xq54VG4OQD7sexwc1uV2DmWsXHQ","hostname":"www.projectbaiu.org","port":"80","addressesResolved":["63.226.21.50"],"addressUsed":"63.226.21.50"}\n["validationRecord"]\t[{"url":"http://www.projectbaiu.org/.well-known/acme-challenge/oQL3Lr39uV-G-Bp4xq54VG4OQD7sexwc1uV2DmWsXHQ","hostname":"www.projectbaiu.org","port":"80","addressesResolved":["63.226.21.50"],"addressUsed":"63.226.21.50"}]\n["validated"]\t"2022-04-28T17:12:46Z"']
Cleaning challenge tokens...
(hook) Cleaning Challenge
(hook) Challenge rule removed from virtual.
(hook) Cleaning Challenge
Traceback (most recent call last):
File "/shared/letsencrypt/hook_script.py", line 120, in
clean_challenge(sys.argv[2:])
File "/shared/letsencrypt/hook_script.py", line 62, in clean_challenge
vip.properties['rules'].remove('/Common/le_challenge_rule')
ValueError: list.remove(x): x not in list
ERROR: clean_challenge hook returned with non-zero exit code