[Feature]: HTCondorCEBackend — grid CE over the shared `_htcondor` core

Open
#23 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python

Research direction

Start by reading the shared _htcondor core and the contract suite for the htcondor stack, then run the remote and htcondor tests. Focus on the fetch-twice destructive_fetch regression, capability narrowing, and kill-path coverage. Done means the CE path is green with IDTOKENS, records its leave_in_queue decision, and exercises partial-batch outcomes.

Written by the indexing model from the issue text.

Description

User Story

As the WMS consumer,
I want an HTCondorCEBackend submitting to an HTCondor-CE with IDTOKENS and spooled file
transfer,
So that the second major grid-CE path exists on the new contract, including the destructive-fetch
semantics that motivated IC-ADR-002.

Feature Description

  • HTCondorCEBackend: JobBackend + OutputRetriever + Cancellable.
    Deliberately no LoadReporter (DIRAC's getCEStatus is an explicit "not supported") and no
    live diagnostics — capability absence is the point, and the integration suite must show the
    narrowing works.
  • Reuses the _htcondor core (submit description, ClassAd parsing, state
    map); differs in transport/auth: remote schedd (-pool/-remote) or CE submission, grid
    universe routing, -spool.
  • Token + proxy together per IC-ADR-003 (its Motivation 7 / all-of kinds): SCITOKENS/IDTOKENS
    channel auth (token file + _CONDOR_* env injection, audience <ce>:9619) plus the X.509
    proxy materialised into the submit description (use_x509userproxy = true) — consumed
    site-side (VOMS attributes → APEL accounting), including the
    _CONDOR_DELEGATE_JOB_GSI_CREDENTIALS=false workaround (HTCONDOR-2904). The pure-IDTOKENS
    basic stack config exercises the token path; a proxy-bearing config is the ADR-002 auth axis.
  • Destructive retrieval: condor_transfer_data then declared destructive=True
    (HTCondor ≥ 25.8: completed spooled job leaves the queue); evaluate leave_in_queue as the
    re-fetchability override and record the decision. Coarse bounded materialisation (submit-time
    transfer_output_files allow-list, transfer timeout, post-hoc size/count check).
  • Phase 2 wiring: contract suite against the htcondor stack's CE path (markers
    remote and htcondor); the fetch-twice destructive_fetch test is the flagship regression
    (versions axis [lts, latest] makes 25.8-class changes red on the Renovate PR).

Definition of Done

  • Contract suite green against the htcondor stack CE path with IDTOKENS
  • Fetch-twice test asserts the declared one-shot semantics; leave_in_queue decision recorded
  • isinstance(be, LoadReporter) is False and the suite's capability-skip path exercises it
  • Kill path green; per-id outcomes on partial batches

Alternatives Considered

  • Expressing it as BatchBackend(HTCondorCETransport, HTCondorScheduler) — open ADR question;
    blocked on a transport-level "destructive get" notion. Start monolithic, revisit (share only
    _htcondor functions, never a subtype).
  • GSI/VOMS auth — out of scope; IDTOKENS-first per ADR-002 §4.

Related Issues

Blocked by: # (_htcondor), #. Validated by: #5 / htcondor stack.

Additional Context

Dominant language
Python
Stars
3
Forks
3
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from DIRACGrid/intercede

All issues in DIRACGrid/intercede

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.