Packages uploaded with --directory --use-temp-repository are not replaced when another upload occurs and DOESN'T use the flag.

Start by reproducing the upload sequence using --directory --use-temp-repository followed by a regular --directory or --file upload, then inspect the resulting repository version package listing and location_href values. Done means the second package with the same NEVRA replaces the temporary-repository package and only the expected package is served.

Summary

When uploading a set of packages using --directory and --use-temp-repository, the resulting packages have a different location_href property, which is one of the "uniqueness" constraints on the package. However, when uploading a package in the regular way (either --directory bare or --file), packages with the same NEVRA do not replace these temp-repo side loaded packages. Instead they're added along side the older package.

Therefore the repository version ends up with two versions of the same package, so when that version is published, the older package is displayed and downloaded from the content listing. I'm not sure if it's always the older package, or if there's some other ordering that causes one to clobber the other.

In my case I ran into #1198, and needed to upload anew to get them signed, and found that the packages still weren't signed, despite the packages correct sha256's showing up in the repository version. Only when inspecting carefully did I notice the duplicate records in the repository version.

Steps to reproduce

Build two versions of a package with the same NEVRA, but different checksums. Upload one to a test repository using --directory --use-temp-repository. Then try to replace it with an upload of the other package. Note that you get a new repository version for each step, but that when downloading, the first package is always served. Also note that the second package is present in the repository version package listing. Also note that the packages' location_href values are different. The ones coming from the temporary repository having a reference to that repository in the location href.

Expected behavior

It is expected that a package with the same NEVRA will replace the existing. If location_href is used to determine uniqueness, then packages uploaded using --use-temp-repository should not set a different value for location_href than a package uploaded the traditional way.

Stacktrace/Error log

Compare
"location_href": "kernel-devel-matched-5.14.0-284.30.1.el9_2.fips.0.9.1.x86_64.rpm", vs "location_href": "Packages/eba14ffc-b413-4698-83db-fea3b036db70/e8258450c7780912/kernel-devel-matched-5.14.0-284.30.1.el9_2.fips.0.9.1.x86_64.rpm". I assume that the uuid and other hash comes from the temporary repository.

Pulp and pulp-cli version info

$ pulp --version
Pulp3 Command Line Interface, Version 0.32.0.dev
Plugin Versions:
  common: 0.32.0.dev
$ pulp status
{
  "versions": [
    {
      "component": "core",
      "version": "3.69.0",
      "package": "pulpcore",
      "module": "pulpcore.app",
      "domain_compatible": true
    },
    {
      "component": "ansible",
      "version": "0.23.1",
      "package": "pulp-ansible",
      "module": "pulp_ansible.app",
      "domain_compatible": false
    },
    {
      "component": "container",
      "version": "2.22.0",
      "package": "pulp-container",
      "module": "pulp_container.app",
      "domain_compatible": false
    },
    {
      "component": "deb",
      "version": "3.4.0",
      "package": "pulp_deb",
      "module": "pulp_deb.app",
      "domain_compatible": false
    },
    {
      "component": "maven",
      "version": "0.8.1",
      "package": "pulp-maven",
      "module": "pulp_maven.app",
      "domain_compatible": false
    },
    {
      "component": "ostree",
      "version": "2.4.4",
      "package": "pulp-ostree",
      "module": "pulp_ostree.app",
      "domain_compatible": true
    },
    {
      "component": "python",
      "version": "3.12.5",
      "package": "pulp-python",
      "module": "pulp_python.app",
      "domain_compatible": true
    },
    {
      "component": "rpm",
      "version": "3.27.3.dev",
      "package": "pulp-rpm",
      "module": "pulp_rpm.app",
      "domain_compatible": true
    },
    {
      "component": "certguard",
      "version": "3.69.0",
      "package": "pulpcore",
      "module": "pulp_certguard.app",
      "domain_compatible": true
    },
    {
      "component": "file",
      "version": "3.69.0",
      "package": "pulpcore",
      "module": "pulp_file.app",
      "domain_compatible": true
    }
  ],
  "online_workers": [
    {
      "pulp_href": "/pulp/api/v3/workers/01961626-02e6-7906-a83e-4007fc4f0f0f/",
      "prn": "prn:core.worker:01961626-02e6-7906-a83e-4007fc4f0f0f",
      "pulp_created": "2025-04-08T16:05:12.040975Z",
      "pulp_last_updated": "2025-04-08T16:05:12.041042Z",
      "name": "1@pulp-worker-757fcbb6df-6c8jw",
      "last_heartbeat": "2025-05-22T04:57:44.073499Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.3.dev",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      },
      "current_task": null
    },
    {
      "pulp_href": "/pulp/api/v3/workers/01961789-2643-7092-89ab-cb46f7609424/",
      "prn": "prn:core.worker:01961789-2643-7092-89ab-cb46f7609424",
      "pulp_created": "2025-04-08T22:33:06.373643Z",
      "pulp_last_updated": "2025-04-08T22:33:06.373661Z",
      "name": "1@pulp-worker-757fcbb6df-jzzqb",
      "last_heartbeat": "2025-05-22T04:57:47.139500Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.3.dev",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      },
      "current_task": null
    }
  ],
  "online_api_apps": [
    {
      "name": "9@pulp-api-5c7dbb4968-kr7wm",
      "last_heartbeat": "2025-05-22T04:57:51.651460Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.3.dev",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      }
    },
    {
      "name": "10@pulp-api-5c7dbb4968-kr7wm",
      "last_heartbeat": "2025-05-22T04:57:51.642939Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.3.dev",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      }
    }
  ],
  "online_content_apps": [
    {
      "name": "10@pulp-content-559495989b-gprv6",
      "last_heartbeat": "2025-05-22T04:57:48.425790Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.2",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      }
    },
    {
      "name": "11@pulp-content-559495989b-gprv6",
      "last_heartbeat": "2025-05-22T04:57:45.453453Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.2",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      }
    }
  ],
  "database_connection": {
    "connected": true
  },
  "redis_connection": {
    "connected": false
  },
  "storage": {
    "total": null,
    "used": 141642827633,
    "free": null
  },
  "content_settings": {
    "content_origin": "https://pulp.example.com",
    "content_path_prefix": "/pulp/content/"
  },
  "domain_enabled": false
}

Additonal context

This is not a typical operation, assuredly, to replace existing packages, however it's an important one, and it should work correctly.