[UX BUG] `arkcli +chat` hallucinates tool identity, misleads users

Inizia dal comando +chat e segui il modo in cui il relativo prompt di sistema o le definizioni dei comandi espongono le conoscenze su arkcli. Leggi il comportamento esistente di +connect, profile e config descritto nell’issue, quindi individua i metadati autorevoli disponibili per il modello. Il lavoro è completato quando le domande su arkcli ricevono risposte fondate o una risposta esplicita di incertezza invece di istruzioni di installazione non pertinenti.

Problem

When asked a question about arkcli itself (its own +connect skills system), arkcli +chat returns a confident but entirely fabricated answer — inventing a false identity for the tool and providing completely wrong installation instructions.

Repro

arkcli +chat "how to install arkcli related skills to repo-local not system-wide"

response:

{
  "id": "resp_0217825291156653cf615ac2c98f7e6dbb140b1d7a15eab131a32",
  "model": "auto",
  "content": "To install `arkcli` (Arkansas CLI, a common command-line tool for Kubernetes/Ark resources) and its related skills **locally to your repository** (not system-wide), follow the steps below, depending on how you manage project dependencies:\n\n---\n\n## 1. Prerequisites\nEnsure you have these tools installed on your machine (they remain system-wide, but `arkcli` itself will be local to your repo):\n- `git`\n- `pip` (for Python-based installations, most `arkcli` variants are Python-based)\n- `npm`/`yarn` (if you are using the Node.js version of `arkcli`)\n\n---\n\n## Method 1: Install via `pip` to a local virtual environment (Recommended for Python projects)\nThis keeps all `arkcli` dependencies isolated to your repo, no system-wide changes.\n\n### Step 1: Create a local virtual environment in your repo\nNavigate to your repository root, then create a venv:\n```bash\ncd /path/to/your/repo\npython3 -m venv .venv\n```\n\n### Step 2: Activate the virtual environment\n```bash\n# Linux/macOS\nsource .venv/bin/activate\n\n# Windows (PowerShell)\n.venv\\Scripts\\Activate.ps1\n```\nAfter activation, any `pip install` will only affect this local venv, not your system.\n\n### Step 3: Install `arkcli` and related skills\nIf `arkcli` is published on PyPI:\n```bash\npip install arkcli\n# Install additional skills/plugins (example, adjust package names as needed)\npip install arkcli-skill-1 arkcli-skill-2\n```\n\nIf you are installing `arkcli` from source:\n```bash\n# Clone arkcli to a local directory in your repo (optional)\ngit clone https://github.com/your-org/arkcli.git local-arkcli\n# Install it in editable mode to the venv\npip install -e ./local-arkcli\n```\n\n### Step 4: Add the venv to `.gitignore`\nTo avoid committing the venv to your repo, add this line to `.gitignore`:\n```\n.venv/\nlocal-arkcli/\n```\n\nTo use `arkcli` later, just re-activate the venv before running it.\n\n---\n\n## Method 2: Install to a local `node_modules` (for Node.js projects)\nIf you are working with a JavaScript/Node.js project, install `arkcli` as a local dev dependency:\n\n### Step 1: Navigate to your repo root\n```bash\ncd /path/to/your/repo\n```\n\n### Step 2: Install locally\n```bash\n# Install arkcli and related skills as dev dependencies\nnpm install --save-dev arkcli arkcli-skill-example\n# Or with yarn\nyarn add --dev arkcli arkcli-skill-example\n```\n\nThis installs `arkcli` to `./node_modules/.bin/arkcli`, not system-wide.\n\n### Step 3: Run local `arkcli`\n```bash\n# Call the local binary directly\n./node_modules/.bin/arkcli --help\n\n# Or add a script to package.json for convenience:\n# \"scripts\": { \"arkcli\": \"arkcli\" }\n# Then run:\nnpm run arkcli -- --help\n```\n\n---\n\n## Method 3: Manual installation to a local directory (any project)\nIf you want to directly download the binary/source to your repo and use it from there:\n\n1. Create a local directory (e.g. `./bin`) in your repo to hold `arkcli`:\n   ```bash\n   mkdir -p /path/to/your/repo/bin\n   ```\n\n2. Download or clone `arkcli` into this directory:\n   ```bash\n   git clone https://github.com/heptio/ark-cli.git ./bin/arkcli\n   ```\n\n3. Add the local `bin` directory to your `PATH` *only for your current shell session* or via a project-specific `.envrc` (with [direnv](https://direnv.net/)):\n   ```bash\n   # Add to .envrc (for direnv)\n   export PATH=\"$PWD/bin:$PATH\"\n   ```\n   direnv will automatically load this path when you `cd` into your repo, and unload it when you leave.\n\n4. Add `bin/` to `.gitignore` if you don't want to commit the `arkcli` source/binary.\n\n---\n\n## Verify the installation\nCheck that you are using the local installation:\n```bash\n# For Python venv/Method 1\nwhich arkcli\n# Should output: /path/to/your/repo/.venv/bin/arkcli\n\n# For Node.js/Method 2\nnpx which arkcli\n# Should output: /path/to/your/repo/node_modules/.bin/arkcli\n```",
  "usage": {
    "prompt_tokens": 58,
    "completion_tokens": 1076,
    "total_tokens": 1134
  },
  "object": "response",
  "status": "completed",
  "created_at": 1782529115,
  "store": false,
  "caching": {
    "type": "disabled"
  }
}

The response (summarized):

• Claims arkcli is "Arkansas CLI, a common tool for Kubernetes/Ark resources"
• Recommends pip install arkcli and pip install arkcli-skill-*
• Recommends npm install --save-dev arkcli
• References github.com/heptio/ark-cli (Heptio/ark is the defunct Velero K8s backup project — unrelated)
• Suggests .venv/bin/arkcli, node_modules/.bin/arkcli, ./bin/arkcli as local installs
• Never once mentions arkcli +connect, which is the actual command the user was asking about

What actually happened

The LLM had no knowledge of arkcli (Volcengine ARK CLI), so it:

1. Guessed "ark" = "Arkansas" / "Kubernetes Ark (Velero)"
2. Guessed it was a Python or Node.js package
3. Conflated multiple unrelated open-source projects into a fictional answer
4. Answered an arkcli tooling question without referencing +connect, profile, or any actual arkcli commands

Why this is a bug

A user asking about arkcli tooling presumably trusts the tool to know itself — the CLI should have perfect self-knowledge of its own commands, architecture, and ecosystem.

Expected behavior

arkcli +chat should answer questions about arkcli itself accurately, leveraging its own command definitions, config structure, profile system, etc. If the model lacks built-in knowledge, the system prompt should inject authoritative metadata (e.g., command tree, config schema, FAQ) to ground the answer. A hallucinated answer is worse than saying "I don't know."