[UX BUG] `arkcli +chat` hallucinates tool identity, misleads users

+chat コマンドから始め、そのシステムプロンプトまたはコマンド定義が arkcli の知識をどのように公開しているかを追跡します。issue に記載されている既存の +connect、profile、config の動作を読み、次にモデルが利用できる信頼できるメタデータを特定します。arkcli に関する質問が、無関係なインストール手順ではなく、根拠のある回答または明示的な不確実性の応答を受け取るようになれば完了です。

Problem

When asked a question about arkcli itself (its own +connect skills system), arkcli +chat returns a confident but entirely fabricated answer — inventing a false identity for the tool and providing completely wrong installation instructions.

Repro

arkcli +chat "how to install arkcli related skills to repo-local not system-wide"

response:

{
  "id": "resp_0217825291156653cf615ac2c98f7e6dbb140b1d7a15eab131a32",
  "model": "auto",
  "content": "To install `arkcli` (Arkansas CLI, a common command-line tool for Kubernetes/Ark resources) and its related skills **locally to your repository** (not system-wide), follow the steps below, depending on how you manage project dependencies:\n\n---\n\n## 1. Prerequisites\nEnsure you have these tools installed on your machine (they remain system-wide, but `arkcli` itself will be local to your repo):\n- `git`\n- `pip` (for Python-based installations, most `arkcli` variants are Python-based)\n- `npm`/`yarn` (if you are using the Node.js version of `arkcli`)\n\n---\n\n## Method 1: Install via `pip` to a local virtual environment (Recommended for Python projects)\nThis keeps all `arkcli` dependencies isolated to your repo, no system-wide changes.\n\n### Step 1: Create a local virtual environment in your repo\nNavigate to your repository root, then create a venv:\n```bash\ncd /path/to/your/repo\npython3 -m venv .venv\n```\n\n### Step 2: Activate the virtual environment\n```bash\n# Linux/macOS\nsource .venv/bin/activate\n\n# Windows (PowerShell)\n.venv\\Scripts\\Activate.ps1\n```\nAfter activation, any `pip install` will only affect this local venv, not your system.\n\n### Step 3: Install `arkcli` and related skills\nIf `arkcli` is published on PyPI:\n```bash\npip install arkcli\n# Install additional skills/plugins (example, adjust package names as needed)\npip install arkcli-skill-1 arkcli-skill-2\n```\n\nIf you are installing `arkcli` from source:\n```bash\n# Clone arkcli to a local directory in your repo (optional)\ngit clone https://github.com/your-org/arkcli.git local-arkcli\n# Install it in editable mode to the venv\npip install -e ./local-arkcli\n```\n\n### Step 4: Add the venv to `.gitignore`\nTo avoid committing the venv to your repo, add this line to `.gitignore`:\n```\n.venv/\nlocal-arkcli/\n```\n\nTo use `arkcli` later, just re-activate the venv before running it.\n\n---\n\n## Method 2: Install to a local `node_modules` (for Node.js projects)\nIf you are working with a JavaScript/Node.js project, install `arkcli` as a local dev dependency:\n\n### Step 1: Navigate to your repo root\n```bash\ncd /path/to/your/repo\n```\n\n### Step 2: Install locally\n```bash\n# Install arkcli and related skills as dev dependencies\nnpm install --save-dev arkcli arkcli-skill-example\n# Or with yarn\nyarn add --dev arkcli arkcli-skill-example\n```\n\nThis installs `arkcli` to `./node_modules/.bin/arkcli`, not system-wide.\n\n### Step 3: Run local `arkcli`\n```bash\n# Call the local binary directly\n./node_modules/.bin/arkcli --help\n\n# Or add a script to package.json for convenience:\n# \"scripts\": { \"arkcli\": \"arkcli\" }\n# Then run:\nnpm run arkcli -- --help\n```\n\n---\n\n## Method 3: Manual installation to a local directory (any project)\nIf you want to directly download the binary/source to your repo and use it from there:\n\n1. Create a local directory (e.g. `./bin`) in your repo to hold `arkcli`:\n   ```bash\n   mkdir -p /path/to/your/repo/bin\n   ```\n\n2. Download or clone `arkcli` into this directory:\n   ```bash\n   git clone https://github.com/heptio/ark-cli.git ./bin/arkcli\n   ```\n\n3. Add the local `bin` directory to your `PATH` *only for your current shell session* or via a project-specific `.envrc` (with [direnv](https://direnv.net/)):\n   ```bash\n   # Add to .envrc (for direnv)\n   export PATH=\"$PWD/bin:$PATH\"\n   ```\n   direnv will automatically load this path when you `cd` into your repo, and unload it when you leave.\n\n4. Add `bin/` to `.gitignore` if you don't want to commit the `arkcli` source/binary.\n\n---\n\n## Verify the installation\nCheck that you are using the local installation:\n```bash\n# For Python venv/Method 1\nwhich arkcli\n# Should output: /path/to/your/repo/.venv/bin/arkcli\n\n# For Node.js/Method 2\nnpx which arkcli\n# Should output: /path/to/your/repo/node_modules/.bin/arkcli\n```",
  "usage": {
    "prompt_tokens": 58,
    "completion_tokens": 1076,
    "total_tokens": 1134
  },
  "object": "response",
  "status": "completed",
  "created_at": 1782529115,
  "store": false,
  "caching": {
    "type": "disabled"
  }
}

The response (summarized):

• Claims arkcli is "Arkansas CLI, a common tool for Kubernetes/Ark resources"
• Recommends pip install arkcli and pip install arkcli-skill-*
• Recommends npm install --save-dev arkcli
• References github.com/heptio/ark-cli (Heptio/ark is the defunct Velero K8s backup project — unrelated)
• Suggests .venv/bin/arkcli, node_modules/.bin/arkcli, ./bin/arkcli as local installs
• Never once mentions arkcli +connect, which is the actual command the user was asking about

What actually happened

The LLM had no knowledge of arkcli (Volcengine ARK CLI), so it:

1. Guessed "ark" = "Arkansas" / "Kubernetes Ark (Velero)"
2. Guessed it was a Python or Node.js package
3. Conflated multiple unrelated open-source projects into a fictional answer
4. Answered an arkcli tooling question without referencing +connect, profile, or any actual arkcli commands

Why this is a bug

A user asking about arkcli tooling presumably trusts the tool to know itself — the CLI should have perfect self-knowledge of its own commands, architecture, and ecosystem.

Expected behavior

arkcli +chat should answer questions about arkcli itself accurately, leveraging its own command definitions, config structure, profile system, etc. If the model lacks built-in knowledge, the system prompt should inject authoritative metadata (e.g., command tree, config schema, FAQ) to ground the answer. A hallucinated answer is worse than saying "I don't know."