Full Tutorial Copy of Jetson Thor Deploying VSS (Video Search and Summarization)

Preparation

Install Docker

Obtain NGC API Key

• Enter https://ngc.nvidia.com/ and log in to your account
• Go to the https://org.ngc.nvidia.com/setup/api-keys page in the user configuration section
• Go to theLegacy API Keypage, clickGenerate Legacy Keyto generate an API Key
• Copy and save the generated API Key as prompted

Obtain Hugging Face Token

• Log in to the Hugging Face platform: visit the URL https://huggingface.co/ , and log in to your account.
• Navigate to the API Key page: Jump to the dedicated API Keys page.
• Generate API Key: As shown in the figure below, click the "Generate New Token" button. Copy the generated token and securely store it in a safe location for future use.

Configure VSS Event Reviewer

• Download Video Search and Summarization
  git clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization.git -b v2.4.1 cd video-search-and-summarization/

• Modify the API Key for the Event Reviewer feature
  cd deploy/docker/event_reviewer vim .env # Update to download Cosmos-Reason2 / other models from HuggingFace HF_TOKEN=hf_*** # Update to download Cosmos-Reason1 from NGC NGC_API_KEY=nvapi-***

• Modify the docker compose configuration (optional)
  ## 添加模型本地缓存 mkdir models # 打开compose.yaml vim compose.yaml
    Modify the compose.yaml file
  services: via-server: image: ${VSS_IMAGE:-nvcr.io/nvidia/blueprint/vss-engine:2.4.1${IS_SBSA:+-sbsa}} networks: - vss-shared-network shm_size: '16gb' runtime: nvidia ports: - "${VSS_BACKEND_PORT:-8000}:${VSS_BACKEND_PORT:-8000}" - "${VSS_FRONTEND_PORT:-9000}:${VSS_FRONTEND_PORT:-9000}" volumes: - "${MODEL_ROOT_DIR:-/dummy}${MODEL_ROOT_DIR:+:${MODEL_ROOT_DIR:-}}" ## 注释以下两行 # - via-ngc-model-cache:/root/.via/ngc_model_cache # - via-hf-cache:/tmp/huggingface ## 添加模型缓存目录 - ./models/via-ngc-model-cache:/root/.via/ngc_model_cache - ./models/via-hf-cache:/tmp/huggingface - "${ALERT_REVIEW_MEDIA_BASE_DIR:-/dummy}${ALERT_REVIEW_MEDIA_BASE_DIR:+:${ALERT_REVIEW_MEDIA_BASE_DIR:-}}" environment: BACKEND_PORT: "${VSS_BACKEND_PORT:-8000}" DISABLE_CA_RAG: "true" DISABLE_FRONTEND: "true" DISABLE_GUARDRAILS: "true" DISABLE_CV_PIPELINE: "true" FRONTEND_PORT: "${VSS_FRONTEND_PORT:-9000}" MODEL_PATH: "${MODEL_PATH:-git:https://huggingface.co/nvidia/Cosmos-Reason2-8B}" VLM_MODEL_TO_USE: "${VLM_MODEL_TO_USE:-cosmos-reason2}" VLLM_GPU_MEMORY_UTILIZATION: "${VLLM_GPU_MEMORY_UTILIZATION:-}" VLM_MAX_MODEL_LEN: "${VLM_MAX_MODEL_LEN:-}" VSS_LOG_LEVEL: "${VSS_LOG_LEVEL:-}" VSS_EXTRA_ARGS: "${VSS_EXTRA_ARGS:-}" VLM_DEFAULT_NUM_FRAMES_PER_CHUNK: "${VLM_DEFAULT_NUM_FRAMES_PER_CHUNK:-}" ALERT_REVIEW_MEDIA_BASE_DIR: "${ALERT_REVIEW_MEDIA_BASE_DIR:-}" VLM_SYSTEM_PROMPT: "You are a helpful assistant. Answer the user's question. " VLM_INPUT_WIDTH: "${VLM_INPUT_WIDTH:-}" VLM_INPUT_HEIGHT: "${VLM_INPUT_HEIGHT:-}" NVIDIA_VISIBLE_DEVICES: "${NVIDIA_VISIBLE_DEVICES:-}" NGC_API_KEY: "${NGC_API_KEY:-}" HF_TOKEN: "${HF_TOKEN:-}" ## 添加科学上网代理（如果需要） HTTP_PROXY: "http://xxx.xxx.xxx.xxx:7890" HTTPS_PROXY: "http://xxx.xxx.xxx.xxx:7890" ulimits: memlock: soft: -1 hard: -1 stack: 67108864 ipc: host healthcheck: test: ["CMD", "curl", "-f", "http://localhost:${VSS_BACKEND_PORT:-8000}/health/live"] interval: 30s timeout: 20s retries: 25 start_period: 90s

• Run VSS Event Reviewer
  # 创建VSS Event Reviewer网络 docker network create vss-shared-network # 如果升级了vss需清空vst数据 rm -rf vst/vst_volume/* # 运行VSS Event Reviewer，包含Alert Bridge, VLM Pipeline, Alert Inspector UI和Video Storage Toolkit功能 ALERT_REVIEW_MEDIA_BASE_DIR=/tmp/alert-media-dir docker compose up -d

  Note:

  1. The first run requires downloading the docker mirroring, and some docker mirroring may fail to download properly. In such cases, you need to manually download the corresponding tar package for loading and modify the name of the docker mirroring currently in use in compose.yaml.

  2. Entering the container will automatically download and load the model
• Close VSS Event Reviewer
  cd deploy/docker/event_reviewer ALERT_REVIEW_MEDIA_BASE_DIR=/tmp/alert-media-dir docker compose down

After invoking the large model, it may cause memory occupation, requiring running commands for cleanup. The executable script is:

sudo sh deploy/scripts/sys_cache_cleaner.sh

Configure cv-event-detector

• Enter the cv-event-detector directory
  # change directory to example of CV detector cd video-search-and-summarization/examples/cv-event-detector

• Modify the docker compose configuration (optional)
  ## 添加模型本地缓存 mkdir models # 打开compose.yaml vim compose.yaml
    Modify the compose.yaml file
  services: nv-cv-event-detector: image: ${NV_CV_EVENT_DETECTOR_IMAGE:-nvcr.io/nvidia/blueprint/nv-cv-event-detector:2.4.0${IS_SBSA:+-sbsa}} networks: - vss-shared-network shm_size: '16gb' runtime: nvidia ports: - "${NV_CV_EVENT_DETECTOR_API_PORT:-23491}:${NV_CV_EVENT_DETECTOR_API_PORT:-23491}" entrypoint: ["/opt/nvidia/nv-cv-event-detector/start_nv_cv_event_detector.sh"] volumes: ## 注释以下两行 # - via-ngc-model-cache:/root/.via/ngc_model_cache # - via-hf-cache:/tmp/huggingface ## 添加模型缓存目录 - ./models/via-ngc-model-cache:/root/.via/ngc_model_cache - ./models/via-hf-cache:/tmp/huggingface - "${ALERT_REVIEW_MEDIA_BASE_DIR:-/dummy}${ALERT_REVIEW_MEDIA_BASE_DIR:+:${ALERT_REVIEW_MEDIA_BASE_DIR:-}}" - cv-input-dir:/tmp - ./gdinoconfig_grpc.txt:/opt/nvidia/nv-cv-event-detector/gdinoconfig_grpc.txt environment: NV_CV_EVENT_DETECTOR_API_PORT: ${NV_CV_EVENT_DETECTOR_API_PORT:-23491} INSTALL_PROPRIETARY_CODECS: "true" USE_GDINO: "${USE_GDINO:-true}" DISABLE_SOM_OVERLAY: "${DISABLE_SOM_OVERLAY:-false}" USE_TRACKER_CONFIG: "${USE_TRACKER_CONFIG:-/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml}" CLIP_CACHE_PRE_EV_TIME: "${CLIP_CACHE_PRE_EV_TIME:-5}" #5s CLIP_CACHE_POST_EV_TIME: "${CLIP_CACHE_POST_EV_TIME:-20}" #25s ENABLE_FILE_STREAMING_MODE: "${ENABLE_FILE_STREAMING_MODE:-false}" NVIDIA_VISIBLE_DEVICES: "${NVIDIA_VISIBLE_DEVICES:-}" LD_LIBRARY_PATH: "/opt/nvidia/deepstream/deepstream/lib:/opt/tritonserver/lib/" ## 添加科学上网代理（如果需要） HTTP_PROXY: "http://xxx.xxx.xxx.xxx:7890" HTTPS_PROXY: "http://xxx.xxx.xxx.xxx:7890" ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65535 hard: 65535 stack: 67108864 ipc: host healthcheck: test: ["CMD", "curl", "-f", "http://localhost:${NV_CV_EVENT_DETECTOR_API_PORT:-23491}/health"] interval: 30s timeout: 20s retries: 50 start_period: 90s

• Run cv-event-detector
  # 运行cv-event-detector，包含cv-ui功能 ALERT_REVIEW_MEDIA_BASE_DIR=/tmp/alert-media-dir docker compose -f compose.yaml -f compose.thor.yaml up -d

  The first run requires downloading the docker mirroring, downloading, and loading the model

• Turn off cv-event-detector
  # change directory to example of CV detector cd examples/cv-event-detector ALERT_REVIEW_MEDIA_BASE_DIR=/tmp/alert-media-dir docker compose down

Enter Inspector WebUI

Set up the video pipeline

1. Open the browser, enter http://0.0.0.0:7862, and access the Computer Vision Pipeline Manager page
2. Select a video file from Examples to modify Detection Parameters and Alert Prompts
3. Click Process Video to start processing the video

View Alert Results

1. Enter http://0.0.0.0:7860 in the browser to access the Alert Inspector page
2. Select the video stream you want to view in the DASHBOARD, click the Chat icon on the far right of the entry to enter the CHAT function and play the video
3. You can enter a question in the chat box and click Ask to obtain the results of VLM