dsrptlab/pickle_vision
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove Dagster pipeline and redesign calibration flow UI

Browse Source
This commit is contained in:
Ruslan Bakiev
2026-03-26 09:28:49 +07:00
parent f14249dec9
commit a3b57c5742
21 changed files with 455 additions and 2215 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
Dockerfile
View File

@@ -26,14 +26,14 @@ RUN pip install --no-cache-dir -r requirements.txt
# Copy project files
COPY . .
# Create data directory
RUN mkdir -p data/dagster_home
# Create data/config directories
RUN mkdir -p data jetson/config
# Add /app to Python path so 'src' module can be imported
ENV PYTHONPATH=/app:$PYTHONPATH
# Expose ports for API (8000) and Dagster UI (3000)
EXPOSE 8000 3000
# Expose web UI/API port
EXPOSE 8080
# Default command - start Dagster webserver
CMD ["dagster", "dev", "-m", "dagster_project", "--host", "0.0.0.0", "--port", "3000"]
# Default command - run Jetson referee web app
CMD ["python3", "jetson/main.py", "--port", "8080"]

183
README.md
View File

@@ -1,151 +1,72 @@
# 🎾 Pickle - Pickleball Ball Tracking
# Pickle Vision
Система трекинга пикабольного мяча с автоматической детекцией корта и преобразованием координат в метры.
Real-time referee system for pickleball with 2 CSI cameras on Jetson.
## Что делает
## Current Product Scope
1. **Детекция корта** - автоматический поиск 4 углов корта (Roboflow модель)
2. **Детекция мяча** - поиск мяча на каждом кадре (YOLO v8)
3. **Трансформация координат** - преобразование пикселей в метры (homography)
4. **Визуализация** - видео с траекторией, графики, тепловая карта
Three tabs in the web UI:
## Быстрый старт
1. `Camera` - live feeds from both cameras.
2. `Calibration` - step-by-step court calibration flow (ROI -> lines -> intersections -> template match -> camera pose).
3. `Trajectory` - 3D court + ball trajectory + VAR overlay.
Dagster pipeline has been removed from active project runtime.
## Run
### Local (Jetson)
```bash
# Запуск
docker-compose up -d
# Открыть Dagster UI
open http://localhost:3000
# Запустить пайплайн
docker exec pickle-dagster dagster asset materialize --select '*' -m dagster_project
python3 jetson/main.py --port 8080
```
## Структура пайплайна
Open: `http://<jetson-ip>:8080`
```
1. extract_video_frames → Извлекает 100 кадров (с 10-й секунды)
↓
2. detect_court_keypoints → Находит 4 угла корта
↓ ↓
3. detect_ball_positions ←┘ Детектит мяч на всех кадрах
↓
4. compute_2d_coordinates → Преобразует пиксели в метры
↓
5. visualize_trajectory → Создает визуализации
```
## Результаты
После выполнения пайплайна в `data/`:
- **extract_video_frames.json** - метаданные видео
- **detect_court_keypoints.json** - координаты углов корта
- **detect_ball_positions.json** - позиции мяча в пикселях
- **compute_2d_coordinates.json** - позиции мяча в метрах
- **visualization.mp4** - видео с траекторией, кортом и координатами
- **frames/** - извлеченные кадры
- **ball_detections/** - кадры с найденным мячом
- **court_detection_preview.jpg** - превью с найденными углами корта
## Структура проекта
```
pickle/
├── dagster_project/ # Dagster пайплайн
│ ├── assets/ # 5 asset'ов пайплайна
│ └── io_managers/ # JSON IO manager
├── src/ # Основной код
│ ├── ball_detector.py # YOLO детекция
│ ├── court_calibrator.py # Калибровка корта
│ ├── ball_tracker.py # Трекинг
│ └── video_processor.py # Обработка видео
├── data/ # Результаты выполнения
├── DJI_0017.MP4 # Видео для обработки
├── docker-compose.yml
└── Dockerfile
```
## Конфигурация
Параметры в `dagster_project/assets/`:
- **video_extraction.py** - `start_sec=10`, `num_frames=100`
- **ball_detection.py** - `confidence_threshold=0.3`, slicing 320x320
- **coordinate_transform.py** - корт 13.4м × 6.1м
## Модели
- **Корт**: `ping-pong-paddle-ai-with-images/pickleball-court-p3chl-7tufp` (Roboflow)
- **Мяч**: `pickleball-detection-1oqlw/1` (Roboflow) → fallback на YOLOv8n
## Требования
- Docker & Docker Compose
- 4GB+ RAM
- Видео файл `DJI_0017.MP4` в корне проекта
## Docker команды
### Docker
```bash
# Билд и запуск
docker-compose up --build -d
# Логи
docker-compose logs -f
# Остановка
docker-compose down
# Выполнить пайплайн
docker exec pickle-dagster dagster asset materialize --select '*' -m dagster_project
# Выполнить один asset
docker exec pickle-dagster dagster asset materialize --select 'detect_ball_positions' -m dagster_project
docker-compose up --build
```
## Dagster UI
Open: `http://localhost:8080`
http://localhost:3000
## Calibration Flow
Показывает:
- Граф зависимостей между assets
- Логи выполнения
- История запусков
- Метаданные результатов
Calibration is launched from the `Calibration` tab with `Run Calibration Flow`.
## Формат данных
Per camera, UI shows each step status:
**compute_2d_coordinates.json**:
```json
[
{
"frame": 6,
"timestamp": 0.2,
"pixel_x": 1234.5,
"pixel_y": 678.9,
"x_m": 5.67,
"y_m": 2.34,
"confidence": 0.85
}
]
1. Court ROI (green mask)
2. White line segments detection
3. Segment merge into court lines
4. Line intersections
5. Template point match
6. Camera pose solve (PnP)
7. Geometry overlay
Output is stored in:
- `jetson/config/cam0_calibration.json`
- `jetson/config/cam1_calibration.json`
## Main Runtime Files
- `jetson/main.py` - dual-camera loop, calibration, detection, VAR events
- `src/web/app.py` - Flask API + tab endpoints
- `src/web/templates/index.html` - 3-tab UI
- `src/calibration/camera_calibrator.py` - camera geometry / projection
- `src/physics/trajectory.py` - trajectory model
- `src/physics/event_detector.py` - close-call logic
## Dependencies
Install from `requirements.txt`.
```bash
pip install -r requirements.txt
```
## Производительность
## Notes
- Извлечение кадров: ~1 сек
- Детекция корта: ~1 сек
- Детекция мяча: ~6 сек (100 кадров, ~15 FPS)
- Трансформация координат: <1 сек
- Визуализация: ~1 сек
**Итого**: ~10 секунд на 100 кадров видео
## Стоимость
**$0** - всё работает локально в Docker, без облачных API
## License
MIT
- Calibration is required before trajectory/VAR logic becomes fully useful.
- Each camera is calibrated independently against known half-court geometry.

32
dagster_project/__init__.py
View File

@@ -1,32 +0,0 @@
"""
Dagster project for pickleball ball tracking with 3D coordinates
"""
from dagster import Definitions
from dagster_project.assets import (
extract_video_frames,
detect_court_keypoints,
detect_ball_positions,
visualize_ball_on_court,
detect_net,
calibrate_camera_3d,
compute_ball_3d_coordinates,
create_interactive_viewer
)
from dagster_project.io_managers.json_io_manager import json_io_manager
defs = Definitions(
assets=[
extract_video_frames,
detect_court_keypoints,
detect_ball_positions,
visualize_ball_on_court,
detect_net,
calibrate_camera_3d,
compute_ball_3d_coordinates,
create_interactive_viewer
],
resources={
"json_io_manager": json_io_manager
}
)

21
dagster_project/assets/__init__.py
View File

@@ -1,21 +0,0 @@
"""Dagster assets for pickleball tracking pipeline"""
from dagster_project.assets.video_extraction import extract_video_frames
from dagster_project.assets.court_detection import detect_court_keypoints
from dagster_project.assets.ball_detection import detect_ball_positions
from dagster_project.assets.visualization import visualize_ball_on_court
from dagster_project.assets.net_detection import detect_net
from dagster_project.assets.camera_calibration import calibrate_camera_3d
from dagster_project.assets.coordinate_transform_3d import compute_ball_3d_coordinates
from dagster_project.assets.interactive_viewer import create_interactive_viewer
__all__ = [
"extract_video_frames",
"detect_court_keypoints",
"detect_ball_positions",
"visualize_ball_on_court",
"detect_net",
"calibrate_camera_3d",
"compute_ball_3d_coordinates",
"create_interactive_viewer"
]

181
dagster_project/assets/ball_detection.py
View File

@@ -1,181 +0,0 @@
"""Asset 3: Detect ball positions using YOLO"""
import sys
import cv2
from pathlib import Path
from typing import Dict, List, Optional
from dagster import asset, AssetExecutionContext
from tqdm import tqdm
# Add src to path to import existing ball detector
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
@asset(
io_manager_key="json_io_manager",
compute_kind="yolo",
description="Detect ball positions on all frames using YOLO"
)
def detect_ball_positions(
context: AssetExecutionContext,
extract_video_frames: Dict
) -> List[Dict]:
"""
Detect ball positions on all extracted frames
Inputs:
- extract_video_frames: metadata from frame extraction
- data/frames/*.jpg: all extracted frames
Outputs:
- data/detect_ball_positions.json
Returns:
List of dicts with:
- frame: frame number
- x: pixel x coordinate (or None if not detected)
- y: pixel y coordinate (or None if not detected)
- confidence: detection confidence (0-1)
- diameter_px: estimated ball diameter in pixels
"""
from src.ball_detector import BallDetector
frames_dir = Path(extract_video_frames['frames_dir'])
num_frames = extract_video_frames['num_frames']
context.log.info(f"Initializing YOLO ball detector...")
# Initialize detector
detector = BallDetector(
model_id="pickleball-moving-ball/5",
confidence_threshold=0.3, # Lower threshold to catch more detections
slice_enabled=False # Disable slicing for faster Hosted API inference
)
context.log.info(f"Processing {num_frames} frames for ball detection...")
detections = []
frames_with_ball = 0
for i in tqdm(range(num_frames), desc="Detecting ball"):
frame_path = frames_dir / f"frame_{i:04d}.jpg"
if not frame_path.exists():
context.log.warning(f"Frame {i} not found: {frame_path}")
detections.append({
"frame": i,
"x": None,
"y": None,
"confidence": 0.0,
"diameter_px": None,
"bbox": None
})
continue
# Load frame
frame = cv2.imread(str(frame_path))
# Detect ball
results = detector.detect(frame)
if results and len(results) > 0:
# Take highest confidence detection
ball = results[0]
# Calculate diameter from bbox
bbox = ball.get('bbox')
diameter_px = None
if bbox:
width = bbox[2] - bbox[0]
height = bbox[3] - bbox[1]
diameter_px = (width + height) / 2
detections.append({
"frame": i,
"x": float(ball['center'][0]),
"y": float(ball['center'][1]),
"confidence": float(ball['confidence']),
"diameter_px": float(diameter_px) if diameter_px else None,
"bbox": [float(b) for b in bbox] if bbox else None
})
frames_with_ball += 1
else:
# No detection
detections.append({
"frame": i,
"x": None,
"y": None,
"confidence": 0.0,
"diameter_px": None,
"bbox": None
})
# Log progress every 20 frames
if (i + 1) % 20 == 0:
detection_rate = frames_with_ball / (i + 1) * 100
context.log.info(f"Processed {i + 1}/{num_frames} frames. Detection rate: {detection_rate:.1f}%")
detection_rate = frames_with_ball / num_frames * 100
context.log.info(f"✓ Ball detected in {frames_with_ball}/{num_frames} frames ({detection_rate:.1f}%)")
# Save ALL detection images
_save_detection_preview(context, frames_dir, detections, num_preview=999)
return detections
def _save_detection_preview(
context: AssetExecutionContext,
frames_dir: Path,
detections: List[Dict],
num_preview: int = 5
):
"""Save preview images showing ball detections"""
run_id = context.run_id
preview_dir = Path(f"data/{run_id}/ball_detections")
preview_dir.mkdir(parents=True, exist_ok=True)
# Find first N frames with detections
detected_frames = [d for d in detections if d['x'] is not None][:num_preview]
for detection in detected_frames:
frame_num = detection['frame']
frame_path = frames_dir / f"frame_{frame_num:04d}.jpg"
if not frame_path.exists():
continue
frame = cv2.imread(str(frame_path))
# Draw ball
x, y = int(detection['x']), int(detection['y'])
cv2.circle(frame, (x, y), 8, (0, 0, 255), -1)
# Draw bbox if available
if detection['bbox']:
bbox = detection['bbox']
cv2.rectangle(
frame,
(int(bbox[0]), int(bbox[1])),
(int(bbox[2]), int(bbox[3])),
(0, 255, 0),
2
)
# Draw confidence
cv2.putText(
frame,
f"Conf: {detection['confidence']:.2f}",
(x + 15, y - 10),
cv2.FONT_HERSHEY_SIMPLEX,
0.6,
(255, 255, 255),
2
)
# Save preview
preview_path = preview_dir / f"detection_frame_{frame_num:04d}.jpg"
cv2.imwrite(str(preview_path), frame)
context.log.info(f"Saved {len(detected_frames)} preview images to {preview_dir}")

143
dagster_project/assets/camera_calibration.py
View File

@@ -1,143 +0,0 @@
"""Asset: Calibrate camera using court corners and net position"""
import cv2
import numpy as np
from typing import Dict
from dagster import asset, AssetExecutionContext
@asset(
io_manager_key="json_io_manager",
compute_kind="opencv",
description="Calibrate camera using cv2.solvePnP with court corners and net"
)
def calibrate_camera_3d(
context: AssetExecutionContext,
detect_court_keypoints: Dict,
detect_net: Dict
) -> Dict:
"""
Calibrate camera to get 3D pose using known 3D↔2D point correspondences
Inputs:
- detect_court_keypoints: 4 court corners in pixels
- detect_net: 4 net corners in pixels
Known 3D points:
- Court: 13.4m × 6.1m rectangle at Z=0
- Net: height 0.914m at middle of court (Y=3.05m)
Outputs:
Camera calibration parameters
Returns:
Dict with:
- camera_matrix: [[fx, 0, cx], [0, fy, cy], [0, 0, 1]]
- rotation_vector: [rx, ry, rz]
- translation_vector: [tx, ty, tz]
- rotation_matrix: 3x3 matrix
- reprojection_error: RMS error in pixels
- calibrated: success flag
"""
# Get 2D points (pixels)
court_corners = np.array(detect_court_keypoints['corners_pixel'], dtype=np.float32)
net_corners = np.array(detect_net['net_corners_pixel'], dtype=np.float32)
# Define 3D points (meters) in world coordinates
# Court corners (Z=0, on ground)
court_3d = np.array([
[0, 0, 0], # TL
[13.4, 0, 0], # TR
[13.4, 6.1, 0], # BR
[0, 6.1, 0] # BL
], dtype=np.float32)
# Net endpoints (2 points)
# Net is at Y=3.05m (middle of 6.1m court width)
# We have 2 endpoints: left and right side at top of net
net_3d = np.array([
[0, 3.05, 0.914], # Left endpoint (top of net)
[13.4, 3.05, 0.914], # Right endpoint (top of net)
], dtype=np.float32)
# Combine all 3D and 2D points
object_points = np.vstack([court_3d, net_3d]) # 6 points total (4 court + 2 net)
image_points = np.vstack([court_corners, net_corners])
context.log.info(f"Calibrating with {len(object_points)} point correspondences")
context.log.info(f"3D points shape: {object_points.shape}")
context.log.info(f"2D points shape: {image_points.shape}")
# Initial camera matrix estimate
# Assume principal point at image center
w = detect_court_keypoints['frame_width']
h = detect_court_keypoints['frame_height']
cx = w / 2
cy = h / 2
# Estimate focal length (typical for drone/action cameras)
focal_length = max(w, h) # Initial guess
camera_matrix = np.array([
[focal_length, 0, cx],
[0, focal_length, cy],
[0, 0, 1]
], dtype=np.float32)
# No lens distortion (assume corrected or minimal)
dist_coeffs = None
# Solve PnP to get camera pose
try:
success, rotation_vec, translation_vec = cv2.solvePnP(
object_points,
image_points,
camera_matrix,
dist_coeffs,
flags=cv2.SOLVEPNP_ITERATIVE
)
if not success:
context.log.error("cv2.solvePnP failed")
return {
"calibrated": False,
"error": "solvePnP failed"
}
# Convert rotation vector to rotation matrix
rotation_matrix, _ = cv2.Rodrigues(rotation_vec)
# Calculate reprojection error
projected_points, _ = cv2.projectPoints(
object_points,
rotation_vec,
translation_vec,
camera_matrix,
dist_coeffs
)
projected_points = projected_points.reshape(-1, 2)
reprojection_error = np.sqrt(np.mean((image_points - projected_points) ** 2))
context.log.info(f"✓ Camera calibration successful")
context.log.info(f" Reprojection error: {reprojection_error:.2f} pixels")
context.log.info(f" Focal length: {focal_length:.1f}")
context.log.info(f" Rotation vector: {rotation_vec.flatten()}")
context.log.info(f" Translation vector: {translation_vec.flatten()}")
return {
"camera_matrix": camera_matrix.tolist(),
"rotation_vector": rotation_vec.flatten().tolist(),
"translation_vector": translation_vec.flatten().tolist(),
"rotation_matrix": rotation_matrix.tolist(),
"reprojection_error": float(reprojection_error),
"focal_length": float(focal_length),
"principal_point": [float(cx), float(cy)],
"image_size": [w, h],
"calibrated": True
}
except Exception as e:
context.log.error(f"Error during camera calibration: {e}")
return {
"calibrated": False,
"error": str(e)
}

130
dagster_project/assets/coordinate_transform.py
View File

@@ -1,130 +0,0 @@
"""Asset 4: Transform pixel coordinates to real-world 2D coordinates"""
import sys
import numpy as np
from pathlib import Path
from typing import Dict, List
from dagster import asset, AssetExecutionContext
# Add src to path to import existing calibrator
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
@asset(
io_manager_key="json_io_manager",
compute_kind="transform",
description="Transform pixel coordinates to real-world 2D court coordinates using homography"
)
def compute_2d_coordinates(
context: AssetExecutionContext,
detect_court_keypoints: Dict,
detect_ball_positions: List[Dict],
extract_video_frames: Dict
) -> List[Dict]:
"""
Transform ball pixel coordinates to real-world court coordinates
Inputs:
- detect_court_keypoints: court corners in pixels
- detect_ball_positions: ball detections in pixels
- extract_video_frames: video metadata (FPS)
Outputs:
- data/compute_2d_coordinates.json
Returns:
List of dicts with:
- frame: frame number
- timestamp: time in seconds
- pixel_x, pixel_y: pixel coordinates
- x_m, y_m: real-world coordinates in meters
- confidence: detection confidence
"""
from src.court_calibrator import CourtCalibrator
context.log.info("Initializing court calibrator...")
calibrator = CourtCalibrator(
court_width_m=detect_court_keypoints['court_width_m'],
court_length_m=detect_court_keypoints['court_length_m']
)
# Calibrate using detected corners
corners = detect_court_keypoints['corners_pixel']
context.log.info(f"Calibrating with corners: {corners}")
success = calibrator.calibrate_manual(corners)
if not success:
raise RuntimeError("Court calibration failed. Check corner points.")
context.log.info("✓ Court calibration successful")
# Transform all ball positions
fps = extract_video_frames['fps']
trajectory = []
detected_count = 0
for detection in detect_ball_positions:
frame_num = detection['frame']
timestamp = frame_num / fps
if detection['x'] is not None and detection['y'] is not None:
# Transform pixel → meters
pixel_coords = [detection['x'], detection['y']]
real_coords = calibrator.pixel_to_real(pixel_coords)
if real_coords is not None:
trajectory.append({
"frame": frame_num,
"timestamp": round(timestamp, 3),
"pixel_x": round(detection['x'], 2),
"pixel_y": round(detection['y'], 2),
"x_m": round(float(real_coords[0]), 3),
"y_m": round(float(real_coords[1]), 3),
"confidence": round(detection['confidence'], 3)
})
detected_count += 1
else:
# Transformation failed (point outside court?)
trajectory.append({
"frame": frame_num,
"timestamp": round(timestamp, 3),
"pixel_x": round(detection['x'], 2),
"pixel_y": round(detection['y'], 2),
"x_m": None,
"y_m": None,
"confidence": round(detection['confidence'], 3)
})
else:
# No detection
trajectory.append({
"frame": frame_num,
"timestamp": round(timestamp, 3),
"pixel_x": None,
"pixel_y": None,
"x_m": None,
"y_m": None,
"confidence": 0.0
})
# Log progress
if (frame_num + 1) % 20 == 0:
context.log.info(f"Transformed {frame_num + 1}/{len(detect_ball_positions)} positions")
transform_rate = detected_count / len(detect_ball_positions) * 100
context.log.info(f"✓ Transformed {detected_count}/{len(detect_ball_positions)} positions ({transform_rate:.1f}%)")
# Calculate statistics
valid_positions = [t for t in trajectory if t['x_m'] is not None]
if valid_positions:
x_coords = [t['x_m'] for t in valid_positions]
y_coords = [t['y_m'] for t in valid_positions]
context.log.info(f"Position statistics:")
context.log.info(f" X range: {min(x_coords):.2f}m to {max(x_coords):.2f}m")
context.log.info(f" Y range: {min(y_coords):.2f}m to {max(y_coords):.2f}m")
context.log.info(f" Court dimensions: {calibrator.court_length}m x {calibrator.court_width}m")
return trajectory

190
dagster_project/assets/coordinate_transform_3d.py
View File

@@ -1,190 +0,0 @@
"""Asset: Transform ball positions to 3D coordinates (X, Y, Z) in meters"""
import cv2
import numpy as np
from typing import Dict, List
from dagster import asset, AssetExecutionContext
@asset(
io_manager_key="json_io_manager",
compute_kind="opencv",
description="Compute 3D ball coordinates (X, Y, Z) using camera calibration"
)
def compute_ball_3d_coordinates(
context: AssetExecutionContext,
detect_court_keypoints: Dict,
detect_ball_positions: List[Dict],
calibrate_camera_3d: Dict
) -> List[Dict]:
"""
Transform ball pixel coordinates to 3D world coordinates (meters)
Strategy:
- If ball on ground: use homography (Z=0)
- If ball in air: use ray casting + bbox size estimation (Z>0)
Inputs:
- detect_court_keypoints: court corners
- detect_ball_positions: ball positions in pixels
- calibrate_camera_3d: camera calibration
Returns:
List of dicts with 3D coordinates for each frame:
[
{
"frame": 0,
"x_m": float or null, # X position on court (0-13.4m)
"y_m": float or null, # Y position on court (0-6.1m)
"z_m": float or null, # Z height above court (meters)
"on_ground": bool, # True if ball touching court
"confidence": float # Detection confidence
},
...
]
"""
if not calibrate_camera_3d.get('calibrated'):
context.log.error("Camera not calibrated, cannot compute 3D coordinates")
return [{"frame": i, "x_m": None, "y_m": None, "z_m": None, "on_ground": False}
for i in range(len(detect_ball_positions))]
# Extract calibration parameters
camera_matrix = np.array(calibrate_camera_3d['camera_matrix'], dtype=np.float32)
rotation_vec = np.array(calibrate_camera_3d['rotation_vector'], dtype=np.float32)
translation_vec = np.array(calibrate_camera_3d['translation_vector'], dtype=np.float32)
rotation_matrix = np.array(calibrate_camera_3d['rotation_matrix'], dtype=np.float32)
fx, fy = camera_matrix[0, 0], camera_matrix[1, 1]
cx, cy = camera_matrix[0, 2], camera_matrix[1, 2]
focal_length = (fx + fy) / 2
# Build homography for ground plane (Z=0)
court_corners_pixel = np.array(detect_court_keypoints['corners_pixel'], dtype=np.float32)
court_corners_meters = np.array([
[0, 0],
[13.4, 0],
[13.4, 6.1],
[0, 6.1]
], dtype=np.float32)
homography_matrix = cv2.getPerspectiveTransform(court_corners_pixel, court_corners_meters)
# Camera position in world coordinates
camera_position = -rotation_matrix.T @ translation_vec.reshape(3, 1)
context.log.info(f"Processing {len(detect_ball_positions)} frames for 3D coordinate transformation")
results = []
for i, ball_det in enumerate(detect_ball_positions):
if ball_det['x'] is None:
# No ball detected
results.append({
"frame": i,
"x_m": None,
"y_m": None,
"z_m": None,
"on_ground": False,
"confidence": 0.0
})
continue
ball_x = ball_det['x']
ball_y = ball_det['y']
ball_diameter = ball_det['diameter_px']
bbox = ball_det['bbox']
confidence = ball_det['confidence']
# Strategy 1: Try ground plane projection (assume Z=0)
ball_point_2d = np.array([[ball_x, ball_y]], dtype=np.float32)
ball_ground = cv2.perspectiveTransform(ball_point_2d.reshape(-1, 1, 2), homography_matrix)
x_ground, y_ground = ball_ground[0][0]
# Check if ball is likely on ground by comparing bbox size
# If ball bbox is large → likely on ground
# Simple heuristic: if diameter > threshold, assume on ground
on_ground_threshold = 30 # pixels - tune this based on typical ball size on ground
if ball_diameter and ball_diameter > on_ground_threshold:
# Ball likely on ground
results.append({
"frame": i,
"x_m": float(x_ground),
"y_m": float(y_ground),
"z_m": 0.0,
"on_ground": True,
"confidence": float(confidence)
})
else:
# Ball likely in air - use ray casting
try:
# Unproject 2D point to 3D ray
point_2d_normalized = np.array([
(ball_x - cx) / fx,
(ball_y - cy) / fy,
1.0
])
# Ray direction in camera coordinates
ray_camera = point_2d_normalized / np.linalg.norm(point_2d_normalized)
# Transform ray to world coordinates
ray_world = rotation_matrix.T @ ray_camera
# Estimate distance using ball size
# Real pickleball diameter: 74mm = 0.074m
ball_diameter_real = 0.074 # meters
ball_diameter_pixels = ball_diameter if ball_diameter else 20 # fallback to 20px
# Distance formula: D = (real_size * focal_length) / pixel_size
distance = (ball_diameter_real * focal_length) / ball_diameter_pixels
# Compute 3D point along ray
ball_3d = camera_position.flatten() + ray_world * distance
x_3d, y_3d, z_3d = ball_3d
# Sanity checks
if z_3d < 0 or z_3d > 10: # Height should be 0-10m
# Invalid, fallback to ground projection
results.append({
"frame": i,
"x_m": float(x_ground),
"y_m": float(y_ground),
"z_m": 0.0,
"on_ground": True,
"confidence": float(confidence)
})
else:
results.append({
"frame": i,
"x_m": float(x_3d),
"y_m": float(y_3d),
"z_m": float(z_3d),
"on_ground": False,
"confidence": float(confidence)
})
except Exception as e:
context.log.warning(f"Frame {i}: Error computing 3D coords, using ground projection: {e}")
results.append({
"frame": i,
"x_m": float(x_ground),
"y_m": float(y_ground),
"z_m": 0.0,
"on_ground": True,
"confidence": float(confidence)
})
if (i + 1) % 20 == 0:
context.log.info(f"Processed {i + 1}/{len(detect_ball_positions)} frames")
# Statistics
detected_count = sum(1 for r in results if r['x_m'] is not None)
on_ground_count = sum(1 for r in results if r.get('on_ground', False))
in_air_count = detected_count - on_ground_count
context.log.info(f"✓ Computed 3D coordinates: {detected_count} detections")
context.log.info(f" On ground: {on_ground_count}, In air: {in_air_count}")
return results

277
dagster_project/assets/court_detection.py
View File

@@ -1,277 +0,0 @@
"""Asset 2: Detect court keypoints using Roboflow Hosted API"""
import os
import cv2
import numpy as np
from pathlib import Path
from typing import Dict, List
from dagster import asset, AssetExecutionContext
from inference_sdk import InferenceHTTPClient
@asset(
io_manager_key="json_io_manager",
compute_kind="roboflow",
description="Detect pickleball court corners using Roboflow keypoint detection model"
)
def detect_court_keypoints(
context: AssetExecutionContext,
extract_video_frames: Dict
) -> Dict:
"""
Detect court keypoints from first frame using Roboflow model
Inputs:
- extract_video_frames: metadata from frame extraction
- data/frames/frame_0000.jpg: first frame
Outputs:
- data/detect_court_keypoints.json
Returns:
Dict with:
- corners_pixel: list of 4 corner coordinates [[x,y], ...]
- court_width_m: court width in meters (6.1)
- court_length_m: court length in meters (13.4)
- keypoints: all detected keypoints
"""
from inference import get_model
frames_dir = Path(extract_video_frames['frames_dir'])
first_frame_path = frames_dir / "frame_0000.jpg"
context.log.info(f"Loading first frame: {first_frame_path}")
if not first_frame_path.exists():
raise FileNotFoundError(f"First frame not found: {first_frame_path}")
# Load frame
frame = cv2.imread(str(first_frame_path))
h, w = frame.shape[:2]
context.log.info(f"Frame dimensions: {w}x{h}")
# Get API key
api_key = os.getenv("ROBOFLOW_API_KEY")
if not api_key:
context.log.warning("ROBOFLOW_API_KEY not set, using estimated corners")
corners = _estimate_court_corners(w, h)
else:
# Try to detect court using Roboflow Hosted API
try:
context.log.info("Detecting court using Roboflow Hosted API...")
client = InferenceHTTPClient(
api_url="https://serverless.roboflow.com",
api_key=api_key
)
result = client.infer(str(first_frame_path), model_id="pickleball-court-cfyv4/1")
# Extract keypoints from result
all_points = []
if result and 'predictions' in result and len(result['predictions']) > 0:
pred = result['predictions'][0]
if 'points' in pred and len(pred['points']) >= 4:
# Модель возвращает много points (линии корта)
all_points = [[p['x'], p['y']] for p in pred['points']]
context.log.info(f"✓ Detected {len(all_points)} keypoints from court lines")
# Находим 4 угла для калибровки (но не для визуализации)
corners = _extract_court_corners_from_points(all_points, w, h)
context.log.info(f"✓ Extracted 4 corners from keypoints")
else:
context.log.warning("No keypoints in prediction, using estimated corners")
corners = _estimate_court_corners(w, h)
else:
context.log.warning("No predictions from model, using estimated corners")
corners = _estimate_court_corners(w, h)
except Exception as e:
context.log.warning(f"Court detection failed: {e}. Using estimated corners.")
corners = _estimate_court_corners(w, h)
context.log.info(f"Court corners: {corners}")
# Save visualization - рисуем ВСЕ точки и линии от модели
vis_frame = frame.copy()
# Рисуем все точки от модели
if len(all_points) > 0:
context.log.info(f"Drawing {len(all_points)} keypoints on visualization")
# Рисуем все точки
for i, point in enumerate(all_points):
x, y = int(point[0]), int(point[1])
cv2.circle(vis_frame, (x, y), 5, (0, 255, 0), -1)
# Подписываем каждую точку номером
cv2.putText(
vis_frame,
str(i),
(x + 8, y),
cv2.FONT_HERSHEY_SIMPLEX,
0.4,
(255, 255, 0),
1
)
# Соединяем все соседние точки линиями
for i in range(len(all_points) - 1):
p1 = tuple(map(int, all_points[i]))
p2 = tuple(map(int, all_points[i + 1]))
cv2.line(vis_frame, p1, p2, (0, 255, 0), 2)
# Save visualization with run_id
run_id = context.run_id
vis_path = Path(f"data/{run_id}/court_detection_preview.jpg")
cv2.imwrite(str(vis_path), vis_frame)
context.log.info(f"Saved court visualization to {vis_path}")
return {
"corners_pixel": corners,
"court_width_m": 6.1,
"court_length_m": 13.4,
"frame_width": w,
"frame_height": h
}
def _estimate_court_corners(width: int, height: int) -> List[List[float]]:
"""
Estimate court corners based on typical DJI camera position
(camera in corner at angle)
Returns corners in order: [TL, TR, BR, BL]
"""
# Assume court takes up ~80% of frame with perspective
margin_x = width * 0.05
margin_y = height * 0.1
# Perspective: far edge narrower than near edge
return [
[margin_x + width * 0.1, margin_y], # Top-left (far)
[width - margin_x - width * 0.1, margin_y], # Top-right (far)
[width - margin_x, height - margin_y], # Bottom-right (near)
[margin_x, height - margin_y] # Bottom-left (near)
]
def _extract_court_corners_from_points(points: List[List[float]], width: int, height: int) -> List[List[float]]:
"""
Extract 4 court corners from many detected points (court lines)
Strategy:
1. Build convex hull from all points
2. Classify hull points into 4 sides (left, right, top, bottom)
3. Fit line for each side using linear regression
4. Find 4 corners as intersections of fitted lines
This works even if one corner is not visible on frame (extrapolation)
"""
if len(points) < 4:
return _estimate_court_corners(width, height)
# Build convex hull from all points
points_array = np.array(points, dtype=np.float32)
hull = cv2.convexHull(points_array)
hull_points = np.array([p[0] for p in hull], dtype=np.float32)
# Classify hull points into 4 sides
# Strategy: sort hull points by angle from centroid, then split into 4 groups
center = hull_points.mean(axis=0)
# Calculate angle for each point relative to center
angles = np.arctan2(hull_points[:, 1] - center[1], hull_points[:, 0] - center[0])
# Sort points by angle
sorted_indices = np.argsort(angles)
sorted_points = hull_points[sorted_indices]
# Split into 4 groups (4 sides)
n = len(sorted_points)
quarter = n // 4
side1 = sorted_points[0:quarter]
side2 = sorted_points[quarter:2*quarter]
side3 = sorted_points[2*quarter:3*quarter]
side4 = sorted_points[3*quarter:]
# Fit lines for each side using cv2.fitLine
def fit_line_coefficients(pts):
if len(pts) < 2:
return None
# cv2.fitLine returns (vx, vy, x0, y0) - direction vector and point on line
line = cv2.fitLine(pts, cv2.DIST_L2, 0, 0.01, 0.01)
vx, vy, x0, y0 = line[0][0], line[1][0], line[2][0], line[3][0]
# Convert to line equation: y = mx + b or vertical line x = c
if abs(vx) < 1e-6: # Vertical line
return ('vertical', x0)
m = vy / vx
b = y0 - m * x0
return ('normal', m, b)
line1 = fit_line_coefficients(side1)
line2 = fit_line_coefficients(side2)
line3 = fit_line_coefficients(side3)
line4 = fit_line_coefficients(side4)
lines = [line1, line2, line3, line4]
# Find intersections between adjacent sides
def line_intersection(line_a, line_b):
if line_a is None or line_b is None:
return None
# Handle vertical lines
if line_a[0] == 'vertical' and line_b[0] == 'vertical':
return None
elif line_a[0] == 'vertical':
x = line_a[1]
m2, b2 = line_b[1], line_b[2]
y = m2 * x + b2
return [float(x), float(y)]
elif line_b[0] == 'vertical':
x = line_b[1]
m1, b1 = line_a[1], line_a[2]
y = m1 * x + b1
return [float(x), float(y)]
else:
m1, b1 = line_a[1], line_a[2]
m2, b2 = line_b[1], line_b[2]
if abs(m1 - m2) < 1e-6: # Parallel lines
return None
x = (b2 - b1) / (m1 - m2)
y = m1 * x + b1
return [float(x), float(y)]
# Find 4 corners as intersections
corners = []
for i in range(4):
next_i = (i + 1) % 4
corner = line_intersection(lines[i], lines[next_i])
if corner:
corners.append(corner)
# If we got 4 corners, return them
if len(corners) == 4:
return corners
# Fallback: use convex hull extreme points
tl = hull_points[np.argmin(hull_points[:, 0] + hull_points[:, 1])].tolist()
tr = hull_points[np.argmax(hull_points[:, 0] - hull_points[:, 1])].tolist()
br = hull_points[np.argmax(hull_points[:, 0] + hull_points[:, 1])].tolist()
bl = hull_points[np.argmin(hull_points[:, 0] - hull_points[:, 1])].tolist()
return [tl, tr, br, bl]
def _extract_court_corners(keypoints: List[Dict], width: int, height: int) -> List[List[float]]:
"""
Extract 4 court corners from detected keypoints (old function for compatibility)
"""
if len(keypoints) < 4:
return _estimate_court_corners(width, height)
points = [[kp['x'], kp['y']] for kp in keypoints]
return _extract_court_corners_from_points(points, width, height)

665
dagster_project/assets/interactive_viewer.py
View File

@@ -1,665 +0,0 @@
"""Asset: Create interactive HTML viewer for frame-by-frame ball tracking"""
from pathlib import Path
from typing import Dict, List
from dagster import asset, AssetExecutionContext
@asset(
io_manager_key="json_io_manager",
compute_kind="html",
description="Create interactive HTML viewer with frame + 3D court visualization"
)
def create_interactive_viewer(
context: AssetExecutionContext,
extract_video_frames: Dict,
compute_ball_3d_coordinates: List[Dict]
) -> Dict:
"""
Create interactive HTML viewer showing:
- Left: Original video frame
- Right: Interactive 3D court (Three.js - rotatable with mouse)
- Controls: Prev/Next buttons + slider
Outputs:
- data/{run_id}/viewer/index.html
Returns:
Dict with viewer_path
"""
run_id = context.run_id
frames_dir = Path(extract_video_frames['frames_dir'])
# Create viewer directory
viewer_dir = Path(f"data/{run_id}/viewer")
viewer_dir.mkdir(parents=True, exist_ok=True)
# Copy frames to viewer directory
import shutil
viewer_frames_dir = viewer_dir / "frames"
viewer_frames_dir.mkdir(exist_ok=True)
# Filter frames with detections
frames_with_ball = [f for f in compute_ball_3d_coordinates if f['x_m'] is not None]
context.log.info(f"Creating viewer for {len(frames_with_ball)} frames with ball detections")
# Copy only frames with detections
for frame_data in frames_with_ball:
frame_num = frame_data['frame']
src = frames_dir / f"frame_{frame_num:04d}.jpg"
dst = viewer_frames_dir / f"frame_{frame_num:04d}.jpg"
if src.exists():
shutil.copy2(src, dst)
context.log.info(f"Copied {len(frames_with_ball)} frames to viewer directory")
# Generate HTML
html_content = _generate_html(frames_with_ball, run_id)
# Save HTML
html_path = viewer_dir / "index.html"
with open(html_path, 'w') as f:
f.write(html_content)
context.log.info(f"✓ Interactive viewer created: {html_path}")
context.log.info(f" Open in browser: file://{html_path.absolute()}")
return {
"viewer_path": str(html_path),
"num_frames": len(frames_with_ball)
}
def _generate_html(frames_data: List[Dict], run_id: str) -> str:
"""Generate HTML with Three.js for real 3D visualization"""
# Convert frames data to JSON
import json
frames_json = json.dumps(frames_data, indent=2)
html = f"""<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Ball Tracking 3D Viewer - Run {run_id[:8]}</title>
<style>
* {{
margin: 0;
padding: 0;
box-sizing: border-box;
}}
body {{
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
background: #1a1a1a;
color: #ffffff;
padding: 20px;
overflow-x: hidden;
}}
.container {{
max-width: 1900px;
margin: 0 auto;
}}
h1 {{
text-align: center;
margin-bottom: 25px;
font-size: 28px;
color: #00ff88;
}}
.viewer {{
display: grid;
grid-template-columns: 1fr 1fr;
gap: 20px;
margin-bottom: 20px;
height: 600px;
}}
.panel {{
background: #2a2a2a;
border-radius: 8px;
padding: 20px;
box-shadow: 0 4px 6px rgba(0,0,0,0.3);
display: flex;
flex-direction: column;
}}
.panel h2 {{
margin-bottom: 15px;
font-size: 18px;
color: #00ff88;
}}
#frameImage {{
width: 100%;
height: 100%;
object-fit: contain;
border-radius: 4px;
}}
#canvas3d {{
width: 100%;
height: 100%;
border-radius: 4px;
display: block;
}}
.controls {{
background: #2a2a2a;
border-radius: 8px;
padding: 20px;
box-shadow: 0 4px 6px rgba(0,0,0,0.3);
}}
.controls-row {{
display: flex;
align-items: center;
gap: 15px;
margin-bottom: 15px;
}}
button {{
background: #00ff88;
color: #1a1a1a;
border: none;
padding: 10px 24px;
border-radius: 4px;
font-size: 16px;
font-weight: 600;
cursor: pointer;
transition: all 0.2s;
}}
button:hover {{
background: #00cc6a;
transform: translateY(-1px);
}}
button:active {{
transform: translateY(0);
}}
button:disabled {{
background: #444;
color: #888;
cursor: not-allowed;
transform: none;
}}
input[type="range"] {{
flex: 1;
height: 6px;
background: #444;
border-radius: 3px;
outline: none;
-webkit-appearance: none;
}}
input[type="range"]::-webkit-slider-thumb {{
-webkit-appearance: none;
width: 18px;
height: 18px;
background: #00ff88;
border-radius: 50%;
cursor: pointer;
}}
input[type="range"]::-moz-range-thumb {{
width: 18px;
height: 18px;
background: #00ff88;
border-radius: 50%;
cursor: pointer;
border: none;
}}
.info {{
display: grid;
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
gap: 15px;
}}
.info-item {{
background: #1a1a1a;
padding: 12px;
border-radius: 4px;
}}
.info-label {{
font-size: 12px;
color: #888;
margin-bottom: 4px;
}}
.info-value {{
font-size: 20px;
font-weight: 600;
color: #00ff88;
}}
.kbd {{
display: inline-block;
padding: 3px 8px;
background: #444;
border-radius: 3px;
font-family: monospace;
font-size: 14px;
margin: 0 4px;
}}
.help {{
margin-top: 15px;
padding: 10px;
background: #1a1a1a;
border-radius: 4px;
font-size: 14px;
color: #888;
}}
</style>
</head>
<body>
<div class="container">
<h1>🎾 Pickleball Ball Tracking 3D Viewer</h1>
<div class="viewer">
<div class="panel">
<h2>📹 Video Frame</h2>
<img id="frameImage" alt="Video frame">
</div>
<div class="panel">
<h2>🗺️ Interactive 3D Court (drag to rotate, scroll to zoom)</h2>
<div id="canvas3d"></div>
</div>
</div>
<div class="controls">
<div class="controls-row">
<button id="prevBtn">← Prev</button>
<input type="range" id="frameSlider" min="0" max="0" value="0">
<button id="nextBtn">Next →</button>
<button id="playBtn">▶ Play</button>
</div>
<div class="info">
<div class="info-item">
<div class="info-label">Frame</div>
<div class="info-value" id="frameNum">-</div>
</div>
<div class="info-item">
<div class="info-label">X Position (m)</div>
<div class="info-value" id="posX">-</div>
</div>
<div class="info-item">
<div class="info-label">Y Position (m)</div>
<div class="info-value" id="posY">-</div>
</div>
<div class="info-item">
<div class="info-label">Z Height (m)</div>
<div class="info-value" id="posZ">-</div>
</div>
<div class="info-item">
<div class="info-label">Confidence</div>
<div class="info-value" id="confidence">-</div>
</div>
<div class="info-item">
<div class="info-label">On Ground</div>
<div class="info-value" id="onGround">-</div>
</div>
</div>
<div class="help">
💡 <strong>Controls:</strong>
<span class="kbd">←/→</span> Navigate frames
<span class="kbd">Space</span> Play/Pause
<span class="kbd">Mouse drag</span> Rotate 3D view
<span class="kbd">Mouse wheel</span> Zoom
</div>
</div>
</div>
<!-- Three.js from CDN -->
<script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r128/three.min.js"></script>
<script>
// Frame data from Python
const framesData = {frames_json};
let currentIndex = 0;
let isPlaying = false;
let playInterval = null;
// DOM elements
const frameImage = document.getElementById('frameImage');
const canvas3dContainer = document.getElementById('canvas3d');
const prevBtn = document.getElementById('prevBtn');
const nextBtn = document.getElementById('nextBtn');
const playBtn = document.getElementById('playBtn');
const frameSlider = document.getElementById('frameSlider');
const frameNum = document.getElementById('frameNum');
const posX = document.getElementById('posX');
const posY = document.getElementById('posY');
const posZ = document.getElementById('posZ');
const confidence = document.getElementById('confidence');
const onGround = document.getElementById('onGround');
// Initialize
frameSlider.max = framesData.length - 1;
// Setup Three.js scene
const scene = new THREE.Scene();
scene.background = new THREE.Color(0x1a1a1a);
const camera = new THREE.PerspectiveCamera(
60,
canvas3dContainer.clientWidth / canvas3dContainer.clientHeight,
0.1,
1000
);
camera.position.set(15, 12, 15);
camera.lookAt(6.7, 3, 0);
const renderer = new THREE.WebGLRenderer({{ antialias: true }});
renderer.setSize(canvas3dContainer.clientWidth, canvas3dContainer.clientHeight);
canvas3dContainer.appendChild(renderer.domElement);
// Lighting
const ambientLight = new THREE.AmbientLight(0xffffff, 0.6);
scene.add(ambientLight);
const directionalLight = new THREE.DirectionalLight(0xffffff, 0.8);
directionalLight.position.set(10, 20, 10);
scene.add(directionalLight);
// Grid helper
const gridHelper = new THREE.GridHelper(20, 20, 0x444444, 0x222222);
scene.add(gridHelper);
// Court dimensions
const courtLength = 13.4;
const courtWidth = 6.1;
const netHeight = 0.914;
// Court floor (green)
const courtGeometry = new THREE.PlaneGeometry(courtLength, courtWidth);
const courtMaterial = new THREE.MeshStandardMaterial({{
color: 0x00aa44,
side: THREE.DoubleSide,
roughness: 0.8
}});
const court = new THREE.Mesh(courtGeometry, courtMaterial);
court.rotation.x = -Math.PI / 2;
court.position.set(courtLength / 2, 0, courtWidth / 2);
scene.add(court);
// Court boundaries (white lines)
const boundaryMaterial = new THREE.LineBasicMaterial({{ color: 0xffffff, linewidth: 3 }});
const boundaryPoints = [
new THREE.Vector3(0, 0.01, 0),
new THREE.Vector3(courtLength, 0.01, 0),
new THREE.Vector3(courtLength, 0.01, courtWidth),
new THREE.Vector3(0, 0.01, courtWidth),
new THREE.Vector3(0, 0.01, 0)
];
const boundaryGeometry = new THREE.BufferGeometry().setFromPoints(boundaryPoints);
const boundary = new THREE.Line(boundaryGeometry, boundaryMaterial);
scene.add(boundary);
// Net (mesh) - positioned at middle of court LENGTH (X=6.7m), spanning full WIDTH (Z axis)
const netGeometry = new THREE.BoxGeometry(0.05, netHeight, courtWidth);
const netMaterial = new THREE.MeshStandardMaterial({{
color: 0xcccccc,
transparent: true,
opacity: 0.7,
side: THREE.DoubleSide
}});
const net = new THREE.Mesh(netGeometry, netMaterial);
net.position.set(courtLength / 2, netHeight / 2, courtWidth / 2);
scene.add(net);
// Net poles at both ends of the net (Z=0 and Z=courtWidth)
const poleGeometry = new THREE.CylinderGeometry(0.05, 0.05, netHeight, 8);
const poleMaterial = new THREE.MeshStandardMaterial({{ color: 0x666666 }});
const pole1 = new THREE.Mesh(poleGeometry, poleMaterial);
pole1.position.set(courtLength / 2, netHeight / 2, 0);
scene.add(pole1);
const pole2 = new THREE.Mesh(poleGeometry, poleMaterial);
pole2.position.set(courtLength / 2, netHeight / 2, courtWidth);
scene.add(pole2);
// Ball (sphere)
let ballMesh = null;
let ballShadow = null;
let ballLine = null;
function createBall() {{
// Remove old ball if exists
if (ballMesh) scene.remove(ballMesh);
if (ballShadow) scene.remove(ballShadow);
if (ballLine) scene.remove(ballLine);
// Create ball
const ballGeometry = new THREE.SphereGeometry(0.074 / 2, 32, 32);
const ballMaterial = new THREE.MeshStandardMaterial({{
color: 0xffff00,
emissive: 0xff4444,
emissiveIntensity: 0.3,
metalness: 0.5,
roughness: 0.3
}});
ballMesh = new THREE.Mesh(ballGeometry, ballMaterial);
// Shadow (circle on ground)
const shadowGeometry = new THREE.CircleGeometry(0.1, 32);
const shadowMaterial = new THREE.MeshBasicMaterial({{
color: 0x000000,
transparent: true,
opacity: 0.3,
side: THREE.DoubleSide
}});
ballShadow = new THREE.Mesh(shadowGeometry, shadowMaterial);
ballShadow.rotation.x = -Math.PI / 2;
scene.add(ballMesh);
scene.add(ballShadow);
}}
function updateBall(x, y, z) {{
if (!ballMesh) createBall();
ballMesh.position.set(x, z, y);
ballShadow.position.set(x, 0.01, y);
// Draw line from shadow to ball if in air
if (z > 0.1) {{
if (ballLine) scene.remove(ballLine);
const lineMaterial = new THREE.LineBasicMaterial({{
color: 0xffffff,
transparent: true,
opacity: 0.3
}});
const linePoints = [
new THREE.Vector3(x, z, y),
new THREE.Vector3(x, 0, y)
];
const lineGeometry = new THREE.BufferGeometry().setFromPoints(linePoints);
ballLine = new THREE.Line(lineGeometry, lineMaterial);
scene.add(ballLine);
}} else {{
if (ballLine) {{
scene.remove(ballLine);
ballLine = null;
}}
}}
}}
// Mouse controls for camera
let isDragging = false;
let previousMousePosition = {{ x: 0, y: 0 }};
let cameraAngle = {{ theta: Math.PI / 4, phi: Math.PI / 3 }};
let cameraDistance = 22;
canvas3dContainer.addEventListener('mousedown', (e) => {{
isDragging = true;
previousMousePosition = {{ x: e.clientX, y: e.clientY }};
}});
document.addEventListener('mouseup', () => {{
isDragging = false;
}});
canvas3dContainer.addEventListener('mousemove', (e) => {{
if (!isDragging) return;
const deltaX = e.clientX - previousMousePosition.x;
const deltaY = e.clientY - previousMousePosition.y;
cameraAngle.theta += deltaX * 0.01;
cameraAngle.phi = Math.max(0.1, Math.min(Math.PI / 2 - 0.1, cameraAngle.phi - deltaY * 0.01));
previousMousePosition = {{ x: e.clientX, y: e.clientY }};
updateCameraPosition();
}});
canvas3dContainer.addEventListener('wheel', (e) => {{
e.preventDefault();
cameraDistance += e.deltaY * 0.01;
cameraDistance = Math.max(5, Math.min(50, cameraDistance));
updateCameraPosition();
}});
function updateCameraPosition() {{
const centerX = courtLength / 2;
const centerZ = courtWidth / 2;
camera.position.x = centerX + cameraDistance * Math.sin(cameraAngle.phi) * Math.cos(cameraAngle.theta);
camera.position.y = cameraDistance * Math.cos(cameraAngle.phi);
camera.position.z = centerZ + cameraDistance * Math.sin(cameraAngle.phi) * Math.sin(cameraAngle.theta);
camera.lookAt(centerX, 0, centerZ);
}}
// Handle window resize
window.addEventListener('resize', () => {{
camera.aspect = canvas3dContainer.clientWidth / canvas3dContainer.clientHeight;
camera.updateProjectionMatrix();
renderer.setSize(canvas3dContainer.clientWidth, canvas3dContainer.clientHeight);
}});
// Animation loop
function animate() {{
requestAnimationFrame(animate);
renderer.render(scene, camera);
}}
animate();
// Render current frame
function render() {{
if (framesData.length === 0) return;
const frame = framesData[currentIndex];
// Update image - frames are in ./frames/ directory relative to HTML
const framePath = `frames/frame_${{String(frame.frame).padStart(4, '0')}}.jpg`;
frameImage.src = framePath;
// Update info
frameNum.textContent = `${{currentIndex + 1}} / ${{framesData.length}} (Frame #${{frame.frame}})`;
posX.textContent = frame.x_m !== null ? frame.x_m.toFixed(2) : 'N/A';
posY.textContent = frame.y_m !== null ? frame.y_m.toFixed(2) : 'N/A';
posZ.textContent = frame.z_m !== null ? frame.z_m.toFixed(2) : 'N/A';
confidence.textContent = frame.confidence.toFixed(2);
onGround.textContent = frame.on_ground ? '✓ Yes' : '✗ No';
// Update slider
frameSlider.value = currentIndex;
// Update buttons
prevBtn.disabled = currentIndex === 0;
nextBtn.disabled = currentIndex === framesData.length - 1;
// Update ball position in 3D
if (frame.x_m !== null && frame.y_m !== null && frame.z_m !== null) {{
updateBall(frame.x_m, frame.y_m, frame.z_m);
}}
}}
// Navigation functions
function nextFrame() {{
if (currentIndex < framesData.length - 1) {{
currentIndex++;
render();
}}
}}
function prevFrame() {{
if (currentIndex > 0) {{
currentIndex--;
render();
}}
}}
function gotoFrame(index) {{
currentIndex = Math.max(0, Math.min(index, framesData.length - 1));
render();
}}
function togglePlay() {{
if (isPlaying) {{
clearInterval(playInterval);
isPlaying = false;
playBtn.textContent = '▶ Play';
}} else {{
isPlaying = true;
playBtn.textContent = '⏸ Pause';
playInterval = setInterval(() => {{
if (currentIndex < framesData.length - 1) {{
nextFrame();
}} else {{
clearInterval(playInterval);
isPlaying = false;
playBtn.textContent = '▶ Play';
currentIndex = 0;
render();
}}
}}, 100); // 10 fps
}}
}}
// Event listeners
prevBtn.addEventListener('click', prevFrame);
nextBtn.addEventListener('click', nextFrame);
playBtn.addEventListener('click', togglePlay);
frameSlider.addEventListener('input', (e) => gotoFrame(parseInt(e.target.value)));
// Keyboard shortcuts
document.addEventListener('keydown', (e) => {{
switch(e.key) {{
case 'ArrowLeft':
prevFrame();
break;
case 'ArrowRight':
nextFrame();
break;
case ' ':
e.preventDefault();
togglePlay();
break;
}}
}});
// Initial render
render();
</script>
</body>
</html>
"""
return html

72
dagster_project/assets/net_detection.py
View File

@@ -1,72 +0,0 @@
"""Asset: Detect tennis/pickleball net using Roboflow"""
import os
import cv2
import numpy as np
from pathlib import Path
from typing import Dict
from dagster import asset, AssetExecutionContext
from inference_sdk import InferenceHTTPClient
@asset(
io_manager_key="json_io_manager",
compute_kind="roboflow",
description="Detect pickleball/tennis net using Roboflow model"
)
def detect_net(
context: AssetExecutionContext,
extract_video_frames: Dict,
detect_court_keypoints: Dict
) -> Dict:
"""
Detect net on first frame using Roboflow model
NO FALLBACKS - if model doesn't detect net, this will fail
Inputs:
- extract_video_frames: frame metadata
- detect_court_keypoints: court corners (for visualization)
Outputs:
- data/{run_id}/net_detection_preview.jpg: visualization
- JSON with net detection results
Returns:
Dict with net detection data
"""
run_id = context.run_id
frames_dir = Path(extract_video_frames['frames_dir'])
first_frame_path = frames_dir / "frame_0000.jpg"
context.log.info(f"Loading first frame: {first_frame_path}")
# Load frame
frame = cv2.imread(str(first_frame_path))
h, w = frame.shape[:2]
context.log.info(f"Frame dimensions: {w}x{h}")
# Get API key
api_key = os.getenv("ROBOFLOW_API_KEY")
if not api_key:
raise ValueError("ROBOFLOW_API_KEY environment variable is not set")
context.log.info("Detecting net using Roboflow model...")
client = InferenceHTTPClient(
api_url="https://serverless.roboflow.com",
api_key=api_key
)
# Call Roboflow model - MODEL_ID WILL BE PROVIDED BY USER
# Placeholder - user will provide correct model
model_id = "MODEL_ID_PLACEHOLDER"
result = client.infer(str(first_frame_path), model_id=model_id)
context.log.info(f"Roboflow response: {result}")
# TODO: Parse result based on actual model output format
# User will provide correct model and we'll update parsing logic
raise NotImplementedError("Waiting for correct Roboflow model from user")

83
dagster_project/assets/video_extraction.py
View File

@@ -1,83 +0,0 @@
"""Asset 1: Extract frames from video"""
import cv2
from pathlib import Path
from typing import Dict
from dagster import asset, AssetExecutionContext
@asset(
io_manager_key="json_io_manager",
compute_kind="opencv",
description="Extract frames from video starting at specified second"
)
def extract_video_frames(context: AssetExecutionContext) -> Dict:
"""
Extract frames from DJI_0017.MP4 video
Inputs:
- DJI_0017.MP4 (video file in root directory)
Outputs:
- data/frames/frame_XXXX.jpg (100 frames)
- data/extract_video_frames.json (metadata)
Returns:
Dict with:
- frames_dir: path to frames directory
- num_frames: number of extracted frames
- fps: video FPS
- start_frame: starting frame number
"""
# Configuration
video_path = "DJI_0017.MP4"
start_sec = 10
num_frames = 100
context.log.info(f"Opening video: {video_path}")
cap = cv2.VideoCapture(video_path)
if not cap.isOpened():
raise RuntimeError(f"Could not open video: {video_path}")
fps = cap.get(cv2.CAP_PROP_FPS)
total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
start_frame = int(start_sec * fps)
context.log.info(f"Video info: {total_frames} frames, {fps} FPS")
context.log.info(f"Extracting {num_frames} frames starting from frame {start_frame} ({start_sec}s)")
# Create output directory with run_id
run_id = context.run_id
frames_dir = Path(f"data/{run_id}/frames")
frames_dir.mkdir(parents=True, exist_ok=True)
# Set starting position
cap.set(cv2.CAP_PROP_POS_FRAMES, start_frame)
# Extract frames
extracted = 0
for i in range(num_frames):
ret, frame = cap.read()
if not ret:
context.log.warning(f"Could not read frame {i}. Stopping extraction.")
break
frame_path = frames_dir / f"frame_{i:04d}.jpg"
cv2.imwrite(str(frame_path), frame)
extracted += 1
if (i + 1) % 20 == 0:
context.log.info(f"Extracted {i + 1}/{num_frames} frames")
cap.release()
context.log.info(f"✓ Extracted {extracted} frames to {frames_dir}")
return {
"frames_dir": str(frames_dir),
"num_frames": extracted,
"fps": fps,
"start_frame": start_frame,
"start_sec": start_sec
}

90
dagster_project/assets/visualization.py
View File

@@ -1,90 +0,0 @@
"""Asset: Draw court polygon with 4 corners"""
import cv2
import numpy as np
from pathlib import Path
from typing import Dict
from dagster import asset, AssetExecutionContext
@asset(
io_manager_key="json_io_manager",
compute_kind="opencv",
description="Draw court polygon with 4 corners on first frame"
)
def visualize_ball_on_court(
context: AssetExecutionContext,
extract_video_frames: Dict,
detect_court_keypoints: Dict
) -> Dict:
"""
Draw court polygon (4 corners) on first frame
Inputs:
- extract_video_frames: frame metadata
- detect_court_keypoints: 4 court corners with perspective
Outputs:
- One image: data/{run_id}/court_polygon.jpg
Returns:
Dict with:
- image_path: path to saved image
"""
run_id = context.run_id
frames_dir = Path(extract_video_frames['frames_dir'])
# Load first frame
first_frame_path = frames_dir / "frame_0000.jpg"
context.log.info(f"Loading first frame: {first_frame_path}")
if not first_frame_path.exists():
raise FileNotFoundError(f"First frame not found: {first_frame_path}")
frame = cv2.imread(str(first_frame_path))
if frame is None:
raise RuntimeError(f"Failed to load frame: {first_frame_path}")
# Get court corners (4 points with perspective)
corners = detect_court_keypoints['corners_pixel']
court_polygon = np.array(corners, dtype=np.int32)
context.log.info(f"Drawing court polygon with 4 corners: {corners}")
# Draw court polygon (4 corners with perspective)
cv2.polylines(
frame,
[court_polygon],
isClosed=True,
color=(0, 255, 0), # Green
thickness=3
)
# Draw court corners as circles
for i, corner in enumerate(corners):
cv2.circle(
frame,
(int(corner[0]), int(corner[1])),
8,
(0, 255, 255), # Yellow
-1
)
# Label corners
cv2.putText(
frame,
str(i),
(int(corner[0]) + 12, int(corner[1])),
cv2.FONT_HERSHEY_SIMPLEX,
0.6,
(0, 255, 255),
2
)
# Save image
output_path = Path(f"data/{run_id}/court_polygon.jpg")
cv2.imwrite(str(output_path), frame)
context.log.info(f"✓ Saved court polygon visualization to {output_path}")
return {
"image_path": str(output_path)
}

1
dagster_project/io_managers/__init__.py
View File

@@ -1 +0,0 @@
"""IO Managers for Dagster assets"""

70
dagster_project/io_managers/json_io_manager.py
View File

@@ -1,70 +0,0 @@
"""JSON IO Manager for storing asset outputs as JSON files"""
import json
from pathlib import Path
from typing import Any
from dagster import IOManager, io_manager, OutputContext, InputContext
class JSONIOManager(IOManager):
"""IO Manager that stores outputs as JSON files in data/{run_id}/ directory"""
def __init__(self, base_path: str = "data"):
self.base_path = Path(base_path)
self.base_path.mkdir(parents=True, exist_ok=True)
def _get_path(self, context) -> Path:
"""Get file path for asset with run_id subdirectory"""
asset_name = context.asset_key.path[-1]
# For InputContext, try upstream run_id first, fallback to finding latest
if isinstance(context, InputContext):
try:
run_id = context.upstream_output.run_id
except:
# If upstream run_id not available, find the most recent run directory
# that contains this asset (for partial re-runs)
run_dirs = sorted([d for d in self.base_path.iterdir() if d.is_dir()],
key=lambda d: d.stat().st_mtime, reverse=True)
for run_dir in run_dirs:
potential_path = run_dir / f"{asset_name}.json"
if potential_path.exists():
return potential_path
# If not found, use the latest run_dir
run_id = run_dirs[0].name if run_dirs else "unknown"
else:
run_id = context.run_id
# Create run-specific directory
run_dir = self.base_path / run_id
run_dir.mkdir(parents=True, exist_ok=True)
return run_dir / f"{asset_name}.json"
def handle_output(self, context: OutputContext, obj: Any):
"""Save asset output to JSON file"""
file_path = self._get_path(context)
with open(file_path, 'w') as f:
json.dump(obj, f, indent=2)
context.log.info(f"Saved {context.asset_key.path[-1]} to {file_path}")
def load_input(self, context: InputContext) -> Any:
"""Load asset input from JSON file"""
file_path = self._get_path(context)
if not file_path.exists():
raise FileNotFoundError(f"Asset output not found: {file_path}")
with open(file_path, 'r') as f:
obj = json.load(f)
context.log.info(f"Loaded {context.asset_key.path[-1]} from {file_path}")
return obj
@io_manager
def json_io_manager():
"""Factory for JSON IO Manager"""
return JSONIOManager(base_path="data")

28
docker-compose.yml
View File

@@ -1,34 +1,20 @@
version: '3.8'
services:
dagster:
pickle-vision:
build: .
container_name: pickle-dagster
container_name: pickle-vision
ports:
- "3000:3000"
- "8080:8080"
volumes:
# Mount data directory for pipeline outputs (frames, detections, JSON)
# Runtime outputs and configs
- ./data:/app/data
# Mount dagster_home for Dagster metadata (history, logs, storage)
- ./dagster_home:/app/dagster_home
# Mount models directory
- ./jetson/config:/app/jetson/config
- ./models:/app/models
# Mount video file
- ./DJI_0017.MP4:/app/DJI_0017.MP4
# Mount source code for hot reload
- ./dagster_project:/app/dagster_project
- ./src:/app/src
- ./jetson:/app/jetson
environment:
- PYTHONUNBUFFERED=1
- DAGSTER_HOME=/app/dagster_home
- ROBOFLOW_API_KEY=JxrPOJZjb5lwHw0pnxey
restart: unless-stopped
command: dagster dev -m dagster_project --host 0.0.0.0 --port 3000
# Optional: Redis for Celery (if you want to add it later)
# redis:
# image: redis:7-alpine
# container_name: pickle-redis
# ports:
# - "6379:6379"
# restart: unless-stopped
command: python3 jetson/main.py --port 8080

159
jetson/main.py
View File

@@ -34,6 +34,19 @@ _cam_readers = {}
_args = None
def _init_calibration_steps():
"""Initialize ordered calibration step status map for UI."""
return {
'green_mask': {'status': 'pending', 'detail': 'Not started'},
'line_segments': {'status': 'pending', 'detail': 'Not started'},
'merged_lines': {'status': 'pending', 'detail': 'Not started'},
'intersections': {'status': 'pending', 'detail': 'Not started'},
'template_match': {'status': 'pending', 'detail': 'Not started'},
'camera_pose': {'status': 'pending', 'detail': 'Not started'},
'overlay': {'status': 'pending', 'detail': 'Not started'},
}
def auto_calibrate():
"""Calibrate cameras by detecting court line intersections and matching
them to the known pickleball court template.
@@ -51,9 +64,21 @@ def auto_calibrate():
results = {}
for sensor_id, reader in _cam_readers.items():
steps = _init_calibration_steps()
frame = reader.grab()
if frame is None:
results[str(sensor_id)] = {'ok': False, 'error': 'No frame available'}
steps['green_mask'] = {'status': 'fail', 'detail': 'No frame available'}
steps['line_segments'] = {'status': 'fail', 'detail': 'Calibration aborted'}
steps['merged_lines'] = {'status': 'fail', 'detail': 'Calibration aborted'}
steps['intersections'] = {'status': 'fail', 'detail': 'Calibration aborted'}
steps['template_match'] = {'status': 'fail', 'detail': 'Calibration aborted'}
steps['camera_pose'] = {'status': 'fail', 'detail': 'Calibration aborted'}
steps['overlay'] = {'status': 'fail', 'detail': 'Calibration aborted'}
results[str(sensor_id)] = {
'ok': False,
'error': f'CAM {sensor_id}: No frame available',
'steps': steps,
}
continue
h, w = frame.shape[:2]
@@ -68,6 +93,13 @@ def auto_calibrate():
green_pct = np.count_nonzero(green_mask) / (w * h) * 100
if green_pct < 5:
steps['green_mask'] = {'status': 'fail', 'detail': f'Green area too small: {green_pct:.1f}%'}
steps['line_segments'] = {'status': 'fail', 'detail': 'Calibration aborted'}
steps['merged_lines'] = {'status': 'fail', 'detail': 'Calibration aborted'}
steps['intersections'] = {'status': 'fail', 'detail': 'Calibration aborted'}
steps['template_match'] = {'status': 'fail', 'detail': 'Calibration aborted'}
steps['camera_pose'] = {'status': 'fail', 'detail': 'Calibration aborted'}
steps['overlay'] = {'status': 'fail', 'detail': 'Calibration aborted'}
cv2.putText(debug_frame, f"FAILED: Green area too small ({green_pct:.0f}%)",
(10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 0, 255), 2)
_, jpeg = cv2.imencode('.jpg', debug_frame, [cv2.IMWRITE_JPEG_QUALITY, 85])
@@ -75,13 +107,19 @@ def auto_calibrate():
'ok': False,
'error': f'CAM {sensor_id}: Green area too small ({green_pct:.0f}%)',
'debug_image': base64.b64encode(jpeg.tobytes()).decode('ascii'),
'steps': steps,
}
continue
steps['green_mask'] = {'status': 'ok', 'detail': f'Green area: {green_pct:.1f}%'}
# Step 2: Detect white line segments on court
white_segments = _detect_white_lines_on_court(frame, green_mask)
for x1, y1, x2, y2 in white_segments:
cv2.line(debug_frame, (x1, y1), (x2, y2), (255, 255, 0), 1)
if white_segments:
steps['line_segments'] = {'status': 'ok', 'detail': f'{len(white_segments)} segments detected'}
else:
steps['line_segments'] = {'status': 'fail', 'detail': '0 segments detected'}
# Step 3: Merge into distinct lines
merged = _merge_line_segments(white_segments)
@@ -89,28 +127,55 @@ def auto_calibrate():
p1, p2 = m['p1'], m['p2']
cv2.line(debug_frame, (int(p1[0]), int(p1[1])),
(int(p2[0]), int(p2[1])), (0, 165, 255), 2)
if merged:
steps['merged_lines'] = {'status': 'ok', 'detail': f'{len(merged)} merged lines'}
else:
steps['merged_lines'] = {'status': 'fail', 'detail': '0 merged lines'}
# Step 4: Find intersections
intersections = _find_line_intersections(merged, w, h)
for ix, iy in intersections:
cv2.circle(debug_frame, (int(ix), int(iy)), 6, (0, 0, 255), -1)
if intersections:
steps['intersections'] = {'status': 'ok', 'detail': f'{len(intersections)} intersections'}
else:
steps['intersections'] = {'status': 'fail', 'detail': '0 intersections'}
cv2.putText(debug_frame,
f"{len(white_segments)} segs -> {len(merged)} lines -> {len(intersections)} pts",
(10, h - 15), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 255), 1)
template = get_half_court_intersections(side)
quad = _find_green_quad(green_mask, w * h)
if len(intersections) < 4:
# Fallback: try green quad + both mappings
quad = _find_green_quad(green_mask, w * h)
if quad is not None:
cal, cam_pos, mapping_info = _calibrate_from_quad(quad, side, w, h)
if cal is not None:
steps['template_match'] = {
'status': 'ok',
'detail': 'Low intersections, used quad fallback mapping'
}
_finalize_calibration(
cal, cam_pos, sensor_id, side, debug_frame, w, h,
results, mapping_info, len(intersections), len(merged)
cal=cal,
cam_pos=cam_pos,
sensor_id=sensor_id,
side=side,
debug_frame=debug_frame,
results=results,
method_info=mapping_info,
steps=steps,
n_segments=len(white_segments),
n_lines=len(merged),
n_intersections=len(intersections),
n_points_matched=0,
)
continue
steps['template_match'] = {'status': 'fail', 'detail': 'Not enough intersections for template fit'}
steps['camera_pose'] = {'status': 'fail', 'detail': 'solvePnP skipped'}
steps['overlay'] = {'status': 'fail', 'detail': 'No calibration output'}
cv2.putText(debug_frame, f"FAILED: Need 4+ intersections, got {len(intersections)}",
(10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 0, 255), 2)
_, jpeg = cv2.imencode('.jpg', debug_frame, [cv2.IMWRITE_JPEG_QUALITY, 85])
@@ -118,14 +183,11 @@ def auto_calibrate():
'ok': False,
'error': f'CAM {sensor_id}: {len(intersections)} intersections (need 4+) from {len(merged)} lines',
'debug_image': base64.b64encode(jpeg.tobytes()).decode('ascii'),
'steps': steps,
}
continue
# Step 5: Match intersections to court template
template = get_half_court_intersections(side)
# Try matching with green quad seeded homography
quad = _find_green_quad(green_mask, w * h)
match = _match_intersections_to_template(
intersections, template, side, quad, w, h
)
@@ -135,12 +197,29 @@ def auto_calibrate():
if quad is not None:
cal, cam_pos, mapping_info = _calibrate_from_quad(quad, side, w, h)
if cal is not None:
steps['template_match'] = {
'status': 'ok',
'detail': 'Template match weak, used quad fallback mapping'
}
_finalize_calibration(
cal, cam_pos, sensor_id, side, debug_frame, w, h,
results, mapping_info, len(intersections), len(merged)
cal=cal,
cam_pos=cam_pos,
sensor_id=sensor_id,
side=side,
debug_frame=debug_frame,
results=results,
method_info=mapping_info,
steps=steps,
n_segments=len(white_segments),
n_lines=len(merged),
n_intersections=len(intersections),
n_points_matched=0,
)
continue
steps['template_match'] = {'status': 'fail', 'detail': 'Could not match intersections to template'}
steps['camera_pose'] = {'status': 'fail', 'detail': 'solvePnP skipped'}
steps['overlay'] = {'status': 'fail', 'detail': 'No calibration output'}
cv2.putText(debug_frame, "FAILED: Could not match court pattern",
(10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 0, 255), 2)
_, jpeg = cv2.imencode('.jpg', debug_frame, [cv2.IMWRITE_JPEG_QUALITY, 85])
@@ -148,9 +227,16 @@ def auto_calibrate():
'ok': False,
'error': f'CAM {sensor_id}: Could not match court pattern',
'debug_image': base64.b64encode(jpeg.tobytes()).decode('ascii'),
'steps': steps,
}
continue
n_matched = len(match['image_points'])
steps['template_match'] = {
'status': 'ok',
'detail': f'Matched {n_matched}/{len(template)} template points'
}
# Draw matched points
for name, (px, py) in match['image_points'].items():
cv2.circle(debug_frame, (int(px), int(py)), 10, (0, 255, 0), 2)
@@ -162,8 +248,22 @@ def auto_calibrate():
pts_3d = np.array(list(match['world_points'].values()), dtype=np.float32)
cal = CameraCalibrator()
cal.calibrate(pts_2d, pts_3d, w, h)
try:
cal.calibrate(pts_2d, pts_3d, w, h)
except RuntimeError as exc:
steps['camera_pose'] = {'status': 'fail', 'detail': str(exc)}
steps['overlay'] = {'status': 'fail', 'detail': 'No calibration output'}
_, jpeg = cv2.imencode('.jpg', debug_frame, [cv2.IMWRITE_JPEG_QUALITY, 85])
results[str(sensor_id)] = {
'ok': False,
'error': f'CAM {sensor_id}: solvePnP failed ({exc})',
'debug_image': base64.b64encode(jpeg.tobytes()).decode('ascii'),
'steps': steps,
}
continue
cam_pos = (-cal.rotation_matrix.T @ cal.translation_vec).flatten()
method_info = f"{n_matched} intersections matched"
# Sanity check
if cam_pos[2] < 0 or cam_pos[2] > 10:
@@ -173,19 +273,39 @@ def auto_calibrate():
cal2, cam_pos2, info = _calibrate_from_quad(quad, side, w, h)
if cal2 is not None:
cal, cam_pos = cal2, cam_pos2
method_info = f"{method_info}; fallback={info}"
n_matched = len(match['image_points'])
_finalize_calibration(
cal, cam_pos, sensor_id, side, debug_frame, w, h,
results, f"{n_matched} intersections matched", len(intersections), len(merged)
cal=cal,
cam_pos=cam_pos,
sensor_id=sensor_id,
side=side,
debug_frame=debug_frame,
results=results,
method_info=method_info,
steps=steps,
n_segments=len(white_segments),
n_lines=len(merged),
n_intersections=len(intersections),
n_points_matched=n_matched,
)
return results
def _finalize_calibration(cal, cam_pos, sensor_id, side, debug_frame, w, h,
results, method_info, n_intersections, n_lines):
def _finalize_calibration(cal, cam_pos, sensor_id, side, debug_frame,
results, method_info, steps, n_segments, n_lines,
n_intersections, n_points_matched):
"""Save calibration and build result dict."""
steps['camera_pose'] = {
'status': 'ok',
'detail': f'Camera pose solved: X={cam_pos[0]:.2f}, Y={cam_pos[1]:.2f}, Z={cam_pos[2]:.2f}m'
}
steps['overlay'] = {
'status': 'ok',
'detail': f'Geometry projected with {n_points_matched} matched points'
}
cv2.putText(debug_frame,
f"CAM{sensor_id}: X={cam_pos[0]:.2f} Y={cam_pos[1]:.2f} Z={cam_pos[2]:.2f}m",
(10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 255, 0), 2)
@@ -208,7 +328,12 @@ def _finalize_calibration(cal, cam_pos, sensor_id, side, debug_frame, w, h,
'camera_position': cam_pos.tolist(),
'debug_image': base64.b64encode(jpeg.tobytes()).decode('ascii'),
'matched_lines_3d': matched_lines_3d,
'points_matched': n_intersections,
'points_matched': n_points_matched,
'intersections_detected': n_intersections,
'lines_detected': n_lines,
'segments_detected': n_segments,
'method': method_info,
'steps': steps,
}
print(f"[CAM {sensor_id}] Calibrated! Camera at "
f"({cam_pos[0]:.2f}, {cam_pos[1]:.2f}, {cam_pos[2]:.2f}) via {method_info}")

9
pyproject.toml
View File

@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "pickle-ball-tracking"
version = "0.1.0"
description = "Pickleball ball tracking system using YOLO and Dagster"
description = "Pickleball referee system with dual-camera calibration and ball tracking"
requires-python = ">=3.11"
dependencies = [
"ultralytics>=8.0.0",
@@ -14,8 +14,6 @@ dependencies = [
"supervision>=0.16.0",
"opencv-python>=4.8.0",
"numpy>=1.24.0",
"dagster>=1.5.0",
"dagster-webserver>=1.5.0",
"matplotlib>=3.8.0",
"fastapi>=0.104.0",
"uvicorn>=0.24.0",
@@ -25,9 +23,6 @@ dependencies = [
"tqdm>=4.66.0",
]
[tool.dagster]
module_name = "dagster_project"
[tool.setuptools.packages.find]
where = ["."]
include = ["dagster_project*", "src*"]
include = ["src*"]

4
requirements.txt
View File

@@ -16,10 +16,6 @@ pydantic>=2.0.0
# celery>=5.3.0
# redis>=5.0.0
# Dagster
dagster>=1.5.0
dagster-webserver>=1.5.0
# Roboflow Hosted Inference
inference-sdk>=0.9.0

2
src/calibration/camera_calibrator.py
View File

@@ -1,6 +1,6 @@
"""
3D camera calibration using court geometry.
Extracted from dagster_project/assets/camera_calibration.py and coordinate_transform_3d.py.
Standalone runtime module for calibration and 3D projection.
For our setup: two cameras mounted at the net center, looking at opposite halves.
Each camera gets its own calibration.

318
src/web/templates/index.html
View File

@@ -12,17 +12,13 @@
background: #0a0a1a;
color: #e0e0e0;
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', system-ui, sans-serif;
overflow: hidden;
height: 100vh;
min-height: 100vh;
overflow-x: hidden;
overflow-y: auto;
}
/* Header */
.header {
position: fixed;
top: 0;
left: 0;
right: 0;
z-index: 10;
display: flex;
align-items: center;
justify-content: space-between;
@@ -68,7 +64,7 @@
gap: 8px;
padding: 8px;
justify-content: center;
height: calc(100vh - 48px);
align-items: flex-start;
}
.cam-box {
flex: 1;
@@ -104,10 +100,10 @@
/* Full-height tab layout */
.tab-content {
padding-top: 48px;
padding: 8px 12px 12px;
}
.tab-full {
height: calc(100vh - 48px - 100px);
min-height: 420px;
}
.tab-full .viewport-3d {
width: 100%;
@@ -115,11 +111,6 @@
border-top: none;
}
.bottom-bar {
position: fixed;
bottom: 0;
left: 0;
right: 0;
z-index: 10;
display: flex;
align-items: center;
justify-content: center;
@@ -127,7 +118,7 @@
padding: 4px 8px;
background: #0d0d20;
border-top: 1px solid #222;
height: 100px;
min-height: 100px;
}
.bottom-card {
width: 120px;
@@ -260,10 +251,9 @@
/* Event banner */
.event-banner {
display: none;
position: fixed;
top: 60px;
left: 50%;
transform: translateX(-50%);
position: relative;
margin: 8px auto;
width: fit-content;
background: rgba(255, 60, 60, 0.95);
color: white;
padding: 12px 32px;
@@ -293,42 +283,111 @@
.info-item .label { color: #666; }
.info-item .value { color: #4ecca3; font-weight: 600; margin-left: 4px; }
/* Calibration split layout */
.cal-split {
display: flex;
height: calc(100vh - 48px);
}
.cal-left {
flex: 1;
min-width: 0;
}
.cal-left .viewport-3d {
width: 100%;
height: 100%;
border-top: none;
}
.cal-right {
width: 360px;
flex-shrink: 0;
/* Calibration flow layout (scrollable) */
.calibration-page {
display: flex;
flex-direction: column;
border-left: 1px solid #2a2a4a;
gap: 12px;
}
.cal-cameras {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 8px;
}
.cal-cam-card {
position: relative;
border: 1px solid #2a2a4a;
border-radius: 6px;
overflow: hidden;
background: #0d0d20;
overflow-y: auto;
}
.cal-cam-card img {
width: 100%;
display: block;
}
.cal-cam-card .cam-label {
z-index: 1;
}
.cal-controls {
padding: 16px;
border-bottom: 1px solid #2a2a4a;
border: 1px solid #2a2a4a;
border-radius: 6px;
background: #0d0d20;
}
.cal-debug-images {
flex: 1;
.flow-grid {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 12px;
}
.flow-card {
border: 1px solid #2a2a4a;
border-radius: 6px;
background: #0d0d20;
padding: 12px;
}
.flow-title {
font-size: 12px;
color: #aaa;
margin-bottom: 8px;
letter-spacing: 0.4px;
text-transform: uppercase;
font-weight: 700;
}
.flow-steps {
display: flex;
flex-direction: column;
gap: 6px;
}
.flow-step {
border: 1px solid #2a2a4a;
border-radius: 4px;
padding: 6px 8px;
background: #10102a;
font-size: 12px;
}
.flow-step .step-head {
display: flex;
justify-content: space-between;
gap: 8px;
}
.flow-step .step-name {
color: #ddd;
font-weight: 600;
}
.flow-step .step-status {
color: #888;
font-size: 11px;
text-transform: uppercase;
}
.flow-step .step-detail {
margin-top: 4px;
color: #8e8eb0;
font-size: 11px;
}
.flow-step.ok .step-status { color: #4ecca3; }
.flow-step.fail .step-status { color: #ff6666; }
.flow-step.pending .step-status { color: #aaa; }
.calibration-3d-wrap {
border: 1px solid #2a2a4a;
border-radius: 6px;
overflow: hidden;
background: #0d0d20;
}
.calibration-3d-wrap .viewport-3d {
border-top: none;
}
.cal-debug-images {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 8px;
padding: 8px;
}
.cal-debug-card {
position: relative;
border: 1px solid #2a2a4a;
border-radius: 6px;
overflow: hidden;
background: #0d0d20;
padding: 8px;
}
.cal-debug-card img {
width: 100%;
@@ -374,6 +433,28 @@
max-height: 95vh;
object-fit: contain;
}
@media (max-width: 980px) {
.header {
flex-direction: column;
align-items: flex-start;
gap: 8px;
}
.status-bar {
flex-wrap: wrap;
gap: 8px;
}
.cameras {
flex-direction: column;
}
.cam-box {
max-width: 100%;
}
.cal-cameras,
.flow-grid,
.cal-debug-images {
grid-template-columns: 1fr;
}
}
</style>
</head>
<body>
@@ -408,41 +489,66 @@
</div>
</div>
<!-- Tab 2: Calibration — split: 3D left, debug right -->
<!-- Tab 2: Calibration — full scroll flow -->
<div class="tab-content" id="tab-calibration">
<div class="cal-split">
<div class="cal-left">
<div class="calibration-page">
<div class="cal-cameras">
<div class="cal-cam-card">
<div class="cam-label">CAM 1 LIVE</div>
<img id="cal-cam1" alt="Camera 1 live">
</div>
<div class="cal-cam-card">
<div class="cam-label">CAM 0 LIVE</div>
<img id="cal-cam0" alt="Camera 0 live">
</div>
</div>
<div class="cal-controls">
<button class="btn-calibrate" id="btnCalibrate" onclick="doCalibrate()" style="padding:8px 24px;font-size:14px">Run Calibration Flow</button>
<div class="calibrate-status" id="calStatus" style="margin-top:8px">
<span id="calStatusText">Not calibrated</span>
</div>
<div id="calError" style="color:#ff4444;font-size:11px;word-break:break-all;display:none;margin-top:8px"></div>
<div id="calPositions" style="display:none;margin-top:8px;font-size:11px">
<div id="calPos0" style="color:#4ecca3"></div>
<div id="calPos1" style="color:#ff88cc"></div>
</div>
</div>
<div class="flow-grid">
<div class="flow-card">
<div class="flow-title">CAM 0 Calibration Steps</div>
<div id="calFlowCam0" class="flow-steps"></div>
</div>
<div class="flow-card">
<div class="flow-title">CAM 1 Calibration Steps</div>
<div id="calFlowCam1" class="flow-steps"></div>
</div>
</div>
<div class="cal-debug-images">
<div class="cal-debug-card">
<div class="cal-debug-label">CAM 0 RESULT</div>
<img id="calDebugImg0" alt="CAM 0 calibration result">
</div>
<div class="cal-debug-card">
<div class="cal-debug-label">CAM 1 RESULT</div>
<img id="calDebugImg1" alt="CAM 1 calibration result">
</div>
</div>
<div class="calibration-3d-wrap">
<div class="flow-title" style="padding:12px 12px 0;margin-bottom:0">3D Reconstruction Preview</div>
<div class="viewport-3d" id="calibration-3d"></div>
</div>
<div class="cal-right">
<div class="cal-controls">
<button class="btn-calibrate" id="btnCalibrate" onclick="doCalibrate()" style="padding:8px 24px;font-size:14px">Calibrate</button>
<div class="calibrate-status" id="calStatus">
<span id="calStatusText">Not calibrated</span>
</div>
<div id="calError" style="color:#ff4444;font-size:11px;word-break:break-all;display:none;margin-top:4px"></div>
<div id="calPositions" style="display:none;margin-top:6px;font-size:11px">
<div id="calPos0" style="color:#4ecca3"></div>
<div id="calPos1" style="color:#ff88cc"></div>
</div>
</div>
<div class="cal-debug-images">
<div class="cal-debug-card">
<div class="cal-debug-label">CAM 0</div>
<img id="calDebugImg0" alt="CAM 0">
<img id="cal-cam0" class="cal-live" alt="CAM 0 live">
</div>
<div class="cal-debug-card">
<div class="cal-debug-label">CAM 1</div>
<img id="calDebugImg1" alt="CAM 1">
<img id="cal-cam1" class="cal-live" alt="CAM 1 live">
</div>
<div class="flow-card">
<div class="flow-title">Workflow Legend</div>
<div style="font-size:12px;color:#aaa;line-height:1.5">
1) detect court ROI → 2) detect line segments → 3) merge lines → 4) get intersections →
5) match to template points → 6) solve camera pose → 7) overlay reconstructed geometry.
Green steps are confirmed, red steps failed, gray steps are pending/skipped.
</div>
</div>
</div>
</div>
<!-- Tab 3: Trajectory — 3D main, cameras small bottom center -->
<!-- Tab 3: Trajectory — 3D main + camera strip -->
<div class="tab-content" id="tab-trajectory">
<div class="tab-full">
<div class="viewport-3d" id="trajectory-3d"></div>
@@ -502,17 +608,78 @@ window.addEventListener('popstate', function() {
if (activeTab !== 'camera') switchTab(activeTab);
// ===================== Calibration =====================
var CALIBRATION_STEP_ORDER = [
'green_mask',
'line_segments',
'merged_lines',
'intersections',
'template_match',
'camera_pose',
'overlay'
];
var CALIBRATION_STEP_LABELS = {
green_mask: '1. Court ROI',
line_segments: '2. White Segments',
merged_lines: '3. Merge Lines',
intersections: '4. Intersections',
template_match: '5. Template Match',
camera_pose: '6. Camera Pose',
overlay: '7. Geometry Overlay'
};
function renderCalibrationSteps(containerId, steps, fallbackError) {
var el = document.getElementById(containerId);
if (!el) return;
var html = '';
for (var i = 0; i < CALIBRATION_STEP_ORDER.length; i++) {
var key = CALIBRATION_STEP_ORDER[i];
var item = steps && steps[key] ? steps[key] : { status: 'pending', detail: 'No data yet' };
var status = item.status || 'pending';
var detail = item.detail || '';
html += '<div class="flow-step ' + status + '">';
html += ' <div class="step-head">';
html += ' <div class="step-name">' + CALIBRATION_STEP_LABELS[key] + '</div>';
html += ' <div class="step-status">' + status + '</div>';
html += ' </div>';
html += ' <div class="step-detail">' + detail + '</div>';
html += '</div>';
}
if (fallbackError) {
html += '<div class="flow-step fail">';
html += ' <div class="step-head"><div class="step-name">Error</div><div class="step-status">fail</div></div>';
html += ' <div class="step-detail">' + fallbackError + '</div>';
html += '</div>';
}
el.innerHTML = html;
}
function updateCalibrationFlow(result) {
var cam0 = result && result['0'] ? result['0'] : null;
var cam1 = result && result['1'] ? result['1'] : null;
renderCalibrationSteps('calFlowCam0', cam0 ? cam0.steps : null, cam0 && cam0.error ? cam0.error : '');
renderCalibrationSteps('calFlowCam1', cam1 ? cam1.steps : null, cam1 && cam1.error ? cam1.error : '');
}
function doCalibrate() {
var btn = document.getElementById('btnCalibrate');
var errEl = document.getElementById('calError');
btn.disabled = true;
btn.textContent = 'Calibrating...';
errEl.style.display = 'none';
updateCalibrationFlow({
'0': { steps: { green_mask: { status: 'pending', detail: 'Running calibration flow...' } } },
'1': { steps: { green_mask: { status: 'pending', detail: 'Running calibration flow...' } } }
});
fetch('/api/calibration/trigger', { method: 'POST' })
.then(function(r) { return r.json(); })
.then(function(data) {
btn.disabled = false;
updateCalibrationFlow(data.result || null);
// Show debug images from calibration
if (data.result) {
@@ -526,7 +693,7 @@ function doCalibrate() {
}
if (data.ok) {
btn.textContent = 'Re-calibrate';
btn.textContent = 'Re-run Calibration Flow';
errEl.style.display = 'none';
updateCalibrationStatus();
@@ -557,7 +724,7 @@ function doCalibrate() {
// Highlight detected court lines on 3D scene
if (data.result) highlightDetectedLines(data.result);
} else {
btn.textContent = 'Calibrate';
btn.textContent = 'Run Calibration Flow';
// Show errors
var errors = [];
if (data.result) {
@@ -572,9 +739,13 @@ function doCalibrate() {
})
.catch(function(e) {
btn.disabled = false;
btn.textContent = 'Calibrate';
btn.textContent = 'Run Calibration Flow';
errEl.textContent = String(e);
errEl.style.display = 'block';
updateCalibrationFlow({
'0': { error: String(e) },
'1': { error: String(e) }
});
});
}
@@ -614,6 +785,7 @@ function updateCalibrationStatus() {
}
// Check on load and periodically
updateCalibrationStatus();
updateCalibrationFlow(null);
setInterval(updateCalibrationStatus, 3000);
var cameraMeshes = [];