HandGesture
Draw on screen with just your index finger, real-time hand landmark detection, gesture classification, and canvas rendering from a live webcam feed. No hardware required.
HandGesture turns your webcam into a drawing canvas. Raise your index finger to draw; hover over on-screen buttons to switch colours, start, stop, or clear. No mouse, stylus, or touchscreen — just a camera and your hand.
Built in Python using OpenCV for video capture and canvas compositing, and MediaPipe Hands for multi-hand 21-landmark detection. The system can track up to 4 hands simultaneously at 30+ FPS, all on a standard laptop CPU, no GPU required.
cv2.line() between the current and previous position onto a persistent canvas layer.cv2.addWeighted() and displayed in real time.OpenCV opens the default webcam, reads frames in a loop, mirrors the image horizontally (so movement feels natural), and converts from BGR → RGB colour space for MediaPipe processing.
MediaPipe Hands processes each RGB frame and returns normalised (x, y, z) coordinates for 21 keypoints per hand, wrist, MCP joints, PIP joints, DIP joints, and fingertips.
A geometric classifier checks the y-coordinates of each finger's tip vs. its proximal knuckle (MCP). If only the index fingertip is above its MCP while all others are below → drawing mode.
When in drawing mode, the app tracks the index fingertip pixel position and calls cv2.line() between the current and previous position onto a persistent canvas layer.
On-screen buttons are fixed pixel regions. Each frame, if the fingertip coordinate falls within a button's bounding box, the action fires. The canvas layer is composited onto the live frame using cv2.addWeighted() and displayed in real time.
MediaPipe returns normalised landmark coordinates in the range [0.0, 1.0]. To convert to pixel coordinates on the actual frame:
pixel_x = landmark.x × frame_width pixel_y = landmark.y × frame_height
Where landmark.x and landmark.y are values from MediaPipe in range [0,1]. Multiplying by frame dimensions gives pixel coordinates, so the same tracking code works identically on a 640×480 or 1920×1080 webcam — the model never needs to know the resolution.
# Webcam frame: 640 × 480 landmark.x = 0.512 landmark.y = 0.347 pixel_x = 0.512 × 640 = 327.68 → 328 px pixel_y = 0.347 × 480 = 166.56 → 167 px # Same landmark, 1920×1080 frame: pixel_x = 0.512 × 1920 = 983.04 → 983 px pixel_y = 0.347 × 1080 = 374.76 → 375 px
is_drawing = (
tip_y[INDEX] < mcp_y[INDEX] AND # index finger up
tip_y[MIDDLE] > mcp_y[MIDDLE] AND # middle finger down
tip_y[RING] > mcp_y[RING] AND # ring finger down
tip_y[PINKY] > mcp_y[PINKY] # pinky down
)Since MediaPipe's y-axis is top-down (y=0 at top), a fingertip above its knuckle means tip_y < mcp_y. That gives a binary classifier which fires only on the drawing pose — index extended, all other fingers curled. No machine learning; pure landmark geometry, checked every frame.
# Hand landmarks (normalised, from MediaPipe) tip_y[INDEX] = 0.31 mcp_y[INDEX] = 0.52 → 0.31 < 0.52 ✓ UP tip_y[MIDDLE] = 0.58 mcp_y[MIDDLE] = 0.54 → 0.58 > 0.54 ✓ DOWN tip_y[RING] = 0.61 mcp_y[RING] = 0.55 → 0.61 > 0.55 ✓ DOWN tip_y[PINKY] = 0.63 mcp_y[PINKY] = 0.56 → 0.63 > 0.56 ✓ DOWN → all 4 conditions true → is_drawing = True
is_hitting_button = (
btn.x1 <= finger_x <= btn.x2 AND
btn.y1 <= finger_y <= btn.y2
)Each on-screen button is an axis-aligned bounding box (AABB), the simplest possible collision test, borrowed straight from 2D game physics. The fingertip's pixel coordinate is checked against every button rectangle each frame — no click event, no debounce beyond a short hold-to-confirm, just geometry re-evaluated 30 times a second.
# "Clear Canvas" button region:
btn = { x1: 500, y1: 20, x2: 620, y2: 70 }
# Fingertip at frame coordinate (560, 45):
500 <= 560 <= 620 → True
20 <= 45 <= 70 → True
→ is_hitting_button = True → canvas.clear()MediaPipe detecting 21 hand landmarks in real time, the index-finger drawing engine tracing strokes on the canvas overlay, and gesture-based button interactions, all at 30+ FPS directly from a standard webcam.
| Control | Trigger | Action |
|---|---|---|
| START | Fingertip hover | Enable drawing mode, fingertip traces strokes |
| STOP | Fingertip hover | Disable drawing, tracking stays active, no strokes |
| CLEAR | Fingertip hover | Wipe entire canvas clean |
| COLOR | Hover color button | Switch brush: Blue / Green / Red / Pink |
| Q | Keyboard | Quit the application |
python handgesture.py. All you need is a webcam and a well-lit room.