Vision-Language Models for Robotic Grasping in Unstructured Environments

This research project explores the use of vision-language models (VLMs) to enable a robotic arm to identify and grasp objects specified through natural-language instructions (e.g., 'pick up the red cup') in a cluttered, unstructured tabletop scene, rather than relying on pre-programmed object coordinates.

A pretrained vision-language grounding model localizes the referenced object in the camera image based on the language command, and the resulting bounding box/segmentation mask is passed to a grasp-planning module that computes a feasible grasp pose for a simulated robotic gripper.

The study evaluates grasp-success rate across a range of object categories and command phrasings (including ambiguous or multi-object scenes), and discusses failure modes — such as referring-expression ambiguity and occlusion — as open challenges for deploying language-conditioned robotic manipulation in real-world settings.

Detailed Project Description

Vision-Language Models for Robotic Grasping in Unstructured Environments applies computer vision to camera frames to detect or recognise what is in view and act on the result automatically.

How It Works
1. The camera captures frames continuously.
2. Frames are pre-processed and fed to the detection/recognition model.
3. Detections are drawn on the output and used to trigger the required action (log, alert, unlock, count).
4. Results and images are stored or streamed to a browser/dashboard.

Technology Roles
- Python
- PyTorch
- Vision-Language Models
- Robotics
- Grasp Planning

Implementation Steps
1. Define requirements and draw the ER / class diagram and screen flow.
2. Set up the environment (Python, PyTorch, ROS / PyBullet Simulator) and create the project skeleton.
3. Design the database schema / data model and seed it with sample data.
4. Implement the core modules with validation and role-based access where needed.
5. Build the user interface and connect it to the backend / data layer.
6. Test each module (unit + manual), fix edge cases and secure inputs (SQL injection, XSS, auth).
7. Prepare documentation, screenshots and a demo build.

Key Features
- Real-time detection
- Annotated output
- Event logging
- Adjustable confidence threshold

Expected Output
On completion the student will have a live video/image feed with detected objects highlighted and the corresponding actions logged.

Deliverables
- Complete source code with comments
- Database schema / data model and screen flow
- Project report with screenshots and results
- Demonstration-ready build