Gesture Controlled Virtual Mouse with ESP32-CAM & OpenCV
Gesture Controlled Virtual Mouse with ESP32-CAM Camera Module & OpenCV with Python Libraries & control mouse & clicks with Python Arduino Code
- Suitable forB.Tech/Diploma ESP32-CAM students
- Build time1-2 weeks
- LevelAdvanced
- Available atAjvika Labs, Kakinada
About this project
Overview
In this article, we will develop
Gesture Controlled Virtual Mouse
with
ESP32-CAM
&
OpenCV
. The
ESP32 Camera Module
along with a
Python
program can be used to control the
mouse tracking
and
clicking
operations wirelessly.
To get started one must have sound knowledge of Python,
Image processing
, Embedded Systems as well as the Internet of Things. First, we will understand how to control the mouse tracking and clicking, and what all requirements are needed to run the python program. We will first test the whole
python script
with a webcam or internal camera of a laptop.
In the second part, we will use an
ESP32-CAM Module
and run the Python code. So, the ESP32-CAM will be used as an input device instead of a PC camera or any other external camera.
Earlier we learned about
Gesture Recognition
& its application in Machine Learning. And we also developed
Gesture Controlled Robot
using Accelerometer. The same motion detection feature has been implemented in
ESP32-CAM Based Squid Game
using OpenCV.
Bill of Materials
The following is the list of Bill of Materials for making a
Gesture Controlled Virtual Mouse
. You can purchase all these components from Amazon.
S.N.
Components
Quantity
Purchase Links
ESP32-CAM Board AI-Thinker
Amazon
AliExpress
FTDI Module
Amazon
AliExpress
Micro-USB Cable
Amazon
AliExpress
Jumper Wires
Amazon
AliExpress
Controlling Mouse Tracking & Clicks with PC Camera
Before jumping to the project part, lets build a
Gesture Controlled Virtual Mouse
using PC
image recognition technology
Installing Python & Required Libraries
For the live stream of video to be visible on our computer we need to write a
Python script
that will enable us to retrieve the frames of the video. The first step is to
install Python
. Go to python.org and download
Python version 3.7.8
. It is important to download this version or downgrade to this version or else a few libraries won’t work.
Once downloaded and installed go to command prompt and run the commands below:
python
--
version
The output should be as below, that is version
3.7.8
Now we have to install a few
libraries
. For that run the following commands below one after another until all the libraries are installed.
pip
install
numpy
pip
install
opencv
python
pip
install
autopy
pip
install
mediapipe
If the python version you installed is correct then installing these won’t be an issue.
Source Code/Program
Now create a folder and inside that folder create a new python file, name the file as
track_hand.py
Now copy and paste the code below, and save it.
import
cv2
import
mediapipe
as
mp
import
time
import
math
import
numpy
as
np
class
handDetector
(
)
:
def
__init__
(
self
mode
=
False
maxHands
=
modelComplexity
=
detectionCon
=
0.5
trackCon
=
0.5
)
:
self
mode
=
mode
self
maxHands
=
maxHands
self
modelComplex
=
modelComplexity
self
detectionCon
=
detectionCon
self
trackCon
=
trackCon
self
mpHands
=
mp
solutions
hands
self
hands
=
self
mpHands
Hands
(
self
mode
self
maxHands
self
modelComplex
self
detectionCon
self
trackCon
)
self
mpDraw
=
mp
solutions
drawing_utils
self
tipIds
=
[
]
def
findHands
(
self
img
draw
=
True
)
:
imgRGB
=
cv2
cvtColor
(
img
cv2
COLOR_BGR2RGB
)
self
results
=
self
hands
process
(
imgRGB
)
# print(results.multi_hand_landmarks)
if
self
results
multi_hand_landmarks
:
for
handLms
in
self
results
multi_hand_landmarks
:
if
draw
:
self
mpDraw
draw_landmarks
(
img
handLms
self
mpHands
HAND_CONNECTIONS
)
return
img
def
findPosition
(
self
img
handNo
=
draw
=
True
)
:
xList
=
[
]
yList
=
[
]
bbox
=
[
]
self
lmList
=
[
]
if
self
results
multi_hand_landmarks
:
myHand
=
self
results
multi_hand_landmarks
[
handNo
]
for
id
lm
in
enumerate
(
myHand
landmark
)
:
# print(id, lm)
h
w
c
=
img
shape
cx
cy
=
int
(
lm
x *
w
)
int
(
lm
y *
h
)
xList
append
(
cx
)
yList
append
(
cy
)
# print(id, cx, cy)
self
lmList
append
(
[
id
cx
cy
]
)
if
draw
:
cv2
circle
(
img
(
cx
cy
)
(
)
cv2
FILLED
)
xmin
xmax
=
min
(
xList
)
max
(
xList
)
ymin
ymax
=
min
(
yList
)
max
(
yList
)
bbox
=
xmin
ymin
xmax
ymax
if
draw
:
cv2
rectangle
(
img
(
xmin
ymin
)
(
xmax
+
ymax
+
)
(
)
)
return
self
lmList
bbox
def
fingersUp
(
self
)
:
fingers
=
[
]
# Thumb
#print(self.lmList)
#print(self.tipIds)
if
self
lmList
[
self
tipIds
[
]
]
[
]
>
self
lmList
[
self
tipIds
[
]
]
[
]
:
fingers
append
(
)
else
:
fingers
append
(
)
# Fingers
for
id
in
range
(
)
:
if
self
lmList
[
self
tipIds
[
id
]
]
[
]
<
self
lmList
[
self
tipIds
[
id
]
]
[
]
:
fingers
append
(
)
else
:
fingers
append
(
)
# totalFingers = fingers.count(1)
return
fingers
def
findDistance
(
self
p1
p2
img
draw
=
True
r
=
t
=
)
:
x1
y1
=
self
lmList
[
p1
]
[
:
]
x2
y2
=
self
lmList
[
p2
]
[
:
]
cx
cy
=
(
x1
+
x2
)
// 2, (y1 + y2) // 2
if
draw
:
cv2
line
(
img
(
x1
y1
)
(
x2
y2
)
(
)
t
)
cv2
circle
(
img
(
x1
y1
)
r
(
)
cv2
FILLED
)
cv2
circle
(
img
(
x2
y2
)
r
(
)
cv2
FILLED
)
cv2
circle
(
img
(
cx
cy
)
r
(
)
cv2
FILLED
)
length
=
math
hypot
(
x2
x1
y2
y1
)
return
length
img
[
x1
y1
x2
y2
cx
cy
]
def
main
(
)
:
pTime
=
cTime
=
cap
=
cv2
VideoCapture
(
)
detector
=
handDetector
(
)
while
True
:
success
img
=
cap
read
(
)
img
=
detector
findHands
(
img
)
lmList
bbox
=
detector
findPosition
(
img
)
if
len
(
lmList
)
!=
:
(
lmList
[
]
)
cTime
=
time
time
(
)
fps
=
/
(
cTime
pTime
)
pTime
=
cTime
fingers
=
detector
fingersUp
(
)
cv2
putText
(
img
str
(
int
(
fps
)
)
(
)
cv2
FONT_HERSHEY_PLAIN
(
)
)
cv2
imshow
(
"Image"
img
)
cv2
waitKey
(
)
if
__name__
==
"__main__"
:
main
(
)
Now inside the same folder create another python file as
final.py
Now copy and paste the code below, and save it. But before saving make sure you change a few attributes:
**
wCam
and
hCam
attributes should be changed accordingly, these are the width and height of your webcam
. **
import
numpy
as
np
import
track_hand
as
htm
import
time
import
autopy
import
cv2
wCam
hCam
=
1280
frameR
=
smoothening
=
pTime
=
plocX
plocY
=
clocX
clocY
=
cap
=
cv2
VideoCapture
(
)
cap
set
(
wCam
)
cap
set
(
hCam
)
detector
=
htm
handDetector
(
maxHands
=
)
wScr
hScr
=
autopy
screen
size
(
)
# print(wScr, hScr)
while
True
:
# 1. Find hand Landmarks
fingers
=
[
]
success
img
=
cap
read
(
)
img
=
detector
findHands
(
img
)
lmList
bbox
=
detector
findPosition
(
img
)
# 2. Get the tip of the index and middle fingers
if
len
(
lmList
)
!=
:
x1
y1
=
lmList
[
]
[
:
]
x2
y2
=
lmList
[
]
[
:
]
# print(x1, y1, x2, y2)
# 3. Check which fingers are up
fingers
=
detector
fingersUp
(
)
# print(fingers)
cv2
rectangle
(
img
(
frameR
frameR
)
(
wCam
frameR
hCam
frameR
)
(
)
)
# 4. Only Index Finger : Moving Mode
if
fingers
[
]
==
and
fing
In this article, we will develop
Gesture Controlled Virtual Mouse
with
ESP32-CAM
&
OpenCV
. The
ESP32 Camera Module
along with a
Python
program can be used to control the
mouse tracking
and
clicking
operations wirelessly.
To get started one must have sound knowledge of Python,
Image processing
, Embedded Systems as well as the Internet of Things. First, we will understand how to control the mouse tracking and clicking, and what all requirements are needed to run the python program. We will first test the whole
python script
with a webcam or internal camera of a laptop.
In the second part, we will use an
ESP32-CAM Module
and run the Python code. So, the ESP32-CAM will be used as an input device instead of a PC camera or any other external camera.
Earlier we learned about
Gesture Recognition
& its application in Machine Learning. And we also developed
Gesture Controlled Robot
using Accelerometer. The same motion detection feature has been implemented in
ESP32-CAM Based Squid Game
using OpenCV.
Bill of Materials
The following is the list of Bill of Materials for making a
Gesture Controlled Virtual Mouse
. You can purchase all these components from Amazon.
S.N.
Components
Quantity
Purchase Links
ESP32-CAM Board AI-Thinker
Amazon
AliExpress
FTDI Module
Amazon
AliExpress
Micro-USB Cable
Amazon
AliExpress
Jumper Wires
Amazon
AliExpress
Controlling Mouse Tracking & Clicks with PC Camera
Before jumping to the project part, lets build a
Gesture Controlled Virtual Mouse
using PC
image recognition technology
Installing Python & Required Libraries
For the live stream of video to be visible on our computer we need to write a
Python script
that will enable us to retrieve the frames of the video. The first step is to
install Python
. Go to python.org and download
Python version 3.7.8
. It is important to download this version or downgrade to this version or else a few libraries won’t work.
Once downloaded and installed go to command prompt and run the commands below:
python
--
version
The output should be as below, that is version
3.7.8
Now we have to install a few
libraries
. For that run the following commands below one after another until all the libraries are installed.
pip
install
numpy
pip
install
opencv
python
pip
install
autopy
pip
install
mediapipe
If the python version you installed is correct then installing these won’t be an issue.
Source Code/Program
Now create a folder and inside that folder create a new python file, name the file as
track_hand.py
Now copy and paste the code below, and save it.
import
cv2
import
mediapipe
as
mp
import
time
import
math
import
numpy
as
np
class
handDetector
(
)
:
def
__init__
(
self
mode
=
False
maxHands
=
modelComplexity
=
detectionCon
=
0.5
trackCon
=
0.5
)
:
self
mode
=
mode
self
maxHands
=
maxHands
self
modelComplex
=
modelComplexity
self
detectionCon
=
detectionCon
self
trackCon
=
trackCon
self
mpHands
=
mp
solutions
hands
self
hands
=
self
mpHands
Hands
(
self
mode
self
maxHands
self
modelComplex
self
detectionCon
self
trackCon
)
self
mpDraw
=
mp
solutions
drawing_utils
self
tipIds
=
[
]
def
findHands
(
self
img
draw
=
True
)
:
imgRGB
=
cv2
cvtColor
(
img
cv2
COLOR_BGR2RGB
)
self
results
=
self
hands
process
(
imgRGB
)
# print(results.multi_hand_landmarks)
if
self
results
multi_hand_landmarks
:
for
handLms
in
self
results
multi_hand_landmarks
:
if
draw
:
self
mpDraw
draw_landmarks
(
img
handLms
self
mpHands
HAND_CONNECTIONS
)
return
img
def
findPosition
(
self
img
handNo
=
draw
=
True
)
:
xList
=
[
]
yList
=
[
]
bbox
=
[
]
self
lmList
=
[
]
if
self
results
multi_hand_landmarks
:
myHand
=
self
results
multi_hand_landmarks
[
handNo
]
for
id
lm
in
enumerate
(
myHand
landmark
)
:
# print(id, lm)
h
w
c
=
img
shape
cx
cy
=
int
(
lm
x *
w
)
int
(
lm
y *
h
)
xList
append
(
cx
)
yList
append
(
cy
)
# print(id, cx, cy)
self
lmList
append
(
[
id
cx
cy
]
)
if
draw
:
cv2
circle
(
img
(
cx
cy
)
(
)
cv2
FILLED
)
xmin
xmax
=
min
(
xList
)
max
(
xList
)
ymin
ymax
=
min
(
yList
)
max
(
yList
)
bbox
=
xmin
ymin
xmax
ymax
if
draw
:
cv2
rectangle
(
img
(
xmin
ymin
)
(
xmax
+
ymax
+
)
(
)
)
return
self
lmList
bbox
def
fingersUp
(
self
)
:
fingers
=
[
]
# Thumb
#print(self.lmList)
#print(self.tipIds)
if
self
lmList
[
self
tipIds
[
]
]
[
]
>
self
lmList
[
self
tipIds
[
]
]
[
]
:
fingers
append
(
)
else
:
fingers
append
(
)
# Fingers
for
id
in
range
(
)
:
if
self
lmList
[
self
tipIds
[
id
]
]
[
]
<
self
lmList
[
self
tipIds
[
id
]
]
[
]
:
fingers
append
(
)
else
:
fingers
append
(
)
# totalFingers = fingers.count(1)
return
fingers
def
findDistance
(
self
p1
p2
img
draw
=
True
r
=
t
=
)
:
x1
y1
=
self
lmList
[
p1
]
[
:
]
x2
y2
=
self
lmList
[
p2
]
[
:
]
cx
cy
=
(
x1
+
x2
)
// 2, (y1 + y2) // 2
if
draw
:
cv2
line
(
img
(
x1
y1
)
(
x2
y2
)
(
)
t
)
cv2
circle
(
img
(
x1
y1
)
r
(
)
cv2
FILLED
)
cv2
circle
(
img
(
x2
y2
)
r
(
)
cv2
FILLED
)
cv2
circle
(
img
(
cx
cy
)
r
(
)
cv2
FILLED
)
length
=
math
hypot
(
x2
x1
y2
y1
)
return
length
img
[
x1
y1
x2
y2
cx
cy
]
def
main
(
)
:
pTime
=
cTime
=
cap
=
cv2
VideoCapture
(
)
detector
=
handDetector
(
)
while
True
:
success
img
=
cap
read
(
)
img
=
detector
findHands
(
img
)
lmList
bbox
=
detector
findPosition
(
img
)
if
len
(
lmList
)
!=
:
(
lmList
[
]
)
cTime
=
time
time
(
)
fps
=
/
(
cTime
pTime
)
pTime
=
cTime
fingers
=
detector
fingersUp
(
)
cv2
putText
(
img
str
(
int
(
fps
)
)
(
)
cv2
FONT_HERSHEY_PLAIN
(
)
)
cv2
imshow
(
"Image"
img
)
cv2
waitKey
(
)
if
__name__
==
"__main__"
:
main
(
)
Now inside the same folder create another python file as
final.py
Now copy and paste the code below, and save it. But before saving make sure you change a few attributes:
**
wCam
and
hCam
attributes should be changed accordingly, these are the width and height of your webcam
. **
import
numpy
as
np
import
track_hand
as
htm
import
time
import
autopy
import
cv2
wCam
hCam
=
1280
frameR
=
smoothening
=
pTime
=
plocX
plocY
=
clocX
clocY
=
cap
=
cv2
VideoCapture
(
)
cap
set
(
wCam
)
cap
set
(
hCam
)
detector
=
htm
handDetector
(
maxHands
=
)
wScr
hScr
=
autopy
screen
size
(
)
# print(wScr, hScr)
while
True
:
# 1. Find hand Landmarks
fingers
=
[
]
success
img
=
cap
read
(
)
img
=
detector
findHands
(
img
)
lmList
bbox
=
detector
findPosition
(
img
)
# 2. Get the tip of the index and middle fingers
if
len
(
lmList
)
!=
:
x1
y1
=
lmList
[
]
[
:
]
x2
y2
=
lmList
[
]
[
:
]
# print(x1, y1, x2, y2)
# 3. Check which fingers are up
fingers
=
detector
fingersUp
(
)
# print(fingers)
cv2
rectangle
(
img
(
frameR
frameR
)
(
wCam
frameR
hCam
frameR
)
(
)
)
# 4. Only Index Finger : Moving Mode
if
fingers
[
]
==
and
fing
Technologies used
Hardware components
- ESP32
- ESP32-CAM
Software & tools
- Arduino IDE
What you get
- Working project / hardware kit, tested before delivery
- Complete source code with explanation
- Project demo and guidance for your viva
- Project report, PPT and IEEE paper support (on request)
Delivery & Payment
Delivery takes a minimum of two weeks from the date your inquiry is confirmed.
Payment is in two parts:
• 50% advance to start the project
• Remaining 50% before delivery
Note:
• Documentation and PPT are not included by default
• If you need documentation/PPT, it can be arranged for extra time and cost
We'll confirm the exact timeline and cost with you after reviewing your requirements.
Related ESP32-CAM projects
ESP32-CAM
Mars Rover Robot Replica using ESP32 CAM & Arduino
Build Mars Rover Robot Replica using ESP32 CAM Module & Arduino with universal rocker-bogie system with assembly guide, source code & app
View project
ESP32-CAM
DIY Spiderbot Hexapod Robot using ESP32-CAM FOV
DIY Spiderbot Hexapod Robot with ESP32-CAM FOV that can traverse multiple terrains & stream video using its camera with Code & 3D Model
View project
ESP32-CAM
QR Code Scanner with ESP32 CAM Module & OpenCV
DIY QR Code Scanner Reader with ESP32 CAM Module & OpenCV Python Libraries, read QR Codes using ESP32 Camera & Arduino Code
View project
ESP32-CAM
Motion Detection Squid Game using ESP32 CAM & OpenCV
Motion Detection based Squid Game with ESP32 CAM or Camera Module & OpenCV Python Code, red light person dead or game over & green light mov...
View project