124 lines
4.9 KiB
Markdown
124 lines
4.9 KiB
Markdown
# Yoloserv 2026 Project
|
|
|
|
## Project Objectives
|
|
|
|
### Repo to large for GitLab / Streamline the repo for Deployment
|
|
|
|
### Move camera_stream from core to YOLO
|
|
|
|
#### Problems with camera_stream
|
|
|
|
Camera stream is a proccess that provides a live video feed to the UI. It has led to better success with the face matching proccess and has allowed for a more seamless user experience. Right now the camera_stream functionalty is in the CORE repo and runs and runs contiously in the background. There are a few issues with this:
|
|
|
|
- It occupies and holds the camera would cause issues with other applications that use core and not the camera_stream
|
|
- It uses a full CPU core to run which as our applications scale is not maintainable.
|
|
|
|
#### Solution
|
|
|
|
The solution was to move the camera_stream functionality to the YOLO repo and make it a UKDI variable that can be enabled or disabled. However this failed as it added to much delay and lagging when loading the stream and moving frames back and forth from yolo to core back to yolo. The decision was made to keep the camera_stream in core for now but init it from a UKDI var.
|
|
|
|
#### Flow Diagram
|
|
|
|
The Flow has been broken into two parts to show the flow of the camera stream.
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
UI[UI]
|
|
CORE[Core]
|
|
YOLO[YOLO]
|
|
CAM[Camera]
|
|
|
|
UI --> | Open Camera Stream | CORE
|
|
|
|
CORE --> | Open Camera Stream | YOLO
|
|
YOLO --> | Open Camera Stream | CAM
|
|
CAM --> | Frames | YOLO
|
|
YOLO --> | Frames | CORE
|
|
```
|
|
|
|
This flow below can be viewed within the method `cam_livefeed` in the file `yoloserv.py`.
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
UI[UI]
|
|
CORE[CORE]
|
|
CAM[Camera]
|
|
|
|
UI --> | Open Camera Stream | CORE
|
|
CORE --> | Open Camera Stream | CAM
|
|
CAM --> | Frames | CORE
|
|
CORE --> | Camera Stream | UI
|
|
```
|
|
|
|
#### Camera Stream Symlink
|
|
|
|
Dispension has moved hardware conenctions to symlinks after dealing with unreliable USB hubs in the field. Below is how to setup a symlink for a camera stream.
|
|
|
|
Do not use /dev/video instead tun this command ```ls -l /dev/v4l/by-id/``` to find the correct camera. In the case where the device provides two /dev/video devies we can run ```v4l2-ctl --list-formats-ext -d /dev/video0``` to find the correct device you'll see the formats it provides and also the resolution for those formats.
|
|
|
|
So I ran ```ls -l /dev/v4l/by-id/``` and ```v4l2-ctl --list-formats-ext -d /dev/video0``` to find the correct device. I then created a symlink to the device with ```sudo ln -s /dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920_628DABFF-video-index0 /dev/facematch_camera```. I confirmed this worked by running ```ls -l /dev/facematch_camera``` and ```v4l2-ctl --list-formats-ext -d /dev/facematch_camera```.
|
|
|
|
|
|
|
|
### Yoloserv pipeline
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant UI
|
|
participant CORE
|
|
participant CAM
|
|
participant YOLO
|
|
participant para
|
|
|
|
rect rgba(200, 220, 255, 0.35)
|
|
UI->>CORE: pic_still
|
|
CORE->>CAM: capture still
|
|
CAM-->>CORE: still image
|
|
CORE-->>UI: still image
|
|
end
|
|
|
|
rect rgba(220, 255, 220, 0.35)
|
|
UI->>CORE: facematch (regula)
|
|
CORE->>YOLO: facematch
|
|
YOLO->>para: facematch
|
|
para-->>YOLO: result
|
|
YOLO-->>CORE: result
|
|
CORE-->>UI: result
|
|
end
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Implement 2D detection
|
|
|
|
Dispension faced the problem of needing a detection model in place for fully autonomous operation. To resolve this we implemented a 2D detection model that can be used to detect faces in a video stream. We implemented paravisions 2D model detection. Refer to para.md for more information on installing models.
|
|
|
|
Here are the results from a live user test:
|
|
|
|
|
|
```bash
|
|
<ValidnessResult (isValid=0, sharpness=-1.000000, quality=-1.000000, acceptability=-1.000000, frontality=-1.000000, mask=-1.000000, image_avg_pixel=115.375671, image_bright_pixel_pct=0.145549, image_dark_pixel_pct=0.030642, face_height_pct=0.318821, face_width_pct=0.208163, face_width=133.224304, face_height=153.033981, face_roll_angle=3.102657, face_positions=[0.377501,-0.001564,0.585664,0.317257])>
|
|
HERE IS THE LIVENESS RESULT <LivenessResult (liveProbability=0.999998, spoofProbability=0.000002)>
|
|
|
|
HERE IS THE VALIDNESS RESULT <ValidnessResult (isValid=0, sharpness=0.023801, quality=-1.000000, acceptability=-1.000000, frontality=97.000000, mask=0.008516, image_avg_pixel=58.403454, image_bright_pixel_pct=0.000000, image_dark_pixel_pct=0.000000, face_height_pct=0.551221, face_width_pct=0.579525, face_width=373.213989, face_height=446.488739, face_roll_angle=0.489146, face_positions=[0.182151,0.242310,0.761676,0.793531])>
|
|
HERE IS THE LIVENESS RESULT <LivenessResult (liveProbability=0.057056, spoofProbability=0.830750)>
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Improvements
|
|
|
|
Ideally we should be using the first frame to complete the facematch process and not for rendering. This would speed things up by a few seconds but is something for the future.
|
|
|
|
Add cleanup functionality to remove localcam files after facematch is complete. |