Models
The Live Sense models provide an interface for interacting with the underlying machine learning (ML) models used for object detection.
Info: Object Detection/Recognition
Describes the combination of object classification, identifying if objects of interest exist in an image, and localization, identifying where the object is located in the image. For example, a Live Sense model that identifies a vehicle in the image and provides a bounding box of where the vehicle was identified.
The core function of every model is to process an RGB image and return a list of detections found within the image.
Each Live Sense model has a set of object classes that it is able to detect. The developer must determine which objects and models are relevant to their use case.
Model types and detection classes
The following table describes the models and their available classes:
Category | Model Name | Label | Recommended Confidence |
Road Basics | RB_day | pedestrian | 47 |
| | rider | 47 |
| | bicycle | 45 |
| | car | 46 |
| | motorcycle | 47 |
| | bus | 50 |
| | truck | 45 |
| | traffic-light | 47 |
| RB_night | pedestrian | 45 |
| | car | 40 |
| | motorcycle | 40 |
| | truck | 40 |
Road Alerts | BrakeLight | car-brake-light-on | 80 |
Road Hazards | cone_barrier | cone | 60 |
| | strip-stand | 60 |
| | rectangular-barrier | 75 |
| | cylindrical-barrier | 60 |
| | delineator-barrier | 60 |
| pothole | pothole | 80 |
| speed_bump_signage | speed-bump-sign | 55 |
Road Signs | RoadSigns | DS_SpeedLimit[5-130] | 55 |
| | DS_AdvisorySpeedLimit[10-85] | 55 |
| | DS_Stop | 55 |
| | DS_DoNotEnter | 55 |
| | DS_NoAccess | 55 |
| | DS_StartConstruction | 55 |
| | DS_PedestrianCrossing | 55 |
| | DS_SpeedBumps | 55 |
| | DS_StopHereOnRed | 55 |
| | DS_NoLeftTurn | 55 |
| | DS_NoRightOrLeftTurn | 55 |
| | DS_NoUTurn | 55 |
| | DS_NoUTurnOrLeftTurn | 55 |
| | DS_ParkingRestriction | 55 |
Note
RB_night
detects supported features in low light conditions.
Note
For the Road Signs model, only the label prefix is given above.
See the Road Signs section below for the full label list with sample images
Models available in beta mode
The models below are available for beta testing:
Category | Model Name | Label | Recommended Confidence |
Road Alerts | TrafficLightCVModel | traffic-light-green | 75 |
| | traffic-light-red | 75 |
| | traffic-light-yellow | 75 |
Road Hazards | bridge_tunnel | bridge | 75 |
| | tunnel | 75 |
| signage | roadworks-going-on | 60 |
| | road-closed | 55 |
| speed_bump_object | cross-walk | 55 |
| height_restriction_signs | height-restriction-sign-Xft-Yin | 60 |
Lane Detection | lane_detector | lane | N/A |
RTG | real_time_guidance | stop-sign | 60 |
| | traffic-light | 60 |
Confidence configuration
The Live Sense SDK allows you to configure the confidence values of the models as well as the individual classes within each model.
Update confidence of a model
The following function updates the confidence of all classes available in the specified model to the provided value.
func updateModelConfidence(modelKey : String, modelConfidence : Float)
The modelKey
can be used from the table described in the Model Types and Detection Classes section.
Update confidence of a class
The following function updates the confidence of the specified class to the provided value. This supersedes the confidence configured for the parent model.
func updateClassConfidence(classLabel : ClassLabelName, classConfidence : Float)
The classLabel
can be used from the table described in the Model Types and Detection Classes section.
Get confidence of a model or a class
func getConfidenceThreshold(modelKey : String, classLabel:ClassLabelName?) -> Float
The modelKey
and classLabel
can be used from the table described in the Model Types and Detection Classes section.
Warning :: Limitations
Using a lower confidence threshold will result in more detections, but an overall lower accuracy. For this reason, applications should keep their confidence threshold at or above the recommended value to maintain a reasonably accurate output.
Detections work better in Landscape
mode.
Note
- In addition, these methods can be called during a running session.
- For the following labels, class-wise confidence is not supported:
brake-light-on
DS_SpeedLimit[5-130]_White_Circle_00
DS_SpeedLimit[5-130]_White_Rectangle_01
Model execution options
Live Sense SDK supports multiple instances for executing the Live Sense models. Exact support depends upon the device and model being used.
Note
If a configuration cannot be used as-is, the SDK will fall back to the next best supported configuration.
Object recognition
A Live Sense LSDRecognition
describes the following properties of a detected object:
- class - Which object was detected. For more information, see Model Types and Detection Classes.
- location - Where the object was found in the image frame.
- confidence score - A number between 0 and 1 that indicates confidence that the object was correctly detected.
Model runtime support for underlying architecture
| CPU | GPU/Metal (A6 Chip) | ANE(A12 Chip) |
Road Basics | YES | YES | YES |
Road Alerts | YES | YES | YES |
Road Signs | YES | YES | YES |
Road Hazards | YES | YES | YES |
Distance and position estimation
Note
This feature is in Beta, the returned distance and relative position values may be inaccurate.
All of the object detection models listed above can provide an estimated distance and position relative to the camera's point of view for each detected object.
The output values for the distance
and relativePosition
properties of a detected object are as follows:
- Height: Distance of the detection from the ground.
- Lateral: Distance of the detection relative to the center of the camera's view. This value may be positive or negative with a negative value signifying that the detection is towards the left side of the camera's view and vise-versa.
- Depth: Distance that the device needs to travel forward in a straight line so that the actual distance is equal to the lateral distance.
- Distance: Actual distance of the detection from the camera's point of view.
All distance values are provided in meters and are used in the following features of Live Sense SDK
- Time-to-collision alerts
- Relative velocity estimation
Road signs
Group | Label | Sample Image |
SpeedLimit | DS_SpeedLimitXX_White_Circle_00 |  |
| DS_SpeedLimitXX_White_Circle_01 |  |
| DS_SpeedLimitXX_White_Rectangle_01 |  |
AdvisorySpeedLimit | DS_AdvisorySpeedLimitXX_Yellow_Rectangle_00 |  |
| DS_AdvisorySpeedLimitXX_Yellow_Rectangle_01 |  |
| DS_AdvisorySpeedLimitXX_Yellow_Rectangle_03 |  |
Stop | DS_Stop_Red_Octogon_00 |  |
DoNotEnter | DS_DoNotEnter_Red_Circle_00 |  |
| DS_DoNotEnter_White_Rectangle_00 |  |
| DS_DoNotEnter_White_Rectangle_01 |  |
NoAccess | DS_NoAccess_White_Circle_00 |  |
StartConstruction | DS_StartConstruction_White_Triangle_00 |  |
Yield | DS_Yield_White_Triangle_00 |  |
PedstrianCrossing | DS_PedestrianCrossing_Blue_Rectangle_00 |  |
| DS_PedestrianCrossing_White_Triangle_00 |  |
| DS_PedestrianCrossing_Yellow_Circle_00 |  |
| DS_PedestrianCrossing_Yellow_Diamond_01 |  |
| DS_PedestrianCrossing_Yellow_Triangle_00 |  |
SpeedBumps | DS_SpeedBumps_Blue_Rectangle_00 |  |
| DS_SpeedBumps_White_Triangle_00 |  |
| DS_SpeedBumps_Yellow_Diamond_01 |  |
StopHereOnRed | DS_StopHereOnRed_White_Rectangle_00 |  |
NoLeftTurn | DS_NoLeftTurn_White_Circle_00 |  |
| DS_NoLeftTurn_White_Rectangle_02 |  |
NoRightOrLeftTurn | DS_NoRightOrLeftTurn_White_Rectangle_00 |  |
| DS_NoRightOrLeftTurn_White_Circle_00 |  |
NoUTurn | DS_NoUTurn_White_Circle_00 |  |
NoUTurnOrLeftTurn | DS_NoUTurnOrLeftTurn_White_Rectangle_00 |  |
ParkingRestriction | DS_ParkingRestriction_Blue_Circle_00 |  |
| DS_ParkingRestriction_White_Circle_01 |  |
| DS_ParkingRestriction_White_Rectangle_03 |  |