Track-to-truth metrics¶
- class stonesoup.metricgenerator.tracktotruthmetrics.SIAPMetrics(position_weighting: numpy.ndarray = None, velocity_weighting: numpy.ndarray = None, position_mapping: numpy.ndarray = None, velocity_mapping: numpy.ndarray = None, position_mapping2: numpy.ndarray = None, velocity_mapping2: numpy.ndarray = None, truth_id: str = None, track_id: str = None)[source]¶
Bases:
stonesoup.metricgenerator.base.MetricGenerator
SIAP Metrics
Computes the Single Integrated Air Picture (SIAP) metrics as defined by the Systems Engineering Task Force. The implementation provided here is derived from [1] and focuses on providing the SIAP attribute measures.
The SIAP metrics provided require provision of ground truth information.
Additionally, when relevant metadata properties
track_id
andtruth_id
are provided, calculates the ID-based SIAPS: ID Completeness (CID), ID Correctness (IDC) and ID Ambiguity (IDA). This implementation assumes that track and ground truth path IDs are implemented via metadata, whereby the stringstrack_id
andtruth_id
are keys to track and truth metadata entries with ID data respectively.In the original paper the calculations are dependent upon \(m\) which corresponds to the identifying number of the sense capability which is being assessed. This is not used in this implementation, with the assumption being that the fused sensor set is being assessed.
Note:
Track
types store metadata outside of their states attribute. Therefore the ID SIAPs make metadata comparisons via the tracks last ID metadata value (as calling track.metadata will return the track’s metadata at the end of its life). To provide a better implementation, one might modifyTrack
types to contain a list of state types that hold their own metadata.- Reference
[1] Single Integrated Air Picture (SIAP) Metrics Implementation, Votruba et al, 29-10-2001
- Parameters
position_weighting (
numpy.ndarray
, optional) – Weighting(s) to be used by euclidean measure in position kinematic accuracy calculations. If None, weights are all 1velocity_weighting (
numpy.ndarray
, optional) – Weighting(s) to be used by euclidean measure in velocity kinematic accuracy calculations. If None, weights are all 1position_mapping (
numpy.ndarray
, optional) – Mapping array which specifies which elements within state space state vectors correspond to positionvelocity_mapping (
numpy.ndarray
, optional) – Mapping array which specifies which elements within state space state vectors correspond to velocityposition_mapping2 (
numpy.ndarray
, optional) – Mapping array which specifies which elements within the ground truth state space state vectors correspond to position. Default is same as position_mappingvelocity_mapping2 (
numpy.ndarray
, optional) – Mapping array which specifies which elements within the ground truth state space state vectors correspond to velocity. Default is same as velocity_mappingtruth_id (
str
, optional) – Metadata key for ID of each ground truth path in datasettrack_id (
str
, optional) – Metadata key for ID of each track in dataset
- position_weighting: numpy.ndarray¶
Weighting(s) to be used by euclidean measure in position kinematic accuracy calculations. If None, weights are all 1
- velocity_weighting: numpy.ndarray¶
Weighting(s) to be used by euclidean measure in velocity kinematic accuracy calculations. If None, weights are all 1
- position_mapping: numpy.ndarray¶
Mapping array which specifies which elements within state space state vectors correspond to position
- velocity_mapping: numpy.ndarray¶
Mapping array which specifies which elements within state space state vectors correspond to velocity
- position_mapping2: numpy.ndarray¶
Mapping array which specifies which elements within the ground truth state space state vectors correspond to position. Default is same as position_mapping
- velocity_mapping2: numpy.ndarray¶
Mapping array which specifies which elements within the ground truth state space state vectors correspond to velocity. Default is same as velocity_mapping
- compute_metric(manager, *args, **kwargs)[source]¶
Compute metric
- Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
- Returns
Generated metrics
- Return type
list of
Metric
objects
- C_single_time(manager, timestamp)[source]¶
SIAP metric C “Completeness” at a specific time
Returns an assessment of the number of true targets currently being tracked compared to the number of true targets at a specific timestamp, \({t}\). The output is a percentage, range \(0:1\), with a target score of 1
\[C_{t} = \frac{J{T_m}({t})}{J({t})}\]- where
\(J{T_m}({t})\) is the number of objects being tracked at timestamp \({t}\)
- and
\(J({t})\) is the number of true objects at timestamp \({t}\).
- Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
timestamp (datetime.datetime) – timestamp at which to compute the metric
- Returns
Contains the metric information
- Return type
- C_time_range(manager)[source]¶
SIAP metric C “Completeness” over time
Returns an assessment of the number of targets currently being tracked compared to the number of true targets over the time range of the dataset. The output is a percentage, range \(0:1\), with a score of 1
\[C = \frac{\sum_{t_{start}}^{t_{end}}J{T_m}({t})} {\sum_{t_{start}}^{t_{end}}J({t})}\]- where
\(J{T_m}({t})\) is the number of objects being tracked at timestamp \({t}\)
- and
\(J({t})\) is the number of true objects at timestamp \({t}\).
- Parameters
manager (MetricManager) – Containing the data to be used to create the metric(s)
- Returns
Contains the metric information
- Return type
- A_single_time(manager, timestamp)[source]¶
SIAP metric A “Ambiguity” at a specific time
Returns an assessment of the number of tracks which are assigned to true objects against the total number of tracks, at a specific timestamp, \({t}\). The output is unbounded with a range of \(0:\infty\). The optimal target score for Ambiguity is 1.
\[A_{t} = \frac{N{A}({t})}{J{T_m}({t})}\]- where
\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\)
- and
\(J{T_m}({t})\) is the number of objects being tracked at timestamp \({t}\).
- Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
timestamp (datetime.datetime) – timestamp at which to compute the metric
- Returns
Contains the metric information
- Return type
- A_time_range(manager)[source]¶
SIAP metric A “Ambiguity” over time
Returns a percentage value which assesses the number of tracks which are assigned to true objects against the total number of tracks. The output is unbounded with a range of \(0:\infty\). The optimal target score for Ambiguity is 1.
\[A = \frac{\sum_{t_{start}}^{t_{end}}N{A}({t})} {\sum_{t_{start}}^{t_{end}}J{T_m}({t})}\]- where
\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\)
- and
\(J{T_m}({t})\) is the number of objects being tracked at timestamp \({t}\).
- Parameters
manager (MetricManager) – Containing the data to be used to create the metric
- Returns
Contains the metric information
- Return type
- S_single_time(manager, timestamp)[source]¶
SIAP metric S “Spuriousness” at a specific time
Returns an assessment of the number of tracks that are deemed to be spurious, i.e. unassigned to true objects, at a specific timestamp, \({t}\). The output is a percentage, range \(0:1\), with a target score of 0.
\[S_{t} = \frac{N({t}) - N{A}({t})}{N({t})}\]- where
\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\)
- and
\(N({t})\) is the number of tracks timestamp \({t}\).
- Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
timestamp (datetime.datetime) – timestamp at which to compute the metric
- Returns
Contains the metric information
- Return type
- S_time_range(manager)[source]¶
SIAP metric S Spuriousness” over time
The average percentage of tracks that are deemed to be spurious, i.e. unassigned to true objects. The output is a percentage, range \(0:1\), with a target score of 0.
\[S = \frac{\sum_{t_{start}}^{t_{end}}[N({t}) - N{A}({t})]} {\sum_{t_{start}}^{t_{end}}N({t})}\]- where
\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\)
- and
\(N({t})\) is the number of tracks timestamp \({t}\).
- Parameters
manager (MetricManager) – Containing the data to be used to create the metric
- Returns
Contains the metric information
- Return type
- LT(manager)[source]¶
SIAP metric LT over time
Returns \(1/{R}\) where \({R}\) is the average number of excess tracks assigned. The output is unbounded with a range of \(0:\infty\), target score is \(LT = \infty\).
- Parameters
manager (MetricManager) – Containing the data to be used to create the metric
- Returns
Contains the metric information
- Return type
- LS(manager)[source]¶
SIAP metric LS over time
Returns the percentage of time that true objects have been tracked across the dataset. The output is a percentage, range \(0:1\), with a target score of 1.
\[LS = \frac{\sum_{j=1}^{J}T{L}_{j}}{\sum_{j=1}^{J}T_{j}}\]- where
\(\sum_{j=1}^{J}T{L}_{j}\) is the total time of the longest track on object \(j\).
- and
\(\sum_{j=1}^{J}T_{j}\) is the total duration of true object \(j\).
- Parameters
manager (MetricManager) – Containing the data to be used to create the metric
- Returns
Contains the metric information
- Return type
- PA_single_time(manager, timestamp)[source]¶
SIAP metric PA at a specific time
Returns an assessment of the average assigned track positional accuracy at a specific timestamp, \({t}\). The output is a distance measure, range \(0:\infty\), with a target score of 0.
\[PA_{t} = \frac{{\sum_{n\in D(t)}PA_{n}(t)}}{NA(t)}\]- where
\(D(t)\) is the set of tracks held at timestamp \(t\) \(PA_{n}(t)\) is the Euclidean distance of track n to its associated truth at timestamp \({t}\)
- and
\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\).
- Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
timestamp (datetime.datetime) – timestamp at which to compute the metric
- Returns
Contains the metric information
- Return type
- PA(manager)[source]¶
SIAP metric PA over time
The average positional accuracy of associated tracks .The output is a distance measure, range \(0:\infty\), with a target score of 0.
\[PA = \frac{\sum_{t_{start}}^{t_{end}}{\sum_{n\in D(t)}PA_{n}(t)}} {\sum_{t_{start}}^{t_{end}}{NA(t)}}\]- where
\(D(t)\) is the set of tracks held at timestamp \(t\) \(PA_{n}(t)\) is the Euclidean distance of track n to its associated truth at timestamp \({t}\)
- and
\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\).
- Parameters
manager (MetricManager) – Containing the data to be used to create the metric
- Returns
Contains the metric information
- Return type
- VA_single_time(manager, timestamp)[source]¶
SIAP metric VA at a specific time
Returns an assessment of the average assigned track velocity accuracy at a specific timestamp, \({t}\). The output is a distance measure, range \(0:\infty\), with a target score of 0.
\[VA_{t} = \frac{{\sum_{n\in D(t)}VA_{n}(t)}}{NA(t)}\]- where
\(D(t)\) is the set of tracks held at timestamp \(t\), \(VA_{n}(t)\) is the Euclidean distance of track n’s velocity components to its associated truth’s corresponding velocities at timestamp \({t}\)
- and
\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\).
- Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
timestamp (datetime.datetime) – timestamp at which to compute the metric
- Returns
Contains the metric information
- Return type
- VA(manager)[source]¶
SIAP metric VA over time
The average velocity accuracy of associated tracks.The output is a distance measure, range \(0:\infty\), with a target score of 0.
\[VA = \frac{\sum_{t_{start}}^{t_{end}}{\sum_{n\in D(t)}VA_{n}(t)}} {\sum_{t_{start}}^{t_{end}}{NA(t)}}\]- where
\(D(t)\) is the set of tracks held at timestamp \(t\) \(VA_{n}(t)\) is the Euclidean distance of track n’s velocity components to its associated truth’s corresponding velocities at timestamp \({t}\)
- and
\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\).
- Parameters
manager (MetricManager) – Containing the data to be used to create the metric
- Returns
Contains the metric information
- Return type
- num_tracks(manager)[source]¶
Calculates the number of tracks stored in the metric manager
- Parameters
manager (MetricManager) – Containing the data to be used to create the metric
- Returns
Contains the metric information
- Return type
- num_truths(manager)[source]¶
Calculates the number of truths stored in the metric manager :param manager: Containing the data to be used to create the metric :type manager: MetricManager
- Returns
Contains the metric information
- Return type
- CID_single_time(manager, timestamp)[source]¶
SIAP metric CID at a specific time
Returns an assessment of the number of targets currently being tracked with assigned tracks with known IDs, compared to the number of targets being tracked at a specific timestamp, \({t}\). The output is a percentage, range math:0:1, with a target score of 1.
\[CID_{t} = \frac{J{T}({t}) - J{U}({t})}{JT({t})}\]- where
\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\)
- and
\(J{U}({t})\) is the number of number of truths tracked with unknown ID at timestamp \({t}\).
- Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
timestamp (datetime.datetime) – timestamp at which to compute the metric
- Returns
Contains the metric information
- Return type
- CID_time_range(manager)[source]¶
SIAP metric CID over time
The average percentage of targets being tracked with assigned tracks with known IDs across the dataset. The output is a percentage, range math:0:1, with a target score of 1.
\[CID = \frac{\sum_{t_{start}}^{t_{end}}[J{T}({t}) - J{U}({T})]} {\sum_{t_{start}}^{t_{end}}J{T}({t})}\]- where
\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\)
- and
\(J{U}({t})\) is the number of number of truths tracked with unknown ID at timestamp \({t}\).
- Parameters
manager (MetricManager) – Containing the data to be used to create the metric
- Returns
Contains the metric information
- Return type
- IDC_single_time(manager, timestamp)[source]¶
SIAP metric IDc at a specific time
Returns an assessment of the number of targets currently being tracked with the correct ID, compared to the number of targets being tracked at a specific timestamp, \({t}\). The output is a percentage, range math:0:1, with a target score of 1.
\[IDC_{t} = \frac{J{C}({t})}{JT({t})}\]- where
\(J{C}({t})\) is the number of number of truths tracked with correct ID at timestamp \({t}\)
- and
\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\).
- Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
timestamp (datetime.datetime) – timestamp at which to compute the metric
- Returns
Contains the metric information
- Return type
- IDC_time_range(manager)[source]¶
SIAP metric IDC over time
The average percentage of targets being tracked with the correct ID across the dataset. The output is a percentage, range math:0:1, with a target score of 1.
\[IDC = \frac{\sum_{t_{start}}^{t_{end}}J{C}({t})}{\sum_{t_{start}}^{t_{end}}J{T}({t})}\]- where
\(J{C}({t})\) is the number of number of truths tracked with correct ID at timestamp \({t}\)
- and
\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\).
- Parameters
manager (MetricManager) – Containing the data to be used to create the metric
- Returns
Contains the metric information
- Return type
- IDA_single_time(manager, timestamp)[source]¶
SIAP metric IDc at a specific time
Returns an assessment of the number of targets currently being tracked with ambiguous ID, compared to the number of targets being tracked at a specific timestamp, \({t}\). An object’s ID is considered ambiguous if it has multiple tracks with correct and incorrect IDs. The output is a percentage, range math:0:1, with a target score of 0.
\[IDA_{t} = \frac{J{A}({t})}{JT({t})}\]- where
\(J{A}({t}) = J{T}({t}) - J{C}({t}) - J{I}({t}) - J{U}({t})\) is the number of number of truths tracked with ambiguous ID at timestamp \({t}\), \(J{C}({t}), J{I}({t}), J{U}({t})\) are the number of truths tracked with correct, incorrect and unkown (no) ID at timestamp \(t\) respectively.
- and
\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\).
- Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
timestamp (datetime.datetime) – timestamp at which to compute the metric
- Returns
Contains the metric information
- Return type
- IDA_time_range(manager)[source]¶
SIAP metric IDC over time
The average percentage of targets being tracked with ambiguous ID across the dataset. The output is a percentage, range math:0:1, with a target score of 0.
\[IDA = \frac{\sum_{t_{start}}^{t_{end}}J{A}({t})}{\sum_{t_{start}}^{t_{end}}J{T}({t})}\]- where
\(J{A}({t}) = J{T}({t}) - J{C}({t}) - J{I}({t}) - J{U}({t})\) is the number of number of truths tracked with ambiguous ID at timestamp \({t}\), \(J{C}({t}), J{I}({t}), J{U}({t})\) are the number of truths tracked with correct, incorrect and unkown (no) ID at timestamp \(t\) respectively.
- and
\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\).
- Parameters
manager (MetricManager) – Containing the data to be used to create the metric
- Returns
Contains the metric information
- Return type