Tracktotruth metrics¶

class
stonesoup.metricgenerator.tracktotruthmetrics.
SIAPMetrics
(position_weighting: numpy.ndarray = None, velocity_weighting: numpy.ndarray = None, position_mapping: numpy.ndarray = None, velocity_mapping: numpy.ndarray = None, truth_id: str = None, track_id: str = None)[source]¶ Bases:
stonesoup.metricgenerator.base.MetricGenerator
SIAP Metrics
Computes the Single Integrated Air Picture (SIAP) metrics as defined by the Systems Engineering Task Force. The implementation provided here is derived from [1] and focuses on providing the SIAP attribute measures.
The SIAP metrics provided require provision of ground truth information.
Additionally, when relevant metadata properties
track_id
andtruth_id
are provided, calculates the IDbased SIAPS: ID Completeness (CID), ID Correctness (IDC) and ID Ambiguity (IDA). This implementation assumes that track and ground truth path IDs are implemented via metadata, whereby the stringstrack_id
andtruth_id
are keys to track and truth metadata entries with ID data respectively.In the original paper the calculations are dependent upon \(m\) which corresponds to the identifying number of the sense capability which is being assessed. This is not used in this implementation, with the assumption being that the fused sensor set is being assessed.
Note:
Track
types store metadata outside of their states attribute. Therefore the ID SIAPs make metadata comparisons via the tracks last ID metadata value (as calling track.metadata will return the track’s metadata at the end of its life). To provide a better implementation, one might modifyTrack
types to contain a list of state types that hold their own metadata. Reference
[1] Single Integrated Air Picture (SIAP) Metrics Implementation, Votruba et al, 29102001
 Parameters
position_weighting (
numpy.ndarray
, optional) – Weighting(s) to be used by euclidean measure in position kinematic accuracy calculations. If None, weights are all 1velocity_weighting (
numpy.ndarray
, optional) – Weighting(s) to be used by euclidean measure in velocity kinematic accuracy calculations. If None, weights are all 1position_mapping (
numpy.ndarray
, optional) – Mapping array which specifies which elements within state space state vectors correspond to positionvelocity_mapping (
numpy.ndarray
, optional) – Mapping array which specifies which elements within state space state vectors correspond to velocitytruth_id (
str
, optional) – Metadata key for ID of each ground truth path in datasettrack_id (
str
, optional) – Metadata key for ID of each track in dataset

position_weighting
: numpy.ndarray¶ Weighting(s) to be used by euclidean measure in position kinematic accuracy calculations. If None, weights are all 1

velocity_weighting
: numpy.ndarray¶ Weighting(s) to be used by euclidean measure in velocity kinematic accuracy calculations. If None, weights are all 1

position_mapping
: numpy.ndarray¶ Mapping array which specifies which elements within state space state vectors correspond to position

velocity_mapping
: numpy.ndarray¶ Mapping array which specifies which elements within state space state vectors correspond to velocity

compute_metric
(manager, *args, **kwargs)[source]¶ Compute metric
 Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
 Returns
Generated metrics
 Return type
list of
Metric
objects

C_single_time
(manager, timestamp)[source]¶ SIAP metric C “Completeness” at a specific time
Returns an assessment of the number of true targets currently being tracked compared to the number of true targets at a specific timestamp, \({t}\). The output is a percentage, range \(0:1\), with a target score of 1
\[C_{t} = \frac{J{T_m}({t})}{J({t})}\] where
\(J{T_m}({t})\) is the number of objects being tracked at timestamp \({t}\)
 and
\(J({t})\) is the number of true objects at timestamp \({t}\).
 Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
timestamp (datetime.datetime) – timestamp at which to compute the metric
 Returns
Contains the metric information
 Return type

C_time_range
(manager)[source]¶ SIAP metric C “Completeness” over time
Returns an assessment of the number of targets currently being tracked compared to the number of true targets over the time range of the dataset. The output is a percentage, range \(0:1\), with a score of 1
\[C = \frac{\sum_{t_{start}}^{t_{end}}J{T_m}({t})} {\sum_{t_{start}}^{t_{end}}J({t})}\] where
\(J{T_m}({t})\) is the number of objects being tracked at timestamp \({t}\)
 and
\(J({t})\) is the number of true objects at timestamp \({t}\).
 Parameters
manager (MetricManager) – Containing the data to be used to create the metric(s)
 Returns
Contains the metric information
 Return type

A_single_time
(manager, timestamp)[source]¶ SIAP metric A “Ambiguity” at a specific time
Returns an assessment of the number of tracks which are assigned to true objects against the total number of tracks, at a specific timestamp, \({t}\). The output is unbounded with a range of \(0:\infty\). The optimal target score for Ambiguity is 1.
\[A_{t} = \frac{N{A}({t})}{J{T_m}({t})}\] where
\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\)
 and
\(J{T_m}({t})\) is the number of objects being tracked at timestamp \({t}\).
 Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
timestamp (datetime.datetime) – timestamp at which to compute the metric
 Returns
Contains the metric information
 Return type

A_time_range
(manager)[source]¶ SIAP metric A “Ambiguity” over time
Returns a percentage value which assesses the number of tracks which are assigned to true objects against the total number of tracks. The output is unbounded with a range of \(0:\infty\). The optimal target score for Ambiguity is 1.
\[A = \frac{\sum_{t_{start}}^{t_{end}}N{A}({t})} {\sum_{t_{start}}^{t_{end}}J{T_m}({t})}\] where
\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\)
 and
\(J{T_m}({t})\) is the number of objects being tracked at timestamp \({t}\).
 Parameters
manager (MetricManager) – Containing the data to be used to create the metric
 Returns
Contains the metric information
 Return type

S_single_time
(manager, timestamp)[source]¶ SIAP metric S “Spuriousness” at a specific time
Returns an assessment of the number of tracks that are deemed to be spurious, i.e. unassigned to true objects, at a specific timestamp, \({t}\). The output is a percentage, range \(0:1\), with a target score of 0.
\[S_{t} = \frac{N({t})  N{A}({t})}{N({t})}\] where
\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\)
 and
\(N({t})\) is the number of tracks timestamp \({t}\).
 Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
timestamp (datetime.datetime) – timestamp at which to compute the metric
 Returns
Contains the metric information
 Return type

S_time_range
(manager)[source]¶ SIAP metric S Spuriousness” over time
The average percentage of tracks that are deemed to be spurious, i.e. unassigned to true objects. The output is a percentage, range \(0:1\), with a target score of 0.
\[S = \frac{\sum_{t_{start}}^{t_{end}}[N({t})  N{A}({t})]} {\sum_{t_{start}}^{t_{end}}N({t})}\] where
\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\)
 and
\(N({t})\) is the number of tracks timestamp \({t}\).
 Parameters
manager (MetricManager) – Containing the data to be used to create the metric
 Returns
Contains the metric information
 Return type

LT
(manager)[source]¶ SIAP metric LT over time
Returns \(1/{R}\) where \({R}\) is the average number of excess tracks assigned. The output is unbounded with a range of \(0:\infty\), target score is \(LT = \infty\).
 Parameters
manager (MetricManager) – Containing the data to be used to create the metric
 Returns
Contains the metric information
 Return type

LS
(manager)[source]¶ SIAP metric LS over time
Returns the percentage of time that true objects have been tracked across the dataset. The output is a percentage, range \(0:1\), with a target score of 1.
\[LS = \frac{\sum_{j=1}^{J}T{L}_{j}}{\sum_{j=1}^{J}T_{j}}\] where
\(\sum_{j=1}^{J}T{L}_{j}\) is the total time of the longest track on object \(j\).
 and
\(\sum_{j=1}^{J}T_{j}\) is the total duration of true object \(j\).
 Parameters
manager (MetricManager) – Containing the data to be used to create the metric
 Returns
Contains the metric information
 Return type

PA_single_time
(manager, timestamp)[source]¶ SIAP metric PA at a specific time
Returns an assessment of the average assigned track positional accuracy at a specific timestamp, \({t}\). The output is a distance measure, range \(0:\infty\), with a target score of 0.
\[PA_{t} = \frac{{\sum_{n\in D(t)}PA_{n}(t)}}{NA(t)}\] where
\(D(t)\) is the set of tracks held at timestamp \(t\) \(PA_{n}(t)\) is the Euclidean distance of track n to its associated truth at timestamp \({t}\)
 and
\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\).
 Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
timestamp (datetime.datetime) – timestamp at which to compute the metric
 Returns
Contains the metric information
 Return type

PA
(manager)[source]¶ SIAP metric PA over time
The average positional accuracy of associated tracks .The output is a distance measure, range \(0:\infty\), with a target score of 0.
\[PA = \frac{\sum_{t_{start}}^{t_{end}}{\sum_{n\in D(t)}PA_{n}(t)}} {\sum_{t_{start}}^{t_{end}}{NA(t)}}\] where
\(D(t)\) is the set of tracks held at timestamp \(t\) \(PA_{n}(t)\) is the Euclidean distance of track n to its associated truth at timestamp \({t}\)
 and
\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\).
 Parameters
manager (MetricManager) – Containing the data to be used to create the metric
 Returns
Contains the metric information
 Return type

VA_single_time
(manager, timestamp)[source]¶ SIAP metric VA at a specific time
Returns an assessment of the average assigned track velocity accuracy at a specific timestamp, \({t}\). The output is a distance measure, range \(0:\infty\), with a target score of 0.
\[VA_{t} = \frac{{\sum_{n\in D(t)}VA_{n}(t)}}{NA(t)}\] where
\(D(t)\) is the set of tracks held at timestamp \(t\), \(VA_{n}(t)\) is the Euclidean distance of track n’s velocity components to its associated truth’s corresponding velocities at timestamp \({t}\)
 and
\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\).
 Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
timestamp (datetime.datetime) – timestamp at which to compute the metric
 Returns
Contains the metric information
 Return type

VA
(manager)[source]¶ SIAP metric VA over time
The average velocity accuracy of associated tracks.The output is a distance measure, range \(0:\infty\), with a target score of 0.
\[VA = \frac{\sum_{t_{start}}^{t_{end}}{\sum_{n\in D(t)}VA_{n}(t)}} {\sum_{t_{start}}^{t_{end}}{NA(t)}}\] where
\(D(t)\) is the set of tracks held at timestamp \(t\) \(VA_{n}(t)\) is the Euclidean distance of track n’s velocity components to its associated truth’s corresponding velocities at timestamp \({t}\)
 and
\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\).
 Parameters
manager (MetricManager) – Containing the data to be used to create the metric
 Returns
Contains the metric information
 Return type

num_tracks
(manager)[source]¶ Calculates the number of tracks stored in the metric manager
 Parameters
manager (MetricManager) – Containing the data to be used to create the metric
 Returns
Contains the metric information
 Return type

num_truths
(manager)[source]¶ Calculates the number of truths stored in the metric manager :param manager: Containing the data to be used to create the metric :type manager: MetricManager
 Returns
Contains the metric information
 Return type

CID_single_time
(manager, timestamp)[source]¶ SIAP metric CID at a specific time
Returns an assessment of the number of targets currently being tracked with assigned tracks with known IDs, compared to the number of targets being tracked at a specific timestamp, \({t}\). The output is a percentage, range math:0:1, with a target score of 1.
\[CID_{t} = \frac{J{T}({t})  J{U}({t})}{JT({t})}\] where
\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\)
 and
\(J{U}({t})\) is the number of number of truths tracked with unknown ID at timestamp \({t}\).
 Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
timestamp (datetime.datetime) – timestamp at which to compute the metric
 Returns
Contains the metric information
 Return type

CID_time_range
(manager)[source]¶ SIAP metric CID over time
The average percentage of targets being tracked with assigned tracks with known IDs across the dataset. The output is a percentage, range math:0:1, with a target score of 1.
\[CID = \frac{\sum_{t_{start}}^{t_{end}}[J{T}({t})  J{U}({T})]} {\sum_{t_{start}}^{t_{end}}J{T}({t})}\] where
\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\)
 and
\(J{U}({t})\) is the number of number of truths tracked with unknown ID at timestamp \({t}\).
 Parameters
manager (MetricManager) – Containing the data to be used to create the metric
 Returns
Contains the metric information
 Return type

IDC_single_time
(manager, timestamp)[source]¶ SIAP metric IDc at a specific time
Returns an assessment of the number of targets currently being tracked with the correct ID, compared to the number of targets being tracked at a specific timestamp, \({t}\). The output is a percentage, range math:0:1, with a target score of 1.
\[IDC_{t} = \frac{J{C}({t})}{JT({t})}\] where
\(J{C}({t})\) is the number of number of truths tracked with correct ID at timestamp \({t}\)
 and
\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\).
 Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
timestamp (datetime.datetime) – timestamp at which to compute the metric
 Returns
Contains the metric information
 Return type

IDC_time_range
(manager)[source]¶ SIAP metric IDC over time
The average percentage of targets being tracked with the correct ID across the dataset. The output is a percentage, range math:0:1, with a target score of 1.
\[IDC = \frac{\sum_{t_{start}}^{t_{end}}J{C}({t})}{\sum_{t_{start}}^{t_{end}}J{T}({t})}\] where
\(J{C}({t})\) is the number of number of truths tracked with correct ID at timestamp \({t}\)
 and
\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\).
 Parameters
manager (MetricManager) – Containing the data to be used to create the metric
 Returns
Contains the metric information
 Return type

IDA_single_time
(manager, timestamp)[source]¶ SIAP metric IDc at a specific time
Returns an assessment of the number of targets currently being tracked with ambiguous ID, compared to the number of targets being tracked at a specific timestamp, \({t}\). An object’s ID is considered ambiguous if it has multiple tracks with correct and incorrect IDs. The output is a percentage, range math:0:1, with a target score of 0.
\[IDA_{t} = \frac{J{A}({t})}{JT({t})}\] where
\(J{A}({t}) = J{T}({t})  J{C}({t})  J{I}({t})  J{U}({t})\) is the number of number of truths tracked with ambiguous ID at timestamp \({t}\), \(J{C}({t}), J{I}({t}), J{U}({t})\) are the number of truths tracked with correct, incorrect and unkown (no) ID at timestamp \(t\) respectively.
 and
\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\).
 Parameters
manager (MetricManager) – containing the data to be used to create the metric(s)
timestamp (datetime.datetime) – timestamp at which to compute the metric
 Returns
Contains the metric information
 Return type

IDA_time_range
(manager)[source]¶ SIAP metric IDC over time
The average percentage of targets being tracked with ambiguous ID across the dataset. The output is a percentage, range math:0:1, with a target score of 0.
\[IDA = \frac{\sum_{t_{start}}^{t_{end}}J{A}({t})}{\sum_{t_{start}}^{t_{end}}J{T}({t})}\] where
\(J{A}({t}) = J{T}({t})  J{C}({t})  J{I}({t})  J{U}({t})\) is the number of number of truths tracked with ambiguous ID at timestamp \({t}\), \(J{C}({t}), J{I}({t}), J{U}({t})\) are the number of truths tracked with correct, incorrect and unkown (no) ID at timestamp \(t\) respectively.
 and
\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\).
 Parameters
manager (MetricManager) – Containing the data to be used to create the metric
 Returns
Contains the metric information
 Return type