Track-to-truth metrics

class stonesoup.metricgenerator.tracktotruthmetrics.SIAPMetrics(position_measure: Measure, velocity_measure: Measure, generator_name: str = 'siap_generator', tracks_key: str = 'tracks', truths_key: str = 'groundtruth_paths')[source]

Bases: MetricGenerator

SIAP Metrics

Computes the Single Integrated Air Picture (SIAP) metrics as defined by the Systems Engineering Task Force. The implementation provided here is derived from [1] and focuses on providing the SIAP attribute measures.

The SIAP metrics provided require provision of ground truth information.

In the original paper the calculations are dependent upon \(m\) which corresponds to the identifying number of the sense capability which is being assessed. This is not used in this implementation, with the assumption being that the fused sensor set is being assessed.

Metrics:

  • Completeness (C): Fraction of true objects being tracked. The output is in the range \(0:1\), with a target score of 1.

  • Ambiguity (A): Number of tracks assigned to a true object. The output is unbounded with a range of \(0:\infty\). The target score is 1.

  • Spuriousness (S): Fraction of tracks that are unassigned to a true object. The output is in the range \(0:1\), with a target score of 0.

  • Positional Accuracy (PA): Positional error of associated tracks to their respective truths. The output is a distance measure, range \(0:\infty\), with a target score of 0.

  • Velocity Accuracy (VA): Velocity error of associated tracks to their respective truths. The output is a distance measure, range \(0:\infty\), with a target score of 0.

  • Rate of track number changes (R): SIAP continuity measure. Rate of number of track changes per truth. The output is in the range \(0:\infty\), with a target score of 0.

  • Longest track Segment (LS): SIAP continuity measure. Proportion of longest associated track segment per truth. The output is in the range \(0:1\), with a target score of 1.

Reference

[1] Single Integrated Air Picture (SIAP) Metrics Implementation, Votruba et al., 29-10-2001

Parameters:

  • position_measure (Measure) – Distance measure used in calculating position accuracy scores.

  • velocity_measure (Measure) – Distance measure used in calculating velocity accuracy scores.

  • generator_name (str, optional) – Unique identifier to use when accessing generated metrics from MultiManager

  • tracks_key (str, optional) – Key to access set of tracks added to MetricManager

  • truths_key (str, optional) – Key to access set of ground truths added to MetricManager. Or key to access a second set of tracks for track-to-track metric generation

position_measure: Measure

Distance measure used in calculating position accuracy scores.

velocity_measure: Measure

Distance measure used in calculating velocity accuracy scores.

generator_name: str

Unique identifier to use when accessing generated metrics from MultiManager

tracks_key: str

Key to access set of tracks added to MetricManager

truths_key: str

Key to access set of ground truths added to MetricManager. Or key to access a second set of tracks for track-to-track metric generation

compute_metric(manager, **kwargs)[source]

Compute metrics:

\[\begin{split}\begin{alignat*}{3} \textrm{Name} &\quad \textrm{At Time} &&\quad \textrm{TimeRange}\\ C &\quad \frac{JT({t})}{J({t})} &&\quad \frac{\sum_{t_{start}}^{t_{end}}JT({t})} {\sum_{t_{start}}^{t_{end}}J({t})}\\ A &\quad \frac{N{A}({t})}{JT({t})} &&\quad \frac{\sum_{t_{start}}^{t_{end}} N{A}({t})}{\sum_{t_{start}}^{t_{end}}JT({t})}\\ S &\quad \frac{N({t}) - N{A}({t})}{N({t})} &&\quad \frac{\sum_{t_{start}}^{t_{end}} [N({t}) - N{A}({t})]}{\sum_{t_{start}}^{t_{end}}N({t})}\\ PA &\quad \frac{{\sum_{n\in tracks}PA_{n}(t)}}{NA(t)} &&\quad \frac{\sum_{t_{start}}^{t_{end}}{\sum_{n\in tracks}PA_{n}(t)}} {\sum_{t_{start}}^{t_{end}}{NA(t)}}\\ VA &\quad \frac{{\sum_{n\in tracks}VA_{n}(t)}}{NA(t)} &&\quad \frac{\sum_{t_{start}}^{t_{end}}{\sum_{n\in tracks}VA_{n}(t)}} {\sum_{t_{start}}^{t_{end}}{NA(t)}}\\ R &\quad -- &&\quad \frac{\sum_{j\in truths}NU_j-1}{\sum_{j\in truths}TT_j}\\ LS &\quad -- &&\quad \frac{\sum_{j\in truths}T{L}_{j}}{\sum_{j\in truths}T_{j}} \end{alignat*}\end{split}\]
Parameters:

manager (MetricManager) – containing the data to be used to create the metric(s)

Returns:

Generated metrics

Return type:

list of Metric objects

static num_truths_at_time(ground_truths, timestamp)[source]

\(J(t)\). Calculate the number of true objects held by manager at timestamp.

Parameters:

  • ground_truths (set or list of GroundTruthPath or Track objects) – Containing the ground truth or track data to be used

  • timestamp (datetime.datetime) – Timestamp at which to compute the value

Returns:

Number of true objects in groundtruth set or list at timestamp

Return type:

float

static num_associated_truths_at_time(manager, ground_truths, timestamp)[source]

\(JT(t)\). Calculate the number of associated true objects held by manager at timestamp.

Parameters:

  • manager (MetricManager) – Containing the data to be used

  • ground_truths (set or list of GroundTruthPath or Track objects) – Containing the groundtruth or track data to be used

  • timestamp (datetime.datetime) – Timestamp at which to compute the value

Returns:

Number of associated true objects held by manager at timestamp

Return type:

float

static num_tracks_at_time(tracks, timestamp)[source]

\(N(t)\). Calculate the number of tracks held by manager at timestamp.

Parameters:

  • tracks (set or list of Track objects) – Containing the track data to be used

  • timestamp (datetime.datetime) – Timestamp at which to compute the value

Returns:

Number of tracks in tracks set or list at timestamp

Return type:

float

static num_associated_tracks_at_time(manager, tracks, timestamp)[source]

\(NA(t)\). Calculate the number of associated tracks held by manager at timestamp.

Parameters:

  • manager (MetricManager) – Containing the data to be used

  • tracks (set or list of Track) – Containing the track data to be used

  • timestamp (datetime.datetime) – Timestamp at which to compute the value

Returns:

Number of associated tracks held by manager at timestamp.

Return type:

float

accuracy_at_time(manager, timestamp, measure)[source]

\(PA(t)\) or \(VA(t)\) (dependent on measure). Calculate the kinematic accuracy of track-truth associations held by manager at timestamp.

Parameters:

  • manager (MetricManager) – Containing the data to be used

  • timestamp (datetime.datetime) – Timestamp at which to compute the value

  • measure (Measure) – Measure used to calculate ‘distance’ between truths and tracks

Returns:

Kinematic accuracy of track-truth associations held by manager at timestamp.

Return type:

float

Note

This method adds the ‘distance’ errors for each and every association. An alternative would be to consider each true object and track at most once.

static truth_track_from_association(association)[source]

Find truth and track from an association.

Parameters:

association (Association) – Association that contains truth and track as its objects

Returns:

True object and track that are the objects of the association

Return type:

GroundTruthPath, Track

static total_time_tracked(manager, truth)[source]

\(TT\). Calculate the total time a truth is tracked for by tracks contained by manager.

Parameters:
Returns:

Number of seconds that truth is tracked for

Return type:

float

static min_num_tracks_needed_to_track(manager, truth)[source]

\(NU_j\). Calculate the minimum number of tracks needed to track truth using tracks held by manager.

Parameters:
Returns:

Minimum number of tracks needed to track truth

Return type:

int

rate_of_track_number_changes(manager, ground_truths)[source]

\(R\). Calculate the average rate of track number changes for true objects held by manager.

Parameters:

  • manager (MetricManager) – Containing the data to be used

  • ground_truths (set or list of GroundTruthPath or Track objects) – Containing the ground truth or track data to be used

Returns:

Average rate of track number changes

Return type:

float

static truth_lifetime(truth)[source]

\(T\). Calculate how long truth exists for.

Parameters:

truth (GroundTruthPath) – True object

Returns:

Number of seconds that truth exists for

Return type:

float

static longest_track_time_on_truth(manager, truth)[source]

\(TL_j\). Calculate the longest time that a single track is associated to truth using associations held by manager.

Parameters:
Returns:

Number of seconds of longest association to truth

Return type:

float

class stonesoup.metricgenerator.tracktotruthmetrics.IDSIAPMetrics(position_measure: Measure, velocity_measure: Measure, truth_id: str, track_id: str, generator_name: str = 'Id_siap_generator', tracks_key: str = 'tracks', truths_key: str = 'groundtruth_paths')[source]

Bases: SIAPMetrics

ID-based SIAP Metrics

SIAP metric generator that additionally computes ID-based SIAP metrics.

ID-based Metrics:

  • ID Completeness (CID): ID-based SIAP. Fraction of true objects with an assigned ID. The output is in the range \(0:1\), with a target score of 1.

  • ID Correctness (IDC): ID-based SIAP. Fraction of true objects with correct ID assignment. The output is in the range \(0:1\), with a target score of 1.

  • ID Ambiguity (IDA): ID-based SIAP. Fraction of true objects with ambiguous ID assignment. The output is in the range \(0:1\), with a target score of 0.

Notes

  • This implementation assumes that track and ground truth path IDs are implemented via metadata, whereby the strings track_id and truth_id are keys to track and truth metadata entries with ID data respectively.

  • Track types store metadata outside of their states attribute. Therefore, the ID SIAPs make metadata comparisons via the tracks’ last ID metadata values (as calling track.metadata will return a track’s metadata at the end of its life). To provide a better implementation, one might modify Track types to contain a list of state types that hold their own metadata.

Reference

[1] Single Integrated Air Picture (SIAP) Metrics Implementation, Votruba et al., 29-10-2001

Parameters:

  • position_measure (Measure) – Distance measure used in calculating position accuracy scores.

  • velocity_measure (Measure) – Distance measure used in calculating velocity accuracy scores.

  • truth_id (str) – Metadata key for ID of each ground truth path in data-set

  • track_id (str) – Metadata key for ID of each track in data-set

  • generator_name (str, optional) – Unique identifier to use when accessing generated metrics from MultiManager

  • tracks_key (str, optional) – Key to access set of tracks added to MetricManager

  • truths_key (str, optional) – Key to access set of ground truths added to MetricManager. Or key to access a second set of tracks for track-to-track metric generation

truth_id: str

Metadata key for ID of each ground truth path in data-set

track_id: str

Metadata key for ID of each track in data-set

generator_name: str

Unique identifier to use when accessing generated metrics from MultiManager

compute_metric(manager, **kwargs)[source]

Compute metrics:

\[\begin{split}\begin{alignat*}{3} \textrm{Name} &\quad \textrm{At Time} &&\quad \textrm{TimeRange}\\ C &\quad \frac{JT({t})}{J({t})} &&\quad \frac{\sum_{t_{start}}^{t_{end}}JT({t})} {\sum_{t_{start}}^{t_{end}}J({t})}\\ A &\quad \frac{N{A}({t})}{JT({t})} &&\quad \frac{\sum_{t_{start}}^{t_{end}} N{A}({t})}{\sum_{t_{start}}^{t_{end}}JT({t})}\\ S &\quad \frac{N({t}) - N{A}({t})}{N({t})} &&\quad \frac{\sum_{t_{start}}^{t_{end}} [N({t}) - N{A}({t})]}{\sum_{t_{start}}^{t_{end}}N({t})}\\ PA &\quad \frac{{\sum_{n\in tracks}PA_{n}(t)}}{NA(t)} &&\quad \frac{\sum_{t_{start}}^{t_{end}}{\sum_{n\in tracks}PA_{n}(t)}} {\sum_{t_{start}}^{t_{end}}{NA(t)}}\\ VA &\quad \frac{{\sum_{n\in tracks}VA_{n}(t)}}{NA(t)} &&\quad \frac{\sum_{t_{start}}^{t_{end}}{\sum_{n\in tracks}VA_{n}(t)}} {\sum_{t_{start}}^{t_{end}}{NA(t)}}\\ R &\quad -- &&\quad \frac{\sum_{j\in truths}NU_j-1}{\sum_{j\in truths}TT_j}\\ LS &\quad -- &&\quad \frac{\sum_{j\in truths}T{L}_{j}}{\sum_{j\in truths}T_{j}}\\ CID &\quad \frac{J{T}({t}) - J{U}({t})}{JT({t})} &&\quad \frac{\sum_{t_{start}}^{t_{end}}[J{T}({t}) - J{U}({T})]} {\sum_{t_{start}}^{t_{end}}J{T}({t})}\\ IDC &\quad \frac{J{C}({t})}{JT({t})} &&\quad \frac{\sum_{t_{start}}^{t_{end}} J{C}({t})}{\sum_{t_{start}}^{t_{end}}J{T}({t})}\\ IDA &\quad \frac{J{A}({t})}{JT({t})} &&\quad \frac{\sum_{t_{start}}^{t_{end}} J{A}({t})}{\sum_{t_{start}}^{t_{end}}J{T}({t})} \end{alignat*}\end{split}\]
Parameters:

manager (MetricManager) – containing the data to be used to create the metric(s)

Returns:

Generated metrics

Return type:

list of Metric objects

find_track_id(track, timestamp)[source]

Find track ID at timestamp.

Parameters:
Returns:

track ID at timestamp

Return type:

any

num_id_truths_at_time(manager, ground_truths, timestamp)[source]
\(JU\), \(JC\), \(JI\). Calculate the number of true objects that are:

Un-identified, correctly identified, incorrectly identified at timestamp according to associations held by manager.

Parameters:

  • manager (MetricManager) – Containing the data to be used

  • ground_truths (set or list of GroundTruthPath or Track objects) – Containing the ground truth or track data to be used

  • timestamp (datetime.datetime) – Timestamp at which to consider associations

Returns:

Number of true objects: Un-identified, correctly identified, incorrectly identified

Return type:

int, int, int

Note

  • A true object is considered to be un-identified if all tracks associated with it at timestamp have no ID.

  • A true object is considered to be correctly identified if all tracks associated with it at timestamp have the same ID as it.

  • A true object is considered to be incorrectly identified if all tracks associated with it at timestamp have a different ID to it.

  • A true object is considered to have ambiguous identification if tracks associated with it have differing ID (this value is calculated in the compute_metric() method).