Track-to-truth metrics

class stonesoup.metricgenerator.tracktotruthmetrics.SIAPMetrics(position_weighting=None, velocity_weighting=None, position_mapping=None, velocity_mapping=None, truth_id=None, track_id=None)[source]

Bases: stonesoup.metricgenerator.base.MetricGenerator

SIAP Metrics

Computes the Single Integrated Air Picture (SIAP) metrics as defined by the Systems Engineering Task Force. The implementation provided here is derived from [1] and focuses on providing the SIAP attribute measures.

The SIAP metrics provided require provision of ground truth information.

Additionally, when relevant metadata properties track_id and truth_id are provided, calculates the ID-based SIAPS: ID Completeness (CID), ID Correctness (IDC) and ID Ambiguity (IDA). This implementation assumes that track and ground truth path IDs are implemented via metadata, whereby the strings track_id and truth_id are keys to track and truth metadata entries with ID data respectively.

In the original paper the calculations are dependent upon \(m\) which corresponds to the identifying number of the sense capability which is being assessed. This is not used in this implementation, with the assumption being that the fused sensor set is being assessed.

Note: Track types store metadata outside of their states attribute. Therefore the ID SIAPs make metadata comparisons via the tracks last ID metadata value (as calling track.metadata will return the track’s metadata at the end of its life). To provide a better implementation, one might modify Track types to contain a list of state types that hold their own metadata.

Reference

[1] Single Integrated Air Picture (SIAP) Metrics Implementation, Votruba et al, 29-10-2001

Parameters
  • position_weighting (numpy.ndarray, optional) – Weighting(s) to be used by euclidean measure in position kinematic accuracy calculations. If None, weights are all 1

  • velocity_weighting (numpy.ndarray, optional) – Weighting(s) to be used by euclidean measure in velocity kinematic accuracy calculations. If None, weights are all 1

  • position_mapping (numpy.ndarray, optional) – Mapping array which specifies which elements within state space state vectors correspond to position

  • velocity_mapping (numpy.ndarray, optional) – Mapping array which specifies which elements within state space state vectors correspond to velocity

  • truth_id (str, optional) – Metadata key for ID of each ground truth path in dataset

  • track_id (str, optional) – Metadata key for ID of each track in dataset

position_weighting: numpy.ndarray

Weighting(s) to be used by euclidean measure in position kinematic accuracy calculations. If None, weights are all 1

velocity_weighting: numpy.ndarray

Weighting(s) to be used by euclidean measure in velocity kinematic accuracy calculations. If None, weights are all 1

position_mapping: numpy.ndarray

Mapping array which specifies which elements within state space state vectors correspond to position

velocity_mapping: numpy.ndarray

Mapping array which specifies which elements within state space state vectors correspond to velocity

truth_id: str

Metadata key for ID of each ground truth path in dataset

track_id: str

Metadata key for ID of each track in dataset

compute_metric(manager, *args, **kwargs)[source]

Compute metric

Parameters

manager (MetricManager) – containing the data to be used to create the metric(s)

Returns

Generated metrics

Return type

list of Metric objects

C_single_time(manager, timestamp)[source]

SIAP metric C at a specific time

Returns an assessment of the number of targets currently being tracked compared to the number of true targets at a specific timestamp, \({t}\). The output is a percentage, range 0:1, with a target score of 1

\[C_{t} = \frac{J{T_m}({t})}{J({t})}\]
where

\(J{T_m}({t})\) is the number of objects being tracked at timestamp \({t}\)

and

\(J({t})\) is the number of true objects at timestamp \({t}\).

Parameters
  • manager (MetricManager) – containing the data to be used to create the metric(s)

  • timestamp (datetime.datetime) – timestamp at which to compute the metric

Returns

Contains the metric information

Return type

SingleTimeMetric

C_time_range(manager)[source]

SIAP metric C over time

Returns an assessment of the number of targets currently being tracked compared to the number of true targets over the time range of the dataset. The output is a percentage, range \(0:1\), with a score of 1

\[C = \frac{\sum_{t_{start}}^{t_{end}}J{T_m}({t})} {\sum_{t_{start}}^{t_{end}}J({t})}\]
where

\(J{T_m}({t})\) is the number of objects being tracked at timestamp \({t}\)

and

\(J({t})\) is the number of true objects at timestamp \({t}\).

Parameters

manager (MetricManager) – Containing the data to be used to create the metric(s)

Returns

Contains the metric information

Return type

TimeRangeMetric

A_single_time(manager, timestamp)[source]

SIAP metric A at a specific time

Returns an assessment of the number of tracks which are assigned to true objects against the total number of tracks, at a specific timestamp, \({t}\). The output is a percentage, range \(0:\infty\), with a target score of 1

\[A_{t} = \frac{N{A}({t})}{J{T_m}({t})}\]
where

\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\)

and

\(J{T_m}({t})\) is the number of objects being tracked at timestamp \({t}\).

Parameters
  • manager (MetricManager) – containing the data to be used to create the metric(s)

  • timestamp (datetime.datetime) – timestamp at which to compute the metric

Returns

Contains the metric information

Return type

SingleTimeMetric

A_time_range(manager)[source]

SIAP metric A over time

Returns a percentage value which assesses the number of tracks which are assigned to true objects against the total number of tracks. The target score is 1.

\[A = \frac{\sum_{t_{start}}^{t_{end}}N{A}({t})} {\sum_{t_{start}}^{t_{end}}J{T_m}({t})}\]
where

\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\)

and

\(J{T_m}({t})\) is the number of objects being tracked at timestamp \({t}\).

Parameters

manager (MetricManager) – Containing the data to be used to create the metric

Returns

Contains the metric information

Return type

TimeRangeMetric

S_single_time(manager, timestamp)[source]

SIAP metric S at a specific time

Returns an assessment of the number of tracks that are deemed to be spurious, i.e. unassigned to true objects, at a specific timestamp, \({t}\). The output is a percentage, range \(0:\infty\), with a target score of 0

\[S_{t} = \frac{N({t}) - N{A}({t})}{N({t})}\]
where

\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\)

and

\(N({t})\) is the number of tracks timestamp \({t}\).

Parameters
  • manager (MetricManager) – containing the data to be used to create the metric(s)

  • timestamp (datetime.datetime) – timestamp at which to compute the metric

Returns

Contains the metric information

Return type

SingleTimeMetric

S_time_range(manager)[source]

SIAP metric S over time

The average percentage of tracks that are deemed to be spurious, i.e. unassigned to true objects.

\[S = \frac{\sum_{t_{start}}^{t_{end}}[N({t}) - N{A}({t})]} {\sum_{t_{start}}^{t_{end}}N({t})}\]
where

\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\)

and

\(N({t})\) is the number of tracks timestamp \({t}\).

Parameters

manager (MetricManager) – Containing the data to be used to create the metric

Returns

Contains the metric information

Return type

TimeRangeMetric

LT(manager)[source]

SIAP metric LT over time

Returns \(1/{R}\) where \({R}\) is the average number of excess tracks assigned. Target score is \(LT = \infty\)

Parameters

manager (MetricManager) – Containing the data to be used to create the metric

Returns

Contains the metric information

Return type

TimeRangeMetric

LS(manager)[source]

SIAP metric LS over time

Returns the percentage of time that true objects have been tracked across the dataset

\[LS = \frac{\sum_{j=1}^{J}T{L}_{j}}{\sum_{j=1}^{J}T_{j}}\]
where

\(\sum_{j=1}^{J}T{L}_{j}\) is the total time of the longest track on object \(j\).

and

\(\sum_{j=1}^{J}T_{j}\) is the total duration of true object \(j\).

Parameters

manager (MetricManager) – Containing the data to be used to create the metric

Returns

Contains the metric information

Return type

TimeRangeMetric

PA_single_time(manager, timestamp)[source]

SIAP metric PA at a specific time

Returns an assessment of the average assigned track positional accuracy at a specific timestamp, \({t}\). The output is a distance measure, range \(0:\infty\), with a target score of 0

\[PA_{t} = \frac{{\sum_{n\in D(t)}PA_{n}(t)}}{NA(t)}\]
where

\(D(t)\) is the set of tracks held at timestamp \(t\) \(PA_{n}(t)\) is the Euclidean distance of track n to its associated truth at timestamp \({t}\)

and

\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\).

Parameters
  • manager (MetricManager) – containing the data to be used to create the metric(s)

  • timestamp (datetime.datetime) – timestamp at which to compute the metric

Returns

Contains the metric information

Return type

SingleTimeMetric

PA(manager)[source]

SIAP metric PA over time

The average positional accuracy of associated tracks.

\[PA = \frac{\sum_{t_{start}}^{t_{end}}{\sum_{n\in D(t)}PA_{n}(t)}} {\sum_{t_{start}}^{t_{end}}{NA(t)}}\]
where

\(D(t)\) is the set of tracks held at timestamp \(t\) \(PA_{n}(t)\) is the Euclidean distance of track n to its associated truth at timestamp \({t}\)

and

\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\).

Parameters

manager (MetricManager) – Containing the data to be used to create the metric

Returns

Contains the metric information

Return type

TimeRangeMetric

VA_single_time(manager, timestamp)[source]

SIAP metric VA at a specific time

Returns an assessment of the average assigned track velocity accuracy at a specific timestamp, \({t}\). The output is a distance measure, range \(0:\infty\), with a target score of 0

\[VA_{t} = \frac{{\sum_{n\in D(t)}VA_{n}(t)}}{NA(t)}\]
where

\(D(t)\) is the set of tracks held at timestamp \(t\), \(VA_{n}(t)\) is the Euclidean distance of track n’s velocity components to its associated truth’s corresponding velocities at timestamp \({t}\)

and

\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\).

Parameters
  • manager (MetricManager) – containing the data to be used to create the metric(s)

  • timestamp (datetime.datetime) – timestamp at which to compute the metric

Returns

Contains the metric information

Return type

SingleTimeMetric

VA(manager)[source]

SIAP metric VA over time

The average velocity accuracy of associated tracks.

\[VA = \frac{\sum_{t_{start}}^{t_{end}}{\sum_{n\in D(t)}VA_{n}(t)}} {\sum_{t_{start}}^{t_{end}}{NA(t)}}\]
where

\(D(t)\) is the set of tracks held at timestamp \(t\) \(VA_{n}(t)\) is the Euclidean distance of track n’s velocity components to its associated truth’s corresponding velocities at timestamp \({t}\)

and

\(N{A}({t})\) is the number of tracks assigned to true objects at timestamp \({t}\).

Parameters

manager (MetricManager) – Containing the data to be used to create the metric

Returns

Contains the metric information

Return type

TimeRangeMetric

num_tracks(manager)[source]

Calculates the number of tracks stored in the metric manager :param manager: Containing the data to be used to create the metric :type manager: MetricManager

Returns

Contains the metric information

Return type

TimeRangeMetric

num_truths(manager)[source]

Calculates the number of truths stored in the metric manager :param manager: Containing the data to be used to create the metric :type manager: MetricManager

Returns

Contains the metric information

Return type

TimeRangeMetric

CID_single_time(manager, timestamp)[source]

SIAP metric CID at a specific time

Returns an assessment of the number of targets currently being tracked with assigned tracks with known IDs, compared to the number of targets being tracked at a specific timestamp, \({t}\). The output is a percentage, range 0:1, with a target score of ?

\[CID_{t} = \frac{J{T}({t}) - J{U}({t})}{JT({t})}\]
where

\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\)

and

\(J{U}({t})\) is the number of number of truths tracked with unknown ID at timestamp \({t}\).

Parameters
  • manager (MetricManager) – containing the data to be used to create the metric(s)

  • timestamp (datetime.datetime) – timestamp at which to compute the metric

Returns

Contains the metric information

Return type

SingleTimeMetric

CID_time_range(manager)[source]

SIAP metric CID over time

The average percentage of targets being tracked with assigned tracks with known IDs across the dataset. The target score is 1.

\[CID = \frac{\sum_{t_{start}}^{t_{end}}[J{T}({t}) - J{U}({T})]} {\sum_{t_{start}}^{t_{end}}J{T}({t})}\]
where

\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\)

and

\(J{U}({t})\) is the number of number of truths tracked with unknown ID at timestamp \({t}\).

Parameters

manager (MetricManager) – Containing the data to be used to create the metric

Returns

Contains the metric information

Return type

TimeRangeMetric

IDC_single_time(manager, timestamp)[source]

SIAP metric IDc at a specific time

Returns an assessment of the number of targets currently being tracked with the correct ID, compared to the number of targets being tracked at a specific timestamp, \({t}\). The output is a percentage, range 0:1, with a target score of 1

\[IDC_{t} = \frac{J{C}({t})}{JT({t})}\]
where

\(J{C}({t})\) is the number of number of truths tracked with correct ID at timestamp \({t}\)

and

\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\).

Parameters
  • manager (MetricManager) – containing the data to be used to create the metric(s)

  • timestamp (datetime.datetime) – timestamp at which to compute the metric

Returns

Contains the metric information

Return type

SingleTimeMetric

IDC_time_range(manager)[source]

SIAP metric IDC over time

The average percentage of targets being tracked with the correct ID across the dataset. The target score is 1.

\[IDC = \frac{\sum_{t_{start}}^{t_{end}}J{C}({t})}{\sum_{t_{start}}^{t_{end}}J{T}({t})}\]
where

\(J{C}({t})\) is the number of number of truths tracked with correct ID at timestamp \({t}\)

and

\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\).

Parameters

manager (MetricManager) – Containing the data to be used to create the metric

Returns

Contains the metric information

Return type

TimeRangeMetric

IDA_single_time(manager, timestamp)[source]

SIAP metric IDc at a specific time

Returns an assessment of the number of targets currently being tracked with ambiguous ID, compared to the number of targets being tracked at a specific timestamp, \({t}\). An object’s ID is considered ambiguous if it has multiple tracks with correct and incorrect IDs.The output is a percentage, range 0:1, with a target score of 1

\[IDA_{t} = \frac{J{A}({t})}{JT({t})}\]
where

\(J{A}({t}) = J{T}({t}) - J{C}({t}) - J{I}({t}) - J{U}({t})\) is the number of number of truths tracked with ambiguous ID at timestamp \({t}\), \(J{C}({t}), J{I}({t}), J{U}({t})\) are the number of truths tracked with correct, incorrect and unkown (no) ID at timestamp \(t\) respectively.

and

\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\).

Parameters
  • manager (MetricManager) – containing the data to be used to create the metric(s)

  • timestamp (datetime.datetime) – timestamp at which to compute the metric

Returns

Contains the metric information

Return type

SingleTimeMetric

IDA_time_range(manager)[source]

SIAP metric IDC over time

The average percentage of targets being tracked with ambiguous ID across the dataset. The target score is 1.

\[IDA = \frac{\sum_{t_{start}}^{t_{end}}J{A}({t})}{\sum_{t_{start}}^{t_{end}}J{T}({t})}\]
where

\(J{A}({t}) = J{T}({t}) - J{C}({t}) - J{I}({t}) - J{U}({t})\) is the number of number of truths tracked with ambiguous ID at timestamp \({t}\), \(J{C}({t}), J{I}({t}), J{U}({t})\) are the number of truths tracked with correct, incorrect and unkown (no) ID at timestamp \(t\) respectively.

and

\(J{T}({t})\) is the number of true objects being tracked at timestamp \({t}\).

Parameters

manager (MetricManager) – Containing the data to be used to create the metric

Returns

Contains the metric information

Return type

TimeRangeMetric