Readers

Reader classes are used for getting data into the framework.

Base classes for different Readers.

class stonesoup.reader.base.Reader[source]

Bases: Base, BufferedGenerator

Reader base class

class stonesoup.reader.base.DetectionReader[source]

Bases: Reader

Detection Reader base class

abstract detections_gen()[source]

Returns a generator of detections for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of Detection – Detections generate in the time step

class stonesoup.reader.base.GroundTruthReader[source]

Bases: Reader

Ground Truth Reader base class

abstract groundtruth_paths_gen()[source]

Returns a generator of ground truth paths for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of GroundTruthPath – Ground truth paths existing in the time step

class stonesoup.reader.base.SensorDataReader[source]

Bases: Reader

Sensor Data Reader base class

abstract sensor_data_gen()[source]

Returns a generator of sensor data for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of SensorData – Sensor data generated in the time step

class stonesoup.reader.base.FrameReader[source]

Bases: SensorDataReader

FrameReader base class

A FrameReader produces SensorData in the form of ImageFrame objects.

abstract frames_gen()[source]

Returns a generator of frames for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of ImageFrame – Generated frame in the time step

sensor_data_gen()[source]

Returns a generator of frames for each time step.

Note

This is just a wrapper around (and therefore performs identically to) frames_gen().

Yields:

datetime.datetime – Datetime of current time step
set of ImageFrame – Generated frame in the time step

Base classes for use with File based readers.

class stonesoup.reader.file.FileReader(path: Path)[source]

Bases: Reader

Base class for file based readers.

Parameters:: path (pathlib.Path) – Path to file to be opened. Str will be converted to path.

path: Path: Path to file to be opened. Str will be converted to path.

class stonesoup.reader.file.BinaryFileReader(path: Path)[source]

Bases: FileReader

Base class for binary file readers.

Parameters:: path (pathlib.Path) – Path to file to be opened. Str will be converted to path.

class stonesoup.reader.file.TextFileReader(path: Path, encoding: str = 'utf-8')[source]

Bases: FileReader

Base class for text file readers.

Parameters:

path (pathlib.Path) – Path to file to be opened. Str will be converted to path.
encoding (str, optional) – File encoding. Must be valid coding. Default ‘utf-8’.

encoding: str: File encoding. Must be valid coding. Default ‘utf-8’.

Generic

Generic readers for Stone Soup.

This is a collection of generic readers for Stone Soup, allowing quick reading of data that is in common formats.

class stonesoup.reader.generic.CSVGroundTruthReader(path: Path, state_vector_fields: Sequence[str], time_field: str, path_id_field: str, encoding: str = 'utf-8', time_field_format: str = None, timestamp: bool = False, metadata_fields: Collection[str] = None, csv_options: Mapping = {})[source]

Bases: GroundTruthReader, _CSVReader

A simple reader for csv files of truth data.

CSV file must have headers, as these are used to determine which fields to use to generate the ground truth state. Those states with the same ID will be put into a GroundTruthPath in sequence, and all paths that are updated at the same time are yielded together, and such assumes file is in time order.

Parameters:

path (pathlib.Path) – Path to file to be opened. Str will be converted to path.
state_vector_fields (Sequence[str]) – List of columns names to be used in state vector
time_field (str) – Name of column to be used as time field
path_id_field (str) – Name of column to be used as path ID
encoding (str, optional) – File encoding. Must be valid coding. Default ‘utf-8’.
time_field_format (str, optional) – Optional datetime format
timestamp (bool, optional) – Treat time field as a timestamp from epoch
metadata_fields (Collection[str], optional) – List of columns to be saved as metadata, default all
csv_options (Mapping, optional) – Keyword arguments for the underlying csv reader

path_id_field: str: Name of column to be used as path ID

groundtruth_paths_gen()[source]

Returns a generator of ground truth paths for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of GroundTruthPath – Ground truth paths existing in the time step

class stonesoup.reader.generic.CSVDetectionReader(path: Path, state_vector_fields: Sequence[str], time_field: str, encoding: str = 'utf-8', time_field_format: str = None, timestamp: bool = False, metadata_fields: Collection[str] = None, csv_options: Mapping = {})[source]

Bases: DetectionReader, _CSVReader

A simple detection reader for csv files of detections.

CSV file must have headers, as these are used to determine which fields to use to generate the detection. Detections at the same time are yielded together, and such assume file is in time order.

Parameters:

path (pathlib.Path) – Path to file to be opened. Str will be converted to path.
state_vector_fields (Sequence[str]) – List of columns names to be used in state vector
time_field (str) – Name of column to be used as time field
encoding (str, optional) – File encoding. Must be valid coding. Default ‘utf-8’.
time_field_format (str, optional) – Optional datetime format
timestamp (bool, optional) – Treat time field as a timestamp from epoch
metadata_fields (Collection[str], optional) – List of columns to be saved as metadata, default all
csv_options (Mapping, optional) – Keyword arguments for the underlying csv reader

detections_gen()[source]

Returns a generator of detections for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of Detection – Detections generate in the time step

YAML

class stonesoup.reader.yaml.YAMLReader(path: Path)[source]

Bases: FileReader, BufferedGenerator

YAML Reader

Parameters:: path (pathlib.Path) – File to read data from

path: Path: File to read data from

class stonesoup.reader.yaml.YAMLDetectionReader(path: Path)[source]

Bases: YAMLReader, DetectionReader

YAML Detection Reader

Parameters:: path (pathlib.Path) – File to read data from

detections_gen()[source]

Returns a generator of detections for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of Detection – Detections generate in the time step

class stonesoup.reader.yaml.YAMLGroundTruthReader(path: Path)[source]

Bases: YAMLReader, GroundTruthReader

YAML Ground Truth Reader

Parameters:: path (pathlib.Path) – File to read data from

groundtruth_paths_gen()[source]

Returns a generator of ground truth paths for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of GroundTruthPath – Ground truth paths existing in the time step

class stonesoup.reader.yaml.YAMLSensorDataReader(path: Path)[source]

Bases: YAMLReader, SensorDataReader

YAML Sensor Data Reader

Parameters:: path (pathlib.Path) – File to read data from

sensor_data_gen()[source]

Returns a generator of sensor data for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of SensorData – Sensor data generated in the time step

class stonesoup.reader.yaml.YAMLTrackReader(path: Path)[source]

Bases: YAMLReader, Tracker

YAML Track Reader

Parameters:: path (pathlib.Path) – File to read data from

HDF5

Generic HDF5 readers for Stone Soup.

This is a collection of generic readers for Stone Soup, allowing quick reading of data that is in HDF5 format, using the h5py library.

class stonesoup.reader.hdf5.HDF5GroundTruthReader(path: Path, state_vector_fields: Sequence[str], time_field: str, path_id_field: str, time_field_format: str = None, timestamp: bool = False, time_res_second: int = 1, time_res_micro: int = 1000000.0, metadata_fields: Collection[str] = None)[source]

Bases: GroundTruthReader, _HDF5Reader

A simple reader for HDF5 files of truth data.

HDF5 files are hierarchically structured data files with embedded metadata. This reader will extract values that are placed anywhere within the hierarchy, but it assumes all datasets are 1D arrays of base types representing ‘columns’ of data. All fields must be the same length, and a ‘row’ of data is constructed from the values at the same positional index in each column. Those states with the same ID will be put into a GroundTruthPath in sequence, and all paths that are updated at the same time are yielded together, and such assumes file is in time order.

Parameters:

path (pathlib.Path) – Path to file to be opened. Str will be converted to path.
state_vector_fields (Sequence[str]) – Paths of datasets to be used in state vector
time_field (str) – Path of dataset to be used as time field
path_id_field (str) – Path of dataset to be used as path ID
time_field_format (str, optional) – Optional datetime format
timestamp (bool, optional) – Treat time field as a timestamp from epoch
time_res_second (int, optional) – Desired maximum resolution of time values in seconds
time_res_micro (int, optional) – Desired maximum sub-second resolution of time values in microseconds
metadata_fields (Collection[str], optional) – Paths of datasets to be saved as metadata, default all

path_id_field: str: Path of dataset to be used as path ID

groundtruth_paths_gen()[source]

Returns a generator of ground truth paths for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of GroundTruthPath – Ground truth paths existing in the time step

class stonesoup.reader.hdf5.HDF5DetectionReader(path: Path, state_vector_fields: Sequence[str], time_field: str, time_field_format: str = None, timestamp: bool = False, time_res_second: int = 1, time_res_micro: int = 1000000.0, metadata_fields: Collection[str] = None)[source]

Bases: DetectionReader, _HDF5Reader

A simple detection reader for HDF5 files of detections.

HDF5 files are hierarchically structured data files with embedded metadata. This reader will extract values that are placed anywhere within the hierarchy, but it assumes all datasets are 1D arrays of base types representing ‘columns’ of data. All fields must be the same length, and a ‘row’ of data is constructed from the values at the same positional index in each column. Detections at the same time are yielded together, and such assume file is in time order.

Parameters:

path (pathlib.Path) – Path to file to be opened. Str will be converted to path.
state_vector_fields (Sequence[str]) – Paths of datasets to be used in state vector
time_field (str) – Path of dataset to be used as time field
time_field_format (str, optional) – Optional datetime format
timestamp (bool, optional) – Treat time field as a timestamp from epoch
time_res_second (int, optional) – Desired maximum resolution of time values in seconds
time_res_micro (int, optional) – Desired maximum sub-second resolution of time values in microseconds
metadata_fields (Collection[str], optional) – Paths of datasets to be saved as metadata, default all

detections_gen()[source]

Returns a generator of detections for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of Detection – Detections generate in the time step

AISHub

class stonesoup.reader.aishub.JSON_AISDetectionReader(path: Path, encoding: str = 'utf-8')[source]

Bases: DetectionReader, TextFileReader

A simple detection reader for JSON files of AIS (maritime transponder) detections.

This particular JSON AIS reader is written to read the JSON format used by files downloaded from ‘www.aishub.net’ (must be a contributor to their AIS database to download their files).

JSON format:

[{"ERROR": "false"},
[{"NAME": "MARTCILINO", "MMSI": 205466990, "LONGITUDE": 3078401,
"TIME": "1516233686", "LATITUDE": 31215032, ...},
{"NAME": "MARTCILINO", "MMSI": 205466990, "LONGITUDE": 3064592,
"TIME": "1516234275", "LATITUDE": 31227463, ...}]]

Notes

TIME is in Linux Epoch format
LONGITUDE and LATITUDE are (long/lat degrees)*(600,000)
MMSI is unique ship identifier
The AIS detection attributes for lattitude, longitude, and timestamp are saved as the attributes of a ‘Detection’; the other attributes are saved as the dictionary ‘metadata’ attribute of a ‘Detection’.

Parameters:

path (pathlib.Path) – Path to file to be opened. Str will be converted to path.
encoding (str, optional) – File encoding. Must be valid coding. Default ‘utf-8’.

detections_gen()[source]

Returns a generator of detections for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of Detection – Detections generate in the time step

Kafka

class stonesoup.reader.kafka.KafkaDetectionReader(topic: str, kafka_config: Dict[str, str], state_vector_fields: List[str], time_field: str, time_field_format: str = None, timestamp: bool = False, metadata_fields: Collection[str] = None, buffer_size: int = 0, timeout: bool = None)[source]

Bases: DetectionReader, _KafkaReader

A detection reader that reads detections from a Kafka broker

It is assumed that each message contains a single detection. The value of each message is a JSON object containing the detection data. The JSON object must contain a field for each element of the state vector and a timestamp. The JSON object may also contain fields for the detection metadata.

Parameters:

topic (str) – The Kafka topic on which to listen for messages.
kafka_config (Dict[str, str]) – Configuration properties for the underlying kafka consumer. See the confluent-kafka documentation for more details.
state_vector_fields (List[str]) – List of columns names to be used in state vector.
time_field (str) – Name of column to be used as time field.
time_field_format (str, optional) – Optional datetime format.
timestamp (bool, optional) – Treat time field as a timestamp from epoch.
metadata_fields (Collection[str], optional) – List of columns to be saved as metadata, default all.
buffer_size (int, optional) – Size of the frame buffer. The frame buffer is used to cache frames in cases where the stream generates messages faster than they are ingested by the reader. If buffer_size is less than or equal to zero, the buffer size is infinite.
timeout (bool, optional) – Timeout (in seconds) when reading from buffer. Defaults to None in which case the reader will block until new data becomes available.

detections_gen()[source]

Returns a generator of detections for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of Detection – Detections generate in the time step

class stonesoup.reader.kafka.KafkaGroundTruthReader(topic: str, kafka_config: Dict[str, str], state_vector_fields: List[str], time_field: str, path_id_field: str, time_field_format: str = None, timestamp: bool = False, metadata_fields: Collection[str] = None, buffer_size: int = 0, timeout: bool = None)[source]

Bases: GroundTruthReader, _KafkaReader

A ground truth reader that reads ground truths from a Kafka broker

It is assumed that each message contains a single ground truth state. The value of each message is a JSON object containing the ground truth data. The JSON object must contain a field for each element of the state vector, a timestamp, and the ground truth path ID. The JSON object may also contain fields for the ground truth metadata.

Parameters:

topic (str) – The Kafka topic on which to listen for messages.
kafka_config (Dict[str, str]) –
Configuration properties for the underlying kafka consumer. See the confluent-kafka documentation for more details.
state_vector_fields (List[str]) – List of columns names to be used in state vector.
time_field (str) – Name of column to be used as time field.
path_id_field (str) – Name of column to be used as path ID.
time_field_format (str, optional) – Optional datetime format.
timestamp (bool, optional) – Treat time field as a timestamp from epoch.
metadata_fields (Collection[str], optional) – List of columns to be saved as metadata, default all.
buffer_size (int, optional) – Size of the frame buffer. The frame buffer is used to cache frames in cases where the stream generates messages faster than they are ingested by the reader. If buffer_size is less than or equal to zero, the buffer size is infinite.
timeout (bool, optional) – Timeout (in seconds) when reading from buffer. Defaults to None in which case the reader will block until new data becomes available.

path_id_field: str: Name of column to be used as path ID.

groundtruth_paths_gen()[source]

Returns a generator of ground truth paths for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of GroundTruthPath – Ground truth paths existing in the time step

Image

class stonesoup.reader.image.SingleImageFileReader(path: Path, timestamp: datetime = None)[source]

Bases: FileReader, FrameReader

ImageFileReader A reader that reads a single image file from a given directory.

Parameters:

path (pathlib.Path) – Path to file to be opened. Str will be converted to path.
timestamp (datetime.datetime, optional) – Timestamp given to the returned frame

timestamp: datetime: Timestamp given to the returned frame

frames_gen()[source]

Returns a generator of frames for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of ImageFrame – Generated frame in the time step

Video

Video readers for Stone Soup.

This is a collection of video readers for Stone Soup, allowing quick reading of video data/streams.

class stonesoup.reader.video.VideoClipReader(path: Path, start_time: timedelta = datetime.timedelta(0), end_time: timedelta = None, timestamp: datetime = None)[source]

Bases: FileReader, FrameReader

A simple reader that uses MoviePy to read video frames from a file.

Usage of MoviePy allows for the application of clip transformations and effects, as per the MoviePy documentation. Upon instantiation, the underlying MoviePy VideoFileClip instance can be accessed through the clip class property. This can then be used as expected, e.g.:

# Rearrange RGB to BGR
def arrange_bgr(image):
    return image[:, :, [2, 1, 0]]

reader = VideoClipReader("path_to_file")
reader.clip = reader.clip.fl_image(arrange_bgr)

for timestamp, frame in reader:
    # The generated frame.pixels will now
    # be arranged in BGR format.
    ...

Parameters:

path (pathlib.Path) – Path to file to be opened. Str will be converted to path.
start_time (datetime.timedelta, optional) – Start time expressed as duration from the start of the clip
end_time (datetime.timedelta, optional) – End time expressed as duration from the start of the clip
timestamp (datetime.datetime, optional) – Timestamp given to the first frame

start_time: timedelta: Start time expressed as duration from the start of the clip

end_time: timedelta: End time expressed as duration from the start of the clip

timestamp: datetime: Timestamp given to the first frame

frames_gen()[source]

Returns a generator of frames for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of ImageFrame – Generated frame in the time step

class stonesoup.reader.video.FFmpegVideoStreamReader(url: ParseResult, buffer_size: int = 1, input_opts: Mapping[str, str] = None, output_opts: Mapping[str, str] = None, filters: Sequence[Tuple[str, Sequence[Any], Mapping[Any, Any]]] = None, frame_size: Tuple[int, int] = None)[source]

Bases: UrlReader, FrameReader

A threaded reader that uses ffmpeg-python to read frames from video streams (e.g. RTSP) in real-time.

Notes

Use of this class requires that FFmpeg is installed on the host machine.
By default, FFmpeg performs internal buffering of frames leading to a slight delay in the incoming frames (0.5-1 sec). To remove the delay it is recommended to set input_opts={'threads': 1, 'fflags': 'nobuffer'} when instantiating a reader, e.g: .

video_reader = FFmpegVideoStreamReader('rtsp://192.168.0.10:554/1/h264minor',
                                       input_opts={'threads': 1, 'fflags': 'nobuffer'})
for timestamp, frame in video_reader:
    ....

Parameters:

url (urllib.parse.ParseResult) – Input source to read video stream from, passed as input url argument. This can include any valid FFmpeg input e.g. rtsp URL, device name when using ‘dshow’/’v4l2’
buffer_size (int, optional) – Size of the frame buffer. The frame buffer is used to cache frames in cases where the stream generates frames faster than they are ingested by the reader. If buffer_size is less than or equal to zero, the buffer size is infinite.
input_opts (Mapping[str, str], optional) – FFmpeg input options, provided in the form of a dictionary, whose keys correspond to option names. (e.g. {'fflags': 'nobuffer'}). The default is {}.
output_opts (Mapping[str, str], optional) – FFmpeg output options, provided in the form of a dictionary, whose keys correspond to option names. The default is {'f': 'rawvideo', 'pix_fmt': 'rgb24'}.
filters (Sequence[Tuple[str, Sequence[Any], Mapping[Any, Any]]], optional) – FFmpeg filters, provided in the form of a list of filter name, sequence of arguments, mapping of key/value pairs (e.g. [('scale', ('320', '240'), {})]). Default None where no filter will be applied. Note that frame_size may need to be set in when video size changed by filter.
frame_size (Tuple[int, int], optional) – Tuple of frame width and height. Default None where it will be detected using ffprobe against the input, but this may yield wrong width/height (e.g. when filters are applied), and such this option can be used to override.

url: ParseResult: Input source to read video stream from, passed as input url argument. This can include any valid FFmpeg input e.g. rtsp URL, device name when using ‘dshow’/’v4l2’

buffer_size: int: Size of the frame buffer. The frame buffer is used to cache frames in cases where the stream generates frames faster than they are ingested by the reader. If buffer_size is less than or equal to zero, the buffer size is infinite.

frame_size: Tuple[int, int]: Tuple of frame width and height. Default None where it will be detected using ffprobe against the input, but this may yield wrong width/height (e.g. when filters are applied), and such this option can be used to override.

input_opts: Mapping[str, str]: FFmpeg input options, provided in the form of a dictionary, whose keys correspond to option names. (e.g. {'fflags': 'nobuffer'}). The default is {}.

output_opts: Mapping[str, str]: FFmpeg output options, provided in the form of a dictionary, whose keys correspond to option names. The default is {'f': 'rawvideo', 'pix_fmt': 'rgb24'}.

filters: Sequence[Tuple[str, Sequence[Any], Mapping[Any, Any]]]: FFmpeg filters, provided in the form of a list of filter name, sequence of arguments, mapping of key/value pairs (e.g. [('scale', ('320', '240'), {})]). Default None where no filter will be applied. Note that frame_size may need to be set in when video size changed by filter.

frames_gen()[source]

Returns a generator of frames for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of ImageFrame – Generated frame in the time step

OpenSky Network

class stonesoup.reader.opensky.OpenSkyNetworkDetectionReader(bbox: Tuple[float, float, float, float] = None, timestep: timedelta = datetime.timedelta(seconds=15))[source]

Bases: _OpenSkyNetworkReader, DetectionReader

OpenSky Network detection reader

This reader uses the OpenSky Network REST API to fetch air traffic control data.

The detection state vector consists of longitude, latitude (in decimal degrees) and altitude (in meters).

Note

By using this reader, you are agreeing to OpenSky Network’s terms of use.

Parameters:

bbox (Tuple[float, float, float, float], optional) – Bounding box to filter data to (left, bottom, right, top). Default None which will include global data.
timestep (datetime.timedelta, optional) – Time of each poll after reported time from OpenSky. Must be greater than 10 seconds. Default 15 seconds.

detections_gen()[source]

Returns a generator of detections for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of Detection – Detections generate in the time step

class stonesoup.reader.opensky.OpenSkyNetworkGroundTruthReader(bbox: Tuple[float, float, float, float] = None, timestep: timedelta = datetime.timedelta(seconds=15))[source]

Bases: _OpenSkyNetworkReader, GroundTruthReader

OpenSky Network groundtruth reader

This reader uses the OpenSky Network REST API to fetch air traffic control data.

The groundtruth state vector consists of longitude, latitude (in decimal degrees) and altitude (in meters).

Paths that are yielded are grouped based on the International Civil Aviation Organisation (ICAO) 24-bit address.

Note

By using this reader, you are agreeing to OpenSky Network’s terms of use.

Parameters:

bbox (Tuple[float, float, float, float], optional) – Bounding box to filter data to (left, bottom, right, top). Default None which will include global data.
timestep (datetime.timedelta, optional) – Time of each poll after reported time from OpenSky. Must be greater than 10 seconds. Default 15 seconds.

groundtruth_paths_gen()[source]

Returns a generator of ground truth paths for each time step.

Yields:

datetime.datetime – Datetime of current time step
set of GroundTruthPath – Ground truth paths existing in the time step

Readers

Generic

YAML

HDF5

AISHub

Kafka

Image

Video

OpenSky Network

Astronomical