Module: formatting

Data formatting nodes.

These nodes typically do not change the actual numeric values contained in the received data streams, but instead change the format of the data (for instance, the shape of the blocks, the properties of the chunks, possibly the number of streams in a packet). The most important use cases are to extract segments from continuous data, and to "play back" a long packet by chopping it into a series of successive streaming packets.

ApplyAxisMask

Apply an axis mask that was previously generated using Generate Axis Mask.

This can be used to filter out the same collection of entries from multiple unrelated data packets (e.g., trials from different modalities).

Version 1.0.0

Ports/Properties

  • set_breakpoint
    Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

    • verbose name: Set Breakpoint (Debug Only)
    • default value: False
    • port type: BoolPort
    • value type: bool (can be None)
  • data
    Data to process.

    • verbose name: Data
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: INOUT
  • mask
    Mask to apply

    • verbose name: Mask
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: IN

BatchInstances

Group and batch instances based on some some criterion along a new feature axis.

The data will be concatenated and padded along that axis. This can be used in cases where you want to group trials (represented as instances in Neuropype) by certain criteria in the instance.data field (typically Marker, but could be any other attribute). You specify the instance.data field holding the criteria (such as 'Marker'), and a feature axis will be created for each unique value found in that field, and the instances (trials) will be stacked along the corresponding axis based on their value (for example, all trials with marker "A" would go in one axis, all trials with marker "B" in another), etc. This allows you to segregate trials into batches for comparative analysis. (Note that if all you want is the mean of trials matching each criteria, instead of the individual trials themselves, you can used the GroupedMean node.)

Version 0.5.1

Ports/Properties

  • set_breakpoint
    Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

    • verbose name: Set Breakpoint (Debug Only)
    • default value: False
    • port type: BoolPort
    • value type: bool (can be None)
  • data
    Data to process.

    • verbose name: Data
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: INOUT
  • group_by
    Group instances that have the same value or combination of values for the given instance field(s). If given, then the grouped instances will be batched along a new axis.

    • verbose name: Group By
    • default value: []
    • port type: ListPort
    • value type: list (can be None)
  • verify_unique
    Ensure that the given fields (if any), are unique within each group. If this is not the case, an error will be thrown.

    • verbose name: Verify Unique
    • default value: []
    • port type: ListPort
    • value type: list (can be None)
  • pad_maxitems
    Optional max # of items for padding. If this is None, it amounts to the max items in any of the groups. If < 1, then it is taken as a quantile of the item counts (e.g., 0.9 would be 90th percentile). If given as an integer >= 1, it is taken as the max # of items.

    • verbose name: Pad Maxitems
    • default value: None
    • port type: FloatPort
    • value type: float (can be None)
  • pad_value
    Value to use for padding. This value will be inserted to fill up groups that have less than the max # of items.

    • verbose name: Pad Value
    • default value: 0.0
    • port type: FloatPort
    • value type: float (can be None)
  • handle_overlength
    How to handle groups that have more items than pad_maxitems. In the truncate-group modes, the excess items are removed from the group, either at the beginning (-pre variant), or at the end (-post variant). In the drop-group mode, the entire group is dropped from the data set.

    • verbose name: Handle Overlength
    • default value: truncate-group-pre
    • port type: EnumPort
    • value type: object (can be None)

ExtractStreams

Extract streams whose names are in the given list of names.

This will create a new packet that has only the matched streams (chunks).

Version 1.0.0

Ports/Properties

  • set_breakpoint
    Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

    • verbose name: Set Breakpoint (Debug Only)
    • default value: False
    • port type: BoolPort
    • value type: bool (can be None)
  • data
    Data to process.

    • verbose name: Data
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: INOUT
  • stream_names
    Names of the streams to extract.

    • verbose name: Stream Names
    • default value: []
    • port type: ListPort
    • value type: list (can be None)
  • support_wildcards
    Support for wildcard matching.

    • verbose name: Support Wildcards
    • default value: False
    • port type: BoolPort
    • value type: bool (can be None)

GetAxisMask

Identifies which slices meet criteria parameters and outputs the result as a mask packet.

Only the first chunk with the axis_type is processed; See ExtractStreams. The mask packet can then be used by ApplyAxisMask to mask out slices along the specified dimension, in a packet with one or more streams with a matching axis. These nodes can be used to slice out trials with bad data.

Version 1.0.0

Ports/Properties

  • set_breakpoint
    Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

    • verbose name: Set Breakpoint (Debug Only)
    • default value: False
    • port type: BoolPort
    • value type: bool (can be None)
  • data
    Data to process.

    • verbose name: Data
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: IN
  • mask
    Output mask.

    • verbose name: Mask
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: OUT
  • axis_type

    • verbose name: Axis Type
    • default value: instance
    • port type: EnumPort
    • value type: object (can be None)
  • comparison_data
    Optionally compare input data to this packet. Any len()>1 axes must match input data exactly.

    • verbose name: Comparison Data
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: IN
  • comparison_value
    Value to compare data to if comparison_data not provided. Default -np.inf and greater_equal will eliminate nan trials.

    • verbose name: Comparison Value
    • default value: None
    • port type: FloatPort
    • value type: float (can be None)
  • comparison_operator

    • verbose name: Comparison Operator
    • default value: greater_equal
    • port type: EnumPort
    • value type: object (can be None)
  • dimension_operator
    When input chunk is multidimensional, this operation will be applied to the result of the comparison across all other axes.

    • verbose name: Dimension Operator
    • default value: all
    • port type: EnumPort
    • value type: object (can be None)
  • invert_selection
    If set then the resulting mask will be inverted.

    • verbose name: Invert Selection
    • default value: False
    • port type: BoolPort
    • value type: bool (can be None)

MergeStreams

Merge the streams (chunks) in the given packets into one packet that contains all streams.

A data packet in NeuroPype can contain one or more named streams, and for each stream it holds one chunk. This node will merge the streams carried by all inputs into one packet. If a stream is encountered whose name is already in the packet, then a fresh name will be generated (as in eeg-1, eeg-2, etc.).

Version 1.0.0

Ports/Properties

  • set_breakpoint
    Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

    • verbose name: Set Breakpoint (Debug Only)
    • default value: False
    • port type: BoolPort
    • value type: bool (can be None)
  • data1
    Input data 1.

    • verbose name: Data1
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: IN
  • data2
    Input data 2.

    • verbose name: Data2
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: IN
  • data3
    Input data 3.

    • verbose name: Data3
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: IN
  • data4
    Input data 4.

    • verbose name: Data4
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: IN
  • data5
    Input data 5.

    • verbose name: Data5
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: IN
  • data6
    Input data 6.

    • verbose name: Data6
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: IN
  • data7
    Input data 7.

    • verbose name: Data7
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: IN
  • data8
    Input data 8.

    • verbose name: Data8
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: IN
  • data9
    Input data 9.

    • verbose name: Data9
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: IN
  • data10
    Input data 10.

    • verbose name: Data10
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: IN
  • replace_if_exists
    Replace streams if they already exist.

    • verbose name: Replace If Exists
    • default value: False
    • port type: BoolPort
    • value type: bool (can be None)
  • outdata
    Data to process.

    • verbose name: Outdata
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: OUT
  • sorting
    Sorting order of chunks in the packet.

    • verbose name: Sorting
    • default value: input
    • port type: EnumPort
    • value type: object (can be None)

RenameStreams

Rename streams in the given data.

This node can flexibly rename streams. Note that such renaming is rarely necessary since almost all nodes identify the streams to act on by their properties rather than name. However, in some settings, such as when multiple streams are merged using the MergeStreams node, or when data is handed off to external applications, renaming streams can be helpful.

Version 1.0.0

Ports/Properties

  • set_breakpoint
    Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

    • verbose name: Set Breakpoint (Debug Only)
    • default value: False
    • port type: BoolPort
    • value type: bool (can be None)
  • data
    Data to process.

    • verbose name: Data
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: INOUT
  • name_changes
    Name changes to apply. This is formatted as in {'myoldname1': 'mynewname1', 'myoldname2': 'mynewname2', ...}. Streams that are not mentioned in this mapping keep their name.

    • verbose name: Name Changes
    • default value: {}
    • port type: DictPort
    • value type: dict (can be None)

Segmentation

Cut fixed-length segments out of a continuous time series.

This node requires input data that has both a multi-channel time-series stream (channels by samples), and a marker stream in it. The node will then cut out a segment around each marker of interest, and return a packet that is a 3d array of extracted same-length segments. Important: Note that the markers of interest first need to be flagged as "target markers", using the Assign Target Markers node (usually segments are only extracted around a subset of marker types). A typical setup consists of the Assign Target Markers node followed by the Segmentation node. Note that the data that comes out of this node is no longer a continuous time series, but instead is now segmented. The resulting array has a new axis that indexes the segments (this is a so-called "instance" axis). This processing step is also called epoching in some other frameworks (e.g., EEGLAB). This node will work both on packets that contain an entire recording (it will then extract one segment for each target marker in the recording), as well as as on packets that are streaming. In the former case, markers that would produce segments that extend beyond data limits are ignored; see RemoveIntersectingMarkers node to remove these markers prior to Segmentation. In the latter case the node consumes chunks and emits segments as soon as all the data that is overlapped by the segment has been received ( therefore, for a given nonempty input chunk, this node may output zero, one, or more segments). There is also an optional mode that extracts a single sliding segment on each update that lies at the end of the data received by the node (see online segmentation mode parameter). The new segments receive new time stamps that are relative to the marker around which they were originally extracted (i.e., the part of the segment that lies before the marker, if any, will have negative time stamps). The absolute time of the marker is stored as well, though (so no information is lost). The instance axis data contains a rec array corresponding to a table of independent variables. Default columns include Marker, TargetValue, TrialIndex, and possibly Subject and Session if they can be determined automatically.

Version 1.0.1

Ports/Properties

  • set_breakpoint
    Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

    • verbose name: Set Breakpoint (Debug Only)
    • default value: False
    • port type: BoolPort
    • value type: bool (can be None)
  • data
    Data to segment.

    • verbose name: Data
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: INOUT
  • time_bounds
    Time limits for extracted segments, relative to the position of the marker, in seconds. For each target marker, a segment will be extracted around it, and the beginning and end of the segment is given by these two values (e.g., [-1, 2] yields segments that begin one second before the marker and end two seconds after the marker). Segments may intersect the marker, or lie before it, or lie after it.

    • verbose name: Segment Time Limits
    • default value: [-1, 2]
    • port type: ListPort
    • value type: list (can be None)
  • select_markers
    (Optional) Data will be segmented around these markers only. This is a list of marker strings. If this is not provided, but target values have been assigned to markers in a previous step (typically using the AssignTargets node), then segmentation will extract segments only around those markers that have target values set. Otherwise this node will fall back to all markers in the data, except for markers that are empty strings (e.g., scrubbed using some preceding marker rewriting node).

    • verbose name: Select Markers
    • default value: None
    • port type: ListPort
    • value type: list (can be None)
  • online_epoching
    Online segment extraction mode. If this is set to marker-locked, then segments will be extracted the same way on streaming data ("online") as on non-streaming data such as imported recordings ("offline"). If this is set to sliding, then for online data, on each tick a single segment will be extracted that lies at the end of the data received by this node. This mode can be handy when the remainder of the pipeline is calibrated on data relative to markers, but during online processing the pipeline should produce outputs continuously, regardless of whether there is any marker or not.

    • verbose name: Online Segmentation Mode
    • default value: marker-locked
    • port type: EnumPort
    • value type: object (can be None)
  • sample_offset
    Optional additional shift of the segment position, in samples. This setting allows to shift all segments by this many samples relative to the marker, which can be used to match the behavior of other systems (e.g., BCILAB or EEGLAB) in a sample-exact fashion.

    • verbose name: Additional Sample Offset
    • default value: 0
    • port type: IntPort
    • value type: int (can be None)
  • max_gap_length
    Maximum length of time-stamp gaps that are allowed in segments. If a gap between successive samples is larger than this duration in seconds, then the affected segment is not extracted. Optional. Note that if your time stamps are jittered, this will generate a lot of false positives unless you use a very generous cutoff. You can use the DejitterTimestamps node early during your processing to fix that.

    • verbose name: Max Gap Length
    • default value: 0.2
    • port type: FloatPort
    • value type: float (can be None)
  • keep_marker_chunk
    Whether or not to keep the selected markers in a chunk called 'segmented-markers'.

    • verbose name: Keep Marker Chunk
    • default value: False
    • port type: BoolPort
    • value type: bool (can be None)
  • verbose
    Verbose output. Can be used for debugging your pipeline.

    • verbose name: Verbose
    • default value: False
    • port type: BoolPort
    • value type: bool (can be None)

SelectInstances

Can select instances (i.e

., trials) based on metadata contained in the instance.data field. This is used to select trials matching certain criteria, such as subject, session, group, task, conditions, etc. See SetInstanceDetails to apply metadata extracted from the file path (using PathIterator) to each trial (which can then be used by this node).

Version 1.1.1

Ports/Properties

  • set_breakpoint
    Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

    • verbose name: Set Breakpoint (Debug Only)
    • default value: False
    • port type: BoolPort
    • value type: bool (can be None)
  • data
    Data to process.

    • verbose name: Data
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: INOUT
  • selection
    Described the instance selection criteria. Several formats are permitted: simple string - e.g. 'Go' - This will be compared against the 'Marker' field. dictionary without 'name' field - e.g. {'Marker': 'Go', 'Result': ['Success', 'Unknown']} - The dict keys indicate the name of the instance data fields, and the dict values are used to compare to the values. dictionary with 'name' field - e.g. {'name': 'Marker', 'value': 'Go'} 'name': Name of the column in the instance axis data table (IV table). 'operator': (optional) The comparison operator. If not provided then the value in 'default_comparison_operator' will be used. Also see default_comparison_operator for the list of possible operators. 'value': (optional) The value that each IV cell in the column will be compared against. If not provided then None is assumed. A list of the aforementioned dictionaries.

    • verbose name: Selection
    • default value: None
    • port type: Port
    • value type: object (can be None)
  • default_comparison_operator
    When selection argument does not provide an operator, this operator is used. 'auto' changes depending on provided 'value'. If 'value' is None or nan, then 'auto' will behave like 'not_equal'. Otherwise ('value' is valid), 'auto' will behave like 'equal'. If 'value' is a string (or list of strings), then some comparison operators will be remapped: greater or greater_equal -> startswith; less or less_equal -> endswith; equal -> fnmatch(test, pattern) not_equal -> not isin

    • verbose name: Default Comparison Operator
    • default value: auto
    • port type: EnumPort
    • value type: object (can be None)
  • combine_selections
    Logical operator to use to combine evaluations specified in selection argument.

    • verbose name: Combine Selections
    • default value: and
    • port type: EnumPort
    • value type: object (can be None)
  • invert_selection
    If set then the resulting of the selection comparison check will be inverted before logical combination with previous IsGood values.

    • verbose name: Invert Selection
    • default value: False
    • port type: BoolPort
    • value type: bool (can be None)
  • combine_previous
    Logical operator to use to combine overall selection result with previous IsGood values. Use 'override' to ignore previous values.

    • verbose name: Combine Previous
    • default value: override
    • port type: EnumPort
    • value type: object (can be None)
  • drop_immediately
    Can drop !IsGood rows immediately. Otherwise merge this output with other packets then use DropBadTrials node.

    • verbose name: Drop Immediately
    • default value: True
    • port type: BoolPort
    • value type: bool (can be None)
  • assign_target_value
    For instances matching the selection criteria, set their TargetValue field to this value. Note that when this value is not None, only the TargetValue is updated and the IsGood field is not updated. A consequence is that calling SelectInstances again on the same packet using combine_previous will give unexpected behaviour.

    • verbose name: Assign Target Value
    • default value: None
    • port type: FloatPort
    • value type: float (can be None)

SeparateSignalChunks

Separate the signal chunks from the non-signal chunks.

An incoming data packet may carry multiple data streams, and for each stream a packet will have one chunk. Some of these chunks may be specifically flagged as 'signal' chunks (multi-channel time series) while other may not be (e.g., feature vectors, marker streams, predictions). Some of the filter nodes will only work on signal chunks and will not apply cleanly on non-signal chunks, so if a packet that has mixed content shall be filtered with such a node, the signal chunks must be separated out first. This node has two outputs: one that carries the signal streams, and another one that carries all other streams. After processing, the chunks may be merged again.

Version 1.0.0

Ports/Properties

  • set_breakpoint
    Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

    • verbose name: Set Breakpoint (Debug Only)
    • default value: False
    • port type: BoolPort
    • value type: bool (can be None)
  • data
    Data to process.

    • verbose name: Data
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: IN
  • signals
    Data with only signal chunks.

    • verbose name: Signals
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: OUT
  • nonsignals
    Data with only non-signal chunks.

    • verbose name: Nonsignals
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: OUT

StreamData

Play back a data set in real time or at a desired rate.

This node requires as input a packet that holds the entire recording to play back, for example the output of one of the data import nodes. A common setup is an import node followed by a Stream Data node, followed by some processing, visualization, or network streaming. You can play back either based on wall-clock time, optionally modified by a speedup factor, or you can use an abstract clock that advances by a deterministic amount from one tick to the next. The latter is more useful when you are debugging your code, e.g., when you need to be able to exactly recreate a situation in which an error occurred previously. This node is also able to simulate a variety of issues that occur in real-world data sources, such as jitter in the emitted chunk sizes and empty packets -- these special features are mainly useful for testing and debugging your own nodes.

Version 1.1.0

Ports/Properties

  • set_breakpoint
    Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

    • verbose name: Set Breakpoint (Debug Only)
    • default value: False
    • port type: BoolPort
    • value type: bool (can be None)
  • data
    Data to stream.

    • verbose name: Data
    • default value: None
    • port type: DataPort
    • value type: Packet (can be None)
    • data direction: INOUT
  • timing
    Timing mode. The wallclock mode means that playback is based on actual elapsed time, and the deterministic mode means that an abstract clock is used that advances by a mathematically determined amount on each tick. The first mode is what is used to simulate a real device, and the second mode can be used for debugging your code (because on each run, exactly the same sequence of data is emitted by the node, which allows you to replicate a problem exactly and drill into it).

    • verbose name: Timing Mode
    • default value: wallclock
    • port type: EnumPort
    • value type: object (can be None)
  • looping
    Looping playback. Whether to stream the data in a loop.

    • verbose name: Looping
    • default value: True
    • port type: BoolPort
    • value type: bool (can be None)
  • speedup
    Speedup factor for wallclock mode. This is the speedup, relative to the actually elapsed time, at which the data is played back (e.g., 2 means that the data is played back at 2x real time). This setting is ignored in deterministic mode.

    • verbose name: Speedup Factor (Wallclock Mode)
    • default value: 1.0
    • port type: FloatPort
    • value type: float (can be None)
  • update_interval
    Average chunk length per tick for deterministic mode, in seconds. The length of each individual chunk is drawn from a Gaussian distribution with this value as the mean and a standard deviation based on the jitter percentage. This setting is ignored in wallclock mode.

    • verbose name: Chunk Length Mean (Deterministic Mode)
    • default value: 0.04
    • port type: FloatPort
    • value type: float (can be None)
  • jitter_percent
    Pseudo-random jitter applied to the chunk length in deterministic mode, in percent. On each tick a chunk is emitted whose length is drawn pseudo-randomly from a Gaussian distribution with a mean value and a standard deviation that is a percentage of that mean. For instance, if the mean chunk length is set to 0.15 (seconds), and this value is set to 20 (percent), then the chunks are 0.15 +/- 0.03 seconds long.

    • verbose name: Chunk Length Std (Deterministic Mode)
    • default value: 5
    • port type: FloatPort
    • value type: float (can be None)
  • randseed
    Random seed for deterministic mode. This value allows to change the series of chunk lengths that is generated by the random number generator.

    • verbose name: Random Seed
    • default value: 34535
    • port type: IntPort
    • value type: int (can be None)
  • hitch_probability
    Probability of simulated hitches, in Percent. The node generates an empty packet with this probability. This can be used to test whether subsequent nodes are robust to empty packets.

    • verbose name: Hitch Probability
    • default value: 0.0
    • port type: FloatPort
    • value type: float (can be None)
  • timestamp_jitter
    Pseudo-random jitter applied to the time stamps. This is the standard deviation of a normal distribution, in seconds.

    • verbose name: Timestamp Jitter
    • default value: 0.0
    • port type: FloatPort
    • value type: float (can be None)
  • start_pos
    Starting position for playback. This is the time offset, in seconds from the beginning of the data, where playback should start.

    • verbose name: Start Pos
    • default value: 0.0
    • port type: FloatPort
    • value type: float (can be None)