Getting Started with NeuroPype and Pipeline Designer
You can download the Neuropype Suite from your account page on neuropype.io. Simply run the downloaded .exe to install the Neuropype Suite (on Windows 10 a dialog box may appear requesting you give permission to the installer).
The installer will install the Neuropype and Pipeline Designer applications, along with some other utilities, resources and examples. It will also create desktop icons for Neuropype and Pipeline Designer by default.
Activating your License
When installation is completed, the Neuropype server application will start automatically. If it's your first time running Neuropype on that computer, you will be prompted to activate your license. A dialog box will pop up asking for your Intheon/Neuropype username and password. These will be the email address and password that you used to create an account on neuropype.io and purchase a Neuropype license (or obtain a free license in the case of the Academic edition).
Changing your license
Upon installation, Neuropype saves your license information in a file called
license.ini that is stored in the
.neuropype\Neuropype <Edition> Suite folder of your user folder. So for example, if you have the Academic Edition and (on Windows) your username is Harry, your
license.ini file is stored at
c:\Users\Harry\.neuropype\Neuropype Academic Suite. If you need to re-input your license information for some reason, such as if you want to assign the Neuropype installation on a computer to a different user, simply delete the
license.ini file. Restart Neuropype and you will be prompted for a license username and password. Enter the new username and password and, providing it is valid, your Neuropype installation will be assigned to the new user (and a new
license.ini file created). Important: do not attempt to edit the
license.ini file as this will cause the license check to fail. If you do edit it by mistake, then delete the file, restart Neuropype, and you will be prompted for your license info again.
The NeuroPype server
Once you have installed the suite and activated your license, the NeuroPype server will be running in the background, which you can see from the small NP icon in your Windows taskbar (lower right-hand corner of the screen; depending on your Windows setup, you may need to click on the taskbar to show the NP icon). Neuropype consumes relatively few resources and does not need to be turned off in most cases, but you can always right-click the icon and choose Exit to terminate it if desired. You can then bring it back up later from the Neuropype desktop icon. Note that you will not be able to run any data processing when the server is not active.
The Pipeline Designer
The NeuroPype Suite includes an open-source Pipeline Designer application, which you can use to design, edit, and execute NeuroPype pipelines using a visual 'drag-and-drop' interface. You can launch the Pipeline Designer from the start menu under Windows or on other platforms as per the installation instructions for your platform. This will bring up the welcome screen (pictured below) where you can create or open a new processing pipeline.
(Note that some of the widgets visible on the screenshots in this documentation may not be available in all editions of the NeuroPype Suite.)
Running your first pipeline
In this user guide, we'll take you through opening, editing and configuring a pipeline in Neuropype and explain the various aspects of working in Neuropype along the way.
Opening A Pipeline
To open an example pipeline, click Open and navigate to the Examples/ folder inside your installation folder (e.g.,
C:\Program Files\Intheon\NeuroPype Startup Suite (64-bit)\Examples). For example, opening the file
SimpleBandpower.pyp will bring up a screen similar to the following:
The large white area in the above screenshot is the 'canvas' that shows your current pipeline, which you can edit using drag-and-drop and double-click actions. On the left you have the widget panel, which has all the available widgets or
nodes that you can drag onto the canvas. You can hover the mouse over any category or widget and see a tooltip that briefly summarizes it. The small box below is the help panel, which shows additional documentation for the current selected widget -- this will be your primary source of documentation for each node, and it usually contains far more information about a node than the small window may suggest. Below the help panel is the toolbar with the main actions buttons, most importantly pause/unpause. The empty white rectangle at the bottom right is the output window. This may not yet be open at first run, but by default it will pop up when the first output is displayed. You can either dock to the bottom of the canvas as shown in the above screen shot, or keep it as a free-floating window that you can, for instance, place on another screen.
To launch the current pipeline, click the orange pause icon in the toolbar (initially engaged) to unpause it. As a result, the Pipeline Designer will ask the NeuroPype server to start executing your pipeline. This will print some output, and after a few seconds, it will show a time series view that looks like the following -- your first signal processing pipeline is running (congrats!).
Note that, if there are multiple EEG streams on the network currently via LSL, this pipeline may pick up the first one that it finds, which may not be what you expect (if so, you can double-click the LSL Input node and change the first entry to
name='My Datastream' as shown below):
You can query information about the current pipeline by clicking the info toolbar button (with the
For more complex pipelines this info will contain generally useful information, so don't skip it!
Changing pipeline settings
Editing Node Settings
You can change the settings of any of the processing nodes in your canvas by double-clicking on it. For instance, when you double-click on the Import XDF node, a window pops up that shows you the settings (including the file name to import).
At times, you may see options being mentioned or shown in screenshots that are not displayed in your node's properties. Some settings of a node require a deeper understanding of the behavior of a node and careful review of its documentation, and are therefore only visible in the "expert mode" of the Pipeline Designer. This can be easily switched by going to the menu item named Options / Settings..., and then checking the "Show expert parameters" checkbox as shown below.
Note that you need to restart the designer for the change to take effect.
Some of the less important settings in the dialogs shown in this guide (usually the ones near the bottom) may not be available in your version (e.g., community), but that's not a problem. You can pause the mouse pointer for a few seconds over individual settings to see a tooltip with detailed information about each one (every setting has a tooltip). In the image below, the reason the file is found, even though the path is not omitted, is that it's in the
Examples/ folder that comes included with the suite.
When you edit a setting of a node (e.g., the filename), it will be instantly applied, and the processing behavior of the pipeline will change (if it's currently unpaused). If you need to edit multiple settings at once, it's a good idea to pause the pipeline before making the changes to avoid crashes because of inconsistent settings. The
Import XDF here is importing a file containing a few minutes of data from a 6-channel InteraXon Muse EEG headset (during the recording, a person was alternatively focusing and relaxing to make it more interesting).
How a pipeline works
Data Flow Basics
When you execute a pipeline in NeuroPype Enterprise, the data flow behaves as follows. It is really quite simple: generally, data exits a node on the right side along the grey wire and continues to the next node, gets processed by it, and then exits that node along its output wire on the right, and so on. Data generally flows from left to right through a node, as shown below.
In this pipeline, the import node has no input wire since it's a data source. It loads the given file, and then outputs it via its output wire. That file then travels into the Stream Data node where it's stored in memory and subsequently streamed out in real time to the next node. The LSL Output node takes that real-time data stream and broadcasts it out over the LSL network (LSL=Lab Streaming Layer, see this video), where it can be picked up by other programs on the computer/LAN/Wi-Fi (e.g., a game), or by the same program.
The interesting thing to note here is that the output of the Import node is the entire recording, i.e., a very long chunk of data, which is also called offline (or batch) processing, while the output of the stream data node and input to the LSL node are small successive chops of the data comprising a real-time stream, often only 100ms or less in length. This is also called online processing. As showcased by this simple pipeline, the data flow engine of NeuroPype can easily do both in the same pipeline.
NeuroPype does online processing as follows. When NeuroPype is running your pipeline, it will periodically perform a
tick (by default 25 times a second) unless you pause it. In each tick, it will go through your pipeline in data flow order (in this example pipeline, it's left to right, top to bottom), update each node in turn, and move whatever data is produced by it through its output wires into the inputs of the next node (then update that node, move its output data to the subsequent node, and so on). This order is called topological order, and it doesn't depend on where nodes are on the screen (just on how they are connected).
Data and Data Sources
The data that travels over the wires may be an entire recording (e.g., what is produced by
Import XDF and accepted by
Stream Data), or it may be just a small chunk of data of some ongoing 'live' data stream (e.g., what is produced by
Stream Data and consumed by
LSL Output). Generally, the
Import nodes will output the content of the entire data file in one large packet, and will output nothing on subsequent ticks. Input nodes in the
Network category (see list of categories in widget panel), such as
LSL Input, will return whatever was received since the last tick (which would usually be a small chunk). If you are doing offline processing, in general you can reset/reload the pipeline by clicking the rightmost button in the toolbar (the circular arrow in your GUI) to re-run any offline processing. Note that some screenshots of this guide show a question mark icon instead (this is from a previous version).
Most of the nodes offered by NeuroPype are processing nodes, such as those shown below (from the upper chain in the
SimpleBandpower pipeline). These nodes accept a packet, and output a processed packet, and if they receive a whole recording, they will output a whole recording, and if they receive a small chunk, they will output a small chunk.
Some exceptions to this behavior are the
Stream Data node that we discussed before, whose jobs are to take a recording and then chop it up into many small chunks which are output over multiple successive ticks (i.e., real-time playback). Other exceptions are special-purpose nodes that may, for instance, buffer a certain amount of streaming data until a condition is met, and then output it in one large chunk (e.g., nodes that need to see a minimum amount of data to calibrate themselves on it). Ideally you should go through the help text of each node (in the help pane) to see how it behaves. Many of the nodes serve very special purposes, and may require data to be in a specific form to work as intended. You will generally find a lot of information on this in the node's help text in the help panel.
Some nodes act as 'data sinks', i.e., they take in data but have no outgoing wires. Their functions include file export, transmission over the network, and visualization, e.g., like the two nodes at the end of the upper chain in our pipeline (one sending the data over the LSL network, and the other plotting the incoming data as a streaming time series).
Adding nodes to your pipeline
Adding New Nodes
Let's peek into the data stream between the
FIR Filter node and the
Squaring node. For this, you can scroll the widget panel down to the
Visualization category and click on it to expand its contents as below:
Now you can grab the
Time Series Plot node with the left mouse button and drag-and-drop it under the
Now you can left-click on the grey arc on the right of the
FIR Filter node, hold the button, and drag a wire to the dashed arc on the left of the
Time Series Plot (1) node, and release.
As a result, a second time-series plot will open itself (if the pipeline is unpaused) and show 4 channels. Note that, on some versions of Windows, new plots may open behind the Pipeline Designer window, but they will be highlighted in the taskbar for easy discovery. You can press the '-' key on your keyboard in order to change the scale of the signal.
Tip: you can also add nodes by dragging a wire out from one node and then releasing it somewhere over the blank canvas. This will bring up a context menu with a categorized list of nodes that you can insert; you can even start typing, and it will prune the list to show the nodes that contain the substring you typed.
Data in Neuropype
In simple pipelines as this one, the data packets that travel along your wires on each tick are all multi-channel time series (i.e., each is a 2D array of channels by time-points). However, in more sophisticated pipelines, you can have data that are 3D arrays, for instances multi-channel time-frequency diagrams, or arrays of even more dimensions, such as the data that is flowing in pipelines that deal with brain connectivity measures. General n-way arrays in NeuroPype sometimes also called
Tensors in NeuroPype are not just a grid of raw numbers, but in fact each of the axes has a type (e.g.,
frequency), and stores information about the scale (such as the time range in seconds for a time axis). Some axis types also have labels (for instance, channel labels, feature labels, or trial labels). It is for this reason that the above plot is able to plot both the channel labels as well as the time correctly, or that a frequency spectrum plot would show the correct frequency axis.
Lastly, the entire chunk of data can have other meta-data attached to it, such as the provenance (what device it came from), the data modality (for instance, EEG, shown on the top), and so on. Some of the nodes in NeuroPype can perform some flexible processing on these n-way tensors, so be sure to take your time to find out what features they offer.
Sometimes you have multiple pieces of data that you would like to transmit together side by side. One example of this is EEG and EOG coming from two different amplifiers. Another example is having EEG plus event markers, and some more examples are having simultaneous data from N subjects in a group experiment, or features plus labels in machine learning. Sometimes these different pieces of data are best treated independently, e.g., when you want to apply different kinds of filters to each, and in that case you can have parallel data paths in your pipeline, each with its own processing nodes. However, in NeuroPype multiple streams can also be bundled together and transmitted together over a single wire. Many nodes can naturally produce multi-stream packets, for instance importing or exporting an EEG file that has both EEG and markers, and there are special nodes to bundle and your data streams as you wish.
NeuroPype has a lot of signal processing nodes (the Pipeline Designer shows these under the
Signal Processing category), for instance the
FIR Filter node, which is one of the most useful of them and whose settings are shown below.
This node as used here will perform a band-pass filter, and retain only the 8-14 Hz frequencies. However, often the rolloff at the band edges is not very sharp (since that would be difficult to realize with a low-order filter), and only frequencies outside 7 and 15 Hz are fully attenuated (the 1 Hz transition bands are partially attenuated). When you change these frequencies, it's good practice to not make the band edges unnecessarily steep. In addition, you can try to reduce the order of the filter (especially if you band edges are loose and if your cutoff frequencies are high), but note that the filter can end up leaking frequencies outside those bands, which can, for instance, cause slow drifts that can break all sorts of subsequent signal processing. A very high order will drive up your CPU usage and can slow down processing. You can also try the IIR Filter as an alternative (though make sure you read up on the principles behind it). You can play with the frequency parameters of this filter and watch how the signal changes that the time-series viewer that you added is plotting.
Another very important node is the
Select Range node, which allows you to select a subset of the data that passes through the node -- make sure you read the help text and tooltips to understand how it operates. For instance, you can select data along any axis, here
space (channels), and the selection can be a range, such as 1...5, or a list of indices (e.g., [0,1,2,3]). There is also a Python-style syntax that uses colons, where 2:7 goes from 2 to 6, but, unlike MATLAB, this syntax does not include the last element in the range. The setting in this pipeline removes the last 2 channels, since they happen to contain digital garbage, which can pollute the visualization if not removed; if you use a different dataset or headset, you will have to change this to remove a different set of garbage channels.
The last three processing nodes compute the sliding-window root-mean-square power of the signal (in the frequency band that the FIR filter retains), and finally averaged across all channels.
The 8-14 Hz band is also known as the 'alpha' frequency band in EEG, which primarily picks up idle activity of neurons in the brain. That is, when one relaxes and/or closes the eyes, the alpha power, which is computed by this filter chain, will go up, and that's why this pipeline computes a very simple relaxation measure.
Getting data in and out of Neuropype
Lab Streaming Layer
You might be asking how the signal makes it from the import node to the FIR filter node, because there is no wire between the two nodes. The reason is that, in between the
LSL Output node at the bottom of the pach and the
LSL Input on the top left the data travels invisibly over the LSL network. LSL works even between different computers on the same WiFi or LAN, i.e., if the LSL output node was on one PC and the LSL input node was placed in a second pipeline that you have open on another PC, data would flow from one to the other in just the same way (unless you computer's Firewall gets in the way). Since there could be many inlets and outlets present on a given WiFi or LAN, there could be quite some confusion, and therefore the outlet generally assigns a unique name to the stream, while the inlet is set to look for a stream with that particular name on the LSL network.
LSL is not only a way to get data from one computer to another, but also to get data from your EEG system, or any other kind of sensor that supports it, into NeuroPype. You can also use it to get data out of NeuroPype into external real-time visualizations, stimulus presentation software, and so on.
This pipeline uses LSL simply to make it very easy for you to switch from playing back pre-recorded data, which is what this pipeline is doing, to processing data from a live EEG system. All you have to do is delete the three nodes at the bottom, and then tell the LSL Input node to read from your EEG amplifier instead.
LSL is probably the easiest way to get live sensor data into NeuroPype. If you have a sensor that is supported by it (see https://github.com/sccn/labstreaminglayer/wiki/SupportedDevices.wiki), then you only need to get the sensor, start the acquisition program, enable LSL streaming in that program (or for some sensors, start additionally an LSL client that translates the acquisition program's output to LSL), and make sure that the
LSL Input node has a query string that matches that sensor. For instance, if you use
type='EEG' as below, then you will pick up data from the first EEG device on the network that it finds:
... this is fine if you're testing this at home, but if you're at a public place where others may be running their own LSL streams (or perhaps you are running more than one stream, such as one playback and one actual device), you must be more specific than this. For instance, you can refine the search using the name of the stream, as in
name='Cognionics Quick-20' to read from a specific Cognionics headset. But if there are more than one headset of that type in use on the network, you won't know which one you're reading from -- in this case, it's best to include your computer's hostname, as well, as in the following example:
As a side node, when you change the LSL stream, you may well end up with a different number of channels (or sampling rate, etc.), than what you had before in the playback data, and when you do that while the pipeline is running, some nodes may be unprepared for that switch. Think of it like changing engine parts of a moving Formula 1 car. You can do a lot of changes while the pipeline is running, but if you do encounter an error, you can click the reload button (the circular arrow next to pause/unpause) to reset the pipeline back into a healthy state. Note that any state that the pipeline may have accumulated (such as buffers or calibration) is cleared by this action, as well.
Using Vendor-Supplied Software
How data from a sensor gets into LSL depends on the specific sensor, as mentioned before -- a good starting point is to download the latest LSL distribution from https://github.com/sccn/labstreaminglayer (which also includes some useful debugging utilities), check whether any matching app is included for your hardware, and if not, check if the vendor hardware happens to have LSL support. There is also a fairly recent copy of an LSL distribution included with the NeuroPype Suite. Or alternatively you can check if your vendor happens to support LSL natively, which many of them do. For instance, in the case of the Muse, a program called muse-io (shown below) is included by the vendor.
This program can be invoked from the command line to stream out over LSL, as shown below:
In this case you can give the stream a name yourself (we used 'Muse' here) which allows you to disambiguate your stream from others that may be on the network, but most vendors will hard-code the stream name.
For more detailed information on the use of various hardware with Lab Streaming Layer please see this section.
Biosignal data processing
Using Artifact Removal
Most robust EEG processing pipelines require some sort of artifact removal. NeuroPype has a number of processing nodes for this job, for instance the
Channel Repair node shown in the below graph, which you can open by loading the file
ArtifactRemoval.pyp in the examples folder.
Be sure to inspect the settings of these nodes when you want to add EEG artifact removal to your own pipeline, and also make sure you visually check the quality of the artifact removal using some time series plots as done here to ensure that your settings are good. You need to remember the following things: (1) you need to remove garbage channels beforehand (done here by the Select Range node, which is here set to keep the first 4 channels, since this example was written for the Muse). (2) you must have a decent high-pass filter before artifact removal that reliably removes drifts (here this is an FIR filter with a sufficiently high order, at least 50db suppression, and 0.5 to 1Hz transition band). If you sensor has enormous drifts, you may need to increase the suppression and/or order. (3) then come the artifact removal nodes, which work only on EEG, not on other kinds of signals. (4) these nodes are adaptive, and they must first see a certain amount of data, e.g., 20 seconds, in order to calibrate themselves.
These nodes will first buffer up several seconds of data until they have enough (during that time the graph swill output nothing), then it calibrates itself, cleans the data that it has buffered, and sends it out as one big chunk; once that's happened, the nodes will clean and pass through any subsequent small chunks that they gets (i.e., operate in real time). While the nodes are collecting calibration data, you will see only one plot showing the raw data streaming, but once they are calibrated, a second plot will open up and shows the clean data next to it, as below.
Accumulating Calibration Data for Machine Learning
Next we will discuss machine learning (specifically supervised learning). If you're not familiar with that, or have no intention to use it at this point, you can skip or gloss over these sections. The machine learning nodes in NeuroPype are also adaptive, but unlike artifact removal, they will not simply buffer up the first few seconds of session data on their own (because that data for calibration must meet certain conditions, for instance it must include labels). Instead, we place a special node early in the pipeline whose job it is to buffer up the calibration data, and then to output it in one big chunk that travels through the rest of the pipeline, causing each subsequent node to update (and calibrate) itself accordingly. From then on, subsequent data packets pass through the graph normally, and all nodes (now calibrated) can transform the raw inputs into predictions as they should. This special node is called
Accumulate Calibration Data.
Imagined Movements Classification
The use of that node is shown in the file
SimpleMotorImagery1.pyp, which you should now open. The pipeline should look as follows:
This pipeline can predict from EEG whether you're currently imagining one of multiple possible limb movements (default: left hand movement vs. right hand movement, a.k.a., two-class classification). The output of this pipeline at any tick is the probability that the person imagines the first type of movement, and the probability for the second type. Note that this pipeline prints its output to the NeuroPype console. If you are not seeing the console, you can open it from the Options menu by selecting
Show Output View. You will almost certainly want to dock the output view to the bottom of the VPE canvas by grabbing the title bar and dragging it over the bottom 10% of the canvas, at which point a visual dock indicator appears, and releasing.
Since the EEG patterns associated with these movements look different for any two people, several nodes (here: Common Spatial Patterns and Logistic Regression) must first adapt themselves based on some calibration data for the particular user, and moreover, it's not enough for the calibration data to be some arbitrary EEG but must meet certain criteria as we already alluded to earlier (while we show it here for imagined movements, the same rules apply to pretty much any use of machine learning on EEG). First, the node needs to obtain example EEG of left-hand movement and example EEG of right-hand movement, respectively. Also, a single trial per class of movement is far from enough, rather the node needs to see closer to 20-50 repeats when you use a full-sized EEG headset. And lastly, and these trials also must be in more or less randomized order, i.e., not simply a block of all-left trials followed by a block of all-right trials. Collecting data in that way is one of the most common beginner mistakes with machine learning on time series, and it is important to avoid it, even if it means collecting some new data.
Working with markers
For the aforementioned reasons, the EEG signal must be annotated such that one can tell at what points in the signal we have example data for class 1 (person imagines left hand movement) and where we have example data of class 2 (right hand movement). One way to do it is to include a special 'trigger channel' in the EEG, which takes on pre-defined signal levels that encode different classes (e.g., 0=nothing, 1=left, 2=right). While NeuroPype can parse trigger channels (by inserting a
Parse Trigger Channel node just before the
Accumulate node), this pipeline does not use them.
Instead, the pipeline assumes that the data packets emitted by the LSL Input node have not just one stream (EEG), but also a second stream that has a list of marker strings plus their time stamps (markers), i.e., they are multi-stream packets and there are consequently two data streams flowing through the entire pipeline. The markers are then interpreted by the rest of the pipeline to indicate the points in time around which the EEG is of a particular class (in this pipeline, a marker with the string 'left' and time-stamp 17.5 would indicate that the EEG at 17.5 seconds into the recording is of class 0, and if the marker had been 'right' it would indicate class 1). Of course the data could contain any amount of other random markers (e.g., 'recording-started', 'user-was-sneezing', 'enter-pressed'), so how does the pipeline know what markers encode classes, and which classes do they encode? This binding is established by the
Assign Targets node, whose settings are shown below (the syntax means that 'left' strings map to class 0, 'right' maps to class 1, and all other strings don't map to anything).
The second question is, given that there's a marker at 17.5 seconds, how does the pipeline know where relative to that point in time we find the relevant pattern in the EEG that captures the imagined movement? Does start a second before the marker and end a second after, or does it start at the marker and ends 10 seconds later? Extracting the right portion of the data is usually handled by the
Segmentation node, which extracts segments of a certain length relative to each marker. The below graphic shows the settings for this pipeline, which are interpreted as follows: extract a segment that starts 0.5 seconds after each marker, and ends 3.5 seconds after that marker (i.e., the segment is 3 seconds long). If you use negative numbers, you can place the segment before the marker.
The third node in the top row is our accumulate node. Double-click it to inspect the settings:
The most important detail here are the
Begin marker and
End marker fields: this node assumes that the beginning and end of the period in the data stream that contains calibration data is identified by markers in the data stream, and these two fields let you choose which marker strings the node is looking for. So in summary, the pipeline expects that your data stream has at some point a marker string named 'calib-begin', followed by a bunch of 'left' and 'right' markers in pseudo-random order, followed by a 'calib-end' marker.
Calibration Marker Generation
So now you want to calibrate the pipeline: how do you get the markers into your EEG stream? If you're using a pre-recorded file (e.g., the example file that the pipeline is loading and playing back in the bottom three nodes), then the required markers are hopefully already in the file (in this case we made sure they are). However, if you're working live, i.e., you're wearing an EEG headset, how do you generate the markers live? This is actually again very easy using the lab streaming layer: basically you need a script (for instance, in Python, MATLAB or C#) that calls an LSL function like send_marker('left') -- have a look at the examples at https://github.com/sccn/labstreaminglayer/wiki/ExampleCode.wiki (look out for examples titled "Sending a stream of strings with irregular timing" or "Sending string-formatted irregular streams" for the language that you're using). For instance, here's how to send a marker in Python:
Of course a real program would send all the markers, and since those need to match what the person is imagining, it would basically manage the entire calibration process and instruct the user when to imagine what movement (that's also where the 0.5 and 3.5 in the segmentation node come from -- it takes about half a second for a person to read the instruction and begin imagining the movement, and he/she will finish about 3 seconds later and get ready for the next trial). The source and binary releases include a small example Python script in the
Extras/ folder where you installed the NeuroPype Suite, which can be used to print simple instructions and emit markers like these.
If you want to throw machine learning at other prediction tasks, e.g., workload, you would also author a corresponding calibration program that, for example, puts the person into high workload (for instance by setting the game difficulty to high) and emit, say, a 'high' marker into LSL, and then puts them into a low workload state and emit a 'low' marker, and so on. Or to calibrate emotion detection you could try to play happy vs. sad music, and so on.
Picking up Marker Streams with LSL
LSL Input node is responsible for returning a marker stream together with the EEG. For this to happen, it first needs to find and pick up that stream on the network, of course. The name of the marker stream that it should look for can be entered in the second settings field, named
Marker query (or it can also be a compound expression to ensure that it's uniquely identified on the network, similarly to the example earlier under Using LSL):
LSL Output node used to send out the data which is being played back by the
Stream Data node at the bottom of the pipeline is set up to send out not just the EEG stream, but also a marker stream:
So in summary, the data file that's being imported by the
Import Set node contains 2 streams (EEG and markers), which you can also verify by importing that file into EEGLAB for inspection (if you have MATLAB). Then, this 2-stream recording travels into the
Stream Data node, gets stored in memory, and then is being played back in real time and chunks of it are forwarded to the
LSL Output node. This node creates 2 LSL streams, which are picked up by the
LSL Input node, and then travel through the entire remainder of the pipeline. And if you delete the three playback nodes at the bottom, you need to have 2 live LSL streams, one from your headset and one from your app that controls the calibration process, in order to calibrate the pipeline. By the way, since the calibration session is actually over 15 minutes long, in this example the
Stream Data node is set up to play back the data at 10x real-time speed, to shorten the wait time for you until something interesting happens (you can dial it back to 1x by editing the node settings).
Using Pre-Recorded Calibration Data
Our pipeline is doing a great job, but wouldn't it be nice if we didn't have to re-collect the calibration data each time we ran the pipeline? Indeed it's often more convenient to record calibration data into a file in a first session, and then just load that file every time we want to run our pipeline. This is actually quite easy, and for this the
Accumulate Calibration Data node needs to be replaced by an
Inject Calibration Data node, which has a second input port where one can pipe a calibration recording in (which we import here using
Import Set). This is implemented in
SimpleMotorImagery2.pyp, and the relevant section looks as follows (rest stays the same):
The imported calibration recording will be injected into the data stream on the first tick, and so it travels through the rest of the pipeline, all nodes can calibrate themselves properly based on it. After that, the regular real-time stream coming from the
LSL Input node will flow through the graph and get transformed by it into predictions.
Recording Calibration Data
Calibration data, just like anything else you can send over LSL, can be recorded using the
LabRecorder program that comes with the LSL distribution. Wwe recommend that you check out the documentation for this program at https://github.com/sccn/labstreaminglayer/wiki/LabRecorder.wiki if you're unsure how it works. The NeuroPype installer will by default install a version of Lab Recorder that should work out of the box. One thing should be noted, however: to prevent NeuroPype from getting confused, make sure that you record only one marker stream and uncheck other marker streams that may perhaps be online on the same network (the recorder lets you uncheck streams that you don't want in its list of streams), also uncheck any other miscellaneous streams that you don't need (e.g., other EEG streams or streams hosted by other users/programs):
You can also check out a screencast that walks you though how to collect calibration data here.
More example pipelines
Next, check out the other examples and their documentation. You may find a pipeline that is already doing what you want to accomplish, or which you can adapt, and can save yourself building one from scratch yourself.
Now you should be ready to roll with NeuroPype!