Trodes Export Utility Functions =============================== .. contents:: --------------------- Trodes Export Utility allows the user to extract specific data features (Spikes, LFP, Raw, etc.) from the ``.rec`` recording file. The extracted features, or data bands, are saved to subfolders in the original data directory in ``.dat`` binary format, and are named according to their data type. This makes it easy to import data into Matlab or Python for further analysis using the `Matlab and Python Import Utilities _` provided with Trodes. Trodes Export Utility can be used from within Trodes by navigating to File > Extract..., or from the command line using the trodesexport application. Additional extraction options are available when using trodesexport from the command line. **Note: In Trodes versions before 2.0.0, export functions were separated into different command line programs for each data type (exportLFP, exportSpikes, etc.). These functions are now deprecated and have been combined into a single export tool called "trodesexport". This program can either be run from the command line, or from Trodes when a recorded file is opened in playback mode. The following instructions are for use with trodesexport.** Trodes splits neural data into three conceptual streams, which are used during extraction: 1. Raw. The only modification that can be applied to raw data in the exporter is whether or not the designated digital reference is subtracted from the signal. No filters are applied. 2. Spikes. This uses bandpass filtering and digital referencing. 3. LFP. This uses lowpass filtering, referencing, and data rate reduction (default to 1500 Hz from 30 kHz). Data Extraction Using the Trodes Interface ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To extract data from within Trodes, open a previously recorded ``.rec`` file in playback mode using the Open Playback File option from the main menu of Trodes. Once the file is open: - Go to File > Extract... - This will open a separate window named "Extract data from file" - On the left side of the window, check the data bands you would like to extract - Click the "Start" button - Export progress is shown in the embedded console window Controlling Export Settings Using the Workspace ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ By default, Trodes exports data using the workspace display settings present at the time of recording (channels, filters, references, LFP channel, spike detection thresholds, etc.). These settings are embedded in the ``.rec`` file when it is initially saved and applied to the playback workspace when opening the file prior to export. These workspace settings control data export. This means that the channels being exported and the export settings can be changed by altering the workspace. This will determine how and which data is displayed/exported, but will not alter the saved data in any way. **IMPORTANT NOTE:** When Trodes loads a data file in Playback mode, it does so using the workspace file that shares the same name as the ``.rec`` file, and is saved to the same data directory as the file being viewed (e.g. DataDir/myfile.rec & DataDir/myfile.trodesconf). By default, this is also true of exporting data using the command line, however it is possible to manually specify the export workspace file using the ``-reconfig`` argument. This can be especially useful when batch exporting data. Changing exporting settings via the workspace can be accomplished in 3 ways: 1. Open, reconfigure and save the original the workspace file that shares a name and is saved to the same directory as the ``.rec`` file. 2. Move the original workspace file to a different directory or delete it and create a new workspace file of the same name that replaces it in the original data directory. 3. Use the ``-reconfig`` argument to specify the workspace used for export. This file does not need to be in the same directory as the ``.rec`` file(s) data is being extracted from. **Example:** You took a 128-channel recording, but have determined that 32 of the channels are unusable (bad shanks happen...). The unusable channels can be removed from export by creating a new workspace file where these channels have been removed from display. In this case, simply open the original workspace file used to record via the Edit existing file... option under Create/Edit Workspace in the Trodes Main menu. Navigate to the Channel Map tab in the workspace editor, and change the nTrode IDs for the bad channels to "Not assigned". Save the workspace, or use the save as option to create a new workspace. When creating a new workspace file, this workspace must have the the same name as the ``.rec`` file. This workspace will be swapped into the original data folder (moving the original workspace to another location). When the data if opened in playback mode using the new workspace, the unwanted channels will not be displayed or included in data export. This method can also be used to update settings such as LFP subsampling rate, spike detection threshold, etc.. Processing Modes (one required; multiple selections allowed) ------------------------------------------------------------ All data bands listed below are available when exporting the data from the command line or using the Trodes GUI. - *-spikes* -- Spike waveform export (spike snippets only). - *-lfp* -- Continuous LFP band export. Sampling rate controlled using the LFP subsampling dropdown in the workspace editor. - *-spikeband* -- Continuous spike band export. - *-raw* -- Continuous raw band export. - *-stim* -- Continuous stimulation band export for stimulation capable recording channels. - *-dio* -- Digital IO channel state change export. - *-analogio* -- Continuous analog IO export. - *-time* -- Trodes system time in samples. - *-mountainsort* -- One mountainsort file per ntrode (using raw band data) - *-kilosort* -- One kilosort file per ntrode (using raw band data). **Exporting Neuropixels Data:** In contrast to many other probe types, the Neuropixels probe does not export a single broadband "raw" trace for each channel. This is because the Neuropixels probe actively filters and divides the incoming data into 2 streams for each channel: Spike band and LFP band. The Spike band signal has been band pass filtered to eliminate LFP oscillations, whereas the LFP band has been low pass filtered and downsampled to 2.5 Khz. This means that exporting Neuropixels data using the "raw" option in Trodes will not result in raw and unfiltered data, but rather spike band data. Exporting using the LFP option will export the LFP band data. Additionally, in versions of Trodes prior to 2.4.2, doing this may trigger an error message. **Important Notices:** 1. A bug has been identified in versions of Trodes prior to 2.5.0, where LFP band data channel mapping can be scrambled on export when the full number of available channels were not used during recording. This most often occurs when recording from multiple probes untethered where the max recordable channels is limited to 400 across all connected probes. 2. Prior to Trodes 2.6.0 gain was not properly accounted for when generating the Voltage_Scaling variable used for converting data to uV scale. This bug incorrectly applied the default value to all data regardless of the user-selected gain. This issue has been addressed in Trodes 2.6.0. For any data exported using a previous version of Trodes, the correct Voltage_Scaling value can be computed using the gain value reported in the the spikeconf section of the workspace file used for data acquisition. This value is computed as follows: Voltage_Scaling = (1000.0/GAIN)*(600.0/32767.0), where GAIN is the reported gain value used when recording. **Exporting for Kilosort:** Data exported from Trodes for Kilosort will include the following supporting files: - .channelmap -- Binary file with simple header that can be read into Matlab and used to create the .m coordinate file required when using Kilosort. Coordinate information is stored in the fields struct. When recording with Neuropixels probes, coordinate information is automatically generated based on your channel configuration. - .timestamps -- This file contains the Trodes timestamps for each sample. This is because Kilosort does not use timestamps, but returns the sample index of the original data. The returned indices can be used to retrieve the original Trodes timestamps when the spikes occurred. These timestamps can be used for aligning spiking activity to event data recorded using the digital IOs. **Low Sampling Rate Mode:** When exporting data recorded using the 2000 Hz sampling rate option, changing the LFP subsampling rate to 1 is recommended. This is because the LFP subsampling rate controls the sampling rate for exported LFP data. Data Extraction from the Command line ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The trodesexport application can be executed from the command line and doing so offers expanded extraction options that are not available when using the Trodes GUI, such as batch processing. To do this, simply open the command line, set your current directory to the Trodes application folder, and call trodesexport with the relevant input arguments. trodesexport Command Line Inputs -------------------------------- **trodesexport** <-h -v> <-spikes -lfp -spikeband -raw -stim> -rec FILENAME <-optionflag1 optionvalue1> ... - If the *-h* flag is entered, a full usage menu will print to the screen - If the *-v* flag is entered, the program's version number will print to the screen - At least one of the processing flags must be provided (such as *-spikes*) - A rec file must be provided after the *-rec* flag - By default, the trodesexport program will use the settings defined when recording the data or the settings defined in an external workspace (see above). Options for overriding this using the command line are detailed below. Optional Flags -------------- - *-rec * -- Recording filename. Required. Muliple *-rec * entries can be used to append data in output. - *-output * -- The base name for the output files. If not specified, the base name of the first .rec file is used. - *-outputdirectory * -- A root directory to extract output files to (default is directory of .rec file). - *-reconfig * -- Use a different workspace than the one embedded in the recording file. - *-interp * -- (Default is -1 for infinite) Maximum number of dropped packets to interpolate over (linear). A value of -1 is infinite. 0 is no interpolation. - *-sortingmode <0,1, or 2>* -- Used by mountainsort and kilosort modes. Can be a value of 0, 1, or 2 (default). 0 combines channels into one file. Channels in the file will be in the same order as that in the workspace, but won't be divided into separate ntrodes. A value of 1 separates files by nTrodes. A value of 2 (default) separates files by their designated sorting groups, which are set manually by the user. Channel map coordinates are also set manually by the user. - *-abortbaddata <1 or 0>* -- Whether or not to abort export if data appears corrupted. - *-paddingbytes * -- Used to add extra bytes to the expected packet size if an override is required Datatype-Specific Options ------------------------- **LFP data band** - *-lfplowpass * -- Low pass filter value for the LFP band. Overrides settings in workspace for all nTrodes. - *-lfpoutputrate * -- (Default 1500) The LFP band output can have a sampling rate less than the full sampling rate. - *-uselfprefs <1 or 0>* -- Override what is in the workspace for lfp band reference on/off settings for all nTrodes. - *-uselfpfilters <1 or 0>* -- Override what is in the workspace for lfp filter on/off settings for all nTrodes. **Spike data band** - *-spikehighpass * -- High pass filter value for the spike band. Overrides settings in workspace for all nTrodes. - *-spikelowpass * -- Low pass filter value for the spike band. Overrides settings in workspace for all nTrodes. - *-invert <1 or 0>* -- (Default 1) Sets whether or not to invert spikes to go upward. - *-usespikerefs <1 or 0>* -- Override what is in the workspace for spike band reference on/off settings for all nTrodes. - *-usespikefilters <1 or 0>* -- Override what is in the workspace for spike filter on/off settings for all nTrodes. **Spike event processing** - *-thresh * -- Overrides the thresholds for each nTrode stored in the workspace and applies the new threshold to all nTrodes. **Raw data band** - *-userawrefs <1 or 0>* -- Override what is in the workspace for raw band reference on/off settings for all nTrodes. Other Export Utilities ~~~~~~~~~~~~~~~~~~~~~~ Some export formats are not included in the main *trodesexport* utility. These are provided as separate command line programs: **exportofflinesorter (**\ `Offline Sorter `__\ **)** Used to extract continous data from a raw rec file and save to binary file that can be imported with Offline Sorter. *exportofflinesorter* -rec INPUTFILENAME OPTION1 VALUE1 OPTION2 VALUE2 ... Input argument defaults: - *-outputrate* -1 (full) - *-usespikefilters* 1 (Filter the channels with the saved spike filters) - *-interp* -1 (interpolation not supported, do not change) - *-combinechannels* 1 (combine neural channels into one file. Instead use one file per channel.) - *-appendauxdata* 0 (do not append DIO and analog data into the file) - *-userefs* 1 (Use the digital reference channels designated in the file) To import the binary file into Offline Sorter, from the main menu of Offline Sorter select File | Import | Binary File with Continuously Digitized Data. Choose the following settings: - Number of channels: the total number of channels in the exported file - Data offset: 0 - Digitization freq: 30000 or 20000 (depending on what you actually used) - A/D Conversion resolution: 16 - Maximum Voltage (mV): 6.39 - Swap bytes: No