Skip to content

Allow concatenating BaseRaw objects#13263

Open
cbrnr wants to merge 11 commits intomne-tools:mainfrom
cbrnr:concatenate-baseraw
Open

Allow concatenating BaseRaw objects#13263
cbrnr wants to merge 11 commits intomne-tools:mainfrom
cbrnr:concatenate-baseraw

Conversation

@cbrnr
Copy link
Copy Markdown
Contributor

@cbrnr cbrnr commented May 27, 2025

As reported in this forum post, mne.concatenate_raws() only works if all objects share the exact same type. In particular, this means that it is not possible to concatenate Raw (an alias for RawFIF) and RawArray objects, and other types like RawEDF, RawEEGLAB, etc. do not work either. However, since all concrete types inherit from BaseRaw, I think we could relax the compatibility check accordingly.

The following example fails on main, but works on this branch:

import numpy as np

import mne

sfreq = 100.0
ch_names = ["EEG 001", "EEG 002"]
ch_types = ["eeg"] * 2
info = mne.create_info(ch_names=ch_names, sfreq=sfreq, ch_types=ch_types)
data = np.random.randn(len(ch_names), 1000)

raw_array = mne.io.RawArray(data, info)
raw_array.save("temp_raw.fif", overwrite=True)
raw_fiff = mne.io.read_raw_fif("temp_raw.fif", preload=True)

mne.concatenate_raws([raw_fiff, raw_array])

@cbrnr cbrnr requested review from drammock and larsoner as code owners May 27, 2025 06:43
@cbrnr cbrnr force-pushed the concatenate-baseraw branch from 72c4701 to 71ad5a7 Compare May 27, 2025 06:45
@cbrnr cbrnr force-pushed the concatenate-baseraw branch from d7aa4dd to b656479 Compare May 27, 2025 06:48
@hoechenberger
Copy link
Copy Markdown
Member

I didn't review any of the changes and don't know if there's any side effects to be expected; but purely from a user's POV, I'd say we should definitely try to support this

Copy link
Copy Markdown
Member

@sappelhoff sappelhoff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think we should support this. Potential issues could arise from concatenating Raw types that support different attributes. For example, the BrainVision Raw supports impedances, but other Raws might not.

@larsoner
Copy link
Copy Markdown
Member

At a bare minimum we would have to ensure all data are preloaded. And as @sappelhoff points out to keep things simple the resulting class should probably be RawArray since IIRC it doesn't have any extra attributes like impedance and such. (We could later maybe think about how to combine/keep them but I'd rather instead move toward removing them entirely.)

@cbrnr cbrnr force-pushed the concatenate-baseraw branch from aeb72bd to 7fc7634 Compare April 23, 2026 09:47
Co-authored-by: Copilot <copilot@github.com>
@cbrnr
Copy link
Copy Markdown
Contributor Author

cbrnr commented Apr 23, 2026

@larsoner I implemented your suggestions, I think this should be ready for review now.

Comment thread mne/io/base.py Outdated
Comment thread mne/io/base.py Outdated
Comment on lines +3248 to +3250
for r in raws:
if not r.preload:
r.load_data()
Copy link
Copy Markdown
Member

@larsoner larsoner Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit surprised you need this. It seems like the logic in raws[0].append(raws[1:], preload) can/should be smart enough to handle this case. Did you try to make this work?

In other words, I would expect something of the form:

if mixed_types:
    raws = list(raws)  # local copy of list
    raws[0] = RawArray(raws[0]._data, raws[0].info, first_samp=raws[0].first_samp)
    preload = True
raws[0].append(raws[1:], preload)

after the events code to be potentially much less memory intensive (by a factor approaching 2), and also fewer lines and easier to follow here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, you may be right. I pushed a fix, but I think I still have to preload raws[0] in order to use raws[0]._data, right? Also, I think we need to copy the annotations.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah you're right, you'd either need to preload that one or use .get_data. And yes probably annotations too

cbrnr and others added 3 commits April 23, 2026 16:45
Co-authored-by: Eric Larson <larson.eric.d@gmail.com>
Co-authored-by: Copilot <copilot@github.com>
@cbrnr
Copy link
Copy Markdown
Contributor Author

cbrnr commented Apr 23, 2026

I mean, I'm starting to doubt if this is really worth it and if people should just do the conversion manually... WDYT?

@larsoner
Copy link
Copy Markdown
Member

Well there are some slightly tricky bits that people can silently get wrong, like the first_samp plus annotations copying. So I think it's probably still worthwhile

@cbrnr
Copy link
Copy Markdown
Contributor Author

cbrnr commented Apr 23, 2026

OK, I think I implemented all of your suggestions.

@larsoner
Copy link
Copy Markdown
Member

Copy link
Copy Markdown
Member

@larsoner larsoner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM +1 for merge once CIs come back green!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants