Pair EMG and orientation recordings from 3DC, build a trial-level fusion baseline, and test whether the second modality contributes useful information.
Download the complete runnable series (Python scripts, shared helpers and setup instructions).
Use 64-bit Python 3.11 and LibEMG 2.0.3. Keep the downloaded files together: the walkthrough uses common.py from that bundle. The figures below were generated by these scripts; results describe the stated recordings and split.
Check what the second modality actually contains
The 3DC device includes an inertial sensor, but the files used here contain four-component orientation exports, not nine columns of raw acceleration, angular velocity and magnetic field. We treat their near-unit four-vectors as quaternion orientation representations. Component order and coordinate-frame conventions are not used to assign physical axes. The accompanying plot labels stored components rather than inventing acceleration units.
This exercise uses only Participant1: eleven gestures, four training repetitions and four test repetitions, giving 44 trials in each split. Pair files by participant, split, repetition and gesture. Matching array row indices would be unsafe: the EMG and orientation recordings have different sample counts, and these exports do not establish sample-level timestamp alignment.
Load paired trials and reject invalid inputs
import ast
import json
import numpy as np
import matplotlib.pyplot as plt
from libemg.utils import get_windows
from libemg.feature_extractor import FeatureExtractor
from libemg.filtering import Filter
from sklearn.metrics import balanced_accuracy_score
from common import DATA, OUT, lda, savefig
base = DATA / '3dc' / 'Participant1'
emg_filter = Filter(1000)
emg_filter.install_filters({'name': 'bandpass', 'cutoff': [20, 450], 'order': 4})
def paired_trials(split):
emg_rows, orientation_rows, labels, repetitions = [], [], [], []
for path in sorted((base/split/'EMG').glob('*.txt')):
repetition, label = map(int, path.stem.split('_')[-2:])
imu_path = base/split/'IMU'/path.name.replace('_EMG_', '_IMU_')
if not imu_path.is_file():
raise FileNotFoundError(f'Unpaired trial: {path.name}')
emg = np.loadtxt(path, delimiter=',')
q = np.array([ast.literal_eval(line) for line in imu_path.read_text().splitlines() if line.strip()])
if q.ndim != 2 or q.shape[1] != 4:
raise ValueError('Expected four-component orientation samples')
norm = np.linalg.norm(q, axis=1)
valid = np.isfinite(q).all(axis=1) & (norm > .5) & (norm < 1.5)
q = q[valid] / norm[valid, None]
if len(q) < 10:
raise ValueError('Too few valid orientation samples')The next block continues inside paired_trials. It summarizes each modality over its complete matched trial, so unequal sampling rates do not require invented synchronized windows. This is an offline static-gesture experiment. A whole-trial representation would not provide a real-time prediction before that trial finishes.
Construct complementary summaries
EMG uses 250 ms windows with a 50 ms hop at 1 kHz, extracts MAV and WL, then averages the resulting twenty-dimensional vectors across the trial. For normalized orientation samples \(q_i\), form \(M=T^{-1}\sum_iq_iq_i^T\) and retain its upper triangle. Since \((-q)(-q)^T=qq^T\), the summary avoids the sign ambiguity of a quaternion representation. Ten retained entries describe a symmetric matrix; they are not all independent because its trace is one.
windows = get_windows(emg_filter.filter(emg), 250, 50)
features = FeatureExtractor().extract_features(['MAV', 'WL'], windows, array=True)
emg_rows.append(features.mean(axis=0))
# q and -q represent the same orientation: qq^T has the same sign invariance.
moment = np.einsum('ni,nj->ij', q, q) / len(q)
orientation_rows.append(moment[np.triu_indices(4)])
labels.append(label)
repetitions.append(repetition)
return np.array(emg_rows), np.array(orientation_rows), np.array(labels), np.array(repetitions)
train_e, train_q, train_y, train_reps = paired_trials('train')
test_e, test_q, test_y, _ = paired_trials('test')
assert len(train_y) == len(test_y) == 44The orientation summary captures a distribution of stored orientations, while averaging discards their temporal order. It can help if pose provides information beyond EMG, or hurt if pose varies incidentally between training and test. We deliberately avoid calling this a general motion-recognition representation.
Compare each modality with the same classifier
Feature-level fusion concatenates \(z=[z_{\mathrm{EMG}};z_{\mathrm{orientation}}]\). Fit a fresh standardized shrinkage-LDA pipeline for EMG alone, orientation alone and their concatenation. The three inputs contain twenty, ten and thirty columns respectively. Keep every condition’s training and test trials identical so the ablation answers whether this particular added representation helps.
conditions = {'EMG': (train_e, test_e), 'Orientation': (train_q, test_q),
'EMG + orientation': (np.c_[train_e, train_q], np.c_[test_e, test_q])}
results = {}
for name, (train, test) in conditions.items():
model = lda().fit(train, train_y)
prediction = model.predict(test)
results[name] = {'dimensions': train.shape[1],
'test_balanced_accuracy': float(balanced_accuracy_score(test_y, prediction)),
'correct_trials': int((prediction == test_y).sum()), 'total_trials': len(test_y)}
(OUT/'07-results.json').write_text(json.dumps(results, indent=2))
print(results)
EMG alone classified 44/44 held-out trials correctly (100%). Orientation alone classified 8/44 (18.18%). Fusion classified 42/44 (95.45%). Adding dimensions did not improve this small experiment. The result does not show that inertial sensing is generally unhelpful; it shows that this whole-trial summary and classifier did not add a benefit for these recordings.
What changes for synchronized or cross-user recognition?
For window-level fusion, obtain timestamps or a documented synchronization relation, define the same physical interval for each modality, and extract features from each interval at its own sampling rate. Do not align solely by row number. Quaternion interpolation also needs to respect sign and rotation geometry rather than applying ordinary interpolation indiscriminately.
For a stronger study, repeat the ablation across participants and sessions, fitting all transforms within each training split. Separate changes of gesture from changes of arm position. Otherwise orientation may provide an incidental contextual shortcut that disappears when gestures are performed in new poses. Missing-modality robustness is another distinct question that requires an explicit evaluation.
Try it yourself
Add another participant and run the same three-condition ablation separately before pooling data. Inspect which gestures lose accuracy under fusion. Then design a pose-controlled split that could distinguish gesture information from position context.
Run the full companion script to regenerate the figures and machine-readable results:
python 07_multimodal.pyLibEMG documentation: offline data and metadata