diff --git a/deep_learning/Q2/ALGORITHM.md b/deep_learning/Q2/ALGORITHM.md new file mode 100644 index 0000000..efa285d --- /dev/null +++ b/deep_learning/Q2/ALGORITHM.md @@ -0,0 +1,1367 @@ +# Q2 多模态情感预测:当前两个核心方案 + +我们目前保留两个主要模型: + +1. **方案一:Mask-aware EarlyConcat + BiGRU** + - 结构简单; + - 参数少; + - 是我们的强基线模型。 + +2. **方案二:MoFE-7 + MLP Router** + - 针对 EarlyConcat 在所有位置使用同一融合形式的限制,显式学习不同模态子集的组合权重; + - 已按数学方案的 Q2 口径重训;本次单种子结果中,Accuracy 略高,但 Macro-F1、回归指标和缺失情景汇总点估计低于 EarlyConcat。 + +二者其实是一条很自然的递进路线: + +> **方案一:把所有可用模态直接交给一个统一模型学习。** +> **方案二:先判断当前更适合使用哪一种模态组合,再交给统一模型学习。** + +--- + +# 一、数据在进入模型之前是什么样子? + +我们使用官方提供的对齐后多模态特征。 + +每条样本被表示为长度为 50 的有序序列: + +\[ +t=1,2,\dots,50. +\] + +注意,这里的 50 个位置是 **wordpiece ordered positions(按文本词片顺序排列的位置)**,不是严格意义上的 50 个等长物理时间片。 + +在每个位置 \(t\),都有三种模态: + +\[ +T_t\in\mathbb{R}^{768} +\] + +表示文本 BERT 特征; + +\[ +A_t\in\mathbb{R}^{74} +\] + +表示音频特征; + +\[ +V_t\in\mathbb{R}^{35} +\] + +表示视觉特征。 + +同时还有三个 observation mask: + +\[ +M_t^T,\quad M_t^A,\quad M_t^V. +\] + +它们告诉模型: + +> 当前位置的 Text / Audio / Vision 到底有没有有效观测。 + +因此 mask 非常重要。 + +例如某个 Audio 特征全为 0: + +- 没有 mask 时,模型不知道这是“真实特征刚好接近 0”; +- 有 mask 时,模型知道这是“Audio 当前缺失”。 + +--- + +# 二、方案一:Mask-aware EarlyConcat + BiGRU + +## 2.1 一句话理解 + +EarlyConcat 的思想非常直接: + +> **同一个位置上的文本、音频、视觉特征全部放在一起,然后交给一个 BiGRU,让模型自己学习三种模态之间以及前后位置之间的关系。** + +整体流程: + +\[ +\boxed{ +T/A/V + Masks +\rightarrow +Early Concatenation +\rightarrow +BiGRU +\rightarrow +Sequence\ Aggregation +\rightarrow +Classification/Regression +} +\] + +--- + +## 2.2 第一步:同位置多模态拼接 + +对于位置 \(t\),我们有: + +\[ +T_t,\quad A_t,\quad V_t. +\] + +当前实现先分别把三个模态投影到 128 维,再加上位置和模态标记,并用 mask 清零缺失模态: + +\[ +z_t^m=M_t^m\left[\operatorname{LN}(\operatorname{GELU}(W_mX_t^m+b_m))+p_t+e_m\right], +\quad m\in\{T,A,V\}. +\] + +随后拼接投影后的特征和三个 mask,再经过一个融合层: + +\[ +x_t^{\mathrm{fuse}}= +\phi([z_t^T;z_t^A;z_t^V;M_t^T;M_t^A;M_t^V]). +\] + +其中 \(\phi\) 是 Linear、GELU、LayerNorm 和 Dropout。这个工程实现仍属于 EarlyConcat:模态在时序主干前合并,但各自先经过独立投影,并保留 mask。 + +这里的符号 + +\[ +[\,;\,] +\] + +表示向量拼接。 + +也就是说,本来三个模态是三份信息: + +```text +Text ── T_t +Audio ── A_t +Vision ── V_t +``` + +现在把三个投影后的模态表示和 mask 放到同一个向量: + +```text +[projected Text | projected Audio | projected Vision | masks] +``` + +这就是 **Early Concatenation(早期拼接)**。 + +之所以叫“Early”,是因为: + +> 三个模态在进入主要时序模型之前就已经融合。 + +--- + +## 2.3 第二步:BiGRU 建模上下文 + +融合层处理以后,我们得到长度为 50 的序列 \(x_t^{\mathrm{fuse}}\): + +\[ +x_1^{\mathrm{fuse}},x_2^{\mathrm{fuse}},\dots,x_{50}^{\mathrm{fuse}}. +\] + +然后送入双向 GRU: + +\[ +H= +BiGRU(x_1^{\mathrm{fuse}},x_2^{\mathrm{fuse}},\dots,x_{50}^{\mathrm{fuse}}). +\] + +BiGRU 包含两个方向。 + +前向 GRU: + +\[ +x_1\rightarrow x_2\rightarrow\cdots\rightarrow x_{50} +\] + +负责利用前面的上下文; + +反向 GRU: + +\[ +x_{50}\rightarrow x_{49}\rightarrow\cdots\rightarrow x_1 +\] + +负责利用后面的上下文。 + +因此当前位置的表示可以同时参考: + +\[ +\text{前文信息} ++ +\text{当前位置多模态信息} ++ +\text{后文信息}. +\] + +这对局部模态缺失尤其有用。 + +例如位置 \(t\) 的 Audio 缺失了: + +```text +t-2 t-1 t t+1 t+2 +Audio ✓ × ✓ ✓ +Text ✓ ✓ ✓ ✓ +Vision ✓ ✓ ✓ ✓ +``` + +模型仍然可以利用: + +1. 当前位置的 Text; +2. 当前位置的 Vision; +3. 前后位置的上下文; +4. mask 告诉模型 Audio 当前缺失。 + +因此 EarlyConcat + BiGRU 本身就具有一定的**隐式缺失补偿能力**。 + +--- + +## 2.4 第三步:得到整条样本表示 + +BiGRU 得到: + +\[ +H=(h_1,h_2,\dots,h_{50}). +\] + +随后按照现有实现对整个序列进行聚合,得到 clip-level representation: + +\[ +h_{\mathrm{clip}}. +\] + +当前实现对 BiGRU 的位置输出做 masked mean:只平均至少有一个模态观测的位置;如果一条样本全部缺失,则保留一个位置用于产生序列级输出。 + +--- + +## 2.5 第四步:同时预测情感极性和情感强度 + +模型最终有两个任务: + +### 任务 A:情感极性分类 + +当前分类头预测三类极性: + +\[ +\text{Negative / Neutral / Positive}. +\] + +得到: + +\[ +\hat y_{\mathrm{cls}}. +\] + +### 任务 B:情感强度回归 + +预测连续情感强度: + +\[ +\hat y_{\mathrm{reg}}. +\] + +因此最终: + +\[ +h_{\mathrm{clip}} +\rightarrow +\begin{cases} +Head_{\mathrm{cls}} +\rightarrow +\hat y_{\mathrm{cls}}\\ +Head_{\mathrm{reg}} +\rightarrow +\hat y_{\mathrm{reg}} +\end{cases} +\] + +训练时采用交叉熵分类损失与强度回归损失的加权和: + +\[ +\mathcal L += +\operatorname{CE}(y_{\mathrm{cls}},\hat y_{\mathrm{cls}}) ++0.5\operatorname{SmoothL1}\left(\frac{\hat y_{\mathrm{reg}}}{3},\frac{y_{\mathrm{reg}}}{3}\right). +\] + +回归头通过 \(3\tanh(\cdot)\) 输出 \([-3,3]\) 范围的强度值。 + +--- + +## 2.6 EarlyConcat 为什么效果不错? + +它最大的优点其实是: + +> **假设非常少。** + +它没有提前规定: + +- Text 一定最重要; +- Audio 一定只是辅助; +- Vision 应该和 Text 对齐到某个共享语义空间; +- 某个模态缺失以后一定应该由另一个模态修复。 + +而是: + +\[ +\boxed{ +\text{保留三个模态的信息,把学习任务交给 BiGRU。} +} +\] + +因此在我们目前只有几千条训练样本的情况下,它是一个非常稳定的强基线。 + +--- + +# 三、方案一的问题:所有情况下都采用同一种融合方式 + +EarlyConcat 的问题也很明显。 + +无论当前数据是什么情况,它始终做: + +\[ +[T;A;V]. +\] + +但是实际情况可能是: + +### 情况 1:三个模态都很好 + +可能: + +\[ +T+A+V +\] + +一起使用最好。 + +### 情况 2:Audio 当前质量不好 + +可能: + +\[ +T+V +\] + +更好。 + +### 情况 3:Vision 缺失 + +可能: + +\[ +T+A +\] + +更合理。 + +### 情况 4:Text 本身已经非常有信息 + +可能: + +\[ +T +\] + +单独使用反而比加入噪声较大的 A/V 更合适。 + +所以我们进一步提出: + +\[ +\boxed{ +\text{为什么所有位置都必须使用同一种融合方式?} +} +\] + +这就产生了第二个方案。 + +--- + +# 四、方案二:MoFE-7 + MLP Router + +## 4.1 一句话理解 + +MoFE 的核心思想是: + +> **不再永远把 T/A/V 一股脑拼起来,而是准备 7 种不同的模态组合方案,再让 Router 根据当前位置的情况决定应该更相信哪些组合。** + +MoFE 全称: + +> **Mixture of Fusion Experts** + +中文可以称为: + +> **融合专家混合模型** + +我们当前还加入了 availability constraint,因此更完整地可以理解成: + +> **Availability-aware Mixture of Fusion Experts** + +即: + +> **可用性感知的融合专家混合模型。** + +--- + +# 五、为什么恰好是 7 个 Expert? + +我们有三个模态: + +\[ +T,\quad A,\quad V. +\] + +三个模态所有非空组合一共有: + +\[ +2^3-1=7 +\] + +种。 + +因此定义: + +\[ +\boxed{ +\mathcal E= +\{ +E_T,E_A,E_V, +E_{TA},E_{TV},E_{AV}, +E_{TAV} +\} +} +\] + +具体来说: + +| Expert | 可以看到的信息 | +|---|---| +| \(E_T\) | Text | +| \(E_A\) | Audio | +| \(E_V\) | Vision | +| \(E_{TA}\) | Text + Audio | +| \(E_{TV}\) | Text + Vision | +| \(E_{AV}\) | Audio + Vision | +| \(E_{TAV}\) | Text + Audio + Vision | + +这里最关键的一点是: + +> **Expert 的分工不是训练以后才希望它自己形成,而是从结构上就已经确定。** + +例如: + +\[ +E_{TA} +\] + +永远不能看 Vision。 + +而: + +\[ +E_{AV} +\] + +永远不能看 Text。 + +因此七个 Expert 天然具有不同的信息来源。 + +--- + +# 六、MoFE 的完整算法流程 + +整体流程可以概括成: + +\[ +\boxed{ +T/A/V +\rightarrow +Private Projection +\rightarrow +7\ Fusion\ Experts +\rightarrow +MLP\ Router +\rightarrow +Weighted\ Fusion +\rightarrow +Shared\ BiGRU +\rightarrow +Emotion\ Prediction +} +\] + +下面逐步解释。 + +--- + +## 6.1 第一步:三个模态分别进行私有投影 + +原始三个模态维度差异很大: + +\[ +T_t\in\mathbb R^{768}, +\] + +\[ +A_t\in\mathbb R^{74}, +\] + +\[ +V_t\in\mathbb R^{35}. +\] + +因此首先分别进行投影: + +\[ +z_t^T=P_T(T_t), +\] + +\[ +z_t^A=P_A(A_t), +\] + +\[ +z_t^V=P_V(V_t). +\] + +统一得到: + +\[ +z_t^T,z_t^A,z_t^V\in\mathbb R^{64}. +\] + +注意: + +\[ +P_T,\quad P_A,\quad P_V +\] + +是三个**独立的投影网络**。 + +这里统一到 64 维只是为了方便后面的计算,并不意味着: + +\[ +z_T=z_A=z_V. +\] + +我们没有强迫三个模态进入所谓的“共享语义空间”。 + +我们的原则仍然是: + +\[ +\boxed{ +\text{保留模态差异,需要交互时再交互。} +} +\] + +--- + +# 七、第二步:构造七个融合 Expert + +例如: + +\[ +E_T(t)=f_T(z_t^T), +\] + +\[ +E_{TA}(t) += +f_{TA}([z_t^T;z_t^A]), +\] + +\[ +E_{TV}(t) += +f_{TV}([z_t^T;z_t^V]), +\] + +\[ +E_{TAV}(t) += +f_{TAV}([z_t^T;z_t^A;z_t^V]). +\] + +其余同理。 + +所有 Expert 最后都输出: + +\[ +E_S(t)\in\mathbb R^{64}. +\] + +这里特别需要说明: + +> **我们的 7 个 Expert 不是 7 个完整的深度网络。** + +每个 Expert 只是一个很轻量的 fusion module。 + +真正参数量较大的时序模型 BiGRU 只有: + +\[ +\boxed{1\text{ 个}} +\] + +而不是 7 个。 + +所以结构是: + +```text +T ─┐ +A ─┼─> 7 个轻量 Fusion Experts +V ─┘ + │ + ▼ + Weighted Sum + │ + ▼ + 一个 Shared BiGRU +``` + +而不是: + +```text +Expert 1 -> BiGRU 1 +Expert 2 -> BiGRU 2 +... +Expert 7 -> BiGRU 7 +``` + +这对于我们的中小规模数据集非常重要。 + +--- + +# 八、第三步:Router 决定当前更应该相信哪些 Expert + +这是 MoFE 最核心的一步。 + +对于每个位置 \(t\),我们使用一个小型 MLP Router。 + +Router 会根据当前位置的信息产生 7 个分数: + +\[ +s_{t,T}, +s_{t,A}, +s_{t,V}, +s_{t,TA}, +s_{t,TV}, +s_{t,AV}, +s_{t,TAV}. +\] + +经过 Softmax 后得到: + +\[ +\alpha_{t,T}, +\alpha_{t,A}, +\alpha_{t,V}, +\alpha_{t,TA}, +\alpha_{t,TV}, +\alpha_{t,AV}, +\alpha_{t,TAV}. +\] + +并满足: + +\[ +\sum_{S\in\mathcal E} +\alpha_{t,S}=1. +\] + +只要当前位置至少有一个模态可用,这些权重就在当前可用的 experts 之间归一化;如果三个模态全缺失,则七个 expert 权重都为 0,并改用 learned missing token。 + +可以把: + +\[ +\alpha_{t,S} +\] + +理解成: + +> **当前位置模型愿意把多少融合权重分配给 Expert \(S\)。** + +--- + +# 九、Router 看什么信息? + +当前 Router 主要使用: + +1. 三个模态的 observation masks; +2. 三个 private projection 的 log-RMS 幅值; +3. 三个模态各自的局部 5-position observed ratio。 + +因此 Router 的输入为 9 个数值,MLP 结构为 \(9\rightarrow16\rightarrow7\),七个输出对应七种模态子集。 + +也就是说 Router 不只是知道: + +> “这个模态有没有。” + +它还能获得一些局部状态信息: + +> “这个模态附近是不是经常缺失?” + +> “当前 private representation 的幅值状态是什么样?” + +然后决定: + +\[ +\boxed{ +\text{当前位置更适合使用哪种模态组合。} +} +\] + +--- + +# 十、第四步:Availability-aware Routing + +我们还加入一个非常重要的约束: + +> **如果某个 Expert 所需要的模态不存在,那么这个 Expert 当前不能被选择。** + +例如: + +\[ +M_t^A=0. +\] + +说明当前位置 Audio 缺失。 + +那么: + +\[ +E_A,\quad +E_{TA},\quad +E_{AV},\quad +E_{TAV} +\] + +全部依赖 Audio,因此这些 Expert 当前不可用。 + +我们直接把对应 Router logit 屏蔽: + +\[ +s_{t,S}=-\infty. +\] + +Softmax 后自然得到: + +\[ +\alpha_{t,S}=0. +\] + +所以 Router 不需要自己慢慢学: + +> “Audio 没了以后不要用 Audio。” + +这个基本事实直接由模型结构保证。 + +Router 真正需要学习的是: + +> **在当前仍然可用的 Fusion Experts 中,应该怎样分配权重。** + +--- + +# 十一、如果三个模态全部缺失怎么办? + +如果某个位置: + +\[ +M_t^T=M_t^A=M_t^V=0, +\] + +那么 7 个 Expert 全部不可用。 + +此时模型使用一个学习得到的: + +\[ +e_{\mathrm{missing}} +\] + +作为 all-missing token。 + +也就是说模型知道: + +> “这个位置没有可靠的模态观测。” + +然后把这个信息继续交给后面的 BiGRU。 + +BiGRU 可以根据前后位置: + +\[ +t-1,\quad t+1,\quad\cdots +\] + +继续进行上下文建模。 + +--- + +# 十二、第五步:对七个 Expert 做加权融合 + +Router 得到权重后: + +\[ +u_t += +\sum_{S\in\mathcal E} +\alpha_{t,S}E_S(t). +\] + +例如某个位置可能学到: + +\[ +\alpha_T=0.50, +\] + +\[ +\alpha_{TA}=0.25, +\] + +\[ +\alpha_{TV}=0.15, +\] + +\[ +\alpha_{TAV}=0.10. +\] + +那么: + +\[ +u_t += +0.50E_T ++ +0.25E_{TA} ++ +0.15E_{TV} ++ +0.10E_{TAV}. +\] + +所以它不是硬选择: + +> “只能选一个 Expert。” + +而是: + +> **根据当前情况,动态组合多个 Expert。** + +--- + +# 十三、第六步:统一进入 Shared BiGRU + +得到: + +\[ +u_1,u_2,\dots,u_{50} +\] + +以后,当前实现把每个 \(u_t\) 与三个 observation masks 拼接,再经过 \(67\rightarrow128\) 的输入投影层;投影后的序列再统一进入: + +\[ +H= +BiGRU(u_1,u_2,\dots,u_{50}). +\] + +因此整个模型的职责划分非常清楚: + +### MoFE 负责 + +\[ +\boxed{ +\text{当前位置应该怎样组合不同模态?} +} +\] + +### BiGRU 负责 + +\[ +\boxed{ +\text{这些位置之间存在怎样的上下文关系?} +} +\] + +也就是: + +\[ +\boxed{ +\text{MoFE 做模态组合选择,BiGRU 做序列上下文建模。} +} +\] + +--- + +# 十四、第七步:完成情感预测 + +BiGRU 输出经过现有序列聚合方式后,得到 clip-level representation: + +\[ +h_{\mathrm{clip}}. +\] + +然后同时完成: + +### 情感极性分类 + +\[ +h_{\mathrm{clip}} +\rightarrow +Head_{\mathrm{cls}} +\rightarrow +\hat y_{\mathrm{cls}} +\] + +### 情感强度回归 + +\[ +h_{\mathrm{clip}} +\rightarrow +Head_{\mathrm{reg}} +\rightarrow +\hat y_{\mathrm{reg}}. +\] + +训练目标继续与 EarlyConcat 相同: + +\[ +\mathcal L +=\operatorname{CE}(y_{\mathrm{cls}},\hat y_{\mathrm{cls}}) ++0.5\operatorname{SmoothL1}\left(\frac{\hat y_{\mathrm{reg}}}{3},\frac{y_{\mathrm{reg}}}{3}\right). +\] + +--- + +# 十五、MoFE 为什么比我们之前的 MoE 更合理? + +我们之前也尝试过 MoE,但是旧 Expert 类似: + +```text +Full Expert +Text-dominant Expert +AV Expert +``` + +问题在于: + +> Expert 到底应该擅长什么,很大程度上需要模型自己学。 + +在只有几千条训练数据的情况下: + +\[ +\text{既要学 Expert specialization} ++ +\text{又要学 Router} +\] + +比较困难。 + +而现在: + +\[ +T,A,V,TA,TV,AV,TAV +\] + +七个 Expert 的分工天然存在。 + +例如: + +```text +E_T -> 永远只看 Text +E_AV -> 永远只看 Audio + Vision +E_TAV -> 永远看三个模态 +``` + +因此模型不再需要先学习: + +> “每个 Expert 到底是什么。” + +它只需要学习: + +\[ +\boxed{ +\text{当前应该如何组合这些已经定义清楚的 Expert。} +} +\] + +这大幅降低了 Router 学习 specialization 的难度。 + +--- + +# 十六、MoFE 还可以得到可解释的模态 Utility + +Router 得到七个权重以后,我们还可以计算每个模态的: + +> **Task-conditioned Modality Utility** + +例如 Text: + +\[ +R_T(t) += +\alpha_T ++ +\alpha_{TA} ++ +\alpha_{TV} ++ +\alpha_{TAV}. +\] + +Audio: + +\[ +R_A(t) += +\alpha_A ++ +\alpha_{TA} ++ +\alpha_{AV} ++ +\alpha_{TAV}. +\] + +Vision: + +\[ +R_V(t) += +\alpha_V ++ +\alpha_{TV} ++ +\alpha_{AV} ++ +\alpha_{TAV}. +\] + +它表示: + +> **Router 当前有多少权重分配给了“包含该模态”的融合路径。** + +需要注意: + +\[ +R_T,R_A,R_V +\] + +不是物理意义上的“信号可靠性”。 + +更准确地说,它们是: + +\[ +\boxed{ +\text{Task-conditioned Modality Utility} +} +\] + +即: + +> **这个模态对于当前情感预测任务有多大的使用价值。** + +--- + +# 十七、一个直观例子 + +假设三个模态都正常。 + +模型可能认为: + +```text +Text 重要 +Text + Audio 有帮助 +Text + Vision 有帮助 +TAV 少量补充 +``` + +于是: + +\[ +u_t += +0.45E_T ++ +0.25E_{TA} ++ +0.20E_{TV} ++ +0.10E_{TAV}. +\] + +如果当前位置 Text 缺失: + +```text +Text × +Audio ✓ +Vision ✓ +``` + +那么所有包含 Text 的 Expert: + +\[ +E_T,E_{TA},E_{TV},E_{TAV} +\] + +直接不可用。 + +Router 只能在: + +\[ +E_A,E_V,E_{AV} +\] + +之间重新分配: + +\[ +u_t += +\alpha_AE_A ++ +\alpha_VE_V ++ +\alpha_{AV}E_{AV}. +\] + +所以模型可以自然地根据模态缺失状态改变融合策略。 + +--- + +# 十八、两个方案最核心的区别 + +| 问题 | EarlyConcat + BiGRU | MoFE-7 + MLP Router | +|---|---|---| +| 模态如何融合 | 直接全部拼接 | 动态组合 7 种模态子集 | +| 是否区分不同融合方式 | 否 | 是 | +| 是否显式利用 mask | 是 | 是 | +| 模态缺失处理 | 交给 BiGRU 隐式学习 | Availability mask + BiGRU | +| 是否有 Router | 否 | 有 | +| Expert 数量 | 无 | 7 | +| 时序主干 | 1 个 BiGRU | 1 个 Shared BiGRU | +| 可解释性 | 一般 | 可以分析 Expert 权重和 Modality Utility | +| 复杂度 | 低 | 略高 | +| 定位 | 强基线;本次重训点估计较强 | 候选;Accuracy 略高,整体优势未确认 | + +--- + +# 十九、用一句话理解两个模型 + +## EarlyConcat + +> **“把三种模态的材料都摆在桌子上,让 BiGRU 自己读。”** + +数学上: + +\[ +\boxed{ +[T;A;V;M] +\rightarrow +BiGRU +\rightarrow +Prediction +} +\] + +--- + +## MoFE-7 + +> **“先准备 7 种不同的模态组合方案,让 Router 根据当前位置的模态状态决定更应该参考哪些组合,再交给统一的 BiGRU 建模上下文。”** + +数学上: + +\[ +\boxed{ +T/A/V +\rightarrow +7\ Fusion\ Experts +\rightarrow +MLP\ Router +\rightarrow +Weighted\ Fusion +\rightarrow +Shared\ BiGRU +\rightarrow +Prediction +} +\] + +--- + +# 二十、按数学方案 Q2 口径重训后的模型性能 + +本节替换旧的检查点重评结果,报告 2026-09-25 在 GPU 上从头重训两个模型后的结果。官方测试集只在检查点确定后进行一次干净评估;缺失鲁棒性在官方验证集的 42 个固定情景上评估。 + +## 20.1 训练与评估设置 + +- 使用官方 `aligned_50.pkl` 划分:训练 3,395 条 / 1,528 个 source video,验证 728 条 / 239 组,测试 727 条 / 381 组;三份划分的 source video 互不重叠。 +- 输入是 50 个有序 wordpiece positions,文本、音频、视觉维度为 768、74、35;不是 Q1 的 50 个物理时间 bins。 +- 两模型共享只在官方训练集观测行上拟合的 median/MAD 缩放器、训练掩码、batch 顺序、seed `20260924` 和联合目标。 +- 连续块训练缺失率为 0%、10%、30%、50%、70%,模式为 single、sync、partial、async;每个被选模态至少保留 20% 原有观测。 +- AdamW:学习率 `3e-4`、weight decay `1e-3`、batch size 64、最多 12 epochs、patience 3。两模型都在第 3 轮选中检查点。 +- checkpoint 选择使用官方验证集上 `0.0/none`、`0.3/single`、`0.3/sync`、`0.5/async` 四情景的平均联合损失。 +- 保留当前两个模型的输出头和 `CE + 0.5 × SmoothL1` 目标。数学方案 C5 使用的概率 Beta-mixture 损失与当前确定性输出头不兼容,因此没有移植该损失。 +- 测试指标为 Accuracy、Macro-F1、MAE、RMSE、Pearson。分类使用 Negative / Neutral / Positive 三类,回归预测裁剪到 \([-3,3]\)。 + +## 20.2 官方测试集结果 + +| 模型 | Accuracy ↑ | Macro-F1 ↑ | MAE ↓ | RMSE ↓ | Pearson ↑ | 参数量 | +|---|---:|---:|---:|---:|---:|---:| +| EarlyConcat + BiGRU | 0.6740 | **0.6157** | **0.6624** | **0.8897** | **0.6585** | 253,124 | +| MoFE-7 + MLP Router | **0.6795** | 0.6049 | 0.6761 | 0.8980 | 0.6427 | 306,523 | + +差值均为 MoFE-7 减 EarlyConcat,置信区间来自 1,000 次按测试 source-video ID 成对 Bootstrap: + +| 指标 | 差值 | 95% Bootstrap 区间 | +|---|---:|---:| +| Accuracy | +0.0055 | [−0.0259, +0.0339] | +| Macro-F1 | −0.0109 | [−0.0475, +0.0216] | +| MAE | +0.0137 | [−0.0065, +0.0353] | +| RMSE | +0.0083 | [−0.0150, +0.0327] | +| Pearson | −0.0159 | [−0.0323, +0.0013] | + +五项指标的区间都包含零。MoFE 的 Accuracy 点估计略高;EarlyConcat 的 Macro-F1、MAE、RMSE 和 Pearson 点估计更好。当前是单种子结果,视频组 Bootstrap 反映样本组抽样不确定性,不反映跨随机种子波动。 + +## 20.3 验证集 42 个缺失情景 + +下表中的平均值是对 41 个非 clean 验证情景不加权求平均;worst 是其中 Macro-F1 最低的情景。它们是描述性汇总,不代替 AURC。 + +| 模型 | Clean Macro-F1 | 41 个缺失情景平均 Macro-F1 | 最差情景 Macro-F1 | 缺失情景平均 MAE | +|---|---:|---:|---:|---:| +| EarlyConcat + BiGRU | **0.5746** | **0.5655** | **0.5289**(0.3/location_start_T) | **0.6371** | +| MoFE-7 + MLP Router | 0.5626 | 0.5513 | 0.5048(0.7/partial) | 0.6453 | + +30% 模态缺失的 Macro-F1 / MAE: + +| 条件 | EarlyConcat | MoFE-7 | +|---|---:|---:| +| Text | 0.5462 / 0.6386 | 0.5455 / 0.6477 | +| Audio + Vision | 0.5817 / 0.6326 | 0.5620 / 0.6350 | +| All-modal | 0.5693 / 0.6352 | 0.5617 / 0.6421 | + +归一化 AURC-MAE 越低越好。差值为 MoFE-7 减 EarlyConcat,区间按验证 source-video ID 成对 Bootstrap: + +| 缺失模式 | EarlyConcat | MoFE-7 | 差值及 95% 区间 | +|---|---:|---:|---:| +| Single | **0.6379** | 0.6494 | +0.0115 [−0.0082, +0.0302] | +| Sync | **0.6387** | 0.6461 | +0.0074 [−0.0120, +0.0250] | +| Partial | **0.6420** | 0.6527 | +0.0107 [−0.0087, +0.0285] | +| Async | **0.6409** | 0.6527 | +0.0119 [−0.0062, +0.0289] | + +四种模式下 EarlyConcat 的 AURC-MAE 点估计均较低,但区间均跨零。 + +## 20.4 结论与边界 + +这次重训的点估计整体偏向 EarlyConcat + BiGRU:测试集 Macro-F1 和回归指标更高,验证集缺失情景的平均 Macro-F1、平均 MAE 与四种 AURC-MAE 也更好;MoFE-7 仅在测试 Accuracy 上略高。由于 Bootstrap 区间均跨零,且本次只使用一个 seed,不能声称 EarlyConcat 已被统计上确认优于 MoFE。当前应将 EarlyConcat 视为本轮较强候选,MoFE-7 继续保留比较,不宣称其具有总体优势。 + +数学汇总文件中的 C5 测试结果为 Accuracy 0.6740、Macro-F1 0.5861、MAE 0.6980、RMSE 0.9674、Pearson 0.6300。C5 的概率输出头与损失函数不同,该结果只作背景参考,不是与本节两个模型的严格同结构消融比较。 + +## 20.5 可复现产物 + +运行入口: + +```bash +uv run python -m q2.train_math_protocol \ + --device auto \\ + --output-dir outputs/followups/R04_math_protocol_retraining_replica +``` + +本次报告、检查点、scaler 和逐条件结果保存在独立目录: + +- [RESULTS.md](outputs/followups/R03_math_protocol_retraining/RESULTS.md):本次结果和解读; +- [official_test_metrics_by_seed.csv](outputs/followups/R03_math_protocol_retraining/official_test_metrics_by_seed.csv):官方测试集指标; +- [official_test_paired_bootstrap.csv](outputs/followups/R03_math_protocol_retraining/official_test_paired_bootstrap.csv):测试集配对 Bootstrap; +- [controlled_metrics_by_scenario.csv](outputs/followups/R03_math_protocol_retraining/controlled_metrics_by_scenario.csv):42 个验证情景的逐项结果; +- [aurc_mae_by_mode_seed.csv](outputs/followups/R03_math_protocol_retraining/aurc_mae_by_mode_seed.csv) 与 [aurc_mae_paired_bootstrap.csv](outputs/followups/R03_math_protocol_retraining/aurc_mae_paired_bootstrap.csv):AURC 与其区间; +- [run_manifest.json](outputs/followups/R03_math_protocol_retraining/run_manifest.json):输入特征 hash、数据划分、训练配置、设备和评估规程。 + +--- + +# 二十一、目前我们对整个方法的最终理解 + +我们的模型设计经历了一个很重要的变化。 + +最开始的问题是: + +> “怎么把三个模态融合起来?” + +EarlyConcat 的答案是: + +\[ +\boxed{ +\text{直接融合,然后统一学习。} +} +\] + +进一步我们发现,更合理的问题其实是: + +> **“在不同样本、不同位置、不同模态缺失状态下,究竟应该使用哪一种模态组合?”** + +于是 MoFE 的答案变成: + +\[ +\boxed{ +\text{不是寻找唯一最优的融合方式, +而是让模型根据当前条件动态选择融合方式。} +} +\] + +因此我们最终的核心思想可以概括为: + +> **保留不同模态自身的信息结构,不预设某个模态永远最重要;将所有可能的非空模态子集定义为轻量融合专家,再通过可用性感知的 MLP Router 在局部位置动态分配专家权重,最后由共享 BiGRU 统一建模序列上下文。** + +最终流程: + +```text + Text + │ + ▼ + Private Projection + │ + ├──────────────┐ + │ │ +Audio ──> Private Projection │ + │ │ + ├──────────┐ │ + │ │ │ +Vision -> Private Projection │ │ + │ │ │ + ▼ ▼ ▼ + + ┌─────────────────────────────┐ + │ 7 Fusion Experts │ + │ │ + │ T A V TA TV AV │ + │ TAV │ + └──────────────┬──────────────┘ + │ + ▼ + MLP Router + │ + Availability Mask + │ + ▼ + Expert Weights α + │ + ▼ + Weighted Expert Mixture + │ + ▼ + Shared BiGRU + │ + ▼ + Sequence Aggregation + │ + ┌────────┴────────┐ + ▼ ▼ + Polarity Head Intensity Head + │ │ + ▼ ▼ + 情感极性分类 情感强度回归 +``` + +一句话总结: + +\[ +\boxed{ +\text{EarlyConcat 是“全部给模型看”, +MoFE 是“让模型决定当前应该怎么看”。} +} +\] diff --git a/deep_learning/Q2/EXPERIMENT_PROTOCOL.md b/deep_learning/Q2/EXPERIMENT_PROTOCOL.md new file mode 100644 index 0000000..736d132 --- /dev/null +++ b/deep_learning/Q2/EXPERIMENT_PROTOCOL.md @@ -0,0 +1,47 @@ +# Q2 后续实验协议 + +本文件只规定后续实验如何公平比较,不保存已经结束的实验数值或结论。当前维护的参照模型为 **EarlyConcat + BiGRU** 和 **MoFE-7 + MLP Router**。 + +## 固定比较条件 + +- 使用附件 2 的官方训练/验证划分和 `aligned_50.pkl`。每个样本的 50 个有序词片位置不能描述成 Q1 的 50 个物理时间箱。 +- 两个参照模型和候选模型使用相同输入特征、显式观测掩码、训练集 median/MAD 标准化,以及相同的联合极性分类和强度回归目标。 +- 训练缺失增强与验证缺失都使用连续块。验证条件包括 clean,以及 Text、Audio、Vision、Audio+Vision、All-modal 五种模式在 10%、20%、30% 缺失率下的表现。 +- 默认种子为 42、3407、2026。候选模型必须与两个参照使用相同的划分、种子和验证条件。 +- 分别报告 Accuracy、Macro-F1、MAE、Pearson;同时报告相对 clean 的变化、平均缺失表现、最差条件和跨种子均值/标准差。主对比可按来源视频 ID 做成对 bootstrap。 +- 不合并指标构造手工总分;测试标签不得用于训练、模型选择或超参数选择。 + +## 参照资产与新实验目录 + +当前保留的模型检查点和训练集标准化参数在 `outputs/mofe_7experts/`。该目录作为只读参照保存;不要把新实验结果写入其中。每一轮实验都使用唯一子目录: + +```text +outputs/followups/<编号_简短假设>/ +``` + +训练入口示例: + +```bash +uv run python -m q2.train_mofe \ + --phase full --seeds 42 3407 2026 \ + --output-dir outputs/followups/F01_local_repair +``` + +每轮只改变一个主要因素。开始训练前先记录假设、唯一改动、预期指标和停止规则;一次改变多个因素时,要拆成能够分别归因的运行。 + +## 增加任务专属 Router 的前置条件 + +在训练 dual-router 前,先运行 `uv run python -m q2.task_preference`,用保留的 single-router 检查点评估七个模态子集 expert 对分类 Macro-F1、回归 MAE 和 Pearson 的偏好。某个子集可用时强制使用该 expert;不可用时沿用已训练 router 的可用 expert 路由,并报告覆盖比例。若分类和回归的 expert 排名基本一致,或赢家不具备跨 seed 稳定性,就停止在诊断阶段,不增加第二个 Router。 + +若未来数据支持稳定的任务偏好分化,再按单因素顺序评估 dual-router:先比较 shared-expert/shared-BiGRU 主模型与 single-router;再比较参数量匹配的加宽 single-router;之后才评估小型 task adapter 或共享首层的 Router。沿用相同输入特征、mask、训练目标、三 seed 和验证条件,并报告分类及回归指标,不以单一综合分决定升级。当前 forced-expert 诊断没有显示稳定分化,详情见 `outputs/followups/D0_task_preference/task_preference_diagnostic.md`。 + +## 每轮记录 + +每个实验目录至少保存: + +- `hypothesis.md`:研究问题、单一改动和预期现象; +- `run_manifest.json`:代码版本、配置、输入特征、划分、种子、设备和检查点来源; +- `metrics_by_condition.csv`、`summary.csv` 和 `paired_bootstrap.csv`; +- 对应模型检查点、训练历史,以及解释结果所需的诊断图。 + +每轮结束时说明:候选模型是否改善平均缺失表现和最差条件;clean 表现是否下降;收益是否跨种子稳定;增加的参数量与训练成本是否值得。若收益只出现在单一缺失模式,应将它报告为该模式的专门化表现,不据此直接替换整体主模型。 diff --git a/deep_learning/Q2/README.md b/deep_learning/Q2/README.md index 4cfeaf8..268516f 100644 --- a/deep_learning/Q2/README.md +++ b/deep_learning/Q2/README.md @@ -1,39 +1,77 @@ -# Q2/Q3 algorithm selection built on Q1 alignment +# Q2:缺失模态下的多模态情感识别 -## Decision about reusing Q1 +本目录包含 Q2 的训练代码、环境配置和后续实验约定。当前只维护两种模型: -The transferable part of Q1 is its explicit time correspondence and observation mask: features from different modalities share ordered positions, missing values are accompanied by masks, and a position can be traced to source time. That interface is useful for both Q2 local-gap handling and Q3 evidence localization. +1. **EarlyConcat + BiGRU**:将文本、音频、视觉特征和观测掩码拼接后,用双向 GRU 建模有序序列,作为简洁基线。 +2. **MoFE-7 + MLP Router**:根据每个位置可用的模态,在七种模态子集专家之间路由,再用共享 BiGRU 建模序列。 -The exact Q1 B1 extraction cannot be rerun over the 4,850 Attachment 2 training examples. Attachment 2 supplies precomputed aligned and unaligned feature tensors, but not the source audio/video or CTC word-time posteriors for the full training set. Its `aligned_50.pkl` also has 50 wordpiece positions and no Q1 `time_bounds_s`; those positions must not be described as the 50 equal-duration physical-time bins exported by `final/Q1`. +旧的模型比较报告、指标表和图表已清理。不要从此 README 推断模型优劣;后续结果应放进独立实验目录并在新报告中解释。 -Accordingly, the Q2 experiment uses the official aligned feature set as its shared wordpiece axis, and compares it with a fixed equal-window pooling control made from the official unaligned audio/vision sequences. This is a downstream alignment-utility check, not a claim that B1 was recomputed on Attachment 2. The official train/validation split is retained; test labels are not used. +## 数据与时序表示 -## Q2 candidates +训练入口读取项目根目录下 `E题数据/附件2-数据集特征文件/aligned_50.pkl` 的官方训练集和验证集。每个样本包含 50 个有序位置,文本、音频、视觉维度分别为 768、74、35,并带有显式观测掩码。这里的 50 个位置是附件 2 提供的词片位置,**不是 50 个等长物理时间箱**。 -All candidates use identical training examples, train-only median/MAD scaling, joint polarity/intensity objectives, and 15 validation corruptions (three contiguous missing rates by five modality patterns). +Q1 的对齐方法为 Q2 提供了有序的跨模态输入组织方式;Q2 在此基础上处理连续块缺失,不重新提取或改写 Q1 特征。根目录 `math/` 和 `final/Q1/` 中的内容只作只读参考。 -| Candidate | Fusion rule | What it tests | -| --- | --- | --- | -| `concat` | Project each modality, concatenate features and availability flags, then run a bidirectional GRU | Strong, simple early-fusion baseline | -| `gate` | Learn per-slot modality weights, mask unavailable modalities, then run a bidirectional GRU | Whether explicit reliability-aware fusion handles local gaps | -| `crossattn` | Apply masked cross-modal attention over the 50 shared slots, then temporal pooling | Whether contextual cross-modal exchange improves robustness | +标准化参数只从训练集拟合。当前保留的两组模型权重及共享标准化参数位于: -The report keeps Macro-F1, MAE, and Pearson separate. The default selection is Macro-F1-first across local corruption conditions; MAE and Pearson remain explicit tradeoffs, not terms in a constructed total score. The selected architecture is also trained on fixed-window-resampled features as an alignment control. A separate validation control shifts audio and vision by 1–10 positions to measure sensitivity to cross-modal timing. - -## Q3 explanation selection - -The selected Q2 model is frozen. Integrated Gradients and five-slot grouped occlusion are compared on held-out Attachment 2 validation clips using deletion comprehensiveness, sufficiency, and local rank stability. Attachment 4 has original videos and transcripts, so B1's CTC hard word-time procedure can be applied to those 20 clips to map high-importance wordpiece positions back to seconds. The saved Attachment 4 pickle files do not include `time_bounds_s`; explanations therefore retain both the model slot and the CTC-derived word interval, with alignment quality recorded. - -## Run - -The project environment is managed by `uv` and installs the CUDA 13.0 PyTorch build: - -```bash -cd deep_learning/Q2 -uv sync -uv run python -m q2.train_compare -cd ../Q3 -uv run --project ../Q2 python -m q3.explain_selection +```text +outputs/mofe_7experts/ +├── aligned_robust_stats.npz +└── models/ + ├── baselines/concat/seed_{42,3407,2026}/model_best.pt + └── B5_mofe_mlp/seed_{42,3407,2026}/model_best.pt ``` -The main outputs are written to `outputs/algorithm_selection/`; plots, CSV metrics, run metadata, and checkpoints stay under this directory. The source data, `math`, and `final/Q1` are read-only inputs. +## 环境 + +本项目使用 `uv` 管理 Python 环境。进入本目录后同步锁定依赖: + +```bash +uv sync --locked +``` + +## 运行 + +先用单个种子执行快速检查。每轮运行使用新的目录,避免覆盖保留的参照权重: + +```bash +uv run python -m q2.train_mofe \ + --phase smoke --seeds 42 \ + --output-dir outputs/followups/F00_smoke +``` + +完整训练和验证示例: + +```bash +uv run python -m q2.train_mofe \ + --phase full --seeds 42 3407 2026 \ + --output-dir outputs/followups/F01_local_repair +``` + +完整运行会训练两种模型,并在 clean、Text、Audio、Vision、Audio+Vision、All-modal 条件下评估 10%、20%、30% 连续块缺失;结果、检查点和诊断图写入指定目录。默认输出目录是 `outputs/mofe_7experts/`,日常新实验应显式设置 `--output-dir`,避免覆盖保留的权重和标准化参数。 + +## 文件索引 + +- `q2/data.py`:官方特征读取、观测掩码、训练集 robust scaling 和连续块缺失。 +- `q2/models.py`:EarlyConcat + BiGRU。 +- `q2/mofe.py`:MoFE-7 专家与 MLP 路由器。 +- `q2/train_mofe.py`:两种保留模型的训练、验证、统计和可视化入口。 +- `q2/task_preference.py`:复用现有 MoFE 权重,比较七个 forced expert 的分类/回归偏好。 +- [ALGORITHM.md](ALGORITHM.md):两个保留模型的算法说明和最新验证集重评结果。 +- [EXPERIMENT_PROTOCOL.md](EXPERIMENT_PROTOCOL.md):后续实验的固定比较条件与记录要求。 +- `outputs/followups/README.md`:新实验目录的命名和存放规则。 + +Q3 暂不在本目录中开展;待 Q2 后续选型完成后再统一规划。 + +## 当前任务偏好诊断 + +在训练 dual-router 前,先用现有 MoFE-7 检查了七个 forced expert 在分类 Macro-F1 与回归 MAE/Pearson 上的偏好。三个 seed、15 种缺失条件的排名大体一致,没有看到稳定的分类—回归 expert 分工;因此当前不启动 dual-router,仍以 single-router MoFE-7 为活动参照。详见[诊断报告](outputs/followups/D0_task_preference/task_preference_diagnostic.md)及逐条件数据。 + +可用以下命令复现该诊断: + +```bash +uv run python -m q2.task_preference \ + --seeds 42 3407 2026 \ + --output-dir outputs/followups/D0_task_preference +``` diff --git a/deep_learning/Q2/RESULTS.md b/deep_learning/Q2/RESULTS.md deleted file mode 100644 index 3d3ea8b..0000000 --- a/deep_learning/Q2/RESULTS.md +++ /dev/null @@ -1,54 +0,0 @@ -# Q2 algorithm selection results - -## Q1 alignment transfer decision - -Q1 B1 aligns BERT word features and audio/vision observations with hard CTC word intervals, projects observed features onto a 0.1-second common grid, exports 50 equal-duration physical-time bins, and keeps observation masks. For Q2, the shared ordered axis and explicit masks transfer directly: a local gap stays a local gap after alignment and can be represented without inventing feature values. - -The exact B1 extraction was not recomputed over Attachment 2. The official 4,850-row feature package contains precomputed aligned and unaligned tensors, but no full-set source audio/video or word-time posterior. Its `aligned_50.pkl` has 50 wordpiece positions and no per-slot `time_bounds_s`; those positions are not Q1's 50 equal-duration bins. This experiment therefore trains on the official aligned features and compares them with an equal-window audio/vision resampling control. The comparison tests the value of an aligned ordered representation for the downstream Q2 task; it does not claim to reproduce B1 on all 4,850 clips. - -## Data and protocol - -- Attachment 2 official split: 3,395 training clips and 728 validation clips. Their source-video ID sets do not overlap. -- Each official aligned example has 50 positions with Text 768-D, Audio 74-D, Vision 35-D features and modality observation masks. -- Attachment 2 test labels were not used. -- The three candidates shared train-only median/MAD normalization, the joint polarity/intensity objective, and training-time contiguous block masking. -- Validation corruption covered 10%, 20%, and 30% of 50 positions for Text, Audio, Vision, Audio+Vision, and all three modalities. This is a wordpiece-position proxy for a continuous time gap; full-set second-level timestamps are not supplied. -- Each candidate was run with seeds 42, 3407, and 2026. Reported `±` values are seed standard deviations over the fixed official validation set and deterministic corruption draws; they are not confidence intervals over new videos. - -## Fusion comparison - -| Model | Clean Accuracy | Clean Macro-F1 | Corrupt Accuracy, mean | Corrupt Macro-F1, mean | Worst condition Macro-F1 | Corrupt MAE | Corrupt Pearson | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| Early concatenation + BiGRU | 0.626 ± 0.013 | 0.580 ± 0.018 | 0.623 ± 0.011 | **0.575 ± 0.017** | **0.516** | **0.640 ± 0.004** | 0.607 ± 0.004 | -| Reliability gate + BiGRU | 0.621 ± 0.011 | 0.570 ± 0.019 | 0.617 ± 0.011 | 0.565 ± 0.018 | 0.498 | 0.643 ± 0.008 | **0.610 ± 0.008** | -| Masked cross-modal attention | 0.604 ± 0.012 | 0.540 ± 0.049 | 0.601 ± 0.009 | 0.539 ± 0.047 | 0.462 | 0.649 ± 0.024 | 0.592 ± 0.016 | - -Early concatenation has the best mean corrupted Macro-F1 and MAE. The gate has slightly higher Pearson, so the metrics do not collapse to one score. Cross-modal attention is lower and more variable at this sample size. It is not selected for the next Q2 stage. - -For the selected concatenation model, the hardest tested case is 30% Text masking: Macro-F1 0.547 and MAE 0.665, compared with clean Macro-F1 0.580 and MAE 0.636. Audio-only or Vision-only masking has a smaller effect in these runs. This is evidence about this feature set and these simulated spans; it does not establish a universal modality ranking. - -## Alignment utility control - -The same concatenation model was trained either on the supplied aligned wordpiece features or on equal-window-resampled audio/vision features from the official unaligned tensors. - -| Representation | Clean Macro-F1 | Corrupt Macro-F1 | Corrupt MAE | Corrupt Pearson | -| --- | ---: | ---: | ---: | ---: | -| Supplied word-aligned 50 positions | 0.580 ± 0.018 | **0.575 ± 0.017** | **0.640 ± 0.004** | **0.607 ± 0.004** | -| Equal-window resampled unaligned input | 0.501 ± 0.014 | 0.504 ± 0.016 | 0.665 ± 0.005 | 0.577 ± 0.010 | - -On the selected model, shifting Audio and Vision by 1–10 positions changed aligned Macro-F1 from 0.580 to 0.557. That is a modest timing-sensitivity signal; it does not prove the model uses precise physical-time correspondence. Together with the fixed-window comparison, the result supports retaining the supplied aligned sequence for Q2. - -## Selected Q2 direction - -Continue with mask-aware early concatenation plus a bidirectional GRU, using local block masking during training. Keep the reliability gate as an ablation because its Pearson is slightly higher. Revisit cross-attention only if a later run has stronger evidence and enough data to control overfitting. - -## Reproducible artifacts - -- [Model and representation summary](outputs/algorithm_selection/summary.csv) -- [Metrics by missing type and rate](outputs/algorithm_selection/validation_metrics_by_condition.csv) -- [Aligned versus fixed-window and temporal-shift controls](outputs/algorithm_selection/alignment_transfer_ablation.csv) -- [Training/data audit and run manifest](outputs/algorithm_selection/data_audit.json), [run manifest](outputs/algorithm_selection/run_manifest.json) -- [Validation plot](outputs/algorithm_selection/missing_rate_comparison.png) -- [Selected seed-42 checkpoint](outputs/algorithm_selection/models/aligned/concat/model_best.pt) - -The fitted checkpoint is for algorithm selection, not the final Attachment 3 submission model. The final model should be trained on train+validation after the architecture and thresholds are frozen. diff --git a/deep_learning/Q2/outputs/algorithm_selection/alignment_transfer_ablation.csv b/deep_learning/Q2/outputs/algorithm_selection/alignment_transfer_ablation.csv deleted file mode 100644 index 2697f09..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/alignment_transfer_ablation.csv +++ /dev/null @@ -1,7 +0,0 @@ -method,representation,condition,n_valid,n_seeds,accuracy,accuracy_sd,macro_f1,macro_f1_sd,mae,mae_sd,pearson,pearson_sd,missing_rate -concat,provided_word_aligned_50,clean,728,3,0.6259157509157509,0.012689016905266455,0.5803030257575642,0.01849482950112713,0.6362011035283407,0.0035474183737517766,0.6131094378711319,0.003049322677391937, -concat,provided_word_aligned_50,audio_vision_shifted_1_to_10_slots,728,3,0.6144688644688645,0.017174907814570563,0.5566710058858901,0.02417454695632849,0.6264231006304423,0.003071737263575265,0.610966440919508,0.00045089311901344093, -concat,provided_word_aligned_50,all_local_corruption_mean,728,3,0.6226800976800977,0.011084117339613314,0.5751264735191365,0.016846851540436875,0.6402280900213454,0.004374268275822229,0.6068152054284395,0.004485336212715804,0.20000000000000004 -concat,equal_window_resampled_unaligned,clean,728,3,0.586996336996337,0.010491244722884272,0.5005019574320758,0.013925639871727626,0.6615431904792786,0.005397772426619935,0.581966026863303,0.009642037378255953, -concat,equal_window_resampled_unaligned,audio_vision_shifted_1_to_10_slots,728,3,0.5956959706959707,0.010309826235528995,0.5144238712048543,0.010932166832616294,0.6616438627243042,0.005509720037298075,0.5807893064362651,0.010274359593119802, -concat,equal_window_resampled_unaligned,all_local_corruption_mean,728,3,0.5905677655677655,0.006480515023060099,0.5042920441199125,0.015580461355158729,0.6651763810051813,0.004962154081458186,0.5766306314815771,0.010310184996660417,0.20000000000000004 diff --git a/deep_learning/Q2/outputs/algorithm_selection/data_audit.json b/deep_learning/Q2/outputs/algorithm_selection/data_audit.json deleted file mode 100644 index 0ff18e7..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/data_audit.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "source": "/home/gloamxun/modeling_zhaocui/E题数据/附件2-数据集特征文件/aligned_50.pkl", - "train_samples": 3395, - "valid_samples": 728, - "train_classes": [ - 967, - 758, - 1670 - ], - "valid_classes": [ - 206, - 184, - 338 - ], - "mean_observed_slots": { - "text": 24.645655375552284, - "audio": 22.626509572901327, - "vision": 21.394108983799704 - }, - "train_valid_video_overlap": 0 -} \ No newline at end of file diff --git a/deep_learning/Q2/outputs/algorithm_selection/missing_rate_comparison.png b/deep_learning/Q2/outputs/algorithm_selection/missing_rate_comparison.png deleted file mode 100644 index c80256d..0000000 Binary files a/deep_learning/Q2/outputs/algorithm_selection/missing_rate_comparison.png and /dev/null differ diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/seed_2026/training_history.csv b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/seed_2026/training_history.csv deleted file mode 100644 index 5915a17..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/seed_2026/training_history.csv +++ /dev/null @@ -1,10 +0,0 @@ -epoch,train_loss,valid_clean_loss -1.0,1.05146148469713,0.9808105859127674 -2.0,0.8792424190927435,0.8854341854105939 -3.0,0.7658277087741427,0.8630200951963991 -4.0,0.7162007325225406,0.8734401222113725 -5.0,0.666606965440291,0.8838760400866414 -6.0,0.6495787705536242,0.9118019499621548 -7.0,0.5989178496378439,0.9559067448416909 -8.0,0.5539200300419772,0.9710462656649914 -9.0,0.5062780554095904,1.0261535592131563 diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/seed_3407/training_history.csv b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/seed_3407/training_history.csv deleted file mode 100644 index 1d6be0c..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/seed_3407/training_history.csv +++ /dev/null @@ -1,11 +0,0 @@ -epoch,train_loss,valid_clean_loss -1.0,1.0488198929362826,0.9925541471649002 -2.0,0.877586845446516,0.9104853272438049 -3.0,0.7854031710712998,0.8691042694416675 -4.0,0.7275559962899597,0.8561014971890293 -5.0,0.6785599307881461,0.8777891502275572 -6.0,0.6318395165381608,0.9196370329175677 -7.0,0.5972505140083807,0.9164495874237228 -8.0,0.5567877353341492,0.9559657193802216 -9.0,0.5066572507774388,1.0156079124618362 -10.0,0.47001609758094504,1.059160087134812 diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/seed_42/training_history.csv b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/seed_42/training_history.csv deleted file mode 100644 index 4878fc9..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/seed_42/training_history.csv +++ /dev/null @@ -1,11 +0,0 @@ -epoch,train_loss,valid_clean_loss -1.0,1.0536950241636347,0.9706045127176977 -2.0,0.8524193581607606,0.8777912927197886 -3.0,0.760877827251399,0.8636277507949661 -4.0,0.7150518761740791,0.8624753559028709 -5.0,0.6774439651657034,0.877832626248454 -6.0,0.6516634187212696,0.8901654671836685 -7.0,0.5939983526865641,0.918816069325248 -8.0,0.5618191918841114,0.9479289251369435 -9.0,0.513365975132695,1.013728333043528 -10.0,0.4933904481154901,1.030043561379988 diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/training_history.csv b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/training_history.csv deleted file mode 100644 index 4878fc9..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/training_history.csv +++ /dev/null @@ -1,11 +0,0 @@ -epoch,train_loss,valid_clean_loss -1.0,1.0536950241636347,0.9706045127176977 -2.0,0.8524193581607606,0.8777912927197886 -3.0,0.760877827251399,0.8636277507949661 -4.0,0.7150518761740791,0.8624753559028709 -5.0,0.6774439651657034,0.877832626248454 -6.0,0.6516634187212696,0.8901654671836685 -7.0,0.5939983526865641,0.918816069325248 -8.0,0.5618191918841114,0.9479289251369435 -9.0,0.513365975132695,1.013728333043528 -10.0,0.4933904481154901,1.030043561379988 diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/model_best.pt b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/model_best.pt deleted file mode 100644 index 2e60a06..0000000 Binary files a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/model_best.pt and /dev/null differ diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_2026/model_best.pt b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_2026/model_best.pt deleted file mode 100644 index b3f963d..0000000 Binary files a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_2026/model_best.pt and /dev/null differ diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_2026/training_history.csv b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_2026/training_history.csv deleted file mode 100644 index dc054e4..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_2026/training_history.csv +++ /dev/null @@ -1,10 +0,0 @@ -epoch,train_loss,valid_clean_loss -1.0,1.0184006403993677,0.9758902355864808 -2.0,0.8721449165432541,0.9131481346193251 -3.0,0.7584562389938919,0.9061737309445391 -4.0,0.70247816046079,0.908711409830785 -5.0,0.6474666976266437,0.9444795060943771 -6.0,0.6255715103061111,0.9916293214965652 -7.0,0.5631065650118722,1.0601372142414471 -8.0,0.5009032366452394,1.1017653536010574 -9.0,0.4493949540235378,1.177744794677902 diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_3407/model_best.pt b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_3407/model_best.pt deleted file mode 100644 index 3df6a29..0000000 Binary files a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_3407/model_best.pt and /dev/null differ diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_3407/training_history.csv b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_3407/training_history.csv deleted file mode 100644 index 805183c..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_3407/training_history.csv +++ /dev/null @@ -1,11 +0,0 @@ -epoch,train_loss,valid_clean_loss -1.0,1.0344635292335793,0.9769261263229034 -2.0,0.8467512428760529,0.8954217656628116 -3.0,0.7528229709024783,0.8857204809293642 -4.0,0.7020640406343672,0.8774335417118702 -5.0,0.6610019422239728,0.9038714190105815 -6.0,0.6052676102629414,0.9758975283130185 -7.0,0.5659312236088293,0.9750687735421317 -8.0,0.5310967906757638,1.049424912903335 -9.0,0.46808629096658144,1.1149894405197311 -10.0,0.4314451297676122,1.177863896548093 diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_42/model_best.pt b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_42/model_best.pt deleted file mode 100644 index 2e60a06..0000000 Binary files a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_42/model_best.pt and /dev/null differ diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_42/training_history.csv b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_42/training_history.csv deleted file mode 100644 index f988f88..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/seed_42/training_history.csv +++ /dev/null @@ -1,9 +0,0 @@ -epoch,train_loss,valid_clean_loss -1.0,1.011995311136599,0.9519830608105921 -2.0,0.8360813723670112,0.8962717344472696 -3.0,0.7534678158936677,0.8991011263249995 -4.0,0.7069296527791906,0.9073571071519957 -5.0,0.6639300579274142,0.9432451947704776 -6.0,0.6353109103661997,0.9912105065125686 -7.0,0.5673890513954339,1.0443138106838687 -8.0,0.5254902547156369,1.1027433977022276 diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/training_history.csv b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/training_history.csv deleted file mode 100644 index 0122891..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/crossattn/training_history.csv +++ /dev/null @@ -1,9 +0,0 @@ -epoch,train_loss,valid_clean_loss -1.0,1.0119953784677718,0.9519833208440425 -2.0,0.8360813088991024,0.8962707964928596 -3.0,0.7534678666679947,0.8991013843934614 -4.0,0.7069298084135409,0.9073559026141743 -5.0,0.6639298437922089,0.9432453493495564 -6.0,0.6353108998801973,0.9912109388099922 -7.0,0.56738873405589,1.0443127351802783 -8.0,0.5254902160829968,1.1027441640476605 diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/model_best.pt b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/model_best.pt deleted file mode 100644 index 64cc6a5..0000000 Binary files a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/model_best.pt and /dev/null differ diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_2026/model_best.pt b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_2026/model_best.pt deleted file mode 100644 index 10ecdc2..0000000 Binary files a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_2026/model_best.pt and /dev/null differ diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_2026/training_history.csv b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_2026/training_history.csv deleted file mode 100644 index 239da3f..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_2026/training_history.csv +++ /dev/null @@ -1,10 +0,0 @@ -epoch,train_loss,valid_clean_loss -1.0,1.0707936783631642,1.0067818826371497 -2.0,0.9140718049473233,0.9000171115110208 -3.0,0.7863181178216581,0.8512143093151051 -4.0,0.7349886541013364,0.851207211122408 -5.0,0.6789810916891804,0.8575038864062383 -6.0,0.6610487986493994,0.8806245772393195 -7.0,0.6122590667671628,0.9165601101550427 -8.0,0.5692078007592095,0.9351011645662916 -9.0,0.5215919649711361,0.99570418714167 diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_3407/model_best.pt b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_3407/model_best.pt deleted file mode 100644 index 576c39f..0000000 Binary files a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_3407/model_best.pt and /dev/null differ diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_3407/training_history.csv b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_3407/training_history.csv deleted file mode 100644 index d7ab83b..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_3407/training_history.csv +++ /dev/null @@ -1,11 +0,0 @@ -epoch,train_loss,valid_clean_loss -1.0,1.0577489623317011,0.9879156252840063 -2.0,0.8716377052995894,0.8907375866240197 -3.0,0.7750455615697084,0.8654810419449439 -4.0,0.7179531797214791,0.8613645519529071 -5.0,0.6809909796273267,0.8796235846949148 -6.0,0.6348593281926932,0.9029495820894347 -7.0,0.6029366790144531,0.9171899249265482 -8.0,0.5716203340777645,0.9463915248493572 -9.0,0.5201758698180869,0.9928344789442125 -10.0,0.5027363620422505,1.0327370245378096 diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_42/model_best.pt b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_42/model_best.pt deleted file mode 100644 index 64cc6a5..0000000 Binary files a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_42/model_best.pt and /dev/null differ diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_42/training_history.csv b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_42/training_history.csv deleted file mode 100644 index 7d2f742..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/seed_42/training_history.csv +++ /dev/null @@ -1,10 +0,0 @@ -epoch,train_loss,valid_clean_loss -1.0,1.0575931425447818,0.9988141858970726 -2.0,0.8773076059641661,0.8911895647153749 -3.0,0.7700778461164899,0.8651458110128131 -4.0,0.7248913248380026,0.8694970201659988 -5.0,0.682613401501267,0.884684423823933 -6.0,0.659738369010113,0.9030193935383807 -7.0,0.6011747334290434,0.9310533351950593 -8.0,0.5665941779260282,0.9628150620303311 -9.0,0.5270494206084145,1.0122937671430818 diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/training_history.csv b/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/training_history.csv deleted file mode 100644 index 7d2f742..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/gate/training_history.csv +++ /dev/null @@ -1,10 +0,0 @@ -epoch,train_loss,valid_clean_loss -1.0,1.0575931425447818,0.9988141858970726 -2.0,0.8773076059641661,0.8911895647153749 -3.0,0.7700778461164899,0.8651458110128131 -4.0,0.7248913248380026,0.8694970201659988 -5.0,0.682613401501267,0.884684423823933 -6.0,0.659738369010113,0.9030193935383807 -7.0,0.6011747334290434,0.9310533351950593 -8.0,0.5665941779260282,0.9628150620303311 -9.0,0.5270494206084145,1.0122937671430818 diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/model_best.pt b/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/model_best.pt deleted file mode 100644 index 717a2a8..0000000 Binary files a/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/model_best.pt and /dev/null differ diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_2026/model_best.pt b/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_2026/model_best.pt deleted file mode 100644 index ab256f9..0000000 Binary files a/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_2026/model_best.pt and /dev/null differ diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_2026/training_history.csv b/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_2026/training_history.csv deleted file mode 100644 index 6fd41f5..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_2026/training_history.csv +++ /dev/null @@ -1,11 +0,0 @@ -epoch,train_loss,valid_clean_loss -1.0,1.0648082616152588,1.0257792996836232 -2.0,0.9508535012050912,0.9455079901349414 -3.0,0.83619585191762,0.9239543119629661 -4.0,0.7835445602734884,0.9075145099189256 -5.0,0.7335039586932571,0.9146714800006741 -6.0,0.7189743309109299,0.9153845460860284 -7.0,0.669045564201143,0.9579015452783186 -8.0,0.6221646765867869,0.9763828352257445 -9.0,0.5898675388760037,1.0184288430999924 -10.0,0.5511867072847154,1.0475065275862976 diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_3407/model_best.pt b/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_3407/model_best.pt deleted file mode 100644 index 04ff6e5..0000000 Binary files a/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_3407/model_best.pt and /dev/null differ diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_3407/training_history.csv b/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_3407/training_history.csv deleted file mode 100644 index 3ce362f..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_3407/training_history.csv +++ /dev/null @@ -1,12 +0,0 @@ -epoch,train_loss,valid_clean_loss -1.0,1.0638797470816859,1.028256350821191 -2.0,0.9404247038894229,0.9602446667440645 -3.0,0.8558459458527742,0.9207163734750433 -4.0,0.7991754125665735,0.9120825791096949 -5.0,0.7538611182460079,0.9113014386250422 -6.0,0.7000082863701714,0.9467641167588287 -7.0,0.6632694422646805,0.9526280204018394 -8.0,0.6257995438796503,0.9969304380836067 -9.0,0.5829168972041872,1.0277221163550576 -10.0,0.556718733575609,1.0591561126184987 -11.0,0.5129955758651098,1.1357925462198781 diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_42/model_best.pt b/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_42/model_best.pt deleted file mode 100644 index 717a2a8..0000000 Binary files a/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_42/model_best.pt and /dev/null differ diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_42/training_history.csv b/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_42/training_history.csv deleted file mode 100644 index d3b2383..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/seed_42/training_history.csv +++ /dev/null @@ -1,11 +0,0 @@ -epoch,train_loss,valid_clean_loss -1.0,1.0713691667274192,1.017243931581686 -2.0,0.9250873768771136,0.9292757629038213 -3.0,0.827078006333775,0.9095677916820233 -4.0,0.7792822652392917,0.9027277290166079 -5.0,0.7457061266457593,0.9076534837156862 -6.0,0.709590431716707,0.9116529927148924 -7.0,0.670481797169756,0.9217575978446793 -8.0,0.6348734536656627,0.9528291304032881 -9.0,0.5908853731773518,0.982715639439258 -10.0,0.5694722047558537,0.9955548689915583 diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/training_history.csv b/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/training_history.csv deleted file mode 100644 index d3b2383..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/models/fixed_window/concat/training_history.csv +++ /dev/null @@ -1,11 +0,0 @@ -epoch,train_loss,valid_clean_loss -1.0,1.0713691667274192,1.017243931581686 -2.0,0.9250873768771136,0.9292757629038213 -3.0,0.827078006333775,0.9095677916820233 -4.0,0.7792822652392917,0.9027277290166079 -5.0,0.7457061266457593,0.9076534837156862 -6.0,0.709590431716707,0.9116529927148924 -7.0,0.670481797169756,0.9217575978446793 -8.0,0.6348734536656627,0.9528291304032881 -9.0,0.5908853731773518,0.982715639439258 -10.0,0.5694722047558537,0.9955548689915583 diff --git a/deep_learning/Q2/outputs/algorithm_selection/run_manifest.json b/deep_learning/Q2/outputs/algorithm_selection/run_manifest.json deleted file mode 100644 index dbd5034..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/run_manifest.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "source_feature": "/home/gloamxun/modeling_zhaocui/E题数据/附件2-数据集特征文件/aligned_50.pkl", - "source_sha256": "66e867aa74bc70a844e806e5571e371c9abb4a35f9e2887ce9b4d97ff2cb8fcd", - "device": "cuda", - "cuda_name": "NVIDIA GeForce RTX 5070 Ti", - "seeds": [ - 42, - 3407, - 2026 - ], - "epochs_max": 32, - "patience": 6, - "batch_size": 64, - "best_epochs": { - "concat_seed_42": 4, - "concat_seed_3407": 4, - "concat_seed_2026": 3, - "gate_seed_42": 3, - "gate_seed_3407": 4, - "gate_seed_2026": 3, - "crossattn_seed_42": 2, - "crossattn_seed_3407": 4, - "crossattn_seed_2026": 3, - "fixed_window_concat_seed_42": 4, - "fixed_window_concat_seed_3407": 5, - "fixed_window_concat_seed_2026": 4 - }, - "selected_macro_f1_first": "concat", - "selection_policy": "report Macro-F1, MAE, and Pearson separately; selected model maximizes mean validation Macro-F1 across 15 contiguous corruption conditions, then uses MAE and lexical model name only as tie-breaks", - "models": [ - "concat", - "gate", - "crossattn" - ], - "corruption_rates": [ - 0.1, - 0.2, - 0.3 - ], - "corruption_patterns": [ - "text", - "audio", - "vision", - "audio_vision", - "all_modalities" - ], - "feature_scaling": "training split median/MAD; fallback to standard deviation for zero-MAD dimensions", - "test_labels_used": false, - "alignment_transfer_limit": "The official aligned_50 data use a 50-slot wordpiece sequence with no per-slot seconds or stored Q1 B1 time_bounds. The fixed-window comparison is a downstream alignment control, not a re-run of Q1 B1 on the full dataset.", - "python": "3.14.7 (main, Aug 10 2026, 00:00:00) [GCC 16.1.1 20260515 (Red Hat 16.1.1-2)]", - "torch": "2.14.0+cu130", - "numpy": "2.5.3", - "created_unix": 1790237371.5417986 -} \ No newline at end of file diff --git a/deep_learning/Q2/outputs/algorithm_selection/selected_method.txt b/deep_learning/Q2/outputs/algorithm_selection/selected_method.txt deleted file mode 100644 index 8d7be13..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/selected_method.txt +++ /dev/null @@ -1 +0,0 @@ -Macro-F1-first validation selection: concat. See summary.csv for the full multi-metric tradeoff. diff --git a/deep_learning/Q2/outputs/algorithm_selection/summary.csv b/deep_learning/Q2/outputs/algorithm_selection/summary.csv deleted file mode 100644 index be92594..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/summary.csv +++ /dev/null @@ -1,5 +0,0 @@ -method,representation,n_seeds,clean_accuracy,clean_accuracy_sd,clean_macro_f1,clean_macro_f1_sd,clean_mae,clean_mae_sd,clean_pearson,clean_pearson_sd,corrupt_accuracy_mean,corrupt_accuracy_sd,corrupt_macro_f1_mean,corrupt_macro_f1_sd,corrupt_macro_f1_worst,corrupt_mae_mean,corrupt_mae_sd,corrupt_pearson_mean,corrupt_pearson_sd,f1_rate_10,accuracy_rate_10,mae_rate_10,f1_rate_20,accuracy_rate_20,mae_rate_20,f1_rate_30,accuracy_rate_30,mae_rate_30,pareto_nondominated -concat,provided_word_aligned_50,3,0.6259157509157509,0.012689016905266455,0.5803030257575643,0.01849482950112713,0.6362011035283407,0.0035474183737517766,0.6131094378711319,0.003049322677391937,0.6226800976800976,0.011084117339613314,0.5751264735191365,0.016846851540436875,0.5160231153138954,0.6402280900213454,0.004374268275822229,0.6068152054284395,0.004485336212715804,0.5774116129988989,0.6244505494505495,0.6361218094825745,0.5732483562656322,0.6214285714285714,0.6398131450017294,0.5747194512928783,0.6221611721611722,0.6447493155797323,True -gate,provided_word_aligned_50,3,0.6213369963369964,0.0106695789356511,0.570042406396691,0.0192417246427628,0.6392609675725301,0.010298306434114075,0.61843647657748,0.008153726338115605,0.6170940170940171,0.011429412792673272,0.5654484786987964,0.018209275544794713,0.4977788775985414,0.6432029167811076,0.00818900735475074,0.6097108251803484,0.008038674969140262,0.5690117947676202,0.6205128205128205,0.6386720657348633,0.5668714434894636,0.6183150183150183,0.6423242449760437,0.5604621978393048,0.6124542124542125,0.6486124396324157,True -crossattn,provided_word_aligned_50,3,0.6039377289377289,0.011682555697960702,0.5402080959251137,0.04931996722297363,0.6486262281735738,0.0247913008377115,0.5962298100136721,0.016314693282960167,0.6013125763125764,0.00904432495782142,0.5394035484113701,0.047085193081398864,0.4615384615384615,0.6492320696512858,0.023553864046032207,0.5917838426035978,0.015944508668008214,0.5437388259395578,0.6055860805860807,0.6467597643534342,0.5379114650369052,0.5998168498168498,0.6491282820701599,0.5365603542576471,0.5985347985347985,0.6518081625302632,False -concat,equal_window_resampled_unaligned,3,0.586996336996337,0.010491244722884272,0.5005019574320758,0.013925639871727626,0.6615431904792786,0.005397772426619935,0.581966026863303,0.009642037378255953,0.5905677655677655,0.006480515023060099,0.5042920441199125,0.015580461355158729,0.4641572706698656,0.6651763810051813,0.004962154081458186,0.5766306314815771,0.010310184996660417,0.5042141077437426,0.5906593406593407,0.6621770620346069,0.503384597978282,0.5899267399267399,0.6627050677935282,0.5052774266377128,0.5911172161172161,0.6706470131874084,True diff --git a/deep_learning/Q2/outputs/algorithm_selection/validation_metrics_by_condition.csv b/deep_learning/Q2/outputs/algorithm_selection/validation_metrics_by_condition.csv deleted file mode 100644 index bc7504f..0000000 --- a/deep_learning/Q2/outputs/algorithm_selection/validation_metrics_by_condition.csv +++ /dev/null @@ -1,199 +0,0 @@ -method,representation,seed,condition,missing_rate,n_valid,accuracy,macro_f1,mae,pearson -concat,provided_word_aligned_50,42,clean,0.0,728,0.6332417582417582,0.587941053090477,0.6350555419921875,0.6157338809096098 -concat,provided_word_aligned_50,42,text,0.1,728,0.6263736263736264,0.5758300795065502,0.6362224221229553,0.6099001459533547 -concat,provided_word_aligned_50,42,audio,0.1,728,0.6318681318681318,0.5859694579820366,0.6346138119697571,0.6186827904447096 -concat,provided_word_aligned_50,42,vision,0.1,728,0.6277472527472527,0.5825072763635762,0.6360324621200562,0.6155412177622099 -concat,provided_word_aligned_50,42,audio_vision,0.1,728,0.635989010989011,0.5919984665799197,0.6340617537498474,0.6171314707027704 -concat,provided_word_aligned_50,42,all_modalities,0.1,728,0.625,0.5767997019394343,0.6347602009773254,0.6127068856220025 -concat,provided_word_aligned_50,42,text,0.2,728,0.6112637362637363,0.5533514537785528,0.6472386121749878,0.5878231360637466 -concat,provided_word_aligned_50,42,audio,0.2,728,0.6346153846153846,0.5913737928222076,0.6337395906448364,0.6167897438460014 -concat,provided_word_aligned_50,42,vision,0.2,728,0.6277472527472527,0.5807552037464527,0.6369960308074951,0.6152903324501466 -concat,provided_word_aligned_50,42,audio_vision,0.2,728,0.6346153846153846,0.5889252158180606,0.6331195831298828,0.6187845560398619 -concat,provided_word_aligned_50,42,all_modalities,0.2,728,0.6263736263736264,0.5759393139946568,0.6418775916099548,0.6097362713984099 -concat,provided_word_aligned_50,42,text,0.3,728,0.6181318681318682,0.5669198053141907,0.6596490740776062,0.57596152605997 -concat,provided_word_aligned_50,42,audio,0.3,728,0.6414835164835165,0.5997485874560443,0.6367413997650146,0.6154119880721182 -concat,provided_word_aligned_50,42,vision,0.3,728,0.6332417582417582,0.5862230880240699,0.6370688080787659,0.61519025568542 -concat,provided_word_aligned_50,42,audio_vision,0.3,728,0.6428571428571429,0.5982133735032932,0.6347481608390808,0.6176032363345405 -concat,provided_word_aligned_50,42,all_modalities,0.3,728,0.625,0.5769720397379972,0.6341773867607117,0.6145213983185148 -concat,provided_word_aligned_50,3407,clean,0.0,728,0.6332417582417582,0.5937554950072336,0.6333680152893066,0.6138300943227009 -concat,provided_word_aligned_50,3407,text,0.1,728,0.6277472527472527,0.5848809977670061,0.6363487243652344,0.6056385975757032 -concat,provided_word_aligned_50,3407,audio,0.1,728,0.635989010989011,0.5968950441819239,0.6318458914756775,0.6158991632022357 -concat,provided_word_aligned_50,3407,vision,0.1,728,0.6332417582417582,0.5933513166880985,0.6328762769699097,0.6142965716303582 -concat,provided_word_aligned_50,3407,audio_vision,0.1,728,0.635989010989011,0.5972272747847399,0.6317789554595947,0.6131521212086648 -concat,provided_word_aligned_50,3407,all_modalities,0.1,728,0.6208791208791209,0.5757563714642701,0.6323474645614624,0.6174836749082849 -concat,provided_word_aligned_50,3407,text,0.2,728,0.6043956043956044,0.5490811521155289,0.6509508490562439,0.5895223408430469 -concat,provided_word_aligned_50,3407,audio,0.2,728,0.635989010989011,0.5977119556159886,0.6304426789283752,0.618383030295478 -concat,provided_word_aligned_50,3407,vision,0.2,728,0.6401098901098901,0.6034867151810619,0.628890335559845,0.6187934617340419 -concat,provided_word_aligned_50,3407,audio_vision,0.2,728,0.6346153846153846,0.600406263153788,0.633434534072876,0.6123211947292676 -concat,provided_word_aligned_50,3407,all_modalities,0.2,728,0.6153846153846154,0.5731546231546232,0.6435555815696716,0.6001160890533341 -concat,provided_word_aligned_50,3407,text,0.3,728,0.6071428571428571,0.5572172310238351,0.6613345146179199,0.5663984666907937 -concat,provided_word_aligned_50,3407,audio,0.3,728,0.6401098901098901,0.602821099359954,0.6328924298286438,0.6157652973695024 -concat,provided_word_aligned_50,3407,vision,0.3,728,0.6277472527472527,0.5882423141524775,0.6326207518577576,0.6153119550822198 -concat,provided_word_aligned_50,3407,audio_vision,0.3,728,0.6428571428571429,0.6062757576614914,0.6341086030006409,0.6170348507761535 -concat,provided_word_aligned_50,3407,all_modalities,0.3,728,0.6277472527472527,0.5839727159969967,0.646858811378479,0.5966126328673234 -concat,provided_word_aligned_50,2026,clean,0.0,728,0.6112637362637363,0.5592125291749822,0.6401797533035278,0.6097643383810852 -concat,provided_word_aligned_50,2026,text,0.1,728,0.6098901098901099,0.5538253146595995,0.6407675743103027,0.6035210214254456 -concat,provided_word_aligned_50,2026,audio,0.1,728,0.614010989010989,0.5630983962746592,0.6385665535926819,0.6110860092543575 -concat,provided_word_aligned_50,2026,vision,0.1,728,0.6181318681318682,0.5661140652625988,0.6405577659606934,0.6110400577216631 -concat,provided_word_aligned_50,2026,audio_vision,0.1,728,0.6098901098901099,0.5587567567850849,0.6381099224090576,0.6107705749941461 -concat,provided_word_aligned_50,2026,all_modalities,0.1,728,0.614010989010989,0.5581636747439855,0.6429373621940613,0.6088956569868328 -concat,provided_word_aligned_50,2026,text,0.2,728,0.6043956043956044,0.5429151983962918,0.6489536762237549,0.5886848356465891 -concat,provided_word_aligned_50,2026,audio,0.2,728,0.6112637362637363,0.5603026186889267,0.636631965637207,0.6105368566655438 -concat,provided_word_aligned_50,2026,vision,0.2,728,0.614010989010989,0.5607756511971072,0.6439995765686035,0.6079712044449676 -concat,provided_word_aligned_50,2026,audio_vision,0.2,728,0.6181318681318682,0.5696462960623787,0.6381081342697144,0.6114528702793328 -concat,provided_word_aligned_50,2026,all_modalities,0.2,728,0.6085164835164835,0.5508998902588577,0.6492584347724915,0.6012091713678648 -concat,provided_word_aligned_50,2026,text,0.3,728,0.5851648351648352,0.5160231153138954,0.6731752753257751,0.5521914823230472 -concat,provided_word_aligned_50,2026,audio,0.3,728,0.6181318681318682,0.5670636517410711,0.636723518371582,0.6097718270787369 -concat,provided_word_aligned_50,2026,vision,0.3,728,0.6085164835164835,0.5560149900702367,0.6435868144035339,0.6098501902389435 -concat,provided_word_aligned_50,2026,audio_vision,0.3,728,0.6098901098901099,0.5653848813301509,0.637891948223114,0.6075333619057974 -concat,provided_word_aligned_50,2026,all_modalities,0.3,728,0.6043956043956044,0.5496991187074715,0.6696622371673584,0.5843647212263229 -gate,provided_word_aligned_50,42,clean,0.0,728,0.6332417582417582,0.5807021489645882,0.649620771408081,0.6141327125880438 -gate,provided_word_aligned_50,42,text,0.1,728,0.6291208791208791,0.5724806267179149,0.6482935547828674,0.607812656143153 -gate,provided_word_aligned_50,42,audio,0.1,728,0.6318681318681318,0.5804741658290519,0.64830082654953,0.6160345444489322 -gate,provided_word_aligned_50,42,vision,0.1,728,0.635989010989011,0.5855122033916645,0.6496425867080688,0.6131362897637801 -gate,provided_word_aligned_50,42,audio_vision,0.1,728,0.6332417582417582,0.5831283881314556,0.6489386558532715,0.6140409647711148 -gate,provided_word_aligned_50,42,all_modalities,0.1,728,0.6332417582417582,0.583341398636846,0.6480620503425598,0.6083974001811914 -gate,provided_word_aligned_50,42,text,0.2,728,0.6071428571428571,0.5464225853270284,0.6594239473342896,0.5807870469081623 -gate,provided_word_aligned_50,42,audio,0.2,728,0.6373626373626373,0.5879790951691172,0.6486459970474243,0.6137318643257155 -gate,provided_word_aligned_50,42,vision,0.2,728,0.6401098901098901,0.5906020456660217,0.6478570103645325,0.6146850468047298 -gate,provided_word_aligned_50,42,audio_vision,0.2,728,0.6428571428571429,0.5956295807308858,0.6458060145378113,0.615765199266891 -gate,provided_word_aligned_50,42,all_modalities,0.2,728,0.6222527472527473,0.5699698676231126,0.6509362459182739,0.6090527347361592 -gate,provided_word_aligned_50,42,text,0.3,728,0.5947802197802198,0.5351473922902494,0.6707870364189148,0.5698602169256346 -gate,provided_word_aligned_50,42,audio,0.3,728,0.6414835164835165,0.5932626146881034,0.6488574147224426,0.6137698439998989 -gate,provided_word_aligned_50,42,vision,0.3,728,0.635989010989011,0.5859542607988605,0.6494671106338501,0.6129307301381368 -gate,provided_word_aligned_50,42,audio_vision,0.3,728,0.6373626373626373,0.5906348593126334,0.6477416753768921,0.6122889569353058 -gate,provided_word_aligned_50,42,all_modalities,0.3,728,0.6304945054945055,0.5730354311449867,0.6414216160774231,0.6210376760306293 -gate,provided_word_aligned_50,3407,clean,0.0,728,0.6181318681318682,0.5815951114421278,0.6290252804756165,0.6278403558722048 -gate,provided_word_aligned_50,3407,text,0.1,728,0.6126373626373627,0.5710061419053845,0.6293500661849976,0.6178298909468922 -gate,provided_word_aligned_50,3407,audio,0.1,728,0.6208791208791209,0.5860650310799707,0.6279832124710083,0.6279555276295478 -gate,provided_word_aligned_50,3407,vision,0.1,728,0.6181318681318682,0.5824584906426339,0.629248321056366,0.6268484455037675 -gate,provided_word_aligned_50,3407,audio_vision,0.1,728,0.6195054945054945,0.5832702882513066,0.6273886561393738,0.6257845884321586 -gate,provided_word_aligned_50,3407,all_modalities,0.1,728,0.6098901098901099,0.5711323268553031,0.6288275122642517,0.6273843507994947 -gate,provided_word_aligned_50,3407,text,0.2,728,0.5947802197802198,0.5362174090491446,0.6508536338806152,0.5974496153645171 -gate,provided_word_aligned_50,3407,audio,0.2,728,0.6236263736263736,0.5893879935672744,0.6285275220870972,0.6276866860061261 -gate,provided_word_aligned_50,3407,vision,0.2,728,0.6126373626373627,0.5806822211441783,0.6233463287353516,0.6299624942730432 -gate,provided_word_aligned_50,3407,audio_vision,0.2,728,0.6236263736263736,0.5941062537491897,0.6226418614387512,0.631152312923411 -gate,provided_word_aligned_50,3407,all_modalities,0.2,728,0.6085164835164835,0.5685926111566246,0.6388359069824219,0.6172122010198969 -gate,provided_word_aligned_50,3407,text,0.3,728,0.5892857142857143,0.5319135217224389,0.6806973814964294,0.5598124161849717 -gate,provided_word_aligned_50,3407,audio,0.3,728,0.614010989010989,0.5787631246271929,0.6292544603347778,0.6264481106992486 -gate,provided_word_aligned_50,3407,vision,0.3,728,0.6112637362637363,0.5785955746513064,0.6262590885162354,0.6269980353567415 -gate,provided_word_aligned_50,3407,audio_vision,0.3,728,0.614010989010989,0.5854704803100748,0.6231685280799866,0.627471635956635 -gate,provided_word_aligned_50,3407,all_modalities,0.3,728,0.603021978021978,0.5649390371505222,0.6471090912818909,0.6058508041636224 -gate,provided_word_aligned_50,2026,clean,0.0,728,0.6126373626373627,0.5478299587833569,0.6391368508338928,0.6133363612721914 -gate,provided_word_aligned_50,2026,text,0.1,728,0.6085164835164835,0.5374777669587797,0.6402928233146667,0.61032865906815 -gate,provided_word_aligned_50,2026,audio,0.1,728,0.614010989010989,0.5504415308771583,0.6376320719718933,0.6137133407936685 -gate,provided_word_aligned_50,2026,vision,0.1,728,0.614010989010989,0.549111309400672,0.6384209990501404,0.6116423863441457 -gate,provided_word_aligned_50,2026,audio_vision,0.1,728,0.6112637362637363,0.548206981836245,0.6367278695106506,0.6134016265067737 -gate,provided_word_aligned_50,2026,all_modalities,0.1,728,0.6153846153846154,0.5510702709999151,0.6409717798233032,0.6055737170119238 -gate,provided_word_aligned_50,2026,text,0.2,728,0.6043956043956044,0.5387500187126036,0.6619775295257568,0.5870594418667304 -gate,provided_word_aligned_50,2026,audio,0.2,728,0.614010989010989,0.5520324803225694,0.6362826824188232,0.6150266095836183 -gate,provided_word_aligned_50,2026,vision,0.2,728,0.614010989010989,0.549868355076148,0.6375797986984253,0.6131134463734892 -gate,provided_word_aligned_50,2026,audio_vision,0.2,728,0.6098901098901099,0.5462836267094535,0.6391955614089966,0.6117577661672767 -gate,provided_word_aligned_50,2026,all_modalities,0.2,728,0.6195054945054945,0.556547508338603,0.642953634262085,0.6004007468165689 -gate,provided_word_aligned_50,2026,text,0.3,728,0.5810439560439561,0.4977788775985414,0.6796172261238098,0.55000542409833 -gate,provided_word_aligned_50,2026,audio,0.3,728,0.614010989010989,0.5490560178594751,0.6381493806838989,0.6113943643034875 -gate,provided_word_aligned_50,2026,vision,0.3,728,0.614010989010989,0.5555904891120482,0.6359769105911255,0.6131701769471742 -gate,provided_word_aligned_50,2026,audio_vision,0.3,728,0.6126373626373627,0.5581596077386942,0.6396954655647278,0.6082282425073746 -gate,provided_word_aligned_50,2026,all_modalities,0.3,728,0.5934065934065934,0.5286316785844464,0.6709842085838318,0.5729928980874585 -crossattn,provided_word_aligned_50,42,clean,0.0,728,0.6043956043956044,0.48998863119202335,0.6759882569313049,0.5777291731774127 -crossattn,provided_word_aligned_50,42,text,0.1,728,0.603021978021978,0.49273388898137477,0.6812499165534973,0.5694193464166294 -crossattn,provided_word_aligned_50,42,audio,0.1,728,0.6112637362637363,0.504821770685027,0.6705242395401001,0.5815934451841169 -crossattn,provided_word_aligned_50,42,vision,0.1,728,0.6043956043956044,0.4937041426648234,0.6775157451629639,0.5754628320692556 -crossattn,provided_word_aligned_50,42,audio_vision,0.1,728,0.6112637362637363,0.5009137863541273,0.6698321104049683,0.5814767002889362 -crossattn,provided_word_aligned_50,42,all_modalities,0.1,728,0.6016483516483516,0.49390809329904145,0.6687620282173157,0.5750133906760472 -crossattn,provided_word_aligned_50,42,text,0.2,728,0.5837912087912088,0.4692083792640309,0.6945626139640808,0.5412206795833109 -crossattn,provided_word_aligned_50,42,audio,0.2,728,0.6057692307692307,0.4919398568662255,0.6648613810539246,0.5871766995932145 -crossattn,provided_word_aligned_50,42,vision,0.2,728,0.603021978021978,0.4986750074706885,0.6775456666946411,0.5770632994127847 -crossattn,provided_word_aligned_50,42,audio_vision,0.2,728,0.6112637362637363,0.506035010012657,0.6634278297424316,0.5873091839764368 -crossattn,provided_word_aligned_50,42,all_modalities,0.2,728,0.5906593406593407,0.48358818056113656,0.6767656803131104,0.5728545007104048 -crossattn,provided_word_aligned_50,42,text,0.3,728,0.5769230769230769,0.4615384615384615,0.7029426693916321,0.5265077727922097 -crossattn,provided_word_aligned_50,42,audio,0.3,728,0.6057692307692307,0.4999911736715954,0.6637417078018188,0.585929811841307 -crossattn,provided_word_aligned_50,42,vision,0.3,728,0.6002747252747253,0.488070946081782,0.6788135766983032,0.5753310885314236 -crossattn,provided_word_aligned_50,42,audio_vision,0.3,728,0.6071428571428571,0.5068731682791942,0.6635814309120178,0.5881054554275786 -crossattn,provided_word_aligned_50,42,all_modalities,0.3,728,0.592032967032967,0.48234759307110725,0.667457640171051,0.5803081998379233 -crossattn,provided_word_aligned_50,3407,clean,0.0,728,0.6153846153846154,0.5885764436484813,0.6276583671569824,0.6085564971364105 -crossattn,provided_word_aligned_50,3407,text,0.1,728,0.6112637362637363,0.5835380749854434,0.6310563683509827,0.599629174971605 -crossattn,provided_word_aligned_50,3407,audio,0.1,728,0.6222527472527473,0.5972489054492567,0.623246431350708,0.6125511992350159 -crossattn,provided_word_aligned_50,3407,vision,0.1,728,0.6098901098901099,0.583342725650418,0.6262677311897278,0.6099948694514816 -crossattn,provided_word_aligned_50,3407,audio_vision,0.1,728,0.6112637362637363,0.5867178901767943,0.623745322227478,0.6093859569553041 -crossattn,provided_word_aligned_50,3407,all_modalities,0.1,728,0.614010989010989,0.5888196778236806,0.6245514750480652,0.6117287980833325 -crossattn,provided_word_aligned_50,3407,text,0.2,728,0.6043956043956044,0.5717745964093709,0.6391856074333191,0.5848733951460722 -crossattn,provided_word_aligned_50,3407,audio,0.2,728,0.6112637362637363,0.5894389643701733,0.6257225871086121,0.6074896792266403 -crossattn,provided_word_aligned_50,3407,vision,0.2,728,0.6112637362637363,0.5851332896742695,0.6264434456825256,0.6127778365291627 -crossattn,provided_word_aligned_50,3407,audio_vision,0.2,728,0.614010989010989,0.5949110192264351,0.6187586188316345,0.6182033017472267 -crossattn,provided_word_aligned_50,3407,all_modalities,0.2,728,0.6016483516483516,0.5773507156134905,0.634911835193634,0.592528289751998 -crossattn,provided_word_aligned_50,3407,text,0.3,728,0.6002747252747253,0.5711675869572423,0.6530678868293762,0.5606507699299987 -crossattn,provided_word_aligned_50,3407,audio,0.3,728,0.614010989010989,0.5913067138879948,0.6232653856277466,0.608846448174426 -crossattn,provided_word_aligned_50,3407,vision,0.3,728,0.6071428571428571,0.5819589420485362,0.6271139979362488,0.6113427502800617 -crossattn,provided_word_aligned_50,3407,audio_vision,0.3,728,0.6098901098901099,0.5897044167286073,0.6164263486862183,0.6193569599948916 -crossattn,provided_word_aligned_50,3407,all_modalities,0.3,728,0.6181318681318682,0.5940082804540815,0.6317148804664612,0.5948538231621964 -crossattn,provided_word_aligned_50,2026,clean,0.0,728,0.592032967032967,0.5420592129348364,0.6422320604324341,0.6024037597271933 -crossattn,provided_word_aligned_50,2026,text,0.1,728,0.5975274725274725,0.5454707818440054,0.6418640613555908,0.6024112136660502 -crossattn,provided_word_aligned_50,2026,audio,0.1,728,0.5934065934065934,0.5436021768815864,0.6399821043014526,0.6049551097626092 -crossattn,provided_word_aligned_50,2026,vision,0.1,728,0.5892857142857143,0.5384039205789096,0.6415307521820068,0.6011185044495995 -crossattn,provided_word_aligned_50,2026,audio_vision,0.1,728,0.603021978021978,0.554150981347194,0.6382296681404114,0.6055650886270252 -crossattn,provided_word_aligned_50,2026,all_modalities,0.1,728,0.6002747252747253,0.5487055723716843,0.6430385112762451,0.5965045050053945 -crossattn,provided_word_aligned_50,2026,text,0.2,728,0.5906593406593407,0.5288690186841035,0.6494663953781128,0.5950657404432197 -crossattn,provided_word_aligned_50,2026,audio,0.2,728,0.5906593406593407,0.5424377950535354,0.6427614688873291,0.601656698677867 -crossattn,provided_word_aligned_50,2026,vision,0.2,728,0.5934065934065934,0.5455594737723953,0.6451690793037415,0.5999133532386576 -crossattn,provided_word_aligned_50,2026,audio_vision,0.2,728,0.5961538461538461,0.5478165327681003,0.6367671489715576,0.6098925146891432 -crossattn,provided_word_aligned_50,2026,all_modalities,0.2,728,0.5892857142857143,0.5359341358069671,0.6405748724937439,0.5950458235969649 -crossattn,provided_word_aligned_50,2026,text,0.3,728,0.5755494505494505,0.5101004599189818,0.6680996417999268,0.565804882523542 -crossattn,provided_word_aligned_50,2026,audio,0.3,728,0.603021978021978,0.5546585571147281,0.6443181037902832,0.6002440646348579 -crossattn,provided_word_aligned_50,2026,vision,0.3,728,0.5989010989010989,0.5523658108429751,0.6397479772567749,0.6069549742183601 -crossattn,provided_word_aligned_50,2026,audio_vision,0.3,728,0.5879120879120879,0.5420598732858583,0.6358648538589478,0.6059215495922518 -crossattn,provided_word_aligned_50,2026,all_modalities,0.3,728,0.5810439560439561,0.5222533299835609,0.6609663367271423,0.5802332350553648 -concat,provided_word_aligned_50,42,audio_vision_shifted_1_to_10_slots,0.0,728,0.6222527472527473,0.5650157195157686,0.6228974461555481,0.6112627581844078 -concat,provided_word_aligned_50,3407,audio_vision_shifted_1_to_10_slots,0.0,728,0.6263736263736264,0.5755677417356012,0.6278499364852905,0.6111890270289838 -concat,provided_word_aligned_50,2026,audio_vision_shifted_1_to_10_slots,0.0,728,0.5947802197802198,0.5294295564063006,0.6285219192504883,0.6104475375451327 -concat,equal_window_resampled_unaligned,42,clean,0.0,728,0.5892857142857143,0.4853955760688928,0.6553117632865906,0.587849225352828 -concat,equal_window_resampled_unaligned,42,text,0.1,728,0.5892857142857143,0.48234450535531,0.6542671918869019,0.5821794113270566 -concat,equal_window_resampled_unaligned,42,audio,0.1,728,0.5879120879120879,0.4840888292569874,0.6544017195701599,0.589859444837917 -concat,equal_window_resampled_unaligned,42,vision,0.1,728,0.5906593406593407,0.48684880290869176,0.6560104489326477,0.5873086725722289 -concat,equal_window_resampled_unaligned,42,audio_vision,0.1,728,0.5961538461538461,0.49502890457555865,0.6592482924461365,0.5895610005108267 -concat,equal_window_resampled_unaligned,42,all_modalities,0.1,728,0.5865384615384616,0.4839129480007753,0.661249577999115,0.5814457099031457 -concat,equal_window_resampled_unaligned,42,text,0.2,728,0.5741758241758241,0.4641572706698656,0.6567733883857727,0.5645263734507251 -concat,equal_window_resampled_unaligned,42,audio,0.2,728,0.5934065934065934,0.49411312080823294,0.6519993543624878,0.5919285095409706 -concat,equal_window_resampled_unaligned,42,vision,0.2,728,0.5851648351648352,0.4807564908587733,0.6555609107017517,0.5864882167667788 -concat,equal_window_resampled_unaligned,42,audio_vision,0.2,728,0.5947802197802198,0.486680557422301,0.6588707566261292,0.5950950548477731 -concat,equal_window_resampled_unaligned,42,all_modalities,0.2,728,0.5824175824175825,0.4826855628872202,0.6649248003959656,0.5778664904235934 -concat,equal_window_resampled_unaligned,42,text,0.3,728,0.5769230769230769,0.47568745986340694,0.6655409932136536,0.5549458556193816 -concat,equal_window_resampled_unaligned,42,audio,0.3,728,0.5865384615384616,0.49062463717636134,0.6570892930030823,0.5903076532449915 -concat,equal_window_resampled_unaligned,42,vision,0.3,728,0.5934065934065934,0.49023200117754157,0.6581456661224365,0.5844521656955831 -concat,equal_window_resampled_unaligned,42,audio_vision,0.3,728,0.6057692307692307,0.5036551721604455,0.6700985431671143,0.5948534685103303 -concat,equal_window_resampled_unaligned,42,all_modalities,0.3,728,0.5989010989010989,0.49385933420149347,0.6675514578819275,0.5765029580102057 -concat,equal_window_resampled_unaligned,42,audio_vision_shifted_1_to_10_slots,0.0,728,0.5961538461538461,0.5023963012985687,0.6553080081939697,0.5869747480356168 -concat,equal_window_resampled_unaligned,3407,clean,0.0,728,0.5961538461538461,0.512827084557042,0.664772629737854,0.5872103830577866 -concat,equal_window_resampled_unaligned,3407,text,0.1,728,0.6002747252747253,0.5143254275091239,0.6675248146057129,0.5790469117683982 -concat,equal_window_resampled_unaligned,3407,audio,0.1,728,0.5961538461538461,0.5141876873569328,0.6630795001983643,0.5887049023755868 -concat,equal_window_resampled_unaligned,3407,vision,0.1,728,0.592032967032967,0.5021484550452703,0.6637163758277893,0.588692970574274 -concat,equal_window_resampled_unaligned,3407,audio_vision,0.1,728,0.6002747252747253,0.5172277536075813,0.6633241176605225,0.592489640322385 -concat,equal_window_resampled_unaligned,3407,all_modalities,0.1,728,0.6043956043956044,0.5264327838402229,0.6659616231918335,0.5874171690472151 -concat,equal_window_resampled_unaligned,3407,text,0.2,728,0.5810439560439561,0.49649824913662327,0.6659781336784363,0.5747007869349695 -concat,equal_window_resampled_unaligned,3407,audio,0.2,728,0.6002747252747253,0.5210696576225557,0.662110447883606,0.5898815380715464 -concat,equal_window_resampled_unaligned,3407,vision,0.2,728,0.603021978021978,0.5134655842561412,0.6635054349899292,0.5874898942649338 -concat,equal_window_resampled_unaligned,3407,audio_vision,0.2,728,0.6043956043956044,0.5213539297376665,0.665846586227417,0.591038037009385 -concat,equal_window_resampled_unaligned,3407,all_modalities,0.2,728,0.592032967032967,0.5111232435941855,0.6659784317016602,0.5810186064947893 -concat,equal_window_resampled_unaligned,3407,text,0.3,728,0.5824175824175825,0.49292032420488613,0.6802199482917786,0.5488280644778166 -concat,equal_window_resampled_unaligned,3407,audio,0.3,728,0.603021978021978,0.5278870449050491,0.6631943583488464,0.587517131217198 -concat,equal_window_resampled_unaligned,3407,vision,0.3,728,0.6043956043956044,0.5144897492123951,0.6652408838272095,0.5849148208986014 -concat,equal_window_resampled_unaligned,3407,audio_vision,0.3,728,0.6057692307692307,0.5222120983952546,0.6709288954734802,0.5894788895188137 -concat,equal_window_resampled_unaligned,3407,all_modalities,0.3,728,0.5934065934065934,0.5118478541545217,0.6919083595275879,0.558672500491863 -concat,equal_window_resampled_unaligned,3407,audio_vision_shifted_1_to_10_slots,0.0,728,0.6057692307692307,0.5237566082920623,0.6653115153312683,0.5864640082461416 -concat,equal_window_resampled_unaligned,2026,clean,0.0,728,0.5755494505494505,0.5032832116702929,0.6645451784133911,0.5708384721792944 -concat,equal_window_resampled_unaligned,2026,text,0.1,728,0.5686813186813187,0.4953554567971044,0.6647433638572693,0.5624346756639816 -concat,equal_window_resampled_unaligned,2026,audio,0.1,728,0.592032967032967,0.524084368831769,0.6619700789451599,0.5727849021508449 -concat,equal_window_resampled_unaligned,2026,vision,0.1,728,0.5837912087912088,0.5121488634353761,0.6646548509597778,0.5731723699341335 -concat,equal_window_resampled_unaligned,2026,audio_vision,0.1,728,0.5879120879120879,0.5138457090126345,0.6634346842765808,0.5783368477304374 -concat,equal_window_resampled_unaligned,2026,all_modalities,0.1,728,0.5837912087912088,0.5112311206228025,0.6690692901611328,0.5634143872695312 -concat,equal_window_resampled_unaligned,2026,text,0.2,728,0.570054945054945,0.4938343318957674,0.6707639694213867,0.5459273537200591 -concat,equal_window_resampled_unaligned,2026,audio,0.2,728,0.5934065934065934,0.5230327607731714,0.6592981815338135,0.5755735913907287 -concat,equal_window_resampled_unaligned,2026,vision,0.2,728,0.5906593406593407,0.5211672989417647,0.6658343076705933,0.5720666083948681 -concat,equal_window_resampled_unaligned,2026,audio_vision,0.2,728,0.5989010989010989,0.5276783237584716,0.6637739539146423,0.583433626473117 -concat,equal_window_resampled_unaligned,2026,all_modalities,0.2,728,0.5851648351648352,0.5131525873114892,0.6693573594093323,0.5605391525623993 -concat,equal_window_resampled_unaligned,2026,text,0.3,728,0.5508241758241759,0.4775661528696233,0.6854801177978516,0.5184409269913169 -concat,equal_window_resampled_unaligned,2026,audio,0.3,728,0.5989010989010989,0.5333065606332371,0.6566126942634583,0.576381050540554 -concat,equal_window_resampled_unaligned,2026,vision,0.3,728,0.592032967032967,0.5197981617020956,0.6703022122383118,0.5691319241770694 -concat,equal_window_resampled_unaligned,2026,audio_vision,0.3,728,0.6016483516483516,0.5344217204682321,0.6686394214630127,0.5850381809653504 -concat,equal_window_resampled_unaligned,2026,all_modalities,0.3,728,0.5728021978021978,0.4906531284411469,0.6887523531913757,0.5344899699772931 -concat,equal_window_resampled_unaligned,2026,audio_vision_shifted_1_to_10_slots,0.0,728,0.5851648351648352,0.5171187040239319,0.6643120646476746,0.5689291630270369 diff --git a/deep_learning/Q2/outputs/followups/D0_task_preference/expert_task_preference_summary.csv b/deep_learning/Q2/outputs/followups/D0_task_preference/expert_task_preference_summary.csv new file mode 100644 index 0000000..ab3898b --- /dev/null +++ b/deep_learning/Q2/outputs/followups/D0_task_preference/expert_task_preference_summary.csv @@ -0,0 +1,9 @@ +expert,n_seeds,conditions_averaged,corrupt_macro_f1_mean,corrupt_macro_f1_seed_sd,corrupt_mae_mean,corrupt_mae_seed_sd,corrupt_pearson_mean,corrupt_pearson_seed_sd,corrupt_available_position_fraction_mean,corrupt_available_position_fraction_seed_sd +T,3,15,0.589757307588876,0.0020352263518919928,0.6427479929394192,0.011099141735671992,0.6055934960695915,0.011249087713636004,0.4713144275877211,0.0009020844190840855 +A,3,15,0.31808865690767957,0.021317873724056042,0.7786133103900484,0.02215640639085551,0.2517374183437017,0.056858036666330804,0.41234615974956085,0.0006123395125548464 +V,3,15,0.4294686868255882,0.00702363166013299,0.746937084197998,0.018509741647116298,0.3397204968144785,0.03755825627467872,0.38980709380573697,0.0007666997275958537 +TA,3,15,0.5669756392100714,0.01133309602845077,0.6594201882680256,0.01470301824398307,0.5769403121670593,0.018611300259088026,0.39344567192925345,0.001209217023352635 +TV,3,15,0.5614920808228451,0.019711265507630443,0.6670825415187411,0.023795316023615953,0.567084624145485,0.0207702286048309,0.3718882878621419,0.0013093468654614014 +AV,3,15,0.39277649494622896,0.020026357257377843,0.7993785394562615,0.035564809067982435,0.34948917366719146,0.02272610332661933,0.37169109582901,0.0008765253565872557 +TAV,3,15,0.5310708763875678,0.04085138769461332,0.7081663846969605,0.04967350356763789,0.5417887241832338,0.025764587032271177,0.353772289885415,0.0014139247968273899 +learned_router,3,15,0.5996814814915562,0.009482724136440013,0.6405956612692939,0.01886112793147847,0.6049016689353811,0.007374661467605528,0.4902148962148963,0.00031441255612441293 diff --git a/deep_learning/Q2/outputs/followups/D0_task_preference/forced_expert_metrics.csv b/deep_learning/Q2/outputs/followups/D0_task_preference/forced_expert_metrics.csv new file mode 100644 index 0000000..4e7346b --- /dev/null +++ b/deep_learning/Q2/outputs/followups/D0_task_preference/forced_expert_metrics.csv @@ -0,0 +1,385 @@ +method,seed,condition,missing_rate,expert,n_valid,available_position_fraction,accuracy,macro_f1,mae,pearson +B5_mofe_mlp,42,clean,0.0,learned_router,728,0.5117582417582418,0.6538461538461539,0.6219292861291515,0.6228764653205872,0.6143991681634768 +B5_mofe_mlp,42,clean,0.0,T,728,0.5117582678794861,0.6304945054945055,0.595530036337148,0.6439221501350403,0.6129421272609571 +B5_mofe_mlp,42,clean,0.0,A,728,0.47049450874328613,0.47527472527472525,0.298145070470697,0.7733861207962036,0.18784970953005917 +B5_mofe_mlp,42,clean,0.0,V,728,0.44483518600463867,0.5027472527472527,0.40175895555984303,0.766243577003479,0.24655662252226374 +B5_mofe_mlp,42,clean,0.0,TA,728,0.47049450874328613,0.6126373626373627,0.5778441259738553,0.6754669547080994,0.5976612786856457 +B5_mofe_mlp,42,clean,0.0,TV,728,0.44483518600463867,0.5659340659340659,0.5313464424314036,0.6711714267730713,0.5439696393509325 +B5_mofe_mlp,42,clean,0.0,AV,728,0.44483518600463867,0.39972527472527475,0.32010564554339865,0.8058619499206543,0.28534741724537543 +B5_mofe_mlp,42,clean,0.0,TAV,728,0.44483518600463867,0.5192307692307693,0.5075932022075595,0.7201082110404968,0.5295943406640352 +B5_mofe_mlp,42,text_10,0.1,learned_router,728,0.5092032967032967,0.6497252747252747,0.6156314005702952,0.6238095760345459,0.6073313719228126 +B5_mofe_mlp,42,text_10,0.1,T,728,0.46244505047798157,0.625,0.586687579750888,0.6396907567977905,0.6069353813986993 +B5_mofe_mlp,42,text_10,0.1,A,728,0.47049450874328613,0.4725274725274725,0.29847247595086945,0.7734847068786621,0.1872762163407035 +B5_mofe_mlp,42,text_10,0.1,V,728,0.44483518600463867,0.5013736263736264,0.40093402112196824,0.7681401968002319,0.23859035367845496 +B5_mofe_mlp,42,text_10,0.1,TA,728,0.4237362742424011,0.6043956043956044,0.5640906610207688,0.6682834029197693,0.5903746167194355 +B5_mofe_mlp,42,text_10,0.1,TV,728,0.40118134021759033,0.5590659340659341,0.5191159133025295,0.6736815571784973,0.5364920078295683 +B5_mofe_mlp,42,text_10,0.1,AV,728,0.44483518600463867,0.39697802197802196,0.31798635878496145,0.8078023195266724,0.280227031807917 +B5_mofe_mlp,42,text_10,0.1,TAV,728,0.40118134021759033,0.5233516483516484,0.5067270736643431,0.7274371385574341,0.5169177462509623 +B5_mofe_mlp,42,audio_10,0.1,learned_router,728,0.5117582417582418,0.6497252747252747,0.6192151480683822,0.6202160716056824,0.6147259952822673 +B5_mofe_mlp,42,audio_10,0.1,T,728,0.5117582678794861,0.6291208791208791,0.5945275158931668,0.6429776549339294,0.6132756084561103 +B5_mofe_mlp,42,audio_10,0.1,A,728,0.42186814546585083,0.5206043956043956,0.3800213409408812,0.7459056973457336,0.34520880073060467 +B5_mofe_mlp,42,audio_10,0.1,V,728,0.44483518600463867,0.5027472527472527,0.40304222030887615,0.7655854225158691,0.2472653367702658 +B5_mofe_mlp,42,audio_10,0.1,TA,728,0.42186814546585083,0.6208791208791209,0.5887443905119517,0.6653785705566406,0.6021204010545738 +B5_mofe_mlp,42,audio_10,0.1,TV,728,0.44483518600463867,0.5645604395604396,0.5312885900336469,0.6704310178756714,0.5439517824131759 +B5_mofe_mlp,42,audio_10,0.1,AV,728,0.3990384638309479,0.4642857142857143,0.3849597770559165,0.7626285552978516,0.3957500275879042 +B5_mofe_mlp,42,audio_10,0.1,TAV,728,0.3990384638309479,0.5343406593406593,0.5227392208880216,0.6968366503715515,0.5554522305845234 +B5_mofe_mlp,42,vision_10,0.1,learned_router,728,0.5117582417582418,0.6497252747252747,0.617790322593887,0.6200696229934692,0.6170344442601211 +B5_mofe_mlp,42,vision_10,0.1,T,728,0.5117582678794861,0.6291208791208791,0.5943509173547352,0.644621729850769,0.6129819496927139 +B5_mofe_mlp,42,vision_10,0.1,A,728,0.47049450874328613,0.4766483516483517,0.2989428209119208,0.7728793621063232,0.1876997393849426 +B5_mofe_mlp,42,vision_10,0.1,V,728,0.3976648449897766,0.521978021978022,0.43087711258901057,0.7396351099014282,0.3554676006672735 +B5_mofe_mlp,42,vision_10,0.1,TA,728,0.47049450874328613,0.6098901098901099,0.5742147768264858,0.6766068339347839,0.5972162162499693 +B5_mofe_mlp,42,vision_10,0.1,TV,728,0.3976648449897766,0.5865384615384616,0.55296383612865,0.6538857817649841,0.5634816451275405 +B5_mofe_mlp,42,vision_10,0.1,AV,728,0.3976648449897766,0.4697802197802198,0.3822322738135533,0.7551642656326294,0.38427666063984023 +B5_mofe_mlp,42,vision_10,0.1,TAV,728,0.3976648449897766,0.5494505494505495,0.5360678828855083,0.6838967800140381,0.5509729751679995 +B5_mofe_mlp,42,audio_vision_10,0.1,learned_router,728,0.5117582417582418,0.6524725274725275,0.6211709858768683,0.620396614074707,0.6150188131976286 +B5_mofe_mlp,42,audio_vision_10,0.1,T,728,0.5117582678794861,0.6304945054945055,0.5962642544403453,0.6440324783325195,0.6128135963865373 +B5_mofe_mlp,42,audio_vision_10,0.1,A,728,0.42197802662849426,0.5123626373626373,0.3684800611034152,0.7456803917884827,0.33199125335675117 +B5_mofe_mlp,42,audio_vision_10,0.1,V,728,0.39917582273483276,0.5288461538461539,0.4393232991026214,0.7295647859573364,0.3589534058253107 +B5_mofe_mlp,42,audio_vision_10,0.1,TA,728,0.42197802662849426,0.6071428571428571,0.5708085735541816,0.6724270582199097,0.6024972861010555 +B5_mofe_mlp,42,audio_vision_10,0.1,TV,728,0.39917582273483276,0.5824175824175825,0.5450890570908877,0.6524452567100525,0.5628685312944325 +B5_mofe_mlp,42,audio_vision_10,0.1,AV,728,0.39917582273483276,0.4807692307692308,0.3902098705265415,0.7563263773918152,0.3809815404834876 +B5_mofe_mlp,42,audio_vision_10,0.1,TAV,728,0.39917582273483276,0.5508241758241759,0.5380789099976825,0.6938035488128662,0.5469243453503756 +B5_mofe_mlp,42,all_modalities_10,0.1,learned_router,728,0.4626098901098901,0.6414835164835165,0.6107226107226107,0.6245603561401367,0.6099878332655955 +B5_mofe_mlp,42,all_modalities_10,0.1,T,728,0.4626098871231079,0.6236263736263736,0.5896180574655738,0.644334077835083,0.6061672135521097 +B5_mofe_mlp,42,all_modalities_10,0.1,A,728,0.42359891533851624,0.47802197802197804,0.29851508448322833,0.7703589200973511,0.19932722734527986 +B5_mofe_mlp,42,all_modalities_10,0.1,V,728,0.401016503572464,0.5096153846153846,0.40743192506208387,0.7642088532447815,0.2536262411288443 +B5_mofe_mlp,42,all_modalities_10,0.1,TA,728,0.42359891533851624,0.6071428571428571,0.5695040612711005,0.6758896708488464,0.5936467906721097 +B5_mofe_mlp,42,all_modalities_10,0.1,TV,728,0.401016503572464,0.5769230769230769,0.5415381488196198,0.672159731388092,0.5447658722890104 +B5_mofe_mlp,42,all_modalities_10,0.1,AV,728,0.401016503572464,0.4024725274725275,0.3202895136648401,0.8006964921951294,0.28858272134328355 +B5_mofe_mlp,42,all_modalities_10,0.1,TAV,728,0.401016503572464,0.5192307692307693,0.5060589429054257,0.7209683060646057,0.5279373251382632 +B5_mofe_mlp,42,text_20,0.2,learned_router,728,0.5065384615384615,0.6304945054945055,0.5904030390809835,0.6350097060203552,0.5945970800346047 +B5_mofe_mlp,42,text_20,0.2,T,728,0.4137362837791443,0.6277472527472527,0.5834715774578197,0.6387249231338501,0.5959559379397134 +B5_mofe_mlp,42,text_20,0.2,A,728,0.47049450874328613,0.47115384615384615,0.2975569493673755,0.7751386761665344,0.17787621670831064 +B5_mofe_mlp,42,text_20,0.2,V,728,0.44483518600463867,0.49313186813186816,0.3957467065496128,0.7714186310768127,0.22587166430826042 +B5_mofe_mlp,42,text_20,0.2,TA,728,0.377692312002182,0.6043956043956044,0.5611937927949194,0.6714186668395996,0.5678669128762814 +B5_mofe_mlp,42,text_20,0.2,TV,728,0.35653847455978394,0.5686813186813187,0.5239313100098085,0.681026816368103,0.526936547599484 +B5_mofe_mlp,42,text_20,0.2,AV,728,0.44483518600463867,0.39972527472527475,0.31990146004230513,0.8130425810813904,0.2707806717799165 +B5_mofe_mlp,42,text_20,0.2,TAV,728,0.35653847455978394,0.5192307692307693,0.49528792280719197,0.7405415773391724,0.5088571560275247 +B5_mofe_mlp,42,audio_20,0.2,learned_router,728,0.5117582417582418,0.6483516483516484,0.6152893789223305,0.6198974251747131,0.6158397838233138 +B5_mofe_mlp,42,audio_20,0.2,T,728,0.5117582678794861,0.6277472527472527,0.5934285449238298,0.6425583362579346,0.6129603628587058 +B5_mofe_mlp,42,audio_20,0.2,A,728,0.37777474522590637,0.5370879120879121,0.4088556651206101,0.721038818359375,0.408803633397513 +B5_mofe_mlp,42,audio_20,0.2,V,728,0.44483518600463867,0.5027472527472527,0.4043940692076286,0.7664347290992737,0.24638264842704208 +B5_mofe_mlp,42,audio_20,0.2,TA,728,0.37777474522590637,0.6291208791208791,0.5983548479511946,0.6565340757369995,0.6034848427820116 +B5_mofe_mlp,42,audio_20,0.2,TV,728,0.44483518600463867,0.5618131868131868,0.5278821116181079,0.6707059144973755,0.5439704521908111 +B5_mofe_mlp,42,audio_20,0.2,AV,728,0.35697802901268005,0.49175824175824173,0.4044220197597692,0.7304497957229614,0.46939738334691383 +B5_mofe_mlp,42,audio_20,0.2,TAV,728,0.35697802901268005,0.554945054945055,0.5389342277853322,0.683380126953125,0.5631088893669088 +B5_mofe_mlp,42,vision_20,0.2,learned_router,728,0.5117582417582418,0.6497252747252747,0.6161050291578356,0.6227805614471436,0.6170723967407132 +B5_mofe_mlp,42,vision_20,0.2,T,728,0.5117582678794861,0.6304945054945055,0.595665938533476,0.6456263065338135,0.612921271986715 +B5_mofe_mlp,42,vision_20,0.2,A,728,0.47049450874328613,0.4739010989010989,0.294076640863259,0.772672712802887,0.19172033154149434 +B5_mofe_mlp,42,vision_20,0.2,V,728,0.3548901081085205,0.5494505494505495,0.4673960507937731,0.7191774845123291,0.4004459126810772 +B5_mofe_mlp,42,vision_20,0.2,TA,728,0.47049450874328613,0.6071428571428571,0.5708690749466393,0.6775083541870117,0.5974902100142974 +B5_mofe_mlp,42,vision_20,0.2,TV,728,0.3548901081085205,0.6016483516483516,0.5649164229707409,0.647191047668457,0.5713132865281553 +B5_mofe_mlp,42,vision_20,0.2,AV,728,0.3548901081085205,0.5137362637362637,0.43167042702402014,0.7291663289070129,0.416186430649655 +B5_mofe_mlp,42,vision_20,0.2,TAV,728,0.3548901081085205,0.5741758241758241,0.5579426783722908,0.6710066795349121,0.5582887705143701 +B5_mofe_mlp,42,audio_vision_20,0.2,learned_router,728,0.5117582417582418,0.6456043956043956,0.6124517381483326,0.6248074173927307,0.6073068166652785 +B5_mofe_mlp,42,audio_vision_20,0.2,T,728,0.5117582678794861,0.6291208791208791,0.5952612725131757,0.6444122791290283,0.6129037391710341 +B5_mofe_mlp,42,audio_vision_20,0.2,A,728,0.3742307722568512,0.5288461538461539,0.393885105990035,0.7293428778648376,0.3882759877201428 +B5_mofe_mlp,42,audio_vision_20,0.2,V,728,0.3539285659790039,0.5494505494505495,0.4675111779446084,0.7149078249931335,0.41871289651004223 +B5_mofe_mlp,42,audio_vision_20,0.2,TA,728,0.3742307722568512,0.6195054945054945,0.5844559446078783,0.669805645942688,0.600856133553038 +B5_mofe_mlp,42,audio_vision_20,0.2,TV,728,0.3539285659790039,0.5892857142857143,0.5537996616712025,0.6531057357788086,0.5563709753247387 +B5_mofe_mlp,42,audio_vision_20,0.2,AV,728,0.3539285659790039,0.4945054945054945,0.4048075528799828,0.7323312163352966,0.42436332986946057 +B5_mofe_mlp,42,audio_vision_20,0.2,TAV,728,0.3539285659790039,0.5535714285714286,0.5374035508293326,0.6750969290733337,0.5556165001105868 +B5_mofe_mlp,42,all_modalities_20,0.2,learned_router,728,0.41145604395604396,0.6401098901098901,0.6082957983542778,0.6285881400108337,0.6113174288636467 +B5_mofe_mlp,42,all_modalities_20,0.2,T,728,0.41145604848861694,0.6112637362637363,0.5753999256800798,0.6470811367034912,0.6110403575185562 +B5_mofe_mlp,42,all_modalities_20,0.2,A,728,0.37502747774124146,0.48214285714285715,0.30971805746653613,0.7708896994590759,0.19843845587271233 +B5_mofe_mlp,42,all_modalities_20,0.2,V,728,0.3545604348182678,0.5,0.40444384179778897,0.768601655960083,0.2478532037462523 +B5_mofe_mlp,42,all_modalities_20,0.2,TA,728,0.37502747774124146,0.6126373626373627,0.581822149187397,0.6791433691978455,0.5855705717441732 +B5_mofe_mlp,42,all_modalities_20,0.2,TV,728,0.3545604348182678,0.5714285714285714,0.5350638931866368,0.6752635836601257,0.5391820306204274 +B5_mofe_mlp,42,all_modalities_20,0.2,AV,728,0.3545604348182678,0.4065934065934066,0.33361946481005195,0.7985524535179138,0.28911163355771957 +B5_mofe_mlp,42,all_modalities_20,0.2,TAV,728,0.3545604348182678,0.5137362637362637,0.5016717439879469,0.7235588431358337,0.5228808277462639 +B5_mofe_mlp,42,text_30,0.3,learned_router,728,0.5034615384615385,0.614010989010989,0.5665382052109811,0.6587778925895691,0.5712246625860047 +B5_mofe_mlp,42,text_30,0.3,T,728,0.36975276470184326,0.6002747252747253,0.5489214966247952,0.6610612273216248,0.5660733797289579 +B5_mofe_mlp,42,text_30,0.3,A,728,0.47049450874328613,0.4739010989010989,0.29743008078567407,0.776188850402832,0.16784848141204534 +B5_mofe_mlp,42,text_30,0.3,V,728,0.44483518600463867,0.489010989010989,0.3906705384925391,0.7730352282524109,0.21533838045151854 +B5_mofe_mlp,42,text_30,0.3,TA,728,0.3367857336997986,0.5769230769230769,0.5309590282919976,0.6845227479934692,0.5380879936631464 +B5_mofe_mlp,42,text_30,0.3,TV,728,0.3170604407787323,0.5563186813186813,0.5074245114833101,0.6991907358169556,0.5079542890645365 +B5_mofe_mlp,42,text_30,0.3,AV,728,0.44483518600463867,0.38873626373626374,0.3106438181055198,0.8153451681137085,0.2564790729278157 +B5_mofe_mlp,42,text_30,0.3,TAV,728,0.3170604407787323,0.510989010989011,0.48202998092174165,0.7513630390167236,0.48075890078323275 +B5_mofe_mlp,42,audio_30,0.3,learned_router,728,0.5117582417582418,0.6414835164835165,0.6099992622632019,0.6212661862373352,0.6127160322638576 +B5_mofe_mlp,42,audio_30,0.3,T,728,0.5117582678794861,0.6304945054945055,0.5962642544403453,0.6413890719413757,0.6140594602256363 +B5_mofe_mlp,42,audio_30,0.3,A,728,0.32620880007743835,0.5480769230769231,0.4494048835498772,0.7181518077850342,0.4130353082081023 +B5_mofe_mlp,42,audio_30,0.3,V,728,0.44483518600463867,0.5027472527472527,0.40552179812400446,0.7659018635749817,0.24744836243929447 +B5_mofe_mlp,42,audio_30,0.3,TA,728,0.32620880007743835,0.6291208791208791,0.5964728532922603,0.6501738429069519,0.6030688423259514 +B5_mofe_mlp,42,audio_30,0.3,TV,728,0.44483518600463867,0.5618131868131868,0.5278372031251927,0.6705804467201233,0.5438638236128189 +B5_mofe_mlp,42,audio_30,0.3,AV,728,0.3069230914115906,0.49862637362637363,0.42544608223551644,0.7236812710762024,0.4526082943905184 +B5_mofe_mlp,42,audio_30,0.3,TAV,728,0.3069230914115906,0.5673076923076923,0.5511576680296482,0.6794568300247192,0.5516785851487167 +B5_mofe_mlp,42,vision_30,0.3,learned_router,728,0.5117582417582418,0.6442307692307693,0.6100360167586674,0.6305182576179504,0.6127016976409972 +B5_mofe_mlp,42,vision_30,0.3,T,728,0.5117582678794861,0.6263736263736264,0.59178963181016,0.6464172005653381,0.6124406407087638 +B5_mofe_mlp,42,vision_30,0.3,A,728,0.47049450874328613,0.47527472527472525,0.29307320983548674,0.7725724577903748,0.18981337716798075 +B5_mofe_mlp,42,vision_30,0.3,V,728,0.31381869316101074,0.5480769230769231,0.46141777713440435,0.7053895592689514,0.4504351953464994 +B5_mofe_mlp,42,vision_30,0.3,TA,728,0.47049450874328613,0.6112637362637363,0.5755727285311548,0.6785207986831665,0.5965958774703438 +B5_mofe_mlp,42,vision_30,0.3,TV,728,0.31381869316101074,0.5947802197802198,0.5518425273811883,0.6574218273162842,0.5668784897408216 +B5_mofe_mlp,42,vision_30,0.3,AV,728,0.31381869316101074,0.5027472527472527,0.413102983693412,0.7221376299858093,0.44235786176147723 +B5_mofe_mlp,42,vision_30,0.3,TAV,728,0.31381869316101074,0.5810439560439561,0.5627987997604266,0.6714197397232056,0.5504938666999736 +B5_mofe_mlp,42,audio_vision_30,0.3,learned_router,728,0.5117582417582418,0.6538461538461539,0.6220936984876739,0.6240691542625427,0.6123047776384958 +B5_mofe_mlp,42,audio_vision_30,0.3,T,728,0.5117582678794861,0.6291208791208791,0.5956657591699687,0.6447410583496094,0.6124816718027329 +B5_mofe_mlp,42,audio_vision_30,0.3,A,728,0.3259340822696686,0.5412087912087912,0.4293427018964002,0.7119898796081543,0.46116924292994144 +B5_mofe_mlp,42,audio_vision_30,0.3,V,728,0.30848902463912964,0.5741758241758241,0.5081685315054737,0.7036085724830627,0.46865253704666515 +B5_mofe_mlp,42,audio_vision_30,0.3,TA,728,0.3259340822696686,0.6167582417582418,0.5796313859552673,0.6652615070343018,0.6066108168106552 +B5_mofe_mlp,42,audio_vision_30,0.3,TV,728,0.30848902463912964,0.6085164835164835,0.5747678972005031,0.6532444953918457,0.5685189504949397 +B5_mofe_mlp,42,audio_vision_30,0.3,AV,728,0.30848902463912964,0.5178571428571429,0.4479560935960237,0.704010009765625,0.4737609457990894 +B5_mofe_mlp,42,audio_vision_30,0.3,TAV,728,0.30848902463912964,0.5769230769230769,0.5590219753958139,0.6671831011772156,0.5613400203043469 +B5_mofe_mlp,42,all_modalities_30,0.3,learned_router,728,0.3595054945054945,0.6442307692307693,0.6106062664769548,0.6119968891143799,0.6240023475516322 +B5_mofe_mlp,42,all_modalities_30,0.3,T,728,0.3595055043697357,0.6332417582417582,0.5994013685816965,0.6420187950134277,0.6142476274070523 +B5_mofe_mlp,42,all_modalities_30,0.3,A,728,0.326043963432312,0.4876373626373626,0.32114976052878363,0.7629672288894653,0.24577611782666536 +B5_mofe_mlp,42,all_modalities_30,0.3,V,728,0.3076648414134979,0.5137362637362637,0.41564992061683226,0.7619249224662781,0.26280751782695616 +B5_mofe_mlp,42,all_modalities_30,0.3,TA,728,0.326043963432312,0.6277472527472527,0.5901208902602798,0.6717715263366699,0.6055637792894115 +B5_mofe_mlp,42,all_modalities_30,0.3,TV,728,0.3076648414134979,0.5851648351648352,0.550384612243989,0.6579670906066895,0.5550817131264711 +B5_mofe_mlp,42,all_modalities_30,0.3,AV,728,0.3076648414134979,0.43131868131868134,0.3524060719327709,0.7886312007904053,0.2950171316134991 +B5_mofe_mlp,42,all_modalities_30,0.3,TAV,728,0.3076648414134979,0.5206043956043956,0.5084554432330729,0.7108278274536133,0.5347220240307088 +B5_mofe_mlp,3407,clean,0.0,learned_router,728,0.5117582417582418,0.635989010989011,0.604973907499834,0.6324349045753479,0.6151046626873287 +B5_mofe_mlp,3407,clean,0.0,T,728,0.5117582678794861,0.6002747252747253,0.589715106019248,0.6271728873252869,0.6222406477372165 +B5_mofe_mlp,3407,clean,0.0,A,728,0.47049450874328613,0.46703296703296704,0.25617731189594245,0.8055213093757629,0.08527234954826798 +B5_mofe_mlp,3407,clean,0.0,V,728,0.44483518600463867,0.45879120879120877,0.3911083700819353,0.7868513464927673,0.2733494259442715 +B5_mofe_mlp,3407,clean,0.0,TA,728,0.47049450874328613,0.5796703296703297,0.5637105051146315,0.6420981884002686,0.5835249000576797 +B5_mofe_mlp,3407,clean,0.0,TV,728,0.44483518600463867,0.6153846153846154,0.5641846149831432,0.6430084109306335,0.5929149001529713 +B5_mofe_mlp,3407,clean,0.0,AV,728,0.44483518600463867,0.3791208791208791,0.31403029968176926,0.8854933381080627,0.27891622253907705 +B5_mofe_mlp,3407,clean,0.0,TAV,728,0.44483518600463867,0.5879120879120879,0.5681219195313041,0.6709685921669006,0.5645502331565663 +B5_mofe_mlp,3407,text_10,0.1,learned_router,728,0.509010989010989,0.6318681318681318,0.5933228635123383,0.633339524269104,0.6112815975781888 +B5_mofe_mlp,3407,text_10,0.1,T,728,0.46123626828193665,0.614010989010989,0.5962559055831377,0.6278678774833679,0.6178578184774396 +B5_mofe_mlp,3407,text_10,0.1,A,728,0.47049450874328613,0.4642857142857143,0.24969978803929094,0.8068049550056458,0.080513190196943 +B5_mofe_mlp,3407,text_10,0.1,V,728,0.44483518600463867,0.4574175824175824,0.3878406396990897,0.7909943461418152,0.2623676162284545 +B5_mofe_mlp,3407,text_10,0.1,TA,728,0.4227197766304016,0.5879120879120879,0.5646643159293967,0.6490334868431091,0.5744750721425774 +B5_mofe_mlp,3407,text_10,0.1,TV,728,0.39942309260368347,0.6085164835164835,0.5474885374714241,0.6448301672935486,0.5895230801704665 +B5_mofe_mlp,3407,text_10,0.1,AV,728,0.44483518600463867,0.37774725274725274,0.3110187465790914,0.8889135718345642,0.2696459458776994 +B5_mofe_mlp,3407,text_10,0.1,TAV,728,0.39942309260368347,0.5934065934065934,0.563693177045021,0.6679768562316895,0.5622421569025232 +B5_mofe_mlp,3407,audio_10,0.1,learned_router,728,0.5117582417582418,0.635989010989011,0.6042690508274062,0.6318331956863403,0.6144226173077325 +B5_mofe_mlp,3407,audio_10,0.1,T,728,0.5117582678794861,0.5989010989010989,0.5878169018954168,0.6271204948425293,0.622383546006487 +B5_mofe_mlp,3407,audio_10,0.1,A,728,0.422802209854126,0.5027472527472527,0.332656408166132,0.7686276435852051,0.25398532291213505 +B5_mofe_mlp,3407,audio_10,0.1,V,728,0.44483518600463867,0.45604395604395603,0.38920601362461826,0.7875474095344543,0.2726621335461414 +B5_mofe_mlp,3407,audio_10,0.1,TA,728,0.422802209854126,0.5947802197802198,0.5806242036699651,0.6359114050865173,0.5923569117539921 +B5_mofe_mlp,3407,audio_10,0.1,TV,728,0.44483518600463867,0.6167582417582418,0.5663217661957952,0.6429393291473389,0.5927652831204141 +B5_mofe_mlp,3407,audio_10,0.1,AV,728,0.3992857336997986,0.44505494505494503,0.40079404466501245,0.8398518562316895,0.37350789652686583 +B5_mofe_mlp,3407,audio_10,0.1,TAV,728,0.3992857336997986,0.5879120879120879,0.5677938460265421,0.6635177135467529,0.5747748964548253 +B5_mofe_mlp,3407,vision_10,0.1,learned_router,728,0.5117582417582418,0.6387362637362637,0.6078729015399154,0.6297808885574341,0.6157341493803992 +B5_mofe_mlp,3407,vision_10,0.1,T,728,0.5117582678794861,0.5989010989010989,0.5881686695855791,0.6271765232086182,0.6224112929540812 +B5_mofe_mlp,3407,vision_10,0.1,A,728,0.47049450874328613,0.4697802197802198,0.2617495652772756,0.8053902387619019,0.08562704392462114 +B5_mofe_mlp,3407,vision_10,0.1,V,728,0.39975276589393616,0.5164835164835165,0.4530382033974704,0.7460650205612183,0.37016954440873484 +B5_mofe_mlp,3407,vision_10,0.1,TA,728,0.47049450874328613,0.5810439560439561,0.5652874494272252,0.6420679092407227,0.5835241242239276 +B5_mofe_mlp,3407,vision_10,0.1,TV,728,0.39975276589393616,0.6208791208791209,0.5716893090185614,0.636868417263031,0.5996523024591878 +B5_mofe_mlp,3407,vision_10,0.1,AV,728,0.39975276589393616,0.4340659340659341,0.3798771393241414,0.8302300572395325,0.3660266205867799 +B5_mofe_mlp,3407,vision_10,0.1,TAV,728,0.39975276589393616,0.5975274725274725,0.5773266951804078,0.6591137051582336,0.5739229519759457 +B5_mofe_mlp,3407,audio_vision_10,0.1,learned_router,728,0.5117582417582418,0.625,0.5961079542334627,0.630139172077179,0.614822589976647 +B5_mofe_mlp,3407,audio_vision_10,0.1,T,728,0.5117582678794861,0.6002747252747253,0.5895426241854899,0.6275323629379272,0.6220930103186073 +B5_mofe_mlp,3407,audio_vision_10,0.1,A,728,0.42197802662849426,0.4793956043956044,0.2964038513186514,0.7789897918701172,0.23988959094939177 +B5_mofe_mlp,3407,audio_vision_10,0.1,V,728,0.39879122376441956,0.4945054945054945,0.44061682531723784,0.7518568634986877,0.3852922869623105 +B5_mofe_mlp,3407,audio_vision_10,0.1,TA,728,0.42197802662849426,0.5906593406593407,0.5759372217914477,0.6386086344718933,0.5889149366423129 +B5_mofe_mlp,3407,audio_vision_10,0.1,TV,728,0.39879122376441956,0.6195054945054945,0.5730831708901883,0.6402206420898438,0.5971609331392899 +B5_mofe_mlp,3407,audio_vision_10,0.1,AV,728,0.39879122376441956,0.4409340659340659,0.39365944462354036,0.8360045552253723,0.3745781671523048 +B5_mofe_mlp,3407,audio_vision_10,0.1,TAV,728,0.39879122376441956,0.5879120879120879,0.5702051410805352,0.6638662219047546,0.5713070522068541 +B5_mofe_mlp,3407,all_modalities_10,0.1,learned_router,728,0.4610164835164835,0.6195054945054945,0.5890880499595165,0.6359822750091553,0.6113081341411937 +B5_mofe_mlp,3407,all_modalities_10,0.1,T,728,0.4610165059566498,0.5934065934065934,0.5840003133946626,0.63011234998703,0.6190677754940976 +B5_mofe_mlp,3407,all_modalities_10,0.1,A,728,0.4225274920463562,0.47115384615384615,0.2646409174971828,0.8042469024658203,0.08459586170402346 +B5_mofe_mlp,3407,all_modalities_10,0.1,V,728,0.399478018283844,0.46016483516483514,0.39465376804402846,0.7862517237663269,0.2773311673393334 +B5_mofe_mlp,3407,all_modalities_10,0.1,TA,728,0.4225274920463562,0.5824175824175825,0.5671818574180514,0.6445117592811584,0.5822325728126405 +B5_mofe_mlp,3407,all_modalities_10,0.1,TV,728,0.399478018283844,0.6043956043956044,0.5508070001152509,0.6478139162063599,0.5896831592571647 +B5_mofe_mlp,3407,all_modalities_10,0.1,AV,728,0.399478018283844,0.3873626373626374,0.32667745632365675,0.8781699538230896,0.28384176543794837 +B5_mofe_mlp,3407,all_modalities_10,0.1,TAV,728,0.399478018283844,0.5796703296703297,0.5603794485883429,0.6692947745323181,0.5686062320667473 +B5_mofe_mlp,3407,text_20,0.2,learned_router,728,0.5066758241758241,0.6195054945054945,0.5755820027236789,0.6454524993896484,0.593560555068977 +B5_mofe_mlp,3407,text_20,0.2,T,728,0.4073077142238617,0.6016483516483516,0.5744089403315806,0.6390801072120667,0.6021358689203483 +B5_mofe_mlp,3407,text_20,0.2,A,728,0.47049450874328613,0.4642857142857143,0.2494552539431357,0.8059768080711365,0.07746170464833917 +B5_mofe_mlp,3407,text_20,0.2,V,728,0.44483518600463867,0.45467032967032966,0.38380648966660313,0.793641984462738,0.24830679616913864 +B5_mofe_mlp,3407,text_20,0.2,TA,728,0.3711263835430145,0.5741758241758241,0.5418661031211773,0.6627365946769714,0.560477116430549 +B5_mofe_mlp,3407,text_20,0.2,TV,728,0.3507142961025238,0.603021978021978,0.5357923985725986,0.6560191512107849,0.57324278460567 +B5_mofe_mlp,3407,text_20,0.2,AV,728,0.44483518600463867,0.3791208791208791,0.3126586105031628,0.8910257816314697,0.2565337001789869 +B5_mofe_mlp,3407,text_20,0.2,TAV,728,0.3507142961025238,0.5934065934065934,0.5604543054498898,0.6754387021064758,0.5450045495235731 +B5_mofe_mlp,3407,audio_20,0.2,learned_router,728,0.5117582417582418,0.6332417582417582,0.6046231856798019,0.6310903429985046,0.6172434069189507 +B5_mofe_mlp,3407,audio_20,0.2,T,728,0.5117582678794861,0.6002747252747253,0.5893745342941709,0.6272591948509216,0.6221016536867588 +B5_mofe_mlp,3407,audio_20,0.2,A,728,0.3748626410961151,0.4945054945054945,0.3373484461000103,0.7447602152824402,0.3567551983127843 +B5_mofe_mlp,3407,audio_20,0.2,V,728,0.44483518600463867,0.46016483516483514,0.3944283930013793,0.788540244102478,0.2723841073830063 +B5_mofe_mlp,3407,audio_20,0.2,TA,728,0.3748626410961151,0.5975274725274725,0.5831680958487365,0.6350200176239014,0.5940989862125469 +B5_mofe_mlp,3407,audio_20,0.2,TV,728,0.44483518600463867,0.6167582417582418,0.5667282248122798,0.6437039375305176,0.5927169318926079 +B5_mofe_mlp,3407,audio_20,0.2,AV,728,0.35401099920272827,0.4697802197802198,0.42693867646678624,0.8040833473205566,0.41986353119227654 +B5_mofe_mlp,3407,audio_20,0.2,TAV,728,0.35401099920272827,0.6002747252747253,0.5813735202449936,0.6587744355201721,0.5783157554155207 +B5_mofe_mlp,3407,vision_20,0.2,learned_router,728,0.5117582417582418,0.635989010989011,0.6063745521197687,0.6304876804351807,0.6140966219851783 +B5_mofe_mlp,3407,vision_20,0.2,T,728,0.5117582678794861,0.6002747252747253,0.5892252758196194,0.6273123025894165,0.6225083846238101 +B5_mofe_mlp,3407,vision_20,0.2,A,728,0.47049450874328613,0.4684065934065934,0.25920850508591314,0.8053038716316223,0.085792392341107 +B5_mofe_mlp,3407,vision_20,0.2,V,728,0.3525000214576721,0.5288461538461539,0.4720058888479941,0.7252842783927917,0.41317305592752934 +B5_mofe_mlp,3407,vision_20,0.2,TA,728,0.47049450874328613,0.5824175824175825,0.5670673740607254,0.6420120000839233,0.584068084198822 +B5_mofe_mlp,3407,vision_20,0.2,TV,728,0.3525000214576721,0.625,0.5781351467902999,0.6339038610458374,0.5999732210847242 +B5_mofe_mlp,3407,vision_20,0.2,AV,728,0.3525000214576721,0.47802197802197804,0.4364142415752362,0.7915685176849365,0.4020386709009841 +B5_mofe_mlp,3407,vision_20,0.2,TAV,728,0.3525000214576721,0.6181318681318682,0.5982469512967439,0.656253457069397,0.5726154586560915 +B5_mofe_mlp,3407,audio_vision_20,0.2,learned_router,728,0.5117582417582418,0.635989010989011,0.609224910808655,0.6240155696868896,0.6185045310430713 +B5_mofe_mlp,3407,audio_vision_20,0.2,T,728,0.5117582678794861,0.5989010989010989,0.5883115478052187,0.6270718574523926,0.6227611193021018 +B5_mofe_mlp,3407,audio_vision_20,0.2,A,728,0.36782968044281006,0.5123626373626373,0.3814657210401891,0.7518494725227356,0.33531566843891375 +B5_mofe_mlp,3407,audio_vision_20,0.2,V,728,0.34728023409843445,0.5247252747252747,0.4843248302906597,0.7358435392379761,0.4214118417347598 +B5_mofe_mlp,3407,audio_vision_20,0.2,TA,728,0.36782968044281006,0.6002747252747253,0.5892792851694453,0.6287823915481567,0.6006773690433412 +B5_mofe_mlp,3407,audio_vision_20,0.2,TV,728,0.34728023409843445,0.6332417582417582,0.5964419732456347,0.6303028464317322,0.6076400918884116 +B5_mofe_mlp,3407,audio_vision_20,0.2,AV,728,0.34728023409843445,0.46565934065934067,0.4355944055944056,0.802980363368988,0.416563986982085 +B5_mofe_mlp,3407,audio_vision_20,0.2,TAV,728,0.34728023409843445,0.6002747252747253,0.5862363095320736,0.6544463038444519,0.585378576750997 +B5_mofe_mlp,3407,all_modalities_20,0.2,learned_router,728,0.40958791208791206,0.6332417582417582,0.6079307048305954,0.634419858455658,0.6044717331613441 +B5_mofe_mlp,3407,all_modalities_20,0.2,T,728,0.40958791971206665,0.6002747252747253,0.5899048320687862,0.6346542835235596,0.6100138467500781 +B5_mofe_mlp,3407,all_modalities_20,0.2,A,728,0.3732692301273346,0.4739010989010989,0.27165625210557365,0.8040054440498352,0.10424704916947851 +B5_mofe_mlp,3407,all_modalities_20,0.2,V,728,0.3524450659751892,0.4642857142857143,0.40252151018656984,0.7802431583404541,0.29135073297647346 +B5_mofe_mlp,3407,all_modalities_20,0.2,TA,728,0.3732692301273346,0.5686813186813187,0.5572575422788313,0.6487622261047363,0.572519010283675 +B5_mofe_mlp,3407,all_modalities_20,0.2,TV,728,0.3524450659751892,0.6126373626373627,0.5629491230638092,0.6498149037361145,0.5810132494000526 +B5_mofe_mlp,3407,all_modalities_20,0.2,AV,728,0.3524450659751892,0.3956043956043956,0.3405710346065853,0.8701844215393066,0.3057071236393479 +B5_mofe_mlp,3407,all_modalities_20,0.2,TAV,728,0.3524450659751892,0.5837912087912088,0.5631923344724951,0.6739726066589355,0.559728632496238 +B5_mofe_mlp,3407,text_30,0.3,learned_router,728,0.5033791208791208,0.6181318681318682,0.5733540937906745,0.6566742658615112,0.566892564181419 +B5_mofe_mlp,3407,text_30,0.3,T,728,0.36063188314437866,0.6112637362637363,0.5806932321558671,0.6485380530357361,0.579561962998976 +B5_mofe_mlp,3407,text_30,0.3,A,728,0.47049450874328613,0.46703296703296704,0.2536900771342752,0.8093824982643127,0.062449242058257734 +B5_mofe_mlp,3407,text_30,0.3,V,728,0.44483518600463867,0.4409340659340659,0.3710188660549014,0.7965743541717529,0.2468807094370837 +B5_mofe_mlp,3407,text_30,0.3,TA,728,0.32774725556373596,0.592032967032967,0.5537539943729123,0.6646007299423218,0.5395408779465738 +B5_mofe_mlp,3407,text_30,0.3,TV,728,0.30873626470565796,0.6016483516483516,0.5385391277047284,0.6617969870567322,0.5533827408893808 +B5_mofe_mlp,3407,text_30,0.3,AV,728,0.44483518600463867,0.37225274725274726,0.3029182824443429,0.8928882479667664,0.25637364632868326 +B5_mofe_mlp,3407,text_30,0.3,TAV,728,0.30873626470565796,0.5865384615384616,0.5490273643806375,0.6738438010215759,0.5345429351694087 +B5_mofe_mlp,3407,audio_30,0.3,learned_router,728,0.5117582417582418,0.6332417582417582,0.604260667390971,0.6289573907852173,0.6173815327526297 +B5_mofe_mlp,3407,audio_30,0.3,T,728,0.5117582678794861,0.6002747252747253,0.5893745342941709,0.6269524693489075,0.6225360827025238 +B5_mofe_mlp,3407,audio_30,0.3,A,728,0.3245879113674164,0.532967032967033,0.41756519321424274,0.7341850399971008,0.41121987797045906 +B5_mofe_mlp,3407,audio_30,0.3,V,728,0.44483518600463867,0.45054945054945056,0.3832106337593439,0.788564920425415,0.27355794624988927 +B5_mofe_mlp,3407,audio_30,0.3,TA,728,0.3245879113674164,0.6085164835164835,0.5930238521146932,0.6325163841247559,0.5979866380346056 +B5_mofe_mlp,3407,audio_30,0.3,TV,728,0.44483518600463867,0.6112637362637363,0.5583419641348294,0.6421341300010681,0.5937514966892242 +B5_mofe_mlp,3407,audio_30,0.3,AV,728,0.30483517050743103,0.5,0.4623444097193838,0.7733940482139587,0.4553589450889685 +B5_mofe_mlp,3407,audio_30,0.3,TAV,728,0.30483517050743103,0.614010989010989,0.5938581135634877,0.6490943431854248,0.594392630535773 +B5_mofe_mlp,3407,vision_30,0.3,learned_router,728,0.5117582417582418,0.6332417582417582,0.6016029959347541,0.6318032145500183,0.6125144009888801 +B5_mofe_mlp,3407,vision_30,0.3,T,728,0.5117582678794861,0.5989010989010989,0.5879942456720697,0.627319872379303,0.622716023566736 +B5_mofe_mlp,3407,vision_30,0.3,A,728,0.47049450874328613,0.4697802197802198,0.26181694968773733,0.8051190972328186,0.08667342142486818 +B5_mofe_mlp,3407,vision_30,0.3,V,728,0.3087087869644165,0.5233516483516484,0.461764143838227,0.7223021984100342,0.4261833345686536 +B5_mofe_mlp,3407,vision_30,0.3,TA,728,0.47049450874328613,0.5824175824175825,0.5672280279314932,0.6419576406478882,0.5838877593870844 +B5_mofe_mlp,3407,vision_30,0.3,TV,728,0.3087087869644165,0.6112637362637363,0.5662069378393744,0.641978919506073,0.5924386504061155 +B5_mofe_mlp,3407,vision_30,0.3,AV,728,0.3087087869644165,0.49862637362637363,0.445456810889948,0.7776786088943481,0.42160082881328553 +B5_mofe_mlp,3407,vision_30,0.3,TAV,728,0.3087087869644165,0.6071428571428571,0.5854633810737848,0.6539671421051025,0.5750920229415292 +B5_mofe_mlp,3407,audio_vision_30,0.3,learned_router,728,0.5117582417582418,0.6318681318681318,0.6067614810631058,0.6228161454200745,0.6214083492458665 +B5_mofe_mlp,3407,audio_vision_30,0.3,T,728,0.5117582678794861,0.6002747252747253,0.5893745342941709,0.627665102481842,0.6221383829594522 +B5_mofe_mlp,3407,audio_vision_30,0.3,A,728,0.324972540140152,0.5425824175824175,0.43415828421831115,0.7195268273353577,0.4252793587336247 +B5_mofe_mlp,3407,audio_vision_30,0.3,V,728,0.30618131160736084,0.5521978021978022,0.5104487621775328,0.7024396657943726,0.48245166107283655 +B5_mofe_mlp,3407,audio_vision_30,0.3,TA,728,0.324972540140152,0.6085164835164835,0.5960454962258536,0.627300500869751,0.605465763886298 +B5_mofe_mlp,3407,audio_vision_30,0.3,TV,728,0.30618131160736084,0.6195054945054945,0.5821910607231708,0.6338788270950317,0.6053651346534237 +B5_mofe_mlp,3407,audio_vision_30,0.3,AV,728,0.30618131160736084,0.510989010989011,0.4810449552418117,0.7675954699516296,0.46180909475521414 +B5_mofe_mlp,3407,audio_vision_30,0.3,TAV,728,0.30618131160736084,0.6085164835164835,0.5928020993486814,0.6524761319160461,0.5899472144346922 +B5_mofe_mlp,3407,all_modalities_30,0.3,learned_router,728,0.3557967032967033,0.6222527472527473,0.5949881199577279,0.6458109617233276,0.5977780881361412 +B5_mofe_mlp,3407,all_modalities_30,0.3,T,728,0.3557967245578766,0.6057692307692307,0.5973699542180001,0.636671781539917,0.6031830619040961 +B5_mofe_mlp,3407,all_modalities_30,0.3,A,728,0.32255494594573975,0.47802197802197804,0.2858992684718562,0.7997276186943054,0.12003216546779098 +B5_mofe_mlp,3407,all_modalities_30,0.3,V,728,0.3045879304409027,0.49175824175824173,0.4332501168084348,0.778567910194397,0.28114115860320466 +B5_mofe_mlp,3407,all_modalities_30,0.3,TA,728,0.32255494594573975,0.5755494505494505,0.5618376925923627,0.6488231420516968,0.5646885904319525 +B5_mofe_mlp,3407,all_modalities_30,0.3,TV,728,0.3045879304409027,0.6153846153846154,0.569951909037207,0.6524090766906738,0.5765150164354017 +B5_mofe_mlp,3407,all_modalities_30,0.3,AV,728,0.3045879304409027,0.41208791208791207,0.3552498095654271,0.8604830503463745,0.28184821072646954 +B5_mofe_mlp,3407,all_modalities_30,0.3,TAV,728,0.3045879304409027,0.5741758241758241,0.5572916475321001,0.676213800907135,0.54624389958209 +B5_mofe_mlp,2026,clean,0.0,learned_router,728,0.5117582417582418,0.6318681318681318,0.596526187825028,0.6597291827201843,0.603269634378733 +B5_mofe_mlp,2026,clean,0.0,T,728,0.5117582678794861,0.6126373626373627,0.596773495897157,0.6489740014076233,0.601146379126596 +B5_mofe_mlp,2026,clean,0.0,A,728,0.47049450874328613,0.4725274725274725,0.2753954425047768,0.8250412344932556,0.2298903631499931 +B5_mofe_mlp,2026,clean,0.0,V,728,0.44483518600463867,0.5151098901098901,0.41609693688045696,0.7419555187225342,0.33524138002400866 +B5_mofe_mlp,2026,clean,0.0,TA,728,0.47049450874328613,0.592032967032967,0.5532162058371736,0.6664567589759827,0.5597609050774838 +B5_mofe_mlp,2026,clean,0.0,TV,728,0.44483518600463867,0.592032967032967,0.5774206516713903,0.7040247917175293,0.5664710859024277 +B5_mofe_mlp,2026,clean,0.0,AV,728,0.44483518600463867,0.3626373626373626,0.34907979624206836,0.8443424105644226,0.23675834043024976 +B5_mofe_mlp,2026,clean,0.0,TAV,728,0.44483518600463867,0.5370879120879121,0.4651554172647243,0.800514817237854,0.5127572796173613 +B5_mofe_mlp,2026,text_10,0.1,learned_router,728,0.5091208791208791,0.6332417582417582,0.5928689292978023,0.6584991812705994,0.596161006853697 +B5_mofe_mlp,2026,text_10,0.1,T,728,0.46167582273483276,0.6126373626373627,0.5930164703753459,0.6494342088699341,0.5946751987953215 +B5_mofe_mlp,2026,text_10,0.1,A,728,0.47049450874328613,0.4725274725274725,0.27541290783115796,0.8262399435043335,0.22445379743900615 +B5_mofe_mlp,2026,text_10,0.1,V,728,0.44483518600463867,0.5123626373626373,0.41266149205541813,0.7444264888763428,0.3297960384984079 +B5_mofe_mlp,2026,text_10,0.1,TA,728,0.4230494499206543,0.5837912087912088,0.5397729831672579,0.6658428907394409,0.5527226923735943 +B5_mofe_mlp,2026,text_10,0.1,TV,728,0.4000000059604645,0.592032967032967,0.571111408272237,0.6985095739364624,0.5560292387489061 +B5_mofe_mlp,2026,text_10,0.1,AV,728,0.44483518600463867,0.3585164835164835,0.34496753089852183,0.8458796739578247,0.23343561887490413 +B5_mofe_mlp,2026,text_10,0.1,TAV,728,0.4000000059604645,0.5384615384615384,0.4615912536399786,0.7843563556671143,0.4964630609894201 +B5_mofe_mlp,2026,audio_10,0.1,learned_router,728,0.5117582417582418,0.6332417582417582,0.5971419540706502,0.658007025718689,0.6047899431200526 +B5_mofe_mlp,2026,audio_10,0.1,T,728,0.5117582678794861,0.6112637362637363,0.594775223911222,0.6496217846870422,0.601103231343751 +B5_mofe_mlp,2026,audio_10,0.1,A,728,0.42263737320899963,0.489010989010989,0.30986648206415496,0.795183539390564,0.32899912608575077 +B5_mofe_mlp,2026,audio_10,0.1,V,728,0.44483518600463867,0.5164835164835165,0.41734454924869885,0.7415561079978943,0.3356207820136009 +B5_mofe_mlp,2026,audio_10,0.1,TA,728,0.42263737320899963,0.5906593406593407,0.5523948427986319,0.6616610288619995,0.5656969509728144 +B5_mofe_mlp,2026,audio_10,0.1,TV,728,0.44483518600463867,0.5906593406593407,0.5756793112725316,0.7050058841705322,0.567017826103781 +B5_mofe_mlp,2026,audio_10,0.1,AV,728,0.3998076915740967,0.40796703296703296,0.4021505731323723,0.8058489561080933,0.33914515576905724 +B5_mofe_mlp,2026,audio_10,0.1,TAV,728,0.3998076915740967,0.5604395604395604,0.5020907193394417,0.7697135210037231,0.5332348404594554 +B5_mofe_mlp,2026,vision_10,0.1,learned_router,728,0.5117582417582418,0.6277472527472527,0.5919759280299669,0.6596941947937012,0.6023122800724011 +B5_mofe_mlp,2026,vision_10,0.1,T,728,0.5117582678794861,0.6112637362637363,0.5949280494683542,0.648731529712677,0.6010584699954675 +B5_mofe_mlp,2026,vision_10,0.1,A,728,0.47049450874328613,0.4725274725274725,0.2753954425047768,0.8233988285064697,0.2309789025689641 +B5_mofe_mlp,2026,vision_10,0.1,V,728,0.3984340727329254,0.5398351648351648,0.43853463696042566,0.7145515084266663,0.4295200877890959 +B5_mofe_mlp,2026,vision_10,0.1,TA,728,0.47049450874328613,0.592032967032967,0.5532162058371736,0.6668615937232971,0.5594488575507512 +B5_mofe_mlp,2026,vision_10,0.1,TV,728,0.3984340727329254,0.6016483516483516,0.5819547002883038,0.6787747144699097,0.5779704289995252 +B5_mofe_mlp,2026,vision_10,0.1,AV,728,0.3984340727329254,0.4258241758241758,0.42253912572720753,0.7936479449272156,0.3612784148549423 +B5_mofe_mlp,2026,vision_10,0.1,TAV,728,0.3984340727329254,0.5714285714285714,0.5087225335512859,0.760180652141571,0.5350946399361437 +B5_mofe_mlp,2026,audio_vision_10,0.1,learned_router,728,0.5117582417582418,0.6291208791208791,0.5949724164706738,0.6590589880943298,0.6034714874557056 +B5_mofe_mlp,2026,audio_vision_10,0.1,T,728,0.5117582678794861,0.614010989010989,0.5982455160173181,0.6492474675178528,0.6011554275378082 +B5_mofe_mlp,2026,audio_vision_10,0.1,A,728,0.42107143998146057,0.4876373626373626,0.3042744149605441,0.7851645350456238,0.3562019133732601 +B5_mofe_mlp,2026,audio_vision_10,0.1,V,728,0.3980769217014313,0.5439560439560439,0.4559201356747369,0.7165518999099731,0.4228907732362493 +B5_mofe_mlp,2026,audio_vision_10,0.1,TA,728,0.42107143998146057,0.5934065934065934,0.559867581706166,0.6601358652114868,0.5680404827910764 +B5_mofe_mlp,2026,audio_vision_10,0.1,TV,728,0.3980769217014313,0.6002747252747253,0.5842152460266312,0.6859722137451172,0.575453733717388 +B5_mofe_mlp,2026,audio_vision_10,0.1,AV,728,0.3980769217014313,0.43131868131868134,0.4262759970983259,0.8031644821166992,0.35346199972803627 +B5_mofe_mlp,2026,audio_vision_10,0.1,TAV,728,0.3980769217014313,0.5535714285714286,0.49118433140019535,0.7740562558174133,0.5279377438591082 +B5_mofe_mlp,2026,all_modalities_10,0.1,learned_router,728,0.45884615384615385,0.6387362637362637,0.6013762119617033,0.6582632660865784,0.5998105774990818 +B5_mofe_mlp,2026,all_modalities_10,0.1,T,728,0.45884615182876587,0.6098901098901099,0.5927342612347185,0.6530159711837769,0.5924213553262381 +B5_mofe_mlp,2026,all_modalities_10,0.1,A,728,0.42030221223831177,0.4739010989010989,0.2799395224420051,0.8183923363685608,0.22552480747030987 +B5_mofe_mlp,2026,all_modalities_10,0.1,V,728,0.3982967138290405,0.5123626373626373,0.41084897900937084,0.7401790618896484,0.34290184431076304 +B5_mofe_mlp,2026,all_modalities_10,0.1,TA,728,0.42030221223831177,0.603021978021978,0.5636157256230538,0.6699337959289551,0.5545966180956201 +B5_mofe_mlp,2026,all_modalities_10,0.1,TV,728,0.3982967138290405,0.5961538461538461,0.5817349795156427,0.7066670656204224,0.5633093560522575 +B5_mofe_mlp,2026,all_modalities_10,0.1,AV,728,0.3982967138290405,0.36813186813186816,0.35589607464607464,0.8376190066337585,0.24289847097047093 +B5_mofe_mlp,2026,all_modalities_10,0.1,TAV,728,0.3982967138290405,0.5288461538461539,0.46033318738535584,0.7966769933700562,0.5109216642892338 +B5_mofe_mlp,2026,text_20,0.2,learned_router,728,0.5064560439560439,0.6277472527472527,0.579286730359655,0.6673612594604492,0.5829749639030503 +B5_mofe_mlp,2026,text_20,0.2,T,728,0.4112088084220886,0.6167582417582418,0.5881921764479171,0.6567916870117188,0.5831459023434139 +B5_mofe_mlp,2026,text_20,0.2,A,728,0.47049450874328613,0.47115384615384615,0.27465560798894134,0.8285160064697266,0.21743928273022034 +B5_mofe_mlp,2026,text_20,0.2,V,728,0.44483518600463867,0.5096153846153846,0.4084128341513546,0.7475921511650085,0.31692232958876915 +B5_mofe_mlp,2026,text_20,0.2,TA,728,0.3752472698688507,0.603021978021978,0.5486119524164519,0.6673292517662048,0.539250775649662 +B5_mofe_mlp,2026,text_20,0.2,TV,728,0.35461539030075073,0.6112637362637363,0.5843667494564081,0.689144492149353,0.544504109553622 +B5_mofe_mlp,2026,text_20,0.2,AV,728,0.44483518600463867,0.3585164835164835,0.3446302940469433,0.8487928509712219,0.2126482794308453 +B5_mofe_mlp,2026,text_20,0.2,TAV,728,0.35461539030075073,0.5425824175824175,0.46587133463364855,0.7634091377258301,0.48214188094638644 +B5_mofe_mlp,2026,audio_20,0.2,learned_router,728,0.5117582417582418,0.6304945054945055,0.5954536143668596,0.6568871736526489,0.6037191294218217 +B5_mofe_mlp,2026,audio_20,0.2,T,728,0.5117582678794861,0.614010989010989,0.5983472125624337,0.650320291519165,0.6009217628625136 +B5_mofe_mlp,2026,audio_20,0.2,A,728,0.3796977996826172,0.5164835164835165,0.3655881236095169,0.7692446112632751,0.3949788114535673 +B5_mofe_mlp,2026,audio_20,0.2,V,728,0.44483518600463867,0.5151098901098901,0.41620583054274807,0.7414965629577637,0.33444942065556305 +B5_mofe_mlp,2026,audio_20,0.2,TA,728,0.3796977996826172,0.5989010989010989,0.5638265304840163,0.6491042971611023,0.5770019410427926 +B5_mofe_mlp,2026,audio_20,0.2,TV,728,0.44483518600463867,0.592032967032967,0.5774181294613681,0.7061969041824341,0.5658046168061596 +B5_mofe_mlp,2026,audio_20,0.2,AV,728,0.35961538553237915,0.4793956043956044,0.4729905027892644,0.7711812257766724,0.3921269492204333 +B5_mofe_mlp,2026,audio_20,0.2,TAV,728,0.35961538553237915,0.5645604395604396,0.509050719882809,0.7490760684013367,0.5341345004529877 +B5_mofe_mlp,2026,vision_20,0.2,learned_router,728,0.5117582417582418,0.6277472527472527,0.5900160130008749,0.6632897853851318,0.6042444043430241 +B5_mofe_mlp,2026,vision_20,0.2,T,728,0.5117582678794861,0.6126373626373627,0.5965577523042379,0.6482207179069519,0.6014239127944337 +B5_mofe_mlp,2026,vision_20,0.2,A,728,0.47049450874328613,0.4725274725274725,0.2753954425047768,0.8218814730644226,0.23031725904419292 +B5_mofe_mlp,2026,vision_20,0.2,V,728,0.3572802245616913,0.5480769230769231,0.45491017561200736,0.7200025916099548,0.41887110693834134 +B5_mofe_mlp,2026,vision_20,0.2,TA,728,0.47049450874328613,0.592032967032967,0.5532162058371736,0.6667653918266296,0.5596752023907243 +B5_mofe_mlp,2026,vision_20,0.2,TV,728,0.3572802245616913,0.6085164835164835,0.5858009850150654,0.6767599582672119,0.5666763582603909 +B5_mofe_mlp,2026,vision_20,0.2,AV,728,0.3572802245616913,0.45604395604395603,0.45257735286592293,0.7822015881538391,0.3511756469807228 +B5_mofe_mlp,2026,vision_20,0.2,TAV,728,0.3572802245616913,0.5673076923076923,0.5113134010717539,0.7461904287338257,0.5258941828977346 +B5_mofe_mlp,2026,audio_vision_20,0.2,learned_router,728,0.5117582417582418,0.625,0.5917418600128784,0.6548266410827637,0.6054018274439124 +B5_mofe_mlp,2026,audio_vision_20,0.2,T,728,0.5117582678794861,0.614010989010989,0.5983472125624337,0.649300754070282,0.6012901186776531 +B5_mofe_mlp,2026,audio_vision_20,0.2,A,728,0.37296703457832336,0.5041208791208791,0.35789310705995514,0.7596713900566101,0.41398276883225044 +B5_mofe_mlp,2026,audio_vision_20,0.2,V,728,0.3531593382358551,0.5796703296703297,0.5048183867639185,0.6984941959381104,0.4714365413048334 +B5_mofe_mlp,2026,audio_vision_20,0.2,TA,728,0.37296703457832336,0.5892857142857143,0.5573263153954385,0.6522572040557861,0.5764869722455894 +B5_mofe_mlp,2026,audio_vision_20,0.2,TV,728,0.3531593382358551,0.6057692307692307,0.5884441789459711,0.6706872582435608,0.581897930872946 +B5_mofe_mlp,2026,audio_vision_20,0.2,AV,728,0.3531593382358551,0.4684065934065934,0.465468176106474,0.767947793006897,0.41865978021168704 +B5_mofe_mlp,2026,audio_vision_20,0.2,TAV,728,0.3531593382358551,0.5728021978021978,0.5181235721920635,0.7481652498245239,0.5424179636227278 +B5_mofe_mlp,2026,all_modalities_20,0.2,learned_router,728,0.40585164835164833,0.6387362637362637,0.6023597229130889,0.6525924205780029,0.601538369275999 +B5_mofe_mlp,2026,all_modalities_20,0.2,T,728,0.40585166215896606,0.6057692307692307,0.5895572046059541,0.6424159407615662,0.5987482734789812 +B5_mofe_mlp,2026,all_modalities_20,0.2,A,728,0.3694780170917511,0.47802197802197804,0.2829329028964813,0.8103722333908081,0.25204782200960785 +B5_mofe_mlp,2026,all_modalities_20,0.2,V,728,0.34881868958473206,0.5178571428571429,0.40966513694882706,0.7393192648887634,0.34226536554912906 +B5_mofe_mlp,2026,all_modalities_20,0.2,TA,728,0.3694780170917511,0.5975274725274725,0.5569987222986782,0.6711844801902771,0.560237183016514 +B5_mofe_mlp,2026,all_modalities_20,0.2,TV,728,0.34881868958473206,0.5906593406593407,0.5720218781867062,0.699638843536377,0.5616554801012917 +B5_mofe_mlp,2026,all_modalities_20,0.2,AV,728,0.34881868958473206,0.37362637362637363,0.3626956690910179,0.8267216682434082,0.25176083351503564 +B5_mofe_mlp,2026,all_modalities_20,0.2,TAV,728,0.34881868958473206,0.5453296703296703,0.4749388892268766,0.7958879470825195,0.5090043003952414 +B5_mofe_mlp,2026,text_30,0.3,learned_router,728,0.5032417582417582,0.6236263736263736,0.5695777746219127,0.6862288117408752,0.5485307360149316 +B5_mofe_mlp,2026,text_30,0.3,T,728,0.35857143998146057,0.6002747252747253,0.5650627890671912,0.6760544180870056,0.5486430119282898 +B5_mofe_mlp,2026,text_30,0.3,A,728,0.47049450874328613,0.4684065934065934,0.2694917313065056,0.8302391171455383,0.21220052810195741 +B5_mofe_mlp,2026,text_30,0.3,V,728,0.44483518600463867,0.5041208791208791,0.40060701323330045,0.7515881657600403,0.3084106819138643 +B5_mofe_mlp,2026,text_30,0.3,TA,728,0.32582417130470276,0.5824175824175825,0.5246945241369286,0.692663848400116,0.4910659178654774 +B5_mofe_mlp,2026,text_30,0.3,TV,728,0.3089011013507843,0.5934065934065934,0.5598344950469888,0.7086329460144043,0.49570851546354966 +B5_mofe_mlp,2026,text_30,0.3,AV,728,0.44483518600463867,0.35714285714285715,0.3435056224920738,0.8512967228889465,0.19852242746832857 +B5_mofe_mlp,2026,text_30,0.3,TAV,728,0.3089011013507843,0.5343406593406593,0.45069366576352604,0.7611382603645325,0.43668715634323624 +B5_mofe_mlp,2026,audio_30,0.3,learned_router,728,0.5117582417582418,0.6277472527472527,0.5928207016211315,0.6564014554023743,0.6057011879455848 +B5_mofe_mlp,2026,audio_30,0.3,T,728,0.5117582678794861,0.6126373626373627,0.5963454618716882,0.6506627798080444,0.6009406388191995 +B5_mofe_mlp,2026,audio_30,0.3,A,728,0.3250274658203125,0.521978021978022,0.38592081478406426,0.7444095611572266,0.43124762537966826 +B5_mofe_mlp,2026,audio_30,0.3,V,728,0.44483518600463867,0.5151098901098901,0.4189500868178652,0.7407843470573425,0.3353742329440497 +B5_mofe_mlp,2026,audio_30,0.3,TA,728,0.3250274658203125,0.6208791208791209,0.5886808752593625,0.6490352749824524,0.570975506988905 +B5_mofe_mlp,2026,audio_30,0.3,TV,728,0.44483518600463867,0.5892857142857143,0.5746190571714002,0.7062152624130249,0.5695050918318726 +B5_mofe_mlp,2026,audio_30,0.3,AV,728,0.3078022003173828,0.4876373626373626,0.4781988617789497,0.7588927745819092,0.41241611684819374 +B5_mofe_mlp,2026,audio_30,0.3,TAV,728,0.3078022003173828,0.5796703296703297,0.5254621943407217,0.7300676703453064,0.5331740156497008 +B5_mofe_mlp,2026,vision_30,0.3,learned_router,728,0.5117582417582418,0.6291208791208791,0.5902326768980343,0.6632499098777771,0.6060668473400238 +B5_mofe_mlp,2026,vision_30,0.3,T,728,0.5117582678794861,0.6112637362637363,0.5951956697149005,0.648055374622345,0.6013617900975071 +B5_mofe_mlp,2026,vision_30,0.3,A,728,0.47049450874328613,0.4725274725274725,0.2753954425047768,0.8200526237487793,0.2290184928714137 +B5_mofe_mlp,2026,vision_30,0.3,V,728,0.307609885931015,0.5714285714285714,0.480133525611997,0.6950517296791077,0.48955757399923455 +B5_mofe_mlp,2026,vision_30,0.3,TA,728,0.47049450874328613,0.592032967032967,0.5533383448560617,0.6673561334609985,0.5595820020363866 +B5_mofe_mlp,2026,vision_30,0.3,TV,728,0.307609885931015,0.6181318681318682,0.5942276325734972,0.6576289534568787,0.5825792146121718 +B5_mofe_mlp,2026,vision_30,0.3,AV,728,0.307609885931015,0.4835164835164835,0.4758194996023056,0.7574149966239929,0.39536644001209464 +B5_mofe_mlp,2026,vision_30,0.3,TAV,728,0.307609885931015,0.5851648351648352,0.5229046773276295,0.7180534601211548,0.542626958107565 +B5_mofe_mlp,2026,audio_vision_30,0.3,learned_router,728,0.5117582417582418,0.625,0.5962838833608418,0.6554995775222778,0.6044590906471965 +B5_mofe_mlp,2026,audio_vision_30,0.3,T,728,0.5117582678794861,0.6126373626373627,0.5966200968456806,0.6497842073440552,0.6010002684577302 +B5_mofe_mlp,2026,audio_vision_30,0.3,A,728,0.3190384805202484,0.521978021978022,0.40390961057275176,0.7342812418937683,0.44376299869270686 +B5_mofe_mlp,2026,audio_vision_30,0.3,V,728,0.301071435213089,0.5686813186813187,0.5040223473945638,0.687127411365509,0.4953386251138901 +B5_mofe_mlp,2026,audio_vision_30,0.3,TA,728,0.3190384805202484,0.5906593406593407,0.5620402015554825,0.6461183428764343,0.5751739467292083 +B5_mofe_mlp,2026,audio_vision_30,0.3,TV,728,0.301071435213089,0.6167582417582418,0.6003175316510284,0.6656390428543091,0.5833222854175106 +B5_mofe_mlp,2026,audio_vision_30,0.3,AV,728,0.301071435213089,0.5,0.49814761477760233,0.7473122477531433,0.4398620166190232 +B5_mofe_mlp,2026,audio_vision_30,0.3,TAV,728,0.301071435213089,0.5673076923076923,0.5181807136427148,0.7298097610473633,0.549260519080892 +B5_mofe_mlp,2026,all_modalities_30,0.3,learned_router,728,0.36043956043956044,0.6167582417582418,0.5778458150682998,0.6775782108306885,0.5767908971520759 +B5_mofe_mlp,2026,all_modalities_30,0.3,T,728,0.36043956875801086,0.603021978021978,0.582619604272029,0.6699808835983276,0.5730899311736721 +B5_mofe_mlp,2026,all_modalities_30,0.3,A,728,0.32840660214424133,0.4725274725274725,0.28157868772104216,0.8073930144309998,0.2329322112177729 +B5_mofe_mlp,2026,all_modalities_30,0.3,V,728,0.31060439348220825,0.5274725274725275,0.4283917020609205,0.7411948442459106,0.35155160333443614 +B5_mofe_mlp,2026,all_modalities_30,0.3,TA,728,0.32840660214424133,0.5755494505494505,0.5352650821255436,0.6817681789398193,0.5363938930111966 +B5_mofe_mlp,2026,all_modalities_30,0.3,TV,728,0.31060439348220825,0.5810439560439561,0.5628840082630775,0.7163251042366028,0.5509194266569809 +B5_mofe_mlp,2026,all_modalities_30,0.3,AV,728,0.31060439348220825,0.3873626373626374,0.37820754147953356,0.8290948271751404,0.2590757927734446 +B5_mofe_mlp,2026,all_modalities_30,0.3,TAV,728,0.31060439348220825,0.5343406593406593,0.46600788776303587,0.795678436756134,0.50343403287812 diff --git a/deep_learning/Q2/outputs/followups/D0_task_preference/rank_concordance.csv b/deep_learning/Q2/outputs/followups/D0_task_preference/rank_concordance.csv new file mode 100644 index 0000000..e7b7299 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/D0_task_preference/rank_concordance.csv @@ -0,0 +1,49 @@ +seed,condition,spearman_macro_f1_vs_mae,spearman_macro_f1_vs_pearson,best_macro_f1_expert,best_mae_expert,best_pearson_expert,macro_f1_order_best_to_worst,mae_order_best_to_worst,pearson_order_best_to_worst +42,all_modalities_10,-0.9285714285714288,0.9642857142857145,T,T,T,T>TA>TV>TAV>V>AV>A,T>TV>TA>TAV>V>A>AV,T>TA>TV>TAV>AV>V>A +42,all_modalities_20,-0.8571428571428573,0.9285714285714288,TA,T,T,TA>T>TV>TAV>V>AV>A,T>TV>TA>TAV>V>A>AV,T>TA>TV>TAV>AV>V>A +42,all_modalities_30,-0.9285714285714288,0.9642857142857145,T,T,T,T>TA>TV>TAV>V>AV>A,T>TV>TA>TAV>V>A>AV,T>TA>TV>TAV>AV>V>A +42,audio_10,-0.8571428571428573,0.8571428571428573,T,T,T,T>TA>TV>TAV>V>AV>A,T>TA>TV>TAV>A>AV>V,T>TA>TAV>TV>AV>A>V +42,audio_20,-0.9285714285714288,0.9285714285714288,TA,T,T,TA>T>TAV>TV>A>AV>V,T>TA>TV>TAV>A>AV>V,T>TA>TAV>TV>AV>A>V +42,audio_30,-0.9285714285714288,0.9285714285714288,TA,T,T,TA>T>TAV>TV>A>AV>V,T>TA>TV>TAV>A>AV>V,T>TA>TAV>TV>AV>A>V +42,audio_vision_10,-0.9285714285714288,0.9642857142857145,T,T,T,T>TA>TV>TAV>V>AV>A,T>TV>TA>TAV>V>A>AV,T>TA>TV>TAV>AV>V>A +42,audio_vision_20,-0.9285714285714288,0.9642857142857145,T,T,T,T>TA>TV>TAV>V>AV>A,T>TV>TA>TAV>V>A>AV,T>TA>TV>TAV>AV>V>A +42,audio_vision_30,-0.9642857142857145,0.9642857142857145,T,T,T,T>TA>TV>TAV>V>AV>A,T>TV>TA>TAV>V>AV>A,T>TA>TV>TAV>AV>V>A +42,clean,-0.9285714285714288,0.9642857142857145,T,T,T,T>TA>TV>TAV>V>AV>A,T>TV>TA>TAV>V>A>AV,T>TA>TV>TAV>AV>V>A +42,text_10,-0.9642857142857145,0.9642857142857145,T,T,T,T>TA>TV>TAV>V>AV>A,T>TA>TV>TAV>V>A>AV,T>TA>TV>TAV>AV>V>A +42,text_20,-0.9642857142857145,0.9642857142857145,T,T,T,T>TA>TV>TAV>V>AV>A,T>TA>TV>TAV>V>A>AV,T>TA>TV>TAV>AV>V>A +42,text_30,-0.9642857142857145,0.9642857142857145,T,T,T,T>TA>TV>TAV>V>AV>A,T>TA>TV>TAV>V>A>AV,T>TA>TV>TAV>AV>V>A +42,vision_10,-0.9642857142857145,0.9642857142857145,T,T,T,T>TA>TV>TAV>V>AV>A,T>TV>TA>TAV>V>AV>A,T>TA>TV>TAV>AV>V>A +42,vision_20,-0.8928571428571429,0.9642857142857145,T,T,T,T>TA>TV>TAV>V>AV>A,T>TV>TAV>TA>V>AV>A,T>TA>TV>TAV>AV>V>A +42,vision_30,-0.8571428571428573,0.9642857142857145,T,T,T,T>TA>TAV>TV>V>AV>A,T>TV>TAV>TA>V>AV>A,T>TA>TV>TAV>V>AV>A +2026,all_modalities_10,-0.8928571428571429,1.0,T,T,T,T>TV>TA>TAV>V>AV>A,T>TA>TV>V>TAV>A>AV,T>TV>TA>TAV>V>AV>A +2026,all_modalities_20,-0.8928571428571429,0.9642857142857145,T,T,T,T>TV>TA>TAV>V>AV>A,T>TA>TV>V>TAV>A>AV,T>TV>TA>TAV>V>A>AV +2026,all_modalities_30,-0.8928571428571429,1.0,T,T,T,T>TV>TA>TAV>V>AV>A,T>TA>TV>V>TAV>A>AV,T>TV>TA>TAV>V>AV>A +2026,audio_10,-0.8928571428571429,0.9642857142857145,T,T,T,T>TV>TA>TAV>V>AV>A,T>TA>TV>V>TAV>A>AV,T>TV>TA>TAV>AV>V>A +2026,audio_20,-0.7142857142857144,0.8571428571428573,T,TA,T,T>TV>TA>TAV>AV>V>A,TA>T>TV>V>TAV>A>AV,T>TA>TV>TAV>A>AV>V +2026,audio_30,-0.8571428571428573,0.8928571428571429,T,TA,T,T>TA>TV>TAV>AV>V>A,TA>T>TV>TAV>V>A>AV,T>TA>TV>TAV>A>AV>V +2026,audio_vision_10,-0.8928571428571429,0.9642857142857145,T,T,T,T>TV>TA>TAV>V>AV>A,T>TA>TV>V>TAV>A>AV,T>TV>TA>TAV>V>A>AV +2026,audio_vision_20,-0.8928571428571429,1.0,T,T,T,T>TV>TA>TAV>V>AV>A,T>TA>TV>V>TAV>A>AV,T>TV>TA>TAV>V>AV>A +2026,audio_vision_30,-0.7857142857142859,0.9285714285714288,TV,TA,T,TV>T>TA>TAV>V>AV>A,TA>T>TV>V>TAV>A>AV,T>TV>TA>TAV>V>A>AV +2026,clean,-0.8928571428571429,1.0,T,T,T,T>TV>TA>TAV>V>AV>A,T>TA>TV>V>TAV>A>AV,T>TV>TA>TAV>V>AV>A +2026,text_10,-0.8928571428571429,1.0,T,T,T,T>TV>TA>TAV>V>AV>A,T>TA>TV>V>TAV>A>AV,T>TV>TA>TAV>V>AV>A +2026,text_20,-0.8928571428571429,0.9642857142857145,T,T,T,T>TV>TA>TAV>V>AV>A,T>TA>TV>V>TAV>A>AV,T>TV>TA>TAV>V>A>AV +2026,text_30,-0.8928571428571429,0.9642857142857145,T,T,T,T>TV>TA>TAV>V>AV>A,T>TA>TV>V>TAV>A>AV,T>TV>TA>TAV>V>A>AV +2026,vision_10,-0.9285714285714288,1.0,T,T,T,T>TV>TA>TAV>V>AV>A,T>TA>TV>V>TAV>AV>A,T>TV>TA>TAV>V>AV>A +2026,vision_20,-0.9285714285714288,1.0,T,T,T,T>TV>TA>TAV>V>AV>A,T>TA>TV>V>TAV>AV>A,T>TV>TA>TAV>V>AV>A +2026,vision_30,-0.9642857142857145,1.0,T,T,T,T>TV>TA>TAV>V>AV>A,T>TV>TA>V>TAV>AV>A,T>TV>TA>TAV>V>AV>A +3407,all_modalities_10,-0.9285714285714288,0.8571428571428573,T,T,T,T>TA>TAV>TV>V>AV>A,T>TA>TV>TAV>V>A>AV,T>TV>TA>TAV>AV>V>A +3407,all_modalities_20,-0.8214285714285715,0.8571428571428573,T,T,T,T>TAV>TV>TA>V>AV>A,T>TA>TV>TAV>V>A>AV,T>TV>TA>TAV>AV>V>A +3407,all_modalities_30,-0.9285714285714288,0.9642857142857145,T,T,T,T>TV>TA>TAV>V>AV>A,T>TA>TV>TAV>V>A>AV,T>TV>TA>TAV>AV>V>A +3407,audio_10,-0.8214285714285715,0.8928571428571429,T,T,T,T>TA>TAV>TV>AV>V>A,T>TA>TV>TAV>A>V>AV,T>TV>TA>TAV>AV>V>A +3407,audio_20,-0.8214285714285715,0.9285714285714288,T,T,T,T>TA>TAV>TV>AV>V>A,T>TA>TV>TAV>A>V>AV,T>TA>TV>TAV>AV>A>V +3407,audio_30,-0.7142857142857144,0.8571428571428573,TAV,T,T,TAV>TA>T>TV>AV>A>V,T>TA>TV>TAV>A>AV>V,T>TA>TAV>TV>AV>A>V +3407,audio_vision_10,-0.9642857142857145,0.9642857142857145,T,T,T,T>TA>TV>TAV>V>AV>A,T>TA>TV>TAV>V>A>AV,T>TV>TA>TAV>V>AV>A +3407,audio_vision_20,-0.8214285714285715,0.8928571428571429,TV,T,T,TV>TA>T>TAV>V>AV>A,T>TA>TV>TAV>V>A>AV,T>TV>TA>TAV>V>AV>A +3407,audio_vision_30,-0.8571428571428573,0.8214285714285715,TA,TA,T,TA>TAV>T>TV>V>AV>A,TA>T>TV>TAV>V>A>AV,T>TA>TV>TAV>V>AV>A +3407,clean,-0.8214285714285715,0.8571428571428573,T,T,T,T>TAV>TV>TA>V>AV>A,T>TA>TV>TAV>V>A>AV,T>TV>TA>TAV>AV>V>A +3407,text_10,-0.8571428571428573,0.8571428571428573,T,T,T,T>TA>TAV>TV>V>AV>A,T>TV>TA>TAV>V>A>AV,T>TV>TA>TAV>AV>V>A +3407,text_20,-0.8214285714285715,0.8214285714285715,T,T,T,T>TAV>TA>TV>V>AV>A,T>TV>TA>TAV>V>A>AV,T>TV>TA>TAV>AV>V>A +3407,text_30,-0.8571428571428573,0.8571428571428573,T,T,T,T>TA>TAV>TV>V>AV>A,T>TV>TA>TAV>V>A>AV,T>TV>TA>TAV>AV>V>A +3407,vision_10,-0.8571428571428573,0.8928571428571429,T,T,T,T>TAV>TV>TA>V>AV>A,T>TV>TA>TAV>V>A>AV,T>TV>TA>TAV>V>AV>A +3407,vision_20,-0.7857142857142859,0.7857142857142859,TAV,T,T,TAV>T>TV>TA>V>AV>A,T>TV>TA>TAV>V>AV>A,T>TV>TA>TAV>V>AV>A +3407,vision_30,-0.8928571428571429,0.8571428571428573,T,T,T,T>TAV>TA>TV>V>AV>A,T>TA>TV>TAV>V>AV>A,T>TV>TA>TAV>V>AV>A diff --git a/deep_learning/Q2/outputs/followups/D0_task_preference/run_manifest.json b/deep_learning/Q2/outputs/followups/D0_task_preference/run_manifest.json new file mode 100644 index 0000000..402540a --- /dev/null +++ b/deep_learning/Q2/outputs/followups/D0_task_preference/run_manifest.json @@ -0,0 +1,41 @@ +{ + "diagnostic": "forced-expert task preference for the retained single-router MoFE-7", + "checkpoint_dir": "/home/gloamxun/modeling_zhaocui/deep_learning/Q2/outputs/mofe_7experts/models/B5_mofe_mlp", + "checkpoint_seeds": [ + 42, + 3407, + 2026 + ], + "feature_file": "/home/gloamxun/modeling_zhaocui/E题数据/附件2-数据集特征文件/aligned_50.pkl", + "scaler_file": "/home/gloamxun/modeling_zhaocui/deep_learning/Q2/outputs/mofe_7experts/aligned_robust_stats.npz", + "scaler_sha256": "3b62802bef92f49a7846773741bc57fda1564aa9eed43f047a3e31a9c357c4e8", + "device": "cuda", + "conditions": [ + "clean", + "text_10", + "audio_10", + "vision_10", + "audio_vision_10", + "all_modalities_10", + "text_20", + "audio_20", + "vision_20", + "audio_vision_20", + "all_modalities_20", + "text_30", + "audio_30", + "vision_30", + "audio_vision_30", + "all_modalities_30" + ], + "forced_expert_policy": "use the requested expert where its modality subset is available; fall back to the trained single router at positions where it is unavailable; all-missing positions use the learned missing token", + "scope_note": "50 official ordered wordpiece positions; no claim about physical-time reliability", + "interpretation_note": "Ranking agreement is descriptive on the supplied validation split; it is a motivation diagnostic, not an unbiased test-set estimate.", + "feature_sha256": "66e867aa74bc70a844e806e5571e371c9abb4a35f9e2887ce9b4d97ff2cb8fcd", + "cuda_device": "NVIDIA GeForce RTX 5070 Ti", + "python_version": "3.14.7 (main, Aug 10 2026, 00:00:00) [GCC 16.1.1 20260515 (Red Hat 16.1.1-2)]", + "torch_version": "2.14.0+cu130", + "numpy_version": "2.5.3", + "valid_examples": 728, + "corruption_seed_protocol": "seed + 13 + pattern_index*101 + int(rate*1000)" +} \ No newline at end of file diff --git a/deep_learning/Q2/outputs/followups/D0_task_preference/task_preference_diagnostic.md b/deep_learning/Q2/outputs/followups/D0_task_preference/task_preference_diagnostic.md new file mode 100644 index 0000000..27b9fd9 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/D0_task_preference/task_preference_diagnostic.md @@ -0,0 +1,47 @@ +# Single-router MoFE 任务偏好诊断 + +本诊断使用保留的 single-router 检查点和验证集,用于判断是否值得增加第二个 router;它不是测试集估计。 + +## Forced-expert 规则 + +所选模态子集可用的位置强制使用对应 expert;该子集不可用时,由已训练 router 在其他可用 expert 中选择;全模态缺失时沿用 learned missing token。可用率表示所选 expert 能被强制使用的位置比例。 + +## 缺失条件平均指标 + +下表先在每个 seed 内对 15 种连续块缺失条件求平均,再汇总三个 seed;seed 标准差见 CSV。 + +| Expert | Macro-F1 ↑ | MAE ↓ | Pearson ↑ | 可强制使用比例 | +| --- | ---: | ---: | ---: | ---: | +| T | 0.590 | 0.643 | 0.606 | 0.471 | +| A | 0.318 | 0.779 | 0.252 | 0.412 | +| V | 0.429 | 0.747 | 0.340 | 0.390 | +| TA | 0.567 | 0.659 | 0.577 | 0.393 | +| TV | 0.561 | 0.667 | 0.567 | 0.372 | +| AV | 0.393 | 0.799 | 0.349 | 0.372 | +| TAV | 0.531 | 0.708 | 0.542 | 0.354 | +| learned_router | 0.600 | 0.641 | 0.605 | 0.490 | + +## 两个任务的 expert 偏好 + +在 48 个 seed—条件组合中,分类 Macro-F1 与回归 MAE 的平均 Spearman ρ 为 **-0.885**。MAE 越低越好,因此负相关表示两个指标倾向于选中相似的 expert。Macro-F1 与 Pearson 的平均 ρ 为 **0.931**。 + +六个重点条件下的相关性先按三个 seed 求平均;最优 expert 一栏显示三个 seed 中的多数结果: + +| 条件 | ρ(Macro-F1, MAE) | ρ(Macro-F1, Pearson) | Macro-F1 最优 | MAE 最优 | Pearson 最优 | +| --- | ---: | ---: | --- | --- | --- | +| Clean | -0.881 | 0.940 | T (3/3) | T (3/3) | T (3/3) | +| Text 30% | -0.905 | 0.929 | T (3/3) | T (3/3) | T (3/3) | +| Audio 30% | -0.833 | 0.893 | T (1/3), TA (1/3), TAV (1/3) | T (2/3) | T (3/3) | +| Vision 30% | -0.905 | 0.940 | T (3/3) | T (3/3) | T (3/3) | +| Audio+Vision 30% | -0.869 | 0.905 | T (1/3), TA (1/3), TV (1/3) | TA (2/3) | T (3/3) | +| All-modal 30% | -0.917 | 0.976 | T (3/3) | T (3/3) | T (3/3) | + +各指标的最优 expert 次数:Macro-F1(T: 40, A: 0, V: 0, TA: 4, TV: 2, AV: 0, TAV: 2);MAE(T: 44, A: 0, V: 0, TA: 4, TV: 0, AV: 0, TAV: 0);Pearson(T: 48, A: 0, V: 0, TA: 0, TV: 0, AV: 0, TAV: 0)。 + +当前排名没有显示稳定的分类—回归 expert 分工:Macro-F1 较高通常同时对应较低 MAE 和较高 Pearson;文本 expert 在分类与回归指标上都是最常见的赢家。因此,这项诊断**没有提供增加第二个 router 所需的任务特异模态偏好证据**。目前保留 single-router 作为活动参照;这不代表两个任务在任何数据或设置下都不可能需要不同路由。 + +## 结论范围 + +输入是官方提供的 50 个有序 wordpiece 位置。结果只反映这些位置及本次缺失掩码下的任务与 expert 关系,不表示物理时间可靠性。 + +逐条件结果见 `forced_expert_metrics.csv` 和 `rank_concordance.csv`;跨 seed 汇总见 `expert_task_preference_summary.csv`。 diff --git a/deep_learning/Q2/outputs/algorithm_selection/aligned_robust_stats.npz b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/aligned_robust_stats.npz similarity index 100% rename from deep_learning/Q2/outputs/algorithm_selection/aligned_robust_stats.npz rename to deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/aligned_robust_stats.npz diff --git a/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/expert_condition_matrix.csv b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/expert_condition_matrix.csv new file mode 100644 index 0000000..be5abb8 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/expert_condition_matrix.csv @@ -0,0 +1,337 @@ +method,seed,condition,expert,available_position_fraction,accuracy,macro_f1,mae,pearson +B5_mofe_mlp,42,clean,T,0.5117582417582418,0.6304945054945055,0.595530036337148,0.6439222097396851,0.6129420612443442 +B5_mofe_mlp,42,clean,A,0.4704945054945055,0.47527472527472525,0.298145070470697,0.7733861207962036,0.18784968852824 +B5_mofe_mlp,42,clean,V,0.44483516483516483,0.5027472527472527,0.40175895555984303,0.766243577003479,0.24655657337265044 +B5_mofe_mlp,42,clean,TA,0.4704945054945055,0.6126373626373627,0.5778441259738553,0.6754669547080994,0.5976613672709937 +B5_mofe_mlp,42,clean,TV,0.44483516483516483,0.5659340659340659,0.5313464424314036,0.6711714863777161,0.5439695555081324 +B5_mofe_mlp,42,clean,AV,0.44483516483516483,0.39972527472527475,0.32010564554339865,0.8058619499206543,0.2853473823193152 +B5_mofe_mlp,42,clean,TAV,0.44483516483516483,0.5192307692307693,0.5075932022075595,0.7201082110404968,0.5295944326442668 +B5_mofe_mlp,42,text_10,T,0.46244505494505495,0.625,0.586687579750888,0.6396908164024353,0.6069353393592744 +B5_mofe_mlp,42,text_10,A,0.4704945054945055,0.4725274725274725,0.29847247595086945,0.7734847068786621,0.1872761811347766 +B5_mofe_mlp,42,text_10,V,0.44483516483516483,0.5013736263736264,0.40093402112196824,0.7681402564048767,0.23859029167275192 +B5_mofe_mlp,42,text_10,TA,0.4237362637362637,0.6043956043956044,0.5640906610207688,0.6682833433151245,0.590374645148395 +B5_mofe_mlp,42,text_10,TV,0.4011813186813187,0.5590659340659341,0.5191159133025295,0.6736815571784973,0.5364919274641561 +B5_mofe_mlp,42,text_10,AV,0.44483516483516483,0.39697802197802196,0.31798635878496145,0.8078023195266724,0.2802269980427409 +B5_mofe_mlp,42,text_10,TAV,0.4011813186813187,0.5233516483516484,0.5067270736643431,0.7274370193481445,0.5169178278994004 +B5_mofe_mlp,42,audio_10,T,0.5117582417582418,0.6291208791208791,0.5945275158931668,0.642977774143219,0.613275492732102 +B5_mofe_mlp,42,audio_10,A,0.4218681318681319,0.5206043956043956,0.3800213409408812,0.7459056973457336,0.3452087882296117 +B5_mofe_mlp,42,audio_10,V,0.44483516483516483,0.5027472527472527,0.40304222030887615,0.7655854225158691,0.2472652676368147 +B5_mofe_mlp,42,audio_10,TA,0.4218681318681319,0.6208791208791209,0.5887443905119517,0.6653785109519958,0.6021205001293886 +B5_mofe_mlp,42,audio_10,TV,0.44483516483516483,0.5645604395604396,0.5312885900336469,0.6704310178756714,0.5439517055201103 +B5_mofe_mlp,42,audio_10,AV,0.39903846153846156,0.4642857142857143,0.3849597770559165,0.7626285552978516,0.395749971509581 +B5_mofe_mlp,42,audio_10,TAV,0.39903846153846156,0.5343406593406593,0.5227392208880216,0.6968365907669067,0.5554522946329644 +B5_mofe_mlp,42,vision_10,T,0.5117582417582418,0.6291208791208791,0.5943509173547352,0.6446217894554138,0.6129818568677179 +B5_mofe_mlp,42,vision_10,A,0.4704945054945055,0.4766483516483517,0.2989428209119208,0.7728794813156128,0.18769972077258862 +B5_mofe_mlp,42,vision_10,V,0.39766483516483514,0.521978021978022,0.43087711258901057,0.7396351099014282,0.3554675683918263 +B5_mofe_mlp,42,vision_10,TA,0.4704945054945055,0.6098901098901099,0.5742147768264858,0.6766067743301392,0.5972162914035549 +B5_mofe_mlp,42,vision_10,TV,0.39766483516483514,0.5865384615384616,0.55296383612865,0.6538858413696289,0.5634815784773998 +B5_mofe_mlp,42,vision_10,AV,0.39766483516483514,0.4697802197802198,0.3822322738135533,0.7551643252372742,0.38427665028451974 +B5_mofe_mlp,42,vision_10,TAV,0.39766483516483514,0.5494505494505495,0.5360678828855083,0.6838967204093933,0.5509730603354713 +B5_mofe_mlp,42,audio_vision_10,T,0.5117582417582418,0.6304945054945055,0.5962642544403453,0.6440325975418091,0.6128134692505169 +B5_mofe_mlp,42,audio_vision_10,A,0.421978021978022,0.5123626373626373,0.3684800611034152,0.7456803917884827,0.33199118394826943 +B5_mofe_mlp,42,audio_vision_10,V,0.3991758241758242,0.5288461538461539,0.4393232991026214,0.7295649647712708,0.3589533434031505 +B5_mofe_mlp,42,audio_vision_10,TA,0.421978021978022,0.6071428571428571,0.5708085735541816,0.6724270582199097,0.6024973312871745 +B5_mofe_mlp,42,audio_vision_10,TV,0.3991758241758242,0.5824175824175825,0.5450890570908877,0.652445375919342,0.5628683926779028 +B5_mofe_mlp,42,audio_vision_10,AV,0.3991758241758242,0.4807692307692308,0.3902098705265415,0.7563263773918152,0.3809814529152993 +B5_mofe_mlp,42,audio_vision_10,TAV,0.3991758241758242,0.5508241758241759,0.5380789099976825,0.6938034892082214,0.5469244359251345 +B5_mofe_mlp,42,all_modalities_10,T,0.4626098901098901,0.6236263736263736,0.5896180574655738,0.644334077835083,0.6061671603432021 +B5_mofe_mlp,42,all_modalities_10,A,0.4235989010989011,0.47802197802197804,0.29851508448322833,0.7703589200973511,0.19932721022116437 +B5_mofe_mlp,42,all_modalities_10,V,0.4010164835164835,0.5096153846153846,0.40743192506208387,0.7642088532447815,0.2536261885599537 +B5_mofe_mlp,42,all_modalities_10,TA,0.4235989010989011,0.6071428571428571,0.5695040612711005,0.6758896708488464,0.5936468866005145 +B5_mofe_mlp,42,all_modalities_10,TV,0.4010164835164835,0.5769230769230769,0.5415381488196198,0.6721597909927368,0.5447657707430857 +B5_mofe_mlp,42,all_modalities_10,AV,0.4010164835164835,0.4024725274725275,0.3202895136648401,0.8006964921951294,0.28858268928350844 +B5_mofe_mlp,42,all_modalities_10,TAV,0.4010164835164835,0.5192307692307693,0.5060589429054257,0.7209683060646057,0.5279374148247811 +B5_mofe_mlp,42,text_20,T,0.4137362637362637,0.6277472527472527,0.5834715774578197,0.6387249827384949,0.595955769366048 +B5_mofe_mlp,42,text_20,A,0.4704945054945055,0.47115384615384615,0.2975569493673755,0.7751386761665344,0.17787620014502312 +B5_mofe_mlp,42,text_20,V,0.44483516483516483,0.49313186813186816,0.3957467065496128,0.7714186310768127,0.22587158419370937 +B5_mofe_mlp,42,text_20,TA,0.3776923076923077,0.6043956043956044,0.5611937927949194,0.6714186668395996,0.5678669912027287 +B5_mofe_mlp,42,text_20,TV,0.3565384615384615,0.5686813186813187,0.5239313100098085,0.681026816368103,0.5269364291198027 +B5_mofe_mlp,42,text_20,AV,0.44483516483516483,0.39972527472527475,0.31990146004230513,0.8130425810813904,0.2707806455648105 +B5_mofe_mlp,42,text_20,TAV,0.3565384615384615,0.5192307692307693,0.49528792280719197,0.7405415773391724,0.5088571803296038 +B5_mofe_mlp,42,audio_20,T,0.5117582417582418,0.6277472527472527,0.5934285449238298,0.6425583362579346,0.61296028794768 +B5_mofe_mlp,42,audio_20,A,0.3777747252747253,0.5370879120879121,0.4088556651206101,0.721038818359375,0.4088036003378612 +B5_mofe_mlp,42,audio_20,V,0.44483516483516483,0.5027472527472527,0.4043940692076286,0.766434907913208,0.2463825958649145 +B5_mofe_mlp,42,audio_20,TA,0.3777747252747253,0.6291208791208791,0.5983548479511946,0.6565340757369995,0.6034849351932553 +B5_mofe_mlp,42,audio_20,TV,0.44483516483516483,0.5618131868131868,0.5278821116181079,0.6707059144973755,0.5439704154498037 +B5_mofe_mlp,42,audio_20,AV,0.356978021978022,0.49175824175824173,0.4044220197597692,0.7304497957229614,0.46939736345936744 +B5_mofe_mlp,42,audio_20,TAV,0.356978021978022,0.554945054945055,0.5389342277853322,0.6833800673484802,0.5631089758912269 +B5_mofe_mlp,42,vision_20,T,0.5117582417582418,0.6304945054945055,0.595665938533476,0.645626425743103,0.6129211456979007 +B5_mofe_mlp,42,vision_20,A,0.4704945054945055,0.4739010989010989,0.294076640863259,0.772672712802887,0.1917203096512453 +B5_mofe_mlp,42,vision_20,V,0.3548901098901099,0.5494505494505495,0.4673960507937731,0.7191774845123291,0.4004458766947187 +B5_mofe_mlp,42,vision_20,TA,0.4704945054945055,0.6071428571428571,0.5708690749466393,0.6775083541870117,0.5974902403406877 +B5_mofe_mlp,42,vision_20,TV,0.3548901098901099,0.6016483516483516,0.5649164229707409,0.647191047668457,0.5713132358596449 +B5_mofe_mlp,42,vision_20,AV,0.3548901098901099,0.5137362637362637,0.43167042702402014,0.7291663289070129,0.41618639794585793 +B5_mofe_mlp,42,vision_20,TAV,0.3548901098901099,0.5741758241758241,0.5579426783722908,0.6710066199302673,0.5582889096221613 +B5_mofe_mlp,42,audio_vision_20,T,0.5117582417582418,0.6291208791208791,0.5952612725131757,0.6444123387336731,0.612903588247494 +B5_mofe_mlp,42,audio_vision_20,A,0.37423076923076926,0.5288461538461539,0.393885105990035,0.7293429374694824,0.3882759036052619 +B5_mofe_mlp,42,audio_vision_20,V,0.3539285714285714,0.5494505494505495,0.4675111779446084,0.7149078249931335,0.41871280252003096 +B5_mofe_mlp,42,audio_vision_20,TA,0.37423076923076926,0.6195054945054945,0.5844559446078783,0.669805645942688,0.6008561243242124 +B5_mofe_mlp,42,audio_vision_20,TV,0.3539285714285714,0.5892857142857143,0.5537996616712025,0.6531057953834534,0.5563709108545314 +B5_mofe_mlp,42,audio_vision_20,AV,0.3539285714285714,0.4945054945054945,0.4048075528799828,0.7323312163352966,0.42436332585578096 +B5_mofe_mlp,42,audio_vision_20,TAV,0.3539285714285714,0.5535714285714286,0.5374035508293326,0.6750969290733337,0.5556165429960925 +B5_mofe_mlp,42,all_modalities_20,T,0.41145604395604396,0.6112637362637363,0.5753999256800798,0.6470812559127808,0.6110403126351307 +B5_mofe_mlp,42,all_modalities_20,A,0.37502747252747254,0.48214285714285715,0.30971805746653613,0.7708897590637207,0.1984384514917167 +B5_mofe_mlp,42,all_modalities_20,V,0.35456043956043953,0.5,0.40444384179778897,0.768601655960083,0.24785317295609371 +B5_mofe_mlp,42,all_modalities_20,TA,0.37502747252747254,0.6126373626373627,0.581822149187397,0.6791433095932007,0.5855706598675895 +B5_mofe_mlp,42,all_modalities_20,TV,0.35456043956043953,0.5714285714285714,0.5350638931866368,0.6752636432647705,0.539181947101854 +B5_mofe_mlp,42,all_modalities_20,AV,0.35456043956043953,0.4065934065934066,0.33361946481005195,0.7985524535179138,0.28911162438768584 +B5_mofe_mlp,42,all_modalities_20,TAV,0.35456043956043953,0.5137362637362637,0.5016717439879469,0.723558783531189,0.522880930319737 +B5_mofe_mlp,42,text_30,T,0.36975274725274726,0.6002747252747253,0.5489214966247952,0.6610613465309143,0.5660732620520418 +B5_mofe_mlp,42,text_30,A,0.4704945054945055,0.4739010989010989,0.29743008078567407,0.7761889696121216,0.1678484467957318 +B5_mofe_mlp,42,text_30,V,0.44483516483516483,0.489010989010989,0.3906705384925391,0.7730352282524109,0.21533832545300696 +B5_mofe_mlp,42,text_30,TA,0.3367857142857143,0.5769230769230769,0.5309590282919976,0.6845227479934692,0.5380880477965313 +B5_mofe_mlp,42,text_30,TV,0.31706043956043956,0.5563186813186813,0.5074245114833101,0.6991908550262451,0.5079541484193177 +B5_mofe_mlp,42,text_30,AV,0.44483516483516483,0.38873626373626374,0.3106438181055198,0.8153451681137085,0.25647904474699446 +B5_mofe_mlp,42,text_30,TAV,0.31706043956043956,0.510989010989011,0.48202998092174165,0.7513630390167236,0.4807588760950029 +B5_mofe_mlp,42,audio_30,T,0.5117582417582418,0.6304945054945055,0.5962642544403453,0.6413891911506653,0.6140593651176108 +B5_mofe_mlp,42,audio_30,A,0.3262087912087912,0.5480769230769231,0.4494048835498772,0.7181517481803894,0.4130354529119704 +B5_mofe_mlp,42,audio_30,V,0.44483516483516483,0.5027472527472527,0.40552179812400446,0.7659019231796265,0.24744831420819957 +B5_mofe_mlp,42,audio_30,TA,0.3262087912087912,0.6291208791208791,0.5964728532922603,0.6501736640930176,0.6030689771166702 +B5_mofe_mlp,42,audio_30,TV,0.44483516483516483,0.5618131868131868,0.5278372031251927,0.6705803871154785,0.5438638322507882 +B5_mofe_mlp,42,audio_30,AV,0.3069230769230769,0.49862637362637363,0.42544608223551644,0.7236812114715576,0.45260841133749147 +B5_mofe_mlp,42,audio_30,TAV,0.3069230769230769,0.5673076923076923,0.5511576680296482,0.6794567704200745,0.5516787374683813 +B5_mofe_mlp,42,vision_30,T,0.5117582417582418,0.6263736263736264,0.59178963181016,0.6464172601699829,0.6124405738784353 +B5_mofe_mlp,42,vision_30,A,0.4704945054945055,0.47527472527472525,0.29307320983548674,0.7725724577903748,0.18981334780850148 +B5_mofe_mlp,42,vision_30,V,0.3138186813186813,0.5480769230769231,0.46141777713440435,0.7053897380828857,0.4504351822566667 +B5_mofe_mlp,42,vision_30,TA,0.4704945054945055,0.6112637362637363,0.5755727285311548,0.6785207986831665,0.5965959233259053 +B5_mofe_mlp,42,vision_30,TV,0.3138186813186813,0.5947802197802198,0.5518425273811883,0.6574218273162842,0.5668784431581148 +B5_mofe_mlp,42,vision_30,AV,0.3138186813186813,0.5027472527472527,0.413102983693412,0.7221376299858093,0.4423578428565793 +B5_mofe_mlp,42,vision_30,TAV,0.3138186813186813,0.5810439560439561,0.5627987997604266,0.6714197397232056,0.5504939616109109 +B5_mofe_mlp,42,audio_vision_30,T,0.5117582417582418,0.6291208791208791,0.5956657591699687,0.6447409987449646,0.6124815887059187 +B5_mofe_mlp,42,audio_vision_30,A,0.32593406593406593,0.5412087912087912,0.4293427018964002,0.7119898796081543,0.4611692059662869 +B5_mofe_mlp,42,audio_vision_30,V,0.308489010989011,0.5741758241758241,0.5081685315054737,0.7036085724830627,0.46865245226218855 +B5_mofe_mlp,42,audio_vision_30,TA,0.32593406593406593,0.6167582417582418,0.5796313859552673,0.6652615666389465,0.6066108477643619 +B5_mofe_mlp,42,audio_vision_30,TV,0.308489010989011,0.6085164835164835,0.5747678972005031,0.6532444953918457,0.5685188687826935 +B5_mofe_mlp,42,audio_vision_30,AV,0.308489010989011,0.5178571428571429,0.4479560935960237,0.704010009765625,0.473760873249426 +B5_mofe_mlp,42,audio_vision_30,TAV,0.308489010989011,0.5769230769230769,0.5590219753958139,0.6671830415725708,0.5613400189071075 +B5_mofe_mlp,42,all_modalities_30,T,0.3595054945054945,0.6332417582417582,0.5994013685816965,0.642018735408783,0.6142476754248422 +B5_mofe_mlp,42,all_modalities_30,A,0.32604395604395603,0.4876373626373626,0.32114976052878363,0.7629672288894653,0.24577614371604897 +B5_mofe_mlp,42,all_modalities_30,V,0.30766483516483517,0.5137362637362637,0.41564992061683226,0.7619248032569885,0.2628075288162841 +B5_mofe_mlp,42,all_modalities_30,TA,0.32604395604395603,0.6277472527472527,0.5901208902602798,0.6717713475227356,0.6055638351449961 +B5_mofe_mlp,42,all_modalities_30,TV,0.30766483516483517,0.5851648351648352,0.550384612243989,0.6579669713973999,0.5550816637819891 +B5_mofe_mlp,42,all_modalities_30,AV,0.30766483516483517,0.43131868131868134,0.3524060719327709,0.7886311411857605,0.2950171215037075 +B5_mofe_mlp,42,all_modalities_30,TAV,0.30766483516483517,0.5206043956043956,0.5084554432330729,0.7108278274536133,0.5347221240292945 +B5_mofe_mlp,3407,clean,T,0.5117582417582418,0.6002747252747253,0.589715106019248,0.6271728873252869,0.6222406977786201 +B5_mofe_mlp,3407,clean,A,0.4704945054945055,0.46703296703296704,0.25617731189594245,0.8055213093757629,0.08527244746812068 +B5_mofe_mlp,3407,clean,V,0.44483516483516483,0.45879120879120877,0.3911083700819353,0.7868512272834778,0.27334952956888975 +B5_mofe_mlp,3407,clean,TA,0.4704945054945055,0.5796703296703297,0.5637105051146315,0.6420981884002686,0.5835249053178219 +B5_mofe_mlp,3407,clean,TV,0.44483516483516483,0.6153846153846154,0.5641846149831432,0.6430085301399231,0.5929148218414205 +B5_mofe_mlp,3407,clean,AV,0.44483516483516483,0.3791208791208791,0.31403029968176926,0.8854933381080627,0.2789162730797137 +B5_mofe_mlp,3407,clean,TAV,0.44483516483516483,0.5879120879120879,0.5681219195313041,0.6709685921669006,0.5645502116442188 +B5_mofe_mlp,3407,text_10,T,0.46123626373626375,0.614010989010989,0.5962559055831377,0.6278678774833679,0.617857871648632 +B5_mofe_mlp,3407,text_10,A,0.4704945054945055,0.4642857142857143,0.24969978803929094,0.8068049550056458,0.08051328125107551 +B5_mofe_mlp,3407,text_10,V,0.44483516483516483,0.4574175824175824,0.3878406396990897,0.7909942269325256,0.26236773636559957 +B5_mofe_mlp,3407,text_10,TA,0.42271978021978024,0.5879120879120879,0.5646643159293967,0.6490334868431091,0.5744751061732449 +B5_mofe_mlp,3407,text_10,TV,0.39942307692307694,0.6085164835164835,0.5474885374714241,0.6448302268981934,0.5895230158639547 +B5_mofe_mlp,3407,text_10,AV,0.44483516483516483,0.37774725274725274,0.3110187465790914,0.8889136910438538,0.2696459968528376 +B5_mofe_mlp,3407,text_10,TAV,0.39942307692307694,0.5934065934065934,0.563693177045021,0.6679768562316895,0.5622421334799538 +B5_mofe_mlp,3407,audio_10,T,0.5117582417582418,0.5989010989010989,0.5878169018954168,0.6271204948425293,0.6223836004743148 +B5_mofe_mlp,3407,audio_10,A,0.4228021978021978,0.5027472527472527,0.332656408166132,0.7686276435852051,0.2539853638071684 +B5_mofe_mlp,3407,audio_10,V,0.44483516483516483,0.45604395604395603,0.38920601362461826,0.7875473499298096,0.27266225636799185 +B5_mofe_mlp,3407,audio_10,TA,0.4228021978021978,0.5947802197802198,0.5806242036699651,0.6359114050865173,0.5923569096429423 +B5_mofe_mlp,3407,audio_10,TV,0.44483516483516483,0.6167582417582418,0.5663217661957952,0.6429393887519836,0.5927651982092371 +B5_mofe_mlp,3407,audio_10,AV,0.3992857142857143,0.44505494505494503,0.40079404466501245,0.8398518562316895,0.37350790006416895 +B5_mofe_mlp,3407,audio_10,TAV,0.3992857142857143,0.5879120879120879,0.5677938460265421,0.6635177135467529,0.5747748813576409 +B5_mofe_mlp,3407,vision_10,T,0.5117582417582418,0.5989010989010989,0.5881686695855791,0.6271764636039734,0.6224112877034929 +B5_mofe_mlp,3407,vision_10,A,0.4704945054945055,0.4697802197802198,0.2617495652772756,0.8053902387619019,0.08562714494931029 +B5_mofe_mlp,3407,vision_10,V,0.39975274725274723,0.5164835164835165,0.4530382033974704,0.7460650205612183,0.3701696777824128 +B5_mofe_mlp,3407,vision_10,TA,0.4704945054945055,0.5810439560439561,0.5652874494272252,0.6420679688453674,0.5835241403957903 +B5_mofe_mlp,3407,vision_10,TV,0.39975274725274723,0.6208791208791209,0.5716893090185614,0.6368684768676758,0.5996522472016457 +B5_mofe_mlp,3407,vision_10,AV,0.39975274725274723,0.4340659340659341,0.3798771393241414,0.8302299380302429,0.366026727245457 +B5_mofe_mlp,3407,vision_10,TAV,0.39975274725274723,0.5975274725274725,0.5773266951804078,0.6591137051582336,0.573922934425617 +B5_mofe_mlp,3407,audio_vision_10,T,0.5117582417582418,0.6002747252747253,0.5895426241854899,0.6275323629379272,0.6220930974326538 +B5_mofe_mlp,3407,audio_vision_10,A,0.421978021978022,0.4793956043956044,0.2964038513186514,0.7789896726608276,0.2398897695596731 +B5_mofe_mlp,3407,audio_vision_10,V,0.3987912087912088,0.4945054945054945,0.44061682531723784,0.7518566846847534,0.38529247892112506 +B5_mofe_mlp,3407,audio_vision_10,TA,0.421978021978022,0.5906593406593407,0.5759372217914477,0.6386086344718933,0.5889149782654084 +B5_mofe_mlp,3407,audio_vision_10,TV,0.3987912087912088,0.6195054945054945,0.5730831708901883,0.6402207016944885,0.5971608635562845 +B5_mofe_mlp,3407,audio_vision_10,AV,0.3987912087912088,0.4409340659340659,0.39365944462354036,0.8360045552253723,0.37457825919086657 +B5_mofe_mlp,3407,audio_vision_10,TAV,0.3987912087912088,0.5879120879120879,0.5702051410805352,0.6638661623001099,0.5713070790736874 +B5_mofe_mlp,3407,all_modalities_10,T,0.4610164835164835,0.5934065934065934,0.5840003133946626,0.63011234998703,0.6190678344486067 +B5_mofe_mlp,3407,all_modalities_10,A,0.4225274725274725,0.47115384615384615,0.2646409174971828,0.8042468428611755,0.08459595129973409 +B5_mofe_mlp,3407,all_modalities_10,V,0.39947802197802196,0.46016483516483514,0.39465376804402846,0.7862517237663269,0.2773313002639839 +B5_mofe_mlp,3407,all_modalities_10,TA,0.4225274725274725,0.5824175824175825,0.5671818574180514,0.6445116400718689,0.5822325816333559 +B5_mofe_mlp,3407,all_modalities_10,TV,0.39947802197802196,0.6043956043956044,0.5508070001152509,0.6478140354156494,0.5896831136236704 +B5_mofe_mlp,3407,all_modalities_10,AV,0.39947802197802196,0.3873626373626374,0.32667745632365675,0.8781698942184448,0.28384183315332934 +B5_mofe_mlp,3407,all_modalities_10,TAV,0.39947802197802196,0.5796703296703297,0.5603794485883429,0.6692947745323181,0.5686062320633599 +B5_mofe_mlp,3407,text_20,T,0.4073076923076923,0.6016483516483516,0.5744089403315806,0.6390801072120667,0.6021358626478962 +B5_mofe_mlp,3407,text_20,A,0.4704945054945055,0.4642857142857143,0.2494552539431357,0.8059768080711365,0.07746180938998821 +B5_mofe_mlp,3407,text_20,V,0.44483516483516483,0.45467032967032966,0.38380648966660313,0.793641984462738,0.24830688010478227 +B5_mofe_mlp,3407,text_20,TA,0.37112637362637363,0.5741758241758241,0.5418661031211773,0.6627365350723267,0.5604770780530056 +B5_mofe_mlp,3407,text_20,TV,0.3507142857142857,0.603021978021978,0.5357923985725986,0.6560192704200745,0.5732426799450642 +B5_mofe_mlp,3407,text_20,AV,0.44483516483516483,0.3791208791208791,0.3126586105031628,0.8910257816314697,0.25653375730622274 +B5_mofe_mlp,3407,text_20,TAV,0.3507142857142857,0.5934065934065934,0.5604543054498898,0.6754387021064758,0.5450045102927684 +B5_mofe_mlp,3407,audio_20,T,0.5117582417582418,0.6002747252747253,0.5893745342941709,0.6272591948509216,0.6221016794037438 +B5_mofe_mlp,3407,audio_20,A,0.37486263736263736,0.4945054945054945,0.3373484461000103,0.7447602152824402,0.3567553342617643 +B5_mofe_mlp,3407,audio_20,V,0.44483516483516483,0.46016483516483514,0.3944283930013793,0.7885401844978333,0.2723842014763785 +B5_mofe_mlp,3407,audio_20,TA,0.37486263736263736,0.5975274725274725,0.5831680958487365,0.6350199580192566,0.5940990561424356 +B5_mofe_mlp,3407,audio_20,TV,0.44483516483516483,0.6167582417582418,0.5667282248122798,0.6437039971351624,0.5927168852036161 +B5_mofe_mlp,3407,audio_20,AV,0.354010989010989,0.4697802197802198,0.42693867646678624,0.8040833473205566,0.41986361746775985 +B5_mofe_mlp,3407,audio_20,TAV,0.354010989010989,0.6002747252747253,0.5813735202449936,0.6587744355201721,0.5783158306220523 +B5_mofe_mlp,3407,vision_20,T,0.5117582417582418,0.6002747252747253,0.5892252758196194,0.6273123025894165,0.6225084305561861 +B5_mofe_mlp,3407,vision_20,A,0.4704945054945055,0.4684065934065934,0.25920850508591314,0.8053038716316223,0.0857924968918223 +B5_mofe_mlp,3407,vision_20,V,0.3525,0.5288461538461539,0.4720058888479941,0.7252842783927917,0.41317316472128224 +B5_mofe_mlp,3407,vision_20,TA,0.4704945054945055,0.5824175824175825,0.5670673740607254,0.6420119404792786,0.5840681638144474 +B5_mofe_mlp,3407,vision_20,TV,0.3525,0.625,0.5781351467902999,0.6339039206504822,0.5999731536004191 +B5_mofe_mlp,3407,vision_20,AV,0.3525,0.47802197802197804,0.4364142415752362,0.7915684580802917,0.4020387452763518 +B5_mofe_mlp,3407,vision_20,TAV,0.3525,0.6181318681318682,0.5982469512967439,0.656253457069397,0.5726154591849658 +B5_mofe_mlp,3407,audio_vision_20,T,0.5117582417582418,0.5989010989010989,0.5883115478052187,0.627071738243103,0.6227612439489552 +B5_mofe_mlp,3407,audio_vision_20,A,0.36782967032967034,0.5123626373626373,0.3814657210401891,0.7518494725227356,0.3353158204477575 +B5_mofe_mlp,3407,audio_vision_20,V,0.3472802197802198,0.5247252747252747,0.4843248302906597,0.7358435392379761,0.4214120030770159 +B5_mofe_mlp,3407,audio_vision_20,TA,0.36782967032967034,0.6002747252747253,0.5892792851694453,0.628782331943512,0.6006774631204904 +B5_mofe_mlp,3407,audio_vision_20,TV,0.3472802197802198,0.6332417582417582,0.5964419732456347,0.6303027868270874,0.6076400985344274 +B5_mofe_mlp,3407,audio_vision_20,AV,0.3472802197802198,0.46565934065934067,0.4355944055944056,0.8029803037643433,0.41656408552542945 +B5_mofe_mlp,3407,audio_vision_20,TAV,0.3472802197802198,0.6002747252747253,0.5862363095320736,0.6544462442398071,0.585378630597988 +B5_mofe_mlp,3407,all_modalities_20,T,0.40958791208791206,0.6002747252747253,0.5899048320687862,0.6346542239189148,0.61001396406442 +B5_mofe_mlp,3407,all_modalities_20,A,0.37326923076923074,0.4739010989010989,0.27165625210557365,0.8040055632591248,0.10424718487552988 +B5_mofe_mlp,3407,all_modalities_20,V,0.35244505494505496,0.4642857142857143,0.40252151018656984,0.7802431583404541,0.29135083510415233 +B5_mofe_mlp,3407,all_modalities_20,TA,0.37326923076923074,0.5686813186813187,0.5572575422788313,0.6487621665000916,0.5725190946033569 +B5_mofe_mlp,3407,all_modalities_20,TV,0.35244505494505496,0.6126373626373627,0.5629491230638092,0.649815022945404,0.5810131620167055 +B5_mofe_mlp,3407,all_modalities_20,AV,0.35244505494505496,0.3956043956043956,0.3405710346065853,0.8701844215393066,0.3057071868398566 +B5_mofe_mlp,3407,all_modalities_20,TAV,0.35244505494505496,0.5837912087912088,0.5631923344724951,0.6739726066589355,0.559728652392748 +B5_mofe_mlp,3407,text_30,T,0.36063186813186815,0.6112637362637363,0.5806932321558671,0.6485379934310913,0.5795620041997364 +B5_mofe_mlp,3407,text_30,A,0.4704945054945055,0.46703296703296704,0.2536900771342752,0.8093824982643127,0.062449346059579636 +B5_mofe_mlp,3407,text_30,V,0.44483516483516483,0.4409340659340659,0.3710188660549014,0.7965743541717529,0.24688085844704077 +B5_mofe_mlp,3407,text_30,TA,0.32774725274725275,0.592032967032967,0.5537539943729123,0.6646007299423218,0.539540874633729 +B5_mofe_mlp,3407,text_30,TV,0.3087362637362637,0.6016483516483516,0.5385391277047284,0.661797046661377,0.553382704054472 +B5_mofe_mlp,3407,text_30,AV,0.44483516483516483,0.37225274725274726,0.3029182824443429,0.8928881883621216,0.2563737225632936 +B5_mofe_mlp,3407,text_30,TAV,0.3087362637362637,0.5865384615384616,0.5490273643806375,0.6738437414169312,0.5345429616136375 +B5_mofe_mlp,3407,audio_30,T,0.5117582417582418,0.6002747252747253,0.5893745342941709,0.6269524693489075,0.6225361454710463 +B5_mofe_mlp,3407,audio_30,A,0.3245879120879121,0.532967032967033,0.41756519321424274,0.7341850399971008,0.4112199578484464 +B5_mofe_mlp,3407,audio_30,V,0.44483516483516483,0.45054945054945056,0.3832106337593439,0.7885648012161255,0.2735580453202714 +B5_mofe_mlp,3407,audio_30,TA,0.3245879120879121,0.6085164835164835,0.5930238521146932,0.6325163841247559,0.5979866887810961 +B5_mofe_mlp,3407,audio_30,TV,0.44483516483516483,0.6112637362637363,0.5583419641348294,0.6421341896057129,0.5937514392972938 +B5_mofe_mlp,3407,audio_30,AV,0.3048351648351648,0.5,0.4623444097193838,0.7733940482139587,0.4553589708631863 +B5_mofe_mlp,3407,audio_30,TAV,0.3048351648351648,0.614010989010989,0.5938581135634877,0.6490943431854248,0.5943926321157222 +B5_mofe_mlp,3407,vision_30,T,0.5117582417582418,0.5989010989010989,0.5879942456720697,0.6273198127746582,0.6227160564904063 +B5_mofe_mlp,3407,vision_30,A,0.4704945054945055,0.4697802197802198,0.26181694968773733,0.8051190972328186,0.08667352225368381 +B5_mofe_mlp,3407,vision_30,V,0.3087087912087912,0.5233516483516484,0.461764143838227,0.7223021984100342,0.42618344773653233 +B5_mofe_mlp,3407,vision_30,TA,0.4704945054945055,0.5824175824175825,0.5672280279314932,0.6419575810432434,0.5838877628981125 +B5_mofe_mlp,3407,vision_30,TV,0.3087087912087912,0.6112637362637363,0.5662069378393744,0.641978919506073,0.5924386628826628 +B5_mofe_mlp,3407,vision_30,AV,0.3087087912087912,0.49862637362637363,0.445456810889948,0.7776786088943481,0.4216009067968943 +B5_mofe_mlp,3407,vision_30,TAV,0.3087087912087912,0.6071428571428571,0.5854633810737848,0.6539670825004578,0.575092050206525 +B5_mofe_mlp,3407,audio_vision_30,T,0.5117582417582418,0.6002747252747253,0.5893745342941709,0.627665102481842,0.6221384257612228 +B5_mofe_mlp,3407,audio_vision_30,A,0.3249725274725275,0.5425824175824175,0.43415828421831115,0.7195268273353577,0.42527940515455587 +B5_mofe_mlp,3407,audio_vision_30,V,0.3061813186813187,0.5521978021978022,0.5104487621775328,0.7024397253990173,0.48245172490908617 +B5_mofe_mlp,3407,audio_vision_30,TA,0.3249725274725275,0.6085164835164835,0.5960454962258536,0.6273004412651062,0.6054657619733786 +B5_mofe_mlp,3407,audio_vision_30,TV,0.3061813186813187,0.6195054945054945,0.5821910607231708,0.6338788866996765,0.6053650582969483 +B5_mofe_mlp,3407,audio_vision_30,AV,0.3061813186813187,0.510989010989011,0.4810449552418117,0.7675954103469849,0.4618091165069954 +B5_mofe_mlp,3407,audio_vision_30,TAV,0.3061813186813187,0.6085164835164835,0.5928020993486814,0.6524761319160461,0.5899471775250983 +B5_mofe_mlp,3407,all_modalities_30,T,0.3557967032967033,0.6057692307692307,0.5973699542180001,0.636671781539917,0.6031831013481838 +B5_mofe_mlp,3407,all_modalities_30,A,0.32255494505494503,0.47802197802197804,0.2858992684718562,0.7997275590896606,0.12003229290681895 +B5_mofe_mlp,3407,all_modalities_30,V,0.3045879120879121,0.49175824175824173,0.4332501168084348,0.7785677909851074,0.28114128593778215 +B5_mofe_mlp,3407,all_modalities_30,TA,0.32255494505494503,0.5755494505494505,0.5618376925923627,0.6488230228424072,0.5646887050197861 +B5_mofe_mlp,3407,all_modalities_30,TV,0.3045879120879121,0.6153846153846154,0.569951909037207,0.6524091362953186,0.5765149523517863 +B5_mofe_mlp,3407,all_modalities_30,AV,0.3045879120879121,0.41208791208791207,0.3552498095654271,0.860482931137085,0.28184827867802903 +B5_mofe_mlp,3407,all_modalities_30,TAV,0.3045879120879121,0.5741758241758241,0.5572916475321001,0.676213800907135,0.5462438459585837 +B5_mofe_mlp,2026,clean,T,0.5117582417582418,0.6126373626373627,0.596773495897157,0.6489740014076233,0.6011464226614753 +B5_mofe_mlp,2026,clean,A,0.4704945054945055,0.4725274725274725,0.2753954425047768,0.8250412344932556,0.2298903607017162 +B5_mofe_mlp,2026,clean,V,0.44483516483516483,0.5151098901098901,0.41609693688045696,0.741955578327179,0.33524135089906154 +B5_mofe_mlp,2026,clean,TA,0.4704945054945055,0.592032967032967,0.5532162058371736,0.6664568185806274,0.559760822402557 +B5_mofe_mlp,2026,clean,TV,0.44483516483516483,0.592032967032967,0.5774206516713903,0.7040248513221741,0.566471089961779 +B5_mofe_mlp,2026,clean,AV,0.44483516483516483,0.3626373626373626,0.34907979624206836,0.8443424105644226,0.23675835533866071 +B5_mofe_mlp,2026,clean,TAV,0.44483516483516483,0.5370879120879121,0.4651554172647243,0.800514817237854,0.5127573095239696 +B5_mofe_mlp,2026,text_10,T,0.4616758241758242,0.6126373626373627,0.5930164703753459,0.6494342088699341,0.5946752236527758 +B5_mofe_mlp,2026,text_10,A,0.4704945054945055,0.4725274725274725,0.27541290783115796,0.8262399435043335,0.22445377406080036 +B5_mofe_mlp,2026,text_10,V,0.44483516483516483,0.5123626373626373,0.41266149205541813,0.7444266080856323,0.3297960027909343 +B5_mofe_mlp,2026,text_10,TA,0.42304945054945053,0.5837912087912088,0.5397729831672579,0.6658428907394409,0.5527226500907891 +B5_mofe_mlp,2026,text_10,TV,0.4,0.592032967032967,0.571111408272237,0.6985096335411072,0.5560292322354257 +B5_mofe_mlp,2026,text_10,AV,0.44483516483516483,0.3585164835164835,0.34496753089852183,0.8458796739578247,0.23343562747276608 +B5_mofe_mlp,2026,text_10,TAV,0.4,0.5384615384615384,0.4615912536399786,0.7843563556671143,0.4964631006483741 +B5_mofe_mlp,2026,audio_10,T,0.5117582417582418,0.6112637362637363,0.594775223911222,0.6496217250823975,0.6011032755471458 +B5_mofe_mlp,2026,audio_10,A,0.4226373626373626,0.489010989010989,0.30986648206415496,0.795183539390564,0.32899907907949955 +B5_mofe_mlp,2026,audio_10,V,0.44483516483516483,0.5164835164835165,0.41734454924869885,0.7415561079978943,0.3356207601336875 +B5_mofe_mlp,2026,audio_10,TA,0.4226373626373626,0.5906593406593407,0.5523948427986319,0.6616610884666443,0.565696879589075 +B5_mofe_mlp,2026,audio_10,TV,0.44483516483516483,0.5906593406593407,0.5756793112725316,0.7050058841705322,0.5670178444167456 +B5_mofe_mlp,2026,audio_10,AV,0.3998076923076923,0.40796703296703296,0.4021505731323723,0.8058488368988037,0.339145168854004 +B5_mofe_mlp,2026,audio_10,TAV,0.3998076923076923,0.5604395604395604,0.5020907193394417,0.7697135210037231,0.5332348529741537 +B5_mofe_mlp,2026,vision_10,T,0.5117582417582418,0.6112637362637363,0.5949280494683542,0.6487315893173218,0.6010584792844983 +B5_mofe_mlp,2026,vision_10,A,0.4704945054945055,0.4725274725274725,0.2753954425047768,0.8233988285064697,0.2309789028597694 +B5_mofe_mlp,2026,vision_10,V,0.39843406593406594,0.5398351648351648,0.43853463696042566,0.714551568031311,0.4295200457733584 +B5_mofe_mlp,2026,vision_10,TA,0.4704945054945055,0.592032967032967,0.5532162058371736,0.6668616533279419,0.5594488174710718 +B5_mofe_mlp,2026,vision_10,TV,0.39843406593406594,0.6016483516483516,0.5819547002883038,0.6787747144699097,0.5779704513615178 +B5_mofe_mlp,2026,vision_10,AV,0.39843406593406594,0.4258241758241758,0.42253912572720753,0.7936479449272156,0.36127841752536083 +B5_mofe_mlp,2026,vision_10,TAV,0.39843406593406594,0.5714285714285714,0.5087225335512859,0.760180652141571,0.5350946573582769 +B5_mofe_mlp,2026,audio_vision_10,T,0.5117582417582418,0.614010989010989,0.5982455160173181,0.6492474675178528,0.6011554930046857 +B5_mofe_mlp,2026,audio_vision_10,A,0.4210714285714286,0.4876373626373626,0.3042744149605441,0.7851645350456238,0.3562020191286223 +B5_mofe_mlp,2026,audio_vision_10,V,0.39807692307692305,0.5439560439560439,0.4559201356747369,0.7165518999099731,0.4228908246742094 +B5_mofe_mlp,2026,audio_vision_10,TA,0.4210714285714286,0.5934065934065934,0.559867581706166,0.6601359844207764,0.5680404697940684 +B5_mofe_mlp,2026,audio_vision_10,TV,0.39807692307692305,0.6002747252747253,0.5842152460266312,0.6859722137451172,0.5754538071328372 +B5_mofe_mlp,2026,audio_vision_10,AV,0.39807692307692305,0.43131868131868134,0.4262759970983259,0.8031644821166992,0.3534621619974544 +B5_mofe_mlp,2026,audio_vision_10,TAV,0.39807692307692305,0.5535714285714286,0.49118433140019535,0.7740561366081238,0.5279378455891054 +B5_mofe_mlp,2026,all_modalities_10,T,0.45884615384615385,0.6098901098901099,0.5927342612347185,0.6530159711837769,0.5924214145617598 +B5_mofe_mlp,2026,all_modalities_10,A,0.4203021978021978,0.4739010989010989,0.2799395224420051,0.8183923363685608,0.2255248405027258 +B5_mofe_mlp,2026,all_modalities_10,V,0.3982967032967033,0.5123626373626373,0.41084897900937084,0.7401790618896484,0.3429018289361354 +B5_mofe_mlp,2026,all_modalities_10,TA,0.4203021978021978,0.603021978021978,0.5636157256230538,0.6699337959289551,0.5545965441192207 +B5_mofe_mlp,2026,all_modalities_10,TV,0.3982967032967033,0.5961538461538461,0.5817349795156427,0.7066670656204224,0.5633093738983236 +B5_mofe_mlp,2026,all_modalities_10,AV,0.3982967032967033,0.36813186813186816,0.35589607464607464,0.8376190066337585,0.24289852502950912 +B5_mofe_mlp,2026,all_modalities_10,TAV,0.3982967032967033,0.5288461538461539,0.46033318738535584,0.7966769933700562,0.5109217307299846 +B5_mofe_mlp,2026,text_20,T,0.4112087912087912,0.6167582417582418,0.5881921764479171,0.6567917466163635,0.5831459100031497 +B5_mofe_mlp,2026,text_20,A,0.4704945054945055,0.47115384615384615,0.27465560798894134,0.8285160064697266,0.217439267929507 +B5_mofe_mlp,2026,text_20,V,0.44483516483516483,0.5096153846153846,0.4084128341513546,0.7475921511650085,0.31692231439961444 +B5_mofe_mlp,2026,text_20,TA,0.37524725274725274,0.603021978021978,0.5486119524164519,0.6673293113708496,0.539250704149763 +B5_mofe_mlp,2026,text_20,TV,0.3546153846153846,0.6112637362637363,0.5843667494564081,0.6891446113586426,0.5445040624635075 +B5_mofe_mlp,2026,text_20,AV,0.44483516483516483,0.3585164835164835,0.3446302940469433,0.8487928509712219,0.21264829535541369 +B5_mofe_mlp,2026,text_20,TAV,0.3546153846153846,0.5425824175824175,0.46587133463364855,0.7634089589118958,0.4821419242332205 +B5_mofe_mlp,2026,audio_20,T,0.5117582417582418,0.614010989010989,0.5983472125624337,0.6503203511238098,0.6009217826600645 +B5_mofe_mlp,2026,audio_20,A,0.3796978021978022,0.5164835164835165,0.3655881236095169,0.7692446112632751,0.3949787935175758 +B5_mofe_mlp,2026,audio_20,V,0.44483516483516483,0.5151098901098901,0.41620583054274807,0.7414965629577637,0.3344494005058201 +B5_mofe_mlp,2026,audio_20,TA,0.3796978021978022,0.5989010989010989,0.5638265304840163,0.6491043567657471,0.5770018901388131 +B5_mofe_mlp,2026,audio_20,TV,0.44483516483516483,0.592032967032967,0.5774181294613681,0.7061969041824341,0.5658046441453625 +B5_mofe_mlp,2026,audio_20,AV,0.3596153846153846,0.4793956043956044,0.4729905027892644,0.7711812257766724,0.39212697776164607 +B5_mofe_mlp,2026,audio_20,TAV,0.3596153846153846,0.5645604395604396,0.509050719882809,0.7490760684013367,0.534134549592931 +B5_mofe_mlp,2026,vision_20,T,0.5117582417582418,0.6126373626373627,0.5965577523042379,0.6482208371162415,0.6014238929804235 +B5_mofe_mlp,2026,vision_20,A,0.4704945054945055,0.4725274725274725,0.2753954425047768,0.8218814730644226,0.2303172585691777 +B5_mofe_mlp,2026,vision_20,V,0.3572802197802198,0.5480769230769231,0.45491017561200736,0.7200025916099548,0.4188711113417921 +B5_mofe_mlp,2026,vision_20,TA,0.4704945054945055,0.592032967032967,0.5532162058371736,0.6667655110359192,0.5596751462352669 +B5_mofe_mlp,2026,vision_20,TV,0.3572802197802198,0.6085164835164835,0.5858009850150654,0.6767599582672119,0.5666763660374662 +B5_mofe_mlp,2026,vision_20,AV,0.3572802197802198,0.45604395604395603,0.45257735286592293,0.7822015881538391,0.3511756724909663 +B5_mofe_mlp,2026,vision_20,TAV,0.3572802197802198,0.5673076923076923,0.5113134010717539,0.7461904883384705,0.5258941930408034 +B5_mofe_mlp,2026,audio_vision_20,T,0.5117582417582418,0.614010989010989,0.5983472125624337,0.6493008136749268,0.6012901521222299 +B5_mofe_mlp,2026,audio_vision_20,A,0.372967032967033,0.5041208791208791,0.35789310705995514,0.7596712112426758,0.41398282191828445 +B5_mofe_mlp,2026,audio_vision_20,V,0.3531593406593407,0.5796703296703297,0.5048183867639185,0.6984941363334656,0.4714365355794056 +B5_mofe_mlp,2026,audio_vision_20,TA,0.372967032967033,0.5892857142857143,0.5573263153954385,0.6522572040557861,0.5764869480931716 +B5_mofe_mlp,2026,audio_vision_20,TV,0.3531593406593407,0.6057692307692307,0.5884441789459711,0.6706872582435608,0.5818979616306791 +B5_mofe_mlp,2026,audio_vision_20,AV,0.3531593406593407,0.4684065934065934,0.465468176106474,0.7679478526115417,0.41865980844292155 +B5_mofe_mlp,2026,audio_vision_20,TAV,0.3531593406593407,0.5728021978021978,0.5181235721920635,0.7481652498245239,0.5424180387206542 +B5_mofe_mlp,2026,all_modalities_20,T,0.40585164835164833,0.6057692307692307,0.5895572046059541,0.6424160003662109,0.5987483157591782 +B5_mofe_mlp,2026,all_modalities_20,A,0.369478021978022,0.47802197802197804,0.2829329028964813,0.8103722929954529,0.2520477595177189 +B5_mofe_mlp,2026,all_modalities_20,V,0.3488186813186813,0.5178571428571429,0.40966513694882706,0.7393193244934082,0.34226531640291846 +B5_mofe_mlp,2026,all_modalities_20,TA,0.369478021978022,0.5975274725274725,0.5569987222986782,0.6711844801902771,0.5602370766890162 +B5_mofe_mlp,2026,all_modalities_20,TV,0.3488186813186813,0.5906593406593407,0.5720218781867062,0.6996389031410217,0.5616554905976145 +B5_mofe_mlp,2026,all_modalities_20,AV,0.3488186813186813,0.37362637362637363,0.3626956690910179,0.8267216682434082,0.25176084156700307 +B5_mofe_mlp,2026,all_modalities_20,TAV,0.3488186813186813,0.5453296703296703,0.4749388892268766,0.7958878874778748,0.5090043666084783 +B5_mofe_mlp,2026,text_30,T,0.3585714285714286,0.6002747252747253,0.5650627890671912,0.6760544776916504,0.5486430219432402 +B5_mofe_mlp,2026,text_30,A,0.4704945054945055,0.4684065934065934,0.2694917313065056,0.8302391171455383,0.21220047686679583 +B5_mofe_mlp,2026,text_30,V,0.44483516483516483,0.5041208791208791,0.40060701323330045,0.7515881657600403,0.30841065019819836 +B5_mofe_mlp,2026,text_30,TA,0.32582417582417583,0.5824175824175825,0.5246945241369286,0.6926639080047607,0.49106579731378935 +B5_mofe_mlp,2026,text_30,TV,0.3089010989010989,0.5934065934065934,0.5598344950469888,0.7086330056190491,0.49570849701649317 +B5_mofe_mlp,2026,text_30,AV,0.44483516483516483,0.35714285714285715,0.3435056224920738,0.8512967228889465,0.1985224132656547 +B5_mofe_mlp,2026,text_30,TAV,0.3089010989010989,0.5343406593406593,0.45069366576352604,0.7611382603645325,0.4366871235743856 +B5_mofe_mlp,2026,audio_30,T,0.5117582417582418,0.6126373626373627,0.5963454618716882,0.6506626605987549,0.6009406951050202 +B5_mofe_mlp,2026,audio_30,A,0.32502747252747255,0.521978021978022,0.38592081478406426,0.7444095611572266,0.4312476657436216 +B5_mofe_mlp,2026,audio_30,V,0.44483516483516483,0.5151098901098901,0.4189500868178652,0.7407843470573425,0.33537420437520643 +B5_mofe_mlp,2026,audio_30,TA,0.32502747252747255,0.6208791208791209,0.5886808752593625,0.6490353345870972,0.570975514331383 +B5_mofe_mlp,2026,audio_30,TV,0.44483516483516483,0.5892857142857143,0.5746190571714002,0.7062152624130249,0.5695051030165077 +B5_mofe_mlp,2026,audio_30,AV,0.3078021978021978,0.4876373626373626,0.4781988617789497,0.7588927745819092,0.41241620046706 +B5_mofe_mlp,2026,audio_30,TAV,0.3078021978021978,0.5796703296703297,0.5254621943407217,0.7300676703453064,0.5331740795732829 +B5_mofe_mlp,2026,vision_30,T,0.5117582417582418,0.6112637362637363,0.5951956697149005,0.648055374622345,0.6013618141882758 +B5_mofe_mlp,2026,vision_30,A,0.4704945054945055,0.4725274725274725,0.2753954425047768,0.8200526237487793,0.2290185059925883 +B5_mofe_mlp,2026,vision_30,V,0.3076098901098901,0.5714285714285714,0.480133525611997,0.6950516700744629,0.4895576056562932 +B5_mofe_mlp,2026,vision_30,TA,0.4704945054945055,0.592032967032967,0.5533383448560617,0.6673561930656433,0.559581971258752 +B5_mofe_mlp,2026,vision_30,TV,0.3076098901098901,0.6181318681318682,0.5942276325734972,0.6576288938522339,0.5825792345472227 +B5_mofe_mlp,2026,vision_30,AV,0.3076098901098901,0.4835164835164835,0.4758194996023056,0.7574149370193481,0.39536651435197256 +B5_mofe_mlp,2026,vision_30,TAV,0.3076098901098901,0.5851648351648352,0.5229046773276295,0.7180534601211548,0.5426270178162265 +B5_mofe_mlp,2026,audio_vision_30,T,0.5117582417582418,0.6126373626373627,0.5966200968456806,0.6497841477394104,0.6010003821616935 +B5_mofe_mlp,2026,audio_vision_30,A,0.31903846153846155,0.521978021978022,0.40390961057275176,0.7342812418937683,0.4437631093762853 +B5_mofe_mlp,2026,audio_vision_30,V,0.30107142857142855,0.5686813186813187,0.5040223473945638,0.6871273517608643,0.4953387356903287 +B5_mofe_mlp,2026,audio_vision_30,TA,0.31903846153846155,0.5906593406593407,0.5620402015554825,0.6461182832717896,0.5751739998185288 +B5_mofe_mlp,2026,audio_vision_30,TV,0.30107142857142855,0.6167582417582418,0.6003175316510284,0.6656390428543091,0.5833224225343602 +B5_mofe_mlp,2026,audio_vision_30,AV,0.30107142857142855,0.5,0.49814761477760233,0.7473122477531433,0.43986219237598306 +B5_mofe_mlp,2026,audio_vision_30,TAV,0.30107142857142855,0.5673076923076923,0.5181807136427148,0.7298097610473633,0.5492606185227297 +B5_mofe_mlp,2026,all_modalities_30,T,0.36043956043956044,0.603021978021978,0.582619604272029,0.6699809432029724,0.5730899875150471 +B5_mofe_mlp,2026,all_modalities_30,A,0.3284065934065934,0.4725274725274725,0.28157868772104216,0.8073928952217102,0.2329321633646142 +B5_mofe_mlp,2026,all_modalities_30,V,0.3106043956043956,0.5274725274725275,0.4283917020609205,0.7411948442459106,0.3515515646987056 +B5_mofe_mlp,2026,all_modalities_30,TA,0.3284065934065934,0.5755494505494505,0.5352650821255436,0.6817682981491089,0.536393753611536 +B5_mofe_mlp,2026,all_modalities_30,TV,0.3106043956043956,0.5810439560439561,0.5628840082630775,0.7163252234458923,0.5509194419000414 +B5_mofe_mlp,2026,all_modalities_30,AV,0.3106043956043956,0.3873626373626374,0.37820754147953356,0.8290948271751404,0.2590758080570932 +B5_mofe_mlp,2026,all_modalities_30,TAV,0.3106043956043956,0.5343406593406593,0.46600788776303587,0.795678436756134,0.5034340527690154 diff --git a/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/figures/comparison_earlyconcat_mofe7.png b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/figures/comparison_earlyconcat_mofe7.png new file mode 100644 index 0000000..e0d6279 Binary files /dev/null and b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/figures/comparison_earlyconcat_mofe7.png differ diff --git a/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/metrics_by_condition.csv b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/metrics_by_condition.csv new file mode 100644 index 0000000..842776f --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/metrics_by_condition.csv @@ -0,0 +1,97 @@ +method,seed,condition,missing_rate,n_valid,accuracy,macro_f1,mae,pearson +B0_early_concat,42,clean,0.0,728,0.6332417582417582,0.587941053090477,0.6350555419921875,0.6157338827731829 +B0_early_concat,42,text_10,0.1,728,0.6263736263736264,0.5758300795065502,0.6362224221229553,0.609900144997639 +B0_early_concat,42,audio_10,0.1,728,0.6318681318681318,0.5859694579820366,0.6346138715744019,0.6186827917630225 +B0_early_concat,42,vision_10,0.1,728,0.6277472527472527,0.5825072763635762,0.6360324621200562,0.6155412236363174 +B0_early_concat,42,audio_vision_10,0.1,728,0.635989010989011,0.5919984665799197,0.6340617537498474,0.6171314734153543 +B0_early_concat,42,all_modalities_10,0.1,728,0.625,0.5767997019394343,0.6347602009773254,0.612706885342579 +B0_early_concat,42,text_20,0.2,728,0.6112637362637363,0.5533514537785528,0.6472386121749878,0.5878231385987681 +B0_early_concat,42,audio_20,0.2,728,0.6346153846153846,0.5913737928222076,0.6337395906448364,0.6167897438685002 +B0_early_concat,42,vision_20,0.2,728,0.6277472527472527,0.5807552037464527,0.6369960904121399,0.6152903325014981 +B0_early_concat,42,audio_vision_20,0.2,728,0.6346153846153846,0.5889252158180606,0.6331196427345276,0.6187845533592816 +B0_early_concat,42,all_modalities_20,0.2,728,0.6263736263736264,0.5759393139946568,0.6418775916099548,0.6097362711446388 +B0_early_concat,42,text_30,0.3,728,0.6181318681318682,0.5669198053141907,0.6596490740776062,0.5759615307863893 +B0_early_concat,42,audio_30,0.3,728,0.6414835164835165,0.5997485874560443,0.6367413997650146,0.6154119892745854 +B0_early_concat,42,vision_30,0.3,728,0.6332417582417582,0.5862230880240699,0.6370688080787659,0.615190259269048 +B0_early_concat,42,audio_vision_30,0.3,728,0.6428571428571429,0.5982133735032932,0.6347481608390808,0.6176032462057502 +B0_early_concat,42,all_modalities_30,0.3,728,0.625,0.5769720397379972,0.6341773867607117,0.6145213988024191 +B5_mofe_mlp,42,clean,0.0,728,0.6538461538461539,0.6219292861291515,0.6228764653205872,0.6143991595201629 +B5_mofe_mlp,42,text_10,0.1,728,0.6497252747252747,0.6156314005702952,0.6238094568252563,0.6073313722070635 +B5_mofe_mlp,42,audio_10,0.1,728,0.6497252747252747,0.6192151480683822,0.6202160716056824,0.614726001096473 +B5_mofe_mlp,42,vision_10,0.1,728,0.6497252747252747,0.617790322593887,0.620069682598114,0.6170344342825693 +B5_mofe_mlp,42,audio_vision_10,0.1,728,0.6524725274725275,0.6211709858768683,0.6203965544700623,0.6150187757034211 +B5_mofe_mlp,42,all_modalities_10,0.1,728,0.6414835164835165,0.6107226107226107,0.6245603561401367,0.6099877885070804 +B5_mofe_mlp,42,text_20,0.2,728,0.6304945054945055,0.5904030390809835,0.6350096464157104,0.5945970846072441 +B5_mofe_mlp,42,audio_20,0.2,728,0.6483516483516484,0.6152893789223305,0.6198973655700684,0.6158397844329069 +B5_mofe_mlp,42,vision_20,0.2,728,0.6497252747252747,0.6161050291578356,0.6227806210517883,0.6170723836770221 +B5_mofe_mlp,42,audio_vision_20,0.2,728,0.6456043956043956,0.6124517381483326,0.6248074173927307,0.6073068132303947 +B5_mofe_mlp,42,all_modalities_20,0.2,728,0.6401098901098901,0.6082957983542778,0.6285881400108337,0.6113174240287709 +B5_mofe_mlp,42,text_30,0.3,728,0.614010989010989,0.5665382052109811,0.6587778925895691,0.57122460274951 +B5_mofe_mlp,42,audio_30,0.3,728,0.6414835164835165,0.6099992622632019,0.6212661266326904,0.6127160793050621 +B5_mofe_mlp,42,vision_30,0.3,728,0.6442307692307693,0.6100360167586674,0.6305183172225952,0.6127016894323144 +B5_mofe_mlp,42,audio_vision_30,0.3,728,0.6538461538461539,0.6220936984876739,0.6240691542625427,0.6123047623188541 +B5_mofe_mlp,42,all_modalities_30,0.3,728,0.6442307692307693,0.6106062664769548,0.6119968295097351,0.6240023303690699 +B0_early_concat,3407,clean,0.0,728,0.6332417582417582,0.5937554950072336,0.6333680152893066,0.6138300851146614 +B0_early_concat,3407,text_10,0.1,728,0.6277472527472527,0.5848809977670061,0.6363487243652344,0.6056385881925804 +B0_early_concat,3407,audio_10,0.1,728,0.635989010989011,0.5968950441819239,0.6318458914756775,0.6158991549580464 +B0_early_concat,3407,vision_10,0.1,728,0.6332417582417582,0.5933513166880985,0.6328762769699097,0.614296568855868 +B0_early_concat,3407,audio_vision_10,0.1,728,0.635989010989011,0.5972272747847399,0.6317789554595947,0.613152110576325 +B0_early_concat,3407,all_modalities_10,0.1,728,0.6208791208791209,0.5757563714642701,0.6323474645614624,0.617483663695453 +B0_early_concat,3407,text_20,0.2,728,0.6043956043956044,0.5490811521155289,0.6509508490562439,0.5895223268279027 +B0_early_concat,3407,audio_20,0.2,728,0.635989010989011,0.5977119556159886,0.6304426789283752,0.6183830231575858 +B0_early_concat,3407,vision_20,0.2,728,0.6401098901098901,0.6034867151810619,0.628890335559845,0.6187934526699497 +B0_early_concat,3407,audio_vision_20,0.2,728,0.6346153846153846,0.600406263153788,0.633434534072876,0.6123211847883981 +B0_early_concat,3407,all_modalities_20,0.2,728,0.6153846153846154,0.5731546231546232,0.6435555815696716,0.6001160788677808 +B0_early_concat,3407,text_30,0.3,728,0.6071428571428571,0.5572172310238351,0.6613345146179199,0.5663984672630126 +B0_early_concat,3407,audio_30,0.3,728,0.6401098901098901,0.602821099359954,0.6328924298286438,0.6157652906372406 +B0_early_concat,3407,vision_30,0.3,728,0.6277472527472527,0.5882423141524775,0.6326207518577576,0.6153119497189492 +B0_early_concat,3407,audio_vision_30,0.3,728,0.6428571428571429,0.6062757576614914,0.6341086030006409,0.6170348487651544 +B0_early_concat,3407,all_modalities_30,0.3,728,0.6277472527472527,0.5839727159969967,0.6468587517738342,0.5966126256796758 +B5_mofe_mlp,3407,clean,0.0,728,0.635989010989011,0.604973907499834,0.6324349641799927,0.6151046893704437 +B5_mofe_mlp,3407,text_10,0.1,728,0.6318681318681318,0.5933228635123383,0.633339524269104,0.6112816469725518 +B5_mofe_mlp,3407,audio_10,0.1,728,0.635989010989011,0.6042690508274062,0.6318331360816956,0.6144226554849131 +B5_mofe_mlp,3407,vision_10,0.1,728,0.6387362637362637,0.6078729015399154,0.6297808885574341,0.6157341874415431 +B5_mofe_mlp,3407,audio_vision_10,0.1,728,0.625,0.5961079542334627,0.630139172077179,0.6148226006976545 +B5_mofe_mlp,3407,all_modalities_10,0.1,728,0.6195054945054945,0.5890880499595165,0.6359822750091553,0.6113081804637202 +B5_mofe_mlp,3407,text_20,0.2,728,0.6195054945054945,0.5755820027236789,0.6454524993896484,0.5935605745999009 +B5_mofe_mlp,3407,audio_20,0.2,728,0.6332417582417582,0.6046231856798019,0.6310902833938599,0.6172434930698147 +B5_mofe_mlp,3407,vision_20,0.2,728,0.635989010989011,0.6063745521197687,0.6304876208305359,0.6140966581910975 +B5_mofe_mlp,3407,audio_vision_20,0.2,728,0.635989010989011,0.609224910808655,0.6240154504776001,0.6185046337968831 +B5_mofe_mlp,3407,all_modalities_20,0.2,728,0.6332417582417582,0.6079307048305954,0.634419858455658,0.6044717652490789 +B5_mofe_mlp,3407,text_30,0.3,728,0.6181318681318682,0.5733540937906745,0.656674325466156,0.5668925651895068 +B5_mofe_mlp,3407,audio_30,0.3,728,0.6332417582417582,0.604260667390971,0.6289573907852173,0.617381543407622 +B5_mofe_mlp,3407,vision_30,0.3,728,0.6332417582417582,0.6016029959347541,0.6318032145500183,0.6125144777620533 +B5_mofe_mlp,3407,audio_vision_30,0.3,728,0.6318681318681318,0.6067614810631058,0.6228161454200745,0.621408382664964 +B5_mofe_mlp,3407,all_modalities_30,0.3,728,0.6222527472527473,0.5949881199577279,0.6458109021186829,0.5977781285960149 +B0_early_concat,2026,clean,0.0,728,0.6112637362637363,0.5592125291749822,0.6401797533035278,0.6097643309752457 +B0_early_concat,2026,text_10,0.1,728,0.6098901098901099,0.5538253146595995,0.6407675743103027,0.60352101329599 +B0_early_concat,2026,audio_10,0.1,728,0.614010989010989,0.5630983962746592,0.6385665535926819,0.6110860034615799 +B0_early_concat,2026,vision_10,0.1,728,0.6181318681318682,0.5661140652625988,0.6405577659606934,0.6110400519655834 +B0_early_concat,2026,audio_vision_10,0.1,728,0.6098901098901099,0.5587567567850849,0.6381099224090576,0.6107705746373118 +B0_early_concat,2026,all_modalities_10,0.1,728,0.614010989010989,0.5581636747439855,0.6429373621940613,0.6088956503302468 +B0_early_concat,2026,text_20,0.2,728,0.6043956043956044,0.5429151983962918,0.6489536762237549,0.5886848272694757 +B0_early_concat,2026,audio_20,0.2,728,0.6112637362637363,0.5603026186889267,0.636631965637207,0.6105368420749809 +B0_early_concat,2026,vision_20,0.2,728,0.614010989010989,0.5607756511971072,0.6439995765686035,0.6079712010559852 +B0_early_concat,2026,audio_vision_20,0.2,728,0.6181318681318682,0.5696462960623787,0.6381081342697144,0.6114528658433437 +B0_early_concat,2026,all_modalities_20,0.2,728,0.6085164835164835,0.5508998902588577,0.6492584347724915,0.6012091639944216 +B0_early_concat,2026,text_30,0.3,728,0.5851648351648352,0.5160231153138954,0.6731752753257751,0.5521914713018387 +B0_early_concat,2026,audio_30,0.3,728,0.6181318681318682,0.5670636517410711,0.636723518371582,0.6097718223127288 +B0_early_concat,2026,vision_30,0.3,728,0.6085164835164835,0.5560149900702367,0.6435868144035339,0.609850184236316 +B0_early_concat,2026,audio_vision_30,0.3,728,0.6098901098901099,0.5653848813301509,0.637891948223114,0.6075333578422618 +B0_early_concat,2026,all_modalities_30,0.3,728,0.6043956043956044,0.5496991187074715,0.6696622371673584,0.5843647118157551 +B5_mofe_mlp,2026,clean,0.0,728,0.6318681318681318,0.596526187825028,0.6597292423248291,0.6032696127085834 +B5_mofe_mlp,2026,text_10,0.1,728,0.6332417582417582,0.5928689292978023,0.6584991812705994,0.5961609550472456 +B5_mofe_mlp,2026,audio_10,0.1,728,0.6332417582417582,0.5971419540706502,0.6580070853233337,0.6047899048858725 +B5_mofe_mlp,2026,vision_10,0.1,728,0.6277472527472527,0.5919759280299669,0.6596941947937012,0.6023122703551795 +B5_mofe_mlp,2026,audio_vision_10,0.1,728,0.6291208791208791,0.5949724164706738,0.6590589880943298,0.60347148906883 +B5_mofe_mlp,2026,all_modalities_10,0.1,728,0.6387362637362637,0.6013762119617033,0.6582632660865784,0.599810574437488 +B5_mofe_mlp,2026,text_20,0.2,728,0.6277472527472527,0.579286730359655,0.667361319065094,0.582974915319036 +B5_mofe_mlp,2026,audio_20,0.2,728,0.6304945054945055,0.5954536143668596,0.6568871736526489,0.6037190808626715 +B5_mofe_mlp,2026,vision_20,0.2,728,0.6277472527472527,0.5900160130008749,0.6632898449897766,0.6042443916006178 +B5_mofe_mlp,2026,audio_vision_20,0.2,728,0.625,0.5917418600128784,0.6548266410827637,0.6054018359666251 +B5_mofe_mlp,2026,all_modalities_20,0.2,728,0.6387362637362637,0.6023597229130889,0.6525924801826477,0.6015383340552747 +B5_mofe_mlp,2026,text_30,0.3,728,0.6236263736263736,0.5695777746219127,0.6862288117408752,0.5485307064435356 +B5_mofe_mlp,2026,audio_30,0.3,728,0.6277472527472527,0.5928207016211315,0.656401515007019,0.605701200975301 +B5_mofe_mlp,2026,vision_30,0.3,728,0.6291208791208791,0.5902326768980343,0.6632499694824219,0.6060668503667017 +B5_mofe_mlp,2026,audio_vision_30,0.3,728,0.625,0.5962838833608418,0.6554995775222778,0.6044591409783642 +B5_mofe_mlp,2026,all_modalities_30,0.3,728,0.6167582417582418,0.5778458150682998,0.6775780916213989,0.5767909598557909 diff --git a/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/modality_utility_by_condition.csv b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/modality_utility_by_condition.csv new file mode 100644 index 0000000..9aa55fe --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/modality_utility_by_condition.csv @@ -0,0 +1,49 @@ +method,seed,condition,utility_text_mean,utility_audio_mean,utility_vision_mean +B5_mofe_mlp,42,clean,0.6618766188621521,0.5899914503097534,0.5503968000411987 +B5_mofe_mlp,42,text_10,0.6631303429603577,0.6045999526977539,0.5672140121459961 +B5_mofe_mlp,42,audio_10,0.6753877997398376,0.5900472402572632,0.5553804636001587 +B5_mofe_mlp,42,vision_10,0.6722928285598755,0.5894219875335693,0.5504083633422852 +B5_mofe_mlp,42,audio_vision_10,0.6965795755386353,0.5897499918937683,0.5503614544868469 +B5_mofe_mlp,42,all_modalities_10,0.663243293762207,0.5896329879760742,0.5504105091094971 +B5_mofe_mlp,42,text_20,0.6646575927734375,0.6176748275756836,0.5843319892883301 +B5_mofe_mlp,42,audio_20,0.6875958442687988,0.5900545716285706,0.5599780678749084 +B5_mofe_mlp,42,vision_20,0.6815856695175171,0.5894845128059387,0.5504598617553711 +B5_mofe_mlp,42,audio_vision_20,0.7309128642082214,0.5897306203842163,0.5503560900688171 +B5_mofe_mlp,42,all_modalities_20,0.6649715900421143,0.589468240737915,0.5503589510917664 +B5_mofe_mlp,42,text_30,0.6656913161277771,0.6293779611587524,0.5994364023208618 +B5_mofe_mlp,42,audio_30,0.7017264366149902,0.5900580883026123,0.5654440522193909 +B5_mofe_mlp,42,vision_30,0.6905444860458374,0.5895596742630005,0.5504684448242188 +B5_mofe_mlp,42,audio_vision_30,0.7655619382858276,0.5896934866905212,0.5503290891647339 +B5_mofe_mlp,42,all_modalities_30,0.666799783706665,0.5894308686256409,0.5503942370414734 +B5_mofe_mlp,3407,clean,0.5965457558631897,0.6089583039283752,0.5700515508651733 +B5_mofe_mlp,3407,text_10,0.5980162024497986,0.6175488829612732,0.5699793100357056 +B5_mofe_mlp,3407,audio_10,0.6005110144615173,0.6088259816169739,0.5817712545394897 +B5_mofe_mlp,3407,vision_10,0.602319598197937,0.6167332530021667,0.5697159767150879 +B5_mofe_mlp,3407,audio_vision_10,0.6381478309631348,0.6084476709365845,0.5697139501571655 +B5_mofe_mlp,3407,all_modalities_10,0.598029613494873,0.6078585386276245,0.5693216323852539 +B5_mofe_mlp,3407,text_20,0.6004789471626282,0.6273719668388367,0.5698307156562805 +B5_mofe_mlp,3407,audio_20,0.604293942451477,0.6087151169776917,0.5934981107711792 +B5_mofe_mlp,3407,vision_20,0.6091526746749878,0.6249159574508667,0.569640576839447 +B5_mofe_mlp,3407,audio_vision_20,0.6846911907196045,0.6084182858467102,0.569659948348999 +B5_mofe_mlp,3407,all_modalities_20,0.6004623174667358,0.607769250869751,0.5691742300987244 +B5_mofe_mlp,3407,text_30,0.6017250418663025,0.6352017521858215,0.5698606371879578 +B5_mofe_mlp,3407,audio_30,0.6071357727050781,0.609175980091095,0.6063627004623413 +B5_mofe_mlp,3407,vision_30,0.6154943108558655,0.6325424909591675,0.5695337653160095 +B5_mofe_mlp,3407,audio_vision_30,0.7215061783790588,0.6085333228111267,0.5696398019790649 +B5_mofe_mlp,3407,all_modalities_30,0.6027611494064331,0.6076351404190063,0.5691232085227966 +B5_mofe_mlp,2026,clean,0.6298863887786865,0.5585612058639526,0.5456792712211609 +B5_mofe_mlp,2026,text_10,0.6311661005020142,0.571927547454834,0.5534814596176147 +B5_mofe_mlp,2026,audio_10,0.6393773555755615,0.5580084919929504,0.5503398776054382 +B5_mofe_mlp,2026,vision_10,0.6332205533981323,0.5693727731704712,0.5460953712463379 +B5_mofe_mlp,2026,audio_vision_10,0.6685320734977722,0.558728039264679,0.5464425086975098 +B5_mofe_mlp,2026,all_modalities_10,0.6309354305267334,0.5586868524551392,0.5456184148788452 +B5_mofe_mlp,2026,text_20,0.6327240467071533,0.5853152871131897,0.5612947344779968 +B5_mofe_mlp,2026,audio_20,0.6477784514427185,0.557928740978241,0.5547807216644287 +B5_mofe_mlp,2026,vision_20,0.6358551979064941,0.5790520906448364,0.5461242198944092 +B5_mofe_mlp,2026,audio_vision_20,0.7063466906547546,0.5585485696792603,0.5466189980506897 +B5_mofe_mlp,2026,all_modalities_20,0.6332317590713501,0.5591967105865479,0.5456498265266418 +B5_mofe_mlp,2026,text_30,0.6339558362960815,0.5997033715248108,0.569423258304596 +B5_mofe_mlp,2026,audio_30,0.658158004283905,0.5578649640083313,0.5603182315826416 +B5_mofe_mlp,2026,vision_30,0.638981282711029,0.590747058391571,0.5463551878929138 +B5_mofe_mlp,2026,audio_vision_30,0.7488567233085632,0.5589607357978821,0.5467853546142578 +B5_mofe_mlp,2026,all_modalities_30,0.6328451037406921,0.5590822696685791,0.5455838441848755 diff --git a/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/models/B5_mofe_mlp/seed_2026/model_best.pt b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/models/B5_mofe_mlp/seed_2026/model_best.pt new file mode 100644 index 0000000..6ce82a8 Binary files /dev/null and b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/models/B5_mofe_mlp/seed_2026/model_best.pt differ diff --git a/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/models/B5_mofe_mlp/seed_3407/model_best.pt b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/models/B5_mofe_mlp/seed_3407/model_best.pt new file mode 100644 index 0000000..2c215d9 Binary files /dev/null and b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/models/B5_mofe_mlp/seed_3407/model_best.pt differ diff --git a/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/models/B5_mofe_mlp/seed_42/model_best.pt b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/models/B5_mofe_mlp/seed_42/model_best.pt new file mode 100644 index 0000000..78b5ac0 Binary files /dev/null and b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/models/B5_mofe_mlp/seed_42/model_best.pt differ diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/seed_2026/model_best.pt b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/models/baselines/concat/seed_2026/model_best.pt similarity index 100% rename from deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/seed_2026/model_best.pt rename to deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/models/baselines/concat/seed_2026/model_best.pt diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/seed_3407/model_best.pt b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/models/baselines/concat/seed_3407/model_best.pt similarity index 100% rename from deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/seed_3407/model_best.pt rename to deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/models/baselines/concat/seed_3407/model_best.pt diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/model_best.pt b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/models/baselines/concat/seed_42/model_best.pt similarity index 100% rename from deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/model_best.pt rename to deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/models/baselines/concat/seed_42/model_best.pt diff --git a/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/paired_bootstrap.csv b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/paired_bootstrap.csv new file mode 100644 index 0000000..c00d680 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/paired_bootstrap.csv @@ -0,0 +1,6 @@ +comparison,candidate,reference,metric,delta_candidate_minus_reference,bootstrap_ci_2p5,bootstrap_ci_97p5,bootstrap_probability_delta_gt_0,bootstrap_replicates,resampling_unit,paired,seed +MoFE-7 MLP vs EarlyConcat,B5_mofe_mlp,B0_early_concat,corrupt_macro_f1,0.024555007972419718,0.01160479835525017,0.038111700197624034,0.999,1000,source video id,True,20260924 +MoFE-7 MLP vs EarlyConcat,B5_mofe_mlp,B0_early_concat,worst_condition_macro_f1,0.023103307323882216,0.007213272437200871,0.04285067710167781,0.996,1000,source video id,True,20260924 +MoFE-7 MLP vs EarlyConcat,B5_mofe_mlp,B0_early_concat,text_30_macro_f1,0.023103307323882216,0.004027805218645245,0.04212429356920461,0.992,1000,source video id,True,20260924 +MoFE-7 MLP vs EarlyConcat,B5_mofe_mlp,B0_early_concat,corrupt_mae,0.00036756197611487185,-0.008981325373224738,0.009240262436166205,0.524,1000,source video id,True,20260924 +MoFE-7 MLP vs EarlyConcat,B5_mofe_mlp,B0_early_concat,corrupt_pearson,-0.0019135250956427985,-0.01366917866941531,0.010852838898108074,0.397,1000,source video id,True,20260924 diff --git a/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/parameter_count.csv b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/parameter_count.csv new file mode 100644 index 0000000..047e5cf --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/parameter_count.csv @@ -0,0 +1,3 @@ +method,trainable_parameters,ratio_to_earlyconcat,within_2x_earlyconcat +B0_early_concat,253124,1.0,True +B5_mofe_mlp,306523,1.2109598457672919,True diff --git a/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/router_weights_by_condition.csv b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/router_weights_by_condition.csv new file mode 100644 index 0000000..e1b8bfc --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/router_weights_by_condition.csv @@ -0,0 +1,49 @@ +method,seed,condition,active_position_fraction,fallback_position_fraction,normalized_router_entropy,fraction_active_positions_max_weight_over_0p8,alpha_T_mean,alpha_A_mean,alpha_V_mean,alpha_TA_mean,alpha_TV_mean,alpha_AV_mean,alpha_TAV_mean +B5_mofe_mlp,42,clean,0.5117582417582418,0.48824175824175825,0.8870560454827882,0.08063130770882543,0.2515257000923157,0.11152571439743042,0.07744884490966797,0.15852881968021393,0.12860679626464844,0.14914916455745697,0.1232161670923233 +B5_mofe_mlp,42,text_10,0.5092032967032967,0.4907967032967033,0.8506916726514709,0.08211491772322632,0.2304162234067917,0.13068588078022003,0.09430146962404251,0.14338572323322296,0.11664345115423203,0.17277489602565765,0.1117931678891182 +B5_mofe_mlp,42,audio_10,0.5117582417582418,0.48824175824175825,0.8449963602081583,0.08616061842387804,0.2751412093639374,0.10004748404026031,0.09071236103773117,0.1420598030090332,0.14774158596992493,0.13385255634784698,0.11044659465551376 +B5_mofe_mlp,42,vision_10,0.5117582417582418,0.48824175824175825,0.8472941162919574,0.08063130770882543,0.2735346555709839,0.1252371221780777,0.06957398355007172,0.1735313981771469,0.11499655246734619,0.13289612531661987,0.11023180931806564 +B5_mofe_mlp,42,audio_vision_10,0.5117582417582418,0.48824175824175825,0.7958907203885603,0.175434829289242,0.32850298285484314,0.10010535269975662,0.06980907171964645,0.14210578799247742,0.11540128290653229,0.1335059553384781,0.11057072132825851 +B5_mofe_mlp,42,all_modalities_10,0.4626098901098901,0.5373901098901099,0.8841173799028046,0.08432804798384702,0.2541447579860687,0.11095340549945831,0.07761403918266296,0.15777607262134552,0.1283317655324936,0.14818961918354034,0.12299200892448425 +B5_mofe_mlp,42,text_20,0.5065384615384615,0.49346153846153845,0.8152469884580759,0.08005206638464042,0.20999735593795776,0.14806155860424042,0.112088643014431,0.12878893315792084,0.10419154912233353,0.19696328043937683,0.09990917891263962 +B5_mofe_mlp,42,audio_20,0.5117582417582418,0.48824175824175825,0.8069851130815402,0.09013313291818767,0.2962163984775543,0.0896897166967392,0.10295859724283218,0.1273450255393982,0.16525308787822723,0.11975577473640442,0.09878180921077728 +B5_mofe_mlp,42,vision_20,0.5117582417582418,0.48824175824175825,0.8111495627938943,0.08063130770882543,0.2933293581008911,0.13767293095588684,0.062129102647304535,0.18726874887943268,0.10258819162845612,0.1186119019985199,0.09839928895235062 +B5_mofe_mlp,42,audio_vision_20,0.5117582417582418,0.48824175824175825,0.7057799858331233,0.2687352372772171,0.40449970960617065,0.08882029354572296,0.06192621961236,0.12605853378772736,0.10232582688331604,0.11834055930376053,0.09802991151809692 +B5_mofe_mlp,42,all_modalities_20,0.41145604395604396,0.588543956043956,0.8795920971657628,0.08853575482406356,0.25787994265556335,0.11065426468849182,0.07723148167133331,0.157210573554039,0.12760961055755615,0.14714252948760986,0.12227214872837067 +B5_mofe_mlp,42,text_30,0.5034615384615385,0.49653846153846154,0.7845061109572483,0.07726727054458146,0.19043782353401184,0.164101243019104,0.12816408276557922,0.11582792550325394,0.0932326465845108,0.218837708234787,0.08939971774816513 +B5_mofe_mlp,42,audio_30,0.5117582417582418,0.48824175824175825,0.7627258720846881,0.09308567747476916,0.32027310132980347,0.0778515487909317,0.11745958030223846,0.11037490516901016,0.18614806234836578,0.10296247154474258,0.08493035286664963 +B5_mofe_mlp,42,vision_30,0.5117582417582418,0.48824175824175825,0.7764460734868434,0.08063130770882543,0.31232842803001404,0.1496150940656662,0.054964445531368256,0.20050112903118134,0.09068562090396881,0.10487580299377441,0.0870303362607956 +B5_mofe_mlp,42,audio_vision_30,0.5117582417582418,0.48824175824175825,0.6149186646941948,0.3631092978312218,0.4812489449977875,0.07732704281806946,0.05401867628097534,0.10968495160341263,0.0891624167561531,0.10309218615293503,0.08546625822782516 +B5_mofe_mlp,42,all_modalities_30,0.3595054945054945,0.6404945054945055,0.8745368649815657,0.09307657038055937,0.26191991567611694,0.11035662144422531,0.07678450644016266,0.1566968709230423,0.12672796845436096,0.14605894684791565,0.12145592272281647 +B5_mofe_mlp,3407,clean,0.5117582417582418,0.48824175824175825,0.8917789142359226,0.08063130770882543,0.21564963459968567,0.16335873305797577,0.12684088945388794,0.1254868507385254,0.09765232354402542,0.11325515806674957,0.15775780379772186 +B5_mofe_mlp,3407,text_10,0.509010989010989,0.490989010989011,0.8547805451037497,0.08031088082901554,0.19791889190673828,0.1904035061597824,0.1427920013666153,0.11356116831302643,0.08846957981586456,0.12491724640130997,0.14193856716156006 +B5_mofe_mlp,3407,audio_10,0.5117582417582418,0.48824175824175825,0.8504781823179828,0.08481855271634099,0.23455235362052917,0.1466052234172821,0.15115498006343842,0.11315018683671951,0.1112954393029213,0.10172916203737259,0.14151319861412048 +B5_mofe_mlp,3407,vision_10,0.5117582417582418,0.48824175824175825,0.8539108633297741,0.08063130770882543,0.2311594933271408,0.1815430223941803,0.11412636935710907,0.1422722488641739,0.08770900219678879,0.10201089829206467,0.14117981493473053 +B5_mofe_mlp,3407,audio_vision_10,0.5117582417582418,0.48824175824175825,0.7997980860345286,0.175434829289242,0.29687586426734924,0.14598697423934937,0.11404559761285782,0.11318352073431015,0.08737359195947647,0.10181976109743118,0.14071525633335114 +B5_mofe_mlp,3407,all_modalities_10,0.4610164835164835,0.5389835164835165,0.8892426095830844,0.08348727727787379,0.2185969352722168,0.16224151849746704,0.12673969566822052,0.12583576142787933,0.09755325317382812,0.11298926174640656,0.1560446321964264 +B5_mofe_mlp,3407,text_20,0.5066758241758241,0.4933241758241758,0.8130542021789052,0.08176543946212655,0.17931020259857178,0.2202228158712387,0.16002467274665833,0.10018529742956161,0.07809362560510635,0.137038454413414,0.12512563169002533 +B5_mofe_mlp,3407,audio_20,0.5117582417582418,0.48824175824175825,0.8087283495175436,0.09002576766158471,0.2538429796695709,0.12989234924316406,0.1755901426076889,0.10037955641746521,0.12468340992927551,0.09022346138954163,0.12538877129554749 +B5_mofe_mlp,3407,vision_20,0.5117582417582418,0.48824175824175825,0.8141886106137566,0.08063130770882543,0.24743759632110596,0.20013628900051117,0.10070110112428665,0.16005626320838928,0.0773327648639679,0.09001021832227707,0.12432596832513809 +B5_mofe_mlp,3407,audio_vision_20,0.5117582417582418,0.48824175824175825,0.6969105807199653,0.28124328967146234,0.3873015344142914,0.1272396296262741,0.09933283925056458,0.09888661652803421,0.07606082409620285,0.08873630315065384,0.12244285643100739 +B5_mofe_mlp,3407,all_modalities_20,0.40958791208791206,0.5904120879120879,0.8837504554280156,0.08867127238580723,0.22335518896579742,0.1614186316728592,0.12589122354984283,0.12545962631702423,0.09687625616788864,0.11222808808088303,0.15477190911769867 +B5_mofe_mlp,3407,text_30,0.5033791208791208,0.4966208791208791,0.77998020566234,0.07853517437100911,0.1615801304578781,0.24538777768611908,0.1755058914422989,0.08944669365882874,0.0692080482840538,0.1480167955160141,0.11085427552461624 +B5_mofe_mlp,3407,audio_30,0.5117582417582418,0.48824175824175825,0.7653189262124539,0.09217307279364398,0.2723240554332733,0.1130220890045166,0.20212848484516144,0.08758600801229477,0.13917095959186554,0.07771366834640503,0.10805478692054749 +B5_mofe_mlp,3407,vision_30,0.5117582417582418,0.48824175824175825,0.7773799776708603,0.08063130770882543,0.26250481605529785,0.21736890077590942,0.08825590461492538,0.17656537890434265,0.06769922375679016,0.07888038456439972,0.10872545838356018 +B5_mofe_mlp,3407,audio_vision_30,0.5117582417582418,0.48824175824175825,0.6151879114518624,0.36498818982177367,0.4589049518108368,0.11260678619146347,0.08762799948453903,0.08767727017402649,0.06704143434762955,0.07825901359319687,0.10788305103778839 +B5_mofe_mlp,3407,all_modalities_30,0.3557967032967033,0.6442032967032967,0.8791751773751674,0.09342907883561115,0.22755521535873413,0.16036975383758545,0.12519316375255585,0.12486399710178375,0.09638682752847672,0.11167588084936142,0.15395526587963104 +B5_mofe_mlp,2026,clean,0.5117582417582418,0.48824175824175825,0.892980397203899,0.08063130770882543,0.2621680796146393,0.13523989915847778,0.11440448462963104,0.1282719075679779,0.10990405827760696,0.12046957015991211,0.12954255938529968 +B5_mofe_mlp,2026,text_10,0.5091208791208791,0.4908791208791209,0.8566292977820127,0.08099503561407295,0.24004511535167694,0.16017793118953705,0.13224223256111145,0.11618264019489288,0.09917686879634857,0.1352321356534958,0.1169428601861 +B5_mofe_mlp,2026,audio_10,0.5117582417582418,0.48824175824175825,0.8513181092443619,0.08616061842387804,0.28532060980796814,0.12107318639755249,0.13112173974514008,0.11523441225290298,0.12272416055202484,0.1084277555346489,0.11609779298305511 +B5_mofe_mlp,2026,vision_10,0.5117582417582418,0.48824175824175825,0.8538046087080117,0.08063130770882543,0.275703102350235,0.15632730722427368,0.10242735594511032,0.14280176162719727,0.09840572625398636,0.1080247089266777,0.11630945652723312 +B5_mofe_mlp,2026,audio_vision_10,0.5117582417582418,0.48824175824175825,0.7992446529652792,0.1772063560231909,0.33916473388671875,0.12080469727516174,0.10259180516004562,0.11497309058904648,0.0985245332121849,0.10807139426469803,0.11586850136518478 +B5_mofe_mlp,2026,all_modalities_10,0.45884615384615385,0.5411538461538462,0.890664129810964,0.08400191593821099,0.2641141414642334,0.13430693745613098,0.11438830941915512,0.1279611438512802,0.10974214971065521,0.1203693225979805,0.12911845743656158 +B5_mofe_mlp,2026,text_20,0.5064560439560439,0.49354395604395607,0.8196418202446201,0.08093300786547328,0.21744248270988464,0.18586474657058716,0.15042437613010406,0.10369155555963516,0.08837979286909103,0.14998102188110352,0.10421676933765411 +B5_mofe_mlp,2026,audio_20,0.5117582417582418,0.48824175824175825,0.8139422624038224,0.0915288812540262,0.30578380823135376,0.10856158286333084,0.14610645174980164,0.1034226343035698,0.134155735373497,0.09755375236272812,0.10441582649946213 +B5_mofe_mlp,2026,vision_20,0.5117582417582418,0.48824175824175825,0.8189807529817417,0.08063130770882543,0.28755468130111694,0.17542824149131775,0.09184657037258148,0.1557445526123047,0.08823608607053757,0.0968705415725708,0.10431937873363495 +B5_mofe_mlp,2026,audio_vision_20,0.5117582417582418,0.48824175824175825,0.7084234520060394,0.27120463817908524,0.41442573070526123,0.10667615383863449,0.09106616675853729,0.10168138891458511,0.0874401405453682,0.09591105580329895,0.10279884934425354 +B5_mofe_mlp,2026,all_modalities_20,0.40585164835164833,0.5941483516483517,0.8838247695015665,0.08962296080687741,0.26898571848869324,0.13424904644489288,0.1132628470659256,0.12779442965984344,0.10867198556661606,0.1192564070224762,0.1277802437543869 +B5_mofe_mlp,2026,text_30,0.5032417582417582,0.49675824175824174,0.7814398465385116,0.08243257997597991,0.19258207082748413,0.21375314891338348,0.1692270040512085,0.09033004194498062,0.07751274853944778,0.16531100869178772,0.09128450602293015 +B5_mofe_mlp,2026,audio_30,0.5117582417582418,0.48824175824175825,0.7664926836291861,0.09711187459738028,0.3314794600009918,0.09291943162679672,0.1653663069009781,0.08855544030666351,0.14884275197982788,0.08355656266212463,0.08927922695875168 +B5_mofe_mlp,2026,vision_30,0.5117582417582418,0.48824175824175825,0.7769904015474858,0.08063130770882543,0.3017820417881012,0.19846147298812866,0.07910744100809097,0.17135068774223328,0.07599635422229767,0.08345004171133041,0.0898519977927208 +B5_mofe_mlp,2026,audio_vision_30,0.5117582417582418,0.48824175824175825,0.6051323653073035,0.3765836375348937,0.49931350350379944,0.09166184812784195,0.07765422761440277,0.0873459056019783,0.07456637173891068,0.08182720094919205,0.08763010799884796 +B5_mofe_mlp,2026,all_modalities_30,0.36043956043956044,0.6395604395604395,0.8852294042453963,0.08887195121951219,0.2681088149547577,0.1340326964855194,0.11355702579021454,0.12770922482013702,0.10893982648849487,0.11956492066383362,0.12808853387832642 diff --git a/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/routing_entropy.csv b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/routing_entropy.csv new file mode 100644 index 0000000..e1b8bfc --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/routing_entropy.csv @@ -0,0 +1,49 @@ +method,seed,condition,active_position_fraction,fallback_position_fraction,normalized_router_entropy,fraction_active_positions_max_weight_over_0p8,alpha_T_mean,alpha_A_mean,alpha_V_mean,alpha_TA_mean,alpha_TV_mean,alpha_AV_mean,alpha_TAV_mean +B5_mofe_mlp,42,clean,0.5117582417582418,0.48824175824175825,0.8870560454827882,0.08063130770882543,0.2515257000923157,0.11152571439743042,0.07744884490966797,0.15852881968021393,0.12860679626464844,0.14914916455745697,0.1232161670923233 +B5_mofe_mlp,42,text_10,0.5092032967032967,0.4907967032967033,0.8506916726514709,0.08211491772322632,0.2304162234067917,0.13068588078022003,0.09430146962404251,0.14338572323322296,0.11664345115423203,0.17277489602565765,0.1117931678891182 +B5_mofe_mlp,42,audio_10,0.5117582417582418,0.48824175824175825,0.8449963602081583,0.08616061842387804,0.2751412093639374,0.10004748404026031,0.09071236103773117,0.1420598030090332,0.14774158596992493,0.13385255634784698,0.11044659465551376 +B5_mofe_mlp,42,vision_10,0.5117582417582418,0.48824175824175825,0.8472941162919574,0.08063130770882543,0.2735346555709839,0.1252371221780777,0.06957398355007172,0.1735313981771469,0.11499655246734619,0.13289612531661987,0.11023180931806564 +B5_mofe_mlp,42,audio_vision_10,0.5117582417582418,0.48824175824175825,0.7958907203885603,0.175434829289242,0.32850298285484314,0.10010535269975662,0.06980907171964645,0.14210578799247742,0.11540128290653229,0.1335059553384781,0.11057072132825851 +B5_mofe_mlp,42,all_modalities_10,0.4626098901098901,0.5373901098901099,0.8841173799028046,0.08432804798384702,0.2541447579860687,0.11095340549945831,0.07761403918266296,0.15777607262134552,0.1283317655324936,0.14818961918354034,0.12299200892448425 +B5_mofe_mlp,42,text_20,0.5065384615384615,0.49346153846153845,0.8152469884580759,0.08005206638464042,0.20999735593795776,0.14806155860424042,0.112088643014431,0.12878893315792084,0.10419154912233353,0.19696328043937683,0.09990917891263962 +B5_mofe_mlp,42,audio_20,0.5117582417582418,0.48824175824175825,0.8069851130815402,0.09013313291818767,0.2962163984775543,0.0896897166967392,0.10295859724283218,0.1273450255393982,0.16525308787822723,0.11975577473640442,0.09878180921077728 +B5_mofe_mlp,42,vision_20,0.5117582417582418,0.48824175824175825,0.8111495627938943,0.08063130770882543,0.2933293581008911,0.13767293095588684,0.062129102647304535,0.18726874887943268,0.10258819162845612,0.1186119019985199,0.09839928895235062 +B5_mofe_mlp,42,audio_vision_20,0.5117582417582418,0.48824175824175825,0.7057799858331233,0.2687352372772171,0.40449970960617065,0.08882029354572296,0.06192621961236,0.12605853378772736,0.10232582688331604,0.11834055930376053,0.09802991151809692 +B5_mofe_mlp,42,all_modalities_20,0.41145604395604396,0.588543956043956,0.8795920971657628,0.08853575482406356,0.25787994265556335,0.11065426468849182,0.07723148167133331,0.157210573554039,0.12760961055755615,0.14714252948760986,0.12227214872837067 +B5_mofe_mlp,42,text_30,0.5034615384615385,0.49653846153846154,0.7845061109572483,0.07726727054458146,0.19043782353401184,0.164101243019104,0.12816408276557922,0.11582792550325394,0.0932326465845108,0.218837708234787,0.08939971774816513 +B5_mofe_mlp,42,audio_30,0.5117582417582418,0.48824175824175825,0.7627258720846881,0.09308567747476916,0.32027310132980347,0.0778515487909317,0.11745958030223846,0.11037490516901016,0.18614806234836578,0.10296247154474258,0.08493035286664963 +B5_mofe_mlp,42,vision_30,0.5117582417582418,0.48824175824175825,0.7764460734868434,0.08063130770882543,0.31232842803001404,0.1496150940656662,0.054964445531368256,0.20050112903118134,0.09068562090396881,0.10487580299377441,0.0870303362607956 +B5_mofe_mlp,42,audio_vision_30,0.5117582417582418,0.48824175824175825,0.6149186646941948,0.3631092978312218,0.4812489449977875,0.07732704281806946,0.05401867628097534,0.10968495160341263,0.0891624167561531,0.10309218615293503,0.08546625822782516 +B5_mofe_mlp,42,all_modalities_30,0.3595054945054945,0.6404945054945055,0.8745368649815657,0.09307657038055937,0.26191991567611694,0.11035662144422531,0.07678450644016266,0.1566968709230423,0.12672796845436096,0.14605894684791565,0.12145592272281647 +B5_mofe_mlp,3407,clean,0.5117582417582418,0.48824175824175825,0.8917789142359226,0.08063130770882543,0.21564963459968567,0.16335873305797577,0.12684088945388794,0.1254868507385254,0.09765232354402542,0.11325515806674957,0.15775780379772186 +B5_mofe_mlp,3407,text_10,0.509010989010989,0.490989010989011,0.8547805451037497,0.08031088082901554,0.19791889190673828,0.1904035061597824,0.1427920013666153,0.11356116831302643,0.08846957981586456,0.12491724640130997,0.14193856716156006 +B5_mofe_mlp,3407,audio_10,0.5117582417582418,0.48824175824175825,0.8504781823179828,0.08481855271634099,0.23455235362052917,0.1466052234172821,0.15115498006343842,0.11315018683671951,0.1112954393029213,0.10172916203737259,0.14151319861412048 +B5_mofe_mlp,3407,vision_10,0.5117582417582418,0.48824175824175825,0.8539108633297741,0.08063130770882543,0.2311594933271408,0.1815430223941803,0.11412636935710907,0.1422722488641739,0.08770900219678879,0.10201089829206467,0.14117981493473053 +B5_mofe_mlp,3407,audio_vision_10,0.5117582417582418,0.48824175824175825,0.7997980860345286,0.175434829289242,0.29687586426734924,0.14598697423934937,0.11404559761285782,0.11318352073431015,0.08737359195947647,0.10181976109743118,0.14071525633335114 +B5_mofe_mlp,3407,all_modalities_10,0.4610164835164835,0.5389835164835165,0.8892426095830844,0.08348727727787379,0.2185969352722168,0.16224151849746704,0.12673969566822052,0.12583576142787933,0.09755325317382812,0.11298926174640656,0.1560446321964264 +B5_mofe_mlp,3407,text_20,0.5066758241758241,0.4933241758241758,0.8130542021789052,0.08176543946212655,0.17931020259857178,0.2202228158712387,0.16002467274665833,0.10018529742956161,0.07809362560510635,0.137038454413414,0.12512563169002533 +B5_mofe_mlp,3407,audio_20,0.5117582417582418,0.48824175824175825,0.8087283495175436,0.09002576766158471,0.2538429796695709,0.12989234924316406,0.1755901426076889,0.10037955641746521,0.12468340992927551,0.09022346138954163,0.12538877129554749 +B5_mofe_mlp,3407,vision_20,0.5117582417582418,0.48824175824175825,0.8141886106137566,0.08063130770882543,0.24743759632110596,0.20013628900051117,0.10070110112428665,0.16005626320838928,0.0773327648639679,0.09001021832227707,0.12432596832513809 +B5_mofe_mlp,3407,audio_vision_20,0.5117582417582418,0.48824175824175825,0.6969105807199653,0.28124328967146234,0.3873015344142914,0.1272396296262741,0.09933283925056458,0.09888661652803421,0.07606082409620285,0.08873630315065384,0.12244285643100739 +B5_mofe_mlp,3407,all_modalities_20,0.40958791208791206,0.5904120879120879,0.8837504554280156,0.08867127238580723,0.22335518896579742,0.1614186316728592,0.12589122354984283,0.12545962631702423,0.09687625616788864,0.11222808808088303,0.15477190911769867 +B5_mofe_mlp,3407,text_30,0.5033791208791208,0.4966208791208791,0.77998020566234,0.07853517437100911,0.1615801304578781,0.24538777768611908,0.1755058914422989,0.08944669365882874,0.0692080482840538,0.1480167955160141,0.11085427552461624 +B5_mofe_mlp,3407,audio_30,0.5117582417582418,0.48824175824175825,0.7653189262124539,0.09217307279364398,0.2723240554332733,0.1130220890045166,0.20212848484516144,0.08758600801229477,0.13917095959186554,0.07771366834640503,0.10805478692054749 +B5_mofe_mlp,3407,vision_30,0.5117582417582418,0.48824175824175825,0.7773799776708603,0.08063130770882543,0.26250481605529785,0.21736890077590942,0.08825590461492538,0.17656537890434265,0.06769922375679016,0.07888038456439972,0.10872545838356018 +B5_mofe_mlp,3407,audio_vision_30,0.5117582417582418,0.48824175824175825,0.6151879114518624,0.36498818982177367,0.4589049518108368,0.11260678619146347,0.08762799948453903,0.08767727017402649,0.06704143434762955,0.07825901359319687,0.10788305103778839 +B5_mofe_mlp,3407,all_modalities_30,0.3557967032967033,0.6442032967032967,0.8791751773751674,0.09342907883561115,0.22755521535873413,0.16036975383758545,0.12519316375255585,0.12486399710178375,0.09638682752847672,0.11167588084936142,0.15395526587963104 +B5_mofe_mlp,2026,clean,0.5117582417582418,0.48824175824175825,0.892980397203899,0.08063130770882543,0.2621680796146393,0.13523989915847778,0.11440448462963104,0.1282719075679779,0.10990405827760696,0.12046957015991211,0.12954255938529968 +B5_mofe_mlp,2026,text_10,0.5091208791208791,0.4908791208791209,0.8566292977820127,0.08099503561407295,0.24004511535167694,0.16017793118953705,0.13224223256111145,0.11618264019489288,0.09917686879634857,0.1352321356534958,0.1169428601861 +B5_mofe_mlp,2026,audio_10,0.5117582417582418,0.48824175824175825,0.8513181092443619,0.08616061842387804,0.28532060980796814,0.12107318639755249,0.13112173974514008,0.11523441225290298,0.12272416055202484,0.1084277555346489,0.11609779298305511 +B5_mofe_mlp,2026,vision_10,0.5117582417582418,0.48824175824175825,0.8538046087080117,0.08063130770882543,0.275703102350235,0.15632730722427368,0.10242735594511032,0.14280176162719727,0.09840572625398636,0.1080247089266777,0.11630945652723312 +B5_mofe_mlp,2026,audio_vision_10,0.5117582417582418,0.48824175824175825,0.7992446529652792,0.1772063560231909,0.33916473388671875,0.12080469727516174,0.10259180516004562,0.11497309058904648,0.0985245332121849,0.10807139426469803,0.11586850136518478 +B5_mofe_mlp,2026,all_modalities_10,0.45884615384615385,0.5411538461538462,0.890664129810964,0.08400191593821099,0.2641141414642334,0.13430693745613098,0.11438830941915512,0.1279611438512802,0.10974214971065521,0.1203693225979805,0.12911845743656158 +B5_mofe_mlp,2026,text_20,0.5064560439560439,0.49354395604395607,0.8196418202446201,0.08093300786547328,0.21744248270988464,0.18586474657058716,0.15042437613010406,0.10369155555963516,0.08837979286909103,0.14998102188110352,0.10421676933765411 +B5_mofe_mlp,2026,audio_20,0.5117582417582418,0.48824175824175825,0.8139422624038224,0.0915288812540262,0.30578380823135376,0.10856158286333084,0.14610645174980164,0.1034226343035698,0.134155735373497,0.09755375236272812,0.10441582649946213 +B5_mofe_mlp,2026,vision_20,0.5117582417582418,0.48824175824175825,0.8189807529817417,0.08063130770882543,0.28755468130111694,0.17542824149131775,0.09184657037258148,0.1557445526123047,0.08823608607053757,0.0968705415725708,0.10431937873363495 +B5_mofe_mlp,2026,audio_vision_20,0.5117582417582418,0.48824175824175825,0.7084234520060394,0.27120463817908524,0.41442573070526123,0.10667615383863449,0.09106616675853729,0.10168138891458511,0.0874401405453682,0.09591105580329895,0.10279884934425354 +B5_mofe_mlp,2026,all_modalities_20,0.40585164835164833,0.5941483516483517,0.8838247695015665,0.08962296080687741,0.26898571848869324,0.13424904644489288,0.1132628470659256,0.12779442965984344,0.10867198556661606,0.1192564070224762,0.1277802437543869 +B5_mofe_mlp,2026,text_30,0.5032417582417582,0.49675824175824174,0.7814398465385116,0.08243257997597991,0.19258207082748413,0.21375314891338348,0.1692270040512085,0.09033004194498062,0.07751274853944778,0.16531100869178772,0.09128450602293015 +B5_mofe_mlp,2026,audio_30,0.5117582417582418,0.48824175824175825,0.7664926836291861,0.09711187459738028,0.3314794600009918,0.09291943162679672,0.1653663069009781,0.08855544030666351,0.14884275197982788,0.08355656266212463,0.08927922695875168 +B5_mofe_mlp,2026,vision_30,0.5117582417582418,0.48824175824175825,0.7769904015474858,0.08063130770882543,0.3017820417881012,0.19846147298812866,0.07910744100809097,0.17135068774223328,0.07599635422229767,0.08345004171133041,0.0898519977927208 +B5_mofe_mlp,2026,audio_vision_30,0.5117582417582418,0.48824175824175825,0.6051323653073035,0.3765836375348937,0.49931350350379944,0.09166184812784195,0.07765422761440277,0.0873459056019783,0.07456637173891068,0.08182720094919205,0.08763010799884796 +B5_mofe_mlp,2026,all_modalities_30,0.36043956043956044,0.6395604395604395,0.8852294042453963,0.08887195121951219,0.2681088149547577,0.1340326964855194,0.11355702579021454,0.12770922482013702,0.10893982648849487,0.11956492066383362,0.12808853387832642 diff --git a/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/run_manifest.json b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/run_manifest.json new file mode 100644 index 0000000..29ff1d8 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/run_manifest.json @@ -0,0 +1,86 @@ +{ + "experiment": "Q2 selected models: EarlyConcat + BiGRU and MoFE-7 + MLP Router", + "created_unix": 1790272294.3490393, + "python_version": "3.14.7 (main, Aug 10 2026, 00:00:00) [GCC 16.1.1 20260515 (Red Hat 16.1.1-2)]", + "torch_version": "2.14.0+cu130", + "numpy_version": "2.5.3", + "device": "cuda", + "cuda_device": "NVIDIA GeForce RTX 5070 Ti", + "feature_file": "/home/gloamxun/modeling_zhaocui/E\u9898\u6570\u636e/\u9644\u4ef62-\u6570\u636e\u96c6\u7279\u5f81\u6587\u4ef6/aligned_50.pkl", + "feature_sha256": "66e867aa74bc70a844e806e5571e371c9abb4a35f9e2887ce9b4d97ff2cb8fcd", + "feature_dimensions": { + "text": 768, + "audio": 74, + "vision": 35 + }, + "sequence_length": 50, + "representation_note": "official ordered 50-wordpiece positions; not 50 physical-time bins", + "train_examples": 3395, + "valid_examples": 728, + "train_source_video_groups": 1528, + "valid_source_video_groups": 239, + "train_only_scaler": "/home/gloamxun/modeling_zhaocui/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/aligned_robust_stats.npz", + "scaler_max_abs_difference_from_reference": 0.0, + "test_labels_used": false, + "seeds": [ + 42, + 3407, + 2026 + ], + "epochs_max": 32, + "patience": 6, + "batch_size": 64, + "optimizer": "AdamW(lr=1.5e-4, weight_decay=1e-4), gradient clip 1.0", + "training_mask_augmentation": "same contiguous-block augment_masks protocol for both models", + "validation_conditions": [ + "clean", + "text_10", + "audio_10", + "vision_10", + "audio_vision_10", + "all_modalities_10", + "text_20", + "audio_20", + "vision_20", + "audio_vision_20", + "all_modalities_20", + "text_30", + "audio_30", + "vision_30", + "audio_vision_30", + "all_modalities_30" + ], + "validation_corruption_seed": "seed + 13 + pattern_index*101 + int(rate*1000)", + "loss": "cross_entropy + 0.5*SmoothL1(intensity/3, regression_label/3)", + "models": { + "B0_early_concat": "project modalities independently, concatenate features and masks, then BiGRU", + "B5_mofe_mlp": { + "experts": [ + "T", + "A", + "V", + "TA", + "TV", + "AV", + "TAV" + ], + "router": "MLP over per-position observed values and local observation statistics", + "availability": "hard mask; unavailable expert weights are zero", + "shared_temporal_backbone": "one BiGRU after position-wise expert mixture" + } + }, + "best_epochs": { + "B0_early_concat_seed_42": 4, + "B5_mofe_mlp_seed_42": 4, + "B0_early_concat_seed_3407": 4, + "B5_mofe_mlp_seed_3407": 4, + "B0_early_concat_seed_2026": 3, + "B5_mofe_mlp_seed_2026": 4 + }, + "paired_bootstrap": { + "replicates": 1000, + "seed": 20260924, + "resampling_unit": "source video id", + "paired": true + } +} \ No newline at end of file diff --git a/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/summary.csv b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/summary.csv new file mode 100644 index 0000000..1780101 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/summary.csv @@ -0,0 +1,3 @@ +method,n_seeds,worst_condition,clean_accuracy,clean_accuracy_sd,corrupt_accuracy_mean,corrupt_accuracy_sd,clean_macro_f1,clean_macro_f1_sd,corrupt_macro_f1_mean,corrupt_macro_f1_sd,clean_mae,clean_mae_sd,corrupt_mae_mean,corrupt_mae_sd,clean_pearson,clean_pearson_sd,corrupt_pearson_mean,corrupt_pearson_sd,worst_condition_macro_f1,worst_single_run_macro_f1,text_30_macro_f1,text_30_macro_f1_sd,audio_30_macro_f1,audio_30_macro_f1_sd,vision_30_macro_f1,vision_30_macro_f1_sd,audio_vision_30_macro_f1,audio_vision_30_macro_f1_sd,all_modalities_30_macro_f1,all_modalities_30_macro_f1_sd,corrupt_macro_f1,corrupt_mae,corrupt_pearson +B0_early_concat,3,text_30,0.6259157509157509,0.012689016905266455,0.6226800976800976,0.011084117339613314,0.5803030257575643,0.01849482950112713,0.5751264735191365,0.016846851540436875,0.6362011035283407,0.0035474183737517766,0.6402280926704407,0.004374266641036123,0.6131094329543634,0.003049326453361512,0.6068152013123896,0.004485339859543551,0.5467200505506405,0.5160231153138954,0.5467200505506405,0.02702334845391397,0.5898777795190231,0.01981725011325887,0.5768267974155946,0.018051809113621656,0.5899580041649785,0.021659392511569595,0.5702146248141552,0.01810847481514832,0.5751264735191365,0.6402280926704407,0.6068152013123896 +B5_mofe_mlp,3,text_30,0.6405677655677655,0.011682555697960713,0.6341575091575092,0.008260560563259616,0.6078097938180046,0.012936809470245413,0.5996814814915562,0.009482724136440013,0.6383468906084696,0.01912445943818777,0.6405956546465555,0.018861146501817867,0.61092448719973,0.006638694954493296,0.6049016762167468,0.007374674358633282,0.5698233578745228,0.5665382052109811,0.5698233578745228,0.003414574297503367,0.6023602104251015,0.008745543947786412,0.6006238965304852,0.009937909445226123,0.6083796876372073,0.012980777465454897,0.5944800671676608,0.01638613384398019,0.5996814814915562,0.6405956546465555,0.6049016762167468 diff --git a/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/aurc_mae_by_mode_seed.csv b/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/aurc_mae_by_mode_seed.csv new file mode 100644 index 0000000..5bba15c --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/aurc_mae_by_mode_seed.csv @@ -0,0 +1,25 @@ +method,seed,mask_mode,aurc_mae,rates_realized +B0_early_concat,42,single,0.6401313925363663,"[0.0, 0.03371342897433643, 0.10101503353205936, 0.16749406682715026, 0.233161167473855]" +B0_early_concat,42,sync,0.6433302939648693,"[0.0, 0.045156406674301666, 0.15049590345982738, 0.2869043063014706, 0.4688425032750801]" +B0_early_concat,42,partial,0.643875960390525,"[0.0, 0.040239075593981495, 0.11254067775371336, 0.19417960020144912, 0.3517369377138981]" +B0_early_concat,42,async,0.6419499029297968,"[0.0, 0.042147033495012594, 0.1402331971734938, 0.2674778555592227, 0.43688540417216304]" +B0_early_concat,3407,single,0.6386396612080375,"[0.0, 0.03371342897433643, 0.10101503353205936, 0.16749406682715026, 0.233161167473855]" +B0_early_concat,3407,sync,0.6426227949801453,"[0.0, 0.045156406674301666, 0.15049590345982738, 0.2869043063014706, 0.4688425032750801]" +B0_early_concat,3407,partial,0.6506563048804127,"[0.0, 0.040239075593981495, 0.11254067775371336, 0.19417960020144912, 0.3517369377138981]" +B0_early_concat,3407,async,0.6473036061422844,"[0.0, 0.042147033495012594, 0.1402331971734938, 0.2674778555592227, 0.43688540417216304]" +B0_early_concat,2026,single,0.6465365400894263,"[0.0, 0.03371342897433643, 0.10101503353205936, 0.16749406682715026, 0.233161167473855]" +B0_early_concat,2026,sync,0.6504719890922472,"[0.0, 0.045156406674301666, 0.15049590345982738, 0.2869043063014706, 0.4688425032750801]" +B0_early_concat,2026,partial,0.652087221009127,"[0.0, 0.040239075593981495, 0.11254067775371336, 0.19417960020144912, 0.3517369377138981]" +B0_early_concat,2026,async,0.6503968739690713,"[0.0, 0.042147033495012594, 0.1402331971734938, 0.2674778555592227, 0.43688540417216304]" +B5_mofe_mlp,42,single,0.6329832291613164,"[0.0, 0.03371342897433643, 0.10101503353205936, 0.16749406682715026, 0.233161167473855]" +B5_mofe_mlp,42,sync,0.6325276544269155,"[0.0, 0.045156406674301666, 0.15049590345982738, 0.2869043063014706, 0.4688425032750801]" +B5_mofe_mlp,42,partial,0.6334337662111412,"[0.0, 0.040239075593981495, 0.11254067775371336, 0.19417960020144912, 0.3517369377138981]" +B5_mofe_mlp,42,async,0.6426767924420232,"[0.0, 0.042147033495012594, 0.1402331971734938, 0.2674778555592227, 0.43688540417216304]" +B5_mofe_mlp,3407,single,0.6373919322630748,"[0.0, 0.03371342897433643, 0.10101503353205936, 0.16749406682715026, 0.233161167473855]" +B5_mofe_mlp,3407,sync,0.6479961326945486,"[0.0, 0.045156406674301666, 0.15049590345982738, 0.2869043063014706, 0.4688425032750801]" +B5_mofe_mlp,3407,partial,0.6479020337042737,"[0.0, 0.040239075593981495, 0.11254067775371336, 0.19417960020144912, 0.3517369377138981]" +B5_mofe_mlp,3407,async,0.6466195743207078,"[0.0, 0.042147033495012594, 0.1402331971734938, 0.2674778555592227, 0.43688540417216304]" +B5_mofe_mlp,2026,single,0.6617258705000735,"[0.0, 0.03371342897433643, 0.10101503353205936, 0.16749406682715026, 0.233161167473855]" +B5_mofe_mlp,2026,sync,0.6610093338534158,"[0.0, 0.045156406674301666, 0.15049590345982738, 0.2869043063014706, 0.4688425032750801]" +B5_mofe_mlp,2026,partial,0.6690499670696615,"[0.0, 0.040239075593981495, 0.11254067775371336, 0.19417960020144912, 0.3517369377138981]" +B5_mofe_mlp,2026,async,0.6662885449740689,"[0.0, 0.042147033495012594, 0.1402331971734938, 0.2674778555592227, 0.43688540417216304]" diff --git a/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/aurc_mae_paired_bootstrap.csv b/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/aurc_mae_paired_bootstrap.csv new file mode 100644 index 0000000..6b0688e --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/aurc_mae_paired_bootstrap.csv @@ -0,0 +1,5 @@ +mode,delta_aurc_mae_mofe_minus_earlyconcat,bootstrap_ci_2p5,bootstrap_ci_97p5,bootstrap_probability_delta_lt_0,replicates,resampling_unit,paired,seed +single,0.0022644793635447913,-0.007326506614258393,0.01085375445537662,0.307,1000,source video id,True,20260926 +sync,0.001702680979206006,-0.008553429202151783,0.011162477203288298,0.37,1000,source video id,True,20260926 +partial,0.0012554269016705755,-0.009299259915437324,0.010371210538351492,0.416,1000,source video id,True,20260926 +async,0.005311509565215755,-0.0041539559800425705,0.013656995740298614,0.149,1000,source video id,True,20260926 diff --git a/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/aurc_mae_summary.csv b/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/aurc_mae_summary.csv new file mode 100644 index 0000000..7915ef7 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/aurc_mae_summary.csv @@ -0,0 +1,9 @@ +method,mask_mode,mean,sd_across_seeds +B0_early_concat,single,0.6417691979446101,0.004195471240907912 +B0_early_concat,sync,0.6454750260124206,0.004341931450312084 +B0_early_concat,partial,0.6488731620933549,0.004386444902871605 +B0_early_concat,async,0.6465501276803842,0.004273596527398931 +B5_mofe_mlp,single,0.6440336773081549,0.015479646013878707 +B5_mofe_mlp,sync,0.6471777069916266,0.014258467000764302 +B5_mofe_mlp,partial,0.6501285889950255,0.01791219144174424 +B5_mofe_mlp,async,0.6518616372456,0.012648641515595156 diff --git a/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/controlled_metrics_by_scenario.csv b/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/controlled_metrics_by_scenario.csv new file mode 100644 index 0000000..a49ea2b --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/controlled_metrics_by_scenario.csv @@ -0,0 +1,253 @@ +method,seed,scenario,realized_additional_global_rate,n_valid,accuracy,macro_f1,mae,rmse,pearson +B0_early_concat,42,0.0/none,0.0,728,0.6332417582417582,0.587941053090477,0.6350555419921875,0.8411936357612951,0.6157338827731829 +B0_early_concat,42,0.1/single,0.03371342897433643,728,0.6304945054945055,0.5841478555667776,0.6329803466796875,0.8388883612450743,0.6156007751289291 +B0_early_concat,42,0.1/sync,0.045156406674301666,728,0.6277472527472527,0.5817500837463079,0.6324868202209473,0.8354757584548449,0.6182538400699824 +B0_early_concat,42,0.1/partial,0.040239075593981495,728,0.6291208791208791,0.5809513824305218,0.6362647414207458,0.8395663845790533,0.6146226232321855 +B0_early_concat,42,0.1/async,0.042147033495012594,728,0.6304945054945055,0.5813440322183995,0.6335626244544983,0.837625919668868,0.6163399508733897 +B0_early_concat,42,0.3/single,0.10101503353205936,728,0.6195054945054945,0.5675794227375214,0.640035092830658,0.8424158498015616,0.6069511611336497 +B0_early_concat,42,0.3/sync,0.15049590345982738,728,0.6263736263736264,0.5780608118498217,0.6299555897712708,0.8375190319916388,0.6124431874344702 +B0_early_concat,42,0.3/partial,0.11254067775371336,728,0.6153846153846154,0.5593362859645706,0.641963541507721,0.8404494474527319,0.6049209455061247 +B0_early_concat,42,0.3/async,0.1402331971734938,728,0.6318681318681318,0.576909502145226,0.6313087344169617,0.8365481276590316,0.6093167943618667 +B0_early_concat,42,0.5/single,0.16749406682715026,728,0.6291208791208791,0.5812458325756198,0.6444581151008606,0.8504945900849837,0.595658614323956 +B0_early_concat,42,0.5/sync,0.2869043063014706,728,0.6126373626373627,0.5593171296296297,0.6449117064476013,0.8485519317960112,0.5940501662945247 +B0_early_concat,42,0.5/partial,0.19417960020144912,728,0.6112637362637363,0.5538751092290113,0.6426721215248108,0.846240606269933,0.5907667712241222 +B0_early_concat,42,0.5/async,0.2674778555592227,728,0.6016483516483516,0.5399077899607511,0.6515048742294312,0.8615156389124333,0.5754295119229905 +B0_early_concat,42,0.7/single,0.233161167473855,728,0.625,0.5688829757001009,0.6452269554138184,0.8590691802596665,0.5839596403467362 +B0_early_concat,42,0.7/sync,0.4688425032750801,728,0.6016483516483516,0.5502346950183244,0.66935795545578,0.8879573422702182,0.5454190465035358 +B0_early_concat,42,0.7/partial,0.3517369377138981,728,0.6016483516483516,0.5386532608428335,0.6552613377571106,0.8613171911951291,0.5706160481156233 +B0_early_concat,42,0.7/async,0.43688540417216304,728,0.6057692307692307,0.5448128152476447,0.6480301022529602,0.8642782952419213,0.5654181079796473 +B0_early_concat,42,0.3/modality_T,0.05864506749840218,728,0.625,0.5763071383484074,0.6379959583282471,0.8473427188282466,0.5955566859385917 +B0_early_concat,42,0.3/modality_A,0.05697909847839254,728,0.6401098901098901,0.5987085232857344,0.6355111002922058,0.8415443767490056,0.6168384537396358 +B0_early_concat,42,0.3/modality_V,0.05703409792921816,728,0.6304945054945055,0.5826704686319633,0.6378379464149475,0.8427720583730931,0.6139202299259341 +B0_early_concat,42,0.3/modality_TA,0.10919187418833474,728,0.6195054945054945,0.5673255633255633,0.6411880850791931,0.8501693809265032,0.593141387871954 +B0_early_concat,42,0.3/modality_TV,0.11437644408481795,728,0.6181318681318682,0.5667943482947192,0.6455541849136353,0.8531380173804407,0.5879106305784599 +B0_early_concat,42,0.3/modality_AV,0.11303721690563862,728,0.6332417582417582,0.589575000050125,0.6335181593894958,0.8394406082973969,0.6190294231457893 +B0_early_concat,42,0.3/modality_TAV,0.16956369042443203,728,0.625,0.5736949459376823,0.636223316192627,0.8385192007210537,0.6136927230613016 +B0_early_concat,42,0.3/location_start_T,0.10048832030833439,728,0.614010989010989,0.5486629805996582,0.639001727104187,0.8471883018318189,0.5899140511730886 +B0_early_concat,42,0.3/location_middle_T,0.10048832030833439,728,0.6195054945054945,0.5621330833277529,0.6401935815811157,0.8398970121305769,0.6011036539570971 +B0_early_concat,42,0.3/location_end_T,0.10048832030833439,728,0.6167582417582418,0.558781569307885,0.6448237895965576,0.8462808587041787,0.595287414298351 +B0_early_concat,42,0.3/span_long_T,0.10048832030833439,728,0.6167582417582418,0.5587239725548967,0.6371496915817261,0.8397095329026115,0.5977651886958409 +B0_early_concat,42,0.3/span_multi_short_T,0.10048832030833439,728,0.6167582417582418,0.5618207578802662,0.6346496939659119,0.8382192825209313,0.5993613309755399 +B0_early_concat,42,0.3/location_start_A,0.10087109490306286,728,0.635989010989011,0.5935620701417008,0.6319957971572876,0.84067373737167,0.6193052962176031 +B0_early_concat,42,0.3/location_middle_A,0.10087109490306286,728,0.6304945054945055,0.5891673048189151,0.6320388317108154,0.8388446986922057,0.6193795942976837 +B0_early_concat,42,0.3/location_end_A,0.10087109490306286,728,0.6304945054945055,0.5891673048189151,0.6308991312980652,0.8360611654771293,0.6216326232820929 +B0_early_concat,42,0.3/span_long_A,0.10087109490306286,728,0.6304945054945055,0.586797819903155,0.6292437314987183,0.8373133664919333,0.6218214161352669 +B0_early_concat,42,0.3/span_multi_short_A,0.10087109490306286,728,0.6332417582417582,0.5918991467740389,0.6312629580497742,0.8393302944773211,0.620064068130471 +B0_early_concat,42,0.3/location_start_V,0.09701596125937247,728,0.6373626373626373,0.5909589685370776,0.638043999671936,0.8432465570143738,0.6133269596558762 +B0_early_concat,42,0.3/location_middle_V,0.09701596125937247,728,0.6291208791208791,0.5802229949228151,0.636878252029419,0.8422387328692155,0.6142276235341132 +B0_early_concat,42,0.3/location_end_V,0.09701596125937247,728,0.6291208791208791,0.5802229949228151,0.6369577050209045,0.8424747861717187,0.6139934605500105 +B0_early_concat,42,0.3/span_long_V,0.09701596125937247,728,0.6304945054945055,0.583777087861221,0.6353855133056641,0.8398962669808279,0.6167031708983985 +B0_early_concat,42,0.3/span_multi_short_V,0.09701596125937247,728,0.6304945054945055,0.5833277915350489,0.6393547654151917,0.8443521540873805,0.6124599424438969 +B0_early_concat,42,0.3/synchrony_sync,0.17627119309968198,728,0.6236263736263736,0.5712425532056137,0.6429818272590637,0.8454751991613966,0.6058335119911614 +B0_early_concat,42,0.3/synchrony_partial,0.11943882005826376,728,0.6304945054945055,0.5832748099288496,0.6332861185073853,0.8400844140439543,0.6035583549739136 +B0_early_concat,42,0.3/synchrony_async,0.16594815654028122,728,0.6332417582417582,0.581960911374582,0.634742021560669,0.8411228109856493,0.6030323439352911 +B0_early_concat,3407,0.0/none,0.0,728,0.6332417582417582,0.5937554950072336,0.6333680152893066,0.8346194991301408,0.6138300851146614 +B0_early_concat,3407,0.1/single,0.03371342897433643,728,0.6277472527472527,0.5863431107430745,0.6328662633895874,0.8353187909586893,0.6112280473426515 +B0_early_concat,3407,0.1/sync,0.045156406674301666,728,0.6263736263736264,0.5838823759644308,0.6331413984298706,0.8319518002110763,0.614769736946681 +B0_early_concat,3407,0.1/partial,0.040239075593981495,728,0.6277472527472527,0.5826595399578411,0.6362914443016052,0.8349912760615754,0.6108295560922031 +B0_early_concat,3407,0.1/async,0.042147033495012594,728,0.6401098901098901,0.5984721457021069,0.6350982785224915,0.8343727940448075,0.6137234888279289 +B0_early_concat,3407,0.3/single,0.10101503353205936,728,0.6085164835164835,0.563921331484755,0.6361177563667297,0.8353053046479242,0.607661728262342 +B0_early_concat,3407,0.3/sync,0.15049590345982738,728,0.6304945054945055,0.5883092160267541,0.6361386775970459,0.8404312208242788,0.6029838091344039 +B0_early_concat,3407,0.3/partial,0.11254067775371336,728,0.6112637362637363,0.5649195311816672,0.642254650592804,0.8373105190635507,0.6025399956015483 +B0_early_concat,3407,0.3/async,0.1402331971734938,728,0.6236263736263736,0.5729838717006412,0.6402410268783569,0.8337487059759023,0.6076096623115538 +B0_early_concat,3407,0.5/single,0.16749406682715026,728,0.6181318681318682,0.5715556497676894,0.6397385597229004,0.8471463687132873,0.5913795821530734 +B0_early_concat,3407,0.5/sync,0.2869043063014706,728,0.6236263736263736,0.575541941133339,0.6427107453346252,0.8423896702594076,0.5942984252907214 +B0_early_concat,3407,0.5/partial,0.19417960020144912,728,0.5961538461538461,0.5415150811859474,0.6579906940460205,0.8591292636831587,0.5722431967077758 +B0_early_concat,3407,0.5/async,0.2674778555592227,728,0.603021978021978,0.5529601029601029,0.6514623165130615,0.8635603164542541,0.5680182640279093 +B0_early_concat,3407,0.7/single,0.233161167473855,728,0.603021978021978,0.5548110166904493,0.653153657913208,0.864040575904769,0.5720059484103266 +B0_early_concat,3407,0.7/sync,0.4688425032750801,728,0.6071428571428571,0.5592133635943569,0.6612244248390198,0.8699659907610593,0.5613123004112319 +B0_early_concat,3407,0.7/partial,0.3517369377138981,728,0.5782967032967034,0.5192448487580333,0.6624062657356262,0.8770518088093162,0.5473815485219784 +B0_early_concat,3407,0.7/async,0.43688540417216304,728,0.5782967032967034,0.5296829018983362,0.6629856824874878,0.8789931529844939,0.5433830120722912 +B0_early_concat,3407,0.3/modality_T,0.05864506749840218,728,0.6167582417582418,0.5696594236400062,0.6442276835441589,0.8481597459797712,0.5886136769457235 +B0_early_concat,3407,0.3/modality_A,0.05697909847839254,728,0.6332417582417582,0.5935566473770096,0.6333451867103577,0.8344743705182801,0.6155545899116962 +B0_early_concat,3407,0.3/modality_V,0.05703409792921816,728,0.6332417582417582,0.5934148419250151,0.6332776546478271,0.8344196193351635,0.6139845146475589 +B0_early_concat,3407,0.3/modality_TA,0.10919187418833474,728,0.6167582417582418,0.569608407866905,0.6469659209251404,0.8517946749213303,0.5848663158386197 +B0_early_concat,3407,0.3/modality_TV,0.11437644408481795,728,0.6057692307692307,0.5520132224603463,0.6410850882530212,0.8440129965170219,0.5908302231341378 +B0_early_concat,3407,0.3/modality_AV,0.11303721690563862,728,0.635989010989011,0.5984543253270477,0.6292101740837097,0.8312502136803833,0.6188758147944096 +B0_early_concat,3407,0.3/modality_TAV,0.16956369042443203,728,0.6222527472527473,0.577255329165107,0.6411339044570923,0.8406993677079958,0.6056894215716182 +B0_early_concat,3407,0.3/location_start_T,0.10048832030833439,728,0.6167582417582418,0.5616277516526954,0.6559486985206604,0.854438684150858,0.5788422305236482 +B0_early_concat,3407,0.3/location_middle_T,0.10048832030833439,728,0.6071428571428571,0.547740563024856,0.6456643342971802,0.8450714289366523,0.59239383541254 +B0_early_concat,3407,0.3/location_end_T,0.10048832030833439,728,0.6071428571428571,0.5468314040316676,0.6502484083175659,0.8484889919328977,0.5891420561980362 +B0_early_concat,3407,0.3/span_long_T,0.10048832030833439,728,0.6016483516483516,0.546928522672282,0.6454964280128479,0.8463376948392111,0.5878408095083768 +B0_early_concat,3407,0.3/span_multi_short_T,0.10048832030833439,728,0.6043956043956044,0.5515248320550723,0.6477841138839722,0.8459924639012224,0.5877258960827487 +B0_early_concat,3407,0.3/location_start_A,0.10087109490306286,728,0.6263736263736264,0.5891968004925136,0.63096684217453,0.8328749627885234,0.618139931953556 +B0_early_concat,3407,0.3/location_middle_A,0.10087109490306286,728,0.6373626373626373,0.5982533903882219,0.6332005262374878,0.8344721205404403,0.6156531655487061 +B0_early_concat,3407,0.3/location_end_A,0.10087109490306286,728,0.6401098901098901,0.6003231457093513,0.6325213313102722,0.8332261294709254,0.6168599173095554 +B0_early_concat,3407,0.3/span_long_A,0.10087109490306286,728,0.6304945054945055,0.5918206496100208,0.6307344436645508,0.8312857068072388,0.6198440817316815 +B0_early_concat,3407,0.3/span_multi_short_A,0.10087109490306286,728,0.6332417582417582,0.595786573091332,0.6304507851600647,0.8323247687322188,0.6185376730913125 +B0_early_concat,3407,0.3/location_start_V,0.09701596125937247,728,0.6318681318681318,0.5918186769129088,0.6308284997940063,0.8326575559153797,0.6164779072814917 +B0_early_concat,3407,0.3/location_middle_V,0.09701596125937247,728,0.6318681318681318,0.5930726844490408,0.6334424018859863,0.835375730759368,0.6128703835769779 +B0_early_concat,3407,0.3/location_end_V,0.09701596125937247,728,0.6304945054945055,0.5915655585223337,0.6334879398345947,0.835227415159139,0.6128021593800222 +B0_early_concat,3407,0.3/span_long_V,0.09701596125937247,728,0.6346153846153846,0.5958501672120969,0.6301674246788025,0.832417931048369,0.6161365463499257 +B0_early_concat,3407,0.3/span_multi_short_V,0.09701596125937247,728,0.6291208791208791,0.588772574029934,0.6311795711517334,0.833889672220184,0.6149213762305303 +B0_early_concat,3407,0.3/synchrony_sync,0.17627119309968198,728,0.6304945054945055,0.5874171274320514,0.6434048414230347,0.8416043656648424,0.6052197775255865 +B0_early_concat,3407,0.3/synchrony_partial,0.11943882005826376,728,0.6153846153846154,0.5725142421088963,0.638130247592926,0.8434438856574996,0.5938085222353164 +B0_early_concat,3407,0.3/synchrony_async,0.16594815654028122,728,0.6263736263736264,0.5815558788981249,0.639380156993866,0.8382507475096762,0.601474525688235 +B0_early_concat,2026,0.0/none,0.0,728,0.6112637362637363,0.5592125291749822,0.6401797533035278,0.8435639776807761,0.6097643309752457 +B0_early_concat,2026,0.1/single,0.03371342897433643,728,0.6126373626373627,0.5601732347712779,0.6388869881629944,0.8425065874583629,0.6090522004630045 +B0_early_concat,2026,0.1/sync,0.045156406674301666,728,0.6208791208791209,0.5649448309789866,0.6389307975769043,0.8412108184623058,0.6108543782865608 +B0_early_concat,2026,0.1/partial,0.040239075593981495,728,0.614010989010989,0.5584513367819262,0.6431105732917786,0.8447519993121942,0.6059585962909831 +B0_early_concat,2026,0.1/async,0.042147033495012594,728,0.614010989010989,0.5589535709422427,0.6394624710083008,0.8424333613384806,0.6086355218373317 +B0_early_concat,2026,0.3/single,0.10101503353205936,728,0.5961538461538461,0.5303411844263911,0.644254207611084,0.8467073887008049,0.6011147024802118 +B0_early_concat,2026,0.3/sync,0.15049590345982738,728,0.6153846153846154,0.5586649169024365,0.6396176815032959,0.8460189546756811,0.6038839691724569 +B0_early_concat,2026,0.3/partial,0.11254067775371336,728,0.6002747252747253,0.5414842809750425,0.6449089646339417,0.8454197856067814,0.6000672714828029 +B0_early_concat,2026,0.3/async,0.1402331971734938,728,0.6167582417582418,0.5574463118580766,0.6412432193756104,0.8442993142001068,0.6000500422354057 +B0_early_concat,2026,0.5/single,0.16749406682715026,728,0.5906593406593407,0.5343478616835018,0.6554132699966431,0.8629948412784115,0.5779273255314922 +B0_early_concat,2026,0.5/sync,0.2869043063014706,728,0.6057692307692307,0.5402298406725984,0.6494545936584473,0.8566324340595707,0.5939648586438693 +B0_early_concat,2026,0.5/partial,0.19417960020144912,728,0.5892857142857143,0.5226659187505066,0.6543736457824707,0.8614206069809918,0.5773399814562676 +B0_early_concat,2026,0.5/async,0.2674778555592227,728,0.5934065934065934,0.5295882956865473,0.6541514992713928,0.8724610071841917,0.5626896612214186 +B0_early_concat,2026,0.7/single,0.233161167473855,728,0.5934065934065934,0.532628227109786,0.6483538150787354,0.870265481387598,0.5673644852821225 +B0_early_concat,2026,0.7/sync,0.4688425032750801,728,0.5961538461538461,0.5301148286882755,0.6787758469581604,0.8906546470660056,0.547949008886643 +B0_early_concat,2026,0.7/partial,0.3517369377138981,728,0.5782967032967034,0.49990870782827085,0.6650825142860413,0.8827671950876751,0.5543444352742108 +B0_early_concat,2026,0.7/async,0.43688540417216304,728,0.5714285714285714,0.5025610545542795,0.6675907373428345,0.8872400843457747,0.5422344198391074 +B0_early_concat,2026,0.3/modality_T,0.05864506749840218,728,0.5947802197802198,0.5351898961754573,0.6460724472999573,0.8546291394157867,0.5876677547086017 +B0_early_concat,2026,0.3/modality_A,0.05697909847839254,728,0.6167582417582418,0.5662772023588923,0.6368343830108643,0.8410307546655318,0.6118634446955162 +B0_early_concat,2026,0.3/modality_V,0.05703409792921816,728,0.6126373626373627,0.557729972907977,0.6418033838272095,0.8440729159094172,0.6100584181871076 +B0_early_concat,2026,0.3/modality_TA,0.10919187418833474,728,0.603021978021978,0.5448842919697842,0.6518500447273254,0.8589307850717696,0.5797191740283412 +B0_early_concat,2026,0.3/modality_TV,0.11437644408481795,728,0.603021978021978,0.541860535364365,0.6528465151786804,0.8566144821832752,0.588955738533617 +B0_early_concat,2026,0.3/modality_AV,0.11303721690563862,728,0.6304945054945055,0.5825187723642906,0.6375922560691833,0.8408415788360272,0.612813220807612 +B0_early_concat,2026,0.3/modality_TAV,0.16956369042443203,728,0.6057692307692307,0.5458579640041461,0.6483514904975891,0.8455093549439832,0.608450763176158 +B0_early_concat,2026,0.3/location_start_T,0.10048832030833439,728,0.5975274725274725,0.5154262536957387,0.6474388837814331,0.8511553849841474,0.5876055423314643 +B0_early_concat,2026,0.3/location_middle_T,0.10048832030833439,728,0.5947802197802198,0.5273134273577562,0.651376485824585,0.8531833936108886,0.5919773613751391 +B0_early_concat,2026,0.3/location_end_T,0.10048832030833439,728,0.5989010989010989,0.5317455371915709,0.6568161845207214,0.8567943666389586,0.5878101230844675 +B0_early_concat,2026,0.3/span_long_T,0.10048832030833439,728,0.592032967032967,0.5220599876513855,0.6480600833892822,0.8553228036066841,0.5854352052042763 +B0_early_concat,2026,0.3/span_multi_short_T,0.10048832030833439,728,0.5906593406593407,0.5159900126701303,0.6425683498382568,0.8479485073875213,0.5947492287997942 +B0_early_concat,2026,0.3/location_start_A,0.10087109490306286,728,0.6126373626373627,0.563729128324724,0.6361474394798279,0.8418471814765383,0.6102622224288402 +B0_early_concat,2026,0.3/location_middle_A,0.10087109490306286,728,0.6153846153846154,0.5629585797451505,0.6369883418083191,0.8422769120407221,0.6104588818493184 +B0_early_concat,2026,0.3/location_end_A,0.10087109490306286,728,0.6126373626373627,0.5599234991449159,0.6361879110336304,0.8414413515665965,0.6112857790951658 +B0_early_concat,2026,0.3/span_long_A,0.10087109490306286,728,0.6181318681318682,0.5672427371505847,0.6351941227912903,0.84013680956134,0.6122003841323984 +B0_early_concat,2026,0.3/span_multi_short_A,0.10087109490306286,728,0.6126373626373627,0.5600733772344474,0.6348678469657898,0.8409436143236783,0.6113205736925018 +B0_early_concat,2026,0.3/location_start_V,0.09701596125937247,728,0.6126373626373627,0.5611197593956215,0.64106684923172,0.8454543313788724,0.6088502283110014 +B0_early_concat,2026,0.3/location_middle_V,0.09701596125937247,728,0.6085164835164835,0.5557324605271149,0.642802894115448,0.8447385930433952,0.6088194139871752 +B0_early_concat,2026,0.3/location_end_V,0.09701596125937247,728,0.6098901098901099,0.5575133773353141,0.6417749524116516,0.8442896777118307,0.6093391576059886 +B0_early_concat,2026,0.3/span_long_V,0.09701596125937247,728,0.6181318681318682,0.5667042220475478,0.6411034464836121,0.844152566453742,0.6097178840476188 +B0_early_concat,2026,0.3/span_multi_short_V,0.09701596125937247,728,0.6153846153846154,0.5651938412949072,0.642799437046051,0.845178842466505,0.6091480103515892 +B0_early_concat,2026,0.3/synchrony_sync,0.17627119309968198,728,0.6057692307692307,0.5482100254697998,0.6528782844543457,0.8552752759399308,0.5958982611163057 +B0_early_concat,2026,0.3/synchrony_partial,0.11943882005826376,728,0.6126373626373627,0.5553496423474522,0.6373363137245178,0.8448557143072355,0.5996811452037548 +B0_early_concat,2026,0.3/synchrony_async,0.16594815654028122,728,0.6208791208791209,0.5618089083713896,0.642960786819458,0.8441256287463826,0.6005135762432049 +B5_mofe_mlp,42,0.0/none,0.0,728,0.6538461538461539,0.6219292861291515,0.6228764653205872,0.8430840725364007,0.6143991595201629 +B5_mofe_mlp,42,0.1/single,0.03371342897433643,728,0.6483516483516484,0.6176568539994053,0.6194556951522827,0.8364675392287196,0.6155295037181994 +B5_mofe_mlp,42,0.1/sync,0.045156406674301666,728,0.646978021978022,0.6131089418994519,0.6230729222297668,0.8388785560201717,0.6141908451088897 +B5_mofe_mlp,42,0.1/partial,0.040239075593981495,728,0.6524725274725275,0.6193363630301347,0.6241474151611328,0.8379474256065216,0.6137031429539084 +B5_mofe_mlp,42,0.1/async,0.042147033495012594,728,0.6497252747252747,0.6159992213876729,0.6239848732948303,0.8404273555935142,0.61353036159771 +B5_mofe_mlp,42,0.3/single,0.10101503353205936,728,0.6373626373626373,0.5985146260620283,0.6270084977149963,0.8386096847780098,0.6054063180216553 +B5_mofe_mlp,42,0.3/sync,0.15049590345982738,728,0.6414835164835165,0.6040789612486607,0.6249569058418274,0.8408457611521547,0.6049185603235804 +B5_mofe_mlp,42,0.3/partial,0.11254067775371336,728,0.6304945054945055,0.5891368041469461,0.6224022507667542,0.827918548761029,0.6144819802426599 +B5_mofe_mlp,42,0.3/async,0.1402331971734938,728,0.6483516483516484,0.6074532789670405,0.6267399191856384,0.8365157435862942,0.60522919392934 +B5_mofe_mlp,42,0.5/single,0.16749406682715026,728,0.6373626373626373,0.5935685808642579,0.641523003578186,0.8501884503989168,0.5866277942269824 +B5_mofe_mlp,42,0.5/sync,0.2869043063014706,728,0.6277472527472527,0.5844696109712227,0.6337242722511292,0.8441922476966766,0.5913227359189296 +B5_mofe_mlp,42,0.5/partial,0.19417960020144912,728,0.6346153846153846,0.5872227018393876,0.6360241174697876,0.8445437900624987,0.5922147321419193 +B5_mofe_mlp,42,0.5/async,0.2674778555592227,728,0.6071428571428571,0.5548667051067803,0.6596632599830627,0.8709813887997953,0.5530374150565661 +B5_mofe_mlp,42,0.7/single,0.233161167473855,728,0.6085164835164835,0.5566056168847249,0.6539682149887085,0.8732921417971317,0.5559331721614461 +B5_mofe_mlp,42,0.7/sync,0.4688425032750801,728,0.6153846153846154,0.5722363903398882,0.6507095694541931,0.8706489761786416,0.5546201796993863 +B5_mofe_mlp,42,0.7/partial,0.3517369377138981,728,0.6112637362637363,0.5491367242946944,0.649608850479126,0.8622736163821897,0.5674264189655753 +B5_mofe_mlp,42,0.7/async,0.43688540417216304,728,0.614010989010989,0.5607072922993648,0.6545283794403076,0.8672503886745678,0.5562671204175875 +B5_mofe_mlp,42,0.3/modality_T,0.05864506749840218,728,0.6373626373626373,0.5968187892381337,0.6317851543426514,0.8464475179199263,0.6006693398835592 +B5_mofe_mlp,42,0.3/modality_A,0.05697909847839254,728,0.646978021978022,0.6163107544157121,0.6207097768783569,0.838498053196448,0.6142489597323474 +B5_mofe_mlp,42,0.3/modality_V,0.05703409792921816,728,0.6442307692307693,0.6107151863557211,0.6259862780570984,0.8335144362752684,0.6153024937943374 +B5_mofe_mlp,42,0.3/modality_TA,0.10919187418833474,728,0.6428571428571429,0.6043780415380584,0.6327931880950928,0.8475631100160345,0.5948025418510873 +B5_mofe_mlp,42,0.3/modality_TV,0.11437644408481795,728,0.6373626373626373,0.5959482244281312,0.6271441578865051,0.8361193735369933,0.6012465766270015 +B5_mofe_mlp,42,0.3/modality_AV,0.11303721690563862,728,0.657967032967033,0.6265967365967366,0.6204225420951843,0.8320528842459516,0.6142369587359615 +B5_mofe_mlp,42,0.3/modality_TAV,0.16956369042443203,728,0.6387362637362637,0.6022829403270885,0.6247004270553589,0.8415388521710019,0.6114450566462162 +B5_mofe_mlp,42,0.3/location_start_T,0.10048832030833439,728,0.6167582417582418,0.5551944119171176,0.6426858305931091,0.8588827630798264,0.5909442057230415 +B5_mofe_mlp,42,0.3/location_middle_T,0.10048832030833439,728,0.6277472527472527,0.5816861376992662,0.6292437314987183,0.8380489244291087,0.6084504680158224 +B5_mofe_mlp,42,0.3/location_end_T,0.10048832030833439,728,0.6291208791208791,0.5828384880080901,0.6327103972434998,0.8456859983335451,0.6009234012556889 +B5_mofe_mlp,42,0.3/span_long_T,0.10048832030833439,728,0.6181318681318682,0.5606029894245005,0.63385009765625,0.8449191717280062,0.5999291487943097 +B5_mofe_mlp,42,0.3/span_multi_short_T,0.10048832030833439,728,0.6236263736263736,0.5675871193351786,0.6286787390708923,0.8434076674373588,0.6027067451671214 +B5_mofe_mlp,42,0.3/location_start_A,0.10087109490306286,728,0.6497252747252747,0.6197366933833278,0.6174412965774536,0.8388514134230686,0.6113825928060521 +B5_mofe_mlp,42,0.3/location_middle_A,0.10087109490306286,728,0.646978021978022,0.6173225209844928,0.616412878036499,0.8338065393313597,0.6161370673969482 +B5_mofe_mlp,42,0.3/location_end_A,0.10087109490306286,728,0.646978021978022,0.6165553163133471,0.6158262491226196,0.8313431379297845,0.6183121117147332 +B5_mofe_mlp,42,0.3/span_long_A,0.10087109490306286,728,0.6483516483516484,0.6173720425147514,0.6175615787506104,0.8363065172492736,0.6136977910950321 +B5_mofe_mlp,42,0.3/span_multi_short_A,0.10087109490306286,728,0.6524725274725275,0.6230131272943935,0.61532062292099,0.8349310975570658,0.6150409227419862 +B5_mofe_mlp,42,0.3/location_start_V,0.09701596125937247,728,0.6442307692307693,0.6101387544638678,0.6222208738327026,0.8278320442101604,0.6172869524300535 +B5_mofe_mlp,42,0.3/location_middle_V,0.09701596125937247,728,0.6442307692307693,0.6093692166419439,0.6222869753837585,0.8253240180231957,0.6206632365514538 +B5_mofe_mlp,42,0.3/location_end_V,0.09701596125937247,728,0.6414835164835165,0.6068048524614181,0.6223054528236389,0.8262935756209701,0.619957370820794 +B5_mofe_mlp,42,0.3/span_long_V,0.09701596125937247,728,0.646978021978022,0.6112351316308698,0.622149646282196,0.8255536446935012,0.6199181054653543 +B5_mofe_mlp,42,0.3/span_multi_short_V,0.09701596125937247,728,0.6428571428571429,0.6082354427182013,0.6236581206321716,0.8262526380234242,0.6195939305316789 +B5_mofe_mlp,42,0.3/synchrony_sync,0.17627119309968198,728,0.6373626373626373,0.6012358038786352,0.6281257271766663,0.8476797355527455,0.6060950536914038 +B5_mofe_mlp,42,0.3/synchrony_partial,0.11943882005826376,728,0.6346153846153846,0.5923570448352682,0.6266241669654846,0.8353940676885147,0.6059126232533022 +B5_mofe_mlp,42,0.3/synchrony_async,0.16594815654028122,728,0.6346153846153846,0.5925850106177974,0.6234601140022278,0.8406986941683526,0.5973964270479869 +B5_mofe_mlp,3407,0.0/none,0.0,728,0.635989010989011,0.604973907499834,0.6324349641799927,0.838804657884871,0.6151046893704437 +B5_mofe_mlp,3407,0.1/single,0.03371342897433643,728,0.6373626373626373,0.6048865594071073,0.6280990242958069,0.8329202621621531,0.6151751620920511 +B5_mofe_mlp,3407,0.1/sync,0.045156406674301666,728,0.6304945054945055,0.5987230879568809,0.6321651935577393,0.8356450719141751,0.6145683643566429 +B5_mofe_mlp,3407,0.1/partial,0.040239075593981495,728,0.6332417582417582,0.598012204601852,0.6343168616294861,0.836387477516575,0.6110104686409112 +B5_mofe_mlp,3407,0.1/async,0.042147033495012594,728,0.635989010989011,0.6058945511736596,0.6329501867294312,0.8365664388900896,0.6153538223070723 +B5_mofe_mlp,3407,0.3/single,0.10101503353205936,728,0.6346153846153846,0.5990035446502852,0.6325004696846008,0.8355625773519788,0.6046990738861794 +B5_mofe_mlp,3407,0.3/sync,0.15049590345982738,728,0.6304945054945055,0.5957759196211425,0.6371961832046509,0.84088180618401,0.602958316177432 +B5_mofe_mlp,3407,0.3/partial,0.11254067775371336,728,0.6208791208791209,0.5825165271433349,0.6410707831382751,0.8389358225978112,0.598158496010818 +B5_mofe_mlp,3407,0.3/async,0.1402331971734938,728,0.6318681318681318,0.5942475648935711,0.6402276158332825,0.839390974198143,0.6007527560377187 +B5_mofe_mlp,3407,0.5/single,0.16749406682715026,728,0.6332417582417582,0.5948964744967374,0.6464778184890747,0.8446069533500644,0.5901295001763649 +B5_mofe_mlp,3407,0.5/sync,0.2869043063014706,728,0.6318681318681318,0.5973188250716287,0.6492260694503784,0.8514326854066492,0.5825697846418139 +B5_mofe_mlp,3407,0.5/partial,0.19417960020144912,728,0.6181318681318682,0.5737695892370467,0.6486934423446655,0.8496342797849444,0.5804087890235984 +B5_mofe_mlp,3407,0.5/async,0.2674778555592227,728,0.6112637362637363,0.5720071090958225,0.6490155458450317,0.8661096765277796,0.559367928353015 +B5_mofe_mlp,3407,0.7/single,0.233161167473855,728,0.5975274725274725,0.5517406725325694,0.6459130644798279,0.8581754114801059,0.5724437675509451 +B5_mofe_mlp,3407,0.7/sync,0.4688425032750801,728,0.5892857142857143,0.552859114928787,0.677151620388031,0.8907122987492757,0.531687592193011 +B5_mofe_mlp,3407,0.7/partial,0.3517369377138981,728,0.5892857142857143,0.5342811882319379,0.6670288443565369,0.8729558978373072,0.5473239468352531 +B5_mofe_mlp,3407,0.7/async,0.43688540417216304,728,0.5810439560439561,0.5320545275034573,0.6657702922821045,0.8800158807557013,0.5376864140920043 +B5_mofe_mlp,3407,0.3/modality_T,0.05864506749840218,728,0.6346153846153846,0.592395637194854,0.6376990079879761,0.8418836438274057,0.5952924527832733 +B5_mofe_mlp,3407,0.3/modality_A,0.05697909847839254,728,0.6346153846153846,0.6053242660314008,0.631351888179779,0.8380616909298862,0.6145230523563534 +B5_mofe_mlp,3407,0.3/modality_V,0.05703409792921816,728,0.6387362637362637,0.6076841472669062,0.6331720948219299,0.8356800930832423,0.610946837414197 +B5_mofe_mlp,3407,0.3/modality_TA,0.10919187418833474,728,0.6208791208791209,0.587877532835486,0.6422289609909058,0.8495119937905492,0.5922293943405987 +B5_mofe_mlp,3407,0.3/modality_TV,0.11437644408481795,728,0.6332417582417582,0.589795993553472,0.6403915286064148,0.8451140292386135,0.5883170312494674 +B5_mofe_mlp,3407,0.3/modality_AV,0.11303721690563862,728,0.6277472527472527,0.5986501178019438,0.6302677392959595,0.8346439585295746,0.6170157327095299 +B5_mofe_mlp,3407,0.3/modality_TAV,0.16956369042443203,728,0.6208791208791209,0.5872355756816936,0.6344102025032043,0.8431307321499237,0.6069219241721779 +B5_mofe_mlp,3407,0.3/location_start_T,0.10048832030833439,728,0.6208791208791209,0.5691816344789541,0.6474371552467346,0.8457314924808049,0.5877471209842607 +B5_mofe_mlp,3407,0.3/location_middle_T,0.10048832030833439,728,0.6332417582417582,0.580026202948566,0.6398654580116272,0.8329578308637743,0.602320103267925 +B5_mofe_mlp,3407,0.3/location_end_T,0.10048832030833439,728,0.6346153846153846,0.5818332297643384,0.6438518762588501,0.8348256140910719,0.6006683747918653 +B5_mofe_mlp,3407,0.3/span_long_T,0.10048832030833439,728,0.6304945054945055,0.5789449562705777,0.638378918170929,0.839944168119987,0.5929227244179269 +B5_mofe_mlp,3407,0.3/span_multi_short_T,0.10048832030833439,728,0.635989010989011,0.5830090515689985,0.6403829455375671,0.8377337542318748,0.5959436129993941 +B5_mofe_mlp,3407,0.3/location_start_A,0.10087109490306286,728,0.6222527472527473,0.5942606106743894,0.6317183375358582,0.8378718803105245,0.6163487843098526 +B5_mofe_mlp,3407,0.3/location_middle_A,0.10087109490306286,728,0.6332417582417582,0.6041073785131045,0.6293778419494629,0.8342761706281807,0.6171306545936357 +B5_mofe_mlp,3407,0.3/location_end_A,0.10087109490306286,728,0.635989010989011,0.6045352623195686,0.6293238401412964,0.8329167198828686,0.6180389298373584 +B5_mofe_mlp,3407,0.3/span_long_A,0.10087109490306286,728,0.6346153846153846,0.6060356885394697,0.6299681067466736,0.8372156941677521,0.614922712649418 +B5_mofe_mlp,3407,0.3/span_multi_short_A,0.10087109490306286,728,0.625,0.5980538669345082,0.6309807896614075,0.8373939802903236,0.6152266066983048 +B5_mofe_mlp,3407,0.3/location_start_V,0.09701596125937247,728,0.646978021978022,0.6167710620102839,0.6295149922370911,0.8281255398154749,0.6150683890220531 +B5_mofe_mlp,3407,0.3/location_middle_V,0.09701596125937247,728,0.6291208791208791,0.5970350664860342,0.6279875040054321,0.8285469923327045,0.6150038375612108 +B5_mofe_mlp,3407,0.3/location_end_V,0.09701596125937247,728,0.6291208791208791,0.5970350664860342,0.628416121006012,0.8286181007474381,0.6152751252437939 +B5_mofe_mlp,3407,0.3/span_long_V,0.09701596125937247,728,0.6346153846153846,0.6033920140660455,0.6285055875778198,0.8288092755139251,0.6145346855118451 +B5_mofe_mlp,3407,0.3/span_multi_short_V,0.09701596125937247,728,0.6373626373626373,0.6050369975450578,0.6291371583938599,0.8286749255438085,0.6143714918917834 +B5_mofe_mlp,3407,0.3/synchrony_sync,0.17627119309968198,728,0.6236263736263736,0.592557354039046,0.6403738856315613,0.8458653177750385,0.6041630287343566 +B5_mofe_mlp,3407,0.3/synchrony_partial,0.11943882005826376,728,0.6373626373626373,0.5974215893808968,0.6407806873321533,0.8445428725721186,0.5908264739709724 +B5_mofe_mlp,3407,0.3/synchrony_async,0.16594815654028122,728,0.6387362637362637,0.6044534328871108,0.6384277939796448,0.8368071907382745,0.6014610241367565 +B5_mofe_mlp,2026,0.0/none,0.0,728,0.6318681318681318,0.596526187825028,0.6597292423248291,0.8489002293710785,0.6032696127085834 +B5_mofe_mlp,2026,0.1/single,0.03371342897433643,728,0.6332417582417582,0.5970636969424238,0.6549333333969116,0.84639998483126,0.6035253872373745 +B5_mofe_mlp,2026,0.1/sync,0.045156406674301666,728,0.6277472527472527,0.5913100989187946,0.6568704843521118,0.8449213938859407,0.603907181971587 +B5_mofe_mlp,2026,0.1/partial,0.040239075593981495,728,0.6401098901098901,0.6024246836534827,0.6604945659637451,0.8481462530229538,0.6010193432821422 +B5_mofe_mlp,2026,0.1/async,0.042147033495012594,728,0.6332417582417582,0.5970167867969366,0.6555314660072327,0.8447819862485282,0.6044813102528398 +B5_mofe_mlp,2026,0.3/single,0.10101503353205936,728,0.6112637362637363,0.5638798050124324,0.6562340259552002,0.8475582224333787,0.5979011819911703 +B5_mofe_mlp,2026,0.3/sync,0.15049590345982738,728,0.6373626373626373,0.5989757153812737,0.6513807773590088,0.8493299353562517,0.595392891922453 +B5_mofe_mlp,2026,0.3/partial,0.11254067775371336,728,0.6181318681318682,0.5709193418039654,0.6664141416549683,0.8523418464954076,0.5914931558714205 +B5_mofe_mlp,2026,0.3/async,0.1402331971734938,728,0.6291208791208791,0.5870988936388463,0.6590703129768372,0.8508121089050296,0.5900941328826028 +B5_mofe_mlp,2026,0.5/single,0.16749406682715026,728,0.6126373626373627,0.5657169221458677,0.667115330696106,0.8635330523409043,0.5791727571395311 +B5_mofe_mlp,2026,0.5/sync,0.2869043063014706,728,0.6291208791208791,0.5857551126223407,0.6644048094749451,0.8677120855890653,0.5713612415242764 +B5_mofe_mlp,2026,0.5/partial,0.19417960020144912,728,0.6071428571428571,0.5531815016401854,0.6734310984611511,0.8739539092129627,0.5636138803720933 +B5_mofe_mlp,2026,0.5/async,0.2674778555592227,728,0.6002747252747253,0.5462966876593826,0.6667307019233704,0.8750358982896606,0.5594822701935701 +B5_mofe_mlp,2026,0.7/single,0.233161167473855,728,0.6057692307692307,0.5505603733738087,0.6735425591468811,0.8787421518357899,0.5616953487570775 +B5_mofe_mlp,2026,0.7/sync,0.4688425032750801,728,0.6332417582417582,0.5849499933341841,0.6716032028198242,0.874264169285277,0.5570492921944442 +B5_mofe_mlp,2026,0.7/partial,0.3517369377138981,728,0.6043956043956044,0.5413011359929573,0.6734654903411865,0.8867611643860307,0.5492658387339109 +B5_mofe_mlp,2026,0.7/async,0.43688540417216304,728,0.5824175824175825,0.5164414243555493,0.6856517791748047,0.9007994081729855,0.5242932038005764 +B5_mofe_mlp,2026,0.3/modality_T,0.05864506749840218,728,0.6291208791208791,0.5851588544385288,0.6609874367713928,0.8549948646993513,0.586805018488965 +B5_mofe_mlp,2026,0.3/modality_A,0.05697909847839254,728,0.6263736263736264,0.5923935684031888,0.6586064100265503,0.8480335925101825,0.60334875155725 +B5_mofe_mlp,2026,0.3/modality_V,0.05703409792921816,728,0.6277472527472527,0.5893350885736607,0.665980339050293,0.8524798430579631,0.6000276703946874 +B5_mofe_mlp,2026,0.3/modality_TA,0.10919187418833474,728,0.6318681318681318,0.587396209053305,0.6626462936401367,0.8586239046697091,0.5775735874993875 +B5_mofe_mlp,2026,0.3/modality_TV,0.11437644408481795,728,0.6222527472527473,0.5738086078934214,0.6704530119895935,0.866302760610593,0.5774709040224134 +B5_mofe_mlp,2026,0.3/modality_AV,0.11303721690563862,728,0.625,0.5916277684292892,0.6536700129508972,0.8439307195897752,0.6074596191049758 +B5_mofe_mlp,2026,0.3/modality_TAV,0.16956369042443203,728,0.6208791208791209,0.5812583811618302,0.6594159007072449,0.849905694610642,0.5970091338142024 +B5_mofe_mlp,2026,0.3/location_start_T,0.10048832030833439,728,0.6153846153846154,0.5618821946453657,0.6588675379753113,0.8592022115222522,0.5808518864395622 +B5_mofe_mlp,2026,0.3/location_middle_T,0.10048832030833439,728,0.635989010989011,0.5824702411988757,0.6568567156791687,0.8484010017651692,0.5967801637287061 +B5_mofe_mlp,2026,0.3/location_end_T,0.10048832030833439,728,0.625,0.5698652866175331,0.6648644208908081,0.8535741441162473,0.5882863910037036 +B5_mofe_mlp,2026,0.3/span_long_T,0.10048832030833439,728,0.6236263736263736,0.562985436305976,0.6574857831001282,0.8555561177828485,0.5918958078441902 +B5_mofe_mlp,2026,0.3/span_multi_short_T,0.10048832030833439,728,0.6277472527472527,0.5726133028329055,0.6584722995758057,0.8555951308170282,0.5927028642691361 +B5_mofe_mlp,2026,0.3/location_start_A,0.10087109490306286,728,0.6318681318681318,0.5946901594150785,0.6520330905914307,0.8433031735725448,0.6073570509720814 +B5_mofe_mlp,2026,0.3/location_middle_A,0.10087109490306286,728,0.6318681318681318,0.5964427935908233,0.6560055017471313,0.8459685792205899,0.6055766417612172 +B5_mofe_mlp,2026,0.3/location_end_A,0.10087109490306286,728,0.6304945054945055,0.5954019090935031,0.6567164659500122,0.8454491143527618,0.606308967978668 +B5_mofe_mlp,2026,0.3/span_long_A,0.10087109490306286,728,0.6346153846153846,0.5984529560616517,0.6542528867721558,0.8456221050542772,0.6045499685036667 +B5_mofe_mlp,2026,0.3/span_multi_short_A,0.10087109490306286,728,0.6291208791208791,0.5940461219915539,0.652047872543335,0.8438097261553195,0.6066816421615474 +B5_mofe_mlp,2026,0.3/location_start_V,0.09701596125937247,728,0.625,0.5846195652173912,0.6589953899383545,0.8472817644100747,0.6052728090813515 +B5_mofe_mlp,2026,0.3/location_middle_V,0.09701596125937247,728,0.6332417582417582,0.5924845794271737,0.6621479392051697,0.8497565832531178,0.604477594980146 +B5_mofe_mlp,2026,0.3/location_end_V,0.09701596125937247,728,0.635989010989011,0.5966448674232786,0.6628890633583069,0.8499398476690037,0.6041605414076311 +B5_mofe_mlp,2026,0.3/span_long_V,0.09701596125937247,728,0.6318681318681318,0.5920418130355286,0.6612062454223633,0.8490313438115848,0.6043638943646987 +B5_mofe_mlp,2026,0.3/span_multi_short_V,0.09701596125937247,728,0.6304945054945055,0.5901013226297688,0.6599566340446472,0.8482730221309799,0.6053465874208716 +B5_mofe_mlp,2026,0.3/synchrony_sync,0.17627119309968198,728,0.6332417582417582,0.5960026820501673,0.661083996295929,0.85475512137696,0.5907962734135933 +B5_mofe_mlp,2026,0.3/synchrony_partial,0.11943882005826376,728,0.625,0.5813376007663962,0.6566504836082458,0.8541285149652549,0.5872303717344893 +B5_mofe_mlp,2026,0.3/synchrony_async,0.16594815654028122,728,0.6346153846153846,0.5921399047232352,0.6548778414726257,0.8516002961616485,0.5907794067608744 diff --git a/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/official_test_metrics_by_seed.csv b/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/official_test_metrics_by_seed.csv new file mode 100644 index 0000000..349f072 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/official_test_metrics_by_seed.csv @@ -0,0 +1,7 @@ +method,seed,n_test,accuracy,macro_f1,mae,rmse,pearson +B0_early_concat,42,727,0.6740027510316369,0.5899364073506269,0.6732717752456665,0.8909249051748226,0.650797420132884 +B0_early_concat,3407,727,0.6616231086657497,0.5828956102942167,0.6562737822532654,0.8698196330495139,0.6629223076526239 +B0_early_concat,2026,727,0.671251719394773,0.5855120860972955,0.6805484890937805,0.8901024339427269,0.6502357150725873 +B5_mofe_mlp,42,727,0.6588720770288858,0.5881517532839975,0.6833341121673584,0.8967927556093251,0.6416499042677164 +B5_mofe_mlp,3407,727,0.6629986244841816,0.5987075270336911,0.6780627965927124,0.8830488445601224,0.6511876181785311 +B5_mofe_mlp,2026,727,0.6740027510316369,0.6049338687002567,0.6956164836883545,0.9003428945074896,0.6396089629278374 diff --git a/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/official_test_paired_bootstrap.csv b/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/official_test_paired_bootstrap.csv new file mode 100644 index 0000000..2655dc1 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/official_test_paired_bootstrap.csv @@ -0,0 +1,6 @@ +comparison,metric,delta_mean_over_seeds,bootstrap_ci_2p5,bootstrap_ci_97p5,bootstrap_probability_delta_gt_0,replicates,resampling_unit,paired,seed +MoFE-7 + MLP Router minus EarlyConcat + BiGRU,accuracy,-0.003668042182485065,-0.017722402708141464,0.011162107799884273,0.295,1000,source video id,True,20260925 +MoFE-7 + MLP Router minus EarlyConcat + BiGRU,macro_f1,0.011149681758602092,-0.008115389318624997,0.03215508906599282,0.888,1000,source video id,True,20260925 +MoFE-7 + MLP Router minus EarlyConcat + BiGRU,mae,0.015639781951904297,0.0053533881902694345,0.027328948676586112,0.998,1000,source video id,True,20260925 +MoFE-7 + MLP Router minus EarlyConcat + BiGRU,rmse,0.009779174169957994,-0.001325107979073983,0.021876485489001028,0.954,1000,source video id,True,20260925 +MoFE-7 + MLP Router minus EarlyConcat + BiGRU,pearson,-0.010502985828003353,-0.0207993259194662,-0.0015367106204170408,0.015,1000,source video id,True,20260925 diff --git a/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/official_test_summary.csv b/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/official_test_summary.csv new file mode 100644 index 0000000..35ef139 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/official_test_summary.csv @@ -0,0 +1,11 @@ +method,metric,mean,sd_across_seeds +B0_early_concat,accuracy,0.6689591930307198,0.006500434148903153 +B0_early_concat,macro_f1,0.5861147012473796,0.0035588712482642632 +B0_early_concat,mae,0.6700313488642374,0.01245755274515561 +B0_early_concat,rmse,0.8836156573890211,0.011954782742238058 +B0_early_concat,pearson,0.6546518142860317,0.007167961602772788 +B5_mofe_mlp,accuracy,0.6652911508482348,0.007821514034494262 +B5_mofe_mlp,macro_f1,0.5972643830059817,0.00848362233805206 +B5_mofe_mlp,mae,0.6856711308161417,0.009007176293739278 +B5_mofe_mlp,rmse,0.8933948315589791,0.009134027413164894 +B5_mofe_mlp,pearson,0.6441488284580283,0.006180597134201404 diff --git a/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/run_manifest.json b/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/run_manifest.json new file mode 100644 index 0000000..e890cbc --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R02_math_protocol_evaluation/run_manifest.json @@ -0,0 +1,44 @@ +{ + "experiment": "Frozen EarlyConcat vs MoFE-7 evaluation under math/Q2 test protocol", + "created_unix": 1790283893.6609955, + "device": "cuda", + "cuda_device": "NVIDIA GeForce RTX 5070 Ti", + "feature_file": "/home/gloamxun/modeling_zhaocui/E\u9898\u6570\u636e/\u9644\u4ef62-\u6570\u636e\u96c6\u7279\u5f81\u6587\u4ef6/aligned_50.pkl", + "feature_sha256": "66e867aa74bc70a844e806e5571e371c9abb4a35f9e2887ce9b4d97ff2cb8fcd", + "representation": "official aligned_50 ordered positions; not physical-time bins", + "train_valid_test_counts": { + "train": 3395, + "valid": 728, + "test": 727 + }, + "source_video_groups": { + "train": 1528, + "valid": 239, + "test": 381 + }, + "official_group_splits_disjoint": true, + "test_evaluation": "test outputs preserved from the earlier single evaluation; no test prediction was rerun", + "test_prediction_performed_this_invocation": false, + "seeds": [ + 42, + 3407, + 2026 + ], + "checkpoint_source": "/home/gloamxun/modeling_zhaocui/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/models", + "train_only_scaler": "/home/gloamxun/modeling_zhaocui/deep_learning/Q2/outputs/followups/R01_selected_model_reevaluation/aligned_robust_stats.npz", + "scaler_max_abs_difference_from_train_refit": 0.0, + "test_labels_used_for_training_or_selection": false, + "controlled_missingness": { + "scenario_seed": 20261833, + "scenario_design": "math/Q2 42-scenario design regenerated on the Q2 models' BERT attention-mask base", + "scenarios": 42, + "AURC": "normalized trapezoidal area of MAE over realized equal-modality-weighted added missing rate, at 0/.1/.3/.5/.7 for single/sync/partial/async" + }, + "bootstrap": { + "replicates": 1000, + "test_seed": 20260925, + "aurc_seed": 20260926, + "unit": "source video id", + "paired": true + } +} \ No newline at end of file diff --git a/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/RESULTS.md b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/RESULTS.md new file mode 100644 index 0000000..088f117 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/RESULTS.md @@ -0,0 +1,73 @@ +# Q2 retraining under the math/Q2 V2 evaluation protocol + +## Protocol + +- Official Attachment 2 split: 3,395 train / 728 validation / 727 test samples; source-video groups are disjoint. +- Seed: `20260924`, matching the math/Q2 run. +- Both models were retrained from scratch with the same train-only median/MAD scaler, training masks, batch order, optimizer, and joint objective. +- Training masks use contiguous spans at rates 0%, 10%, 30%, 50%, and 70%, with `single`, `sync`, `partial`, or `async` patterns. At least 20% of originally observed positions are retained in each selected modality. +- AdamW: learning rate `3e-4`, weight decay `1e-3`, batch size 64, maximum 12 epochs, early-stopping patience 3. +- Checkpoint selection uses the mean validation `CE + 0.5 × SmoothL1` over `0.0/none`, `0.3/single`, `0.3/sync`, and `0.5/async`. Both models selected epoch 3. +- The official test set was evaluated once, after checkpoint selection, on its clean observation masks. No test label was used for fitting or selection. +- The 42 controlled missingness scenarios and AURC-MAE were evaluated on validation data. Confidence intervals use 1,000 paired bootstrap resamples of source-video groups. + +The existing deterministic model heads and joint CE + SmoothL1 objective were retained. The math/Q2 C5 probabilistic Beta-mixture loss cannot be applied to these two architectures without changing the models. Thus the split, missingness, checkpoint discipline, and test metrics follow the math/Q2 protocol, while the training objective is the existing Q2 objective. + +## Official test results + +| Model | Accuracy ↑ | Macro-F1 ↑ | MAE ↓ | RMSE ↓ | Pearson ↑ | +|---|---:|---:|---:|---:|---:| +| EarlyConcat + BiGRU | 0.6740 | **0.6157** | **0.6624** | **0.8897** | **0.6585** | +| MoFE-7 + MLP Router | **0.6795** | 0.6049 | 0.6761 | 0.8980 | 0.6427 | + +Paired bootstrap differences are `MoFE-7 − EarlyConcat`; all 95% intervals include zero. + +| Metric | Difference | 95% source-video bootstrap interval | +|---|---:|---:| +| Accuracy | +0.0055 | [-0.0259, +0.0339] | +| Macro-F1 | -0.0109 | [-0.0475, +0.0216] | +| MAE | +0.0137 | [-0.0065, +0.0353] | +| RMSE | +0.0083 | [-0.0150, +0.0327] | +| Pearson | -0.0159 | [-0.0323, +0.0013] | + +The point estimates favor EarlyConcat on Macro-F1 and regression metrics; MoFE-7 has slightly higher Accuracy. The group-bootstrap intervals do not establish a clear difference on this single-seed run. + +## Validation missingness results + +Unweighted averages over the 41 non-clean rows of the 42-scenario audit are supplementary summaries; they are not the AURC statistic. + +| Model | Clean Macro-F1 | Mean Macro-F1 over 41 masked scenarios | Worst masked scenario Macro-F1 | Mean MAE over 41 masked scenarios | +|---|---:|---:|---:|---:| +| EarlyConcat + BiGRU | **0.5746** | **0.5655** | **0.5289** (`0.3/location_start_T`) | **0.6371** | +| MoFE-7 + MLP Router | 0.5626 | 0.5513 | 0.5048 (`0.7/partial`) | 0.6453 | + +| 42-scenario AURC-MAE mode | EarlyConcat | MoFE-7 | MoFE-7 − EarlyConcat (95% group-bootstrap interval) | +|---|---:|---:|---:| +| Single | **0.6379** | 0.6494 | +0.0115 [-0.0082, +0.0302] | +| Sync | **0.6387** | 0.6461 | +0.0074 [-0.0120, +0.0250] | +| Partial | **0.6420** | 0.6527 | +0.0107 [-0.0087, +0.0285] | +| Async | **0.6409** | 0.6527 | +0.0119 [-0.0062, +0.0289] | + +Lower AURC-MAE is better. EarlyConcat has the lower point estimate in all four modes; each interval crosses zero. + +At 30% modality-specific missingness: + +| Scenario | EarlyConcat Macro-F1 / MAE | MoFE-7 Macro-F1 / MAE | +|---|---:|---:| +| Text only | 0.5462 / 0.6386 | 0.5455 / 0.6477 | +| Audio + Vision | 0.5817 / 0.6326 | 0.5620 / 0.6350 | +| All modalities | 0.5693 / 0.6352 | 0.5617 / 0.6421 | + +## Reading the result + +Under this retraining run, EarlyConcat is the stronger overall choice by Macro-F1, MAE, Pearson, mean masked-scenario metrics, and all four AURC-MAE point estimates. MoFE-7's small Accuracy advantage is uncertain. Because this is one seed, the results should be read as a protocol-matched run rather than a multi-seed stability estimate. + +The math/Q2 report's C5 test result (Accuracy 0.6740, Macro-F1 0.5861, MAE 0.6980, RMSE 0.9674, Pearson 0.6300) is included only as context: C5 has a different probabilistic output head and loss, so it is not a like-for-like architecture comparison. + +## Artifacts + +- `run_manifest.json`: data, split, training, validation, test, and bootstrap protocol. +- `official_test_metrics_by_seed.csv`, `official_test_paired_bootstrap.csv`: final test metrics and paired group intervals. +- `controlled_metrics_by_scenario.csv`, `aurc_mae_by_mode_seed.csv`, `aurc_mae_paired_bootstrap.csv`: controlled validation evaluation. +- `training_history.csv`, `training_mask_distribution.csv`, `parameter_count.csv`: training diagnostics. +- `models/`: selected checkpoints and per-model histories. diff --git a/deep_learning/Q2/outputs/algorithm_selection/fixed_window_robust_stats.npz b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/aligned_robust_stats.npz similarity index 76% rename from deep_learning/Q2/outputs/algorithm_selection/fixed_window_robust_stats.npz rename to deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/aligned_robust_stats.npz index 5515e58..3d7956d 100644 Binary files a/deep_learning/Q2/outputs/algorithm_selection/fixed_window_robust_stats.npz and b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/aligned_robust_stats.npz differ diff --git a/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/aurc_mae_by_mode_seed.csv b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/aurc_mae_by_mode_seed.csv new file mode 100644 index 0000000..bd22ed5 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/aurc_mae_by_mode_seed.csv @@ -0,0 +1,9 @@ +method,seed,mask_mode,aurc_mae,rates_realized +B0_early_concat,20260924,single,0.6378559862834174,"[0.0, 0.03371342897433643, 0.10101503353205936, 0.16749406682715026, 0.233161167473855]" +B0_early_concat,20260924,sync,0.6387340663515244,"[0.0, 0.045156406674301666, 0.15049590345982738, 0.2869043063014706, 0.4688425032750801]" +B0_early_concat,20260924,partial,0.6419565330457959,"[0.0, 0.040239075593981495, 0.11254067775371336, 0.19417960020144912, 0.3517369377138981]" +B0_early_concat,20260924,async,0.6408780951419853,"[0.0, 0.042147033495012594, 0.1402331971734938, 0.2674778555592227, 0.43688540417216304]" +B5_mofe_mlp,20260924,single,0.6493678370419695,"[0.0, 0.03371342897433643, 0.10101503353205936, 0.16749406682715026, 0.233161167473855]" +B5_mofe_mlp,20260924,sync,0.6461235184522032,"[0.0, 0.045156406674301666, 0.15049590345982738, 0.2869043063014706, 0.4688425032750801]" +B5_mofe_mlp,20260924,partial,0.6526607283449438,"[0.0, 0.040239075593981495, 0.11254067775371336, 0.19417960020144912, 0.3517369377138981]" +B5_mofe_mlp,20260924,async,0.6527283697618514,"[0.0, 0.042147033495012594, 0.1402331971734938, 0.2674778555592227, 0.43688540417216304]" diff --git a/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/aurc_mae_paired_bootstrap.csv b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/aurc_mae_paired_bootstrap.csv new file mode 100644 index 0000000..170e951 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/aurc_mae_paired_bootstrap.csv @@ -0,0 +1,5 @@ +mask_mode,delta_aurc_mae_mofe_minus_earlyconcat,bootstrap_ci_2p5,bootstrap_ci_97p5,bootstrap_probability_delta_lt_0,replicates,resampling_unit,paired,seed +single,0.0115118507585521,-0.008162294605830693,0.0301810704036703,0.147,1000,source video id,True,20260926 +sync,0.007389452100678762,-0.011956058947051679,0.024951573659996493,0.259,1000,source video id,True,20260926 +partial,0.0107041952991479,-0.008660331894343192,0.028507998549019406,0.161,1000,source video id,True,20260926 +async,0.011850274619866097,-0.006150395373279235,0.028900548840914045,0.115,1000,source video id,True,20260926 diff --git a/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/controlled_metrics_by_scenario.csv b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/controlled_metrics_by_scenario.csv new file mode 100644 index 0000000..3def47f --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/controlled_metrics_by_scenario.csv @@ -0,0 +1,85 @@ +method,seed,scenario,realized_additional_global_rate,n_valid,accuracy,macro_f1,mae,rmse,pearson +B0_early_concat,20260924,0.0/none,0.0,728,0.6153846153846154,0.5746473167631652,0.6343159675598145,0.8547929855610467,0.6094602929761117 +B0_early_concat,20260924,0.1/single,0.03371342897433643,728,0.6085164835164835,0.5664719454034405,0.6328904032707214,0.85180821504718,0.6094495047174566 +B0_early_concat,20260924,0.1/sync,0.045156406674301666,728,0.6098901098901099,0.572216012695813,0.6333207488059998,0.8524984763011572,0.6076810890989575 +B0_early_concat,20260924,0.1/partial,0.040239075593981495,728,0.6112637362637363,0.5706507860891473,0.6365461945533752,0.8541368192404588,0.6058859344866191 +B0_early_concat,20260924,0.1/async,0.042147033495012594,728,0.6181318681318682,0.57972935597245,0.6354024410247803,0.852031893370522,0.6096049530317527 +B0_early_concat,20260924,0.3/single,0.10101503353205936,728,0.614010989010989,0.574220027013609,0.6334993839263916,0.854360969277559,0.6003605729983688 +B0_early_concat,20260924,0.3/sync,0.15049590345982738,728,0.6167582417582418,0.5730892523600383,0.630831241607666,0.8516494286640907,0.6037950917057342 +B0_early_concat,20260924,0.3/partial,0.11254067775371336,728,0.5961538461538461,0.5524892450018223,0.6370627880096436,0.8509616306790629,0.5979238186423941 +B0_early_concat,20260924,0.3/async,0.1402331971734938,728,0.6112637362637363,0.5642957517004324,0.6346827745437622,0.8484940848967643,0.6016212338363784 +B0_early_concat,20260924,0.5/single,0.16749406682715026,728,0.6085164835164835,0.5670184055357302,0.641832172870636,0.8632332628816644,0.584865558206941 +B0_early_concat,20260924,0.5/sync,0.2869043063014706,728,0.6167582417582418,0.5778898483774816,0.6371996998786926,0.8516882007461904,0.5924273843364904 +B0_early_concat,20260924,0.5/partial,0.19417960020144912,728,0.6043956043956044,0.5555283579204368,0.6441753506660461,0.8556262354706008,0.58203768440478 +B0_early_concat,20260924,0.5/async,0.2674778555592227,728,0.6016483516483516,0.5562522166489335,0.6466042995452881,0.8691973721270065,0.5700682850933873 +B0_early_concat,20260924,0.7/single,0.233161167473855,728,0.5947802197802198,0.5485981680163076,0.6481859087944031,0.8809218965997698,0.5635469635745481 +B0_early_concat,20260924,0.7/sync,0.4688425032750801,728,0.5989010989010989,0.5626788755858914,0.6574939489364624,0.8839283533082283,0.5530534634060382 +B0_early_concat,20260924,0.7/partial,0.3517369377138981,728,0.5865384615384616,0.5362416499854459,0.649185299873352,0.8699182696154274,0.5583734809756835 +B0_early_concat,20260924,0.7/async,0.43688540417216304,728,0.5906593406593407,0.5452478413289441,0.6452565789222717,0.8735732640099089,0.5557409742879581 +B0_early_concat,20260924,0.3/modality_T,0.05864506749840218,728,0.5934065934065934,0.5462207369967149,0.6385592818260193,0.8588612841667281,0.5881575690894479 +B0_early_concat,20260924,0.3/modality_A,0.05697909847839254,728,0.6195054945054945,0.5811293545443177,0.6334327459335327,0.8554246446919079,0.6094268235049153 +B0_early_concat,20260924,0.3/modality_V,0.05703409792921816,728,0.6085164835164835,0.5686488097327915,0.6339969038963318,0.8540853174750868,0.6090671394991262 +B0_early_concat,20260924,0.3/modality_TA,0.10919187418833474,728,0.6071428571428571,0.5602184289914843,0.6519798636436462,0.8699653056231473,0.5768200282380198 +B0_early_concat,20260924,0.3/modality_TV,0.11437644408481795,728,0.6016483516483516,0.5551175139246832,0.6393751502037048,0.8553959366977271,0.586114371400815 +B0_early_concat,20260924,0.3/modality_AV,0.11303721690563862,728,0.6181318681318682,0.5816591704230339,0.6326149702072144,0.849950156433407,0.6139148157800808 +B0_early_concat,20260924,0.3/modality_TAV,0.16956369042443203,728,0.6098901098901099,0.5693316484741863,0.635162353515625,0.8552724883114987,0.603834533975782 +B0_early_concat,20260924,0.3/location_start_T,0.10048832030833439,728,0.592032967032967,0.5289423490769057,0.6485703587532043,0.8620428808968518,0.58162843015804 +B0_early_concat,20260924,0.3/location_middle_T,0.10048832030833439,728,0.603021978021978,0.5536695658136993,0.6376606822013855,0.8514152889573482,0.5897726128244986 +B0_early_concat,20260924,0.3/location_end_T,0.10048832030833439,728,0.6043956043956044,0.5556056212627483,0.6402567625045776,0.8542722582486372,0.5891006734385348 +B0_early_concat,20260924,0.3/span_long_T,0.10048832030833439,728,0.5989010989010989,0.5420906602882528,0.6381194591522217,0.8522570166440261,0.5853301068267182 +B0_early_concat,20260924,0.3/span_multi_short_T,0.10048832030833439,728,0.5975274725274725,0.5440495562446782,0.6356171369552612,0.8507890951392163,0.5876002847201148 +B0_early_concat,20260924,0.3/location_start_A,0.10087109490306286,728,0.6167582417582418,0.5812912542814239,0.6293162107467651,0.8535272872442308,0.6114908900853474 +B0_early_concat,20260924,0.3/location_middle_A,0.10087109490306286,728,0.625,0.5898724710749466,0.6324224472045898,0.8539363426497163,0.6110184042952445 +B0_early_concat,20260924,0.3/location_end_A,0.10087109490306286,728,0.6263736263736264,0.5905371029206817,0.6318912506103516,0.8519847767795174,0.6123387287503104 +B0_early_concat,20260924,0.3/span_long_A,0.10087109490306286,728,0.6181318681318682,0.583726857007959,0.6294587254524231,0.8517886220306001,0.6128556406770401 +B0_early_concat,20260924,0.3/span_multi_short_A,0.10087109490306286,728,0.6126373626373627,0.5775040835891835,0.6295161247253418,0.8519002262223094,0.6129813734377794 +B0_early_concat,20260924,0.3/location_start_V,0.09701596125937247,728,0.6167582417582418,0.5784311415271869,0.6317656636238098,0.8524039774314564,0.6094092868851544 +B0_early_concat,20260924,0.3/location_middle_V,0.09701596125937247,728,0.6126373626373627,0.5758902475296572,0.6318002343177795,0.8514377607625061,0.6102280937754011 +B0_early_concat,20260924,0.3/location_end_V,0.09701596125937247,728,0.6085164835164835,0.5710858841987924,0.6308775544166565,0.8521441651682452,0.6096283880242306 +B0_early_concat,20260924,0.3/span_long_V,0.09701596125937247,728,0.6098901098901099,0.5712207484964352,0.6293761730194092,0.8494720698811937,0.6119813338293026 +B0_early_concat,20260924,0.3/span_multi_short_V,0.09701596125937247,728,0.6112637362637363,0.5724471470092743,0.6319549083709717,0.8523396436835378,0.609626822827826 +B0_early_concat,20260924,0.3/synchrony_sync,0.17627119309968198,728,0.6071428571428571,0.5695955472926512,0.6377636194229126,0.8563529196373093,0.6014507911006194 +B0_early_concat,20260924,0.3/synchrony_partial,0.11943882005826376,728,0.614010989010989,0.5684426287766249,0.637352466583252,0.8497644743263203,0.5966725971941069 +B0_early_concat,20260924,0.3/synchrony_async,0.16594815654028122,728,0.6126373626373627,0.5674224398503759,0.6281272768974304,0.8457619380102747,0.6032017513238488 +B5_mofe_mlp,20260924,0.0/none,0.0,728,0.6057692307692307,0.5626447626447627,0.6368198394775391,0.8559372894743668,0.6043374371906683 +B5_mofe_mlp,20260924,0.1/single,0.03371342897433643,728,0.603021978021978,0.5577024380655372,0.6354461312294006,0.8543030622177138,0.6043309492398073 +B5_mofe_mlp,20260924,0.1/sync,0.045156406674301666,728,0.6016483516483516,0.5578604237235317,0.6360094547271729,0.8566200486967864,0.6013163340141866 +B5_mofe_mlp,20260924,0.1/partial,0.040239075593981495,728,0.6043956043956044,0.5556321720066021,0.6396713852882385,0.8584938039739364,0.6010106097514567 +B5_mofe_mlp,20260924,0.1/async,0.042147033495012594,728,0.6016483516483516,0.5565284878755451,0.6399586200714111,0.8586023845707027,0.6003751700281593 +B5_mofe_mlp,20260924,0.3/single,0.10101503353205936,728,0.6016483516483516,0.541701381711408,0.6420438289642334,0.8560677438828654,0.6001244469401612 +B5_mofe_mlp,20260924,0.3/sync,0.15049590345982738,728,0.614010989010989,0.5693500431359637,0.6388541460037231,0.855140553404774,0.6004973824912962 +B5_mofe_mlp,20260924,0.3/partial,0.11254067775371336,728,0.6002747252747253,0.539363851343671,0.6399300694465637,0.852451385499069,0.600576671641639 +B5_mofe_mlp,20260924,0.3/async,0.1402331971734938,728,0.6112637362637363,0.5574919295976676,0.6433636546134949,0.8590869073880102,0.5949047660811583 +B5_mofe_mlp,20260924,0.5/single,0.16749406682715026,728,0.6153846153846154,0.5586891334012748,0.6575657725334167,0.8704040602456963,0.5845020723393936 +B5_mofe_mlp,20260924,0.5/sync,0.2869043063014706,728,0.6236263736263736,0.5743149665205417,0.6494404673576355,0.861759830163145,0.5863964302751469 +B5_mofe_mlp,20260924,0.5/partial,0.19417960020144912,728,0.6071428571428571,0.5355878409861167,0.6588982939720154,0.8727261767123125,0.5771476109647259 +B5_mofe_mlp,20260924,0.5/async,0.2674778555592227,728,0.6071428571428571,0.5480764166366612,0.6661632657051086,0.8843453864002215,0.5607269390754831 +B5_mofe_mlp,20260924,0.7/single,0.233161167473855,728,0.5975274725274725,0.5338090526934354,0.6756492257118225,0.8963052076097996,0.5586007231842708 +B5_mofe_mlp,20260924,0.7/sync,0.4688425032750801,728,0.6085164835164835,0.565745648642757,0.6606540679931641,0.8915316333902972,0.5541614851997242 +B5_mofe_mlp,20260924,0.7/partial,0.3517369377138981,728,0.5879120879120879,0.5047590693139797,0.6689532995223999,0.8808325448358977,0.5652797535160367 +B5_mofe_mlp,20260924,0.7/async,0.43688540417216304,728,0.592032967032967,0.5358739987222959,0.6561869382858276,0.8794836563621672,0.5588249328667847 +B5_mofe_mlp,20260924,0.3/modality_T,0.05864506749840218,728,0.6057692307692307,0.5454628701133298,0.6476889252662659,0.8713019083833805,0.5849661557179732 +B5_mofe_mlp,20260924,0.3/modality_A,0.05697909847839254,728,0.6098901098901099,0.5668229481750353,0.6372511386871338,0.8576914509064657,0.6038199443939388 +B5_mofe_mlp,20260924,0.3/modality_V,0.05703409792921816,728,0.6085164835164835,0.5658571865320935,0.635873556137085,0.8509052084722697,0.6033888741448421 +B5_mofe_mlp,20260924,0.3/modality_TA,0.10919187418833474,728,0.6085164835164835,0.551328601397119,0.6482953429222107,0.8684097474486374,0.5835353299125547 +B5_mofe_mlp,20260924,0.3/modality_TV,0.11437644408481795,728,0.6098901098901099,0.5553867994131184,0.6396345496177673,0.8586720452609139,0.587121236120106 +B5_mofe_mlp,20260924,0.3/modality_AV,0.11303721690563862,728,0.603021978021978,0.5619825935453588,0.6350018382072449,0.8494499320015197,0.60596049079492 +B5_mofe_mlp,20260924,0.3/modality_TAV,0.16956369042443203,728,0.6043956043956044,0.5616850220473674,0.6421029567718506,0.8664177923921365,0.5936093549207478 +B5_mofe_mlp,20260924,0.3/location_start_T,0.10048832030833439,728,0.5824175824175825,0.5074646922437304,0.6651872396469116,0.8883185050888599,0.5708209590446683 +B5_mofe_mlp,20260924,0.3/location_middle_T,0.10048832030833439,728,0.6071428571428571,0.5255359991732577,0.6560966968536377,0.8699031271111219,0.5868176227041925 +B5_mofe_mlp,20260924,0.3/location_end_T,0.10048832030833439,728,0.6071428571428571,0.5241847210214233,0.6566184759140015,0.8703728673739797,0.5880970537326943 +B5_mofe_mlp,20260924,0.3/span_long_T,0.10048832030833439,728,0.592032967032967,0.5179866041214112,0.6569079756736755,0.8722153699980981,0.5832300823948291 +B5_mofe_mlp,20260924,0.3/span_multi_short_T,0.10048832030833439,728,0.5975274725274725,0.5217755256072807,0.6542142033576965,0.8695648872333397,0.5861358594281869 +B5_mofe_mlp,20260924,0.3/location_start_A,0.10087109490306286,728,0.6153846153846154,0.5693105809612327,0.6393567323684692,0.8605473930921826,0.6030681972111498 +B5_mofe_mlp,20260924,0.3/location_middle_A,0.10087109490306286,728,0.6002747252747253,0.551121695858538,0.6374183297157288,0.8570439020438321,0.6040473734185878 +B5_mofe_mlp,20260924,0.3/location_end_A,0.10087109490306286,728,0.6043956043956044,0.5553975345642012,0.6361827850341797,0.8545916515771931,0.6057378355308779 +B5_mofe_mlp,20260924,0.3/span_long_A,0.10087109490306286,728,0.6071428571428571,0.5568552904967875,0.6383994817733765,0.8585218529627641,0.603334225974666 +B5_mofe_mlp,20260924,0.3/span_multi_short_A,0.10087109490306286,728,0.6085164835164835,0.5639201931530282,0.6405740976333618,0.8599686392899067,0.6018797003793258 +B5_mofe_mlp,20260924,0.3/location_start_V,0.09701596125937247,728,0.614010989010989,0.5706553787814322,0.6286044120788574,0.843568535117206,0.607697827670399 +B5_mofe_mlp,20260924,0.3/location_middle_V,0.09701596125937247,728,0.6057692307692307,0.5625629140502925,0.631475567817688,0.8447082164734656,0.6066031598751728 +B5_mofe_mlp,20260924,0.3/location_end_V,0.09701596125937247,728,0.6016483516483516,0.5591251566861323,0.6326090097427368,0.8460766185600626,0.6050984837070122 +B5_mofe_mlp,20260924,0.3/span_long_V,0.09701596125937247,728,0.6153846153846154,0.5732705025992542,0.629095733165741,0.8440032861383474,0.6077357791248426 +B5_mofe_mlp,20260924,0.3/span_multi_short_V,0.09701596125937247,728,0.6098901098901099,0.5658045462820314,0.6311919093132019,0.8445015138936631,0.6068331137272684 +B5_mofe_mlp,20260924,0.3/synchrony_sync,0.17627119309968198,728,0.6085164835164835,0.5675933141063153,0.6438319683074951,0.8657874068623187,0.5957094024638004 +B5_mofe_mlp,20260924,0.3/synchrony_partial,0.11943882005826376,728,0.6057692307692307,0.542036764873725,0.6417625546455383,0.8593849528343224,0.5939879772923478 +B5_mofe_mlp,20260924,0.3/synchrony_async,0.16594815654028122,728,0.6153846153846154,0.5670250898329069,0.6416157484054565,0.859539120348063,0.5924952604080301 diff --git a/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/hypothesis.md b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/hypothesis.md new file mode 100644 index 0000000..857d823 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/hypothesis.md @@ -0,0 +1,13 @@ +# R03: 按 math/Q2 V2 口径重训两种保留模型 + +## 假设 + +在保持 EarlyConcat + BiGRU 与 MoFE-7 + MLP Router 结构及共同监督目标不变的情况下,使用数学方案中的官方划分、连续块缺失训练和 42 个固定验证情景,可以公平比较两种模型的干净测试表现与缺失鲁棒性。 + +## 唯一实验改动 + +相对现有检查点,本轮重新训练时将缺失训练改为 0/10/30/50/70% 与 single/sync/partial/async,每个被选模态至少保留 20% 观测;训练和批次顺序在两个模型间配对。数学方案中的 C5 概率损失不适用于现有确定性分类/回归头,因此保留项目既有的 CE + 0.5 SmoothL1 联合目标。 + +## 数据使用 + +标准化器只在官方训练集观测行上拟合;官方验证集只用于早停与缺失评估;官方测试集在全部检查点确定后做一次干净评估。 diff --git a/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/models/B0_early_concat/seed_20260924/model_best.pt b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/models/B0_early_concat/seed_20260924/model_best.pt new file mode 100644 index 0000000..4eb048b Binary files /dev/null and b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/models/B0_early_concat/seed_20260924/model_best.pt differ diff --git a/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/models/B0_early_concat/seed_20260924/training_history.csv b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/models/B0_early_concat/seed_20260924/training_history.csv new file mode 100644 index 0000000..33e158f --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/models/B0_early_concat/seed_20260924/training_history.csv @@ -0,0 +1,7 @@ +method,seed,epoch,train_loss,valid_selection_loss,valid_clean_loss +B0_early_concat,20260924,1,1.0061404870616064,0.9110350304252499,0.9014195590228825 +B0_early_concat,20260924,2,0.8046501874923706,0.889426393004564,0.8862256545286912 +B0_early_concat,20260924,3,0.7282454095504902,0.8806566258708199,0.8769266769126222 +B0_early_concat,20260924,4,0.6743332914732121,0.9174083739846618,0.9197112072955121 +B0_early_concat,20260924,5,0.6134337760784008,0.9341701424711354,0.9390361531750187 +B0_early_concat,20260924,6,0.558312753284419,1.000857290330824,1.0128365755081177 diff --git a/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/models/B5_mofe_mlp/seed_20260924/model_best.pt b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/models/B5_mofe_mlp/seed_20260924/model_best.pt new file mode 100644 index 0000000..f3ec56b Binary files /dev/null and b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/models/B5_mofe_mlp/seed_20260924/model_best.pt differ diff --git a/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/models/B5_mofe_mlp/seed_20260924/training_history.csv b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/models/B5_mofe_mlp/seed_20260924/training_history.csv new file mode 100644 index 0000000..3d6dfed --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/models/B5_mofe_mlp/seed_20260924/training_history.csv @@ -0,0 +1,7 @@ +method,seed,epoch,train_loss,valid_selection_loss,valid_clean_loss +B5_mofe_mlp,20260924,1,1.0217058570296675,0.9436172379569692,0.9375626621665535 +B5_mofe_mlp,20260924,2,0.8520832823382484,0.9506990911213906,0.952043721964071 +B5_mofe_mlp,20260924,3,0.8015031770423606,0.8901680516345161,0.887631527015141 +B5_mofe_mlp,20260924,4,0.7390829468214953,0.9029482278850052,0.9031652598590642 +B5_mofe_mlp,20260924,5,0.6777735639501501,0.9294115293484467,0.9311579852313786 +B5_mofe_mlp,20260924,6,0.6290948126051161,0.9776710203060737,0.9763853563057198 diff --git a/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/official_test_metrics_by_seed.csv b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/official_test_metrics_by_seed.csv new file mode 100644 index 0000000..9c03040 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/official_test_metrics_by_seed.csv @@ -0,0 +1,3 @@ +method,seed,best_epoch,n_test,accuracy,macro_f1,mae,rmse,pearson +B0_early_concat,20260924,3,727,0.6740027510316369,0.6157432417472255,0.6623634099960327,0.8897308076984263,0.6585211684420424 +B5_mofe_mlp,20260924,3,727,0.6795048143053645,0.6048829559870884,0.6760958433151245,0.8979946869648381,0.6426699367924718 diff --git a/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/official_test_paired_bootstrap.csv b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/official_test_paired_bootstrap.csv new file mode 100644 index 0000000..e8a5737 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/official_test_paired_bootstrap.csv @@ -0,0 +1,6 @@ +comparison,metric,delta,bootstrap_ci_2p5,bootstrap_ci_97p5,bootstrap_probability_delta_gt_0,replicates,resampling_unit,paired,seed +B5_mofe_mlp minus B0_early_concat,accuracy,0.005502063273727598,-0.02588999712195796,0.03394678025644185,0.617,1000,source video id,True,20260925 +B5_mofe_mlp minus B0_early_concat,macro_f1,-0.01086028576013709,-0.04748880693804554,0.021613230578814074,0.251,1000,source video id,True,20260925 +B5_mofe_mlp minus B0_early_concat,mae,0.013732433319091797,-0.006543658673763275,0.0352545291185379,0.916,1000,source video id,True,20260925 +B5_mofe_mlp minus B0_early_concat,rmse,0.008263879266411811,-0.01499332646297088,0.03268860651949909,0.762,1000,source video id,True,20260925 +B5_mofe_mlp minus B0_early_concat,pearson,-0.015851231649570696,-0.032304558903424575,0.0012594110305533422,0.036,1000,source video id,True,20260925 diff --git a/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/official_test_summary.csv b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/official_test_summary.csv new file mode 100644 index 0000000..2b22dba --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/official_test_summary.csv @@ -0,0 +1,11 @@ +method,metric,mean,sd_across_seeds,n_seeds +B0_early_concat,accuracy,0.6740027510316369,0.0,1 +B0_early_concat,macro_f1,0.6157432417472255,0.0,1 +B0_early_concat,mae,0.6623634099960327,0.0,1 +B0_early_concat,rmse,0.8897308076984263,0.0,1 +B0_early_concat,pearson,0.6585211684420424,0.0,1 +B5_mofe_mlp,accuracy,0.6795048143053645,0.0,1 +B5_mofe_mlp,macro_f1,0.6048829559870884,0.0,1 +B5_mofe_mlp,mae,0.6760958433151245,0.0,1 +B5_mofe_mlp,rmse,0.8979946869648381,0.0,1 +B5_mofe_mlp,pearson,0.6426699367924718,0.0,1 diff --git a/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/parameter_count.csv b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/parameter_count.csv new file mode 100644 index 0000000..268dd3a --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/parameter_count.csv @@ -0,0 +1,3 @@ +method,parameters_total,parameters_trainable,best_epoch +B0_early_concat,253124,253124,3 +B5_mofe_mlp,306523,306523,3 diff --git a/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/run_manifest.json b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/run_manifest.json new file mode 100644 index 0000000..71888d0 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/run_manifest.json @@ -0,0 +1,87 @@ +{ + "experiment": "Retrained EarlyConcat and MoFE-7 + MLP Router using math/Q2 V2-compatible protocol", + "created_unix": 1790293163.7924957, + "device": "cuda", + "cuda_device": "NVIDIA GeForce RTX 5070 Ti", + "feature_file": "/home/gloamxun/modeling_zhaocui/E\u9898\u6570\u636e/\u9644\u4ef62-\u6570\u636e\u96c6\u7279\u5f81\u6587\u4ef6/aligned_50.pkl", + "feature_sha256": "66e867aa74bc70a844e806e5571e371c9abb4a35f9e2887ce9b4d97ff2cb8fcd", + "representation": "official aligned_50 ordered positions; not Q1 physical-time bins", + "train_valid_test_counts": { + "train": 3395, + "valid": 728, + "test": 727 + }, + "source_video_groups": { + "train": 1528, + "valid": 239, + "test": 381 + }, + "official_group_splits_disjoint": true, + "train_only_scaler": "/home/gloamxun/modeling_zhaocui/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/aligned_robust_stats.npz", + "scaler_fit": "median and 1.4826*MAD on observed training rows only; zero-MAD fallback to std then 1", + "seed": 20260924, + "model_seeds": [ + 20260924 + ], + "training_configuration": { + "epoch_limit": 12, + "early_stopping_patience": 3, + "batch_size": 64, + "optimizer": "AdamW", + "learning_rate": 0.0003, + "weight_decay": 0.001, + "gradient_clip_norm": 1.0, + "early_stopping_metric": "mean validation joint CE + 0.5*SmoothL1 over 0.0/none, 0.3/single, 0.3/sync, 0.5/async", + "architecture_preserved": { + "B0_early_concat": "EarlyConcat + BiGRU", + "B5_mofe_mlp": "MoFE-7 + MLP Router" + }, + "objective": "cross entropy + 0.5 * SmoothL1(intensity/3, label/3); same objective for both methods", + "training_corruption": { + "rates": [ + 0.0, + 0.1, + 0.3, + 0.5, + 0.7 + ], + "patterns": [ + "single", + "sync", + "partial", + "async" + ], + "preserve_at_least_fraction_per_selected_modality": 0.2, + "generator_seed": 20261227, + "same_sample_masks_and_batch_orders_across_models": true + } + }, + "validation_protocol": { + "scenario_seed": 20261833, + "scenario_count": 42, + "same_fixed_masks_for_both_models": true, + "scenario_design": "math/Q2 42 controlled continuous-mask scenarios regenerated on each sample's original observation mask", + "selection_scenarios": [ + "0.0/none", + "0.3/single", + "0.3/sync", + "0.5/async" + ], + "selection_note": "Deterministic-model adaptation; uses joint supervised loss instead of C5's probabilistic selection NLL.", + "aurc": "normalized trapezoidal MAE area over realized equal-modality-weighted additional missing rate for single/sync/partial/async at 0/.1/.3/.5/.7" + }, + "test_protocol": { + "official_test_final_clean_passes": 1, + "test_used_for_training_or_checkpoint_selection": false, + "metrics": [ + "accuracy", + "macro_f1", + "mae", + "rmse", + "pearson" + ], + "paired_group_bootstrap_replicates": 1000, + "bootstrap_unit": "source video id", + "bootstrap_seed": 20260925 + } +} \ No newline at end of file diff --git a/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/training_history.csv b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/training_history.csv new file mode 100644 index 0000000..0ea7d20 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/training_history.csv @@ -0,0 +1,13 @@ +method,seed,epoch,train_loss,valid_selection_loss,valid_clean_loss +B0_early_concat,20260924,1,1.0061404870616064,0.9110350304252499,0.9014195590228825 +B0_early_concat,20260924,2,0.8046501874923706,0.889426393004564,0.8862256545286912 +B0_early_concat,20260924,3,0.7282454095504902,0.8806566258708199,0.8769266769126222 +B0_early_concat,20260924,4,0.6743332914732121,0.9174083739846618,0.9197112072955121 +B0_early_concat,20260924,5,0.6134337760784008,0.9341701424711354,0.9390361531750187 +B0_early_concat,20260924,6,0.558312753284419,1.000857290330824,1.0128365755081177 +B5_mofe_mlp,20260924,1,1.0217058570296675,0.9436172379569692,0.9375626621665535 +B5_mofe_mlp,20260924,2,0.8520832823382484,0.9506990911213906,0.952043721964071 +B5_mofe_mlp,20260924,3,0.8015031770423606,0.8901680516345161,0.887631527015141 +B5_mofe_mlp,20260924,4,0.7390829468214953,0.9029482278850052,0.9031652598590642 +B5_mofe_mlp,20260924,5,0.6777735639501501,0.9294115293484467,0.9311579852313786 +B5_mofe_mlp,20260924,6,0.6290948126051161,0.9776710203060737,0.9763853563057198 diff --git a/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/training_mask_distribution.csv b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/training_mask_distribution.csv new file mode 100644 index 0000000..1a7a9e9 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/R03_math_protocol_retraining/training_mask_distribution.csv @@ -0,0 +1,41 @@ +method,seed,rate_mode,sample_epoch_assignments +B0_early_concat,20260924,0.0/async,1034 +B0_early_concat,20260924,0.0/partial,1074 +B0_early_concat,20260924,0.0/single,1084 +B0_early_concat,20260924,0.0/sync,1065 +B0_early_concat,20260924,0.1/async,1059 +B0_early_concat,20260924,0.1/partial,1029 +B0_early_concat,20260924,0.1/single,1034 +B0_early_concat,20260924,0.1/sync,969 +B0_early_concat,20260924,0.3/async,1047 +B0_early_concat,20260924,0.3/partial,989 +B0_early_concat,20260924,0.3/single,978 +B0_early_concat,20260924,0.3/sync,1014 +B0_early_concat,20260924,0.5/async,1004 +B0_early_concat,20260924,0.5/partial,1017 +B0_early_concat,20260924,0.5/single,1013 +B0_early_concat,20260924,0.5/sync,986 +B0_early_concat,20260924,0.7/async,975 +B0_early_concat,20260924,0.7/partial,1006 +B0_early_concat,20260924,0.7/single,982 +B0_early_concat,20260924,0.7/sync,1011 +B5_mofe_mlp,20260924,0.0/async,1034 +B5_mofe_mlp,20260924,0.0/partial,1074 +B5_mofe_mlp,20260924,0.0/single,1084 +B5_mofe_mlp,20260924,0.0/sync,1065 +B5_mofe_mlp,20260924,0.1/async,1059 +B5_mofe_mlp,20260924,0.1/partial,1029 +B5_mofe_mlp,20260924,0.1/single,1034 +B5_mofe_mlp,20260924,0.1/sync,969 +B5_mofe_mlp,20260924,0.3/async,1047 +B5_mofe_mlp,20260924,0.3/partial,989 +B5_mofe_mlp,20260924,0.3/single,978 +B5_mofe_mlp,20260924,0.3/sync,1014 +B5_mofe_mlp,20260924,0.5/async,1004 +B5_mofe_mlp,20260924,0.5/partial,1017 +B5_mofe_mlp,20260924,0.5/single,1013 +B5_mofe_mlp,20260924,0.5/sync,986 +B5_mofe_mlp,20260924,0.7/async,975 +B5_mofe_mlp,20260924,0.7/partial,1006 +B5_mofe_mlp,20260924,0.7/single,982 +B5_mofe_mlp,20260924,0.7/sync,1011 diff --git a/deep_learning/Q2/outputs/followups/README.md b/deep_learning/Q2/outputs/followups/README.md new file mode 100644 index 0000000..75705a1 --- /dev/null +++ b/deep_learning/Q2/outputs/followups/README.md @@ -0,0 +1,9 @@ +# 后续实验输出 + +每项新实验在本目录下建立唯一子目录,例如 `F01_local_repair/`。把假设、运行配置、逐条件指标、统计比较、检查点和诊断图都放在该子目录中;不要覆盖已保留的参照权重与标准化参数: + +```text +../mofe_7experts/ +``` + +新实验的统一条件和记录要求见 [Q2 实验协议](../../EXPERIMENT_PROTOCOL.md)。 diff --git a/deep_learning/Q2/outputs/mofe_7experts/aligned_robust_stats.npz b/deep_learning/Q2/outputs/mofe_7experts/aligned_robust_stats.npz new file mode 100644 index 0000000..3d7956d Binary files /dev/null and b/deep_learning/Q2/outputs/mofe_7experts/aligned_robust_stats.npz differ diff --git a/deep_learning/Q2/outputs/mofe_7experts/models/B5_mofe_mlp/seed_2026/model_best.pt b/deep_learning/Q2/outputs/mofe_7experts/models/B5_mofe_mlp/seed_2026/model_best.pt new file mode 100644 index 0000000..6ce82a8 Binary files /dev/null and b/deep_learning/Q2/outputs/mofe_7experts/models/B5_mofe_mlp/seed_2026/model_best.pt differ diff --git a/deep_learning/Q2/outputs/mofe_7experts/models/B5_mofe_mlp/seed_3407/model_best.pt b/deep_learning/Q2/outputs/mofe_7experts/models/B5_mofe_mlp/seed_3407/model_best.pt new file mode 100644 index 0000000..2c215d9 Binary files /dev/null and b/deep_learning/Q2/outputs/mofe_7experts/models/B5_mofe_mlp/seed_3407/model_best.pt differ diff --git a/deep_learning/Q2/outputs/mofe_7experts/models/B5_mofe_mlp/seed_42/model_best.pt b/deep_learning/Q2/outputs/mofe_7experts/models/B5_mofe_mlp/seed_42/model_best.pt new file mode 100644 index 0000000..78b5ac0 Binary files /dev/null and b/deep_learning/Q2/outputs/mofe_7experts/models/B5_mofe_mlp/seed_42/model_best.pt differ diff --git a/deep_learning/Q2/outputs/mofe_7experts/models/baselines/concat/seed_2026/model_best.pt b/deep_learning/Q2/outputs/mofe_7experts/models/baselines/concat/seed_2026/model_best.pt new file mode 100644 index 0000000..ac1d764 Binary files /dev/null and b/deep_learning/Q2/outputs/mofe_7experts/models/baselines/concat/seed_2026/model_best.pt differ diff --git a/deep_learning/Q2/outputs/mofe_7experts/models/baselines/concat/seed_3407/model_best.pt b/deep_learning/Q2/outputs/mofe_7experts/models/baselines/concat/seed_3407/model_best.pt new file mode 100644 index 0000000..e553b7d Binary files /dev/null and b/deep_learning/Q2/outputs/mofe_7experts/models/baselines/concat/seed_3407/model_best.pt differ diff --git a/deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/seed_42/model_best.pt b/deep_learning/Q2/outputs/mofe_7experts/models/baselines/concat/seed_42/model_best.pt similarity index 100% rename from deep_learning/Q2/outputs/algorithm_selection/models/aligned/concat/seed_42/model_best.pt rename to deep_learning/Q2/outputs/mofe_7experts/models/baselines/concat/seed_42/model_best.pt diff --git a/deep_learning/Q2/pyproject.toml b/deep_learning/Q2/pyproject.toml index e91a7a8..1dc1c3c 100644 --- a/deep_learning/Q2/pyproject.toml +++ b/deep_learning/Q2/pyproject.toml @@ -1,5 +1,5 @@ [project] -name = "deep-learning-q2-q3-selection" +name = "deep-learning-q2" version = "0.1.0" requires-python = ">=3.14" dependencies = [ @@ -7,7 +7,6 @@ dependencies = [ "numpy>=2.5.3", "scikit-learn>=1.9.1", "torch>=2.14.0", - "transformers>=5.17.0", ] [tool.uv.sources] diff --git a/deep_learning/Q2/q2/__init__.py b/deep_learning/Q2/q2/__init__.py index 56d18d5..8214e10 100644 --- a/deep_learning/Q2/q2/__init__.py +++ b/deep_learning/Q2/q2/__init__.py @@ -1 +1 @@ -"""Q2 robustness and Q3 explanation-selection experiments.""" +"""Q2 multimodal emotion-recognition experiments.""" diff --git a/deep_learning/Q2/q2/evaluate_math_protocol.py b/deep_learning/Q2/q2/evaluate_math_protocol.py new file mode 100644 index 0000000..c064cbd --- /dev/null +++ b/deep_learning/Q2/q2/evaluate_math_protocol.py @@ -0,0 +1,614 @@ +"""Score the frozen EarlyConcat and MoFE checkpoints using the math-Q2 protocol. + +This script performs no training and selects no models. It evaluates the saved +three-seed checkpoints on the official labeled test split once, and reuses the +fixed 42-scenario validation-mask audit as the controlled-missingness protocol. +All generated files stay under deep_learning/Q2/outputs/followups/. +""" +from __future__ import annotations + +import csv +import argparse +import hashlib +import json +import math +import statistics +import time +from collections import defaultdict +from pathlib import Path +from typing import Any + +import numpy as np +import torch +from sklearn.metrics import accuracy_score, f1_score, mean_absolute_error, mean_squared_error +from torch import nn + +from .data import ( + ATTACHMENT2, + MODALITIES, + RobustStats, + Split, + _ids_and_targets, + _text_mask, + _unpickle, + apply_robust_stats, + fit_robust_stats, + load_aligned, +) +from .models import AlignedFusionModel +from .mofe import MixtureOfFusionExperts +from .train_mofe import MODEL_CONFIG, _predict, _device_for, EARLYCONCAT, MOFE7_MLP + + +Q2_ROOT = Path(__file__).resolve().parents[1] +REPO_ROOT = Q2_ROOT.parents[1] +REFERENCE_DIR = Q2_ROOT / "outputs" / "followups" / "R01_selected_model_reevaluation" +OUTPUT_DIR = Q2_ROOT / "outputs" / "followups" / "R02_math_protocol_evaluation" +SEEDS = (42, 3407, 2026) +BOOTSTRAP_REPS = 1000 +TEST_BOOTSTRAP_SEED = 20260925 +AURC_BOOTSTRAP_SEED = 20260926 +SCENARIO_SEED = 20261833 +METHODS = (EARLYCONCAT, MOFE7_MLP) +CURVE_MODES = ("single", "sync", "partial", "async") +CURVE_RATES = (0.0, 0.1, 0.3, 0.5, 0.7) + + +def write_csv(path: Path, rows: list[dict[str, Any]]) -> None: + if not rows: + return + path.parent.mkdir(parents=True, exist_ok=True) + fields = list(dict.fromkeys(key for row in rows for key in row)) + with path.open("w", newline="", encoding="utf-8-sig") as stream: + writer = csv.DictWriter(stream, fieldnames=fields) + writer.writeheader() + writer.writerows(rows) + + +def sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return digest.hexdigest() + + +def split_from_part(part: dict[str, Any]) -> Split: + xs = tuple(np.asarray(part[name], dtype=np.float32) for name in MODALITIES) + masks = [ + _text_mask(part), + np.any(np.isfinite(xs[1]) & (xs[1] != 0), axis=-1), + np.any(np.isfinite(xs[2]) & (xs[2] != 0), axis=-1), + ] + ids, y_cls, y_reg = _ids_and_targets(part) + return Split(xs, np.stack(masks, axis=-1), y_cls, y_reg, ids) + + +def load_splits(feature_path: Path) -> dict[str, Split]: + raw = _unpickle(feature_path) + usual = load_aligned(feature_path) + splits = {"train": usual["train"], "valid": usual["valid"], "test": split_from_part(raw["test"])} + groups = { + name: {sample_id.split("$_$", 1)[0] for sample_id in split.ids} + for name, split in splits.items() + } + for first, second in (("train", "valid"), ("train", "test"), ("valid", "test")): + overlap = groups[first] & groups[second] + if overlap: + raise ValueError(f"official {first}/{second} source-video groups overlap: {len(overlap)}") + for name, split in splits.items(): + expected = np.where(split.y_reg < 0, 0, np.where(split.y_reg == 0, 1, 2)) + if not np.array_equal(expected, split.y_cls): + raise ValueError(f"{name}: classification labels disagree with strict sign of regression labels") + return splits + + +def metrics(split: Split, logits: np.ndarray, intensity: np.ndarray, indices: np.ndarray | None = None) -> dict[str, float]: + if indices is None: + indices = np.arange(split.n) + y_cls = split.y_cls[indices] + y_reg = split.y_reg[indices] + pred_cls = np.asarray(logits)[indices].argmax(axis=-1) + pred_reg = np.clip(np.asarray(intensity).reshape(-1)[indices], -3.0, 3.0) + return { + "accuracy": float(accuracy_score(y_cls, pred_cls)), + "macro_f1": float(f1_score(y_cls, pred_cls, labels=[0, 1, 2], average="macro", zero_division=0)), + "mae": float(mean_absolute_error(y_reg, pred_reg)), + "rmse": float(math.sqrt(mean_squared_error(y_reg, pred_reg))), + "pearson": float(np.corrcoef(y_reg, pred_reg)[0, 1]) if np.std(y_reg) > 0 and np.std(pred_reg) > 0 else float("nan"), + } + + +def load_model(method: str, seed: int, dims: tuple[int, int, int], device: torch.device) -> nn.Module: + if method == EARLYCONCAT: + checkpoint = REFERENCE_DIR / "models" / "baselines" / "concat" / f"seed_{seed}" / "model_best.pt" + model: nn.Module = AlignedFusionModel("concat", dims=dims).to(device) + state = torch.load(checkpoint, map_location=device, weights_only=False) + if state.get("kind") != "concat" or int(state.get("seed", -1)) != seed: + raise ValueError(f"unexpected EarlyConcat checkpoint: {checkpoint}") + elif method == MOFE7_MLP: + checkpoint = REFERENCE_DIR / "models" / MOFE7_MLP / f"seed_{seed}" / "model_best.pt" + model = MixtureOfFusionExperts(dims=dims, **MODEL_CONFIG).to(device) + state = torch.load(checkpoint, map_location=device, weights_only=False) + if state.get("config") != MODEL_CONFIG or int(state.get("seed", -1)) != seed: + raise ValueError(f"unexpected MoFE checkpoint: {checkpoint}") + else: + raise ValueError(f"unknown model {method}") + if tuple(state.get("dims", ())) != dims: + raise ValueError(f"feature dimensions do not match checkpoint: {checkpoint}") + model.load_state_dict(state["state_dict"]) + model.eval() + return model + + +def best_interval(visible: np.ndarray, wanted: int, cap: int, location: str, rng: np.random.Generator) -> tuple[int, int] | None: + steps = len(visible) + candidates: list[tuple[int, int, int, int]] = [] + for left in range(steps): + hits = 0 + for right in range(left, steps): + hits += int(visible[right]) + count = min(hits, cap) + if count: + candidates.append((abs(count - wanted), right - left + 1, left, right)) + if not candidates: + return None + best = min((error, span) for error, span, _, _ in candidates) + tied = [(left, right) for error, span, left, right in candidates if (error, span) == best] + if location == "start": + return min(tied, key=lambda pair: (pair[0], pair[1])) + if location == "end": + return max(tied, key=lambda pair: (pair[1], pair[0])) + if location == "middle": + center = (steps - 1) / 2 + return min(tied, key=lambda pair: (abs((pair[0] + pair[1]) / 2 - center), pair[0])) + if location != "random": + raise ValueError(f"unknown interval location: {location}") + return tied[int(rng.integers(0, len(tied)))] + + +def spread_short_spans(visible: np.ndarray, wanted: int, cap: int) -> np.ndarray: + positions = np.flatnonzero(visible) + count = min(int(wanted), int(cap), len(positions)) + chosen = np.zeros(len(visible), dtype=bool) + if count <= 0: + return chosen + n_spans = min(3, count) + chunks = np.array_split(positions, n_spans) + allocations = [count // n_spans + int(i < count % n_spans) for i in range(n_spans)] + for chunk, amount in zip(chunks, allocations): + if amount <= 0 or len(chunk) == 0: + continue + amount = min(amount, len(chunk)) + start = max(0, (len(chunk) - amount) // 2) + chosen[chunk[start:start + amount]] = True + return chosen + + +def continuous_mask( + original: np.ndarray, + rate: float, + mode: str, + rng: np.random.Generator, + *, + modalities: tuple[int, ...] | None = None, + location: str = "random", + span_structure: str = "long", +) -> np.ndarray: + """Reproduce math/Q2 continuous masking on this model's observed positions.""" + observed = np.asarray(original, dtype=bool) + result = observed.copy() + if rate <= 0 or mode == "none": + return result + steps, modality_count = observed.shape + present = [m for m in range(modality_count) if observed[:, m].any()] + if not present: + return result + if modalities is not None: + selected = [int(m) for m in modalities if int(m) in present] + if not selected: + return result + elif mode == "single": + selected = [int(rng.choice(present))] + elif mode in {"sync", "partial", "async"}: + if len(present) == 1: + selected = present + else: + count = int(rng.integers(2, min(3, len(present)) + 1)) + selected = sorted(int(v) for v in rng.choice(present, size=count, replace=False)) + else: + raise ValueError(f"unknown mask mode: {mode}") + + def max_hide(modality: int) -> int: + count = int(observed[:, modality].sum()) + keep = max(1, int(math.ceil(0.2 * count))) + return max(0, count - keep) + + target = {m: min(max_hide(m), int(round(rate * int(observed[:, m].sum())))) for m in selected} + if mode == "sync": + span = max(1, int(round(rate * steps))) + if location == "start": + left = 0 + elif location == "end": + left = steps - span + elif location == "middle": + left = (steps - span) // 2 + elif location == "random": + left = int(rng.integers(0, max(1, steps - span + 1))) + else: + raise ValueError(f"unknown interval location: {location}") + right = min(steps - 1, left + span - 1) + for m in selected: + candidates = np.flatnonzero(observed[left:right + 1, m]) + left + amount = min(len(candidates), max_hide(m), target[m]) + if amount: + offset = 0 if location != "end" else len(candidates) - amount + result[candidates[max(0, offset):max(0, offset) + amount], m] = False + else: + common_span = max(1, int(round(rate * steps))) + for rank, m in enumerate(selected): + wanted = target[m] + if wanted <= 0: + continue + cap = max_hide(m) + if span_structure == "multi_short": + hide = spread_short_spans(observed[:, m], wanted, cap) + elif span_structure != "long": + raise ValueError(f"unknown span structure: {span_structure}") + elif mode == "single" and location != "random": + # Place a contiguous block at the requested relative location + # among observed positions, while keeping the selected-source + # missing amount fixed. This avoids treating padding as time. + interval = best_interval(observed[:, m], wanted, cap, location, rng) + hide = np.zeros(steps, dtype=bool) + if interval is not None: + left, right = interval + candidates = np.flatnonzero(observed[left:right + 1, m]) + left + amount = min(len(candidates), wanted, cap) + if amount: + offset = 0 if location != "end" else len(candidates) - amount + hide[candidates[max(0, offset):max(0, offset) + amount]] = True + elif mode in {"partial", "async"}: + if mode == "partial": + base_left = int(rng.integers(0, max(1, steps - common_span + 1))) if location == "random" else ( + 0 if location == "start" else steps - common_span if location == "end" else (steps - common_span) // 2 + ) + offset = int(round(rank * common_span * 0.5)) + else: + base_left = 0 if location == "random" else ( + 0 if location == "start" else steps - common_span if location == "end" else (steps - common_span) // 2 + ) + available = max(1, steps - common_span + 1) + offsets = np.rint(np.linspace(0, max(0, available - 1), len(selected))).astype(int) + if location == "random": + rng.shuffle(offsets) + offset = int(offsets[rank]) + left = min(max(0, base_left + offset), max(0, steps - common_span)) + right = min(steps - 1, left + common_span - 1) + hide = np.zeros(steps, dtype=bool) + candidates = np.flatnonzero(observed[left:right + 1, m]) + left + amount = min(len(candidates), wanted, cap) + if amount: + hide[candidates[:amount]] = True + else: + interval = best_interval(observed[:, m], wanted, cap, location, rng) + hide = np.zeros(steps, dtype=bool) + if interval is not None: + left, right = interval + candidates = np.flatnonzero(observed[left:right + 1, m]) + left + amount = min(len(candidates), wanted, cap) + if amount: + offset = 0 if location != "end" else len(candidates) - amount + hide[candidates[max(0, offset):max(0, offset) + amount]] = True + result[hide, m] = False + return result + + +def scenario_seed(seed: int, sample_id: str, key: str) -> int: + return int.from_bytes(hashlib.sha256(f"{seed}:{sample_id}:{key}".encode()).digest()[:8], "little") + + +def make_scenarios(valid: Split, seed: int = SCENARIO_SEED) -> dict[str, np.ndarray]: + scenarios = {"0.0/none": valid.mask.copy()} + for rate in CURVE_RATES[1:]: + for mode in CURVE_MODES: + key = f"{rate:.1f}/{mode}" + scenarios[key] = np.stack([ + continuous_mask(mask, rate, mode, np.random.default_rng(scenario_seed(seed, sample_id, key))) + for sample_id, mask in zip(valid.ids, valid.mask) + ]) + modality_sets = (((0,), "T"), ((1,), "A"), ((2,), "V"), ((0, 1), "TA"), ((0, 2), "TV"), ((1, 2), "AV"), ((0, 1, 2), "TAV")) + for selected, label in modality_sets: + key = f"0.3/modality_{label}" + scenarios[key] = np.stack([ + continuous_mask(mask, 0.3, "sync", np.random.default_rng(scenario_seed(seed, sample_id, key)), modalities=selected) + for sample_id, mask in zip(valid.ids, valid.mask) + ]) + for modality_index, label in enumerate(("T", "A", "V")): + for location in ("start", "middle", "end"): + key = f"0.3/location_{location}_{label}" + scenarios[key] = np.stack([ + continuous_mask(mask, 0.3, "single", np.random.default_rng(scenario_seed(seed, sample_id, key)), modalities=(modality_index,), location=location) + for sample_id, mask in zip(valid.ids, valid.mask) + ]) + for structure in ("long", "multi_short"): + key = f"0.3/span_{structure}_{label}" + scenarios[key] = np.stack([ + continuous_mask(mask, 0.3, "single", np.random.default_rng(scenario_seed(seed, sample_id, key)), modalities=(modality_index,), span_structure=structure) + for sample_id, mask in zip(valid.ids, valid.mask) + ]) + for mode in ("sync", "partial", "async"): + key = f"0.3/synchrony_{mode}" + scenarios[key] = np.stack([ + continuous_mask(mask, 0.3, mode, np.random.default_rng(scenario_seed(seed, sample_id, key)), modalities=(0, 1, 2)) + for sample_id, mask in zip(valid.ids, valid.mask) + ]) + return scenarios + + +def actual_additional_rates(base: np.ndarray, scenarios: dict[str, np.ndarray]) -> dict[str, np.ndarray]: + result = {} + observed = base.sum(axis=1) + for scenario, current in scenarios.items(): + newly_hidden = base & ~current + hidden_count = newly_hidden.sum(axis=1) + by_modality = np.divide( + hidden_count, + observed, + out=np.full(hidden_count.shape, np.nan, dtype=np.float64), + where=observed > 0, + ) + result[scenario] = np.nanmean(by_modality, axis=1) + return result + + +def aurc_from_curve(rates: list[float], maes: list[float]) -> float: + order = np.argsort(np.asarray(rates), kind="stable") + x = np.asarray(rates, dtype=np.float64)[order] + y = np.asarray(maes, dtype=np.float64)[order] + unique_x, inverse = np.unique(x, return_inverse=True) + unique_y = np.asarray([y[inverse == i].mean() for i in range(len(unique_x))]) + if len(unique_x) <= 1 or unique_x[-1] <= 0: + return float(maes[0]) + return float(np.trapezoid(unique_y, unique_x) / unique_x[-1]) + + +def curve_scenarios(mode: str) -> list[str]: + return ["0.0/none"] + [f"{rate:.1f}/{mode}" for rate in CURVE_RATES[1:]] + + +def group_indices(ids: list[str]) -> tuple[list[str], dict[str, np.ndarray]]: + groups = sorted({sample_id.split("$_$", 1)[0] for sample_id in ids}) + mapping = {group: np.flatnonzero(np.asarray([x.split("$_$", 1)[0] == group for x in ids])) for group in groups} + return groups, mapping + + +def bootstrap_clean_test( + split: Split, + preds: dict[tuple[str, int], dict[str, np.ndarray]], +) -> list[dict[str, Any]]: + groups, mapping = group_indices(split.ids) + rng = np.random.default_rng(TEST_BOOTSTRAP_SEED) + draws: dict[str, list[float]] = defaultdict(list) + for _ in range(BOOTSTRAP_REPS): + chosen = rng.choice(groups, size=len(groups), replace=True) + indices = np.concatenate([mapping[group] for group in chosen]) + per_method = {} + for method in METHODS: + per_seed = [metrics(split, preds[(method, seed)]["logits"], preds[(method, seed)]["intensity"], indices) for seed in SEEDS] + per_method[method] = {key: float(np.mean([row[key] for row in per_seed])) for key in per_seed[0]} + for metric in per_method[EARLYCONCAT]: + draws[metric].append(per_method[MOFE7_MLP][metric] - per_method[EARLYCONCAT][metric]) + rows = [] + for metric, values in draws.items(): + rows.append({ + "comparison": "MoFE-7 + MLP Router minus EarlyConcat + BiGRU", + "metric": metric, + "delta_mean_over_seeds": float(np.mean([r[metric] for r in [ + metrics(split, preds[(MOFE7_MLP, seed)]["logits"], preds[(MOFE7_MLP, seed)]["intensity"]) + for seed in SEEDS + ]]) - np.mean([r[metric] for r in [ + metrics(split, preds[(EARLYCONCAT, seed)]["logits"], preds[(EARLYCONCAT, seed)]["intensity"]) + for seed in SEEDS + ]])), + "bootstrap_ci_2p5": float(np.quantile(values, 0.025)), + "bootstrap_ci_97p5": float(np.quantile(values, 0.975)), + "bootstrap_probability_delta_gt_0": float(np.mean(np.asarray(values) > 0)), + "replicates": BOOTSTRAP_REPS, + "resampling_unit": "source video id", + "paired": True, + "seed": TEST_BOOTSTRAP_SEED, + }) + return rows + + +def bootstrap_aurc( + valid: Split, + predictions: dict[tuple[str, int, str], dict[str, np.ndarray]], + scenarios: dict[str, np.ndarray], + rates_by_sample: dict[str, np.ndarray], +) -> list[dict[str, Any]]: + groups, mapping = group_indices(valid.ids) + rng = np.random.default_rng(AURC_BOOTSTRAP_SEED) + delta_by_mode: dict[str, list[float]] = {mode: [] for mode in CURVE_MODES} + for _ in range(BOOTSTRAP_REPS): + chosen = rng.choice(groups, size=len(groups), replace=True) + indices = np.concatenate([mapping[group] for group in chosen]) + for mode in CURVE_MODES: + keys = curve_scenarios(mode) + model_aucs: dict[str, list[float]] = {method: [] for method in METHODS} + for method in METHODS: + for seed in SEEDS: + xs = [float(np.nanmean(rates_by_sample[key][indices])) for key in keys] + ys = [float(np.abs(valid.y_reg[indices] - predictions[(method, seed, key)]["intensity"][indices]).mean()) for key in keys] + model_aucs[method].append(aurc_from_curve(xs, ys)) + delta_by_mode[mode].append(float(np.mean(model_aucs[MOFE7_MLP]) - np.mean(model_aucs[EARLYCONCAT]))) + point = {} + for mode in CURVE_MODES: + model_aucs = {} + for method in METHODS: + model_aucs[method] = [] + for seed in SEEDS: + keys = curve_scenarios(mode) + xs = [float(np.nanmean(rates_by_sample[key])) for key in keys] + ys = [float(np.abs(valid.y_reg - predictions[(method, seed, key)]["intensity"]).mean()) for key in keys] + model_aucs[method].append(aurc_from_curve(xs, ys)) + point[mode] = float(np.mean(model_aucs[MOFE7_MLP]) - np.mean(model_aucs[EARLYCONCAT])) + rows = [] + for mode, values in delta_by_mode.items(): + rows.append({ + "mode": mode, + "delta_aurc_mae_mofe_minus_earlyconcat": point[mode], + "bootstrap_ci_2p5": float(np.quantile(values, 0.025)), + "bootstrap_ci_97p5": float(np.quantile(values, 0.975)), + "bootstrap_probability_delta_lt_0": float(np.mean(np.asarray(values) < 0)), + "replicates": BOOTSTRAP_REPS, + "resampling_unit": "source video id", + "paired": True, + "seed": AURC_BOOTSTRAP_SEED, + }) + return rows + + +def run(device_name: str = "auto", batch_size: int = 64, masks_only: bool = False) -> None: + OUTPUT_DIR.mkdir(parents=True, exist_ok=True) + device = _device_for(device_name) + torch.set_num_threads(4) + torch.backends.cudnn.deterministic = True + torch.backends.cudnn.benchmark = False + + feature_path = ATTACHMENT2 / "aligned_50.pkl" + with (REFERENCE_DIR / "run_manifest.json").open("r", encoding="utf-8") as stream: + reference_manifest = json.load(stream) + if sha256(feature_path) != reference_manifest["feature_sha256"]: + raise ValueError("current official feature file hash differs from the checkpoint evaluation manifest") + + raw_splits = load_splits(feature_path) + train = raw_splits["train"] + valid = raw_splits["valid"] + test = raw_splits["test"] + scaler_path = REFERENCE_DIR / "aligned_robust_stats.npz" + stats = RobustStats.load(scaler_path) + computed = fit_robust_stats(train) + scaler_diff = max( + max(float(np.max(np.abs(a - b))) for a, b in zip(computed.center, stats.center)), + max(float(np.max(np.abs(a - b))) for a, b in zip(computed.scale, stats.scale)), + ) + if scaler_diff > 1e-6: + raise ValueError(f"checkpoint scaler is not the train-only scaler (max difference {scaler_diff})") + valid = apply_robust_stats(valid, stats) + test = apply_robust_stats(test, stats) + dims = tuple(x.shape[-1] for x in train.x) + + if not masks_only: + # Final, clean official-test evaluation; no retraining or selection occurs here. + test_predictions: dict[tuple[str, int], dict[str, np.ndarray]] = {} + test_rows: list[dict[str, Any]] = [] + for method in METHODS: + for seed in SEEDS: + model = load_model(method, seed, dims, device) + prediction = _predict(model, test, test.mask, device, batch_size) + test_predictions[(method, seed)] = prediction + test_rows.append({"method": method, "seed": seed, "n_test": test.n, **metrics(test, prediction["logits"], prediction["intensity"])}) + del model + if torch.cuda.is_available(): + torch.cuda.empty_cache() + + summary_rows = [] + for method in METHODS: + subset = [row for row in test_rows if row["method"] == method] + for metric in ("accuracy", "macro_f1", "mae", "rmse", "pearson"): + values = [float(row[metric]) for row in subset] + summary_rows.append({"method": method, "metric": metric, "mean": float(np.mean(values)), "sd_across_seeds": float(np.std(values, ddof=1))}) + write_csv(OUTPUT_DIR / "official_test_metrics_by_seed.csv", test_rows) + write_csv(OUTPUT_DIR / "official_test_summary.csv", summary_rows) + write_csv(OUTPUT_DIR / "official_test_paired_bootstrap.csv", bootstrap_clean_test(test, test_predictions)) + + # Reproduce the math-Q2 42-scenario design with a per-sample stable seed, + # while applying it to the observation masks used to train these models. + scenario_masks = make_scenarios(valid) + rates_by_sample = actual_additional_rates(valid.mask, scenario_masks) + condition_predictions: dict[tuple[str, int, str], dict[str, np.ndarray]] = {} + condition_rows: list[dict[str, Any]] = [] + for method in METHODS: + for seed in SEEDS: + model = load_model(method, seed, dims, device) + for scenario, masks in scenario_masks.items(): + prediction = _predict(model, valid, masks, device, batch_size) + condition_predictions[(method, seed, scenario)] = prediction + values = metrics(valid, prediction["logits"], prediction["intensity"]) + condition_rows.append({ + "method": method, + "seed": seed, + "scenario": scenario, + "realized_additional_global_rate": float(np.nanmean(rates_by_sample[scenario])), + "n_valid": valid.n, + **values, + }) + del model + if torch.cuda.is_available(): + torch.cuda.empty_cache() + write_csv(OUTPUT_DIR / "controlled_metrics_by_scenario.csv", condition_rows) + + auc_rows: list[dict[str, Any]] = [] + for method in METHODS: + for seed in SEEDS: + for mode in CURVE_MODES: + keys = curve_scenarios(mode) + xs = [float(np.nanmean(rates_by_sample[key])) for key in keys] + ys = [float(np.abs(valid.y_reg - condition_predictions[(method, seed, key)]["intensity"]).mean()) for key in keys] + auc_rows.append({"method": method, "seed": seed, "mask_mode": mode, "aurc_mae": aurc_from_curve(xs, ys), "rates_realized": json.dumps(xs)}) + write_csv(OUTPUT_DIR / "aurc_mae_by_mode_seed.csv", auc_rows) + auc_summary = [] + for method in METHODS: + for mode in CURVE_MODES: + values = [row["aurc_mae"] for row in auc_rows if row["method"] == method and row["mask_mode"] == mode] + auc_summary.append({"method": method, "mask_mode": mode, "mean": float(np.mean(values)), "sd_across_seeds": float(np.std(values, ddof=1))}) + write_csv(OUTPUT_DIR / "aurc_mae_summary.csv", auc_summary) + write_csv(OUTPUT_DIR / "aurc_mae_paired_bootstrap.csv", bootstrap_aurc(valid, condition_predictions, scenario_masks, rates_by_sample)) + + manifest = { + "experiment": "Frozen EarlyConcat vs MoFE-7 evaluation under math/Q2 test protocol", + "created_unix": time.time(), + "device": str(device), + "cuda_device": torch.cuda.get_device_name(0) if device.type == "cuda" else None, + "feature_file": str(feature_path), + "feature_sha256": sha256(feature_path), + "representation": "official aligned_50 ordered positions; not physical-time bins", + "train_valid_test_counts": {name: split.n for name, split in raw_splits.items()}, + "source_video_groups": {name: len({sample_id.split("$_$", 1)[0] for sample_id in split.ids}) for name, split in raw_splits.items()}, + "official_group_splits_disjoint": True, + "test_evaluation": ( + "one final clean evaluation on official labeled test split; no training/model selection/calibration" + if not masks_only else "test outputs preserved from the earlier single evaluation; no test prediction was rerun" + ), + "test_prediction_performed_this_invocation": not masks_only, + "seeds": list(SEEDS), + "checkpoint_source": str(REFERENCE_DIR / "models"), + "train_only_scaler": str(scaler_path), + "scaler_max_abs_difference_from_train_refit": scaler_diff, + "test_labels_used_for_training_or_selection": False, + "controlled_missingness": { + "scenario_seed": SCENARIO_SEED, + "scenario_design": "math/Q2 42-scenario design regenerated on the Q2 models' BERT attention-mask base", + "scenarios": len(scenario_masks), + "AURC": "normalized trapezoidal area of MAE over realized equal-modality-weighted added missing rate, at 0/.1/.3/.5/.7 for single/sync/partial/async", + }, + "bootstrap": { + "replicates": BOOTSTRAP_REPS, + "test_seed": TEST_BOOTSTRAP_SEED, + "aurc_seed": AURC_BOOTSTRAP_SEED, + "unit": "source video id", + "paired": True, + }, + } + (OUTPUT_DIR / "run_manifest.json").write_text(json.dumps(manifest, indent=2), encoding="utf-8") + print(f"wrote math-protocol comparison to {OUTPUT_DIR}") + print(f"n_test={test.n}; n_valid={valid.n}; device={device}; scenarios={len(scenario_masks)}") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--masks-only", action="store_true", help="Recompute validation mask scenarios without rerunning official-test inference") + args = parser.parse_args() + run(masks_only=args.masks_only) diff --git a/deep_learning/Q2/q2/finalize_summary.py b/deep_learning/Q2/q2/finalize_summary.py index b015d47..2cd2a7a 100644 --- a/deep_learning/Q2/q2/finalize_summary.py +++ b/deep_learning/Q2/q2/finalize_summary.py @@ -9,7 +9,7 @@ from .train_compare import _plot, _summary, _write_csv def main() -> None: parser = argparse.ArgumentParser(description="Rebuild Q2 summary tables from saved validation predictions") - parser.add_argument("--output-dir", default=str(Path(__file__).resolve().parents[1] / "outputs" / "algorithm_selection")) + parser.add_argument("--output-dir", default=str(Path(__file__).resolve().parents[1] / "outputs" / "followups" / "earlyconcat_standalone")) args = parser.parse_args() output = Path(args.output_dir) with (output / "validation_metrics_by_condition.csv").open(encoding="utf-8-sig", newline="") as stream: diff --git a/deep_learning/Q2/q2/models.py b/deep_learning/Q2/q2/models.py index 5c725d8..debe0b4 100644 --- a/deep_learning/Q2/q2/models.py +++ b/deep_learning/Q2/q2/models.py @@ -5,6 +5,8 @@ from torch import nn class AlignedFusionModel(nn.Module): + """Early concatenation + BiGRU model for the supplied aligned sequence.""" + def __init__( self, kind: str, @@ -14,8 +16,8 @@ class AlignedFusionModel(nn.Module): dropout: float = 0.15, ) -> None: super().__init__() - if kind not in {"concat", "gate", "crossattn"}: - raise ValueError(f"unknown model kind: {kind}") + if kind != "concat": + raise ValueError(f"only the selected EarlyConcat model is maintained; got: {kind}") self.kind = kind self.hidden = hidden self.projections = nn.ModuleList( @@ -25,31 +27,9 @@ class AlignedFusionModel(nn.Module): self.position = nn.Parameter(torch.randn(1, steps, hidden) * 0.02) self.modality = nn.Parameter(torch.randn(1, 1, 3, hidden) * 0.02) self.dropout = nn.Dropout(dropout) - - if kind == "concat": - self.fusion = nn.Sequential( - nn.Linear(hidden * 3 + 3, hidden), nn.GELU(), nn.LayerNorm(hidden), nn.Dropout(dropout) - ) - elif kind == "gate": - self.gate_score = nn.Sequential(nn.Linear(hidden, hidden // 2), nn.Tanh(), nn.Linear(hidden // 2, 1)) - self.fusion = nn.Sequential( - nn.Linear(hidden + 3, hidden), nn.GELU(), nn.LayerNorm(hidden), nn.Dropout(dropout) - ) - else: - layer = nn.TransformerEncoderLayer( - d_model=hidden, - nhead=4, - dim_feedforward=hidden * 2, - dropout=dropout, - activation="gelu", - batch_first=True, - norm_first=True, - ) - self.cross_encoder = nn.TransformerEncoder(layer, num_layers=2, enable_nested_tensor=False) - self.fusion = nn.Sequential( - nn.Linear(hidden + 3, hidden), nn.GELU(), nn.LayerNorm(hidden), nn.Dropout(dropout) - ) - + self.fusion = nn.Sequential( + nn.Linear(hidden * 3 + 3, hidden), nn.GELU(), nn.LayerNorm(hidden), nn.Dropout(dropout) + ) self.temporal = nn.GRU( input_size=hidden, hidden_size=hidden // 2, @@ -72,38 +52,16 @@ class AlignedFusionModel(nn.Module): encoded.append(token) stack = torch.stack(encoded, dim=2) # B x T x M x D availability = masks.to(stack.dtype) - gate_weights = None - - if self.kind == "concat": - fused = self.fusion(torch.cat((stack.flatten(2), availability), dim=-1)) - elif self.kind == "gate": - scores = self.gate_score(stack).squeeze(-1) - scores = scores.masked_fill(~masks, -1e4) - gate_weights = torch.softmax(scores, dim=-1) * availability - gate_weights = gate_weights / gate_weights.sum(dim=-1, keepdim=True).clamp_min(1e-8) - weighted = (stack * gate_weights[..., None]).sum(dim=2) - fused = self.fusion(torch.cat((weighted, availability), dim=-1)) - else: - batch, steps, modalities, hidden = stack.shape - flat = stack.reshape(batch, steps * modalities, hidden) - valid = masks.reshape(batch, steps * modalities).clone() - empty = ~valid.any(dim=1) - if empty.any(): - valid[empty, 0] = True - flat[empty, 0] = 0.0 - attended = self.cross_encoder(flat, src_key_padding_mask=~valid) - attended = attended.reshape(batch, steps, modalities, hidden) - observed_count = availability.sum(dim=2, keepdim=True) - pooled = (attended * availability[..., None]).sum(dim=2) / observed_count.clamp_min(1.0) - fused = self.fusion(torch.cat((pooled, availability), dim=-1)) + fused = self.fusion(torch.cat((stack.flatten(2), availability), dim=-1)) temporal, _ = self.temporal(self.dropout(fused)) time_weight = masks.any(dim=-1).to(temporal.dtype) empty_time = time_weight.sum(dim=1, keepdim=True) <= 0 if empty_time.any(): time_weight[empty_time.squeeze(1), 0] = 1.0 - pooled = (temporal * time_weight[..., None]).sum(dim=1) / time_weight.sum(dim=1, keepdim=True).clamp_min(1.0) + pooled = (temporal * time_weight[..., None]).sum(dim=1) + pooled = pooled / time_weight.sum(dim=1, keepdim=True).clamp_min(1.0) hidden = self.head(pooled) logits = self.classifier(hidden) intensity = 3.0 * torch.tanh(self.regressor(hidden).squeeze(-1)) - return {"logits": logits, "intensity": intensity, "gate": gate_weights} + return {"logits": logits, "intensity": intensity} diff --git a/deep_learning/Q2/q2/mofe.py b/deep_learning/Q2/q2/mofe.py new file mode 100644 index 0000000..615e986 --- /dev/null +++ b/deep_learning/Q2/q2/mofe.py @@ -0,0 +1,224 @@ +from __future__ import annotations + +from typing import Any + +import torch +import torch.nn.functional as F +from torch import nn + + +SUBSETS: dict[str, tuple[int, ...]] = { + "T": (0,), + "A": (1,), + "V": (2,), + "TA": (0, 1), + "TV": (0, 2), + "AV": (1, 2), + "TAV": (0, 1, 2), +} +EXPERT_NAMES = tuple(SUBSETS) +EXPERT_BITS = { + name: tuple(int(i in indices) for i in range(3)) + for name, indices in SUBSETS.items() +} + + +class MixtureOfFusionExperts(nn.Module): + """Seven-subset, hard-availability MoFE with the selected MLP router. + + Each modality has a private projection. Experts only receive the private + projections belonging to their subset. The weighted result is passed + through one shared temporal backbone and one shared prediction head. + """ + + def __init__( + self, + dims: tuple[int, int, int], + router: str = "mlp", + expert_names: tuple[str, ...] = EXPERT_NAMES, + availability_mode: str = "hard", + steps: int = 50, + latent_dim: int = 64, + hidden: int = 128, + dropout: float = 0.15, + ) -> None: + super().__init__() + if router != "mlp": + raise ValueError(f"only the selected MLP router is maintained; got: {router}") + if availability_mode != "hard": + raise ValueError(f"only hard availability masking is maintained; got: {availability_mode}") + if tuple(expert_names) != EXPERT_NAMES: + raise ValueError("the selected MoFE uses all seven modality-subset experts") + + self.dims = dims + self.router_kind = router + self.expert_names = tuple(expert_names) + self.availability_mode = availability_mode + self.steps = steps + self.latent_dim = latent_dim + self.hidden = hidden + + # These projections are private to each modality and are not tied. + self.private_projections = nn.ModuleList( + nn.Sequential(nn.Linear(size, latent_dim), nn.GELU()) for size in dims + ) + self.experts = nn.ModuleDict() + for name in self.expert_names: + n_modalities = len(SUBSETS[name]) + self.experts[name] = nn.Sequential( + nn.Linear(n_modalities * latent_dim, hidden), + nn.GELU(), + nn.Dropout(dropout), + nn.Linear(hidden, latent_dim), + nn.LayerNorm(latent_dim), + ) + + router_input_dim = 9 + self.router = nn.Sequential( + nn.Linear(router_input_dim, 16), + nn.GELU(), + nn.Linear(16, len(self.expert_names)), + ) + + # Shared early-fusion projection, BiGRU, and task heads. + self.all_missing_token = nn.Parameter(torch.zeros(1, 1, latent_dim)) + self.input_projection = nn.Sequential( + nn.Linear(latent_dim + 3, hidden), + nn.GELU(), + nn.LayerNorm(hidden), + nn.Dropout(dropout), + ) + self.dropout = nn.Dropout(dropout) + self.temporal = nn.GRU( + input_size=hidden, + hidden_size=hidden // 2, + num_layers=1, + batch_first=True, + bidirectional=True, + ) + self.head = nn.Sequential(nn.Linear(hidden, hidden // 2), nn.GELU(), nn.Dropout(dropout)) + self.classifier = nn.Linear(hidden // 2, 3) + self.regressor = nn.Linear(hidden // 2, 1) + + @staticmethod + def _availability(masks: torch.Tensor, names: tuple[str, ...]) -> torch.Tensor: + masks = masks.bool() + columns = [masks[..., list(SUBSETS[name])].all(dim=-1) for name in names] + return torch.stack(columns, dim=-1) + + def _router_features( + self, + private: tuple[torch.Tensor, torch.Tensor, torch.Tensor], + masks: torch.Tensor, + ) -> torch.Tensor: + observed = masks.to(dtype=private[0].dtype) + magnitude = torch.stack( + [torch.sqrt(x.square().mean(dim=-1) + 1e-8) for x in private], dim=-1 + ) + local_ratio = F.avg_pool1d( + observed.transpose(1, 2), kernel_size=5, stride=1, padding=2, count_include_pad=False + ).transpose(1, 2) + return torch.cat((observed, torch.log1p(magnitude), local_ratio), dim=-1) + + def _route( + self, + router_features: torch.Tensor, + availability: torch.Tensor, + force_expert: str | None, + ) -> torch.Tensor: + scores = self.router(router_features) + scores = scores.masked_fill(~availability, -1e4) + weights = torch.softmax(scores, dim=-1) * availability.to(scores.dtype) + # In the full seven-expert model this is exactly the all-modalities- + # missing case. It also safely handles ablations with no eligible set. + has_expert = availability.any(dim=-1, keepdim=True) + weights = weights * has_expert.to(weights.dtype) + weights = weights / weights.sum(dim=-1, keepdim=True).clamp_min(1e-8) + + if force_expert is not None: + if force_expert not in self.expert_names: + raise ValueError(f"expert {force_expert} is not enabled in this model") + expert_idx = self.expert_names.index(force_expert) + forced = torch.zeros_like(weights) + forced[..., expert_idx] = 1.0 + # Force the requested expert where its modality subset is present; + # where it is unavailable, use the learned router over eligible + # experts instead of replacing observed information with zeros. + return torch.where(availability[..., expert_idx, None], forced, weights) + + return weights + + def forward( + self, + xs: tuple[torch.Tensor, torch.Tensor, torch.Tensor], + masks: torch.Tensor, + force_expert: str | None = None, + ) -> dict[str, Any]: + masks = masks.bool() + if masks.ndim != 3 or masks.shape[-1] != 3: + raise ValueError(f"masks must have shape B x T x 3, got {tuple(masks.shape)}") + if masks.shape[1] > self.steps: + raise ValueError(f"sequence has {masks.shape[1]} steps, model supports {self.steps}") + + private_values = [] + for modality, (projector, x) in enumerate(zip(self.private_projections, xs)): + projected = projector(x) + projected = projected * masks[..., modality, None].to(projected.dtype) + private_values.append(projected) + private = tuple(private_values) + router_features = self._router_features(private, masks) + availability = self._availability(masks, self.expert_names) + + local_expert_outputs = [] + for name in self.expert_names: + indices = SUBSETS[name] + expert_input = torch.cat([private[i] for i in indices], dim=-1) + local_expert_outputs.append(self.experts[name](expert_input)) + expert_stack = torch.stack(local_expert_outputs, dim=-2) + + alpha_local = self._route(router_features, availability, force_expert) + fused = (expert_stack * alpha_local[..., None]).sum(dim=-2) + has_expert = availability.any(dim=-1) + fused = torch.where( + has_expert[..., None], fused, self.all_missing_token.expand_as(fused) + ) + + # Restore a stable seven-column interface for saved diagnostics, + # including expert-set ablations. + alpha = masks.new_zeros((*masks.shape[:2], len(EXPERT_NAMES)), dtype=private[0].dtype) + expert_outputs = private[0].new_zeros((*masks.shape[:2], len(EXPERT_NAMES), self.latent_dim)) + for local_idx, name in enumerate(self.expert_names): + global_idx = EXPERT_NAMES.index(name) + alpha[..., global_idx] = alpha_local[..., local_idx] + expert_outputs[..., global_idx, :] = expert_stack[..., local_idx, :] + + fused_with_masks = torch.cat((fused, masks.to(fused.dtype)), dim=-1) + encoded = self.input_projection(fused_with_masks) + temporal, _ = self.temporal(self.dropout(encoded)) + time_weight = masks.any(dim=-1).to(temporal.dtype) + empty_time = time_weight.sum(dim=1, keepdim=True) <= 0 + if empty_time.any(): + time_weight[empty_time.squeeze(1), 0] = 1.0 + pooled = (temporal * time_weight[..., None]).sum(dim=1) + pooled = pooled / time_weight.sum(dim=1, keepdim=True).clamp_min(1.0) + hidden = self.head(pooled) + logits = self.classifier(hidden) + intensity = 3.0 * torch.tanh(self.regressor(hidden).squeeze(-1)) + + bits = torch.tensor( + [EXPERT_BITS[name] for name in EXPERT_NAMES], + dtype=alpha.dtype, + device=alpha.device, + ) + utility = torch.einsum("bte,em->btm", alpha, bits) + return { + "logits": logits, + "intensity": intensity, + "fused": fused, + "alpha": alpha, + "utility": utility, + "availability": availability, + "expert_outputs": expert_outputs, + "fallback": ~has_expert, + "router_features": router_features, + } diff --git a/deep_learning/Q2/q2/task_preference.py b/deep_learning/Q2/q2/task_preference.py new file mode 100644 index 0000000..d504a22 --- /dev/null +++ b/deep_learning/Q2/q2/task_preference.py @@ -0,0 +1,298 @@ +from __future__ import annotations + +import argparse +import csv +import json +import sys +from pathlib import Path +from typing import Any + +import numpy as np +import torch + +from .data import RobustStats, apply_robust_stats, load_aligned +from .mofe import EXPERT_NAMES +from .mofe import MixtureOfFusionExperts +from .train_mofe import ( + MOFE7_MLP, + SEEDS, + _conditions, + _metric_dict, + _predict, + _sha256, + _write_csv, +) + + +ROOT = Path(__file__).resolve().parents[1] +REFERENCE_DIR = ROOT / "outputs" / "mofe_7experts" +DEFAULT_DIAGNOSTIC_OUTPUT = ROOT / "outputs" / "followups" / "D0_task_preference" +MODEL_CONFIG: dict[str, Any] = { + "router": "mlp", + "expert_names": EXPERT_NAMES, + "availability_mode": "hard", +} + + +def _load_model(seed: int, dims: tuple[int, int, int], device: torch.device) -> MixtureOfFusionExperts: + checkpoint = REFERENCE_DIR / "models" / MOFE7_MLP / f"seed_{seed}" / "model_best.pt" + saved = torch.load(checkpoint, map_location=device, weights_only=False) + if saved.get("config") != MODEL_CONFIG or tuple(saved.get("dims", ())) != dims: + raise ValueError(f"checkpoint does not match the selected single-router model: {checkpoint}") + if int(saved.get("seed", -1)) != seed: + raise ValueError(f"checkpoint seed mismatch: expected {seed}, found {saved.get('seed')}") + model = MixtureOfFusionExperts(dims=dims, **MODEL_CONFIG).to(device) + model.load_state_dict(saved["state_dict"]) + return model.eval() + + +def _rank_rows(rows: list[dict[str, Any]]) -> list[dict[str, Any]]: + groups: dict[tuple[int, str], list[dict[str, Any]]] = {} + for row in rows: + if row["expert"] == "learned_router": + continue + groups.setdefault((int(row["seed"]), str(row["condition"])), []).append(row) + + result: list[dict[str, Any]] = [] + for (seed, condition), values in sorted(groups.items()): + by_name = {str(row["expert"]): row for row in values} + ordered = [by_name[name] for name in EXPERT_NAMES] + f1 = [float(row["macro_f1"]) for row in ordered] + mae = [float(row["mae"]) for row in ordered] + pearson = [float(row["pearson"]) for row in ordered] + result.append({ + "seed": seed, + "condition": condition, + "spearman_macro_f1_vs_mae": _spearman(f1, mae), + "spearman_macro_f1_vs_pearson": _spearman(f1, pearson), + "best_macro_f1_expert": EXPERT_NAMES[int(np.argmax(f1))], + "best_mae_expert": EXPERT_NAMES[int(np.argmin(mae))], + "best_pearson_expert": EXPERT_NAMES[int(np.argmax(pearson))], + "macro_f1_order_best_to_worst": ">".join(EXPERT_NAMES[i] for i in np.argsort(-np.asarray(f1), kind="stable")), + "mae_order_best_to_worst": ">".join(EXPERT_NAMES[i] for i in np.argsort(np.asarray(mae), kind="stable")), + "pearson_order_best_to_worst": ">".join(EXPERT_NAMES[i] for i in np.argsort(-np.asarray(pearson), kind="stable")), + }) + return result + + +def _spearman(left: list[float], right: list[float]) -> float: + def average_ranks(values: list[float]) -> np.ndarray: + array = np.asarray(values, dtype=np.float64) + order = np.argsort(array, kind="stable") + ranks = np.empty(len(array), dtype=np.float64) + start = 0 + while start < len(array): + end = start + 1 + while end < len(array) and array[order[end]] == array[order[start]]: + end += 1 + ranks[order[start:end]] = (start + 1 + end) / 2 + start = end + return ranks + + left_ranks = average_ranks(left) + right_ranks = average_ranks(right) + if np.std(left_ranks) == 0 or np.std(right_ranks) == 0: + return 0.0 + return float(np.corrcoef(left_ranks, right_ranks)[0, 1]) + + +def _summary_rows(rows: list[dict[str, Any]]) -> list[dict[str, Any]]: + result = [] + for expert in (*EXPERT_NAMES, "learned_router"): + matching = [row for row in rows if row["expert"] == expert and row["condition"] != "clean"] + per_seed: dict[int, list[dict[str, Any]]] = {} + for row in matching: + per_seed.setdefault(int(row["seed"]), []).append(row) + seed_means = [] + for seed, seed_rows in sorted(per_seed.items()): + seed_means.append({ + metric: float(np.mean([float(row[metric]) for row in seed_rows])) + for metric in ("macro_f1", "mae", "pearson", "available_position_fraction") + }) + if not seed_means: + continue + out: dict[str, Any] = {"expert": expert, "n_seeds": len(seed_means), "conditions_averaged": len(matching) // len(seed_means)} + for metric in ("macro_f1", "mae", "pearson", "available_position_fraction"): + values = [item[metric] for item in seed_means] + out[f"corrupt_{metric}_mean"] = float(np.mean(values)) + out[f"corrupt_{metric}_seed_sd"] = float(np.std(values, ddof=1)) if len(values) > 1 else 0.0 + result.append(out) + return result + + +def _write_readout(output: Path, summary: list[dict[str, Any]], ranks: list[dict[str, Any]]) -> None: + rho_f1_mae = np.asarray([float(row["spearman_macro_f1_vs_mae"]) for row in ranks]) + rho_f1_pearson = np.asarray([float(row["spearman_macro_f1_vs_pearson"]) for row in ranks]) + best_f1 = {name: sum(row["best_macro_f1_expert"] == name for row in ranks) for name in EXPERT_NAMES} + best_mae = {name: sum(row["best_mae_expert"] == name for row in ranks) for name in EXPERT_NAMES} + best_pearson = {name: sum(row["best_pearson_expert"] == name for row in ranks) for name in EXPERT_NAMES} + rank_by_condition: dict[str, list[dict[str, Any]]] = {} + for row in ranks: + rank_by_condition.setdefault(str(row["condition"]), []).append(row) + + def winners(condition: str, column: str) -> str: + matching = rank_by_condition[condition] + counts = {name: sum(row[column] == name for row in matching) for name in EXPERT_NAMES} + max_count = max(counts.values()) + names = [name for name, count in counts.items() if count == max_count] + return ", ".join(f"{name} ({max_count}/{len(matching)})" for name in names) + + by_name = {str(row["expert"]): row for row in summary} + lines = [ + "# Single-router MoFE 任务偏好诊断", + "", + "本诊断使用保留的 single-router 检查点和验证集,用于判断是否值得增加第二个 router;它不是测试集估计。", + "", + "## Forced-expert 规则", + "", + "所选模态子集可用的位置强制使用对应 expert;该子集不可用时,由已训练 router 在其他可用 expert 中选择;全模态缺失时沿用 learned missing token。可用率表示所选 expert 能被强制使用的位置比例。", + "", + "## 缺失条件平均指标", + "", + "下表先在每个 seed 内对 15 种连续块缺失条件求平均,再汇总三个 seed;seed 标准差见 CSV。", + "", + "| Expert | Macro-F1 ↑ | MAE ↓ | Pearson ↑ | 可强制使用比例 |", + "| --- | ---: | ---: | ---: | ---: |", + ] + for name in (*EXPERT_NAMES, "learned_router"): + row = by_name[name] + lines.append( + f"| {name} | {float(row['corrupt_macro_f1_mean']):.3f} | {float(row['corrupt_mae_mean']):.3f} | " + f"{float(row['corrupt_pearson_mean']):.3f} | {float(row['corrupt_available_position_fraction_mean']):.3f} |" + ) + lines.extend([ + "", + "## 两个任务的 expert 偏好", + "", + f"在 {len(ranks)} 个 seed—条件组合中,分类 Macro-F1 与回归 MAE 的平均 Spearman ρ 为 **{rho_f1_mae.mean():.3f}**。MAE 越低越好,因此负相关表示两个指标倾向于选中相似的 expert。Macro-F1 与 Pearson 的平均 ρ 为 **{rho_f1_pearson.mean():.3f}**。", + "", + "六个重点条件下的相关性先按三个 seed 求平均;最优 expert 一栏显示三个 seed 中的多数结果:", + "", + "| 条件 | ρ(Macro-F1, MAE) | ρ(Macro-F1, Pearson) | Macro-F1 最优 | MAE 最优 | Pearson 最优 |", + "| --- | ---: | ---: | --- | --- | --- |", + ]) + key_conditions = ( + ("clean", "Clean"), + ("text_30", "Text 30%"), + ("audio_30", "Audio 30%"), + ("vision_30", "Vision 30%"), + ("audio_vision_30", "Audio+Vision 30%"), + ("all_modalities_30", "All-modal 30%"), + ) + for condition, label in key_conditions: + condition_rows = rank_by_condition[condition] + rho_mae = float(np.mean([float(row["spearman_macro_f1_vs_mae"]) for row in condition_rows])) + rho_pearson = float(np.mean([float(row["spearman_macro_f1_vs_pearson"]) for row in condition_rows])) + lines.append( + f"| {label} | {rho_mae:.3f} | {rho_pearson:.3f} | " + f"{winners(condition, 'best_macro_f1_expert')} | {winners(condition, 'best_mae_expert')} | " + f"{winners(condition, 'best_pearson_expert')} |" + ) + lines.extend([ + "", + f"各指标的最优 expert 次数:Macro-F1({_format_counts(best_f1)});MAE({_format_counts(best_mae)});Pearson({_format_counts(best_pearson)})。", + "", + "当前排名没有显示稳定的分类—回归 expert 分工:Macro-F1 较高通常同时对应较低 MAE 和较高 Pearson;文本 expert 在分类与回归指标上都是最常见的赢家。因此,这项诊断**没有提供增加第二个 router 所需的任务特异模态偏好证据**。目前保留 single-router 作为活动参照;这不代表两个任务在任何数据或设置下都不可能需要不同路由。", + "", + "## 结论范围", + "", + "输入是官方提供的 50 个有序 wordpiece 位置。结果只反映这些位置及本次缺失掩码下的任务与 expert 关系,不表示物理时间可靠性。", + "", + "逐条件结果见 `forced_expert_metrics.csv` 和 `rank_concordance.csv`;跨 seed 汇总见 `expert_task_preference_summary.csv`。", + ]) + (output / "task_preference_diagnostic.md").write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def _format_counts(counts: dict[str, int]) -> str: + return ", ".join(f"{name}: {count}" for name, count in counts.items()) + + +def run(args: argparse.Namespace) -> None: + output = args.output_dir.resolve() + output.mkdir(parents=True, exist_ok=True) + device = torch.device("cuda" if args.device == "auto" and torch.cuda.is_available() else "cpu") if args.device == "auto" else torch.device(args.device) + torch.set_num_threads(args.threads) + + raw = load_aligned() + scaler_path = REFERENCE_DIR / "aligned_robust_stats.npz" + stats = RobustStats.load(scaler_path) + valid = apply_robust_stats(raw["valid"], stats) + dims = tuple(int(x.shape[-1]) for x in valid.x) + + metric_rows: list[dict[str, Any]] = [] + for seed in args.seeds: + model = _load_model(seed, dims, device) + conditions = _conditions(valid, seed) + for condition, rate, masks in conditions: + predictions = { + "learned_router": _predict(model, valid, masks, device, args.batch_size), + **{ + expert: _predict(model, valid, masks, device, args.batch_size, force_expert=expert) + for expert in EXPERT_NAMES + }, + } + for expert, prediction in predictions.items(): + coverage = 1.0 + if expert != "learned_router": + expert_index = EXPERT_NAMES.index(expert) + coverage = float(model._availability(torch.as_tensor(masks, dtype=torch.bool, device=device), EXPERT_NAMES)[..., expert_index].float().mean().item()) + else: + coverage = float(prediction["availability"].astype(bool).any(axis=-1).mean()) + metric_rows.append({ + "method": MOFE7_MLP, + "seed": seed, + "condition": condition, + "missing_rate": rate, + "expert": expert, + "n_valid": valid.n, + "available_position_fraction": coverage, + **_metric_dict(valid.y_cls, valid.y_reg, prediction["logits"], prediction["intensity"]), + }) + print(f"forced-expert diagnostic complete for seed={seed} on {device}", flush=True) + del model + if torch.cuda.is_available(): + torch.cuda.empty_cache() + + rank_rows = _rank_rows(metric_rows) + summary = _summary_rows(metric_rows) + _write_csv(output / "forced_expert_metrics.csv", metric_rows) + _write_csv(output / "expert_task_preference_summary.csv", summary) + _write_csv(output / "rank_concordance.csv", rank_rows) + _write_readout(output, summary, rank_rows) + feature_path = ROOT.parents[1] / "E题数据" / "附件2-数据集特征文件" / "aligned_50.pkl" + metadata = { + "diagnostic": "forced-expert task preference for the retained single-router MoFE-7", + "checkpoint_dir": str(REFERENCE_DIR / "models" / MOFE7_MLP), + "checkpoint_seeds": list(args.seeds), + "feature_file": str(feature_path), + "feature_sha256": _sha256(feature_path), + "scaler_file": str(scaler_path), + "scaler_sha256": _sha256(scaler_path), + "device": str(device), + "cuda_device": torch.cuda.get_device_name(0) if device.type == "cuda" else None, + "python_version": sys.version, + "torch_version": torch.__version__, + "numpy_version": np.__version__, + "valid_examples": valid.n, + "conditions": [condition for condition, _, _ in _conditions(valid, args.seeds[0])], + "corruption_seed_protocol": "seed + 13 + pattern_index*101 + int(rate*1000)", + "forced_expert_policy": "use the requested expert where its modality subset is available; fall back to the trained single router at positions where it is unavailable; all-missing positions use the learned missing token", + "scope_note": "50 official ordered wordpiece positions; no claim about physical-time reliability", + "interpretation_note": "Ranking agreement is descriptive on the supplied validation split; it is a motivation diagnostic, not an unbiased test-set estimate.", + } + (output / "run_manifest.json").write_text(json.dumps(metadata, indent=2, ensure_ascii=False), encoding="utf-8") + print(f"saved forced-expert diagnostic to {output}", flush=True) + + +def main() -> None: + parser = argparse.ArgumentParser(description="Measure classification/regression preferences across existing MoFE experts.") + parser.add_argument("--seeds", type=int, nargs="+", default=list(SEEDS)) + parser.add_argument("--batch-size", type=int, default=128) + parser.add_argument("--threads", type=int, default=4) + parser.add_argument("--device", default="auto") + parser.add_argument("--output-dir", type=Path, default=DEFAULT_DIAGNOSTIC_OUTPUT) + run(parser.parse_args()) + + +if __name__ == "__main__": + main() diff --git a/deep_learning/Q2/q2/train_compare.py b/deep_learning/Q2/q2/train_compare.py index c8d76bf..7716b60 100644 --- a/deep_learning/Q2/q2/train_compare.py +++ b/deep_learning/Q2/q2/train_compare.py @@ -43,7 +43,7 @@ PATTERNS = { "audio_vision": (1, 2), "all_modalities": (0, 1, 2), } -KINDS = ("concat", "gate", "crossattn") +KINDS = ("concat",) def seed_everything(seed: int) -> None: @@ -292,7 +292,7 @@ def _summary(rows: list[dict[str, Any]]) -> list[dict[str, Any]]: def _plot(summary: list[dict[str, Any]], rows: list[dict[str, Any]], path: Path) -> None: path.parent.mkdir(parents=True, exist_ok=True) - colors = {"concat": "#4e79a7", "gate": "#f28e2b", "crossattn": "#59a14f"} + colors = {"concat": "#4e79a7"} fig, axes = plt.subplots(1, 2, figsize=(11, 4.4), constrained_layout=True) for row in summary: kind = row["method"] @@ -474,14 +474,14 @@ def _run(args: argparse.Namespace) -> None: def main() -> None: - parser = argparse.ArgumentParser(description="Q2 local-missingness model and alignment transfer selection") + parser = argparse.ArgumentParser(description="Train the EarlyConcat baseline and its alignment-transfer control") parser.add_argument("--seeds", type=int, nargs="+", default=[42, 3407, 2026]) parser.add_argument("--epochs", type=int, default=32) parser.add_argument("--patience", type=int, default=6) parser.add_argument("--batch-size", type=int, default=64) parser.add_argument("--threads", type=int, default=4) parser.add_argument("--device", default="auto") - parser.add_argument("--output-dir", default=str(Path(__file__).resolve().parents[1] / "outputs" / "algorithm_selection")) + parser.add_argument("--output-dir", default=str(Path(__file__).resolve().parents[1] / "outputs" / "followups" / "earlyconcat_standalone")) args = parser.parse_args() _run(args) diff --git a/deep_learning/Q2/q2/train_math_protocol.py b/deep_learning/Q2/q2/train_math_protocol.py new file mode 100644 index 0000000..f2471ab --- /dev/null +++ b/deep_learning/Q2/q2/train_math_protocol.py @@ -0,0 +1,513 @@ +"""Retrain the two maintained Q2 models under the math/Q2 V2 protocol. + +The model architectures and joint CE + SmoothL1 objective stay unchanged. +Training masks, official splits, validation scenarios, and final-test handling +follow the corresponding math/Q2 protocol where those choices apply. +""" +from __future__ import annotations + +import argparse +import csv +import hashlib +import json +import math +import random +import time +from collections import Counter, defaultdict +from pathlib import Path +from typing import Any + +import numpy as np +import torch +import torch.nn.functional as F +from sklearn.metrics import accuracy_score, f1_score, mean_absolute_error, mean_squared_error +from torch import nn + +from .data import ATTACHMENT2, RobustStats, Split, apply_robust_stats, fit_robust_stats +from .evaluate_math_protocol import ( + AURC_BOOTSTRAP_SEED, + BOOTSTRAP_REPS, + CURVE_MODES, + METHODS, + SCENARIO_SEED, + TEST_BOOTSTRAP_SEED, + actual_additional_rates, + aurc_from_curve, + continuous_mask, + curve_scenarios, + load_splits, + make_scenarios, + metrics, + scenario_seed, + sha256, + write_csv, +) +from .models import AlignedFusionModel +from .mofe import MixtureOfFusionExperts +from .train_mofe import EARLYCONCAT, MODEL_CONFIG, MOFE7_MLP, _predict +from .train_compare import _loss, seed_everything + + +Q2_ROOT = Path(__file__).resolve().parents[1] +OUTPUT_DIR = Q2_ROOT / "outputs" / "followups" / "R03_math_protocol_retraining" +SEED = 20260924 +TRAIN_MASK_SEED = 20261227 +BATCH_SIZE = 64 +EPOCH_LIMIT = 12 +PATIENCE = 3 +LEARNING_RATE = 3e-4 +WEIGHT_DECAY = 1e-3 +SELECTION_SCENARIOS = ("0.0/none", "0.3/single", "0.3/sync", "0.5/async") +TRAIN_RATES = (0.0, 0.1, 0.3, 0.5, 0.7) +TRAIN_MODES = ("single", "sync", "partial", "async") + + +def device_for(name: str) -> torch.device: + if name == "auto": + return torch.device("cuda" if torch.cuda.is_available() else "cpu") + return torch.device(name) + + +def set_deterministic(seed: int) -> None: + seed_everything(seed) + torch.set_num_threads(4) + torch.backends.cudnn.deterministic = True + torch.backends.cudnn.benchmark = False + + +def build_model(method: str, dims: tuple[int, int, int], device: torch.device) -> nn.Module: + if method == EARLYCONCAT: + return AlignedFusionModel("concat", dims=dims).to(device) + if method == MOFE7_MLP: + return MixtureOfFusionExperts(dims=dims, **MODEL_CONFIG).to(device) + raise ValueError(f"unknown method: {method}") + + +def model_state(model: nn.Module, method: str) -> dict[str, Any]: + state: dict[str, Any] = { + "method": method, + "dims": tuple(int(x) for x in model_dims(model)), + "state_dict": model.state_dict(), + "seed": SEED, + "protocol": "math/Q2 V2 adapted deterministic-model training", + } + if method == EARLYCONCAT: + state["kind"] = "concat" + else: + state["config"] = MODEL_CONFIG + return state + + +def model_dims(model: nn.Module) -> tuple[int, int, int]: + if isinstance(model, AlignedFusionModel): + return tuple(layer[0].in_features for layer in model.projections) # type: ignore[return-value] + if isinstance(model, MixtureOfFusionExperts): + return tuple(layer[0].in_features for layer in model.private_projections) # type: ignore[return-value] + raise TypeError(type(model)) + + +def train_masks_for_epoch(split: Split, epoch: int) -> tuple[np.ndarray, Counter[str]]: + """Sample reproducible math-protocol rates/patterns per training example.""" + rows: list[np.ndarray] = [] + counts: Counter[str] = Counter() + for sample_id, observed in zip(split.ids, split.mask): + rng = np.random.default_rng(scenario_seed(TRAIN_MASK_SEED + SEED, sample_id, f"train/{epoch}")) + rate = float(rng.choice(TRAIN_RATES)) + mode = str(rng.choice(TRAIN_MODES)) + key = f"{rate:.1f}/{mode}" + counts[key] += 1 + row = continuous_mask(observed, rate, mode, rng) + rows.append(row) + return np.stack(rows), counts + + +def _batched_loss( + model: nn.Module, + split: Split, + masks: np.ndarray, + device: torch.device, + batch_size: int, +) -> float: + model.eval() + losses: list[float] = [] + weights: list[int] = [] + with torch.inference_mode(): + for start in range(0, split.n, batch_size): + end = min(start + batch_size, split.n) + xs = tuple(torch.as_tensor(x[start:end], dtype=torch.float32, device=device) for x in split.x) + mb = torch.as_tensor(masks[start:end], dtype=torch.bool, device=device) + y_cls = torch.as_tensor(split.y_cls[start:end], dtype=torch.long, device=device) + y_reg = torch.as_tensor(split.y_reg[start:end], dtype=torch.float32, device=device) + losses.append(float(_loss(model(xs, mb), y_cls, y_reg).item())) + weights.append(end - start) + return float(np.average(losses, weights=weights)) + + +def selection_loss(model: nn.Module, valid: Split, scenarios: dict[str, np.ndarray], device: torch.device) -> float: + return float(np.mean([ + _batched_loss(model, valid, scenarios[key], device, BATCH_SIZE) + for key in SELECTION_SCENARIOS + ])) + + +def train_one( + method: str, + train: Split, + valid: Split, + valid_scenarios: dict[str, np.ndarray], + orders: list[np.ndarray], + output_dir: Path, + device: torch.device, +) -> tuple[nn.Module, int, list[dict[str, Any]], Counter[str]]: + set_deterministic(SEED) + model = build_model(method, tuple(x.shape[-1] for x in train.x), device) + optimizer = torch.optim.AdamW(model.parameters(), lr=LEARNING_RATE, weight_decay=WEIGHT_DECAY) + xs = tuple(torch.as_tensor(x, dtype=torch.float32, device=device) for x in train.x) + y_cls = torch.as_tensor(train.y_cls, dtype=torch.long, device=device) + y_reg = torch.as_tensor(train.y_reg, dtype=torch.float32, device=device) + checkpoint_path = output_dir / "model_best.pt" + history: list[dict[str, Any]] = [] + train_mask_counts: Counter[str] = Counter() + best_loss = math.inf + best_epoch = 0 + stale = 0 + + for epoch in range(1, EPOCH_LIMIT + 1): + model.train() + epoch_masks, epoch_counts = train_masks_for_epoch(train, epoch) + train_mask_counts.update(epoch_counts) + batch_losses: list[float] = [] + order = orders[epoch - 1] + for start in range(0, train.n, BATCH_SIZE): + indices_np = order[start:start + BATCH_SIZE] + indices = torch.as_tensor(indices_np, dtype=torch.long, device=device) + mb = torch.as_tensor(epoch_masks[indices_np], dtype=torch.bool, device=device) + output = model(tuple(x.index_select(0, indices) for x in xs), mb) + loss = _loss(output, y_cls.index_select(0, indices), y_reg.index_select(0, indices)) + optimizer.zero_grad(set_to_none=True) + loss.backward() + nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) + optimizer.step() + batch_losses.append(float(loss.detach().item())) + + valid_selection_loss = selection_loss(model, valid, valid_scenarios, device) + row = { + "method": method, + "seed": SEED, + "epoch": epoch, + "train_loss": float(np.mean(batch_losses)), + "valid_selection_loss": valid_selection_loss, + "valid_clean_loss": _batched_loss(model, valid, valid.mask, device, BATCH_SIZE), + } + history.append(row) + print( + f"[{method}] epoch={epoch:02d} train={row['train_loss']:.4f} " + f"valid_selection={valid_selection_loss:.4f} clean={row['valid_clean_loss']:.4f}", + flush=True, + ) + if valid_selection_loss < best_loss - 1e-4: + best_loss = valid_selection_loss + best_epoch = epoch + stale = 0 + torch.save(model_state(model, method) | {"best_epoch": best_epoch}, checkpoint_path) + else: + stale += 1 + if stale >= PATIENCE: + break + + saved = torch.load(checkpoint_path, map_location=device, weights_only=False) + model.load_state_dict(saved["state_dict"]) + model.eval() + write_csv(output_dir / "training_history.csv", history) + return model, best_epoch, history, train_mask_counts + + +def _group_map(ids: list[str]) -> tuple[list[str], dict[str, np.ndarray]]: + source_ids = [sample_id.split("$_$", 1)[0] for sample_id in ids] + groups = sorted(set(source_ids)) + mapping = { + group: np.flatnonzero(np.asarray([source == group for source in source_ids])) + for group in groups + } + return groups, mapping + + +def test_group_bootstrap(test: Split, predictions: dict[str, dict[str, np.ndarray]]) -> list[dict[str, Any]]: + groups, mapping = _group_map(test.ids) + rng = np.random.default_rng(TEST_BOOTSTRAP_SEED) + draws: dict[str, list[float]] = defaultdict(list) + for _ in range(BOOTSTRAP_REPS): + selected = rng.choice(groups, size=len(groups), replace=True) + indices = np.concatenate([mapping[group] for group in selected]) + values = { + method: metrics(test, predictions[method]["logits"], predictions[method]["intensity"], indices) + for method in METHODS + } + for name in values[EARLYCONCAT]: + draws[name].append(values[MOFE7_MLP][name] - values[EARLYCONCAT][name]) + point = { + name: metrics(test, predictions[MOFE7_MLP]["logits"], predictions[MOFE7_MLP]["intensity"])[name] + - metrics(test, predictions[EARLYCONCAT]["logits"], predictions[EARLYCONCAT]["intensity"])[name] + for name in draws + } + return [{ + "comparison": f"{MOFE7_MLP} minus {EARLYCONCAT}", + "metric": name, + "delta": point[name], + "bootstrap_ci_2p5": float(np.quantile(values, 0.025)), + "bootstrap_ci_97p5": float(np.quantile(values, 0.975)), + "bootstrap_probability_delta_gt_0": float(np.mean(np.asarray(values) > 0)), + "replicates": BOOTSTRAP_REPS, + "resampling_unit": "source video id", + "paired": True, + "seed": TEST_BOOTSTRAP_SEED, + } for name, values in draws.items()] + + +def validation_aurc_bootstrap( + valid: Split, + predictions: dict[tuple[str, str], dict[str, np.ndarray]], + rates_by_sample: dict[str, np.ndarray], +) -> list[dict[str, Any]]: + groups, mapping = _group_map(valid.ids) + rng = np.random.default_rng(AURC_BOOTSTRAP_SEED) + deltas: dict[str, list[float]] = {mode: [] for mode in CURVE_MODES} + + def score(method: str, mode: str, indices: np.ndarray) -> float: + keys = curve_scenarios(mode) + xs = [float(np.nanmean(rates_by_sample[key][indices])) for key in keys] + ys = [ + float(np.abs(valid.y_reg[indices] - predictions[(method, key)]["intensity"][indices]).mean()) + for key in keys + ] + return aurc_from_curve(xs, ys) + + for _ in range(BOOTSTRAP_REPS): + selected = rng.choice(groups, size=len(groups), replace=True) + indices = np.concatenate([mapping[group] for group in selected]) + for mode in CURVE_MODES: + deltas[mode].append(score(MOFE7_MLP, mode, indices) - score(EARLYCONCAT, mode, indices)) + rows = [] + for mode in CURVE_MODES: + all_indices = np.arange(valid.n) + values = deltas[mode] + rows.append({ + "mask_mode": mode, + "delta_aurc_mae_mofe_minus_earlyconcat": score(MOFE7_MLP, mode, all_indices) - score(EARLYCONCAT, mode, all_indices), + "bootstrap_ci_2p5": float(np.quantile(values, 0.025)), + "bootstrap_ci_97p5": float(np.quantile(values, 0.975)), + "bootstrap_probability_delta_lt_0": float(np.mean(np.asarray(values) < 0)), + "replicates": BOOTSTRAP_REPS, + "resampling_unit": "source video id", + "paired": True, + "seed": AURC_BOOTSTRAP_SEED, + }) + return rows + + +def run(device_name: str = "auto", output_dir: Path = OUTPUT_DIR) -> None: + if output_dir.exists() and any(output_dir.iterdir()): + raise FileExistsError(f"refusing to overwrite non-empty result directory: {output_dir}") + output_dir.mkdir(parents=True, exist_ok=True) + device = device_for(device_name) + if device.type == "cuda" and not torch.cuda.is_available(): + raise RuntimeError("CUDA was requested but is unavailable") + + feature_path = ATTACHMENT2 / "aligned_50.pkl" + raw_splits = load_splits(feature_path) + train_raw, valid_raw, test_raw = raw_splits["train"], raw_splits["valid"], raw_splits["test"] + stats = fit_robust_stats(train_raw) + train, valid, test = (apply_robust_stats(s, stats) for s in (train_raw, valid_raw, test_raw)) + stats_path = output_dir / "aligned_robust_stats.npz" + stats.save(stats_path) + dims = tuple(int(x.shape[-1]) for x in train.x) + valid_scenarios = make_scenarios(valid, SCENARIO_SEED) + if len(valid_scenarios) != 42: + raise ValueError(f"expected 42 controlled scenarios, got {len(valid_scenarios)}") + rates_by_sample = actual_additional_rates(valid.mask, valid_scenarios) + + set_deterministic(SEED) + order_rng = np.random.default_rng(SEED + 809) + orders = [order_rng.permutation(train.n) for _ in range(EPOCH_LIMIT)] + best_epochs: dict[str, int] = {} + training_rows: list[dict[str, Any]] = [] + mask_count_rows: list[dict[str, Any]] = [] + parameter_rows: list[dict[str, Any]] = [] + + for method in METHODS: + model_dir = output_dir / "models" / method / f"seed_{SEED}" + model_dir.mkdir(parents=True, exist_ok=True) + model, best_epoch, history, mask_counts = train_one( + method, train, valid, valid_scenarios, orders, model_dir, device + ) + best_epochs[method] = best_epoch + training_rows.extend(history) + parameter_rows.append({ + "method": method, + "parameters_total": sum(p.numel() for p in model.parameters()), + "parameters_trainable": sum(p.numel() for p in model.parameters() if p.requires_grad), + "best_epoch": best_epoch, + }) + for key, count in sorted(mask_counts.items()): + mask_count_rows.append({"method": method, "seed": SEED, "rate_mode": key, "sample_epoch_assignments": count}) + del model + if torch.cuda.is_available(): + torch.cuda.empty_cache() + write_csv(output_dir / "training_history.csv", training_rows) + write_csv(output_dir / "training_mask_distribution.csv", mask_count_rows) + write_csv(output_dir / "parameter_count.csv", parameter_rows) + + # Reload the selected checkpoints, then conduct one final official-test pass. + test_predictions: dict[str, dict[str, np.ndarray]] = {} + test_rows: list[dict[str, Any]] = [] + condition_predictions: dict[tuple[str, str], dict[str, np.ndarray]] = {} + condition_rows: list[dict[str, Any]] = [] + for method in METHODS: + checkpoint_path = output_dir / "models" / method / f"seed_{SEED}" / "model_best.pt" + saved = torch.load(checkpoint_path, map_location=device, weights_only=False) + model = build_model(method, dims, device) + model.load_state_dict(saved["state_dict"]) + model.eval() + + test_prediction = _predict(model, test, test.mask, device, BATCH_SIZE) + test_predictions[method] = test_prediction + test_rows.append({ + "method": method, + "seed": SEED, + "best_epoch": best_epochs[method], + "n_test": test.n, + **metrics(test, test_prediction["logits"], test_prediction["intensity"]), + }) + + for scenario, masks in valid_scenarios.items(): + prediction = _predict(model, valid, masks, device, BATCH_SIZE) + condition_predictions[(method, scenario)] = prediction + condition_rows.append({ + "method": method, + "seed": SEED, + "scenario": scenario, + "realized_additional_global_rate": float(np.nanmean(rates_by_sample[scenario])), + "n_valid": valid.n, + **metrics(valid, prediction["logits"], prediction["intensity"]), + }) + print(f"[valid/{method}] {scenario} done", flush=True) + del model + if torch.cuda.is_available(): + torch.cuda.empty_cache() + + write_csv(output_dir / "official_test_metrics_by_seed.csv", test_rows) + write_csv(output_dir / "official_test_paired_bootstrap.csv", test_group_bootstrap(test, test_predictions)) + write_csv(output_dir / "controlled_metrics_by_scenario.csv", condition_rows) + + test_summary = [] + for method in METHODS: + row = next(r for r in test_rows if r["method"] == method) + for metric in ("accuracy", "macro_f1", "mae", "rmse", "pearson"): + test_summary.append({"method": method, "metric": metric, "mean": row[metric], "sd_across_seeds": 0.0, "n_seeds": 1}) + write_csv(output_dir / "official_test_summary.csv", test_summary) + + aurc_rows: list[dict[str, Any]] = [] + for method in METHODS: + for mode in CURVE_MODES: + keys = curve_scenarios(mode) + xs = [float(np.nanmean(rates_by_sample[key])) for key in keys] + ys = [ + float(np.abs(valid.y_reg - condition_predictions[(method, key)]["intensity"]).mean()) + for key in keys + ] + aurc_rows.append({ + "method": method, + "seed": SEED, + "mask_mode": mode, + "aurc_mae": aurc_from_curve(xs, ys), + "rates_realized": json.dumps(xs), + }) + write_csv(output_dir / "aurc_mae_by_mode_seed.csv", aurc_rows) + write_csv(output_dir / "aurc_mae_paired_bootstrap.csv", validation_aurc_bootstrap(valid, condition_predictions, rates_by_sample)) + + manifest = { + "experiment": "Retrained EarlyConcat and MoFE-7 + MLP Router using math/Q2 V2-compatible protocol", + "created_unix": time.time(), + "device": str(device), + "cuda_device": torch.cuda.get_device_name(0) if device.type == "cuda" else None, + "feature_file": str(feature_path), + "feature_sha256": sha256(feature_path), + "representation": "official aligned_50 ordered positions; not Q1 physical-time bins", + "train_valid_test_counts": {name: split.n for name, split in raw_splits.items()}, + "source_video_groups": {name: len({sid.split("$_$", 1)[0] for sid in split.ids}) for name, split in raw_splits.items()}, + "official_group_splits_disjoint": True, + "train_only_scaler": str(stats_path), + "scaler_fit": "median and 1.4826*MAD on observed training rows only; zero-MAD fallback to std then 1", + "seed": SEED, + "model_seeds": [SEED], + "training_configuration": { + "epoch_limit": EPOCH_LIMIT, + "early_stopping_patience": PATIENCE, + "batch_size": BATCH_SIZE, + "optimizer": "AdamW", + "learning_rate": LEARNING_RATE, + "weight_decay": WEIGHT_DECAY, + "gradient_clip_norm": 1.0, + "early_stopping_metric": "mean validation joint CE + 0.5*SmoothL1 over 0.0/none, 0.3/single, 0.3/sync, 0.5/async", + "architecture_preserved": { + EARLYCONCAT: "EarlyConcat + BiGRU", + MOFE7_MLP: "MoFE-7 + MLP Router", + }, + "objective": "cross entropy + 0.5 * SmoothL1(intensity/3, label/3); same objective for both methods", + "training_corruption": { + "rates": list(TRAIN_RATES), + "patterns": list(TRAIN_MODES), + "preserve_at_least_fraction_per_selected_modality": 0.2, + "generator_seed": TRAIN_MASK_SEED, + "same_sample_masks_and_batch_orders_across_models": True, + }, + }, + "validation_protocol": { + "scenario_seed": SCENARIO_SEED, + "scenario_count": len(valid_scenarios), + "same_fixed_masks_for_both_models": True, + "scenario_design": "math/Q2 42 controlled continuous-mask scenarios regenerated on each sample's original observation mask", + "selection_scenarios": list(SELECTION_SCENARIOS), + "selection_note": "Deterministic-model adaptation; uses joint supervised loss instead of C5's probabilistic selection NLL.", + "aurc": "normalized trapezoidal MAE area over realized equal-modality-weighted additional missing rate for single/sync/partial/async at 0/.1/.3/.5/.7", + }, + "test_protocol": { + "official_test_final_clean_passes": 1, + "test_used_for_training_or_checkpoint_selection": False, + "metrics": ["accuracy", "macro_f1", "mae", "rmse", "pearson"], + "paired_group_bootstrap_replicates": BOOTSTRAP_REPS, + "bootstrap_unit": "source video id", + "bootstrap_seed": TEST_BOOTSTRAP_SEED, + }, + } + (output_dir / "run_manifest.json").write_text(json.dumps(manifest, indent=2), encoding="utf-8") + (output_dir / "hypothesis.md").write_text( + "# R03: 按 math/Q2 V2 口径重训两种保留模型\n\n" + "## 假设\n\n" + "在保持 EarlyConcat + BiGRU 与 MoFE-7 + MLP Router 结构及共同监督目标不变的情况下," + "使用数学方案中的官方划分、连续块缺失训练和 42 个固定验证情景,可以公平比较两种模型的干净测试表现与缺失鲁棒性。\n\n" + "## 唯一实验改动\n\n" + "相对现有检查点,本轮重新训练时将缺失训练改为 0/10/30/50/70% 与 single/sync/partial/async," + "每个被选模态至少保留 20% 观测;训练和批次顺序在两个模型间配对。数学方案中的 C5 概率损失不适用于现有确定性分类/回归头," + "因此保留项目既有的 CE + 0.5 SmoothL1 联合目标。\n\n" + "## 数据使用\n\n" + "标准化器只在官方训练集观测行上拟合;官方验证集只用于早停与缺失评估;官方测试集在全部检查点确定后做一次干净评估。\n", + encoding="utf-8", + ) + + print(f"wrote retraining results to {output_dir}", flush=True) + print(f"train/valid/test={train.n}/{valid.n}/{test.n}; device={device}; best_epochs={best_epochs}", flush=True) + for row in test_rows: + print( + f"{row['method']}: Acc={row['accuracy']:.4f} Macro-F1={row['macro_f1']:.4f} " + f"MAE={row['mae']:.4f} RMSE={row['rmse']:.4f} Pearson={row['pearson']:.4f}", + flush=True, + ) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--device", default="auto", choices=("auto", "cuda", "cpu")) + parser.add_argument("--output-dir", type=Path, default=OUTPUT_DIR) + arguments = parser.parse_args() + run(device_name=arguments.device, output_dir=arguments.output_dir) diff --git a/deep_learning/Q2/q2/train_mofe.py b/deep_learning/Q2/q2/train_mofe.py new file mode 100644 index 0000000..a22b6d4 --- /dev/null +++ b/deep_learning/Q2/q2/train_mofe.py @@ -0,0 +1,787 @@ +from __future__ import annotations + +import argparse +import csv +import hashlib +import json +import math +import random +import sys +import time +from pathlib import Path +from typing import Any + +import numpy as np +import torch +from sklearn.metrics import accuracy_score, f1_score, mean_absolute_error +from torch import nn + +from .data import ( + ATTACHMENT2, + MODALITIES, + RobustStats, + Split, + apply_robust_stats, + augment_masks, + corrupt_masks, + fit_robust_stats, + load_aligned, +) +from .models import AlignedFusionModel +from .mofe import EXPERT_NAMES, SUBSETS, MixtureOfFusionExperts +from .train_compare import PATTERNS, _loss, _pearson, _train_one, seed_everything + + +ROOT = Path(__file__).resolve().parents[1] +REFERENCE_OUTPUT = ROOT / "outputs" / "mofe_7experts" +DEFAULT_OUTPUT = ROOT / "outputs" / "mofe_7experts" +EARLYCONCAT = "B0_early_concat" +MOFE7_MLP = "B5_mofe_mlp" +SEEDS = (42, 3407, 2026) +RATES = (0.10, 0.20, 0.30) +HIDDEN = 128 +LATENT_DIM = 64 +MODEL_CONFIG: dict[str, Any] = { + "router": "mlp", + "expert_names": EXPERT_NAMES, + "availability_mode": "hard", +} +SUMMARY_METRICS = ( + "corrupt_macro_f1", + "worst_condition_macro_f1", + "text_30_macro_f1", + "corrupt_mae", + "corrupt_pearson", +) + + +def _write_csv(path: Path, rows: list[dict[str, Any]]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + if not rows: + return + fields = list(dict.fromkeys(key for row in rows for key in row)) + with path.open("w", newline="", encoding="utf-8-sig") as stream: + writer = csv.DictWriter(stream, fieldnames=fields) + writer.writeheader() + writer.writerows(rows) + + +def _read_csv(path: Path) -> list[dict[str, str]]: + if not path.exists(): + return [] + with path.open("r", newline="", encoding="utf-8-sig") as stream: + return list(csv.DictReader(stream)) + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return digest.hexdigest() + + +def _device_for(name: str) -> torch.device: + if name == "auto": + return torch.device("cuda" if torch.cuda.is_available() else "cpu") + return torch.device(name) + + +def _conditions(valid: Split, seed: int) -> list[tuple[str, float, np.ndarray]]: + rows = [("clean", 0.0, valid.mask.copy())] + for rate in RATES: + for pattern_idx, (pattern, modalities) in enumerate(PATTERNS.items()): + masks = corrupt_masks( + valid.mask, + rate, + modalities, + seed + 13 + pattern_idx * 101 + int(rate * 1000), + ) + rows.append((f"{pattern}_{int(rate * 100)}", rate, masks)) + return rows + + +def _metric_dict( + y_cls: np.ndarray, + y_reg: np.ndarray, + logits: np.ndarray, + intensity: np.ndarray, +) -> dict[str, float]: + predicted_class = np.asarray(logits).argmax(axis=-1) + predicted_intensity = np.clip(np.asarray(intensity).reshape(-1), -3.0, 3.0) + return { + "accuracy": float(accuracy_score(y_cls, predicted_class)), + "macro_f1": float(f1_score(y_cls, predicted_class, labels=[0, 1, 2], average="macro", zero_division=0)), + "mae": float(mean_absolute_error(y_reg, predicted_intensity)), + "pearson": _pearson(y_reg, predicted_intensity), + } + + +def _validation_loss(model: nn.Module, valid: Split, device: torch.device, batch_size: int) -> float: + model.eval() + values: list[float] = [] + weights: list[int] = [] + with torch.inference_mode(): + for start in range(0, valid.n, batch_size): + end = min(start + batch_size, valid.n) + xs = tuple(torch.as_tensor(x[start:end], dtype=torch.float32, device=device) for x in valid.x) + masks = torch.as_tensor(valid.mask[start:end], dtype=torch.bool, device=device) + y_cls = torch.as_tensor(valid.y_cls[start:end], dtype=torch.long, device=device) + y_reg = torch.as_tensor(valid.y_reg[start:end], dtype=torch.float32, device=device) + values.append(float(_loss(model(xs, masks), y_cls, y_reg).item())) + weights.append(end - start) + return float(np.average(values, weights=weights)) + + +def _train_mofe( + train: Split, + valid: Split, + output_dir: Path, + device: torch.device, + seed: int, + epochs: int, + patience: int, + batch_size: int, + reuse_checkpoint: bool, +) -> tuple[MixtureOfFusionExperts, int, list[dict[str, Any]]]: + dims = tuple(int(x.shape[-1]) for x in train.x) + checkpoint_path = output_dir / "model_best.pt" + history_path = output_dir / "training_history.csv" + if reuse_checkpoint and checkpoint_path.exists(): + saved = torch.load(checkpoint_path, map_location=device, weights_only=False) + if saved.get("config") != MODEL_CONFIG or tuple(saved.get("dims", ())) != dims or int(saved.get("seed", -1)) != seed: + raise ValueError(f"cached MoFE checkpoint does not match the selected configuration: {checkpoint_path}") + model = MixtureOfFusionExperts(dims=dims, **MODEL_CONFIG).to(device) + model.load_state_dict(saved["state_dict"]) + history = [ + {"method": MOFE7_MLP, "seed": seed, **{key: float(value) for key, value in row.items() if key in {"epoch", "train_loss", "valid_clean_loss"}}} + for row in _read_csv(history_path) + ] + return model.eval(), int(saved.get("best_epoch", 0)), history + + output_dir.mkdir(parents=True, exist_ok=True) + seed_everything(seed) + model = MixtureOfFusionExperts(dims=dims, **MODEL_CONFIG).to(device) + optimizer = torch.optim.AdamW(model.parameters(), lr=1.5e-4, weight_decay=1e-4) + xs = tuple(torch.as_tensor(x, dtype=torch.float32, device=device) for x in train.x) + base_masks = train.mask + y_cls = torch.as_tensor(train.y_cls, dtype=torch.long, device=device) + y_reg = torch.as_tensor(train.y_reg, dtype=torch.float32, device=device) + rng = np.random.default_rng(seed + 809) + best_loss = math.inf + best_epoch = 0 + stale_epochs = 0 + history: list[dict[str, Any]] = [] + + for epoch in range(1, epochs + 1): + model.train() + order = rng.permutation(train.n) + batch_losses: list[float] = [] + for start in range(0, train.n, batch_size): + ids_np = order[start:start + batch_size] + ids = torch.as_tensor(ids_np, dtype=torch.long, device=device) + masks_np = augment_masks(base_masks[ids_np], rng) + masks = torch.as_tensor(masks_np, dtype=torch.bool, device=device) + output = model(tuple(x.index_select(0, ids) for x in xs), masks) + loss = _loss(output, y_cls.index_select(0, ids), y_reg.index_select(0, ids)) + optimizer.zero_grad(set_to_none=True) + loss.backward() + nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) + optimizer.step() + batch_losses.append(float(loss.detach().item())) + + valid_loss = _validation_loss(model, valid, device, batch_size) + row = { + "method": MOFE7_MLP, + "seed": seed, + "epoch": epoch, + "train_loss": float(np.mean(batch_losses)), + "valid_clean_loss": valid_loss, + } + history.append(row) + print(f"[MoFE-7 MLP] seed={seed} epoch={epoch:02d} train={row['train_loss']:.4f} valid={valid_loss:.4f}", flush=True) + if valid_loss < best_loss - 1e-4: + best_loss = valid_loss + best_epoch = epoch + stale_epochs = 0 + torch.save({ + "method": MOFE7_MLP, + "config": MODEL_CONFIG, + "dims": dims, + "state_dict": model.state_dict(), + "seed": seed, + "best_epoch": epoch, + }, checkpoint_path) + else: + stale_epochs += 1 + if stale_epochs >= patience: + break + + saved = torch.load(checkpoint_path, map_location=device, weights_only=False) + model.load_state_dict(saved["state_dict"]) + model.eval() + _write_csv(history_path, history) + return model, best_epoch, history + + +def _load_or_train_concat( + train: Split, + valid: Split, + output_dir: Path, + device: torch.device, + seed: int, + epochs: int, + patience: int, + batch_size: int, + reuse_checkpoint: bool, +) -> tuple[AlignedFusionModel, int, list[dict[str, Any]]]: + checkpoint_path = output_dir / "model_best.pt" + dims = tuple(int(x.shape[-1]) for x in train.x) + if reuse_checkpoint and checkpoint_path.exists(): + saved = torch.load(checkpoint_path, map_location=device, weights_only=False) + if saved.get("kind") != "concat" or tuple(saved.get("dims", ())) != dims or int(saved.get("seed", -1)) != seed: + raise ValueError(f"cached EarlyConcat checkpoint does not match: {checkpoint_path}") + model = AlignedFusionModel("concat", dims=dims).to(device) + model.load_state_dict(saved["state_dict"]) + history = [ + {"method": EARLYCONCAT, "seed": seed, **{key: float(value) for key, value in row.items() if key in {"epoch", "train_loss", "valid_clean_loss"}}} + for row in _read_csv(output_dir / "training_history.csv") + ] + return model.eval(), int(saved.get("best_epoch", 0)), history + + model, best_epoch, history = _train_one( + "concat", train, valid, output_dir, device, seed, epochs, patience, batch_size + ) + rows = [{"method": EARLYCONCAT, "seed": seed, **row} for row in history] + return model.eval(), best_epoch, rows + + +@torch.inference_mode() +def _predict( + model: nn.Module, + split: Split, + masks: np.ndarray, + device: torch.device, + batch_size: int, + force_expert: str | None = None, +) -> dict[str, np.ndarray]: + fields = ["logits", "intensity"] + if isinstance(model, MixtureOfFusionExperts): + fields.extend(("alpha", "utility", "availability", "fallback")) + chunks: dict[str, list[np.ndarray]] = {name: [] for name in fields} + for start in range(0, split.n, batch_size): + end = min(start + batch_size, split.n) + xs = tuple(torch.as_tensor(x[start:end], dtype=torch.float32, device=device) for x in split.x) + mask_batch = torch.as_tensor(masks[start:end], dtype=torch.bool, device=device) + output = model(xs, mask_batch, force_expert=force_expert) if isinstance(model, MixtureOfFusionExperts) else model(xs, mask_batch) + for name in fields: + value = output[name] + chunks[name].append(value.float().cpu().numpy()) + result = {name: np.concatenate(values, axis=0) for name, values in chunks.items()} + result["intensity"] = np.clip(result["intensity"].reshape(-1), -3.0, 3.0) + return result + + +def _condition_row( + method: str, + seed: int, + condition: str, + rate: float, + split: Split, + prediction: dict[str, np.ndarray], +) -> dict[str, Any]: + return { + "method": method, + "seed": seed, + "condition": condition, + "missing_rate": rate, + "n_valid": split.n, + **_metric_dict(split.y_cls, split.y_reg, prediction["logits"], prediction["intensity"]), + } + + +def _diagnostics( + seed: int, + condition: str, + masks: np.ndarray, + prediction: dict[str, np.ndarray], +) -> tuple[dict[str, Any], dict[str, Any]]: + alpha = prediction["alpha"] + availability = prediction["availability"].astype(bool) + active = availability.any(axis=-1) + active_alpha = alpha[active] + if active_alpha.size: + means = active_alpha.mean(axis=0) + entropy = -(active_alpha * np.log(np.maximum(active_alpha, 1e-12))).sum(axis=-1) / np.log(len(EXPERT_NAMES)) + high_weight = (active_alpha.max(axis=-1) > 0.8).mean() + else: + means = np.zeros(len(EXPERT_NAMES), dtype=np.float64) + entropy = np.zeros(0, dtype=np.float64) + high_weight = 0.0 + route_row: dict[str, Any] = { + "method": MOFE7_MLP, + "seed": seed, + "condition": condition, + "active_position_fraction": float(active.mean()), + "fallback_position_fraction": float((~active).mean()), + "normalized_router_entropy": float(entropy.mean()) if entropy.size else 0.0, + "fraction_active_positions_max_weight_over_0p8": float(high_weight), + } + for index, name in enumerate(EXPERT_NAMES): + route_row[f"alpha_{name}_mean"] = float(means[index]) + utility = prediction["utility"] + utility_row: dict[str, Any] = {"method": MOFE7_MLP, "seed": seed, "condition": condition} + for modality, name in enumerate(MODALITIES): + observed = masks[..., modality] + utility_row[f"utility_{name}_mean"] = float(utility[..., modality][observed].mean()) if observed.any() else 0.0 + return route_row, utility_row + + +def _summary_rows(rows: list[dict[str, Any]]) -> list[dict[str, Any]]: + summaries: list[dict[str, Any]] = [] + for method in (EARLYCONCAT, MOFE7_MLP): + matching = [row for row in rows if row["method"] == method] + seeds = sorted({int(row["seed"]) for row in matching}) + conditions = list(dict.fromkeys(row["condition"] for row in matching)) + by_seed_condition = {(int(row["seed"]), row["condition"]): row for row in matching} + clean = [by_seed_condition[(seed, "clean")] for seed in seeds] + corrupt_conditions = [condition for condition in conditions if condition != "clean"] + corrupt_by_seed = { + seed: [by_seed_condition[(seed, condition)] for condition in corrupt_conditions] + for seed in seeds + } + condition_f1 = { + condition: float(np.mean([by_seed_condition[(seed, condition)]["macro_f1"] for seed in seeds])) + for condition in corrupt_conditions + } + worst_condition = min(condition_f1, key=condition_f1.get) + row: dict[str, Any] = {"method": method, "n_seeds": len(seeds), "worst_condition": worst_condition} + for metric in ("accuracy", "macro_f1", "mae", "pearson"): + clean_values = [float(item[metric]) for item in clean] + corrupt_values = [float(np.mean([item[metric] for item in corrupt_by_seed[seed]])) for seed in seeds] + row[f"clean_{metric}"] = float(np.mean(clean_values)) + row[f"clean_{metric}_sd"] = float(np.std(clean_values, ddof=1)) if len(clean_values) > 1 else 0.0 + row[f"corrupt_{metric}_mean"] = float(np.mean(corrupt_values)) + row[f"corrupt_{metric}_sd"] = float(np.std(corrupt_values, ddof=1)) if len(corrupt_values) > 1 else 0.0 + row["worst_condition_macro_f1"] = condition_f1[worst_condition] + row["worst_single_run_macro_f1"] = min( + item["macro_f1"] for seed in seeds for item in corrupt_by_seed[seed] + ) + text_30 = [by_seed_condition[(seed, "text_30")] for seed in seeds] + row["text_30_macro_f1"] = float(np.mean([item["macro_f1"] for item in text_30])) + row["text_30_macro_f1_sd"] = float(np.std([item["macro_f1"] for item in text_30], ddof=1)) if len(text_30) > 1 else 0.0 + for condition in ("audio_30", "vision_30", "audio_vision_30", "all_modalities_30"): + values = [by_seed_condition[(seed, condition)]["macro_f1"] for seed in seeds] + row[f"{condition}_macro_f1"] = float(np.mean(values)) + row[f"{condition}_macro_f1_sd"] = float(np.std(values, ddof=1)) if len(values) > 1 else 0.0 + row["corrupt_macro_f1"] = row["corrupt_macro_f1_mean"] + row["corrupt_mae"] = row["corrupt_mae_mean"] + row["corrupt_pearson"] = row["corrupt_pearson_mean"] + summaries.append(row) + return summaries + + +def _bootstrap_distributions( + method: str, + predictions: dict[tuple[str, int, str], dict[str, np.ndarray]], + valid: Split, + seeds: list[int], + conditions: list[str], + group_counts: np.ndarray, +) -> dict[str, np.ndarray]: + group_names = sorted({sample_id.split("$_$", 1)[0] for sample_id in valid.ids}) + group_index = {name: index for index, name in enumerate(group_names)} + row_group = np.asarray([group_index[sample_id.split("$_$", 1)[0]] for sample_id in valid.ids], dtype=np.int64) + n_groups = len(group_names) + n_slots = len(seeds) * len(conditions) + confusion_by_group = np.zeros((n_groups, n_slots, 9), dtype=np.float64) + regression_by_group = np.zeros((n_groups, n_slots, 7), dtype=np.float64) + for seed_index, seed in enumerate(seeds): + for condition_index, condition in enumerate(conditions): + slot = seed_index * len(conditions) + condition_index + pred = predictions[(method, seed, condition)] + predicted_class = pred["logits"].argmax(axis=-1) + code = valid.y_cls * 3 + predicted_class + np.add.at(confusion_by_group[:, slot, :], (row_group, code), 1.0) + intensity = np.clip(pred["intensity"].reshape(-1), -3.0, 3.0) + values = np.stack(( + np.ones(valid.n), + np.abs(valid.y_reg - intensity), + valid.y_reg, + valid.y_reg ** 2, + intensity, + intensity ** 2, + valid.y_reg * intensity, + ), axis=-1) + for statistic in range(values.shape[-1]): + np.add.at(regression_by_group[:, slot, statistic], row_group, values[:, statistic]) + + weighted_confusion = np.einsum("rg,gsk->rsk", group_counts, confusion_by_group, optimize=True) + cm = weighted_confusion.reshape(len(group_counts), len(seeds), len(conditions), 3, 3) + true_count = cm.sum(axis=-1) + predicted_count = cm.sum(axis=-2) + true_positive = np.diagonal(cm, axis1=-2, axis2=-1) + denominator = true_count + predicted_count + class_f1 = np.divide(2.0 * true_positive, denominator, out=np.zeros_like(true_positive), where=denominator > 0) + macro_f1 = class_f1.mean(axis=-1) + + weighted_regression = np.einsum("rg,gsk->rsk", group_counts, regression_by_group, optimize=True) + regression = weighted_regression.reshape(len(group_counts), len(seeds), len(conditions), 7) + count = np.maximum(regression[..., 0], 1.0) + mae = regression[..., 1] / count + sum_y, sum_y2, sum_pred, sum_pred2, sum_yp = (regression[..., index] for index in range(2, 7)) + covariance = sum_yp - sum_y * sum_pred / count + variance_y = np.maximum(sum_y2 - sum_y ** 2 / count, 0.0) + variance_pred = np.maximum(sum_pred2 - sum_pred ** 2 / count, 0.0) + denominator_corr = np.sqrt(variance_y * variance_pred) + pearson = np.divide(covariance, denominator_corr, out=np.zeros_like(covariance), where=denominator_corr > 1e-12) + text_30_index = conditions.index("text_30") + return { + "corrupt_macro_f1": macro_f1[:, :, 1:].mean(axis=(1, 2)), + "worst_condition_macro_f1": macro_f1[:, :, 1:].mean(axis=1).min(axis=1), + "text_30_macro_f1": macro_f1[:, :, text_30_index].mean(axis=1), + "corrupt_mae": mae[:, :, 1:].mean(axis=(1, 2)), + "corrupt_pearson": pearson[:, :, 1:].mean(axis=(1, 2)), + } + + +def _paired_bootstrap( + predictions: dict[tuple[str, int, str], dict[str, np.ndarray]], + valid: Split, + seeds: list[int], + conditions: list[str], + reps: int, + bootstrap_seed: int, + summaries: list[dict[str, Any]], +) -> list[dict[str, Any]]: + groups = sorted({sample_id.split("$_$", 1)[0] for sample_id in valid.ids}) + rng = np.random.default_rng(bootstrap_seed) + draws = rng.integers(0, len(groups), size=(reps, len(groups))) + group_counts = np.zeros((reps, len(groups)), dtype=np.float64) + for rep in range(reps): + group_counts[rep] = np.bincount(draws[rep], minlength=len(groups)) + candidate = _bootstrap_distributions(MOFE7_MLP, predictions, valid, seeds, conditions, group_counts) + reference = _bootstrap_distributions(EARLYCONCAT, predictions, valid, seeds, conditions, group_counts) + summary_map = {row["method"]: row for row in summaries} + point_keys = { + "corrupt_macro_f1": "corrupt_macro_f1", + "worst_condition_macro_f1": "worst_condition_macro_f1", + "text_30_macro_f1": "text_30_macro_f1", + "corrupt_mae": "corrupt_mae", + "corrupt_pearson": "corrupt_pearson", + } + rows = [] + for metric in SUMMARY_METRICS: + delta = candidate[metric] - reference[metric] + key = point_keys[metric] + rows.append({ + "comparison": "MoFE-7 MLP vs EarlyConcat", + "candidate": MOFE7_MLP, + "reference": EARLYCONCAT, + "metric": metric, + "delta_candidate_minus_reference": float(summary_map[MOFE7_MLP][key] - summary_map[EARLYCONCAT][key]), + "bootstrap_ci_2p5": float(np.quantile(delta, 0.025)), + "bootstrap_ci_97p5": float(np.quantile(delta, 0.975)), + "bootstrap_probability_delta_gt_0": float(np.mean(delta > 0.0)), + "bootstrap_replicates": reps, + "resampling_unit": "source video id", + "paired": True, + "seed": bootstrap_seed, + }) + return rows + + +def _plot_summary(output: Path, summaries: list[dict[str, Any]]) -> None: + import matplotlib + matplotlib.use("Agg") + import matplotlib.pyplot as plt + + labels = ["EarlyConcat + BiGRU", "MoFE-7 + MLP Router"] + by_method = {row["method"]: row for row in summaries} + methods = (EARLYCONCAT, MOFE7_MLP) + metrics = ("clean_macro_f1", "corrupt_macro_f1", "worst_condition_macro_f1") + names = ("Clean", "Mean corrupted", "Worst condition") + x = np.arange(len(names)) + width = 0.34 + fig, ax = plt.subplots(figsize=(8.6, 4.8), constrained_layout=True) + for offset, method, label, color in ( + (-width / 2, methods[0], labels[0], "#4e79a7"), + (width / 2, methods[1], labels[1], "#f28e2b"), + ): + values = [by_method[method][metric] for metric in metrics] + ax.bar(x + offset, values, width, label=label, color=color) + ax.set_xticks(x, names) + ax.set_ylabel("Macro-F1") + ax.set_ylim(0, 1) + ax.set_title("Q2 selected-model validation comparison") + ax.legend(frameon=False) + output.mkdir(parents=True, exist_ok=True) + fig.savefig(output / "comparison_earlyconcat_mofe7.png", dpi=180) + plt.close(fig) + + +def _parameter_rows(dims: tuple[int, int, int], device: torch.device) -> list[dict[str, Any]]: + models: dict[str, nn.Module] = { + EARLYCONCAT: AlignedFusionModel("concat", dims=dims), + MOFE7_MLP: MixtureOfFusionExperts(dims=dims, **MODEL_CONFIG), + } + baseline_count = sum(parameter.numel() for parameter in models[EARLYCONCAT].parameters() if parameter.requires_grad) + rows = [] + for name, model in models.items(): + count = sum(parameter.numel() for parameter in model.parameters() if parameter.requires_grad) + rows.append({ + "method": name, + "trainable_parameters": count, + "ratio_to_earlyconcat": count / baseline_count, + "within_2x_earlyconcat": bool(count <= 2 * baseline_count), + }) + return rows + + +def _smoke_test(train: Split, output: Path, device: torch.device, seed: int) -> dict[str, Any]: + seed_everything(seed) + dims = tuple(int(x.shape[-1]) for x in train.x) + count = min(4, train.n) + xs = tuple(torch.as_tensor(x[:count], dtype=torch.float32, device=device) for x in train.x) + masks = torch.as_tensor(train.mask[:count].copy(), dtype=torch.bool, device=device) + masks[0] = True + if count > 1: + masks[1, 5:12, 0] = False + if count > 2: + masks[2, 18:23, :] = False + target_class = torch.as_tensor(train.y_cls[:count], dtype=torch.long, device=device) + target_intensity = torch.as_tensor(train.y_reg[:count], dtype=torch.float32, device=device) + reports: dict[str, Any] = {} + models: dict[str, nn.Module] = { + EARLYCONCAT: AlignedFusionModel("concat", dims=dims).to(device), + MOFE7_MLP: MixtureOfFusionExperts(dims=dims, **MODEL_CONFIG).to(device), + } + for name, model in models.items(): + model.train() + result = model(xs, masks) + loss = _loss(result, target_class, target_intensity) + loss.backward() + gradient = sum(float(p.grad.detach().abs().sum().cpu()) for p in model.parameters() if p.grad is not None) + reports[name] = { + "logits_shape": list(result["logits"].shape), + "intensity_shape": list(result["intensity"].shape), + "finite_loss": bool(torch.isfinite(loss).item()), + "gradient_l1": gradient, + } + mofe_model = models[MOFE7_MLP] + mo = mofe_model(xs, masks) + active = mo["availability"].any(dim=-1) + alpha_sums = mo["alpha"].sum(dim=-1) + alpha_error = float((alpha_sums[active] - 1).abs().max().cpu()) if active.any() else 0.0 + unavailable_weights = float(mo["alpha"].masked_select(~mo["availability"]).abs().max().cpu()) if (~mo["availability"]).any() else 0.0 + expert_gradients = { + name: sum(float(parameter.grad.detach().abs().sum().cpu()) for parameter in expert.parameters() if parameter.grad is not None) + for name, expert in mofe_model.experts.items() + } + router_gradient = sum(float(parameter.grad.detach().abs().sum().cpu()) for parameter in mofe_model.router.parameters() if parameter.grad is not None) + if alpha_error > 1e-6 or unavailable_weights > 1e-8: + raise RuntimeError(f"MoFE routing mask invariant failed: sum_error={alpha_error}, unavailable={unavailable_weights}") + if not all(value > 0 for value in expert_gradients.values()) or router_gradient <= 0: + raise RuntimeError(f"MoFE expert/router gradients are incomplete: {expert_gradients}; router={router_gradient}") + report = { + "passed": all(item["finite_loss"] and item["gradient_l1"] > 0 for item in reports.values()), + "seed": seed, + "device": str(device), + "cuda_device": torch.cuda.get_device_name(0) if device.type == "cuda" else None, + "batch_size_checked": count, + "steps": train.steps, + "models": reports, + "mofe_experts": list(EXPERT_NAMES), + "mofe_alpha_shape": list(mo["alpha"].shape), + "mofe_max_weight_sum_error": alpha_error, + "mofe_max_weight_on_unavailable_experts": unavailable_weights, + "mofe_expert_gradient_l1": expert_gradients, + "mofe_router_gradient_l1": router_gradient, + "parameter_count": {row["method"]: row["trainable_parameters"] for row in _parameter_rows(dims, device)}, + } + output.mkdir(parents=True, exist_ok=True) + (output / "smoke_test.json").write_text(json.dumps(report, indent=2), encoding="utf-8") + return report + + +def _run(args: argparse.Namespace) -> None: + output = args.output_dir.resolve() + output.mkdir(parents=True, exist_ok=True) + device = _device_for(args.device) + torch.set_num_threads(args.threads) + torch.backends.cudnn.deterministic = True + torch.backends.cudnn.benchmark = False + + raw = load_aligned() + computed_stats = fit_robust_stats(raw["train"]) + reference_stats_path = REFERENCE_OUTPUT / "aligned_robust_stats.npz" + if reference_stats_path.exists(): + stats = RobustStats.load(reference_stats_path) + scaler_diff = max( + max(float(np.max(np.abs(a - b))) for a, b in zip(computed_stats.center, stats.center)), + max(float(np.max(np.abs(a - b))) for a, b in zip(computed_stats.scale, stats.scale)), + ) + else: + stats = computed_stats + scaler_diff = 0.0 + train = apply_robust_stats(raw["train"], stats) + valid = apply_robust_stats(raw["valid"], stats) + stats.save(output / "aligned_robust_stats.npz") + dims = tuple(int(x.shape[-1]) for x in train.x) + feature_path = ATTACHMENT2 / "aligned_50.pkl" + if not feature_path.exists(): + raise FileNotFoundError(f"official aligned feature file not found: {feature_path}") + + if args.phase == "smoke": + report = _smoke_test(train, output, device, args.seeds[0]) + report["scaler_max_abs_difference_from_reference"] = scaler_diff + (output / "smoke_test.json").write_text(json.dumps(report, indent=2), encoding="utf-8") + print(f"selected-model smoke: passed={report['passed']} device={device}", flush=True) + return + + seeds = list(args.seeds) + metrics_rows: list[dict[str, Any]] = [] + predictions: dict[tuple[str, int, str], dict[str, np.ndarray]] = {} + router_rows: list[dict[str, Any]] = [] + utility_rows: list[dict[str, Any]] = [] + expert_rows: list[dict[str, Any]] = [] + history_rows: list[dict[str, Any]] = [] + best_epochs: dict[str, int] = {} + condition_names: list[str] = [] + + for seed in seeds: + baseline_dir = output / "models" / "baselines" / "concat" / f"seed_{seed}" + baseline, baseline_epoch, baseline_history = _load_or_train_concat( + train, valid, baseline_dir, device, seed, args.epochs, args.patience, + args.batch_size, args.reuse_checkpoints and not args.force_retrain, + ) + mofe_dir = output / "models" / MOFE7_MLP / f"seed_{seed}" + mofe, mofe_epoch, mofe_history = _train_mofe( + train, valid, mofe_dir, device, seed, args.epochs, args.patience, + args.batch_size, args.reuse_checkpoints and not args.force_retrain, + ) + best_epochs[f"{EARLYCONCAT}_seed_{seed}"] = baseline_epoch + best_epochs[f"{MOFE7_MLP}_seed_{seed}"] = mofe_epoch + history_rows.extend(baseline_history) + history_rows.extend(mofe_history) + + conditions = _conditions(valid, seed) + names = [condition for condition, _, _ in conditions] + if condition_names and names != condition_names: + raise RuntimeError("validation condition ordering changed between seeds") + condition_names = names + for method, model in ((EARLYCONCAT, baseline), (MOFE7_MLP, mofe)): + for condition, rate, masks in conditions: + prediction = _predict(model, valid, masks, device, args.batch_size) + predictions[(method, seed, condition)] = prediction + metrics_rows.append(_condition_row(method, seed, condition, rate, valid, prediction)) + if method == MOFE7_MLP: + route_row, utility_row = _diagnostics(seed, condition, masks, prediction) + router_rows.append(route_row) + utility_rows.append(utility_row) + for expert in EXPERT_NAMES: + forced = _predict(model, valid, masks, device, args.batch_size, force_expert=expert) + observed = masks[..., list(SUBSETS[expert])].all(axis=-1) + expert_rows.append({ + "method": MOFE7_MLP, + "seed": seed, + "condition": condition, + "expert": expert, + "available_position_fraction": float(observed.mean()), + **_metric_dict(valid.y_cls, valid.y_reg, forced["logits"], forced["intensity"]), + }) + print(f"evaluated {method}/seed{seed}", flush=True) + del baseline, mofe + if torch.cuda.is_available(): + torch.cuda.empty_cache() + + summaries = _summary_rows(metrics_rows) + paired = _paired_bootstrap( + predictions, valid, seeds, condition_names, args.bootstrap_reps, + args.bootstrap_seed, summaries, + ) if args.bootstrap_reps > 0 else [] + parameter_rows = _parameter_rows(dims, device) + output_rows = { + "metrics_by_condition.csv": metrics_rows, + "summary.csv": summaries, + "paired_bootstrap.csv": paired, + "parameter_count.csv": parameter_rows, + "router_weights_by_condition.csv": router_rows, + "routing_entropy.csv": router_rows, + "modality_utility_by_condition.csv": utility_rows, + "expert_condition_matrix.csv": expert_rows, + "training_history.csv": history_rows, + } + for filename, rows in output_rows.items(): + _write_csv(output / filename, rows) + _plot_summary(output / "figures", summaries) + + manifest = { + "experiment": "Q2 selected models: EarlyConcat + BiGRU and MoFE-7 + MLP Router", + "created_unix": time.time(), + "python_version": sys.version, + "torch_version": torch.__version__, + "numpy_version": np.__version__, + "device": str(device), + "cuda_device": torch.cuda.get_device_name(0) if device.type == "cuda" else None, + "feature_file": str(feature_path), + "feature_sha256": _sha256(feature_path), + "feature_dimensions": dict(zip(MODALITIES, dims)), + "sequence_length": train.steps, + "representation_note": "official ordered 50-wordpiece positions; not 50 physical-time bins", + "train_examples": train.n, + "valid_examples": valid.n, + "train_source_video_groups": len({sample_id.split("$_$", 1)[0] for sample_id in train.ids}), + "valid_source_video_groups": len({sample_id.split("$_$", 1)[0] for sample_id in valid.ids}), + "train_only_scaler": str(output / "aligned_robust_stats.npz"), + "scaler_max_abs_difference_from_reference": scaler_diff, + "test_labels_used": False, + "seeds": seeds, + "epochs_max": args.epochs, + "patience": args.patience, + "batch_size": args.batch_size, + "optimizer": "AdamW(lr=1.5e-4, weight_decay=1e-4), gradient clip 1.0", + "training_mask_augmentation": "same contiguous-block augment_masks protocol for both models", + "validation_conditions": condition_names, + "validation_corruption_seed": "seed + 13 + pattern_index*101 + int(rate*1000)", + "loss": "cross_entropy + 0.5*SmoothL1(intensity/3, regression_label/3)", + "models": { + EARLYCONCAT: "project modalities independently, concatenate features and masks, then BiGRU", + MOFE7_MLP: { + "experts": list(EXPERT_NAMES), + "router": "MLP over per-position observed values and local observation statistics", + "availability": "hard mask; unavailable expert weights are zero", + "shared_temporal_backbone": "one BiGRU after position-wise expert mixture", + }, + }, + "best_epochs": best_epochs, + "paired_bootstrap": { + "replicates": args.bootstrap_reps, + "seed": args.bootstrap_seed, + "resampling_unit": "source video id", + "paired": True, + }, + } + (output / "run_manifest.json").write_text(json.dumps(manifest, indent=2), encoding="utf-8") + print(f"selected-model results saved to {output}", flush=True) + + +def main() -> None: + parser = argparse.ArgumentParser(description="Train and compare the two retained Q2 models.") + parser.add_argument("--phase", choices=("smoke", "full"), default="full") + parser.add_argument("--epochs", type=int, default=32) + parser.add_argument("--patience", type=int, default=6) + parser.add_argument("--batch-size", type=int, default=64) + parser.add_argument("--threads", type=int, default=4) + parser.add_argument("--device", default="auto") + parser.add_argument("--seeds", type=int, nargs="+", default=list(SEEDS)) + parser.add_argument("--bootstrap-reps", type=int, default=1000) + parser.add_argument("--bootstrap-seed", type=int, default=20260924) + parser.add_argument("--reuse-checkpoints", action="store_true") + parser.add_argument("--force-retrain", action="store_true") + parser.add_argument("--output-dir", type=Path, default=DEFAULT_OUTPUT) + _run(parser.parse_args()) + + +if __name__ == "__main__": + main() diff --git a/deep_learning/Q3/.gitignore b/deep_learning/Q3/.gitignore deleted file mode 100644 index f0ccc32..0000000 --- a/deep_learning/Q3/.gitignore +++ /dev/null @@ -1,3 +0,0 @@ -.venv/ -__pycache__/ -*.py[cod] diff --git a/deep_learning/Q3/README.md b/deep_learning/Q3/README.md deleted file mode 100644 index b13e909..0000000 --- a/deep_learning/Q3/README.md +++ /dev/null @@ -1,18 +0,0 @@ -# Q3 explainability algorithm selection - -Q3 reuses the Q2-selected predictor and its train-only robust scaler. It does not train a different predictor just to make an attribution method look better. - -The experiment compares Integrated Gradients with five-slot grouped occlusion on the held-out Attachment 2 validation split. It measures deletion comprehensiveness, sufficiency, local rank stability under small input noise, and runtime. Polarity and continuous intensity explanations are selected separately because the two outputs can depend on different evidence. - -For the 20 Attachment 4 videos, the script reads the aligned feature pickle and matching MP4, predicts polarity/intensity, then applies the Q1 hard CTC Viterbi word-time procedure to each transcript and source audio. Text wordpieces and aligned audio/vision slots inherit the CTC word interval. The output includes CTC quality and validity flags; CTC timestamps are a weak temporal reference, not human event annotations or ground truth. - -## Run - -The environment is the `uv`-managed Q2 environment, which contains the same CUDA PyTorch, Transformers, NumPy, and plotting dependencies: - -```bash -cd deep_learning/Q3 -uv run --project ../Q2 python -m q3.explain_selection -``` - -Results are written to `deep_learning/Q3/outputs/explanation_selection/`. Q2 model checkpoints and normalization statistics are read from `deep_learning/Q2/outputs/algorithm_selection/`. diff --git a/deep_learning/Q3/RESULTS.md b/deep_learning/Q3/RESULTS.md deleted file mode 100644 index 86558f0..0000000 --- a/deep_learning/Q3/RESULTS.md +++ /dev/null @@ -1,42 +0,0 @@ -# Q3 explanation algorithm selection results - -## Predictor and validation setup - -Q3 reuses the Q2-selected early-concatenation + BiGRU classifier/regressor. Explanations were compared on all 728 held-out Attachment 2 validation clips; the predictor was trained on the official training split. Five-slot groups give 30 possible modality/time regions per clip. The class target is each clip's predicted-class probability; the intensity target is the predicted continuous score. - -## Explanation comparison - -| Explainer | Target | Signed change after deleting top 30% | Absolute change after deleting top 30% | Error when keeping only top 30% | Deletion AUC, 10–50% | Runtime for 728 clips | -| --- | --- | ---: | ---: | ---: | ---: | ---: | -| Grouped occlusion | Class probability | **+0.260** | 0.274 | **0.024** | **0.099** | **0.40 s** | -| Integrated Gradients | Class probability | +0.258 | **0.283** | 0.040 | 0.093 | 4.72 s | -| Random-region control | Class probability | +0.057 | 0.077 | 0.163 | 0.023 | — | -| Grouped occlusion | Intensity | −0.132 | 0.588 | **0.070** | −0.058 | **0.40 s** | -| Integrated Gradients | Intensity | −0.060 | **0.632** | 0.114 | −0.036 | 4.72 s | -| Random-region control | Intensity | −0.031 | 0.185 | 0.377 | −0.011 | — | - -Grouped occlusion is selected for polarity: it produces a slightly larger signed class-probability drop, lower sufficiency error, higher deletion AUC, and runs about 12 times faster. For intensity, the result is a tradeoff. Integrated Gradients causes a larger prediction change when its top evidence is removed; grouped occlusion better preserves the prediction when only its top regions remain. The displayed intensity regions use grouped occlusion, with Integrated Gradients retained as a directional cross-check. The negative signed intensity changes mean that removing the selected regions raises the predicted score on average; intensity evidence is bidirectional. - -Under standardized input noise with σ=0.02, the top-region rank Spearman correlations were 0.9993–0.9999 and top-30% Jaccard overlap was 0.987–0.998 on 120 balanced validation clips. This shows stability to that small perturbation, not stability across retrained models or a different dataset. - -## Mapping Attachment 4 evidence to video time - -All 20 Attachment 4 MP4 files were found, and all 20 BERT token sequences matched the supplied feature token IDs. Q1's hard CTC Viterbi word-time procedure aligned at least one word in every clip; 19 clips had full transcript word coverage, with mean word coverage 99.3%. Text wordpieces and corresponding aligned audio/vision slots inherit the transcript word interval, allowing a selected five-slot region to be shown in clip seconds. - -CTC times are weak alignment references, not human event labels. One transcript has partial coverage. The CTC path score is uncalibrated, so it is recorded for review and is not presented as a probability or ground truth. Attachment 4's pkl files themselves do not contain Q1 `time_bounds_s`; the script computes word times from the supplied video audio and transcript. - -Example timeline for clip 01: - -![Attachment 4 clip 01 explanation timeline](/home/gloamxun/modeling_zhaocui/deep_learning/Q3/outputs/explanation_selection/attachment4_01_evidence_timeline.png) - -## Artifacts - -- [Explainer faithfulness and stability summary](outputs/explanation_selection/q3_explanation_method_summary.csv) -- [Deletion/sufficiency curves](outputs/explanation_selection/q3_deletion_curves.csv) -- [Rank stability under small input noise](outputs/explanation_selection/q3_explanation_stability.csv) -- [Selected explanation methods and intensity tradeoff](outputs/explanation_selection/q3_explainer_selection.json) -- [Attachment 4 predictions](outputs/explanation_selection/attachment4_predictions.csv) -- [Top regions with word/time evidence](outputs/explanation_selection/attachment4_top_evidence.csv) -- [Attachment 4 alignment coverage audit](outputs/explanation_selection/attachment4_alignment_audit.json) - -The top-region CSV carries slot indices, transcript words, CTC-derived start/end seconds, uncalibrated alignment quality, prediction outputs, and modality-specific importance. It can be used to inspect individual samples or prepare the Chapter 4 evidence examples. diff --git a/deep_learning/Q3/outputs/explanation_selection/attachment4_01_evidence_timeline.png b/deep_learning/Q3/outputs/explanation_selection/attachment4_01_evidence_timeline.png deleted file mode 100644 index 8e95e7c..0000000 Binary files a/deep_learning/Q3/outputs/explanation_selection/attachment4_01_evidence_timeline.png and /dev/null differ diff --git a/deep_learning/Q3/outputs/explanation_selection/attachment4_alignment_audit.json b/deep_learning/Q3/outputs/explanation_selection/attachment4_alignment_audit.json deleted file mode 100644 index 03fb0b9..0000000 --- a/deep_learning/Q3/outputs/explanation_selection/attachment4_alignment_audit.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "n_samples": 20, - "n_video_files_found": 20, - "n_ctc_any_words_aligned": 20, - "n_ctc_full_word_coverage": 19, - "mean_transcript_word_coverage": 0.9932742662282303, - "n_bert_token_sequences_matching_pickle": 20, - "time_mapping": "Q1 B1 CTC Viterbi hard word intervals computed from the supplied Attachment 4 video audio and transcript; subword slots inherit their transcript word interval", - "quality_note": "CTC path score is uncalibrated. These intervals are localization references for interpretation, not human-annotated ground truth." -} \ No newline at end of file diff --git a/deep_learning/Q3/outputs/explanation_selection/attachment4_predictions.csv b/deep_learning/Q3/outputs/explanation_selection/attachment4_predictions.csv deleted file mode 100644 index 5277382..0000000 --- a/deep_learning/Q3/outputs/explanation_selection/attachment4_predictions.csv +++ /dev/null @@ -1,21 +0,0 @@ -sample_id,predicted_class,predicted_class_id,predicted_class_probability,predicted_intensity,transcript,video_file_exists,ctc_alignment_status,ctc_word_coverage,ctc_aligned_words,transcript_words,bert_token_ids_match_pickle -01,Neutral,1,0.602049708366394,-0.12586602568626404,Replacing these wear components when replacing the timing belt is essential to ensuring the new belt performs to its mileage requirements,True,ok,1.0,21,21,True -02,Positive,2,0.4676651060581207,0.24043764173984528,We want to live by each other’s happiness - not by each other’s misery.,True,partial,0.9285714285714286,13,14,True -03,Negative,0,0.4883529841899872,-0.6055137515068054,"There's one lender at the moment which I think is just Bankwest who don't take rental income into account, they take rental yield into account.",True,ok,1.0,25,25,True -04,Negative,0,0.5609740614891052,0.06384700536727905,"If I blow it at the team exercise, should I kiss my chances of cheering ""GO BLUE"" goodbye?] Absolutely not.",True,ok,1.0,20,20,True -05,Positive,2,0.8418908715248108,1.0688594579696655,"Hi, my name is Chloe, video marketer for Red Wagon Marketing.",True,ok,1.0,11,11,True -06,Positive,2,0.8087986707687378,0.8882662057876587,"If you're a fan of dancing in that sense, just like to watch people dance, see impressive dance moves then you might want to check out this movie solely for that",True,ok,1.0,31,31,True -07,Positive,2,0.7762729525566101,0.8499683141708374,"As Linn’s associate editor Michael Baadke reports in our November 28 issue, attendees “enthusiastically discussed the topics of growing the hobby, the future of stamp shows, and dealers and philatelic partnerships, along with ways the leading organizations involved in the stamp hobby can work together to make it succeed and grow",True,ok,0.9803921568627451,50,51,True -08,Positive,2,0.8577593564987183,1.020017147064209,"That brings us to tonight, the Universal Design Grand Challenge",True,ok,1.0,10,10,True -09,Negative,0,0.9559774994850159,-1.5653929710388184,"(uhh) I did not like this movie at all, I would not recommend it",True,ok,1.0,14,14,True -10,Negative,0,0.8787182569503784,-0.9965534806251526,(umm) And you know I really do like to see fluffy chick flicks sometimes so I'm not against that but this one was pretty terrible,True,ok,1.0,25,25,True -11,Negative,0,0.6878972053527832,-0.6313911080360413,I would be ashamed to have made this film if I was a director,True,ok,1.0,14,14,True -12,Negative,0,0.7832551002502441,-1.2948027849197388,"Or worse, an individual previously had good credit, but usually by no fault of their own, or perhaps by fault of their own, they have let their credit sag, and credit scores is very low.",True,ok,1.0,35,35,True -13,Neutral,1,0.4873892366886139,0.27714914083480835,"For example, I could take a set of data and from that data, I can find a relationship between any two of the given factors or more.",True,ok,1.0,27,27,True -14,Positive,2,0.6718549728393555,0.6427467465400696,He is the co-founder of Rossen and Vettese Limited and the former Executive Director of Uniform Final Examination (UFE) courses at Toronto's York University.,True,ok,1.0,24,24,True -15,Positive,2,0.9403521418571472,1.3164536952972412,"However, despite their poverty, the family prioritize education because they believed in its power to transform lives",True,ok,1.0,17,17,True -16,Negative,0,0.9597424864768982,-1.8123797178268433,"It's a terrible, this is a terrible movie",True,ok,1.0,8,8,True -17,Positive,2,0.9568064212799072,1.3108301162719727,"Applying these four design concepts to your presentations is simple, easy and will make people think you turned into a design guru.",True,ok,1.0,22,22,True -18,Neutral,1,0.4668891727924347,-0.5185524225234985,"-And in Denmark - the first Baltic Cod fishery has been MSC – certified -Meanwhile, the Faeroese Mackerel Fishery has been denied MSC certification based on the fact that the fishery has failed to reach an agreement on mackerel quotas with Norway and the European Union.",True,ok,0.9565217391304348,44,46,True -19,Negative,0,0.5051047205924988,-0.09867963194847107,"People are surprisingly forgiving brands when they own up to mistakes, and unfortunately some haters out there love to point fingers and jump all over imperfections, but for the most part, people understand",True,ok,1.0,33,33,True -20,Positive,2,0.9234767556190491,1.2413525581359863,"And of course, click in the link of the description of this video for more, and we'll have more live updates and a stock market video (wrap-up) at the end of the day today.",True,ok,1.0,34,34,True diff --git a/deep_learning/Q3/outputs/explanation_selection/attachment4_top_evidence.csv b/deep_learning/Q3/outputs/explanation_selection/attachment4_top_evidence.csv deleted file mode 100644 index 4e44555..0000000 --- a/deep_learning/Q3/outputs/explanation_selection/attachment4_top_evidence.csv +++ /dev/null @@ -1,178 +0,0 @@ -sample_id,modality,block_index,slot_start_index,slot_end_index_exclusive,slot_indices,tokens_or_wordpieces,matched_words,word_indices,time_start_s,time_end_s,ctc_quality_uncalibrated_mean,ctc_words_covered,ctc_word_coverage_clip,class_importance,intensity_importance,class_explainer,intensity_explainer,ctc_alignment_status -01,text,1,5,10,"5,6,7,8,9",when replacing the timing belt,when replacing the timing belt,"4,5,6,7,8",1.9625,3.2225000000000006,1.2557723595913322e-13,5,1.0,0.0734131932258606,0.08602334558963776,grouped_occlusion,grouped_occlusion,ok -01,text,0,0,5,"1,2,3,4",replacing these wear components,Replacing these wear components,"0,1,2,3",0.2225,1.9425,1.0916685621897051e-13,4,1.0,0.053081393241882324,0.12229763716459274,grouped_occlusion,grouped_occlusion,ok -01,text,3,15,20,"15,16,17,18,19",new belt performs to its,new belt performs to its,"14,15,16,17,18",5.202500000000001,7.1625000000000005,9.867163331619365e-14,5,1.0,0.04383492469787598,0.09824259579181671,grouped_occlusion,grouped_occlusion,ok -01,audio,3,15,20,"15,16,17,18,19",new belt performs to its,new belt performs to its,"14,15,16,17,18",5.202500000000001,7.1625000000000005,9.867163331619365e-14,5,1.0,0.0138014554977417,0.061192527413368225,grouped_occlusion,grouped_occlusion,ok -01,audio,1,5,10,"5,6,7,8,9",when replacing the timing belt,when replacing the timing belt,"4,5,6,7,8",1.9625,3.2225000000000006,1.2557723595913322e-13,5,1.0,0.013497352600097656,0.025103554129600525,grouped_occlusion,grouped_occlusion,ok -01,audio,2,10,15,"10,11,12,13,14",is essential to ensuring the,is essential to ensuring the,"9,10,11,12,13",3.6025000000000005,5.1825,6.037600744192027e-14,5,1.0,0.006180107593536377,0.051678575575351715,grouped_occlusion,grouped_occlusion,ok -01,vision,2,10,15,"10,11,12,13,14",is essential to ensuring the,is essential to ensuring the,"9,10,11,12,13",3.6025000000000005,5.1825,6.037600744192027e-14,5,1.0,0.006771266460418701,0.02889835834503174,grouped_occlusion,grouped_occlusion,ok -01,vision,0,0,5,"1,2,3,4",replacing these wear components,Replacing these wear components,"0,1,2,3",0.2225,1.9425,1.0916685621897051e-13,4,1.0,0.003824293613433838,0.00939151644706726,grouped_occlusion,grouped_occlusion,ok -01,vision,3,15,20,"15,16,17,18,19",new belt performs to its,new belt performs to its,"14,15,16,17,18",5.202500000000001,7.1625000000000005,9.867163331619365e-14,5,1.0,0.002546370029449463,0.02828623354434967,grouped_occlusion,grouped_occlusion,ok -02,text,1,5,10,"5,6,7,8,9",by each other ’ s,by each other’s,"4,5,6",0.9824999999999999,1.4625,2.631730448927397e-13,3,0.9285714285714286,0.14046677947044373,0.2743243873119354,grouped_occlusion,grouped_occlusion,partial -02,text,2,10,15,"10,12,13,14",happiness not by each,happiness not by each,"7,9,10,11",1.5025,2.4025000000000003,1.0520036340214618e-13,4,0.9285714285714286,0.12429457902908325,0.22381268441677094,grouped_occlusion,grouped_occlusion,partial -02,text,0,0,5,"1,2,3,4",we want to live,We want to live,"0,1,2,3",0.2425,0.9425,1.6290645074933628e-13,4,0.9285714285714286,0.05786612629890442,0.0930139571428299,grouped_occlusion,grouped_occlusion,partial -02,audio,1,5,10,"5,6,7,8,9",by each other ’ s,by each other’s,"4,5,6",0.9824999999999999,1.4625,2.631730448927397e-13,3,0.9285714285714286,0.02898383140563965,0.03369395434856415,grouped_occlusion,grouped_occlusion,partial -02,audio,0,0,5,"1,2,3,4",we want to live,We want to live,"0,1,2,3",0.2425,0.9425,1.6290645074933628e-13,4,0.9285714285714286,0.017408668994903564,0.032251402735710144,grouped_occlusion,grouped_occlusion,partial -02,audio,2,10,15,"10,12,13,14",happiness not by each,happiness not by each,"7,9,10,11",1.5025,2.4025000000000003,1.0520036340214618e-13,4,0.9285714285714286,0.0049620866775512695,0.005869343876838684,grouped_occlusion,grouped_occlusion,partial -02,vision,2,10,15,"10,12,13,14",happiness not by each,happiness not by each,"7,9,10,11",1.5025,2.4025000000000003,1.0520036340214618e-13,4,0.9285714285714286,0.06942322850227356,0.14820988476276398,grouped_occlusion,grouped_occlusion,partial -02,vision,1,5,10,"5,6,7,8,9",by each other ’ s,by each other’s,"4,5,6",0.9824999999999999,1.4625,2.631730448927397e-13,3,0.9285714285714286,0.06549379229545593,0.16973082721233368,grouped_occlusion,grouped_occlusion,partial -02,vision,3,15,20,"15,16,17,18,19",other ’ s misery .,other’s misery.,"12,13",2.4825000000000004,3.3025000000000007,7.766181684510053e-13,2,0.9285714285714286,0.04377242922782898,0.09742923080921173,grouped_occlusion,grouped_occlusion,partial -03,text,4,20,25,"20,21,22,23,24",t take rental income into,don't take rental income into,"13,14,15,16,17",4.522500000000001,6.1225000000000005,1.960631140554847e-11,5,1.0,0.0903124213218689,0.2127276062965393,grouped_occlusion,grouped_occlusion,ok -03,text,5,25,30,"25,26,27,28,29","account , they take rental","account, they take rental","18,19,20,21",6.242500000000001,8.3225,6.012753101977051e-13,4,1.0,0.08981853723526001,0.18761926889419556,grouped_occlusion,grouped_occlusion,ok -03,text,0,0,5,"1,2,3,4",there ' s one,There's one,"0,1",0.0425,0.9824999999999999,4.302478284875174e-12,2,1.0,0.049811989068984985,0.13886994123458862,grouped_occlusion,grouped_occlusion,ok -03,audio,4,20,25,"20,21,22,23,24",t take rental income into,don't take rental income into,"13,14,15,16,17",4.522500000000001,6.1225000000000005,1.960631140554847e-11,5,1.0,0.02670571208000183,0.05279737710952759,grouped_occlusion,grouped_occlusion,ok -03,audio,2,10,15,"10,11,12,13,14",which i think is just,which I think is just,"6,7,8,9,10",2.1825000000000006,3.2225000000000006,1.63512502285273e-11,5,1.0,0.0200861394405365,0.05790430307388306,grouped_occlusion,grouped_occlusion,ok -03,audio,5,25,30,"25,26,27,28,29","account , they take rental","account, they take rental","18,19,20,21",6.242500000000001,8.3225,6.012753101977051e-13,4,1.0,0.017825692892074585,0.05010336637496948,grouped_occlusion,grouped_occlusion,ok -03,vision,4,20,25,"20,21,22,23,24",t take rental income into,don't take rental income into,"13,14,15,16,17",4.522500000000001,6.1225000000000005,1.960631140554847e-11,5,1.0,0.013105422258377075,0.026691555976867676,grouped_occlusion,grouped_occlusion,ok -03,vision,5,25,30,"25,26,27,28,29","account , they take rental","account, they take rental","18,19,20,21",6.242500000000001,8.3225,6.012753101977051e-13,4,1.0,0.013015061616897583,0.018819868564605713,grouped_occlusion,grouped_occlusion,ok -03,vision,3,15,20,"15,16,17,18,19",bank ##west who don ',Bankwest who don't,"11,12,13",3.3025000000000007,4.702500000000001,3.167890863874076e-11,3,1.0,0.01059773564338684,0.01818716526031494,grouped_occlusion,grouped_occlusion,ok -04,text,2,10,15,"10,11,12,13,14",should i kiss my chances,should I kiss my chances,"8,9,10,11,12",2.9025000000000003,4.062500000000001,1.5775802776191961e-10,5,1.0,0.21443644165992737,0.4394690990447998,grouped_occlusion,grouped_occlusion,ok -04,text,0,0,5,"1,2,3,4",if i blow it,If I blow it,"0,1,2,3",0.0025000000000000005,1.4025,7.158484560430912e-12,4,1.0,0.20325228571891785,0.4211195707321167,grouped_occlusion,grouped_occlusion,ok -04,text,4,20,25,"20,21,22,23,24",""" goodbye ? ] absolutely","BLUE"" goodbye?] Absolutely","16,17,18",5.102500000000001,6.7225,2.2125524584117753e-13,3,1.0,0.12389594316482544,0.23821038007736206,grouped_occlusion,grouped_occlusion,ok -04,audio,3,15,20,"15,16,17,18,19","of cheering "" go blue","of cheering ""GO BLUE""","13,14,15,16",4.2225,5.5025,1.1504596424116164e-11,4,1.0,0.061087846755981445,0.1968434453010559,grouped_occlusion,grouped_occlusion,ok -04,audio,2,10,15,"10,11,12,13,14",should i kiss my chances,should I kiss my chances,"8,9,10,11,12",2.9025000000000003,4.062500000000001,1.5775802776191961e-10,5,1.0,0.05859661102294922,0.1886153370141983,grouped_occlusion,grouped_occlusion,ok -04,audio,4,20,25,"20,21,22,23,24",""" goodbye ? ] absolutely","BLUE"" goodbye?] Absolutely","16,17,18",5.102500000000001,6.7225,2.2125524584117753e-13,3,1.0,0.042976558208465576,0.1398433893918991,grouped_occlusion,grouped_occlusion,ok -04,vision,2,10,15,"10,11,12,13,14",should i kiss my chances,should I kiss my chances,"8,9,10,11,12",2.9025000000000003,4.062500000000001,1.5775802776191961e-10,5,1.0,0.05346882343292236,0.12295112013816833,grouped_occlusion,grouped_occlusion,ok -04,vision,3,15,20,"15,16,17,18,19","of cheering "" go blue","of cheering ""GO BLUE""","13,14,15,16",4.2225,5.5025,1.1504596424116164e-11,4,1.0,0.05310636758804321,0.10312038660049438,grouped_occlusion,grouped_occlusion,ok -04,vision,1,5,10,"5,6,7,8,9","at the team exercise ,","at the team exercise,","4,5,6,7",1.5625,2.8025000000000007,9.294841968446489e-11,4,1.0,0.04525059461593628,0.11550295352935791,grouped_occlusion,grouped_occlusion,ok -05,text,2,10,15,"10,11,12,13,14",##er for red wagon marketing,marketer for Red Wagon Marketing.,"6,7,8,9,10",3.5825000000000005,5.4625,4.2536280223424595e-13,5,1.0,0.030606567859649658,0.023810386657714844,grouped_occlusion,grouped_occlusion,ok -05,text,0,0,5,"1,2,3,4","hi , my name","Hi, my name","0,1,2",1.2625,2.2825000000000006,9.345394556125576e-12,3,1.0,0.029161453247070312,0.012323379516601562,grouped_occlusion,grouped_occlusion,ok -05,text,1,5,10,"5,6,7,8,9","is chloe , video market","is Chloe, video marketer","3,4,5,6",2.3225000000000002,4.022500000000001,8.972688035370665e-12,4,1.0,0.015018045902252197,0.057985544204711914,grouped_occlusion,grouped_occlusion,ok -05,audio,1,5,10,"5,6,7,8,9","is chloe , video market","is Chloe, video marketer","3,4,5,6",2.3225000000000002,4.022500000000001,8.972688035370665e-12,4,1.0,0.022373735904693604,0.12835413217544556,grouped_occlusion,grouped_occlusion,ok -05,audio,0,0,5,"1,2,3,4","hi , my name","Hi, my name","0,1,2",1.2625,2.2825000000000006,9.345394556125576e-12,3,1.0,0.006277620792388916,0.018590211868286133,grouped_occlusion,grouped_occlusion,ok -05,audio,3,15,20,15,.,Marketing.,10,4.982500000000001,5.4625,1.7565947322020906e-13,1,1.0,0.003395378589630127,0.015216469764709473,grouped_occlusion,grouped_occlusion,ok -05,vision,1,5,10,"5,6,7,8,9","is chloe , video market","is Chloe, video marketer","3,4,5,6",2.3225000000000002,4.022500000000001,8.972688035370665e-12,4,1.0,0.03512507677078247,0.09098595380783081,grouped_occlusion,grouped_occlusion,ok -05,vision,0,0,5,"1,2,3,4","hi , my name","Hi, my name","0,1,2",1.2625,2.2825000000000006,9.345394556125576e-12,3,1.0,0.02607184648513794,0.10822361707687378,grouped_occlusion,grouped_occlusion,ok -05,vision,3,15,20,15,.,Marketing.,10,4.982500000000001,5.4625,1.7565947322020906e-13,1,1.0,0.007893681526184082,0.017847895622253418,grouped_occlusion,grouped_occlusion,ok -06,text,3,15,20,"15,16,17,18,19","to watch people dance ,","to watch people dance,","11,12,13,14",2.5225000000000004,3.5825000000000005,9.533516620804992e-13,4,1.0,0.0643836259841919,0.11163991689682007,grouped_occlusion,grouped_occlusion,ok -06,text,4,20,25,"20,21,22,23,24",see impressive dance moves then,see impressive dance moves then,"15,16,17,18,19",3.7225000000000006,5.442500000000001,1.003787440114341e-12,5,1.0,0.05716830492019653,0.10976755619049072,grouped_occlusion,grouped_occlusion,ok -06,text,2,10,15,"10,11,12,13,14","that sense , just like","that sense, just like","7,8,9,10",1.6625,2.5025000000000004,6.328522515592245e-13,4,1.0,0.05548006296157837,0.1172025203704834,grouped_occlusion,grouped_occlusion,ok -06,audio,6,30,35,"30,31,32,33,34",out this movie solely for,out this movie solely for,"25,26,27,28,29",6.482500000000001,7.562500000000001,5.6144654493064985e-12,5,1.0,0.00836336612701416,0.047490835189819336,grouped_occlusion,grouped_occlusion,ok -06,audio,2,10,15,"10,11,12,13,14","that sense , just like","that sense, just like","7,8,9,10",1.6625,2.5025000000000004,6.328522515592245e-13,4,1.0,0.006695687770843506,0.0182039737701416,grouped_occlusion,grouped_occlusion,ok -06,audio,3,15,20,"15,16,17,18,19","to watch people dance ,","to watch people dance,","11,12,13,14",2.5225000000000004,3.5825000000000005,9.533516620804992e-13,4,1.0,0.005690395832061768,0.0055931806564331055,grouped_occlusion,grouped_occlusion,ok -06,vision,2,10,15,"10,11,12,13,14","that sense , just like","that sense, just like","7,8,9,10",1.6625,2.5025000000000004,6.328522515592245e-13,4,1.0,0.024024665355682373,0.06122779846191406,grouped_occlusion,grouped_occlusion,ok -06,vision,3,15,20,"15,16,17,18,19","to watch people dance ,","to watch people dance,","11,12,13,14",2.5225000000000004,3.5825000000000005,9.533516620804992e-13,4,1.0,0.022018134593963623,0.061392247676849365,grouped_occlusion,grouped_occlusion,ok -06,vision,4,20,25,"20,21,22,23,24",see impressive dance moves then,see impressive dance moves then,"15,16,17,18,19",3.7225000000000006,5.442500000000001,1.003787440114341e-12,5,1.0,0.02075207233428955,0.05445140600204468,grouped_occlusion,grouped_occlusion,ok -07,text,4,20,25,"20,21,22,23,24",“ enthusiastically discussed the topics,“enthusiastically discussed the topics,"13,14,15,16",4.362500000000001,7.5025,5.132190603623443e-13,4,0.9803921568627451,0.05012655258178711,0.07725489139556885,grouped_occlusion,grouped_occlusion,ok -07,text,5,25,30,"25,26,27,28,29","of growing the hobby ,","of growing the hobby,","17,18,19,20",7.562500000000001,9.1225,6.009222892922947e-13,4,0.9803921568627451,0.03437221050262451,0.05201399326324463,grouped_occlusion,grouped_occlusion,ok -07,text,9,45,50,"45,46,47,48",with ways the leading,with ways the leading,"32,33,34,35",14.3225,15.3825,4.788019704647536e-13,4,0.9803921568627451,0.03220874071121216,0.0367276668548584,grouped_occlusion,grouped_occlusion,ok -07,audio,3,15,20,"15,17,18,19","november issue , attendees","November issue, attendees","9,11,12",3.1025000000000005,4.3425,6.891936173584844e-14,3,0.9803921568627451,0.009746789932250977,0.013783574104309082,grouped_occlusion,grouped_occlusion,ok -07,audio,1,5,10,"5,6,7,8,9",s associate editor michael ba,Linn’s associate editor Michael Baadke,"1,2,3,4,5",0.7224999999999999,2.4025000000000003,6.973944706871624e-12,5,0.9803921568627451,0.008990466594696045,0.007673025131225586,grouped_occlusion,grouped_occlusion,ok -07,audio,2,10,15,"10,11,12,13,14",##ad ##ke reports in our,Baadke reports in our,"5,6,7,8",2.0825000000000005,3.0825000000000005,7.431383019894585e-12,4,0.9803921568627451,0.0072026848793029785,0.008348703384399414,grouped_occlusion,grouped_occlusion,ok -07,vision,7,35,40,"35,36,37,38,39",", and dealers and phil","shows, and dealers and philatelic","25,26,27,28,29",10.522499999999999,12.8825,1.0360054611198636e-12,5,0.9803921568627451,0.004808366298675537,0.029618024826049805,grouped_occlusion,grouped_occlusion,ok -07,vision,3,15,20,"15,17,18,19","november issue , attendees","November issue, attendees","9,11,12",3.1025000000000005,4.3425,6.891936173584844e-14,3,0.9803921568627451,0.0032321810722351074,0.0074433088302612305,grouped_occlusion,grouped_occlusion,ok -07,vision,2,10,15,"10,11,12,13,14",##ad ##ke reports in our,Baadke reports in our,"5,6,7,8",2.0825000000000005,3.0825000000000005,7.431383019894585e-12,4,0.9803921568627451,0.0024552345275878906,0.008127868175506592,grouped_occlusion,grouped_occlusion,ok -08,text,0,0,5,"1,2,3,4",that brings us to,That brings us to,"0,1,2,3",0.4825,1.3425,4.985850672359178e-13,4,1.0,0.11082303524017334,0.11384594440460205,grouped_occlusion,grouped_occlusion,ok -08,text,1,5,10,"5,6,7,8,9","tonight , the universal design","tonight, the Universal Design","4,5,6,7",1.4825,3.1225000000000005,1.5200054500503443e-13,4,1.0,0.1038968563079834,0.21401238441467285,grouped_occlusion,grouped_occlusion,ok -08,text,2,10,15,"10,11",grand challenge,Grand Challenge,"8,9",3.3825000000000003,4.322500000000001,1.194004156129736e-14,2,1.0,0.04137396812438965,0.019734859466552734,grouped_occlusion,grouped_occlusion,ok -08,audio,2,10,15,"10,11",grand challenge,Grand Challenge,"8,9",3.3825000000000003,4.322500000000001,1.194004156129736e-14,2,1.0,0.0033051371574401855,0.022482693195343018,grouped_occlusion,grouped_occlusion,ok -08,audio,1,5,10,"5,6,7,8,9","tonight , the universal design","tonight, the Universal Design","4,5,6,7",1.4825,3.1225000000000005,1.5200054500503443e-13,4,1.0,0.0019735097885131836,0.06727790832519531,grouped_occlusion,grouped_occlusion,ok -08,audio,0,0,5,"1,2,3,4",that brings us to,That brings us to,"0,1,2,3",0.4825,1.3425,4.985850672359178e-13,4,1.0,0.0007883310317993164,0.06953203678131104,grouped_occlusion,grouped_occlusion,ok -08,vision,0,0,5,"1,2,3,4",that brings us to,That brings us to,"0,1,2,3",0.4825,1.3425,4.985850672359178e-13,4,1.0,0.004726111888885498,0.004060029983520508,grouped_occlusion,grouped_occlusion,ok -08,vision,1,5,10,"5,6,7,8,9","tonight , the universal design","tonight, the Universal Design","4,5,6,7",1.4825,3.1225000000000005,1.5200054500503443e-13,4,1.0,0.004242956638336182,0.008825302124023438,grouped_occlusion,grouped_occlusion,ok -08,vision,2,10,15,"10,11",grand challenge,Grand Challenge,"8,9",3.3825000000000003,4.322500000000001,1.194004156129736e-14,2,1.0,0.002269923686981201,0.007381081581115723,grouped_occlusion,grouped_occlusion,ok -09,text,1,5,10,"5,6,7,8,9",i did not like this,I did not like this,"1,2,3,4,5",1.1824999999999999,2.2025000000000006,1.8337023088832634e-11,5,1.0,0.05766040086746216,0.3939073085784912,grouped_occlusion,grouped_occlusion,ok -09,text,2,10,15,"10,11,12,13,14","movie at all , i","movie at all, I","6,7,8,9",2.2825000000000006,3.1025000000000005,1.7216472685121577e-10,4,1.0,0.05135905742645264,0.33904457092285156,grouped_occlusion,grouped_occlusion,ok -09,text,0,0,5,"1,2,3,4",( uh ##h ),(uhh),0,0.7825,1.1425,6.173012658529708e-13,1,1.0,0.04096817970275879,0.3692760467529297,grouped_occlusion,grouped_occlusion,ok -09,audio,0,0,5,"1,2,3,4",( uh ##h ),(uhh),0,0.7825,1.1425,6.173012658529708e-13,1,1.0,0.004597127437591553,0.05938518047332764,grouped_occlusion,grouped_occlusion,ok -09,audio,1,5,10,"5,6,7,8,9",i did not like this,I did not like this,"1,2,3,4,5",1.1824999999999999,2.2025000000000006,1.8337023088832634e-11,5,1.0,0.0028375983238220215,0.0619351863861084,grouped_occlusion,grouped_occlusion,ok -09,audio,3,15,20,"15,16,17,18",would not recommend it,would not recommend it,"10,11,12,13",3.2025000000000006,4.942500000000001,2.9955450915683833e-12,4,1.0,0.0018830299377441406,0.029448747634887695,grouped_occlusion,grouped_occlusion,ok -09,vision,2,10,15,"10,11,12,13,14","movie at all , i","movie at all, I","6,7,8,9",2.2825000000000006,3.1025000000000005,1.7216472685121577e-10,4,1.0,0.0035586953163146973,0.010941743850708008,grouped_occlusion,grouped_occlusion,ok -09,vision,3,15,20,"15,16,17,18",would not recommend it,would not recommend it,"10,11,12,13",3.2025000000000006,4.942500000000001,2.9955450915683833e-12,4,1.0,0.002703547477722168,0.007805347442626953,grouped_occlusion,grouped_occlusion,ok -09,vision,0,0,5,"1,2,3,4",( uh ##h ),(uhh),0,0.7825,1.1425,6.173012658529708e-13,1,1.0,0.001896202564239502,0.0043097734451293945,grouped_occlusion,grouped_occlusion,ok -10,text,5,25,30,"25,26,27,28,29",but this one was pretty,but this one was pretty,"19,20,21,22,23",6.742500000000001,9.1625,3.4013181527343934e-12,5,1.0,0.10162562131881714,0.3853045701980591,grouped_occlusion,grouped_occlusion,ok -10,text,2,10,15,"10,11,12,13,14",like to see fluffy chick,like to see fluffy chick,"7,8,9,10,11",2.4825000000000004,3.8425000000000002,5.712801915197975e-12,5,1.0,0.07766342163085938,0.3524249792098999,grouped_occlusion,grouped_occlusion,ok -10,text,3,15,20,"15,16,17,18,19",flick ##s sometimes so i,flicks sometimes so I'm,"12,13,14,15",3.9025000000000003,5.242500000000001,9.396792278478092e-09,4,1.0,0.07552039623260498,0.3311324715614319,grouped_occlusion,grouped_occlusion,ok -10,audio,5,25,30,"25,26,27,28,29",but this one was pretty,but this one was pretty,"19,20,21,22,23",6.742500000000001,9.1625,3.4013181527343934e-12,5,1.0,0.010288834571838379,0.08406132459640503,grouped_occlusion,grouped_occlusion,ok -10,audio,1,5,10,"5,6,7,8,9",you know i really do,you know I really do,"2,3,4,5,6",1.3425,2.4225000000000003,9.51819295384607e-13,5,1.0,0.008134961128234863,0.08679646253585815,grouped_occlusion,grouped_occlusion,ok -10,audio,2,10,15,"10,11,12,13,14",like to see fluffy chick,like to see fluffy chick,"7,8,9,10,11",2.4825000000000004,3.8425000000000002,5.712801915197975e-12,5,1.0,0.007106482982635498,0.06629914045333862,grouped_occlusion,grouped_occlusion,ok -10,vision,3,15,20,"15,16,17,18,19",flick ##s sometimes so i,flicks sometimes so I'm,"12,13,14,15",3.9025000000000003,5.242500000000001,9.396792278478092e-09,4,1.0,0.005765736103057861,0.02699226140975952,grouped_occlusion,grouped_occlusion,ok -10,vision,5,25,30,"25,26,27,28,29",but this one was pretty,but this one was pretty,"19,20,21,22,23",6.742500000000001,9.1625,3.4013181527343934e-12,5,1.0,0.005424618721008301,0.004094421863555908,grouped_occlusion,grouped_occlusion,ok -10,vision,4,20,25,"20,21,22,23,24",' m not against that,I'm not against that,"15,16,17,18",5.1825,6.6225000000000005,9.396402256608505e-09,4,1.0,0.0023061037063598633,0.0014348030090332031,grouped_occlusion,grouped_occlusion,ok -11,text,0,0,5,"1,2,3,4",i would be ashamed,I would be ashamed,"0,1,2,3",0.5225,1.3825,2.4343081894167942e-11,4,1.0,0.17121726274490356,0.3522495925426483,grouped_occlusion,grouped_occlusion,ok -11,text,1,5,10,"5,6,7,8,9",to have made this film,to have made this film,"4,5,6,7,8",3.2225000000000006,4.202500000000001,1.8143959640334076e-12,5,1.0,0.050980210304260254,0.09193223714828491,grouped_occlusion,grouped_occlusion,ok -11,text,2,10,15,"10,11,12,13,14",if i was a director,if I was a director,"9,10,11,12,13",4.442500000000001,5.702500000000001,1.195583991709281e-10,5,1.0,0.03379368782043457,0.03823119401931763,grouped_occlusion,grouped_occlusion,ok -11,audio,2,10,15,"10,11,12,13,14",if i was a director,if I was a director,"9,10,11,12,13",4.442500000000001,5.702500000000001,1.195583991709281e-10,5,1.0,0.07368618249893188,0.19450706243515015,grouped_occlusion,grouped_occlusion,ok -11,audio,1,5,10,"5,6,7,8,9",to have made this film,to have made this film,"4,5,6,7,8",3.2225000000000006,4.202500000000001,1.8143959640334076e-12,5,1.0,0.026468276977539062,0.03296065330505371,grouped_occlusion,grouped_occlusion,ok -11,audio,0,0,5,"1,2,3,4",i would be ashamed,I would be ashamed,"0,1,2,3",0.5225,1.3825,2.4343081894167942e-11,4,1.0,0.006807446479797363,0.017702996730804443,grouped_occlusion,grouped_occlusion,ok -11,vision,0,0,5,"1,2,3,4",i would be ashamed,I would be ashamed,"0,1,2,3",0.5225,1.3825,2.4343081894167942e-11,4,1.0,0.019326627254486084,0.04640209674835205,grouped_occlusion,grouped_occlusion,ok -11,vision,1,5,10,"5,6,7,8,9",to have made this film,to have made this film,"4,5,6,7,8",3.2225000000000006,4.202500000000001,1.8143959640334076e-12,5,1.0,0.006760776042938232,0.037277281284332275,grouped_occlusion,grouped_occlusion,ok -11,vision,2,10,15,"10,11,12,13,14",if i was a director,if I was a director,"9,10,11,12,13",4.442500000000001,5.702500000000001,1.195583991709281e-10,5,1.0,0.0025706887245178223,0.00287705659866333,grouped_occlusion,grouped_occlusion,ok -12,text,6,30,35,"30,31,32,33,34",let their credit sa ##g,"let their credit sag,","25,26,27,28",9.2625,10.7425,2.0235579326793276e-13,4,1.0,0.07106262445449829,0.21757328510284424,grouped_occlusion,grouped_occlusion,ok -12,text,3,15,20,"15,16,17,18,19","fault of their own ,","fault of their own,","12,13,14,15",5.5825000000000005,6.3825,1.8391183721102136e-13,4,1.0,0.060820698738098145,0.17711377143859863,grouped_occlusion,grouped_occlusion,ok -12,text,4,20,25,"20,21,22,23,24",or perhaps by fault of,or perhaps by fault of,"16,17,18,19,20",6.4225,8.1025,1.301446420968425e-12,5,1.0,0.05820423364639282,0.14634323120117188,grouped_occlusion,grouped_occlusion,ok -12,audio,5,25,30,"25,26,27,28,29","their own , they have","their own, they have","21,22,23,24",8.1425,9.2225,2.640871901066012e-13,4,1.0,0.021765828132629395,0.05425441265106201,grouped_occlusion,grouped_occlusion,ok -12,audio,1,5,10,"5,6,7,8,9",individual previously had good credit,"individual previously had good credit,","3,4,5,6,7",1.2625,3.4425000000000003,1.981996783456316e-13,5,1.0,0.020217180252075195,0.01235973834991455,grouped_occlusion,grouped_occlusion,ok -12,audio,2,10,15,"10,11,12,13,14",", but usually by no","credit, but usually by no","7,8,9,10,11",3.0825000000000005,5.4625,5.096125070013053e-13,5,1.0,0.01965177059173584,0.05803334712982178,grouped_occlusion,grouped_occlusion,ok -12,vision,0,0,5,"1,2,3,4","or worse , an","Or worse, an","0,1,2",0.5824999999999999,1.2425,1.2646860654133281e-13,3,1.0,0.00710904598236084,0.015213608741760254,grouped_occlusion,grouped_occlusion,ok -12,vision,1,5,10,"5,6,7,8,9",individual previously had good credit,"individual previously had good credit,","3,4,5,6,7",1.2625,3.4425000000000003,1.981996783456316e-13,5,1.0,0.0066879987716674805,0.025673270225524902,grouped_occlusion,grouped_occlusion,ok -12,vision,2,10,15,"10,11,12,13,14",", but usually by no","credit, but usually by no","7,8,9,10,11",3.0825000000000005,5.4625,5.096125070013053e-13,5,1.0,0.005474686622619629,0.0271909236907959,grouped_occlusion,grouped_occlusion,ok -13,text,1,5,10,"5,6,7,8,9",could take a set of,could take a set of,"3,4,5,6,7",0.9624999999999999,1.8225,1.1293682043937327e-10,5,1.0,0.03829997777938843,0.08469116687774658,grouped_occlusion,grouped_occlusion,ok -13,text,0,0,5,"1,2,3,4","for example , i","For example, I","0,1,2",0.0025000000000000005,0.9025,3.8531617620664426e-12,3,1.0,0.03210568428039551,0.017864346504211426,grouped_occlusion,grouped_occlusion,ok -13,text,4,20,25,"20,21,22,23,24",relationship between any two of,relationship between any two of,"17,18,19,20,21",5.0825000000000005,6.782500000000001,7.070703467858767e-12,5,1.0,0.01825752854347229,0.03968304395675659,grouped_occlusion,grouped_occlusion,ok -13,audio,4,20,25,"20,21,22,23,24",relationship between any two of,relationship between any two of,"17,18,19,20,21",5.0825000000000005,6.782500000000001,7.070703467858767e-12,5,1.0,0.01745942234992981,0.03462590277194977,grouped_occlusion,grouped_occlusion,ok -13,audio,2,10,15,"10,11,12,13,14",data and from that data,"data and from that data,","8,9,10,11,12",1.8425,4.2625,9.633590059405473e-12,5,1.0,0.01131179928779602,0.007808178663253784,grouped_occlusion,grouped_occlusion,ok -13,audio,0,0,5,"1,2,3,4","for example , i","For example, I","0,1,2",0.0025000000000000005,0.9025,3.8531617620664426e-12,3,1.0,0.006338447332382202,0.00720486044883728,grouped_occlusion,grouped_occlusion,ok -14,text,2,10,15,"10,11,12,13,14",and vet ##tes ##e limited,and Vettese Limited,"6,7,8",2.0025000000000004,2.8425000000000002,4.791347081170585e-13,3,1.0,0.04705315828323364,0.09162890911102295,grouped_occlusion,grouped_occlusion,ok -14,text,4,20,25,"20,21,22,23,24",of uniform final examination (,of Uniform Final Examination (UFE),"14,15,16,17,18",4.8425,6.9225,5.581173689339982e-13,5,1.0,0.040479302406311035,0.05692708492279053,grouped_occlusion,grouped_occlusion,ok -14,text,0,0,5,"1,2,3,4",he is the co,He is the co-founder,"0,1,2,3",0.5025,1.3225,1.8809520903830734e-13,4,1.0,0.0321732759475708,0.08200758695602417,grouped_occlusion,grouped_occlusion,ok -14,audio,5,25,30,"25,26,27,28,29",u ##fe ) courses at,(UFE) courses at,"18,19,20",6.742500000000001,8.362499999999999,1.0878076875022951e-13,3,1.0,0.02120727300643921,0.04397076368331909,grouped_occlusion,grouped_occlusion,ok -14,audio,2,10,15,"10,11,12,13,14",and vet ##tes ##e limited,and Vettese Limited,"6,7,8",2.0025000000000004,2.8425000000000002,4.791347081170585e-13,3,1.0,0.020826101303100586,0.029214560985565186,grouped_occlusion,grouped_occlusion,ok -14,audio,1,5,10,"5,6,7,8,9",- founder of ross ##en,co-founder of Rossen,"3,4,5",0.7825,1.9224999999999999,6.402270448983464e-13,3,1.0,0.017303526401519775,0.04407292604446411,grouped_occlusion,grouped_occlusion,ok -14,vision,6,30,35,"30,31,32,33,34",toronto ' s york university,Toronto's York University.,"21,22,23",8.3825,9.6625,3.617856135575942e-12,3,1.0,0.03959810733795166,0.06995487213134766,grouped_occlusion,grouped_occlusion,ok -14,vision,5,25,30,"25,26,27,28,29",u ##fe ) courses at,(UFE) courses at,"18,19,20",6.742500000000001,8.362499999999999,1.0878076875022951e-13,3,1.0,0.03660660982131958,0.030756652355194092,grouped_occlusion,grouped_occlusion,ok -14,vision,4,20,25,"20,21,22,23,24",of uniform final examination (,of Uniform Final Examination (UFE),"14,15,16,17,18",4.8425,6.9225,5.581173689339982e-13,5,1.0,0.03492516279220581,0.029532790184020996,grouped_occlusion,grouped_occlusion,ok -15,text,3,15,20,"15,16,17,18,19",believed in its power to,believed in its power to,"10,11,12,13,14",4.602500000000001,6.3825,1.3666649254190005e-11,5,1.0,0.03316134214401245,0.08834660053253174,grouped_occlusion,grouped_occlusion,ok -15,text,2,10,15,"10,11,12,13,14",##iti ##ze education because they,prioritize education because they,"6,7,8,9",2.4225000000000003,4.522500000000001,5.053813400666634e-14,4,1.0,0.022731482982635498,0.05859649181365967,grouped_occlusion,grouped_occlusion,ok -15,text,1,5,10,"5,6,7,8,9","poverty , the family prior","poverty, the family prioritize","3,4,5,6",1.3025,2.9825000000000004,6.197984013383666e-14,4,1.0,0.013731598854064941,0.04108023643493652,grouped_occlusion,grouped_occlusion,ok -15,audio,0,0,5,"1,2,3,4","however , despite their","However, despite their","0,1,2",0.10250000000000001,1.2825,3.8566196054618246e-14,3,1.0,0.008936703205108643,0.06280517578125,grouped_occlusion,grouped_occlusion,ok -15,audio,1,5,10,"5,6,7,8,9","poverty , the family prior","poverty, the family prioritize","3,4,5,6",1.3025,2.9825000000000004,6.197984013383666e-14,4,1.0,0.007372438907623291,0.04567360877990723,grouped_occlusion,grouped_occlusion,ok -15,audio,2,10,15,"10,11,12,13,14",##iti ##ze education because they,prioritize education because they,"6,7,8,9",2.4225000000000003,4.522500000000001,5.053813400666634e-14,4,1.0,0.004992425441741943,0.07340216636657715,grouped_occlusion,grouped_occlusion,ok -15,vision,0,0,5,"1,2,3,4","however , despite their","However, despite their","0,1,2",0.10250000000000001,1.2825,3.8566196054618246e-14,3,1.0,0.006294310092926025,0.04669678211212158,grouped_occlusion,grouped_occlusion,ok -15,vision,2,10,15,"10,11,12,13,14",##iti ##ze education because they,prioritize education because they,"6,7,8,9",2.4225000000000003,4.522500000000001,5.053813400666634e-14,4,1.0,0.003808140754699707,0.05000507831573486,grouped_occlusion,grouped_occlusion,ok -15,vision,1,5,10,"5,6,7,8,9","poverty , the family prior","poverty, the family prioritize","3,4,5,6",1.3025,2.9825000000000004,6.197984013383666e-14,4,1.0,0.0037418007850646973,0.05190694332122803,grouped_occlusion,grouped_occlusion,ok -16,text,1,5,10,"5,6,7,8,9","terrible , this is a","terrible, this is a","2,3,4,5",0.8025,1.9224999999999999,6.726967737989785e-13,4,1.0,0.17667824029922485,0.8273677825927734,grouped_occlusion,grouped_occlusion,ok -16,text,0,0,5,"1,2,3,4",it ' s a,It's a,"0,1",0.4625,0.7224999999999999,5.886124124070357e-10,2,1.0,0.08441895246505737,0.6347545385360718,grouped_occlusion,grouped_occlusion,ok -16,text,2,10,15,"10,11",terrible movie,terrible movie,"6,7",2.0625000000000004,2.8425000000000002,3.447706729088321e-13,2,1.0,0.012092411518096924,0.09678518772125244,grouped_occlusion,grouped_occlusion,ok -16,audio,1,5,10,"5,6,7,8,9","terrible , this is a","terrible, this is a","2,3,4,5",0.8025,1.9224999999999999,6.726967737989785e-13,4,1.0,0.005522668361663818,0.07541024684906006,grouped_occlusion,grouped_occlusion,ok -16,audio,0,0,5,"1,2,3,4",it ' s a,It's a,"0,1",0.4625,0.7224999999999999,5.886124124070357e-10,2,1.0,0.004132866859436035,0.0680011510848999,grouped_occlusion,grouped_occlusion,ok -16,audio,2,10,15,"10,11",terrible movie,terrible movie,"6,7",2.0625000000000004,2.8425000000000002,3.447706729088321e-13,2,1.0,0.0015830397605895996,0.02998960018157959,grouped_occlusion,grouped_occlusion,ok -16,vision,1,5,10,"5,6,7,8,9","terrible , this is a","terrible, this is a","2,3,4,5",0.8025,1.9224999999999999,6.726967737989785e-13,4,1.0,0.0030817389488220215,0.010976672172546387,grouped_occlusion,grouped_occlusion,ok -16,vision,0,0,5,"1,2,3,4",it ' s a,It's a,"0,1",0.4625,0.7224999999999999,5.886124124070357e-10,2,1.0,0.0018818974494934082,0.008708953857421875,grouped_occlusion,grouped_occlusion,ok -16,vision,2,10,15,"10,11",terrible movie,terrible movie,"6,7",2.0625000000000004,2.8425000000000002,3.447706729088321e-13,2,1.0,0.0007146596908569336,0.0037450790405273438,grouped_occlusion,grouped_occlusion,ok -17,text,3,15,20,"15,16,17,18,19",make people think you turned,make people think you turned,"13,14,15,16,17",6.982500000000001,8.202499999999999,5.0308304651950566e-14,5,1.0,0.03196984529495239,0.10473096370697021,grouped_occlusion,grouped_occlusion,ok -17,text,4,20,25,"20,21,22,23,24",into a design guru .,into a design guru.,"18,19,20,21",8.3025,9.5425,6.876211439212395e-12,4,1.0,0.025771260261535645,0.09248185157775879,grouped_occlusion,grouped_occlusion,ok -17,text,2,10,15,"10,11,12,13,14","simple , easy and will","simple, easy and will","9,10,11,12",4.8025,6.942500000000001,8.080162655757211e-13,4,1.0,0.02379608154296875,0.15895986557006836,grouped_occlusion,grouped_occlusion,ok -17,audio,0,0,5,"1,2,3,4",applying these four design,Applying these four design,"0,1,2,3",0.0025000000000000005,2.1225000000000005,3.907075057276878e-13,4,1.0,0.001038670539855957,0.02217411994934082,grouped_occlusion,grouped_occlusion,ok -17,audio,1,5,10,"5,6,7,8,9",concepts to your presentations is,concepts to your presentations is,"4,5,6,7,8",2.1825000000000006,4.4225,1.3654575542430864e-13,5,1.0,0.0008327364921569824,0.029980182647705078,grouped_occlusion,grouped_occlusion,ok -17,audio,4,20,25,"20,21,22,23,24",into a design guru .,into a design guru.,"18,19,20,21",8.3025,9.5425,6.876211439212395e-12,4,1.0,0.0007992982864379883,0.01889348030090332,grouped_occlusion,grouped_occlusion,ok -17,vision,3,15,20,"15,16,17,18,19",make people think you turned,make people think you turned,"13,14,15,16,17",6.982500000000001,8.202499999999999,5.0308304651950566e-14,5,1.0,0.003390192985534668,0.05066335201263428,grouped_occlusion,grouped_occlusion,ok -17,vision,2,10,15,"10,11,12,13,14","simple , easy and will","simple, easy and will","9,10,11,12",4.8025,6.942500000000001,8.080162655757211e-13,4,1.0,0.0014863014221191406,0.04471385478973389,grouped_occlusion,grouped_occlusion,ok -17,vision,4,20,25,"20,21,22,23,24",into a design guru .,into a design guru.,"18,19,20,21",8.3025,9.5425,6.876211439212395e-12,4,1.0,0.0008729696273803711,0.02221369743347168,grouped_occlusion,grouped_occlusion,ok -18,text,0,0,5,"1,2,3,4",- and in denmark,-And in Denmark,"0,1,2",0.5625,1.2025,7.339091369947377e-14,3,0.9565217391304348,0.040209442377090454,0.07814651727676392,grouped_occlusion,grouped_occlusion,ok -18,text,1,5,10,"6,7,8,9",the first baltic cod,the first Baltic Cod,"4,5,6,7",1.2825,2.7225000000000006,1.9238576151557236e-12,4,0.9565217391304348,0.03645741939544678,0.07523351907730103,grouped_occlusion,grouped_occlusion,ok -18,text,7,35,40,"35,36,37,38,39",fact that the fishery has,fact that the fishery has,"27,28,29,30,31",11.4625,12.6625,3.051134360523701e-14,5,0.9565217391304348,0.02193853259086609,0.09061205387115479,grouped_occlusion,grouped_occlusion,ok -18,audio,1,5,10,"6,7,8,9",the first baltic cod,the first Baltic Cod,"4,5,6,7",1.2825,2.7225000000000006,1.9238576151557236e-12,4,0.9565217391304348,0.011634111404418945,0.011849582195281982,grouped_occlusion,grouped_occlusion,ok -18,audio,3,15,20,"15,16,17,18,19","certified - meanwhile , the","certified -Meanwhile, the","13,14,15",4.522500000000001,7.242500000000001,2.0090313043947478e-13,3,0.9565217391304348,0.009550690650939941,0.00717240571975708,grouped_occlusion,grouped_occlusion,ok -18,audio,4,20,25,"20,21,22,23,24",fae ##ro ##ese mack ##ere,Faeroese Mackerel,"16,17",7.2625,8.1625,7.552127939731886e-13,2,0.9565217391304348,0.0075453221797943115,0.0038139820098876953,grouped_occlusion,grouped_occlusion,ok -18,vision,3,15,20,"15,16,17,18,19","certified - meanwhile , the","certified -Meanwhile, the","13,14,15",4.522500000000001,7.242500000000001,2.0090313043947478e-13,3,0.9565217391304348,0.015370607376098633,0.00964266061782837,grouped_occlusion,grouped_occlusion,ok -18,vision,4,20,25,"20,21,22,23,24",fae ##ro ##ese mack ##ere,Faeroese Mackerel,"16,17",7.2625,8.1625,7.552127939731886e-13,2,0.9565217391304348,0.01368647813796997,0.025389909744262695,grouped_occlusion,grouped_occlusion,ok -18,vision,7,35,40,"35,36,37,38,39",fact that the fishery has,fact that the fishery has,"27,28,29,30,31",11.4625,12.6625,3.051134360523701e-14,5,0.9565217391304348,0.012452512979507446,0.0016064047813415527,grouped_occlusion,grouped_occlusion,ok -19,text,5,25,30,"25,26,27,28,29",and jump all over imperfect,"and jump all over imperfections,","21,22,23,24,25",8.9225,10.7025,2.4467552179386886e-13,5,1.0,0.14123356342315674,0.26041585206985474,grouped_occlusion,grouped_occlusion,ok -19,text,2,10,15,"10,11,12,13,14","up to mistakes , and","up to mistakes, and","8,9,10,11",3.6025000000000005,5.2625,2.756182849760793e-14,4,1.0,0.1259310245513916,0.27155396342277527,grouped_occlusion,grouped_occlusion,ok -19,text,1,5,10,"5,6,7,8,9",##giving brands when they own,forgiving brands when they own,"3,4,5,6,7",1.8025,3.4025000000000003,4.482843688564828e-15,5,1.0,0.10481253266334534,0.20342162251472473,grouped_occlusion,grouped_occlusion,ok -19,audio,5,25,30,"25,26,27,28,29",and jump all over imperfect,"and jump all over imperfections,","21,22,23,24,25",8.9225,10.7025,2.4467552179386886e-13,5,1.0,0.0375896692276001,0.0982179045677185,grouped_occlusion,grouped_occlusion,ok -19,audio,7,35,40,"35,36,37,38,39","most part , people understand","most part, people understand","29,30,31,32",11.8425,13.5625,6.956549052765396e-14,4,1.0,0.03040042519569397,0.054060935974121094,grouped_occlusion,grouped_occlusion,ok -19,audio,4,20,25,"20,21,22,23,24",there love to point fingers,there love to point fingers,"16,17,18,19,20",6.942500000000001,8.862499999999999,2.039331089220153e-13,5,1.0,0.021963000297546387,0.04821614921092987,grouped_occlusion,grouped_occlusion,ok -19,vision,3,15,20,"15,16,17,18,19",unfortunately some hate ##rs out,unfortunately some haters out,"12,13,14,15",5.3425,6.9225,4.614146997521066e-14,4,1.0,0.031420767307281494,0.07558748126029968,grouped_occlusion,grouped_occlusion,ok -19,vision,1,5,10,"5,6,7,8,9",##giving brands when they own,forgiving brands when they own,"3,4,5,6,7",1.8025,3.4025000000000003,4.482843688564828e-15,5,1.0,0.029108166694641113,0.07041062414646149,grouped_occlusion,grouped_occlusion,ok -19,vision,5,25,30,"25,26,27,28,29",and jump all over imperfect,"and jump all over imperfections,","21,22,23,24,25",8.9225,10.7025,2.4467552179386886e-13,5,1.0,0.02759939432144165,0.028621017932891846,grouped_occlusion,grouped_occlusion,ok -20,text,7,35,40,"35,36,37,38,39",) at the end of,(wrap-up) at the end of,"26,27,28,29,30",8.5025,10.1025,1.1233773471618156e-12,5,1.0,0.022437691688537598,0.02704167366027832,grouped_occlusion,grouped_occlusion,ok -20,text,0,0,5,"1,2,3,4","and of course ,","And of course,","0,1,2",0.0225,1.0425,7.323313211132221e-13,3,1.0,0.01609170436859131,0.03773140907287598,grouped_occlusion,grouped_occlusion,ok -20,text,5,25,30,"25,26,27,28,29",updates and a stock market,updates and a stock market,"20,21,22,23,24",6.6625000000000005,8.1625,6.493302127844331e-12,5,1.0,0.01604229211807251,0.03772282600402832,grouped_occlusion,grouped_occlusion,ok -20,audio,8,40,45,"40,41,42,43",the day today .,the day today.,"31,32,33",10.1425,10.8025,5.934335502395015e-12,3,1.0,0.0015968680381774902,0.016044139862060547,grouped_occlusion,grouped_occlusion,ok -20,audio,3,15,20,"15,16,17,18,19","for more , and we","for more, and we'll","13,14,15,16",4.3825,5.522500000000001,1.612313071618996e-11,4,1.0,0.0011889338493347168,0.01394963264465332,grouped_occlusion,grouped_occlusion,ok -20,audio,1,5,10,"5,6,7,8,9",click in the link of,click in the link of,"3,4,5,6,7",1.1025,2.1425000000000005,2.2686710998425595e-12,5,1.0,0.0008327364921569824,0.015181779861450195,grouped_occlusion,grouped_occlusion,ok -20,vision,2,10,15,"10,11,12,13,14",the description of this video,the description of this video,"8,9,10,11,12",2.2425000000000006,3.8225000000000007,1.1555191343633454e-11,5,1.0,0.006613016128540039,0.03737950325012207,grouped_occlusion,grouped_occlusion,ok -20,vision,3,15,20,"15,16,17,18,19","for more , and we","for more, and we'll","13,14,15,16",4.3825,5.522500000000001,1.612313071618996e-11,4,1.0,0.004205465316772461,0.032741665840148926,grouped_occlusion,grouped_occlusion,ok -20,vision,1,5,10,"5,6,7,8,9",click in the link of,click in the link of,"3,4,5,6,7",1.1025,2.1425000000000005,2.2686710998425595e-12,5,1.0,0.0037073493003845215,0.016587018966674805,grouped_occlusion,grouped_occlusion,ok diff --git a/deep_learning/Q3/outputs/explanation_selection/q3_deletion_curves.csv b/deep_learning/Q3/outputs/explanation_selection/q3_deletion_curves.csv deleted file mode 100644 index 858619c..0000000 --- a/deep_learning/Q3/outputs/explanation_selection/q3_deletion_curves.csv +++ /dev/null @@ -1,37 +0,0 @@ -method,target,fraction_removed,mean_signed_drop,mean_absolute_change,n_valid -integrated_gradients,predicted_class_probability,0.1,0.1318911910057068,0.1598358154296875,728 -integrated_gradients,predicted_class_probability,0.2,0.2255048155784607,0.2508196234703064,728 -integrated_gradients,predicted_class_probability,0.3,0.2575814723968506,0.2828781306743622,728 -integrated_gradients,predicted_class_probability,0.4,0.2559848725795746,0.28699347376823425,728 -integrated_gradients,predicted_class_probability,0.5,0.25572627782821655,0.29257047176361084,728 -integrated_gradients,predicted_class_probability,-0.3,0.039929818361997604,0.039929818361997604,728 -integrated_gradients,intensity,0.1,-0.04874260723590851,0.37119486927986145,728 -integrated_gradients,intensity,0.2,-0.03821782395243645,0.5629051327705383,728 -integrated_gradients,intensity,0.3,-0.05994963273406029,0.6319279074668884,728 -integrated_gradients,intensity,0.4,-0.13912875950336456,0.653501570224762,728 -integrated_gradients,intensity,0.5,-0.20183886587619781,0.6705618500709534,728 -integrated_gradients,intensity,-0.3,0.1144869476556778,0.1144869476556778,728 -grouped_occlusion,predicted_class_probability,0.1,0.18875761330127716,0.20476296544075012,728 -grouped_occlusion,predicted_class_probability,0.2,0.2506031095981598,0.2624903619289398,728 -grouped_occlusion,predicted_class_probability,0.3,0.26046913862228394,0.27427902817726135,728 -grouped_occlusion,predicted_class_probability,0.4,0.2578567862510681,0.27872106432914734,728 -grouped_occlusion,predicted_class_probability,0.5,0.2540586292743683,0.2836124897003174,728 -grouped_occlusion,predicted_class_probability,-0.3,0.024163084104657173,0.024163084104657173,728 -grouped_occlusion,intensity,0.1,-0.056311462074518204,0.4420826733112335,728 -grouped_occlusion,intensity,0.2,-0.08403468877077103,0.5550013184547424,728 -grouped_occlusion,intensity,0.3,-0.13223451375961304,0.5881773829460144,728 -grouped_occlusion,intensity,0.4,-0.20568883419036865,0.6224051117897034,728 -grouped_occlusion,intensity,0.5,-0.2683386206626892,0.6487718820571899,728 -grouped_occlusion,intensity,-0.3,0.0703125074505806,0.0703125074505806,728 -random,predicted_class_probability,0.1,0.017329903319478035,0.03110884316265583,728 -random,predicted_class_probability,0.2,0.0376301035284996,0.05323585495352745,728 -random,predicted_class_probability,0.3,0.057446520775556564,0.07715633511543274,728 -random,predicted_class_probability,0.4,0.07688438147306442,0.09651552885770798,728 -random,predicted_class_probability,0.5,0.0910244807600975,0.11337775737047195,728 -random,predicted_class_probability,-0.3,0.16268795728683472,0.16268795728683472,728 -random,intensity,0.1,-0.01041108462959528,0.07401501387357712,728 -random,intensity,0.2,-0.012165687046945095,0.12958525121212006,728 -random,intensity,0.3,-0.031106332316994667,0.18503554165363312,728 -random,intensity,0.4,-0.03734554722905159,0.2288973033428192,728 -random,intensity,0.5,-0.04232712835073471,0.2670633792877197,728 -random,intensity,-0.3,0.37735068798065186,0.37735068798065186,728 diff --git a/deep_learning/Q3/outputs/explanation_selection/q3_explainer_selection.json b/deep_learning/Q3/outputs/explanation_selection/q3_explainer_selection.json deleted file mode 100644 index f1aa63f..0000000 --- a/deep_learning/Q3/outputs/explanation_selection/q3_explainer_selection.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "q2_predictor": "concat", - "classification_explainer": "grouped_occlusion", - "intensity_explainer_primary": "grouped_occlusion", - "intensity_explainer_crosscheck": "integrated_gradients", - "intensity_tradeoff": { - "integrated_gradients_abs_change_at_30": 0.6319279074668884, - "integrated_gradients_sufficiency_error_top_30": 0.1144869476556778, - "grouped_occlusion_abs_change_at_30": 0.5881773829460144, - "grouped_occlusion_sufficiency_error_top_30": 0.0703125074505806 - }, - "selection_basis": "For polarity, grouped occlusion has the larger signed target-probability drop, lower sufficiency error, higher deletion AUC, and lower runtime. For intensity, IG causes a larger deletion change but grouped occlusion has lower top-evidence sufficiency error; grouped occlusion is used for displayed segments and IG is retained as a cross-check. No combined explanation score is used.", - "valid_samples": 728, - "stability_samples": 120, - "integrated_gradients_runtime_seconds": 4.718544340998051, - "grouped_occlusion_runtime_seconds": 0.40209812500688713, - "ctc_time_map_for_attachment4": "Q1 hard CTC Viterbi word boundaries from source video audio; not human alignment ground truth" -} \ No newline at end of file diff --git a/deep_learning/Q3/outputs/explanation_selection/q3_explanation_faithfulness.png b/deep_learning/Q3/outputs/explanation_selection/q3_explanation_faithfulness.png deleted file mode 100644 index d4b2998..0000000 Binary files a/deep_learning/Q3/outputs/explanation_selection/q3_explanation_faithfulness.png and /dev/null differ diff --git a/deep_learning/Q3/outputs/explanation_selection/q3_explanation_method_summary.csv b/deep_learning/Q3/outputs/explanation_selection/q3_explanation_method_summary.csv deleted file mode 100644 index 5e7c186..0000000 --- a/deep_learning/Q3/outputs/explanation_selection/q3_explanation_method_summary.csv +++ /dev/null @@ -1,7 +0,0 @@ -method,target,comprehensiveness_signed_drop_at_30,absolute_prediction_change_at_30,sufficiency_abs_error_top_30,deletion_drop_auc_10_to_50,runtime_seconds,spearman_rank_correlation,top_30_percent_jaccard,n_samples,input_noise_sigma -integrated_gradients,predicted_class_probability,0.2575814723968506,0.2828781306743622,0.039929818361997604,0.09328798949718475,4.718544340998051,0.9998964070157896,0.9966666666666666,120,0.02 -integrated_gradients,intensity,-0.05994963273406029,0.6319279074668884,0.1144869476556778,-0.03625869527459145,4.718544340998051,0.9999221890615111,0.9983333333333333,120,0.02 -grouped_occlusion,predicted_class_probability,0.26046913862228394,0.27427902817726135,0.024163084104657173,0.09903371557593346,0.40209812500688713,0.9993435630984816,0.99,120,0.02 -grouped_occlusion,intensity,-0.13223451375961304,0.5881773829460144,0.0703125074505806,-0.05842830780893564,0.40209812500688713,0.9994847562909268,0.9866666666666667,120,0.02 -random,predicted_class_probability,0.057446520775556564,0.07715633511543274,0.16268795728683472,0.022613819781690837,0.40209812500688713,nan,nan,, -random,intensity,-0.031106332316994667,0.18503554165363312,0.37735068798065186,-0.010698667308315635,0.40209812500688713,nan,nan,, diff --git a/deep_learning/Q3/outputs/explanation_selection/q3_explanation_stability.csv b/deep_learning/Q3/outputs/explanation_selection/q3_explanation_stability.csv deleted file mode 100644 index 0bd2686..0000000 --- a/deep_learning/Q3/outputs/explanation_selection/q3_explanation_stability.csv +++ /dev/null @@ -1,5 +0,0 @@ -method,target,spearman_rank_correlation,top_30_percent_jaccard,n_samples,input_noise_sigma -integrated_gradients,predicted_class_probability,0.9998964070157896,0.9966666666666666,120,0.02 -grouped_occlusion,predicted_class_probability,0.9993435630984816,0.99,120,0.02 -integrated_gradients,intensity,0.9999221890615111,0.9983333333333333,120,0.02 -grouped_occlusion,intensity,0.9994847562909268,0.9866666666666667,120,0.02 diff --git a/deep_learning/Q3/q3/__init__.py b/deep_learning/Q3/q3/__init__.py deleted file mode 100644 index db72d13..0000000 --- a/deep_learning/Q3/q3/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Q3 interpretation and evidence-localization experiments.""" diff --git a/deep_learning/Q3/q3/ctc_time.py b/deep_learning/Q3/q3/ctc_time.py deleted file mode 100644 index 79cf83d..0000000 --- a/deep_learning/Q3/q3/ctc_time.py +++ /dev/null @@ -1,148 +0,0 @@ -from __future__ import annotations - -import re -import subprocess -from dataclasses import dataclass -from pathlib import Path - -import numpy as np -import torch -from transformers import AutoModelForCTC, AutoTokenizer - - -SAMPLE_RATE = 16_000 -MODEL_ID = "facebook/wav2vec2-base-960h" - - -@dataclass -class WordInterval: - word: str - start_s: float - end_s: float - quality: float - valid: bool - - -def decode_audio(video_path: Path) -> np.ndarray: - result = subprocess.run( - [ - "ffmpeg", "-v", "error", "-i", str(video_path), "-map", "0:a:0", - "-ac", "1", "-ar", str(SAMPLE_RATE), "-f", "f32le", "pipe:1", - ], - check=True, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - ) - waveform = np.frombuffer(result.stdout, dtype=" tuple[list[int], list[list[int]]]: - vocab = tokenizer.get_vocab() - delimiter = int(tokenizer.convert_tokens_to_ids(tokenizer.word_delimiter_token or "|")) - unknown = int(tokenizer.unk_token_id) - targets: list[int] = [] - per_word: list[list[int]] = [[] for _ in words] - for word_index, raw_word in enumerate(words): - if word_index: - targets.append(delimiter) - normalized = re.sub(r"[^a-z']", "", raw_word.lower()) - for character in normalized: - per_word[word_index].append(len(targets)) - targets.append(int(vocab.get(character, unknown))) - return targets, per_word - - -def _viterbi(log_probs: np.ndarray, targets: list[int], blank: int) -> np.ndarray | None: - if not targets or log_probs.ndim != 2: - return None - states = np.full(2 * len(targets) + 1, blank, dtype=np.int64) - states[1::2] = np.asarray(targets, dtype=np.int64) - frames, count = log_probs.shape[0], len(states) - if frames == 0 or frames < len(targets): - return None - previous = np.full(count, -np.inf, dtype=np.float64) - previous[0] = float(log_probs[0, blank]) - previous[1] = float(log_probs[0, states[1]]) - back = np.zeros((frames, count), dtype=np.uint8) - skip = np.zeros(count, dtype=bool) - if count > 2: - skip[2:] = (states[2:] != blank) & (states[2:] != states[:-2]) - for frame in range(1, frames): - stay = previous - one = np.full(count, -np.inf, dtype=np.float64) - one[1:] = previous[:-1] - two = np.full(count, -np.inf, dtype=np.float64) - if skip.any(): - two[skip] = previous[np.flatnonzero(skip) - 2] - candidates = np.stack((stay, one, two), axis=0) - choice = candidates.argmax(axis=0).astype(np.uint8) - previous = candidates[choice, np.arange(count)] + log_probs[frame, states] - back[frame] = choice - state = count - 1 if previous[-1] >= previous[-2] else count - 2 - path = np.empty(frames, dtype=np.int32) - path[-1] = state - for frame in range(frames - 1, 0, -1): - state -= int(back[frame, state]) - path[frame - 1] = state - return path - - -def model_time_constants(model) -> tuple[float, float]: - config = model.config - stride = int(np.prod(config.conv_stride)) - receptive = 1 - jump = 1 - for kernel, local_stride in zip(config.conv_kernel, config.conv_stride): - receptive += (int(kernel) - 1) * jump - jump *= int(local_stride) - return stride / SAMPLE_RATE, receptive / (2 * SAMPLE_RATE) - - -def align_words( - waveform: np.ndarray, - words: list[str], - tokenizer, - model, - device: torch.device, -) -> list[WordInterval]: - targets, word_targets = _ctc_targets(words, tokenizer) - blank = int(tokenizer.pad_token_id) - if not targets or not len(waveform): - return [WordInterval(w, float("nan"), float("nan"), 0.0, False) for w in words] - with torch.inference_mode(): - values = torch.as_tensor(waveform, dtype=torch.float32, device=device).unsqueeze(0) - logits = model(input_values=values).logits[0].float() - log_probs = torch.log_softmax(logits, dim=-1).cpu().numpy() - path = _viterbi(log_probs, targets, blank) - frame_step, center_s = model_time_constants(model) - duration = len(waveform) / SAMPLE_RATE - intervals: list[WordInterval] = [] - if path is None: - return [WordInterval(w, float("nan"), float("nan"), 0.0, False) for w in words] - for word, target_indices in zip(words, word_targets): - states = np.asarray([2 * index + 1 for index in target_indices], dtype=np.int32) - frame_indices = np.flatnonzero(np.isin(path, states)) if len(states) else np.empty(0, dtype=np.int64) - if not len(frame_indices): - intervals.append(WordInterval(word, float("nan"), float("nan"), 0.0, False)) - continue - first, last = int(frame_indices[0]), int(frame_indices[-1]) - start = max(0.0, first * frame_step + center_s - frame_step / 2) - end = min(duration, (last + 1) * frame_step + center_s - frame_step / 2) - char_scores = [] - for target_index in target_indices: - selected = np.flatnonzero(path == 2 * target_index + 1) - if len(selected): - char_scores.extend(log_probs[selected, targets[target_index]].tolist()) - quality = float(np.exp(np.mean(char_scores))) if char_scores else 0.0 - valid = end > start - intervals.append(WordInterval(word, start, end, quality, valid)) - return intervals - - -def load_ctc(device: torch.device): - tokenizer = AutoTokenizer.from_pretrained(MODEL_ID) - model = AutoModelForCTC.from_pretrained(MODEL_ID).to(device).eval() - return tokenizer, model diff --git a/deep_learning/Q3/q3/explain_selection.py b/deep_learning/Q3/q3/explain_selection.py deleted file mode 100644 index 4ce6177..0000000 --- a/deep_learning/Q3/q3/explain_selection.py +++ /dev/null @@ -1,622 +0,0 @@ -from __future__ import annotations - -import argparse -import csv -import json -import pickle -import re -import sys -import time -from pathlib import Path -from typing import Any - -Q2_PROJECT = Path(__file__).resolve().parents[2] / "Q2" -sys.path.insert(0, str(Q2_PROJECT)) - -import matplotlib -matplotlib.use("Agg") -import matplotlib.pyplot as plt -import numpy as np -import torch -from scipy.stats import spearmanr -from transformers import AutoTokenizer - -from .ctc_time import align_words, decode_audio, load_ctc -from q2.data import MODALITIES, ROOT, RobustStats, Split, apply_robust_stats, load_aligned -from q2.models import AlignedFusionModel -from q2.train_compare import _score_arrays, _write_csv - - -ATTACHMENT4 = ROOT / "E题数据" / "附件4-可解释专项视频样本与特征文件" / "附件4-可解释专项视频样本与特征文件" / "对齐版本" -MODALITY_LABELS = {0: "text", 1: "audio", 2: "vision"} -CLASS_NAMES = {0: "Negative", 1: "Neutral", 2: "Positive"} -BLOCK = 5 -N_BLOCKS = 50 // BLOCK - - -def _model_from_run(output: Path, device: torch.device): - method = (output / "selected_method.txt").read_text(encoding="utf-8").split(":", 1)[1].split(".", 1)[0].strip() - checkpoint = torch.load(output / "models" / "aligned" / method / "model_best.pt", map_location=device, weights_only=False) - model = AlignedFusionModel(method, tuple(checkpoint["dims"])).to(device) - model.load_state_dict(checkpoint["state_dict"]) - model.eval() - return method, model - - -def _selected_scores( - model: AlignedFusionModel, - split: Split, - mask: np.ndarray, - device: torch.device, - batch: int = 128, - target_classes: np.ndarray | None = None, -): - model.eval() - all_logits, all_reg = [], [] - with torch.inference_mode(): - for start in range(0, split.n, batch): - stop = min(start + batch, split.n) - xs = tuple(torch.as_tensor(x[start:stop], dtype=torch.float32, device=device) for x in split.x) - mb = torch.as_tensor(mask[start:stop], dtype=torch.bool, device=device) - result = model(xs, mb) - all_logits.append(result["logits"].float().cpu().numpy()) - all_reg.append(result["intensity"].float().cpu().numpy()) - logits = np.concatenate(all_logits) - intensity = np.clip(np.concatenate(all_reg), -3.0, 3.0) - pred_class = logits.argmax(axis=-1) - selected_class = pred_class if target_classes is None else np.asarray(target_classes, dtype=np.int64) - prob = torch.softmax(torch.as_tensor(logits), dim=-1).numpy()[np.arange(split.n), selected_class] - return logits, pred_class, prob, intensity - - -def _integrated_groups( - model: AlignedFusionModel, - split: Split, - masks: np.ndarray, - device: torch.device, - steps: int = 16, - batch_size: int = 48, -) -> tuple[np.ndarray, np.ndarray]: - """Absolute Integrated Gradients grouped into three modalities x ten 5-slot blocks.""" - model.eval() - class_scores = np.zeros((split.n, 3, N_BLOCKS), dtype=np.float32) - reg_scores = np.zeros_like(class_scores) - for start in range(0, split.n, batch_size): - stop = min(start + batch_size, split.n) - xb = tuple(torch.as_tensor(x[start:stop], dtype=torch.float32, device=device) for x in split.x) - mb = torch.as_tensor(masks[start:stop], dtype=torch.bool, device=device) - with torch.no_grad(): - base = model(xb, mb) - target = base["logits"].argmax(dim=-1) - grad_class = [torch.zeros_like(x) for x in xb] - grad_reg = [torch.zeros_like(x) for x in xb] - # cuDNN's fused GRU does not support backward while the module is in - # eval mode; the non-fused implementation is mathematically identical. - with torch.backends.cudnn.flags(enabled=False): - for alpha in torch.linspace(1.0 / steps, 1.0, steps, device=device): - inputs = tuple((x * alpha).detach().requires_grad_(True) for x in xb) - output = model(inputs, mb) - target_prob = torch.softmax(output["logits"], dim=-1).gather(1, target[:, None]).sum() - gradients = torch.autograd.grad(target_prob, inputs, retain_graph=True) - reg_gradients = torch.autograd.grad(output["intensity"].sum(), inputs) - for modality in range(3): - grad_class[modality] += gradients[modality].detach() - grad_reg[modality] += reg_gradients[modality].detach() - for modality in range(3): - attr_class = (xb[modality] * grad_class[modality] / steps).abs().sum(dim=-1) - attr_reg = (xb[modality] * grad_reg[modality] / steps).abs().sum(dim=-1) - attr_class = attr_class.reshape(stop - start, N_BLOCKS, BLOCK).sum(dim=-1) - attr_reg = attr_reg.reshape(stop - start, N_BLOCKS, BLOCK).sum(dim=-1) - class_scores[start:stop, modality] = attr_class.float().cpu().numpy() - reg_scores[start:stop, modality] = attr_reg.float().cpu().numpy() - print(f"[IG] explained validation rows {start}:{stop}/{split.n}", flush=True) - return class_scores, reg_scores - - -def _occlusion_groups( - model: AlignedFusionModel, - split: Split, - masks: np.ndarray, - device: torch.device, -) -> tuple[np.ndarray, np.ndarray]: - """Measure the prediction change when one aligned five-slot modality block is hidden.""" - full_logits, full_class, full_prob, full_reg = _selected_scores(model, split, masks, device) - class_scores = np.zeros((split.n, 3, N_BLOCKS), dtype=np.float32) - reg_scores = np.zeros_like(class_scores) - for modality in range(3): - for block in range(N_BLOCKS): - changed = masks.copy() - left, right = block * BLOCK, (block + 1) * BLOCK - changed[:, left:right, modality] = False - _, _, prob, reg = _selected_scores(model, split, changed, device, target_classes=full_class) - class_scores[:, modality, block] = np.abs(full_prob - prob) - reg_scores[:, modality, block] = np.abs(full_reg - reg) - print(f"[occlusion] finished {MODALITY_LABELS[modality]}", flush=True) - return class_scores, reg_scores - - -def _rank_delete_masks(base: np.ndarray, scores: np.ndarray, fraction: float, keep: bool = False) -> np.ndarray: - n, steps, modalities = base.shape - count = max(1, int(round(fraction * 3 * N_BLOCKS))) - ranked = np.argsort(-scores.reshape(n, -1), axis=1) - result = np.zeros_like(base) if keep else base.copy() - for row in range(n): - for flat_index in ranked[row, :count]: - modality, block = divmod(int(flat_index), N_BLOCKS) - left, right = block * BLOCK, (block + 1) * BLOCK - if keep: - result[row, left:right, modality] = base[row, left:right, modality] - else: - result[row, left:right, modality] = False - return result - - -def _faithfulness_curves( - model: AlignedFusionModel, - split: Split, - base_masks: np.ndarray, - explanations: dict[str, tuple[np.ndarray, np.ndarray]], - device: torch.device, - seed: int, -) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]: - logits, pred_class, full_prob, full_reg = _selected_scores(model, split, base_masks, device) - rng = np.random.default_rng(seed) - random_cls = rng.random((split.n, 3, N_BLOCKS), dtype=np.float32) - random_reg = random_cls.copy() - curve_rows: list[dict[str, Any]] = [] - for method, (class_scores, reg_scores) in [*explanations.items(), ("random", (random_cls, random_reg))]: - for target, scores in (("predicted_class_probability", class_scores), ("intensity", reg_scores)): - for fraction in (0.10, 0.20, 0.30, 0.40, 0.50): - delete_masks = _rank_delete_masks(base_masks, scores, fraction, keep=False) - _, _, after_prob, after_reg = _selected_scores(model, split, delete_masks, device, - target_classes=pred_class) - if target == "predicted_class_probability": - difference = full_prob - after_prob - abs_difference = np.abs(difference) - else: - difference = full_reg - after_reg - abs_difference = np.abs(difference) - curve_rows.append({ - "method": method, "target": target, "fraction_removed": fraction, - "mean_signed_drop": float(np.mean(difference)), - "mean_absolute_change": float(np.mean(abs_difference)), - "n_valid": split.n, - }) - keep_masks = _rank_delete_masks(base_masks, scores, 0.30, keep=True) - _, _, keep_prob, keep_reg = _selected_scores(model, split, keep_masks, device, - target_classes=pred_class) - if target == "predicted_class_probability": - sufficiency = np.abs(full_prob - keep_prob) - else: - sufficiency = np.abs(full_reg - keep_reg) - curve_rows.append({ - "method": method, "target": target, "fraction_removed": -0.30, - "mean_signed_drop": float(np.mean(sufficiency)), - "mean_absolute_change": float(np.mean(sufficiency)), - "n_valid": split.n, - }) - - summary_rows: list[dict[str, Any]] = [] - for method in explanations.keys() | {"random"}: - for target in ("predicted_class_probability", "intensity"): - local = [r for r in curve_rows if r["method"] == method and r["target"] == target] - removal30 = next(r for r in local if r["fraction_removed"] == 0.30) - sufficiency = next(r for r in local if r["fraction_removed"] == -0.30) - removal = [r for r in local if r["fraction_removed"] > 0] - auc = float(np.trapezoid([r["mean_signed_drop"] for r in removal], [r["fraction_removed"] for r in removal])) - summary_rows.append({ - "method": method, - "target": target, - "comprehensiveness_signed_drop_at_30": removal30["mean_signed_drop"], - "absolute_prediction_change_at_30": removal30["mean_absolute_change"], - "sufficiency_abs_error_top_30": sufficiency["mean_absolute_change"], - "deletion_drop_auc_10_to_50": auc, - }) - return curve_rows, summary_rows - - -def _noise_split(split: Split, seed: int, sigma: float = 0.02) -> Split: - rng = np.random.default_rng(seed) - xs = [] - for modality, x in enumerate(split.x): - noise = rng.normal(0.0, sigma, size=x.shape).astype(np.float32) - noise *= split.mask[:, :, modality, None] - xs.append((x + noise).astype(np.float32)) - return Split(tuple(xs), split.mask.copy(), split.y_cls, split.y_reg, split.ids) - - -def _balanced_subset(split: Split, count: int, seed: int) -> Split: - rng = np.random.default_rng(seed) - selected: list[int] = [] - per_class = max(1, count // 3) - for label in (0, 1, 2): - available = np.flatnonzero(split.y_cls == label) - take = min(per_class, len(available)) - selected.extend(rng.choice(available, size=take, replace=False).tolist()) - if len(selected) < count: - remaining = np.setdiff1d(np.arange(split.n), np.asarray(selected, dtype=int)) - extra = min(count - len(selected), len(remaining)) - selected.extend(rng.choice(remaining, size=extra, replace=False).tolist()) - ids = np.asarray(sorted(selected[:count]), dtype=int) - return Split(tuple(x[ids] for x in split.x), split.mask[ids], split.y_cls[ids], split.y_reg[ids], [split.ids[i] for i in ids]) - - -def _rank_stability(original: np.ndarray, changed: np.ndarray) -> tuple[float, float]: - correlations, overlaps = [], [] - n, modalities, blocks = original.shape - top_n = max(1, int(round(modalities * blocks * 0.30))) - for row in range(n): - a = original[row].reshape(-1) - b = changed[row].reshape(-1) - corr = spearmanr(a, b).statistic - correlations.append(float(corr) if np.isfinite(corr) else 0.0) - top_a = set(np.argsort(-a)[:top_n].tolist()) - top_b = set(np.argsort(-b)[:top_n].tolist()) - overlaps.append(len(top_a & top_b) / max(1, len(top_a | top_b))) - return float(np.mean(correlations)), float(np.mean(overlaps)) - - -def _plot_faithfulness(curves: list[dict[str, Any]], output: Path) -> None: - fig, axes = plt.subplots(1, 2, figsize=(10, 4.1), constrained_layout=True) - styles = {"integrated_gradients": "#4e79a7", "grouped_occlusion": "#f28e2b", "random": "#999999"} - for ax, target, title, ylabel in ( - (axes[0], "predicted_class_probability", "Polarity evidence deletion", "probability drop"), - (axes[1], "intensity", "Intensity evidence deletion", "absolute intensity change"), - ): - for method in styles: - rows = sorted([r for r in curves if r["target"] == target and r["method"] == method and r["fraction_removed"] > 0], key=lambda r: r["fraction_removed"]) - if rows: - metric = "mean_signed_drop" if target == "predicted_class_probability" else "mean_absolute_change" - ax.plot([r["fraction_removed"] for r in rows], [r[metric] for r in rows], marker="o", label=method, color=styles[method]) - ax.set(title=title, xlabel="top evidence blocks removed", ylabel=ylabel) - ax.grid(alpha=0.25) - ax.legend(frameon=False) - output.parent.mkdir(parents=True, exist_ok=True) - fig.savefig(output, dpi=180) - plt.close(fig) - - -def _word_spans(text: str) -> list[tuple[int, int, str]]: - return [(m.start(), m.end(), m.group(0)) for m in re.finditer(r"\S+", text)] - - -def _offset_to_word(offset: tuple[int, int], spans: list[tuple[int, int, str]]) -> int | None: - start, end = int(offset[0]), int(offset[1]) - if end <= start: - return None - overlaps = [max(0, min(end, right) - max(start, left)) for left, right, _ in spans] - if not overlaps or max(overlaps) == 0: - return None - return int(np.argmax(overlaps)) - - -def _attachment4_raw() -> tuple[list[dict[str, Any]], list[Path]]: - records, videos = [], [] - pkl_paths = sorted(ATTACHMENT4.glob("*.pkl")) - for path in pkl_paths: - with path.open("rb") as stream: - record = pickle.load(stream, encoding="latin1") - records.append(record) - videos.append(ATTACHMENT4 / "videos" / f"{record['id']}.mp4") - if len(records) != 20: - raise ValueError(f"expected 20 aligned Attachment 4 clips; found {len(records)} in {ATTACHMENT4}") - return records, videos - - -def _attachment4_split(records: list[dict[str, Any]]) -> Split: - xs = [[], [], []] - masks = [] - ids = [] - for record in records: - xs[0].append(np.asarray(record["text"], dtype=np.float32)) - xs[1].append(np.asarray(record["audio"], dtype=np.float32)) - xs[2].append(np.asarray(record["vision"], dtype=np.float32)) - token = np.asarray(record["text_bert"]) - masks.append(np.stack((token[1].astype(bool), np.any(record["audio"] != 0, axis=-1), np.any(record["vision"] != 0, axis=-1)), axis=-1)) - ids.append(str(record["id"])) - return Split(tuple(np.stack(x) for x in xs), np.stack(masks), np.zeros(len(records), dtype=np.int64), np.zeros(len(records), dtype=np.float32), ids) - - -def _block_value_per_slot(group_scores: np.ndarray, masks: np.ndarray) -> np.ndarray: - n = group_scores.shape[0] - slots = np.zeros((n, 3, 50), dtype=np.float32) - for modality in range(3): - for block in range(N_BLOCKS): - left, right = block * BLOCK, (block + 1) * BLOCK - active = masks[:, left:right, modality] - count = active.sum(axis=1).clip(min=1) - each = group_scores[:, modality, block] / count - slots[:, modality, left:right] = each[:, None] - return slots - - -def _run_attachment4( - model: AlignedFusionModel, - output: Path, - stats: RobustStats, - device: torch.device, - bert_tokenizer, - class_method: str, - reg_method: str, -) -> None: - records, videos = _attachment4_raw() - raw = _attachment4_split(records) - split = apply_robust_stats(raw, stats) - logits, pred_class, prob, intensity = _selected_scores(model, split, split.mask, device) - need_ig = class_method == "integrated_gradients" or reg_method == "integrated_gradients" - need_occ = class_method == "grouped_occlusion" or reg_method == "grouped_occlusion" - ig_class, ig_reg = _integrated_groups(model, split, split.mask, device) if need_ig else (None, None) - occ_class, occ_reg = _occlusion_groups(model, split, split.mask, device) if need_occ else (None, None) - class_group = ig_class if class_method == "integrated_gradients" else occ_class - reg_group = ig_reg if reg_method == "integrated_gradients" else occ_reg - - predictions: list[dict[str, Any]] = [] - evidence: list[dict[str, Any]] = [] - word_mappings: dict[str, list[dict[str, Any]]] = {} - ctc_word_coverages: list[float] = [] - ctc_tokenizer, ctc_model = load_ctc(device) - for index, (record, video_path) in enumerate(zip(records, videos)): - clip_id = str(record["id"]) - text = str(record["raw_text"]) - words = text.split() - time_status = "ok" - try: - waveform = decode_audio(video_path) - intervals = align_words(waveform, words, ctc_tokenizer, ctc_model, device) - except Exception as exc: - intervals = [] - time_status = f"ctc_failed:{type(exc).__name__}" - valid_word_count = sum(interval.valid for interval in intervals) - ctc_coverage = valid_word_count / max(1, len(words)) - ctc_word_coverages.append(ctc_coverage) - if time_status == "ok": - time_status = "ok" if ctc_coverage >= 0.95 else ("partial" if valid_word_count else "failed") - encoded = bert_tokenizer(text, padding="max_length", truncation=True, max_length=50, - return_offsets_mapping=True, return_tensors="np") - offsets = encoded["offset_mapping"][0] - model_tokens = np.asarray(record["text_bert"])[0] - input_ids_match = bool(np.array_equal(encoded["input_ids"][0], model_tokens)) - pieces = bert_tokenizer.convert_ids_to_tokens(model_tokens.tolist()) - spans = _word_spans(text) - token_word = [_offset_to_word(tuple(offsets[i]), spans) for i in range(50)] - local_words = [] - for slot in range(50): - word_index = token_word[slot] - interval = intervals[word_index] if word_index is not None and word_index < len(intervals) else None - local_words.append({ - "slot": slot, - "token": pieces[slot], - "word_index": word_index, - "word": spans[word_index][2] if word_index is not None else "", - "start_s": interval.start_s if interval and interval.valid else float("nan"), - "end_s": interval.end_s if interval and interval.valid else float("nan"), - "ctc_quality": interval.quality if interval and interval.valid else 0.0, - "ctc_valid": bool(interval and interval.valid), - }) - word_mappings[clip_id] = local_words - - predictions.append({ - "sample_id": clip_id, - "predicted_class": CLASS_NAMES[int(pred_class[index])], - "predicted_class_id": int(pred_class[index]), - "predicted_class_probability": float(prob[index]), - "predicted_intensity": float(intensity[index]), - "transcript": text, - "video_file_exists": video_path.is_file(), - "ctc_alignment_status": time_status, - "ctc_word_coverage": ctc_coverage, - "ctc_aligned_words": valid_word_count, - "transcript_words": len(words), - "bert_token_ids_match_pickle": input_ids_match, - }) - - for modality in range(3): - block_values = class_group[index, modality] - available_blocks = [ - block for block in range(N_BLOCKS) - if np.any(split.mask[index, block * BLOCK:(block + 1) * BLOCK, modality]) - ] - top_blocks = sorted(available_blocks, key=lambda block: -float(block_values[block]))[:3] - for block in top_blocks: - left, right = block * BLOCK, (block + 1) * BLOCK - local_slots = [slot for slot in range(left, right) - if split.mask[index, slot, modality] and local_words[slot]["ctc_valid"]] - if not local_slots: - continue - maps = [local_words[slot] for slot in local_slots] - word_rows = {} - for mapping in maps: - if mapping["word_index"] is not None: - word_rows[int(mapping["word_index"])] = mapping - unique_words = [word_rows[key] for key in sorted(word_rows)] - if not unique_words: - continue - evidence.append({ - "sample_id": clip_id, - "modality": MODALITY_LABELS[modality], - "block_index": int(block), - "slot_start_index": int(left), - "slot_end_index_exclusive": int(right), - "slot_indices": ",".join(str(slot) for slot in local_slots), - "tokens_or_wordpieces": " ".join(local_words[slot]["token"] for slot in local_slots), - "matched_words": " ".join(mapping["word"] for mapping in unique_words), - "word_indices": ",".join(str(mapping["word_index"]) for mapping in unique_words), - "time_start_s": min(mapping["start_s"] for mapping in unique_words), - "time_end_s": max(mapping["end_s"] for mapping in unique_words), - "ctc_quality_uncalibrated_mean": float(np.mean([mapping["ctc_quality"] for mapping in unique_words])), - "ctc_words_covered": len(unique_words), - "ctc_word_coverage_clip": ctc_coverage, - "class_importance": float(class_group[index, modality, block]), - "intensity_importance": float(reg_group[index, modality, block]), - "class_explainer": class_method, - "intensity_explainer": reg_method, - "ctc_alignment_status": time_status, - }) - - _write_csv(output / "attachment4_predictions.csv", predictions) - _write_csv(output / "attachment4_top_evidence.csv", evidence) - _plot_attachment4_example(records, videos, predictions, evidence, output) - (output / "attachment4_alignment_audit.json").write_text(json.dumps({ - "n_samples": len(records), - "n_video_files_found": sum(x.is_file() for x in videos), - "n_ctc_any_words_aligned": sum(row["ctc_aligned_words"] > 0 for row in predictions), - "n_ctc_full_word_coverage": sum(row["ctc_alignment_status"] == "ok" for row in predictions), - "mean_transcript_word_coverage": float(np.mean(ctc_word_coverages)), - "n_bert_token_sequences_matching_pickle": sum(row["bert_token_ids_match_pickle"] for row in predictions), - "time_mapping": "Q1 B1 CTC Viterbi hard word intervals computed from the supplied Attachment 4 video audio and transcript; subword slots inherit their transcript word interval", - "quality_note": "CTC path score is uncalibrated. These intervals are localization references for interpretation, not human-annotated ground truth.", - }, ensure_ascii=False, indent=2), encoding="utf-8") - - -def _plot_attachment4_example(records, videos, predictions, evidence, output: Path) -> None: - eligible = [row for row in predictions if row["ctc_alignment_status"] in {"ok", "partial"}] - if not eligible: - return - chosen = eligible[0] - clip_id = chosen["sample_id"] - transcript = str(next(r["raw_text"] for r in records if str(r["id"]) == clip_id)) - local = [row for row in evidence if row["sample_id"] == clip_id - and float(row["time_end_s"]) > float(row["time_start_s"])] - if not local: - return - word_salience: dict[tuple[str, int, str], float] = {} - word_times: dict[tuple[str, int, str], tuple[float, float, str]] = {} - for row in local: - key = (row["modality"], int(row["block_index"]), row["matched_words"]) - word_salience[key] = word_salience.get(key, 0.0) + float(row["class_importance"]) - word_times[key] = (float(row["time_start_s"]), float(row["time_end_s"]), row["matched_words"]) - if not word_times: - return - max_time = max(value[1] for value in word_times.values()) - fig, ax = plt.subplots(figsize=(12, 4.2), constrained_layout=True) - palette = {"text": "#4e79a7", "audio": "#f28e2b", "vision": "#59a14f"} - max_value = max(word_salience.values(), default=1.0) or 1.0 - y_levels = {"text": 2, "audio": 1, "vision": 0} - for key, salience in word_salience.items(): - modality, _, word = key - if key not in word_times: - continue - start, end, _ = word_times[key] - alpha = 0.25 + 0.75 * min(1.0, salience / max_value) - y = y_levels[modality] - ax.broken_barh([(start, max(0.01, end - start))], (y - 0.3, 0.6), - facecolors=palette[modality], alpha=alpha, edgecolors="white", linewidth=0.35) - ax.text((start + end) / 2, y, word, ha="center", va="center", fontsize=6, rotation=55) - ax.set_yticks([0, 1, 2], labels=["Vision", "Audio", "Text"]) - ax.set_xlim(0, max(0.1, max_time)) - ax.set_xlabel("seconds from clip start (Q1 CTC word-time mapping)") - ax.set_title(f"Attachment 4 example {clip_id}: {chosen['predicted_class']} / intensity {chosen['predicted_intensity']:.2f}") - ax.grid(axis="x", alpha=0.2) - output.mkdir(parents=True, exist_ok=True) - fig.savefig(output / f"attachment4_{clip_id}_evidence_timeline.png", dpi=180) - plt.close(fig) - - -def _run(args: argparse.Namespace) -> None: - output = Path(args.output_dir) - output.mkdir(parents=True, exist_ok=True) - q2_output = Path(args.q2_output_dir) - device = torch.device("cuda" if args.device == "auto" and torch.cuda.is_available() else ("cpu" if args.device == "auto" else args.device)) - torch.set_num_threads(args.threads) - method, model = _model_from_run(q2_output, device) - stats = RobustStats.load(q2_output / "aligned_robust_stats.npz") - valid = apply_robust_stats(load_aligned()["valid"], stats) - - started = time.perf_counter() - ig_class, ig_reg = _integrated_groups(model, valid, valid.mask, device, steps=args.ig_steps, batch_size=args.batch_size) - ig_seconds = time.perf_counter() - started - started = time.perf_counter() - occ_class, occ_reg = _occlusion_groups(model, valid, valid.mask, device) - occ_seconds = time.perf_counter() - started - explanations = {"integrated_gradients": (ig_class, ig_reg), "grouped_occlusion": (occ_class, occ_reg)} - curves, summary = _faithfulness_curves(model, valid, valid.mask, explanations, device, args.seed) - - stability_subset = _balanced_subset(valid, args.stability_samples, args.seed + 17) - noisy_subset = _noise_split(stability_subset, args.seed + 23, sigma=args.noise_sigma) - stable_ig_class, stable_ig_reg = _integrated_groups(model, noisy_subset, noisy_subset.mask, device, - steps=args.ig_steps, batch_size=args.batch_size) - stable_occ_class, stable_occ_reg = _occlusion_groups(model, noisy_subset, noisy_subset.mask, device) - stability_rows = [] - for name, original, perturbed in ( - ("integrated_gradients", ig_class[np.isin(np.asarray(valid.ids), stability_subset.ids)], stable_ig_class), - ("grouped_occlusion", occ_class[np.isin(np.asarray(valid.ids), stability_subset.ids)], stable_occ_class), - ): - corr, jaccard = _rank_stability(original, perturbed) - stability_rows.append({"method": name, "target": "predicted_class_probability", "spearman_rank_correlation": corr, - "top_30_percent_jaccard": jaccard, "n_samples": len(stability_subset.ids), - "input_noise_sigma": args.noise_sigma}) - for name, original, perturbed in ( - ("integrated_gradients", ig_reg[np.isin(np.asarray(valid.ids), stability_subset.ids)], stable_ig_reg), - ("grouped_occlusion", occ_reg[np.isin(np.asarray(valid.ids), stability_subset.ids)], stable_occ_reg), - ): - corr, jaccard = _rank_stability(original, perturbed) - stability_rows.append({"method": name, "target": "intensity", "spearman_rank_correlation": corr, - "top_30_percent_jaccard": jaccard, "n_samples": len(stability_subset.ids), - "input_noise_sigma": args.noise_sigma}) - - for row in summary: - row["runtime_seconds"] = ig_seconds if row["method"] == "integrated_gradients" else occ_seconds - stability = next((s for s in stability_rows if s["method"] == row["method"] and s["target"] == row["target"]), None) - if stability: - row.update(stability) - else: - row.update({"spearman_rank_correlation": float("nan"), "top_30_percent_jaccard": float("nan")}) - _write_csv(output / "q3_explanation_method_summary.csv", summary) - _write_csv(output / "q3_deletion_curves.csv", curves) - _write_csv(output / "q3_explanation_stability.csv", stability_rows) - _plot_faithfulness(curves, output / "q3_explanation_faithfulness.png") - - # Select classification and intensity explainers separately, based on direct validation probes. - cls = [r for r in summary if r["target"] == "predicted_class_probability" and r["method"] != "random"] - reg = [r for r in summary if r["target"] == "intensity" and r["method"] != "random"] - class_method = sorted(cls, key=lambda r: (-r["comprehensiveness_signed_drop_at_30"], r["sufficiency_abs_error_top_30"], r["method"]))[0]["method"] - reg_ig = next(r for r in reg if r["method"] == "integrated_gradients") - reg_occ = next(r for r in reg if r["method"] == "grouped_occlusion") - # There is a real tradeoff for the regression head: IG changes the output more - # after deletion, while occlusion better retains it when only the selected - # evidence is kept. Use direct intervention for the displayed segments and - # retain IG as a directional cross-check. - reg_method = "grouped_occlusion" - selection = { - "q2_predictor": method, - "classification_explainer": class_method, - "intensity_explainer_primary": reg_method, - "intensity_explainer_crosscheck": "integrated_gradients", - "intensity_tradeoff": { - "integrated_gradients_abs_change_at_30": reg_ig["absolute_prediction_change_at_30"], - "integrated_gradients_sufficiency_error_top_30": reg_ig["sufficiency_abs_error_top_30"], - "grouped_occlusion_abs_change_at_30": reg_occ["absolute_prediction_change_at_30"], - "grouped_occlusion_sufficiency_error_top_30": reg_occ["sufficiency_abs_error_top_30"], - }, - "selection_basis": "For polarity, grouped occlusion has the larger signed target-probability drop, lower sufficiency error, higher deletion AUC, and lower runtime. For intensity, IG causes a larger deletion change but grouped occlusion has lower top-evidence sufficiency error; grouped occlusion is used for displayed segments and IG is retained as a cross-check. No combined explanation score is used.", - "valid_samples": valid.n, - "stability_samples": len(stability_subset.ids), - "integrated_gradients_runtime_seconds": ig_seconds, - "grouped_occlusion_runtime_seconds": occ_seconds, - "ctc_time_map_for_attachment4": "Q1 hard CTC Viterbi word boundaries from source video audio; not human alignment ground truth", - } - (output / "q3_explainer_selection.json").write_text(json.dumps(selection, ensure_ascii=False, indent=2), encoding="utf-8") - print(f"Q3 explainers: classification={class_method}; intensity={reg_method}", flush=True) - - tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased", use_fast=True) - _run_attachment4(model, output, stats, device, tokenizer, class_method, reg_method) - print(f"saved Q3 explanation selection and Attachment 4 evidence to {output}", flush=True) - - -def main() -> None: - parser = argparse.ArgumentParser(description="Compare faithful Q3 explanations and map Attachment 4 evidence to video time") - parser.add_argument("--q2-output-dir", default=str(Q2_PROJECT / "outputs" / "algorithm_selection")) - parser.add_argument("--output-dir", default=str(Path(__file__).resolve().parents[1] / "outputs" / "explanation_selection")) - parser.add_argument("--device", default="auto") - parser.add_argument("--threads", type=int, default=4) - parser.add_argument("--batch-size", type=int, default=48) - parser.add_argument("--ig-steps", type=int, default=16) - parser.add_argument("--stability-samples", type=int, default=120) - parser.add_argument("--noise-sigma", type=float, default=0.02) - parser.add_argument("--seed", type=int, default=42) - args = parser.parse_args() - _run(args) - - -if __name__ == "__main__": - main()