Add aligned missingness analyses and results
This commit is contained in:
@@ -60,7 +60,7 @@ C6 在这组验证点指标中取得较高的 Accuracy、Macro-F1、MAE 和 RMSE
|
||||
|
||||
测试集类别支持数为负向 207、中性 158、正向 362；相应召回率为 0.7440、0.2089、0.8370。温度校准参数为 1.2127。测试集指标、预测、组 Bootstrap 区间及门控诊断见 [Q2/results](Q2/results/)。
|
||||
|
||||
### 连续缺失控制
|
||||
### 连续缺失控制与缺失类型/率的规律分析
|
||||
|
||||
验证集共执行 42 个固定缺失情景。C5 的归一化 AURC-MAE 及相对 C0 的差值如下；差值置信区间由来源视频组配对 Bootstrap 得到。
|
||||
|
||||
@@ -71,13 +71,87 @@ C6 在这组验证点指标中取得较高的 Accuracy、Macro-F1、MAE 和 RMSE
|
||||
| 部分重叠 | 0.6883 | -0.0774 | [-0.1146, -0.0429] |
|
||||
| 异步 | 0.6880 | -0.0894 | [-0.1261, -0.0554] |
|
||||
|
||||
#### 缺失率对 C5 性能的影响
|
||||
|
||||
以下受控遮蔽评估只在官方验证集进行（728 条、239 个来源视频组），不使用测试集选择遮蔽方案。主表模型为正式选定的 C5；所有已有 C0–C7 消融模型使用相同验证样本和相同掩码。自然缺失率约为 36.2%。新增遮蔽只作用于原本可观测的特征行。置信区间由 1,000 次来源视频组配对 Bootstrap 得到。
|
||||
|
||||
横轴使用论文定义的实际新增缺失率；“最终缺失率”包含 aligned 数据本来的缺失。请求遮蔽率和实际率不同，是因为音频、视频原本已有缺行，且连续区间内可遮蔽的有效行数受时间位置影响。
|
||||
|
||||
|
||||
|
||||
| 遮蔽模式 | 请求率 | 实际新增率 | 最终缺失率 | Accuracy | Macro-F1 | MAE | 相对自然缺失 ΔMAE（95% 区间） |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 自然缺失 | 0% | 0.0% | 36.2% | 0.6168 | 0.5472 | 0.6615 | — |
|
||||
| 单模态 | 10% | 3.4% | 38.3% | 0.6181 | 0.5489 | 0.6596 | -0.0019 [-0.0085, 0.0037] |
|
||||
| 单模态 | 30% | 10.1% | 42.7% | 0.6113 | 0.5380 | 0.6692 | +0.0077 [-0.0007, 0.0163] |
|
||||
| 单模态 | 50% | 16.7% | 46.7% | 0.6071 | 0.5285 | 0.6678 | +0.0063 [-0.0065, 0.0202] |
|
||||
| 单模态 | 70% | 23.2% | 51.0% | 0.6085 | 0.5318 | 0.6935 | +0.0320 [0.0172, 0.0466] |
|
||||
| 同步 | 10% | 5.7% | 40.6% | 0.6154 | 0.5396 | 0.6589 | -0.0025 [-0.0115, 0.0049] |
|
||||
| 同步 | 30% | 18.4% | 50.0% | 0.6126 | 0.5414 | 0.6741 | +0.0126 [-0.0093, 0.0332] |
|
||||
| 同步 | 50% | 33.0% | 60.1% | 0.6058 | 0.5213 | 0.6964 | +0.0349 [0.0110, 0.0627] |
|
||||
| 同步 | 70% | 50.5% | 71.3% | 0.5934 | 0.5084 | 0.7390 | +0.0775 [0.0461, 0.1100] |
|
||||
| 部分重叠 | 10% | 5.3% | 40.4% | 0.6181 | 0.5483 | 0.6612 | -0.0003 [-0.0117, 0.0126] |
|
||||
| 部分重叠 | 30% | 14.8% | 48.7% | 0.6113 | 0.5357 | 0.6784 | +0.0169 [-0.0005, 0.0357] |
|
||||
| 部分重叠 | 50% | 23.6% | 56.5% | 0.6236 | 0.5500 | 0.6918 | +0.0304 [0.0049, 0.0568] |
|
||||
| 部分重叠 | 70% | 39.1% | 67.4% | 0.6058 | 0.5297 | 0.7297 | +0.0682 [0.0400, 0.1002] |
|
||||
| 异步 | 10% | 5.6% | 40.4% | 0.6168 | 0.5432 | 0.6623 | +0.0008 [-0.0119, 0.0154] |
|
||||
| 异步 | 30% | 17.5% | 49.4% | 0.6099 | 0.5325 | 0.6756 | +0.0142 [0.0022, 0.0269] |
|
||||
| 异步 | 50% | 31.2% | 59.0% | 0.6030 | 0.5244 | 0.6998 | +0.0384 [0.0166, 0.0629] |
|
||||
| 异步 | 70% | 48.3% | 70.5% | 0.6058 | 0.5185 | 0.7206 | +0.0591 [0.0311, 0.0884] |
|
||||
|
||||
整体上实际新增缺失率升高时，C5 的 MAE 和 Macro-F1 多数变差；样本级结果并非严格单调。到 70% 请求率时，四种模式的 MAE 增幅置信区间都高于 0；同步模式的 MAE 最高（0.7390），同时它的实际新增率也最高，因此不能把这项差异单独归因于“同步”结构。10% 请求率下，各模式的 MAE 变化区间均覆盖 0。完整曲线与每个模式的组 Bootstrap 明细见 `Q2/results/controlled_missingness.csv` 和 `Q2/results/controlled_group_bootstrap.csv`。
|
||||
|
||||
#### 缺失模态类型：匹配新增缺失量
|
||||
|
||||
为单独比较缺失类型，另做匹配遮蔽：每个验证样本在 T、A、V、TA、TV、AV、TAV 七种条件下新增相同数量的缺失特征行，且各模态新增行数之和逐样本相同。每个样本最多新增 15 行，并按音频和视频各自的可遮蔽容量取共同上限；728 个样本中 25 个没有足够的共同容量，预算为 0。每种类型总计新增 8,932 行（平均每样本 12.27 行），遮蔽以连续时间段构造，七种类型最终总缺失率均为 44.34%。
|
||||
|
||||
表中“论文口径实际新增率”按各模态原本可观测行数作等模态归一，因此相同新增行数在 T/A/V 上会得到不同百分比；跨类型比较以匹配的实际新增行数为准。ΔMAE 为 C5 在该类型下相对自然缺失条件的变化；负值表示误差下降。C5–C0 为同一掩码下的配对 MAE 差值。
|
||||
|
||||
| 缺失类型 | 论文口径实际新增率 | C5 Accuracy | C5 Macro-F1 | C5 MAE | C5 ΔMAE vs 自然（95% 区间） | C5–C0 ΔMAE（95% 区间） |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| T | 8.2% | 0.6099 | 0.5352 | 0.6866 | +0.0251 [0.0079, 0.0450] | -0.0771 [-0.1160, -0.0355] |
|
||||
| A | 19.6% | 0.6099 | 0.5341 | 0.6658 | +0.0043 [-0.0024, 0.0109] | -0.0680 [-0.1061, -0.0314] |
|
||||
| V | 20.3% | 0.6209 | 0.5553 | 0.6525 | -0.0089 [-0.0181, 0.0001] | -0.0838 [-0.1220, -0.0467] |
|
||||
| TA | 13.9% | 0.6154 | 0.5420 | 0.6685 | +0.0070 [-0.0105, 0.0259] | -0.0706 [-0.1077, -0.0318] |
|
||||
| TV | 14.3% | 0.6071 | 0.5354 | 0.6680 | +0.0065 [-0.0057, 0.0188] | -0.0778 [-0.1175, -0.0360] |
|
||||
| AV | 20.0% | 0.6181 | 0.5486 | 0.6586 | -0.0029 [-0.0093, 0.0023] | -0.0745 [-0.1117, -0.0388] |
|
||||
| TAV | 16.1% | 0.6154 | 0.5436 | 0.6651 | +0.0036 [-0.0077, 0.0161] | -0.0675 [-0.1073, -0.0308] |
|
||||
|
||||
匹配总量后，只有 T 单模态遮蔽的 C5 MAE 增幅区间不含 0（+0.0251）；其余类型的 MAE 变化区间均覆盖 0，不能据此排出稳定的模态脆弱性次序。A 单模态遮蔽的 Macro-F1 相对自然条件下降约 0.0131，配对区间为 [-0.0305, -0.0006]。七种类型下 C5 的 MAE 均低于 C0，配对差值区间均低于 0。匹配掩码逐样本审计、模型指标和组 Bootstrap 见 `Q2/results/matched_missing_type.csv`、`matched_missing_type_audit.csv` 和 `matched_missing_type_bootstrap.csv`；复现实验脚本为 `Q2/run_matched_missing_type.py`。
|
||||
|
||||
#### 结构消融在缺失率曲线上的表现
|
||||
|
||||
上面的验证消融表给出自然条件的 Accuracy、Macro-F1、MAE 等指标；下表进一步给出四种连续遮蔽曲线上的归一化 AURC-MAE 相对 C0 的差值。负值代表平均预测误差更低。C0–C7 及单因素诊断共 12 个模型版本均使用相同 42 个验证遮蔽情景；每列的 95% 组 Bootstrap 区间见 `controlled_group_bootstrap.csv`。
|
||||
|
||||
| 模型 | 单模态 ΔAURC | 同步 ΔAURC | 部分重叠 ΔAURC | 异步 ΔAURC |
|
||||
|---|---:|---:|---:|---:|
|
||||
| C0 | 0.000 | 0.000 | 0.000 | 0.000 |
|
||||
| C1 | -0.050 | -0.043 | -0.046 | -0.065 |
|
||||
| C2 | -0.022 | -0.022 | -0.022 | -0.016 |
|
||||
| C3 | -0.020 | -0.024 | -0.020 | -0.037 |
|
||||
| C4 | -0.040 | -0.044 | -0.035 | -0.052 |
|
||||
| C5（最终选定） | -0.076 | -0.076 | -0.077 | -0.089 |
|
||||
| C6 | -0.091 | -0.092 | -0.084 | -0.108 |
|
||||
| C6 无距离/跨度惩罚 | -0.062 | -0.075 | -0.073 | -0.078 |
|
||||
| C6 无辅助重构 | -0.041 | -0.046 | -0.049 | -0.055 |
|
||||
| C6 点遮蔽 | -0.050 | -0.063 | -0.057 | -0.072 |
|
||||
| C7 仅蒸馏 | -0.061 | -0.059 | -0.055 | -0.073 |
|
||||
| C7 仅组风险 | -0.041 | -0.027 | -0.026 | -0.045 |
|
||||
|
||||
C5 的 AURC-MAE 相对 C0 的配对 95% 区间依次为：单模态 [-0.1118, -0.0389]、同步 [-0.1105, -0.0417]、部分重叠 [-0.1146, -0.0429]、异步 [-0.1261, -0.0554]，均低于 0。C6 在四条曲线上的 AURC 点估计最低；正式模型仍依论文规定按验证选择损失选择 C5。相对 C6，取消距离/跨度惩罚、辅助重构或连续遮蔽的点估计，四模式平均 AURC 分别约增加 0.022、0.046 和 0.033，支持保留这些结构与训练项。由于 aligned_50 不含逐行质量分数，质量到噪声映射消融仍不可识别。
|
||||
|
||||
#### 位置、跨度与同步方式的检查
|
||||
|
||||
位置、长跨度/多短跨度和同步/部分重叠/异步的全部点指标与组 Bootstrap 明细也保存在 42 情景结果表中。文本模态起始/中段/末段遮蔽的实际新增率相同（约 10.1%），对应 C5 MAE 为 0.6982、0.6674、0.6767；这是描述性对比，未做位置两两的配对检验。音频和视觉自然缺行更多，同为 30% 请求率时实际新增率差异较大（例如音频起始 20.0%、末段 1.2%；视觉起始 19.3%、末段 1.2%），不据此宣称位置本身造成的因果差异。长跨度与多短跨度的同模态遮蔽、以及同步结构的实际遮蔽量同样应结合审计列阅读。
|
||||
|
||||
附件三有 30 条无标签样本，仅导出预测和审计，不计算 Accuracy 或 F1。`aligned_50.pkl` 没有逐行质量分数，因此按算法回退规则对可见行取 `q*=1, J_Q=0`；无人工边界或附件三标签的项目不报告虚构的监督指标。
|
||||
|
||||
## 主要结果文件
|
||||
|
||||
- Q1 五折模型对照：`Q1/results/model_comparison_v2/comparison_summary.csv`、`fold_metrics.csv`、`oof_predictions.csv`、`group_bootstrap_deltas.csv`。
|
||||
- Q1 特征文件与样本清单：`Q1/features_v2/`。
|
||||
- Q1 随机样本词到原始音频/视频位置图：`Q1/results/alignment_query_random_seed20260925.png`；重绘脚本为 `Q1/visualize_alignment.py`。
|
||||
- Q1 随机样本词到原始音频/视频位置图：`Q1/results/alignment_query_example.png`；重绘脚本为 `Q1/visualize_alignment.py`。
|
||||
- Q2 验证消融与正式指标：`Q2/results/ablation_validation.csv`、`validation_metrics.json`、`test_metrics.json`。
|
||||
- Q2 缺失控制与组区间：`Q2/results/controlled_missingness.csv`、`controlled_group_bootstrap.csv`、`group_bootstrap_ci.csv`。
|
||||
- Q2 匹配缺失类型及图表：`Q2/results/matched_missing_type.csv`、`matched_missing_type_audit.csv`、`matched_missing_type_bootstrap.csv`、`matched_missing_type_manifest.json`、`aligned_missingness_effects.png`；脚本见 `Q2/run_matched_missing_type.py` 和 `Q2/plot_missingness_effects.py`。
|
||||
- Q2 附件三输出：`Q2/results/attachment3_predictions.csv`、`attachment3_audit.csv`。
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
"""Plot the aligned-data rate sweep and matched missing-type response."""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
from pathlib import Path
|
||||
|
||||
import matplotlib.pyplot as plt
|
||||
import numpy as np
|
||||
|
||||
|
||||
RESULTS = Path(__file__).resolve().parent / "results"
|
||||
|
||||
|
||||
def read_csv(name: str) -> list[dict[str, str]]:
|
||||
with (RESULTS / name).open(encoding="utf-8-sig", newline="") as stream:
|
||||
return list(csv.DictReader(stream))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
rate_rows = read_csv("controlled_missingness.csv")
|
||||
rate_bootstrap = read_csv("controlled_group_bootstrap.csv")
|
||||
type_bootstrap = read_csv("matched_missing_type_bootstrap.csv")
|
||||
|
||||
colors = {"single": "#3b82f6", "sync": "#dc2626", "partial": "#16a34a", "async": "#9333ea"}
|
||||
labels = {"single": "Single modality", "sync": "Synchronous", "partial": "Partial overlap", "async": "Asynchronous"}
|
||||
fig, (ax_rate, ax_type) = plt.subplots(1, 2, figsize=(12.4, 4.8), gridspec_kw={"width_ratios": [1.35, 1.0]})
|
||||
|
||||
baseline = next(row for row in rate_rows if row["model"] == "C5" and row["mask_pattern"] == "none")
|
||||
baseline_ci = next(row for row in rate_bootstrap if row["model"] == "C5" and row["scenario"] == "0.0/none" and row["metric"] == "mae")
|
||||
for mode in ("single", "sync", "partial", "async"):
|
||||
rows = [baseline] + sorted(
|
||||
(row for row in rate_rows if row["model"] == "C5" and row["mask_pattern"] == mode),
|
||||
key=lambda row: float(row["rate_requested_per_selected_source"]),
|
||||
)
|
||||
x, y, lower, upper = [], [], [], []
|
||||
for row in rows:
|
||||
if row["mask_pattern"] == "none":
|
||||
ci = baseline_ci
|
||||
scenario = "0.0/none"
|
||||
else:
|
||||
scenario = f"{float(row['rate_requested_per_selected_source']):.1f}/{mode}"
|
||||
ci = next(item for item in rate_bootstrap if item["model"] == "C5" and item["scenario"] == scenario and item["metric"] == "mae")
|
||||
x.append(float(row["rate_realized_additional_global"]))
|
||||
y.append(float(row["regression_mae"]))
|
||||
lower.append(float(ci["ci_2_5"]))
|
||||
upper.append(float(ci["ci_97_5"]))
|
||||
ax_rate.errorbar(
|
||||
x, y, yerr=[np.asarray(y) - np.asarray(lower), np.asarray(upper) - np.asarray(y)],
|
||||
color=colors[mode], marker="o", linewidth=1.7, markersize=4.5,
|
||||
capsize=2.5, label=labels[mode], alpha=0.95,
|
||||
)
|
||||
ax_rate.set_title("C5 performance across missing rates")
|
||||
ax_rate.set_xlabel("Added missing rate (paper definition)")
|
||||
ax_rate.set_ylabel("Regression MAE (95% group-bootstrap CI)")
|
||||
ax_rate.grid(axis="both", color="#d1d5db", linewidth=0.7, alpha=0.65)
|
||||
ax_rate.legend(frameon=False, fontsize=8.5, loc="upper left")
|
||||
|
||||
type_order = ("T", "A", "V", "TA", "TV", "AV", "TAV")
|
||||
point, low, high = [], [], []
|
||||
for label in type_order:
|
||||
scenario = f"matched_type_{label}"
|
||||
boot = next(row for row in type_bootstrap if row["model"] == "C5" and row["scenario"] == scenario and row["metric"] == "mae")
|
||||
point.append(float(boot["delta_to_natural"]))
|
||||
low.append(float(boot["delta_to_natural_ci_2_5"]))
|
||||
high.append(float(boot["delta_to_natural_ci_97_5"]))
|
||||
positions = np.arange(len(type_order))
|
||||
ax_type.errorbar(
|
||||
positions, point, yerr=[np.asarray(point) - low, high - np.asarray(point)],
|
||||
fmt="o", color="#2563eb", ecolor="#2563eb", capsize=3, linewidth=1.4,
|
||||
markersize=5,
|
||||
)
|
||||
ax_type.axhline(0, color="#374151", linewidth=1, linestyle="--")
|
||||
ax_type.set_xticks(positions, type_order)
|
||||
ax_type.set_title("Matched missing-modality types")
|
||||
ax_type.set_xlabel("Hidden modality set")
|
||||
ax_type.set_ylabel("MAE change from natural condition")
|
||||
ax_type.grid(axis="y", color="#d1d5db", linewidth=0.7, alpha=0.65)
|
||||
ax_type.text(
|
||||
0.02, 0.02, "Same added feature-row count per sample and type",
|
||||
transform=ax_type.transAxes, fontsize=7.5, color="#4b5563",
|
||||
)
|
||||
|
||||
fig.tight_layout(pad=1.2)
|
||||
output = RESULTS / "aligned_missingness_effects.png"
|
||||
fig.savefig(output, dpi=200, bbox_inches="tight", facecolor="white")
|
||||
print(output)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 177 KiB |
@@ -0,0 +1,17 @@
|
||||
model,scenario,rate_realized_additional_global,rate_realized_additional_by_modality,natural_missing_rate_global,natural_missing_rate_by_modality,rate_final_total_missing_global,rate_final_total_missing_by_modality,synchronous_no_observation_rate,matched_added_rows_total,n,accuracy,macro_f1,negative_support,neutral_support,positive_support,negative_recall,middle_recall,positive_recall,regression_mae,regression_rmse,pearson,brier,classification_nll,ece_15,interval_90_coverage,interval_90_mean_width,selection_nll,matched_added_rows_mean_per_sample,matched_zero_budget_samples,predictive_variance_mean_uncalibrated,within_trajectory_variance_mean,between_trajectory_variance_mean,predictive_mean_mean_calibrated,predictive_variance_mean_calibrated
|
||||
C0,0.0/none,0.0,"[0.0, 0.0, 0.0]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.0,0,728,0.592032967032967,0.5559346126510305,206,184,338,0.5825242718446602,0.34782608695652173,0.7307692307692307,0.7334139347076416,0.9371736594281902,0.5192425847053528,0.5145634913739343,0.8668775768593128,0.03562808104126608,0.9093406593406593,3.1437528133392334,0.9035571651175792,12.26923076923077,25,,,,,
|
||||
C0,matched_type_T,0.08236053480866758,"[0.24538461538461645, 0.0, 0.0]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.44335164835164836,"[0.24538461538461645, 0.5295054945054959, 0.5551648351648366]",0.10975274725274724,8932,728,0.5769230769230769,0.5393829339564863,206,184,338,0.5679611650485437,0.33152173913043476,0.7159763313609467,0.7636505961418152,0.9703582826584921,0.49724081158638,0.5187844733458724,0.8722964721219175,0.04336370695691294,0.8901098901098901,3.135228395462036,0.9112614179448717,12.26923076923077,25,,,,,
|
||||
C0,matched_type_A,0.19564514931792046,"[0.0, 0.582904297899134, 0.0]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.44335164835164836,"[0.0, 0.7748901098901099, 0.5551648351648366]",0.0,8932,728,0.5865384615384616,0.555780304844736,206,184,338,0.5776699029126213,0.375,0.7071005917159763,0.7338536977767944,0.9391900723295753,0.5181482434272766,0.519898950732422,0.8767403526355085,0.036376168171705664,0.9148351648351648,3.144644021987915,0.9135486459482965,12.26923076923077,25,,,,,
|
||||
C0,matched_type_V,0.20339971766384882,"[0.0, 0.0, 0.618757345880629]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.44335164835164836,"[0.0, 0.5295054945054959, 0.8005494505494513]",0.0,8932,728,0.5851648351648352,0.5509321523095454,206,184,338,0.5631067961165048,0.3641304347826087,0.7189349112426036,0.7363642454147339,0.9375345223746303,0.5202035903930664,0.516828397263377,0.8695349724293657,0.039149007102292924,0.9107142857142857,3.1420364379882812,0.906339212626309,12.26923076923077,25,,,,,
|
||||
C0,matched_type_TA,0.13874580351273275,"[0.12291208791208741, 0.2904665492020848, 0.0]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.44335164835164836,"[0.12291208791208742, 0.6519780219780212, 0.5551648351648366]",0.055714285714285716,8932,728,0.5810439560439561,0.5474719462821344,206,184,338,0.5776699029126213,0.34782608695652173,0.7100591715976331,0.7391047477722168,0.9450906540515626,0.5127956867218018,0.5180875149838958,0.870366247055919,0.03234895390179779,0.9052197802197802,3.1427252292633057,0.9075628422776769,12.26923076923077,25,,,,,
|
||||
C0,matched_type_TV,0.14297275336278015,"[0.12271978021977972, 0.0, 0.3096324011835494]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.44335164835164836,"[0.12271978021977974, 0.5295054945054959, 0.6778296703296693]",0.057637362637362646,8932,728,0.5892857142857143,0.5513653212302171,206,184,338,0.5825242718446602,0.33152173913043476,0.7337278106508875,0.745841920375824,0.9528627503863888,0.5077762007713318,0.5170688084607442,0.8702722730912172,0.03295859232782085,0.9010989010989011,3.137592077255249,0.9079227278806173,12.26923076923077,25,,,,,
|
||||
C0,matched_type_AV,0.19963931250831526,"[0.0, 0.29155520915710675, 0.30962899938872834]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.44335164835164836,"[0.0, 0.6522527472527463, 0.6778021978021966]",0.0,8932,728,0.5837912087912088,0.5462339793823264,206,184,338,0.558252427184466,0.3423913043478261,0.7307692307692307,0.7330833673477173,0.9383928815388751,0.5195925831794739,0.5155379554942412,0.8677943862756456,0.047162958173387164,0.9052197802197802,3.1403942108154297,0.9046343445559706,12.26923076923077,25,,,,,
|
||||
C0,matched_type_TAV,0.16096515249542598,"[0.08175824175824226, 0.19401270719267957, 0.2080956029821996]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.44335164835164836,"[0.08175824175824226, 0.6112912087912086, 0.6370054945054938]",0.040027472527472525,8932,728,0.5906593406593407,0.5530669662179554,206,184,338,0.5728155339805825,0.3423913043478261,0.7366863905325444,0.7325779795646667,0.9408177479302549,0.5191915035247803,0.5132234186440678,0.8658605898856188,0.04369982477779119,0.9052197802197802,3.142317533493042,0.9027755010067487,12.26923076923077,25,,,,,
|
||||
C5,0.0/none,0.0,"[0.0, 0.0, 0.0]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.0,0,728,0.6167582417582418,0.5472420438041464,206,184,338,0.6990291262135923,0.1956521739130435,0.7958579881656804,0.6614819765090942,0.8930210542087126,0.6175611019134521,0.49179526594346484,0.8373559713363647,0.05974927303064,0.8928571428571429,2.381861686706543,3.535538673400879,12.26923076923077,25,0.5485242605209351,0.5485153794288635,8.777557923167478e-06,0.1391230672597885,0.5960303544998169
|
||||
C5,matched_type_T,0.08236053480866758,"[0.24538461538461645, 0.0, 0.0]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.44335164835164836,"[0.24538461538461645, 0.5295054945054959, 0.5551648351648366]",0.10975274725274724,8932,728,0.6098901098901099,0.5351839563874109,206,184,338,0.6747572815533981,0.17391304347826086,0.8076923076923077,0.6865675449371338,0.937496344241326,0.5820114612579346,0.4981574696956791,0.8469253182411194,0.0442884711773841,0.8956043956043956,2.453551769256592,3.5504415035247803,12.26923076923077,25,0.5814554691314697,0.5812084674835205,0.0002470039762556553,0.15884242951869965,0.6281879544258118
|
||||
C5,matched_type_A,0.19564514931792046,"[0.0, 0.582904297899134, 0.0]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.44335164835164836,"[0.0, 0.7748901098901099, 0.5551648351648366]",0.0,8932,728,0.6098901098901099,0.5340993909191584,206,184,338,0.7038834951456311,0.16847826086956522,0.7928994082840237,0.6658172011375427,0.8955359667696297,0.6198774576187134,0.4916545771006669,0.8370434045791626,0.058128268427246214,0.8942307692307693,2.3901164531707764,3.53391432762146,12.26923076923077,25,0.5517944693565369,0.5517837405204773,1.0674714758351911e-05,0.12758490443229675,0.600457489490509
|
||||
C5,matched_type_V,0.20339971766384882,"[0.0, 0.0, 0.618757345880629]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.44335164835164836,"[0.0, 0.5295054945054959, 0.8005494505494513]",0.0,8932,728,0.6208791208791209,0.5552627088646019,206,184,338,0.6990291262135923,0.21195652173913043,0.7958579881656804,0.6525367498397827,0.8809055223239688,0.6240981817245483,0.49228886462649224,0.8382279872894287,0.05256490151469524,0.8873626373626373,2.3522565364837646,3.5415308475494385,12.26923076923077,25,0.5376319885253906,0.5376207828521729,1.121730929298792e-05,0.13418912887573242,0.5840601921081543
|
||||
C5,matched_type_TA,0.13874580351273275,"[0.12291208791208741, 0.2904665492020848, 0.0]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.44335164835164836,"[0.12291208791208742, 0.6519780219780212, 0.5551648351648366]",0.055714285714285716,8932,728,0.6153846153846154,0.5419674263773049,206,184,338,0.6990291262135923,0.1793478260869565,0.8017751479289941,0.6685264706611633,0.9048146028456692,0.609342634677887,0.49276868260838724,0.8386208415031433,0.04161169175263289,0.8928571428571429,2.419700860977173,3.536464214324951,12.26923076923077,25,0.5647855401039124,0.5646933913230896,9.222278458764777e-05,0.14660096168518066,0.6122171878814697
|
||||
C5,matched_type_TV,0.14297275336278015,"[0.12271978021977972, 0.0, 0.3096324011835494]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.44335164835164836,"[0.12271978021977974, 0.5295054945054959, 0.6778296703296693]",0.057637362637362646,8932,728,0.6071428571428571,0.5353544129527634,206,184,338,0.6796116504854369,0.1793478260869565,0.7958579881656804,0.6680172681808472,0.9005903572763313,0.6099472045898438,0.49241561221321045,0.83706134557724,0.05255568387744191,0.8942307692307693,2.405548334121704,3.531872510910034,12.26923076923077,25,0.5589443445205688,0.5588531494140625,9.121741459239274e-05,0.14908139407634735,0.6059207320213318
|
||||
C5,matched_type_AV,0.19963931250831526,"[0.0, 0.29155520915710675, 0.30962899938872834]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.44335164835164836,"[0.0, 0.6522527472527463, 0.6778021978021966]",0.0,8932,728,0.6181318681318682,0.5485750437972842,206,184,338,0.7038834951456311,0.1956521739130435,0.7958579881656804,0.6586098074913025,0.8902075943845942,0.6201772689819336,0.491709285683531,0.8372098207473755,0.05521660641982003,0.8914835164835165,2.3736307621002197,3.536386251449585,12.26923076923077,25,0.5457169413566589,0.5457064509391785,1.0502969416847918e-05,0.13100601732730865,0.5931695699691772
|
||||
C5,matched_type_TAV,0.16096515249542598,"[0.08175824175824226, 0.19401270719267957, 0.2080956029821996]",0.36155677655677654,"[0.0, 0.5295054945054959, 0.5551648351648366]",0.44335164835164836,"[0.08175824175824226, 0.6112912087912086, 0.6370054945054938]",0.040027472527472525,8932,728,0.6153846153846154,0.5436390160113765,206,184,338,0.6990291262135923,0.18478260869565216,0.7988165680473372,0.6651139259338379,0.8969123500264189,0.6153786182403564,0.4912526448827642,0.8370823860168457,0.052326494479899864,0.896978021978022,2.3980393409729004,3.535360813140869,12.26923076923077,25,0.5562692284584045,0.5562043786048889,6.489618681371212e-05,0.1406826674938202,0.6039722561836243
|
||||
|
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,49 @@
|
||||
model,scenario,metric,estimate,bootstrap_median,ci_2_5,ci_97_5,replicates,unit,C5_minus_C0,C5_minus_C0_bootstrap_median,C5_minus_C0_ci_2_5,C5_minus_C0_ci_97_5,delta_to_natural,delta_to_natural_bootstrap_median,delta_to_natural_ci_2_5,delta_to_natural_ci_97_5
|
||||
C0,0.0/none,accuracy,0.592032967032967,0.5913224969833732,0.5543807466530893,0.6289861583970531,1000,paired source-video group resample,0.02472527472527475,0.025136950236025524,-0.006831068942286053,0.05586404164352515,,,,
|
||||
C0,0.0/none,macro_f1,0.5559346126510305,0.5555506288566419,0.5180087213671375,0.5904483452391245,1000,paired source-video group resample,-0.008692568846884074,-0.00742421382336883,-0.04789524506814869,0.028977594075404574,,,,
|
||||
C0,0.0/none,mae,0.7334139347076416,0.7324737906455994,0.6878134265542031,0.7832536712288857,1000,paired source-video group resample,-0.07193195819854736,-0.071920245885849,-0.10994510054588318,-0.03534466326236726,,,,
|
||||
C0,matched_type_T,accuracy,0.5769230769230769,0.5770046147955585,0.5408827979799479,0.6145150501672241,1000,paired source-video group resample,0.03296703296703307,0.032942839563492476,0.0,0.06374791111316701,-0.015109890109890167,-0.015031447315988988,-0.029895387859348983,0.0
|
||||
C0,matched_type_T,macro_f1,0.5393829339564863,0.5385649983604828,0.5026451207194949,0.5780419310118782,1000,paired source-video group resample,-0.0041989775690753905,-0.0034015426051711706,-0.042662169816292075,0.03254199150929852,-0.016551678694544214,-0.01639816840472469,-0.03355851780613857,0.0001821587002871506
|
||||
C0,matched_type_T,mae,0.7636505961418152,0.7628006935119629,0.7182668462395668,0.8176649451255799,1000,paired source-video group resample,-0.0770830512046814,-0.07778486609458923,-0.11603872776031494,-0.035469788312912,0.030236661434173584,0.03057163953781128,0.013966797292232514,0.04837098419666289
|
||||
C0,matched_type_A,accuracy,0.5865384615384616,0.5864031531712329,0.5506038818445845,0.626185627618545,1000,paired source-video group resample,0.02335164835164838,0.024373741707647056,-0.00977026523349212,0.058040787337662346,-0.005494505494505475,-0.00589970501474929,-0.02439190084835633,0.012613656973580748
|
||||
C0,matched_type_A,macro_f1,0.555780304844736,0.5551606953545224,0.5176488312713191,0.5925956170859047,1000,paired source-video group resample,-0.021680913925577583,-0.02165753935569914,-0.05884198222022191,0.01951280756673255,-0.00015430780629455132,-0.0008347498161657696,-0.022996330038185877,0.021114250080972417
|
||||
C0,matched_type_A,mae,0.7338536977767944,0.7332103848457336,0.6873686507344245,0.7827591091394425,1000,paired source-video group resample,-0.06803649663925171,-0.06791001558303833,-0.10614908784627915,-0.03143596947193146,0.00043976306915283203,0.000182420015335083,-0.010727481544017791,0.01074945032596588
|
||||
C0,matched_type_V,accuracy,0.5851648351648352,0.5846153846153846,0.5474133614934856,0.6229592409611499,1000,paired source-video group resample,0.0357142857142857,0.03624167603563083,0.001372530395136773,0.06980799337693908,-0.006868131868131844,-0.006958945610779421,-0.019640913928759766,0.006723887118899058
|
||||
C0,matched_type_V,macro_f1,0.5509321523095454,0.5504003169237217,0.5137349977346363,0.589068371691351,1000,paired source-video group resample,0.004330556555056542,0.00461833151432145,-0.03706279068242269,0.044783543906377433,-0.005002460341485104,-0.005314663059922509,-0.019918844377567746,0.009815314781836821
|
||||
C0,matched_type_V,mae,0.7363642454147339,0.735662192106247,0.6903349027037621,0.7864542722702026,1000,paired source-video group resample,-0.08382749557495117,-0.0833868682384491,-0.12203920781612396,-0.04667305201292038,0.002950310707092285,0.003127545118331909,-0.0027874454855918883,0.009412758052349089
|
||||
C0,matched_type_TA,accuracy,0.5810439560439561,0.5802639807774597,0.5417263913439392,0.6193116710875332,1000,paired source-video group resample,0.03434065934065933,0.034979152744967046,0.0012862796833772747,0.06878390592691291,-0.01098901098901095,-0.011392423317454436,-0.02572360248447213,0.004161177008443304
|
||||
C0,matched_type_TA,macro_f1,0.5474719462821344,0.5460971289211473,0.509503682800298,0.5847959654674446,1000,paired source-video group resample,-0.005504519904829475,-0.00498077796565094,-0.04295062127825049,0.0369578431753336,-0.008462666368896143,-0.009472600395273478,-0.024997034956830633,0.008817364825479677
|
||||
C0,matched_type_TA,mae,0.7391047477722168,0.7382183074951172,0.694877165555954,0.7883736655116081,1000,paired source-video group resample,-0.07057827711105347,-0.07113125920295715,-0.10771856904029846,-0.0317716732621193,0.005690813064575195,0.0057299137115478516,-0.006073255836963653,0.017568321526050566
|
||||
C0,matched_type_TV,accuracy,0.5892857142857143,0.5890986823551531,0.5519334049409237,0.6273248530952489,1000,paired source-video group resample,0.017857142857142794,0.018826946873885808,-0.017573583857830474,0.051515677311970695,-0.0027472527472527375,-0.002670226969292422,-0.01650904754102729,0.010247273926095887
|
||||
C0,matched_type_TV,macro_f1,0.5513653212302171,0.5506651348324003,0.5140847005460136,0.5903611501572952,1000,paired source-video group resample,-0.01601090827745366,-0.014849160751929047,-0.06125595462076885,0.028038069526667368,-0.004569291420813415,-0.004106826914776929,-0.020845162983090575,0.010857893839506496
|
||||
C0,matched_type_TV,mae,0.745841920375824,0.7452885210514069,0.6976982250809669,0.7954563602805138,1000,paired source-video group resample,-0.0778246521949768,-0.07787632942199707,-0.11753073632717133,-0.036039607226848604,0.012427985668182373,0.012515097856521606,0.002442780137062073,0.022447265684604645
|
||||
C0,matched_type_AV,accuracy,0.5837912087912088,0.5837820358701102,0.5435346577255309,0.6228792423761839,1000,paired source-video group resample,0.03434065934065933,0.03398086279098411,0.0012879788639365306,0.06790232296587549,-0.008241758241758212,-0.008281577443322585,-0.020327277252364308,0.0028656387211147935
|
||||
C0,matched_type_AV,macro_f1,0.5462339793823264,0.5450550063694277,0.5062522623876544,0.5839731840580099,1000,paired source-video group resample,0.0023410644149577386,0.0019090907894763753,-0.0374221104021527,0.0433665663761355,-0.009700633268704073,-0.009747149375830044,-0.024668950398918317,0.003485043541411635
|
||||
C0,matched_type_AV,mae,0.7330833673477173,0.7323378324508667,0.6874367862939834,0.7837248966097832,1000,paired source-video group resample,-0.0744735598564148,-0.07418161630630493,-0.11166073977947236,-0.03880458921194077,-0.0003305673599243164,-0.0003624260425567627,-0.005563387274742126,0.0048500016331672665
|
||||
C0,matched_type_TAV,accuracy,0.5906593406593407,0.5892252317008092,0.5516665698209322,0.6291562766410912,1000,paired source-video group resample,0.02472527472527475,0.025740025740025707,-0.009678232361719696,0.061311696037009734,-0.0013736263736263687,-0.0013898601060891025,-0.015807456709416606,0.013639807162534464
|
||||
C0,matched_type_TAV,macro_f1,0.5530669662179554,0.5514799414851768,0.5122892491372637,0.5920116915662829,1000,paired source-video group resample,-0.009427950206578828,-0.008313162966268994,-0.05234725069468815,0.032884553762760074,-0.002867646433075133,-0.0029838757839551477,-0.019391363633015066,0.014507233669741196
|
||||
C0,matched_type_TAV,mae,0.7325779795646667,0.731381893157959,0.6869693398475647,0.7818331733345986,1000,paired source-video group resample,-0.06746405363082886,-0.06719186902046204,-0.1072593554854393,-0.030761821568012254,-0.0008359551429748535,-0.001007169485092163,-0.008481840789318084,0.006332886219024657
|
||||
C5,0.0/none,accuracy,0.6167582417582418,0.6166978776529338,0.5809674694817873,0.6550346304991086,1000,paired source-video group resample,0.02472527472527475,0.025136950236025524,-0.006831068942286053,0.05586404164352515,,,,
|
||||
C5,0.0/none,macro_f1,0.5472420438041464,0.5469260158058056,0.5096860090973042,0.5860718123814138,1000,paired source-video group resample,-0.008692568846884074,-0.00742421382336883,-0.04789524506814869,0.028977594075404574,,,,
|
||||
C5,0.0/none,mae,0.6614819765090942,0.6603323519229889,0.6108811289072037,0.7183952122926712,1000,paired source-video group resample,-0.07193195819854736,-0.071920245885849,-0.10994510054588318,-0.03534466326236726,,,,
|
||||
C5,matched_type_T,accuracy,0.6098901098901099,0.609271523178808,0.5747819418053616,0.646381613781627,1000,paired source-video group resample,0.03296703296703307,0.032942839563492476,0.0,0.06374791111316701,-0.006868131868131844,-0.007042267491107923,-0.02122653771168629,0.00662564460054878
|
||||
C5,matched_type_T,macro_f1,0.5351839563874109,0.5350491048096742,0.4995437583617357,0.5701451203231437,1000,paired source-video group resample,-0.0041989775690753905,-0.0034015426051711706,-0.042662169816292075,0.03254199150929852,-0.01205808741673553,-0.012524860857425157,-0.03253195925729562,0.00718955733681344
|
||||
C5,matched_type_T,mae,0.6865675449371338,0.6868902146816254,0.6341559991240502,0.7454844802618027,1000,paired source-video group resample,-0.0770830512046814,-0.07778486609458923,-0.11603872776031494,-0.035469788312912,0.02508556842803955,0.024865955114364624,0.007885064184665681,0.0449989840388298
|
||||
C5,matched_type_A,accuracy,0.6098901098901099,0.6097400724335759,0.5731360815602836,0.6497304916021891,1000,paired source-video group resample,0.02335164835164838,0.024373741707647056,-0.00977026523349212,0.058040787337662346,-0.006868131868131844,-0.006711409395973145,-0.01663233657693487,0.0014265844711636722
|
||||
C5,matched_type_A,macro_f1,0.5340993909191584,0.5347423198579184,0.49929644898774395,0.5717139506905322,1000,paired source-video group resample,-0.021680913925577583,-0.02165753935569914,-0.05884198222022191,0.01951280756673255,-0.01314265288498806,-0.012725867310055677,-0.030530061845670146,-0.0006382757566359616
|
||||
C5,matched_type_A,mae,0.6658172011375427,0.6649405360221863,0.6158120661973954,0.7202810019254684,1000,paired source-video group resample,-0.06803649663925171,-0.06791001558303833,-0.10614908784627915,-0.03143596947193146,0.004335224628448486,0.004407644271850586,-0.0024298295378684994,0.01089831292629242
|
||||
C5,matched_type_V,accuracy,0.6208791208791209,0.6210256755846082,0.584963195832761,0.6598774140528694,1000,paired source-video group resample,0.0357142857142857,0.03624167603563083,0.001372530395136773,0.06980799337693908,0.004120879120879106,0.004081632653061218,-0.0028011204481792618,0.011050488973232705
|
||||
C5,matched_type_V,macro_f1,0.5552627088646019,0.5548154105397096,0.5188237206853148,0.5936986399490214,1000,paired source-video group resample,0.004330556555056542,0.00461833151432145,-0.03706279068242269,0.044783543906377433,0.008020665060455512,0.007616129713741926,-0.0009067843389306632,0.01848125000692546
|
||||
C5,matched_type_V,mae,0.6525367498397827,0.6515619158744812,0.6022072985768319,0.7089235290884971,1000,paired source-video group resample,-0.08382749557495117,-0.0833868682384491,-0.12203920781612396,-0.04667305201292038,-0.008945226669311523,-0.00854673981666565,-0.01810423731803894,0.0001220732927322365
|
||||
C5,matched_type_TA,accuracy,0.6153846153846154,0.6150122684404544,0.5784156654373092,0.6536993582520942,1000,paired source-video group resample,0.03434065934065933,0.034979152744967046,0.0012862796833772747,0.06878390592691291,-0.0013736263736263687,-0.00135963501517683,-0.01654031883935727,0.013146313955040857
|
||||
C5,matched_type_TA,macro_f1,0.5419674263773049,0.5417557193177193,0.5047802290907606,0.5799203132022882,1000,paired source-video group resample,-0.005504519904829475,-0.00498077796565094,-0.04295062127825049,0.0369578431753336,-0.0052746174268415436,-0.0050068654127332635,-0.02809887871993042,0.016375842304308087
|
||||
C5,matched_type_TA,mae,0.6685264706611633,0.66790372133255,0.6156516760587692,0.7259220972657203,1000,paired source-video group resample,-0.07057827711105347,-0.07113125920295715,-0.10771856904029846,-0.0317716732621193,0.007044494152069092,0.006769269704818726,-0.010480178892612458,0.025946144759654996
|
||||
C5,matched_type_TV,accuracy,0.6071428571428571,0.607167555819008,0.5717907954014195,0.6450396176467932,1000,paired source-video group resample,0.017857142857142794,0.018826946873885808,-0.017573583857830474,0.051515677311970695,-0.009615384615384692,-0.009615384615384581,-0.020718232044198873,0.0013587882063736843
|
||||
C5,matched_type_TV,macro_f1,0.5353544129527634,0.5354645081878824,0.49819271440188406,0.5745457156706383,1000,paired source-video group resample,-0.01601090827745366,-0.014849160751929047,-0.06125595462076885,0.028038069526667368,-0.011887630851383002,-0.011946300166675305,-0.02800474254119283,0.0033269332252161493
|
||||
C5,matched_type_TV,mae,0.6680172681808472,0.6666520535945892,0.6168728619813919,0.7224325031042099,1000,paired source-video group resample,-0.0778246521949768,-0.07787632942199707,-0.11753073632717133,-0.036039607226848604,0.00653529167175293,0.00642704963684082,-0.005669380724430084,0.0188097670674324
|
||||
C5,matched_type_AV,accuracy,0.6181318681318682,0.6176066024759285,0.5811098396645663,0.6577088653256504,1000,paired source-video group resample,0.03434065934065933,0.03398086279098411,0.0012879788639365306,0.06790232296587549,0.0013736263736263687,0.001367054637333387,-0.005479639964672922,0.007052435195588585
|
||||
C5,matched_type_AV,macro_f1,0.5485750437972842,0.5487752308088052,0.5115179758083219,0.5869957723657893,1000,paired source-video group resample,0.0023410644149577386,0.0019090907894763753,-0.0374221104021527,0.0433665663761355,0.00133299999313774,0.0012425549092640042,-0.008037309549542104,0.009641827431609167
|
||||
C5,matched_type_AV,mae,0.6586098074913025,0.6580310165882111,0.6077357083559036,0.7151366353034974,1000,paired source-video group resample,-0.0744735598564148,-0.07418161630630493,-0.11166073977947236,-0.03880458921194077,-0.002872169017791748,-0.002576887607574463,-0.009267038106918335,0.002250625193119048
|
||||
C5,matched_type_TAV,accuracy,0.6153846153846154,0.6153846153846154,0.5786798646362098,0.6536938208747975,1000,paired source-video group resample,0.02472527472527475,0.025740025740025707,-0.009678232361719696,0.061311696037009734,-0.0013736263736263687,-0.0013764631433757502,-0.01276641078336559,0.009736083077316392
|
||||
C5,matched_type_TAV,macro_f1,0.5436390160113765,0.544465986990263,0.5070496304871545,0.5830016204071233,1000,paired source-video group resample,-0.009427950206578828,-0.008313162966268994,-0.05234725069468815,0.032884553762760074,-0.003603027792769886,-0.00333310846437912,-0.02114415046888372,0.01257102810185335
|
||||
C5,matched_type_TAV,mae,0.6651139259338379,0.6645334362983704,0.6112550914287567,0.7212894827127456,1000,paired source-video group resample,-0.06746405363082886,-0.06719186902046204,-0.1072593554854393,-0.030761821568012254,0.0036319494247436523,0.003264307975769043,-0.00766083300113678,0.016097874939441675
|
||||
|
@@ -0,0 +1,37 @@
|
||||
{
|
||||
"input": "E题数据/附件2-数据集特征文件/aligned_50.pkl",
|
||||
"input_sha256": "66e867aa74bc70a844e806e5571e371c9abb4a35f9e2887ce9b4d97ff2cb8fcd",
|
||||
"evaluation_split": "official validation",
|
||||
"validation_samples": 728,
|
||||
"source_video_groups": 239,
|
||||
"models": [
|
||||
"C0",
|
||||
"C5"
|
||||
],
|
||||
"modalities": {
|
||||
"T": "text",
|
||||
"A": "audio",
|
||||
"V": "vision"
|
||||
},
|
||||
"matched_type_sets": [
|
||||
"T",
|
||||
"A",
|
||||
"V",
|
||||
"TA",
|
||||
"TV",
|
||||
"AV",
|
||||
"TAV"
|
||||
],
|
||||
"mask_rule": "per-sample target=min(per_sample_cap, audio_hide_capacity, vision_hide_capacity); split target evenly across selected modalities; continuous intervals",
|
||||
"per_sample_cap_rows": 15,
|
||||
"total_added_feature_rows_per_type": 8932,
|
||||
"mean_added_feature_rows_per_sample": 12.26923076923077,
|
||||
"zero_budget_samples": 25,
|
||||
"mask_seed": 20262133,
|
||||
"C5_evaluation_seed": 20262134,
|
||||
"bootstrap_seed": 20262135,
|
||||
"bootstrap_repeats": 1000,
|
||||
"C5_temperature": 1.212728800581531,
|
||||
"C0_temperature": 2.032287887301264,
|
||||
"test_split_used": false
|
||||
}
|
||||
@@ -0,0 +1,369 @@
|
||||
"""Matched-volume missing-modality evaluation for the official validation split.
|
||||
|
||||
This complements the standard requested-rate sweep. Every type condition hides
|
||||
the same number of originally observed feature rows in each validation sample;
|
||||
the affected rows are placed in one contiguous span per selected modality.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from sklearn.metrics import accuracy_score, f1_score, mean_absolute_error
|
||||
|
||||
import train
|
||||
from crg import INPUT_DIMS, StructuredGaussianImputer
|
||||
from data import ALIGNED_PATH, fit_preprocessor, load_official_splits, transform_split
|
||||
|
||||
|
||||
RESULTS = Path(__file__).resolve().parent / "results"
|
||||
MODALITY_SETS = (
|
||||
((0,), "T"), ((1,), "A"), ((2,), "V"),
|
||||
((0, 1), "TA"), ((0, 2), "TV"), ((1, 2), "AV"), ((0, 1, 2), "TAV"),
|
||||
)
|
||||
MODALITY_NAMES = ("text", "audio", "vision")
|
||||
|
||||
|
||||
def make_matched_type_masks(
|
||||
split: Any, seed: int, per_sample_cap: int = 15,
|
||||
) -> tuple[dict[str, np.ndarray], list[dict[str, Any]]]:
|
||||
original = np.asarray(split.mask, dtype=bool)
|
||||
counts = original.sum(axis=1).astype(np.int64)
|
||||
keep_minimum = np.maximum(1, np.ceil(0.2 * counts).astype(np.int64))
|
||||
capacity = np.maximum(0, counts - keep_minimum)
|
||||
# Match the same feasible volume per sample for every modality set. The
|
||||
# least observed of audio/vision determines the cap, so no condition can
|
||||
# gain an advantage by applying its mask to a different subset of samples.
|
||||
budget = np.minimum(per_sample_cap, np.minimum(capacity[:, 1], capacity[:, 2]))
|
||||
masks: dict[str, np.ndarray] = {"0.0/none": original.copy()}
|
||||
audit: list[dict[str, Any]] = []
|
||||
|
||||
for selected, label in MODALITY_SETS:
|
||||
key = f"matched_type_{label}"
|
||||
current = original.copy()
|
||||
for row_index, sample_id in enumerate(split.ids):
|
||||
total = int(budget[row_index])
|
||||
base, remainder = divmod(total, len(selected))
|
||||
sample_seed = int.from_bytes(
|
||||
hashlib.sha256(f"{seed}:{sample_id}:{key}".encode("utf-8")).digest()[:8],
|
||||
"little",
|
||||
)
|
||||
rng = np.random.default_rng(sample_seed)
|
||||
allocation = np.full(len(selected), base, dtype=np.int64)
|
||||
if remainder:
|
||||
allocation[rng.permutation(len(selected))[:remainder]] += 1
|
||||
starts: list[str] = []
|
||||
ends: list[str] = []
|
||||
hidden_by_modality = np.zeros(3, dtype=np.int64)
|
||||
for modality, amount_value in zip(selected, allocation):
|
||||
amount = int(amount_value)
|
||||
if amount == 0:
|
||||
starts.append("")
|
||||
ends.append("")
|
||||
continue
|
||||
interval = train._best_interval(
|
||||
original[row_index, :, modality], amount,
|
||||
int(capacity[row_index, modality]), "random", rng,
|
||||
)
|
||||
if interval is None:
|
||||
raise RuntimeError(f"no feasible interval for {sample_id}/{label}/{MODALITY_NAMES[modality]}")
|
||||
left, right = interval
|
||||
positions = np.flatnonzero(original[row_index, left:right + 1, modality]) + left
|
||||
if len(positions) != amount:
|
||||
raise RuntimeError(f"matched interval hid {len(positions)} rows, expected {amount}")
|
||||
current[row_index, positions, modality] = False
|
||||
hidden_by_modality[modality] = amount
|
||||
starts.append(str(int(left)))
|
||||
ends.append(str(int(right)))
|
||||
actual_total = int(np.sum(original[row_index] & ~current[row_index]))
|
||||
if actual_total != total:
|
||||
raise RuntimeError(f"matched volume differs for {sample_id}: {actual_total} != {total}")
|
||||
audit.append({
|
||||
"scenario": key,
|
||||
"sample_id": sample_id,
|
||||
"source_video_id": str(split.groups[row_index]),
|
||||
"selected_modalities": json.dumps([MODALITY_NAMES[m] for m in selected]),
|
||||
"base_mask_seed": int(seed),
|
||||
"sample_mask_seed": sample_seed,
|
||||
"matched_added_rows_target": total,
|
||||
"matched_added_rows_actual": actual_total,
|
||||
"hidden_text_rows": int(hidden_by_modality[0]),
|
||||
"hidden_audio_rows": int(hidden_by_modality[1]),
|
||||
"hidden_vision_rows": int(hidden_by_modality[2]),
|
||||
"span_start_by_selected_modality": json.dumps(starts),
|
||||
"span_end_by_selected_modality": json.dumps(ends),
|
||||
})
|
||||
if not np.array_equal(np.sum(original & ~current, axis=(1, 2)), budget):
|
||||
raise RuntimeError(f"per-sample matched-volume invariant failed for {label}")
|
||||
masks[key] = current
|
||||
|
||||
total_masked = int(budget.sum())
|
||||
if len({int(np.sum(original & ~mask)) for key, mask in masks.items() if key != "0.0/none"}) != 1:
|
||||
raise RuntimeError("matched modality scenarios do not have identical total missing volume")
|
||||
print(
|
||||
f"matched type masks: samples={split.n}, added_rows_per_scenario={total_masked}, "
|
||||
f"mean_per_sample={budget.mean():.3f}, zero_budget_samples={int(np.sum(budget == 0))}",
|
||||
flush=True,
|
||||
)
|
||||
return masks, audit
|
||||
|
||||
|
||||
def metric_values(split: Any, prediction: dict[str, np.ndarray], indices: np.ndarray) -> dict[str, float]:
|
||||
return {
|
||||
"accuracy": float(accuracy_score(split.class_y[indices], prediction["predicted_class"][indices])),
|
||||
"macro_f1": float(f1_score(
|
||||
split.class_y[indices], prediction["predicted_class"][indices],
|
||||
labels=[0, 1, 2], average="macro", zero_division=0,
|
||||
)),
|
||||
"mae": float(mean_absolute_error(
|
||||
split.regression_y[indices], prediction["predicted_score"][indices],
|
||||
)),
|
||||
}
|
||||
|
||||
|
||||
def evaluate_matched_masks(
|
||||
model_name: str,
|
||||
split: Any,
|
||||
arrays: dict[str, np.ndarray],
|
||||
masks: dict[str, np.ndarray],
|
||||
temperature: float,
|
||||
*,
|
||||
model: Any | None = None,
|
||||
c0_state: dict[str, Any] | None = None,
|
||||
device: torch.device | None = None,
|
||||
seed: int = 0,
|
||||
) -> tuple[list[dict[str, Any]], dict[str, dict[str, np.ndarray]]]:
|
||||
rows = []
|
||||
predictions = {}
|
||||
for scenario, mask in masks.items():
|
||||
if model_name == "C0":
|
||||
if c0_state is None:
|
||||
raise ValueError("C0 state is required")
|
||||
metrics, prediction = train.evaluate_c0(c0_state, split, arrays, temperature, mask)
|
||||
else:
|
||||
if model is None or device is None:
|
||||
raise ValueError("neural model and device are required")
|
||||
scenario_seed = train._scenario_seed(seed, split.name, scenario)
|
||||
with train.fixed_torch_seed(scenario_seed, device):
|
||||
metrics, prediction = train.evaluate(
|
||||
model, arrays, split, device, 64, masks=mask, temperature=temperature,
|
||||
)
|
||||
predictions[scenario] = prediction
|
||||
rates = train._missing_rate_summary(split.mask, mask)
|
||||
row = {
|
||||
"model": model_name,
|
||||
"scenario": scenario,
|
||||
"rate_realized_additional_global": rates["additional_global"],
|
||||
"rate_realized_additional_by_modality": json.dumps(
|
||||
[None if not np.isfinite(value) else float(value)
|
||||
for value in np.nanmean(rates["additional_by_modality"], axis=0)]
|
||||
),
|
||||
"natural_missing_rate_global": rates["natural_global"],
|
||||
"natural_missing_rate_by_modality": json.dumps(
|
||||
np.mean(rates["natural_by_modality"], axis=0).tolist()
|
||||
),
|
||||
"rate_final_total_missing_global": rates["final_global"],
|
||||
"rate_final_total_missing_by_modality": json.dumps(
|
||||
np.mean(rates["final_by_modality"], axis=0).tolist()
|
||||
),
|
||||
"synchronous_no_observation_rate": float(np.mean(rates["synchronous_no_observation"])),
|
||||
"matched_added_rows_total": int(np.sum(split.mask & ~mask)),
|
||||
**metrics,
|
||||
}
|
||||
rows.append(row)
|
||||
return rows, predictions
|
||||
|
||||
|
||||
def paired_source_video_bootstrap(
|
||||
split: Any,
|
||||
predictions: dict[str, dict[str, dict[str, np.ndarray]]],
|
||||
repeats: int,
|
||||
seed: int,
|
||||
) -> list[dict[str, Any]]:
|
||||
scenarios = list(predictions["C0"])
|
||||
groups = np.unique(split.groups)
|
||||
group_indices = {group: np.flatnonzero(split.groups == group) for group in groups}
|
||||
names = tuple(predictions)
|
||||
point = {
|
||||
(model, scenario, metric): value
|
||||
for model in names
|
||||
for scenario in scenarios
|
||||
for metric, value in metric_values(split, predictions[model][scenario], np.arange(split.n)).items()
|
||||
}
|
||||
draws = {key: [] for key in point}
|
||||
within_natural = {
|
||||
(model, scenario, metric): []
|
||||
for model in names for scenario in scenarios if scenario != "0.0/none"
|
||||
for metric in ("accuracy", "macro_f1", "mae")
|
||||
}
|
||||
model_deltas = {
|
||||
(scenario, metric): []
|
||||
for scenario in scenarios for metric in ("accuracy", "macro_f1", "mae")
|
||||
}
|
||||
rng = np.random.default_rng(seed)
|
||||
for _ in range(repeats):
|
||||
chosen = rng.choice(groups, size=len(groups), replace=True)
|
||||
indices = np.concatenate([group_indices[group] for group in chosen])
|
||||
replicate = {}
|
||||
for model in names:
|
||||
for scenario in scenarios:
|
||||
for metric, value in metric_values(split, predictions[model][scenario], indices).items():
|
||||
replicate[(model, scenario, metric)] = value
|
||||
draws[(model, scenario, metric)].append(value)
|
||||
for model in names:
|
||||
for scenario in scenarios:
|
||||
if scenario == "0.0/none":
|
||||
continue
|
||||
for metric in ("accuracy", "macro_f1", "mae"):
|
||||
within_natural[(model, scenario, metric)].append(
|
||||
replicate[(model, scenario, metric)] - replicate[(model, "0.0/none", metric)]
|
||||
)
|
||||
for scenario in scenarios:
|
||||
for metric in ("accuracy", "macro_f1", "mae"):
|
||||
model_deltas[(scenario, metric)].append(
|
||||
replicate[("C5", scenario, metric)] - replicate[("C0", scenario, metric)]
|
||||
)
|
||||
|
||||
def interval(values: list[float]) -> tuple[float, float, float]:
|
||||
values_np = np.asarray(values, dtype=np.float64)
|
||||
return (float(np.median(values_np)), float(np.percentile(values_np, 2.5)),
|
||||
float(np.percentile(values_np, 97.5)))
|
||||
|
||||
rows = []
|
||||
for model in names:
|
||||
for scenario in scenarios:
|
||||
for metric in ("accuracy", "macro_f1", "mae"):
|
||||
median, lower, upper = interval(draws[(model, scenario, metric)])
|
||||
row: dict[str, Any] = {
|
||||
"model": model, "scenario": scenario, "metric": metric,
|
||||
"estimate": point[(model, scenario, metric)],
|
||||
"bootstrap_median": median, "ci_2_5": lower, "ci_97_5": upper,
|
||||
"replicates": repeats, "unit": "paired source-video group resample",
|
||||
}
|
||||
if scenario != "0.0/none":
|
||||
delta = point[(model, scenario, metric)] - point[(model, "0.0/none", metric)]
|
||||
d_median, d_lower, d_upper = interval(within_natural[(model, scenario, metric)])
|
||||
row.update({
|
||||
"delta_to_natural": delta,
|
||||
"delta_to_natural_bootstrap_median": d_median,
|
||||
"delta_to_natural_ci_2_5": d_lower,
|
||||
"delta_to_natural_ci_97_5": d_upper,
|
||||
})
|
||||
model_delta = point[("C5", scenario, metric)] - point[("C0", scenario, metric)]
|
||||
md_median, md_lower, md_upper = interval(model_deltas[(scenario, metric)])
|
||||
row.update({
|
||||
"C5_minus_C0": model_delta,
|
||||
"C5_minus_C0_bootstrap_median": md_median,
|
||||
"C5_minus_C0_ci_2_5": md_lower,
|
||||
"C5_minus_C0_ci_97_5": md_upper,
|
||||
})
|
||||
rows.append(row)
|
||||
return rows
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--seed", type=int, default=20260924 + 1209)
|
||||
parser.add_argument("--per-sample-cap", type=int, default=15)
|
||||
parser.add_argument("--bootstrap-repeats", type=int, default=1000)
|
||||
parser.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
|
||||
args = parser.parse_args()
|
||||
train.seed_everything(args.seed)
|
||||
device = torch.device(args.device)
|
||||
|
||||
official = load_official_splits()
|
||||
fit, heldout_train = train.split_calibration(official["train"], 20260924)
|
||||
_, temperature_calibration = train.split_calibration(heldout_train, 20260925, fraction=0.5)
|
||||
fitted = fit_preprocessor(fit)
|
||||
transformed = {name: transform_split(split, fitted) for name, split in official.items()}
|
||||
transformed["fit"] = transform_split(fit, fitted)
|
||||
transformed["temperature_calibration"] = transform_split(temperature_calibration, fitted)
|
||||
|
||||
imputer = StructuredGaussianImputer(INPUT_DIMS)
|
||||
imputer.load_state_dict(torch.load(RESULTS / "structured_imputer.pt", map_location="cpu", weights_only=True))
|
||||
with (RESULTS / "validation_metrics.json").open(encoding="utf-8") as stream:
|
||||
validation_metadata = json.load(stream)
|
||||
if validation_metadata.get("selected_model") != "C5":
|
||||
raise RuntimeError(f"expected selected C5 model, found {validation_metadata.get('selected_model')}")
|
||||
temperature = float(validation_metadata["temperature"])
|
||||
selected_reliability = (0.3, 0.05, 0.05, 0.05)
|
||||
model = train._make_variant("C5", imputer, selected_reliability).to(device)
|
||||
model.load_state_dict(torch.load(RESULTS / "crg_student.pt", map_location="cpu", weights_only=True))
|
||||
model.eval()
|
||||
|
||||
_, c0_state = train.fit_c0(fit, official["valid"], transformed)
|
||||
train.calibrate_c0_interval(c0_state, temperature_calibration, transformed["temperature_calibration"])
|
||||
_, c0_calibration = train.evaluate_c0(c0_state, temperature_calibration, transformed["temperature_calibration"])
|
||||
c0_temperature = train.fit_temperature(c0_calibration["probabilities"], temperature_calibration.class_y)
|
||||
|
||||
masks, audit = make_matched_type_masks(official["valid"], args.seed, args.per_sample_cap)
|
||||
metrics_c0, pred_c0 = evaluate_matched_masks(
|
||||
"C0", official["valid"], transformed["valid"], masks, c0_temperature,
|
||||
c0_state=c0_state,
|
||||
)
|
||||
metrics_c5, pred_c5 = evaluate_matched_masks(
|
||||
"C5", official["valid"], transformed["valid"], masks, temperature,
|
||||
model=model, device=device, seed=args.seed + 1,
|
||||
)
|
||||
total_masked = int(np.sum(official["valid"].mask & ~masks["matched_type_T"]))
|
||||
per_sample_budget_mean = total_masked / official["valid"].n
|
||||
zero_budget_samples = sum(
|
||||
1 for row in audit if row["scenario"] == "matched_type_T" and row["matched_added_rows_target"] == 0
|
||||
)
|
||||
summary_rows = []
|
||||
for row in metrics_c0 + metrics_c5:
|
||||
row["matched_added_rows_mean_per_sample"] = per_sample_budget_mean
|
||||
row["matched_zero_budget_samples"] = zero_budget_samples
|
||||
summary_rows.append(row)
|
||||
train.write_csv(RESULTS / "matched_missing_type.csv", summary_rows)
|
||||
train.write_csv(RESULTS / "matched_missing_type_audit.csv", audit)
|
||||
bootstrap_rows = paired_source_video_bootstrap(
|
||||
official["valid"], {"C0": pred_c0, "C5": pred_c5},
|
||||
args.bootstrap_repeats, args.seed + 2,
|
||||
)
|
||||
train.write_csv(RESULTS / "matched_missing_type_bootstrap.csv", bootstrap_rows)
|
||||
manifest = {
|
||||
"input": str(ALIGNED_PATH.relative_to(train.ROOT)),
|
||||
"input_sha256": train.sha256(ALIGNED_PATH),
|
||||
"evaluation_split": "official validation",
|
||||
"validation_samples": official["valid"].n,
|
||||
"source_video_groups": int(len(np.unique(official["valid"].groups))),
|
||||
"models": ["C0", "C5"],
|
||||
"modalities": {"T": "text", "A": "audio", "V": "vision"},
|
||||
"matched_type_sets": [label for _, label in MODALITY_SETS],
|
||||
"mask_rule": "per-sample target=min(per_sample_cap, audio_hide_capacity, vision_hide_capacity); split target evenly across selected modalities; continuous intervals",
|
||||
"per_sample_cap_rows": args.per_sample_cap,
|
||||
"total_added_feature_rows_per_type": total_masked,
|
||||
"mean_added_feature_rows_per_sample": per_sample_budget_mean,
|
||||
"zero_budget_samples": zero_budget_samples,
|
||||
"mask_seed": args.seed,
|
||||
"C5_evaluation_seed": args.seed + 1,
|
||||
"bootstrap_seed": args.seed + 2,
|
||||
"bootstrap_repeats": args.bootstrap_repeats,
|
||||
"C5_temperature": temperature,
|
||||
"C0_temperature": c0_temperature,
|
||||
"test_split_used": False,
|
||||
}
|
||||
(RESULTS / "matched_missing_type_manifest.json").write_text(
|
||||
json.dumps(manifest, indent=2, ensure_ascii=False), encoding="utf-8",
|
||||
)
|
||||
print(f"C0 temperature={c0_temperature:.6f}; C5 temperature={temperature:.6f}", flush=True)
|
||||
for row in summary_rows:
|
||||
if row["model"] != "C5" or row["scenario"] == "0.0/none":
|
||||
continue
|
||||
print(
|
||||
f"{row['model']} {row['scenario']}: added={row['rate_realized_additional_global']:.4f} "
|
||||
f"final={row['rate_final_total_missing_global']:.4f} "
|
||||
f"Acc={row['accuracy']:.4f} MacroF1={row['macro_f1']:.4f} "
|
||||
f"MAE={row['regression_mae']:.4f}",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user