structure.md

Model structure

中文 · Project README

TactiWeave-VP implements the following sequence classifier:

tactile frames ── shared ResNet-18 ── visual projection ──┐
                                                         ├─ frame-wise fusion
physical states ───────── physical MLP ──────────────────┘
        └─ causal TCN ── attention pooling ── classifier ── logits

The implementation lives in model/ and uses [batch, time, features] for sequence features.

Inputs

For frame t, the aligned observation is

$$ \mathcal{O}_t=(I_t,p_t),\qquad I_t\in\mathbb{R}^{1\times350\times350},\qquad p_t=[F_x,F_y,F_z,F_{2D},\mu,A]^\top\in\mathbb{R}^{6}. $$

The six metadata fields, in model order, are Fx, Fy, Fz, F2D, mu, and area. The physical vector is standardized with training-split statistics only:

$$ \widetilde p_{t,j}=\frac{p_{t,j}-m_j}{\max(s_j,\varepsilon)}, $$

where m and s are the population mean and standard deviation computed from unique training frames. The saved statistics are reused unchanged for validation, testing, offline inference, and real-time inference.

Visual encoder

All frames share one ResNet-18. The batch and time dimensions are flattened to process the frames together:

$$ [B,T,1,H,W]\rightarrow[BT,1,H,W]\rightarrow[BT,512]. $$

The classification layer is removed. For ImageNet initialization, the first RGB convolution weights are averaged across the channel dimension to initialize the single input channel. A projection produces a 256-dimensional visual token:

$$ r_t=\operatorname{ResNet18}(I_t),\qquad v_t=\operatorname{LayerNorm}(\operatorname{GELU}(W_vr_t+b_v)), $$

with r_t in R^512 and v_t in R^256.

Physical encoder

The normalized six-dimensional state passes through a two-layer MLP:

$$ s_t=\operatorname{LayerNorm}!\left(W_{p2}\operatorname{GELU}(W_{p1}\widetilde p_t+b_{p1})+b_{p2}\right). $$

The default dimensions are 6 -> 64 -> 128, so s_t is a 128-dimensional physical token.

Frame-wise fusion

Aligned visual and physical tokens are concatenated at each time step and projected back to 256 dimensions:

$$ x_t=\operatorname{LayerNorm}!\left(\operatorname{GELU}(W_f[v_t;s_t]+b_f)\right), $$

corresponding to 256 + 128 -> 256. Invalid padded positions, when a mask is supplied, are zeroed before temporal encoding.

Causal TCN

The fused sequence is processed by four residual TCN blocks. Each block contains two left-padded causal 1-D convolutions, GELU activation, dropout, and a residual connection. Defaults are:

• hidden dimension: 256;
• kernel size: 3;
• block dilations: (1, 2, 4, 8);
• convolution dropout: 0.2.

A causal convolution at time t reads only the current and previous positions. With two convolutions per residual block, the temporal receptive field is

$$ R=1+2(k-1)\sum_{l=1}^{L}d_l =1+2(3-1)(1+2+4+8)=61\text{ frames}. $$

The TCN preserves sequence length and returns h_t in R^256 for every frame.

Attention pooling and classifier

Attention pooling converts the temporal features into one fixed-size vector:

$$ e_t=w_a^\top\tanh(W_ah_t+b_a),\qquad \alpha_t=\frac{\exp(e_t)}{\sum_{\tau\in\mathcal V}\exp(e_\tau)},\qquad z=\sum_{t\in\mathcal V}\alpha_t h_t, $$

where V is the set of valid positions from the optional mask. The weights are non-negative and sum to one over valid positions.

The classifier is Linear(256, 128) -> GELU -> Dropout(0.3) -> Linear(128, C). It returns unnormalized logits. Training passes those logits directly to cross-entropy loss; probabilities are computed only when needed for inference or reporting.

Default dimension flow

Stage	Per-frame or per-sequence shape
Tactile image	1 x 350 x 350 per frame
ResNet-18 feature	512 per frame
Visual token	256 per frame
Physical input/token	6 -> 128 per frame
Concatenated/fused token	384 -> 256 per frame
Causal TCN output	256 per frame
Attention-pooled feature	256 per sequence
Classifier output	C logits per sequence

See Data pipeline for how aligned sequences are built and Usage for training and inference commands.