-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.ejs
More file actions
1093 lines (1067 loc) · 57.7 KB
/
Copy pathindex.ejs
File metadata and controls
1093 lines (1067 loc) · 57.7 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!doctype html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<script src="https://distill.pub/template.v2.js"></script>
<style><%= require("raw-loader!../static/style.css") %></style>
<script src="https://d3js.org/d3.v4.min.js"></script>
<script src="https://d3js.org/d3-selection-multi.v1.min.js"></script>
<script src="https://ariutta.github.io/svg-pan-zoom/dist/svg-pan-zoom.js"></script>
</head>
<body>
<d-front-matter>
<script type="text/json">{
"title": "Feature-Wise Transformations",
"description": "A simple and surprisingly effective family of conditioning mechanisms.",
"authors": [
{
"author": "Vincent Dumoulin",
"authorURL": "https://vdumoulin.github.io",
"affiliations": [{"name": "Google Brain", "url": "https://ai.google/research/teams/brain"}]
},
{
"author": "Ethan Perez",
"authorURL": "http://ethanperez.net/",
"affiliations": [
{"name": "Rice University", "url": "http://www.rice.edu/"},
{"name": "MILA", "url": "https://mila.quebec/en/"}
]
},
{
"author": "Nathan Schucher",
"authorURL": "https://nathanschucher.com/",
"affiliations": [{"name": "Element AI", "url": "https://element.ai/"}]
},
{
"author": "Florian Strub",
"authorURL": "https://fstrub95.github.io/",
"affiliations": [{"name": "Univ. of Lille, Inria", "url": "https://team.inria.fr/sequel/"}]
},
{
"author": "Harm de Vries",
"authorURL": "http://www-etud.iro.umontreal.ca/~devries/",
"affiliations": [{"name": "MILA", "url": "https://mila.quebec/en/"}]
},
{
"author": "Aaron Courville",
"authorURL": "https://aaroncourville.wordpress.com/",
"affiliations": [{"name": "MILA", "url": "https://mila.quebec/en/"}]
},
{
"author": "Yoshua Bengio",
"authorURL": "http://www.iro.umontreal.ca/~bengioy/yoshua_en/",
"affiliations": [{"name": "MILA", "url": "https://mila.quebec/en/"}]
}
],
"katex": {
"delimiters": [
{
"left": "$",
"right": "$",
"display": false
},
{
"left": "$$",
"right": "$$",
"display": true
}
]
}
}</script>
</d-front-matter>
<d-title>
<h1>Feature-Wise Transformations</h1>
<p>A simple and surprisingly effective family of conditioning mechanisms.</p>
<div class="l-page" id="vtoc"></div>
</d-title>
<d-article>
<p>
Many real-world problems require integrating multiple sources of information.
Sometimes these problems involve multiple, distinct modalities of
information — vision, language, audio, etc. — as is required
to understand a scene in a movie or answer a question about an image.
Other times, these problems involve multiple sources of the same
kind of input, i.e. when summarizing several documents or drawing one
image in the style of another.
</p>
<figure class="l-body-outset">
<%= require("../static/diagrams/multimodal.svg") %>
</figure>
<p>
When approaching such problems, it often makes sense to process one source
of information <em>in the context of</em> another; for instance, in the
right example above, one can extract meaning from the image in the context
of the question. In machine learning, we often refer to this context-based
processing as <em>conditioning</em>: the computation carried out by a model
is conditioned or <em>modulated</em> by information extracted from an
auxiliary input.
</p>
<p>
Finding an effective way to condition on or fuse sources of information
is an open research problem, and
<!-- Introduction -->
in this article, we concentrate on a specific family of approaches we call
<em>feature-wise transformations</em>.
<!-- Related Work -->
We will examine the use of feature-wise transformations in many neural network
architectures to solve a surprisingly large and diverse set of problems;
<!-- Experiments -->
their success, we will argue, is due to being flexible enough to learn an
effective representation of the conditioning input in varied settings.
In the language of multi-task learning, where the conditioning signal is
taken to be a task description, feature-wise transformations
learn a task representation which allows them to capture and leverage the
relationship between multiple sources of information, even in remarkably
different problem settings.
</p>
<hr/>
<h2 id="feature-wise-transformations">Feature-wise transformations</h2>
<p>
To motivate feature-wise transformations, we start with a basic example,
where the two inputs are images and category labels, respectively. For the
purpose of this example, we are interested in building a generative model of
images of various classes (puppy, boat, airplane, etc.). The model takes as
input a class and a source of random noise (e.g., a vector sampled from a
normal distribution) and outputs an image sample for the requested class.
</p>
<figure class="l-body">
<%= require("../static/diagrams/conditional-generation.svg") %>
</figure>
<p>
Our first instinct might be to build a separate model for each
class. For a small number of classes this approach is not too bad a solution,
but for thousands of classes, we quickly run into scaling issues, as the number
of parameters to store and train grows with the number of classes.
We are also missing out on the opportunity to leverage commonalities between
classes; for instance, different types of dogs (puppy, terrier, dalmatian,
etc.) share visual traits and are likely to share computation when
mapping from the abstract noise vector to the output image.
</p>
<p>
Now let's imagine that, in addition to the various classes, we also need to
model attributes like size or color. In this case, we can't
reasonably expect to train a separate network for <em>each</em> possible
conditioning combination! Let's examine a few simple options.
</p>
<p>
A quick fix would be to concatenate a representation of the conditioning
information to the noise vector and treat the result as the model's input.
This solution is quite parameter-efficient, as we only need to increase
the size of the first layer's weight matrix. However, this approach makes the implicit
assumption that the input is where the model needs to use the conditioning information.
Maybe this assumption is correct, or maybe it's not; perhaps the
model does not need to incorporate the conditioning information until late
into the generation process (e.g., right before generating the final pixel
output when conditioning on texture). In this case, we would be forcing the model to
carry this information around unaltered for many layers.
</p>
<p>
Because this operation is cheap, we might as well avoid making any such
assumptions and concatenate the conditioning representation to the input of
<em>all</em> layers in the network. Let's call this approach
<em>concatenation-based conditioning</em>.
</p>
<figure class="l-body">
<%= require("../static/diagrams/concatenation-based-conditioning.svg") %>
</figure>
<p>
Another efficient way to integrate conditioning information into the network
is via <em>conditional biasing</em>, namely, by adding a <em>bias</em> to
the hidden layers based on the conditioning representation.
</p>
<figure class="l-body">
<%= require("../static/diagrams/conditional-biasing.svg") %>
</figure>
<p>
Interestingly, conditional biasing can be thought of as another way to
implement concatenation-based conditioning. Consider a fully-connected
linear layer applied to the concatenation of an input
<d-math>\mathbf{x}</d-math> and a conditioning representation
<d-math>\mathbf{z}</d-math>:
<d-footnote>
The same argument applies to convolutional networks, provided we ignore
the border effects due to zero-padding.
</d-footnote>
</p>
<figure class="l-body">
<%= require("../static/diagrams/concatenation-as-biasing.svg") %>
</figure>
<p>
Yet another efficient way to integrate class information into the network is
via <em>conditional scaling</em>, i.e., scaling hidden layers
based on the conditioning representation.
</p>
<figure class="l-body">
<%= require("../static/diagrams/conditional-scaling.svg") %>
</figure>
<p>
A special instance of conditional scaling is feature-wise sigmoidal gating:
we scale each feature by a value between <d-math>0</d-math> and
<d-math>1</d-math> (enforced by applying the logistic function), as a
function of the conditioning representation. Intuitively, this gating allows
the conditioning information to select which features are passed forward
and which are zeroed out.
</p>
<p>
Given that both additive and multiplicative interactions seem natural and
intuitive, which approach should we pick? One argument in favor of
<em>multiplicative</em> interactions is that they are useful in learning
relationships between inputs, as these interactions naturally identify
"matches": multiplying elements that agree in sign yields larger values than
multiplying elements that disagree. This property is why dot products are
often used to determine how similar two vectors are.
<d-footnote>
Multiplicative interactions alone have had a history of success in various
domains — see <a href="#bibliographic-notes">Bibliographic Notes</a>.
</d-footnote>
One argument in favor of <em>additive</em> interactions is that they are
more natural for applications that are less strongly dependent on the
joint values of two inputs, like feature aggregation or feature detection
(i.e., checking if a feature is present in either of two inputs).
</p>
<p>
In the spirit of making as few assumptions about the problem as possible,
we may as well combine <em>both</em> into a
conditional <em>affine transformation</em>.
<d-footnote>
An affine transformation is a transformation of the form
<d-math>y = m * x + b</d-math>.
</d-footnote>
</p>
<p>
All methods outlined above share the common trait that they act at the
<em>feature</em> level; in other words, they leverage <em>feature-wise</em>
interactions between the conditioning representation and the conditioned
network. It is certainly possible to use more complex interactions,
but feature-wise interactions often strike a happy compromise between
effectiveness and efficiency: the number of scaling and/or shifting
coefficients to predict scales linearly with the number of features in the
network. Also, in practice, feature-wise transformations (often compounded
across multiple layers) frequently have enough capacity to model complex
phenomenon in various settings.
</p>
<p>
Lastly, these transformations only enforce a limited inductive bias and
remain domain-agnostic. This quality can be a downside, as some problems may
be easier to solve with a stronger inductive bias. However, it is this
characteristic which also enables these transformations to be so widely
effective across problem domains, as we will later review.
</p>
<h3 id="nomenclature">Nomenclature</h3>
<p>
To continue the discussion on feature-wise transformations we need to
abstract away the distinction between multiplicative and additive
interactions. Without losing generality, let's focus on feature-wise affine
transformations, and let's adopt the nomenclature of Perez et al.
<d-cite key="perez2018film"></d-cite>, which formalizes conditional affine
transformations under the acronym <em>FiLM</em>, for Feature-wise Linear
Modulation.
<d-footnote>
Strictly speaking, <em>linear</em> is a misnomer, as we allow biasing, but
we hope the more rigorous-minded reader will forgive us for the sake of a
better-sounding acronym.
</d-footnote>
</p>
<p>
We say that a neural network is modulated using FiLM, or <em>FiLM-ed</em>,
after inserting <em>FiLM layers</em> into its architecture. These layers are
parametrized by some form of conditioning information, and the mapping from
conditioning information to FiLM parameters (i.e., the shifting and scaling
coefficients) is called the <em>FiLM generator</em>.
In other words, the FiLM generator predicts the parameters of the FiLM
layers based on some auxiliary input.
Note that the FiLM parameters are parameters in one network but predictions
from another network, so they aren't learnable parameters with fixed
weights as in the fully traditional sense.
For simplicity, you can assume that the FiLM generator outputs the
concatenation of all FiLM parameters for the network architecture.
</p>
<figure class="l-body">
<%= require("../static/diagrams/film-architecture.svg") %>
</figure>
<p>
As the name implies, a FiLM layer applies a feature-wise affine
transformation to its input. By <em>feature-wise</em>, we mean that scaling
and shifting are applied element-wise, or in the case of convolutional
networks, feature map -wise.
<d-footnote>
To expand a little more on the convolutional case, feature maps can be
thought of as the same feature detector being evaluated at different
spatial locations, in which case it makes sense to apply the same affine
transformation to all spatial locations.
</d-footnote>
In other words, assuming <d-math>\mathbf{x}</d-math> is a FiLM layer's
input, <d-math>\mathbf{z}</d-math> is a conditioning input, and
<d-math>\gamma</d-math> and <d-math>\beta</d-math> are
<d-math>\mathbf{z}</d-math>-dependent scaling and shifting vectors,
<d-math block>
\textrm{FiLM}(\mathbf{x}) = \gamma(\mathbf{z}) \odot \mathbf{x}
+ \beta(\mathbf{z}).
</d-math>
You can interact with the following fully-connected and convolutional FiLM
layers to get an intuition of the sort of modulation they allow:
</p>
<figure class="l-body-outset" id="film-layer-diagram">
<%= require("../static/diagrams/film-layer.svg") %>
</figure>
<p>
In addition to being a good abstraction of conditional feature-wise
transformations, the FiLM nomenclature lends itself well to the notion of a
<em>task representation</em>. From the perspective of multi-task learning,
we can view the conditioning signal as the task description. More
specifically, we can view the concatenation of all FiLM scaling and shifting
coefficients as both an instruction on <em>how to modulate</em> the
conditioned network and a <em>representation</em> of the task at hand. We
will explore and illustrate this idea later on.
</p>
<hr/>
<h2 id="in-literature">Feature-wise transformations in the literature</h2>
<p>
Feature-wise transformations find their way into methods applied to many
problem settings, but because of their simplicity, their effectiveness is
seldom highlighted in lieu of other novel research contributions. Below are
a few notable examples of feature-wise transformations in the literature,
grouped by application domain. The diversity of these applications
underscores the flexible, general-purpose ability of feature-wise
interactions to learn effective task representations.
</p>
<div style="width: 100%">
<button class="expand-collapse-button" content-type="literature">expand all</button>
</div>
<button class="collapsible" content-name="vqa" content-type="literature">Visual question-answering<span style="float: right;">+</span></button>
<p class="content" content-name="vqa" content-type="literature">
Perez et al. <d-cite key="perez2017learning,perez2018film"></d-cite> use
FiLM layers to build a visual reasoning model
trained on the CLEVR dataset <d-cite key="johnson2017clevr"></d-cite> to
answer multi-step, compositional questions about synthetic images.
</p>
<figure class="content l-body-outset" content-name="vqa" content-type="literature">
<%= require("../static/diagrams/film-clevr.svg") %>
</figure>
<p class="content" content-name="vqa" content-type="literature">
The model's linguistic pipeline is a FiLM generator which
extracts a question representation that is linearly mapped to
FiLM parameter values. Using these values, FiLM layers inserted within each
residual block condition the visual pipeline. The model is trained
end-to-end on image-question-answer triples. Strub et al.
<d-cite key="strub2018visual"></d-cite> later on improved on the model by
using an attention mechanism to alternate between attending to the language
input and generating FiLM parameters layer by layer. This approach was
better able to scale to settings with longer input sequences such as
dialogue and was evaluated on the GuessWhat?! <d-cite key="vries2017guesswhat"></d-cite>
and ReferIt <d-cite key="kazemzadeh2014referitgame"></d-cite> datasets.
</p>
<p class="content" content-name="vqa" content-type="literature">
de Vries et al. <d-cite key="vries2017modulating"></d-cite> leverage FiLM
to condition a pre-trained network. Their model's linguistic pipeline
modulates the visual pipeline via conditional batch normalization,
which can be viewed as a special case of FiLM. The model learns to answer natural language questions about
real-world images on the GuessWhat?! <d-cite key="vries2017guesswhat"></d-cite>
and VQAv1 <d-cite key="agrawal2015vqa"></d-cite> datasets.
</p>
<figure class="content l-body-outset" content-name="vqa" content-type="literature">
<%= require("../static/diagrams/guesswhat.svg") %>
</figure>
<p class="content" content-name="vqa" content-type="literature">
The visual pipeline consists of a pre-trained residual network that is
fixed throughout training. The linguistic pipeline manipulates the visual
pipeline by perturbing the residual network's batch normalization
parameters, which re-scale and re-shift feature maps after activations
have been normalized to have zero mean and unit variance. As hinted
earlier, conditional batch normalization can be viewed as an instance of
FiLM where the post-normalization feature-wise affine transformation is
replaced with a FiLM layer.
</p>
<button class="collapsible" content-name="style-transfer" content-type="literature">Style transfer<span style="float: right;">+</span></button>
<p class="content" content-name="style-transfer" content-type="literature">
Dumoulin et al. <d-cite key="dumoulin2017learned"></d-cite> use
feature-wise affine transformations — in the form of conditional
instance normalization layers — to condition a style transfer
network on a chosen style image. Like conditional batch normalization
discussed in the previous subsection,
conditional instance normalization can be seen as an instance of FiLM
where a FiLM layer replaces the post-normalization feature-wise affine
transformation. For style transfer, the network models each style as a separate set of
instance normalization parameters, and it applies normalization with these
style-specific parameters.
</p>
<figure class="content l-body-outset" content-name="style-transfer" id="alrfas-diagram" content-type="literature">
<%= require("../static/diagrams/alrfas.svg") %>
</figure>
<p class="content" content-name="style-transfer" content-type="literature">
Dumoulin et al. <d-cite key="dumoulin2017learned"></d-cite> use a simple
embedding lookup to produce instance normalization parameters, while
Ghiasi et al. <d-cite key="ghiasi2017exploring"></d-cite> further
introduce a <em>style prediction network</em>, trained jointly with the
style transfer network to predict the conditioning parameters directly from
a given style image. In this article we opt to use the FiLM nomenclature
because it is decoupled from normalization operations, but the FiLM
layers used by Perez et al. <d-cite key="perez2018film"></d-cite> were
themselves heavily inspired by the conditional normalization layers used
by Dumoulin et al. <d-cite key="dumoulin2017learned"></d-cite>.
</p>
<p class="content" content-name="style-transfer" content-type="literature">
Yang et al. <d-cite key="yang2018efficient"></d-cite> use a related
architecture for video object segmentation — the task of segmenting a
particular object throughout a video given that object's segmentation in the
first frame. Their model conditions an image segmentation network over a
video frame on the provided first frame segmentation using feature-wise
scaling factors, as well as on the previous frame using position-wise
biases.
</p>
<p class="content" content-name="style-transfer" content-type="literature">
So far, the models we covered have two sub-networks: a primary
network in which feature-wise transformations are applied and a secondary
network which outputs parameters for these transformations. However, this
distinction between <em>FiLM-ed network</em> and <em>FiLM generator</em>
is not strictly necessary. As an example, Huang and Belongie
<d-cite key="huang2017arbitrary"></d-cite> propose an alternative
style transfer network that uses adaptive instance normalization layers,
which compute normalization parameters using a simple heuristic.
</p>
<figure class="content l-body-outset" content-name="style-transfer" id="adain-diagram" content-type="literature">
<%= require("../static/diagrams/adain.svg") %>
</figure>
<p class="content" content-name="style-transfer" content-type="literature">
Adaptive instance normalization can be interpreted as inserting a FiLM
layer midway through the model. However, rather than relying
on a secondary network to predict the FiLM parameters from the style
image, the main network itself is used to extract the style features
used to compute FiLM parameters. Therefore, the model can be seen as
<em>both</em> the FiLM-ed network and the FiLM generator.
</p>
<button class="collapsible" content-name="image-recognition" content-type="literature">Image recognition<span style="float: right;">+</span></button>
<p class="content" content-name="image-recognition" content-type="literature">
As discussed in previous subsections, there is nothing preventing us from considering a
neural network's activations <em>themselves</em> as conditioning
information. This idea gives rise to
<em>self-conditioned</em> models.
</p>
<figure class="content l-body" content-name="image-recognition" content-type="literature">
<%= require("../static/diagrams/self-conditioning.svg") %>
</figure>
<p class="content" content-name="image-recognition" content-type="literature">
Highway Networks <d-cite key="srivastava2015highway"></d-cite> are a prime
example of applying this self-conditioning principle. They take inspiration
from the LSTMs' <d-cite key="hochreiter1997long"></d-cite> heavy use of
feature-wise sigmoidal gating in their input, forget, and output gates to
regulate information flow:
</p>
<figure class="content l-body" content-name="image-recognition" content-type="literature">
<%= require("../static/diagrams/highway-networks.svg") %>
</figure>
<p class="content" content-name="image-recognition" content-type="literature">
The ImageNet 2017 winning model <d-cite key="hu2017squeeze"></d-cite> also
employs feature-wise sigmoidal gating in a self-conditioning manner, as a
way to "recalibrate" a layer's activations conditioned on themselves.
</p>
<figure class="content l-body" content-name="image-recognition" content-type="literature">
<%= require("../static/diagrams/squeeze-and-excitation.svg") %>
</figure>
<button class="collapsible" content-name="nlp" content-type="literature">Natural language processing<span style="float: right;">+</span></button>
<p class="content" content-name="nlp" content-type="literature">
For statistical language modeling (i.e., predicting the next word
in a sentence), the LSTM <d-cite key="hochreiter1997long,melis2017lstm"></d-cite>
constitutes a popular class of recurrent network architectures. The LSTM
relies heavily on feature-wise sigmoidal gating to control the
information flow in and out of the memory or context cell
<d-math>\mathbf{c}</d-math>, based on the hidden states
<d-math>\mathbf{h}</d-math> and inputs <d-math>\mathbf{x}</d-math> at
every timestep <d-math>\mathbf{t}</d-math>.
</p>
<figure class="content l-body" content-name="nlp" content-type="literature">
<%= require("../static/diagrams/lstm.svg") %>
</figure>
<p class="content" content-name="nlp" content-type="literature">
Also in the domain of language modeling, Dauphin et al. <d-cite key="dauphin2017language"></d-cite> use sigmoidal
gating in their proposed <em>gated linear unit</em>, which uses half of the
input features to apply feature-wise sigmoidal gating to the other half.
Gehring et al. <d-cite key="gehring2017convolutional"></d-cite> adopt this
architectural feature, introducing a fast, parallelizable model for machine
translation in the form of a fully convolutional network.
</p>
<figure class="content l-body" content-name="nlp" content-type="literature">
<%= require("../static/diagrams/convseq2seq.svg") %>
</figure>
<p class="content" content-name="nlp" content-type="literature">
The Gated-Attention Reader <d-cite key="dhingra2017gated"></d-cite>
uses feature-wise scaling, extracting information
from text by conditioning a document-reading network on a query. Its
architecture consists of multiple Gated-Attention modules, which involve
element-wise multiplications between document representation tokens and
token-specific query representations extracted via soft attention on the
query representation tokens.
</p>
<figure class="content l-body" content-name="nlp" content-type="literature">
<%= require("../static/diagrams/gated-attention-reader.svg") %>
</figure>
<button class="collapsible" content-name="reinforcement-learning" content-type="literature">Reinforcement learning<span style="float: right;">+</span></button>
<p class="content" content-name="reinforcement-learning" content-type="literature">
The Gated-Attention architecture <d-cite key="chaplot2017gated"></d-cite>
uses feature-wise sigmoidal gating to fuse linguistic and visual
information in an agent trained to follow simple "go-to" language
instructions in the VizDoom <d-cite key="kempa2016vizdoom"></d-cite> 3D
environment.
</p>
<figure class="content l-body" content-name="reinforcement-learning" content-type="literature">
<%= require("../static/diagrams/gated-attention.svg") %>
</figure>
<p class="content" content-name="reinforcement-learning" content-type="literature">
Bahdanau et al.<d-cite key="bahdanau2018learning"></d-cite> use FiLM
layers to condition Neural Module Network<d-cite key="andreas2016neural"></d-cite>
and LSTM<d-cite key="hochreiter1997long"></d-cite> -based policies to follow
basic, compositional language instructions (arranging objects and going
to particular locations) in a 2D grid world. They train this policy
in an adversarial manner using rewards from another FiLM-based network,
trained to discriminate between ground-truth examples of achieved
instruction states and failed policy trajectories states.
</p>
<p class="content" content-name="reinforcement-learning" content-type="literature">
Outside instruction-following, Kirkpatrick et al.
<d-cite key="kirkpatrick2017overcoming"></d-cite> also use
game-specific scaling and biasing to condition a shared policy network
trained to play 10 different Atari games.
</p>
<button class="collapsible" content-name="generative-modeling" content-type="literature">Generative modeling<span style="float: right;">+</span></button>
<p class="content" content-name="generative-modeling" content-type="literature">
The conditional variant of DCGAN <d-cite key="radford2016unsupervised"></d-cite>,
a well-recognized network architecture for generative adversarial networks
<d-cite key="goodfellow2014generative"></d-cite>, uses concatenation-based
conditioning. The class label is broadcasted as a feature map and then
concatenated to the input of convolutional and transposed convolutional
layers in the discriminator and generator networks.
</p>
<figure class="content l-body" content-name="generative-modeling" content-type="literature">
<%= require("../static/diagrams/dcgan.svg") %>
</figure>
<p class="content" content-name="generative-modeling" content-type="literature">
For convolutional layers, concatenation-based conditioning requires the
network to learn redundant convolutional parameters to interpret each
constant, conditioning feature map; as a result, directly applying a
conditional bias is more parameter efficient, but the two approaches are
still mathematically equivalent.
</p>
<p class="content" content-name="generative-modeling" content-type="literature">
PixelCNN <d-cite key="oord2016conditional"></d-cite>
and WaveNet <d-cite key="oord2016wavenet"></d-cite> — two recent
advances in autoregressive, generative modeling of images and audio,
respectively — use conditional biasing. The simplest form of
conditioning in PixelCNN adds feature-wise biases to all convolutional layer
outputs. In FiLM parlance, this operation is equivalent to inserting FiLM
layers after each convolutional layer and setting the scaling coefficients
to a constant value of 1.
<d-footnote>
The authors also describe a location-dependent biasing scheme which
cannot be expressed in terms of FiLM layers due to the absence of the
feature-wise property.
</d-footnote>
</p>
<figure class="content l-body" content-name="generative-modeling" content-type="literature">
<%= require("../static/diagrams/pixelcnn.svg") %>
</figure>
<p class="content" content-name="generative-modeling" content-type="literature">
WaveNet describes two ways in which conditional biasing allows external
information to modulate the audio or speech generation process based on
conditioning input:
</p>
<ol class="content" content-name="generative-modeling" content-type="literature">
<li>
<strong>Global conditioning</strong> applies the same conditional bias
to the whole generated sequence and is used e.g. to condition on speaker
identity.
</li>
<li>
<strong>Local conditioning</strong> applies a conditional bias which
varies across time steps of the generated sequence and is used e.g. to
let linguistic features in a text-to-speech model influence which sounds
are produced.
</li>
</ol>
<p class="content" content-name="generative-modeling" content-type="literature">
As in PixelCNN, conditioning in WaveNet can be viewed as inserting FiLM
layers after each convolutional layer. The main difference lies in how
the FiLM-generating network is defined: global conditioning
expresses the FiLM-generating network as an embedding lookup which is
broadcasted to the whole time series, whereas local conditioning expresses
it as a mapping from an input sequence of conditioning information to an
output sequence of FiLM parameters.
</p>
<button class="collapsible" content-name="speech-recognition" content-type="literature">Speech recognition<span style="float: right;">+</span></button>
<p class="content" content-name="speech-recognition" content-type="literature">
Kim et al.<d-cite key="kim2017dynamic"></d-cite> modulate a deep
bidirectional LSTM using a form
of conditional normalization. As discussed in the
<em>Visual question-answering</em> and <em>Style transfer</em> subsections,
conditional normalization can be seen as an instance of FiLM where
the post-normalization feature-wise affine transformation is replaced
with a FiLM layer.
</p>
<figure class="content l-body-outset" content-name="speech-recognition" content-type="literature">
<%= require("../static/diagrams/dln.svg") %>
</figure>
<p class="content" content-name="speech-recognition" content-type="literature">
The key difference here is that the conditioning signal does not come from
an external source but rather from utterance
summarization feature vectors extracted in each layer to adapt the model.
</p>
<button class="collapsible" content-name="domain-adaptation" content-type="literature">Domain adaptation and few-shot learning<span style="float: right;">+</span></button>
<p class="content" content-name="domain-adaptation" content-type="literature">
For domain adaptation, Li et al. <d-cite key="li2018adaptive"></d-cite>
find it effective to update the per-channel batch normalization
statistics (mean and variance) of a network trained on one domain with that
network's statistics in a new, target domain. As discussed in the
<em>Style transfer</em> subsection, this operation is akin to using the network as
both the FiLM generator and the FiLM-ed network. Notably, this approach,
along with Adaptive Instance Normalization, has the particular advantage of
not requiring any extra trainable parameters.
</p>
<p class="content" content-name="domain-adaptation" content-type="literature">
For few-shot learning, Oreshkin et al.
<d-cite key="oreshkin2018tadam"></d-cite> explore the use of FiLM layers to
provide more robustness to variations in the input distribution across
few-shot learning episodes. The training set for a given episode is used to
produce FiLM parameters which modulate the feature extractor used in a
Prototypical Networks <d-cite key="snell2017prototypical"></d-cite>
meta-training procedure.
</p>
<hr/>
<h2 id="related-ideas">Related ideas</h2>
<p>
Aside from methods which make direct use of feature-wise transformations,
the FiLM framework connects more broadly with the following methods and
concepts.
</p>
<div style="width: 100%">
<button class="expand-collapse-button" content-type="related">expand all</button>
</div>
<button class="collapsible" content-name="zero-shot" content-type="related">Zero-shot learning<span style="float: right;">+</span></button>
<p class="content" content-name="zero-shot" content-type="related">
The idea of learning a task representation shares a strong connection with
zero-shot learning approaches. In zero-shot learning, semantic task
embeddings may be learned from external information and then leveraged to
make predictions about classes without training examples. For instance, to
generalize to unseen object categories for image classification, one may
construct semantic task embeddings from text-only descriptions and exploit
objects' text-based relationships to make predictions for unseen image
categories. Frome et al. <d-cite key="frome2013devise"></d-cite>, Socher et
al. <d-cite key="socher2013zero"></d-cite>, and Norouzi et al.
<d-cite key="norouzi2014zero"></d-cite> are a few notable exemplars
of this idea.
</p>
<button class="collapsible" content-name="hypernetworks" content-type="related">HyperNetworks<span style="float: right;">+</span></button>
<p class="content" content-name="hypernetworks" content-type="related">
The notion of a secondary network predicting the parameters of a primary
network is also well exemplified by HyperNetworks <d-cite
key="ha2016hypernetworks"></d-cite>, which predict weights for entire layers
(e.g., a recurrent neural network layer). From this perspective, the FiLM
generator is a specialized HyperNetwork that predicts the FiLM parameters of
the FiLM-ed network. The main distinction between the two resides in the
number and specificity of predicted parameters: FiLM requires predicting far
fewer parameters than Hypernetworks, but also has less modulation potential.
The ideal trade-off between a conditioning mechanism's capacity,
regularization, and computational complexity is still an ongoing area of
investigation, and many proposed approaches lie on the spectrum between FiLM
and HyperNetworks (see <a href="#bibliographic-notes">Bibliographic Notes</a>).
</p>
<button class="collapsible" content-name="attention" content-type="related">Attention<span style="float: right;">+</span></button>
<p class="content" content-name="attention" content-type="related">
Some parallels can be drawn between attention and FiLM, but the two operate
in different ways which are important to disambiguate.
</p>
<figure class="content l-body-outset" content-name="attention" content-type="related" id="film-vs-attention-diagram">
<%= require("../static/diagrams/film-vs-attention.svg") %>
</figure>
<p class="content" content-name="attention" content-type="related">
This difference stems from distinct intuitions underlying attention and
FiLM: the former assumes that specific spatial locations or time steps
contain the most useful information, whereas the latter assumes that
specific features or feature maps contain the most useful information.
</p>
<button class="collapsible" content-name="bilinear" content-type="related">Bilinear transformations<span style="float: right;">+</span></button>
<p class="content" content-name="bilinear" content-type="related">
With a little bit of stretching, FiLM can be seen as a special case of a
bilinear transformation
<d-cite key="tenenbaum2000separating"></d-cite> with low-rank weight
matrices. A bilinear transformation defines the relationship between two
inputs <d-math>\mathbf{x}</d-math> and <d-math>\mathbf{z}</d-math> and the
<d-math>k^{th}</d-math> output feature <d-math>y_k</d-math> as
<d-math block>
y_k = \mathbf{x}^T W_k \mathbf{z}.
</d-math>
Note that for each output feature <d-math>y_k</d-math> we have a separate
matrix <d-math>W_k</d-math>, so the full set of weights forms a
multi-dimensional array.
</p>
<figure class="content l-body-outset" content-name="bilinear" content-type="related">
<%= require("../static/diagrams/bilinear.svg") %>
</figure>
<p class="content" content-name="bilinear" content-type="related">
If we view <d-math>\mathbf{z}</d-math> as the concatenation of the scaling
and shifting vectors <d-math>\gamma</d-math> and <d-math>\beta</d-math> and
if we augment the input <d-math>\mathbf{x}</d-math> with a 1-valued feature,
<d-footnote>
As is commonly done to turn a linear transformation into an affine
transformation.
</d-footnote>
we can represent FiLM using a bilinear transformation by zeroing out the
appropriate weight matrix entries:
</p>
<figure class="content l-body-outset" content-name="bilinear" content-type="related">
<%= require("../static/diagrams/film-as-bilinear.svg") %>
</figure>
<p class="content" content-name="bilinear" content-type="related">
For some applications of bilinear transformations,
see the <a href="#bibliographic-notes">Bibliographic Notes</a>.
</p>
<hr/>
<h2 id="properties-of-learned-representation">Properties of the learned task representation</h2>
<p>
As hinted earlier, in adopting the FiLM perspective we implicitly introduce
a notion of <em>task representation</em>: each task — be it a question
about an image or a painting style to imitate — elicits a different
set of FiLM parameters via the FiLM generator which can be understood as its
representation in terms of how to modulate the FiLM-ed network. To help
better understand the properties of this representation, let's focus on two
FiLM-ed models used in fairly different problem settings:
</p>
<ul>
<li>
The visual reasoning model of Perez et al.
<d-cite key="perez2017learning,perez2018film"></d-cite>, which uses FiLM
to modulate a visual processing pipeline based off an input question.
<figure class="l-body-outset">
<%= require("../static/diagrams/film-clevr.svg") %>
</figure>
</li>
<li>
The artistic style transfer model of Ghiasi et al.
<d-cite key="ghiasi2017exploring"></d-cite>, which uses FiLM to modulate a
feed-forward style transfer network based off an input style image.
<figure class="l-body-outset">
<%= require("../static/diagrams/alrfas.svg") %>
</figure>
</li>
</ul>
<p>
As a starting point, can we discern any pattern in the FiLM parameters as a
function of the task description? One way to visualize the FiLM parameter
space is to plot <d-math>\gamma</d-math> against <d-math>\beta</d-math>,
with each point corresponding to a specific task description and a specific
feature map. If we color-code each point according to the feature map it
belongs to we observe the following:
</p>
<figure class="l-body" id="gamma-beta-diagram">
<%= require("../static/diagrams/gamma-beta.svg") %>
</figure>
<p>
The plots above allow us to make several interesting observations. First,
FiLM parameters cluster by feature map in parameter space, and the cluster
locations are not uniform across feature maps. The orientation of these
clusters is also not uniform across feature maps: the main axis of variation
can be <d-math>\gamma</d-math>-aligned, <d-math>\beta</d-math>-aligned, or
diagonal at varying angles. These findings suggest that the affine
transformation in FiLM layers is not modulated in a single, consistent way,
i.e., using <d-math>\gamma</d-math> only, <d-math>\beta</d-math> only, or
<d-math>\gamma</d-math> and <d-math>\beta</d-math> together in some specific
way. Maybe this is due to the affine transformation being overspecified, or
maybe this shows that FiLM layers can be used to perform modulation
operations in several distinct ways.
</p>
<p>
Nevertheless, the fact that these parameter clusters are often somewhat
"dense" may help explain why the style transfer model of Ghiasi et al.
<d-cite key="ghiasi2017exploring"></d-cite> is able to perform style
interpolations: any convex combination of FiLM parameters is likely to
correspond to a meaningful parametrization of the FiLM-ed network.
</p>
<figure class="l-body-outset" id="style-interpolation-diagram">
<%= require("../static/diagrams/style-interpolation.svg") %>
</figure>
<p>
To some extent, the notion of interpolating between tasks using FiLM
parameters can be applied even in the visual question-answering setting.
Using the model trained in Perez et al. <d-cite key="perez2018film"></d-cite>,
we interpolated between the model's FiLM parameters for two pairs of CLEVR
questions. Here we visualize the input locations responsible for
the globally max-pooled features fed to the visual pipeline's output classifier:
</p>
<figure class="l-body" id="question-interpolation-diagram">
<%= require("../static/diagrams/question-interpolation.svg") %>
</figure>
<p>
The network seems to be softly switching where in the image it is looking,
based on the task description. It is quite interesting that these semantically
meaningful interpolation behaviors emerge, as the network has not been
trained to act this way.
</p>
<p>
Despite these similarities across problem settings, we also observe
qualitative differences in the way in which FiLM parameters cluster as a
function of the task description. Unlike the style transfer model, the
visual reasoning model sometimes exhibits several FiLM parameter
sub-clusters for a given feature map.
</p>
<figure class="l-body" id="clevr-subcluster-diagram">
<%= require("../static/diagrams/clevr-subcluster.svg") %>
</figure>
<p>
At the very least, this may indicate that FiLM learns to operate in ways
that are problem-specific, and that we should not expect to find a unified
and problem-independent explanation for FiLM's success in modulating FiLM-ed
networks. Perhaps the compositional or discrete nature of visual reasoning
requires the model to implement several well-defined modes of operation
which are less necessary for style transfer.
</p>
<p>
Focusing on individual feature maps which exhibit sub-clusters, we can try
to infer how questions regroup by color-coding the scatter plots by question
type.
</p>
<figure class="l-body" id="clevr-subcluster-color-diagram">
<%= require("../static/diagrams/clevr-subcluster-color.svg") %>
</figure>
<p>
Sometimes a clear pattern emerges, as in the right plot, where color-related
questions concentrate in the top-right cluster — we observe that
questions either are of type <em>Query color</em> or <em>Equal color</em>,
or contain concepts related to color. Sometimes it is harder to draw a
conclusion, as in the left plot, where question types are scattered across
the three clusters.
</p>
<p>
In cases where question types alone cannot explain the clustering of the
FiLM parameters, we can turn to the conditioning content itself to gain
an understanding of the mechanism at play. Let's take a look at two more
plots: one for feature map 26 as in the previous figure, and another
for a different feature map, also exhibiting several subclusters. This time
we regroup points by the words which appear in their associated question.
</p>
<figure class="l-body" id="clevr-subcluster-color-words-diagram">
<%= require("../static/diagrams/clevr-subcluster-color-words.svg") %>
</figure>
<p>
In the left plot, the left subcluster corresponds to questions involving
objects positioned <em>in front</em> of other objects, while the right
subcluster corresponds to questions involving objects positioned
<em>behind</em> other objects. In the right plot we see some evidence of
separation based on object material: the left subcluster corresponds to
questions involving <em>matte</em> and <em>rubber</em> objects, while the
right subcluster contains questions about <em>shiny</em> or
<em>metallic</em> objects.
</p>
<p>
The presence of sub-clusters in the visual reasoning model also suggests
that question interpolations may not always work reliably, but these
sub-clusters don't preclude one from performing arithmetic on the question
representations, as Perez et al. <d-cite key="perez2018film"></d-cite>
report.
</p>
<figure class="l-body">
<%= require("../static/diagrams/analogy.svg") %>
</figure>
<p>
Perez et al. <d-cite key="perez2018film"></d-cite> report that this sort of
task analogy is not always successful in correcting the model's answer, but
it does point to an interesting fact about FiLM-ed networks: sometimes the
model makes a mistake not because it is incapable of computing the correct
output, but because it fails to produce the correct FiLM parameters for a
given task description. The reverse can also be true: if the set of tasks
the model was trained on is insufficiently rich, the computational
primitives learned by the FiLM-ed network may be insufficient to ensure good
generalization. For instance, a style transfer model may lack the ability to
produce zebra-like patterns if there are no stripes in the styles it was
trained on. This could explain why Ghiasi et al.
<d-cite key="ghiasi2017exploring"></d-cite> report that their style transfer
model's ability to produce pastiches for new styles degrades if it has been
trained on an insufficiently large number of styles. Note however that in
that case the FiLM generator's failure to generalize could also play a role,
and further analysis would be needed to draw a definitive conclusion.
</p>
<p>
This points to a separation between the various computational
primitives learned by the FiLM-ed network and the "numerical recipes"
learned by the FiLM generator: the model's ability to generalize depends
both on its ability to parse new forms of task descriptions and on it having
learned the required computational primitives to solve those tasks. We note
that this multi-faceted notion of generalization is inherited directly from
the multi-task point of view adopted by the FiLM framework.
</p>
<p>
Let's now turn our attention back to the overal structural properties of FiLM
parameters observed thus far. The existence of this structure has already
been explored, albeit more indirectly, by Ghiasi et al.
<d-cite key="ghiasi2017exploring"></d-cite> as well as Perez et al.
<d-cite key="perez2018film"></d-cite>, who applied t-SNE
<d-cite key="maaten2008visualizing"></d-cite> on the FiLM parameter values.
</p>
<div class="l-body-outset">
<svg width="888" height="46" xmlns="http://www.w3.org/2000/svg">
<text x="0" y="20" dy="1em" class="figure-text">
t-SNE projection of FiLM parameters for many task descriptions.
</text>
<path d="M0,45l888,0" style="stroke-width: 1px; stroke: #666; opacity: 0.15;"/>
</svg>
</div>
<figure class="l-body-outset" id="tsne-diagram">
<%= require("../static/diagrams/tsne.svg") %>
</figure>
<p>
The projection on the left is inspired by a similar projection done by Perez
et al. <d-cite key="perez2018film"></d-cite> for their visual reasoning
model trained on CLEVR and shows how questions group by question type.
The projection on the right is inspired by a similar projection done by
Ghiasi et al. <d-cite key="ghiasi2017exploring"></d-cite> for their style
transfer network. The projection does not cluster artists as neatly as the
projection on the left, but this is to be expected, given that an artist's
style may vary widely over time. However, we can still detect interesting
patterns in the projection: note for instance the isolated cluster (circled
in the figure) in which paintings by Ivan Shishkin and Rembrandt are
aggregated. While these two painters exhibit fairly different styles, the
cluster is a grouping of their sketches.
</p>
<figure class="l-body" id="style-explained-diagram">
<%= require("../static/diagrams/style-explained.svg") %>
</figure>
<p>
To summarize, the way neural networks learn to use FiLM layers seems to
vary from problem to problem, input to input, and even from feature to
feature; there does not seem to be a single mechanism by which the
network uses FiLM to condition computation. This flexibility may
explain why FiLM-related methods have been successful across such a
wide variety of domains.
</p>
<hr/>
<h2 id="discussion">Discussion</h2>
<p>
Looking forward, there are still many unanswered questions.
Do these experimental observations on FiLM-based architectures generalize to
other related conditioning mechanisms, such as conditional biasing, sigmoidal
gating, HyperNetworks, and bilinear transformations? When do feature-wise
transformations outperform methods with stronger inductive biases and vice
versa? Recent work combines feature-wise transformations with stronger
inductive bias methods
<d-cite key="strub2018visual,bahdanau2018learning,yang2018dataset"></d-cite>,
which could be an optimal middle ground. Also, to what extent are FiLM's
task representation properties
inherent to FiLM, and to what extent do they emerge from other features
of neural networks (i.e. non-linearities, FiLM generator
depth, etc.)? If you are interested in exploring these or other
questions about FiLM, we recommend looking into the code bases for
FiLM models for <a href="https://github.com/ethanjperez/film">visual reasoning</a>
<d-cite key="perez2018film"></d-cite> and <a href="https://github.com/tensorflow/magenta/tree/master/magenta/models/arbitrary_image_stylization">style transfer</a>
<d-cite key="ghiasi2017exploring"></d-cite> which we used as a
starting point for our experiments here.
</p>
<p>
Finally, the fact that changes on the feature level alone are able to
compound into large and meaningful modulations of the FiLM-ed network is
still very surprising to us, and hopefully future work will uncover deeper
explanations. For now, though, it is a question that
evokes the even grander mystery of how neural networks in general compound
simple operations like matrix multiplications and element-wise
non-linearities into semantically meaningful transformations.
</p>
</d-article>
<d-appendix>
<h3 id="bibliographic-notes">Bibliographic Notes</h3>
<p>
Multiplicative interactions have succeeded on various tasks, ever since
they were introduced in vision as "mapping units" <d-cite key="hinton1981a"></d-cite>
and "dynamic mappings" <d-cite key="vonderMalsburg1994the"></d-cite>
around 40 years ago. These tasks include Character-level Language
Modeling<d-cite key="sutskever2011generating"></d-cite>,
Image Denoising<d-cite key="tang2012boltzmann"></d-cite>,
Pose Estimation<d-cite key="taylor2009factored"></d-cite>,
Tracking<d-cite key="ross2006combining,denil2012learning"></d-cite>,
Action Recognition<d-cite key="le2011learning,taylor2010convolutional"></d-cite>,
and, more generally, tasks involving relating or matching inputs, such as
from different modalities or points in time