"What I cannot create, I do not understand." – R. Feynman
Jonas Kubilius, Annelies Baeck

From What Scale of Representation Does Multivariate Pattern Analysis Decode Information?

The Journal of Neuroscience , 30 (20) , 6811-6812 article pdf

Review ofSwisher et al.

Functional magnetic resonance imaging (fMRI) is severely limited in the extent of detail that it can reveal. In practice, most fMRI studies investigate large areas subtending centimeters, such as the primary visual cortex or face-selective regions.Electrophysiological recordings, however, have demonstrated much smaller scales of organization, for example, columns codingonly for specific orientations within the primary visual cortex. These orientation columns are substantially smaller than the conventionally available fMRI resolution of 3 x 3 mm, so it was assumed that fMRI would be insensitive to these finely spaced orientation columns.

Nonetheless, by using multivariate analysis techniques, Kamitani and Tong (2005) demonstrated that it is possible to decode orientation information in the human brain even with conventional fMRI resolution. These multivariate techniques combine the information of multiple voxels (Haxby et al., 2001), thereby revealing a sensitivity to different orientations, even though individual voxels provide only a very weak sensitivity to this distinction. This sensitivity was argued to result from an unequal distribution of different orientation columns within each voxel, providing a subtle bias that multivariate techniques can exploit to discriminate between orientations (for review, see Norman et al., 2006, or Haynes & Rees, 2006).

However, given that there are also larger scales of organization within the primary visual cortex, such as those deriving from a preference for radial orientations (Sasaki et al., 2006) or the overrepresentation of cardinal orientations (Furmanski & Engel, 2000), is it plausible to suppose that the demonstrated decoding depends on fine-scale columnar organization rather than these larger structures? A possible role for such larger scales of organization was suggested by Op de Beeck’s (2010) demonstration that the decoding of orientation was unaffected by large-scale (8 mm) smoothing. If orientation decoding really depends on fine-scale variability in the distribution of columns, it is unclear how this signal could still be detected when blurred via smoothing. Swisher and colleagues (2010) attempted to resolve this question regarding the scale of representation that contributes to a successful orientation classification by testing classification performance with high resolution fMRI in cats (9.4 T) and humans (7 T) who viewed oriented gratings. In the fMRI data, the authors considered classification performance by applying high or low pass spatial filters, enabling them to determine the contribution of different spatial scales to successful multivariate classification analysis.

On one level, Swisher et al.’s (2010) results are clear: when scanning at very high spatial resolution, both human and cat orientation decoding performance appears to depend on a small, millimeter scale of organization. In particular, the authors found that including all spatial scales up to 1.2 mm in cats and 4–5 mm in humans results in a saturation point, beyond which adding even larger spatial scales does not improve multivariate pattern classification (Swisher et al., red lines in their Figures 1B and 2B). This result therefore rules out the possibility that orientation decoding relies on larger scales of organization in this case and suggests that the critical information for classification performance is included within these millimeter scales of activation. There are, of course, large-scale orientation maps present as well, such as radial bias and differential sensitivity to cardinal versus oblique orientations, but, crucially, the authors show that these organizations only manifested at a centimeter scale and were undetected at the fine scales where successful orientation decoding was achieved. These large-scale maps could be decoded when no fine-scale information was present, but this decoding ability increased as more of this fine-scale information was included (Swisher et al., a blue line in their Figures 1B and 2B).

Thus, Swisher et al. (2010) successfully demonstrated that orientation columns can, in principle, contribute to multivariate orientation decoding. However, as the authors suggest, this evidence should not be taken as a confirmation that decoding necessarily ** results from columns. To illustrate this point, consider the fact that even at scales substantially larger than orientation columns (greater than a few millimeters), the decoding performance remains significantly above chance. It is possible that there is some information of orientation columns remaining at these scales, which enables successful decoding. But it is also possible that such decoding results from some millimeter scale organization, based on some local orientation grouping. Indeed, several alternatives have already been described in the literature, including pinwheel organization at around 1.5–2 mm and a relationship between the ocular dominance columns and the orientation columns (Yacoub et al., 2008). In fact, other scales of organization might also be present, yet unknown at this point. Therefore, understanding the levels of orientation column organization and its strength might be interesting not only for the field of multivariate pattern analysis, but also in a broader context of investigating the organizational principles underlying the primary visual cortex. Potentially, such scales of organization could be observed in spatial autocorrelations of the orientation map data. Consider, for example, Supplementary Figure 2 in Yacoub et al. (2008), where such autocorrelation reveals a curious increase at the scale of 1.5 mm (as compared to shuffled maps), presumably indicating some columnar grouping at a few millimeter scale. At different scanning resolution, these millimeter scales might become more important for decoding than the columns themselves.

A comparison study between the present results and the commonly used 3 × 3 mm resolution scanning could address these questions. Although Swisher et al. (2010) found orientation columns to contribute most to the decoding at 1 × 1 mm resolution, this result might not generalize to lower resolution scans. In particular, at lower resolutions fine spatial frequency signals are greatly diminished, while millimeter scale organization might become more prominent. Thus, at this lower resolution (typically used in decoding studies), the decoding might actually reflect ways in which orientation columns are organized in groups rather than inhomogeneous biases in columnar placement. Swisher et al. (2010) provide data consistent with this possibility when reanalyzing Kamitani and Tong’s (2005) original data (collected at a conventional 3 × 3 mm resolution), in which large amounts of smoothing diminished classification performance only slightly, as if there was a stronger influence of larger scale maps. A comparison of classification performance of the data collected at a lower resolution and smoothed data of high resolution data on the same subjects could clarify this issue. If similar levels of organization are used in decoding orientation, both cases should elicit similar classification performance. Such results would be a strong indication that even in conventional fMRI, orientation decoding is closely linked to columnar organization and not millimeter-scale maps. However, until this evidence is obtained, at a conventional resolution orientation decoding can only be regarded as related to any organization from subvoxel size columns to a few millimeter local groupings.

Another possibility for addressing this question stems from the observation by Swisher et al. (2010) of a correspondence between the average distance of columns with a similar orientation preference and the saturation point in cats. The saturation point reflects the stage at which adding even larger scale information stops contributing to increased classification performance, which is achieved at 1.2–1.4 mm for cats (Swisher et al., Figure 1B). If column level structures really underpin classification performance, there should be a link between a saturation point and a column size. While for cats this notion is intuitive, as an average orientation column size matched the saturation point, it is less obvious for humans where saturation point was around 4–5 mm (much greater than the average spacing of columns). There are many reasons for such difference, for example, the increased head motion in humans (compared to anesthetized cats) or different voxel sizes used (.3125 × .3125 mm in cats, and 1 × 1 mm in humans), since noise will be diminished more rapidly when adding together information from smaller voxels, resulting in earlier saturation. On the other hand, the difference could in fact reflect millimeter scale organization, and not columns, contributing mostly to the decoding. So if the orientation columns were indeed underlying decoding performance in this case, in principle, such a relationship would be reflected in correlations between the observed individual differences in column size (Yacoub et al., 2008) and individual differences in the saturation point.

In summary, the observation that orientation decoding could be achieved by multivariate techniques within the primary visual cortex was originally assumed to reflect sensitivity to random biases in the placement of different orientation columns. However, the existence of larger scales of organization in the primary visual cortex suggested another scale of organization that could underpin successful decoding. Swisher et al. (2010) provide an important advance on this debate by demonstrating that, although such larger scales of organization exist, they do not contribute to classification performance. Challenges lie ahead in trying to dissociate the source of such information between subvoxel scale (columns) and millimeter scale (local clusters of columns) organization. We have highlighted several steps that could be taken to further our understanding of the scale of organization exploited with conventional fMRI resolutions. In particular, we proposed several ways to test the possibility that there may exist scales of organization smaller than the cardinal and radial bias ruled out by the reviewed paper that might reflect some means by which columns are organized to group together.


We are grateful to Lee de-Wit for his ideas and overall help with this paper, Marijke Brants, Hans Op de Beeck, and Jascha Swisher for enlightening discussions and feedback on the manuscript, and Essa Yacoub for clarifying comments. This work was supported by the Research Council of K.U. Leuven (IMPH/06/GHW) and by a Methusalem grant (METH/08/02) from the Flemish Government.


  • Furmanski CS, Engel SA (2000) An oblique effect in human primary visual cortex. Nat Neurosci 3:535-536.
  • Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P (2001) Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex. Science 293: 2425-2430.
  • Haynes J, Rees G (2006) Decoding mental states from brain activity in humans. Nat Rev Neurosci 7:523-534.
  • Kamitani Y, Tong F (2005) Decoding the visual and subjective contents of the human brain. Nat Neurosci 8:679-685.
  • Norman KA, Polyn SM, Detre GJ, Haxby JV (2006) Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends in Cognitive Sciences 10:424-430.
  • Op de Beeck HP (2010) Against hyperacuity in brain reading: Spatial smoothing does not hurt multivariate fMRI analyses? NeuroImage 49:1943-1948.
  • Sasaki Y, Rajimehr R, Kim BW, Ekstrom LB, Vanduffel W, Tootell RB (2006) The Radial Bias: A Different Slant on Visual Orientation Sensitivity in Human and Nonhuman Primates. Neuron 51: 661-670.
  • Swisher JD, Gatenby JC, Gore JC, Wolfe BA, Moon C, Kim S, Tong F (2010) Multiscale Pattern Analysis of Orientation-Selective Activity in the Primary Visual Cortex. J Neurosci 30:325-330.
  • Yacoub E, Harel N, Uğurbil K (2008) High-field fMRI unveils orientation columns in humans. Proceedings of the National Academy of Sciences 105:10607-10612.