Citing Permutation Test

2022-06-28

Source: vignettes/citing-permutation-test.Rmd

How to cite results using permuco

The permuco package includes several permutation methods as well as several multiple comparisons procedures. We propose here the citation commonly used in the literature to cite these methods. When reporting a results using permuco, we suggest to cite the permuco package (Frossard and Renaud 2021), the permutation methods and the multiple comparisons procedure (when using the clusterlm(...)):

Citing the permutation method

The method argument set the permutation method for the aovperm(...), lmperm(...) and clusterlm(...) functions. If you are using a fixed effect linear model (without + Error(...) in the formula) you can use the following references:

method = Reference Comments
freedmand_lane Freedman and Lane (1983) Default method in permuco
kennedy Kennedy (1995)
manly Manly (1991)
huh_jhun Huh and Jhun (2001)
Kherad Pajouh and Renaud (2010) 		Huh and Jhun (2001) propose it for factorial design
Kherad Pajouh and Renaud (2010) generalized it to linear model
terBraak ter Braak (1992)
dekker Dekker, Krackhardt, and Snijders (2007)

For repeated measures ANOVA/ANCOVA using aovperm(...) and clusterlm(...) (when the formula contains + Error(...)), the methods are:

method = Reference Comments
Rd_kheradPajouh_renaud Kherad-Pajouh and Renaud (2015) Default method in permuco
Rde_kheradPajouh_renaud Kherad-Pajouh and Renaud (2015)

Citing the multiple comparisons procedure

The clusterlm(...) function has several multiple comparisons procedures implemented. It can be chosen using the multcomp argument.

multcomp = Reference Parameters Comments
clustermass Maris and Oostenveld (2007)
Bullmore et al. (1999) 		threshold, aggr_FUN Default method in permuco.
Maris and Oostenveld (2007) is used by the EEG community.
troendle Troendle (1995)
minP Westfall and Young (1993) H, E The default parameters are suggested by Pernet et al. (2015).
tfce Smith and Nichols (2009)
bonferroni Dunn (1958)
holm Holm (1979)
benjamini_hochberg Benjamini and Hochberg (1995) Control the FDR.
clusterdepth Frossard and Renaud (2022) threshold

For completeness and full reproducibility of the results, you can also report the parameters used in the clustermass,tfce or clusterdepth methods.

Citing the dataset

The permuco package offers 3 data-sets as example. You will find the reference of the data below:

data-set Reference
attentionshifting_signal
attentionshifting_design 		Tipura, Renaud, and Pegna (2019)
emergencycost Heritier et al. (2009)
jpah2016 Cheval et al. (2016)

Reference

Benjamini, Yoav, and Yosef Hochberg. 1995. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society B 57 (1): 289–300.
Bullmore, Edward T., John Suckling, Stephan Overmeyer, Sophia Rabe-Hesketh, Eric Taylor, and Michael J. Brammer. 1999. “Global, Voxel, and Cluster Tests, by Theory and Permutation, for a Difference Between Two Groups of Structural MR Images of the Brain.” IEEE Transactions on Medical Imaging 18 (1): 32–42. https://doi.org/10.1109/42.750253.
Cheval, Boris, Philippe Sarrazin, Luc Pelletier, and Malte Friese. 2016. “Effect of Retraining Approach-Avoidance Tendencies on an Exercise Task: A Randomized Controlled Trial.” Journal of Physical Activity and Health 13 (12): 1396–1403. https://doi.org/10.1123/jpah.2015-0597.
Dekker, David, David Krackhardt, and Tom A. B. Snijders. 2007. “Sensitivity of MRQAP Tests to Collinearity and Autocorrelation Conditions.” Psychometrika 72 (4): 563–81. https://doi.org/10.1007/s11336-007-9016-1.
Dunn, Olive Jean. 1958. “Estimation of the Means of Dependent Variables.” The Annals of Mathematical Statistics 29 (4): 1095–1111.
Freedman, David, and David Lane. 1983. “A Nonstochastic Interpretation of Reported Significance Levels.” Journal of Business & Economic Statistics 1 (4): 292. https://doi.org/10.2307/1391660.
Frossard, Jaromil, and Olivier Renaud. 2021. Permutation Tests for Regression, ANOVA, and Comparison of Signals: The permuco Package. Journal of Statistical Software. Vol. 99. https://doi.org/10.18637/jss.v099.i15.
———. 2022. “The Cluster Depth Tests: Toward Point-Wise Strong Control of the Family-Wise Error Rate in Massively Univariate Tests with Application to M/EEG.” NeuroImage 247 (February): 118824. https://doi.org/10.1016/j.neuroimage.2021.118824.
Heritier, Stephane, Eva Cantoni, Samuel Copt, and Maria-Pia Victoria-Feser. 2009. Robust Methods in Biostatistics. John Wiley & Sons.
Holm, Sture. 1979. “A Simple Sequentially Rejective Multiple Test Procedure.” Scandinavian Journal of Statistics 6 (2): 65–70.
Huh, Myung-Hoe, and Myoungshic Jhun. 2001. “Random Permutation Testing in Multiple Linear Regression.” Communications in Statistics - Theory and Methods 30 (10): 2023–32. https://doi.org/10.1081/STA-100106060.
Kennedy, Peter E. 1995. “Randomization Tests in Econometrics.” Journal of Business & Economic Statistics 13 (1): 85. https://doi.org/10.2307/1392523.
Kherad Pajouh, Sara, and Olivier Renaud. 2010. “An Exact Permutation Method for Testing Any Effect in Balanced and Unbalanced Fixed Effect ANOVA.” Computational Statistics & Data Analysis 54: 1881–93. https://doi.org/10.1016/j.csda.2010.02.015.
Kherad-Pajouh, Sara, and Olivier Renaud. 2015. “A General Permutation Approach for Analyzing Repeated Measures ANOVA and Mixed-Model Designs.” Statistical Papers 56 (4): 947–67. https://doi.org/10.1007/s00362-014-0617-3.
Manly, Bryan F. J. 1991. Randomization, Bootstrap and Monte Carlo Methods in Biology. Chapman and Hall/CRC.
Maris, Eric, and Robert Oostenveld. 2007. “Nonparametric Statistical Testing of EEG- and MEG-Data.” Journal of Neuroscience Methods 164 (1): 177–90. https://doi.org/10.1016/j.jneumeth.2007.03.024.
Pernet, C. R., M. Latinus, T. E. Nichols, and G. A. Rousselet. 2015. “Cluster-Based Computational Methods for Mass Univariate Analyses of Event-Related Brain Potentials/Fields: A Simulation Study.” Journal of Neuroscience Methods 250: 85–93. https://doi.org/10.1016/j.jneumeth.2014.08.003.
Smith, S, and T Nichols. 2009. “Threshold-Free Cluster Enhancement: Addressing Problems of Smoothing, Threshold Dependence and Localisation in Cluster Inference.” NeuroImage 44 (1): 83–98. https://doi.org/10.1016/j.neuroimage.2008.03.061.
ter Braak, Cajo J. F. 1992. “Permutation Versus Bootstrap Significance Tests in Multiple Regression and Anova.” In Bootstrapping and Related Techniques, edited by Karl-Heinz Jöckel, Günter Rothe, and Wolfgang Sendler, 79–85. Berlin, Heidelberg: Springer-Verlag. https://doi.org/10.1007/978-3-642-48850-4_10.
Tipura, E., O. Renaud, and A. J. Pegna. 2019. “Attention Shifting and Subliminal Cueing Under High Attentional Load: An EEG Study Using Emotional Faces.” Neuroreport, October. https://doi.org/10.1097/WNR.0000000000001349.
Troendle, James F. 1995. “A Stepwise Resampling Method of Multiple Hypothesis Testing.” Journal of the American Statistical Association 90 (429): 370–78. https://doi.org/10.1080/01621459.1995.10476522.
Westfall, Peter H., and S. Stanley Young. 1993. Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment. 1 edition. New York: John Wiley & Sons.