outsider - package review

Reviewer: @nuest

This is my first review for ROpenSci - exciting! Thank you for the opportunity!

Review Submitted:




This report contains documents associated with the review of rOpenSci submitted package:

outsider: ropensci/software-review issue #282.


Package info

Description:

Install and run external command-line programs in R through use of β€˜Docker’ https://www.docker.com/ and β€˜GitHub’ http://github.com/.

Author: Dom Bennett dominic.john.bennett@gmail.com [aut, cre] (https://orcid.org/0000-0003-2722-1359)

repo url: https://antonellilab.github.io/outsider

website url: https://antonellilab.github.io/outsider/

Review info

See reviewer guidelines for further information on the rOpenSci review process.

key review checks:

  • Does the code comply with general principles in the Mozilla reviewing guide?
  • Does the package comply with the ROpenSci packaging guide?
  • Are there improvements that could be made to the code style?
  • Is there code duplication in the package that should be reduced?
  • Are there user interface improvements that could be made?
  • Are there performance improvements that could be made?
  • Is the documentation (installation instructions/vignettes/examples/demos) clear and sufficient?

Please be respectful and kind to the authors in your reviews. The rOpenSci code of conduct is mandatory for everyone involved in our review process.


session info

sessionInfo()
R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.2 LTS

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=en_GB.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=de_DE.UTF-8        LC_COLLATE=en_GB.UTF-8    
 [5] LC_MONETARY=de_DE.UTF-8    LC_MESSAGES=en_GB.UTF-8   
 [7] LC_PAPER=de_DE.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=de_DE.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base     

other attached packages:
[1] magrittr_1.5

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.0         xmlparsedata_1.0.2 compiler_3.5.2    
 [4] prettyunits_1.0.2  remotes_2.0.2      tools_3.5.2       
 [7] testthat_2.0.1     digest_0.6.18      pkgbuild_1.0.2    
[10] pkgload_1.0.2      praise_1.0.0       jsonlite_1.6      
[13] evaluate_0.13      memoise_1.1.0      rlang_0.3.1       
[16] rex_1.1.2          whoami_1.2.0       cli_1.0.1         
[19] rstudioapi_0.9.0   yaml_2.2.0         xopen_1.0.0       
[22] xfun_0.5           cyclocomp_1.1.0    pkgreviewr_0.1.0  
[25] xml2_1.2.0         httr_1.4.0         withr_2.1.2       
[28] knitr_1.21         desc_1.2.0         fs_1.2.6          
[31] devtools_2.0.1     rprojroot_1.3-2    glue_1.3.0        
[34] R6_2.4.0           processx_3.2.1     rcmdcheck_1.3.2   
[37] rmarkdown_1.11     sessioninfo_1.1.1  lintr_1.0.3       
[40] callr_3.1.1        covr_3.2.1         backports_1.1.3   
[43] ps_1.3.0           clisymbols_1.2.0   htmltools_0.3.6   
[46] usethis_1.4.0      rsconnect_0.8.13   assertthat_0.2.0  
[49] goodpractice_1.0.2 lazyeval_0.2.1     crayon_1.3.4      

Test installation

test local outsider install:

devtools::install(pkg_dir, dependencies = T, build_vignettes = T)
These packages have more recent versions available.
Which would you like to update?

1:   knitr   (1.21  -> 1.22 ) [CRAN]
2:   openssl (1.2.1 -> 1.2.2) [CRAN]
3:   CRAN packages only
4:   All
5:   None

Enter one or more numbers separated by spaces, or an empty line to cancel
4
knitr   (1.21  -> 1.22 ) [CRAN]
openssl (1.2.1 -> 1.2.2) [CRAN]
Installing 2 packages: knitr, openssl
Installing packages into β€˜/home/daniel/R/x86_64-pc-linux-gnu-library/3.5’
(as β€˜lib’ is unspecified)
trying URL 'https://cloud.r-project.org/src/contrib/knitr_1.22.tar.gz'
Content type 'application/x-gzip' length 928365 bytes (906 KB)
==================================================
downloaded 906 KB

trying URL 'https://cloud.r-project.org/src/contrib/openssl_1.2.2.tar.gz'
Content type 'application/x-gzip' length 1196460 bytes (1.1 MB)
==================================================
downloaded 1.1 MB

* installing *source* package β€˜knitr’ ...
** package β€˜knitr’ successfully unpacked and MD5 sums checked
** R
** demo
** inst
** tests
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (knitr)
* installing *source* package β€˜openssl’ ...
** package β€˜openssl’ successfully unpacked and MD5 sums checked
Found pkg-config cflags and libs!
Using PKG_CFLAGS=
Using PKG_LIBS=-l:libssl.so.1.1 -l:libcrypto.so.1.1
** libs
rm -f aes.o base64.o bignum.o cert.o compatibility.o diffie.o envelope.o error.o hash.o info.o keygen.o keys.o onload.o openssh.o password.o pem.o pkcs12.o pkcs7.o rand.o rsa.o signing.o ssl.o stream.o write.o openssl.so
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c aes.c -o aes.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c base64.c -o base64.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c bignum.c -o bignum.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c cert.c -o cert.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c compatibility.c -o compatibility.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c diffie.c -o diffie.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c envelope.c -o envelope.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c error.c -o error.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c hash.c -o hash.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c info.c -o info.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c keygen.c -o keygen.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c keys.c -o keys.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c onload.c -o onload.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c openssh.c -o openssh.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c password.c -o password.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c pem.c -o pem.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c pkcs12.c -o pkcs12.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c pkcs7.c -o pkcs7.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c rand.c -o rand.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c rsa.c -o rsa.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c signing.c -o signing.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c ssl.c -o ssl.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c stream.c -o stream.o
gcc -std=gnu99 -I"/usr/share/R/include" -DNDEBUG      -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c write.c -o write.o
gcc -std=gnu99 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o openssl.so aes.o base64.o bignum.o cert.o compatibility.o diffie.o envelope.o error.o hash.o info.o keygen.o keys.o onload.o openssh.o password.o pem.o pkcs12.o pkcs7.o rand.o rsa.o signing.o ssl.o stream.o write.o -l:libssl.so.1.1 -l:libcrypto.so.1.1 -L/usr/lib/R/lib -lR
installing to /home/daniel/R/x86_64-pc-linux-gnu-library/3.5/openssl/libs
** R
** inst
** tests
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (openssl)

The downloaded source packages are in
    β€˜/tmp/RtmpatNG5b/downloaded_packages’
Adding β€˜knitr_1.22.tar.gz’ to the cache
Adding β€˜openssl_1.2.2.tar.gz’ to the cache
Adding β€˜knitr_1.22_R_x86_64-pc-linux-gnu.tar.gz’ to the cache
Adding β€˜openssl_1.2.2_R_x86_64-pc-linux-gnu.tar.gz’ to the cache
  
   checking for file β€˜/home/daniel/Documents/2019_ROpenSci-review/outsider/DESCRIPTION’ ...
  
βœ”  checking for file β€˜/home/daniel/Documents/2019_ROpenSci-review/outsider/DESCRIPTION’

  
─  preparing β€˜outsider’:

  
   checking DESCRIPTION meta-information ...
  
   checking DESCRIPTION meta-information ... 
  
βœ”  checking DESCRIPTION meta-information

  
─  checking for LF line-endings in source and make files and shell scripts

  
─  checking for empty or unneeded directories
   Removed empty directory β€˜outsider/.github’
   Removed empty directory β€˜outsider/docs’
   Removed empty directory β€˜outsider/examples’
   Removed empty directory β€˜outsider/other’
   Removed empty directory β€˜outsider/paper’
   Removed empty directory β€˜outsider/pkgdown’
   Removed empty directory β€˜outsider/vignettes’

  
─  building β€˜outsider_0.0.0.tar.gz’

  
   

Running /usr/lib/R/bin/R CMD INSTALL \
  /tmp/RtmpatNG5b/outsider_0.0.0.tar.gz --install-tests 

-
* installing to library β€˜/home/daniel/R/x86_64-pc-linux-gnu-library/3.5’

|
* installing *source* package β€˜outsider’ ...

-
** R

|
** inst

-
** tests

|
** byte-compile and prepare package for lazy loading

-

|

-
** help

|
*** installing help indices

-
** building package indices

|
** testing if installed package can be loaded

-
* DONE (outsider)

|

-

 
remove.packages("outsider")
Removing package from β€˜/home/daniel/R/x86_64-pc-linux-gnu-library/3.5’
(as β€˜lib’ is unspecified)
Error in remove.packages : there is no package called β€˜outsider’

comments:

  • Installation works smoothly from source.
  • You list β€œdocker” as a system dependency. Have considered checking for existence of Docker at installation time? Really curious here because I also have a Docker-related package.

test install of outsider from GitHub with:

devtools::install_github("AntonelliLab/outsider", dependencies = T, build_vignettes = T)
Downloading GitHub repo AntonelliLab/outsider@master
  
   checking for file β€˜/tmp/RtmpatNG5b/remotes37c91ced7d13/AntonelliLab-outsider-2b31d50/DESCRIPTION’ ...
  
βœ”  checking for file β€˜/tmp/RtmpatNG5b/remotes37c91ced7d13/AntonelliLab-outsider-2b31d50/DESCRIPTION’

  
─  preparing β€˜outsider’:

  
   checking DESCRIPTION meta-information ...
  
βœ”  checking DESCRIPTION meta-information

  
─  checking for LF line-endings in source and make files and shell scripts

  
─  checking for empty or unneeded directories
   Removed empty directory β€˜outsider/.github’
   Removed empty directory β€˜outsider/docs’
   Removed empty directory β€˜outsider/examples’
   Removed empty directory β€˜outsider/other’
   Removed empty directory β€˜outsider/paper’

  
   Removed empty directory β€˜outsider/pkgdown’
   Removed empty directory β€˜outsider/vignettes’

  
─  building β€˜outsider_0.0.0.tar.gz’

  
   
Installing package into β€˜/home/daniel/R/x86_64-pc-linux-gnu-library/3.5’
(as β€˜lib’ is unspecified)
* installing *source* package β€˜outsider’ ...
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (outsider)

comments:

  • no problems to install from GitHub
  • the version number 0.0.0 seems curious to me though, I suggest to either go with the .9000 versioning as described here if you think this is a development state, if you think you have a first working version (as the β€œactive” repo status suggests), why not go with 0.1.0?

Check package integrity

run checks on outsider source:

devtools::check(pkg_dir)

comments:

  • during check, one example and one test fails on my system:
    • running the code from module_search.R manually works, but fails during check:
    E  checking examples (1.6s)
     Running examples in β€˜outsider-Ex.R’ failed
     The error most likely occurred in:
    
     > base::assign(".ptime", proc.time(), pos = "CheckExEnv")
     > ### Name: module_details
     > ### Title: Look up details on module(s)
     > ### Aliases: module_details
     > 
     > ### ** Examples
     > 
     > library(outsider)
     > # return table of ALL available modules on GitHub
     > # NOT RUN - takes too long
     > # (available_modules <- module_search())
     > 
     > # look-up whether specific module exists
     > # NOT RUN
     > # repo <- 'dombennett/om..goodbye.world'
     > # (suppressWarnings(module_exists(repo = repo)))
     > repo <- 'dombennett/om..hello.world'
     > (module_exists(repo = repo))
     Error in open.connection(con, "rb") : HTTP error 401.
     Calls: module_exists ... parse_and_simplify -> parseJSON -> parse_con -> open -> open.connection
     Execution halted
    • _ module_details() works (@test-search.R#94)_
    ── 1. Error: module_details() works (@test-search.R#94)  ───────────
    HTTP error 401.
    1: module_details(repo = c(repo, "dombennett/om..mafft")) at testthat/test-search.R:94
    2: lapply(X = repo, FUN = repo_search)
    3: FUN(X[[i]], ...)
    4: jsonlite::fromJSON(paste0(gh_search_repo_url, search_args))
    5: parse_and_simplify(txt = txt, simplifyVector = simplifyVector, simplifyDataFrame = simplifyDataFrame, 
           simplifyMatrix = simplifyMatrix, flatten = flatten, ...)
    6: parseJSON(txt, bigint_as_char)
    7: parse_con(txt, bigint_as_char)
    8: open(con, "rb")
    9: open.connection(con, "rb")

run tests on outsider source:

devtools::test(pkg_dir)
Loading outsider

Attaching package: β€˜testthat’

The following objects are masked from β€˜package:magrittr’:

    equals, is_less_than, not

Testing outsider
βœ” | OK F W S | Context

⠏ |  0       | Testing 'args'
β ‹ |  1       | Testing 'args'
β ™ |  2       | Testing 'args'
β Ή |  3       | Testing 'args'
β Έ |  4       | Testing 'args'
β Ό |  5       | Testing 'args'
β ΄ |  6       | Testing 'args'
β ¦ |  7       | Testing 'args'
β § |  8       | Testing 'args'
β ‡ |  9       | Testing 'args'
⠏ | 10       | Testing 'args'
β ‹ | 11       | Testing 'args'
β ™ | 12       | Testing 'args'
β Ή | 13       | Testing 'args'
β Έ | 14       | Testing 'args'
β Ό | 15       | Testing 'args'
β ΄ | 16       | Testing 'args'
β ¦ | 17       | Testing 'args'
β § | 18       | Testing 'args'
β ‡ | 19       | Testing 'args'
⠏ | 20       | Testing 'args'
β ‹ | 21       | Testing 'args'
βœ” | 21       | Testing 'args'

β ™ | 22       | Testing 'args'
β Ή | 23       | Testing 'args'
β Έ | 24       | Testing 'args'
β Ό | 25       | Testing 'args'
β ΄ | 26       | Testing 'args'
β ¦ | 27       | Testing 'args'this: 'that'
other ... 
... a: 'a'
... b: 'b'

β § | 28       | Testing 'args'
β ‡ | 29       | Testing 'args'
⠏ | 30       | Testing 'args'
β ‹ | 31       | Testing 'args'
β ™ | 32       | Testing 'args'
β Ή | 33       | Testing 'args'
β Έ | 34       | Testing 'args'
β Ό | 35       | Testing 'args'
β ΄ | 36       | Testing 'args'Hah! The module works! You are impressive!

β ¦ | 37       | Testing 'args'A truly unfavourable circumstance: The module is not working....
But keep on making! You're doing fortunately!

β § | 38       | Testing 'args'
⠏ |  0       | Testing 'console'
β ‹ |  1       | Testing 'console'
β ™ |  2       | Testing 'console'
β Ή |  3       | Testing 'console'cat this

β Έ |  4       | Testing 'console'Wow! The module works! You are neat!
Huzzah! The module works! You are rad!
Gee! The module works! You are cool!
Huh! The module works! You are funkadelic!
Whee! The module works! You are epic!
Yahoo! The module works! You are rad!
Oh! The module works! You are flawless!
Mmm! The module works! You are awe-inspiring!
Whoa! The module works! You are remarkable!
Mmm! The module works! You are awesome!

β Ό |  5       | Testing 'console'Well drop the goat. This is not great. The module is not working....
But keep on forming! You're doing frankly!
Ah shucks.... The module is not working....
But keep on making! You're doing gently!
Ah shucks.... The module is not working....
But keep on setting up! You're doing gently!
I am sorry. I feel bad. The module is not working....
But keep on brewing! You're doing eagerly!
Damn damn damn. The module is not working....
But keep on designing! You're doing eagerly!
Well drop the goat. This is not great. The module is not working....
But keep on forging! You're doing elegantly!
Grrrr..... The module is not working....
But keep on organizing! You're doing tenderly!
Well drop the goat. This is not great. The module is not working....
But keep on assembling! You're doing honestly!
A truly unfavourable circumstance: The module is not working....
But keep on making! You're doing justly!
A truly unfavourable circumstance: The module is not working....
But keep on devising! You're doing quietly!

β ΄ |  6       | Testing 'console'
βœ” |  6       | Testing 'console'

⠏ |  0       | Testing 'container'
β ‹ |  1       | Testing 'container'
β ™ |  2       | Testing 'container'
β Ή |  3       | Testing 'container'
β Έ |  4       | Testing 'container'
β Ό |  5       | Testing 'container'────────────────────────────────────────────────────────────────────
Docker container details:
Image 'dombennett/om_hello.world'
Container 'om_hello.world_0'
Tag 'latest'
Status 'This is a mock'
────────────────────────────────────────────────────────────────────
Outsider module details:
Repo 'dombennett/om..hello.world'
R package 'om..hello.world..dombennett'
Program 'hello.world'
────────────────────────────────────────────────────────────────────

β ΄ |  6       | Testing 'container'
-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

 

β ¦ |  7       | Testing 'container'
-

|

-

|

-

|

-

 

β § |  8       | Testing 'container'
-

|

 

β ‡ |  9       | Testing 'container'
-

|

 

⠏ | 10       | Testing 'container'
β ‹ | 11       | Testing 'container'
β ™ | 12       | Testing 'container'
β Ή | 13       | Testing 'container'
-

|

-

 

β Έ | 14       | Testing 'container'
β Ό | 15       | Testing 'container'
β ΄ | 16       | Testing 'container'
-

|

-

|

 

-

|

 

β ¦ | 17       | Testing 'container'
-

|

 

βœ” | 17       | Testing 'container' [16.2 s]

⠏ |  0       | Testing 'docker'
β ‹ |  1       | Testing 'docker'
β ™ |  2       | Testing 'docker'
β Ή |  3       | Testing 'docker'
β Έ |  4       | Testing 'docker'
β Ό |  5       | Testing 'docker'
β ΄ |  6       | Testing 'docker'
β ¦ |  7       | Testing 'docker'
-


|
Usage:  docker [OPTIONS] COMMAND

A self-sufficient runtime for containers

Options:
      --config string      Location of client config files (default
                           "/home/daniel/.docker")
  -D, --debug              Enable debug mode
  -H, --host list          Daemon socket(s) to connect to
  -l, --log-level string   Set the logging level
                           ("debug"|"info"|"warn"|"error"|"fatal")
                           (default "info")
      --tls                Use TLS; implied by --tlsverify
      --tlscacert string   Trust certs signed only by this CA (default
                           "/home/daniel/.docker/ca.pem")
      --tlscert string     Path to TLS certificate file (default
                           "/home/daniel/.docker/cert.pem")
      --tlskey string      Path to TLS key file (default
                           "/home/daniel/.docker/key.pem")
      --tlsverify          Use TLS and verify the remote
  -v, --version            Print version information and quit

Management Commands:
  builder     Manage builds
  config      Manage Docker configs
  container   Manage containers
  engine      Manage the docker engine
  image       Manage images
  network     Manage networks
  node        Manage Swarm nodes
  plugin      Manage plugins
  secret      Manage Docker secrets
  service     Manage services
  stack       Manage Docker stacks
  swarm       Manage Swarm
  system      Manage Docker
  trust       Manage trust on Docker images
  volume      Manage volumes

Commands:
  attach      Attach local standard input, output, and error streams to a running container
  build       Build an image from a Dockerfile
  commit      Create a new image from a container's changes
  cp          Copy files/folders between a container and the local filesystem
  create      Create a new container
  diff        Inspect changes to files or directories on a container's filesystem
  events      Get real time events from the server
  exec        Run a co
-
mmand in a running container
  export      Export a container's filesystem as a tar archive
  history     Show the history of an image
  images      List images
  import      Import the contents from a tarball to create a filesystem image
  info        Display system-wide information
  inspect     Return low-level information on Docker objects
  kill        Kill one or more running containers
  load        Load an image from a tar archive or STDIN
  login       Log in to a Docker registry
  logout      Log out from a Docker registry
  logs        Fetch the logs of a container
  pause       Pause all processes within one or more containers
  port        List port mappings or a specific mapping for the container
  ps          List containers
  pull        Pull an image or a repository from a registry
  push        Push an image or a repository to a registry
  rename      Rename a container
  restart     Restart one or more containers
  rm          Remove one or more containers
  rmi         Remove one or more images
  run         Run a command in a new container
  save        Save one or more images to a tar archive (streamed to STDOUT by default)
  search      Search the Docker Hub for images
  start       Start one or more stopped containers
  stats       Display a live stream of container(s) resource usage statistics
  stop        Stop one or more running containers
  tag         Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
  top         Display the running processes of a container
  unpause     Unpause all processes within one or more containers
  update      Update configuration of one or more containers
  version     Show the Docker version information
  wait        Block until one or more containers stop, then print their exit codes

Run 'docker COMMAND --help' for more information on a command.

|

-

 

β § |  8       | Testing 'docker'
-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

 

β ‡ |  9       | Testing 'docker'
-

|

 
unable to prepare context: path "url" not found

-

|

 

⠏ | 10       | Testing 'docker'
-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

|

-

 

β ‹ | 11       | Testing 'docker'
-

|

 

β ™ | 12       | Testing 'docker'
β Ή | 13       | Testing 'docker'
β Έ | 14       | Testing 'docker'
β Ό | 15       | Testing 'docker'
β ΄ | 16       | Testing 'docker'
β ¦ | 17       | Testing 'docker'
βœ” | 17       | Testing 'docker' [16.9 s]

⠏ |  0       | Testing 'identities'
β ‹ |  1       | Testing 'identities'
β ™ |  2       | Testing 'identities'
β Ή |  3       | Testing 'identities'
β Έ |  4       | Testing 'identities'
β Ό |  5       | Testing 'identities'
β ΄ |  6       | Testing 'identities'
β ¦ |  7       | Testing 'identities'
β § |  8       | Testing 'identities'
βœ” |  8       | Testing 'identities'

⠏ |  0       | Testing 'install'
β ‹ |  1       | Testing 'install'
β ™ |  2       | Testing 'install'
β Ή |  3       | Testing 'install'
β Έ |  4       | Testing 'install'
β Ό |  5       | Testing 'install'
β ΄ |  6       | Testing 'install'
β ¦ |  7       | Testing 'install'
β § |  8       | Testing 'install'
β ‡ |  9       | Testing 'install'
⠏ | 10       | Testing 'install'
β ‹ | 11       | Testing 'install'
β ™ | 12       | Testing 'install'
β Ή | 13       | Testing 'install'
β Έ | 14       | Testing 'install'
β Ό | 15       | Testing 'install'
β ΄ | 16       | Testing 'install'
β ¦ | 17       | Testing 'install'
β § | 18       | Testing 'install'
βœ” | 18       | Testing 'install' [0.7 s]

⠏ |  0       | Testing 'log'
β ‹ |  1       | Testing 'log'
β ™ |  2       | Testing 'log'
β Ή |  3       | Testing 'log'
β Έ |  4       | Testing 'log'
β Ό |  5       | Testing 'log'
β ΄ |  6       | Testing 'log'
β ¦ |  7       | Testing 'log'
βœ” |  7       | Testing 'log'

⠏ |  0       | Testing 'outsider'
β ‹ |  1       | Testing 'outsider'────────────────────────────────────────────────────────────────────
Outsider module:
Repo 'user/repo'
Package 'repo..user'
Command NA
Args 
Files to send 
Working dir 
Container image 
Container name 
Container tag 
Container status 'This is a mock'
────────────────────────────────────────────────────────────────────

β ™ |  2       | Testing 'outsider'
β Ή |  3       | Testing 'outsider'
β Έ |  4       | Testing 'outsider'
β Ό |  5       | Testing 'outsider'────────────────────────────────────────────────────────────────────
Outsider module:
Repo 'user/repo'
Package 'repo..user'
Command 'cmd'
Args 
Files to send file
Working dir 'wd'
Container image 
Container name 
Container tag 
Container status 'Not running'
────────────────────────────────────────────────────────────────────

β ΄ |  6       | Testing 'outsider'────────────────────────────────────────────────────────────────────
Outsider module:
Repo 'user/repo'
Package 'repo..user'
Command 'cmd'
Args 
Files to send file
Working dir 'wd'
Container image 
Container name 
Container tag 
Container status 'Not running'
────────────────────────────────────────────────────────────────────

β ¦ |  7       | Testing 'outsider'────────────────────────────────────────────────────────────────────
Outsider module:
Repo 'user/repo'
Package 'repo..user'
Command 'cmd'
Args 
Files to send file
Working dir 'wd'
Container image 
Container name 
Container tag 
Container status 'This is a mock'
────────────────────────────────────────────────────────────────────

β § |  8       | Testing 'outsider'
βœ” |  8       | Testing 'outsider' [0.8 s]

⠏ |  0       | Testing 'search'
β ‹ |  1       | Testing 'search'
β ™ |  2       | Testing 'search'
β Ή |  3       | Testing 'search'
β Έ |  4       | Testing 'search'
β Ό |  5       | Testing 'search'
β ΄ |  6       | Testing 'search'
β ¦ |  7       | Testing 'search'
β § |  8       | Testing 'search'
β ‡ |  9       | Testing 'search'
⠏ | 10       | Testing 'search'
β ‹ | 11       | Testing 'search'
β ™ | 12       | Testing 'search'
β Ή | 12   1   | Testing 'search'
β Έ | 13   1   | Testing 'search'
β Ό | 14   1   | Testing 'search'
βœ” | 14   1   | Testing 'search' [5.3 s]
────────────────────────────────────────────────────────────────────
test-search.R:94: warning: module_details() works
No 'dombennett/om..mafft' found.
────────────────────────────────────────────────────────────────────

⠏ |  0       | Testing 'internal'
β ‹ |  1       | Testing 'internal'
βœ” |  1       | Testing 'internal'

⠏ |  0       | Testing 'test'
β ‹ |  1       | Testing 'test'
β ™ |  2       | Testing 'test'
β Ή |  3       | Testing 'test'
β Έ |  4       | Testing 'test'
β Ό |  5       | Testing 'test'
β ΄ |  6       | Testing 'test'
βœ” |  6       | Testing 'test'

⠏ |  0       | Testing 'unittest'
β ‹ |  1       | Testing 'unittest'
β ™ |  2       | Testing 'unittest'
βœ” |  2       | Testing 'unittest'

══ Results ═════════════════════════════════════════════════════════
Duration: 41.2 s

OK:       142
Failed:   0
Warnings: 1
Skipped:  0

Woot!

comments:

  • no failing test here :tada: !

check outsider for goodpractice:

goodpractice::gp(pkg_dir)
Preparing: covr
Preparing: cyclocomp
  
   checking for file β€˜/tmp/RtmpatNG5b/remotes37c99256552/outsider/DESCRIPTION’ ...
  
βœ”  checking for file β€˜/tmp/RtmpatNG5b/remotes37c99256552/outsider/DESCRIPTION’

  
─  preparing β€˜outsider’:

  
   checking DESCRIPTION meta-information ...
  
βœ”  checking DESCRIPTION meta-information

  
─  checking for LF line-endings in source and make files and shell scripts

  
─  checking for empty or unneeded directories

  
─  building β€˜outsider_0.0.0.tar.gz’

  
   
* installing *source* package β€˜outsider’ ...
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (outsider)
Preparing: description
Preparing: lintr
Preparing: namespace
Preparing: rcmdcheck
── GP outsider ─────────────────────────────────────────────────────

It is good practice to

  βœ– write unit tests for all functions, and all
    package code in general. 94% of code lines are covered
    by test cases.

    R/args.R:138:NA
    R/container.R:34:NA
    R/container.R:106:NA
    R/container.R:135:NA
    R/container.R:149:NA
    ... and 33 more lines

  βœ– fix this R CMD check ERROR: Running examples
    in β€˜outsider-Ex.R’ failed The error most likely occurred
    in: > ### Name: module_details > ### Title: Look up
    details on module(s) > ### Aliases: module_details > >
    ### ** Examples > > library(outsider) > # return table
    of ALL available modules on GitHub > # NOT RUN - takes
    too long > # (available_modules <- module_search()) > >
    # look-up whether specific module exists > # NOT RUN > #
    repo <- 'dombennett/om..goodbye.world' > #
    (suppressWarnings(module_exists(repo = repo))) > repo <-
    'dombennett/om..hello.world' > (module_exists(repo =
    repo)) Error in open.connection(con, "rb") : HTTP error
    401. Calls: module_exists ... parse_and_simplify ->
    parseJSON -> parse_con -> open -> open.connection
    Execution halted
  βœ– checking tests ... ERROR Running the tests in
    β€˜tests/test-all.R’ failed. Last 13 lines of output: 3:
    FUN(X[[i]], ...) 4:
    jsonlite::fromJSON(paste0(gh_search_repo_url,
    search_args)) 5: parse_and_simplify(txt = txt,
    simplifyVector = simplifyVector, simplifyDataFrame =
    simplifyDataFrame, simplifyMatrix = simplifyMatrix,
    flatten = flatten, ...) 6: parseJSON(txt,
    bigint_as_char) 7: parse_con(txt, bigint_as_char) 8:
    open(con, "rb") 9: open.connection(con, "rb") ══
    testthat results
    ═══════════════════════════════════════════════ OK: 142
    SKIPPED: 0 FAILED: 1 1. Error: module_details() works
    (@test-search.R#94) Error: testthat unit tests failed
    Execution halted
──────────────────────────────────────────────────────────────────── 

comments:

  • some problems here as before
  • IMHO the coverage is reasonable

Check package metadata files

spell check

devtools::spell_check(pkg_dir)
  WORD       FOUND IN
etc        build.Rmd:45
filepath   docker_cp.Rd:10,12
           file_create.Rd:15
           is_filepath.Rd:5,17
revbayes   README.md:73
ropensci   README.md:8
           README.Rmd:18

comments:

  • in README.md: I recommend to have an empty line after headlines and a space after the #/## in the source code for readability
  • spellcheck: all fine
  • Add phylotaR and ape to Suggests because you use them in vignettes

Check documentation

online documentation: https://antonellilab.github.io/outsider/

  • Is the documentation (installation instructions/vignettes/examples/demos) clear and sufficient?

test outsider function help files:

help(package = "outsider")

comments:

  • …

test outsider vignettes:

vignette(package = "outsider")

comments:

  • The contents links on https://antonellilab.github.io/outsider/articles/advanced_build.html do not work yet/content is missing
  • outsider covers a pretty complex problem! With the existing examples of one very basic β€œhello world” and one (to me very) complex analysis pipeline, I see room for a further example that shows a bit more but is not that domain specific, e.g.Β loading some data from a CSV and running a regression model, saving that to a file. For me as a user it would be great to know how can I load my data, how call my function, and how access the output data.
  • Get Started
    • The code chunk using input_sequences.fasta does not work - would it be possible to let a user download (with R code) such a file and actually run that? I was a bit disappointed when I went though all previous steps and then was β€œstopped” unexpectedly.
    • users often want to know what’s going on, and outsider is doing quite a lot of magic - I suggest to mention log_set already in the get started documentation.
    • β€œUninstalling” - for users new to Docker, you could mention here how to removed the downloaded images.
  • phyologenetic vignette
    • suggets to add a link to the outsider module on GitHub (https://github.com/DomBennett/om..mafft), potentially even to specific files, to help users explore this more complex module with the help of the vignette
    • is there are particular reason for the repeated β€œlibrary(outsider)” statements in each section?
    • Can you make the output of module_help(repo = 'dombennett/om..mafft', fname = 'mafft') more helpful? Right now it only says β€œRun mafft”. I think for the β€œstory” of the vignette it would be much better to actually have some documentation there, e.g.Β reference papers and describe the function.
    • In section β€œAlignment” - what should I look for as a user at the end? Can you add some visualisation here or describe what a user could do now with the output of the process?

Test functionality:

  • Are there user interface improvements that could be made?
  • Are there performance improvements that could be made?
library("outsider")
exports <- ls("package:outsider")
exports
# partial examples only!

outsider:::vars_get

outsider:::.run.outsider

outsider:::repo_search("nuest/outsider")
Sys.getenv("GITHUB_PAT")

comments:

  • Is there a way to list all function names in a module? The Get Started vignette says β€œOnce a function name is known of a particular ..” - can I only do that via the help file? MAybe a module_list(repo = 'dombennett/om..mafft') is feasible to implement.
  • Do you have any further development plans? I can imagine further developments (like building pipelines using a chain of modules), maybe you can put your ideas into GitHub issues for everyone to learn about them?
  • see comment about dev-functions starting with .

Inspect code:

pkgreviewr::pkgreview_print_source("outsider")

* might not be suitable for large packages with many exported functions



comments:

  • Idea: .module_validate(..) to validate the formal structure of a module - I now see there is .module_check :-) - I suggest to put this β€œTODO” into a GitHub issue so that potential collaborators see this is an open task, and not have a function that a user cann access with ::: without any functionality. Or at least add a warning("not implemented")
  • I think the package does a good job to distinguish between user facing functions and internal functions (like test())
  • formatting is alright
  • for pkgdetails_get I personally would rely on the package desc to do the parsing - you already have a number of imports and this lightweight one should not hurt
  • .module_travis - your approach means that the function will fail if the repo mooves, or if there is no internet connection… for me it’s always a signal to reconsider the approach if you have a constant like a URL within a function body. I think it would be better to manage the β€œtemplate” file within the codebase of the package.
  • container.R: your package code probable precedes stevedore, but if you find yourself writing code around calling docker with sys::exec.. then it might be worth the effort to switch; things like running containers remotely (if that is what you mean by β€œremote hosting”) would come very cheaply then;
  • search.R:
    • β€œcode you wrote 6 months ago might just have been written by someone else”: I suggest to add a warning after tkn <- NULL to make it more obvious when requests fail because of a missing token (and not because the token is not valid etc.)
    • info[['url']] <- paste0('https://github.com/', info[['repo']]) > reuse gh_url from the top of the same file
    • module_exists is not implemented fully, see suggestion above (warning and/or GitHub issue)

Review test suite:

See guidance on testing for further details.

test coverage

covr::package_coverage(pkg_dir)
outsider Coverage: 94.10%
R/install.R: 86.30%
R/identities.R: 90.91%
R/search.R: 92.08%
R/docker.R: 92.42%
R/test.R: 93.02%
R/outsider.R: 93.75%
R/container.R: 94.12%
R/args.R: 98.04%
R/build.R: 100.00%
R/console.R: 100.00%
R/log.R: 100.00%

inspect tests

comments:

  • good job on using mocks, very educational for me to read a testsuite with mocks; hard for me though to give suggestions to improve, except: It might be worth the tradeoff of having some β€œintegration tests” without any mocks, because with all the mocks it’s hard for me to tell if you actually test a relevant amount of the control flow; you can always skip those tests if it is about speed, e.g.Β skip them unless and environment variable is set, and just let them run on Travis, where time is not that critical. Tests that I think are worth having as integration tests are the interaction with the GitHub API in test-search.R, the β€œmain” function testing in test-outsider.R, test-log.R (i.e.Β run actual workflows, capture the output, and see if the user is (not) shown what she should see). You could also run integration tests with incomplete modules and see if they fail at the expected point.
  • the tests for the skeleton could be extended to cover the minimal contents of the module, not just the directory. I think this might be worth the effort towards an indirect formal specification how a module looks like.
  • in test-docker.R
    • Tests that actually interact with Docker, e.g.Β using docker_pull, should be skipped on CRAN, see ?testthat::skip_on_cran, or you add your own test function skip_if_docker_is_not_available() (later I saw a comment β€œTODO: add skips if no docker” - go for it :-) ! )
    • the test β€œdocker_build() and docker_img_rm() works”
      • relies on the img variable from the previous test - that can quickly break. I suggest to either duplicate img, or move it outside of the test_that(..) function
      • you should also test that the second execution of docker_img_rm() returns FALSE
    • In general, consider checking the state outside of your own functions, e.g. expect_warning(expect_equal(attr(system2("docker", args = c("inspect", img), stdout = TRUE, stderr = TRUE), "status"), 1)), or with the package stevedore (also applies to test-container.R)
  • in test-console.R
    • I suggest to use expect_output(..), e.g. expect_output(outsider:::cat_line('cat this'), "cat this") (and also for celebrate() and comfort())
  • outsider:::vars_get() - the function is only for testing, right? Then I think it would be better to add it to the test codebase, e.g.Β in test-all.R
LS0tCm91dHB1dDoKICAgIGh0bWxfbm90ZWJvb2s6CiAgICAgICAgdG9jOiB0cnVlCiAgICAgICAgdG9jX2Zsb2F0OiB0cnVlCmVkaXRvcl9vcHRpb25zOgogIGNodW5rX291dHB1dF90eXBlOiBpbmxpbmUKLS0tCgo8IS0tIFJFQURNRS5tZCBpcyBnZW5lcmF0ZWQgZnJvbSBSRUFETUUuUm1kLiBQbGVhc2UgZWRpdCB0aGF0IGZpbGUgLS0+CgpgYGB7ciBzZXR1cCwgaW5jbHVkZSA9IEZBTFNFfQprbml0cjo6b3B0c19jaHVuayRzZXQoCiAgY29sbGFwc2UgPSBUUlVFLAogIGNvbW1lbnQgPSAiIz4iLAogIGZpZy5wYXRoID0gIm1hbi9maWd1cmVzL1JFQURNRS0iLAogIG91dC53aWR0aCA9ICIxMDAlIgopCgpsaWJyYXJ5KG1hZ3JpdHRyKQpgYGAKCiMgYG91dHNpZGVyYCAtIHBhY2thZ2UgcmV2aWV3CgojIyAqKlJldmlld2VyOioqIFtcQG51ZXN0XShodHRwczovL2dpdGh1Yi5jb20vbnVlc3QpCgpUaGlzIGlzIG15IGZpcnN0IHJldmlldyBmb3IgUk9wZW5TY2kgLSBleGNpdGluZyEKVGhhbmsgeW91IGZvciB0aGUgb3Bwb3J0dW5pdHkhCgojIyMgUmV2aWV3IFN1Ym1pdHRlZDoKKipgciBjYXQoc3ByaW50ZigiKipMYXN0IHVwZGF0ZWQ6KiogJXMiLCBTeXMuRGF0ZSgpKSlgKioKCioqKgoKPGJyPgoKVGhpcyByZXBvcnQgY29udGFpbnMgZG9jdW1lbnRzIGFzc29jaWF0ZWQgd2l0aCB0aGUgcmV2aWV3IG9mICoqck9wZW5TY2kqKiBzdWJtaXR0ZWQgcGFja2FnZToKCiMjIyAqKmBvdXRzaWRlcmA6IHJvcGVuc2NpL3NvZnR3YXJlLXJldmlldyoqICBpc3N1ZSBbXCMyODJdKGh0dHBzOi8vZ2l0aHViLmNvbS9yb3BlbnNjaS9vbmJvYXJkaW5nL2lzc3Vlcy8yODIpLgoKPGJyPgoKIyMgUGFja2FnZSBpbmZvCgoqKkRlc2NyaXB0aW9uOioqCgpJbnN0YWxsIGFuZCBydW4gZXh0ZXJuYWwgY29tbWFuZC1saW5lIHByb2dyYW1zIGluIFIgdGhyb3VnaCB1c2Ugb2YKICAgICdEb2NrZXInIDxodHRwczovL3d3dy5kb2NrZXIuY29tLz4gYW5kICdHaXRIdWInIDxodHRwOi8vZ2l0aHViLmNvbS8+LgoKKipBdXRob3I6KiogYHIgcGVyc29uKCJEb20iLCAiQmVubmV0dCIsIHJvbGUgPSBjKCJhdXQiLCAiY3JlIiksCiAgICBlbWFpbCA9ICJkb21pbmljLmpvaG4uYmVubmV0dEBnbWFpbC5jb20iLAogICAgY29tbWVudCA9IGMoT1JDSUQgPSAiMDAwMC0wMDAzLTI3MjItMTM1OSIpKWAKCioqcmVwbyB1cmw6KiogPGh0dHBzOi8vYW50b25lbGxpbGFiLmdpdGh1Yi5pby9vdXRzaWRlcj4KCioqd2Vic2l0ZSB1cmw6KiogPGh0dHBzOi8vYW50b25lbGxpbGFiLmdpdGh1Yi5pby9vdXRzaWRlci8+CgojIyBSZXZpZXcgaW5mbwoKCiMjIyMgU2VlIFtyZXZpZXdlciBndWlkZWxpbmVzXShodHRwczovL3JvcGVuc2NpLmdpdGh1Yi5pby9kZXZfZ3VpZGUvcmV2aWV3ZXJndWlkZS5odG1sKSBmb3IgZnVydGhlciBpbmZvcm1hdGlvbiBvbiB0aGUgck9wZW5TY2kgcmV2aWV3IHByb2Nlc3MuCgoqKmtleSByZXZpZXcgY2hlY2tzOioqCgotIERvZXMgdGhlIGNvZGUgY29tcGx5IHdpdGggKipnZW5lcmFsIHByaW5jaXBsZXMgaW4gdGhlIFtNb3ppbGxhIHJldmlld2luZyBndWlkZV0oaHR0cHM6Ly9tb3ppbGxhc2NpZW5jZS5naXRodWIuaW8vY29kZVJldmlldy9yZXZpZXcuaHRtbCkqKj8KLSBEb2VzIHRoZSBwYWNrYWdlICoqY29tcGx5IHdpdGggdGhlIFtST3BlblNjaSBwYWNrYWdpbmcgZ3VpZGVdKGh0dHBzOi8vcm9wZW5zY2kuZ2l0aHViLmlvL2Rldl9ndWlkZS9idWlsZGluZy5odG1sKSoqPwotIEFyZSB0aGVyZSAqKmltcHJvdmVtZW50cyoqIHRoYXQgY291bGQgYmUgbWFkZSB0byB0aGUgKipjb2RlIHN0eWxlPyoqCi0gSXMgdGhlcmUgKipjb2RlIGR1cGxpY2F0aW9uKiogaW4gdGhlIHBhY2thZ2UgdGhhdCBzaG91bGQgYmUgcmVkdWNlZD8KLSBBcmUgdGhlcmUgKip1c2VyIGludGVyZmFjZSBpbXByb3ZlbWVudHMqKiB0aGF0IGNvdWxkIGJlIG1hZGU/Ci0gQXJlIHRoZXJlICoqcGVyZm9ybWFuY2UgaW1wcm92ZW1lbnRzKiogdGhhdCBjb3VsZCBiZSBtYWRlPwotIElzIHRoZSBbKipkb2N1bWVudGF0aW9uKipdKGh0dHBzOi8vcm9wZW5zY2kuZ2l0aHViLmlvL2Rldl9ndWlkZS9idWlsZGluZy5odG1sI2RvY3VtZW50YXRpb24pIChpbnN0YWxsYXRpb24gaW5zdHJ1Y3Rpb25zL3ZpZ25ldHRlcy9leGFtcGxlcy9kZW1vcykgKipjbGVhciBhbmQgc3VmZmljaWVudCoqPwoKUGxlYXNlIGJlIHJlc3BlY3RmdWwgYW5kIGtpbmQgdG8gdGhlIGF1dGhvcnMgaW4geW91ciByZXZpZXdzLiBUaGUgck9wZW5TY2kgW2NvZGUgb2YgY29uZHVjdF0oaHR0cHM6Ly9yb3BlbnNjaS5naXRodWIuaW8vZGV2X2d1aWRlL3BvbGljaWVzLmh0bWwjY29kZS1vZi1jb25kdWN0KSBpcyBtYW5kYXRvcnkgZm9yIGV2ZXJ5b25lIGludm9sdmVkIGluIG91ciByZXZpZXcgcHJvY2Vzcy4KCioqKgoKIyMjIHNlc3Npb24gaW5mbwoKCmBgYHtyIHNlc3Npb25JbmZvfQpzZXNzaW9uSW5mbygpCmBgYAoKCmBgYHtyIHBrZ19kaXIsIGVjaG8gPSBGfQpwa2dfZGlyIDwtICIvaG9tZS9kYW5pZWwvRG9jdW1lbnRzLzIwMTlfUk9wZW5TY2ktcmV2aWV3L291dHNpZGVyIgpgYGAKCiMjIFRlc3QgaW5zdGFsbGF0aW9uCgojIyMgdGVzdCBsb2NhbCBgb3V0c2lkZXJgIGluc3RhbGw6CgpgYGB7ciB0ZXN0LWxvY2FsfQpkZXZ0b29sczo6aW5zdGFsbChwa2dfZGlyLCBkZXBlbmRlbmNpZXMgPSBULCBidWlsZF92aWduZXR0ZXMgPSBUKQpgYGAKCmBgYHtyIGdpdGh1Yi1ybX0KcmVtb3ZlLnBhY2thZ2VzKCJvdXRzaWRlciIpCmBgYAoKIyMjIyAqKmNvbW1lbnRzOioqCgo8IS0tIHJlY29yZCBjb21tZW50cyBvbiBsb2NhbCBpbnN0YWxsIGhlcmUgLS0+CgotIEluc3RhbGxhdGlvbiB3b3JrcyBzbW9vdGhseSBmcm9tIHNvdXJjZS4KLSBZb3UgbGlzdCAiZG9ja2VyIiBhcyBhIHN5c3RlbSBkZXBlbmRlbmN5LiBfSGF2ZSBjb25zaWRlcmVkIGNoZWNraW5nIGZvciBleGlzdGVuY2Ugb2YgRG9ja2VyIGF0IGluc3RhbGxhdGlvbiB0aW1lP18gUmVhbGx5IGN1cmlvdXMgaGVyZSBiZWNhdXNlIEkgYWxzbyBoYXZlIGEgRG9ja2VyLXJlbGF0ZWQgcGFja2FnZS4KCioqKgoKIyMjIHRlc3QgaW5zdGFsbCBvZiBgb3V0c2lkZXJgIGZyb20gR2l0SHViIHdpdGg6CgpgYGB7ciB0ZXN0LWdpdGh1Yn0KZGV2dG9vbHM6Omluc3RhbGxfZ2l0aHViKCJBbnRvbmVsbGlMYWIvb3V0c2lkZXIiLCBkZXBlbmRlbmNpZXMgPSBULCBidWlsZF92aWduZXR0ZXMgPSBUKQpgYGAKCiMjIyMgKipjb21tZW50czoqKgoKPCEtLSByZWNvcmQgY29tbWVudHMgb24gZ2l0aHViIGluc3RhbGwgaGVyZSAtLT4KCi0gbm8gcHJvYmxlbXMgdG8gaW5zdGFsbCBmcm9tIEdpdEh1YgotIHRoZSB2ZXJzaW9uIG51bWJlciBgMC4wLjBgIHNlZW1zIGN1cmlvdXMgdG8gbWUgdGhvdWdoLCBJIHN1Z2dlc3QgdG8gZWl0aGVyIGdvIHdpdGggdGhlIGAuOTAwMGAgdmVyc2lvbmluZyBbYXMgZGVzY3JpYmVkIGhlcmVdKGh0dHA6Ly9yLXBrZ3MuaGFkLmNvLm56L2Rlc2NyaXB0aW9uLmh0bWwjdmVyc2lvbikgaWYgeW91IHRoaW5rIHRoaXMgaXMgYSBkZXZlbG9wbWVudCBzdGF0ZSwgaWYgeW91IHRoaW5rIHlvdSBoYXZlIGEgZmlyc3Qgd29ya2luZyB2ZXJzaW9uIChhcyB0aGUgImFjdGl2ZSIgcmVwbyBzdGF0dXMgc3VnZ2VzdHMpLCB3aHkgbm90IGdvIHdpdGggYDAuMS4wYD8KCioqKgoKCiMjIENoZWNrIHBhY2thZ2UgaW50ZWdyaXR5CgojIyMgcnVuIGNoZWNrcyBvbiBgb3V0c2lkZXJgIHNvdXJjZToKCmBgYHtyIGNoZWNrLWNoZWNrc30KZGV2dG9vbHM6OmNoZWNrKHBrZ19kaXIpCmBgYAojIyMjICoqY29tbWVudHM6KioKCjwhLS0gcmVjb3JkIGNvbW1lbnRzIG9uIGNoZWNrcyBoZXJlIC0tPgoKLSBkdXJpbmcgY2hlY2ssIG9uZSBleGFtcGxlIGFuZCBvbmUgdGVzdCBmYWlscyBvbiBteSBzeXN0ZW06CiAgLSBydW5uaW5nIHRoZSBjb2RlIGZyb20gYG1vZHVsZV9zZWFyY2guUmAgbWFudWFsbHkgd29ya3MsIGJ1dCBmYWlscyBkdXJpbmcgY2hlY2s6CiAgYGBgCiAgRSAgY2hlY2tpbmcgZXhhbXBsZXMgKDEuNnMpCiAgIFJ1bm5pbmcgZXhhbXBsZXMgaW4g4oCYb3V0c2lkZXItRXguUuKAmSBmYWlsZWQKICAgVGhlIGVycm9yIG1vc3QgbGlrZWx5IG9jY3VycmVkIGluOgogICAKICAgPiBiYXNlOjphc3NpZ24oIi5wdGltZSIsIHByb2MudGltZSgpLCBwb3MgPSAiQ2hlY2tFeEVudiIpCiAgID4gIyMjIE5hbWU6IG1vZHVsZV9kZXRhaWxzCiAgID4gIyMjIFRpdGxlOiBMb29rIHVwIGRldGFpbHMgb24gbW9kdWxlKHMpCiAgID4gIyMjIEFsaWFzZXM6IG1vZHVsZV9kZXRhaWxzCiAgID4gCiAgID4gIyMjICoqIEV4YW1wbGVzCiAgID4gCiAgID4gbGlicmFyeShvdXRzaWRlcikKICAgPiAjIHJldHVybiB0YWJsZSBvZiBBTEwgYXZhaWxhYmxlIG1vZHVsZXMgb24gR2l0SHViCiAgID4gIyBOT1QgUlVOIC0gdGFrZXMgdG9vIGxvbmcKICAgPiAjIChhdmFpbGFibGVfbW9kdWxlcyA8LSBtb2R1bGVfc2VhcmNoKCkpCiAgID4gCiAgID4gIyBsb29rLXVwIHdoZXRoZXIgc3BlY2lmaWMgbW9kdWxlIGV4aXN0cwogICA+ICMgTk9UIFJVTgogICA+ICMgcmVwbyA8LSAnZG9tYmVubmV0dC9vbS4uZ29vZGJ5ZS53b3JsZCcKICAgPiAjIChzdXBwcmVzc1dhcm5pbmdzKG1vZHVsZV9leGlzdHMocmVwbyA9IHJlcG8pKSkKICAgPiByZXBvIDwtICdkb21iZW5uZXR0L29tLi5oZWxsby53b3JsZCcKICAgPiAobW9kdWxlX2V4aXN0cyhyZXBvID0gcmVwbykpCiAgIEVycm9yIGluIG9wZW4uY29ubmVjdGlvbihjb24sICJyYiIpIDogSFRUUCBlcnJvciA0MDEuCiAgIENhbGxzOiBtb2R1bGVfZXhpc3RzIC4uLiBwYXJzZV9hbmRfc2ltcGxpZnkgLT4gcGFyc2VKU09OIC0+IHBhcnNlX2NvbiAtPiBvcGVuIC0+IG9wZW4uY29ubmVjdGlvbgogICBFeGVjdXRpb24gaGFsdGVkCiAgYGBgCiAgCiAgLSBfIG1vZHVsZV9kZXRhaWxzKCkgd29ya3MgKEB0ZXN0LXNlYXJjaC5SIzk0KV8KICAKICBgYGAKICDilIDilIAgMS4gRXJyb3I6IG1vZHVsZV9kZXRhaWxzKCkgd29ya3MgKEB0ZXN0LXNlYXJjaC5SIzk0KSAg4pSA4pSA4pSA4pSA4pSA4pSA4pSA4pSA4pSA4pSA4pSACiAgSFRUUCBlcnJvciA0MDEuCiAgMTogbW9kdWxlX2RldGFpbHMocmVwbyA9IGMocmVwbywgImRvbWJlbm5ldHQvb20uLm1hZmZ0IikpIGF0IHRlc3R0aGF0L3Rlc3Qtc2VhcmNoLlI6OTQKICAyOiBsYXBwbHkoWCA9IHJlcG8sIEZVTiA9IHJlcG9fc2VhcmNoKQogIDM6IEZVTihYW1tpXV0sIC4uLikKICA0OiBqc29ubGl0ZTo6ZnJvbUpTT04ocGFzdGUwKGdoX3NlYXJjaF9yZXBvX3VybCwgc2VhcmNoX2FyZ3MpKQogIDU6IHBhcnNlX2FuZF9zaW1wbGlmeSh0eHQgPSB0eHQsIHNpbXBsaWZ5VmVjdG9yID0gc2ltcGxpZnlWZWN0b3IsIHNpbXBsaWZ5RGF0YUZyYW1lID0gc2ltcGxpZnlEYXRhRnJhbWUsIAogICAgICAgICBzaW1wbGlmeU1hdHJpeCA9IHNpbXBsaWZ5TWF0cml4LCBmbGF0dGVuID0gZmxhdHRlbiwgLi4uKQogIDY6IHBhcnNlSlNPTih0eHQsIGJpZ2ludF9hc19jaGFyKQogIDc6IHBhcnNlX2Nvbih0eHQsIGJpZ2ludF9hc19jaGFyKQogIDg6IG9wZW4oY29uLCAicmIiKQogIDk6IG9wZW4uY29ubmVjdGlvbihjb24sICJyYiIpCiAgYGBgCgojIyMgcnVuIHRlc3RzIG9uIGBvdXRzaWRlcmAgc291cmNlOgoKYGBge3IgY2hlY2stdGVzdHN9CmRldnRvb2xzOjp0ZXN0KHBrZ19kaXIpCmBgYAojIyMjICoqY29tbWVudHM6KioKCjwhLS0gcmVjb3JkIGNvbW1lbnRzIG9uIHRlc3RzIGhlcmUgLS0+CgotIG5vIGZhaWxpbmcgdGVzdCBoZXJlIDp0YWRhOiAhCgojIyMgY2hlY2sgYG91dHNpZGVyYCBmb3IgZ29vZHByYWN0aWNlOgoKYGBge3IgdGVzdC1nb29kcHJhY3RpY2V9Cmdvb2RwcmFjdGljZTo6Z3AocGtnX2RpcikKYGBgCgojIyMjICoqY29tbWVudHM6KioKCjwhLS0gcmVjb3JkIGNvbW1lbnRzIG9uIGdvb2RwcmFjdGljZSBoZXJlIC0tPgoKLSBzb21lIHByb2JsZW1zIGhlcmUgYXMgYmVmb3JlCi0gSU1ITyB0aGUgY292ZXJhZ2UgaXMgcmVhc29uYWJsZQoKIyMgQ2hlY2sgcGFja2FnZSBtZXRhZGF0YSBmaWxlcwoKIyMjIGluc3BlY3QKCi0gIyMjIyBbUkVBRE1FXShodHRwczovL2FudG9uZWxsaWxhYi5naXRodWIuaW8vb3V0c2lkZXIpCi0gIyMjIyBbREVTQ1JJUFRJT05dKGh0dHBzOi8vYW50b25lbGxpbGFiLmdpdGh1Yi5pby9vdXRzaWRlci9ibG9iL21hc3Rlci9ERVNDUklQVElPTikKLSAjIyMjIFtOQU1FU1BBQ0VdKGh0dHBzOi8vYW50b25lbGxpbGFiLmdpdGh1Yi5pby9vdXRzaWRlci9ibG9iL21hc3Rlci9OQU1FU1BBQ0UpCgojIyMgc3BlbGwgY2hlY2sKCmBgYHtyIHNwZWxsLWNoZWNrfQpkZXZ0b29sczo6c3BlbGxfY2hlY2socGtnX2RpcikKYGBgCgojIyMjICoqY29tbWVudHM6KioKCjwhLS0gcmVjb3JkIGNvbW1lbnRzIG9uIG1ldGFkYXRhIGZpbGVzIGhlcmUgLS0+CgotIGluIGBSRUFETUUubWRgOiBJIHJlY29tbWVuZCB0byBoYXZlIGFuIGVtcHR5IGxpbmUgYWZ0ZXIgaGVhZGxpbmVzIGFuZCBhIHNwYWNlIGFmdGVyIHRoZSBgI2AvYCMjYCBpbiB0aGUgc291cmNlIGNvZGUgZm9yIHJlYWRhYmlsaXR5Ci0gc3BlbGxjaGVjazogYWxsIGZpbmUKLSBBZGQgYHBoeWxvdGFSYCBhbmQgYGFwZWAgdG8gYFN1Z2dlc3RzYCBiZWNhdXNlIHlvdSB1c2UgdGhlbSBpbiB2aWduZXR0ZXMKCiMjIENoZWNrIGRvY3VtZW50YXRpb24KCm9ubGluZSBkb2N1bWVudGF0aW9uOiAqKjxodHRwczovL2FudG9uZWxsaWxhYi5naXRodWIuaW8vb3V0c2lkZXIvPioqCgoqIElzIHRoZSBbZG9jdW1lbnRhdGlvbl0oaHR0cHM6Ly9yb3BlbnNjaS5naXRodWIuaW8vZGV2X2d1aWRlL2J1aWxkaW5nLmh0bWwjZG9jdW1lbnRhdGlvbikgKGluc3RhbGxhdGlvbiBpbnN0cnVjdGlvbnMvdmlnbmV0dGVzL2V4YW1wbGVzL2RlbW9zKSBjbGVhciBhbmQgc3VmZmljaWVudD8KCiMjIyB0ZXN0IGBvdXRzaWRlcmAgZnVuY3Rpb24gaGVscCBmaWxlczoKCmBgYHtyIHRlc3QtaGVscH0KaGVscChwYWNrYWdlID0gIm91dHNpZGVyIikKYGBgCgojIyMjICoqY29tbWVudHM6KioKCjwhLS0gcmVjb3JkIGNvbW1lbnRzIG9uIGhlbHAgZmlsZXMgaGVyZSAtLT4KCi0gLi4uCgojIyMgdGVzdCBgb3V0c2lkZXJgIHZpZ25ldHRlczoKCmBgYHtyIHRlc3QtdmlnbmV0dGVzfQp2aWduZXR0ZShwYWNrYWdlID0gIm91dHNpZGVyIikKYGBgCgojIyMjICoqY29tbWVudHM6KioKCjwhLS0gcmVjb3JkIGNvbW1lbnRzIG9uIHZpZ25ldHRlcyBoZXJlIC0tPgoKLSBUaGUgY29udGVudHMgbGlua3Mgb24gaHR0cHM6Ly9hbnRvbmVsbGlsYWIuZ2l0aHViLmlvL291dHNpZGVyL2FydGljbGVzL2FkdmFuY2VkX2J1aWxkLmh0bWwgZG8gbm90IHdvcmsgeWV0L2NvbnRlbnQgaXMgbWlzc2luZwotIGBvdXRzaWRlcmAgY292ZXJzIGEgcHJldHR5IGNvbXBsZXggcHJvYmxlbSEgV2l0aCB0aGUgZXhpc3RpbmcgZXhhbXBsZXMgb2Ygb25lIHZlcnkgYmFzaWMgImhlbGxvIHdvcmxkIiBhbmQgb25lICh0byBtZSB2ZXJ5KSBjb21wbGV4IGFuYWx5c2lzIHBpcGVsaW5lLCBJIHNlZSByb29tIGZvciBhIGZ1cnRoZXIgZXhhbXBsZSB0aGF0IHNob3dzIGEgYml0IG1vcmUgYnV0IGlzIG5vdCB0aGF0IGRvbWFpbiBzcGVjaWZpYywgZS5nLiBsb2FkaW5nIHNvbWUgZGF0YSBmcm9tIGEgQ1NWIGFuZCBydW5uaW5nIGEgcmVncmVzc2lvbiBtb2RlbCwgc2F2aW5nIHRoYXQgdG8gYSBmaWxlLiBGb3IgbWUgYXMgYSB1c2VyIGl0IHdvdWxkIGJlIGdyZWF0IHRvIGtub3cgX2hvdyBjYW4gSSBsb2FkIG15IGRhdGEsIGhvdyBjYWxsIG15IGZ1bmN0aW9uLCBhbmQgaG93IGFjY2VzcyB0aGUgb3V0cHV0IGRhdGFfLgotICoqR2V0IFN0YXJ0ZWQqKgogIC0gVGhlIGNvZGUgY2h1bmsgdXNpbmcgYGlucHV0X3NlcXVlbmNlcy5mYXN0YWAgZG9lcyBub3Qgd29yayAtIHdvdWxkIGl0IGJlIHBvc3NpYmxlIHRvIGxldCBhIHVzZXIgZG93bmxvYWQgKHdpdGggUiBjb2RlKSBzdWNoIGEgZmlsZSBhbmQgYWN0dWFsbHkgcnVuIHRoYXQ/IEkgd2FzIGEgYml0IGRpc2FwcG9pbnRlZCB3aGVuIEkgd2VudCB0aG91Z2ggYWxsIHByZXZpb3VzIHN0ZXBzIGFuZCB0aGVuIHdhcyAic3RvcHBlZCIgdW5leHBlY3RlZGx5LgogIC0gdXNlcnMgb2Z0ZW4gd2FudCB0byBrbm93IHdoYXQncyBnb2luZyBvbiwgYW5kIG91dHNpZGVyIGlzIGRvaW5nIHF1aXRlIGEgbG90IG9mIG1hZ2ljIC0gSSBzdWdnZXN0IHRvIG1lbnRpb24gYGxvZ19zZXRgIGFscmVhZHkgaW4gdGhlIGdldCBzdGFydGVkIGRvY3VtZW50YXRpb24uCiAgLSAiVW5pbnN0YWxsaW5nIiAtIGZvciB1c2VycyBuZXcgdG8gRG9ja2VyLCB5b3UgY291bGQgbWVudGlvbiBoZXJlIGhvdyB0byByZW1vdmVkIHRoZSBkb3dubG9hZGVkIGltYWdlcy4KLSAqKnBoeW9sb2dlbmV0aWMgdmlnbmV0dGUqKgogIC0gc3VnZ2V0cyB0byBhZGQgYSBsaW5rIHRvIHRoZSBvdXRzaWRlciBtb2R1bGUgb24gR2l0SHViIChodHRwczovL2dpdGh1Yi5jb20vRG9tQmVubmV0dC9vbS4ubWFmZnQpLCBwb3RlbnRpYWxseSBldmVuIHRvIHNwZWNpZmljIGZpbGVzLCB0byBoZWxwIHVzZXJzIGV4cGxvcmUgdGhpcyBtb3JlIGNvbXBsZXggbW9kdWxlIHdpdGggdGhlIGhlbHAgb2YgdGhlIHZpZ25ldHRlCiAgLSBpcyB0aGVyZSBhcmUgcGFydGljdWxhciByZWFzb24gZm9yIHRoZSByZXBlYXRlZCAibGlicmFyeShvdXRzaWRlcikiIHN0YXRlbWVudHMgaW4gZWFjaCBzZWN0aW9uPwogIC0gQ2FuIHlvdSBtYWtlIHRoZSBvdXRwdXQgb2YgYG1vZHVsZV9oZWxwKHJlcG8gPSAnZG9tYmVubmV0dC9vbS4ubWFmZnQnLCBmbmFtZSA9ICdtYWZmdCcpYCBtb3JlIGhlbHBmdWw/IFJpZ2h0IG5vdyBpdCBvbmx5IHNheXMgIlJ1biBtYWZmdCIuIEkgdGhpbmsgZm9yIHRoZSAic3RvcnkiIG9mIHRoZSB2aWduZXR0ZSBpdCB3b3VsZCBiZSBtdWNoIGJldHRlciB0byBhY3R1YWxseSBoYXZlIHNvbWUgZG9jdW1lbnRhdGlvbiB0aGVyZSwgZS5nLiByZWZlcmVuY2UgcGFwZXJzIGFuZCBkZXNjcmliZSB0aGUgZnVuY3Rpb24uCiAgLSBJbiBzZWN0aW9uICJBbGlnbm1lbnQiIC0gd2hhdCBzaG91bGQgSSBsb29rIGZvciBhcyBhIHVzZXIgYXQgdGhlIGVuZD8gQ2FuIHlvdSBhZGQgc29tZSB2aXN1YWxpc2F0aW9uIGhlcmUgb3IgZGVzY3JpYmUgd2hhdCBhIHVzZXIgY291bGQgZG8gbm93IHdpdGggdGhlIG91dHB1dCBvZiB0aGUgcHJvY2Vzcz8KCiMjIFRlc3QgZnVuY3Rpb25hbGl0eToKCi0gQXJlIHRoZXJlICoqdXNlciBpbnRlcmZhY2UgaW1wcm92ZW1lbnRzKiogdGhhdCBjb3VsZCBiZSBtYWRlPwotIEFyZSB0aGVyZSAqKnBlcmZvcm1hbmNlIGltcHJvdmVtZW50cyoqIHRoYXQgY291bGQgYmUgbWFkZT8KCmBgYHtyIGZyZWUtc3R5bGV9CmxpYnJhcnkoIm91dHNpZGVyIikKYGBgCgpgYGB7ciBwYXJzZS1mdW5jdGlvbnN9CmV4cG9ydHMgPC0gbHMoInBhY2thZ2U6b3V0c2lkZXIiKQpleHBvcnRzCmBgYAoKPCEtLSBleHBlcmltZW50IHdpdGggcGFja2FnZSBmdW5jdGlvbnMgLS0+CgpgYGB7ciBleHAtY2h1bmt9CiMgcGFydGlhbCBleGFtcGxlcyBvbmx5IQoKb3V0c2lkZXI6Ojp2YXJzX2dldAoKb3V0c2lkZXI6OjoucnVuLm91dHNpZGVyCgpvdXRzaWRlcjo6OnJlcG9fc2VhcmNoKCJudWVzdC9vdXRzaWRlciIpClN5cy5nZXRlbnYoIkdJVEhVQl9QQVQiKQpgYGAKCiMjIyMgKipjb21tZW50czoqKgoKPCEtLSByZWNvcmQgY29tbWVudHMgb24gcGFja2FnZSBleHBlcmltZW50YXRpb24gaGVyZSAtLT4KCi0gX0lzIHRoZXJlIGEgd2F5IHRvIGxpc3QgYWxsIGZ1bmN0aW9uIG5hbWVzIGluIGEgbW9kdWxlP18gVGhlIEdldCBTdGFydGVkIHZpZ25ldHRlIHNheXMgIk9uY2UgYSBmdW5jdGlvbiBuYW1lIGlzIGtub3duIG9mIGEgcGFydGljdWxhciAuLiIgLSBjYW4gSSBvbmx5IGRvIHRoYXQgdmlhIHRoZSBoZWxwIGZpbGU/IE1BeWJlIGEgYG1vZHVsZV9saXN0KHJlcG8gPSAnZG9tYmVubmV0dC9vbS4ubWFmZnQnKWAgaXMgZmVhc2libGUgdG8gaW1wbGVtZW50LgotIF9EbyB5b3UgaGF2ZSBhbnkgZnVydGhlciBkZXZlbG9wbWVudCBwbGFucz9fIEkgY2FuIGltYWdpbmUgZnVydGhlciBkZXZlbG9wbWVudHMgKGxpa2UgYnVpbGRpbmcgcGlwZWxpbmVzIHVzaW5nIGEgY2hhaW4gb2YgbW9kdWxlcyksIG1heWJlIHlvdSBjYW4gcHV0IHlvdXIgaWRlYXMgaW50byBHaXRIdWIgaXNzdWVzIGZvciBldmVyeW9uZSB0byBsZWFybiBhYm91dCB0aGVtPwotIHNlZSBjb21tZW50IGFib3V0IGRldi1mdW5jdGlvbnMgc3RhcnRpbmcgd2l0aCBgLmAKCiMjIEluc3BlY3QgY29kZToKCi0gRG9lcyB0aGUgcGFja2FnZSAqKmNvbXBseSB3aXRoIHRoZSBbUk9wZW5TY2kgcGFja2FnaW5nIGd1aWRlXShodHRwczovL3JvcGVuc2NpLmdpdGh1Yi5pby9kZXZfZ3VpZGUvYnVpbGRpbmcuaHRtbCkqKj8KICAgICogZ29vZCBbZnVuY3Rpb24gJiB2YXJpYWJsZSBuYW1pbmc/XShodHRwczovL3JvcGVuc2NpLmdpdGh1Yi5pby9kZXZfZ3VpZGUvYnVpbGRpbmcuaHRtbCNmdW5jdGlvbi1hbmQtYXJndW1lbnQtbmFtaW5nKQogICAgKiBnb29kIFtkZXBlbmRlbmN5IG1hbmFnZW1lbnRdKGh0dHBzOi8vcm9wZW5zY2kuZ2l0aHViLmlvL2Rldl9ndWlkZS9idWlsZGluZy5odG1sI3BhY2thZ2UtZGVwZW5kZW5jaWVzKT8KLSBBcmUgdGhlcmUgKippbXByb3ZlbWVudHMqKiB0aGF0IGNvdWxkIGJlIG1hZGUgdG8gdGhlIFsqKmNvZGUgc3R5bGU/KipdKGh0dHBzOi8vcm9wZW5zY2kuZ2l0aHViLmlvL2Rldl9ndWlkZS9idWlsZGluZy5odG1sI2NvZGUtc3R5bGUpCi0gSXMgdGhlcmUgKipjb2RlIGR1cGxpY2F0aW9uKiogaW4gdGhlIHBhY2thZ2UgdGhhdCBzaG91bGQgYmUgcmVkdWNlZD8KCmBgYHtyIGluc3BlY3QtY29kZX0KcGtncmV2aWV3cjo6cGtncmV2aWV3X3ByaW50X3NvdXJjZSgib3V0c2lkZXIiKQpgYGAKKipcKiBtaWdodCBub3QgYmUgc3VpdGFibGUgZm9yIGxhcmdlIHBhY2thZ2VzIHdpdGggbWFueSBleHBvcnRlZCBmdW5jdGlvbnMqKgoKPGJyPgo8YnI+CgojIyMjICoqY29tbWVudHM6KioKCjwhLS0gcmVjb3JkIGNvbW1lbnRzIG9uIHBhY2thZ2Ugc291cmNlIGNvZGUgaGVyZSAtLT4KCi0gSWRlYTogYC5tb2R1bGVfdmFsaWRhdGUoLi4pYCB0byB2YWxpZGF0ZSB0aGUgZm9ybWFsIHN0cnVjdHVyZSBvZiBhIG1vZHVsZSAtIEkgbm93IHNlZSB0aGVyZSBpcyBgLm1vZHVsZV9jaGVja2AgOi0pIC0gSSBzdWdnZXN0IHRvIHB1dCB0aGlzICJUT0RPIiBpbnRvIGEgR2l0SHViIGlzc3VlIHNvIHRoYXQgcG90ZW50aWFsIGNvbGxhYm9yYXRvcnMgc2VlIHRoaXMgaXMgYW4gb3BlbiB0YXNrLCBhbmQgbm90IGhhdmUgYSBmdW5jdGlvbiB0aGF0IGEgdXNlciBjYW5uIGFjY2VzcyB3aXRoIGA6OjpgIHdpdGhvdXQgYW55IGZ1bmN0aW9uYWxpdHkuIE9yIGF0IGxlYXN0IGFkZCBhIGB3YXJuaW5nKCJub3QgaW1wbGVtZW50ZWQiKWAKLSBJIHRoaW5rIHRoZSBwYWNrYWdlIGRvZXMgYSBnb29kIGpvYiB0byBkaXN0aW5ndWlzaCBiZXR3ZWVuIHVzZXIgZmFjaW5nIGZ1bmN0aW9ucyBhbmQgaW50ZXJuYWwgZnVuY3Rpb25zIChsaWtlIGB0ZXN0KClgKQotIGZvcm1hdHRpbmcgaXMgYWxyaWdodAotIGZvciBgcGtnZGV0YWlsc19nZXRgIEkgcGVyc29uYWxseSB3b3VsZCByZWx5IG9uIHRoZSBwYWNrYWdlIFtgZGVzY2BdKGh0dHBzOi8vY3Jhbi5yLXByb2plY3Qub3JnL3BhY2thZ2U9ZGVzYykgdG8gZG8gdGhlIHBhcnNpbmcgLSB5b3UgYWxyZWFkeSBoYXZlIGEgbnVtYmVyIG9mIGltcG9ydHMgYW5kIHRoaXMgbGlnaHR3ZWlnaHQgb25lIHNob3VsZCBub3QgaHVydAotIGAubW9kdWxlX3RyYXZpc2AgLSB5b3VyIGFwcHJvYWNoIG1lYW5zIHRoYXQgdGhlIGZ1bmN0aW9uIHdpbGwgZmFpbCBpZiB0aGUgcmVwbyBtb292ZXMsIG9yIGlmIHRoZXJlIGlzIG5vIGludGVybmV0IGNvbm5lY3Rpb24uLi4gZm9yIG1lIGl0J3MgYWx3YXlzIGEgc2lnbmFsIHRvIHJlY29uc2lkZXIgdGhlIGFwcHJvYWNoIGlmIHlvdSBoYXZlIGEgY29uc3RhbnQgbGlrZSBhIFVSTCB3aXRoaW4gYSBmdW5jdGlvbiBib2R5LiBJIHRoaW5rIGl0IHdvdWxkIGJlIGJldHRlciB0byBtYW5hZ2UgdGhlICJ0ZW1wbGF0ZSIgZmlsZSB3aXRoaW4gdGhlIGNvZGViYXNlIG9mIHRoZSBwYWNrYWdlLgotIGBjb250YWluZXIuUmA6IHlvdXIgcGFja2FnZSBjb2RlIHByb2JhYmxlIHByZWNlZGVzIGBzdGV2ZWRvcmVgLCBidXQgaWYgeW91IGZpbmQgeW91cnNlbGYgd3JpdGluZyBjb2RlIGFyb3VuZCBjYWxsaW5nIGRvY2tlciB3aXRoIGBzeXM6OmV4ZWMuLmAgdGhlbiBpdCBtaWdodCBiZSB3b3J0aCB0aGUgZWZmb3J0IHRvIHN3aXRjaDsgdGhpbmdzIGxpa2UgW3J1bm5pbmcgY29udGFpbmVycyByZW1vdGVseV0oaHR0cHM6Ly9naXRodWIuY29tL0FudG9uZWxsaUxhYi9vdXRzaWRlci9pc3N1ZXMvMSkgKGlmIHRoYXQgaXMgd2hhdCB5b3UgbWVhbiBieSAicmVtb3RlIGhvc3RpbmciKSB3b3VsZCBjb21lIHZlcnkgY2hlYXBseSB0aGVuOwotIGBzZWFyY2guUmA6CiAgLSAiY29kZSB5b3Ugd3JvdGUgNiBtb250aHMgYWdvIG1pZ2h0IGp1c3QgaGF2ZSBiZWVuIHdyaXR0ZW4gYnkgc29tZW9uZSBlbHNlIjogSSBzdWdnZXN0IHRvIGFkZCBhIHdhcm5pbmcgYWZ0ZXIgYHRrbiA8LSBOVUxMYCB0byBtYWtlIGl0IG1vcmUgb2J2aW91cyB3aGVuIHJlcXVlc3RzIGZhaWwgYmVjYXVzZSBvZiBhIG1pc3NpbmcgdG9rZW4gKGFuZCBub3QgYmVjYXVzZSB0aGUgdG9rZW4gaXMgbm90IHZhbGlkIGV0Yy4pCiAgLSBgaW5mb1tbJ3VybCddXSA8LSBwYXN0ZTAoJ2h0dHBzOi8vZ2l0aHViLmNvbS8nLCBpbmZvW1sncmVwbyddXSlgID4gcmV1c2UgYGdoX3VybGAgZnJvbSB0aGUgdG9wIG9mIHRoZSBzYW1lIGZpbGUKICAtIGBtb2R1bGVfZXhpc3RzYCBpcyBub3QgaW1wbGVtZW50ZWQgZnVsbHksIHNlZSBzdWdnZXN0aW9uIGFib3ZlICh3YXJuaW5nIGFuZC9vciBHaXRIdWIgaXNzdWUpCgojIyBSZXZpZXcgdGVzdCBzdWl0ZToKClNlZSBndWlkYW5jZSBvbiBbdGVzdGluZ10oaHR0cHM6Ly9yb3BlbnNjaS5naXRodWIuaW8vZGV2X2d1aWRlL2J1aWxkaW5nLmh0bWwjdGVzdGluZykgZm9yIGZ1cnRoZXIgZGV0YWlscy4KCiMjIyB0ZXN0IGNvdmVyYWdlCgpgYGB7ciBwa2dfY292ZXJhZ2V9CmNvdnI6OnBhY2thZ2VfY292ZXJhZ2UocGtnX2RpcikKCmBgYAoKIyMjIGluc3BlY3QgW3Rlc3RzXShodHRwczovL2FudG9uZWxsaWxhYi5naXRodWIuaW8vb3V0c2lkZXIvYmxvYi9tYXN0ZXIvdGVzdHMvdGVzdHRoYXQpCgoKIyMjIyAqKmNvbW1lbnRzOioqCgo8IS0tIHJlY29yZCBjb21tZW50cyBvbiB0ZXN0aW5nIHN1aXRlIGhlcmUgLS0+CgotIGdvb2Qgam9iIG9uIHVzaW5nIG1vY2tzLCB2ZXJ5IGVkdWNhdGlvbmFsIGZvciBtZSB0byByZWFkIGEgdGVzdHN1aXRlIHdpdGggbW9ja3M7IGhhcmQgZm9yIG1lIHRob3VnaCB0byBnaXZlIHN1Z2dlc3Rpb25zIHRvIGltcHJvdmUsIGV4Y2VwdDogSXQgbWlnaHQgYmUgd29ydGggdGhlIHRyYWRlb2ZmIG9mIGhhdmluZyBzb21lICJpbnRlZ3JhdGlvbiB0ZXN0cyIgd2l0aG91dCBhbnkgbW9ja3MsIGJlY2F1c2Ugd2l0aCBhbGwgdGhlIG1vY2tzIGl0J3MgaGFyZCBmb3IgbWUgdG8gdGVsbCBpZiB5b3UgYWN0dWFsbHkgdGVzdCBhIHJlbGV2YW50IGFtb3VudCBvZiB0aGUgY29udHJvbCBmbG93OyB5b3UgY2FuIGFsd2F5cyBza2lwIHRob3NlIHRlc3RzIGlmIGl0IGlzIGFib3V0IHNwZWVkLCBlLmcuIHNraXAgdGhlbSB1bmxlc3MgYW5kIGVudmlyb25tZW50IHZhcmlhYmxlIGlzIHNldCwgYW5kIGp1c3QgbGV0IHRoZW0gcnVuIG9uIFRyYXZpcywgd2hlcmUgdGltZSBpcyBub3QgdGhhdCBjcml0aWNhbC4gVGVzdHMgdGhhdCBJIHRoaW5rIGFyZSB3b3J0aCBoYXZpbmcgYXMgKippbnRlZ3JhdGlvbiB0ZXN0cyoqIGFyZSB0aGUgaW50ZXJhY3Rpb24gd2l0aCB0aGUgR2l0SHViIEFQSSBpbiBgdGVzdC1zZWFyY2guUmAsIHRoZSAibWFpbiIgZnVuY3Rpb24gdGVzdGluZyBpbiBgdGVzdC1vdXRzaWRlci5SYCwgYHRlc3QtbG9nLlJgIChpLmUuIHJ1biBhY3R1YWwgd29ya2Zsb3dzLCBjYXB0dXJlIHRoZSBvdXRwdXQsIGFuZCBzZWUgaWYgdGhlIHVzZXIgaXMgKG5vdCkgc2hvd24gd2hhdCBzaGUgc2hvdWxkIHNlZSkuIFlvdSBjb3VsZCBhbHNvIHJ1biBpbnRlZ3JhdGlvbiB0ZXN0cyB3aXRoIGluY29tcGxldGUgbW9kdWxlcyBhbmQgc2VlIGlmIHRoZXkgZmFpbCBhdCB0aGUgZXhwZWN0ZWQgcG9pbnQuCi0gdGhlIHRlc3RzIGZvciB0aGUgc2tlbGV0b24gY291bGQgYmUgZXh0ZW5kZWQgdG8gY292ZXIgdGhlIG1pbmltYWwgY29udGVudHMgb2YgdGhlIG1vZHVsZSwgbm90IGp1c3QgdGhlIGRpcmVjdG9yeS4gSSB0aGluayB0aGlzIG1pZ2h0IGJlIHdvcnRoIHRoZSBlZmZvcnQgdG93YXJkcyBhbiBpbmRpcmVjdCBmb3JtYWwgc3BlY2lmaWNhdGlvbiBob3cgYSBtb2R1bGUgbG9va3MgbGlrZS4KLSBpbiBgdGVzdC1kb2NrZXIuUmAKICAtIFRlc3RzIHRoYXQgYWN0dWFsbHkgaW50ZXJhY3Qgd2l0aCBEb2NrZXIsIGUuZy4gdXNpbmcgYGRvY2tlcl9wdWxsYCwgc2hvdWxkIGJlIHNraXBwZWQgb24gQ1JBTiwgc2VlIGA/dGVzdHRoYXQ6OnNraXBfb25fY3JhbmAsIG9yIHlvdSBhZGQgeW91ciBvd24gdGVzdCBmdW5jdGlvbiBgc2tpcF9pZl9kb2NrZXJfaXNfbm90X2F2YWlsYWJsZSgpYCAobGF0ZXIgSSBzYXcgYSBjb21tZW50ICJUT0RPOiBhZGQgc2tpcHMgaWYgbm8gZG9ja2VyIiAtIGdvIGZvciBpdCA6LSkgISApCiAgLSB0aGUgdGVzdCAiZG9ja2VyX2J1aWxkKCkgYW5kIGRvY2tlcl9pbWdfcm0oKSB3b3JrcyIKICAgIC0gcmVsaWVzIG9uIHRoZSBgaW1nYCB2YXJpYWJsZSBmcm9tIHRoZSBwcmV2aW91cyB0ZXN0IC0gdGhhdCBjYW4gcXVpY2tseSBicmVhay4gSSBzdWdnZXN0IHRvIGVpdGhlciBkdXBsaWNhdGUgYGltZ2AsIG9yIG1vdmUgaXQgb3V0c2lkZSBvZiB0aGUgYHRlc3RfdGhhdCguLilgIGZ1bmN0aW9uCiAgICAtIHlvdSBzaG91bGQgYWxzbyB0ZXN0IHRoYXQgdGhlIHNlY29uZCBleGVjdXRpb24gb2YgYGRvY2tlcl9pbWdfcm0oKWAgcmV0dXJucyBgRkFMU0VgCiAgLSBJbiBnZW5lcmFsLCAqKmNvbnNpZGVyIGNoZWNraW5nIHRoZSBzdGF0ZSBvdXRzaWRlIG9mIHlvdXIgb3duIGZ1bmN0aW9ucyoqLCBlLmcuIGBleHBlY3Rfd2FybmluZyhleHBlY3RfZXF1YWwoYXR0cihzeXN0ZW0yKCJkb2NrZXIiLCBhcmdzID0gYygiaW5zcGVjdCIsIGltZyksIHN0ZG91dCA9IFRSVUUsIHN0ZGVyciA9IFRSVUUpLCAic3RhdHVzIiksIDEpKWAsIG9yIHdpdGggdGhlIHBhY2thZ2Ugc3RldmVkb3JlIChhbHNvIGFwcGxpZXMgdG8gYHRlc3QtY29udGFpbmVyLlJgKQotIGluIGB0ZXN0LWNvbnNvbGUuUmAKICAtIEkgc3VnZ2VzdCB0byB1c2UgYGV4cGVjdF9vdXRwdXQoLi4pYCwgZS5nLiBgZXhwZWN0X291dHB1dChvdXRzaWRlcjo6OmNhdF9saW5lKCdjYXQgdGhpcycpLCAiY2F0IHRoaXMiKWAgKGFuZCBhbHNvIGZvciBgY2VsZWJyYXRlKClgIGFuZCBgY29tZm9ydCgpYCkKLSBgb3V0c2lkZXI6Ojp2YXJzX2dldCgpYCAtIHRoZSBmdW5jdGlvbiBpcyBvbmx5IGZvciB0ZXN0aW5nLCByaWdodD8gVGhlbiBJIHRoaW5rIGl0IHdvdWxkIGJlIGJldHRlciB0byBhZGQgaXQgdG8gdGhlIHRlc3QgY29kZWJhc2UsIGUuZy4gaW4gYHRlc3QtYWxsLlJgCg==