MDBF-1124 Extend MTRTest _save_logs() in master-migration config

[grooverdan]

gzip isn't the fasted but its available. And there's no parallel here, hopefully max-test fail limits cores that way, but heres the test and outcome:

$ time find . -iregex ".*/core\(\.[0-9]+\)" -ls  -exec gzip {} \;
 11834851 851988 -rw-------   1 dan      dan      872431616 Sep 17 23:40 ./s3/20/log/s3.clone/mysqld.1/data/core.142
 11834968 579840 -rw-------   1 dan      dan      593752064 Sep 17 23:40 ./s3/30/log/s3.arguments/mysqld.1/data/core.126
 11835069 854800 -rw-------   1 dan      dan      875311104 Sep 17 23:40 ./s3/24/log/s3.innodb/mysqld.1/data/core.148
 11835035 864004 -rw-------   1 dan      dan      884736000 Sep 17 23:40 ./s3/5/log/s3.replication_partition-mix/mysqld.1/data/core.152
 11835030 881544 -rw-------   1 dan      dan      902696960 Sep 17 23:40 ./s3/5/log/s3.replication_partition-mix/mysqld.2/data/core.160
 11834907 579836 -rw-------   1 dan      dan      593747968 Sep 17 23:40 ./s3/27/log/s3.amazon/mysqld.1/data/core.141
 11835137 579836 -rw-------   1 dan      dan      593747968 Sep 17 23:40 ./s3/29/log/s3.debug/mysqld.1/data/core.125
 11834829 579840 -rw-------   1 dan      dan      593752064 Sep 17 23:40 ./s3/28/log/s3.partition/mysqld.1/data/core.135
 11834814 582772 -rw-------   1 dan      dan      596754432 Sep 17 23:40 ./s3/6/log/s3.replication_stmt-stmt/mysqld.1/data/core.145
 11834808 848372 -rw-------   1 dan      dan      868728832 Sep 17 23:40 ./s3/6/log/s3.replication_stmt-stmt/mysqld.2/data/core.157
 11835009 579836 -rw-------   1 dan      dan      593747968 Sep 17 23:40 ./s3/23/log/s3.basic/mysqld.1/data/core.127
 11834877 589944 -rw-------   1 dan      dan      604098560 Sep 17 23:40 ./s3/4/log/s3.replication_mixed-mix/mysqld.1/data/core.149
 11834871 845164 -rw-------   1 dan      dan      865443840 Sep 17 23:40 ./s3/4/log/s3.replication_mixed-mix/mysqld.2/data/core.158
 11835098 862968 -rw-------   1 dan      dan      883675136 Sep 17 23:40 ./s3/16/log/s3.partition_create_fail/mysqld.1/data/core.144
 11835155 579848 -rw-------   1 dan      dan      593760256 Sep 17 23:40 ./s3/10/log/s3.encryption/mysqld.1/data/core.140
  7783906 838384 -rw-------   1 dan      dan      858501120 Sep 17 23:40 ./s3/19/log/s3.alter/mysqld.1/data/core.151
 11834991 873408 -rw-------   1 dan      dan      894365696 Sep 17 23:40 ./s3/3/log/s3.replication_delayed-mix/mysqld.1/data/core.147
 11834984 859892 -rw-------   1 dan      dan      880525312 Sep 17 23:40 ./s3/3/log/s3.replication_delayed-mix/mysqld.2/data/core.159
 11835052 579832 -rw-------   1 dan      dan      593743872 Sep 17 23:40 ./s3/14/log/s3.discovery/mysqld.1/data/core.146
 11835203 579836 -rw-------   1 dan      dan      593747968 Sep 17 23:40 ./s3/33/log/s3.mysqldump/mysqld.1/data/core.134
 11835173 579836 -rw-------   1 dan      dan      593747968 Sep 17 23:40 ./s3/25/log/s3.alter2/mysqld.1/data/core.139
 11835188 579836 -rw-------   1 dan      dan      593747968 Sep 17 23:40 ./s3/31/log/s3.backup/mysqld.1/data/core.136

real	0m54.004s
user	0m49.123s
sys	0m2.910s
(base) 
~/Downloads/logs 
$ find . -iregex ".*/core.*" -ls
  4505587   3668 -rw-------   1 dan      dan       3754842 Sep 17 23:02 ./normal/log/innodb.innodb-32k-crash/mysqld.1/data/core.115127.gz
 11834945   3748 -rw-------   1 dan      dan       3835736 Sep 17 23:40 ./s3/20/log/s3.clone/mysqld.1/data/core.142.gz
 11834851   3084 -rw-------   1 dan      dan       3156835 Sep 17 23:40 ./s3/30/log/s3.arguments/mysqld.1/data/core.126.gz
 11834946   3752 -rw-------   1 dan      dan       3841323 Sep 17 23:40 ./s3/24/log/s3.innodb/mysqld.1/data/core.148.gz
 11834947   3792 -rw-------   1 dan      dan       3882667 Sep 17 23:40 ./s3/5/log/s3.replication_partition-mix/mysqld.1/data/core.152.gz
 11834948   3816 -rw-------   1 dan      dan       3904975 Sep 17 23:40 ./s3/5/log/s3.replication_partition-mix/mysqld.2/data/core.160.gz
 11834949   3084 -rw-------   1 dan      dan       3155325 Sep 17 23:40 ./s3/27/log/s3.amazon/mysqld.1/data/core.141.gz
 11834890   3732 -rw-------   1 dan      dan       3819141 Sep 17 23:40 ./s3/log/s3.partition_move/mysqld.1/data/core.143.gz
 11834907   3080 -rw-------   1 dan      dan       3150668 Sep 17 23:40 ./s3/29/log/s3.debug/mysqld.1/data/core.125.gz
 11834950   3084 -rw-------   1 dan      dan       3157110 Sep 17 23:40 ./s3/28/log/s3.partition/mysqld.1/data/core.135.gz
 11834829   3116 -rw-------   1 dan      dan       3189444 Sep 17 23:40 ./s3/6/log/s3.replication_stmt-stmt/mysqld.1/data/core.145.gz
 11834814   3776 -rw-------   1 dan      dan       3863740 Sep 17 23:40 ./s3/6/log/s3.replication_stmt-stmt/mysqld.2/data/core.157.gz
 11834808   3080 -rw-------   1 dan      dan       3149829 Sep 17 23:40 ./s3/23/log/s3.basic/mysqld.1/data/core.127.gz
 11834968   3120 -rw-------   1 dan      dan       3192983 Sep 17 23:40 ./s3/4/log/s3.replication_mixed-mix/mysqld.1/data/core.149.gz
 11834877   3776 -rw-------   1 dan      dan       3862969 Sep 17 23:40 ./s3/4/log/s3.replication_mixed-mix/mysqld.2/data/core.158.gz
 11834871   3764 -rw-------   1 dan      dan       3851403 Sep 17 23:40 ./s3/16/log/s3.partition_create_fail/mysqld.1/data/core.144.gz
 11835009   3092 -rw-------   1 dan      dan       3165202 Sep 17 23:40 ./s3/10/log/s3.encryption/mysqld.1/data/core.140.gz
  7783910   3736 -rw-------   1 dan      dan       3822734 Sep 17 23:40 ./s3/19/log/s3.alter/mysqld.1/data/core.151.gz
 11835030   3796 -rw-------   1 dan      dan       3886605 Sep 17 23:40 ./s3/3/log/s3.replication_delayed-mix/mysqld.1/data/core.147.gz
 11834991   3784 -rw-------   1 dan      dan       3872642 Sep 17 23:40 ./s3/3/log/s3.replication_delayed-mix/mysqld.2/data/core.159.gz
 11834984   3084 -rw-------   1 dan      dan       3155308 Sep 17 23:40 ./s3/14/log/s3.discovery/mysqld.1/data/core.146.gz
 11835035   3080 -rw-------   1 dan      dan       3153438 Sep 17 23:40 ./s3/33/log/s3.mysqldump/mysqld.1/data/core.134.gz
 11835052   3080 -rw-------   1 dan      dan       3153404 Sep 17 23:40 ./s3/25/log/s3.alter2/mysqld.1/data/core.139.gz
 11835069   3080 -rw-------   1 dan      dan       3153690 Sep 17 23:40 ./s3/31/log/s3.backup/mysqld.1/data/core.136.gz

[RazvanLiviuVarzaru]

@grooverdan
Sorry for stealing your pull request, let me know if you're okay with 291606f

Saving cores + plugins + mariadbd scenario: https://buildbot.dev.mariadb.org/#/builders/535/builds/11
No cores: https://buildbot.dev.mariadb.org/#/builders/573/builds/27

First case, var.tar.gz logs/ will contain:

• all err, log, core files per test type / per mtr worker, where core files are compressed
• compressed mariadbd
• compressed plugins
• test results in xml format

Second case:

• all err, log files per test type / per mtr worker
• test results in xml format

[RazvanLiviuVarzaru]

@fauust this patch will save more artifacts to help developers debug.
See comment: #844 (comment)

When at least one core file is generated, the script will save:

• all .so plugin shared libraries
• mariadbd binary
• compressed core files

The first two are required so developers can effectively analyse the core files using gdb. @grooverdan and @dr-m will be happy :)

If you want and have time, have a look at this patch.
I want you to know that the storage requirements at least for the builds in #844 (comment) was:

• ~ 200 MB compressed tar for when core files are generated
• ~ 10 MB for when no core files are generated

So please let me know if you have any concerns with the overall storage capabilities of ci.mariadb.org as
this logic of storing will apply to all builders in master-migration

[RazvanLiviuVarzaru]

Asking a plugins Q here, the first comment has already a lot of information.

How to save the plugins when running from the installed tree? E.g. https://buildbot.dev.mariadb.org/#/builders/573/builds/28

usr/lib/mysql/plugin looks constant to me across distros so I might set:

 plugin_dir=$( [ -d /usr/lib/mysql/plugin ] && echo /usr/lib/mysql/plugin || echo "$vardir/plugins" )

then gzip any .so in the path.

Suggestions welcomed.

[grooverdan]

Sorry for stealing your pull request, let me know if you're okay with 291606f

I'm very happy, I was stuck how to reference the mariadbd in its number of variants without breaking some boundaries within the code.

Asking a plugins Q here, the first comment has already a lot of information.

How to save the plugins when running from the installed tree? E.g. https://buildbot.dev.mariadb.org/#/builders/573/builds/28

usr/lib/mysql/plugin looks constant to me across distros so I might set:

 plugin_dir=$( [ -d /usr/lib/mysql/plugin ] && echo /usr/lib/mysql/plugin || echo "$vardir/plugins" )

then gzip any .so in the path.

Suggestions welcomed.

If its an install tree, then there are packages already, and those can be used to pull the binaries. So I don't think this case needs to be considered.

[grooverdan]

one minor correction (of mine) other great!

[RazvanLiviuVarzaru]

Sorry for stealing your pull request, let me know if you're okay with 291606f

I'm very happy, I was stuck how to reference the mariadbd in its number of variants without breaking some boundaries within the code.

Asking a plugins Q here, the first comment has already a lot of information.
How to save the plugins when running from the installed tree? E.g. https://buildbot.dev.mariadb.org/#/builders/573/builds/28
usr/lib/mysql/plugin looks constant to me across distros so I might set:

 plugin_dir=$( [ -d /usr/lib/mysql/plugin ] && echo /usr/lib/mysql/plugin || echo "$vardir/plugins" )

then gzip any .so in the path.
Suggestions welcomed.

If its an install tree, then there are packages already, and those can be used to pull the binaries. So I don't think this case needs to be considered.

@grooverdan
Well, here comes a very important detail.
I configured the -migration builders not to save the packages if at least one test_type fails.
I did this with the idea in mind that when we migrate all the builders to the new-framework, we shouldn't publish the MariaDB packages if there are failing tests.
I realize now that this may not be realistic.

Currently, on the existing release builders, even if the builders that trigger 'autobake' have a failed test, the packages will still be published.
This isn't ideal, but it seems to be an accepted behavior.

In order to have the plugins available as packages, we would have to continue accepting this behavior.
Plus, we don’t save the packages on CI for Pull Requests, for valid storage-related reasons.
So, anyone who needs to debug a pull request will need access to the plugins.

Back to the scope of this patch,
if /usr/lib/mysql/plugin is constant across distros, I'd prefer to save them ...