Orchestrating multiple tests

[Julioevm]

Hey all,
First, impressive job, its a very easy tool to pick up and write tests with!

I have a questions, when running tests locally, whats the way people are running a series of tests?
A parent yml acting as a test suite calling several flows inside?
A script that runs the flows and maybe does some set up / tear down with the emulatos?
Some test runner on top of this?

Cheers,

[mikedawson]

We're using bash scripts that do some setup and tear down of the backend server instance and record videos using ADB, and generate some basic result summary output. I think mobile.dev also have some cloud stuff that can be used to run multiple flows.

We put each test in a directory (each of which has its own script that can copy files the test needs etc).

The overall test runner looks like this:

#!/bin/bash

#Parse command line arguments as per
# /usr/share/doc/util-linux/examples/getopt-example.bash
TEMP=$(getopt -o 's:u:p:e:t:a:' --long 'serial1:,username:,password:,endpoint:,tests:,apk:' -n 'run-maestro-tests.sh' -- "$@")

eval set -- "$TEMP"
unset TEMP

TESTUSER="admin"
TESTPASS="testpass"
WORKDIR=$(pwd)
SCRIPTDIR=$(realpath $(dirname $0))
TESTAPK=$SCRIPTDIR/../../app-android-launcher/build/outputs/apk/release/app-android-launcher-release.apk

# Alias for maestro with common variables

while true; do
        case "$1" in
             '-s'|'--serial1')
                	TESTSERIAL=$2
                        shift 2
                        continue
                ;;
             '-u'|'--username')
                	TESTUSER=$2
                        shift 2
                        continue
                ;;   
             '-p'|'--password')
                   	TESTPASS=$2
                        shift 2
                       continue
                ;;
              '-e'|'--endpoint')
                    ENDPOINT=$2
                      shift 2
                     continue
               ;;
               '-t'|'--tests')
                     echo "Set tests to $2"
                     TESTS=$2
                     shift 2
                     continue
               ;;
               '-a'|'--apk')
                     echo "Set APK to $2"
                     TESTAPK=$2
                     shift 2
                     continue
               ;;
               '--')
                        shift
                        break
                ;;
        
	esac
done

BASEDIR=$(pwd)

if [ "$TESTS" == "" ]; then
  TESTS=$(ls $SCRIPTDIR/tests/*/runtest.sh)
fi

if [ "$TESTSERIAL" == "" ]; then
  echo "ERROR: ADB device serial required: please specify with --serial1 "
  exit 1
fi

if [ ! -f $TESTAPK ]; then
  echo "ERROR: APK File $TESTAPK does not exist! By default this will be the release APK file."
  echo "Alternatively, you can specify a different APK by using --apk path/to/file.apk "
  exit 1
fi

TESTAPK=$(realpath $TESTAPK)

echo "Serial1=$TESTSERIAL"
echo "Username=$TESTUSER"

IPADDR=$(ifconfig | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p' | head -n 1)
if [ "$ENDPOINT" == "" ]; then
    ENDPOINT="http://$IPADDR:8087/"
fi

echo "ENDPOINT=$ENDPOINT"
MAESTRO_BASE_OPTS="--device=$TESTSERIAL test -e ENDPOINT=$ENDPOINT -e USERNAME=$TESTUSER -e PASSWORD=$TESTPASS "

export ANDROID_SERIAL=$TESTSERIAL
if [ "$(adb shell pm list packages com.toughra.ustadmobile)" != "" ]; then
  adb shell pm uninstall com.toughra.ustadmobile
fi
adb -s $TESTSERIAL install $TESTAPK

if [ ! -d $SCRIPTDIR/results ]; then
  mkdir $SCRIPTDIR/results
fi

if [ -e $SCRIPTDIR/results/results-summary.txt ]; then
  rm $SCRIPTDIR/results/results-summary.txt
fi


for TESTFILE in $TESTS; do
  TESTABSPATH=$(realpath $TESTFILE)
  export ANDROID_SERIAL=$TESTSERIAL
  adb shell rm /sdcard/Download/*

  if [ -e $TESTABSPATH/results ]; then
    rm -rf $TESTABSPATH/results
  fi

  TESTDIR=$(dirname $TESTABSPATH)
  cd $TESTDIR
  source $TESTABSPATH

  if [ "$(cat $TESTDIR/results/result)" == "pass" ]; then
    echo "$(basename $TESTDIR) : PASS" >> $SCRIPTDIR/results/results-summary.txt
  fi

  if [ "$(cat $TESTDIR/results/result)" == "fail" ]; then
    echo "$(basename $TESTDIR) : FAIL" >> $SCRIPTDIR/results/results-summary.txt
  fi

  cd $BASEDIR
done

cd $WORKDIR

[axelniklasson]

Hi folks! Please check the docs for test suites and reports available here. Thanks!