Welcome to Xtesting’s documentation!

Xtesting have leveraged on Functest efforts to provide a reference testing framework:

Xtesting aims at allowing a smooth integration of new Functest Kubernetes testcases.

But, more generally, it eases building any CI/CD toolchain for other domains than testing Virtualized Infrastructure Managers (VIM) such as OpenStack.

It now offers a possible reuse of our framework in other OpenSource projects such as ONAP: integration_demo_E2E_chain.pdf

Technical guidelines

  • to support both python2 and python3
  • to be fully covered by unit tests
  • to be well rated by pylint (only local exceptions are accepted on purpose)
  • to be released as a python package and then to be unlinked to OPNFV Milestones
  • to provide Docker containers and manifests for both architectures supported by OPNFV: amd64 and arm64
  • to publish the API documentation online

Try it!

  • run xtesting container:

    $ sudo docker run opnfv/xtesting

  • run xtesting via package (python2):

    $ virtualenv xtesting-py2
$ . xtesting-py2/bin/activate
$ pip install xtesting
$ sudo xtesting-py2/bin/run_tests -t all
$ deactivate

  • run xtesting via package (python3):

    $ virtualenv xtesting-py3 -p python3
$ . xtesting-py3/bin/activate
$ pip install xtesting
$ sudo xtesting-py3/bin/run_tests -t all
$ deactivate

Contents:

xtesting

xtesting package

Subpackages
xtesting.ci package
Submodules
xtesting.ci.run_tests module

The entry of running tests: 1) Parses xtesting/ci/testcases.yaml to check which testcase(s) to be run 2) Execute the common operations on every testcase (run, push results to db…) 3) Return the right status code

exception xtesting.ci.run_tests.BlockingTestFailed

Bases: Exception

Exception when the blocking test fails

class xtesting.ci.run_tests.Result

Bases: enum.Enum

The overall result in enumerated type

EX_ERROR = -1
EX_OK = 0
class xtesting.ci.run_tests.RunTestsParser

Bases: object

Parser to run tests

parse_args(argv=None)

Parse arguments.

It can call sys.exit if arguments are incorrect.

Returns:
the arguments from cmdline
class xtesting.ci.run_tests.Runner

Bases: object

Runner class

static get_dict_by_test(testname)
static get_run_dict(testname)

Obtain the ‘run’ block of the testcase from testcases.yaml

main(**kwargs)

Entry point of class Runner

run_all()

Run all available testcases

run_test(test)

Run one test case

run_tier(tier)

Run one tier

static source_envfile(rc_file='/var/lib/xtesting/conf/env_file')

Source the env file passed as arg

summary(tier=None)

To generate xtesting report showing the overall results

xtesting.ci.run_tests.main()

Entry point

xtesting.ci.tier_builder module

TierBuilder class to parse testcases config file

class xtesting.ci.tier_builder.TierBuilder(testcases_file)

Bases: object

generate_tiers()
get_test(test_name)
get_tests(tier_name)
get_tier(tier_name)
get_tier_name(test_name)
get_tier_names()
get_tiers()
read_test_yaml()
xtesting.ci.tier_handler module

Tier and TestCase classes to wrap the testcases config file

class xtesting.ci.tier_handler.TestCase(name, enabled, skipped, criteria, blocking, description='', project='')

Bases: object

get_criteria()
get_name()
get_project()
is_blocking()
is_enabled()
is_skipped()
class xtesting.ci.tier_handler.Tier(name, description='')

Bases: object

add_test(testcase)
get_name()
get_skipped_test()
get_test(test_name)
get_test_names()
get_tests()
is_test(test_name)
skip_test(testcase)
xtesting.ci.tier_handler.split_text(text, max_len)
xtesting.core package
Submodules
xtesting.core.behaveframework module

Define classes required to run any Behave test suites.

class xtesting.core.behaveframework.BehaveFramework(**kwargs)

Bases: xtesting.core.testcase.TestCase

BehaveFramework runner.

dir_results = '/var/lib/xtesting/results'
parse_results()

Parse output.json and get the details in it.

run(**kwargs)

Run the BehaveFramework feature files

Here are the steps:
  • create the output directories if required,
  • run behave features with parameters
  • get the results in output.json,
Args:
kwargs: Arbitrary keyword arguments.
Returns:
EX_OK if all suites ran well. EX_RUN_ERROR otherwise.
xtesting.core.feature module

Define the parent classes of all Xtesting Features.

Feature is considered as TestCase offered by Third-party. It offers helpers to run any python method or any bash command.

class xtesting.core.feature.BashFeature(**kwargs)

Bases: xtesting.core.feature.Feature

Class designed to run any bash command.

execute(**kwargs)

Execute the cmd passed as arg

Args:
kwargs: Arbitrary keyword arguments.
Returns:
0 if cmd returns 0, -1 otherwise.
class xtesting.core.feature.Feature(**kwargs)

Bases: xtesting.core.testcase.TestCase

Base model for single feature.

execute(**kwargs)

Execute the Python method.

The subclasses must override the default implementation which is false on purpose.

The new implementation must return 0 if success or anything else if failure.

Args:
kwargs: Arbitrary keyword arguments.
run(**kwargs)

Run the feature.

It allows executing any Python method by calling execute().

It sets the following attributes required to push the results to DB:

  • result,
  • start_time,
  • stop_time.

It doesn’t fulfill details when pushing the results to the DB.

Args:
kwargs: Arbitrary keyword arguments.
Returns:
TestCase.EX_OK if execute() returns 0, TestCase.EX_RUN_ERROR otherwise.
xtesting.core.robotframework module

Define classes required to run any Robot suites.

class xtesting.core.robotframework.ResultVisitor

Bases: robot.result.visitor.ResultVisitor

Visitor to get result details.

get_data()

Get the details of the result.

visit_test(test)

Implements traversing through tests.

Can be overridden to allow modifying the passed in test without calling start_test() or end_test() nor visiting keywords.

class xtesting.core.robotframework.RobotFramework(**kwargs)

Bases: xtesting.core.testcase.TestCase

RobotFramework runner.

dir_results = '/var/lib/xtesting/results'
generate_report()

Generate html and xunit outputs

parse_results()

Parse output.xml and get the details in it.

run(**kwargs)

Run the RobotFramework suites

Here are the steps:
  • create the output directories if required,
  • get the results in output.xml,
  • delete temporary files.
Args:
kwargs: Arbitrary keyword arguments.
Returns:
EX_OK if all suites ran well. EX_RUN_ERROR otherwise.
xtesting.core.testcase module

Define the parent class of all Xtesting TestCases.

class xtesting.core.testcase.TestCase(**kwargs)

Bases: object

Base model for single test case.

EX_OK = 0

everything is OK

EX_PUBLISH_ARTIFACTS_ERROR = 66

publish_artifacts() failed

EX_PUSH_TO_DB_ERROR = 69

push_to_db() failed

EX_RUN_ERROR = 70

run() failed

EX_TESTCASE_FAILED = 68

results are false

EX_TESTCASE_SKIPPED = 67

requirements are unmet

check_requirements()

Check the requirements of the test case.

It can be overriden on purpose.

clean()

Clean the resources.

It can be overriden if resources must be deleted after running the test case.

dir_results = '/var/lib/xtesting/results'
get_duration()

Return the duration of the test case.

Returns:
duration if start_time and stop_time are set “XX:XX” otherwise.
headers = {'Content-Type': 'application/json'}
is_successful()

Interpret the result of the test case.

It allows getting the result of TestCase. It completes run() which only returns the execution status.

It can be overriden if checking result is not suitable.

Returns:
TestCase.EX_OK if result is ‘PASS’. TestCase.EX_TESTCASE_SKIPPED if test case is skipped. TestCase.EX_TESTCASE_FAILED otherwise.
publish_artifacts()

Push the artifacts to the S3 repository.

It allows publishing the artifacts.

It could be overriden if the common implementation is not suitable.

The credentials must be configured before publishing the artifacts:

  • fill ~/.aws/credentials or ~/.boto,
  • set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in env.

The next vars must be set in env:

Returns:
TestCase.EX_OK if artifacts were published to repository. TestCase.EX_PUBLISH_ARTIFACTS_ERROR otherwise.
push_to_db()

Push the results of the test case to the DB.

It allows publishing the results and checking the status.

It could be overriden if the common implementation is not suitable.

The following attributes must be set before pushing the results to DB:

  • project_name,
  • case_name,
  • result,
  • start_time,
  • stop_time.

The next vars must be set in env:

  • TEST_DB_URL,
  • INSTALLER_TYPE,
  • DEPLOY_SCENARIO,
  • NODE_NAME,
  • BUILD_TAG.
Returns:
TestCase.EX_OK if results were pushed to DB. TestCase.EX_PUSH_TO_DB_ERROR otherwise.
run(**kwargs)

Run the test case.

It allows running TestCase and getting its execution status.

The subclasses must override the default implementation which is false on purpose.

The new implementation must set the following attributes to push the results to DB:

  • result,
  • start_time,
  • stop_time.
Args:
kwargs: Arbitrary keyword arguments.
xtesting.core.unit module

Define the parent class to run unittest.TestSuite as TestCase.

class xtesting.core.unit.Suite(**kwargs)

Bases: xtesting.core.testcase.TestCase

Base model for running unittest.TestSuite.

generate_html(stream)

Generate html report from subunit stream

Raises:
CalledProcessError
classmethod generate_stats(stream)

Generate stats from subunit stream

Raises:
Exception
generate_xunit(stream)

Generate junit report from subunit stream

Raises:
Exception
run(**kwargs)

Run the test suite.

It allows running any unittest.TestSuite and getting its execution status.

By default, it runs the suite defined as instance attribute. It can be overriden by passing name as arg. It must conform with TestLoader.loadTestsFromName().

It sets the following attributes required to push the results to DB:

  • result,
  • start_time,
  • stop_time,
  • details.
Args:
kwargs: Arbitrary keyword arguments.
Return:
TestCase.EX_OK if any TestSuite has been run TestCase.EX_RUN_ERROR otherwise.
xtesting.core.vnf module

Define the parent class of all VNF TestCases.

exception xtesting.core.vnf.OrchestratorDeploymentException

Bases: Exception

Raise when orchestrator cannot be deployed.

exception xtesting.core.vnf.VnfDeploymentException

Bases: Exception

Raise when VNF cannot be deployed.

class xtesting.core.vnf.VnfOnBoarding(**kwargs)

Bases: xtesting.core.testcase.TestCase

Base model for VNF test cases.

clean()

Clean VNF test case.

It is up to the test providers to delete resources used for the tests. By default we clean:

  • the user,
  • the tenant
deploy_orchestrator()

Deploy an orchestrator (optional).

If this method is overriden then raise orchestratorDeploymentException if error during orchestrator deployment

deploy_vnf()

Deploy the VNF

This function MUST be implemented by vnf test cases. The details section MAY be updated in the vnf test cases.

The deployment can be executed via a specific orchestrator or using build-in orchestrators such as heat, OpenBaton, cloudify, juju, onap, …

Returns:
True if the VNF is properly deployed False if the VNF is not deployed

Raise VnfDeploymentException if error during VNF deployment

prepare()

Prepare the environment for VNF testing:

Returns base.TestCase.EX_OK if preparation is successfull

Raise VnfPreparationException in case of problem

run(**kwargs)

Run of the VNF test case:

  • Deploy an orchestrator if needed (e.g. heat, cloudify, ONAP,…),
  • Deploy the VNF,
  • Perform tests on the VNF

A VNF test case is successfull when the 3 steps are PASS If one of the step is FAIL, the test case is FAIL

Returns:
TestCase.EX_OK if result is ‘PASS’. TestCase.EX_TESTCASE_FAILED otherwise.
test_vnf()

Test the VNF

This function MUST be implemented by vnf test cases. The details section MAY be updated in the vnf test cases.

Once a VNF is deployed, it is assumed that specific test suite can be run to validate the VNF. Please note that the same test suite can be used on several test case (e.g. clearwater test suite can be used whatever the orchestrator used for the deployment)

Returns:
True if VNF tests are PASS False if test suite is FAIL

Raise VnfTestException if error during VNF test

exception xtesting.core.vnf.VnfPreparationException

Bases: Exception

Raise when VNF preparation cannot be executed.

exception xtesting.core.vnf.VnfTestException

Bases: Exception

Raise when VNF cannot be tested.

xtesting.utils package
Submodules
xtesting.utils.constants module
xtesting.utils.decorators module
xtesting.utils.decorators.can_dump_request_to_file(method)
xtesting.utils.env module
xtesting.utils.env.get(env_var)
xtesting.utils.env.string()

Indices and tables