Retesteth¶

Note

This document is a tutorial. For reference on the retesteth options, look here.

Retesteth through the Web¶

The easiest way to run the tests is through the web interface.

Request Helper¶

To run an existing state test, you can use the request helper. You set these parameters:

Parameter	Meaning	Sample Value
GeneralStateTests/	test suite	stExample
--singletest	name of test	add11
--clients	client to use	t8ntool (the value for geth)
--singlenet	fork to use	Berlin
--vmtrace	trace to produce	raw
--verbosity	log verbosity	none

When a test file contains multiple tests you can restrict which ones you’ll run with the -d, -g, and -v parameters.

Request Single File¶

You can run a single test, either state test or blockchain test, using the request single file option. You specify the test type and then upload a test file. Here are the parameters:

Parameter	Meaning	Sample Value
-t	test suite	test type (state or blockchain)
--testfile	The file to test	you upload this file
--clients	client to use	t8ntool (the value for geth)
--vmtrace	trace to produce	raw
--filltests	test file type	see below

If --filltests is set to none, you need to upload a generated test file. You can find those here (for state tests), and here (for blockchain tests).

If --filltests is set to filltests then you can upload a filler test file, which you can write yourself. This is documented in this tutorial (for state tests) and this one (for blockchain tests).

Custom Command¶

The command line parameters for retesteth are documented here. You can use this option to run whatever parameters you want.

Retesteth in a Docker Container¶

If you want to run the tests locally you can run retesteth inside a Docker container.

These directions are written using Debian Linux 11 on Google Cloud Platform (using a 20 GB disk - the default 10 GB is not enough), but should work with minor changes on any other version of Linux running anywhere else with an Internet connection.

Install docker. You may need to reboot afterwards to get the latest kernel version.
```
sudo apt install -y wget docker docker.io
```

Download the latest retesteth docker image

wget http://retesteth.ethdevops.io/dretesteth.tar

Load the docker image:

sudo docker image load --input dretesteth.tar

Download the dretesteth.sh script.

wget https://raw.githubusercontent.com/ethereum/retesteth/master/dretesteth.sh
chmod +x dretesteth.sh

Download the tests:

git clone --branch develop https://github.com/ethereum/tests.git

Run a test. This has two purposes:

Create the retesteth configuration directories in ~/tests/config, where you can modify them.
A sanity check (that you can run tests successfully).

sudo ./dretesteth.sh -t GeneralStateTests/stExample -- \
 --testpath ~/tests --datadir /tests/config

The output should be similar to:

Running tests using path: /tests
Running 1 test case...
Retesteth config path: /tests/config
Active client configurations: 't8ntool '
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "stExample": (1 of 1)
25%...
50%...
75%...
100%

*** No errors detected
*** Total Tests Run: 12

Note

The /tests directory is referenced inside the docker container. It is the same as the ~/tests directory outside it.

If you get the following error:

Tests folder does not exists, creating test folder: '/tests/GeneralStateTests/stExample'
WARNING: /tests/src/GeneralStateTestsFiller/stExample does not exist!
WARNING: stExample no tests detected in folder!
Running tests for config 'Ethereum GO on StateTool' 2

*** No errors detected
WARNING: /tests/src/GeneralStateTestsFiller does not exist!
*** Total Tests Run: 0

try moving the dretesteth.sh file and cloned tests folder to your home directory (~).

To avoid having to run with sudo all the time, add yourself to the docker group and open a new console.
```
sudo usermod -a -G docker `whoami`
```

How Does This Work?¶

A docker container is similar to a virtual machine, except that it doesn’t run a separate instance of the operating system inside itself so it takes far less resources. One of the features of docker is that it can mount a directory of the host computer inside its own file system. The --testpath parameter to dretesteth.sh tells it what directory to mount, in this case ~/tests which you just cloned from github. It mounts it as /tests inside the container.

By default the retesteth configuration files are in ~/.retesteth. However, that directory is not accessible to us outside the docker. Instead, we use --datadir /tests/config to tell it to use (or create) the configuration in what appears to us to be ~/tests/config, which is easily accessible.

Test Against Your Client¶

There is an instance of geth inside the docker container that you can run tests against. However, unless you are specifically developing tests what you want is to test your client. There are several ways to do this:

Replace the geth inside the docker.
Keep the client on the outside and keep the configuration files intact
Put your client, and any prerequisites, inside the docker and change the configuration files
Keep your client on the outside and connect to it through the network and change the configuration files

When we ran the test in the previous section we also created those configuration files in ~/tests/config, but they were created as being owned by root. If you need to edit them, change the permissions of the config files. To change the configuration files to your own user, run this command:

sudo find ~/tests/config -exec chown $USER {} \; -print

If you look inside ~/tests/config, you’ll see a directory for each configured client. Typically this directory has these files:

config, which contains the configuration for the client:
- The communication protocol to use with the client (typically TCP)
- The address(es) to use with that protocol
- The forks the client supports
- The exceptions the client can throw, and how retesteth should interpret them. This is particularly important when testing the client’s behavior when given invalid blocks.
start.sh, which starts the client inside the docker image
stop.sh, which stops the client instance(s)
genesis, a directory which includes the genesis blocks for various forks the client supports. If this directory does not exist for a client, it uses the genesis blocks for the default client.

Click here for additional documentation. Warning: This documentation may not be up to date

Replace geth Inside the Docker¶

If you want to test a modified version of geth, you can just build it inside the docker container. To do so, recreate the docker container. Before you run sudo ./dretesteth.sh build edit the line that specifies the geth repository. Use whatever repository you use to store your modified version of geth.

Client Outside the Docker, Keep Configuration Files Intact¶

If you want to run your client outside the docker without changing the configuration, these are the steps to follow.

Make sure that the routing works in both directions (from the docker to the client and from the client back to the docker). You may need to configure network address translation.

Run your client. Make sure that the client accepts requests that don’t come from localhost. For example, to run geth use:

geth --http --http.addr 0.0.0.0 retesteth

To run besu use:

docker run -p 8545:8545 -p 13001:30303 \
     hyperledger/besu:latest retesteth --rpc-http-port 8545 \
     --host-allowlist '*'

Run the test the same way you would for a client that runs inside docker, but with the addition of the --nodes parameter. Also, make sure the --clients parameter is set to the client you’re testing.

./dretesteth.sh -t BlockchainTests/ValidBlocks/VMTests -- \
   --testpath ~/tests --datadir /tests/config --clients geth \
   --nodes \<ip\>:\<port, usually defaults to 8545\>

Client Inside the Docker, Modify Configuration Files¶

If you want to run your client inside the docker, follow these steps:

Move the client into ~/tests, along with any required infrastructure (virtual machine software, etc). If you just want to test the directions right now, you can download geth here.
Modify the appropriate start.sh to run your version of the client instead. For example, you might edit ~/tests/config/geth/start.sh to replace geth with /tests/geth in line ten if you put your version of geth in ~/tests.
Run the tests, adding the --clients <name of client> parameter to ensure you’re using the correct configuration. For example, run this command to run the virtual machine tests on geth:
```
./dretesteth.sh -t BlockchainTests/ValidBlocks/VMTests -- --testpath \
~/tests --datadir /tests/config --clients geth
```

Client Outside the Docker, Modify Configuration Files¶

If you want to run your client outside the docker and specify the connectivity in the configuration files, these are the steps to follow:

Create a client in ~/tests/config that doesn’t have start.sh and stop.sh. Typically you would do this by copying an existing client, for example:
```
mkdir ~/tests/config/gethOutside
cp ~/tests/config/geth/config ~/tests/config/gethOutside
```
If you want to specify the IP address and port in the config file, modify the host in the socketAddress to the appropriate remote address. This address needs to work with the JSON over RPC test protocol.

For example,
```
{
   "name" : "Ethereum GO on TCP",
   "socketType" : "tcp",
   "socketAddress" : [ "10.128.0.14:8545" ],
   ...
}
```
Make sure that the routing works in both directions (from the docker to the client and from the client back to the docker). You may need to configure network address translation.
Run your client. Make sure that the client accepts requests that don’t come from localhost. For example, to run geth use:
```
geth --http --http.addr 0.0.0.0 retesteth
```

Run the test the same way you would for a client that runs inside docker:

./dretesteth.sh -t BlockchainTests/ValidBlocks/VMTests -- \
    --testpath ~/tests --datadir /tests/config --clients gethOutside

Running Multiple Threads¶

To improve performance you can run tests across multiple threats. To do this:

If you are using start.sh start multiple nodes with different ports
Provide the IP addresses and ports of the nodes, either in the config file or the –nodes parameter
Run with the parameters -j <number of threads>.

Using the Latest Version¶

The version of retesteth published as a docker file may not have the latest updates. If you want the latest features, you need to build an image from the develop branch yourself:

Install docker.

sudo apt install -y wget docker docker.io

Download the dretesteth.sh script and the Dockerfile. Make sure to do this in an otherwise empty directory, because the docker builder copies everything in or below the directory where Dockerfile is located.

mkdir ~/retestethBuild
cd ~/retestethBuild
wget https://raw.githubusercontent.com/ethereum/retesteth/develop/dretesteth.sh
chmod +x dretesteth.sh
wget https://raw.githubusercontent.com/ethereum/retesteth/develop/Dockerfile

Modify the RUN git clone line in the Dockerfile for repo “retesteth” to change branch -b from master to develop. Do not modify repo branches for “winsvega/solidity” [LLLC opcode support] and “go-ethereum”.
Build the docker image yourself:
```
sudo ./dretesteth.sh build
```
Note

This is a slow process. It took me about an hour on a GCP e2-medium instance.

Conclusion¶

In most cases people don’t start their own client from scratch, but modify an existing client. If the existing client is already configured to support retesteth, you should now be able to run tests on a modified version to ensure it still conforms to Ethereum specifications. If you are writing a completely new client, you still need to implement the RPC calls that retesteth uses and to write the appropriate configuration (config, start.sh, and stop.sh) for it.

There are several actions you might want to do with retesteth beyond testing a new version of an existing client. Here are links to documentation. Note that it hasn’t been updated in a while, so it may not be accurate.

Add configuration for a new client. To do this you need to add retesteth support to the client itself and create a new config for it
Test with a new fork of Ethereum. New forks usually mean new opcodes. Therefore, you will need a docker with a new version of lllc.

If you want to write your own tests, read the next tutorial.