![Sync](http://ctestnet.eosio.se/steem/eosio_sync_header.gif) Updated: April 13, 2018 _Author:_@xebb82 _contributors:_ @summerskin, @michaelyeates, @zaratustra, @nsjames, @v2c, @marcinup # Outline * [Intro and purpose](#intro-and-purpose) * [Installation and configuration](#installation-and-configuration) - Build dawn-v3.0.0 on Ubuntu 16.04, 16.10 or 17.10 - One way to run multiple testnets on the same server - Connect to the Community Testnet - ... or start your own testnet * [Making our setups a bit more realistic](#making-the-setup-a-bit-more-realistic) * [Load testing](#performance-testing) - Hardware setup - Configuration - [Results](#results) - Conclusions * What's next? * [Links](#links) ###### Disclaimer: *This post outlines one way to run multiple testnets on the same server. Several excellent posts which details how to get the software up and running already exists: [Bohdan](https://github.com/CryptoLions/EOS-Jungle-Testnet), [Eugene](https://busy.org/@eluzgin/how-to-join-eosnet-io-testnet-for-bp-candidates), [Marcin](https://github.com/eosgreen/EOS.IO-Community-Testnet-Reactivation), [Michael](https://github.com/michaeljyeates/10mill/). However we still get a lot of questions. This post describes how I do it. This is not the only way. This is not even the [best way](#other-ways). But it is [my way](https://www.youtube.com/watch?v=5AVOpNR2PIs) :)* # Intro and purpose Technically versed people that have either been playing with the eosio software or who understand how to use and follow the official instructions located [here](https://github.com/eosio/eos/wiki) can save some time and jump straight down to: [Making the setup a bit more realistic](#making-the-setup-a-bit-more-realistic). _Everything in this post is subject to change as more information is released and as new tools come out._ This post covers how to download, build and configure the software to connect to a testnet, as well as how to create your own testnet. It also goes through some technical aspects to consider when planning how to run your multi-node setup. Lastly, we detail some performance tests that we have run on a testnet using multiple different setups. We believe some of the community will enjoy the information provided in this post and therefore, without further adieu, here it is. # Installation and configuration In this part we are assuming that the reader has a basic understanding on how to read and execute bash scripts using a Linux environment. The scripts provided have been tested on Xubuntu 17.10 but they should work on Ubuntu >= 16.04. Since early January we have had the Community Testnet more or less stable. We were however forced to reset the chain as we upgraded it to the branch that we thought would become dawn3 - but instead it turned out to be SuperDawn (the producers in the Community Testnet are named after superheroes and we got a tag named after us!). ![tenor.gif](http://ctestnet.eosio.se/tenor.gif) $ ./client.sh get block 1 | jq -e ".timestamp" "2018-02-03T21:46:00.000" Our first block was created a little over than 2 months ago. During that time: $ du -h --max-depth=1 2,1G ./blocks 1,1G ./shared_mem 3,2G . ... the directory holding the blockchain has grown to 3.2G in total. As long as the chain is made up of mostly empty blocks it does not consume much disk. And neither does the software: $ du -h --max-depth=1 1,5G ./eos 1,5G . One assumption I'm making throughout this post is that disk can be seen as "free". To someone running their machines in the cloud this assumption is probably not correct (_you might want to use one directory where you build the eosio software, and then only keep the binaries_), but for those running on dedicated hardware it is probably true enough. ## Build dawn-v3.0.0 on Ubuntu 16.04, 16.10 or 17.10 If you want to do it in a standardized and simple way you can just: $ cd && git clone https://github.com/eosio/eos --recursive $ cd eos && git checkout tags/dawn-v3.0.0 $ ./eosio_build.sh ... or you can: ## One way to run multiple testnets on the same server One way to run multiple testnets on different versions and setups on the same machine while lowering the risk of mixing them up is to do something like the following: $ cd && mkdir testnets && mkdir testnets/[testnetName] $ screen -SU [testnetName] $ cd testnets/[testnetName] $ mkdir wallet data-dir config-dir $ touch wallet/start.sh start.sh cleos.sh $ git clone https://github.com/eosio/eos --recursive $ cd eos && git checkout tags/dawn-v3.0.0 $ git submodule update --init --recursive $ ./eosio_build.sh Output from the build process should look like this: ![tenor.gif](http://ctestnet.eosio.se/eosio_dawn3_built.png) 1. We start by creating one directory (_testnets_) under which we create one directory per testnet. 2. A tip that helps me keep track of my different testnets: I start a [screen](https://www.nixtutor.com/linux/introduction-to-gnu-screen/) session for each and everyone of them. These run in the background and then can easily be attached as you log back on or switch between the different sessions. I personally let nodeos run in one window (mostly because I like the [colored output](https://github.com/EOSIO/eos/issues/2372)). Then I use another window where I call the client (note that we in a real environment will want to log to disk). 3. Clone and build one copy of the eosio software per testnet. This makes it quick and easy to upgrade one specific testnet with limited impact on the other. 4. Create three scripts per testnet; * _start.sh_ - start the daemon * _cleos.sh_ - call the client * _wallet/start.sh_ - start the wallet daemon Add something like the following to _start.sh_ (we will modify this one later): #!/bin/bash ./eos/build/programs/nodeos/nodeos --data-dir data-dir/ --config-dir config-dir/ $@ _cleos.sh_: #!/bin/bash ./eos/build/programs/cleos/cleos -H localhost -p 8888 --wallet-host localhost --wallet-port 8000 $@ _wallet/start.sh_: Here we bind the wallet (_keosd_) to localhost so no other system can access our private keys. You do not want to run the wallet plug-in with your _nodeos_. #!/bin/bash ../eos/build/programs/keosd/keosd --http-server-address localhost:8000 $@ & Next step is to make _cleos.sh_ and _start.sh_ executable, start the wallet daemon and import the keys: $ cd && cd testnets/[testnetName]/wallet && chmod u+x start.sh $ ./start.sh $ cd .. && chmod u+x start.sh && chmod u+x cleos.sh # ( !!! make sure you save the password from the following command !!! ) $ ./cleos.sh wallet create $ ./cleos.sh wallet import # (show all private keys in the wallet) $ ./cleos.sh wallet keys ### Other ways There are multiple ways of doing this same thing and it is important to find your own specific flavor. A lot of good inspiration can be found in a [tool](https://github.com/zrts/run_eos) written by zaratustra, intended to make it easier to launch and manage multiple EOS.IO chains. Eduardo also provide some [docker images](https://hub.docker.com/r/camponez/bp-eosio/). ## Connect to the Community Testnet With small modifications the information below can be used to connect to any and all of the different testnets available. First we need to get the genesis.json and the config.ini: $ cd && cd testnets/[testnetName] $ wget https://raw.githubusercontent.com/michaeljyeates/10mill/master/genesis.json $ wget https://raw.githubusercontent.com/michaeljyeates/10mill/master/config.ini Certain things need to be modified in _config.ini_ genesis-json = /home//testnets/[testnetName]/config-dir/genesis.json http-server-address = 0.0.0.0:8888 p2p-listen-endpoint = 0.0.0.0:9876 p2p-server-address = : agent-name = "" Setting http-server-address to 0.0.0.0:8888 means that it will listen on all network interfaces, on the port 8888. If you have something else running on this port you will receive an error when trying to start the daemon, it will say something like: "*bind: Address already in use*" The p2p-server-address is afaik not used for anything other than identifying your node. It should to be set to something unique. To start syncing the Community Testnet blockchain: $ cd && cd testnets/[testnetName] $ ./start.sh It is normal if it looks something like the following: ![Dawn3-v3.0.0 syncing process](http://ctestnet.eosio.se/eosio_dawn3_syncing.png) 1. look at our [monitor](http://superhero3monitor.eosgreen.io) (_handled by @marcinup_, built by @cryptolions) to see which block we are currently at 2. execute _cleos.sh_ to see which block you are at $ cd && cd testnets/[testnetName] $ ./cleos.sh get info Once they match you are fully synced. Come join the fun at https://t.me/CommunityTestnet . Unfortunately we don't accept new producers until we have BP voting functionality. ## Once you are up and running Once everything is working as it should it might be a good idea for you to start running nodeos in the background and re-direct the output to a log file on disk instead of logging directly to the window in your screen. If you edit your _start.sh_ to look something like this: #!/bin/bash ./eos/build/programs/nodeos/nodeos --data-dir data-dir/ --config-dir config-dir/ > data-dir/log.txt 2>&1 & ... then you will redirect the output from nodeos to go to a text file. You can do something like this to keep an eye on the last 30 lines: $ watch tail -n 30 data-dir/log.txt In the future you will want to add some automated way to keep track of what is going on in this file. If you want you can add: "_echo $! > data-dir/nodeos.pid_" to _start.sh_, which will save the pid in a file (data-dir/nodeos.pid). This pid can be killed (use -2 och -15) if you need to restart _nodeos_. ## Connect to one of the other community driven testnets Under the [Links](#links) section we have included information to get in touch with all community driven testnets that we are aware of. Both the Scholar testnet and Arrowhead are currently accepting new producers. ## ... or start your own testnet First we need to create a genesis file. We recommend placing this at: testnets/`[testnetName]`/config-dir/genesis.json but it can be anywhere you want. The following genesis is what we used when we started the Community Testnet on dawn-v3.0.0 (_a gold star to the first one who decodes the chain_id_). Make sure to generate and use your own initial_key (_./cleos.sh create key_ is your friend). { "initial_key": "EOS55xgWhiYohaHtFTxcPMorB9MzwBfe3JgUuQEujrurmHLHiXmna", "initial_timestamp": "2018-04-06T00:00:00", "initial_parameters": { "maintenance_interval": 86400, "maintenance_skip_slots": 3, "maximum_transaction_size": 2048, "maximum_block_size": 2048000000, "maximum_time_until_expiration": 86400, "maximum_producer_count": 1001 }, "immutable_parameters": { "min_producer_count": 42 }, "initial_chain_id": "00000000000000000000000000000000003c332053555045524845524f203c33" } #### Boot the chain What is written under this topic will probably be replaced by Alexandre Bourgets [EOS Bios Launch](https://github.com/eoscanada/eos-bios-launch-data) in the near future. Until then, this will serve: Michael put together a bash script that can be used to help with the following steps of the boot process: 1. Set the eosio.bios contract 2. Create producers based on a csv-file 3. Set the eosio.system contract, issue 1000000000 EOS and fund each producer with 1 000 000 EOS (we have a problem with showing balances and possibly setting the eosio.token contract fixes this - but it has yet to be confirmed (thanks @cryptolions)) Download the script from our highly official github. $ wget https://raw.githubusercontent.com/michaeljyeates/10mill/master/boot_chain.sh Some things in *boot_chain.sh* needs to be edited to fit your environment: #!/bin/bash CLEOS=/home//testnets/[testnetName]/eos/build/cleos/cleos HOST=localhost PORT=8888 WALLET_HOST=localhost WALLET_PORT=8000 EOS_BUILD_DIR=/home//testnets/[testnetName]/eos/build EOSIO_KEY=EOS7rgcwNPrpgd9FSGAyg6Qn5pHE3sV1nYhiXDRDB2rtVkoh7BX7H Create the file _producers.csv_ (producer_name,public_key) and put it in the same directory as your *boot_chain.sh*. It could look something like this: (make sure each producer have the matching private keys to the public keys you are using): firestorm,EOS5cMGmNturu4C1R4T8dB2LVx5SMNoiL9uhYdNKJVxmdLuAnEqo1 ironman,EOS59vUXs8FXvZshSCMBgzseDpu7jcnzWmyv29XXyH3gqUtZ84t41 aquaman,EOS64Vgd3PVJfoTvB19K8fShiy9e1ZGo1iN33PgLrJfUDeTXSmSuG phoenix,EOS62ZdqFkz4yWhdF3vE2AzMLDSRgEqSfLm5NESkabGY6V4HQ8PkB batman,EOS5wEcKNZ8YdVnCSCnGmwpWLwb2cCqP5FSyWoW1vBASr78KKu8G3 Create a _config.ini_. Make sure: 1. you are not connecting to any peers: leave p2p-peer-address commented out (and use a port that noone will connect to) 2. the private-key used as initial_key in the genesis.json is included as: private-key = ["EOS...", "..."] 3. producer-name = eosio Then start the chain. Validate that _eosio_ is producing blocks, import the private key for eosio and then run the script: $ cd && cd testnets/[testnetName] $ ./cleos.sh wallet import $ chmod u+x boot_chain.sh $ ./boot_chain.sh At this point you can use the steps detailed in [Connect to the Community Testnet](#connect-to-the-community-testnet) to launch another instance of _nodeos_ to start producing with your new producers (firestorm,ironman,aquaman,phoenix and batman) #### Import the ERC20 distribution As far as I know there is currently no software available from block.one on how to import the ERC20 token distribution. Therefore I quickly whipped together two bash scripts that can be used for this (found at the [ScholarTestnets](https://github.com/ScholarTestnet/scholar-scripts) github). You need a snapshot of the distribution. Either you can [create](https://github.com/EOSIO/genesis) one yourself, or you can cheat and download a snapshot made by Robert Allen (found [here](https://github.com/eosdac/airdrop)). The script I wrote loops the csv-file, extracts the amount and the public key. It then randomizes an account name, creates the account and transfers tokens to it. It had at least two small bugs that the Scholars might have fixed by now: * a short _sleep_ could be added after every 300-600 transactions, else you'll get the *block exhausted allowed resources*-message and your transactions will not go through (if you are lazy you can just stop the script, and then run it again after waiting a few seconds) * "0000" needs to be added to the amount to make sure the correct distribution is sent to each user # Making our setups a bit more realistic As we are getting closer to launch it is probably time to start taking our setups more serious. In this post we decided to test a few slightly different setups and document how they each perform under load. #### 1) Single producer ![Single location single producer](http://ctestnet.eosio.se/steem/single_location_single_producer.png "Shared firewall") ##### Incoming traffic The firewall forwards incoming traffic on ports 9876 and 8888 to the producing node which provide users and other producers access to the blockchain. ##### Outgoing traffic None. #### 2) Shared firewall ![Single location multiple node shared firewall](http://ctestnet.eosio.se/steem/single_location_one_node_one_producer_shared_fw.png "Shared firewall") ##### Incoming traffic The firewall forwards incoming traffic on ports 9876 and 8888 to the full node which provide users and other producers access to the blockchain. ##### Outgoing traffic Both the full node and the producing node will establish multiple connections to the nodes making up the blockchain. #### 3) Hidden producer ![Single location hidden producer](http://ctestnet.eosio.se/steem/single_location_one_node_hidden_producer.png "Hidden producer") ##### Incoming traffic The firewall forwards incoming connections on ports 9876 and 8888 to the full node which allows other producers and users access to the blockchain. ##### Outgoing traffic The producer is connected only to the full node which in turn handles the connection to the multiple nodes making up the blockchain. #### 4) Two full nodes, one balancer and one hidden producer The initial tests with one hidden producer and one full node ([2](hidden-producer)) seemed to indicate that the amount of transactions included in each block went down slightly as the producer was not connected to the sender, therefore we added a test with dual full nodes. More on this under [results](#results). ![Two full nodes one balancer and one hidden producer](http://ctestnet.eosio.se/steem/single_location_two_nodes_one_balancer.png "Two full nodes, one balancer and one hidden producer") ##### Incoming traffic The load balancer forwards incoming connections on ports 9876,9776 and 8888,8788 to the full nodes which allows other producers and users access to the blockchain. ##### Outgoing traffic The producer is connected to the two full nodes. #### 5) Six nodeos on two full nodes, one balancer and one hidden producer This is a variation of [4)](#two-full-nodes-one-balancer-one-hidden-producer) where we run three instances of _nodeos_ per full node. ##### Incoming traffic The load balancer forwards incoming connections on ports 9876,9866,9856, 9776,9766,9756 and 8888,8878,8868, 8788,8778,8768 to the full nodes which allows other producers and users access to the blockchain. ##### Outgoing traffic The producer is connected only to the full nodes. The full nodes are not connecting to any node. # Load testing ## Hardware setup We decided to use three different machines in these tests, they are all running Ubuntu 17.10, are connected via 1 gbps network and have the following hardware specifications; 1. AMD Ryzen 1800x, 32GB, Samsung 960 PRO m.2 512GB 2. Dual Xeon X5670 6 core @ 2.93 GHz, 64GB, 4x148GB 10k rpm raid5 3. Dual Xeon X5670 6 core @ 2.93 GHz, 128GB, 2x240GB Intel SSD Pro raid1 One thing we wanted to investigate was how the different servers stand compared to each other. To do this we have been moving the producing accounts (_six in total_) between the different servers: de-facto turning producers into full nodes and vice-versa. Since I only have one machine running on AMD; all _AMD - AMD_ tests are done using the same server. For Intel I have made sure to use both machines; one as producer, one as full node. ### Configuration _This script test far from all aspects in need of testing._ We decided to focus this article mostly on running this script on multiple machines, with different configurations and with small variations to the script (such as sending from multiple accounts / adding a _&_ to the end of the line starting with _./cleos.sh_... starting a new thread for each round in the loop and so forth): #!/bin/bash x=1 while [ "$x" -lt "601" ] ; do ./cleos.sh transfer eosio storm $x test -f let "x++" done ## Results This section can seem a bit spammy but I want to include as much data as possible. It is the only way I can see to make sure we will receive good input regarding the (_lack of_) conclusions drawn below. _I have left the first line as is, and then cleaned the rest to fit the logging output into fewer lines_ ### 1) [Single producing node](#single-producing-node) The client run cleos which connects directly to the producing node to push the transactions into the queue. ##### 400 transactions - single process ###### Producer: Intel ###### Client : AMD storm generated block 69d80e1a... #936319 @ 2018-04-12T09:18:52.500 with 32 trxs, lib: 936272 storm generated block eae7e8e2... #936320... 38 trxs, lib: 936272 gargamel generated block 4b8bd1ba... #936321... 38 trxs, lib: 936284 gargamel generated block b8f5ff12... #936322... 38 trxs, lib: 936284 gargamel generated block 1b3011a4... #936323... 38 trxs, lib: 936284 gargamel generated block bacca3fd... #936324... 39 trxs, lib: 936284 gargamel generated block 9a7f3f63... #936325... 38 trxs, lib: 936284 gargamel generated block 75b86a35... #936326... 37 trxs, lib: 936284 gargamel generated block c2b90933... #936327... 39 trxs, lib: 936284 gargamel generated block e083987a... #936328... 38 trxs, lib: 936284 gargamel generated block bd3483fa... #936329... 25 trxs, lib: 936284 _Accepted transactions:_ 400 ###### Client : Intel eosio generated block 02416a28... #934068 @ 2018-04-12T09:00:06.500 with 11 trxs, lib: 934029 eosio generated block b0064143... #934069... 23 trxs, lib: 934029 eosio generated block be9d02de... #934070... 23 trxs, lib: 934029 eosio generated block 7f49eb8c... #934071... 23 trxs, lib: 934029 eosio generated block ca3e3016... #934072... 24 trxs, lib: 934029 eosio generated block adee9e8d... #934073... 23 trxs, lib: 934029 eosio generated block f282b9bd... #934074... 24 trxs, lib: 934029 eosio generated block d39450d5... #934075... 23 trxs, lib: 934029 eosio generated block 8d9d0280... #934076... 23 trxs, lib: 934029 eosio generated block 038c7c13... #934077... 24 trxs, lib: 934029 storm generated block 13be901d... #934078... 25 trxs, lib: 934041 storm generated block 601f8483... #934079... 23 trxs, lib: 934041 storm generated block 4cf71e87... #934080... 24 trxs, lib: 934041 storm generated block 83458b1b... #934081... 23 trxs, lib: 934041 storm generated block 02777f90... #934082... 24 trxs, lib: 934041 storm generated block 587224ae... #934083... 23 trxs, lib: 934041 storm generated block e720dcf3... #934084... 23 trxs, lib: 934041 storm generated block f1d46148... #934085... 14 trxs, lib: 934041 _Accepted transactions_: 400 ##### Producer AMD ##### 400 transactions - multiple threads ###### Client : AMD batgirl generated block bbc4465f... #938719 @ 2018-04-12T09:38:52.500 with 51 trxs, lib: 938672 batgirl generated block bdcc5bd9... #938720... 148 trxs, lib: 938672 noheart generated block 51f744e2... #938721... 67 trxs, lib: 938684 noheart generated block ccd7271f... #938722... 123 trxs, lib: 938684 noheart generated block c45f4dae... #938723... 8 trxs, lib: 938684 noheart generated block f7e15287... #938725... 3 trxs, lib: 938684 _Accepted transactions_: 400 ###### Client : Intel noheart generated block 047016a3... #940382 @ 2018-04-12T09:52:44.000 with 12 trxs, lib: 940340 noheart generated block 6cc4ea81... #940383... 98 trxs, lib: 940340 robin generated block 916c95c0... #940392... 59 trxs, lib: 940352 robin generated block 4fb18ebf... #940393... 76 trxs, lib: 940352 eosio generated block a37dd8d0... #940402... 64 trxs, lib: 940364 _Accepted transactions_: 309 #### Producer Intel ###### Client : Intel eosio generated block e42da57c... #944230 @ 2018-04-12T10:27:07.500 with 39 trxs, lib: 944189 eosio generated block 89bfd68b... #944231... 183 trxs, lib: 944189 eosio generated block 36ad237d... #944232... 36 trxs, lib: 944189 eosio generated block 15ceb810... #944233... 133 trxs, lib: 944189 eosio generated block 70a091b8... #944234... 9 trxs, lib: 944189 _Accepted transactions_: 400 ###### Client : AMD noheart generated block 58359209... #944996 @ 2018-04-12T10:33:30.500 with 197 trxs, lib: 944957 noheart generated block ab52eddb... #944997... with 2 trxs, lib: 944957 noheart generated block 6c3fc2bc... #944998... with 194 trxs, lib: 944957 noheart generated block e7fa13fe... #945000... with 7 trxs, lib: 944957 _Accepted transactions_: 400 ### 2) [Shared firewall](shared-firewall) #### 400 transactions - single thread ##### Producer AMD ###### Client : AMD eosio generated block 238d0f5b... #474716 @ 2018-04-09T14:33:43.500 with 17 trxs, lib: 474675 eosio generated block b9346754... #474717... 37 trxs, lib: 474675 eosio generated block 3bb1c121... #474718... 35 trxs, lib: 474675 eosio generated block 7fe86788... #474719... 37 trxs, lib: 474675 eosio generated block 9df12ec0... #474720... 36 trxs, lib: 474675 eosio generated block 2b673f6f... #474721... 37 trxs, lib: 474675 eosio generated block 57ef6011... #474722... 37 trxs, lib: 474675 eosio generated block 1b413e1f... #474723... 37 trxs, lib: 474675 storm generated block 7fb9b53a... #474724... 37 trxs, lib: 474687 storm generated block 7263eb30... #474725... 35 trxs, lib: 474687 storm generated block ccd4d263... #474726... 37 trxs, lib: 474687 storm generated block 28870f83... #474727... 18 trxs, lib: 474687 _Accepted: 400_ ###### Client : Intel storm generated block 7b75f0ea... #974247 @ 2018-04-12T14:37:25.500 with 7 trxs, lib: 974206 storm generated block 3508866e... #974248... 23 trxs, lib: 974206 storm generated block ce1c9b79... #974249... 23 trxs, lib: 974206 storm generated block cf16b535... #974250... 23 trxs, lib: 974206 storm generated block d11ab6bd... #974251... 23 trxs, lib: 974206 storm generated block 86096b8c... #974252... 24 trxs, lib: 974206 storm generated block d74e0c97... #974253... 23 trxs, lib: 974206 storm generated block 546e1dc6... #974254... 23 trxs, lib: 974206 gargamel generated block 4534f0bd... #974255... 23 trxs, lib: 974218 gargamel generated block 9eae1196... #974256... 23 trxs, lib: 974218 gargamel generated block 46b0e19d... #974257... 24 trxs, lib: 974218 gargamel generated block 5db95d6b... #974258... 23 trxs, lib: 974218 gargamel generated block c2e6c8e0... #974259... 24 trxs, lib: 974218 gargamel generated block 0ff75754... #974260... 23 trxs, lib: 974218 gargamel generated block e2ea9cbc... #974261... 24 trxs, lib: 974218 gargamel generated block 64b8cb62... #974262... 23 trxs, lib: 974218 gargamel generated block 2094d945... #974263... 23 trxs, lib: 974218 gargamel generated block f42f750e... #974264... 21 trxs, lib: 974218 _Accepted:_ 400 ##### Producer Intel ###### Client : AMD eosio generated block e14aa8f5... #961940 @ 2018-04-12T12:54:47.000 with 5 trxs, lib: 961892 storm generated block ded09ff3... #961941... 39 trxs, lib: 961904 storm generated block 57b182da... #961942... 38 trxs, lib: 961904 storm generated block 42bd953a... #961943... 38 trxs, lib: 961904 storm generated block bfe23d4f... #961944... 39 trxs, lib: 961904 storm generated block dd81dee3... #961945... 38 trxs, lib: 961904 storm generated block 6896bda7... #961946... 38 trxs, lib: 961904 storm generated block 42f2dd71... #961947... 39 trxs, lib: 961904 storm generated block 0ae1c8ea... #961948... 38 trxs, lib: 961904 storm generated block 38d85fdd... #961949... 39 trxs, lib: 961904 storm generated block 2f679555... #961950... 39 trxs, lib: 961904 _Accepted:_ 390 ###### Client : Intel gargamel generated block 742491b3... #962173 @ 2018-04-12T12:56:43.500 with 25 trxs, lib: 962132 gargamel generated block 8c4d8053... #962174... 26 trxs, lib: 962132 gargamel generated block 25df3125... #962175... 28 trxs, lib: 962132 gargamel generated block 09724881... #962176... 26 trxs, lib: 962132 gargamel generated block b9205c5f... #962177... 26 trxs, lib: 962132 gargamel generated block 38010e88... #962178... 27 trxs, lib: 962132 gargamel generated block 13b11d88... #962179... 27 trxs, lib: 962132 gargamel generated block bc023a92... #962180... 27 trxs, lib: 962132 batgirl generated block d16e54af... #962181... 27 trxs, lib: 962144 batgirl generated block 4fc23dd6... #962182... 27 trxs, lib: 962144 batgirl generated block 6730e0bb... #962183... 25 trxs, lib: 962144 batgirl generated block f43079b3... #962184... 27 trxs, lib: 962144 batgirl generated block 1d0e376f... #962185... 26 trxs, lib: 962144 batgirl generated block d6414136... #962186... 27 trxs, lib: 962144 batgirl generated block d0be491f... #962187... 27 trxs, lib: 962144 batgirl generated block cd3893e5... #962188... 2 trxs, lib: 962144 _Accepted:_ 400 #### 400 transactions - multiple threads ##### Producer AMD ###### Client : AMD storm generated block 59c6f1ae... #974609 @ 2018-04-12T14:40:26.500 with 4 trxs, lib: 974566 storm generated block 60533d50... #974610... 148 trxs, lib: 974566 storm generated block efd26313... #974611... 24 trxs, lib: 974566 storm generated block 0ca9b026... #974612... 162 trxs, lib: 974566 storm generated block 45141188... #974614... 1 trxs, lib: 974566 gargamel generated block 03a3d02a... #974616... 61 trxs, lib: 974578 _Accepted:_ 400 ###### Client : Intel batgirl generated block 79ba6f1f... #986730 @ 2018-04-12T16:21:28.500 with 12 trxs, lib: 986683 batgirl generated block ab749e72... #986731... 206 trxs, lib: 986683 noheart generated block ca968e1e... #986732... 8 trxs, lib: 986695 noheart generated block 9a9aeda9... #986733... 153 trxs, lib: 986695 noheart generated block 665c9fa4... #986737... 11 trxs, lib: 986695 _Accepted:_ 390 ##### Producer : Intel ###### Client : AMD noheart generated block 19e82a4f... #987164 @ 2018-04-12T16:25:05.500 with 3 trxs, lib: 987127 noheart generated block 46a45013... #987165... 208 trxs, lib: 987127 noheart generated block a40159bb... #987167... 185 trxs, lib: 987127 noheart generated block dbc9579f... #987169... 2 trxs, lib: 987127 noheart generated block 770f4636... #987171... 2 trxs, lib: 987127 _Accepted:_ 397 ###### Client : Intel gargamel generated block b2388402... #986996 @ 2018-04-12T16:23:41.500 with 178 trxs, lib: 986959 gargamel generated block 85c48888... #986997... 56 trxs, lib: 986959 gargamel generated block 686602cc... #986998... 166 trxs, lib: 986959 _Accepted:_ 400 ### 3) [Hidden producer](#hidden-producer) The client run cleos which connects directly to the full node to get the transactions into the queue. #### 400 transactions - single thread ##### Producer AMD ###### Client : AMD noheart generated block 6135feb2... #990185 @ 2018-04-12T16:50:17.500 with 21 trxs, lib: 990148 noheart generated block f945bc8e... #990186... 36 trxs, lib: 990148 noheart generated block e202dbb0... #990187... 36 trxs, lib: 990148 noheart generated block b3f42ccb... #990188... 36 trxs, lib: 990148 noheart generated block 469f55f6... #990189... 35 trxs, lib: 990148 noheart generated block 4bdcdcbc... #990190... 37 trxs, lib: 990148 noheart generated block 83cc0ca1... #990191... 34 trxs, lib: 990148 noheart generated block bb4b7291... #990192... 36 trxs, lib: 990148 noheart generated block acac2070... #990193... 36 trxs, lib: 990148 noheart generated block 076ce3f3... #990194... 36 trxs, lib: 990148 noheart generated block 8eb40b5d... #990195... 36 trxs, lib: 990148 noheart generated block 2c47515d... #990196... 21 trxs, lib: 990148 _Accepted:_ 400 ###### Client : Intel robin generated block c089fac4... #975381 @ 2018-04-12T14:46:52.500 with 24 trxs, lib: 975334 robin generated block 18c3d110... #975382... 26 trxs, lib: 975334 eosio generated block 0d7ef01a... #975383... 25 trxs, lib: 975346 eosio generated block 1708ad64... #975384... 27 trxs, lib: 975346 eosio generated block 261799dc... #975385... 26 trxs, lib: 975346 eosio generated block 00df4a86... #975386... 25 trxs, lib: 975346 eosio generated block 48118069... #975387... 26 trxs, lib: 975346 eosio generated block 3dc8603d... #975388... 26 trxs, lib: 975346 eosio generated block afb66413... #975389... 25 trxs, lib: 975346 eosio generated block db60d0c9... #975390... 27 trxs, lib: 975346 eosio generated block b9290be7... #975391... 26 trxs, lib: 975346 eosio generated block 84719f28... #975392... 26 trxs, lib: 975346 eosio generated block fad10576... #975393... 25 trxs, lib: 975346 eosio generated block 82243480... #975394... 27 trxs, lib: 975346 storm generated block c25f7cb3... #975395... 26 trxs, lib: 975358 storm generated block 64eed439... #975396... 13 trxs, lib: 975358 _Accepted:_ 400 ##### Producer Intel ###### Client : AMD gargamel generated block 8f40d25c... #977215 @ 2018-04-12T15:02:10.500 with 18 trxs, lib: 977168 gargamel generated block 1e44eb1c... #977216... 36 trxs, lib: 977168 batgirl generated block c078a0a5... #977217... 34 trxs, lib: 977180 batgirl generated block e92eed1e... #977218... 34 trxs, lib: 977180 batgirl generated block 91f32026... #977219... 37 trxs, lib: 977180 batgirl generated block 81187f25... #977220... 36 trxs, lib: 977180 batgirl generated block 4cc9b790... #977221... 37 trxs, lib: 977180 batgirl generated block c20d3a35... #977222... 36 trxs, lib: 977180 batgirl generated block ed8d3a54... #977223... 37 trxs, lib: 977180 batgirl generated block 116323be... #977224... 36 trxs, lib: 977180 batgirl generated block c9d763d0... #977225... 35 trxs, lib: 977180 batgirl generated block b8c9284f... #977226... 24 trxs, lib: 977180 _Accepted:_ 400 ###### Client : Intel gargamel generated block 1317f42b... #975994 @ 2018-04-12T14:51:59.000 with 13 trxs, lib: 975946 batgirl generated block b30ce053... #975995... 24 trxs, lib: 975958 batgirl generated block d167c02a... #975996... 24 trxs, lib: 975958 batgirl generated block f905a24d... #975997... 24 trxs, lib: 975958 batgirl generated block 5dc01ddf... #975998... 23 trxs, lib: 975958 batgirl generated block b0cb5f06... #975999... 24 trxs, lib: 975958 batgirl generated block cb75ed06... #976000... 24 trxs, lib: 975958 batgirl generated block 11a2bd13... #976001... 24 trxs, lib: 975958 batgirl generated block 5b13e951... #976002... 23 trxs, lib: 975958 batgirl generated block 0de45bf3... #976003... 22 trxs, lib: 975958 batgirl generated block 66287858... #976004... 24 trxs, lib: 975958 batgirl generated block 96733ea0... #976005... 22 trxs, lib: 975958 batgirl generated block 90f284ea... #976006... 24 trxs, lib: 975958 noheart generated block 7dfa12ab... #976008... 24 trxs, lib: 975970 noheart generated block e9908e45... #976009... 24 trxs, lib: 975970 noheart generated block 83dad3bf... #976010... 24 trxs, lib: 975970 noheart generated block 159ca01a... #976011... 23 trxs, lib: 975970 _Accepted:_ 400 #### 400 transactions - multiple threads ##### Producer AMD ###### Client : AMD gargamel generated block 4a5a65c0... #987934 @ 2018-04-12T16:31:30.500 with 168 trxs, lib: 987895 gargamel generated block d425df94... #987935... 55 trxs, lib: 987895 gargamel generated block 7541a6bc... #987936... 169 trxs, lib: 987895 gargamel generated block df27c1b1... #987938... 8 trxs, lib: 987895 _Accepted:_ 400 ###### Client : Intel noheart generated block 2bf9aca9... #975512 @ 2018-04-12T14:47:58.000 with 15 trxs, lib: 975466 noheart generated block 89eb0170... #975513... 191 trxs, lib: 975466 noheart generated block 3ad8c5c4... #975514... 5 trxs, lib: 975466 robin generated block 65afde20... #975515... 165 trxs, lib: 975478 robin generated block 3050f077... #975517... 13 trxs, lib: 975478 robin generated block 064b05d6... #975519... 11 trxs, lib: 975478 _Accepted:_ 400 ##### Producer Intel ###### Client : AMD batgirl generated block 22c8e863... #989741 @ 2018-04-12T16:46:35.500 with 63 trxs, lib: 989704 batgirl generated block 7e37c62f... #989742... 144 trxs, lib: 989704 batgirl generated block ace33dd9... #989743... 73 trxs, lib: 989704 batgirl generated block 2f0b5b5b... #989744... 112 trxs, lib: 989704 batgirl generated block 40c63b87... #989745... 8 trxs, lib: 989704 _Accepted:_ 400 ###### Client : Intel robin generated block 00888535... #989918 @ 2018-04-12T16:48:04.000 with 7 trxs, lib: 989872 robin generated block 0dac6abf... #989919... 197 trxs, lib: 989872 eosio generated block 55c4c6bf... #989921... 189 trxs, lib: 989884 eosio generated block ca1684f6... #989923... 7 trxs, lib: 989884 _Accepted:_ 400 #### 4) [Two full nodes, one balancer and one hidden producer](#two-full-nodes-one-balancer-one-hidden-producer) The client run cleos which connects to the load balancer and the connection is then redirected to one of the full nodes using the round robin algorithm. #### 400 transactions - single thread ##### Producer Intel ###### Client : AMD robin generated block 3e54884c... #1009891 @ 2018-04-12T19:34:52.500 with 5 trxs, lib: 1009844 robin generated block b327b7fe... #1009892... 31 trxs, lib: 1009844 eosio generated block b6a13e6c... #1009894... 29 trxs, lib: 1009856 eosio generated block d2b78eea... #1009895... 29 trxs, lib: 1009856 eosio generated block 2780d072... #1009898... 29 trxs, lib: 1009856 eosio generated block 9890a2e0... #1009899... 31 trxs, lib: 1009856 eosio generated block 525b7610... #1009902... 29 trxs, lib: 1009856 eosio generated block 83e6b94b... #1009903... 31 trxs, lib: 1009856 storm generated block 9dc05f72... #1009905... 6 trxs, lib: 1009868 _Accepted:_ 220 ###### Client : Intel robin generated block 4dbd01f5... #1009162 @ 2018-04-12T19:28:48.000 with 23 trxs, lib: 1009124 robin generated block 51b1642e... #1009162... 23 trxs, lib: 1009124 robin generated block 43074497... #1009164... 23 trxs, lib: 1009124 robin generated block 76b37ce9... #1009165... 24 trxs, lib: 1009124 robin generated block f82ec33c... #1009166... 22 trxs, lib: 1009124 robin generated block 47efb29e... #1009167... 23 trxs, lib: 1009124 robin generated block 26e96977... #1009168... 22 trxs, lib: 1009124 robin generated block dc71cfdb... #1009169... 24 trxs, lib: 1009124 robin generated block d17920bd... #1009170... 23 trxs, lib: 1009124 robin generated block ecc93fc3... #1009171... 23 trxs, lib: 1009124 robin generated block 95cff703... #1009172... 23 trxs, lib: 1009124 eosio generated block 564990e9... #1009173... 22 trxs, lib: 1009136 eosio generated block f71e4cba... #1009174... 24 trxs, lib: 1009136 eosio generated block 5ecaf0cb... #1009175... 22 trxs, lib: 1009136 eosio generated block 51dda329... #1009176... 23 trxs, lib: 1009136 eosio generated block 2faa2d7e... #1009177... 23 trxs, lib: 1009136 eosio generated block 3b3c968d... #1009178... 22 trxs, lib: 1009136 eosio generated block 9f436a6c... #1009179... 11 trxs, lib: 1009136 _Accepted:_ 400 #### 400 transactions - multiple threads ##### Producer Intel ###### Client : AMD gargamel generated block 76088e6e... #1011221 @ 2018-04-12T19:45:57.500 with 23 trxs, lib: 1011176 gargamel generated block 22c47c4c... #1011222... 29 trxs, lib: 1011176 gargamel generated block 270fcab9... #1011224... 29 trxs, lib: 1011176 batgirl generated block 28294504... #1011225... 29 trxs, lib: 1011188 batgirl generated block 1e07003b... #1011227... 29 trxs, lib: 1011188 batgirl generated block 9979bfe0... #1011229... 28 trxs, lib: 1011188 batgirl generated block 82a924cc... #1011232... 29 trxs, lib: 1011188 batgirl generated block ad943338... #1011233... 28 trxs, lib: 1011188 batgirl generated block fb482b21... #1011234... 26 trxs, lib: 1011188 _Accepted:_ 250 ###### Client : Intel noheart generated block 865c19c4... #1010166 @ 2018-04-12T19:37:10.000 with 38 trxs, lib: 1010120 noheart generated block 16f541d6... #1010167... 266 trxs, lib: 1010120 robin generated block f0d55775... #1010169... 96 trxs, lib: 1010132 _Accepted:_ 400 #### 5) [Six nodeos on two full nodes, one balancer and one hidden producer](#six-nodes-one-balancer-one-hidden-producer) The client run cleos which connects to the load balancer and the connection is then redirected to one of the full nodes using the round robin algorithm. #### 400 transactions - single thread ##### Producer AMD ###### Client : AMD _Note that the machine connects to the load balancer instead of "localhost" as with the other AMD - AMD tests._ batgirl generated block c86380e6... #440608 @ 2018-04-09T09:30:23.500 with 13 trxs, lib: 440571 batgirl generated block 9e3be2ea... #440610... 28 trxs, lib: 440571 batgirl generated block 2a7839d6... #440612... 29 trxs, lib: 440571 batgirl generated block 1fe1e4c1... #440613... 29 trxs, lib: 440571 batgirl generated block 696635e5... #440615... 29 trxs, lib: 440571 batgirl generated block a169f276... #440616... 29 trxs, lib: 440571 batgirl generated block bba16c71... #440618... 29 trxs, lib: 440571 batgirl generated block 4ac5e117... #440619... 29 trxs, lib: 440571 noheart generated block 2f7b65bf... #440620... 29 trxs, lib: 440583 noheart generated block 5ba2e2f8... #440621... 29 trxs, lib: 440583 noheart generated block ae11e657... #440622... 4 trxs, lib: 440583 _Accepted: 277_ ###### Client : Intel eosio generated block 48f9e16d... #440935 @ 2018-04-09T09:33:07.000 with 18 trxs, lib: 440895 eosio generated block d92bc787... #440936... 23 trxs, lib: 440895 eosio generated block 6115797e... #440937... 24 trxs, lib: 440895 eosio generated block 8e01a0e7... #440938... 23 trxs, lib: 440895 eosio generated block 3c7c8dc1... #440939... 22 trxs, lib: 440895 eosio generated block 20d96484... #440940... 23 trxs, lib: 440895 eosio generated block ede12df7... #440941... 23 trxs, lib: 440895 eosio generated block 9c246a10... #440942... 23 trxs, lib: 440895 eosio generated block 96cc48aa... #440943... 23 trxs, lib: 440895 storm generated block 0722e343... #440944... 23 trxs, lib: 440907 storm generated block 0d234a47... #440945... 23 trxs, lib: 440907 storm generated block 11313d91... #440946... 23 trxs, lib: 440907 storm generated block 81c2ec3e... #440947... 23 trxs, lib: 440907 storm generated block fd9f6558... #440948... 23 trxs, lib: 440907 storm generated block 03ad3c89... #440949... 23 trxs, lib: 440907 storm generated block e1970063... #440950... 23 trxs, lib: 440907 storm generated block 5c4bcda4... #440951... 23 trxs, lib: 440907 storm generated block 7cdc16c8... #440952... 14 trxs, lib: 440907 _Accepted: 400_: ### 400 transactions - multiple threads ##### Producer Intel ###### Client : Intel gargamel generated block 93595d2e... #1076472 @ 2018-04-13T04:50:07.000 with 164 trxs, lib: 1076432 gargamel generated block 5a1b8b9e... #1076473... 171 trxs, lib: 1076432 gargamel generated block aef2734b... #1076474... 65 trxs, lib: 1076432 _Accepted: 400_ ###### Increasing about of transactions to 800 noheart generated block 5ab7f541... #1077963 @ 2018-04-13T05:02:54.000 with 99 trxs, lib: 1077925 noheart generated block 27d98fa3... #1077964... 267 trxs, lib: 1077925 noheart generated block 51ffd3b9... #1077965... 65 trxs, lib: 1077925 noheart generated block a151fc51... #1077966... 301 trxs, lib: 1077925 noheart generated block 15ed9bf6... #1077968... 9 trxs, lib: 1077925 noheart generated block aa95cbc1... #1077970... 59 trxs, lib: 1077925 _Accepted transactions_: 800 ###### Increasing about of transactions to 1200 batgirl generated block 75b7305d... #1078526 @ 2018-04-13T05:07:35.500 with 62 trxs, lib: 1078489 batgirl generated block 4f9507b5... #1078527... 248 trxs, lib: 1078489 batgirl generated block 2050f7bb... #1078528... 129 trxs, lib: 1078489 batgirl generated block 8ed44119... #1078529... 287 trxs, lib: 1078489 batgirl generated block 118f12fc... #1078530... 253 trxs, lib: 1078489 batgirl generated block a74369b3... #1078531... 91 trxs, lib: 1078489 batgirl generated block 813a3d6d... #1078532... 1 trxs, lib: 1078489 batgirl generated block 7b145dcb... #1078533... 2 trxs, lib: 1078489 noheart generated block c3f2c546... #1078547... 34 trxs, lib: 1078501 robin generated block 7c3edb6d... #1078561... 7 trxs, lib: 1078513 eosio generated block e6740849... #1078562... 12 trxs, lib: 1078525 eosio generated block 27829485... #1078570... 1 trxs, lib: 1078525 storm generated block 1ee016b4... #1078576... 4 trxs, lib: 1078537 gargamel generated block df538f17... #1078592... 27 trxs, lib: 1078549 gargamel generated block a61bcaaf... #1078594... 3 trxs, lib: 1078549 _Accepted transactions_: 1161 Got some messages like this (which I believe accounts for the 39 missing transactions) Please increase the expiration time of your transaction! Error Details: Transaction is expired, now is 2018-04-13T05:32:24.500, expiration is ${trx.exp} Error 3030022: Expired Transaction #### 6) Bonus ### 600 transactions - multiple threads Here I ran both the producing nodeos and three nodes accepting incoming transactions on the AMD machine. All nodes connected to each other and the client connected through the balancer. ``` eosio generated block c5460277... #441227 @ 2018-04-09T09:35:33.000 with 29 trxs, lib: 441183 eosio generated block d0f96953... #441228... 31 trxs, lib: 441183 eosio generated block 66358b58... #441231... 29 trxs, lib: 441183 storm generated block 82329467... #443403... 284 trxs, lib: 443355 gargamel generated block 3d505b1e... #443404... 114 trxs, lib: 443367 ``` _Sum: 487_ The producer forked off and two of the full nodes got stuck with different errors: ![Bad file descriptor](http://ctestnet.eosio.se/steem/bad_file_descriptor_spam.png) _Bad file descriptor_ ![Block not accepted](http://ctestnet.eosio.se/steem/block_not_accepted_spam.png) _Block not accepted_ The only way I found to restore the chain was to shut down all nodes except one, and then resync the other. I believe that this has been [addressed here](https://github.com/EOSIO/eos/commit/d4146a883d67731712d10d6a0713d8589c4a6587) and hopefully it is no longer a problem in master. We did however find some other issues when running on master in the Jungle testnet, so I will have to wait to re-test this. ## Conclusions _It is way too early to draw any conclusions from these tests. Especially regarding what kind of load an eosio blockchain will be able to handle. This article has not even started to scratch the surface. What I have done here cannot be seen as anything more than extremely limited testing. However, I will not let that stop me from speculating:_ It seems that my AMD machine might be responsible for the dropped transactions reported above. I do not know why - but when I exclusively use the Intel machines the chain seems to be more stable and I no longer get dropped transactions that I cannot account for. I will try to find time to re-install that machine and see if I can get it to work better with a more clean setup. It does seem to work when I run _AMD - AMD_, meaning it could possibly be something with the network interface. Once I started playing with the different _timeout_ settings in the balancer I could increase the amount of transactions put into the blocks quite heavily. This needs to be more thoroughly investigated and I will try to prioritize it as we move forward. The following screenshot was taken while sending a few hundred tx/s to the four instances of nodeos running on that machine. ![Glances](http://ctestnet.eosio.se/steem/glances.png) Before running these tests I thought that the limiting factor on how many transactions we could fit into each block would be the single core performance of our producing machines. This might be true at a later stage, but not exactly in the way I have seen it before. To get to that point seems to require a more advanced setup than I had previously assumed. Each individual instance of _nodeos_ acting as a full node and accepting incoming transactions is quite easy to spam down, whereby it stops accepting transactions for a few milliseconds. It seems that a possible work-around here is to run multiple instances of _nodeos_ on our full nodes, and balance the load between them. I will test this hypothesis with Charles. # What's next? More than anything, I hope this post will be seen as an invitation to come join the community driven testnets. Please pick a testnet and help us test both performance and stability, as well as our setups. Some suggestions: #### Running in multiple locations The blockchain doesn't care if a block producer is replaced by another. But to us as individual producers it is very important that we make sure our setups are as safe and secure as we can make them. Producers looking to run on bare metal will probably want to run producing nodes at multiple physical locations to decentralize within our own operations. However, we can only allow one producing machine to produce each block. Currently within the eosio software there is no functionality to handle this. ##### Handover functionality We are in discussions with block.one regarding them helping the community to build some sort of fail-over mechanism. There are multiple ways of doing this. If we haven't either received something or built something ourselves by mid May: I am planning on putting together something simple to help us at least automate the process a bit. ### Removing the cap Error 3020004: block exhausted allowed resources Error Details: Block has insufficient cpu resources When pushing transactions it is very easy to hit the a cpu limit. We believe this can be configured by setting the [max_block_cpu_usage](https://github.com/EOSIO/eos/blob/master/contracts/eosio.system/eosio.system.abi#L100) on the eosio_parameters type on the eosio.system contract. We have not found time to investigate this further. Instead I started spamming transactions from multiple users. ### DDoS protection Charles (@ankh2054) and I are planning to start looking at different solutions to help protect BPs from incoming DDoS attacks. Stay tuned for that article. ### Virtualization How big of an impact (if any) on performance does running both clients and servers as virtual machines have? I'm planning on finding time to start testing this sometime during the coming weeks. If you have some ideas or want to help, please get in contact with me. ### Docker Bohdan of @cryptolions will look into testing performance and load using a dockerized setup. ### Keep track of resources It would be highly appreciated if someone could help run more advanced performance tests and also keep a close eye on the load on the servers and clients involved. I have some ideas that could possibly help if someone wants to run these tests but don't know how; feel free to contact me on telegram if you need input. # Links Community Testnet (https://t.me/CommunityTestnet) Jungle testnet (https://t.me/jungletestnet) Scholar (https://t.me/ScholarTestnet) Arrowhead (https://t.me/joinchat/GjJY7BJrnbIW_S67Zchexg)