NEAR Stake Wars III (NEAR TestNet Guide)

In this phase of Stake Wars, validators will learn all about delegation and how to prepare technically to join MainNet.

Official links to NEAR Stake Wars III: Website, Discord, GitHub

Server Requirements for NEAR TestNet Node

Rent servers on: Hetzner | Vultr | Linode | Contabo or similar services.

How to Setup NEAR TestNet Node Full Install Guide

Rent a server

Safe to say that in first days of Stake Wars event minimal requirements was enough, but now after trying to install the node it was overloaded CPU and RAM, so best choice will be default dedicated Hetzner AX41-NVME for 34€.

Choose terminal

All actions described in this guide will be made in terminal (command line), so no any icons or desktop you are used to. Of course you can use your standard Windows or Mac terminal, but I personally prefer Termius. It’s more convenient and has a free version.

Challenge 001 – Create your Shardnet wallet & deploy the NEAR CLI

Create a wallet

Use this link https://wallet.shardnet.near.org/ and hit create account

Then choose your ID and click Reserve My Account ID

Safe to use Passphrase method and click Continue

Then hit copy, save your passphrase and click continue

Enter needed word and click verify and complete

Enter ALL 12 word you have just saved and click Find My Account

Congratulation! you have just created ShardNet wallet with test tokens

Install and test NEAR CLI

Update Linux

sudo apt update && sudo apt upgrade -y

Install developer tools, Node.js, and npm

curl -sL https://deb.nodesource.com/setup_18.x | sudo -E bash -  
sudo apt install build-essential nodejs
PATH="$PATH"

Check Node.js (must be v18.x.x or higher)

node -v

Check npm version (must be v8.x.x or higher)

npm -v

Install NEAR-CLI (Github Repository for NEAR CLI)

sudo npm install -g near-cli

Validator Stats Test

Environment

The environment will need to be set each time a new shell is launched to select the correct network.

Networks:

• GuildNet
• TestNet
• MainNet
• ShardNet (network being used for Stake Wars)

set ShardNet enviroment

export NEAR_ENV=shardnet

You can also run this command to set the Near TestNet Environment persistent:

echo 'export NEAR_ENV=shardnet' >> ~/.bashrc
echo 'export NEAR_ENV=shardnet' >> ~/.bash_profile
source $HOME/.bash_profile

NEAR CLI Commands Guide:

Proposals (indicates they would like to enter the validator set, in order for a proposal to be accepted it must meet the minimum seat price)

near proposals

Validators Current (shows a list of active validators in the current epoch, the number of blocks produced, number of blocks expected, and online rate. Used to monitor if a validator is having issues)

near validators current

Validators Next (shows validators whose proposal was accepted one epoch ago, and that will enter the validator set in the next epoch)

near validators next

Challenge 002 – Deploying a node / activating the node as a validator

Setup your node

Before you start, you may want to confirm that your machine has the right CPU features.

lscpu | grep -P '(?=.*avx )(?=.*sse4.2 )(?=.*cx16 )(?=.*popcnt )' > /dev/null \
  && echo "Supported" \
  || echo "Not supported"

Must show “Supported” otherwise you are using wrong server, rent a suitable one.

Install developer tools:

sudo apt install -y git binutils-dev libcurl4-openssl-dev zlib1g-dev libdw-dev libiberty-dev cmake gcc g++ python3 docker.io protobuf-compiler libssl-dev pkg-config clang llvm cargo

If you have problems with installing python or docker.io on Ubuntu, try this command:

sudo apt install python3
sudo apt install docker-ce

Install Python pip:

sudo apt install python3-pip

Set the configuration:

USER_BASE_BIN=$(python3 -m site --user-base)/bin
export PATH="$USER_BASE_BIN:$PATH"

Install Building env

sudo apt install clang build-essential make

Install Rust & Cargo

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

You will see the following:

Press 1 and after press Enter

Source the environment

source $HOME/.cargo/env

Clone the nearcore repository from GitHub

git clone https://github.com/near/nearcore
cd nearcore
git fetch

Checkout to the commit in this file and insert it below replacing <commit>

git checkout <commit>

Compile nearcore binary. In the nearcore folder run the following commands:

cargo build -p neard --release --features shardnet

The binary path is target/release/neard. If you are seeing issues, it is possible that cargo command is not found. Compiling nearcore binary may take a little while.

Initialize working directory. In order to work properly, the NEAR node requires a working directory and a couple of configuration files. Generate the initial required working directory by running:

./target/release/neard --home ~/.near init --chain-id shardnet --download-genesis

This command will create the directory structure and will generate config.json, node_key.json, and genesis.json on the network you have passed.

Replace the config.json

rm ~/.near/config.json
wget -O ~/.near/config.json https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/shardnet/config.json

Run the node

cd ~/nearcore
./target/release/neard --home ~/.near run

The node is now running you can see log outputs in your console. Your node should be find peers, download headers to 100%, and then download blocks. It could take a while, just wait until 100% sync.

Activating the node as validator

Authorize Wallet Locally. A full access key needs to be installed locally to be able to sign transactions via NEAR-CLI.

near login

Copy the link you see in terminal in your browser

Connect your wallet

Grant Access to Near CLI

After Grant, you will see a page like this, it’s ok, don’t worry, go back to console

Enter your wallet and press Enter

Check the validator_key.json

cat ~/.near/validator_key.json

If a validator_key.json is not present, follow these steps to create one

Create a validator_key.json. <pool_id> —> xx.factory.shardnet.near WHERE xx is you pool name

near generate-key <pool_id>

Copy the file generated to shardnet folder: Make sure to replace <pool_id> by your pool name

cp ~/.near-credentials/shardnet/YOUR_WALLET.json ~/.near/validator_key.json

Edit “account_id” => xx.factory.shardnet.near, where xx is your PoolName. Change private_key to secret_key

account_id must match the staking pool contract name or you will not be able to sign blocks

{
  "account_id": "xx.factory.shardnet.near",
  "public_key": "ed25519:HeaBJ3xLgvZacQWmEctTeUqyfSU4SDEnEwckWxd92W2G",
  "secret_key": "ed25519:****"
}

Start the validator node

target/release/neard run

Setup Systemd

sudo vi /etc/systemd/system/neard.service

Paste. Change USER to your paths

[Unit]
Description=NEARd Daemon Service

[Service]
Type=simple
User=<USER>
#Group=near
WorkingDirectory=/home/<USER>/.near
ExecStart=/home/<USER>/nearcore/target/release/neard run
Restart=on-failure
RestartSec=30
KillSignal=SIGINT
TimeoutStopSec=45
KillMode=mixed

[Install]
WantedBy=multi-user.target

Next

sudo systemctl enable neard

sudo systemctl start neard

f you need to make a change to service because of an error in the file. It has to be reloaded:

sudo systemctl reload neard

Watch logs

journalctl -n 100 -f -u neard

Make log output in pretty print

sudo apt install ccze

View Logs with color

journalctl -n 100 -f -u neard | ccze -A

Becoming a Validator

In order to become a validator and enter the validator set, a minimum set of success criteria must be met.

• The node must be fully synced
• The validator_key.json must be in place
• The contract must be initialised with the public_key in validator_key.json
• The account_id must be set to the staking pool contract id
• There must be enough delegations to meet the minimum seat price. See the seat price here.
• A proposal must be submitted by pinging the contract
• Once a proposal is accepted a validator must wait 2-3 epoch to enter the validator set
• Once in the validator set the validator must produce great than 90% of assigned blocks

Check running status of validator node. If “Validator” is showing up, your pool is selected in the current validators list.

Challenge 003 – Deploy staking pool for your validator. Delegate and stake NEAR.

Mounting a staking pool

You need to replace:

• Pool Name: Staking pool name, the factory automatically adds its name to this parameter, creating {pool_name}.{staking_pool_factory} Examples:
• If pool id is stakewars will create : stakewars.factory.shardnet.near
• Owner ID: The SHARDNET account (i.e. stakewares.shardnet.near) that will manage the staking pool.
• Public Key: The public key in your validator_key.json file.
• 5: The fee the pool will charge (e.g. in this case 5 over 100 is 5% of fees).
• Account Id: The SHARDNET account deploying and signing the mount tx. Usually the same as the Owner ID.

Be sure to have at least 30 NEAR available, it is the minimum required for storage. Example : near call stake_wars_validator.factory.shardnet.near –amount 30 –accountId stakewars.shardnet.near –gas=300000000000000

near call factory.shardnet.near create_staking_pool '{"staking_pool_id": "<pool name>", "owner_id": "<accountId>", "stake_public_key": "<public key>", "reward_fee_fraction": {"numerator": 5, "denominator": 100}, "code_hash":"DD428g9eqLL8fWUxv8QSpVFzyHi1Qd16P8ephYCTmMSZ"}' --accountId="<accountId>" --amount=30 --gas=300000000000000

To change the pool parameters, such as changing the amount of commission charged to 1% in the example below, use this command:

near call <pool_id> update_reward_fee_fraction '{"reward_fee_fraction": {"numerator": 1, "denominator": 100}}' --accountId <account_id> --gas=300000000000000

Check your pool is now visible on https://explorer.shardnet.near.org/nodes/validators

Transactions Guide

Deposit and Stake NEAR

near call <pool_id> deposit_and_stake --amount <amount> --accountId <accountId> --gas=300000000000000

Unstake NEAR (Amount in yoctoNEAR)

near call <pool_id> unstake '{"amount": "<amount yoctoNEAR>"}' --accountId <accountId> --gas=300000000000000

Unstake ALL

near call <pool_id> unstake_all --accountId <accountId> --gas=300000000000000

Withdraw (Unstaking takes 2-3 epochs to complete, after that period you can withdraw in YoctoNEAR from pool.)

near call <pool_id> withdraw '{"amount": "<amount yoctoNEAR>"}' --accountId <accountId> --gas=300000000000000

Withdraw ALL

near call <pool_id> withdraw_all --accountId <accountId> --gas=300000000000000

Ping (ping issues a new proposal and updates the staking balances for your delegators. A ping should be issued each epoch to keep reported rewards current.)

near call <pool_id> ping '{}' --accountId <accountId> --gas=300000000000000

Balances Total

near view <pool_id> get_account_total_balance '{"account_id": "<accountId>"}'

Staked Balance

near view <pool_id> get_account_staked_balance '{"account_id": "<accountId>"}'

Unstaked Balance

near view get_account_unstaked_balance '{"account_id": ""}'

Available for Withdrawal (You can only withdraw funds from a contract if they are unlocked.)

near view <pool_id> is_account_unstaked_balance_available '{"account_id": "<accountId>"}'

Pause Staking

near call <pool_id> pause_staking '{}' --accountId <accountId>

Resume Staking

near call <pool_id> resume_staking '{}' --accountId <accountId>

Try to play with all commands to understand and test how it works

Challenge 004 – Check your Node

Monitor and make alerts

An email notification can make it more comfortable to maintain a validator up and running. Achieve to be a validator confirming transactions on testnet and get >95% of uptime.

Log Files (The log file is stored either in the ~/.nearup/logs directory or in systemd depending on your setup.)

journalctl -n 100 -f -u neard | ccze -A

Log file sample (Validator | 1 validator)

INFO stats: #85079829 H1GUabkB7TW2K2yhZqZ7G47gnpS7ESqicDMNyb9EE6tf Validator 73 validators 30 peers ⬇ 506.1kiB/s ⬆ 428.3kiB/s 1.20 bps 62.08 Tgas/s CPU: 23%, Mem: 7.4 GiB

• Validator: A “Validator” will indicate you are an active validator
• 73 validators: Total 73 validators on the network
• 30 peers: You current have 30 peers. You need at least 3 peers to reach consensus and start validating
• #46199418: block – Look to ensure blocks are moving

RPC

Any node within the network offers RPC services on port 3030 as long as the port is open in the nodes firewall. The NEAR-CLI uses RPC calls behind the scenes. Common uses for RPC are to check on validator stats, node version and to see delegator stake, although it can be used to interact with the blockchain, accounts and contracts overall. More deep commands list and how to use them in more detail here, but we need for this challenge only commands below.

Install

sudo apt install curl jq

Check your node version

curl -s http://127.0.0.1:3030/status | jq .version

Check Delegators and Stake

near view <your pool>.factory.shardnet.near get_accounts '{"from_index": 0, "limit": 10}' --accountId <accountId>.shardnet.near

Check Reason Validator Kicked

curl -s -d '{"jsonrpc": "2.0", "method": "validators", "id": "dontcare", "params": [null]}' -H 'Content-Type: application/json' 127.0.0.1:3030 | jq -c '.result.prev_epoch_kickout[] | select(.account_id | contains ("<POOL_ID>"))' | jq .reason

Check Blocks Produced / Expected

curl -r -s -d '{"jsonrpc": "2.0", "method": "validators", "id": "dontcare", "params": [null]}' -H 'Content-Type: application/json' 127.0.0.1:3030 | jq -c '.result.current_validators[] | select(.account_id | contains ("POOL_ID"))'

Challenge 005 – Make an article

Setup a running validator node for shardnet on any one of the most popular cloud providers and document the process to create an article about it.

• Amazon Web Services
• Google Cloud Platform
• Microsoft Azure
• IBM Cloud
• DigitalOcean
• Hetzner

Deliverables

Article doing a step-by-step guide on how to mount a node validator. (GitHub or Medium)

Acceptance criteria

• Document the process and publish it (GitHub or Medium)
• Include step-by-step instructions to mount a node validator using Stake Wars instructions (Challenge 001, 002, 003 and 004).
• Include detailed screenshots and descriptions on the process.
• Include pricing for running the validator.
• Article can be done in any language. A review can be done for acceptance.

Challenge 006 – Create a cron task on the machine running node validator that allows ping to network automatically

Steps

Create a new file on /home/<USER_ID>/scripts/ping.sh

#!/bin/sh
# Ping call to renew Proposal added to crontab

export NEAR_ENV=shardnet
export LOGS=/home/<USER_ID>/logs
export POOLID=<YOUR_POOL_ID>
export ACCOUNTID=<YOUR_ACCOUNT_ID>

echo "---" >> $LOGS/all.log
date >> $LOGS/all.log
near call $POOLID.factory.shardnet.near ping '{}' --accountId $ACCOUNTID.shardnet.near --gas=300000000000000 >> $LOGS/all.log
near proposals | grep $POOLID >> $LOGS/all.log
near validators current | grep $POOLID >> $LOGS/all.log
near validators next | grep $POOLID >> $LOGS/all.log

Create logs folder

mkdir $HOME/logs

Change execute permission for ping.sh file

chmod +x $HOME/scripts/ping.sh

Create a new crontab, running every 2 hours

crontab -e
0 */2 * * * sh /home/<USER_ID>/scripts/ping.sh

List crontab to see it is running

crontab -l

Review your logs

cat $HOME/logs/all.log

Acceptance criteria

Ping is done periodically to network

Optional

Use croncat to create a ping task. Cron.cat is service running over NEAR Protocol that allows you to automatize the create a ping for validators. In fact, it is the more easy way to do it in testnet and mainnet. For Stake Wars III is now available to run the ping on Shardnet.

Challenge 007 – Data Science for Staking

Description

Blockchain is big data. We believe that very interesting data insights could be found out through investigating staking data on NEAR. Please try to find out some valuable data from staking and validators and build a visualized data dashboard for everyone to access.

If you don’t have anything in mind to start with, please consider the follow examples:

• Lifecycle performance of validators. (How many blocks have they produced, did they ever get kicked out, etc..)
• How does the reward APY change from epoch to epoch, what is affecting it?
• The token flow of big delegation whales. When did they decide to stake and when did they unstake.
• Historical data of the number of delegators (and delegation amount) on each validator node.

Deliverable

• A chartio-like data dashboard, a report built with Jupyter Notebook, or a well-structured spreadsheet. You’re encouraged to use whatever data visualization tool or machine learning algorithm that you love.
• A brief summary of what we could learn from those data and how NEAR can use those data to improve staking.

Acceptance Criteria

• Please provide data insights other than what we can directly get from the NEAR explorer / NEAR wallet.
• As for the outcome, a visualized data dashboard is preferred, also a spreadsheet with well-structured data is also acceptable.
• What we can learn from the data actually matters, so please also provide a summary of how you interpret the data and how we can improve the NEAR system based on your work.
• Analytics with NEAR mainnet data is strongly recommended. You can also analyze data from testnet or shardnet, but mainnet have more real and complex transactions.

Judging Criteria

• A clear and meaningful goal is set. (What kind of data are you investigating and what kind of result are you expecting) (2 points)
• Being able to correctly and effectively grab the data from indexer, rpc-node, etc. (2 points)
• Transform and aggregate the data to form the final result. (3 points)
• A straightforward data dashboard. (3 points)

Challenge 008 – Deploy a smart contract on the owner account of a staking pool

This is a code provided by community, there is not knowledge of auditments. Do your own review of code to ensure security. Use it at your own risk.

Steps

Install cargo and Rust in case you don’t have it. This command is for Linux or MacOS.

curl --proto '=https' --tlsv1.2 https://sh.rustup.rs -sSf | sh

source $HOME/.cargo/env

Add the wasm32-unknown-unknown toolchain

rustup target add wasm32-unknown-unknown

Clone the project found here

git clone https://github.com/zavodil/near-staking-pool-owner

Compile smart contract

cd near-staking-pool-owner/contract
cargo build --target wasm32-unknown-unknown --release

Deploy smart contract on your owner account. Adjust the path to .wasm file if required

NEAR_ENV=shardnet near deploy <OWNER_ID>.shardnet.near --wasmFile target/wasm32-unknown-unknown/release/contract.wasm

Initialise the smart contract picking accounts for splitting revenue

CONTRACT_ID=<OWNER_ID>.shardnet.near

# Change numerator and denomitor to adjust the % for split.
NEAR_ENV=shardnet near call $CONTRACT_ID new '{"staking_pool_account_id": "<STAKINGPOOL_ID>.factory.shardnet.near", "owner_id":"<OWNER_ID>.shardnet.near", "reward_receivers": [["<SPLITED_ACCOUNT_ID_1>.shardnet.near", {"numerator": 3, "denominator":10}], ["<SPLITED_ACCOUNT_ID_2>.shardnet.near", {"numerator": 70, "denominator":100}]]}' --accountId $CONTRACT_ID

Wait until you start receiving rewards on your node staking pool. Do a withdraw of rewards

CONTRACT_ID=<OWNER_ID>.shardnet.near
NEAR_ENV=shardnet near call $CONTRACT_ID withdraw '{}' --accountId $CONTRACT_ID --gas 200000000000000

Acceptance criteria

• Withdraw is successfully distributed to 2 accounts.
• In case a bug is found, detailed feedback can alternately be taken on count.

Challenge 009 – Monitor and RPC

This challenge has 2 objectives:

• Monitor uptime above 70% on ShardNet
• Open the RPC port 3030 for analytics / reporting

Tip: On MainNet, Uptime has to be greater than 95%, or you will be kicked for up to 3 epochs losing rewards and possibly delegators.

Validator scoreboard for ShardNet where you can monitor your uptime and metrics of your validator: ShardNet Uptime Leaderboard

If a validator issues a ping to join the network before being fully synced it is considered an attack on the network!

Steps

Objective 1 – Monitor uptime

• Check your current uptime and manage it to above 70% on ShardNet
• Fix issues with producing chunks. See the troubleshooting section below.
• Implement monitoring scripts

Objective 2 – Open Port 3030 for Diagnostic reporting

Check to see if PORT 3030 is open

sudo iptables -L | grep 3030

Open the port if not open

sudo iptables -A INPUT -p tcp --dport 3030 -j ACCEPT

Save the config for server restarts. You can use one of the 2 solutions:

Using iptables-persistent

sudo apt install iptables-persistent

or if already installed

sudo dpkg-reconfigure iptables-persistent

Using files

iptables-save > /etc/iptables/rules.v4
ip6tables-save > /etc/iptables/rules.v6

Validate the port is open by visiting http://<YOUR IP>:3030/status

In some cases the port may also need to be opened in your cloud provider / datacenter firewall.

Acceptance criteria

• Validator reporting above 70%
• Port can be accessed publicly http://<IP Address>:3030/status

Challenge 010 – ShardNet NEAR Delegation

This challenge automatically delegates ShardNet NEAR to pools with uptime >= 60%.

Snapshot 2022-08-09: One-time 1000 ShardNet NEAR delegation to validators with >= 60% “% Chunks Online” for ALL TIME.

Starting 2022-08-10: Daily 100 ShardNet NEAR delegation to validators with >= 60% “% Chunks Online” in the last 25 EPOCHS.

Monitor uptime

• Check your current uptime and manage it to above 60% on the ShardNet Uptime Leaderboard
• Fix issues with producing chunks (See the troubleshooting guide below)
• Debug Node Issues
  • Triage issues with Missed Chunks by reviewing your report on: http://localhost:3030/debug
  • Submit PR to StakeWars Github with the reason for missing Chunks https://github.com/near/stakewars-iii/issues
• Implement monitoring scripts

Challenge 011 – Staking Farm 2.0

Description

Aurora launched the first staking farm on NEAR early this year and there’s a lot of fun. However there are still some unresolved problems associated with it, such as how to make the APY sustainable, how to make staking farm more decentralised and how one user could earn multiple staking farm rewards at the same time, etc. Please propose an improved staking farm solution which could help to solve some of these challenges.

Deliverable

• A working staking farm 2.0 smart contract
• Design documents and technical notes.

Acceptance criteria

• The solution should be based on the existing NEAR staking system.
• Either a working smart contract or a practical proposal is welcomed.
• A clear illustration of how the new proposed solution can improve existing staking farm is required.

Judging criterias

• Full understanding of the current staking farm mechanism. (1 point)
• The proposed solution has improvements in one or more dimensions, and no significant drawbacks are introduced. (4 points)
• The proposed solution is feasible based on the current NEAR ecosystem. (2 points)
• A working NEAR smart contract or a detailed tech doc is provided. (3 points)

Challenge 012 – TBA

This challenge will be updated after official update by NEAR Team. Stay tuned.

Challenge 013 – Staking Farm 2.0

Description

In this challenge, participants will learn how to update their node, migrate keys, and set up a BACKUP node.

Please note that this challenge includes multiple parts:

• Best Practices: is an informational section that outlines the optimal way of updating a validator node (no points)
• Task: in this section, participants will learn how to migrate validator keys. Only this part of the challenge will be evaluated. (25 UNP & 10 DNP)

Acceptance Criteria

• Run a backup node
• Successfully migrate validator keys from the MAIN node to the BACKUP node

Best Practices

Before updating your node, make sure that you have checked

• An official announcement for the update.
• Release Notes on GitHub.
  Be aware of DB migration or other breaking changes
• Validator chats.
  Review how many validators have been updated?
  Were there any noticeable problems?
• Update TestNet/backup node.
  Spend some time testing it on your backup node before moving to the main validator node.

It is recommended that you keep the previous versions of compiled binaries in case of an urgent downgrade

Update binary

Create a new directory, e.g sources & clone or update nearcore project from GitHub

cd sources
git clone https://github.com/near/nearcore
cd nearcore
git fetch origin --tags

Checkout to the branch you need. The latest unstable release is recommended if you are running on the TestNet or ShardNet and the latest stable version is recommended if you are running on the MainNet. Please check the releases page on GitHub

git checkout <version>

Compile nearcore binary

make release

./target/release/neard --version

Don’t forget to check the version and hash of the binary, it should be the same as displayed on releases page

Replace binary in your work directory

cp ./target/release/neard <WORK_DIR>/bin/

Restart the node and check the log file

Task

Try to use different folders for binaries & sources for version check

Backup node

In case of missed blocks or chunks, the validator can be removed from the active validation set in the next auction. If you want to secure your validator NEAR node from a high downtime, you can deploy a backup server with the preconfigured NEAR node. Having two provisioned nodes allows you to quickly switch from one server to another by migrating your validator keys, so you can continue producing blocks with minimal downtime. If the nearcore release has long database migration, or you must maintain/scale the server, we also recommend migrating your validator keys to the backup node.

Always analyze your node activities and be ready to determine a problem and move the main validator keys to the backup node.

Validator keys migration

Firstly, you need to have a backup server, a NEAR node with a new node_key.json and one more generated validator_key.json. To switch quickly, prepare folders for the validator keys and reserve keys

Be careful! Don’t let two nodes run with the same keys! This will make your node slashed. This means that your node will be excluded from the active validation set and your funds may be burned.

If the server is unavailable (e.g. no network connection) deal with the hosting provider, get access and only then start the key migration procedure! Always make sure that the MAIN node doesn’t work

If the server is available

• Stop the MAIN node
• Make sure the MAIN node isn’t working

ps aux | grep neard
netstat -tlpn | grep neard

Replace main validator keys on the MAIN node with the reserve ones (optional, if you are sure the node won’t start accidentally)

cd mainnet
rm validator_key.json node_key.json
cp ../<RESERVE_KEYS>/validator_key.json ../<RESERVE_KEYS>/node_key.json .

• Stop the BACKUP node
• Make sure the BACKUP node isn’t working

ps aux | grep neard
netstat -tlpn | grep neard

Replace reserve keys on the BACKUP node with main validator

cd mainnet
rm validator_key.json node_key.json
cp ../<VALIDATOR_KEYS>/validator_key.json ../<VALIDATOR_KEYS>/node_key.json .

Check the validator_key.json and node_key.json

You can check file identity using md5sum command for validator_state.json and node_key.json on both nodes. If the checksums match, then the files are identical.

Or you use diff command

diff <WORK_DIR>/mainnet/validator_key.json ../<RESERVE_KEYS>/validator_key.json
diff <WORK_DIR>/mainnet/node_key.json ../<RESERVE_KEYS>/node_key.json

diff <WORK_DIR>/mainnet/validator_key.json ../<VALIDATOR_KEYS>/validator_key.json
diff <WORK_DIR>/mainnet/node_key.json ../<VALIDATOR_KEYS>/node_key.json

Start the BACKUP node

After starting the node in the log, you used to see the line with peer_id that was equal to public_key in /<WORK_DIR>/mainnet/node_key.json & ../<VALIDATOR_KEYS>/node_key.json, but now just make sure that your backup node is validating. In the logs of the BACKUP node you should see “Validator |”

2022-07-29T13:37:03.984160Z  INFO stats: #70892058 F6jTU9iyXRJ6jiQq8XUP6ENpzvHZ8FfYJpuS8w9zHb82 Validator | 100 validators 35 peers ⬇ 1.53 MB/s ⬆ 2.24 MB/s 0.80 bps 22.2 Tgas/s CPU: 39%, Mem: 6.91 GB
2022-07-29T13:37:13.984741Z  INFO stats: #70892066 2HwdtnRkosQ9mLkmKZpa3X28NNtzajfvct2YBBg63QW2 Validator | 100 validators 35 peers ⬇ 1.56 MB/s ⬆ 2.21 MB/s 0.80 bps 66.8 Tgas/s CPU: 38%, Mem: 6.91 GB

Normal logs

2022-07-29T13:44:20.905273Z  INFO stats: #70892419 8uHiGM2CuibYAyA1Sry15M9vcyyXrZD8CG3nf8nvetdU 100 validators 36 peers ⬇ 1.35 MB/s ⬆ 1.30 MB/s 0.80 bps 25.8 Tgas/s CPU: 42%, Mem: 6.01 GB
2022-07-29T13:44:30.905814Z  INFO stats: #70892427 4C7VPH79nx6auKD9JpPK81YtQmZAq35BC2JVJxNJgeWy 100 validators 36 peers ⬇ 1.36 MB/s ⬆ 1.31 MB/s 0.80 bps 22.9 Tgas/s CPU: 37%, Mem: 5.81 GB

More challenges

I’ll add more challenges as soon as they will be available… Stay tuned

Troubleshooting

My node is not producing chunks/blocks

Not producing means:

"num_expected_blocks": XXX,
"num_expected_chunks": XXX, 
"num_produced_blocks": 0,
"num_produced_chunks": 0,

Try this 2:

near view <POOL_ID> get_staking_key '{}'

cat ~/.near/validator_key.json | grep public_key

Both keys must match. If they DO NOT update the staking pool key

Update staking pool key:

near call <POOL_ID> update_staking_key '{"stake_public_key": "<PUBLIC_KEY>"}' --accountId <ACCOUNT_ID>

If match: Check validator_key.json

cat ~/.near/validator_key.json

account_id must be like : xx.factory.shardnet.near If not update it, save and stop/start neard

Explorer is not showing my transactions/not updating – not showing my account / pool

In case that explorer.shardnet.near.org is not working, review the RPC status on the corner.

OK (Green indicator) / RPC not available (Yellow indicator)

If it is in other color than green it should be passing unstable behavior. Wait until in stabilize. In case you can user near CLI to review validators information.

near proposals
near validators current
near validators next

I cannot run neard, it shows ‘Failed to open the database’

This is common because there is a current service using the files. Probably another neard service. Try running the top command on the terminal and verify there is no neard service already running.

top

In case it is already running you can verify it status using the following

journalctl -n 100 -f -u neard | ccze -A

My staking pool lost the tokens I delegate to it

In case you delegated tokens to your staking pools before one the hard forks made to the shardnet network, your account could be deleted by the fork. Just create a new account and delegate enough tokens.

near call <staking_pool_id> deposit_and_stake --amount <amount> --accountId <accountId> --gas=300000000000000

My log show: WARN telemetry: Failed to send telemetry data err=Connect(Timeout)

This is related to RPC issue, don’t worry about it. You will maybe have some timeout, you can use your localnode by adding --nodeUrl 127.0.0.1:3030

I want to move my node to other VPS/Cloud provider

Do regular steps to mount a node on the new VPS/Cloud and take in consideration this:

• Sync your NEW node without validator key
• Stop the NEW node
• Copy over node_key.json from OLD node to NEW node
• Copy over validator_key.json from OLD node to NEW node
• Stop the OLD node
• Start the NEW node

A hardfork is announced, what should I do?

On July 27th a third hardfork was done during stake wars to Shardnet. This for upgrading core code and keep nodes with higher stability.

Run the following to upgrade

sudo systemctl stop neard
rm ~/.near/data/*

cd ~/nearcore
git fetch
git checkout 68bfa84ed1455f891032434d37ccad696e91e4f5
cargo build -p neard --release --features shardnet

cd ~/.near
rm genesis.json
wget https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/shardnet/genesis.json

rm ~/.near/config.json
wget -O ~/.near/config.json https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/shardnet/config.json

sudo systemctl start neard && journalctl -n 100 -f -u neard | ccze -A

Also, if you owner account or staking pool doesn’t appear is probably that it was removed during hard fork. Make those again.

NEAR command worked before but now “no matching pair found”

Please always check if there is word “testnet” somewhere in the output of failing command. NEAR CLI is configured to use the “testnet” chain by default, so we need to always explicitly tell it to use “shardnet” by setting environment variable named “NEAR_ENV” (taken from Challenge 1)

export NEAR_ENV=shardnet

Please always check if there is word “testnet” somewhere in the output of failing command. NEAR CLI is configured to use the “testnet” chain by default, so we need to always explicitly tell it to use “shardnet” by setting environment variable named “NEAR_ENV” (taken from Challenge 1)

echo 'export NEAR_ENV=shardnet' >> ~/.bashrc
echo 'export NEAR_ENV=shardnet' >> ~/.bash_profile
source $HOME/.bash_profile

Issues with block/chuck production

Hardware and Internet meet min specs?

sudo apt install speedtest-cli && speedtest-cli https://www.vpsbenchmarks.com/

Out of RAM? Set a 16GB Swap file.

sudo rm /swapfile
sudo fallocate -l 16G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo echo "/swapfile   none    swap    sw    0   0" >> /etc/fstab
sudo swapon /swapfile```

## Uptime calculation
"% chunks online" = (CHUNKS PRODUCED / CHUNKS EXPECTED) * (VALIDATED EPOCHS / TOTAL EPOCHS)

## ***Common Node Errors and Solutions*** by Open Shards Alliance
In case none of the above worked, you can use this guide. In this document you will find a general rules on how to solve problems related to a node validator running on NEAR Protocol. 

* [Common Node Errors and Solutions](https://near-nodes.io/troubleshooting/common-errors) Guide by OSA.

## Ask community

[Join official NEAR Discord server](https://discord.gg/V3Z6CsEJ7Y), where you fill find the #stakewars channel for solving questions.This is a community fueled channel, feel free to contribute.

Useful links

Wallet: https://wallet.shardnet.near.org/

Explorer: https://explorer.shardnet.near.org/

Check ShardNet seat price: https://explorer.shardnet.near.org/nodes/validators

FAQ

Thank you for reading, hope it helped. Drop a comment if you have any questions.

Subscribe to our Telegram and join Discord for latest updates and be a part of community!

In case you want to support my work and cover some servers cost: