Multinode cloud setup (our node is falling behind)

Hi all,

I’ve been trying to build private network with two nodes in AWS environment.
I am using two EC2 instances, launching bootstrap-validator in the first instance and another validator in the second instance.
I followed all instructions in Benchmark a Cluster | Solana Docs, then successfully did create-vote-account.

Problem

AWS Instance 1:

  1. Set up genesis config NDEBUG=1 ./multinode-demo/setup.sh
  2. Launch bootstrap validator solana-validator --require-tower --ledger /home/ubuntu/solana/net/../config/bootstrap-validator --rpc-port 8899 --snapshot-interval-slots 200 --no-incremental-snapshots --identity /home/ubuntu/solana/net/../config/bootstrap-validator/identity.json --vote-account /home/ubuntu/solana/net/../config/bootstrap-validator/vote-account.json --rpc-faucet-address 127.0.0.1:9900 --no-poh-speed-test --no-os-network-limits-test --full-rpc-api --gossip-port 8001 --log - --public-rpc-address {FIRST INSTANCE IP ADDRESS}:8899 --no-wait-for-vote-to-start-leader --enable-rpc-transaction-history

AWS Instance 2:
Set Entrypoint using {FIRST INSTANCE IP ADDRESS} and launched solana-validator with the following configuration.

  1. Launch the second validator. solana-validator --no-poh-speed-test --no-os-network-limits-test --entrypoint {FIRST INSTANCE IP ADDRESS}:8001 --enable-rpc-transaction-history --rpc-faucet-address {SECOND INSTANCE IP ADDRESS}:9900 --identity validator/validator-keypair.json --vote-account validator/vote-account.json --ledger validator --log - --full-rpc-api --no-incremental-snapshots --require-tower --public-rpc-address {SECOND INSTANCE IP ADDRESS}:8899 --gossip-port 8001 --rpc-port 8899

Then I can check if two nodes are in network using solana gossip.

However, after successfully downloading snapshot tar file from first node once connecting to the first node, the slot of the second node remains unchanged in second node like the following. The gap between two nodes are consistently increasing and eventually it causes that stake-delegation to this validator is not possible.
solana catchup validator-keypair.json
스크린샷 2022-07-04 오후 5 33 43

How can the second validator keep fetching latest slot from the leader node(AWS instance 1)?

This is resolved. I used --gossip-host command to set up gossip address.