Running full node with filter-on


#1

Hello EOS community!

I hope this is the right place to ask this question.

My company wanted to join EOS network. Our requirements are that we need to have api_history_plugin be able to answer queries about historical actions of arbitrary accounts. I realized that for that I need to have “filter-on = eosio.token:transfer:” in my config.ini. (if I am wrong, please advise). We need to be able to answer those queries ourselves, connecting to some already configured node or API service is not an option.
Realizing that it is a bit computation heavy, I dedicated one largish Digital Ocean box just for this purpose - 16GB Ram and 6CPUs.
I used this guide to set up node natively: https://github.com/CryptoLions/EOS-MainNet. I did not change any config.ini values except IP address and filter-on
Because we dont intend to produce blocks, I did not set up block producer variables.

My issue is this - first 10milion blocks sync up quickly. But after that everything slows down, gradually. At 20 milion blocks mark, node suddenly dies. It keeps dieing. And at the restart error log shows that the dirty flag is set, so that we need to start it with --replay-blockchain --hard-replay-blockchain. But, during the replay, it always fails with errors like this:

debug 2018-12-03T21:45:23.424 thread-0 controller.cpp:893 push_scheduled_trans ] 3050003 eosio_assert_message_exception: eosio_assert_message assertion failure
assertion failure with message: refund request not found
{“s”:“refund request not found”}
thread-0 wasm_interface.cpp:934 eosio_assert
pending console output:
{“console”:""}
thread-0 apply_context.cpp:72 exec_one

So right now I am stuck. I can’t get it to even start syncing anymore, it fails at replaying. So I guess the issue is twofold:

  1. why does it get those errors while replaying?
  2. why does it keep failing (and before slow down sync a lot, from 500k blocks per hour for first few milions to like 50k. I did not measure this is a rough estimate). Is it that filter-on with such values is killing it? Should I increase some parameters to let it consume more?

What should I do?
Should I modify some parameters in config.ini to let it consume more resources? But this error looks like there is some weird bug, that state got messed up.
Should I run it in a different way?

I am very new to EOS, and I would much appreciate any help, so that we can join the EOS ecosystem.

Thanks!