Mar 16 11:11:13 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:13.828 WARNING processor_handlers] Max occupancy was not provided by transaction processor: beb33070a255d370296a9cedadede205704beab7428248425e776c9b9f11814dfaab2981e2d449f02b48212505634bc3b28c0f5236eb3f2a1f10d7e646540200. Using default max occupancy: 10 Mar 16 11:11:13 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:13.829 INFO processor_handlers] registered transaction processor: connection_id=beb33070a255d370296a9cedadede205704beab7428248425e776c9b9f11814dfaab2981e2d449f02b48212505634bc3b28c0f5236eb3f2a1f10d7e646540200, family=svg-tp, version=1.0, namespaces=['6c6e09'], max_occupancy=10 Mar 16 11:11:13 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:13.830 WARNING processor_handlers] Max occupancy was not provided by transaction processor: beb33070a255d370296a9cedadede205704beab7428248425e776c9b9f11814dfaab2981e2d449f02b48212505634bc3b28c0f5236eb3f2a1f10d7e646540200. Using default max occupancy: 10 Mar 16 11:11:13 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:13.830 INFO processor_handlers] registered transaction processor: connection_id=beb33070a255d370296a9cedadede205704beab7428248425e776c9b9f11814dfaab2981e2d449f02b48212505634bc3b28c0f5236eb3f2a1f10d7e646540200, family=svg-exchange-tp, version=1.0, namespaces=['36f4f9'], max_occupancy=10 Mar 16 11:11:13 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:13.868 INFO (unknown file)] [src/journal/block_validator.rs: 265] Block 2fd4f35274e2bb481cc75920fe683bb0e0f69b8d25878e7053382f2e4576966851882ca535de7781c602bae13da89002ac37ede10dfbc6889628c79a0c4a2ea5 passed validation Mar 16 11:11:13 svg-node pbft-engine[11243]: INFO | pbft_engine::node:48 | (V(817), view 817, seq 288): Updated to view 817 Mar 16 11:11:13 svg-node pbft-engine[11243]: INFO | pbft_engine::node:39 | (PP, view 817, seq 288): Received f + 1 ViewChange messages; starting early view change Mar 16 11:11:13 svg-node pbft-engine[11243]: INFO | pbft_engine::node:16 | (PP, view 817, seq 288): Starting change to view 818 Mar 16 11:11:13 svg-node pbft-engine[11243]: thread '' panicked at 'Unable to route reply: "SendError(..)"', src/libcore/result.rs:1188:5 Mar 16 11:11:13 svg-node pbft-engine[11243]: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace. Mar 16 11:11:13 svg-node pbft-engine[11243]: DEBUG | zmq:489 | socket dropped Mar 16 11:11:13 svg-node pbft-engine[11243]: DEBUG | zmq:489 | socket dropped Mar 16 11:11:13 svg-node pbft-engine[11243]: INFO | pbft_engine::node:48 | (V(818), view 818, seq 288): Updated to view 818 Mar 16 11:11:13 svg-node pbft-engine[11243]: INFO | pbft_engine::node:39 | (PP, view 818, seq 288): Received f + 1 ViewChange messages; starting early view change Mar 16 11:11:13 svg-node pbft-engine[11243]: INFO | pbft_engine::node:16 | (PP, view 818, seq 288): Starting change to view 819 Mar 16 11:11:13 svg-node pbft-engine[11243]: thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "PoisonError { inner: .. }"', src/libcore/result.rs:1188:5 Mar 16 11:11:13 svg-node systemd[1]: sawtooth-pbft-engine.service: Main process exited, code=exited, status=101/n/a Mar 16 11:11:13 svg-node systemd[1]: sawtooth-pbft-engine.service: Failed with result 'exit-code'. Mar 16 11:11:14 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:14.065 INFO (unknown file)] [src/journal/block_validator.rs: 265] Block 63d63a916942385ae2ea6cd44c1f696e34bf0d07fdcd6424f5c89b0a93b5288852a4a407ba51f3d480f03ca4820693fac47d4c366ceaac2548a201b0005b07b6 passed validation Mar 16 11:11:14 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:14.176 INFO (unknown file)] [src/journal/block_validator.rs: 265] Block ac3f210bfebd3c1c9bab12b38d0241ffbceff0d32d9656e64d8f231f46e3843f58f4b0b128b2d535ce3950c8db144f175d4eaa9836ea919b70d450a2d2da421b passed validation Mar 16 11:11:14 svg-node systemd[1]: sawtooth-pbft-engine.service: Service hold-off time over, scheduling restart. Mar 16 11:11:14 svg-node systemd[1]: sawtooth-pbft-engine.service: Scheduled restart job, restart counter is at 1. Mar 16 11:11:14 svg-node systemd[1]: Stopped Sawtooth PBFT Consensus Engine. Mar 16 11:11:14 svg-node systemd[1]: Started Sawtooth PBFT Consensus Engine. Mar 16 11:11:14 svg-node pbft-engine[15074]: INFO | pbft_engine:88 | Sawtooth PBFT Engine (1.0.2) Mar 16 11:11:14 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:14.210 INFO handlers] Consensus engine registered: pbft 1.0 (additional protocols: []) Mar 16 11:11:14 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:14.211 INFO proxy] Consensus engine activated: pbft 1.0 Mar 16 11:11:17 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:17.351 DEBUG interconnect] No response from dfb8573fb13467ec47d0e3a044a4beb32e133172415e070d64d64131756d8561eb4f1ec7a2772feed269063da6b2250bd34083b66a679702718b5d1ed85e7be2 in 10.006935358047485 seconds - beginning heartbeat pings. Mar 16 11:11:17 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:17.353 DEBUG interconnect] No response from 3e24be64abc7a95a8b89cb1b675985fd31fb524fd7a8780da18195325c39ae995a1ed7b07713940ecf3df0cc6dc79c2d9bf2944e74a93f4ec8e42d133030ff58 in 10.007160186767578 seconds - beginning heartbeat pings. Mar 16 11:11:17 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:17.354 DEBUG interconnect] No response from 41d6ac689a9f14f0e0591600059805b2905c440b51cce183fc1b8c10ff63f259b86d1cadb34da9d6abdcb7951bdb5142160429b530e21c57abd28b973d799f8d in 10.006773948669434 seconds - beginning heartbeat pings. Mar 16 11:11:24 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:24.068 WARNING notifier] Consensus notification CONSENSUS_NOTIFY_BLOCK_VALID timed out Mar 16 11:11:25 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:25.887 DEBUG interconnect] No response from d3a66c2f13457c9e875fc44d6951279787dbf5ba59742bb66beb37cb7e2feb4c8f726e1da5e1ae3e7954f4270b847d80e8ded861fa7bf942b577cbb27f6fd418 in 12.013467788696289 seconds - beginning heartbeat pings. Mar 16 11:11:25 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:25.888 DEBUG interconnect] No response from be6b094718ad6e721ce618de33aa64de6096b81d85cc9b10c805b7b51eb95cf6a1a4fb375b6cf650f6feb61cc53c6901c9213447b07dba117e18be8a74e6eb81 in 11.677692174911499 seconds - beginning heartbeat pings. Mar 16 11:11:27 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:27.364 DEBUG interconnect] No response from dfb8573fb13467ec47d0e3a044a4beb32e133172415e070d64d64131756d8561eb4f1ec7a2772feed269063da6b2250bd34083b66a679702718b5d1ed85e7be2 in 10.003259897232056 seconds - beginning heartbeat pings. Mar 16 11:11:27 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:27.365 DEBUG interconnect] No response from e6f2eb73ed4a5c6b53e383d08118a16e76b9983f5bc39f65713cbf26bde4478aecc8fa7ee33795121251361c127dba2ea130fc1e1a57f65182a62eb89d936382 in 13.286112785339355 seconds - beginning heartbeat pings. Mar 16 11:11:27 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:27.366 DEBUG interconnect] No response from 3e24be64abc7a95a8b89cb1b675985fd31fb524fd7a8780da18195325c39ae995a1ed7b07713940ecf3df0cc6dc79c2d9bf2944e74a93f4ec8e42d133030ff58 in 10.005412101745605 seconds - beginning heartbeat pings. Mar 16 11:11:27 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:27.368 DEBUG interconnect] No response from 41d6ac689a9f14f0e0591600059805b2905c440b51cce183fc1b8c10ff63f259b86d1cadb34da9d6abdcb7951bdb5142160429b530e21c57abd28b973d799f8d in 10.005525350570679 seconds - beginning heartbeat pings. Mar 16 11:11:27 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:27.373 DEBUG interconnect] No response from beb33070a255d370296a9cedadede205704beab7428248425e776c9b9f11814dfaab2981e2d449f02b48212505634bc3b28c0f5236eb3f2a1f10d7e646540200 in 13.219372987747192 seconds - beginning heartbeat pings. Mar 16 11:11:34 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:34.068 WARNING notifier] Consensus notification CONSENSUS_NOTIFY_BLOCK_VALID timed out Mar 16 11:11:34 svg-node pbft-engine[15074]: INFO | pbft_engine::engine: | Startup state received from validator: StartupState { chain_head: Block(block_num: 287, block_id: [196, 242, 114, 177, 247, 123, 127, 193, 7, 161, 130, 82, 134, 26, 183, 177, 225, 228, 41, 229, 32, 37, 136, 85, 131, 176, 143, 67, 90, 141, 124, 23, 101, 231, 90, 117, 176, 18, 163, 54, 136, 163, 79, 180, 98, 109, 50, 229, 40, 116, 224, 71, 180, 184, 180, 7, 246, 84, 100, 120, 32, 165, 76, 24], previous_id: [78, 58, 105, 73, 47, 117, 214, 227, 169, 185, 67, 172, 103, 215, 209, 29, 187, 141, 49, 23, 213, 79, 125, 22, 111, 179, 216, 182, 216, 234, 187, 96, 17, 151, 91, 154, 2, 32, 98, 236, 173, 27, 241, 187, 33, 31, 38, 0, 231, 168, 191, 131, 137, 150, 88, 76, 16, 189, 210, 20, 83, 88, 77, 152], signer_id: [2, 77, 188, 14, 247, 215, 7, 76, 217, 99, 229, 156, 8, 54, 62, 105, 92, 226, 253, 110, 146, 145, 10, 236, 132, 85, 84, 118, 32, 133, 11, 27, 20], payload: 0a2f0a045365616c10a906189e022221024dbc0ef7d7074cd963e59c08363e695ce2fd6e92910aec8455547620850b1b1412404e3a69492f75d6e3a9b943ac67d7d11dbb8d3117d54f7d166fb3d8b6d8eabb6011975b9a022062ecad1bf1bb211f2600e7a8bf838996584c10bdd21453584d981ab3020a780a2102df872813d60b154e2afdb15cb991f0822e29ace534a6c2ca5fcc4626de93df4d12408cc324f4640e9c9e535403b19e95f6395122604ef364f7384ecd831307c66abeb599e2695d7cb627c5d741ca06031564441f645065dab2070d1f787603560f161a04706266742203312e302a06436f6d6d69741240561ef1892d06be080cde4aafc576e62d78d423291093857552435648b88c068c3efe15e33d60f457ba9e3918c156e518b4cc3f5e8b7337fd562c5044cfe22c961a750a310a06436f6d6d697410a906189e02222102df872813d60b154e2afdb15cb991f0822e29ace534a6c2ca5fcc4626de93df4d12404e3a69492f75d6e3a9b943ac67d7d11dbb8d3117d54f7d166fb3d8b6d8eabb6011975b9a022062ecad1bf1bb211f2600e7a8bf838996584c10bdd21453584d981ab3020a780a2103ae0ac6cdf9b1741801e47ec044bb016aca93d04b199b2f89a45b9cd363a2cf97124097d6fe02abfef6326607ca4521daa82cac970079ae08ab50339d871fbf38fda183348a82bc0ea2bf5900dd4adc9a7d0a2767ccf3012fd6cd087a0db5bde928151a04706266742203312e302a06436f6d6d69741240e4db0159db3222b88b331bb8addfca5deb852cb79984b63c40172adcb91ae0bb036e9ab7609f35b974a5c0096eb3c0a41becd4f7f7eb582d517c556f441893b81a750a310a06436f6d6d697410a906189e02222103ae0ac6cdf9b1741801e47ec044bb016aca93d04b199b2f89a45b9cd363a2cf9712404e3a69492f75d6e3a9b943ac67d7d11dbb8d3117d54f7d166fb3d8b6d8eabb6011975b9a022062ecad1bf1bb211f2600e7a8bf838996584c10bdd21453584d981ab3020a780a2102d12c821b3afceb652cab387798752fae1008e95f513c7db7937448f129543a031240e13b3f25c3f5a16e72f3a61b80f5f20e533f615ddac1b1a0b52448fc53afd08e1ad5b7c594827f47ad42b4e70397410d9ecafc55bb90652b23252fe40da264f81a04706266742203312e302a06436f6d6d69741240338783db65eff988746bc7ccdea342f85e31c25b974768b151f3372480c60255729687b68d97cac7ab40a40367aa429b7cbf1f298064bc89a0e655dbf8f79c491a750a310a06436f6d6d697410a906189e02222102d12c821b3afceb652cab387798752fae1008e95f513c7db7937448f129543a0312404e3a69492f75d6e3a9b943ac67d7d11dbb8d3117d54f7d166fb3d8b6d8eabb6011975b9a022062ecad1bf1bb211f2600e7a8bf838996584c10bdd21453584d98, summary: 4ac67ce81020bf4e9c4f514021e8da6747393f8951a2b9f348149bc97d93de69), peers: [PeerInfo { peer_id: [3, 174, 10, 198, 205, 249, 177, 116, 24, 1, 228, 126, 192, 68, 187, 1, 106, 202, 147, 208, 75, 25, 155, 47, 137, 164, 91, 156, 211, 99, 162, 207, 151] }, PeerInfo { peer_id: [2, 209, 44, 130, 27, 58, 252, 235, 101, 44, 171, 56, 119, 152, 117, 47, 174, 16, 8, 233, 95, 81, 60, 125, 183, 147, 116, 72, 241, 41, 84, 58, 3] }, PeerInfo { peer_id: [2, 223, 135, 40, 19, 214, 11, 21, 78, 42, 253, 177, 92, 185, 145, 240, 130, 46, 41, 172, 229, 52, 166, 194, 202, 95, 204, 70, 38, 222, 147, 223, 77] }], local_peer_info: PeerInfo { peer_id: [3, 1, 176, 2, 61, 245, 90, 74, 249, 87, 239, 47, 0, 84, 233, 183, 232, 13, 198, 56, 38, 152, 230, 84, 136, 97, 66, 2, 197, 68, 84, 33, 78] } } Mar 16 11:11:34 svg-node pbft-engine[15074]: DEBUG | pbft_engine::config: | Getting on-chain settings for config Mar 16 11:11:34 svg-node pbft-engine[15074]: INFO | pbft_engine::engine: | PBFT config loaded: PbftConfig { members: [[3, 174, 10, 198, 205, 249, 177, 116, 24, 1, 228, 126, 192, 68, 187, 1, 106, 202, 147, 208, 75, 25, 155, 47, 137, 164, 91, 156, 211, 99, 162, 207, 151], [3, 1, 176, 2, 61, 245, 90, 74, 249, 87, 239, 47, 0, 84, 233, 183, 232, 13, 198, 56, 38, 152, 230, 84, 136, 97, 66, 2, 197, 68, 84, 33, 78], [2, 77, 188, 14, 247, 215, 7, 76, 217, 99, 229, 156, 8, 54, 62, 105, 92, 226, 253, 110, 146, 145, 10, 236, 132, 85, 84, 118, 32, 133, 11, 27, 20], [2, 223, 135, 40, 19, 214, 11, 21, 78, 42, 253, 177, 92, 185, 145, 240, 130, 46, 41, 172, 229, 52, 166, 194, 202, 95, 204, 70, 38, 222, 147, 223, 77], [2, 209, 44, 130, 27, 58, 252, 235, 101, 44, 171, 56, 119, 152, 117, 47, 174, 16, 8, 233, 95, 81, 60, 125, 183, 147, 116, 72, 241, 41, 84, 58, 3]], block_publishing_delay: 2s, update_recv_timeout: 10ms, exponential_retry_base: 100ms, exponential_retry_max: 60s, idle_timeout: 30s, commit_timeout: 10s, view_change_duration: 5s, forced_view_change_interval: 50, max_log_size: 10000, storage_location: "memory" } Mar 16 11:11:34 svg-node pbft-engine[15074]: INFO | pbft_engine::engine: | PBFT state created: (PP, view 0, seq 288) Mar 16 11:11:34 svg-node pbft-engine[15074]: INFO | pbft_engine::node:77 | Updated view to 809 on startup Mar 16 11:11:34 svg-node pbft-engine[15074]: INFO | pbft_engine::node:39 | (PP, view 809, seq 288): Received f + 1 ViewChange messages; starting early view change Mar 16 11:11:34 svg-node pbft-engine[15074]: INFO | pbft_engine::node:16 | (PP, view 809, seq 288): Starting change to view 836 Mar 16 11:11:34 svg-node pbft-engine[15074]: INFO | pbft_engine::node:48 | (V(836), view 836, seq 288 *): Updated to view 836 Mar 16 11:11:34 svg-node pbft-engine[15074]: DEBUG | pbft_engine::node:37 | Ignoring stale view change message for view 836 Mar 16 11:11:36 svg-node pbft-engine[15074]: ERROR | pbft_engine::engine: | InternalError: Couldn't find 2f commit messages in the message log for building a seal Mar 16 11:11:37 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:37.378 DEBUG interconnect] No response from 3e24be64abc7a95a8b89cb1b675985fd31fb524fd7a8780da18195325c39ae995a1ed7b07713940ecf3df0cc6dc79c2d9bf2944e74a93f4ec8e42d133030ff58 in 10.000837087631226 seconds - beginning heartbeat pings. Mar 16 11:11:37 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:37.380 DEBUG interconnect] No response from 41d6ac689a9f14f0e0591600059805b2905c440b51cce183fc1b8c10ff63f259b86d1cadb34da9d6abdcb7951bdb5142160429b530e21c57abd28b973d799f8d in 10.000658988952637 seconds - beginning heartbeat pings. Mar 16 11:11:38 svg-node pbft-engine[15074]: ERROR | pbft_engine::engine: | InternalError: Couldn't find 2f commit messages in the message log for building a seal Mar 16 11:11:40 svg-node pbft-engine[15074]: ERROR | pbft_engine::engine: | InternalError: Couldn't find 2f commit messages in the message log for building a seal Mar 16 11:11:42 svg-node pbft-engine[15074]: ERROR | pbft_engine::engine: | InternalError: Couldn't find 2f commit messages in the message log for building a seal Mar 16 11:11:44 svg-node pbft-engine[15074]: ERROR | pbft_engine::engine: | InternalError: Couldn't find 2f commit messages in the message log for building a seal Mar 16 11:11:46 svg-node pbft-engine[15074]: ERROR | pbft_engine::engine: | InternalError: Couldn't find 2f commit messages in the message log for building a seal Mar 16 11:11:47 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:47.391 DEBUG interconnect] No response from e6f2eb73ed4a5c6b53e383d08118a16e76b9983f5bc39f65713cbf26bde4478aecc8fa7ee33795121251361c127dba2ea130fc1e1a57f65182a62eb89d936382 in 13.242107391357422 seconds - beginning heartbeat pings. Mar 16 11:11:47 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:47.395 DEBUG interconnect] No response from 3e24be64abc7a95a8b89cb1b675985fd31fb524fd7a8780da18195325c39ae995a1ed7b07713940ecf3df0cc6dc79c2d9bf2944e74a93f4ec8e42d133030ff58 in 10.002259492874146 seconds - beginning heartbeat pings. Mar 16 11:11:47 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:47.396 DEBUG interconnect] No response from 41d6ac689a9f14f0e0591600059805b2905c440b51cce183fc1b8c10ff63f259b86d1cadb34da9d6abdcb7951bdb5142160429b530e21c57abd28b973d799f8d in 10.00757622718811 seconds - beginning heartbeat pings. Mar 16 11:11:47 svg-node sawtooth-validator[11225]: [2020-03-16 11:11:47.399 DEBUG interconnect] No response from beb33070a255d370296a9cedadede205704beab7428248425e776c9b9f11814dfaab2981e2d449f02b48212505634bc3b28c0f5236eb3f2a1f10d7e646540200 in 13.002572536468506 seconds - beginning heartbeat pings. Mar 16 11:11:48 svg-node pbft-engine[15074]: ERROR | pbft_engine::engine: | InternalError: Couldn't find 2f commit messages in the message log for building a seal Mar 16 11:11:50 svg-node pbft-engine[15074]: ERROR | pbft_engine::engine: | InternalError: Couldn't find 2f commit messages in the message log for building a seal Mar 16 11:11:52 svg-node pbft-engine[15074]: ERROR | pbft_engine::engine: | InternalError: Couldn't find 2f commit messages in the message log for building a seal Mar 16 11:11:54 svg-node pbft-engine[15074]: ERROR | pbft_engine::engine: | InternalError: Couldn't find 2f commit messages in the message log for building a seal