rjones (Thu, 23 Mar 2017 04:49:14 GMT):
Dan
Dan (Thu, 23 Mar 2017 13:09:25 GMT):
@jjason @cianx @jsmitchell @pschwarz can we pick up the consensus interface discussion here?
cianx (Thu, 23 Mar 2017 13:09:25 GMT):
Has joined the channel.
pschwarz (Thu, 23 Mar 2017 13:09:25 GMT):
Has joined the channel.
jsmitchell (Thu, 23 Mar 2017 13:09:25 GMT):
Has joined the channel.
cianx (Thu, 23 Mar 2017 13:09:53 GMT):
?
Dan (Thu, 23 Mar 2017 13:10:36 GMT):
@jjason was looking for input on passing a view or a view factory into the consenus methods yesterday on #sawtoothlake
Dan (Thu, 23 Mar 2017 13:10:59 GMT):
sec.. maybe I can link to the original posts
Dan (Thu, 23 Mar 2017 13:11:29 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=MgYXMLcHh7f4ZAmuW)
Dan (Thu, 23 Mar 2017 13:11:59 GMT):
My trailing questions...
Dan (Thu, 23 Mar 2017 13:12:21 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=g3pZzwSS8SZyJu8Kx)
Dan (Thu, 23 Mar 2017 13:12:59 GMT):
And jjason's response:
Dan (Thu, 23 Mar 2017 13:13:00 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=iyawYeZQEq3vBn4Ek)
Dan (Thu, 23 Mar 2017 13:13:00 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=g3pZzwSS8SZyJu8Kx) [ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=iyawYeZQEq3vBn4Ek)
cianx (Thu, 23 Mar 2017 13:17:28 GMT):
The consensus interface needs to pass the stateviewfactory to the BlockPublisher and BlockVerifier, so they can create the stateviews desired. That is the cleanest simplest solution. It pushes 1 line of additional code into the consensus implementations if they need a stateview but allows them to get a stateview on any block they desire.
cianx (Thu, 23 Mar 2017 13:24:51 GMT):
Unclear as to what additional questions need to be answered I favor @jjason #1 as it is cleaner and more consistent but both accomplish the goal of allowing consensus implementations to access stateviews anywhere in the chain.
boydjohnson (Thu, 23 Mar 2017 13:47:31 GMT):
Has joined the channel.
agunde (Thu, 23 Mar 2017 13:48:01 GMT):
Has joined the channel.
jjason (Thu, 23 Mar 2017 14:00:51 GMT):
I don't feel strongly enough to fight for or against either. Going with #1.
mfford (Thu, 23 Mar 2017 14:22:05 GMT):
Has joined the channel.
rjones (Thu, 23 Mar 2017 17:34:26 GMT):
Has left the channel.
mahesh_rao (Thu, 23 Mar 2017 18:48:57 GMT):
Has joined the channel.
drozd (Thu, 23 Mar 2017 19:14:51 GMT):
Has joined the channel.
kelly_ (Thu, 23 Mar 2017 19:15:33 GMT):
Has joined the channel.
cianx (Thu, 23 Mar 2017 20:56:23 GMT):
the triumph of ambivalence
pschwarz (Thu, 23 Mar 2017 21:25:57 GMT):
Chatting a bit out of band with @jsmitchell, @cianx and @Dan on the subject of the genesis block on non-genesis validators, and we arrived at the solution of introducing a GenesisConsensus module that would be the default consensus as configured from the empty state. This change requires that a genesis.batch needs some settings transactions for configuring the consensus modules for blocks 1+ - not an unreasonable ask since we're planning on making sawtooth_settings vote-only with a minimum of one approved public key, which changes `sawtooth admin genesis` to require at least one set of batches -- unlike now, where it has none.
pschwarz (Thu, 23 Mar 2017 21:26:04 GMT):
(moved from #sawtoothlake )
pschwarz (Thu, 23 Mar 2017 21:26:40 GMT):
Part of the validation that the genesis consensus could do is fail validation if used on a block with a previous block.
Dan (Thu, 23 Mar 2017 21:27:38 GMT):
@jjason does poet already handle genesis block (or I guess generally any non-poet block) when it comes to stats like aggregate local mean?
jjason (Thu, 23 Mar 2017 21:28:07 GMT):
@dan The assumption has always been that all blocks in the chain belong to a single consensus.
jsmitchell (Thu, 23 Mar 2017 21:28:41 GMT):
it feels like if this is the first poet block then it should be an initialization process, regardless of where the first poet block occurs in the block sequence
jsmitchell (Thu, 23 Mar 2017 21:28:41 GMT):
it feels like if this is the first poet block in the current sequence then it should be an initialization process, regardless of where the first poet block occurs in the block sequence
jjason (Thu, 23 Mar 2017 21:28:44 GMT):
I would have to just put code in the PoET consensus to ignore blocks where the .consensus field does not contain what we expect.
jjason (Thu, 23 Mar 2017 21:30:51 GMT):
I think that if the intent was to use poet going forward - for example, the genesis block contained the config setting `sawtooth.consensus.algorithm=poet` then that block should also contain the signup information for the first validator.
jsmitchell (Thu, 23 Mar 2017 21:31:19 GMT):
yep, but that still represents a transition from the empty state to the state containing those values
jsmitchell (Thu, 23 Mar 2017 21:31:25 GMT):
which is the application of the genesis block
jsmitchell (Thu, 23 Mar 2017 21:31:39 GMT):
and that needs to be done under the regime of the prior consensus
jsmitchell (Thu, 23 Mar 2017 21:31:47 GMT):
which is nothing because state is empty
jsmitchell (Thu, 23 Mar 2017 21:31:53 GMT):
so it should be a default
jjason (Thu, 23 Mar 2017 21:37:35 GMT):
Yeah...that is fine....if we have this "genesis consensus" and it can do all of the legwork to then bootstrap us into PoET consensus...then cool.
Dan (Thu, 23 Mar 2017 21:44:59 GMT):
I don't see anything in the validator registry transaction handler that would be an issue other than this freshness problem we have regardless of genesis approach: https://github.com/hyperledger/sawtooth-core/blob/master/consensus/poet/families/sawtooth_validator_registry/validator_registry/processor/handler.py#L359 I wonder if that would be better as most recent block_id rather than most recent wait_cert_id.
then you aren't special casing genesis differently than if you dev-moded for 20 blocks and then switched to poet.
pschwarz (Thu, 23 Mar 2017 22:05:32 GMT):
Created a story for the genesis consensus in the next sprint, assigned to me. If anyone has things they'd like to add to the description, please do
boydjohnson (Fri, 24 Mar 2017 01:23:27 GMT):
There was talk about changing consensus algorithms after the genesis block has been created. '''[01:18:08.436 INFO chain] on_block_validated: 03e465dc(1, S:8e200cdd, P:9e9b4953)
[01:18:08.437 INFO chain] Chain head updated to: 03e465dc(1, S:8e200cdd, P:9e9b4953)
[01:18:08.437 INFO publisher] Now building on top of block: 03e465dc(1, S:8e200cdd, P:9e9b4953)
[01:18:08.550 INFO poet_enclave_factory] Load PoET enclave module: sawtooth_poet_simulator.poet_enclave_simulator.poet_enclave_simulator
[01:18:08.551 INFO poet_enclave_factory] sawtooth.poet.target_wait_time: 10.000000
[01:18:08.551 INFO poet_enclave_factory] sawtooth.poet.initial_wait_time: 3000.000000
[01:18:08.551 INFO poet_enclave_factory] sawtooth.poet.certificate_sample_length: 50
[01:18:08.551 INFO poet_enclave_factory] sawtooth.poet.fixed_duration_blocks: 50
[01:18:08.551 INFO poet_enclave_factory] sawtooth.poet.minimum_wait_time: 1.000000
[01:18:08.725 CRITICAL publisher] on_chain_updated exception.
[01:18:08.725 ERROR publisher] Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "/project/sawtooth-core/validator/sawtooth_validator/journal/publisher.py", line 332, in on_chain_updated
self._candidate_block = self._build_block(chain_head)
File "/project/sawtooth-core/validator/sawtooth_validator/journal/publisher.py", line 134, in _build_block
if not self._consensus.initialize_block(block_builder.block_header):
File "/project/sawtooth-core/consensus/poet/core/sawtooth_poet/poet_consensus/poet_block_publisher.py", line 211, in initialize_block
self._register_signup_information(block_header=block_header)
File "/project/sawtooth-core/consensus/poet/core/sawtooth_poet/poet_consensus/poet_block_publisher.py", line 100, in _register_signup_information
poet_enclave_module=self._poet_enclave_module).identifier
File "/project/sawtooth-core/consensus/poet/core/sawtooth_poet/poet_consensus/utils.py", line 54, in deserialize_wait_certificate
wait_certificate_dict = json.loads(block.header.consensus.decode())
File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)'''
jjason (Fri, 24 Mar 2017 03:05:19 GMT):
yeah.....right now all hell will break loose.....the PoET consensus module will expect that it can deserialize the contents of block_header.consensus into a wait certificate. The assumption up until this afternoon was that all blocks will have been created by the same consensus module ;)
Dan (Mon, 27 Mar 2017 22:03:17 GMT):
@jjason PR-265 LGTM. I couldn't break it with my mind. I did not run it on a computer though. Usually they break things better.
jjason (Mon, 27 Mar 2017 22:04:41 GMT):
@drozd found a problem if you run 5 validators, but it is deep down in the bowels of abc.py seemingly caused by the cryptography package :(
drozd (Mon, 27 Mar 2017 22:05:26 GMT):
I haven't been able to reproduce it though
drozd (Mon, 27 Mar 2017 22:06:50 GMT):
what's more concerning to me is that the publisher frequently gets stuck in a loop when it gets an error
drozd (Mon, 27 Mar 2017 22:07:02 GMT):
an extremely verbose error loop
jjason (Mon, 27 Mar 2017 22:08:30 GMT):
That may be my fault if I am letting an exception sneak out of `initialize_block()`.
boydjohnson (Mon, 27 Mar 2017 22:13:35 GMT):
Hey @jjason, do you have that abc.py stack trace? I want to add it to my collection.
jjason (Mon, 27 Mar 2017 22:15:59 GMT):
```
validator-2_1 | [03:45:58.672 ERROR publisher] issubclass() arg 2 must be a class or tuple of classes
validator-2_1 | Traceback (most recent call last):
validator-2_1 | File "/project/sawtooth-core/validator/sawtooth_validator/journal/publisher.py", line 414, in on_check_publish_block
validator-2_1 | self._candidate_block = self._build_block(self._chain_head)
validator-2_1 | File "/project/sawtooth-core/validator/sawtooth_validator/journal/publisher.py", line 135, in _build_block
validator-2_1 | if not self._consensus.initialize_block(block_builder.block_header):
validator-2_1 | File "/project/sawtooth-core/consensus/poet/core/sawtooth_poet/poet_consensus/poet_block_publisher.py", line 211, in initialize_block
validator-2_1 | factory.PoetEnclaveFactory.get_poet_enclave_module(state_view)
validator-2_1 | File "/project/sawtooth-core/consensus/poet/core/sawtooth_poet/poet_consensus/poet_enclave_factory.py", line 124, in get_poet_enclave_module
validator-2_1 | module = importlib.import_module(module_name)
validator-2_1 | File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
validator-2_1 | return _bootstrap._gcd_import(name[level:], package, level)
validator-2_1 | File "
jjason (Mon, 27 Mar 2017 22:16:08 GMT):
Per @drozd
drozd (Mon, 27 Mar 2017 22:19:12 GMT):
the publisher doesn't raise when it gets an exception, it just logs it
drozd (Mon, 27 Mar 2017 22:19:27 GMT):
it gets called in a loop by journal, so that loop doesn't stop
jjason (Mon, 27 Mar 2017 22:22:39 GMT):
@drozd You might also start up validators with -vv so you get more verbose logging while you are debugging.
drozd (Mon, 27 Mar 2017 22:26:29 GMT):
with three validators, -vv results in getting buried in interconnect chatter
jjason (Mon, 27 Mar 2017 22:29:33 GMT):
Well...if you can't see the error, then it must not exist ;)
cianx (Tue, 28 Mar 2017 03:47:11 GMT):
@jjason Do you remember what the issue was that casued problems in PoET with speculatively building on the just published? It was missing from one of the stores or caches.
jjason (Tue, 28 Mar 2017 04:22:23 GMT):
The problem I ran into was that because blocks were speculatively built, it caused a wait timer to be created (because `initialize_block()` was called) and by the time `finalize_block()` was called, because it might be some time before transactions were submitted, the timer had not only expired, but it had timed out so the call to create the wait certificate would fail.
cianx (Tue, 28 Mar 2017 05:24:16 GMT):
Thx, that was different than I remembered.
travin (Wed, 29 Mar 2017 14:28:27 GMT):
Has joined the channel.
MicBowman (Wed, 29 Mar 2017 17:40:45 GMT):
Has joined the channel.
Dan (Fri, 31 Mar 2017 21:13:47 GMT):
@jjason I think you were intending to use validator_id to construct a file name? per PR 288 https://github.com/hyperledger/sawtooth-core/pull/288/
I see in your unit tests you use a _friendly_ string like we once (still?) had for validator *name*. In your calls it looks like you will instantiate in practice with the public key of the validator.
jjason (Fri, 31 Mar 2017 21:15:06 GMT):
The intention is that this will really be the validator's string-encoded public key. This is also what the validator registry keys off of.
Dan (Fri, 31 Mar 2017 21:23:43 GMT):
Id is used elsewhere as a function of the pk. If this param is to be used as a pk then consider calling it as such. I'm still not clear on the use though. Is this for a file name, or creating signup info, ...?
jjason (Fri, 31 Mar 2017 21:29:21 GMT):
It is meant to be a persistent, unique identifier for a validator. The validator registry uses it as its key so it can look up signup information. The consensus state store will use it to guarantee a unique name for its underlying LMDB file.
drozd (Fri, 31 Mar 2017 21:43:16 GMT):
`[06:46:52.992 INFO poet_fork_resolver] Chain with new fork head 28c0c6b1...b16bdf17 longer (5) than current chain head 9a55aed4...24233547 (4)`
drozd (Fri, 31 Mar 2017 21:43:23 GMT):
fancy new logging statement
jjason (Fri, 31 Mar 2017 21:43:53 GMT):
@drozd Yeah - I wanted some visibility into fork resolution.
Dan (Fri, 31 Mar 2017 22:04:55 GMT):
Got it. Thanks.
jjason (Fri, 31 Mar 2017 22:12:55 GMT):
It doesn't necessarily _have_ to be the public key of the validator, but currently that is the only unique identifier we have. In 0-7 I believe it was the bitcoin-style address derived from the public key. It may be that this is a larger question about what we should use as a persistent, unique ID for a validator.
jjason (Fri, 31 Mar 2017 22:13:26 GMT):
Maybe not _the_ only...but certainly the one that the PoET consensus module has access to when verifying blocks, etc.
drozd (Fri, 31 Mar 2017 22:56:11 GMT):
@jjason I've been trying variations of the test, and I haven't seen them get out of consensus so far
drozd (Fri, 31 Mar 2017 22:56:44 GMT):
what's something that you think might be more difficult?
jjason (Fri, 31 Mar 2017 22:56:46 GMT):
Oh....I can pretty reliably cause all sorts of havoc :)
jjason (Fri, 31 Mar 2017 22:57:09 GMT):
This branch is pretty good at breaking things:https://github.com/jjason/sawtooth-core/tree/DebugForkResolution
jjason (Fri, 31 Mar 2017 22:57:23 GMT):
It sets target wait time to 5 and initial wait time to 0.
jjason (Fri, 31 Mar 2017 22:57:36 GMT):
Increases odds of forks tremendously.
jjason (Fri, 31 Mar 2017 23:02:04 GMT):
That branch also sets the validators to -vv logging level. What I have been doing is running the test as such:
`run_docker_test poet-smoke.yaml | tee output`
That way I can watch for anomlies (like timeouts, etc.) and also capture to a file so I can post-process.
drozd (Fri, 31 Mar 2017 23:02:36 GMT):
damn, I was trying to figure out how to do that
jjason (Fri, 31 Mar 2017 23:03:00 GMT):
Here are a couple of scripts I use to post-process the log (strips out color stuff and messages we don't really care about). It also takes the test results summary and makes it a little easier to read.
jjason (Fri, 31 Mar 2017 23:05:02 GMT):
Message Attachments
jjason (Fri, 31 Mar 2017 23:05:29 GMT):
Message Attachments
jjason (Fri, 31 Mar 2017 23:06:25 GMT):
@drozd After capturing the log, you can run:
`./process_log output`
It has hard-coded assumption right now for 3 validators, so produces 4 files. One that is the results of the PoET smoke test and one file for the log messages emitted by that particular validator.
boydjohnson (Fri, 31 Mar 2017 23:29:28 GMT):
Hey @jjason, in that branch you posted. the second to last commit has block, and batch broadcast commented out. Is that what @jsmitchell was talking about was causing the weird stack corruption errors?
jjason (Fri, 31 Mar 2017 23:30:52 GMT):
I think so.
jjason (Fri, 31 Mar 2017 23:31:19 GMT):
Is that commit message something like "GARBAGE...."
boydjohnson (Fri, 31 Mar 2017 23:33:00 GMT):
Yeah, there are two commit messages "GARBAGE GARBAGE GARBAGE NEVER MERGE", and this one has @jsmitchell as the Author and you as the Committer.
jjason (Fri, 31 Mar 2017 23:33:53 GMT):
yeah. His commit did that as well as added some more logging and changed some PoET parameters in the configuration. The other one is mine that adds more logging and turns up the verbosity of the validator to _11_.
boydjohnson (Fri, 31 Mar 2017 23:36:50 GMT):
:) I would like to write some unitish tests for these weird errors that would use one component but in a mult-threaded manner, like it would be used in the actual code.
boydjohnson (Fri, 31 Mar 2017 23:37:34 GMT):
Perhaps we could narrow the concurrency bugs down to components.
jjason (Fri, 31 Mar 2017 23:37:34 GMT):
Yeah....being able to ferret out threading issues with unit tests would be suh-weet.
jjason (Fri, 31 Mar 2017 23:40:17 GMT):
Now that I got those other PRs up and done, I am back to debugging. I have found that when the fork happens, it _seems_ that the validator that is out of consensus has validated a block but because the chain head changed underneath it, it wants to re-verify. I am not trying to figure out if it does and if it doesn't, why not. Once that happens, the block just sits in the pending queue and all blocks that build on the block also just sit in the pending queue.
jjason (Fri, 31 Mar 2017 23:40:17 GMT):
Now that I got those other PRs up and done, I am back to debugging. I have found that when the fork happens, it _seems_ that the validator that is out of consensus has validated a block but because the chain head changed underneath it, it wants to re-verify. I am now trying to figure out if it does and if it doesn't, why not. Once that happens, the block just sits in the pending queue and all blocks that build on the block also just sit in the pending queue.
jjason (Fri, 31 Mar 2017 23:43:01 GMT):
One thing I have noticed with testing is that if the default PoET settings are used, the PoET smoke test passes pretty reliably. However, once ratcheting it up (for example - setting target wait time to 5 so that wait timers are shorter and increase the likelihood of forks) that is when the wheels start to fall off.
boydjohnson (Fri, 31 Mar 2017 23:44:00 GMT):
Yeah, @drozd says there are other ways to get errors, too. Like different peering combinations, especially more of a mesh network.
boydjohnson (Fri, 31 Mar 2017 23:45:16 GMT):
But I would think those are more of our concurrency issues in the networking layer problems.
boydjohnson (Fri, 31 Mar 2017 23:46:19 GMT):
I'm taking off. Have a good weekend, @jjason!
jjason (Fri, 31 Mar 2017 23:46:49 GMT):
@boydjohnson You too.
drozd (Fri, 31 Mar 2017 23:55:17 GMT):
@jjason I'm working on a new test setup, so get ready to throw that hard-coded assumption in the trash
jjason (Sat, 01 Apr 2017 00:04:07 GMT):
@drozd That's cool. The scripts were thrown together quickly for quick debugging. I was thinking of changing to a single Python script that you could just pipe output of test while it is running and would clean up and produce output files in real time. Could also make it take as a command line parameter a file that has messages to strip out.
jjason (Sat, 01 Apr 2017 16:09:35 GMT):
@jsmitchell @boydjohnson @drozd @cianx @pschwarz I found what _appears_ to be an issue in fork resolution/chain processing that is leading to the unrecoverable forks seeing when running the PoET smoke test with aggressive target wait time (5). I have some potential fixes that I am going to test today.
jsmitchell (Sat, 01 Apr 2017 16:12:34 GMT):
:woo:
jjason (Sun, 02 Apr 2017 03:51:21 GMT):
@jsmitchell @boydjohnson @cianx @drozd @pschwarz I made a couple of potential fixes to the the chain controller code (in `ChainController.on_block_validated`).
The first problem I found was that, because target and initial wait times are small, blocks are coming quickly and I found that validators would get into this state where they kept running the same block through validation. The reason was that each time it had made a decision about the block, it would find that the chain head was updated while that was happening. So, I changed the code so that that if a block validates successfully, but the chain head changed while it was being validated, and one or more descendant blocks arrived while the block was being validated, instead of rescheduling that block it schedules the descendants. If not, then it simply reschedules the block. There is still a potential problem with that I haven't bothered working out yet.
The second problem I found was that if a validator checks a block, decides it likes the current chain head better, and it doesn't detect that the chain head changed while it was processing it, if a descendant had arrived while that block was being processed it will stay in the pending blocks map and it, and all of its descendants, will never be checked (because each descendant block checks to see if its ancestor is in the pending blocks map and if so, it assumes that it will eventually be processed). Essentially, the original block and all descendants are on a fork that that validator will never bother checking. The fix was that if a block is valid, but is not to become the chain head, if it has immediate descendants in the pending blocks map, then the descendants are validated.
Here is my branch with the proposed fixes (as well as copious amounts of logging messages):
https://github.com/jjason/sawtooth-core/tree/DebugForkResolution
If you search for `JLJ` you'll find the two places I updated in the code.
I am still tracking down one problem I am seeing. For some reason, the executor thread stops running once the PoET smoke test starts getting the results. If you look at the log messages in my next post, you notice that all of a sudden the chain controller stops processing even though the validator keeps responding to message from REST API.
jjason (Sun, 02 Apr 2017 03:51:21 GMT):
@jsmitchell @boydjohnson @cianx @drozd @pschwarz I made a couple of potential fixes to the the chain controller code (in `ChainController.on_block_validated`).
The first problem I found was that, because target and initial wait times are small, blocks are coming quickly and I found that validators would get into this state where they kept running the same block through validation. The reason was that each time it had made a decision about the block, it would find that the chain head was updated while that was happening. So, I changed the code so that that if a block validates successfully, but the chain head changed while it was being validated, and one or more descendant blocks arrived while the block was being validated, instead of rescheduling that block it schedules the descendants. If not, then it simply reschedules the block. There is still a potential problem with that I haven't bothered working out yet.
The second problem I found was that if a validator checks a block, decides it likes the current chain head better, and it doesn't detect that the chain head changed while it was processing it, if a descendant had arrived while that block was being processed it will stay in the pending blocks map and it, and all of its descendants, will never be checked (because each descendant block checks to see if its ancestor is in the pending blocks map and if so, it assumes that it will eventually be processed). Essentially, the original block and all descendants are on a fork that that validator will never bother checking. The fix was that if a block is valid, but is not to become the chain head, if it has immediate descendants in the pending blocks map, then the descendants are validated.
Here is my branch with the proposed fixes (as well as copious amounts of logging messages):
https://github.com/jjason/sawtooth-core/tree/DebugForkResolution
I am still tracking down one problem I am seeing. For some reason, the executor thread stops running once the PoET smoke test starts getting the results. If you look at the log messages in my next post, you notice that all of a sudden the chain controller stops processing even though the validator keeps responding to message from REST API.
jjason (Sun, 02 Apr 2017 03:52:54 GMT):
```validator-1_1 | [02:15:03.255 INFO chain] Block wrapper status valid: 89fed805(16, S:c67da7e4, P:29fe84f5)
validator-1_1 | [02:15:03.256 INFO chain] Finish self.validate_block: 89fed805(16, S:c67da7e4, P:29fe84f5)
validator-1_1 | [02:15:03.256 INFO chain] Start self.validate_block: b852fc69(17, S:c0814a75, P:89fed805)
validator-1_1 | [02:15:03.256 INFO chain] Block wrapper status valid: b852fc69(17, S:c0814a75, P:89fed805)
validator-1_1 | [02:15:03.256 INFO chain] Finish self.validate_block: b852fc69(17, S:c0814a75, P:89fed805)
validator-1_1 | [02:15:03.257 INFO chain] Start self.validate_block: 40f1b20b(18, S:bb0bc0fa, P:b852fc69)
validator-1_1 | [02:15:03.257 INFO chain] Block wrapper status valid: 40f1b20b(18, S:bb0bc0fa, P:b852fc69)
validator-1_1 | [02:15:03.257 INFO chain] Finish self.validate_block: 40f1b20b(18, S:bb0bc0fa, P:b852fc69)
validator-1_1 | [02:15:03.257 INFO chain] Start self.validate_block: 08b6c692(19, S:c28bb8b9, P:40f1b20b)
validator-1_1 | [02:15:03.258 INFO chain] Block wrapper status valid: 08b6c692(19, S:c28bb8b9, P:40f1b20b)
validator-1_1 | [02:15:03.258 INFO chain] Finish self.validate_block: 08b6c692(19, S:c28bb8b9, P:40f1b20b)
validator-1_1 | [02:15:03.258 INFO chain] Start self.validate_block: 68fcdc24(20, S:0cb4daf9, P:08b6c692)
validator-1_1 | [02:15:14.622 DEBUG interconnect] OutboundConnectionThread-tcp://validator-0:8800 receiving NETWORK_PING message: 70 bytes
validator-1_1 | [02:15:14.635 DEBUG interconnect] OutboundConnectionThread-tcp://validator-0:8800 sending NETWORK_ACK to tcp://validator-0:8800
validator-1_1 | [02:15:18.639 DEBUG interconnect] ServerThread sending NETWORK_PING to b'd67529ad395db2b3f38ab74-555ddcfc6bb35c8f3618e26'
validator-1_1 | [02:15:18.652 DEBUG interconnect] ServerThread receiving NETWORK_ACK message: 70 bytes
validator-1_1 | [02:15:18.654 DEBUG interconnect] message round trip: NETWORK_ACK 0.014345884323120117
validator-1_1 | [02:15:28.872 DEBUG interconnect] ServerThread receiving CLIENT_BLOCK_LIST_REQUEST message: 69 bytes
validator-1_1 | [02:15:28.877 DEBUG interconnect] ServerThread sending CLIENT_BLOCK_LIST_RESPONSE to b'e2eb927b89484c5d'
validator-1_1 | [02:15:30.067 DEBUG interconnect] ServerThread receiving CLIENT_BLOCK_LIST_REQUEST message: 69 bytes
validator-1_1 | [02:15:30.073 DEBUG interconnect] ServerThread sending CLIENT_BLOCK_LIST_RESPONSE to b'e2eb927b89484c5d'
validator-1_1 | [02:15:30.160 DEBUG interconnect] ServerThread receiving CLIENT_BLOCK_LIST_REQUEST message: 69 bytes
validator-1_1 | [02:15:30.165 DEBUG interconnect] ServerThread sending CLIENT_BLOCK_LIST_RESPONSE to b'e2eb927b89484c5d'
validator-1_1 | [02:15:34.643 DEBUG interconnect] OutboundConnectionThread-tcp://validator-0:8800 receiving NETWORK_PING message: 70 bytes
validator-1_1 | [02:15:34.646 DEBUG interconnect] OutboundConnectionThread-tcp://validator-0:8800 sending NETWORK_ACK to tcp://validator-0:8800
validator-1_1 | [02:15:38.660 DEBUG interconnect] ServerThread sending NETWORK_PING to b'd67529ad395db2b3f38ab74-555ddcfc6bb35c8f3618e26'
validator-1_1 | [02:15:38.671 DEBUG interconnect] ServerThread receiving NETWORK_ACK message: 70 bytes
validator-1_1 | [02:15:38.672 DEBUG interconnect] message round trip: NETWORK_ACK 0.01240682601928711
validator-1_1 | [02:15:44.662 DEBUG interconnect] OutboundConnectionThread-tcp://validator-0:8800 receiving NETWORK_PING message: 70 bytes
validator-1_1 | [02:15:44.667 DEBUG interconnect] OutboundConnectionThread-tcp://validator-0:8800 sending NETWORK_ACK to tcp://validator-0:8800
validator-1_1 | [02:15:48.673 DEBUG interconnect] ServerThread sending NETWORK_PING to b'd67529ad395db2b3f38ab74-555ddcfc6bb35c8f3618e26'
validator-1_1 | [02:15:48.688 DEBUG interconnect] ServerThread receiving NETWORK_ACK message: 70 bytes
validator-1_1 | [02:15:48.689 DEBUG interconnect] message round trip: NETWORK_ACK 0.01577305793762207
validator-1_1 | [02:16:04.681 DEBUG interconnect] OutboundConnectionThread-tcp://validator-0:8800 receiving NETWORK_PING message: 70 bytes
validator-1_1 | [02:16:04.685 DEBUG interconnect] OutboundConnectionThread-tcp://validator-0:8800 sending NETWORK_ACK to tcp://validator-0:8800```
jjason (Sun, 02 Apr 2017 03:54:21 GMT):
There are more interconnect messages (for the next 1.5 minutes while test finishes), but no messages from chain. All three validators show this same behavior. So, I don't know if it is something funky going on in the docker containers or what is going on.
jsmitchell (Sun, 02 Apr 2017 21:30:57 GMT):
@jjason: I've seen situations where exceptions are getting swallowed. You might want to put a try except block around the contents of run() and log and exception that is caught
jjason (Sun, 02 Apr 2017 21:48:00 GMT):
@jsmitchell Will do. It is strange that all three validators just stop processing blocks at the same time.
jsmitchell (Sun, 02 Apr 2017 22:09:42 GMT):
Would make sense if there is a condition that is deterministically causing them to error :)
jjason (Mon, 03 Apr 2017 15:42:09 GMT):
`ChainController.run` is already wrapped to catch `Exception`. Going to dive into excecutor code to see what is going on.
jjason (Mon, 03 Apr 2017 16:30:28 GMT):
I think that there is one more thing I need to check in `ChainController.on_block_validated()`. If a block is considered invalid, then I think that the all descendant blocks need to be removed from the pending blocks queue as they will never be added to the chain as they are on a fork with an invalid block.
Dan (Tue, 04 Apr 2017 01:31:34 GMT):
@jjason nice work on PR 291... 5 neatly packaged commits, full unit tests for new functionality, clear comments. And in 1200 LOC I can't find anything wrong.
jjason (Tue, 04 Apr 2017 01:38:38 GMT):
Thanks @Dan. Give @pschwarz credit for the commit messages.
jjason (Tue, 04 Apr 2017 01:38:38 GMT):
Thanks @Dan. Give @pschwarz credit for the commit messages.
jjason (Tue, 04 Apr 2017 01:39:38 GMT):
Oh, and let's just leave it at most unit tests for new functionality :( I have a few more I need to add.
cianx (Thu, 06 Apr 2017 22:57:09 GMT):
```$ run_tests
[---Running cli tests---]
INFO:__main__:Bringing up with ['docker-compose', '-p', 'latest', '-f', './cli/tests/unit_cli.yaml', 'up', '--abort-on-container-exit']
Creating network "latest_default" with the default driver
Pulling cli (sawtooth-dev-test:latest)...
ERROR: repository sawtooth-dev-test not found: does not exist or no pull access
ERROR:__main__:Failed to start test.
ERROR:__main__:Command '['docker-compose', '-p', 'latest', '-f', './cli/tests/unit_cli.yaml', 'up', '--abort-on-container-exit']' returned non-zero exit status 1
Traceback (most recent call last):
File "/project/sawtooth-core/bin/run_docker_test", line 101, in main
subprocess.run(compose_up, check=True, timeout=timer.remaining())
File "/usr/lib/python3.5/subprocess.py", line 708, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker-compose', '-p', 'latest', '-f', './cli/tests/unit_cli.yaml', 'up', '--abort-on-container-exit']' returned non-zero exit status 1
INFO:__main__:Shutting down with: ['docker-compose', '-p', 'latest', '-f', './cli/tests/unit_cli.yaml', 'down', '--remove-orphans']
Removing network latest_default
```
cianx (Thu, 06 Apr 2017 22:57:09 GMT):
Error Number 1 today(that I am sharing): ```$ run_tests
[---Running cli tests---]
INFO:__main__:Bringing up with ['docker-compose', '-p', 'latest', '-f', './cli/tests/unit_cli.yaml', 'up', '--abort-on-container-exit']
Creating network "latest_default" with the default driver
Pulling cli (sawtooth-dev-test:latest)...
ERROR: repository sawtooth-dev-test not found: does not exist or no pull access
ERROR:__main__:Failed to start test.
ERROR:__main__:Command '['docker-compose', '-p', 'latest', '-f', './cli/tests/unit_cli.yaml', 'up', '--abort-on-container-exit']' returned non-zero exit status 1
Traceback (most recent call last):
File "/project/sawtooth-core/bin/run_docker_test", line 101, in main
subprocess.run(compose_up, check=True, timeout=timer.remaining())
File "/usr/lib/python3.5/subprocess.py", line 708, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker-compose', '-p', 'latest', '-f', './cli/tests/unit_cli.yaml', 'up', '--abort-on-container-exit']' returned non-zero exit status 1
INFO:__main__:Shutting down with: ['docker-compose', '-p', 'latest', '-f', './cli/tests/unit_cli.yaml', 'down', '--remove-orphans']
Removing network latest_default
```
cianx (Thu, 06 Apr 2017 22:57:35 GMT):
Thought maybe this was because I forgot to do a build all.
cianx (Thu, 06 Apr 2017 22:58:00 GMT):
so ....
cianx (Thu, 06 Apr 2017 22:58:04 GMT):
```(SAWTOOTH)ubuntu-xenial /project/sawtooth-core
$ time build_all
[--- Building images with sawtooth mounted ---]
[--- Building java ---]
[--- Building sawtooth-build-java... ---]
Sending build context to Docker daemon 48.64kB
Step 1/2 : FROM maven:3-jdk-8
3-jdk-8: Pulling from library/maven
6d827a3ef358: Pull complete
2726297beaf1: Pull complete
7d27bd3d7fec: Pull complete
e61641c845ed: Pull complete
cce4cca5b76b: Pull complete
6826227500b0: Pull complete
c03b117ffd91: Pull complete
821a1547b435: Pull complete
2bd47f6b1b42: Pull complete
e4cf3e9f705c: Pull complete
3733107c5c01: Pull complete
Digest: sha256:18e8bd367c73c93e29d62571ee235e106b18bf6718aeb235c7a07840328bba71
Status: Downloaded newer image for maven:3-jdk-8
---> 9997d8483b2f
Step 2/2 : CMD /project/sawtooth-core/bin/build_java_sdk && /project/sawtooth-core/bin/build_java_intkey
---> Running in 42487c1c551d
---> e1bf9ff1bfa9
Removing intermediate container 42487c1c551d
Successfully built e1bf9ff1bfa9
Setting HTTP proxy to (proxy-us.intel.com, 911/)
Setting HTTPS proxy to (proxy-us.intel.com, 912/)
[INFO] Scanning for projects...
[INFO] Downloading: https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[ERROR] Unresolveable build extension: Plugin kr.motd.maven:os-maven-plugin:1.4.0.Final or one of its dependencies could not be resolved: Failed to read artifact descriptor for kr.motd.maven:os-maven-plugin:jar:1.4.0.Final @
@
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project sawtooth:sdk:1.0-SNAPSHOT (/project/sawtooth-core/sdk/java/pom.xml) has 1 error
[ERROR] Unresolveable build extension: Plugin kr.motd.maven:os-maven-plugin:1.4.0.Final or one of its dependencies could not be resolved: Failed to read artifact descriptor for kr.motd.maven:os-maven-plugin:jar:1.4.0.Final: Could not transfer artifact kr.motd.maven:os-maven-plugin:pom:1.4.0.Final from/to central (https://repo.maven.apache.org/maven2): Connect to proxy-us.intel.com:443 [proxy-us.intel.com/10.7.211.16] failed: Connection refused (Connection refused) -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/PluginManagerException
real 0m59.633s
user 0m0.156s
sys 0m0.256s
cianx (Thu, 06 Apr 2017 23:16:45 GMT):
Testing but this may be due to a trailing slash on my proxy environment settings.
cianx (Thu, 06 Apr 2017 23:17:40 GMT):
@adamludvik ^ can you harden your docker build scripts to not be tripped up by a trailing slash on the proxy environment variables?
adamludvik (Thu, 06 Apr 2017 23:17:40 GMT):
Has joined the channel.
haridhakshini (Sun, 09 Apr 2017 02:12:11 GMT):
Has joined the channel.
Lin-YiTang (Tue, 11 Apr 2017 21:15:03 GMT):
Has joined the channel.
jaredcorduan (Thu, 13 Apr 2017 19:28:42 GMT):
Has joined the channel.
sklymenko (Fri, 14 Apr 2017 12:07:38 GMT):
Has joined the channel.
schwentker2 (Mon, 17 Apr 2017 12:25:50 GMT):
Has joined the channel.
amundson (Mon, 17 Apr 2017 22:36:46 GMT):
Has joined the channel.
DerekTiffany (Mon, 24 Apr 2017 18:01:36 GMT):
Has joined the channel.
pancake (Thu, 27 Apr 2017 08:59:49 GMT):
Has joined the channel.
Dan (Sat, 29 Apr 2017 16:09:37 GMT):
@jjason with your spate of commits over the last couple weeks I think we are now at feature complete on the poet - sawtooth 0.8 integration? I know there are testing stories and some refactoring / cleanup, but from a functionality perspective we are golden right?
jjason (Sat, 29 Apr 2017 20:11:31 GMT):
@Dan I think from the perspective of the core PoET consensus, we are 99% of the way. We do have a couple of small stories about removing the old kwargs from the PoET enclaves. However, those stories are not part of the core PoET consensus, so from that perspective you could consider it done.
tkuhrt (Tue, 02 May 2017 17:22:47 GMT):
Has joined the channel.
nitingaur (Fri, 05 May 2017 22:20:01 GMT):
Has joined the channel.
jaemin.lim (Thu, 11 May 2017 13:11:10 GMT):
Has joined the channel.
dhuseby (Thu, 11 May 2017 14:14:48 GMT):
Has joined the channel.
chrisconway (Mon, 15 May 2017 16:07:08 GMT):
Has joined the channel.
vinoypm (Tue, 16 May 2017 09:32:21 GMT):
Has joined the channel.
Donald Liu (Wed, 17 May 2017 01:59:21 GMT):
Has joined the channel.
kellyo (Tue, 23 May 2017 16:01:22 GMT):
Has joined the channel.
kenmazsyma (Wed, 24 May 2017 11:51:20 GMT):
Has joined the channel.
kenmazsyma (Wed, 24 May 2017 23:39:31 GMT):
Has left the channel.
desongchen (Fri, 26 May 2017 03:30:22 GMT):
Has joined the channel.
jjason (Fri, 26 May 2017 23:56:00 GMT):
Heads up to anyone using PoET consensus. With recent merge, you now need to put the report key public PEM in the config settings on the blockchain (`sawtooth.poet.report_public_key_pem`). For example:
```
sawtooth admin keygen validator
sawtooth config genesis --key ~/sawtooth/keys/validator.priv -o config-genesis.batch
sawtooth config proposal create -k ~/sawtooth/keys/validator.priv sawtooth.consensus.algorithm=poet sawtooth.poet.report_public_key_pem="$(cat /project/sawtooth-core/consensus/poet/simulator/packaging/simulator_rk_pub.pem)" -o config.batch
poet genesis -k ~/sawtooth/keys/validator.priv -o poet_genesis.batch
sawtooth admin genesis config-genesis.batch config.batch poet_genesis.batch
```
adamludvik (Tue, 30 May 2017 19:38:25 GMT):
@drozd @jjason Does this break https://github.com/hyperledger/sawtooth-core/blob/master/integration/sawtooth_integration/tests/node_controller.py#L141?
adamludvik (Tue, 30 May 2017 19:38:51 GMT):
Or is that only for SGX?
jjason (Tue, 30 May 2017 19:40:53 GMT):
The change is PoET-specific, either simulator or SGX. What you are referencing uses PoET simulator.
jjason (Tue, 30 May 2017 19:40:53 GMT):
@adamludvik The change is PoET-specific, either simulator or SGX. What you are referencing uses PoET simulator.
adamludvik (Tue, 30 May 2017 19:42:50 GMT):
What is the failure mode if you don't set that config option?
jjason (Tue, 30 May 2017 19:46:56 GMT):
The PEM I assume you mean. The failure is that the genesis node validator registry transaction cannot be validated and the genesis validator will not start.
jjason (Tue, 30 May 2017 19:51:00 GMT):
You wouild likely see a message something like the following in the validators
jjason (Tue, 30 May 2017 19:51:00 GMT):
You wouild likely see a message something like the following in the validator's log:
```
[12:48:34.086 ERROR cli] Unable to create genesis block, due to batch 1b4321dc95ba19e1f6f94c13c68db48309f91bc474f6eeafd515437bcfbae0eb381f7a499c3b7109875ea3639fe0099a90d2656da03a583ff4e1fa7e65aec7a7
```
jjason (Tue, 30 May 2017 19:56:58 GMT):
And in the validator registry TP log, you would see something similar to:
```
[12:48:34 WARNING core] Invalid Transaction Invalid Signup Info: poet_public_key: "03950845be76e144edc58289c2afc88856566d4779de053a945e5d451b0179301e" proof_data: "{\"evidence_payload\": {\"pse_manifest\": \"YTdjYzYzNjM3NmI5N2FhYzhkMjM3YjkyYjYxMzUyMjNhNWZiNzFmYmU1NjM0ZWQ2NDc3MjZjNTlkNjU5MTkyYw==\"}, \"signature\": \"Qrr+AAd5D1t+Ki1UYsFXNRVlM27fodTDU9vVX098RtoPAeTMywPKJ47dEBU0OH54BSXA0BQLQqtp2ig0RORS5QOKPLPAfmdKdOF82ljUSYfIL3zFmnC6loWHA8/MUjjUomsCPYoxdoRC7w/65uJksn+riEGVjNjenHpiDaZXSb2y1Kbv8487zZ6vYwDU/96xE3nwC75UiYypHO7VeovFM/jT+Hw8jfLhJMFIKoUQ3toFC96O4edFXzMK7erhkH147vSCpEVX5USrkq+20Ih+7JUQDSD0H2JGgjPs0reHP+w07qFT8mSkWCtRwsFZDFFZ6D/V2mw8ZiRay1ZuAE5rbQ==\", \"verification_report\": \"{\\\"epidPseudonym\\\": \\\"84ef48385ce91ec2802a88e100ddb3530d86d4e7749f27e65a41fa92ff42e4f2\\\", \\\"id\\\": \\\"ZTUwNjRmNThmNTE2OGE2MGMxM2U0Nzg2OWQyYzhhODVjYWUyMjhmOTYyZWNkNjM2MTIzY2IzYmY2NWUyZGRlZA==\\\", \\\"isvEnclaveQuoteBody\\\": \\\"AAAAAAAAAAAAAAAAAAAAALeFxYt3FSy+f9Ve44UcSZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMmfIZVeONuwPSyoONOvbkPvQ4km7QLbTMcpOAyMehdOAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA2Y+QClFcV0M09DEZdyLf4Qsa6nys5ZZpM7xY9R6KvdwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==\\\", \\\"isvEnclaveQuoteStatus\\\": \\\"OK\\\", \\\"nonce\\\": \\\"0000000000000000\\\", \\\"pseManifestHash\\\": \\\"ec02c5dca90d042ca178c7de222977e802c32f70b3606983d7137158245968dc\\\", \\\"pseManifestStatus\\\": \\\"OK\\\", \\\"timestamp\\\": \\\"2017-05-30T12:48:15.437003\\\"} \"}", anti_sybil_id: "84ef48385ce91ec2802a88e100ddb3530d86d4e7749f27e65a41fa92ff42e4f2", nonce: "0000000000000000" \"}", Reason: Report public key configuration setting (sawtooth.poet.report_public_key_pem) not found
```
adamludvik (Tue, 30 May 2017 19:58:02 GMT):
I think I figured it out
adamludvik (Wed, 31 May 2017 14:13:37 GMT):
@jjason Can you remind why this error occurs? `Failed to create wait certificate: Cannot create wait certificate because timer has timed out`
jjason (Wed, 31 May 2017 14:24:06 GMT):
It is taking too long between the time the wait timer for the block expired and the corresponding wait certificate was created.
adamludvik (Wed, 31 May 2017 14:43:54 GMT):
What happens in that interval?
adamludvik (Wed, 31 May 2017 14:44:08 GMT):
(roughly)
adamludvik (Wed, 31 May 2017 14:55:15 GMT):
Also, I am seeing a lot of poet errors that seem to be related to the c, k, z tests. Examples:
`[14:53:39.730 ERROR poet_block_publisher] Reject building on block 4e632c39: Validator signup information not committed in a timely manner.`
adamludvik (Wed, 31 May 2017 14:55:29 GMT):
`[14:53:46.728 ERROR consensus_state] Validator validator-036c3364 (ID=036c3364...e1782255): Committed in block 3, trying to claim block 4, must wait until block 5`
adamludvik (Wed, 31 May 2017 14:55:41 GMT):
`[14:53:46.731 ERROR poet_block_publisher] Reject building on block ee4908d3: Validator has not waited long enough since registering validator information.`
adamludvik (Wed, 31 May 2017 14:55:55 GMT):
`[14:54:19.322 ERROR poet_block_publisher] Reject building on block 7165ee16: Validator signup information not committed in a timely manner.`
adamludvik (Wed, 31 May 2017 14:56:45 GMT):
Does anyone know if these were happening previously in the dynamic-network test? @drozd
drozd (Wed, 31 May 2017 15:53:53 GMT):
those are new to me
jsmitchell (Wed, 31 May 2017 16:41:31 GMT):
I think those are normal if it's reporting it for itself
jsmitchell (Wed, 31 May 2017 16:41:50 GMT):
if it publishes a block and other validators are rejecting it, that seems like an issue
jjason (Wed, 31 May 2017 16:42:00 GMT):
@adamludvik Yes.
jjason (Wed, 31 May 2017 16:42:44 GMT):
Basically the validator is deciding that it has not waited the number of blocks required and so isn't even going to try to build a block as it knows that the other validators will reject it. This is expected behavior.
jjason (Wed, 31 May 2017 16:44:34 GMT):
I don't like the last error as much but it is still normal - basically that one means that when a validator went to register it thought that a certain block was the head of the chain. However, when other blocks received the block containing its registration information, that block is not building on top of the expected block. So, they are rejecting. However, the error message you see causes that validator to generate new signup information.
jjason (Thu, 01 Jun 2017 22:46:42 GMT):
Heads up to anyone using PoET consensus. With recent merge, you now need to put the expected enclave measurement in the config settings on the blockchain before and validator registry transactions are submitted. For example:
```
sawtooth config proposal create -k ~/sawtooth/keys/validator.priv sawtooth.consensus.algorithm=poet sawtooth.poet.report_public_key_pem="$(cat /project/sawtooth-core/consensus/poet/simulator/packaging/simulator_rk_pub.pem)" sawtooth.poet.valid_enclave_measurements=$(poet enclave measurement) -o config.batch
```
The above command, usually used when firing up the genesis validator, gets that validator's enclave measurement value as the enclave measurement value that the validator registry transaction processor will use when verifying signup information. If the validator registry transaction processor is unable to find the value `sawtooth.poet.valid_enclave_measurements` in the on-chain config settings, it will reject the validator registry transaction.
Dan (Fri, 02 Jun 2017 18:45:37 GMT):
@jjason what is the enclave measurement for the simulated enclave?
jjason (Fri, 02 Jun 2017 19:46:24 GMT):
@Dan `c99f21955e38dbb03d2ca838d3af6e43ef438926ed02db4cc729380c8c7a174e`
jjason (Fri, 02 Jun 2017 19:46:30 GMT):
We can make it pretty much anything we like.
jjason (Fri, 02 Jun 2017 19:46:50 GMT):
You can also run `poet enclave measurement` as the default is to use the simulator.
jjason (Fri, 02 Jun 2017 20:00:16 GMT):
Final piece of on-chain configuration settings for the validator registry transaction processor has been merged. Moving forward, if you are using PoET consensus, when launching a the genesis validator manually, you need to add `sawtooth.poet.valid_enclave_basenames` to the settings before the first validator is registered. You can use `poet enclave basename` to get the genesis validator's basename and use it for testing. For example:
```
sawtooth config proposal create -k ~/sawtooth/keys/validator.priv sawtooth.consensus.algorithm=poet sawtooth.poet.report_public_key_pem="$(cat /project/sawtooth-core/consensus/poet/simulator/packaging/simulator_rk_pub.pem)" sawtooth.poet.valid_enclave_measurements=$(poet enclave measurement) sawtooth.poet.valid_enclave_basenames=$(poet enclave basename) -o config.batch
```
Dan (Fri, 02 Jun 2017 20:38:37 GMT):
sweet! i hadn't had occasion to use the poet CLI yet
h4xr (Sat, 03 Jun 2017 05:35:07 GMT):
Has joined the channel.
benoit.razet (Tue, 06 Jun 2017 17:47:55 GMT):
Has joined the channel.
trbs (Thu, 08 Jun 2017 16:10:40 GMT):
Has joined the channel.
yahtoo (Wed, 14 Jun 2017 08:51:06 GMT):
Has joined the channel.
yahtoo (Wed, 14 Jun 2017 08:54:25 GMT):
Has left the channel.
dataharvest (Thu, 22 Jun 2017 15:59:12 GMT):
Has joined the channel.
dplumb (Mon, 10 Jul 2017 15:05:38 GMT):
Has joined the channel.
Dan (Tue, 11 Jul 2017 16:36:17 GMT):
in stl-394 I'm cleaning up some validator registry state management. In doing that it looks like we have no use for the registered field of ValidatorInfo (consensus/poet/families/sawtooth_validator_registry/protos/validator_registry.proto:23)
Dan (Tue, 11 Jul 2017 16:36:56 GMT):
I may remove that field from the wire protocol in the same PR. Anyone can let me know if they see danger there.
jsmitchell (Tue, 11 Jul 2017 19:09:00 GMT):
@agunde @adamludvik ^
agunde (Tue, 11 Jul 2017 19:20:45 GMT):
@Dan Instead of setting the ValidatorInfo.registred = "revoked", are you just going to delete the info?
Dan (Tue, 11 Jul 2017 19:25:32 GMT):
yes, I'm going to delete the entry for that validator. and then I propose also changing the protobuf definition to not contain the registered field.
Dan (Tue, 11 Jul 2017 19:25:54 GMT):
registration is implicit in being in state. and 'freshness' is checked regularly by poet.
agunde (Tue, 11 Jul 2017 19:26:42 GMT):
Then I don't see a reason to keep the registered field.
Dan (Fri, 14 Jul 2017 20:12:39 GMT):
I thought we had a 'stochastic' re-registration policy, but maybe we only talked about it. We wanted to guard against a flurry of machines having to re-register because their registrations timed out.
I was looking here: consensus/poet/core/build/lib/sawtooth_poet/poet_consensus/poet_block_publisher.py:380 but I don't see a policy besides, it reached its limit.
Dan (Fri, 14 Jul 2017 20:12:39 GMT):
I thought we had a 'stochastic' registration policy, but maybe we only talked about it. We wanted to guard against a flurry of machines having to re-register because their registrations timed out.
I was looking here: consensus/poet/core/build/lib/sawtooth_poet/poet_consensus/poet_block_publisher.py:380 but I don't see a policy besides, it reached its limit.
tim-d-blue (Wed, 19 Jul 2017 15:05:48 GMT):
Has joined the channel.
MaximeLandon (Tue, 25 Jul 2017 11:42:25 GMT):
Has joined the channel.
Dan (Tue, 01 Aug 2017 14:36:28 GMT):
@cianx @adamludvik @rbuysse can I get eyes on these PRs #727, #728, #730 when you get a chance. The first two are baby seals.
rbuysse (Tue, 01 Aug 2017 14:36:28 GMT):
Has joined the channel.
Dan (Tue, 01 Aug 2017 14:37:09 GMT):
apparently purple baby seals.
wbsmith (Fri, 04 Aug 2017 06:02:07 GMT):
Has joined the channel.
Dan (Tue, 08 Aug 2017 14:11:58 GMT):
@cianx do you think we still need the bridge functions in poet? Or is that obviated now that we don't have to worry about msvc?
cianx (Tue, 08 Aug 2017 16:36:44 GMT):
those functions do argument validation and convertions, you can move that else where if you want to rewrite the python bindings to a new api specification.
Dan (Tue, 08 Aug 2017 16:56:28 GMT):
i'm not planning to do anything soon. just curious if the structure was no longer mandated because the compiler situation changed. maybe peel a layer or two out. (noting reduce maintenance cost might not justify the cost to refactor all of the middle.)
Dan (Tue, 08 Aug 2017 20:07:59 GMT):
@cianx @adamludvik @drozd the unit tests under poet/core seem implicitly simulator. One of the tests checks for replay of wait timers. That seems irrelevant. At least the simulator won't have any protection from that without trying to simulate a HW counter - which I don't think is useful.
i can either @skip it or remove it.
If these tests are intended to run against either sgx or simulator I should skip it, if they are intended only for simulator then I should remove it.
adamludvik (Tue, 08 Aug 2017 21:17:19 GMT):
I vote skip it. If they are only implicitly referencing the simulator and are testing details like wait timer replay, I'm guessing they were meant to be used against both the simulator and sgx.
Dan (Fri, 11 Aug 2017 16:18:31 GMT):
@adamludvik @cianx @drozd please look at #730 when you get a chance. @rbuysse tested overnight without issue.
@cianx note I've updated the top comment to include Jira's for your earlier feedback.
Dan (Fri, 11 Aug 2017 16:18:36 GMT):
https://github.com/hyperledger/sawtooth-core/pull/730
Arun.S.M (Fri, 11 Aug 2017 16:45:07 GMT):
Has joined the channel.
zhangchao (Sat, 12 Aug 2017 13:39:33 GMT):
Has joined the channel.
Dan (Sat, 12 Aug 2017 14:17:12 GMT):
I've got an itchy merge finger.
rbuysse (Sun, 13 Aug 2017 21:08:49 GMT):
we're at 11661 blocks re: ^
yiting (Sun, 13 Aug 2017 22:40:26 GMT):
Has joined the channel.
Dan (Mon, 14 Aug 2017 13:41:59 GMT):
@rbuysse can I get a :thumbsup: on the PR?
Dan (Mon, 14 Aug 2017 13:41:59 GMT):
@rbuysse can I get a :thumbsup: on the approval?
rbuysse (Mon, 14 Aug 2017 14:01:35 GMT):
:thumbsup:
Dan (Mon, 14 Aug 2017 14:36:42 GMT):
thx
Dan (Mon, 14 Aug 2017 14:38:02 GMT):
@cianx please double check I have addressed your earlier feedback. I'd like to get this PR merged asap.
SameerF (Wed, 16 Aug 2017 17:38:25 GMT):
Has joined the channel.
SameerF (Wed, 16 Aug 2017 18:16:44 GMT):
Hi, my name is Sameer Ahmed and I work at Wind River. Recently, we have been working on software parts (sparts) project using sawtooth lake. We have already implemented 4 transaction families. Now, I need some guidance on the POET consensus.
SameerF (Wed, 16 Aug 2017 18:23:20 GMT):
Currently, I have docker containers running on different machines. Validator, poet-validator-registry, rest-api is running on each of the container.
boydjohnson (Wed, 16 Aug 2017 18:23:32 GMT):
What are your questions, @SameerF ?
SameerF (Wed, 16 Aug 2017 18:25:12 GMT):
I have already initialized genesis for poet. What should be the next steps? I did browser thru the documentation but I need some guidance here.
boydjohnson (Wed, 16 Aug 2017 18:26:57 GMT):
Check out this docker-compose file. It is how we run a 3 validator poet integration test. https://github.com/hyperledger/sawtooth-core/blob/master/integration/sawtooth_integration/docker/test_poet_smoke.yaml
boydjohnson (Wed, 16 Aug 2017 18:28:04 GMT):
There are three steps for the poet genesis validator. `sawtooth config genesis`, `poet genesis`, and `sawtooth admin genesis`.
SameerF (Wed, 16 Aug 2017 18:28:34 GMT):
Thanks.
SameerF (Wed, 16 Aug 2017 18:28:34 GMT):
I did try to execute the sawtooth config proposal create command in my container and got the following message.
boydjohnson (Wed, 16 Aug 2017 18:28:53 GMT):
No problem. Ask again if you need more guidance.
Dan (Wed, 16 Aug 2017 20:05:06 GMT):
I'm looking to do some system cleanup on validator shutdown. Is there some event stream I can tap into for that?
Specifically this is for poet consensus. Maybe something the poet block publisher knows about would be handy.
Dan (Wed, 16 Aug 2017 20:56:45 GMT):
meh, nevermind. i think I'd have to plumb threading events up into the consensus interface which would be yucky.
SameerF (Wed, 16 Aug 2017 21:00:25 GMT):
Message Attachments
SameerF (Wed, 16 Aug 2017 21:01:12 GMT):
Am I doing something different?
SameerF (Wed, 16 Aug 2017 21:05:22 GMT):
I did try to execute the sawtooth config proposal command from the yaml file provided above and got the above message.
SameerF (Wed, 16 Aug 2017 21:22:18 GMT):
I also did try to remove begin public key and end public key argument from pem file and executed the command again. Here is the output:
SameerF (Wed, 16 Aug 2017 21:22:52 GMT):
Message Attachments
MikeMayori (Thu, 17 Aug 2017 15:58:37 GMT):
Has joined the channel.
tkuhrt (Tue, 22 Aug 2017 20:05:00 GMT):
Are you joining the training meeting today?
ksachdeva (Wed, 23 Aug 2017 17:08:03 GMT):
Has joined the channel.
JuanmaArcas (Wed, 30 Aug 2017 07:18:21 GMT):
Has joined the channel.
lmrln (Fri, 01 Sep 2017 10:11:03 GMT):
Has joined the channel.
detroitpro (Mon, 11 Sep 2017 17:35:54 GMT):
Has joined the channel.
skbodwell (Mon, 11 Sep 2017 20:42:30 GMT):
Has joined the channel.
Southside (Wed, 13 Sep 2017 23:36:05 GMT):
Has joined the channel.
leogzl (Tue, 19 Sep 2017 11:06:47 GMT):
Has joined the channel.
avi-nyc (Sun, 24 Sep 2017 00:02:19 GMT):
Has joined the channel.
mikykey (Sun, 01 Oct 2017 09:45:10 GMT):
Has joined the channel.
mikykey (Sun, 01 Oct 2017 14:14:08 GMT):
Hello to everybody! I'm reading the documentation but I have a doubt. Is it necessary to run an Intel Processor with SGX or is it possible to use another kind of processor too?
mikykey (Sun, 01 Oct 2017 14:14:08 GMT):
Hello to everybody! I'm reading the documentation but I have a doubt. Is it necessary to run an Intel Processor with SGX or is it possible to use another kind of processor too? Thank you
kelly_ (Mon, 02 Oct 2017 14:07:40 GMT):
Hey @mikykey You do not need an Intel processor for Sawtooth
kelly_ (Mon, 02 Oct 2017 14:08:00 GMT):
However Sawtooth does support a consensus mechanism called PoET that can be run in SGX for more protection
kelly_ (Mon, 02 Oct 2017 14:08:35 GMT):
there is also a version of PoET that runs without SGX, although it has weaker security gaurantees
carlos_frijoles (Mon, 02 Oct 2017 14:14:37 GMT):
Has joined the channel.
mikykey (Tue, 03 Oct 2017 08:41:01 GMT):
Hi @kelly_ thank you very much :) you told me exactly the same things I though :)
lovesh (Tue, 03 Oct 2017 14:02:58 GMT):
Has joined the channel.
kiro112 (Wed, 04 Oct 2017 05:28:11 GMT):
Has joined the channel.
cre8bidio (Wed, 04 Oct 2017 15:37:07 GMT):
Has joined the channel.
parisa (Wed, 11 Oct 2017 17:19:36 GMT):
Has joined the channel.
karthikworks (Fri, 27 Oct 2017 08:39:26 GMT):
Has joined the channel.
sam (Wed, 01 Nov 2017 16:31:46 GMT):
Has joined the channel.
nicolapaoli (Wed, 01 Nov 2017 19:22:51 GMT):
Has joined the channel.
askmish (Thu, 02 Nov 2017 09:58:56 GMT):
Has joined the channel.
teious (Thu, 02 Nov 2017 18:37:01 GMT):
Has joined the channel.
hhimanshu (Sun, 05 Nov 2017 16:36:59 GMT):
Has joined the channel.
cam.ross (Mon, 06 Nov 2017 19:46:11 GMT):
Has joined the channel.
grkvlt (Tue, 07 Nov 2017 13:37:22 GMT):
Has joined the channel.
Southside (Wed, 08 Nov 2017 14:32:59 GMT):
Has anyone done any work regarding alternative consensus mechanisms for Sawtooth?
We are building some prototypes for a fairly large system (100,000 users+) and are looking for possible alternatives should PoET not scale as effectively as we would hope.
All comments suggestions welcome.
adamludvik (Wed, 08 Nov 2017 15:24:58 GMT):
I am currently working on a Raft prototype for Sawtooth.
Southside (Wed, 08 Nov 2017 15:27:06 GMT):
_googles Raft_
adamludvik (Wed, 08 Nov 2017 15:31:53 GMT):
This is really great explainer of how Raft works. http://thesecretlivesofdata.com/raft/
adamludvik (Wed, 08 Nov 2017 15:32:26 GMT):
It is like a simplified Paxos. Or PBFT with the byzantine fault tolerance.
adamludvik (Wed, 08 Nov 2017 15:32:26 GMT):
It is like a simplified Paxos. Or PBFT without the byzantine fault tolerance.
Southside (Wed, 08 Nov 2017 15:43:55 GMT):
tyvm
Jimskie (Wed, 08 Nov 2017 16:51:01 GMT):
Has joined the channel.
srongzhe (Thu, 09 Nov 2017 00:51:48 GMT):
Has joined the channel.
kelly_ (Thu, 09 Nov 2017 15:03:05 GMT):
@southside traditional consensus mechanisms like raft and pbft will likely not be very good for very very large networks
kelly_ (Thu, 09 Nov 2017 15:03:55 GMT):
@Southside you may want to look at some of the recent academic papers like honeybadger BFT or algorand, or the work that dfinity is doing with BLS signatures
GhadiShayban (Thu, 09 Nov 2017 15:54:48 GMT):
Has joined the channel.
GhadiShayban (Thu, 09 Nov 2017 15:55:53 GMT):
algorand is brilliant
hhimanshu (Thu, 09 Nov 2017 16:54:06 GMT):
Has left the channel.
Southside (Thu, 09 Nov 2017 17:25:43 GMT):
yes we were seriously impressed by Algorand earlier this year when we were at a much earlier stage. I was over-ruled on that as it is still low on the Technilogy Readiness Level
Southside (Thu, 09 Nov 2017 17:26:24 GMT):
Has much further work been published on it, other than Silvio Micali's paper?
Southside (Thu, 09 Nov 2017 17:27:45 GMT):
which is seriously academic and as one crypto researcher who I have great respect for put it "takes 70 pages to say what any other person could put across in less than 30"
GhadiShayban (Thu, 09 Nov 2017 17:34:39 GMT):
there is a 24 page pager
GhadiShayban (Thu, 09 Nov 2017 17:34:39 GMT):
there is a 24 page paper
GhadiShayban (Thu, 09 Nov 2017 17:35:00 GMT):
https://people.csail.mit.edu/nickolai/papers/gilad-algorand-eprint.pdf
GhadiShayban (Thu, 09 Nov 2017 17:35:40 GMT):
I agree the 74-pager is a bit much, but i guess it's justified this is a protocol description, not a simple algorithm
GhadiShayban (Thu, 09 Nov 2017 17:35:40 GMT):
I agree the 74-pager is a bit much, but i guess it's justified b/c this is a protocol description, not a simple algorithm
Southside (Thu, 09 Nov 2017 17:42:34 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus?msg=W9cY7YCaccFNubNRr) @kelly_ Thank you
AuHuR (Mon, 13 Nov 2017 10:20:20 GMT):
Has joined the channel.
dkhungme (Thu, 16 Nov 2017 08:22:44 GMT):
Has joined the channel.
chandrairawan (Fri, 17 Nov 2017 03:35:57 GMT):
Has joined the channel.
amolk (Fri, 17 Nov 2017 06:44:27 GMT):
Has joined the channel.
nitefrog (Mon, 20 Nov 2017 09:07:17 GMT):
Has joined the channel.
Timote (Mon, 20 Nov 2017 12:52:25 GMT):
Has joined the channel.
umairmansha (Mon, 20 Nov 2017 13:26:15 GMT):
Has joined the channel.
chandrairawan (Thu, 23 Nov 2017 02:17:06 GMT):
Has left the channel.
vinit_3101 (Fri, 24 Nov 2017 07:49:24 GMT):
Has joined the channel.
ruchit 2 (Fri, 24 Nov 2017 12:32:35 GMT):
Has joined the channel.
ajenie (Sat, 25 Nov 2017 12:39:27 GMT):
Has joined the channel.
Vthot4 (Sat, 25 Nov 2017 20:34:40 GMT):
Has joined the channel.
mikykey (Sun, 26 Nov 2017 07:43:09 GMT):
Has left the channel.
mk3595 (Mon, 27 Nov 2017 21:30:29 GMT):
Has joined the channel.
Johnjam (Tue, 28 Nov 2017 16:06:14 GMT):
Has joined the channel.
GulshanKumar (Thu, 30 Nov 2017 12:34:57 GMT):
Has joined the channel.
GulshanKumar (Thu, 30 Nov 2017 12:35:02 GMT):
hello guys. I really new to Blockchain-Hyperledger and i want learn more about it.
i want to know about consensus protocols used in the various Hyperledger projects
GulshanKumar (Thu, 30 Nov 2017 12:39:24 GMT):
can anyone please help me?
Dan (Thu, 30 Nov 2017 13:46:04 GMT):
@GulshanKumar here's Proof of Elapsed Time consensus: https://sawtooth.hyperledger.org/docs/core/releases/latest/architecture/poet.html
GulshanKumar (Thu, 30 Nov 2017 13:52:02 GMT):
@Dan thank you so much....do u also know about other protocols?
iamprem (Thu, 30 Nov 2017 18:19:07 GMT):
Has joined the channel.
Dan (Thu, 30 Nov 2017 20:33:14 GMT):
this is a foundational paper "PBFT" http://pmg.csail.mit.edu/papers/osdi99.pdf
Dan (Thu, 30 Nov 2017 20:33:49 GMT):
this describes proof of work https://bitcoin.org/bitcoin.pdf
Dan (Thu, 30 Nov 2017 20:33:54 GMT):
happy reading :)
ddcprg (Sat, 02 Dec 2017 18:55:20 GMT):
Has joined the channel.
Johnjam (Sun, 03 Dec 2017 14:19:47 GMT):
Hi everyone, I was reading a document about consensus models (really basic stuff from here : https://www.persistent.com/wp-content/uploads/2017/04/WP-Understanding-Blockchain-Consensus-Models.pdf) and I'm wondering which consensus, between PoET and PoS, has a superior transactions rate and if it's possible to control it ? This document indicates only a "Medium" rate for PoET, some advice ? Thank you !
Dan (Sun, 03 Dec 2017 19:31:01 GMT):
Hi @Johnjam I don't think there are any complete Proof of Stake algorithms yet. The ethereum community has been working on casper for a while but I'm not aware of any breakthroughs yet. Confusingly, they have 2 Casper protocols. One of which will go into production soon, but it's not really proof of stake. It's tag line is "finality gadget".
Dan (Sun, 03 Dec 2017 19:41:01 GMT):
There's some material inaccuracies in the PDF you linked by the way. You might want to research a different source. Maybe look at the papers I listed just above here on Nov 30.
Johnjam (Mon, 04 Dec 2017 08:19:22 GMT):
Thank you @Dan for your precisions ! I'll check your link
reggie (Tue, 05 Dec 2017 10:02:48 GMT):
Has joined the channel.
ken.momma (Tue, 05 Dec 2017 13:48:53 GMT):
Has joined the channel.
vizillion (Thu, 07 Dec 2017 03:23:01 GMT):
Has joined the channel.
van0303 (Fri, 08 Dec 2017 07:43:56 GMT):
Has joined the channel.
eclairamb (Fri, 08 Dec 2017 22:26:26 GMT):
Has joined the channel.
thiago-moreira (Fri, 08 Dec 2017 22:29:13 GMT):
Has joined the channel.
formax (Tue, 12 Dec 2017 22:40:36 GMT):
Has joined the channel.
FaysalAhmed (Thu, 14 Dec 2017 09:51:15 GMT):
Has joined the channel.
iswariyar (Thu, 14 Dec 2017 11:06:35 GMT):
Has joined the channel.
Aires.Marques (Sun, 17 Dec 2017 18:23:02 GMT):
Has joined the channel.
Sam_Liban (Mon, 18 Dec 2017 19:07:22 GMT):
Has joined the channel.
Sam_Liban (Mon, 18 Dec 2017 19:14:49 GMT):
Newb questions: In the documentation found at https://sawtooth.hyperledger.org/docs/core/releases/latest/introduction.html#proof-of-elapsed-time-poet I find the mention of "secure CPU instructions which are becoming widely available in consumer and enterprise processors".
1. Does this refer to TPM (trusted platform modules) or more general cryptographic hardware chips?
2. When the "enclave" is mentioned in the same document, is this then the "instruction" created on these hardware chips as mentioned in the quote above? ("using new secure CPU instructions")
Sam_Liban (Mon, 18 Dec 2017 19:14:49 GMT):
Newb questions: In the documentation found at https://sawtooth.hyperledger.org/docs/core/releases/latest/introduction.html#proof-of-elapsed-time-poet I find the mention of "secure CPU instructions which are becoming widely available in consumer and enterprise processors".
1. Does this refer to TPM (trusted platform modules) or more general cryptographic hardware chips?
2. When the "enclave" is mentioned ("enclave(a trusted function)") in the same document, is this then the "instruction" created on these hardware chips as mentioned in the quote above? ("using new secure CPU instructions") If not, why is it "trusted"?
jmugambi (Mon, 18 Dec 2017 20:50:54 GMT):
Has joined the channel.
Dan (Mon, 18 Dec 2017 21:05:53 GMT):
Hi @Sam_Liban I think that reference is to Trusted Execution Environments (TEEs). One of those is Intel SGX. An enclave is a function or group of functions protected by the TEE.
Dan (Mon, 18 Dec 2017 21:06:31 GMT):
In SGX the enclaves are protected by special instructions that prevent manipulation or reading by other processes including the host OS.
Sam_Liban (Tue, 19 Dec 2017 07:33:13 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus?msg=cJxHvRYKusRj7nr5Z) @Dan So have the issues the BSD devs been resolved or are these referencing another solution from Intel? Its really confusing for me at the moment: While I really love the idea of using the hardware, of course we need to ensure, we can trust the hardware manufacturers...so it would need to be a lot of them and not only Intel, right?
Maybe an open hardware consortia would be an answer to this issue?
This is the article I am referencing here: https://arstechnica.com/information-technology/2013/12/we-cannot-trust-intel-and-vias-chip-based-crypto-freebsd-developers-say/
Thanks for your suppoer, Dan. I appreciate it!
Dan (Tue, 19 Dec 2017 14:59:35 GMT):
Hi @Sam_Liban PoET is designed to explore how to use a trusted execution environment. It has two implementations. One using a simulated enclave with no hardware protections. Kind of a reference implementation that could be applied to any TEE. And one implementation that makes use of SGX. I'm sure the decisions for how and when to trust any vendor are complex for each organization. It's our hope that making this software open source will enable organizations to make more informed decisions, and also help the community to explore this new TEE space.
Sam_Liban (Tue, 19 Dec 2017 15:00:40 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus?msg=CzMAKcB8M8ftGJ9vZ) @Dan Thanks Dan, making it OpenSource is what I needed to relax my brain ;-)
Johnjam (Wed, 20 Dec 2017 14:44:45 GMT):
Hi everyone ! I would like to know if someone has the same problem with the devmode consensus model. I'm able to edit the sawtooth.consensus.max_wait_time and the sawtooth.consensus.min_wait_time records in the settings transaction family, but when I add a new block (with intkey for example), the check function (check_publish_block in sawtooth_validator/journal/consensus/dev_mode/dev_mode_consensus.py) seems to be called before the initialize one (initialize_block in sawtooth_validator/journal/consensus/dev_mode/dev_mode_consensus.py). That way, the check function checks a value from the initalized value from the block - 1. Any clues ? Thanks !
wheeze (Sat, 23 Dec 2017 14:56:58 GMT):
Has joined the channel.
PallabPaul (Mon, 25 Dec 2017 06:50:33 GMT):
Has joined the channel.
PallabPaul (Mon, 25 Dec 2017 08:57:42 GMT):
Hello I hope i am in the right channel. So I am testing out the XO game tutorial found here: https://sawtooth.hyperledger.org/docs/core/releases/latest/app_developers_guide/intro_xo_transaction_family.html#xo-transaction-family-specification and I am running into an issue when I try to create game1. when i do the command 'xo list' game1 does not appear and these are the issues that appear on my validator:
PallabPaul (Mon, 25 Dec 2017 08:57:58 GMT):
Clipboard - December 25, 2017 3:57 AM
PallabPaul (Mon, 25 Dec 2017 08:59:25 GMT):
i think there may be an issue with the ports i am using because after setting up my validator i have to kill port 8080 for my REST API to run properly. please let me know if anyone can help. thanks
RTLagot (Thu, 28 Dec 2017 14:49:37 GMT):
Has joined the channel.
marc0o (Sat, 30 Dec 2017 19:24:35 GMT):
Has joined the channel.
elias_p (Tue, 02 Jan 2018 10:44:05 GMT):
Has joined the channel.
RahulBansal (Thu, 04 Jan 2018 07:35:55 GMT):
Has joined the channel.
davod (Wed, 10 Jan 2018 13:16:15 GMT):
Has joined the channel.
sealthedeal (Wed, 10 Jan 2018 16:09:48 GMT):
Has joined the channel.
sealthedeal (Wed, 10 Jan 2018 16:12:41 GMT):
Hey, I've been deeply interested in Hyperledger Sawtooth lately, but when I was recently asked to explain Proof of Elapsed Time, I wasn't able to explain in detail how it scales comparably to PoW. I've tried to youtube it, but there's little to no resources I've been able to locate that expand on the subject other than "Pluggable election strategy set to a permissioned, voting-based strategy by default that scales well and it byzantine fault tolerant. However, finality may be delayed due to forks that must be resolved."
Dan (Wed, 10 Jan 2018 17:51:37 GMT):
Both are random leader election. The difference is in how you make sure you only get 1 leader in the desired block interval. In PoW the probablility of finding a partial hash collision enforces an exponential distribution of times. In PoET its a trusted timer.
Dan (Wed, 10 Jan 2018 17:59:45 GMT):
@jsmitchell @pschwarz @cianx James mentioned unexpectedly long wait times on the LR networks last month. I put this PR up to limit the population estimates. Upside is it really cuts down wait times. Down side is it really cuts down on wait times. i.e. it shifts the average wait time off of e.g. 30s to 9s.
I can do an uglier hack that will continue to pass the original pop estimates around and should keep the target in tact but I'd rather not if we aren't seeing problematic outliers.
That is I don't want to fix something that isn't broken, nor put in awkward changes to make PoET work smoother on small networks when its really designed for big networks. Notes in: https://github.com/hyperledger/sawtooth-core/pull/1275
islandDeveloper242 (Thu, 11 Jan 2018 17:38:37 GMT):
Has joined the channel.
nycycr (Mon, 15 Jan 2018 00:13:30 GMT):
Has joined the channel.
pekayde (Mon, 15 Jan 2018 09:56:53 GMT):
Has joined the channel.
Rak (Tue, 16 Jan 2018 10:47:21 GMT):
Has joined the channel.
liushengxi (Wed, 17 Jan 2018 12:08:59 GMT):
Has joined the channel.
mehmetshams (Thu, 18 Jan 2018 12:12:56 GMT):
Has joined the channel.
IsraelOyinlola (Thu, 18 Jan 2018 19:48:31 GMT):
Has joined the channel.
arthurgreef (Sat, 20 Jan 2018 23:39:42 GMT):
Has joined the channel.
AmarBagineni (Sun, 21 Jan 2018 06:34:07 GMT):
Has joined the channel.
AmarBagineni (Sun, 21 Jan 2018 06:37:08 GMT):
Hello good people,
AmarBagineni (Sun, 21 Jan 2018 06:37:45 GMT):
i have tried out Hyperledger yesterday, as my dive into implementing blockchain technology
AmarBagineni (Sun, 21 Jan 2018 06:38:28 GMT):
I have an understanding that, the data/payload/transaction gets submitted to the network .. and then it has to be commited
AmarBagineni (Sun, 21 Jan 2018 06:39:14 GMT):
but i am not able to decipher what i am missing, while submitting the batches .. the command "sawtooth block list" does not change
AmarBagineni (Sun, 21 Jan 2018 06:39:27 GMT):
and shows only one row (index = 0)
AmarBagineni (Sun, 21 Jan 2018 06:39:42 GMT):
here is the code in my repo for what i tried - https://github.com/amarbagineni/govt-frame/blob/master/modules/add_block.py
pschwarz (Mon, 22 Jan 2018 15:46:20 GMT):
@AmarBagineni Might be better to try your question in #sawtooth . This channel is more specifically focused on the consensus algorithms used by Sawtooth (e.g. PoET)
RachnaChadha (Fri, 26 Jan 2018 12:51:41 GMT):
Has joined the channel.
RachnaChadha (Fri, 26 Jan 2018 12:54:29 GMT):
I have been trying to understand PoET. Can anyone explain what does wait time mean for nodes ? I am not understanding gist of algorithm but just reading words that are not making sense. Who is creating blocks here ? is TEE on ever node. Is there steps guide for this algorithm?
kelly_ (Fri, 26 Jan 2018 19:34:38 GMT):
@RachnaChadha wait time is how long a node but wait before publishing a block
kelly_ (Fri, 26 Jan 2018 19:34:43 GMT):
each node gets a different wait time
kelly_ (Fri, 26 Jan 2018 19:35:08 GMT):
when the wait time expires, the tee creates a proof that that node is an eligible leader
kelly_ (Fri, 26 Jan 2018 19:35:23 GMT):
that node can package that proof up along with a block of transaction and broadcast to the network
kelly_ (Fri, 26 Jan 2018 19:35:38 GMT):
the rest of the nodes check the transactions and the proof, and if they are valid, it appends it to the blockchain
kelly_ (Fri, 26 Jan 2018 19:35:47 GMT):
at that point the nodes all restart their wait timers again
kelly_ (Fri, 26 Jan 2018 19:36:02 GMT):
every node that wants to generate blocks needs a TEE
RachnaChadha (Fri, 26 Jan 2018 20:15:43 GMT):
@kelly_ Thanks. You said “each node gets a different wait time” who is coming up with wait time a nd what is deriving the wait time ?
kelly_ (Fri, 26 Jan 2018 20:16:41 GMT):
there is more information here - https://sawtooth.hyperledger.org/docs/core/releases/latest/architecture/poet.html
kelly_ (Fri, 26 Jan 2018 20:17:05 GMT):
the short story is that it is a deterministic function based off a unique processor diea
kelly_ (Fri, 26 Jan 2018 20:17:07 GMT):
id*
kelly_ (Fri, 26 Jan 2018 20:18:43 GMT):
https://sawtooth.hyperledger.org/docs/core/releases/latest/architecture/poet.html
kelly_ (Fri, 26 Jan 2018 20:18:46 GMT):
oops thats the same link
kelly_ (Fri, 26 Jan 2018 20:18:47 GMT):
hold on
kelly_ (Fri, 26 Jan 2018 20:19:05 GMT):
https://sawtooth.hyperledger.org/docs/core/releases/latest/architecture/poet.html#createwaittimer-localmean-waitcertid-n
RachnaChadha (Sat, 27 Jan 2018 13:51:32 GMT):
I do not understand how PoET works for forking ? I s there concept if longest chain here as in Pow in bitcoin ? I get that PoET is more capital cost so 51% attack is preventable.What about forking though ? Does it happen ?
Dan (Sun, 28 Jan 2018 19:32:07 GMT):
All random leader election protocols fork. What you want is only short lived forks. PoET has a fork resolution algorithm which is roughly that the chain with the most work and most population wins.
Dan (Sun, 28 Jan 2018 19:32:07 GMT):
All random leader election protocols fork. What you want is only short lived forks (i.e. the network quickly comes into agreement). PoET has a fork resolution algorithm which is roughly that the chain with the most work and most population wins.
shanlusun (Tue, 30 Jan 2018 15:08:22 GMT):
Has joined the channel.
Verticom (Tue, 30 Jan 2018 22:19:43 GMT):
Has joined the channel.
toddinpal (Wed, 31 Jan 2018 00:51:10 GMT):
Has joined the channel.
JOYELIN (Thu, 01 Feb 2018 05:42:00 GMT):
Has joined the channel.
ricardor (Thu, 01 Feb 2018 15:38:29 GMT):
Has joined the channel.
vijayin26 (Fri, 02 Feb 2018 22:45:07 GMT):
Has joined the channel.
vs0901 (Sat, 03 Feb 2018 07:39:22 GMT):
Has joined the channel.
atonit (Sat, 03 Feb 2018 12:26:07 GMT):
Has joined the channel.
nutthasakrotchirat (Tue, 06 Feb 2018 19:18:08 GMT):
Has joined the channel.
keyur (Wed, 07 Feb 2018 05:44:39 GMT):
Has joined the channel.
keyur (Wed, 07 Feb 2018 05:47:37 GMT):
Hi devs and Dan Middleton, thanks for open sourcing an awesome Blockchain project!
I cannot find any documentation around Sawtooth "PoET Simulated" implementation. PoET + SGX would be very secure, but how does PoET Simulated hold up in terms of security? Where can I find out more about security and integrity considerations behind using this as the consensus algo?
I am also wondering whether there are any plans to add Proof-of-Stake (PoS) algo as one of the consensus options in Sawtooth in near future?
Thank you!
keyur (Wed, 07 Feb 2018 05:47:37 GMT):
Hi devs and Dan Middleton, thanks for open sourcing an awesome Blockchain project!
I cannot find any documentation around Sawtooth "PoET Simulated" implementation. PoET + SGX would be very secure, but how does PoET Simulated hold up in terms of security? Where can I find out more about security and integrity considerations behind using this as a consensus algo in a permissionless BC?
I am also wondering whether there are any plans to add Proof-of-Stake (PoS) algo as one of the consensus options in Sawtooth in near future?
Thank you!
Dan (Wed, 07 Feb 2018 17:33:39 GMT):
Hi, good point, I don't recall if we clearly define the distinction somewhere. Basically for simulator the enclave is run in software. It is a separate python implementation from the c++ enclave made for SGX, but the behavior should be the same. So what you forego with the simulated enclave are the trusted execution protections, which reduces the byzantine protections - specifically around fairness. The other poet mechanisms (validator registry, c,z,&K tests) will mitigate or prevent other byzantine attacks.
Dan (Wed, 07 Feb 2018 17:34:48 GMT):
I haven't seen a full proof of stake algorithm yet. Would be interested in adding one when there's a mature one. What I have seen are a few algorithms labeled as proof of stake that are veneers over something like POW or voting.
candidosg (Wed, 07 Feb 2018 20:18:29 GMT):
Has joined the channel.
keyur (Thu, 08 Feb 2018 06:31:14 GMT):
Thanks Dan! What are your thoughts on Ethereum Casper PoS algo in the works? https://github.com/ethereum/wiki/wiki/Proof-of-Stake-FAQ
keyur (Thu, 08 Feb 2018 06:33:41 GMT):
One suggestion I have for the upcoming Hackathon in LA is to have a 15-30 minute walkthrough of PoET: it is an important innovation that needs to be highlighted more.
manxiaqu (Thu, 08 Feb 2018 06:56:27 GMT):
Has joined the channel.
manxiaqu (Thu, 08 Feb 2018 07:19:19 GMT):
Has left the channel.
kakuzu (Thu, 08 Feb 2018 18:44:09 GMT):
Has joined the channel.
Dan (Thu, 08 Feb 2018 21:33:34 GMT):
Since Casper is in the works its kinda hard to say. Which is the problem. I think if they solve it totally it will be a huge breakthrough. Last I checked they had two different things going on though. One part that was still very much in reasearch that was the POS algorithm proper. And the other was what vitalik was calling "Casper the Friendly Finality Gadget" or something like that. The gadget is interesting and useful, but is still really POW with a limit on how big a fork can get.
Dan (Thu, 08 Feb 2018 21:33:57 GMT):
I'll have to read through the entirety of the faq you posted though. probably some new stuff there I'm not up to speed on.
Dan (Thu, 08 Feb 2018 21:34:17 GMT):
Happy to talk about PoET in LA. Feel free to throw it on the community agenda.
Johnjam (Fri, 09 Feb 2018 08:09:00 GMT):
Hi, I post my question in this channel, maybe it's more "consensus specific" : I struggle to understand the real meaning of "local mean" in sawtooth validator code and PoET consensus code. The documentation describe it a little bit in the PoET specifications 1.0 (https://sawtooth.hyperledger.org/docs/core/releases/latest/architecture/poet.html?highlight=local%20mean) but if someone have an article or some explanations, it would be helpful !
Dan (Fri, 09 Feb 2018 13:58:15 GMT):
Hi @Johnjam each node keeps track of the average wait times it sees. Using that average it infers the size of the population of other validators. The next wait times are drawn from a pool that is proportionate to the size of the network population.
When a wait certificate is generated (the certificate is the proof that the node ran the poet protocol correctly) to claim a block, it includes the mean that node computed locally. That localmean then is used by the other validators (along with all the previous localmeans announced in the prior blocks in the chain) to estimate the network population and the whole cycle repeats.
Johnjam (Fri, 09 Feb 2018 15:28:41 GMT):
Thanks for your answer @Dan ! I'm still a litlle bit confused by one thing : how can a node deduct the size of the population of other validators from the average wait times it sees ? And if I understood well, when you precise "average wait time it sees", you mean the localmean in wait certificates sent by other validators ?
MicBowman (Fri, 09 Feb 2018 16:37:20 GMT):
@Johnjam since the wait times are all exponentially distributed... the minimum is also exponential.. we use the minimum wait times as samples of that distribution and "reverse" the process to determine the parameters for the distribution (including population)
Johnjam (Fri, 09 Feb 2018 17:03:13 GMT):
Thank you @MicBowman
mdoan (Sat, 10 Feb 2018 01:44:11 GMT):
Has joined the channel.
Dan (Sun, 11 Feb 2018 17:32:46 GMT):
It's mathemagic
pankajgoyal (Tue, 13 Feb 2018 03:08:56 GMT):
Has joined the channel.
askmish (Tue, 13 Feb 2018 14:25:05 GMT):
:woo:
kakuzu (Tue, 13 Feb 2018 20:09:49 GMT):
:woot:
kakuzu (Tue, 13 Feb 2018 21:05:26 GMT):
Transaction header is signed by the signer’s private key (not sent with the transaction) and the resulting signature is stored in header_signature. this is fine if i do transaction i sign it . The batcher_public_key field must match the public key used to sign the batch in which this transaction is contained. who will sign it. Does it mean single transaction in a batch
JasonG (Wed, 14 Feb 2018 07:33:46 GMT):
Has joined the channel.
JasonG (Wed, 14 Feb 2018 07:34:19 GMT):
Hi I want to ask about PoET consensus algorithm. In Proof of Stake, there's this 'nothing at stake' problem when there's a branch in the blockchain. Namely, the forger can choose to bet the stake at two branches at the same time to maximize his/her own interests. In PoET, we have the same problem -- whenever there's a branch in the blockchain, validators can choose to run PoET using the same SGX hardware on two branches at the same time right? How do we prevent this from happening?
JasonG (Wed, 14 Feb 2018 07:34:45 GMT):
Thank you guys in advance for your help!
JasonG (Wed, 14 Feb 2018 07:38:25 GMT):
Does SGX prevent 'double running' PoET on the hardware? In PoET specification 1.0 https://sawtooth.hyperledger.org/docs/core/releases/latest/architecture/poet.html, which sepcification helps with this?
Dan (Wed, 14 Feb 2018 14:48:25 GMT):
Hi @JasonG branches or rather forks are resolved by the fork resolution portion of the algorithm. In brief, fork resolution will try to pick the fork with the most work on it, and only build new blocks on that fork. All nodes will run the same algorithm and choose the same fork.
Yes, there is a restriction on double running of poet with SGX. it uses monotonic counters. Unfortunately the spec needs to be updated there with the exact approach. But the short version is you bind the validator registration to a hardware counter. if the enclave can't find that counter (because another instance is using it) the enclave won't produce a wait certificate.
JasonG (Wed, 14 Feb 2018 15:47:06 GMT):
@Dan Thank you very much for the answer! What if the malicious entity has multiple validator keys registered? He/She can just bind two keys to run two PoET on the same hardware?
JasonG (Wed, 14 Feb 2018 15:47:06 GMT):
@Dan Thank you very much for the answer! What if the malicious entity has multiple validator keys registered? He/She can then bind two keys to run two PoET on the same hardware?
JasonG (Wed, 14 Feb 2018 15:49:42 GMT):
I thinking of using an open blockchain model where everyone can join and leave the validator pool freely
kelly_ (Wed, 14 Feb 2018 16:37:57 GMT):
@JasonG right now we enforce 1 PoET instance per CPU
kelly_ (Wed, 14 Feb 2018 16:38:18 GMT):
so think about PoET as similar to PoW, but without the operation cost
kelly_ (Wed, 14 Feb 2018 16:39:00 GMT):
so there is still the capital cost to buy the "miner"/CPU, and therefore not a nothing-at-stake problem
kakuzu (Wed, 14 Feb 2018 16:47:44 GMT):
validator in the business network will be trusted nodes so they have brand image and lawsuit at stake :)
kakuzu (Wed, 14 Feb 2018 16:50:33 GMT):
open network is good but with some thing at stake is a good choice. If something valuable is not to lose, people won't fear of losing and becomes bad actors
LuisMarado (Wed, 14 Feb 2018 18:01:52 GMT):
Has joined the channel.
JasonG (Wed, 14 Feb 2018 18:16:35 GMT):
@kelly_ appreciate the explanation! By 'enforce', you mean it can enforced using SGX or it's enforced based on permission system? How do you use SGX to enforce 1 PoET per CPU?
Dan (Wed, 14 Feb 2018 19:05:31 GMT):
using the monotonic counter mechanism i mentioned above
JasonG (Wed, 14 Feb 2018 20:57:46 GMT):
@Dan got it thanks! will check the detailed implementation in the code
kakuzu (Wed, 14 Feb 2018 20:58:40 GMT):
no image is available how can i run seth
kakuzu (Wed, 14 Feb 2018 21:00:51 GMT):
```Pulling seth (sawtooth-seth:latest)...
ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.
Continue with the new image? [yN]y
Pulling seth (sawtooth-seth:latest)...
ERROR: pull access denied for sawtooth-seth, repository does not exist or may require 'docker login'
```
kakuzu (Wed, 14 Feb 2018 21:01:09 GMT):
i already login
Dan (Wed, 14 Feb 2018 21:27:06 GMT):
could you repost to #sawtooth-seth ?
kakuzu (Wed, 14 Feb 2018 21:29:15 GMT):
sorry
Dan (Wed, 14 Feb 2018 21:30:04 GMT):
no worries.. not always clear which channel to post things to.
gurpreetlucky (Fri, 16 Feb 2018 07:18:41 GMT):
Has joined the channel.
ColinAlstad (Fri, 16 Feb 2018 17:10:50 GMT):
Has joined the channel.
ColinAlstad (Fri, 16 Feb 2018 17:17:15 GMT):
Is it possible to test out PoET with SGX without submitting a certificate to Intel?
Dan (Fri, 16 Feb 2018 19:07:39 GMT):
i dont remember if we have links in the docs, but yes you will need a developer credential. that lets you access the attestation verification service.
Dan (Fri, 16 Feb 2018 19:07:56 GMT):
there's no fee or anything for that. its just key management.
TheOnlyJoey (Tue, 20 Feb 2018 13:52:55 GMT):
Has joined the channel.
TheOnlyJoey (Tue, 20 Feb 2018 13:54:13 GMT):
Good day, it seems that I can't find documentation regarding using different consensus on the sawtooth docker, any tips on how to do this? seems to only do dev consensus
TheOnlyJoey (Tue, 20 Feb 2018 14:01:37 GMT):
I would think that the validator would have a API call of some sort where you change out the consensus algorithm to whatever
formax (Tue, 20 Feb 2018 20:35:10 GMT):
@TheOnlyJoey you declare you consensus algo in the Genesis block (you can change it later, however).
The required interfaces to be implemented are here --> https://sawtooth.hyperledger.org/docs/core/releases/latest/architecture/journal.html#the-consensus-interface
and, of course, "Use the source, Luke!" :relaxed:
jsmitchell (Tue, 20 Feb 2018 21:10:40 GMT):
@TheOnlyJoey if you take a look at https://github.com/hyperledger/sawtooth-core/blob/master/integration/sawtooth_integration/docker/test_poet_liveness.yaml you can see an example of the sawset and poet cli commands being used to set the sawtooth.consensus.algorithm to poet and establish other required settings. These are built into the genesis block, but can also be set on the fly while the system is running.
these are on chain settings
look at the 'validator-0' block in the yaml file
grkvlt (Wed, 21 Feb 2018 15:22:34 GMT):
+1 this is what I used to find the commands to start PoET in my Apache Brooklyn catalog entries for Sawtooth networks
nage (Wed, 21 Feb 2018 18:46:56 GMT):
Has joined the channel.
kdenhartog (Wed, 21 Feb 2018 18:47:08 GMT):
Has joined the channel.
TreyZhong (Thu, 22 Feb 2018 04:22:48 GMT):
Has joined the channel.
Gabisan (Thu, 22 Feb 2018 12:24:40 GMT):
Has joined the channel.
kyogesh91 (Thu, 22 Feb 2018 20:28:45 GMT):
Has joined the channel.
GulshanKrish (Mon, 26 Feb 2018 11:46:16 GMT):
Has joined the channel.
brian038 (Mon, 26 Feb 2018 19:18:40 GMT):
Has joined the channel.
pmettu (Mon, 26 Feb 2018 22:46:59 GMT):
Has joined the channel.
azanux (Tue, 27 Feb 2018 01:35:35 GMT):
Has joined the channel.
ttvanderveen (Wed, 28 Feb 2018 09:05:01 GMT):
Has joined the channel.
LaurenNeely (Sun, 04 Mar 2018 21:41:30 GMT):
Has joined the channel.
cuevrob (Mon, 05 Mar 2018 17:49:17 GMT):
Has joined the channel.
Snixells (Tue, 06 Mar 2018 12:04:20 GMT):
Has joined the channel.
eramitg (Tue, 06 Mar 2018 19:04:21 GMT):
Has joined the channel.
SharedMocha (Tue, 06 Mar 2018 23:58:50 GMT):
Has joined the channel.
SharedMocha (Wed, 07 Mar 2018 00:01:55 GMT):
one basic question regarding POET -As i understand POET uses Intel SGX which can work on any Intel chip/processes on servers/machines anywhere. Now, will POET work on non Intel processors -Say for example AMD ?
kelly_ (Wed, 07 Mar 2018 04:28:50 GMT):
@SharedMocha the current implementation is for Intel SGX, but PoET could potentially be implemented on other trusted execution environments besides SGX
cheukchan (Wed, 07 Mar 2018 06:13:36 GMT):
Has joined the channel.
Dan (Wed, 07 Mar 2018 07:53:58 GMT):
@SharedMocha we have two versions of poet. One which is secured with SGX and the other with a simulated enclave. The latter does not require any trusted execution (i.e. does not require SGX) but does not provide BFT fairness.
st (Wed, 07 Mar 2018 14:36:57 GMT):
Has joined the channel.
SharedMocha (Wed, 07 Mar 2018 19:24:02 GMT):
@bill.keller @Dan Thanks for the info
bill.keller (Wed, 07 Mar 2018 19:24:02 GMT):
Has joined the channel.
nbajjuri (Wed, 07 Mar 2018 23:48:36 GMT):
Has joined the channel.
AndrewRy 1 (Thu, 08 Mar 2018 03:25:00 GMT):
Has joined the channel.
Ryan2 (Thu, 08 Mar 2018 03:25:51 GMT):
Has joined the channel.
SmoDav (Thu, 08 Mar 2018 10:40:50 GMT):
Has joined the channel.
abrahamjum (Fri, 09 Mar 2018 00:53:01 GMT):
Has joined the channel.
Gandalf (Fri, 09 Mar 2018 03:43:29 GMT):
Has joined the channel.
Gandalf (Fri, 09 Mar 2018 04:10:15 GMT):
@Dan Is there a reason that PoET is not vulnerable to the type of side channel attacks on SGX? https://www.darkreading.com/vulnerabilities---threats/intel-sgx-can-be-used-to-hide-execute-malware/d/d-id/1331211?
https://www.theregister.co.uk/2017/03/07/eggheads_slip_a_note_under_intels_door_sgx_can_leak_crypto_keys/
vijayendrabvs (Fri, 09 Mar 2018 05:01:09 GMT):
Has joined the channel.
MicBowman (Fri, 09 Mar 2018 15:40:51 GMT):
@Gandalf PoET was designed with the assumption that compromise was possible (even if we didn't know how it could happen). The "z-test" limits the number of times a given validator could win the election
MicBowman (Fri, 09 Mar 2018 15:41:43 GMT):
So long as an attack isn't "scalabe" (meaning, that it could easily be reproduced over a large number of nodes) the z-test provides some probabalistic guarantees of liveness
Gandalf (Fri, 09 Mar 2018 15:50:29 GMT):
Thanks for the reply Mic, but that is the issue, no? The attack referenced above could be scalable, unless it was a very large network of nodes. Even then, once identified, could it not attack the underlying counter mechanism upon which the selector is based. I am thinking not only of double spending type of attacks but denial of service types as well.
Gandalf (Fri, 09 Mar 2018 15:50:29 GMT):
Thanks for the reply Mic, but that is the issue, no? The attack referenced above could be scalable, unless it was a very large network of nodes. Even then, once identified, could it not attack the underlying counter mechanism upon which the selector is based. I am thinking not only of double spending type of attacks but denial of service types as well. Maybe since a sizable number would be running on a simulated enclave, it would be harder to attack both ways at once (different vulnerabilities).
Gandalf (Fri, 09 Mar 2018 15:52:09 GMT):
Also, has their been any throughput testing of PoET running on a AWS type system that we can reference? I am trying to compare it to something like Thunderella, which we are also considering using in our Sawtooth implementation
Gandalf (Fri, 09 Mar 2018 15:52:09 GMT):
Also, has their been any throughput testing of PoET running on a AWS type system that we can reference? I am trying to compare it to something like Thunderella, Honeybadger or Algorand, which we are also considering using in our Sawtooth implementation. Thanks again for the reply.
MicBowman (Fri, 09 Mar 2018 15:56:15 GMT):
Possibly... side channel attacks still require a presence on the server where the attack takes place...
MicBowman (Fri, 09 Mar 2018 15:56:54 GMT):
So having a diverse set of hosting organizations for the validator pool limits the scalability
MicBowman (Fri, 09 Mar 2018 15:57:55 GMT):
Regarding throughput... the sawtooth channel is probably best for that question
MicBowman (Fri, 09 Mar 2018 15:58:53 GMT):
Regarding denial of service... we've run simulations of z-test with up to 30% compromised nodes... throughput (and time to commit transactions) is deeply affected... however, we were able to get about half the blocks from "good" validators
MicBowman (Fri, 09 Mar 2018 16:00:11 GMT):
and obviously... simulated enclaves are MUCH easier to attack... there has to be some kind of trust in rational behavior... but that is true of other consensus algorithms that are being used in HL
Gandalf (Fri, 09 Mar 2018 16:04:13 GMT):
Yes, on the simulated enclave. Not yet sure what standard config would be of our average Validator Node. If on laptop/desktop then SGX is handy, if many are trying to set up on a cheap VPS, then not so sure. Love Sawtooth modularity, btw - not completely sold yet on PoET for our needs.
Gandalf (Fri, 09 Mar 2018 16:05:23 GMT):
It (PoET) truly is a novel idea, however and I suspect that future consensus models will have some sort of hardware component check
Gandalf (Fri, 09 Mar 2018 16:05:23 GMT):
It (PoET) truly is a novel idea, however and I suspect that future consensus models will have some sort of hardware component check. It makes a lot of sense in one way almost as a "two factor" i.e. physical validator. Would LOVE to see something like that at a mobile level. Not sure how it will work in today's cloud environment.
Gandalf (Fri, 09 Mar 2018 16:10:00 GMT):
Thanks for info on your simulations. You say getting 1/2 from good validators. How many does PoET require to finalize/confirm block?
Gandalf (Fri, 09 Mar 2018 16:10:00 GMT):
Thanks for info on your simulations and for taking time to answer my questions. You say getting 1/2 from good validators. How many does PoET require to finalize/confirm block?
MicBowman (Fri, 09 Mar 2018 17:05:28 GMT):
PoET is a form of nakamoto consensus... so the leader proposes the block and others commit implicitly by building on it. When i say 1/2 i mean that half of the blocks in the chain come from good validators... so transactions are accepted "fairly"
MicBowman (Fri, 09 Mar 2018 17:06:23 GMT):
assuming less than 50% compromised... there is no way to incorporate a "bad" transaction into the chain (every good validator rejects a block with bad transactions no matter how it was elected)
MicBowman (Fri, 09 Mar 2018 17:06:43 GMT):
however... a compromised validator could censor good transactions
MicBowman (Fri, 09 Mar 2018 17:06:47 GMT):
that's the power of the leader
MicBowman (Fri, 09 Mar 2018 17:07:11 GMT):
the z-test ensures that, while throughput is degraded, good transactions will eventually be committed
jkmart (Fri, 09 Mar 2018 18:12:12 GMT):
Has joined the channel.
PabitraMartha (Sun, 11 Mar 2018 08:16:08 GMT):
Has joined the channel.
kasmobir (Sun, 11 Mar 2018 16:43:55 GMT):
Has joined the channel.
adityasingh177 (Mon, 12 Mar 2018 05:44:27 GMT):
Has joined the channel.
nissshh (Mon, 12 Mar 2018 14:47:39 GMT):
Has joined the channel.
groussarda (Wed, 14 Mar 2018 18:43:27 GMT):
Has joined the channel.
tubevip (Fri, 16 Mar 2018 03:26:41 GMT):
Has joined the channel.
smgulley (Fri, 16 Mar 2018 14:12:36 GMT):
Has joined the channel.
anrgbndhu (Sun, 18 Mar 2018 07:23:28 GMT):
Has joined the channel.
LeifuChen (Sun, 18 Mar 2018 16:19:33 GMT):
Has joined the channel.
AntoniosHadji (Sun, 18 Mar 2018 16:54:30 GMT):
Has joined the channel.
TomBarnes (Mon, 19 Mar 2018 17:31:29 GMT):
Has joined the channel.
ShikarSharma (Tue, 20 Mar 2018 22:56:15 GMT):
Has joined the channel.
rnagler (Wed, 21 Mar 2018 18:58:12 GMT):
Has joined the channel.
brycebudd (Thu, 22 Mar 2018 21:08:54 GMT):
Has joined the channel.
kasmobir (Thu, 22 Mar 2018 21:13:48 GMT):
Has left the channel.
cheetara (Fri, 23 Mar 2018 14:33:15 GMT):
Has joined the channel.
arthurgreef (Fri, 23 Mar 2018 16:57:20 GMT):
Hi - can someone tell me what constitutes a quorum of validators that agree on the validity of a block? Is it 51% of static peers? Does it change for dynamic peers? Thanks.
Dan (Fri, 23 Mar 2018 18:35:41 GMT):
With PoET it's random leader election not voting. In something like PBFT or raft the nodes 'vote' at a view change (leader election). In PoW, PoET, etc. leaders emerge randomly each round. Other nodes follow that leader using a deterministic mechanism, so that with high probability all nodes follow the same leader.
FrankCastellucci (Fri, 23 Mar 2018 23:29:27 GMT):
Has joined the channel.
aj07 (Sun, 25 Mar 2018 19:56:28 GMT):
Has joined the channel.
saurabharora (Sun, 25 Mar 2018 20:35:19 GMT):
Has joined the channel.
sivo4kin (Mon, 26 Mar 2018 11:35:40 GMT):
Has joined the channel.
Gandalf (Mon, 26 Mar 2018 12:16:32 GMT):
@Dan If I understand it, it is similar to the RAFT protocol with the timer running at the SGX level?
Dan (Tue, 27 Mar 2018 12:56:37 GMT):
More like POW, but without the energy burn.
Wilsonda Fonseca (Tue, 27 Mar 2018 17:24:27 GMT):
Has joined the channel.
pavithrans (Wed, 28 Mar 2018 11:57:53 GMT):
Has joined the channel.
pavithrans (Wed, 28 Mar 2018 12:05:15 GMT):
any idea how to clear this error
unable to prepare context: unable to evaluate symlinks in Dockerfile path: GetFileAttributesEx C:\c: The system cannot find the file specified.
rnagler (Wed, 28 Mar 2018 12:56:09 GMT):
I would like do know, if the current PoET implementation is considered safe regarding new side-channel attacks, see for example:
https://www.theregister.co.uk/2018/03/28/intel_shrugs_off_new_sidechannel_attacks_on_branch_prediction_units_and_sgx/
KOttoni (Wed, 28 Mar 2018 16:57:36 GMT):
Has joined the channel.
obo20 (Fri, 30 Mar 2018 00:02:35 GMT):
Has joined the channel.
obo20 (Fri, 30 Mar 2018 00:05:35 GMT):
are you able to use a difference consensus mechanism like POA in sawtooth?
obo20 (Fri, 30 Mar 2018 00:05:42 GMT):
so far all I've read is about PoET
obo20 (Fri, 30 Mar 2018 00:05:55 GMT):
and I honestly don't trust it
ousooners2834 (Fri, 30 Mar 2018 16:19:56 GMT):
Has joined the channel.
mfoege41 (Sat, 31 Mar 2018 00:30:30 GMT):
Has joined the channel.
Steep (Sat, 31 Mar 2018 11:18:23 GMT):
Has joined the channel.
seet 3 (Sun, 01 Apr 2018 12:16:53 GMT):
Has joined the channel.
Rumeel_Hussain (Mon, 02 Apr 2018 10:45:09 GMT):
Has joined the channel.
audreyle (Mon, 02 Apr 2018 17:16:13 GMT):
Has joined the channel.
hanen (Mon, 02 Apr 2018 23:07:21 GMT):
Has joined the channel.
shonblock (Tue, 03 Apr 2018 15:44:29 GMT):
Has joined the channel.
thomasknoll (Wed, 04 Apr 2018 01:32:46 GMT):
Has joined the channel.
sunil_hyper (Thu, 05 Apr 2018 06:29:24 GMT):
Has joined the channel.
sunil_hyper (Thu, 05 Apr 2018 06:29:38 GMT):
Hi.. I'm Sunil, very new to sawtooth. I have some queries please give answers for it. 1) Which database Sawtooth is using in BCT..? 2) is CA required for it..? 3) can we sawtooth support partial nodes..?
harimohanchoudhary (Thu, 05 Apr 2018 09:29:18 GMT):
Has joined the channel.
cheeky (Thu, 05 Apr 2018 19:58:45 GMT):
Has joined the channel.
tommling (Thu, 05 Apr 2018 21:17:18 GMT):
Has joined the channel.
Dan (Fri, 06 Apr 2018 13:28:00 GMT):
Hi Sunil, this channel is best for consensus questions. For general sawtooth questions #sawtooth is a better channel. I'm not sure what BCT means, but Sawtooth uses the Lightning database. That database is fully managed by sawtooth and users/operators don't need to know its there. 2) CA is not required. All permissioning can be handled on-chain using the identity transaction family. 3) no, all sawtooth nodes are uniform and must operate all services required by the particularly network.
Dan (Fri, 06 Apr 2018 16:45:33 GMT):
@adamludvik do you know what the trigger is for rBFT to publish a block?
adamludvik (Fri, 06 Apr 2018 21:26:29 GMT):
No, I am not yet familiar with rBFT.
adamludvik (Fri, 06 Apr 2018 21:29:21 GMT):
The general impression I have is that most voting based consensus algorithms are described without blockchain specifically in mind, and so there is usually some work that goes into defining that trigger. It probably makes sense to set a timer and max block size and publish whenever one of those thresholds is reached.
sv2011 (Sat, 07 Apr 2018 23:11:52 GMT):
Has joined the channel.
indirajith (Sun, 08 Apr 2018 22:18:27 GMT):
Has joined the channel.
ThangDC (Mon, 09 Apr 2018 07:33:00 GMT):
Has joined the channel.
tommling (Mon, 09 Apr 2018 16:06:23 GMT):
Can someone kindly point me to a guide on quickstarting poet?
tommling (Mon, 09 Apr 2018 16:06:32 GMT):
=]
tommling (Mon, 09 Apr 2018 16:06:46 GMT):
via the simulator is fine
benoit.razet (Mon, 09 Apr 2018 16:41:26 GMT):
@tommling have you looked into https://github.com/hyperledger/sawtooth-core/blob/master/integration/sawtooth_integration/docker/test_poet_liveness.yaml
benoit.razet (Mon, 09 Apr 2018 16:54:23 GMT):
but that's not really a guide
fedotovcorp (Tue, 10 Apr 2018 10:15:55 GMT):
Has joined the channel.
Dan (Wed, 11 Apr 2018 14:24:50 GMT):
@tommling here's some other docker scripts that will spin everything up for you. https://sawtooth.hyperledger.org/docs/core/nightly/master/app_developers_guide/creating_sawtooth_network.html#
You can also use the contents of those to see what to do natively on ubuntu if you prefer.
Dan (Wed, 11 Apr 2018 14:24:50 GMT):
@tommling here's some docker scripts that will spin everything up for you. https://sawtooth.hyperledger.org/docs/core/nightly/master/app_developers_guide/creating_sawtooth_network.html#
You can also use the contents of those to see what to do natively on ubuntu if you prefer.
eramitg (Wed, 11 Apr 2018 15:38:25 GMT):
Hi Folks , I am an Phd Candidate in www.nitrr.ac.in my Linkedind Profile is https://www.linkedin.com/in/eramitg/ for sake of earning an Phd Degree i was proposed Blockchain Technology research work area to my guide so oom I request all of you gyus ,please guide me and assign me some research oriented task so that we mutullay benifited research related to Hyperledger Umbrella Project , All of you feel free to catch me on twitter or skype to https://twitter.com/eramitg1 or amitg.iitb skype id also in Zoom to in Zoom ID 3649222703 or whatsapp +917773011100 Regards
eramitg (Wed, 11 Apr 2018 15:38:25 GMT):
Hi Folks , I am an Phd Candidate in www.nitrr.ac.in my Linkedind Profile is https://www.linkedin.com/in/eramitg/ for sake of earning an Phd Degree i was proposed Blockchain Technology research work area to my guide so I request all of you guys ,please guide me and assign me some research oriented task so that we mutually benefited research related to Hyperledger Umbrella Project , All of you feel free to catch me on twitter or skype to https://twitter.com/eramitg1 or amitg.iitb skype id also in Zoom to in Zoom ID 3649222703 or whatsapp +917773011100 Regards Amit
fahad.suhaib (Thu, 12 Apr 2018 08:10:43 GMT):
Has joined the channel.
WLash (Thu, 12 Apr 2018 10:44:59 GMT):
Has joined the channel.
ShashankChndkr (Thu, 12 Apr 2018 10:46:06 GMT):
Has joined the channel.
ShashankChndkr (Thu, 12 Apr 2018 10:47:29 GMT):
how are smart contracts defined in sawtooth
askmish (Thu, 12 Apr 2018 12:12:10 GMT):
@ShashankChndkr Take a look at this: https://sawtooth.hyperledger.org/docs/core/releases/latest/transaction_family_specifications.html
Samikkumar (Thu, 12 Apr 2018 13:47:49 GMT):
Has joined the channel.
WLash (Thu, 12 Apr 2018 14:17:43 GMT):
Hi, I’m analyzing the PoET for my thesis and I’ve got two questions which I can’t after reading the PoET part in the documentation:```
1. How do the validators decide that an election is over? Do they recognize when all eligible nodes sent a wait certificate or is there a max-wait-time for wait certificates?```
2. How is an election for a new block triggered? Does it start immediately after consensus over the predecessor is reached?
```
```
WLash (Thu, 12 Apr 2018 14:18:26 GMT):
Hi, I’m analyzing the PoET for my thesis and I’ve got two questions which I can’t after reading the PoET part in the documentation:
WLash (Thu, 12 Apr 2018 14:18:30 GMT):
1. How do the validators decide that an election is over? Do they recognize when all eligible nodes sent a wait certificate or is there a max-wait-time for wait certificates?
WLash (Thu, 12 Apr 2018 14:18:40 GMT):
2. How is an election for a new block triggered? Does it start immediately after consensus over the predecessor is reached?
WLash (Thu, 12 Apr 2018 14:18:53 GMT):
I hope someone can help me out :wink:
amolk (Thu, 12 Apr 2018 14:29:56 GMT):
@WLash PoET provides probabilistic finality, so there's always a small chance that a block might come in with a lower overall wait time at a later time. The probability of a block being final increases as more blocks get added to the chain. As soon as a claim block is sent out, the validator starts working on creating the next block.
WLash (Thu, 12 Apr 2018 14:44:57 GMT):
@amolk Thanks for your answer. So in the case the chain has already 567 blocks and the validator receives a claim block for position 566 with a lower waiting time than the current block on this position. The validator simply reacts to it by setting the new block on that position and checks if block 567 is still valid?
amolk (Thu, 12 Apr 2018 14:47:02 GMT):
No, all blocks after 566 will be invalidated because they will now be pointing to an old block. The transactions from 567 will end up in some other block, likely the next one published.
MicBowman (Thu, 12 Apr 2018 14:52:29 GMT):
@WLash i can send you some additional information if it would help
WLash (Thu, 12 Apr 2018 15:09:26 GMT):
@amolk Ok, I think I got it. Thank you :)
zscole (Thu, 12 Apr 2018 21:47:56 GMT):
Has joined the channel.
tommling (Fri, 13 Apr 2018 17:57:06 GMT):
@Dan @benoit.razet Thanks for the links ^^, I find that I don't need actual instances of poet set up yet. I am working on a library that models the expected behavior of a network by enumerating the initial inputs such as sampleLenght, initWaitTime, targetWaitTime, minWaitTime, Vertices, Edges, etc. and calculating the the populationSize / localMean as described in the PoET 1.0 specs. The end goal is to visualize information to pick reasonable parameters for our use case.
tommling (Fri, 13 Apr 2018 17:57:40 GMT):
I am having some issues understanding what the local mean is, besides the average performance of a node's validation times.
tommling (Fri, 13 Apr 2018 17:57:52 GMT):
I've found information about why its useful: `PoET uses the measure of aggregate local mean (a measure of the total amount of time spent waiting) to determine the valid fork,` [1.2 docs. consensus interface].
tommling (Fri, 13 Apr 2018 17:58:53 GMT):
This makes sense, but when I run my models, the local average is always orders of magnitudes larger than the target wait time.
tommling (Fri, 13 Apr 2018 18:00:35 GMT):
*Really, I am looking for insight into how these values are calculated and what the inputs `sw` and `sm` are for calculating populationSize*
tommling (Fri, 13 Apr 2018 18:00:35 GMT):
*Really, I am looking for insight into how and why these values are calculated and what they mean. Is the local mean a measure of all the validator's expected wait time on a specific block slot? Is the populationSize a measure of the effective number of nodes in the network? *
tommling (Fri, 13 Apr 2018 18:02:52 GMT):
I guess this is sw = sampleWait and sm = sampleMean?
sswabhiman (Sun, 15 Apr 2018 15:11:22 GMT):
Has joined the channel.
Dan (Mon, 16 Apr 2018 13:35:47 GMT):
@tommling good question and local mean is kind of the linchpin of poet.
the local mean is the validator's observed wait times in the fork it is operating on. It's used as a mechanism to infer the population. (https://sawtooth.hyperledger.org/docs/core/releases/latest/architecture/poet.html#population-size-and-local-mean-computation)
There's a similarly named metric in the code "aggregate local mean" which is the sum of all the local means of the blocks in a fork. That sum is used to infer the population of participants in a fork.
(Note that the initial wait time is poorly named variable. It is really more like the anticipated local mean once the initial validators are launched. It is meant to help provide a ramp mechanism as a network is starting. It is essentially irrelevant after the ramp time is complete.)
Dan (Mon, 16 Apr 2018 13:35:47 GMT):
@tommling good question and local mean is kind of the lynch pin of poet.
the local mean is the validator's observed wait times in the fork it is operating on. It's used as a mechanism to infer the population. (https://sawtooth.hyperledger.org/docs/core/releases/latest/architecture/poet.html#population-size-and-local-mean-computation)
There's a similarly named metric in the code "aggregate local mean" which is the sum of all the local means of the blocks in a fork. That sum is used to infer the population of participants in a fork.
(Note that the initial wait time is poorly named variable. It is really more like the anticipated local mean once the initial validators are launched. It is meant to help provide a ramp mechanism as a network is starting. It is essentially irrelevant after the ramp time is complete.)
vascosotomaior (Tue, 17 Apr 2018 09:53:11 GMT):
Has joined the channel.
tommling (Tue, 17 Apr 2018 20:39:20 GMT):
@Dan Thanks for the clarification. On an unrelated note, we have been experiencing forks in dev mode running only one validator. This is strange because that means our node is inconsistent with itself (ha-ha...). This is a pastebin with the syslogs, It goes on for hundreds of blocks, and really messes with performance because every block has be compared. https://pastebin.com/t1N05SRd
tommling (Tue, 17 Apr 2018 20:39:20 GMT):
@Dan Thanks for the clarification. On an unrelated note, we have been experiencing forks in dev mode running only one validator. This is strange because that mean the node is inconsistent with itself. I will try to pastebin some logs, but my hunch is it has to do with the scheduler. https://pastebin.com/t1N05SRd
Dan (Wed, 18 Apr 2018 17:32:16 GMT):
That's weird! Everytime a new block comes in it's technically considered as a fork so you'll see messages about forks even when the network isn't forking in the way we'd all think of it. However, that pastebin looks like there's actual deep fork inspections going on.
tommling (Thu, 19 Apr 2018 17:20:22 GMT):
@Dan , this goes on for ~ 80 something blocks, or wherever our chainhead is currently at. The /var/sawtooth data directory may be the culprit, but oddly I do not use a persistent volume in our k8 deployment
jsmitchell (Thu, 19 Apr 2018 21:15:42 GMT):
if you look at it more closely, it's just publishing blocks in sequence and then validating them
jsmitchell (Thu, 19 Apr 2018 21:15:45 GMT):
there are no forks
jsmitchell (Thu, 19 Apr 2018 21:15:55 GMT):
the log message is confusing
jsmitchell (Thu, 19 Apr 2018 21:16:53 GMT):
that pastebin is only 76 lines long though, so maybe you mean to post another one with different behavior?
hviewtiful (Fri, 20 Apr 2018 19:35:37 GMT):
Has joined the channel.
PavelG 1 (Sat, 21 Apr 2018 19:36:59 GMT):
Has joined the channel.
PavelG 1 (Sat, 21 Apr 2018 19:38:31 GMT):
Hi! I'm building a proof of concept where two organizations would have a shared sawtooth network and make their transactions visible to each other. Would like to achieve a throughput of 10 transactions per second and minimize time to commit.
Are there recommendations/good reading on network sizing, PoET config and other considerations for throughput/latency?
Dan (Mon, 23 Apr 2018 17:46:22 GMT):
I don't know if we have good docs out there for sizing but I've heard a few others ask similar questions. In general with a blockchain the number of nodes should match (or be a multiple of) the number of participating companies. PoET is designed for large networks. If you just have two organizations that doesn't warrant a lot of nodes.
LoveshHarchandani_1 (Wed, 25 Apr 2018 15:12:04 GMT):
Has joined the channel.
IVictorFeng (Thu, 26 Apr 2018 02:33:41 GMT):
Has joined the channel.
obo20 (Sat, 28 Apr 2018 17:49:18 GMT):
Has left the channel.
MabelOza (Sun, 29 Apr 2018 17:20:25 GMT):
Has joined the channel.
bodhisha (Wed, 02 May 2018 14:58:33 GMT):
Has joined the channel.
jeremie.inblocks (Wed, 02 May 2018 15:30:43 GMT):
Has joined the channel.
JCourech (Thu, 03 May 2018 04:33:29 GMT):
Has joined the channel.
Gabisan (Fri, 04 May 2018 08:50:13 GMT):
Regarding PoET certification for deploying in production a multi-node, multi-companies oragnization: at what level do we need to obtain these certificates? at node, company or organization level?
I couldn't find any reference if anybody could guide me or answer this please.
Dan (Fri, 04 May 2018 15:57:19 GMT):
I think logically it would be the consortium level and practically it would be one company (or if the consortium is a legal entity, then the consortium itself) would sign the enclave. And the network would only trust that identity for enclave validity.
Gabisan (Fri, 04 May 2018 16:50:47 GMT):
Ok, thank you!
tungdt_socoboy (Sat, 05 May 2018 11:39:45 GMT):
Has joined the channel.
pravindia (Sun, 06 May 2018 17:09:45 GMT):
Has joined the channel.
dsl (Mon, 07 May 2018 05:53:35 GMT):
Has joined the channel.
ltseeley (Tue, 08 May 2018 17:20:52 GMT):
Has joined the channel.
ThangDC (Wed, 09 May 2018 05:09:02 GMT):
Has left the channel.
nbajjuri (Wed, 09 May 2018 17:18:48 GMT):
hi
nbajjuri (Wed, 09 May 2018 17:18:48 GMT):
is there any documentation about consenus? like how sawtooth consensus work
Dan (Wed, 09 May 2018 18:54:19 GMT):
Hi Sawtooth provides Dynamic Consensus - the ability to change consensus on the fly. We currently provide devmode targeted at app development and two flavors of PoET. Work is in progress on adding additional consensus algorithms to be released later this year.
A slightly dated post of Dynamic Consensus (originally called unpluggable consensus) https://www.hyperledger.org/blog/2017/11/22/un-pluggable-consensus-with-hyperledger-sawtooth
And this is specifically how PoET works:
https://sawtooth.hyperledger.org/docs/core/nightly/master/architecture/poet.html
lucienlu (Thu, 10 May 2018 11:58:17 GMT):
Has joined the channel.
temas (Thu, 10 May 2018 15:55:55 GMT):
Has joined the channel.
nbajjuri (Thu, 10 May 2018 16:37:08 GMT):
thanks dan
mastys100 (Sat, 12 May 2018 22:23:53 GMT):
Has joined the channel.
varinder (Tue, 15 May 2018 00:03:38 GMT):
Has joined the channel.
varinder (Tue, 15 May 2018 00:06:35 GMT):
I am quite new to Sawtooth architecture but I have experience with Fabric network and development consensus methods. My question is does Sawtooth require a certain processor for it to be deployed on a network? I ask because I understand it uses a specific SGX extension. Does that mean the consensus is limited to those processors that allow this feature?
sauravverma (Tue, 15 May 2018 10:31:36 GMT):
Has joined the channel.
Dan (Tue, 15 May 2018 13:00:30 GMT):
@varinder that is a common question :). SGX is only needed if you use the hardened version of PoET. We also have a version of PoET that just uses conventional software. There is a third consensus, devmode, which developers use across small networks. We also are in the process of adding a fourth consensus, RAFT and may add other consensus engines this year.
RyanBennett (Tue, 15 May 2018 17:05:01 GMT):
Has joined the channel.
paul.sitoh (Wed, 16 May 2018 16:20:42 GMT):
Has joined the channel.
harper58 (Thu, 17 May 2018 02:50:23 GMT):
Has joined the channel.
svanschalkwyk (Fri, 18 May 2018 21:11:21 GMT):
Has joined the channel.
RichardHill (Tue, 22 May 2018 19:37:16 GMT):
Has joined the channel.
jeffcoop9 (Tue, 22 May 2018 23:40:41 GMT):
Has joined the channel.
st (Wed, 23 May 2018 07:09:33 GMT):
Hello to all, I want to experiment a little bit with consensus, which docker image files should I get in order to have a working network of two or more nodes with PoET consensus? (no CPU SGX support, I need the conventional version)
Gandalf (Wed, 23 May 2018 14:56:22 GMT):
@Dan Raft seems very similar to PoET in some ways in its leader selection, with PoET using a (theoretically independent) CPU(virtualized) clock. Raft is noted to provide good crash protection but no byzantine of course - its also easier to implement than most any paxos flavor. Is there any chat or git where there is more discussion on the work that is being done to implement?
askmish (Wed, 23 May 2018 15:06:17 GMT):
@st Here's the poet setup info: https://sawtooth.hyperledger.org/docs/core/nightly/master/app_developers_guide/creating_sawtooth_network.html
kelly_ (Wed, 23 May 2018 16:50:02 GMT):
@Gandalf raft and poet differ a bit in leader election and some of the properties
kelly_ (Wed, 23 May 2018 16:51:14 GMT):
I believe the RAFT implementation is starting in the next few weeks now that the new consensus interface is almost complete
kelly_ (Wed, 23 May 2018 16:51:23 GMT):
this would be the channel to discuss that though
Dan (Wed, 23 May 2018 18:02:33 GMT):
Please join the Hyperledger Sawtooth Technical Forum on Thursday, May 24th at 10am CDT for the following discussion:
PoET 2.0 Preview (Ashish Mishra)
Join from PC, Mac, Linux, iOS or Android:
https://zoom.us/my/hyperledger.community
SjirNijssen (Wed, 23 May 2018 19:49:33 GMT):
Has joined the channel.
kc1116 (Wed, 23 May 2018 19:50:09 GMT):
Has joined the channel.
kc1116 (Wed, 23 May 2018 19:51:16 GMT):
Anyone know if any docs exist describing how to plug and play the consensus
Dan (Thu, 24 May 2018 13:49:44 GMT):
There's a stale RFC for the new consensus engine interface. I think it gives the conceptual ideas but won't be prescriptive on the interface. Are you looking at adding a consensus protocol? https://github.com/hyperledger/sawtooth-rfcs/pull/4/files
kc1116 (Thu, 24 May 2018 14:18:04 GMT):
@Dan was thinking about how a classic POW would work
kc1116 (Thu, 24 May 2018 14:18:39 GMT):
I thought Sawtooth came out the box with a bunch of different consensus algo's ready
Dan (Thu, 24 May 2018 14:24:10 GMT):
yeah you can just switch consensus using settings transactions
kc1116 (Thu, 24 May 2018 14:28:55 GMT):
Yea but that doesn't work unless interface is implemented correct ?
kc1116 (Thu, 24 May 2018 14:29:11 GMT):
and currently POET is only interface besides dev mode correct ?
kc1116 (Thu, 24 May 2018 14:30:13 GMT):
Also why can't people comment on that RFC ? I have a bunch of comments and questions, is it being discussed anywhere ?
kc1116 (Thu, 24 May 2018 14:31:56 GMT):
nvm just noticed, theres a entire RFC process
kelly_ (Thu, 24 May 2018 20:09:14 GMT):
@kc1116 here is probably a fine place to discuss it
kelly_ (Thu, 24 May 2018 20:09:18 GMT):
or the mailing list
kelly_ (Thu, 24 May 2018 20:09:25 GMT):
I can see if it is supposed to be open for comment
kelly_ (Thu, 24 May 2018 20:09:43 GMT):
also there is a RAFT prototype that is being modified for the new interface
kelly_ (Thu, 24 May 2018 20:09:48 GMT):
so that will give 4 consensus options
kelly_ (Thu, 24 May 2018 20:09:59 GMT):
devmode, PoET, PoET-SGX, RAFT
kelly_ (Thu, 24 May 2018 20:10:29 GMT):
would be cool to see an POW style added to the option
Gandalf (Thu, 24 May 2018 20:58:34 GMT):
@kelly_ Cant see why anyone would add a PoW to the mix. Seriously doubt any of the currently operating blockchains would launch a PoW chain if they could start afresh today. Some DPoS perhaps like Algorand would be more to my liking - or the ability to shift algos a la Thunderella - running RAFT until leader corruption and then it shifts to something with BFT like tendermint or even PoET 2.0 - how you are running network (public, private, consortium) is a big factor tho on optimal consensus config
kelly_ (Thu, 24 May 2018 22:05:18 GMT):
agreed
kelly_ (Thu, 24 May 2018 22:05:41 GMT):
there is one place that I think PoW makes sense, and that's for public networks that don't have an initial coin distribution
kelly_ (Thu, 24 May 2018 22:09:31 GMT):
i guess also for large scale public networks where you don't want to relay on some set of staking nodes and don't trust SGX
kelly_ (Thu, 24 May 2018 22:09:42 GMT):
though at this point staking seems to be more decentralized than mining
kc1116 (Fri, 25 May 2018 14:57:51 GMT):
@kelly_ @Gandalf Yes Kelly I think for public networks with no coin distribution this makes sense because it should be easy using sawtooth to then move the chain to one of the stake based options. Also if this also makes sense for completeness, unless Sawtooth is going to be a very opinionated framework
sanjeet_kumar (Mon, 28 May 2018 11:30:17 GMT):
Has joined the channel.
donatopellegrino (Mon, 28 May 2018 13:26:04 GMT):
Has joined the channel.
BALACHUPPALA (Tue, 29 May 2018 06:45:51 GMT):
Has joined the channel.
AlexanderZhovnuvaty (Tue, 29 May 2018 14:04:59 GMT):
Has joined the channel.
jeffcoop9 (Tue, 29 May 2018 16:05:53 GMT):
Has left the channel.
sidhujag (Thu, 31 May 2018 23:57:56 GMT):
Has joined the channel.
sidhujag (Fri, 01 Jun 2018 00:15:51 GMT):
hi guys for permissionless setup does this mean that you do not need to setup a whitelist for validators and validators can come and go without affecting the consensus algo in terms of orphans etc?
sidhujag (Fri, 01 Jun 2018 00:18:30 GMT):
@gandalf the only way to go for POW is merged mining atm and otherwise would have to be a fault tolerant consensus that is probably PoW/PoS hybrid. PoW is still cheaper than PoS in the game theoretical sense as paul sztorc highlighted http://www.truthcoin.info/blog/pow-cheapest/
sidhujag (Fri, 01 Jun 2018 00:18:56 GMT):
For me its either TEE completely or PoW+PoS hybrid with fraud proof design
Jasminess (Fri, 01 Jun 2018 06:30:37 GMT):
Has joined the channel.
Aswath8687 (Sat, 02 Jun 2018 20:21:52 GMT):
Has joined the channel.
quan2d (Mon, 04 Jun 2018 02:40:36 GMT):
Has joined the channel.
Dan (Mon, 04 Jun 2018 13:37:15 GMT):
@askmish Here's a consideration for you as you write poet2: https://jira.hyperledger.org/browse/STL-963. Didn't get around to doing this with PoET1 before we released sawtooth 1.0 and now it's pretty hard to make a change that is both helpful and backwards compatible. In a nutshell be wary how you initialize default values or whether you have default values at all or whether you can change default values.
BlackCat1397 (Wed, 06 Jun 2018 12:48:20 GMT):
Has joined the channel.
shresthi (Wed, 06 Jun 2018 12:49:29 GMT):
Has joined the channel.
BlackCat1397 (Wed, 06 Jun 2018 12:50:06 GMT):
Hello! Is there any plans about implementing PoET for ARM TrustZone?
Dan (Wed, 06 Jun 2018 13:17:56 GMT):
Always welcome to contributions
BlackCat1397 (Wed, 06 Jun 2018 16:16:26 GMT):
Any tutorials about replacing consensus?
bebinu (Wed, 06 Jun 2018 21:48:19 GMT):
Has joined the channel.
bebinu (Wed, 06 Jun 2018 21:48:29 GMT):
tendermint
bebinu (Wed, 06 Jun 2018 21:57:02 GMT):
Hi all, I came across the following JIRA ticket https://jira.hyperledger.org/browse/STL-725 that mentions the following "Produce a design document which describes an overall approach for integrating Tendermint consensus with Sawtooth for the purposes of providing a PBFT-like consensus module, while continuing to support PoET/SGX, PoET/Simulator, and devmode consensus modules." Was this document or a write up of the findings published anywhere? If so, where is it?
Dan (Wed, 06 Jun 2018 22:13:55 GMT):
kind of. there's a sawtooth-rfc up for a consensus interface that is intended to improve on the existing consensus abstraction. https://github.com/hyperledger/sawtooth-rfcs/pull/4/files
bebinu (Wed, 06 Jun 2018 22:39:31 GMT):
@Dan , thanks, I'll have a read. Do you know if any of the individuals involved in that investigation (e.g. Adam Ludvik or Mark Ford) hang out on RocketChat? I'd love to have a conversation with them about their findings, assuming they did the work.
Johannes2511 (Thu, 07 Jun 2018 09:58:27 GMT):
Has joined the channel.
Dan (Thu, 07 Jun 2018 12:44:58 GMT):
Yep. They are all still very involved. I don't have links handy but there's PRs and a branch on sawtooth core developing to that RFC (or some evolution there of).
amundson (Thu, 07 Jun 2018 18:27:18 GMT):
@bebinu there is a lot of development happening on that front right now. any specific questions?
adamludvik (Thu, 07 Jun 2018 18:31:37 GMT):
@bebinu Hi, I am the Adam Ludvik you are interested in :) What's up?
bridgerherman (Thu, 07 Jun 2018 18:36:29 GMT):
Has joined the channel.
abraham (Fri, 08 Jun 2018 04:53:51 GMT):
Has joined the channel.
rys_rahul (Fri, 08 Jun 2018 09:04:31 GMT):
Has joined the channel.
Johannes2511 (Fri, 08 Jun 2018 15:53:16 GMT):
How can I use the simulated PoET with a docker/virtual machine network? What is the maximum network size, which I am able to emulate?
Dan (Fri, 08 Jun 2018 16:17:26 GMT):
We chose a bad word for PoET simulator as it seems to cause confusion. The enclave (the code that would run in trusted execution) runs in normal memory and so that trusted execution is simulated. That's really the only difference.
Dan (Fri, 08 Jun 2018 16:17:43 GMT):
And you can run as many as you like. See how many you can add before it breaks :)
Dan (Fri, 08 Jun 2018 16:18:15 GMT):
For our continuous integration testing we typically launch networks with 10 nodes and 30s interblock times (== target wait time).
sidhujag (Fri, 08 Jun 2018 20:46:08 GMT):
how would tendermint consensus work in conjunction with TEE's?
sidhujag (Fri, 08 Jun 2018 20:46:53 GMT):
would it be for example useful to have a PBFT to control the account states for EVM while the execution can be through SGX?
sidhujag (Fri, 08 Jun 2018 20:48:59 GMT):
if you have PBFT in which context does using SGX make sense then as you are then going towards the fraud proof model which has other game theoretical implications
sidhujag (Fri, 08 Jun 2018 20:49:50 GMT):
can you somehow do TEE through a web-of-trust model or perhaps through a PBFT consensus to decide which TEE's are valid?
Dan (Fri, 08 Jun 2018 20:49:54 GMT):
the consensus has to be designed to use the TEE.
sidhujag (Fri, 08 Jun 2018 20:51:18 GMT):
ok can you step me through what you meant by saying it has to be designed to use the TEE, doesn't that mean your common denominator becomes TEE because its a trusted setup, why the need for PBFT, in which context does it make sense?
Dan (Fri, 08 Jun 2018 23:56:10 GMT):
If you want to take advantage of a tee your software component has to be designed for it. There isn't an automatic protection that comes from using a tee. Also in case this is a point of confusion Sawtooth does not require a tee it just happens that we have a component that is designed to take advantage of it in poet.
Johannes2511 (Sat, 09 Jun 2018 10:47:34 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus?msg=JQgZnhPxtatq8MsJQ) @Dan Where can I see *all* settings related to configuring the PoET enclave? I was targeting roughly 1 new block / 5sec, and ~100+ nodes. Do you have recommendations regarding maximum batch/transaction size
Dan (Sun, 10 Jun 2018 17:52:36 GMT):
These might be in the docs somewhere but you can see them explicitly here:
Dan (Sun, 10 Jun 2018 17:52:40 GMT):
https://github.com/hyperledger/sawtooth-core/blob/master/consensus/poet/core/sawtooth_poet/poet_consensus/poet_settings_view.py
Dan (Sun, 10 Jun 2018 17:53:32 GMT):
I'd set target wait time to 30s (default is 20s and assumes smaller networks). If you set your interblock time too small you will have many leaders each round resulting in more forks which will end up slowing the network down.
Dan (Sun, 10 Jun 2018 17:54:23 GMT):
I don't have recommendations on max batch or txn size.
Johannes2511 (Mon, 11 Jun 2018 07:10:03 GMT):
Thanks :-)
Johannes2511 (Mon, 11 Jun 2018 07:19:02 GMT):
Currently, I tried the following:
`sawtooth.consensus.algorithm=poet
sawtooth.poet.report_public_key_pem="$(cat /etc/sawtooth/simulator_rk_pub.pem)"
sawtooth.poet.valid_enclave_measurements=$(poet enclave measurement)
sawtooth.poet.valid_enclave_basenames=$(poet enclave basename)
sawtooth.poet.target_wait_time=5
sawtooth.poet.initial_wait_time=5
sawtooth.publisher.max_batches_per_block=100`
For validation, I ran in a single node (validator with REST and settings+poet+intkey TPs) `intkey set foobar` and afterwards manually ran `intkey inc foobar 1` until I had 50 blocks (==population_estimate_sample_size) in the blockchain. Therefore, "bootstrapping" was finished. I wanted to determine, if it is possible to roughly append a block every 5-10 seconds and I ran: `watch -n 6 'intkey inc foobar 1'`, which would try to increase the foobar value by one every six seconds. Unfortunately, I ran into severe problems: the validator outputs `Consensus not ready to build candidate block` every 100 ms! I am no longer able to submit *any* transaction. The same behavior is reproducible and also occures, if i use two nodes (same configuration as single node setup).
Johannes2511 (Mon, 11 Jun 2018 07:19:02 GMT):
Currently, I tried the following with the following settings:
```sawtooth.consensus.algorithm=poet
sawtooth.poet.report_public_key_pem="$(cat /etc/sawtooth/simulator_rk_pub.pem)"
sawtooth.poet.valid_enclave_measurements=$(poet enclave measurement)
sawtooth.poet.valid_enclave_basenames=$(poet enclave basename)
sawtooth.poet.target_wait_time=5
sawtooth.poet.initial_wait_time=5
sawtooth.publisher.max_batches_per_block=100```
For validation, I ran in a single node (validator with REST and settings+poet+intkey TPs) `intkey set foobar` and afterwards manually ran `intkey inc foobar 1` until I had 50 blocks (==population_estimate_sample_size) in the blockchain. Therefore, "bootstrapping" was finished. I wanted to determine, if it is possible to roughly append a block every 5-10 seconds and I ran: `watch -n 6 'intkey inc foobar 1'`, which would try to increase the foobar value by one every six seconds. Unfortunately, I ran into severe problems: the validator outputs `Consensus not ready to build candidate block` every 100 ms! I am no longer able to submit *any* transaction. The same behavior is reproducible and also occures, if i use two nodes (same configuration as single node setup).
Johannes2511 (Mon, 11 Jun 2018 09:22:55 GMT):
hyperledger_sawtooth001_11_06_2018_11_22_17.png
Johannes2511 (Mon, 11 Jun 2018 11:35:54 GMT):
Ok. This behavior occus after a validator claims blocks too frequently (see "last valid" log msg below). So far, I couldn't find recovery from this state. Is this a bug or intendet?
```[2018-06-11 13:05:54.987 INFO poet_block_publisher] Reject building on block 52779e59: Validator is claiming blocks too frequently.`
Johannes2511 (Mon, 11 Jun 2018 11:35:54 GMT):
Ok. This behavior occus after a validator claims blocks too frequently (see "last valid" log msg below). So far, I couldn't find recovery from this state. Is this a bug or intendet?
```[2018-06-11 13:05:54.987 INFO poet_block_publisher] Reject building on block 52779e59: Validator is claiming blocks too frequently.```
Dan (Mon, 11 Jun 2018 14:25:00 GMT):
claiming too frequently is a security check. if you want to disable it you need to bump up the z test values
Dan (Mon, 11 Jun 2018 14:25:31 GMT):
If a validator can't claim blocks it's because it's registration didn't get in.
Dan (Mon, 11 Jun 2018 14:25:53 GMT):
the startup phase can be kind of sensitive
Johannes2511 (Mon, 11 Jun 2018 15:07:26 GMT):
Do you mean `sawtooth.poet.ztest_maximum_win_deviation`and `sawtooth.poet.ztest_maximum_win_count` settings? I assume those parameters are dependent on the network size or are they only relevant for the startup phase?
Dan (Mon, 11 Jun 2018 15:11:19 GMT):
min win count
Dan (Mon, 11 Jun 2018 15:11:28 GMT):
if you make that a huge number the test will never trip
Dan (Mon, 11 Jun 2018 15:11:55 GMT):
you can also make win deviation huge, but it's asymptotic so it's harder to work with
Johannes2511 (Mon, 11 Jun 2018 15:12:44 GMT):
Again thanks for that quic response :-)
Johannes2511 (Mon, 11 Jun 2018 15:13:01 GMT):
*quick
Johannes2511 (Mon, 11 Jun 2018 15:13:52 GMT):
`max_win_count` defaults to 3 wins. Are there any experiments variying the size of that Score? Are there already some research papers regarding that PoET algorithm?
Johannes2511 (Mon, 11 Jun 2018 15:18:00 GMT):
Maybe some links to further literature would helpful in the documentation.
Johannes2511 (Mon, 11 Jun 2018 15:18:28 GMT):
I will try to set max_win_count to some larger number for now :-)
Dan (Mon, 11 Jun 2018 15:23:08 GMT):
_ZTEST_MINIMUM_WIN_COUNT_ (min means that at least that many blocks must have been claimed before we start testing for statistically improbable winning)
Dan (Mon, 11 Jun 2018 15:23:08 GMT):
`_ZTEST_MINIMUM_WIN_COUNT ` (min means that at least that many blocks must have been claimed before we start testing for statistically improbable winning)
Dan (Mon, 11 Jun 2018 15:28:12 GMT):
there's also `_KEY_BLOCK_CLAIM_LIMIT_` which you may also want to make big. That one controls how often a validator has to re-register. If you have a small network you can deadlock it if all the validators exhaust their block claim limits sort of simultaneously.
rranjan3 (Mon, 11 Jun 2018 17:01:05 GMT):
Has joined the channel.
JonathanLevi (Mon, 11 Jun 2018 17:11:57 GMT):
Has joined the channel.
markg 17 (Mon, 11 Jun 2018 22:35:25 GMT):
Has joined the channel.
rkrish82 (Tue, 12 Jun 2018 02:58:08 GMT):
Has joined the channel.
ravipatisivaramaprasad (Tue, 12 Jun 2018 04:42:08 GMT):
Has joined the channel.
ravipatisivaramaprasad (Tue, 12 Jun 2018 05:07:18 GMT):
@Dan Hi. I want to deploy my sawtooth application on two node sawtooth network(Docker version). Initially, my compose file is as https://pastebin.com/raw/2Xs6JLLv. Everything working fine when I did compose-up and transaction initiation.
Test case 1 : I just did stop&start of validator-1 container. After I started validator-1 container it is just rebuilding its DB from Validator-0 and continuing with its further functionality once I submit the transaction batch.
Test case 2 : I just did stop&start of validator-0(seed) container. After I started validator-0 container it is just exited. Generally, validator-0 has to rebuild its DB from validator-1 but that doesn't happened. Because when I started Validator-0 it is trying to create genesis block again and at the same time it is rebuilding its DB from validator-1. So genesis block conflict occurred.
To resolve this, I just tried to externalize the validator-0 DB by modifying the compose file as https://pastebin.com/raw/BpuBPVYF. I just written a script that will do compose-up. Please first look at the script file[https://pastebin.com/raw/Fsa5VSze] and then look at compose file[https://pastebin.com/raw/BpuBPVYF]. I just have executed script file then in console the control stuck at a particular point as mentioned in the below snippet.
ravipatisivaramaprasad (Tue, 12 Jun 2018 05:07:53 GMT):
Clipboard - June 12, 2018 10:37 AM
ravipatisivaramaprasad (Tue, 12 Jun 2018 05:09:22 GMT):
Validator-0 log file /sawtooth/logs/validator-debug.log is showing the below debug messages. `Consensus not ready to build candidate block`
ravipatisivaramaprasad (Tue, 12 Jun 2018 05:10:24 GMT):
Clipboard - June 12, 2018 10:39 AM
ravipatisivaramaprasad (Tue, 12 Jun 2018 05:11:53 GMT):
@Dan I stuck at this particular point. I am debugging. But I didn't get the reason why `Consensus not ready to build candidate block`. Can you please help me in resolving this issue.
ravipatisivaramaprasad (Tue, 12 Jun 2018 05:31:15 GMT):
Clipboard - June 12, 2018 10:59 AM
tauras (Tue, 12 Jun 2018 06:32:36 GMT):
Has joined the channel.
tauras (Tue, 12 Jun 2018 06:33:02 GMT):
I would like to implement a *Will application*, which would over time trigger some functionality on its own and not based on any transaction. i.e After 25 years the ownership of the asset need to be changed to the in-laws. Is it possible to perform such kind of *Autonomous event* in sawtooth, which is purely automatic and nothing driven based on a transaction?
tauras (Tue, 12 Jun 2018 06:33:02 GMT):
Hi all,
I would like to implement a *Will application*, which would over time trigger some functionality on its own and not based on any transaction. i.e After 25 years the ownership of the asset need to be changed to the in-laws. Is it possible to perform such kind of *Autonomous event* in sawtooth, which is purely automatic and nothing driven based on a transaction?
amundson (Tue, 12 Jun 2018 13:58:27 GMT):
@tauras you can send/inject a transaction periodically to do that, either via a client or via the batch injector; unfortunately, the batch injector's API is not currently exposed (it is used in the block info txn family). The transaction submitted would look through the wills and update them appropriately.
Dan (Tue, 12 Jun 2018 19:07:39 GMT):
@ravipatisivaramaprasad usually that error is a sign that something broke in the registration process. That might have happened if you've lost files the container was managing (i.e. the location of that file wasn't mapped to the host file system so you just had ephemeral storage in the container.) I would need the log file to know more. Rather than dig through the log though, make sure you have a good reason to run the validator in a container (or are you solving a problem you wouldn't have in production) and if you do then make sure that you are mapping all of the volumes.
tauras (Wed, 13 Jun 2018 03:35:42 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus?msg=kLmbRMK846ME4xXZr) @amundson
Ok thanks but could you please elaborate on how a client side or batch injector call can* look through the wills*?Is there any filetered api for retrieving the transactions based on its value?
tauras (Wed, 13 Jun 2018 03:35:42 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus?msg=kLmbRMK846ME4xXZr) @amundson
Ok thanks but could you please elaborate on how a client side or batch injector call can *look through the wills* ?Is there any filetered api for retrieving the transactions based on its value?
Rekha (Wed, 13 Jun 2018 07:22:03 GMT):
Has joined the channel.
amundson (Wed, 13 Jun 2018 08:39:58 GMT):
@tauras we should take this to #sawtooth since it's not a consensus topic
ravipatisivaramaprasad (Wed, 13 Jun 2018 11:15:53 GMT):
@Dan Ok, Thank you.
csengor (Thu, 14 Jun 2018 05:18:39 GMT):
Has joined the channel.
ravipatisivaramaprasad (Thu, 14 Jun 2018 07:20:56 GMT):
@Dan Hi. I just have understood the multi-node sawtooth network workflow as mentioned below. Please see and provide your comments on my understanding.
ravipatisivaramaprasad (Thu, 14 Jun 2018 07:21:02 GMT):
Sawtooth client submits transaction batch to the validator through the corresponding REST-API
PoET Consensus takes the submitted batch of transactions and builds a candidate block. Once candidate blocks are completed, they are delivered to the Chain Controller for validation and fork resolution.
Each validator within the network requests a wait time from an enclave, or a trusted function. This is where the 'Elapsed Time' comes into play.
The validator with the shortest wait time for a specific block is appointed the leader, and creates the block to be committed to the ledger.
The transaction validator with the shortest wait time publishes the transaction block.
The transaction block is then broadcast to the publishing nodes.
Each node within the network receives the transaction block, and validators verify whether the transaction is valid or not. If the transaction is validated, the global state is updated.
ravipatisivaramaprasad (Thu, 14 Jun 2018 07:21:48 GMT):
What is meant by chain controller here?
ravipatisivaramaprasad (Thu, 14 Jun 2018 07:21:48 GMT):
`What is meant by chain controller here?`
Dan (Thu, 14 Jun 2018 13:36:40 GMT):
chain controller is an internal component of the validator. It manages adding or removing blocks from the chain for that particular validator.
Johannes2511 (Thu, 14 Jun 2018 13:39:48 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus?msg=AKQZisGhAXnJBDgwH) Hey Dan. Thanks for the advies
Johannes2511 (Thu, 14 Jun 2018 13:40:40 GMT):
When i run with two machines and are now alternating somehow with that message but they continue to add blocks to the chain
Johannes2511 (Thu, 14 Jun 2018 13:41:35 GMT):
Unfortunately, at some point, the system starts to fork. When I run with many ("5") nodes is occures faster than with two
Johannes2511 (Thu, 14 Jun 2018 13:41:47 GMT):
How is that supposed to happen?
Johannes2511 (Thu, 14 Jun 2018 13:42:01 GMT):
Shouldn't a blockchain or consensus avoid such situations?
Dan (Thu, 14 Jun 2018 14:04:08 GMT):
The networks should always converge. Note that there are a number of log statements that talk about forks but that's only because each new block is evaluated as a potential fork. If you see that `sawtooth block list` on each validator is actually divergent then you have a real fork. There's also a newer command `sawnet` that can help you diagnose as well.
Johannes2511 (Thu, 14 Jun 2018 14:18:34 GMT):
Yes. I already used sawnet compare-chains. And at some points the net seems to split
Johannes2511 (Thu, 14 Jun 2018 14:19:29 GMT):
screenshot
Johannes2511 (Thu, 14 Jun 2018 14:19:32 GMT):
In the screenshot it is currently only one node drifting away. But usually I have two nodes featuring one fork and the other three featuring another
Johannes2511 (Thu, 14 Jun 2018 14:20:42 GMT):
I used only a slightly different docker file:
```
joe@term0:~ $ wget https://sawtooth.hyperledger.org/docs/core/nightly/master/app_developers_guide/sawtooth-default-poet.yaml
joe@term0:~ $ cp sawtooth-default-poet.yaml sawtooth-joe-poet.yaml
joe@term0:~ $ vim sawtooth-joe-poet.yaml # configure run file
joe@term0:~ $ diff sawtooth-default-poet.yaml sawtooth-joe-poet.yaml
49,50c49,51
< sawtooth.poet.target_wait_time=5 \
< sawtooth.poet.initial_wait_time=25 \
---
> sawtooth.poet.target_wait_time=30 \
> sawtooth.poet.initial_wait_time=300 \
> sawtooth.poet.key_block_claim_limit=128000\
55c55
< sawtooth-validator -v \
---
> sawtooth-validator -vv \
joe@term0:~ $ docker-compose -f sawtooth-joe-poet.yaml up
# in other bash shell login:
joe@term1:~ $ docker exec -it sawtooth-shell-default bash
root@container:/ $ intkey set foo 1 --url http://rest-api-0:8008
root@container:/ $ intkey inc foo 1 --url http://rest-api-0:8008
root@container:/ $ intkey list --url http://rest-api-0:8008
# everything works so far. Now we start one transaction every 5 seconds
root@container:/ $ watch -n 5 'intkey inc foo 1 --url http://rest-api-0:8008'
# or execute randomly:
root@container:/ $ watch -n 5 "bash -c 'intkey inc foo 1 --url http://rest-api-\$(( \$RANDOM % 4 )):8008'"
```
Johannes2511 (Thu, 14 Jun 2018 14:29:03 GMT):
output of chain comparison:
```
$ sawnet compare-chains http://129.217.186.228:8000 http://129.217.186.228:8001 http://129.217.186.228:8002 http://129.217.186.228:8003 http://129.217.186.228:8004
Failed to reach common height with node 0: http://129.217.186.228:8000
Failed to reach common height with node 2: http://129.217.186.228:8002
Failed to reach common height with node 3: http://129.217.186.228:8003
Failed to reach common height with node 4: http://129.217.186.228:8004
COMMON HEIGHT BLOCKS
---------------------------
ANCESTOR 76 73bb3c99
HEIGHT 77 {'ef2c8b69'}
NODE HEAD HEIGHT LAG DIVERG
-------------------------------
1 ef2c8b69 77 0 1
``
Johannes2511 (Thu, 14 Jun 2018 14:29:03 GMT):
output of chain comparison:
```
$ sawnet compare-chains http://129.217.186.228:8000 http://129.217.186.228:8001 http://129.217.186.228:8002 http://129.217.186.228:8003 http://129.217.186.228:8004
Failed to reach common height with node 0: http://129.217.186.228:8000
Failed to reach common height with node 2: http://129.217.186.228:8002
Failed to reach common height with node 3: http://129.217.186.228:8003
Failed to reach common height with node 4: http://129.217.186.228:8004
COMMON HEIGHT BLOCKS
---------------------------
ANCESTOR 76 73bb3c99
HEIGHT 77 {'ef2c8b69'}
NODE HEAD HEIGHT LAG DIVERG
-------------------------------
1 ef2c8b69 77 0 1
```
Johannes2511 (Thu, 14 Jun 2018 14:29:03 GMT):
output of chain comparison:
```
$ sawnet compare-chains http://129.217.186.228:8000 http://129.217.186.228:8001 http://129.217.186.228:8002 http://129.217.186.228:8003 http://129.217.186.228:8004
Failed to reach common height with node 0: http://129.217.186.228:8000
Failed to reach common height with node 2: http://129.217.186.228:8002
Failed to reach common height with node 3: http://129.217.186.228:8003
Failed to reach common height with node 4: http://129.217.186.228:8004
COMMON HEIGHT BLOCKS
---------------------------
ANCESTOR 76 73bb3c99
HEIGHT 77 {'ef2c8b69'}
NODE HEAD HEIGHT LAG DIVERG
-------------------------------
1 ef2c8b69 77 0 1
```
(plz note that i made rest-api-X ports available via docker hosts public ip & port 8000+X to access it via my laptop)
Johannes2511 (Thu, 14 Jun 2018 14:29:03 GMT):
output of chain comparison:
```
$ sawnet compare-chains http://129.217.186.228:8000 http://129.217.186.228:8001 http://129.217.186.228:8002 http://129.217.186.228:8003 http://129.217.186.228:8004
Failed to reach common height with node 0: http://129.217.186.228:8000
Failed to reach common height with node 2: http://129.217.186.228:8002
Failed to reach common height with node 3: http://129.217.186.228:8003
Failed to reach common height with node 4: http://129.217.186.228:8004
COMMON HEIGHT BLOCKS
---------------------------
ANCESTOR 76 73bb3c99
HEIGHT 77 {'ef2c8b69'}
NODE HEAD HEIGHT LAG DIVERG
-------------------------------
1 ef2c8b69 77 0 1
```
(plz note that i made rest-api-X ports available via docker hosts public ip to access it via my laptop. this leads to port number 8000+X ;-) )
Johannes2511 (Thu, 14 Jun 2018 14:43:04 GMT):
Ok. It seems like that node1 has not forked, but i only a lot of blocks "behind". If I parse the latest block_id of node1, it is in all other chains at the same position
Johannes2511 (Thu, 14 Jun 2018 14:43:31 GMT):
However, node1 is in everyone's peer list. shouldn't it sync up somehow?
Johannes2511 (Thu, 14 Jun 2018 15:12:29 GMT):
bild2.png
Johannes2511 (Thu, 14 Jun 2018 15:13:44 GMT):
The longer the run, more nodes start to drift. Node 2&4 seem to be aligned. 3 is behind, 0 even more behind and node 1 far from good and evil. this is also reflected in an asynchronous state:
```
for x in {0..4}; do echo "node$x" && intkey list --url http://129.217.186.228:800${x} && echo ""; done
node0
foo: 719
node1
foo: 248
node2
foo: 822
node3
foo: 807
node4
foo: 822
```
Johannes2511 (Thu, 14 Jun 2018 15:18:43 GMT):
This seems to be the most easy use case, when incrementing a number on 5 nodes. Is there a more stable version, which runs more properly? Sawtooth seems to be a very nice project. The code base looks very clean and well maintained. And also the PoET algorithm is a very interesting concept. Therefore, I would like to leverage Sawtooth for my blockchain project. But I would need a PoC that it is able to operate a setup of ~100 validator nodes.
Dan (Thu, 14 Jun 2018 16:56:04 GMT):
are you running the above as docker containers on the same machine? Like you are running 5 nodes in containers on a laptop? If so maybe you are just resource constrained.
For integeration testing we typically run 10 VMs (medium sized instances, I believe) on AWS and verify that they stay in sync - depending on the test for a day or a week.
tkuhrt (Thu, 14 Jun 2018 18:06:00 GMT):
[17:32:59.853 [ConnectionManager] gossip DEBUG] Endpoint has not completed authorizati
on in 1280 seconds: tcp://10.188.64.95:8801
tkuhrt (Thu, 14 Jun 2018 18:06:09 GMT):
@rbuysse ^
Matsw7 (Fri, 15 Jun 2018 09:56:44 GMT):
Has joined the channel.
rock_martin (Fri, 15 Jun 2018 13:05:06 GMT):
Has joined the channel.
kc1116 (Fri, 15 Jun 2018 14:24:23 GMT):
@Dan any movement on the Consensus Engine API to support proof of stake ?
Dan (Fri, 15 Jun 2018 14:49:27 GMT):
I don't think there's a PoS algorithm out yet. The two things I've seen for PoS are 1. basically voting branded as PoS and 2. Casper FFG - which imo isn't really PoS - it just caps finality after n blocks, it doesn't change leader election away from PoW. We do have some RAFT coming and an update to PoET. Both of which will be on the new consensus engine interface.
DuncanMuhoro (Sun, 17 Jun 2018 11:16:52 GMT):
Has joined the channel.
rjones (Sun, 17 Jun 2018 16:29:34 GMT):
Has joined the channel.
rjones (Sun, 17 Jun 2018 16:29:51 GMT):
Room name changed to: sawtooth-consensus-dev by rjones
rjones (Sun, 17 Jun 2018 16:30:03 GMT):
Has left the channel.
ravipatisivaramaprasad (Tue, 19 Jun 2018 04:44:37 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=AwgaPiTNAqwCxHSWW) @Dan Thank you. Can you please provide me clarification on the following questions. I know something. But I want to get clarity. 1) Once a transaction is submitted to the validator, what are the different actions performed by PoET consensus in a sequential manner?
2) Is PoET consensus has any relation with `Global state` directly?
3) Who will commit the block to the blockchain once it is validated in multinode validator network? And Who will update the State(Is consensus update the state?)?
4) What PoET Consensus will do and what it won't do? I mean What is the exact role of PoET Consensus?
ravipatisivaramaprasad (Tue, 19 Jun 2018 04:44:37 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=AwgaPiTNAqwCxHSWW) @Dan Thank you. Can you please clarify me one thing. 1) Once a transaction is submitted to the validator, what are the different actions performed by PoET consensus in a sequential manner?
2) Is PoET consensus has any relation with `Global state` directly?
3) Who will commit the block to the blockchain once it is validated in multinode validator network? And Who will update the State(Is consensus update the state?)?
4) What PoET Consensus will do and what it won't do? I mean What is the exact role of PoET Consensus?
yadhuksp (Tue, 19 Jun 2018 05:12:00 GMT):
Has joined the channel.
honeyjills (Tue, 19 Jun 2018 09:10:08 GMT):
Has joined the channel.
Dan (Tue, 19 Jun 2018 18:31:35 GMT):
The other parts of the validator manage state and check the validity of transactions. PoET is just responsible for checking when to publish blocks or when blocks are received from other validators checking the consensus rules for fork resolution.
Dan (Tue, 19 Jun 2018 21:19:14 GMT):
@adamludvik that's a really nice Raft PR. Despite the overall size of the PR the commit discipline made it much more tractable to review. And double kudos for getting the documentation in there.
ravipatisivaramaprasad (Wed, 20 Jun 2018 04:17:57 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=PruhaBao4FCir8X2B) @Dan Thank You
bmv025 (Wed, 20 Jun 2018 05:15:22 GMT):
Has joined the channel.
veena.s7 (Wed, 20 Jun 2018 05:30:18 GMT):
Has joined the channel.
kdnmih (Wed, 20 Jun 2018 10:01:04 GMT):
Has joined the channel.
kdnmih (Wed, 20 Jun 2018 12:07:45 GMT):
I asked this question in #sawtooth channel but nobody answer.
kdnmih (Wed, 20 Jun 2018 12:08:10 GMT):
Does anyone provide me about the novel consensus paper?
kdnmih (Wed, 20 Jun 2018 12:08:49 GMT):
Or.. any good resource to start to study about novel consensus
kdnmih (Wed, 20 Jun 2018 12:09:35 GMT):
Seriously I could not find an article about the novel consensus.
Dan (Wed, 20 Jun 2018 13:48:23 GMT):
if you are talking about poet you can read this: https://sawtooth.hyperledger.org/docs/core/releases/latest/architecture/poet.html
Dan (Wed, 20 Jun 2018 13:50:33 GMT):
if you are talking about the novel ability to change consensus on the fly in a running blockchain you can read this:
https://www.hyperledger.org/blog/2017/11/22/un-pluggable-consensus-with-hyperledger-sawtooth
(note that we rebranded it "dynamic consensus" before we launched 1.0).
kdnmih (Thu, 21 Jun 2018 04:10:19 GMT):
Thank you Dan. Could you look at the introduction comment of Sawtooth in https://www.hyperledger.org/projects?
kdnmih (Thu, 21 Jun 2018 04:10:19 GMT):
Thank you @Dan . Could you look at the introduction comment of Sawtooth in https://www.hyperledger.org/projects?
kdnmih (Thu, 21 Jun 2018 04:10:38 GMT):
It is written that "Hyperledger Sawtooth is a modular platform for building, deploying, and running distributed ledgers. Hyperledger Sawtooth includes a novel consensus algorithm, Proof of Elapsed Time (PoET), which targets large distributed validator populations with minimal resource consumption."
kdnmih (Thu, 21 Jun 2018 04:12:39 GMT):
You means novel consensus algorithm == novel consensus algorithm?
kdnmih (Thu, 21 Jun 2018 04:12:39 GMT):
You means novel consensus algorithm == PoET algorithm?
kdnmih (Thu, 21 Jun 2018 04:13:46 GMT):
What is the mean of novel ability?
Dan (Thu, 21 Jun 2018 13:48:36 GMT):
correct
bridgerherman (Thu, 21 Jun 2018 14:12:28 GMT):
Here's an RFC proposing a simple PBFT consensus algorithm for Sawtooth: https://github.com/hyperledger/sawtooth-rfcs/pull/19
bridgerherman (Thu, 21 Jun 2018 14:12:43 GMT):
Any thoughts, comments, or reviews would be greatly appreciated!
adamludvik (Thu, 21 Jun 2018 15:32:17 GMT):
@Dan thanks
jsmitchell (Thu, 21 Jun 2018 15:49:14 GMT):
@bridgerherman this is really well written - nice job!
bridgerherman (Thu, 21 Jun 2018 16:07:36 GMT):
@jsmitchell Thank you!
Johannes2511 (Fri, 22 Jun 2018 12:35:35 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=2BfivcKZSgC5RLEqq) @Dan Both. I ran it on laptop, a virtualized server and now run on a bare metal server using the docker-compose. The system has more than enough ressources (8 core xeon cpu, 16GB ram, ssd) so resource constrains shouldn't be an issue.
Johannes2511 (Fri, 22 Jun 2018 13:07:16 GMT):
I also tried increasing the number of nodes in the network to 10 as you suggested. But unfortunately, I still have no success :-(
Johannes2511 (Mon, 25 Jun 2018 07:10:38 GMT):
@Dan How do your integration tests look like? are they included in the public github repository?
guyfawkes (Mon, 25 Jun 2018 11:38:26 GMT):
Has joined the channel.
Johannes2511 (Mon, 25 Jun 2018 15:07:15 GMT):
@dan I think i found the issue. The documentation (https://sawtooth.hyperledger.org/docs/core/nightly/master/app_developers_guide/creating_sawtooth_network.html#docker-start-a-multiple-node-sawtooth-network) links to the `awtooth-default-poet.yaml` in which the dockerimages `hyperledger/sawtooth-XXX:1.0` are used. When i compiled images from the git-repo using latest branch (1.1.0) it works for me :-)
veena.s7 (Tue, 26 Jun 2018 09:14:00 GMT):
Hi all, We have setup a 5-Node network.
Added INTKEY TP for 2 nodes and exectuted an INT-KEY txn. Consequently, the nodes without the INTKEY went to "Waiting" mode, and the subsequent txns were added to its queue, and no more txn were validated by those nodes. The 3 validators(without INTKEY) will be activated back only if INTKEY is added to those validators.
So,is it not considering the TPs in majority nodes(3 nodes are not having INTKEY)?
I would like to know how the PoET consensus and aggregate local mean works here.
Thanks in advance!
Dan (Tue, 26 Jun 2018 17:20:38 GMT):
Hi. Since those three machines aren't really participating on the network they won't be counted. It seems like that's the majority of the network because you have 5 machines, but since those 3 aren't processing transactions they aren't really part of the network and so your network is 2 machines.
veena.s7 (Wed, 27 Jun 2018 06:20:01 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=6b6wE4tERktJw4Y9t) @Dan So if I understood you correctly,..those validators without INTKEY are not at all considered in network even though they are majority(3 out of 5).
On considering a system of 10 nodes, suppose only the 9&10th node have INTKEY and rest 8 have XO alone, shouldn't be the 9&10th considered bad nodes in the network?? Sorry, I'm trying to get the concept clear..
rootDistress (Wed, 27 Jun 2018 07:23:25 GMT):
Has joined the channel.
pschwarz (Wed, 27 Jun 2018 09:21:06 GMT):
Every node in the network has to have the transaction processors for all the transactions to be applied. If a node doesn't have a TP - XO in your above example, it can't make progress - either publishing new blocks or validating old blocks. When it encounters a transaction of a given family, if it doesn't have an available transaction processor, it will wait until one connects. This blocks progress, so essentially the node stops participating in the network.
veena.s7 (Wed, 27 Jun 2018 10:13:19 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=GCEQySYDASHFai5RX) @pschwarz So, like if I have a network of 10 nodes, 10th one tries to break the system..
this bad node add a new TP which is not in any of the other nodes, and executed a Tnx with the new TP.
Can this bring down the entire network (bcos 9 nodes are not at all participating in the network)?
pschwarz (Wed, 27 Jun 2018 10:14:34 GMT):
Yes, but you can constrain what TP's are required/allowed via an on-chain setting
pschwarz (Wed, 27 Jun 2018 10:16:25 GMT):
`sawtooth.validator.transaction_families`
pschwarz (Wed, 27 Jun 2018 10:17:30 GMT):
It is an json string like: `[{"name": "MyFamily", "version": "1.0"}, ...]`
pschwarz (Wed, 27 Jun 2018 10:17:48 GMT):
Put that in your genesis block
pschwarz (Wed, 27 Jun 2018 10:18:21 GMT):
The basic dev setup is not locked down at all to make it easier for developers
veena.s7 (Wed, 27 Jun 2018 10:20:59 GMT):
great thankyou! and one more doubt, what if someone is trying to remove a TP from an already running network..can this be permissioned as well?
pschwarz (Wed, 27 Jun 2018 11:06:51 GMT):
Do you mean by altering that setting?
kelly_ (Thu, 28 Jun 2018 15:42:52 GMT):
@bridgerherman nice job on the PBFT writeup! I just got a chance to go through it and it looks good. I had a few questions/suggestions. feel free to shoot me a message on here if any of them weren't clear
kelly_ (Thu, 28 Jun 2018 16:04:17 GMT):
@bridgerherman if you are working on the implementation as well you may be interested in this if you haven't seen it: https://github.com/jpmorganchase/quorum/tree/master/consensus/istanbul
kelly_ (Thu, 28 Jun 2018 16:04:43 GMT):
^this is a bft implementation for Ethereum, if you want to look online 'instanbul bft' is the name and it was done by a company called 'amis'
bridgerherman (Thu, 28 Jun 2018 16:14:54 GMT):
@kelly_ Thanks for the comments - Looking at them now!
jnessbcf (Thu, 28 Jun 2018 17:26:21 GMT):
Has joined the channel.
willyko (Thu, 28 Jun 2018 20:37:22 GMT):
Has joined the channel.
sidhujag (Sat, 30 Jun 2018 20:45:32 GMT):
hi guys how does Self Attestation work from a high level design point of view for sawtooth SGX? Is the motivation similar to LiberateSGX? where your removing dependence from using Intel for every quote verification? I assume your cpu still needs to get verified by Intel first before you become a validator and then the remove attestations and quotes are verified by a CA? hows it work? Is it similar to the model described here as the "Antarctica" model so you don't rely on Intel? https://www.idc.ac.il/en/schools/cs/research/Documents/jackson-msc-thesis.pdf
sidhujag (Sat, 30 Jun 2018 20:45:32 GMT):
hi guys how does Self Attestation work from a high level design point of view for sawtooth SGX? Is the motivation similar to LiberateSGX? where your removing dependence from using Intel for every quote verification? I assume your cpu still needs to get verified by Intel first before you become a validator and then the remote attestations and quotes are verified by a CA? hows it work? Is it similar to the model described here as the "Antarctica" model so you don't rely on Intel? https://www.idc.ac.il/en/schools/cs/research/Documents/jackson-msc-thesis.pdf
sidhujag (Sat, 30 Jun 2018 20:52:07 GMT):
for reference Liberating Remote Attestation, which means Intel should allow anyone, not just its own IAS servers to perform Quote verifications (of course, it’s perfectly fine, and probably justified if Intel will continue to operate and offer IAS to those customers who find such a service convenient)
sidhujag (Sat, 30 Jun 2018 20:52:59 GMT):
should be possible.. which is why im wondering if Self Attestation does that: https://software.intel.com/en-us/articles/intel-software-guard-extensions-product-licensing-faq see there someone asked the question if they can become an IAS service
sidhujag (Sat, 30 Jun 2018 20:54:57 GMT):
My other question is more to do with loading enclaves but maybe you guys have figured it out: Does sawtooth allow to load enclaves signed with any key, not just ones “blessed” by Intel. But at the same time still allowing for meaningful Remote Attestation to work. By meaningful attestation I understand one that performs signing of the Quote using keys derived from the processor’s private key, of course.
sidhujag (Sat, 30 Jun 2018 20:55:20 GMT):
I found this link perhaps relevant to that discussion, let me know if I am offbase https://software.intel.com/en-us/sgx-sdk-dev-reference-using-a-different-extended-intel-epid-group-for-remote-attestation
sidhujag (Sat, 30 Jun 2018 20:57:46 GMT):
IMHO it would be awesome to verify integrity of the TEE through a software analysis via timing and thus removing the need for whitelist checking... I wonder if you guys have thoughts on how to remove the bottleneck of verifiying that your HW is adequate by connecting to a central authority rather than an algorithm that may do it deterministically
MicBowman (Mon, 02 Jul 2018 23:26:53 GMT):
is there an RFC for poet2?
agunde (Mon, 02 Jul 2018 23:44:14 GMT):
@MicBowman https://github.com/hyperledger/sawtooth-rfcs/pull/12
amolk (Tue, 03 Jul 2018 09:15:42 GMT):
@MicBowman that's the very initial version. We have an update almost ready. Will send you once its done.
amundson (Tue, 03 Jul 2018 17:45:06 GMT):
@amolk looking forward to seeing the update!
Dan (Tue, 03 Jul 2018 19:57:03 GMT):
@sidhujag we use sgx attestation to join a validator to the list of authorized block publishers. That registration lasts for k blocks. So I think that's similar to the model you are talking about. We don't go back to the intel attestation service to verify quotes but for those registration transactions.
Dan (Tue, 03 Jul 2018 19:59:35 GMT):
poet's use of SGX is whatever is enabled in the SGX SDK. We don't have anything special for enclave authorization.
sidhujag (Wed, 04 Jul 2018 03:11:38 GMT):
yea just for the initial verification right @Dan and then use self-attestation after that once its in the endpoint registry you ensure it stays in there by requiring a resign every k blocks from IAS again?
sidhujag (Wed, 04 Jul 2018 03:12:30 GMT):
I assume then if you get blacklisted or something in the meantime after kth block that you wont be able to verify quotes anymore from those enclaves right
sidhujag (Wed, 04 Jul 2018 03:51:01 GMT):
also have you coded with cache attacks in mind, like use stack instead of heap
sidhujag (Wed, 04 Jul 2018 03:51:01 GMT):
also have you coded with cache attacks in mind, like use stack instead of heap wherever possible?
Dan (Wed, 04 Jul 2018 15:27:13 GMT):
@sidhujag yes that's right on the registration flow.
Dan (Wed, 04 Jul 2018 15:28:46 GMT):
it's been a while since the initial code was written, so I don't recall for sure if there's much heap use. There's little state though, so most of what needs to be in memory is getting read in from sealed storage as local variables in the enclave. I'm more than happy to get a code review on what's there and PRs for any important security fixes.
sidhujag (Thu, 05 Jul 2018 04:15:28 GMT):
cool yea we can def take a look and try to apply best practices to see if theres code there to fix.. what was that saying in bitcoin core for every 10 lines of code theres a bug... so I assume unit testing/black box testing would be critical here, not sure how to detect that other than actively trying to attack it and see if the code is safe. Design wise though, I think most of what you got here makes sense for what I want to do. Is there a project you know or code you know that creates an incentive layer to see what the best way to do it is? I'm also thinking of a bonded validator approach because it will be a permissionless network so validators would require some stake.. so any thoughts on how to best do that? In terms of the EVM, the state storage is stored on all nodes right (any thoughts on IPFS or a content addressable distributed storage medium to store the contracts and just store the balances on chain?).. to run the contracts only 1 node needs to run instead of all validators? That one enclave would submit the evm contract tx after running it through the TEE and the submitter would verify it is the correct state before letting the enclave submit, that is how I kinda imagine it would work efficiently. Each block producer doesn;t need to reverify the transactions because they are signed by enclaves, maybe a sig check is done prior to block inclusion of txs.
MicBowman (Thu, 05 Jul 2018 14:07:15 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=TpXRn3GRXkz7yzfBf) @agunde thanks!
Dan (Thu, 05 Jul 2018 15:53:53 GMT):
@sidhujag regarding incentives.. @MicBowman has done a lot of work there. not sure if anything has been opensourced though.
MicBowman (Thu, 05 Jul 2018 15:55:26 GMT):
@sidhujag we did a project a couple years ago that used SGX & a blockchain to create "burnable tokens"... these could be used for incentives fairly easily
MicBowman (Thu, 05 Jul 2018 15:56:29 GMT):
it was research code against an older version of sawtooth
sidhujag (Thu, 05 Jul 2018 20:32:11 GMT):
I did see this one aswell https://fullmetalhealth.com/cryptoasset-framework-intels-hyperledger-sawtooth-part-one/ @Dan
sidhujag (Thu, 05 Jul 2018 21:09:25 GMT):
@MicBowman interesting.. is it on your github or any reference code to take a look at? I saw that this link I posted designed it I wonder if the design is good though @Dan there are things I would do differently but if the idea is right in which classes to extend and build from then its a good starting point perhaps
sidhujag (Thu, 05 Jul 2018 21:09:25 GMT):
@MicBowman interesting.. is it on your github or any reference code to take a look at? I saw that this link I posted designed it I wonder if the design is good though @Dan there are things I would do differently but if the idea is right in which classes to extend and build from then its a good starting point pehaps
danny_lee (Thu, 05 Jul 2018 21:32:55 GMT):
Has joined the channel.
Dan (Fri, 06 Jul 2018 00:36:12 GMT):
I haven't reviewed that in detail @sidhujag but I believe the authors are often on the #sawtooth channel. You might be able to hail them there by mentioning pokitdok.
sidhujag (Fri, 06 Jul 2018 16:57:54 GMT):
@dan thanks I joined that weekly call and have another question. What is the intuition regarding how EVM will work with sawtooth. I would think since you can deterministically prove that you are running TEE can you not run the smart contract and modify the internal state inside the database by creating that transaction through protected enclave? This way you only need 1 validator to run the contract and post the tx. The same as any other transaction. The rest of the network doesn't need to verify if its run through protected hw as gatekeepers of the blockchain.
sidhujag (Fri, 06 Jul 2018 16:57:54 GMT):
@dan thanks I joined that weekly call and have another question. What is the intuition regarding how EVM will work with sawtooth. I would think since you can deterministically prove that you are running TEE can you not run the smart contract and modify the internal accounts db by creating that transaction through protected enclave? This way you only need 1 validator to run the contract and post the tx. The same as any other transaction. The rest of the network doesn't need to verify if its run through protected hw as gatekeepers of the blockchain.
Dan (Mon, 09 Jul 2018 14:16:55 GMT):
Hi @sidhujag. While someone could write a consensus that used SGX in a way that all validators trusted the results of a block without checking each transaction, that's not how poet or sawtooth is written. At the moment the only SGX usage in sawtooth is the optional PoET enclave that handles random leader election - the part of PoW that wastes energy.
Sawtooth as a whole is entirely independent of SGX. So when it comes to the EVM there's no special SGX behavior (unless you want to create that feature :) )
pschwarz (Mon, 09 Jul 2018 14:41:14 GMT):
@Dan You better look at https://github.com/hyperledger/sawtooth-core/pull/1757 before it gets merged
Dan (Mon, 09 Jul 2018 15:02:04 GMT):
Already in it.
bridgerherman (Mon, 09 Jul 2018 19:44:47 GMT):
Hey everyone, I updated the RFC for PBFT a bit ago to address comments from @kelly_ . Feel free to take a look and comment!
bridgerherman (Mon, 09 Jul 2018 19:44:47 GMT):
Hey everyone, I updated the RFC for PBFT a bit ago to address comments from @kelly_ . Feel free to take a look and comment!
https://github.com/hyperledger/sawtooth-rfcs/pull/19
Dan (Mon, 09 Jul 2018 23:09:29 GMT):
@adamludvik regarding the thread safety thing we were discussing you said that the poet engine is single threaded. Is the engine concept single threaded, like that should be the case for Raft too and all consumers of the SDKS or is this specific to Python or to just PoET?
adamludvik (Tue, 10 Jul 2018 02:25:59 GMT):
@Dan The consensus engine process, in general, is multi-threaded. An object (the driver) is created and passed an object that implements an Engine interface. The interface is basically just a `start()` method that receives a couple objects used to communicate with it. When the driver is started, it creates a background thread that handles communications with the validator and then calls the engine's start method from whatever thread called `driver.start()`, which is the main thread in both the PoET engine and Raft engine. It is expected that the engine will sit in a loop and process updates from the validator (which is receives on a queue or channel) and make requests to the validator (using a service object provided by the driver). Any data used by the engine is private to the engine and it is up to the engine implementer to decide what kind of threading model to use. In both the PoET engine and the Raft engine, we have opted for a single-threaded Engine implementation, but this is not a requirement.
Dan (Tue, 10 Jul 2018 02:43:25 GMT):
thanks @adamludvik
sidhujag (Tue, 10 Jul 2018 04:10:18 GMT):
@adamludvik does the communication thread also do the signature verification single threaded?
sidhujag (Tue, 10 Jul 2018 04:10:18 GMT):
@adamludvik does the communication thread also do the signature verification single threaded ( the memory pool verification not block processing) ?
adamludvik (Tue, 10 Jul 2018 12:49:21 GMT):
consensus engines run in a separate process.signature verification happens in the validator process and is concurrent.
sidhujag (Tue, 10 Jul 2018 16:46:07 GMT):
concurrent as in global to all transactions? or to an individual tx (more that 1 signature verification per tx)
sidhujag (Tue, 10 Jul 2018 16:46:49 GMT):
also does it relay concurrently as well then or right away with some recourse if a signature verification failed?
amolk (Tue, 10 Jul 2018 17:04:35 GMT):
An updated PoET 2.0 RFC has been posted. https://github.com/hyperledger/sawtooth-rfcs/pull/20. This deprecates the earlier RFC (#12), which has been left in place for anyone trying to get context. @MicBowman
MicBowman (Wed, 11 Jul 2018 17:31:35 GMT):
thanks, @amolk
Southside (Wed, 11 Jul 2018 22:42:06 GMT):
Maidsafe have released the code for PARSEC.
https://safenetforum.org/t/parsec-the-release-of-the-code/24469
Very different approach (putting it mildly) from other consensus mechanisms. Could this possibly become a Pluggable Consensus Mechanism for Sawtooth?
jsmitchell (Thu, 12 Jul 2018 03:22:59 GMT):
ooh, it's in rust too!
sidhujag (Thu, 12 Jul 2018 05:07:39 GMT):
@dan how do you keep https://sawtooth.hyperledger.org/docs/core/releases/0.8.8/examples/private_utxo/overview.html the UTXO model private.. the data is encrypted to which keys? are those keys available to all enclaves network wide to be able to validate the transactions?
sergefdrv (Thu, 12 Jul 2018 13:04:06 GMT):
Has joined the channel.
sergefdrv (Thu, 12 Jul 2018 13:06:20 GMT):
Hi there! We just release an initial version of implementation of a consensus protocol called MinBFT. Please have a look if you are interested https://github.com/nec-blockchain/minbft
sergefdrv (Thu, 12 Jul 2018 13:06:20 GMT):
Hi there! We just published an initial version of implementation of a consensus protocol called MinBFT. Please have a look if you are interested https://github.com/nec-blockchain/minbft
sergefdrv (Thu, 12 Jul 2018 13:07:00 GMT):
Any feedback is highly appreciated!
Southside (Thu, 12 Jul 2018 14:30:22 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=5fuMh6n4p8iroHneC) @jsmitchell The power of C++ without the dangers
Dan (Thu, 12 Jul 2018 14:35:35 GMT):
@sidhujag that was a concept we put up like a year ago. We decided not to mature it because it seemed fragile. You are more than welcome to use it as a starting point though and see if you can come up with something better.
The original author on that has since moved on to other projects.
Dan (Thu, 12 Jul 2018 14:43:03 GMT):
@sergefdrv that looks pretty cool! Would you be up for presenting at our tech forum next thursday? (https://calendar.google.com/calendar/r/customweek/2018/7/19?eid=Y3RqdjgwMTltMTVrOGkwZmdmdDZvaG1xN2NfMjAxODA3MTlUMTUwMDAwWiBsaW51eGZvdW5kYXRpb24ub3JnX25mOXU2NGc5azlydmQ5Zjh2cDR2dXIyM2IwQGc&ctz=UTC&sf=true)
sergefdrv (Thu, 12 Jul 2018 15:12:32 GMT):
@Dan sorry, unfortunately, I have a conflicting appointment for this time. Maybe the week after would work?
sergefdrv (Thu, 12 Jul 2018 15:13:42 GMT):
@Dan I mean, Aug 2
kelly_ (Thu, 12 Jul 2018 17:20:40 GMT):
https://github.com/askmish/sawtooth-poet2
kelly_ (Thu, 12 Jul 2018 17:20:45 GMT):
^ PoET 2 repo
sidhujag (Thu, 12 Jul 2018 17:54:19 GMT):
@dan i was just asking because I wanted to see if there was a novel way to keep the data private in context with discussions I'm having with @MicBowman regarding the same type of thing to encrypt data in enclaves
sidhujag (Thu, 12 Jul 2018 17:54:19 GMT):
@dan i was just asking because I wanted to see if there was a novel way to keep the data private in context with discussions I'm having with @MicBowman regarding the same type of thing to encrypt data to enclave ephemeral keys
sidhujag (Thu, 12 Jul 2018 17:54:19 GMT):
@Dan i was just asking because I wanted to see if there was a novel way to keep the data private in context of discussions I'm having with @MicBowman regarding the same type of thing to encrypt data to enclave ephemeral keys
sidhujag (Thu, 12 Jul 2018 17:54:19 GMT):
@Dan i was just asking because I wanted to see if there was a novel way to keep the data private in context with discussions I'm having with @MicBowman regarding the same type of thing to encrypt data to enclave ephemeral keys
Dan (Thu, 12 Jul 2018 18:24:14 GMT):
@sergefdrv that sounds good to me.
@mfford do you think we can schedule a tech forum for 8/2 at the normal slot?
Dan (Thu, 12 Jul 2018 18:24:36 GMT):
@mfford (checking tag)
PHeinz (Thu, 12 Jul 2018 18:58:22 GMT):
Has joined the channel.
amundson (Thu, 12 Jul 2018 20:03:39 GMT):
@Dan @kelly_ we are going to start going through PR review of PBFT consensus engine code. current plan is to do this here: https://github.com/bitwiseio/sawtooth-pbft since we don't have a PBFT repo yet for it in hyperledger. but if there is enough interest, we could do it in a hyperledger repo so it's easier to get more reviewers. opinions? (I know we want the new poet code reviewed by the core maintainers, like I said before, but I don't know if you feel the same about pbft or not)
Dan (Thu, 12 Jul 2018 22:51:40 GMT):
sure
amundson (Thu, 12 Jul 2018 22:52:30 GMT):
sounds like one of your more strongly held opinions, @Dan
Dan (Thu, 12 Jul 2018 22:53:17 GMT):
too many chats today. learning non-verbose responses.
kelly_ (Thu, 12 Jul 2018 23:09:46 GMT):
i think HL repo is fine
kelly_ (Thu, 12 Jul 2018 23:16:10 GMT):
with devmode - should we call this proof of authority? seems pretty similar to what ethereum etc. are doing, though I guess devmode doesn't have an actual 'authority' list registered on chain
kelly_ (Thu, 12 Jul 2018 23:17:37 GMT):
i think once we have devmode, poet-sim, poet-sgx, poet-sgx-without-PSE, raft, and pbft. sawtooth will have the most consensus choices
kelly_ (Thu, 12 Jul 2018 23:17:45 GMT):
out of all platforms
Southside (Thu, 12 Jul 2018 23:19:59 GMT):
Has anyone thought about PARSEC? https://safenetforum.org/t/parsec-the-release-of-the-code/24469
amolk (Fri, 13 Jul 2018 16:12:13 GMT):
@sergefdrv I just went through the minBFT protocol. Thanks for sharing it, it looks very interesting! I have a question about the monotonic counter (USIG). I've just looked at the code superficially and perhaps you're already handling this case. It seems there's no way to prevent an untrusted (i.e. outside the enclave) component from sending an earlier version of the sealed data. So a compromised component could potentially send two messages with the same counter value by just keeping a copy of a previous version of the sealed data?
sergefdrv (Fri, 13 Jul 2018 16:21:29 GMT):
@amolk Thanks for your question! In our implementation of USIG, a key pair is generated and sealed when a USIG enclave instance is initialized the first time. That key pair is used afterwards. However, every USIG enclave generates an ephemeral random value, epoch, that is unique for each enclave instance. This value, as well as the counter value, is never sealed. We use the epoch value to detect an attempt of rollback attack.
amolk (Fri, 13 Jul 2018 16:29:31 GMT):
I see. So as long as the enclave is loaded and has these values in memory, the protocol can continue. If the enclave gets unloaded, these values are destroyed. So a new instance of an enclave appears as a new node to the network.
sergefdrv (Fri, 13 Jul 2018 16:32:27 GMT):
@amolk right, in current implementation, we capture the epoch value of the first valid UI. Then every subsequent UI from that replica is checked against that value
sergefdrv (Fri, 13 Jul 2018 16:34:31 GMT):
@amolk we plan to investigate possible ways to implement faulty replica recovery, including the USIG recreation issue. It seems to be possible using a stable checkpoint retrieved from any of the correct replicas
amolk (Fri, 13 Jul 2018 16:35:20 GMT):
On receiving nodes do you maintain a list of epochs that you've encountered? Do you need to protect that list?
amolk (Fri, 13 Jul 2018 16:35:43 GMT):
and the corresponding counters, of course.
sergefdrv (Fri, 13 Jul 2018 16:39:10 GMT):
every replica maintains a list of the captured epoch value and the last accepted counter values per each peer. This data doesn't have to be protected by TEE, because at most f replicas are assumed to be possibly faulty. The f+1 correct replicas will ensure safety and liveness according to the protocol
amolk (Fri, 13 Jul 2018 17:18:00 GMT):
Thanks for the explanation.
Dan (Fri, 13 Jul 2018 19:56:00 GMT):
That sounds similar to what we are doing with PoET2. We were looking to remove hardware counters which were used to enforce a sort of singleton pattern with the enclave per machine. Each validator is registered on the network (stored in state on the blockchain). That registration had contained the monotonic counter ID which would let the registration safely persist across reboots etc. Doing away with hw counters in PoET 2 the node needs to register every time the process starts since we can't trust persisting any identity token. (You can correct me if I'm remembering wrong Amol. I'll need to double check if that detail is in the RFC too).
Southside (Fri, 13 Jul 2018 19:58:19 GMT):
Where can I read an overview of PoET2?
Dan (Fri, 13 Jul 2018 20:24:51 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=JQWTQS6b2j8z7mBiD) @Southside
amolk (Sat, 14 Jul 2018 05:58:04 GMT):
@dan, that's right. We go a bit further than that by not persisting signing keys either.
Southside (Sun, 15 Jul 2018 19:12:29 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=rjfaJdqqJsAEs4xiw) @Dan Thank you I have had a chance to read this over in depth now. Is it the case that embracing PoET2 would simplify the use of the protocol in the cloud?
sergefdrv (Mon, 16 Jul 2018 08:18:09 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=5bZp55v7K6Nn8zRC8) @amolk We considered to get by using ephemeral keys in USIG, that we would never seal/unseal. But the rational not to do so was that this would make online remote attestation mandatory. Whereas allowing to seal/unseal the keys and using a random enclave instance "ID" would allow to do remote attestation before the consensus network starts, "offline".
kelly_ (Mon, 16 Jul 2018 14:40:25 GMT):
@Southside PoET 2 will be able to run on Intel SGX servers that are available in the cloud. I believe IBM and Azure both offer these today
Dan (Tue, 17 Jul 2018 14:03:02 GMT):
@Southside yes that's the gist. By reducing the HW requirements we hope that it increases the available platforms/environments.
kelly_ (Tue, 17 Jul 2018 23:32:57 GMT):
https://media.consensys.net/scaling-consensus-for-enterprise-explaining-the-ibft-algorithm-ba86182ea668
kelly_ (Tue, 17 Jul 2018 23:33:03 GMT):
^ fyi @bridgerherman
bridgerherman (Wed, 18 Jul 2018 13:43:20 GMT):
@kelly_ Thanks!
johnfranklin (Thu, 19 Jul 2018 06:10:19 GMT):
Has joined the channel.
johnsourour (Thu, 19 Jul 2018 17:14:36 GMT):
Has joined the channel.
kirkwood (Mon, 23 Jul 2018 04:00:48 GMT):
Has joined the channel.
amolk (Tue, 24 Jul 2018 17:18:59 GMT):
@MicBowman @amundson Could you please take a look at the PoET2 RFC and provide comments? https://github.com/hyperledger/sawtooth-rfcs/pull/20
MicBowman (Tue, 24 Jul 2018 17:22:42 GMT):
how soon?
Dan (Tue, 24 Jul 2018 18:30:23 GMT):
before lunch ;)
Dan (Tue, 24 Jul 2018 18:32:16 GMT):
In particular, @MicBowman please look whether we've capture properly the case where a new validator joins an existing network. I'm not sure off the top of my head how we handle setting the chain clock. I'm sure it's either trivial or completely a mess. :D
amundson (Tue, 24 Jul 2018 20:13:34 GMT):
@amolk yes, I will soon (but not before lunch, @Dan ). we should have some other maintainers read through it too that are probably too busy to do it immediately
MicBowman (Tue, 24 Jul 2018 21:08:30 GMT):
@amolk biggest comment... the WaitTime and LocalMean are completely redundant in the wait certificate
MicBowman (Tue, 24 Jul 2018 21:08:46 GMT):
they have to be computed by the validators anyway... why make the enclave do it
naviat (Thu, 26 Jul 2018 09:04:55 GMT):
Has joined the channel.
askmish (Fri, 27 Jul 2018 05:41:59 GMT):
@adamludvik @amundson Would the validator-registry-tp for poet1.0 still ship as a separate package, without any additional deps. If not, I'm going to implement it just as a separate module as part of the poet2.0 instead of a tp.
amundson (Fri, 27 Jul 2018 05:48:05 GMT):
@askmish not sure what you mean. your question seems to be about packaging, but 'instead of a tp' seems to be about how you plan to store data. these seem like unrelated things.
amundson (Fri, 27 Jul 2018 05:48:58 GMT):
it is certainly possible to package stuff up in such a way as to share a TP between both implementations (assuming the data is reasonably similar)
amundson (Fri, 27 Jul 2018 05:48:58 GMT):
it is certainly possible to package stuff up in such a way as to share a TP between both implementations (which would make sense if the data is reasonably similar)
askmish (Fri, 27 Jul 2018 05:49:59 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=JKdi2PgiYYkSBj93J) @amundson Yep that's what I meant. My intention was to reuse the validator-registry TP for both poet1.0 and poet2.0 engines.
amundson (Fri, 27 Jul 2018 05:52:58 GMT):
seems beneficial and easier to support an upgrade path that way
amundson (Fri, 27 Jul 2018 05:57:42 GMT):
the easiest way to manage that is going to be to make your new code a subdirectory in the current poet repo, so you can make changes to the TP in your branch as necessary to support the new poet, etc., etc.
amundson (Fri, 27 Jul 2018 06:00:44 GMT):
@askmish are you starting from scratch out of a desire to switch to rust, or because the design actually has nothing in common except the registry?
askmish (Fri, 27 Jul 2018 06:03:28 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=rr9k82sduTsNYdxqw) @amundson both. Though there's a few differences in the way the validator registry is currently closely implemented for poet1.0 settings and stuff.
askmish (Fri, 27 Jul 2018 06:04:10 GMT):
Are there any plans to update the validator-registry TP?
amundson (Fri, 27 Jul 2018 06:04:52 GMT):
No, not outside what you are working on.
amundson (Fri, 27 Jul 2018 06:07:15 GMT):
The plan is to make sgx optional via a crate feature and thus conditional compilation instead of it being run-time dynamically determined?
amundson (Fri, 27 Jul 2018 06:08:45 GMT):
That will present some new packaging fun -- needing to compile twice to get all the artifacts generated, etc.
amundson (Fri, 27 Jul 2018 06:09:06 GMT):
If it is possible, that would be made easier with three options 'sgx, nosgx, both'
amundson (Fri, 27 Jul 2018 06:09:24 GMT):
where we could use 'both' when doing deb packaging, for example
amundson (Fri, 27 Jul 2018 06:10:42 GMT):
also, the binaries should differ in name so the packages can be installed at the same time
amundson (Fri, 27 Jul 2018 06:11:26 GMT):
it is possible to have multiple [bin] directives in the crate, but I'm not sure if they can be conditionally generated
askmish (Fri, 27 Jul 2018 06:16:15 GMT):
Yep, I'm sure it will impact both our crates and it'll be twice the fun solving them. It would have been easier though, if it was a separate, standalone tp. Perhaps, we just stick with implementing parts of it at runtime right now, with plans to separate the tp into its own repo at a later time. Or, I'll have to go back and rewrite the whole functionality for poet2.0.
askmish (Fri, 27 Jul 2018 06:16:15 GMT):
Yep, I'm sure it will impact both our crates and it'll be twice the fun solving them. It would have been easier though, if it was a separate, standalone tp. Perhaps, we just stick with implementing parts of it at runtime right now, with plans to separate the tp into its own repo at a later time. Or, I'll have to go back and rewrite the whole functionality for poet2.0. Ofcourse with intention to make it poet-agnostic.
askmish (Fri, 27 Jul 2018 06:16:15 GMT):
Yep, I'm sure it will impact both our crates and it'll be twice the fun solving them. It would have been easier though, if it was a separate, standalone tp. Perhaps, we just stick with implementing parts of it at runtime right now, with plans to separate the tp into its own repo at a later time. Or, I'll have to go back and rewrite the whole functionality for poet2.0. Ofcourse with intention to make it poet-version-agnostic.
amundson (Fri, 27 Jul 2018 06:25:34 GMT):
we really don't want to deal with multiple poet repos
askmish (Fri, 27 Jul 2018 06:28:41 GMT):
Is the directory strcuture in sawtooth-poet repo going to change?
amundson (Fri, 27 Jul 2018 06:28:55 GMT):
do you want it to change?
askmish (Fri, 27 Jul 2018 06:29:44 GMT):
both poet1.0 and poet2.0 are mostly different and don't have many parts of poet1.0
amundson (Fri, 27 Jul 2018 06:30:43 GMT):
why are we calling it poet if it's not the same thing?
amundson (Fri, 27 Jul 2018 06:30:47 GMT):
lack of imagination? :)
amundson (Fri, 27 Jul 2018 06:31:50 GMT):
seriously though, sawtooth 0.8 shared no code with sawtooth 0.7, and we are currently rewriting portions of sawtooth in rust, and we managed to do that all without creating a new repo from scratch each time
askmish (Fri, 27 Jul 2018 06:33:04 GMT):
Are you suggesting multiple branches?
amundson (Fri, 27 Jul 2018 06:34:20 GMT):
I wasn't above, no. But it does seem appropriate to create a feature branch off of master to do the new poet work, since you won't be able to get any code into master without it working and passing LR networks.
askmish (Fri, 27 Jul 2018 06:34:47 GMT):
We do need to maintain both poet1.0 and poet2.0 simultaneously.
amundson (Fri, 27 Jul 2018 06:36:04 GMT):
poet/cft and the new poet. presumably not the older poet/bft given no one is bringing that packaging work forward to the new consensus engine model
amundson (Fri, 27 Jul 2018 06:38:39 GMT):
is there really no opportunity for code sharing though? if poet/cft was rewritten in rust?
amundson (Fri, 27 Jul 2018 06:39:30 GMT):
I think poet/cft probably gets simpler when you throw away any desire to have sgx support
amundson (Fri, 27 Jul 2018 06:42:16 GMT):
though, maybe that doesn't make sense if we believe the reason for keeping poet/cft is only because it is better tested
amundson (Fri, 27 Jul 2018 06:44:47 GMT):
also, in packaging, you can absolutely not use "2.0", because we won't be able to package it like that (it doesn't replace the current poet) so if Intel continues with this version thing instead of coming up with a cooler designation, you should consider it part of it's name, not its version number. which probably means "poet2" not "poet2.0" since "poet2.0" can't be a package name.
amolk (Fri, 27 Jul 2018 06:49:07 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=7LLRWeXdjuQoXoPt8) @amundson If the existing PoET were to be rewritten in rust, there would certainly be many areas where we would be able to share code.
amundson (Fri, 27 Jul 2018 06:49:10 GMT):
to be clearer, if the debian package name was "poet-1.1.3" (we will be using 1.1.x for the current poet engine I imagine, since 1.0.x was taken up by sawtooth 1.0), you can't have "poet-2.0.0" because it would replace the "poet-1.1.3" package. and you need to allow them to co-exist if we are supporting both.
amundson (Fri, 27 Jul 2018 06:49:35 GMT):
But, you can do "poet2-0.1.0"
amundson (Fri, 27 Jul 2018 06:49:55 GMT):
because "poet-1.1.3" and "poet2-0.1.0" are different package names
amolk (Fri, 27 Jul 2018 06:51:41 GMT):
Would it make you happier if we called it poet-csp? :slight_smile:
amundson (Fri, 27 Jul 2018 06:51:55 GMT):
absolutely. what does csp stand for?
amolk (Fri, 27 Jul 2018 06:52:17 GMT):
something cool :slight_smile:
amolk (Fri, 27 Jul 2018 06:52:27 GMT):
I'll think of it and let you know
amolk (Fri, 27 Jul 2018 06:52:41 GMT):
:wink:
amundson (Fri, 27 Jul 2018 06:52:41 GMT):
I like it even better now
amundson (Fri, 27 Jul 2018 06:53:20 GMT):
poet-spv ?
amundson (Fri, 27 Jul 2018 06:53:28 GMT):
second-poet-version
amundson (Fri, 27 Jul 2018 06:53:33 GMT):
:)
amolk (Fri, 27 Jul 2018 06:53:37 GMT):
There you go!
amundson (Fri, 27 Jul 2018 06:53:44 GMT):
though, we are really on like version 3 or 4
amundson (Fri, 27 Jul 2018 06:54:48 GMT):
as in, poet 0, poet 1, poet 2 --- you would may call "poet 2" the tpv since it's the third in the list, not the sedond
amundson (Fri, 27 Jul 2018 06:54:48 GMT):
as in, poet 0, poet 1, poet 2 --- you would may call "poet 2" the tpv since it's the third in the list, not the second
amundson (Fri, 27 Jul 2018 06:59:43 GMT):
in addition to the TP code, would it make sense to share the CLI for both versions?
amundson (Fri, 27 Jul 2018 07:00:13 GMT):
is ias_(client|proxy) still relevant for the new poet?
amolk (Fri, 27 Jul 2018 07:05:58 GMT):
Yes, partially. The PSE manifest portion is no longer relevant.
Dan (Fri, 27 Jul 2018 14:25:29 GMT):
Let me summarize to make sure we're all on the same page.
No changes to date with poet other than moving the exiting poet code and the validator registry TP to a new sawtooth-poet repo.
The poet2 development should probably be a branch in the sawtooth-poet repo.
This will facilitate reusing the TP and CLI.
I still recommend deprecating poet1 after poet2 is considered well reviewed and stable.
PoET1/Sim/CFT will have no particular benefit over PoET2 compiled with sgx simulation.
Maintaining both will just be confusing because we'll have a matrix of 4 poet flavors.
But maybe that naming part is different conversation.
bridgerherman (Fri, 27 Jul 2018 14:36:31 GMT):
Hi everyone, I've updated the PBFT consensus RFC again to provide a bit more information on states and state transitions, and to more accurately reflect the prototype implementation I've been working on: https://github.com/hyperledger/sawtooth-rfcs/pull/19
kelly_ (Fri, 27 Jul 2018 16:32:06 GMT):
since we are already on poet3.1 i recommend we rename poet2 to Poet NT
kelly_ (Fri, 27 Jul 2018 16:32:24 GMT):
NT can stand for network time vs the traditional trusted timers
kelly_ (Fri, 27 Jul 2018 16:33:24 GMT):
any future poet versions can be poet95, poetXP, etc
Dan (Fri, 27 Jul 2018 18:24:40 GMT):
:facepalm_tone1:
agunde (Fri, 27 Jul 2018 18:37:07 GMT):
@Dan @kelly_ @amundson @adamludvik @TomBarnes @pschwarz It would be great if we could get some more reviews on @bridgerherman PBFT rfc. Please take a look when you have time. Thanks!
zZz (Sat, 28 Jul 2018 09:16:29 GMT):
Has joined the channel.
henrytill (Sat, 28 Jul 2018 19:10:16 GMT):
Has joined the channel.
Dan (Tue, 31 Jul 2018 04:35:40 GMT):
Sorry for the delay on feedback here. I got part way through today and hopefully can finish up tomorrow. Meanwhile, @bridgerherman is the "Seal" concept part of the original PBFT paper or is that novel in this RFC? Thanks.
Dan (Tue, 31 Jul 2018 12:17:25 GMT):
@sergefdrv Hi, are we still on for your presentation of MinBFT on 8/2?
sergefdrv (Tue, 31 Jul 2018 12:31:26 GMT):
@Dan Yes, I'm preparing the slides :slight_smile:
Dan (Tue, 31 Jul 2018 12:51:45 GMT):
Excellent :)
bridgerherman (Tue, 31 Jul 2018 13:28:07 GMT):
@Dan As I recall, the original paper mentions a "consensus proof," but does not go into much detail. I read about Istanbul BFT consensus, which led me to believe that the "seal" is a good idea: https://github.com/ethereum/EIPs/issues/650
Dan (Tue, 31 Jul 2018 14:00:21 GMT):
thx
kelly_ (Tue, 31 Jul 2018 18:22:06 GMT):
@bridgerherman just went through the new RFC, looks great
kelly_ (Tue, 31 Jul 2018 18:22:44 GMT):
I have one question, do you think the RFC needs to document how a node that has been offline comes back online/syncs?
kelly_ (Tue, 31 Jul 2018 18:24:09 GMT):
for example, how does the node get from the last checkpoint to the current head
bridgerherman (Tue, 31 Jul 2018 18:29:55 GMT):
@kelly_ I'm currently working on a way to do that - the consensus API now has PeerConnected and PeerDisconnected updates, so theoretically the primary should be able to help the offline node sync back up with the network.
MicBowman (Tue, 31 Jul 2018 21:57:42 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=WGfAxLoGpY5iPoroD) @Dan FYI... poetV1 is still substantially more efficient than poetV2 (and more resilient to attack)
kelly_ (Tue, 31 Jul 2018 22:01:46 GMT):
@bridgerherman cool, got it!
Dan (Tue, 31 Jul 2018 22:12:36 GMT):
thanks @MicBowman. In terms of efficiency do you mean forking or what do you have in mind there?
I think we can only maintain one and that it will be too confusing for users to have ~4 poet variants. With the wider hardware footprint available to poet2 it seems we'll get more for our time with that protocol.
martinadolfi (Wed, 01 Aug 2018 12:31:42 GMT):
Has joined the channel.
diegos (Wed, 01 Aug 2018 18:12:44 GMT):
Has joined the channel.
Dan (Thu, 02 Aug 2018 14:12:20 GMT):
Please join us in less than an hour for a presentation on the consensus algorithm, MinBFT (https://github.com/nec-blockchain/minbft) by
Sergey Fedorov
sergefdrv (Thu, 02 Aug 2018 15:01:04 GMT):
https://drive.google.com/file/d/1LSNwYM2f7j-DRdCRvJu_K6JG6Oc468lr/view?usp=sharing
LeonardoCarvalho (Wed, 15 Aug 2018 20:43:13 GMT):
Has joined the channel.
LeonardoCarvalho (Thu, 16 Aug 2018 12:21:11 GMT):
Hello all. I've got some inquisitions about a way of getting Oracles on Sawtooth, is this the correct forum ? :)
amundson (Thu, 16 Aug 2018 19:35:34 GMT):
@LeonardoCarvalho here or sawtooth-core-dev should be fine, depending on the scope, etc.
LeonardoCarvalho (Fri, 17 Aug 2018 11:51:22 GMT):
Ah, ok.
LeonardoCarvalho (Fri, 17 Aug 2018 11:54:21 GMT):
So, let me explain the scopes I've asked about
Accountants sometimes need external confirmations on certain processes (fees, slips, etc), so II am inclined to consider this external confirmation as an Oracle, that all validators would confirm.
Being the validators sitting in external networks, the HTTP requests to confirm on the foreign servers would be safe from Eclipse Attacks, or simple forgeries.
LeonardoCarvalho (Fri, 17 Aug 2018 11:55:09 GMT):
So, I would like to bind the confirmation of each transaction to a external process, asynchronously
Gabe (Fri, 17 Aug 2018 20:25:42 GMT):
Has joined the channel.
tungdt_socoboy (Sat, 18 Aug 2018 17:20:48 GMT):
Hi @LeonardoCarvalho , should it be the External confirmations will sent a Tx to push the confirmation data into the TPs, then the TPs will based on received information, to process its terms and conditions?
tungdt_socoboy (Sat, 18 Aug 2018 17:22:10 GMT):
Actually, the TP could call to external services to get the data, but in term of its security and data trust, the external should sign-off a Tx and sent confirmation data into the blockchain instead. How do you think?
LeonardoCarvalho (Sat, 18 Aug 2018 17:50:15 GMT):
That would invalidate the access to the confirmation service from different networks. You see, if the validators do the job, it would be out of the TP control. Being on the TP control, you got out the distributed validation of the confirmation, and easily "eclipseable".
LeonardoCarvalho (Sat, 18 Aug 2018 17:52:38 GMT):
The current architecture supports the service being handled on the TP.
Dan (Mon, 20 Aug 2018 02:56:47 GMT):
@ltseeley in the Raft code (node.rs) where is RawNode defined? It looks like it comes from the raft namespace ... I thought I'd find a module in the tikv repo but I'm coming up empty grepping in there.
YueYin (Mon, 20 Aug 2018 04:30:01 GMT):
Has joined the channel.
ltseeley (Mon, 20 Aug 2018 14:16:33 GMT):
It's in the raft-rs repo: https://github.com/pingcap/raft-rs/blob/master/src/raw_node.rs
LakshmanTavag (Tue, 21 Aug 2018 19:15:14 GMT):
Has joined the channel.
alchmeina (Thu, 23 Aug 2018 14:16:26 GMT):
Has joined the channel.
amundson (Thu, 23 Aug 2018 15:56:30 GMT):
@amolk thanks for the presentation. that was really good.
amundson (Thu, 23 Aug 2018 15:59:32 GMT):
With respect to the SGX/non-SGX compilation flag decision, I don't think it is necessarily the right approach. It might be better to have two rust 'bin' projects and one 'lib' project that the bin projects use. all the code could go in the lib project, but the compilation time to produce both binaries would be roughly the same as it is for one binary. I think that would already be very close to what you are planning with less compilation time.
amundson (Thu, 23 Aug 2018 16:01:24 GMT):
maybe that doesn't solve the lower-level problem of conditional compilation within the library though, so its probably more complex than that
amundson (Thu, 23 Aug 2018 16:02:30 GMT):
though maybe that code should be at the bin levle
amundson (Thu, 23 Aug 2018 16:02:30 GMT):
though maybe that code should be at the bin level
jsmitchell (Thu, 23 Aug 2018 16:03:38 GMT):
I think rust projects also support multiple targets
jsmitchell (Thu, 23 Aug 2018 16:06:02 GMT):
https://doc.rust-lang.org/cargo/reference/manifest.html#configuring-a-target
jsmitchell (Thu, 23 Aug 2018 16:06:19 GMT):
you can have a [[lib]] section in addition to multiple [[bin]] sections
amundson (Thu, 23 Aug 2018 16:17:09 GMT):
yeah, that might be better if you can conditionally turn off one of the [[bin]] sections
amundson (Thu, 23 Aug 2018 16:25:30 GMT):
@amolk @Dan how does updating the key when the enclave restarts remove the need for the monotonic counter? what prevents a validator from retrying for a more attractive duration multiple times?
amolk (Thu, 23 Aug 2018 16:27:31 GMT):
The enclave will only create one Wait Certificate per block number - the block number-WaitCertificate mapping is maintained in the enclave. So an attacker cannot request the duration multiple times.
amolk (Thu, 23 Aug 2018 16:29:03 GMT):
An attacker may choose to unload the enclave, which will remove the mapping but also the keys. The C test will then prevent the attacker from trying to get a new duration immediately.
amundson (Thu, 23 Aug 2018 16:29:10 GMT):
ok, I was associating it with boot of the machine. but you mean initializing the enclave. so if we had a validator try and use the enclave multiple times, each would require a different key.
jsmitchell (Thu, 23 Aug 2018 16:29:32 GMT):
how does that one wait cert per block number work with forks that cause an N block rewind?
amolk (Thu, 23 Aug 2018 16:30:29 GMT):
But the EPID pseudonym can only be associated with a single enclave instance at a time, so a validator cannot create multiple instances simultaneously and pick whichever gives the best duration.
amundson (Thu, 23 Aug 2018 16:30:54 GMT):
@jsmitchell the enclave code wouldn't know about forks presumably
jsmitchell (Thu, 23 Aug 2018 16:31:30 GMT):
if we rewind N blocks, the network needs to republish new blocks reusing block numbers which have already been mapped in the enclaves
amundson (Thu, 23 Aug 2018 16:32:22 GMT):
yeah, I get your point, really you need previous block id, not block height
amundson (Thu, 23 Aug 2018 16:32:35 GMT):
block height is bogus out of context
amundson (Thu, 23 Aug 2018 16:33:54 GMT):
you could do a map of (genesis_block_id, height) -> wait_cert or (previous_block_id) -> wait_cert fine though
amundson (Thu, 23 Aug 2018 16:34:09 GMT):
well, no, the first doesn't work (duh)
jsmitchell (Thu, 23 Aug 2018 16:34:14 GMT):
come on
amundson (Thu, 23 Aug 2018 16:34:16 GMT):
LOL
amundson (Thu, 23 Aug 2018 16:35:36 GMT):
@amolk ok, so if you use previous_block_id->wait_cert, you need some mechanism to prevent attacks that consume memory (which could be caused in this case by excessive forking)
amundson (Thu, 23 Aug 2018 16:35:52 GMT):
like, by not having all that in memory
jsmitchell (Thu, 23 Aug 2018 16:36:18 GMT):
i guess you are just reusing the pre-rolled winner for that block height, which probably isn't that bad, but it would seem to be better to do it the other way
amundson (Thu, 23 Aug 2018 16:36:39 GMT):
is there an estimate of the amount of memory required if the enclave is up for a couple years doing this?
amolk (Thu, 23 Aug 2018 16:39:11 GMT):
One of the implications of the rewinding scenario in the current design is that the validator will not be able to publish any blocks until the fork catches up. Need to think about that.
amolk (Thu, 23 Aug 2018 16:40:02 GMT):
If we were to limit ourselves to simply the last block published and the corresponding wait certificate, then we need a fixed amount of memory.
jsmitchell (Thu, 23 Aug 2018 16:56:03 GMT):
@amolk that probably applies to all the validators, and is therefore a liveness issue
jsmitchell (Thu, 23 Aug 2018 19:08:03 GMT):
@amolk could you post those slides here, please?
blackbutterflysailing (Sat, 25 Aug 2018 15:49:32 GMT):
Has joined the channel.
blackbutterflysailing (Sat, 25 Aug 2018 15:49:42 GMT):
poet
sergefdrv (Mon, 27 Aug 2018 09:51:30 GMT):
minbft
LakshmanTavag (Mon, 27 Aug 2018 13:58:29 GMT):
Hello, I'm looking for some documentation on how to additional keys to the validator-registry for Poet ( simulator ). I've been able to add the first key in the genesis block using 'poet registration create', but I'm not sure how to add subsequent keys. Could anyone point me in the right direction here ?
jsmitchell (Mon, 27 Aug 2018 14:02:57 GMT):
@LakshmanTavag this channel is for discussing consensus development. Please ask your question in #sawtooth
LakshmanTavag (Mon, 27 Aug 2018 14:04:39 GMT):
Sure, appreciate it.
amolk (Tue, 28 Aug 2018 08:17:38 GMT):
PSE-Free POET v3.pdf
amolk (Tue, 28 Aug 2018 08:17:46 GMT):
@jsmitchell here you go.
amolk (Tue, 28 Aug 2018 08:23:04 GMT):
@jsmitchell @amundson With the proposed fork resolution mechanism, the order of checks is: longest fork, lowest ChainClock, lowest WaitCertificate.Duration. We will never be in a situation where we have to rewind the chain. A validator will always pick the longest fork even if a smaller fork has a lower CC. So a wait certificate will always be issued for blocks numbers higher than the last issued one.
jsmitchell (Tue, 28 Aug 2018 13:03:16 GMT):
No.
jsmitchell (Tue, 28 Aug 2018 13:03:16 GMT):
I don’t think that will work
jsmitchell (Tue, 28 Aug 2018 13:11:43 GMT):
A small clique which has published a slightly longer fork than the bulk of the network rejoins the network and its fork wins?
jsmitchell (Tue, 28 Aug 2018 13:12:25 GMT):
Therefore making the majority of the network replace their last N blocks?
luckycharms810 (Tue, 28 Aug 2018 14:25:34 GMT):
Has joined the channel.
amundson (Tue, 28 Aug 2018 14:45:58 GMT):
@amolk "We will never be in a situation where we have to rewind the chain." <-- can you elaborate on what you mean by this? I assume you are not saying that forks will not exist since you are talking about fork resolution
amolk (Tue, 28 Aug 2018 14:48:53 GMT):
What I mean is that if a block comes in that forks the chain, it will be persisted but the validator won't switch to that fork. The switch will happen only if the fork has equal or more blocks
amolk (Tue, 28 Aug 2018 14:49:14 GMT):
so we will not get into a situation where we switch to a fork that has lesser number of blocks
amundson (Tue, 28 Aug 2018 14:49:29 GMT):
longest fork being the primary method of fork resolution means only chain head comparisons matter, and comparisons should not done at a specific height (chain A, block 5 vs. chain B, block 5) like today. correct?
amolk (Tue, 28 Aug 2018 14:50:55 GMT):
For equal length chains we also do a chain clock comparison and switch to the fork with the lower chain clock
amolk (Tue, 28 Aug 2018 14:52:17 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=Tm3KTfvYFeA8C5wae) @amundson Only so far as both chains contain only 5 blocks. We will not compare blocks further down in the chain.
amolk (Tue, 28 Aug 2018 14:53:22 GMT):
This is a simpler mechanism. Of course, not having the SGX to enforce wait times means there's always a possibility for someone to game the system.
jsmitchell (Tue, 28 Aug 2018 14:54:14 GMT):
this seems to ignore the reality of split/rejoining networks
jsmitchell (Tue, 28 Aug 2018 14:55:06 GMT):
and in a literal sense, the network is really always N split networks separated by the transmission and processing latency
amundson (Tue, 28 Aug 2018 14:55:21 GMT):
it would certainly mean that blocks that where non-winning could becoming winning blocks later, just need a successful clique
jsmitchell (Tue, 28 Aug 2018 14:56:27 GMT):
@amolk lets say a 2-clique gets separated from the network and gets lucky with timers and published 100 more blocks than the mainline network
jsmitchell (Tue, 28 Aug 2018 14:57:14 GMT):
@amolk and then rejoins. Since chain length wins, the mainline network needs to apply that fork, and now all those nodes have to wait 100 blocks before they can publish again?
amundson (Tue, 28 Aug 2018 14:57:20 GMT):
and say those 2-clique where started at genesis to make it worse (they weren't initially connected to the network properly)
amolk (Tue, 28 Aug 2018 14:59:49 GMT):
@jsmitchell We still have the Z test though. Before blocks can be considered for consensus, they need to pass that test. Also, suppose we do switch to the other fork, other validators would be immediately able to publish from that chain head. The restriction on block numbers is only for preventing a validator going back, not going forward on the chain.
amundson (Tue, 28 Aug 2018 15:01:32 GMT):
@amolk would be cool to see a network run where we didn't run off z test :)
jsmitchell (Tue, 28 Aug 2018 15:02:22 GMT):
yeah, i mixed up the direction.
amundson (Tue, 28 Aug 2018 15:09:09 GMT):
@amolk the focus on only comparing chain head could result in a parallelization boost in transaction processing, by feeding more than one block to the parallel scheduler at a time. currently we serialize at block boundaries but if block-height comparisons are irrelevant, then there is no reason to not try and get that efficiency boost.
amolk (Tue, 28 Aug 2018 15:10:54 GMT):
That's an interesting thought. We can certainly give it a try.
jsmitchell (Tue, 28 Aug 2018 15:12:03 GMT):
@amolk in poet 1 testing we basically disable z-test because it can kill liveness with even 10 node networks
amundson (Tue, 28 Aug 2018 15:27:18 GMT):
FYI - I removed branch ryjones-patch-1 from sawtooth-raft repo
amundson (Tue, 28 Aug 2018 15:27:53 GMT):
which, as it turns out, is kind of an elaborate process
amundson (Tue, 28 Aug 2018 15:29:21 GMT):
The permissions and settings on sawtooth-raft have changed. The permissions now (almost) match the maintainers file, and the settings are more in-line with our other sawtooth repos.
kelly_ (Tue, 28 Aug 2018 20:30:50 GMT):
just chiming in here on the PoET2 thread - a very high level observation is that longest chain seems inconsistent with how nakamoto consensus is typically done, which is really 'most work' chain
kelly_ (Tue, 28 Aug 2018 20:32:38 GMT):
so if you had a long running fork, where there were 3 validators with a lower local mean (difficulty), and 7 validators at higher local mean (difficulty), for a given block height you should select the one with a higher local mean
kelly_ (Tue, 28 Aug 2018 20:33:59 GMT):
chain length can be manipulated by adding a lot of nodes after difficulty is reset
kelly_ (Tue, 28 Aug 2018 20:38:11 GMT):
or let's say a network forked at genesis, 3 nodes on one chain and 7 on another. if the 3 node chain is at 20 blocks, and the 7 node is at 10, the 7 node chain is still the 'most work' chain (3*20 < 10*7)
kelly_ (Tue, 28 Aug 2018 20:41:40 GMT):
otherwise you aren't getting security up to 49% attack, it is just 'luckiest' fork wins
kelly_ (Tue, 28 Aug 2018 21:09:20 GMT):
Bitcoin's block chain system is really two quite separate systems, and they are easily confused. The first one is the block tree and the second is the active chain.
The block tree consists of all valid blocks whose entire ancestry is known, up to the genesis block. The rules for validness include no double spending, valid signatures, no introduction of more currency than allowed, ... These are the network rules, and every full Bitcoin node verifies them.
The active chain is one path from genesis block at the top to some leaf node at the bottom of the block tree. Every such path is a valid choice, but nodes are expected to pick the one with the most "work" in it they know about (where work is loosely defined as the sum of the difficulties). Relativity and technological constraints prevent us from doing instant communication across the globe, so two nodes can not be expected to pick the same chain as the active one. This is no problem: the mining mechanism makes sure that the chance two nodes disagree about blocks in the past decreases exponentially as they are older.
So no, there is not one "right chain", there are many. Nodes choose for themselves, but the system is designed to make sure consensus arises quickly.
The rules in practice are this: when a new block arrives, and it extends the previous active chain, we just append it to the active chain. If not, it depends on whether the branch it extends now has more work than the currently active branch. If not, we store the block and stop. If it does have more work, we do a so called "reorganisation": deactivating blocks from the old branch, and activating blocks from the new branch.
kelly_ (Tue, 28 Aug 2018 21:10:06 GMT):
^thats a description of Bitcoin's system, so I guess the thing I'm talking about is making sure a long chain with low difficulty doesn't become the active chain
Dan (Thu, 30 Aug 2018 02:56:18 GMT):
@MicBowman can you shed light on the fork resolution prioritization of longest chain wins in poet2 vs. largest aggregate local mean in poet1?
MicBowman (Thu, 30 Aug 2018 20:22:09 GMT):
@dan in the spec we developed, the block contains the random number generated... that number can be used by each validator to compute the expected wait time (by computing the expected local mean and multiplying it by the random number). the nice part about the random number is that it abstracts the local mean calculation... so a small population will *probably* generate larger average random numbers than a large population. So... for equal length chains, you can always pick the one with the smaller sum of the random numbers
MicBowman (Thu, 30 Aug 2018 20:26:31 GMT):
if one chain has n blocks and the other m blocks (n>m), then another easy test... if the sum of the random numbers in chain 1 through block m is less than the sum of chain 2 (through its last block m), then you pick chain 1
MicBowman (Thu, 30 Aug 2018 20:29:12 GMT):
so the only interesting case is where you chain one has a higher random number sum than chain 2 through block m... question is do you wait for the next block on chain 2 (which still could be preferred over chain 1) or just take the longer chain
wchang (Fri, 31 Aug 2018 00:13:53 GMT):
Has joined the channel.
knkski (Fri, 31 Aug 2018 14:17:46 GMT):
Has joined the channel.
Dan (Fri, 31 Aug 2018 15:17:45 GMT):
Hmmm the ole SSoRNaSH method eh? (Smallest Sum of Random Numbers at Same Height) ;)
I'm not coming up with a reasonable case where I can challenge longest fork first is wrong.
One I thought about was a long-lived small clique, but that clique will be disadvantaged especially when it first forks and it believes its part of a larger network.
MicBowman (Fri, 31 Aug 2018 15:36:37 GMT):
@Dan you're pretty much where we've been... the distribution of the raw random numbers (which was implicit in the local mean/duration computation from poet v1) is a reasonable indicator for population size
MicBowman (Fri, 31 Aug 2018 15:37:15 GMT):
don't know if hart follows this channel, but he's been doing some work on alternate distributions and estimators that looks very promising
Dan (Fri, 31 Aug 2018 16:05:24 GMT):
I also don't know if @hartm follows this channel or if my blatant at'ing there will work ;)
hartm (Fri, 31 Aug 2018 16:05:24 GMT):
Has joined the channel.
sureshtedla (Fri, 31 Aug 2018 17:31:44 GMT):
Has joined the channel.
hartm (Fri, 31 Aug 2018 20:33:20 GMT):
Your at-ing does work! There are too many channels to follow these days...
hartm (Fri, 31 Aug 2018 20:34:56 GMT):
Yes, we have been doing a theoretical analysis of PoET. I can talk more about it if you like.
hartm (Fri, 31 Aug 2018 20:38:16 GMT):
The focus is on 1) a proof of security for the consensus algorithm and 2) mathematically rigorous statements about optimal distributions and estimators.
hartm (Fri, 31 Aug 2018 20:39:08 GMT):
We also have a couple of protocol tweaks that we think might help (but aren't sure yet) for certain networks.
Dan (Fri, 31 Aug 2018 21:28:35 GMT):
Yes way too many channels. :)
Definitely interested to know more about both points.
This current thread of discussion is about fork resolution for what we're calling PoET 2, which is PoET that doesn't rely on the TEE to enforce wait time, only relying on the TEE to generate the fair random.
The assertion in this model is that the longest chain is the best fork resolution decision. When their are two equal length chains, sum the random numbers and choose the smallest sum (as the randoms are analogs of wait times).
Previously we relied on population estimates, but those are also just functions of the distribution of the random numbers so in a sense summing the randoms is more direct.
If you had some thoughts on the distributions that would effect that thinking please chime in.
And if not sorry to have tricked you into reading this much babble. :)
Dan (Fri, 31 Aug 2018 21:30:31 GMT):
@amolk DM'd me that he has some updates based on @MicBowman 's feedback but DCO or something is in the way. Anyway the current version of the spec is here: https://github.com/hyperledger/sawtooth-rfcs/pull/20
hartm (Fri, 31 Aug 2018 21:40:28 GMT):
OK. I'm going to be gone for the next week and a half, but I was planning on talking to @MicBowman about some of this stuff when I get back. I'd be happy to include you if you like.
hartm (Fri, 31 Aug 2018 21:48:36 GMT):
Wait, you're just going on longest chain? It seems to me like the algorithm wouldn't work in this case--a small group could fork, wait for the "difficulty" to go down, and then add in a bunch of new members and use these to pass the main chain in terms of length. Why isn't this a valid attack? I don't see how you can avoid incorporating a population estimate (or some correlated statistic) to compare forks of different lengths if the "mining difficulty"
is allowed to change quickly.
jsmitchell (Fri, 31 Aug 2018 22:48:00 GMT):
👆
LeonardoCarvalho (Sat, 01 Sep 2018 12:25:01 GMT):
Guys, I was talking with a friend, and got a interesting thought... Bitcoin (like all cryptocoins) is based on a finite set of resources that people compete (mine) for, and the competition generate the chain.
Sawtooth, in the other hand, is based on trust of the information inside the blocks, in a no-competing way.
I think that, based on that, the concept of "longest chain" could be changed to something more like the Channels on Fabric.
Say, if a Validator was kept out of the network for, say, 1 hours, and only get transactions that it deemed valid and only in addresses that weren't seen on the "main block", why would it drop all of them ?
LeonardoCarvalho (Sat, 01 Sep 2018 12:25:23 GMT):
"main chain", I mean
amundson (Mon, 03 Sep 2018 16:18:49 GMT):
@hartm if you haven't had a chance to read through it, there is a PoET 2 RFC doc here - https://github.com/hyperledger/sawtooth-rfcs/pull/12 ; it is nearing acceptance I think, but now is still a good time to add comments and/or questions
hartm (Mon, 03 Sep 2018 16:55:30 GMT):
@amundson Thanks for the link! I added a question.
MicBowman (Tue, 04 Sep 2018 16:04:50 GMT):
@hartm see above for what I mean by "longest chain"
MicBowman (Tue, 04 Sep 2018 16:06:25 GMT):
we keep the raw random number generated in the blocks & use that as the differentiator... that is... its the pre-difficulty level differentiator
MicBowman (Tue, 04 Sep 2018 16:08:13 GMT):
the only question that is really of issue is under what circumstances a short chain (that had a lower sum of random numbers at equal length... which kills all of the small group attacks) should be chosen over a longer chain
tommling (Tue, 04 Sep 2018 22:09:25 GMT):
Hi I am experiencing some difficulties with the sawtooth_sdk.consensus.engine interfaces. Specifically related to the message types being passed around by ZMQ. I noticed in the ZMQ driver can process peer connect/disconnect events. I based my implementation of engine off of the https://github.com/hyperledger/sawtooth-poet.git repository.
My bootstrap/genesis node keeps outputting the following logs when I try to add more peers. They negotiate their connection fine on the validator logs, but this is what I see on the engine logs:
```
[2018-09-04 21:24:17.519 DEBUG selector_events] Using selector: ZMQSelector
[2018-09-04 21:24:22.913 INFO oracle] Loading signing key: /etc/sawtooth/keys/validator.priv
[2018-09-04 21:24:22.914 DEBUG selector_events] Using selector: ZMQSelector
[2018-09-04 21:24:23.019 DEBUG consensus_state_store] Create consensus store: /var/lib/sawtooth/poet_consensus_state-024a58d5.lmdb
[2018-09-04 21:24:23.021 DEBUG cs_key_state_store] Create PoET key state store: /var/lib/sawtooth/poet-key-state-024a58d5.lmdb
[2018-09-04 21:24:23.039 INFO cs_enclave_factory] Load PoET enclave module: chronoshift_simulator.chronoshift_enclave_simulator.chronoshift_enclave_simulator; Target wait time: 20.000000; Initial wait time: 5.000000; Population estimate sample size: 50;
[2018-09-04 21:24:23.220 DEBUG chronoshift_enclave_simulator] PoET enclave simulator creating anti-Sybil ID from: 2018-09-04T21:24:23.220345
[2018-09-04 21:24:23.225 DEBUG cs_block_publisher] PoET public key 02be3692...a1863208 in validator registry not found in key state store. Sign up again
[2018-09-04 21:24:23.228 INFO cs_block_publisher] Register Validator Name=validator-024a58d5, ID=024a58d5...7ff7d950, PoET public key=033c8941...8624e8aa, Nonce=1aea563f474783d2627b45912161aa06
[2018-09-04 21:24:23.234 INFO cs_block_publisher] Save key state PPK=033c8941...8624e8aa => SSD=eyJjb3Vu...OGFhIn0=
[2018-09-04 21:25:20.279 DEBUG engine] Received message: CONSENSUS_NOTIFY_PEER_CONNECTED
[2018-09-04 21:25:20.279 ERROR engine] Unknown type tag: CONSENSUS_NOTIFY_PEER_CONNECTED
```
Any Ideas? The logs on the validator side during this negotiation looks like this :
```
[2018-09-04 21:25:18.256 DEBUG base] ALLOWED (CURVE allow any client)
[2018-09-04 21:25:18.256 DEBUG base] ZAP reply code=b'200' text=b'OK'
[2018-09-04 21:25:18.257 DEBUG handlers] got connect message from 4aeb992cce80c263c6acc67cb976278a3b67dd7ad927c3ceb855f6d091a25b56af99936f6c6405712e496781d889b1c807d21dd0ec5c42ac94b0c5400df066ef. sending ack
[2018-09-04 21:25:18.258 DEBUG handlers] Endpoint of connecting node is tcp://berlin-service.tommling.svc:8800
[2018-09-04 21:25:18.258 DEBUG interconnect] Determining whether inbound connection should be allowed. num connections: 1 max 100
[2018-09-04 21:25:18.258 DEBUG handlers] Allowing incoming connection: 4aeb992cce80c263c6acc67cb976278a3b67dd7ad927c3ceb855f6d091a25b56af99936f6c6405712e496781d889b1c807d21dd0ec5c42ac94b0c5400df066ef
[2018-09-04 21:25:18.262 DEBUG handlers] Connection: 4aeb992cce80c263c6acc67cb976278a3b67dd7ad927c3ceb855f6d091a25b56af99936f6c6405712e496781d889b1c807d21dd0ec5c42ac94b0c5400df066ef is approved
[2018-09-04 21:25:18.274 DEBUG gossip_handlers] Got peers request message from 4aeb992cce80c263c6acc67cb976278a3b67dd7ad927c3ceb855f6d091a25b56af99936f6c6405712e496781d889b1c807d21dd0ec5c42ac94b0c5400df066ef
[2018-09-04 21:25:18.279 DEBUG handlers] got disconnect message from 4aeb992cce80c263c6acc67cb976278a3b67dd7ad927c3ceb855f6d091a25b56af99936f6c6405712e496781d889b1c807d21dd0ec5c42ac94b0c5400df066ef. sending ack
```
Dan (Tue, 04 Sep 2018 22:10:41 GMT):
are you more or less on head of sawtooth-core?
hartm (Tue, 04 Sep 2018 22:17:46 GMT):
@MicBowman That's exactly the question I'm asking. Suppose f(chain, block) = the sum of all the population estimates for all of the blocks after the input block. Why couldn't you just always take the fork with the higher value of f?
hartm (Tue, 04 Sep 2018 22:26:43 GMT):
This is the most direct analogy to picking which fork has had the most "work" done on it.
MicBowman (Tue, 04 Sep 2018 23:39:40 GMT):
@hartm i just sent the analysis that we did (a year ago) on this. it should have come through on the comments on the rfc
MicBowman (Tue, 04 Sep 2018 23:41:19 GMT):
a lot of this is trying to reconstruct the logic that went behind the design decisions... i definitely need to write better notes to myself
Dan (Wed, 05 Sep 2018 13:58:39 GMT):
_(also pick up milk today)_
tommling (Wed, 05 Sep 2018 20:11:28 GMT):
@Dan Yeah, if that was directed towards me. Was there a recent change that introduced these messages?
Dan (Wed, 05 Sep 2018 20:16:10 GMT):
I recall something in the commit history (about consensus engine connections not milk) lemme see if I can pull it up.
tommling (Wed, 05 Sep 2018 20:17:37 GMT):
Many thanks
Dan (Wed, 05 Sep 2018 20:17:59 GMT):
I think this is what i was thinking of... not real recent
```commit 5b2465073f2444e2be9787ccb63cf908575de1c0
Author: Logan Seeley
tommling (Wed, 05 Sep 2018 20:23:41 GMT):
I see
tommling (Wed, 05 Sep 2018 20:23:58 GMT):
This may be affecting us because I am using the nightly builds for my engine
hartm (Wed, 05 Sep 2018 22:36:44 GMT):
@MicBowman Can you send it again? I didn't get it. And is there a simple answer why what I proposed won't work? Thanks!
Dan (Thu, 06 Sep 2018 01:18:36 GMT):
I think short answer is that the pop est. is derived from the randoms so it's all functions of the same source information. Mic has a more thorough reply though. https://github.com/hyperledger/sawtooth-rfcs/pull/20
qiang0723 (Thu, 06 Sep 2018 01:27:15 GMT):
Has joined the channel.
hartm (Thu, 06 Sep 2018 09:23:21 GMT):
@Dan Thanks! I'll respond to him there.
MicBowman (Thu, 06 Sep 2018 14:12:12 GMT):
@hartm not sure how best to have the discussion... but just to point out the that your W effectively reduces to sum of the random numbers, right?
MicBowman (Thu, 06 Sep 2018 14:15:05 GMT):
not exactly... because of the minimum of a set of number taken from uniform distribution is not uniform... 1/(n+1) right?
MicBowman (Thu, 06 Sep 2018 14:17:19 GMT):
that is E(min(x1...xn)) == 1/(n+1) if xn are uniformly distributed 0 to 1
hartm (Fri, 07 Sep 2018 06:41:25 GMT):
@MicBowman Not exactly. If my fork has more blocks with lower random numbers and yours has fewer blocks with higher random numbers, the sum would be the same, but my W would be much higher than yours. This is intended behavior--my fork should clearly win here.
hartm (Fri, 07 Sep 2018 06:43:27 GMT):
If we did the (terrible, and not good) population estimator of 1 / (random number) * c for some constant c, then W would reduce to something like c * \sum_{i} { 1/ rand_{i}}, which is clearly not the same as summing random numbers.
hartm (Fri, 07 Sep 2018 06:43:56 GMT):
(Ugh, my "*" got interpreted as bold()
hartm (Fri, 07 Sep 2018 06:44:44 GMT):
(And my statement should say, "the sum *could* be the same" not *would*). It's early in Europe.
hartm (Fri, 07 Sep 2018 10:03:12 GMT):
To "sum" this up: suppose I have a fork with 8 blocks with numbers (1, 2, 2, 3, 2, 1, 3, 1). Now suppose Mic has a fork with 4 blocks with numbers (4, 4, 3, 5). Note that the sum of the random numbers in both cases is 15, but my fork should clearly win. If we use the (bad!!) population estimator c (x) 1/r (where r is the randomness value), my value for W = c (x) [ 1/1 + 1/2 + 1/2 + 1/3 + 1/2 + 1/1 + 1/3 + 1/1] whereas Mic's W = c (x) [1/4 + 1/4 + 1/3 + 1/5]. Clearly my W is higher than Mic's as desired.
hartm (Fri, 07 Sep 2018 10:03:42 GMT):
Is this making sense at all (Mic or others)?
MicBowman (Fri, 07 Sep 2018 14:53:44 GMT):
@hartm but if you look at my description... everything is couched in terms of chain length... the only case that is of any interest at all is where a shorter chain has a higher population estimate for an equal length prefex... and in that case, you have to drop the math and start thinking about systems... *why* did it end up that way
hartm (Sun, 09 Sep 2018 08:29:44 GMT):
Hmmmm. My point is that "total chain work" is a much better first metric than chain length. I'm in rural Bavaria right now and my internet is approximately a trickle, but I'll follow up later on this.
raviyelleni (Sun, 09 Sep 2018 09:09:10 GMT):
Has joined the channel.
arsulegai (Tue, 11 Sep 2018 18:57:48 GMT):
Has joined the channel.
hartm (Thu, 13 Sep 2018 18:55:42 GMT):
OK, finally back in the office. Would people be OK having a discussion on this at the hackfest in Montreal (and do we have enough of the core people to discuss this)? I've managed to convince myself that judging which fork to take based on "total work" is a better metric than going first with chain length, then considering random numbers (they may be equivalent, but this is non-obvious and needs to be proven if it is indeed the case), but I doubt I'll be able to convince others without a formal paper or time in-person.
kelly_ (Thu, 13 Sep 2018 19:08:52 GMT):
I think @Dan @MicBowman and I will be in Montreal
Dan (Thu, 13 Sep 2018 19:12:30 GMT):
sounds good to me
kelly_ (Thu, 13 Sep 2018 19:15:44 GMT):
@hartm i'm with you on the total work thing. I also don't doubt that Mic is right re: chain length (cause he's a smart dude) but it's not intuitive to me
kelly_ (Thu, 13 Sep 2018 19:16:10 GMT):
I'm also not a PoET2 expert either though..
hartm (Thu, 13 Sep 2018 19:17:34 GMT):
Awesome, let's chat in Montreal then.
hartm (Thu, 13 Sep 2018 19:20:24 GMT):
@kelly_ Yeah, it's entirely possible that the cases I know that are "bad" for the chain length metric are not reachable in practice, but we need to prove this if we want to use the chain length metric (for instance, if somehow I have a fork with numbers 1,2,1,2,2 and Mic has a fork with numbers 4,5,6,8,1 then Mic's policy would dictate we take his fork and not mine, although we should probably take mine--but this fork pattern is unlikely since my fork will have probably existed in its current state before Mic's third and fourth blocks).
Dan (Fri, 14 Sep 2018 13:10:33 GMT):
temporary network segmentation is a reasonable consideration
hartm (Fri, 14 Sep 2018 17:08:04 GMT):
@Dan Yeah, I think I can prove security if we fork based on a "total work" metric. I'm not sure about the case where you go with chain length first, and I'm worried that there may in fact be network segmentation attacks if the chain length forking policy doesn't agree with the total work policy.
Southside (Fri, 14 Sep 2018 18:35:03 GMT):
This worries me......
https://people.cs.kuleuven.be/~jo.vanbulck/ccs18.pdf
What is the reaction from the folk in the know?
Southside (Fri, 14 Sep 2018 18:35:35 GMT):
Is this the effective end of SGX and hence PoET?
Southside (Fri, 14 Sep 2018 18:55:08 GMT):
My apologies if this sounds somewhat alarmist but it seems like a serious vulnerability - Hopefully you good people can reassure me but right now, I need to look at alternatives to PoET
hartm (Fri, 14 Sep 2018 19:09:40 GMT):
@Southside You're a paper behind! The current paper is much more damaging: https://www.usenix.org/system/files/conference/usenixsecurity18/sec18-van_bulck.pdf. However, these are mostly just system "bugs" and I believe that the most damaging attacks are in the process of being patched, if they haven't been patched already (someone from Intel on this list can say more). It's definitely not the end of SGX, but definitely qualifies as a "zero-day" bug.
hartm (Fri, 14 Sep 2018 19:11:06 GMT):
@Southside I take that back. Sorry! You're not a paper behind, I was too hasty.
hartm (Fri, 14 Sep 2018 21:15:27 GMT):
@Southside It is a serious vulnerability, but I doubt it will be the effective end of SGX (or trusted hardware in general, since the attack applies to a lot of trusted hardware). If you're really concerned, you may want to PM some of the Intel people on this channel.
MicBowman (Fri, 14 Sep 2018 22:12:25 GMT):
PoET is designed to accommodate non-scalable attacks on the TEE... and many who run PoET today, do so without SGX (the z-test is very effective in any permissioned environment)
Southside (Sat, 15 Sep 2018 16:04:30 GMT):
Thank you @MicBowman and @hartm
amundson (Mon, 17 Sep 2018 16:52:52 GMT):
Sawtooth PBFT component is being (formally) proposed in #sawtooth-governance. Related to https://github.com/hyperledger/sawtooth-rfcs/pull/19 and https://github.com/bitwiseio/sawtooth-pbft.
adamludvik (Tue, 18 Sep 2018 15:39:55 GMT):
@tommling RE: `[2018-09-04 21:25:20.279 ERROR engine] Unknown type tag: CONSENSUS_NOTIFY_PEER_CONNECTED `, I believe in the PoET engine that this should actually be logged at the WARNING or INFO level, as the message should is just ignored (as it should be with PoET). In general, if you are building a new consensus engine, I would not look to the Sawtooth PoET Python implementation as a good reference, as it is adapting an old interface to a new interface. It would be better to take a look at how the Sawtooth Raft or Sawtooth PBFT engines were implemented, as these were built from the ground up to work with the new API.
tommling (Tue, 18 Sep 2018 20:07:16 GMT):
@adamludvik Thank-you for the response. I became that it was not an error after I got the endpoints correct. The consensus algorithm I wrote actually just hybridizes another transaction family with PoET. Now there is a Stake transaction family and a Registry transaction family. The stake addresses are used instead of the anti sybil id, because the stake is finite it can be used instead.
tommling (Tue, 18 Sep 2018 20:07:16 GMT):
@adamludvik Thank-you for the response. I became aware that it was not an error after I got the endpoints correct. The consensus algorithm I wrote actually just hybridizes another transaction family with PoET. Now there is a Stake transaction family and a Registry transaction family. The stake addresses are used instead of the anti sybil id, because the stake is finite it can be used instead.
tommling (Tue, 18 Sep 2018 20:07:16 GMT):
@adamludvik Thank-you for the response. I became aware that it was not an error after I got the endpoints correct. The consensus algorithm I wrote actually just hybridizes another transaction family with PoET. Now there is a Stake transaction family and a Registry transaction family. The stake addresses are used instead of the anti sybil id, because the stake is finite and unique it can be used instead.
amundson (Tue, 18 Sep 2018 21:49:20 GMT):
@Dan @kelly_ @hartm based on the above conversation, is it correct that we are not ready yet for an FCP vote for the PoET RFC, and that we will do one after the hackfest?
Dan (Tue, 18 Sep 2018 23:52:57 GMT):
If we are going to resolve this at the hackfest we'll want it on the agenda.
Dan (Tue, 18 Sep 2018 23:55:05 GMT):
Ok, added to an almost empty agenda
amundson (Wed, 19 Sep 2018 00:05:10 GMT):
do we know of any other open issues with the RFC that we need to discuss?
hartm (Wed, 19 Sep 2018 00:40:17 GMT):
Have you all looked at the following paper: https://eprint.iacr.org/2016/1048.pdf? There are some interesting attacks on the general bitcoin protocol without modifications (see https://eprint.iacr.org/2013/868.pdf for an example of such an attack on the difficulty mechanism) that I'd like to make sure can't be used on PoET. I'd like to make sure that these kinds of attacks can't be done, even in the presence of a (small-ish) number of compromised enclaves.
hartm (Wed, 19 Sep 2018 00:53:16 GMT):
I have another question. Is there an up-to-date document explaining the z-test? I can find the one for PoET 1, but the PoET 2 spec doesn't have it. I'm a bit concerned about the following fact: the z-test only addresses block wins rather than the randomness sampled. I'm worried that some of the block difficulty attacks on the bitcoin backbone can get amplified big time here, since we are estimating population based on the randomness sampled, and the z-test documentation that I found doesn't address the randomness itself, other than the block wins. Intuitively, if I'm sampling a 256-bit number, and I sample, say, 1, then the z-test should toss this. Otherwise the impact on the population estimate may be huge, particularly if I have enough compromised enclaves to cover the population estimator's window. There are relatively easy mitigations to this attack, but it still might be good to check by *how much* someone is winning blocks.
hartm (Thu, 20 Sep 2018 14:01:07 GMT):
I'll ask again: is this the current z-test being used: https://sawtooth.hyperledger.org/docs/core/releases/1.0/architecture/poet.html#z-test ?
hartm (Thu, 20 Sep 2018 14:01:08 GMT):
Thanks!
kelly_ (Thu, 20 Sep 2018 14:09:44 GMT):
@amolk ^
amolk (Thu, 20 Sep 2018 14:41:41 GMT):
Yes
kelly_ (Thu, 20 Sep 2018 15:00:45 GMT):
I think @hartm was also wondering if it has changed since 1.0, and if there is documentation on z-test in 2.0
amolk (Thu, 20 Sep 2018 17:07:33 GMT):
We're using the same Z-test. There's no separate documentation about the various tests in the PoET 2 RFC.
hartm (Thu, 20 Sep 2018 17:30:56 GMT):
@amolk @kelly_ Thanks! I discussed this with Mic some this morning and we're planning on having a follow-up discussion tomorrow.
Dan (Thu, 20 Sep 2018 20:52:40 GMT):
@adamludvik @ltseeley have we done any crash fault tests with sawtooth-raft?
adamludvik (Thu, 20 Sep 2018 21:01:39 GMT):
Yeah, Raft continues to make progress when an engine goes down but the raft engine was not restarting correctly. We have been spending the last week fixing it and this should be the last change needed: https://github.com/hyperledger/sawtooth-raft/pull/30
adamludvik (Fri, 21 Sep 2018 20:28:30 GMT):
I have updated the Consensus Engine API RFC to reflect changes made during implementation. Going to work on documentation next: https://github.com/hyperledger/sawtooth-rfcs/pull/4/commits/7355af743cd0c15674318a7fbd0e8b9112c440a4
adamludvik (Fri, 21 Sep 2018 20:32:15 GMT):
@Dan ^
Dan (Tue, 25 Sep 2018 15:21:38 GMT):
@ltseeley what is the longest LR run you've done with sawtooth-raft?
ltseeley (Tue, 25 Sep 2018 15:22:32 GMT):
The longest run I recorded was almost 17 hours
Dan (Tue, 25 Sep 2018 15:23:00 GMT):
cool. did that end at 17hrs because of failure or were you just done with some other criteria?
ltseeley (Tue, 25 Sep 2018 15:24:07 GMT):
I stopped it to run it with different parameters; no failures.
ltseeley (Tue, 25 Sep 2018 15:24:27 GMT):
Are you interested in running a Raft network for a longer period of time?
Dan (Tue, 25 Sep 2018 15:25:02 GMT):
Our release criteria for sawtooth 1.1/bumper is raft passing LR1
ltseeley (Tue, 25 Sep 2018 15:30:48 GMT):
Got it. Couple questions: is there a specific node count that is required for this LR1 run to meet the criteria? And has there been any discussion yet on a particular commit to use for the release?
Dan (Tue, 25 Sep 2018 15:39:35 GMT):
hmmm good questions. I don't have a strong opinion on the node count. Top end would be 10. I feel like 5 is low end. People might run 3 node nets but that seems kind of pointless for a blockchain.
Dan (Tue, 25 Sep 2018 15:41:28 GMT):
As far as commit.. one criteria is whether we introduce new dependencies that need a licensing scan. I recall the LRU addition recently. I think that's already being scanned for sawtooth core but not completely sure about that.
amundson (Tue, 25 Sep 2018 15:44:46 GMT):
the commit will be the head of master at the time we tag it
thou_shalt (Thu, 27 Sep 2018 06:38:42 GMT):
Has joined the channel.
thou_shalt (Thu, 27 Sep 2018 06:46:02 GMT):
Hello, can somebody help with instructions how to write own consensus engine using pyhon sdk from sawtooth_sdk.consensus .Some simple example for instance .
Dan (Thu, 27 Sep 2018 12:40:22 GMT):
the only example atm is poet. poet is shimmed in from the approach that preceded engines so it's not a good example. The more recent examples are written in rust. This design doc may be of help to you:
https://github.com/hyperledger/sawtooth-rfcs/pull/4/files
sivak2018 (Sat, 29 Sep 2018 04:25:52 GMT):
Has joined the channel.
AshishBanerjee (Sat, 29 Sep 2018 07:37:57 GMT):
Has joined the channel.
fz (Sun, 30 Sep 2018 01:50:01 GMT):
Has joined the channel.
phaniac (Mon, 01 Oct 2018 18:17:45 GMT):
Has joined the channel.
adamludvik (Wed, 03 Oct 2018 16:07:01 GMT):
@Dan @kelly_ @amundson Any objections to moving the consensus RFC to FCP? https://github.com/hyperledger/sawtooth-rfcs/pull/4
Dan (Wed, 03 Oct 2018 17:19:09 GMT):
I think we all need to vote to accept it first? and then we move it to FCP?
Dan (Wed, 03 Oct 2018 17:19:09 GMT):
I think we all need to vote to accept it first and then we move it to FCP 7 days therafter.
Dan (Wed, 03 Oct 2018 17:33:42 GMT):
n/m I think we had all thumbs up or approved the PR in the past and then moved to FCP. It doesn't explicitly say that in the governance RFC though. It's just when we hit a steady state that the shepherd proposes FCP.
adamludvik (Wed, 03 Oct 2018 18:21:34 GMT):
Having written the process down, you'd think I would remember it..
amolk (Thu, 04 Oct 2018 05:29:02 GMT):
@hartm @MicBowman Did you get a chance to discuss the fork resolution for PoET at Montreal?
hartm (Thu, 04 Oct 2018 12:14:40 GMT):
@amolk It's on the agenda for today.
bur (Mon, 08 Oct 2018 08:31:47 GMT):
Has joined the channel.
hedrickw (Mon, 08 Oct 2018 14:40:41 GMT):
Has joined the channel.
hedrickw (Mon, 08 Oct 2018 14:56:33 GMT):
ltavag
Dan (Mon, 08 Oct 2018 20:18:12 GMT):
Here's my notes @amolk https://lists.hyperledger.org/g/tsc/message/1697
amolk (Tue, 09 Oct 2018 01:48:44 GMT):
Thanks @Dan @MicBowman @hartm for the clarification on the fork resolution. So to summarize, the winner will be the fork having the larger population, rather than the longest chain. Ties will be broken by the smaller duration value in the wait certificate.
hartm (Tue, 09 Oct 2018 04:19:43 GMT):
Just to be clear, it's the sum of the population estimates of all of the blocks on the fork.
adamludvik (Wed, 10 Oct 2018 00:56:07 GMT):
Apologies in advance for the long post.. calling @Dan, @MicBowman, @hartm, @jsmitchell, @amundson for your input on the following consensus problem.
I am researching if and how consensus algorithms solve the "fair ordering" problem. In general, fair ordering means that a Byzantine node can not order transactions in a way that unfairly benefits that node. This problem is not interesting in lottery-type algorithms, because the probability of a single node winning and being able to take advantage of unfair ordering is small enough that it is ignored.
This problem is important in voting-type algorithms with a long-term leader, such as many PBFT variants. A malicious leader can, in some cases, participate in a protocol perfectly but manipulate the ordering to its benefit by:
1. Manipulating the order of a given set of transaction to its benefit
2. Generating and inserting new transactions into the ordering to its benefit
3. Withholding submitted transactions from the ordering to its benefit
One thing that makes this problem hard is that, without application-specific knowledge from the transactions, it is difficult to tell if an ordering actually benefits the leader or if "random noise" caused the ordering to be substantially different than the expected ordering.
I have been able to come up with two partial solutions to this problem within the context of solving it for sawtooth-pbft. The first is to have followers compute a heuristic or "probability that the ordering is fair" and to start a new election if they determine the leader appears to be manipulating the order. The second is inspired by the original PBFT paper by Castro and Liskov and applies to the ordering of batches within blocks.
I have described one example of a heuristic that prevents excluding batches indefinitely here: https://gist.github.com/aludvik/164a9c0a7419b758e54190c4a0dfa72b This solution is somewhat inspired by the PoET z-test.
For the second solution, the relevant portion of the PBFT paper is section 4.6 on non-determinism. (Paper found here: http://pmg.csail.mit.edu/papers/osdi99.pdf) If we treat the batch ordering as non-deterministic (because of network latency, dictionary iteration order, etc.), then we can use some application of what is described in 4.6. An example would be to have all followers vote on the batch ordering for the next block before it is published and to require that the leader decide which batches to include by some deterministic computation over 2f+1 of the votes. This creates a smaller new problem to deal with, which is that malicious followers can vote for bogus batch ids, but I think a clever computation should be able to resolve it.
Has anyone else out there considered this problem in any detail? Does anyone have any thoughts or feedback on the two solutions I proposed, or have any ideas for extending them that would improve how they handle the problem?
adamludvik (Wed, 10 Oct 2018 01:19:39 GMT):
Sorry, and @kelly_
kodonnel (Wed, 10 Oct 2018 16:04:07 GMT):
Has joined the channel.
MicBowman (Wed, 10 Oct 2018 16:11:30 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=cxw2hPBwY3yzrjYKp) @amolk And to be clear... any time the longest chain is not chosen we need to engage a more thorough evaluation of *why* we got there because that is a potential indication of a more serious problem
hartm (Wed, 10 Oct 2018 16:17:56 GMT):
Mic is correct. If the longest chain is not the chain with the "most work done" or the "highest difficulty chain" as is put in the literature, then it seems like there is almost certainly adversarial activity.
kelly_ (Wed, 10 Oct 2018 20:23:02 GMT):
@adamludvik I haven't had a chance to look at your proposals but I can think of a couple techniques that could maybe partially solve the problem, though they would add a bunch of complexity
kelly_ (Wed, 10 Oct 2018 20:23:43 GMT):
one would be some sort of commit reveal scheme where commitments are ordered and then the reveals are applied in the order of the commitments
kelly_ (Wed, 10 Oct 2018 20:23:52 GMT):
there are probably a bunch of edge cases there
kelly_ (Wed, 10 Oct 2018 20:24:53 GMT):
the other cryptographic technique you could potentially use to prevent item #2 in your list would be requiring some sort of verifiable delay function to be added as a 'proof-of-wait' to a transaction before it could be included
kelly_ (Wed, 10 Oct 2018 20:25:23 GMT):
if this function was longer than the pbft timeout then maybe the leader couldn't put a bunch of new transactions into a block he is creating
kelly_ (Wed, 10 Oct 2018 20:26:05 GMT):
but there are also ways around that, e.g. withhold one transaction while you generate the 'proof-of-wait'
kelly_ (Wed, 10 Oct 2018 20:26:47 GMT):
a third idea would be to force view changes on a regular basis so you limit the impact of the leader
kelly_ (Wed, 10 Oct 2018 20:27:32 GMT):
I don't think 'front-running' has really been solved for any consensus algo (including nakamoto-style), but to be fair i haven't really researched it
hartm (Wed, 10 Oct 2018 20:33:54 GMT):
@kelly_ @adamludvik Our front-running strategy has been to use time-locked encryption for blockchain. Basically, you encrypt a transaction which is automatically decrypted at a certain block (i.e., in the consensus proof, there is a threshold signature done on the block number. This signature can be used as a key to decrypt transactions encrypted to the block utilizing the standard IBE/signature duality). This only works for traditional BFT algorithms though, where you can have a supemajority of the BFT nodes sign something that is known in advance.
hartm (Wed, 10 Oct 2018 20:34:35 GMT):
This doesn't work against traffic analysis, etc., but is a starting point.
kelly_ (Wed, 10 Oct 2018 20:35:05 GMT):
oh that's net
kelly_ (Wed, 10 Oct 2018 20:35:07 GMT):
neat*
hartm (Wed, 10 Oct 2018 20:36:39 GMT):
In general, you'll need to rigorously define what a fair ordering is before you can prove things about it. For instance, something like "a transaction received by 2/3 n + 1 of the nodes before another one always precedes it in the consensus output" or a similar flavor.
adamludvik (Wed, 10 Oct 2018 21:06:26 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=BT8iZsNpDbpXEhRju) This is basically what Tendermint does: https://www.tendermint.com/docs/introduction/introduction.html#consensus-overview
adamludvik (Wed, 10 Oct 2018 21:06:44 GMT):
They have a new node propose the block at each height.
adamludvik (Wed, 10 Oct 2018 21:07:03 GMT):
@hartm good point about formalizing the definition of fair.
adamludvik (Wed, 10 Oct 2018 23:00:05 GMT):
@hartm your front-running strategy sounds interesting. Can you point me at anything that has more detail on how it works?
hartm (Wed, 10 Oct 2018 23:20:42 GMT):
@adamludvik I'll try to find the detailed materials. We unfortunately don't have a paper--it was intended as something "directly to business".
Dan (Thu, 11 Oct 2018 17:06:14 GMT):
can you spot the elections?
Dan (Thu, 11 Oct 2018 17:06:18 GMT):
block publishing on raft looks cool. I should have been doing more LR review before.
Dan (Thu, 11 Oct 2018 17:31:34 GMT):
@pankajgoyal what params are you using on that raft_new network? It looks like a lot of leader change. (not sure what's objectively a lot, but I'm going to claim that's a lot :) )
pankajgoyal (Thu, 11 Oct 2018 18:02:50 GMT):
@Dan for raft_new network, I'm using below params:
sawtooth.consensus.raft.period=3000
sawtooth.consensus.raft.election_tick=20
sawtooth.consensus.raft.heartbeat_tick=2
sawtooth.publisher.max_batches_per_block=500
Dan (Thu, 11 Oct 2018 18:25:31 GMT):
@ltseeley @adamludvik Reporters ( @kirkwood ) of https://jira.hyperledger.org/browse/STL-1461 have a proposed resolution. When you are able, please review and comment.
Dan (Thu, 11 Oct 2018 18:25:31 GMT):
@ltseeley @adamludvik Reporters of https://jira.hyperledger.org/browse/STL-1461 have a proposed resolution. When you are able, please review and comment.
cuevrob (Thu, 11 Oct 2018 21:11:24 GMT):
Has left the channel.
Dan (Fri, 12 Oct 2018 20:00:54 GMT):
@ltseeley @adamludvik I'm looking through ~2hrs of 1.1 logs from a raft leader and a replica (out of a 10 node network). The leader has dropped 2 duplicate blocks and the replica dealt with about 175 duplicate blocks. As the Raft network is required to be fully connected right now, do you have any thoughts on gossiping blocks?
adamludvik (Mon, 15 Oct 2018 16:13:55 GMT):
Forwarding is very aggressive right now (100% chance to forward I think) so there are going to be a lot of duplicates for followers. With a 100% chance and a fully connected network, all of the forwarding is probably unnecessary. I think @jsmitchell and @boydjohnson are experimenting with different gossip rates right now. If the rate is <100% you would still need it turned on. It may even make sense to make the gossip rate configurable for situations like this.
ltseeley (Mon, 15 Oct 2018 18:37:46 GMT):
I'm currently working on implementing consensus engine changes and I’m considering what is the best strategy for notifying a consensus engine when it’s unregistered (i.e. the on-chain setting is updated and no longer corresponds to the registered engine). I'm proposing introducing a general `ConfigChange` notification to the consensus API that will be used to notify the engine about any important configuration changes. For now, it’ll just be used for telling the engine about a change to the algorithm name/version settings (i.e. an unregister), but in the future it could check other settings. This could be useful for updating the engine's configuration based on on-chain settings (like the peers list, block period, and election/heartbeat tick settings used by Raft); the engine could just pass a list of settings to the validator when it registers, then each time a block is committed, the validator can check those settings and notify the engine of any changes. This would make this task much simpler from the engine’s perspective if it can rely on the validator to keep it informed. Any feedback/discussion around this idea would be great.
adamludvik (Mon, 15 Oct 2018 19:29:42 GMT):
I think this is a really interesting proposal. All of the consensus engines we have written so far have depended on some on-chain settings, and have had to choose between either polling state every time a block is committed to see if a setting has changed (more network traffic), or reading the setting once on startup and assuming it doesn't change (less configurable). Allowing engine's to subscribe to push notifications about settings, and possibly other addresses, seems like a good way way to avoid that tradeoff altogether.
This is also a lot like a state delta subscriber, and we may be able to reuse parts of that subsystem to implement this.
@Dan @amundson do you have any thoughts on this?
adamludvik (Mon, 15 Oct 2018 19:29:42 GMT):
I think this is a really interesting proposal. All of the consensus engines we have written so far have depended on some on-chain settings, and have had to choose between either polling state every time a block is committed to see if a setting has changed (more network traffic), or reading the setting once on startup and assuming it doesn't change (less configurable). Allowing engine's to subscribe to push notifications about settings, and possibly other addresses, seems like a good way way to avoid that tradeoff altogether.
This is also a lot like a state delta subscriber, and we may be able to reuse parts of that subsystem to implement this.
@Dan @amundson do you have any thoughts on this?
adamludvik (Mon, 15 Oct 2018 19:31:16 GMT):
@Dan ?
Dan (Mon, 15 Oct 2018 19:54:18 GMT):
state delta for consensus engines sounds intriguing. it's been a while since I hacked on poet.. I need to go back and look at how we handled walking back in time while evaluating forks.
amundson (Tue, 16 Oct 2018 00:55:01 GMT):
@ltseeley @adamludvik sure, sounds like a good idea. maybe SettingsChange if it is settings specific
MohitJuneja (Tue, 16 Oct 2018 03:26:48 GMT):
Has joined the channel.
kelly_ (Tue, 16 Oct 2018 05:47:36 GMT):
Screenshot_20181015-224652.png
kelly_ (Tue, 16 Oct 2018 05:48:13 GMT):
@Dan @amolk how much epc does poet use?
sergefdrv (Tue, 16 Oct 2018 08:40:12 GMT):
@adamludvik I am wondering if the configuration delta could arrive to the consensus engine asynchronously. Could it be that the delta comes after some other thing to be handled considering the configuration has already been changed?
> Allowing engine's to subscribe to push notifications about settings, and possibly other addresses, seems like a good way way to avoid that tradeoff altogether.
sergefdrv (Tue, 16 Oct 2018 08:40:47 GMT):
sorry for my English :laughing:
amolk (Tue, 16 Oct 2018 09:34:49 GMT):
@kelly_
amolk (Tue, 16 Oct 2018 14:35:54 GMT):
@kelly_ about that tweet, in a VM setting the EPC is also virtualized and so appears as 30 MB or whatever's configured to each VM.
Dan (Tue, 16 Oct 2018 14:43:56 GMT):
@adamludvik just curious why `summary` instead of `digest` or `hash`? https://github.com/hyperledger/sawtooth-core/blob/master/protos/consensus.proto#L42
kelly_ (Tue, 16 Oct 2018 15:07:26 GMT):
@amolk so one megabyte?
amolk (Tue, 16 Oct 2018 15:13:35 GMT):
Yup
adamludvik (Tue, 16 Oct 2018 15:34:41 GMT):
@Dan It's a somewhat unique concept, in that it is a digest of some subset of the contents of a block-like-thing before it is actually a block with a signature. So I was looking for a new term that was similar to digest, but also answered the question "digest of what?". In other words, "the block summary" should be unambiguous, whereas "the block digest" would need further clarification about what you are digesting (the entire block serialized? the header serialized? something else?).
adamludvik (Tue, 16 Oct 2018 15:39:44 GMT):
@sergefdrv Like the other updates (block commit, peer message) the proposal is to have the settings updates arrive asynchronously. The update would occur sometime after a new block had been committed, and might need to include the block id of the associated block.
> Could it be that the delta comes after some other thing to be handled considering the configuration has already been changed?
This is a good point. How would an engine know that the settings have or have not changed after a new block is committed? It might make more sense to include the configuration change in the BlockCommit update instead of in a separate update.
Dan (Tue, 16 Oct 2018 15:43:19 GMT):
@adamludvik there is never anything that is unambiguous in variable names. ;)
Is the summary exclusively over the batch list?
adamludvik (Tue, 16 Oct 2018 15:44:00 GMT):
I think so, although we had talked about including the previous block id in there too
adamludvik (Tue, 16 Oct 2018 15:45:26 GMT):
It was unusual enough to make you ask the question though :)))
ltseeley (Tue, 16 Oct 2018 16:30:26 GMT):
@adamludvik @sergefdrv I think including configuration changes with the BlockCommit update is a good idea. That is already what engines use to verify when a block is committed, so I that's probably the most reliable way to notify consistently.
ltseeley (Tue, 16 Oct 2018 16:30:26 GMT):
@adamludvik @sergefdrv I think including configuration changes with the BlockCommit update is a good idea. That is already what engines use to verify when a block is committed, so that's probably the most reliable way to notify consistently.
grapebaba (Wed, 17 Oct 2018 01:43:19 GMT):
Has joined the channel.
kelly_ (Wed, 17 Oct 2018 16:37:07 GMT):
@adamludvik have you guys done a LRN on pbft?
kelly_ (Wed, 17 Oct 2018 16:37:49 GMT):
one of the reasons i put poet2 and pbft as 'under development' is because we haven't put them under an lrn for the 1.1 release and i thought they both had new featuers being added
kelly_ (Wed, 17 Oct 2018 16:38:02 GMT):
we did raft on LRN and we're able to hit 12-16 hours and then encountered bugs
kelly_ (Wed, 17 Oct 2018 16:38:53 GMT):
if pbft works today and can run for 10 or 12 hours under load I think that's fine to put it as alpha
kelly_ (Wed, 17 Oct 2018 16:39:05 GMT):
i was just concerned about pointing developers at something that is fragile
adamludvik (Wed, 17 Oct 2018 16:48:21 GMT):
That is good point. @amundson, @agunde, or @bridgerherman did you do any PBFT LR testing this summer?
agunde (Wed, 17 Oct 2018 16:51:03 GMT):
I know @bridgerherman and @rberg2 were in the process of trying to set up an LR network before Bridger's internship ended but they ran into issues. But I do not know the specifics.
agunde (Wed, 17 Oct 2018 16:51:03 GMT):
I know @bridgerherman and @rberg2 were in the process of trying to set up an LR network before Bridger's internship ended but they ran into issues. But I do not remember the specifics.
rberg2 (Wed, 17 Oct 2018 16:51:03 GMT):
Has joined the channel.
jsmitchell (Wed, 17 Oct 2018 17:12:49 GMT):
@kelly_ retest raft. There was a deadlocking issue that @boydjohnson fixed in master.
jsmitchell (Wed, 17 Oct 2018 17:13:26 GMT):
we are curious to see if the raft LR issue is related
kirkwood (Wed, 17 Oct 2018 17:13:44 GMT):
Was this a recent commit?
jsmitchell (Wed, 17 Oct 2018 17:13:49 GMT):
yeah
jsmitchell (Wed, 17 Oct 2018 17:14:26 GMT):
https://github.com/hyperledger/sawtooth-core/pull/1896
kirkwood (Wed, 17 Oct 2018 17:15:07 GMT):
oh, in core. gotcha
kelly_ (Wed, 17 Oct 2018 17:23:34 GMT):
@jsmitchell awesome - will do and and report back
kelly_ (Wed, 17 Oct 2018 17:25:18 GMT):
@amolk may know if the Raft run included that PR
pankajgoyal (Wed, 17 Oct 2018 18:21:03 GMT):
@kelly_ Raft run didn't include that PR...we were running Raft with sawtooth 1.1 and the PR is raised on sawtooth-core (master)
kelly_ (Wed, 17 Oct 2018 19:07:44 GMT):
thanks @pankajgoyal
kelly_ (Wed, 17 Oct 2018 19:07:53 GMT):
we'll kick off a run on master shortly
ltseeley (Thu, 18 Oct 2018 16:23:27 GMT):
@adamludvik @amundson @jsmitchell @Dan re: sending settings updates to consensus engines, can we have a discussion on the pros/cons of including this information in the BlockCommit messages sent by the validator? Specifically, we are are considering including updated values of requested settings, as well as some kind of unregister flag.
Dan (Thu, 18 Oct 2018 16:32:48 GMT):
and @askmish
Monitoring state/settings was a little messy in poet1. not sure how it's shaping up in poet2.
I can commit to reviewing the poet1 requirements this week and be prepared to discuss in more detail early next week.
adamludvik (Thu, 18 Oct 2018 16:50:43 GMT):
@ltseeley what are some of the pros/cons you have considered so far?
ltseeley (Thu, 18 Oct 2018 18:57:52 GMT):
Pros:
- Easier for engines to track on-chain settings (no polling)
- Better synchronicity than supplying an update before or after the BlockCommit message
Cons:
- Adds complexity to BlockCommit message
- Changes protobufs and SDK
ltseeley (Thu, 18 Oct 2018 18:58:12 GMT):
@adamludvik any that I'm missing from our previous discussions?
adamludvik (Thu, 18 Oct 2018 22:52:17 GMT):
We would also need an API for subscribing to the changes and there would be some work in the validator to monitor for changes and send them to the engine.
warren13245 (Fri, 19 Oct 2018 13:34:34 GMT):
Has joined the channel.
amundson (Fri, 19 Oct 2018 14:39:06 GMT):
do we feel that this is something to address for 1.1? it starts to feel like it to me.
Dan (Fri, 19 Oct 2018 16:09:56 GMT):
I feel like that will kick 1.1 back. We've talked about a 1.2 tentatively for EOY. Assuming we had additions to the interface they could be added at that time.
amundson (Fri, 19 Oct 2018 16:22:36 GMT):
ok, should probably take this to the release channel
Dan (Fri, 19 Oct 2018 16:23:20 GMT):
wait, when did you get that Tech Ambassador tag? Did that just happen?
amundson (Fri, 19 Oct 2018 16:25:45 GMT):
tag?
Dan (Fri, 19 Oct 2018 16:35:57 GMT):
Clipboard - October 19, 2018 11:35 AM
amundson (Fri, 19 Oct 2018 16:36:57 GMT):
neat
amundson (Fri, 19 Oct 2018 16:37:15 GMT):
I think the tags were just added to rocketchat
silasdavis (Fri, 19 Oct 2018 18:51:38 GMT):
Has joined the channel.
yusra (Sun, 21 Oct 2018 06:48:06 GMT):
Has joined the channel.
danintel (Mon, 22 Oct 2018 15:03:55 GMT):
Has joined the channel.
adamludvik (Mon, 22 Oct 2018 17:40:13 GMT):
@ltseeley Seeing no strong objections to pushing configuration changes with block commits, I think you are good to draft an initial RFC for this.
amundson (Mon, 22 Oct 2018 21:21:39 GMT):
@adamludvik @ltseeley is the plan within pbft then to support both the 1.1 consensus API where you have to ask for that information every block, in addition to the new way for 1.2 where it comes with block commit messages? if so, I'm in favor of it.
ltseeley (Mon, 22 Oct 2018 21:23:06 GMT):
I have no objections to that
amundson (Mon, 22 Oct 2018 21:24:54 GMT):
we could maintain a 1.1-specific pbft branch too, but the longer we can get away with just a single branch the better in my opinion
ltseeley (Mon, 22 Oct 2018 21:35:29 GMT):
I don't think a 1.1-specific branch would be necessary, since the differences between the two should be minimal. @adamludvik @knkski will the membership change procedure rely on updating the `peers` setting like Raft does? And is updating other configuration values from on-chain settings something we want to have in PBFT 1.0?
adamludvik (Mon, 22 Oct 2018 21:43:43 GMT):
@amundson I agree that we shouldn't need a separate 1.1 branch. Receiving push notifications would be an optional optimization.
adamludvik (Mon, 22 Oct 2018 21:44:24 GMT):
@ltseeley I don't think we've hashed out the first question about reading the setting for membership changes yet. I think the WIP branch was based on responding to PeerConnect and PeerDisconnect messages, but I think that is probably the wrong approach. I think you need membership changes to happen on block commit boundaries so you don't have nodes committing blocks based on vote count thresholds from different node counts. I also think it is wrong to initiate a membership change when a node faults.
adamludvik (Mon, 22 Oct 2018 21:46:18 GMT):
> And is updating other configuration values from on-chain settings something we want to have in PBFT 1.0?
This seems more like a nice-to-have, low-priority feature in general, and would be done time-permitting.
achenette (Tue, 23 Oct 2018 16:51:55 GMT):
Has joined the channel.
ltseeley (Tue, 23 Oct 2018 20:52:57 GMT):
This PR introduces new requirements for configuring consensus engines: https://github.com/hyperledger/sawtooth-core/pull/1916 It requires that the settings `sawtooth.consensus.algorithm.name` and `sawtooth.consensus.algorithm.version` be set for an engine to register and interact with the validator. @amundson @Dan @achenette can we have a discussion about this change and how/if we should handle backporting this to 1.1? @adamludvik has suggested that we mark the `sawtooth.consensus.algorithm` setting as deprecated and the two new settings as required for the 1.1 release, but do not actually enforce it (i.e. do not backport the PR).
amundson (Wed, 24 Oct 2018 16:45:19 GMT):
@ltseeley "Validator only accepts consensus engine registration requests from an engine with name and version matching the on-chain settings" -- this seems like a very broken approach. what should happen is that any consensus engines can register with the validator, and the settings drive which consensus engine is asked to process messages. for rolling upgrades, you should be able to connect a new consensus engine to all the validators, then make the settings change to switch over. you shouldn't need to coordinate a network-wide sysadmin party to bring the new consensus engine online after changing the setting.
ltseeley (Wed, 24 Oct 2018 16:54:52 GMT):
Maybe we are being a little loose with the term "registered" and should re-evaluate that. What we mean by "registered" in this context is which consensus engine is actively being used. Any consensus engine can be started up and connected to the validator, and will attempt to register with the validator indefinitely. If the engine doesn't match the on-chain settings, the validator will simply ignore the registration, which leaves the engine in an infinite loop until the on-chain settings are changed to match the engine's. Once the settings are changed, the validator will respond to the engine's request, the engine will be asked to process messages, and the validator will accept service requests from it.
ltseeley (Wed, 24 Oct 2018 16:56:06 GMT):
So the consensus engine can be brought online and be ready before it's used; the validator just won't acknowledge it until the settings match up.
amundson (Wed, 24 Oct 2018 16:56:28 GMT):
@ltseeley that is significantly different than the TP design, for exactly the same problem
amundson (Wed, 24 Oct 2018 16:59:02 GMT):
what I'd want as a sysadmin is to start it up, have it register, be able to confirm with a command (sawadm probably) that it's registered, and then report back to the consortium that my validator is all ready to go
amundson (Wed, 24 Oct 2018 17:05:29 GMT):
another disadvantage of this approach is that it puts the polling on the consensus engine, which has no real knowledge. say we set it to retry once every 10 seconds, then you have on average a 5s gap when making the switch. but worse, if you used exponential backoff so you aren't making so many requests, let's say we make the cap a minute. that means on average we would have a 30s gap when the settings change occurs. which depending on the network may or may not be a big deal. but in the reverse, if it's just a routing change because it's already registered, then there is no delay and no polling on either side (just heartbeat perhaps to make sure the connection remains working).
ltseeley (Wed, 24 Oct 2018 17:15:18 GMT):
Makes sense to me. Any engine can register, but only one can be "active" (send/receive messages). Checking registration with a CLI command seems reasonable as well. @adamludvik do you have any opinions on this?
adamludvik (Wed, 24 Oct 2018 17:49:03 GMT):
@amundson it is not "exactly the same problem". While TPs and consensus engines are similar, the major difference is that TPs are passive and only respond to requests from the validator whereas engines are active and send requests to the validator. So while a TP can sit idle indefinitely with little harm, once started a consensus engine is going to react to internal state and events and try to interact with the validator. What we don't want is an engine to connect and think it is the "active" engine, but have everything it tries to do be rejected by the validator so that it gets into some weird state or crashes.
You bring up a valid point about polling being a bad model, which I don't think is hard to resolve. We can define the states as {Connected, Registered, Active} instead of {Connected, Registered} and allow engines to connect and register their name/version with the validator. The validator can then sends "you are now active" and "you are now inactive" messages to the registered engines when the setting changes to let them know when they should start monitoring internal state and events.
Dan (Wed, 24 Oct 2018 18:48:59 GMT):
I think this 3 state approach also satisfies the requirements imposed by block history like we have for TPs. It's not just that we need a smooth launch for a new consensus but also that a fresh validator - or for any reason you need to play history across consensus changes - you'll need all consensus engines running that were invoked during that history.
Dan (Wed, 24 Oct 2018 18:48:59 GMT):
I think this 3 state approach also satisfies the requirements imposed by block history like we have for TPs. It's not just that we need a smooth launch for a new consensus but also that a fresh validator or for any reason you need to play history across consensus changes you'll need all consensus engines running that were invoked during that history.
pravindia (Sun, 28 Oct 2018 18:23:17 GMT):
Has left the channel.
Mahi_3 (Mon, 29 Oct 2018 05:41:07 GMT):
Has joined the channel.
athulramesh (Mon, 29 Oct 2018 13:24:26 GMT):
Has joined the channel.
srinivas (Tue, 30 Oct 2018 17:53:12 GMT):
Has joined the channel.
chainsaw (Wed, 31 Oct 2018 14:28:55 GMT):
Has joined the channel.
shobhaa (Sun, 04 Nov 2018 17:43:48 GMT):
Has joined the channel.
Dan (Mon, 12 Nov 2018 15:01:42 GMT):
@ltseeley can you help me find the commit(s) that deprecates consensus.algorithm to consensus.algorithm.
ltseeley (Mon, 12 Nov 2018 15:22:41 GMT):
The `consensus.algorithm.
ltseeley (Mon, 12 Nov 2018 15:25:22 GMT):
The `consensus.algorithm.
ltseeley (Mon, 12 Nov 2018 15:25:22 GMT):
The `consensus.algorithm.
ltseeley (Mon, 12 Nov 2018 15:26:04 GMT):
Are you seeing problems with poet v. bumper?
Dan (Mon, 12 Nov 2018 15:38:23 GMT):
I think poet checks settings for which algorithm is set.
Dan (Mon, 12 Nov 2018 15:38:54 GMT):
As far as errors see #sawtooth-core-dev
I'm kind of wandering around channels :)
Dan (Mon, 12 Nov 2018 15:42:18 GMT):
Not to be confusing, but I don't think I linked any error related to settings there. I just remember seeing something over the weekend where poet may have errored on getting settings. Of course I don't see that now. Sometimes when those liveness-style tests kill a validator there's a bunch of spurious errors.
ltseeley (Mon, 12 Nov 2018 15:47:26 GMT):
Hmm, that PublisherThread error is strange, I have not seen that before. Doesn't seem related to settings, though.
Dan (Mon, 12 Nov 2018 15:58:58 GMT):
no, separate things. I'm trying to find the settings error I ran into but it was unreleated to the thing I posted over there.
Dan (Mon, 12 Nov 2018 15:59:18 GMT):
and I also can't substantiate my claim yet that poet actually does anything with that algorithm setting :)
Dan (Mon, 12 Nov 2018 16:00:20 GMT):
but anyway, is there a predecessor to https://github.com/hyperledger/sawtooth-core/pull/1916? the bumper release notes say that the setting is deprecated but I couldn't find a commit related to that.
ltseeley (Mon, 12 Nov 2018 16:03:04 GMT):
What do you mean by "predecessor"?
Dan (Mon, 12 Nov 2018 16:04:14 GMT):
I guess a logical predecessor. Does it relate to or depend on an earlier commit that changed or extended the consenus settings?
Dan (Mon, 12 Nov 2018 16:05:09 GMT):
I assume from the release notes that there's already a commit in place that deprecates that setting and then this PR #1916 is moving along that deprecation goal.
Dan (Mon, 12 Nov 2018 16:05:43 GMT):
Unless it's just that the release notes are warning that we see this change coming so fair warning to stop using consensus.algorithm.
ltseeley (Mon, 12 Nov 2018 16:07:48 GMT):
Your last comment is correct. Until the PR you referenced (1916) gets merged, the on-chain consensus settings have never been enforced in any way. We are including the deprecation in the release notes to prepare everyone for when those settings are actually enforced.
Dan (Mon, 12 Nov 2018 19:35:08 GMT):
So @ltseeley if you run `bin/run_tests -m integration` in poet against the bumper aftifacts (that include your most recent change) do you see liveness failures like what I reported?
ltseeley (Mon, 12 Nov 2018 19:41:52 GMT):
I'll test it out
ltseeley (Mon, 12 Nov 2018 20:10:15 GMT):
I am getting that failure as well
ltseeley (Tue, 13 Nov 2018 15:27:00 GMT):
Hey Dan, could you review this PR when you have a sec? We want to tag it for 0.1.1
ltseeley (Tue, 13 Nov 2018 15:27:01 GMT):
https://github.com/hyperledger/sawtooth-raft/pull/43
agunde (Tue, 13 Nov 2018 15:32:20 GMT):
@Dan ^
Dan (Tue, 13 Nov 2018 15:35:55 GMT):
only if @pschwarz gives me a second :thumbsup: here: https://github.com/hyperledger/sawtooth-poet/pull/12 ;)
Dan (Tue, 13 Nov 2018 15:38:36 GMT):
and here: https://github.com/hyperledger/sawtooth-poet/pull/11 :)
Dan (Tue, 13 Nov 2018 15:39:12 GMT):
or @TomBarnes you could also review #11 ^
pschwarz (Tue, 13 Nov 2018 16:07:35 GMT):
Both approved @Dan, but it does look like you have some CI issues.
pschwarz (Tue, 13 Nov 2018 16:09:08 GMT):
They look like Dynamic network test failure, so I restarted the builds to see if they will right themselves
Dan (Tue, 13 Nov 2018 16:21:35 GMT):
thx. that's the existing issue that was patched into sawtooth-core/1-1 but not fixed in master.
Dan (Tue, 13 Nov 2018 16:21:35 GMT):
that's the existing issue that was patched into sawtooth-core/1-1 but not fixed in master.
pschwarz (Tue, 13 Nov 2018 17:11:28 GMT):
Yeah, I had that explained to me
pschwarz (Tue, 13 Nov 2018 17:11:31 GMT):
:)
pschwarz (Tue, 13 Nov 2018 17:11:38 GMT):
Catching back up
Dan (Tue, 13 Nov 2018 17:23:02 GMT):
that does beg the question about how to handle these merges. It seems bad practice to merge when jenkins has a failure. I could put a PR disabling the offending tests and merge these reviewed PRs. I think I prefer that to waiting until sawtooth-core <--> sawtooth-poet integration is resolved.
Another variant of that is I could alter those tests to use the core bumper nightly instead of 1.0 artifacts.
That would address the first of the issues but it seems we still have a liveness failure that seems releated to sawtooth-core.
Dan (Tue, 13 Nov 2018 17:24:54 GMT):
oh and the other condiration is getting these commits ahead of tagging for bumper
rbuysse (Tue, 13 Nov 2018 17:35:08 GMT):
@dan we're starting to produce nightly docker images for this reason. once we're confident about this approach, we'll update the tests to use them so we can do faster turn around for things like this
rbuysse (Tue, 13 Nov 2018 17:35:08 GMT):
@Dan we're starting to produce nightly docker images for this reason. once we're confident about this approach, we'll update the tests to use them so we can do faster turn around for things like this
rbuysse (Tue, 13 Nov 2018 18:46:18 GMT):
hm. disregard what I said. it looks like we use the apt repos instead of dockerhub for this one.
pschwarz (Tue, 13 Nov 2018 21:49:33 GMT):
@Dan I think this should fix it: https://github.com/hyperledger/sawtooth-poet/pull/13
pschwarz (Tue, 13 Nov 2018 21:49:50 GMT):
Will have to wait to see if it builds, of course
Dan (Tue, 13 Nov 2018 21:59:45 GMT):
muchas gracias!
I expect that it will not show a dynamic network failure but will still see a liveness failure. Expected error is something like `validator-0_1 | thread 'PublisherThread' panicked at 'BlockInjector.block_start failed: PyErr { ptype:
pschwarz (Tue, 13 Nov 2018 22:03:45 GMT):
Is that something that is fixed in one of the other open PR's?
benoit.razet (Tue, 13 Nov 2018 22:18:35 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=mRQtyw8sQsqBWe2vB) @Dan Interestingly we have been dealing with this error too today, playing around with the docker-compose*.yaml to avoid volume mounting code to prevent this error
Dan (Tue, 13 Nov 2018 22:20:06 GMT):
@pschwarz no not yet. but i'm honing in on it like the Warren Commission.
pschwarz (Tue, 13 Nov 2018 22:21:13 GMT):
I had the dynamic network test passing locally with the changes in that PR, so we'll see
Dan (Tue, 13 Nov 2018 22:21:51 GMT):
yeah it passes locally for me too. it's the test_poet_liveness that fails.
Dan (Tue, 13 Nov 2018 22:22:20 GMT):
every time I run it, it goes back and to the left. very curious.
pschwarz (Tue, 13 Nov 2018 22:24:37 GMT):
It's more of a crap shoot, than a sniper shoot, @Dan
benoit.razet (Tue, 13 Nov 2018 22:24:37 GMT):
@Dan disregard my message, I'm still getting used to the new error message from the Rust validator, and I thought I had recognized something we encounter, but it's not related
Dan (Tue, 13 Nov 2018 22:36:05 GMT):
test_poet_liveness passes if I remove block_info and the associated block_validation_rules
so my current analysis is that this is a trap set by boyd to trick me into learning the batch injector.
pschwarz (Tue, 13 Nov 2018 23:01:24 GMT):
Ok, dynamic network test failed
pschwarz (Tue, 13 Nov 2018 23:02:34 GMT):
Though, not due to a timeout, like the other builds - maybe a fluke. I'll kick it off one more time, just to be sure
Dan (Wed, 14 Nov 2018 00:42:39 GMT):
@pschwarz it passed the second time. From a real quick look, the first session seemed to just not reach consensus which might be just bad luck on timing. I'm firing off a third to see how reproducible it is.
pschwarz (Wed, 14 Nov 2018 00:52:36 GMT):
Ah, you fired that third one off
pschwarz (Wed, 14 Nov 2018 00:52:43 GMT):
That was the merge window!!
pschwarz (Wed, 14 Nov 2018 00:52:46 GMT):
;)
pschwarz (Wed, 14 Nov 2018 02:57:23 GMT):
Looks like that one passed too, so we could mege it, and then rebase all the remaining PR's that you want
pschwarz (Wed, 14 Nov 2018 03:09:51 GMT):
Merged - rebase away
ra_w (Wed, 14 Nov 2018 08:46:12 GMT):
Has joined the channel.
Dan (Wed, 14 Nov 2018 14:40:52 GMT):
cool. thanks. I'm confused now though... why did this pass? :)
I thought we had only fixed the dynamic network test failure. @ltseeley had separately confirmed seeing a failure with the test_poet_liveness. Maybe it was because we had only partially hacked up the test settings to use local builds of the validator. Anyway, I'm rebuilding poet and will verify locally and rebase those other PRs.
JayeshJawale2 (Mon, 19 Nov 2018 09:11:02 GMT):
Has joined the channel.
MaherBouidani (Wed, 21 Nov 2018 17:34:05 GMT):
Has joined the channel.
MaherBouidani (Wed, 21 Nov 2018 17:35:52 GMT):
Hi Guys, I have a question regarding consensus in general and I wish to find help....
If we have a total nodes N, now the general reliability assumption to achieve consensus is that the faulty nodes F should be LESS thank N/K
MaherBouidani (Wed, 21 Nov 2018 17:36:00 GMT):
where K is {2,3}
MaherBouidani (Wed, 21 Nov 2018 17:36:34 GMT):
which makes sense, the faulty nodes should be less than either half or one third of total nodes
MaherBouidani (Wed, 21 Nov 2018 17:36:47 GMT):
the question is, what do you call K in this case?
MaherBouidani (Wed, 21 Nov 2018 17:36:54 GMT):
What is K?
Dan (Wed, 21 Nov 2018 18:05:20 GMT):
if you are looking for a name, there might be a term in the original PBFT paper.
Dan (Wed, 21 Nov 2018 18:05:56 GMT):
http://pmg.csail.mit.edu/papers/osdi99.pdf
aKesav (Sat, 24 Nov 2018 08:19:40 GMT):
Has joined the channel.
OviiyaDominic (Tue, 27 Nov 2018 09:18:09 GMT):
Has joined the channel.
luckycharms810 (Tue, 27 Nov 2018 17:02:32 GMT):
Hi, I've been looking in to working on a custom consensus algorithm, and I was wondering if there is the ability for state to be modified as part of the block creation process. This state modification would not occur in the constituent transactions, but it would use the contents of the transactions to make the appropriate state modification.
jsmitchell (Tue, 27 Nov 2018 17:04:50 GMT):
@luckycharms810 take a look at how block_info transactions are injected to support sawtooth-seth
jsmitchell (Tue, 27 Nov 2018 17:08:21 GMT):
https://github.com/hyperledger/sawtooth-core/blob/master/families/block_info/sawtooth_block_info_injector/injector.py
jsmitchell (Tue, 27 Nov 2018 17:08:46 GMT):
and https://github.com/hyperledger/sawtooth-core/blob/master/validator/sawtooth_validator/journal/batch_injector.py
jsmitchell (Tue, 27 Nov 2018 17:09:29 GMT):
transactions are the mechanism for state mutation
jsmitchell (Tue, 27 Nov 2018 17:10:03 GMT):
avoid the temptation to mutate state outside a transaction
kdenhartog (Tue, 27 Nov 2018 19:52:21 GMT):
Has left the channel.
luckycharms810 (Tue, 27 Nov 2018 20:23:57 GMT):
Can you access other transactions in the block injector ?
boydjohnson (Tue, 27 Nov 2018 20:35:24 GMT):
@luckycharms810 Depending on how much delay is able to happen, you can just broadcast out a transaction from the consensus engine. https://sawtooth.hyperledger.org/docs/core/nightly/master/rust_sdk/doc/sawtooth_sdk/consensus/service/trait.Service.html#tymethod.broadcast
jsmitchell (Tue, 27 Nov 2018 20:37:01 GMT):
the interface doesn't expose the details of the candidate block to the injector, but it does provide a state view. I believe if you inject at the end of the block, and a transaction mutated state in the block, you could inspect the ephemeral state view during your injected txn construction.
jsmitchell (Tue, 27 Nov 2018 20:37:18 GMT):
that would be worth testing, since my confidence in that answer is low ^
jsmitchell (Tue, 27 Nov 2018 20:37:58 GMT):
you _may_ only be able to see state as of the parent block
luckycharms810 (Tue, 27 Nov 2018 20:39:09 GMT):
Ah interesting is the consensus engine all implemented in rust after 1.1 ?
boydjohnson (Tue, 27 Nov 2018 20:43:11 GMT):
The python sdk has support for writing consensus engines, too.
jsmitchell (Tue, 27 Nov 2018 20:48:50 GMT):
I know we had talked at some point about exposing the transactions in the candidate block to the injector
jsmitchell (Tue, 27 Nov 2018 20:49:10 GMT):
I don't know if they have been executed at that point or not (for injecting at the end)
jsmitchell (Tue, 27 Nov 2018 20:49:16 GMT):
@boydjohnson @adamludvik ^
Gabe (Wed, 28 Nov 2018 00:56:05 GMT):
https://www.hyperledger.org/blog/2018/11/09/hyperledger-sawtooth-blockchain-security-part-one
Gabe (Wed, 28 Nov 2018 00:56:58 GMT):
Sawtooth’s Pluggable Consensus
Sawtooth supports “pluggable” consensus algorithms—that is it is configurable. Not only is the consensus configurable at blockchain genesis (block 0), but the consensus algorithm can be changed at any point after after blockchain creation with an in-chain setting, sawtooth.consensus.algorithm.
Another Nakamoto-style consensus algorithm, still in the experimental stage, is Proof of Stake (PoS), where the winner has the most of something (
Gabe (Wed, 28 Nov 2018 00:56:58 GMT):
Proof of Stake - are there any PoS options for Sawtooth?
Sawtooth’s Pluggable Consensus
Sawtooth supports “pluggable” consensus algorithms—that is it is configurable. Not only is the consensus configurable at blockchain genesis (block 0), but the consensus algorithm can be changed at any point after after blockchain creation with an in-chain setting, sawtooth.consensus.algorithm.
Another Nakamoto-style consensus algorithm, still in the experimental stage, is Proof of Stake (PoS)
Other consensus algorithms are in the works, such as PBFT. The Sawtooth community encourages contributions of other consensus algorithms by third parties.
Gabe (Wed, 28 Nov 2018 00:56:58 GMT):
Proof of Stake - are there any PoS options for Sawtooth?
Sawtooth’s Pluggable Consensus
Sawtooth supports “pluggable” consensus algorithms—that is it is configurable. Not only is the consensus configurable at blockchain genesis (block 0), but the consensus algorithm can be changed at any point after after blockchain creation with an in-chain setting, sawtooth.consensus.algorithm.
Other consensus algorithms are in the works, such as PBFT. The Sawtooth community encourages contributions of other consensus algorithms by third parties.
Another Nakamoto-style consensus algorithm, still in the experimental stage, is Proof of Stake (PoS)
What does in the experimental stage mean?
Gabe (Wed, 28 Nov 2018 00:56:58 GMT):
Proof of Stake - are there any PoS options for Sawtooth?
Sawtooth’s Pluggable Consensus
Sawtooth supports “pluggable” consensus algorithms—that is it is configurable. Not only is the consensus configurable at blockchain genesis (block 0), but the consensus algorithm can be changed at any point after after blockchain creation with an in-chain setting, sawtooth.consensus.algorithm.
Other consensus algorithms are in the works, such as PBFT. The Sawtooth community encourages contributions of other consensus algorithms by third parties.
Another Nakamoto-style consensus algorithm, still in the experimental stage, is Proof of Stake (PoS)
What does in the experimental stage mean?
Is PoS for Sawtooth on any roadmaps?
Humphries (Wed, 28 Nov 2018 03:48:43 GMT):
Has joined the channel.
amundson (Wed, 28 Nov 2018 15:35:29 GMT):
@Gabe current consensus mechanisms: PoET/CFT (Also called PoET Simulator, PoET 1, etc.); PBFT, Raft, and DevMode
amundson (Wed, 28 Nov 2018 15:36:00 GMT):
Soon PoET 2 (SGX and non-SGX) will be available as well.
amundson (Wed, 28 Nov 2018 15:37:42 GMT):
PoET/CFT is the default currently; PBFT will have its first stable release probably very early January; Raft is a prototype; DevMode is stable but only for development
adamludvik (Wed, 28 Nov 2018 16:30:32 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=BhcG3mHG3moWQAr9o) @jsmitchell @jsmitchell This never made it past the design phase, because of time constraints and concerns about how it would affect performance.
kc1116 (Fri, 30 Nov 2018 15:49:55 GMT):
@amundson I am having a lot of trouble running the consensus and validator images within a kubernetes cluster. Is there any specific hardware I need. There is some issue with the code code in there after I submit a transaction I get a Failed to panic error unwrapping and validator restarts
digysol (Fri, 30 Nov 2018 18:04:20 GMT):
Has joined the channel.
Dan (Fri, 30 Nov 2018 19:32:33 GMT):
@kc1116 , no - no special hardware required.
Are you aware of https://sawtooth.hyperledger.org/docs/core/nightly/master/app_developers_guide/kubernetes.html
arsulegai (Tue, 04 Dec 2018 14:47:15 GMT):
Need guidance, any idea why would validator keep waiting/in hung state when summarize_block is called via consensus SDK?
arsulegai (Tue, 04 Dec 2018 16:09:34 GMT):
@amundson ^
Gabe (Tue, 04 Dec 2018 20:40:17 GMT):
@amundson thanks for the update on current consensus mechanisms (familiar with all of those). We are happy with PoET for permissioned private and consortium networks.
But we wish to develop and deploy a public network as well using ERC-20 or ERC-1404 compliant tokens for payments and to implement a PoS style consensus mechanism for public incentives (as per many of the other public chain business models).
From what we can see, the pluggable consensus architecture enables us to develop and deploy a PoS algorithm for a public chain.
My questions are:
1. Are there any experimental stage PoS algorithms available?
2. Are there any plans on the road map for developing a PoS algorithm?
If this is not right forum for this discussion? please let me know and we can engage directly with Intel.
Gabriel
jsmitchell (Tue, 04 Dec 2018 20:56:18 GMT):
I am not aware of any PoS consensuses (consensi?) available for sawtooth or in development. There aren't any on the roadmap.
Dan (Tue, 04 Dec 2018 21:43:21 GMT):
It would be very cool to see some though.
Dan (Tue, 04 Dec 2018 22:13:18 GMT):
@arsulegai I think that message gets handled by the proxy (see diagram here: https://github.com/hyperledger/sawtooth-rfcs/blob/master/text/0000-consensus-api.md#consensus-engine-architecture-integration
Do you see a log message like this?
```LOGGER.exception("ConsensusSummarizeBlock")```
If it's just hanging though it might be waiting on the response.
I believe the proxy routes that summarize request to the block publisher and thence to a candidate block.
The most relevant code is probably in the candidate_block.rs:
https://github.com/hyperledger/sawtooth-core/blob/master/validator/src/journal/candidate_block.rs#L341
I see the potential for GIL contention and lots of other stuff going on in there. You might try adding more logging statements and see if it's stuck in there.
Dan (Tue, 04 Dec 2018 22:13:18 GMT):
@arsulegai I think that message gets handled by the proxy (see diagram here: https://github.com/hyperledger/sawtooth-rfcs/blob/master/text/0000-consensus-api.md#consensus-engine-architecture-integration
Do you see a log message like this:
```LOGGER.exception("ConsensusSummarizeBlock")```
If it's just hanging though it might be waiting on the response.
I believe the proxy routes that summarize request to the block publisher and thence to a candidate block.
The most relevant code is probably in the candidate_block.rs:
https://github.com/hyperledger/sawtooth-core/blob/master/validator/src/journal/candidate_block.rs#L341
I see the potential for GIL contention and lots of other stuff going on in there. You might try adding more logging statements and see if it's stuck in there.
Gabe (Tue, 04 Dec 2018 22:33:12 GMT):
@Dan thanks - we're looking at moving a PoS development along in the absence of anything on the road map
arsulegai (Wed, 05 Dec 2018 00:47:13 GMT):
@Dan thanks, issue turned out to be because of transaction family version mismatch in client's request.
I see a potential for handling it better. Validator has transactions, but relevant version transaction family is not available when consensus engine sends summarize_block. Validator could rather say nothing to do instead of waiting indefinitely at this point. Any thoughts?
Dan (Wed, 05 Dec 2018 02:33:44 GMT):
There might be log messages about missing the TP?
arsulegai (Wed, 05 Dec 2018 03:08:22 GMT):
No, validator doesn't print anything at this stage of summarizing block. In history of logs from beginning, I see a trace about TP registering with other version.
arsulegai (Wed, 05 Dec 2018 03:08:22 GMT):
No, verbosity of logs with 3 v's, validator doesn't print anything at this stage of summarizing block. However in history of logs from beginning, I see a trace about TP registering with other version.
Brenwen (Wed, 05 Dec 2018 09:04:48 GMT):
Has joined the channel.
Brenwen (Wed, 05 Dec 2018 09:06:15 GMT):
Is there any instruction how to install and run raft engine? I read the documentation (https://sawtooth.hyperledger.org/docs/raft/nightly/master/configuring_deploying.html#starting-a-raft-engine) but it assymes that engine is installed and running already.
Brenwen (Wed, 05 Dec 2018 09:06:15 GMT):
Is there any instruction how to install and run raft engine? I read the documentation (https://sawtooth.hyperledger.org/docs/raft/nightly/master/configuring_deploying.html#starting-a-raft-engine) but it assumes that engine is installed and running already.
arsulegai (Wed, 05 Dec 2018 11:31:26 GMT):
This might help
Brenwen (Wed, 05 Dec 2018 11:36:06 GMT):
do I need to build it from source on github?
arsulegai (Wed, 05 Dec 2018 16:10:31 GMT):
@Brenwen https://github.com/hyperledger/sawtooth-raft/blob/master/adhoc/node.yaml
This docker compose file might be helpful
Brenwen (Fri, 07 Dec 2018 07:10:52 GMT):
I built the raft-engine, now i'm trying to connect 2 nodes into network but have some troubles. When I launch second validator and run raft engine on it - the raft engine of first node crashes with Encoding Error: UnexpectedWireType(WireTypeVariant). When I restart it - the second raft engine crashes with the same error.
nhrishi (Mon, 10 Dec 2018 17:23:28 GMT):
Has joined the channel.
ltseeley (Mon, 10 Dec 2018 20:37:08 GMT):
@Brenwen could you explain in a bit more detail how you're starting up the network? Are you using the provided docker compose files or something else?
Brenwen (Tue, 11 Dec 2018 06:55:36 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=tmQ4gddzSnqu8TTGo) @ltseeley I'm starting it in ubuntu on virtualbox.
Brenwen (Tue, 11 Dec 2018 06:55:36 GMT):
@ltseeley I'm starting it in ubuntu on virtualbox.
Brenwen (Tue, 11 Dec 2018 06:55:36 GMT):
@ltseeley I'm starting it from ubuntu on virtualbox.
DatNguyen (Tue, 11 Dec 2018 07:22:19 GMT):
Has joined the channel.
ltseeley (Tue, 11 Dec 2018 16:48:15 GMT):
Are you following a process outlined somewhere in the documentation? Could you share the commands you are using to start the validators, Raft engines, etc.?
arsulegai (Tue, 11 Dec 2018 18:48:28 GMT):
@Brenwen are you trying to dynamically setup network?
Brenwen (Thu, 13 Dec 2018 07:49:15 GMT):
I just updated my dev sawtooth network on real machines from 1.0 to 1.1 version and now I have some stability and performance issues. I use poet-simulator consensus. After upgrading sawtooth components it didn't work, so I added "consensus:tcp://127.0.0.1:5050" to validator.toml bind, installed python3-sawtooth-poet-engine package and run *poet-engine -vvv* in sepparate terminal window. At last it began to work, but much slower than 1.0 version. I have only 2 nodes active now and if I send several requests to validator at once it can freeze for several minutes, while 1.0 version could process more than 50 async post requests at the same time in 15 seconds. Did I do everything correct when upgrading and is there any way to improve the system peformance?
Brenwen (Thu, 13 Dec 2018 07:49:15 GMT):
I just updated my dev sawtooth network on real machines from 1.0 to 1.1 version and now I have some stability and performance issues. I use poet-simulator consensus, i raised network as it is explained in docs (https://sawtooth.hyperledger.org/docs/core/releases/1.1/app_developers_guide/creating_sawtooth_network.html#ubuntu-add-a-node-to-the-single-node-environment) and everything was working well. After upgrading sawtooth components it didn't work, so I added "consensus:tcp://127.0.0.1:5050" to validator.toml bind, installed python3-sawtooth-poet-engine package and run *poet-engine -vvv* in sepparate terminal window. At last it began to work, but much slower than 1.0 version. I have only 2 nodes active now and if I send several requests to validator at once it can freeze for several minutes, while 1.0 version could process more than 50 async post requests at the same time in 15 seconds. Did I do everything correct when upgrading and is there any way to improve the system peformance?
Brenwen (Thu, 13 Dec 2018 07:49:15 GMT):
I just updated my dev sawtooth network on real machines from 1.0 to 1.1 version and now I have some stability and performance issues. I use poet-simulator consensus and raised network as it is explained in docs (https://sawtooth.hyperledger.org/docs/core/releases/1.1/app_developers_guide/creating_sawtooth_network.html#ubuntu-add-a-node-to-the-single-node-environment) and everything was working well. After upgrading sawtooth components it didn't work, so I added "consensus:tcp://127.0.0.1:5050" to validator.toml bind, installed python3-sawtooth-poet-engine package and run *poet-engine -vvv* in sepparate terminal window. At last it began to work, but much slower than 1.0 version. I have only 2 nodes active now and if I send several requests to validator at once it can freeze for several minutes, while 1.0 version could process more than 50 async post requests at the same time in 15 seconds. Did I do everything correct when upgrading and is there any way to improve the system peformance?
Brenwen (Thu, 13 Dec 2018 07:49:15 GMT):
I just updated my dev sawtooth network on real machines from 1.0 to 1.1 version and now I have some stability and performance issues. I use poet-simulator consensus and raised network as it is explained in docs (https://sawtooth.hyperledger.org/docs/core/releases/1.1/app_developers_guide/creating_sawtooth_network.html#ubuntu-add-a-node-to-the-single-node-environment) and everything was working well. After upgrading sawtooth components it didn't work, so I added "consensus:tcp://127.0.0.1:5050" to validator.toml bind, installed python3-sawtooth-poet-engine package and run *poet-engine -vvv* in sepparate terminal window. At last it began to work, but much slower than 1.0 version. I have only 2 nodes active now and if I send several requests to validator at once it can freeze for several minutes, while 1.0 version validator could process more than 50 async post requests at the same time in 15 seconds. Did I do everything correct when upgrading and is there any way to improve the system peformance?
Brenwen (Thu, 13 Dec 2018 07:49:28 GMT):
updated network to latest dev version, poet-engine remains 1.1, having same issues with sending multiple post requests. Is there any solution to help solving this problem?
Brenwen (Thu, 13 Dec 2018 08:20:27 GMT):
it freezes after log messages of such type:
[2018-12-13 11:17:01.209 INFO engine] Received Block(block_num: 36, block_id: f3548a7d085f231a89b0feeb066eb154ef0a215af61d6478e75a01b58249adfe171913c3741d898746dac5764a8a213202f76ec0fdf47595e7a71d1a2080d6bb, previous_id: c88e2099a2578106f73f34f10adbf90c0bd262176722115063a9b8db778cd4bf4257ab107d65d54c482a32efd44a7f7cb6104e41e3925db1537cb93536a553d0, signer_id: 037b9204d04b1d93e1d2b29457e581d7676f9d41712b94edaed726746954d9d229, payload: b'{"SerializedCertificate": "{\\"block_hash\\": \\"b\'\\\\\\\\x96\\\\\\\\xa4~\\\\\\\\x89!=\\\\\\\\x04\\\\\\\\x04;\\\\\\\\xd9\\\\\\\\xd0\\\\\\\\x15XV\\\\\\\\xc0l$f\\\\\\\\xd2\\\\\\\\xb0q\\\\\\\\xa4?\\\\\\\\xb9F\\\\\\\\x8ffe\\\\\\\\xc0\\\\\\\\t\\\\\\\\x1b\\\\\\\\x18\'\\", \\"duration\\": 1.419965844994794, \\"local_mean\\": 14.799999999999997, \\"nonce\\": \\"d3f2c8ef1d1d9d70c0238b7f42070a80f2e8145a82759adf3b02326e69bbb0b6\\", \\"previous_certificate_id\\": \\"GVRGKZTCHEZWENJX\\", \\"request_time\\": 1544689019.2977455, \\"validator_address\\": \\"037b9204d04b1d93e1d2b29457e581d7676f9d41712b94edaed726746954d9d229\\"}", "Signature": "634a1f37ad446961e58ac3a292e1cceba0c45f114b71811ca3aafd4de7c191c06d39cec2e1d35a902be7ba0f8fc9ad67e1639e6e896d924f40fb3601f2b44d32"}', summary: 96a47e89213d04043bd9d0155856c06c2466d2b071a43fb9468f6665c0091b18)
[2018-12-13 11:17:01.217 DEBUG poet_block_verifier] Block Signer Name=validator-037b9204, ID=037b9204...54d9d229, PoET public key=03aac0fe...44c54caf
arsulegai (Sat, 15 Dec 2018 07:46:41 GMT):
@amundson @jsmitchell Looks like validator published images on docker-hub or the ones published as debian package, doesn't still support changed consensus SDK behavior. Consensus engine registers with validator but validator doesn't send activated message to consensus SDK. Could you please confirm it?
arsulegai (Mon, 17 Dec 2018 16:54:56 GMT):
^ @rberg2
ltseeley (Mon, 17 Dec 2018 18:30:30 GMT):
@arsulegai currently the SDK only works with the nightly validator. I'm updating the SDK right now to support both the 1.1 and nightly versions of the validator.
arsulegai (Mon, 17 Dec 2018 18:50:41 GMT):
Thanks
ltseeley (Mon, 17 Dec 2018 21:03:28 GMT):
Here's the PR for this SDK update: https://github.com/hyperledger/sawtooth-core/pull/1976
Dan (Mon, 17 Dec 2018 23:23:12 GMT):
is that really <=1.1 or just handling for 1.1 specifically? Like the consensus SDK probably doesn't apply to any release but 1.1 and greater. Not being pedantic just trying to understand if I'm forgetting something.
Eddiwar (Mon, 17 Dec 2018 23:55:57 GMT):
Has joined the channel.
arsulegai (Tue, 18 Dec 2018 04:32:08 GMT):
@ltseeley another question unanswered is that we always picked validator from "deb [arch=amd64] http://repo.sawtooth.me/ubuntu/nightly xenial universe" repository and it was not sending activation message. Confused here, so was new nightly validator image published only for bionic?
mfford (Tue, 18 Dec 2018 15:54:29 GMT):
The Hyperledger Sawtooth Team will be hosting a Technical Forum on Thursday, December 20th from 10-11am CDT. The technical forums feature presentations and discussions for the Hyperledger Sawtooth platform. We invite Hyperledger community members interested in Hyperledger Sawtooth to join our forum!
Agenda
Introductions-Mark Ford
Hyperledger Sawtooth Consensus API-Logan Seeley
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/my/hyperledger.community.backup
Or iPhone one-tap :
US: +16465588656,,6223336701# or +16699006833,,6223336701#
Or Telephone:
Dial(for higher quality, dial a number based on your current location):
US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
Meeting ID: 622 333 6701
International numbers available: https://zoom.us/u/bAaJoyznp
ltseeley (Tue, 18 Dec 2018 18:50:10 GMT):
@Dan that's a good catch, you're right. I've updated the comments and commit message to reflect that.
ltseeley (Tue, 18 Dec 2018 18:51:17 GMT):
@rbuysse are the nightly validator images only being published for bionic, or are they published for xenial as well? Re: @arsulegai's question
rbuysse (Tue, 18 Dec 2018 18:53:29 GMT):
we are publishing nightly bumper images for everything and nightly master images for everything except sawtooth-shell
rbuysse (Tue, 18 Dec 2018 18:53:47 GMT):
we'll have shell soon once the poet repo is upgraded to bionic
ltseeley (Tue, 18 Dec 2018 18:57:36 GMT):
@arsulegai so it looks like you are using the nightly master image for the validator. What consensus engine are you using?
arsulegai (Thu, 20 Dec 2018 17:21:09 GMT):
It's PoET2 consensus engine which is under review. Dockerfile is written to consider latest nightly debian package from "deb [arch=amd64] http://repo.sawtooth.me/ubuntu/nightly xenial universe"
rbuysse (Thu, 20 Dec 2018 18:10:32 GMT):
before that PR is merged any dockerfiles should be updated to use ubuntu:bionic and the bionic distro for the sawtooth repo
Dan (Thu, 20 Dec 2018 18:18:30 GMT):
As far as the first PR for the poet2 branch, I'm inclined not to muddy it. let's get that first PR through and then back into a normal PR cadence with digestible commits.
Dan (Thu, 20 Dec 2018 18:18:30 GMT):
As far as the first PR for the poet2 branch, I'm inclined not to muddy it. let's get that first PR through and then back into a normal PR cadence with digestible ~commits~ PRs.
Dan (Thu, 20 Dec 2018 19:31:47 GMT):
If this bionic thing is an issue a different option is to peg the poet2 branch to sawtooth 1.1 / xenial. That would also protect the poet2 testing from moving targets in validator master.
amundson (Thu, 20 Dec 2018 20:39:45 GMT):
its fine with me if the poet2 branch only works with sawtooth 1.1 initially
amundson (Thu, 20 Dec 2018 20:40:23 GMT):
however, it is probably pretty important to get it working w/master sooner rather than later
PhuongDo (Fri, 21 Dec 2018 03:05:48 GMT):
Has joined the channel.
sergefdrv (Fri, 21 Dec 2018 11:58:34 GMT):
@ltseeley Yestarday, during the tech forum, you seemed to mention that sawtooth implementation of PBFT changes the primary/leader node every so often. This is something not described in PBFT paper. I wonder what is the scheme of leader rotation you use?
ltseeley (Fri, 21 Dec 2018 15:49:42 GMT):
Hi, @sergefdrv! There is an RFC that describes our solution here: https://github.com/hyperledger/sawtooth-rfcs/pull/29
sergefdrv (Fri, 21 Dec 2018 16:01:36 GMT):
thanks, will have a look
ltseeley (Fri, 21 Dec 2018 16:05:14 GMT):
If you have any questions or feedback, feel free to let us know
merq (Sat, 22 Dec 2018 03:40:02 GMT):
Has joined the channel.
hidura (Thu, 27 Dec 2018 04:59:57 GMT):
Has joined the channel.
tuckerg (Thu, 27 Dec 2018 06:54:58 GMT):
Has joined the channel.
Dan (Fri, 28 Dec 2018 17:20:44 GMT):
Need another maintainer check here: https://github.com/hyperledger/sawtooth-poet/pull/15 for bionic
sergefdrv (Fri, 28 Dec 2018 20:22:48 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=w6KrmgHfmkZpGbCr8) @ltseeley @ltseeley I've looked through the RFC and wondering how this would impact performance if some of the PBFT nodes appear faulty. In that scenario, regularly changing the leader should mean regularly selecting every faulty node as the leader, as well. If so, this will regularly stop the whole system for the idle timeout. In contract, the origianl PBFT protocol would not do that, but rather stick to the leader as long as it appears to behave correctly. But the original PBFT approach would require the consensus engine to know about individual batches...
ltseeley (Wed, 02 Jan 2019 15:18:17 GMT):
@sergefdrv I think your understanding is correct; this approach definitely has some limitations. In the future this is a problem that could probably be solved in a different way, but for the time being, we think this is sufficient.
MicBowman (Sat, 05 Jan 2019 00:23:36 GMT):
@Dan we've made some fairly significant improvements in how we use SGX with attestation... if anyone working on PoET or PoET2 is interested, we should probably talk about the protocol for verifying an attestation
Dan (Mon, 07 Jan 2019 21:23:29 GMT):
Sounds cool. I think that would be @amolk and @arsulegai
amolk (Tue, 08 Jan 2019 00:12:11 GMT):
I'll get in touch with you, Mic.
arsulegai (Tue, 08 Jan 2019 08:44:51 GMT):
^ @manojgop let's sync with Amol, this might help in other case we were discussing
manojgop (Tue, 08 Jan 2019 08:44:52 GMT):
Has joined the channel.
dokany (Tue, 08 Jan 2019 13:05:16 GMT):
Has joined the channel.
satish67 (Wed, 09 Jan 2019 09:23:22 GMT):
Has joined the channel.
satish67 (Wed, 09 Jan 2019 09:23:29 GMT):
Hello, everyone, i have a general question regarding sawtooth architecture, as sawtooth can execute the parallel transaction processor, does it mean for each transaction processor ledger will be different like channel in fabric, thanks in advance
arsulegai (Wed, 09 Jan 2019 14:06:40 GMT):
@satish67 there's single ledger in sawtooth. Whether to execute transactions in parallel or not is decided by input/output addresses field in transaction header.
muniyaraj (Fri, 11 Jan 2019 12:26:57 GMT):
Has joined the channel.
muniyaraj (Fri, 11 Jan 2019 12:27:43 GMT):
anyone can explain about payload?
muniyaraj (Fri, 11 Jan 2019 12:28:23 GMT):
and why we need input and output address
muniyaraj (Fri, 11 Jan 2019 12:28:26 GMT):
?
arsulegai (Fri, 11 Jan 2019 15:10:51 GMT):
Input addresses tell transaction professor the addresses it can read in Merle tree, output addresses are the ones it can write to.
arsulegai (Fri, 11 Jan 2019 15:12:53 GMT):
It's also used to decide if transactions can be executed in parallel.
Payload is information you want your client to send to TP, do you've specific questions?
dokany (Fri, 11 Jan 2019 21:25:30 GMT):
Hi everyone! Is there a good reference explaining why PoET with SGX is byzantine fault tolerant, aside from the sawtooth docs?
nanspro (Sat, 12 Jan 2019 10:08:23 GMT):
Has joined the channel.
arsulegai (Mon, 14 Jan 2019 13:55:16 GMT):
Hello,
I've a question regarding ZMQ service and connection implementation for SDK. It's seen that send_msg() in rpc is made blocking call and doesn't return until validator tells it has read the message. Is my understanding correct?
arsulegai (Mon, 14 Jan 2019 13:55:44 GMT):
@amundson @jsmitchell ^
jsmitchell (Mon, 14 Jan 2019 14:40:10 GMT):
Which language?
arsulegai (Mon, 14 Jan 2019 15:03:16 GMT):
Rust
jsmitchell (Mon, 14 Jan 2019 15:55:33 GMT):
Have you tried adding a no_wait to see what the effect is?
arsulegai (Mon, 14 Jan 2019 15:58:44 GMT):
Changing from 0 to 1?
jsmitchell (Mon, 14 Jan 2019 16:00:20 GMT):
you are referring to this line? https://github.com/hyperledger/sawtooth-core/blob/master/sdk/rust/src/messaging/zmq_stream.rs#L123
jsmitchell (Mon, 14 Jan 2019 16:01:29 GMT):
https://github.com/hyperledger/sawtooth-core/blob/master/sdk/rust/src/messaging/zmq_stream.rs#L291
jsmitchell (Mon, 14 Jan 2019 16:08:49 GMT):
https://docs.rs/zmq/0.8.2/zmq/static.DONTWAIT.html
arsulegai (Mon, 14 Jan 2019 16:23:46 GMT):
Yes, you're right.. This is the one I was referring to
arsulegai (Mon, 14 Jan 2019 16:24:13 GMT):
Changed it to 1, to make it DONTWAIT, other terms from 0 to 1
arsulegai (Mon, 14 Jan 2019 16:32:15 GMT):
I could see behavior changed because of this modification, however later at line https://github.com/hyperledger/sawtooth-core/blob/6c12eb5faca3c14d8ce4838aef19cbaafd1b630a/sdk/rust/src/consensus/zmq_service.rs#L64 there's one more block when it comes to ZmqService
jsmitchell (Mon, 14 Jan 2019 16:32:39 GMT):
what do you mean "you could see behavior changed"? What is the effect?
arsulegai (Mon, 14 Jan 2019 16:34:08 GMT):
Ok, I'll explain the observation.. But to complete previous statement the https://github.com/hyperledger/sawtooth-core/blob/6c12eb5faca3c14d8ce4838aef19cbaafd1b630a/sdk/rust/src/consensus/zmq_driver.rs#L40 the service timeout of 300 seconds is applied.. Thus blocking it
arsulegai (Mon, 14 Jan 2019 16:41:23 GMT):
Changed behavior is that in earlier case when the value was *not* DONTWAIT, trace at https://github.com/hyperledger/sawtooth-core/blob/6c12eb5faca3c14d8ce4838aef19cbaafd1b630a/sdk/rust/src/messaging/zmq_stream.rs#L290 printed in log and it took long time to print (forgot actual number, it was long enough to fail Raft consensus) https://github.com/hyperledger/sawtooth-core/blob/6c12eb5faca3c14d8ce4838aef19cbaafd1b630a/sdk/rust/src/messaging/zmq_stream.rs#L262, moreover there were no other activities in between
arsulegai (Mon, 14 Jan 2019 16:43:13 GMT):
However with latter configuration, observed that Raft consensus engine ran a little longer. Could see it at least received VoteRequest from others when waiting for service timeout.
dokany (Tue, 15 Jan 2019 12:59:06 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=npoMGjtW5XGBr5Bf3) Anyone?
Dan (Tue, 15 Jan 2019 20:10:04 GMT):
This is the best reference. https://sawtooth.hyperledger.org/docs/core/releases/latest/architecture/poet.html
TLDR the waiting time is enforced in an enclave. There's more defense in depth than that but that's the gist.
What PoW does with repeatedly hashing you get with a trusted timer in PoET.
hartm (Thu, 17 Jan 2019 19:57:47 GMT):
@dokany There isn't a formal proof, to my knowledge (it's something that I'm attempting to write, though).
adeelm (Fri, 18 Jan 2019 16:40:15 GMT):
Has joined the channel.
adeelm (Fri, 18 Jan 2019 16:41:17 GMT):
Hi, I am trying to create a sawtooth network on 5 nodes, I understand from the documentation that all nodes should have sawtooth installed with PoET consensus, I am not able to find how to install PoET, can someone help me?
manojgop (Sun, 20 Jan 2019 07:11:03 GMT):
Do consensus engine implementation need wait for block commit message (Update::BlockCommit) from Validator before calling initialize_block(), or can it start building the next block immediately after calling commit_block() by specifying "previous_id" in initialize_block() same as block_id in commit_block().
manojgop (Sun, 20 Jan 2019 07:11:03 GMT):
Should consensus engine implementation need to wait for block commit message (Update::BlockCommit) from Validator before calling initialize_block(), or can it start building the next block immediately after calling commit_block() by specifying "previous_id" in initialize_block() same as block_id in commit_block().
jsmitchell (Sun, 20 Jan 2019 16:27:38 GMT):
@manojgop I think it needs to wait so the validator has access to the applied base state from the prior block. This could probably be done on top of ephemeral state, but would require changes to the validator.
danintel (Mon, 21 Jan 2019 03:27:49 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=axnkBg34megjLBDNC) @adeelm A good channel for imstallation and configuration questions is #sawtooth. For your question see https://sawtooth.hyperledger.org/docs/core/nightly/master/app_developers_guide/creating_sawtooth_network.html
arsulegai (Mon, 21 Jan 2019 09:13:09 GMT):
Question: Consider the case as follows (from Raft consensus)
1) Client sends request to a non-leader validator node
2) Client's request is invalid
3) Follower node here broadcasted invalid transaction to all other nodes as well
4) Leader node's consensus engine asked leader validator to start forming the block - Leader node got chance to execute the transactions, it saw invalid transaction and rejected it
5) But follower node doesn't get any intimation from other to reject the block (this is by nature of blockchain)
arsulegai (Mon, 21 Jan 2019 09:13:40 GMT):
Problem is follower never became leader, follower's queue is full with all invalid transactions.. They never get removed from it
arsulegai (Mon, 21 Jan 2019 09:14:00 GMT):
Is my understanding correct or something wrong here?
ltseeley (Mon, 21 Jan 2019 15:33:24 GMT):
@manojgop @jsmitchell I'm actually not sure that's true. I think you can initialize the new block on top of a specified previous block as long as the validator has the previous block. That being said, I belive we've only operated under the assumption that it's best to wait for confirmation that the block was committed before starting the next one, so I wouldn't want to make any guarantees.
jsmitchell (Mon, 21 Jan 2019 15:40:17 GMT):
Try it and see :)
arsulegai (Mon, 21 Jan 2019 17:32:27 GMT):
@jsmitchell @ltseeley help me to understand transaction scheduling module ^
jsmitchell (Mon, 21 Jan 2019 17:33:22 GMT):
invalid transactions are currently only removed from the pending queue when the validator successfully publishes a block
jsmitchell (Mon, 21 Jan 2019 17:36:12 GMT):
clearly, this wouldn't be expected to happen with any regularity for consensuses where the leader role either doesn't change or changes slowly
jsmitchell (Mon, 21 Jan 2019 17:37:35 GMT):
pending queue maintenance in such scenarios needs to be re-designed, imo
arsulegai (Mon, 21 Jan 2019 17:38:14 GMT):
Thanks, that explains the behaviour observed
adeelm (Mon, 21 Jan 2019 19:05:58 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=88dc988d-50dc-4bdf-9789-70e1be91a5b6) @danintel Thank you @danintel I am following this URL that you have sent but I am not able to start the validator properly and the first node is able to show the peers but the other nodes do not display the peers. I am trying to setup a network of 5 nodes.
manojgop (Tue, 22 Jan 2019 09:32:21 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=9qNyrgiuuMymDzwoG) @jsmitchell I observed that if we call finalize_block() without calling summarize_block(), then finalize_block() will return InvalidState("Cannot finalize block in current state") when I send invalid transaction. For example, raft consensus was not calling summarize_block() before calling finalize_block() and raft node crashes if we send invalid transaction. ```
When I checked the journal/publisher.rs, it looks like InvalidState error occurs since finalize_block() is returning "FinalizeBlockError::BlockNotInitialized". finalize_block() should have ideally called restart_block() and returned FinalizeBlockError::BlockEmpty. summarize_block() is calling restart_block() and returning FinalizeBlockError::BlockEmpty if there are invalid transactions.
```
manojgop (Tue, 22 Jan 2019 09:32:21 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=9qNyrgiuuMymDzwoG) @jsmitchell I observed that if we call finalize_block() without calling summarize_block(), then finalize_block() will return InvalidState("Cannot finalize block in current state") when I send invalid transaction. For example, raft consensus was not calling summarize_block() before calling finalize_block() and raft node crashes if we send invalid transaction. ```
When I checked the journal/publisher.rs, it looks like InvalidState error occurs since finalize_block() is returning "FinalizeBlockError::BlockNotInitialized". finalize_block() should have ideally called restart_block() and returned FinalizeBlockError::BlockEmpty if there are invalid transactions in batch. summarize_block() is calling restart_block() and returning FinalizeBlockError::BlockEmpty if there are invalid transactions.
```
manojgop (Tue, 22 Jan 2019 09:32:21 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=9qNyrgiuuMymDzwoG) @jsmitchell I observed that if we call finalize_block() without calling summarize_block(), then finalize_block() will return InvalidState("Cannot finalize block in current state") when I send invalid transaction. For example, raft consensus was not calling summarize_block() before calling finalize_block() and raft node crashes if we send invalid transaction. ```
When I checked the journal/publisher.rs, it looks like InvalidState error occurs since finalize_block() is returning "FinalizeBlockError::BlockNotInitialized". finalize_block() should have ideally called restart_block() and returned FinalizeBlockError::BlockEmpty if there are invalid transactions in batch. summarize_block() is calling restart_block() and returning FinalizeBlockError::BlockEmpty if there are invalid transactions. So for now, I need to call summarize_block() before calling finalize_block() to avoid crashing raft node.
```
manojgop (Tue, 22 Jan 2019 09:32:21 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=9qNyrgiuuMymDzwoG) @jsmitchell @ltseeley I observed that if we call finalize_block() without calling summarize_block(), then finalize_block() will return InvalidState("Cannot finalize block in current state") when I send invalid transaction. For example, raft consensus was not calling summarize_block() before calling finalize_block() and raft node crashes if we send invalid transaction. ```
When I checked the journal/publisher.rs, it looks like InvalidState error occurs since finalize_block() is returning "FinalizeBlockError::BlockNotInitialized". finalize_block() should have ideally called restart_block() and returned FinalizeBlockError::BlockEmpty if there are invalid transactions in batch. summarize_block() is calling restart_block() and returning FinalizeBlockError::BlockEmpty if there are invalid transactions. So for now, I need to call summarize_block() before calling finalize_block() to avoid crashing raft node in case of invalid transaction.
```
muniyaraj (Tue, 22 Jan 2019 09:51:05 GMT):
How to fix the issue on ubuntu. this is supply chain project
I showing "Step 1/11 : FROM hyperledger/sawtooth-shell:nightly
ERROR: Service 'shell' failed to build: manifest for hyperledger/sawtooth-shell:nightly not found"
amolk (Tue, 22 Jan 2019 09:55:36 GMT):
@muniyaraj Please take your query to the #sawtooth-supply-chain channel
muniyaraj (Tue, 22 Jan 2019 09:57:05 GMT):
I have sent but dont have reply
muniyaraj (Tue, 22 Jan 2019 09:57:11 GMT):
thank u.
danintel (Tue, 22 Jan 2019 18:33:18 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=XxYCyRP72gy7ZbJgo) @adeelm One possibility is that networking between the nodes is not working then. TCP port 8800 needs to be open between peer nodes.
ltseeley (Tue, 22 Jan 2019 20:16:41 GMT):
@manojgop you should be able to catch and handle the `InvalidState` error when calling `finalize_block`; currently if `finalize_block` returns any error other than `BlockNotReady`, Raft will simply `panic`.
ltseeley (Tue, 22 Jan 2019 20:19:07 GMT):
https://github.com/hyperledger/sawtooth-raft/blob/master/src/node.rs#L205
youngerjo (Wed, 23 Jan 2019 03:44:48 GMT):
Has joined the channel.
manojgop (Wed, 23 Jan 2019 04:39:18 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=wPDJupcGjDLR3y23s) @ltseeley I'm expecting `BlockNotReady `if we sent invalid transactions instead of `InvalidState` error. But if I call `summarize_block()` before `finalize_block()` I'm getting `BlockNotReady` as expected . So for now I've modified the Raft code to call `summarize_block()` before `finalize_block()`
Psypher246 (Wed, 23 Jan 2019 06:46:21 GMT):
Has joined the channel.
Psypher246 (Wed, 23 Jan 2019 07:27:47 GMT):
Hello all,
I've been working on orchestrating a cloud deployment of a sawtooth network based on 1.1.2 with poet and have been struggling with some of the requirements. My problem is specifically with a circular dependency between the validator and the poet engine containers as per the docker-compose file found here: `https://github.com/hyperledger/sawtooth-core/blob/master/docker/compose/sawtooth-default-poet.yaml` In that deployment the logic flow is as such:
launch validator and poet engine container in parallel
validator creates keys and copies them to a shared volume
poet engine runs poet enclave commands and saves the output to shared volume
poet engine copies a key to the shared volume
poet engine runs `poet create registration` and saves the `poet.batch` file to the shared volume, at the same time writing the data to a store in `/var/lib/sawtooth`
poet engine waits for validator key to be copied to shared volume and runs the `poet-engine` command
validator waits for poet enclave files and key file and with it creates a `config.batch` file using `sawset proposal create`
validator waits for poet.batch file and creates a genesis block with it and the config.batch by running `sawadm genesis`
validator runs `sawtooth-validator` command
My problem lies with the back and forth copying of the keys and poet.batch file as the orchestration service I'm using (AWS CloudFormation) requires that the validator container has completed launching before it's automatically generated and registered ip and hostname is available to use downstream in the automation, in this case, as an argument for `poet-engine` to connect to. IE: I can't run validator till I run the poet-engine but I can't run the poet engine till I run the validator.
Now one way I think I can get around this is by installing poet-cli on the validator and running all the poet commands there, but at first this was not working since the poet engine also needed the content in the keystore in `/var/lib/sawtooth`. The result of not having this keystore but yet having the poet.batch file is that everything starts up fine, no errors, but when you try submit any transactions nothing happens, all you get is a pending tx url, no logs appear on any of the containers, which made this hard to troubleshoot. I got a hint yesterday about the keystore and did a simple test with docker compose and was able to replicate a broken node as I described, and fixed it by copying the keystore files from the validator to the poet-engine. I need to do more testing but this did seem to work, however this is not pretty and I don't know what other effects this will have. It's not too hard to get around this but given that the deployment goes onto a distributed computing environment, where I don't have access to shared storage, my best course of action is to use an object store like S3 to copy the keyfiles into and then back again to the poet-engine when it's booting. Additionally I have to install the awscli onto the container to achieve this, which adds bloat and means I have to build my own container.
So my questions are, if I was to go with copying the keystore from the validator to the poet engine will that cause any unforeseen issues?
And is there a better way to do this? If not at this time but in future releases could this be made to work better in a loosely coupled fashion? It seems fine when working with a single docker-compose file on one machine but starts to become problematic when decoupled.
Thanks in advance.
Psypher246 (Wed, 23 Jan 2019 07:27:47 GMT):
Hello all,
I've been working on orchestrating a cloud deployment of a sawtooth network based on 1.1.2 with poet and have been struggling with some of the requirements. My problem is specifically with a circular dependency between the validator and the poet engine containers as per the docker-compose file found here: `https://github.com/hyperledger/sawtooth-core/blob/master/docker/compose/sawtooth-default-poet.yaml` In that deployment the logic flow is as such:
launch validator and poet engine container in parallel
validator creates keys and copies them to a shared volume
poet engine runs poet enclave commands and saves the output to shared volume
poet engine copies a key to the shared volume
poet engine runs `poet create registration` and saves the `poet.batch` file to the shared volume, at the same time writing the data to a store in `/var/lib/sawtooth`
poet engine waits for validator key to be copied to shared volume and runs the `poet-engine` command
validator waits for poet enclave files and key file and with it creates a `config.batch` file using `sawset proposal create`
validator waits for poet.batch file and creates a genesis block with it and the config.batch by running `sawadm genesis`
validator runs `sawtooth-validator` command
My problem lies with the back and forth copying of the keys and poet.batch file as the orchestration service I'm using (AWS CloudFormation) requires that the validator container has completed launching before it's automatically generated and registered ip and hostname is available to use downstream in the automation, in this case, as an argument for `poet-engine` to connect to. IE: I can't run validator till I run the poet-engine but I can't run the poet engine till I run the validator.
Now one way I think I can get around this is by installing poet-cli on the validator and running all the poet commands there, but at first this was not working since the poet engine also needed the content in the keystore in `/var/lib/sawtooth`. The result of not having this keystore but yet having the poet.batch file is that everything starts up fine, no errors, but when you try submit any transactions nothing happens, all you get is a pending tx url, no logs appear on any of the containers, which made this hard to troubleshoot. I got a hint yesterday about the keystore and did a simple test with docker compose and was able to replicate a broken node as I described, and fixed it by copying the keystore files from the validator to the poet-engine. I need to do more testing but this did seem to work, however this is not pretty and I don't know what other effects this will have. It's not too hard to get around this but given that the deployment goes onto a distributed computing environment, where I don't have access to shared storage, my best course of action is to use an object store like S3 to copy the keyfiles into and then back again to the poet-engine when it's booting. Additionally I have to install the awscli onto the container to achieve this, which adds bloat and means I have to build my own container.
So my questions are, if I was to go with copying the keystore from the validator to the poet engine will that cause any unforeseen issues?
And is there a better way to do this? If not at this time but in future releases could this be made to work better in a loosely coupled fashion? It seems fine when working with a single docker-compose file on one machine but starts to become problematic when decoupled.
Thanks in advance.
EDIT: Turns out that worked but I had to jump through some new hoops since the keystore file is a 1TB sparse file. As per a tip here: `https://stackoverflow.com/questions/13252682/copying-a-1tb-sparse-file` I had to use `bsdtar Sczvf` on the validator and copy the tgz to S3 and then `tar Szxvf` on the poet engine to unzip it. For the first time I have a node working with poet but when I create a game with XO I noticed it takes about 8 seconds before it appears in `xo list` which seems quite long, is this normal?
Now going to try connect a second node.
Psypher246 (Wed, 23 Jan 2019 07:27:47 GMT):
Hello all,
I've been working on orchestrating a cloud deployment of a sawtooth network based on 1.1.2 with poet and have been struggling with some of the requirements. My problem is specifically with a circular dependency between the validator and the poet engine containers as per the docker-compose file found here: `https://github.com/hyperledger/sawtooth-core/blob/master/docker/compose/sawtooth-default-poet.yaml` In that deployment the logic flow is as such:
launch validator and poet engine container in parallel
validator creates keys and copies them to a shared volume
poet engine runs poet enclave commands and saves the output to shared volume
poet engine copies a key to the shared volume
poet engine runs `poet create registration` and saves the `poet.batch` file to the shared volume, at the same time writing the data to a store in `/var/lib/sawtooth`
poet engine waits for validator key to be copied to shared volume and runs the `poet-engine` command
validator waits for poet enclave files and key file and with it creates a `config.batch` file using `sawset proposal create`
validator waits for poet.batch file and creates a genesis block with it and the config.batch by running `sawadm genesis`
validator runs `sawtooth-validator` command
My problem lies with the back and forth copying of the keys and poet.batch file as the orchestration service I'm using (AWS CloudFormation) requires that the validator container has completed launching before it's automatically generated and registered ip and hostname is available to use downstream in the automation, in this case, as an argument for `poet-engine` to connect to. IE: I can't run validator till I run the poet-engine but I can't run the poet engine till I run the validator.
Now one way I think I can get around this is by installing poet-cli on the validator and running all the poet commands there, but at first this was not working since the poet engine also needed the content in the keystore in `/var/lib/sawtooth`. The result of not having this keystore but yet having the poet.batch file is that everything starts up fine, no errors, but when you try submit any transactions nothing happens, all you get is a pending tx url, no logs appear on any of the containers, which made this hard to troubleshoot. I got a hint yesterday about the keystore and did a simple test with docker compose and was able to replicate a broken node as I described, and fixed it by copying the keystore files from the validator to the poet-engine. I need to do more testing but this did seem to work, however this is not pretty and I don't know what other effects this will have. It's not too hard to get around this but given that the deployment goes onto a distributed computing environment, where I don't have access to shared storage, my best course of action is to use an object store like S3 to copy the keyfiles into and then back again to the poet-engine when it's booting. Additionally I have to install the awscli onto the container to achieve this, which adds bloat and means I have to build my own container.
So my questions are, if I was to go with copying the keystore from the validator to the poet engine will that cause any unforeseen issues?
And is there a better way to do this? If not at this time but in future releases could this be made to work better in a loosely coupled fashion? It seems fine when working with a single docker-compose file on one machine but starts to become problematic when decoupled.
Thanks in advance.
EDIT: Turns out that worked but I had to jump through some new hoops since the keystore file is a 1TB sparse file. As per a tip here: `https://stackoverflow.com/questions/13252682/copying-a-1tb-sparse-file` I had to use `bsdtar Sczvf` on the validator and copy the tgz to S3 and then `tar Szxvf` on the poet engine to unzip it. For the first time I have a node working with poet but when I create a game with XO I noticed it takes about 8 seconds before it appears in `xo list` which seems quite long, is this normal? I don't recall devmode on sawtooth 1.0.5 being this slow
Now going to try connect a second node.
pankajgoyal (Wed, 23 Jan 2019 09:32:44 GMT):
Regarding Raft Consensus, I'm stuck that all nodes are not taking intkey transaction.
Setup: 4 nodes NUCs, ubuntu/bumper/nightly, default raft consensus parametes.
My nodes are: N1, N2, N3, N4. N2 is the genesis and Leader node as well.
I submitted an intkey transaction to N3 rest-endpoint. Transaction reaches to all nodes. Verified by seeing TP log.
When I query rest-endpoints of all nodes, only N3 and N4 are showing intkey transaction.
I've tried this many times but the behaviour is same.
amundson (Thu, 24 Jan 2019 06:48:48 GMT):
@Psypher246 the inter-block time w/PoET will vary; the target wait time can be modified, but usually 30s is a good figure. running a PoET network with too few nodes will cause the actual wait time to vary widely (>=5 nodes is preferred). if it only took 8s, that seems within norms; if you are running only one validator, that wait time could have ended up being 5 minutes if you weren't lucky with the random number generation (more nodes averages this out). Throughput is a combination of the average wait time and the number of batches per block. devmode on the other hand, just produces blocks without any waiting.
amundson (Thu, 24 Jan 2019 06:54:27 GMT):
@pankajgoyal maybe try it with 1.1 and see if you get the same result. it would be an interesting data point. Raft needs more testing and refinement, for sure. If you want something more stable, use PBFT, we will actively chase after any bugs in PBFT.
Psypher246 (Thu, 24 Jan 2019 07:02:35 GMT):
@amundson thanks for the explanation. Now if I only could get more than one node to work...
Also any comment about the other pains?
amundson (Thu, 24 Jan 2019 07:05:40 GMT):
@Psypher246 I think @rberg2 could potentially give some insights, he has been doing kubernetes deployments of 1.1+PoET and had to solve these things in that context too.
manojgop (Thu, 24 Jan 2019 13:13:33 GMT):
I've a question on Sawtooth Raft design related to Log entry. The Log entries maintained in Sawtooth Raft corresponds to block id. My understanding is, in current design, the Leader node commits the block if the followers has the corresponding block in its log entry. But leader doesn't ensure if the follower node has received all the batches corresponding to this block before commiting this block. I guess the assumption here is follower will receive the batches via gossip network from its peers before leader commits the block. ? After running raft for more than 24 hrs I've seen the follower nodes sending too many batch request and leader node responding to batch request.
manojgop (Thu, 24 Jan 2019 13:13:33 GMT):
I've a question on Sawtooth Raft design related to Log entry. The Log entries maintained in Sawtooth Raft corresponds to block id. My understanding is, in current design, the Leader node commits the block if the followers has the corresponding block in its log entry. But leader doesn't ensure if the follower node has received all the batches corresponding to this block before commiting this block. I guess the assumption here is follower will receive the batches of the block via gossip network from its peers before leader commits the block. ? After running raft for more than 24 hrs I've seen the follower nodes sending too many batch request and leader node responding to batch request.
manojgop (Thu, 24 Jan 2019 13:13:33 GMT):
I've a question on Sawtooth Raft design related to Log entry. The Log entries maintained in Sawtooth Raft corresponds to block id. My understanding is, in current design, the Leader node commits the block if the followers has the corresponding block in its log entry. But leader doesn't ensure if the follower node has received all the batches corresponding to this block before commiting this block. I guess the assumption here is follower will receive the batches of the block via gossip network from its peers before leader commits the block. ? After running raft for more than 24 hrs I've seen the follower nodes sending too many batch requests and leader node responding to batch request. Could someone give any insight on this behavior. Because of this Eventually the leader nodes gets lot of messages to process and its unable to process the headet beat pings from the peer validators and eventually removes the connection. So there is some bottleneck in the system due to which message processing is getting slow
manojgop (Thu, 24 Jan 2019 13:13:33 GMT):
I've a question on Sawtooth Raft design related to Log entry. The Log entries maintained in Sawtooth Raft corresponds to block id. My understanding is, in current design, the Leader node commits the block if the followers has the corresponding block in its log entry. But leader doesn't ensure if the follower node has received all the batches corresponding to this block before commiting this block. I guess the assumption here is follower will receive the batches in the block via gossip network from its peers before leader commits the block. ? After running raft for more than 24 hrs I've seen the follower nodes sending too many batch requests and leader node responding to batch request. Could someone give any insight on this behavior. Because of this Eventually the leader nodes gets lot of messages to process and its unable to process the headet beat pings from the peer validators and eventually removes the connection. So there is some bottleneck in the system due to which message processing is getting slow
manojgop (Thu, 24 Jan 2019 13:13:33 GMT):
I've a question on Sawtooth Raft design related to Log entry. The Log entries maintained in Sawtooth Raft corresponds to block id. My understanding is, in current design, the Leader node commits the block if the followers has the corresponding block in its log entry. But leader doesn't ensure if the follower node has received all the batches corresponding to this block before commiting this block. I guess the assumption here is follower will receive the batches in the block via gossip network from its peers before leader commits the block. ? After running raft for more than 24 hrs I've seen the follower nodes sending too many batch requests and leader node responding to batch request. Could someone give any insight on this behavior. Because of this eventually the leader nodes gets lot of messages to process and its unable to process the heart beat pings from the peer validators and eventually removes the peer connection after timeout. So there is some bottleneck in the system due to which message processing is getting slow
rberg2 (Thu, 24 Jan 2019 15:56:36 GMT):
Hi @Psypher246 everything I have been doing in Kubernetes uses the /poet-shared stuff with the the shell loops waiting for files to arrive. I am not quite clear what about AWS CloudFormation
rberg2 (Thu, 24 Jan 2019 15:56:36 GMT):
Hi @Psypher246 everything I have been doing in Kubernetes uses the /poet-shared stuff with the the shell loops waiting for files to arrive. I am not quite clear what about AWS CloudFormation prevents you from using volumes within a pod edit: the send button is way too close to the cursor and I keep sending when I dont mean to :\
jsmitchell (Thu, 24 Jan 2019 17:36:22 GMT):
@manojgop how many nodes and where were you submitting the batches in this scenario?
manojgop (Fri, 25 Jan 2019 04:10:39 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=YgJ97zqWbY67uxbBJ) @jsmitchell 3 nodes
manojgop (Fri, 25 Jan 2019 04:10:39 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=YgJ97zqWbY67uxbBJ) @jsmitchell 3 nodes. Docker based. All nodes running on same VM
manojgop (Fri, 25 Jan 2019 04:10:39 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=YgJ97zqWbY67uxbBJ) @jsmitchell 3 nodes . Docker based. All nodes running on same VM. Running intkey work load at 5TPS
Behzad 2 (Sat, 26 Jan 2019 20:41:28 GMT):
Has joined the channel.
jastisriradheshyam (Sun, 27 Jan 2019 08:27:59 GMT):
Has joined the channel.
pankajgoyal (Mon, 28 Jan 2019 09:24:11 GMT):
I see that "version" and "name" are hardcoded for poet (https://github.com/hyperledger/sawtooth-poet/blob/master/engine/sawtooth_poet_engine/engine.py) and devmode (https://github.com/hyperledger/sawtooth-devmode/blob/master/src/engine.rs) and for Raft (https://github.com/hyperledger/sawtooth-raft/blob/master/src/engine.rs) version is CARGO_PKG_VERSION and name is CARGO_PKG_NAME. In case of debian raft package, CARGO_PKG_VERSION is a mismatch to sawtooth.consensus.algorithm.version.
pankajgoyal (Mon, 28 Jan 2019 09:36:52 GMT):
PBFT is also using CARGO_PKG_VERSION. Has anyone installed and run PBFT using debian package?
moulika (Mon, 28 Jan 2019 11:04:33 GMT):
Has joined the channel.
ltseeley (Mon, 28 Jan 2019 17:26:49 GMT):
@pankajgoyal yes, I've installed and run PBFT using a debian package and it has worked for me.
pankajgoyal (Mon, 28 Jan 2019 17:27:40 GMT):
what are the pbft settings that you set?
ltseeley (Mon, 28 Jan 2019 17:28:56 GMT):
The settings should be: ```
sawtooth.consensus.algorithm.name=pbft
sawtooth.consensus.algorithm.version=0.1.0
```
ltseeley (Mon, 28 Jan 2019 17:29:28 GMT):
That is, if the `name` and `version` variables in the `Cargo.toml` that is used to build the debian package are `pbft` and `0.1.0` respectively
arsulegai (Tue, 29 Jan 2019 08:38:09 GMT):
@ltseeley Right this setting works for debian released under 0.1.0 version, if debian is from nightly it differs (ref: https://github.com/hyperledger/sawtooth-pbft/blob/8a8224dffadd7780cd4adac76e201f06331358b3/Dockerfile-installed-bionic#L51 ~same for Raft), this information is not documented.
arsulegai (Tue, 29 Jan 2019 08:38:09 GMT):
@ltseeley Right the version field is fetched from contents set in VERSION file, this will not match 0.1.0 for nightly releases. There is need to document a way to fetch this version number.
arsulegai (Tue, 29 Jan 2019 08:38:09 GMT):
@ltseeley Right the version field is fetched from contents set in VERSION file, this will not match 0.1.0 for nightly releases. There is need to document a way to fetch this version number, in these cases eventually a version check command "sawtooth-raft-engine --version" would give the value.
muniyaraj (Wed, 30 Jan 2019 08:41:59 GMT):
hi all, Can you please tell how to use sawtooth explorer
Have any guidance for that?
pschwarz (Wed, 30 Jan 2019 16:18:48 GMT):
@muniyaraj that's a question for the general #sawtooth channel
robinrob (Wed, 30 Jan 2019 19:42:15 GMT):
Has joined the channel.
amundson (Thu, 31 Jan 2019 16:54:18 GMT):
@arsulegai @kelly_ @TomBarnes is there a transaction processor spec I can look at for PoET 2's valdiator registry? I want to compare it to the PoET 1 registry.
arsulegai (Fri, 01 Feb 2019 03:37:49 GMT):
@amundson Validator registry TP follows same implementation as in PoET 1, however you will see slight deviations when it comes to signup info verification. PSE manifest params are not verified in PoET 2.
manojgop (Fri, 01 Feb 2019 04:24:18 GMT):
@amundson Removing PSE (Platform Service Enclave) dependencies including monotonic counters is one of the major difference of POET2 SGX enclave compared with POET1. So PSE Manifest checks have been removed in POET2 (in IAS verification report)
manojgop (Fri, 01 Feb 2019 04:24:18 GMT):
@amundson Removing PSE (Platform Service Enclave) dependencies including monotonic counters is one of the major difference of POET2 SGX enclave compared with POET1. So PSE Manifest checks have been removed in POET2 (in IAS verification report). This change was done to support running POET on Intel Xeon servers where Platform Services (monotonic counters, trusted time) are not available
manojgop (Fri, 01 Feb 2019 04:24:18 GMT):
@amundson Removing PSE (Platform Service Enclave) dependencies including monotonic counters is one of the major difference of POET2 SGX enclave compared with POET1. So PSE Manifest checks have been removed in POET2 (in IAS verification report). This change was done to support running POET on Intel Xeon servers with SGX support but Platform Services (monotonic counters, trusted time) are not available
Zeshun (Mon, 04 Feb 2019 00:18:16 GMT):
Has joined the channel.
manojgop (Mon, 04 Feb 2019 12:17:28 GMT):
@amundson . Couple of points regarding POET2 PR -> https://github.com/hyperledger/sawtooth-poet/pull/10```
```
manojgop (Mon, 04 Feb 2019 12:17:28 GMT):
@amundson . Couple of points regarding POET2 PR -> https://github.com/hyperledger/sawtooth-poet/pull/10```
```
manojgop (Mon, 04 Feb 2019 12:19:47 GMT):
@amundson. Below are some points which provides some background of POET2 PR ->https://github.com/hyperledger/sawtooth-poet/pull/10
manojgop (Mon, 04 Feb 2019 12:19:47 GMT):
@amundson Below are some points which provides some background of POET2 PR ->https://github.com/hyperledger/sawtooth-poet/pull/10
manojgop (Mon, 04 Feb 2019 12:19:47 GMT):
@amundson Below are some points which provides some background on POET2 PR ->https://github.com/hyperledger/sawtooth-poet/pull/10
manojgop (Mon, 04 Feb 2019 12:22:33 GMT):
1. New/Modified components in POET2 is not designed to be backward compatible with existing POET1.
2. After POET2 functionality is enabled we don't plan to continue supporting POET1 in future releases.
3. Not all POET1 files/folders are deleted. Some of the existing POET1 files are reused and are moved to new directory structure
4. Existing PR raised have lot of commits. If existing PR have to be split into independent components which can be built independently then that would require modifying the git commit history. This would need lot of rework. We can consider doing this - For example, raise independent PRs for VRTP, POET2 Enclave, RUST-C++ SGX Enclave FFI, IAS client/proxy
manojgop (Mon, 04 Feb 2019 12:22:33 GMT):
1. New/Modified components in POET2 is not designed to be backward compatible with existing POET1.
2. After POET2 functionality is enabled we don't plan to continue supporting POET1 in future releases.
3. Not all POET1 files/folders are deleted. Some of the existing POET1 files are reused and are moved to new directory structure
4. Existing PR raised have lot of commits. If existing PR have to be split into independent components which can be built independently then that would require modifying the git commit history. We can consider doing this - For example, raise independent PRs for VRTP, POET2 Enclave, RUST-C++ SGX Enclave FFI, IAS client/proxy. Once these PRs get merged we can raise PR for main POET2 engine. This would need lot of rework.
manojgop (Mon, 04 Feb 2019 12:22:33 GMT):
1. New/Modified components in POET2 is not designed to be backward compatible with existing POET1.
2. After POET2 functionality is enabled we don't plan to continue supporting POET1 in future releases.
3. Not all POET1 files/folders are deleted. Some of the existing POET1 files are reused and are moved to new directory structure
4. Existing PR raised have lot of commits. If existing PR have to be split into independent components which can be built independently then that would require modifying the git commit history. We can consider doing this - For example, raise independent smaller PRs for VRTP, POET2 Enclave, RUST-C++ SGX Enclave FFI, IAS client/proxy. Once these PRs get merged we can raise another PR for main POET2 engine. This would need lot of rework and modifying git commit history.
manojgop (Mon, 04 Feb 2019 12:22:33 GMT):
1. New/Modified components in POET2 is not designed to be backward compatible with existing POET1.
2. After POET2 functionality is enabled we don't plan to continue supporting POET1 in future releases.
3. Not all POET1 files/folders are deleted. Some of the existing POET1 files are reused and are moved to new directory structure
4. Existing PR raised have lot of commits. If existing PR have to be split into independent components which can be built independently then that would require modifying the git commit history. We can consider doing this - For example, raise independent smaller PRs for Validator Registry TP, POET2 SGX Enclave, RUST-C++ SGX Enclave FFI, IAS client/proxy. Once these PRs get merged we can raise another PR for main POET2 engine. This would need lot of rework and modifying git commit history.
manojgop (Mon, 04 Feb 2019 12:22:33 GMT):
1. New/Modified components in POET2 is not designed to be backward compatible with existing POET1.
2. After POET2 functionality is enabled we don't plan to continue supporting POET1 in future releases.
3. Not all POET1 files/folders are deleted. Some of the existing POET1 files are reused and are moved to new directory structure
4. Existing PR raised have lot of commits. If existing PR have to be split into independent smaller PRs which can be built independently then that would require modifying the git commit history. We can consider doing this - For example, raise independent smaller PRs for Validator Registry TP, POET2 SGX Enclave, RUST-C++ SGX Enclave FFI, IAS client/proxy. Once these PRs get merged we can raise another PR for main POET2 engine. This would need lot of rework and modifying git commit history.
manojgop (Mon, 04 Feb 2019 12:22:33 GMT):
1. New/Modified components in POET2 is not designed to be backward compatible with existing POET1.
2. After POET2 functionality is enabled we don't plan to continue supporting POET1 in future releases.
3. Not all POET1 files/folders are deleted. Some of the existing POET1 files are reused and are moved to new directory structure
4. Existing PR raised have lot of commits. If existing PR have to be split into independent smaller PRs which can be built independently then that would require modifying the git commit history. We can consider doing this - For example, raise independent smaller PRs for Validator Registry TP, POET2 SGX Enclave, RUST-C++ SGX Enclave FFI, IAS client/proxy. ( Currently these components and enhancements are added as commits in same PR). Once these smaller PRs get merged we can raise another PR for main POET2 engine. But this would need lot of rework and modifying git commit history.
manojgop (Mon, 04 Feb 2019 12:28:06 GMT):
Would you be able to review the code in https://github.com/manojgop/sawtooth-poet/tree/poet2-dev. This is the development repo for POET2. We could setup a call to give a quick walk though of main components of POET2 and differences compared to POET1. This could probably help to address review comments and expedite the review process. Let me know how you would like to take this PR forward
manojgop (Mon, 04 Feb 2019 12:28:06 GMT):
Would you be able to review the code in https://github.com/manojgop/sawtooth-poet/tree/poet2-dev. This is the development repo for POET2. We could setup a call to give a quick walk through of main components of POET2 and differences compared to POET1. This could probably help to address review comments and expedite the review process. Let me know how you would like to take this PR forward
amundson (Mon, 04 Feb 2019 17:18:12 GMT):
@manojgop Deprecating PoET in its current form before we have a viable, tested alternative seems premature. Dropping support for the current PoET (especially without an appropriate upgrade path) seems like a non-starter to me without support for state checkpointing, since we need to be able to re-process our existing chains with existing blocks.
amundson (Mon, 04 Feb 2019 17:27:06 GMT):
@manojgop With respect to reviews, PRs need to be submitted in reasonable chunks. Not +13722,-32000. I don't think the commit history in the PR is particularly useful or relevant as-is, because it was neither peer-reviewed or broken up in a way meant to be reviewable. (Thought the later commits in the PR look substantially better than the first.)
amundson (Mon, 04 Feb 2019 17:36:33 GMT):
I'm willing to be an advocate for getting these PRs reviewed and merged in, but you have to start taking some advice, because otherwise I can't help you. My current advice is to submit a seperate PR against poet2 branch with only the new validator registory TP. I'm interested in seeing that eventually support the current version of PoET even if your team isn't working on it, so it would be beneficial to do it separately so we can easily cherry-pick those commits from poet2 branch to master (adding/modifying it to support PoET 1).
amundson (Mon, 04 Feb 2019 17:37:53 GMT):
That seems sufficiently self-contained, and once we get through that simplified PR, then we can select another piece for the next PR, etc.
manojgop (Tue, 05 Feb 2019 05:46:25 GMT):
@amundson Sounds good. We'll create a design document which highlights the changes made in POET2 and differences w.r.t POET1. Going forward can we review the design first to avoid lot of code rework for future PRs. Shall we share the design document via this channel or do you prefer any other forum where we can discuss this. We can start reviewing the design of validator registry TP followed by other components - POET2 SGX Enclave, RUST-C++ SGX Enclave FFI, IAS client/proxy and then POET2 engine.
amundson (Tue, 05 Feb 2019 06:37:51 GMT):
@manojgop ok, looking forward to the TP PR. I suggest just starting by putting together the PR first, then going from there based on the PR comments.
manojgop (Tue, 05 Feb 2019 09:06:25 GMT):
@amundson Will work on the PR for validator registry TP. But one concern I've is if we don't review the design first it may result in lot of code rework during PR review. For example, in case of validator registry TP we went ahead with design assumption that it need not be backward compatible with POET1.
Psypher246 (Tue, 05 Feb 2019 09:14:15 GMT):
@rberg2 thanks for the confirmation, I went back and did some more research on the capabilities of AWS ECS and Fargate (which is where the problem really lay not CFN) and have been able to now get this to work in a similar fashion. Thanks again.
nanspro (Wed, 06 Feb 2019 15:26:24 GMT):
Has left the channel.
circlespainter (Sat, 09 Feb 2019 10:27:29 GMT):
Has joined the channel.
joaohsenger (Tue, 12 Feb 2019 13:52:42 GMT):
Has joined the channel.
jaypersanchez (Wed, 13 Feb 2019 11:31:11 GMT):
Has joined the channel.
jaypersanchez (Wed, 13 Feb 2019 11:32:23 GMT):
Hello everyone. How can I start to contribute to Sawtooth?
david-kim-tpe (Thu, 14 Feb 2019 08:21:50 GMT):
Has joined the channel.
coldmind (Tue, 19 Feb 2019 13:35:21 GMT):
Has joined the channel.
coldmind (Tue, 19 Feb 2019 13:35:48 GMT):
Hi!
I need to integrate Tendermint as a consensus provider for my Sawtooth project.
My thoughts are to create custom consensus engine, which will send some info
to the Tendermint via simple ABCI application: https://img42.com/CjdQL+
For example, Hyperledger Burrow is running a Tendermint node as a subprocess,
and somehow sending its own transactions to it.
I think I can do something similar: write a consensus engine for Sawtooth in Go,
which will give ability to run Tendermint as a subprocess too instead of separate node.
But, I'm conserned about which info I should send to Tendermint.
The main idea is to maintain blocks order state in Tendermint:
1. Validation step of Sawtooth requires to reach consensus in its own consensus engine.
2. I can send proposed block number and last block header hash to Tendermint.
3. After Tendermint will reach consensus (in other words stores this info among its nodes),
this will be a signal for the engine of Sawtooth that block is ready to be finalized.
Can this be a working solution to be integrated with Tendermint?
amundson (Tue, 19 Feb 2019 14:48:56 GMT):
@coldmind I looked at this a while ago. What you are saying sounds like it could work. My impression of ABCI is that it is similar to our interface for TPs (for writing apps) and not at all an interface for consensus. That approach would probably require Tendermint nodes still communicate to each other over a separate network unless that code is modified within Tendermint to be pluggable.
amundson (Tue, 19 Feb 2019 15:09:37 GMT):
I'm looking back at some of @adamludvik 's notes on the topic. With "Sawtooth as a Tendermint App" (using ABCI), the concerns were basically duplication of concerns between Tendermint and Sawtooth - specifically, they would both do p2p communication, block construction/signing, and consensus. Sawtooth would provide state management, transaction execution, transaction signing (there may be this in Tendermint too?), and client query handling (events, etc.).
amundson (Tue, 19 Feb 2019 15:12:48 GMT):
However, the problem w/Tendermint as just a consensus algorithm is that the current code base (this was from 2017) isn't modular and doesn't expose an API for using the just the consensus algorithm itself. So it is monolithic in that way, and would require extensive reworking of the Tendermint code to separate out the algorithm from the rest of the code.
amundson (Tue, 19 Feb 2019 15:12:48 GMT):
The other approach would be to use only the algorithm. However, the problem w/Tendermint as just a consensus algorithm is that the current code base (this was from 2017) isn't modular and doesn't expose an API for using the just the consensus algorithm itself. So it is monolithic in that way, and would require extensive reworking of the Tendermint code to separate out the algorithm from the rest of the code.
amundson (Tue, 19 Feb 2019 15:15:43 GMT):
I believe Burrow is a fairly traditional Tendermint application in terms of its relationship w/Tendermint.
amundson (Tue, 19 Feb 2019 15:17:18 GMT):
@coldmind If you need help understanding or working with the Sawtooth consensus engine API, @ltseeley has done work on PBFT and Raft consensus engines.
coldmind (Tue, 19 Feb 2019 19:07:41 GMT):
@amundson thank you for response. Indeed, Tendermint nodes will require to communicate to each other. The point is to have benefit of ABCI app, which will allow some nice opportunities along with built-in consensus engine. I will try to make it work.
amundson (Tue, 19 Feb 2019 19:10:31 GMT):
@coldmind how will you coorelate a sawtooth node to a tendermint node?
amundson (Tue, 19 Feb 2019 19:11:20 GMT):
or maybe it doesn't matter, its whatever node it's locally talking to. nevermind. :)
amundson (Tue, 19 Feb 2019 19:11:30 GMT):
good luck, let us know how it goes and how we can help
MelodyMcIntyre (Thu, 21 Feb 2019 06:48:28 GMT):
Has joined the channel.
duncanjw (Fri, 22 Feb 2019 11:04:09 GMT):
Has joined the channel.
duncanjw (Fri, 22 Feb 2019 11:05:05 GMT):
@amundson rookie question but what is the status of PoET-SGX on 1.1 as both the 1.1 and nightly docs say it is not supported but will be be end of 2018?
duncanjw (Fri, 22 Feb 2019 11:05:05 GMT):
@amundson rookie question but what is the status of PoET-SGX on 1.1 as both the 1.1 and nightly docs say it is not supported but will be by end of 2018?
duncanjw (Fri, 22 Feb 2019 11:06:50 GMT):
https://sawtooth.hyperledger.org/docs/core/nightly/master/sysadmin_guide/configure_sgx.html
danintel (Fri, 22 Feb 2019 18:51:47 GMT):
@duncanjw PoET2 is currently on hold due to a higher priority project (stabilizing Raft). So for now the options is using PoET-CFT on 1.1 or PoET-SGX on 1.0. Others directly involved might have more details.
duncanjw (Fri, 22 Feb 2019 19:19:04 GMT):
@danintel many thanks for clarifying
duncanjw (Fri, 22 Feb 2019 19:37:08 GMT):
@danintel worth pointing out that the system admin documentation I referenced doesn't mention Raft only PBFT and PoET-CFT so it's still a work in progress?
duncanjw (Fri, 22 Feb 2019 19:37:38 GMT):
Relevant section is https://sawtooth.hyperledger.org/docs/core/nightly/master/sysadmin_guide/setting_up_sawtooth_poet-sim.html
danintel (Fri, 22 Feb 2019 19:44:51 GMT):
@duncanjw That may be an oversight. Raft is mentioned at https://sawtooth.hyperledger.org/docs/core/nightly/master/introduction.html#dynamic-consensus-algorithms
Raft is further along than PBFT, last I heard. I'll let others chime in on that.
ltseeley (Fri, 22 Feb 2019 19:49:12 GMT):
Actually, PBFT is practically ready to use. We are getting very close to a stable release.
ltseeley (Fri, 22 Feb 2019 19:51:06 GMT):
There are a few little changes that we're working on right now, but most of our effort right now is being spent on testing and stabilizing.
ltseeley (Fri, 22 Feb 2019 19:51:06 GMT):
There are a few little changes that we're working on, but most of our effort right now is being spent on testing and stabilizing.
duncanjw (Fri, 22 Feb 2019 21:50:57 GMT):
@danintel according to current Raft documentation it is of limited use in production https://sawtooth.hyperledger.org/docs/raft/nightly/master/introduction.html
duncanjw (Fri, 22 Feb 2019 21:51:32 GMT):
The Sawtooth Raft consensus engine may be right for you if your Sawtooth deployment will:Consist of a small number of nodes (roughly 1 to 10)
Have a mostly fixed membership
Not require Byzantine fault tolerance
duncanjw (Fri, 22 Feb 2019 21:51:32 GMT):
The Sawtooth Raft consensus engine may be right for you if your Sawtooth deployment will:
1. Consist of a small number of nodes (roughly 1 to 10)
2. Have a mostly fixed membership
3. Not require Byzantine fault tolerance
duncanjw (Fri, 22 Feb 2019 21:53:39 GMT):
First two points means it is to quote Douglas Adams "mostly harmless" but in practice - if true - utterly useless
duncanjw (Fri, 22 Feb 2019 21:53:39 GMT):
First two points mean it is to quote Douglas Adams "mostly harmless" but in practice - if true - utterly useless
duncanjw (Fri, 22 Feb 2019 21:54:45 GMT):
Not sure I understand why this was given priority (if it was) over making sure PoET-SGX was compatible with Sawtooth 1.1
jsmitchell (Fri, 22 Feb 2019 21:56:41 GMT):
it's a do-ocracy. The people/teams that write the code make the decisions about what they are going to work on.
danintel (Fri, 22 Feb 2019 22:02:58 GMT):
@duncanjw I would add:
4. Finality--no forking of the blockchain as with Nakamoto (Lottery)-based consensus
amundson (Sat, 23 Feb 2019 02:50:34 GMT):
PBFT definitely not behind Raft. Recommend PBFT over Raft.
duncanjw (Sun, 24 Feb 2019 10:19:33 GMT):
@jsmitchell a do-ocracy is fine but as a community we have annouced Sawtooth 1.1 and given the impression it is building on what has gone before by adding support for new consensus mechanisms and if we want it adopted we can't afford to drop the ball on core features
duncanjw (Sun, 24 Feb 2019 10:19:33 GMT):
@jsmitchell a do-ocracy is fine but as a community we have announced Sawtooth 1.1 and given the impression it is building on what has gone before by adding support for new consensus mechanisms and if we want it adopted we can't afford to drop the ball on core features
duncanjw (Sun, 24 Feb 2019 10:19:33 GMT):
@jsmitchell a do-ocracy is fine but as a community we have announced Sawtooth 1.1 and given the impression it is building on what has gone before by adding support for new consensus mechanisms and if we want it adopted we can't afford to drop the ball on key features
duncanjw (Sun, 24 Feb 2019 10:22:36 GMT):
The release notes imply the same and - yes - they have a caveat re PoET-SGX if you read the small print there is CMA language -
>PoET-SGX has not been validated on Sawtooth 1.1. Users relying on PoET-SGX are recommended to remain on Sawtooth 1.0. We are working on a new implementation of poet and its TEE enclave, which is anticipated for a point release in the near future.
duncanjw (Sun, 24 Feb 2019 10:22:36 GMT):
The release notes imply the same and - yes - they have a caveat re PoET-SGX if you read the small print there is CMA language -
>PoET-SGX has not been validated on Sawtooth 1.1. Users relying on PoET-SGX are recommended to remain on Sawtooth 1.0. We are working on a new implementation of poet and its TEE enclave, which is anticipated for a point release in the near future.
duncanjw (Sun, 24 Feb 2019 10:22:36 GMT):
The release notes imply the same and - yes - they have a caveat re PoET-SGX if you read the small print there is CMA language -
>PoET-SGX has not been validated on Sawtooth 1.1. Users relying on PoET-SGX are recommended to remain on Sawtooth 1.0. We are working on a new implementation of poet and its TEE enclave, which is anticipated for a point release in the near future.
https://sawtooth.hyperledger.org/release/bumper/#consensus
duncanjw (Sun, 24 Feb 2019 10:22:36 GMT):
The release notes imply the same and - yes - they have a caveat re PoET-SGX if you read the small print there is the following caveat -
>PoET-SGX has not been validated on Sawtooth 1.1. Users relying on PoET-SGX are recommended to remain on Sawtooth 1.0. We are working on a new implementation of poet and its TEE enclave, which is anticipated for a point release in the near future.
https://sawtooth.hyperledger.org/release/bumper/#consensus
duncanjw (Sun, 24 Feb 2019 10:26:06 GMT):
The press release and release notes went out Dec 6 so to be told it's a do-ocracy (which is tantamount to flipping the bird!) nearly three months on doesn't cut it if as I say we want people to take us seriously
duncanjw (Sun, 24 Feb 2019 10:26:06 GMT):
The press release and release notes went out Dec 6 so to be told it's a do-ocracy (which is tantamount to being flipped the bird or at least that's how it feels!) nearly three months on doesn't cut it if as I say we want people to take us seriously
duncanjw (Sun, 24 Feb 2019 10:39:13 GMT):
@jsmitchell you've convinced me on the read with prefix point so maybe I can convince you on this point :)
duncanjw (Sun, 24 Feb 2019 10:44:57 GMT):
The problem is that by remaining on Sawtooth 1.0 someone cannot from benefit from all the improvements, bug fixes etc listed in the release note https://sawtooth.hyperledger.org/release/bumper/#core-sawtooth-components
duncanjw (Sun, 24 Feb 2019 10:44:57 GMT):
The problem is someone reading this will realised that by remaining on Sawtooth 1.0 they cannot from benefit from all the improvements, bug fixes etc listed in the release note https://sawtooth.hyperledger.org/release/bumper/#core-sawtooth-components
duncanjw (Sun, 24 Feb 2019 10:44:57 GMT):
The problem is someone reading this will realised that by remaining on Sawtooth 1.0 they cannot from benefit from all the improvements, *bug fixes* etc listed in the release note https://sawtooth.hyperledger.org/release/bumper/#core-sawtooth-components
duncanjw (Sun, 24 Feb 2019 10:44:57 GMT):
The problem is someone reading this will very quickly realise that by remaining on Sawtooth 1.0 they cannot from benefit from all the improvements, *bug fixes* etc listed in the release note https://sawtooth.hyperledger.org/release/bumper/#core-sawtooth-components
duncanjw (Sun, 24 Feb 2019 10:49:27 GMT):
Happy to have a call to discuss but right now it reads like PoET-SGX is essentially deprecated in Sawtooth 1.1
jsmitchell (Sun, 24 Feb 2019 14:40:05 GMT):
@duncanjw my point was that Intel has a team that has ostensibly “signed up” to maintain the poet code. So, if they decide to focus on something else for whatever reason, that’s up to them, because they are doing that work. I’m not going to attempt to convince them otherwise. Not my place. Now, if someone else decided that they wanted this fixed with priority and contributed some PRs, I’m sure those would be welcomed.
firevenus2019 (Mon, 25 Feb 2019 03:37:58 GMT):
Has joined the channel.
rbnaraujo (Tue, 26 Feb 2019 14:50:21 GMT):
Has joined the channel.
rbnaraujo (Tue, 26 Feb 2019 14:51:09 GMT):
Clipboard - February 26, 2019 10_08 AM.png
danintel (Tue, 26 Feb 2019 21:54:10 GMT):
@rbnaraujo I see the same question on stackoverflow and answered there. Basically you need to preserver file ownership and permissions when copying the files. Use the `cp -p` option. That is
`cp -p --sparse=always [file] [file_backup]` using your example at https://stackoverflow.com/questions/54892920/backup-lmdb-database-from-hyperledger-sawtooth/
prabu3192 (Wed, 27 Feb 2019 06:00:26 GMT):
Has joined the channel.
haggs (Sun, 03 Mar 2019 21:39:46 GMT):
Has joined the channel.
kodonnel (Tue, 05 Mar 2019 16:13:59 GMT):
anyone else noticing that download.01.org is 504 timing out, and therefore breaking the sawtooth-poet builds?
kodonnel (Tue, 05 Mar 2019 16:13:59 GMT):
anyone else noticing that download.01.org(where sgx bins are downloaded from) is 504 timing out, and therefore breaking the sawtooth-poet builds?
danintel (Tue, 05 Mar 2019 16:52:40 GMT):
Intel's https://download.01.org/ and subdirectories seem live to me. Maybe it was temporary or there is a specific file missing?
kodonnel (Wed, 06 Mar 2019 13:40:01 GMT):
back now for me too. It was the whole site for quite a few hours yesterday. c'est la vie.
pankajcheema (Thu, 07 Mar 2019 05:50:33 GMT):
Has joined the channel.
arsulegai (Thu, 07 Mar 2019 18:26:52 GMT):
Need advice - I've a question on the Sawtooth Raft consensus engine (it may well be a question on raft-rs)
Can applying snapshot make changes to the committed log entries?
arsulegai (Thu, 07 Mar 2019 18:28:37 GMT):
If so, will committed_entries field from ready contain the delta of those log entry changes? The delta of log entries which are either added or removed.
arsulegai (Thu, 07 Mar 2019 18:38:39 GMT):
^ @amundson
amundson (Thu, 07 Mar 2019 21:01:52 GMT):
@arsulegai I'm not that familiar with the raft code
arsulegai (Fri, 08 Mar 2019 00:22:00 GMT):
Ok. Other Sawtooth Raft developers, any help here is much appreciated
arsulegai (Fri, 08 Mar 2019 04:13:58 GMT):
@amundson Do you mean question is outside the of scope of sawtooth-raft and you're not familiar with raft-rs library part?
arsulegai (Fri, 08 Mar 2019 11:08:00 GMT):
@ltseeley Could you please help me with your understanding?
hyper_learner_ak (Wed, 13 Mar 2019 08:55:30 GMT):
Has joined the channel.
ltseeley (Wed, 13 Mar 2019 14:04:52 GMT):
@arsulegai based on my understanding of the Raft paper, it seems like the snapshot should NOT make changes to any log entries that are already committed (they should be final). However, I'm not sure to what extent the raft-rs library enforces that. It's been a while since I've done a lot with Raft.
arsulegai (Wed, 13 Mar 2019 14:23:13 GMT):
Thanks @ltseeley that resonates with the observation so far.
arsulegai (Wed, 13 Mar 2019 14:36:43 GMT):
We've a scenario
1. Leader node has proposed a new block, it received a go signal from maximum follower node. Ready status has a committable_entry.
2. Leader node added the block corresponding to this committable_entry in its Blockchain.
3. However the follower node went down before it could receive leader's commit confirmation.
4. Leader went ahead and proposed a new block, because it still has majority of followers (info on timers: block propose time is lesser than the election tick)
5. Follower node comes back online. Raft synced information from leader and received latest commit log entry.
arsulegai (Wed, 13 Mar 2019 14:46:54 GMT):
Problem in this case is that the follower node didn't receive latest block or the block that is already committed when follower node went down.
arsulegai (Wed, 13 Mar 2019 14:46:54 GMT):
Problem in this case is that the follower node didn't receive latest block or the block that is already committed when it went down.
arsulegai (Wed, 13 Mar 2019 14:50:28 GMT):
This scenario will be recovered when leader publishes a new block after committing current block. But what if at step 3, majority of followers go down?
arsulegai (Wed, 13 Mar 2019 14:55:17 GMT):
Wanted to hear opinion on these scenarios. Do we claim in such case that Raft is not CFT if majority of nodes are down?
manojgop (Wed, 13 Mar 2019 15:11:48 GMT):
In Raft its possible that a log entry that's committed in leader node can be overwritten later if a follower becomes leader. So is the block added to block chain by leader node. ```
Entry in Raft is committed only if it satisfies two conditions 1) It is majority stored and 2) at least one entry from leaders term is also majority stored
```
manojgop (Wed, 13 Mar 2019 15:11:48 GMT):
In Raft its possible that a log entry that's committed in leader node can be overwritten later if a follower becomes leader. So is the block added to block chain by leader node, the head could be overwritten later```
Entry in Raft is committed only if it satisfies two conditions 1) It is majority stored and 2) at least one entry from leaders term is also majority stored
```
manojgop (Wed, 13 Mar 2019 15:11:48 GMT):
In Raft its possible that a log entry that's committed in leader node can be overwritten later if a follower becomes leader. So if the block added to block chain by leader node, the head could be overwritten later```
Entry in Raft is committed only if it satisfies two conditions 1) It is majority stored and 2) at least one entry from leaders term is also majority stored
``` ```
This video explains this case it towards the end https://www.youtube.com/watch?v=RHDP_KCrjUc
```
manojgop (Wed, 13 Mar 2019 15:11:48 GMT):
In Raft its possible that a log entry that's committed in leader node can be overwritten later if a follower becomes leader. So if the block added to block chain by leader node, the head could be overwritten later```
Entry in Raft is committed only if it satisfies two conditions 1) It is majority stored and 2) at least one entry from leaders term is also majority stored
``` ```
This video explains this case towards the end *https://www.youtube.com/watch?v=RHDP_KCrjUc*
```
arsulegai (Wed, 13 Mar 2019 15:30:51 GMT):
Video at 19:10 explains safe log commitment scenario
arsulegai (Wed, 13 Mar 2019 15:31:23 GMT):
I don't know if raft-rs considers a entry as "committable_entry" if it's safe as in the video
arsulegai (Wed, 13 Mar 2019 15:37:39 GMT):
From name of the variable and the behavior observed, the committable_entry in Ready status doesn't appear to be a safe committed entry
arsulegai (Wed, 13 Mar 2019 15:39:10 GMT):
@manojgop you're right, blocks added to validator may need to be removed, making space for new blocks coming in. To be in sync with Raft log entries.
arsulegai (Wed, 13 Mar 2019 15:43:49 GMT):
I can think of one example scenario:
1. Leader's messages were delayed to the follower nodes and follower nodes elected a new leader amongst them.
2. The followers with new leader started committing blocks.
3. Old leader syncs back with the network.
Boom! New leader and old leader, both of them have different committed log entries with them. One of them should sacrifice and come to agreement.
arsulegai (Wed, 13 Mar 2019 15:46:30 GMT):
^ @ltseeley thoughts?
manojgop (Wed, 13 Mar 2019 15:56:11 GMT):
@arsulegai New leaders term will be higher and old leader will accept new leader. Old leader then will update it's raft log entry (overwrite old entries if necessary) and also update the chain head (if need be). Raft consensus for block chain (based on implementation) may not be strictly non-forking for some corner cases as explained in the video where some nodes goes down https://www.youtube.com/watch?v=RHDP_KCrjUc*
manojgop (Wed, 13 Mar 2019 15:56:11 GMT):
@arsulegai New leaders term will be higher and old leader will accept new leader. Old leader then will update it's raft log entry (overwrite old entries if necessary) and also update the chain head (if need be). Raft consensus for block chain (based on implementation) may not be strictly non-forking for some corner cases as explained in the video when some nodes goes down https://www.youtube.com/watch?v=RHDP_KCrjUc*
manojgop (Wed, 13 Mar 2019 15:56:11 GMT):
@arsulegai New leaders term will be higher and old leader will accept new leader. Old leader then will update it's raft log entry (overwrite old entries if necessary) and also update the chain head (if need be). Raft consensus for block chain (based on implementation) may not be strictly non-forking for some corner cases as explained in the video when some nodes goes down https://www.youtube.com/watch?v=RHDP_KCrjUc
ltseeley (Wed, 13 Mar 2019 15:58:43 GMT):
That sounds right, @manojgop
arsulegai (Wed, 13 Mar 2019 16:02:45 GMT):
Right!
arsulegai (Wed, 13 Mar 2019 16:04:01 GMT):
Forgetting the case of maximum node failure.. The line at https://github.com/hyperledger/sawtooth-raft/blob/38e52660fe3765f46660954ffb8bc6a332f30a5e/src/block_queue.rs#L96 in sawtooth-raft makes it impossible to apply fork.
arsulegai (Wed, 13 Mar 2019 16:04:52 GMT):
Removing the second part of && condition will solve this forking issue.
arsulegai (Wed, 13 Mar 2019 16:05:16 GMT):
However if follower node is restarted, it'll not be in sync until a new block is published by current leader.
arsulegai (Wed, 13 Mar 2019 16:05:57 GMT):
Completer part of validator has to request the missing previous block id, that then reaches to consensus engine, it then starts committing blocks.
arsulegai (Wed, 13 Mar 2019 16:42:53 GMT):
@manojgop @ltseeley let me go to one more scenario, continuing our earlier discussions
1. Follower nodes is ideally supposed to sync back, when it comes up. Validator's completer part will request for all missing blocks in chain.
2. As per procedure all the block need to pass consensus check before getting committed.
3. If the follower node again goes down before committing the block. There's possibility that log entry is updated with latest leader's information.
4. "commit_queue" & "validator_backlog" in sawtooth-raft's runtime memory is lost.
5. Completer part again gets all missing blocks. But sawtooth-raft will never be able to commit any more blocks.
arsulegai (Wed, 13 Mar 2019 16:42:53 GMT):
@manojgop @ltseeley let me go to one more scenario, continuing our earlier discussions
1. Follower node is ideally supposed to sync back, when it comes up. Validator's completer part will request for all missing blocks in chain.
2. As per procedure all the block need to pass consensus check before getting committed.
3. If the follower node again goes down before committing the block. There's possibility that log entry is updated with latest leader's information.
4. "commit_queue" & "validator_backlog" in sawtooth-raft's runtime memory is lost.
5. Completer part again gets all missing blocks. But sawtooth-raft will never be able to commit any more blocks.
arsulegai (Wed, 13 Mar 2019 16:43:51 GMT):
"commit_queue" will not have next block to be added to Blockchain. It will however gets all other blocks from leader. But validator will not add a block unless it finds the next block as per Blockchain entry.
arsulegai (Wed, 13 Mar 2019 16:44:22 GMT):
So along with if condition, we also need a mechanism and remember if a block has been committed and needs update.
manojgop (Wed, 13 Mar 2019 17:07:12 GMT):
@arsulegai "commit_queue" which is a data structure maintained by sawtooth raft should be in sync with Raft log entries (maintained by raft-rs library). Looks like after a node restarts and "commit_queue" is synced with Raft log entries, Raft consensus need to get blocks which was missing. Will "get_blocks" API retrieve missing blocks from peer validators if its not present locally? @ltseeley may be able to confirm this scenario ?
manojgop (Wed, 13 Mar 2019 17:07:12 GMT):
@arsulegai "commit_queue" which is a data structure maintained by sawtooth raft should be in sync with Raft log entries (maintained by raft-rs library). Looks like after a node restarts and "commit_queue" is synced with Raft log entries, Raft consensus need to get missing blocks . Will "get_blocks" API retrieve missing blocks from peer validators if its not present locally? @ltseeley may be able to confirm this scenario ?
manojgop (Wed, 13 Mar 2019 17:07:12 GMT):
@arsulegai "commit_queue" which is a data structure maintained by sawtooth raft should be in sync with Raft log entries (maintained by raft-rs library). Looks like after a node restarts and "commit_queue" is synced with Raft log entries, Raft consensus need to get missing blocks (if any) and validate it (check _blocks) before it can commit it to the chain . Will "get_blocks" API retrieve missing blocks from peer validators if its not present locally? @ltseeley may be able to confirm this scenario ?
manojgop (Wed, 13 Mar 2019 17:07:12 GMT):
@arsulegai "commit_queue" which is a data structure maintained by sawtooth raft should be in sync with Raft log entries (maintained by raft-rs library). Looks like after a node restarts and "commit_queue" is synced with Raft log entries, Raft consensus need to get missing blocks (if any) and validate it before it can commit it to the chain . Will "get_blocks" API retrieve missing blocks from peer validators if its not present locally? @ltseeley may be able to confirm this scenario ?
arsulegai (Wed, 13 Mar 2019 17:46:44 GMT):
My scenario was "commit_queue" lost because of one more restart before the block is committed. Upon next power-on, the raft doesn't consider the block_id again in Ready status.
manojgop (Thu, 14 Mar 2019 04:04:53 GMT):
@arsulegai This looks to be the case where raft log entries are committed (and persisted to disk), but before applying these entries to state machine (in sawtooth raft case it is committing the blocks to blockchain), the node restarted. So after restart, we need to consider the entries in raft log which are not yet applied to state machine. Looks like this is not implemented in sawtooth raft. Checkout the function "get_applied" in https://docs.rs/raft/0.5.0/raft/struct.RaftLog.html#method.get_applied. @ltseeley Please chime in.
ltseeley (Thu, 14 Mar 2019 15:15:02 GMT):
Sorry guys, it's been quite a while since I've looked at Raft. I'm not sure I have a whole lot to add to this discussion; I think you should be able to compare the committed entries with the blocks that are committed to the chain and implement some logic to get the blockchain to match up with Raft's committed entries.
j0g1 (Fri, 15 Mar 2019 16:53:32 GMT):
Has joined the channel.
j0g1 (Fri, 15 Mar 2019 16:55:19 GMT):
Hi All. Is there any url when I can get deep introduction to PoET? I would like to know the whole workflow to establish consensus. Documentation provide general overview information wthout real example
danintel (Fri, 15 Mar 2019 23:34:39 GMT):
https://sawtooth.hyperledger.org/docs/core/releases/latest/architecture/poet.html is the PoET spec (1.0)
https://github.com/hyperledger/sawtooth-rfcs/pull/20/files is the PoET 2.0 RFC, with much good background info
https://drive.google.com/file/d/1IvwMExtAkrCTyO29X6_qaqtpAS1b1wpu/view is a PoET2 video presentation
https://drive.google.com/file/d/0B_NJV6eJXAA1TGdfMjJlT0Qtb0U/view is an older PoET 1 video presentation
j0g1 (Sat, 16 Mar 2019 19:34:05 GMT):
@danintel first of course I know but is very general. Today I will investigate others links . Thank You!
MelodyMcIntyre (Thu, 21 Mar 2019 05:59:17 GMT):
Hi, I run into error SSL PEM while config poet-sgx. I have the SPID from Intel. IS that possible because of ias_url? what is the url I should use? the one in the example file "https://test-as.sgx.trustedservices.intel.com:443" does not work for me
MicBowman (Thu, 21 Mar 2019 14:12:22 GMT):
@MelodyMcIntyre i have not looked at the poet sgx code in awhile... i do know that IAS EOL'd version2
MicBowman (Thu, 21 Mar 2019 14:13:35 GMT):
we've streamlined the certificate verification quite a bit in the private data objects lab... we've been having some discussions with the india team about how to get those incorporated into sawtooth
Nonj (Thu, 21 Mar 2019 17:50:09 GMT):
Has joined the channel.
Nonj (Thu, 21 Mar 2019 17:50:55 GMT):
Hi folks, I'm currently on the sawtooth/NEXT project and we are looking to upgrade our sawtooth_sdk to version 1.1.4 from 1.0.5. The issue I'm currently running into is that I see that the protobuf package in 1.1.4 got replaced by a package called "consensus" in 1.0.5. Can anyone confirm that? Or am I missing something crucial?
kc1116 (Fri, 22 Mar 2019 00:44:38 GMT):
is pbft or raft ready for primetime ?
amundson (Fri, 22 Mar 2019 03:44:01 GMT):
@kc1116 pbft is ready. it's been well-tested at this point (both long-running testing under heavy load and many unit tests). it will be 1.0 soon.
MelodyMcIntyre (Fri, 22 Mar 2019 05:03:43 GMT):
@MicBowman Thank you so much for you reply. I have been trying to fix the problem but I did not find the solution yet. I am still using sawtooth version 1.0.5 since the latest version does not support POET-SGX yet.
st (Fri, 22 Mar 2019 08:39:18 GMT):
Has left the channel.
danintel (Fri, 22 Mar 2019 21:08:07 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=E37wsis4rXbCDygkL) @Nonj No, but close. Protobuf is still needed and is not replaced for 1.1.
With Sawtooth 1.1, the consensus engine (DevMode, PoET, or whatever) is a separate process and separate package. You need to install the package(s) containing your favorite consensus engine.
You also need to set `sawtooth.consensus.algorithm.name` and `sawtooth.consensus.algorithm.version` in 1.1 (which replaces `sawtooth.consensus.algorithm` in 1.0)
mayankmk14 (Mon, 25 Mar 2019 11:15:06 GMT):
Has joined the channel.
mjalalzai (Mon, 25 Mar 2019 13:48:48 GMT):
Has joined the channel.
amolk (Mon, 25 Mar 2019 15:21:55 GMT):
@amundson @jsmitchell would like to get your thoughts on one particular issue with Raft - not sure if you've encountered it with PBFT. In Raft, a follower node will forward transactions to the leader, who will execute them. The transactions will be validated, assembled in a block that will be sent to the followers. The followers will now start validating transactions, pulling them from their queue.
In case the leader encounters an error while validating a transaction, that transaction is discarded and never appears in a block. These transactions will remain in the follower nodes' queues with pending status. The only time these transactions will be cleared from the queue is in case a follower becomes a leader.
In the past, we've discussed possible ways to clear the queue by periodically trying the validate transactions on the followers so they discard invalid transactions from the queue.
However, while validating these pending transactions, the state on the followers may be different from the state on the leader when it rejected the transactions. Transactions rejected by the validator my succeed on the follower. Therefore, for consensus mechanisms that have a semi-permanent leader, it would be ideal to have a mechanism to convey the status of invalid transactions to followers. The block is essentially a list of successful transactions, perhaps we need a similar mechanism to list and propagate rejected transactions from the leader.
Thoughts?
amundson (Mon, 25 Mar 2019 16:19:34 GMT):
Sending a list of invalid transactions isn't that useful. Each node would have to independently verify that those transactions were indeed invalid for BFT reasons. So the list of invalid transactions from the leader would essentially be a proposed subset of the pending queue that needs additional consideration. It would be better and just as efficient (normally) to just process the entire pending queue to detect invalid transactions.
amundson (Mon, 25 Mar 2019 16:21:13 GMT):
Also, if we attached this information to the block message (not necessarily the block it self, but the message that has the block), then it could potentially get very large in the negative case and we have to consider DoS carefully. If it was handled in other messages, that increases the network load and would have to have a very careful to avoid DoS on the network due to invalid submissions.
jsmitchell (Mon, 25 Mar 2019 16:41:19 GMT):
Some kind of block-horizon based cleanup of the pending queue independent of the publishing process is worth exploring.
amundson (Mon, 25 Mar 2019 16:52:56 GMT):
Sure, you wouldn't need to detect invalid transactions every block. It could be relatively infrequent.
amundson (Mon, 25 Mar 2019 16:54:33 GMT):
You could catch most of them as they come in though, before putting them in the pending queue. Then leave the cleanup for those that changed from valid to invalid over time.
MelodyMcIntyre (Mon, 25 Mar 2019 22:21:09 GMT):
From the log, the sawtooth poet sgx is still using Intel® SGX API (IAS API) version2 but version2 is end of life. does anyone know how to fix it ?
MelodyMcIntyre (Mon, 25 Mar 2019 22:21:48 GMT):
is current poet-sgx is not available I am working on 1.0.5 version since i knew 1.1 version does not work definitely
jsmitchell (Mon, 25 Mar 2019 22:28:39 GMT):
@Danintel @Dan ^
danintel (Mon, 25 Mar 2019 22:35:37 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=Yx3RNaRhMjRYXRWA2) @MelodyMcIntyre I'm not a SGX expert, but I'll try. Various SGX SDK versions are available here on Intel's Open Source website, https://01.org/intel-software-guard-extensions/downloads
Also is there a reason you cannot use a newer version of the SGX API? Does a Docker or other configuration file tie you to an old version?
danintel (Mon, 25 Mar 2019 22:35:37 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=Yx3RNaRhMjRYXRWA2) @MelodyMcIntyre I'm not a SGX expert, but I'll try. Various SGX SDK versions are available here on Intel's Open Source website,
https://01.org/intel-software-guard-extensions/downloads
Also is there a reason you cannot use a newer version of the SGX API? Does a Docker or other configuration file tie you to an old version?
MicBowman (Mon, 25 Mar 2019 22:53:22 GMT):
@MelodyMcIntyre there is a relatively simple change to the URL, however since sawtooth hard codes the IAS verification keys... it might not work to just update the URL... suggest you contact bvavala in the #private-data-objects channel
MicBowman (Mon, 25 Mar 2019 22:53:47 GMT):
he's been doing a lot of the leg work for us on making IAS verification more complete
amiec (Sat, 30 Mar 2019 15:46:08 GMT):
Has joined the channel.
GianlucaSchoefer (Sun, 31 Mar 2019 00:57:16 GMT):
Has joined the channel.
GianlucaSchoefer (Sun, 31 Mar 2019 01:00:51 GMT):
Hello, I'm currently trying to get a better mental model of Sawtooth's validation process. Do I assume correctly, that in Sawtooth one validator creates a block and distributes it to the rest of the network and every single receiving validator replays the transactions with its own TPs and as a result should receive the same state results/radix tree adresses? If this is correct, everything that is non-deterministic should not be allowed to be part of a TP, e.g. http, timestamps, random, etc.?
jsmitchell (Sun, 31 Mar 2019 02:53:48 GMT):
Correct
GianlucaSchoefer (Sun, 31 Mar 2019 04:36:40 GMT):
Thank you :)
GiorgosT (Sun, 31 Mar 2019 14:55:22 GMT):
Has joined the channel.
amundson (Mon, 01 Apr 2019 01:42:00 GMT):
nice summary @GianlucaSchoefer
kumble (Tue, 02 Apr 2019 01:17:27 GMT):
Has joined the channel.
kumble (Tue, 02 Apr 2019 01:19:06 GMT):
*Excuse me if the Question is dumb:* Does Sawtooth currently support a _PBFT hardened RAFT_ consensus ?
jsmitchell (Tue, 02 Apr 2019 01:37:23 GMT):
https://github.com/hyperledger/sawtooth-pbft
jsmitchell (Tue, 02 Apr 2019 01:38:06 GMT):
Raft and PBFT are two separate things, but yes, it supports PBFT
kumble (Tue, 02 Apr 2019 01:39:50 GMT):
Hi, thanks for the revert⁄
kumble (Tue, 02 Apr 2019 01:39:50 GMT):
Hi, thanks for the revert!
kumble (Tue, 02 Apr 2019 01:41:24 GMT):
BTW, there are version of PBFT hardened RAFT that was authored some time ago called Tanagroa: www.scs.stanford.edu/14au-cs244b/labs/projects/copeland_zhong.pdf
kumble (Tue, 02 Apr 2019 01:41:24 GMT):
BTW, there are version of PBFT hardened RAFT that was authored some time ago called Tangaroa: www.scs.stanford.edu/14au-cs244b/labs/projects/copeland_zhong.pdf
kumble (Tue, 02 Apr 2019 01:41:33 GMT):
Have you come across this?
jsmitchell (Tue, 02 Apr 2019 01:47:54 GMT):
I do remember seeing that a while ago, yes. The PBFT implementation we’ve written is pretty much straight from the Castro paper.
jsmitchell (Tue, 02 Apr 2019 01:49:07 GMT):
There are four sawtooth PBFT RFCs here if you want more details: https://github.com/hyperledger/sawtooth-rfcs/tree/master/text
Mohit_Python (Tue, 02 Apr 2019 04:37:35 GMT):
Has joined the channel.
kumble (Tue, 02 Apr 2019 05:48:02 GMT):
Great, I will have a look.
Mohit_Python (Tue, 02 Apr 2019 06:46:42 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=vKeaLsYWsHCyY7zTW) @danintel I have one question here .. What consensus algorithm is being used in 1.0.5 Version?
amiec (Wed, 03 Apr 2019 01:49:09 GMT):
Hi all,
I’m conducting some performance and scalability research with PoET and was hoping this community could help clarify some of the detailed workings for me. More specifically, I have been using Hyperledger Caliper to deliver workloads to small Sawtooth simulated networks of 1, 2, 4, 8, 12 and 16 nodes spun up with Docker containers. Some of my experimental results report a transaction throughput of 0 tps despite a non-zero delivery rate and an experimental duration of at least a minute. This has led me to question my choice of various network parameters, and to wonder about what is happening in a more detailed way with PoET that I perhaps don’t understand. My strange throughput results happened with a choice of (target_wait_time=5, initial_wait_time=25) AND (target_wait_time=5, initial_wait_time=5). Here are some specific questions I have, and it would be SO RAD if I could get some of your brain power. (I know the code itself can be read, but I really need some human help) :
1. In PoET, peers sample an exponentially distributed random variable to achieve a wait time. Take, for example, a target_wait_time of 5 seconds. If I wanted to choose wait times from an exp distribution that averaged ~5s, I would sample from a distribution where lambda=1/5, because the mean of an exp distribution is 1/lambda, and 1/(1/5) = 5, my desired average. Is that what is happening in PoET, or are there nuances to the form of the exponential distribution used for sampling? Additionally, how does the estimated population_size affect the sampling distribution? In a network of size X, does lambda decrease by exactly a factor of X? I want to know this for mental reasoning about the likelihood of anomalous block intervals using CDFs and such.
2. When creating a PoET network, one can specify the target_wait_time and the initial_wait_time. I understand that the initial_wait_time is used to help determine the localMean before the network size can be adequately determined (not enough data). In theory this is something that the administrator of the network can choose based on his/her knowledge of the size of the initial network? What are potential issues that could arise if an inappropriate value of initial_wait_time was chosen for the network? Too many collisions? Too much time between blocks being published?
3. initial_wait_time factors into the localMean calculation until sampleLength (<—referring to the PoETdocumentation) blocks have been published on the chain. In the code this variable is called population_estimate_sample_size or something? And I know it defaults to 50. Where does that number 50 come from and can it be changed by a network administrator without altering source code?
4. At a higher, mechanistic/conceptual level, how does PoET determine an estimate of the network size from block publishing data?
5. When a node is experiencing backpressure (?.. a full queue) does anything about its performance change? Does it engage in any other behavior? I ask because I expect to deliver workloads that are too fast for the network to handle, and my assumption is that the validator simply keeps processing transactions from the queue and if the queue is full, simply drops anything incoming.
I should stop here, but finally - does anyone have suggestions for parameter values I should choose to have a well-functioning network? Starting with small 1, 2, 4, 8, 12, 16 - node networks (I know PoET shouldn’t be used in production with less than about 10 nodes, but this is performance experimental). Parameters:
target_wait_time
initial_wait_time
batches_per_block (my workload currently delivers intkey transactions at 20 txns/batch)
min/max peer connectivity
and any other settings that might relevant but I’m not using?
Maybe it’s just the case that I need a much longer warmup or “bootstrapping” phase before measuring experimental data.
Thank you so much! Please enlighten me.
jsmitchell (Wed, 03 Apr 2019 13:57:55 GMT):
@amiec these are best answered by a poet expert like @Dan or @MicBowman
Kirill_Vusik (Wed, 03 Apr 2019 15:23:24 GMT):
Has joined the channel.
amiec (Wed, 03 Apr 2019 18:44:01 GMT):
Thanks @jsmitchell ! Am I asking this in the right place? Or should I redirect?
amiec (Wed, 03 Apr 2019 18:44:01 GMT):
Thanks @jsmitchell ! Am I asking this in the right place? Or should I redirect?
Dan (Wed, 03 Apr 2019 18:50:13 GMT):
This is the right place. PoET takes some time to ramp using the parameters that you identified above. They are really not that important though. What you want to do is get the network through those first 50 blocks or so. Then you can do a meaningful performance test on a "warm" network. You probably want multiple blocks to establish performance numbers. I'm picking a number out of the air but at least 100 depending on the kind of experiment you are running.
Dan (Wed, 03 Apr 2019 18:51:13 GMT):
During backpressure the node will reject new transactions submitted to it by clients. That should provide the behavior you want during your performance testing. Basically it ignores what it thinks it can't handle.
amiec (Wed, 03 Apr 2019 18:52:13 GMT):
One more for you: If a validator's queue is full, what happens to an incoming transaction? Does the validator keep track of it in any way? Report a timeout? Or does it simply drop it into the abyss and ignore it completely? I ask because I think that dropped transactions may be confusing Caliper, which may be waiting for txn timeout information (to declare failure) that never manifests?
Dan (Wed, 03 Apr 2019 18:53:21 GMT):
I must have some recommended values around somewhere, but off the top of my head ...
target wait time: 30 s
initial wait time: Target_wait_time * population
I don't mess with batches per block or min/max peers but you are welcome to. I would suggest experimenting with batches per block.
Dan (Wed, 03 Apr 2019 18:54:59 GMT):
I don't recall exactly the behavior for how the client's transactions are responded to. I think it may give an HTTP busy error over the REST API. @pschwarz would know more.
amiec (Wed, 03 Apr 2019 19:01:05 GMT):
Thank you so much for the responses! That target wait time is much longer than I would have guessed, is this just to prevent the likelihood of collisions/stale blocks? I will experiment with batches/block - I think the main knob that this controls is the theoretical maximum throughput, which would be (txns/batch)*(batches/block)*(blocks/second) but please correct me if I'm missing an aspect of that (perhaps latency increases due to fat blocks if it gets too high?). Finally, if anyone has notes or answers to questions 1, 2, 3 and 4 above, I'd be very curious. Thank you!
pschwarz (Wed, 03 Apr 2019 19:16:39 GMT):
Yes, you will get a "queue full" error, as an HTTP 429 error status
Dan (Wed, 03 Apr 2019 19:42:07 GMT):
@amiec yes, too small of a target wait time will cause too many forks. i.e. lots of nodes will claim to be the leader and then all have to resolve which candidate block is really the new head of the chain. 30 seconds is good for a reasonably distributed group of about 10 validators. If you have a smaller number of collocated validators you might be able to tune that down.
j0g1 (Thu, 04 Apr 2019 14:00:08 GMT):
Suppose I would like to implement my consensus algorithm. According to the documentation, this is relatively simple because the consensus module is separated from the validator. Is there any tutorial on how to add theconsensus algorithm engine to Sawtooth ?
Dan (Thu, 04 Apr 2019 14:39:18 GMT):
Sorry no tutorials on that particular topic. :(
The best I can suggest is to look at the existing consensus implementations for examples.
j0g1 (Thu, 04 Apr 2019 14:42:49 GMT):
Ok :)
j0g1 (Thu, 04 Apr 2019 14:43:18 GMT):
If I win with implementation I will try create such tutorial for everybody :)
Dan (Thu, 04 Apr 2019 14:43:31 GMT):
that would be fantastic!
j0g1 (Thu, 04 Apr 2019 14:44:39 GMT):
consensus code is in one place in repo or use modules from different places(sorry for dumb question but I don't look at code yet)
Dan (Thu, 04 Apr 2019 14:48:54 GMT):
here's a few consensus implementations that use the sawtooth interface. the first (pbft) is probably the cleanest example:
https://github.com/hyperledger/sawtooth-pbft
https://github.com/hyperledger/sawtooth-raft
https://github.com/hyperledger/sawtooth-poet
Dan (Thu, 04 Apr 2019 14:49:16 GMT):
All of the validator APIs are expressed as protobufs and can be found in the core repo here:
https://github.com/hyperledger/sawtooth-core/tree/master/protos
Dan (Thu, 04 Apr 2019 14:49:24 GMT):
including the consensus api.
Dan (Thu, 04 Apr 2019 14:51:53 GMT):
And here is the RFC that introduced the idea of separating consensus as it's own process. Note the actual implementation of the interfaces will have evolved since this was written but the ideas should still be relevant.
https://github.com/hyperledger/sawtooth-rfcs/blob/master/text/0004-consensus-api.md
j0g1 (Thu, 04 Apr 2019 14:56:31 GMT):
Great! Thank you so much for your support :) I hope to present my consensus proposal soon and tutorial for that also! :)
ltseeley (Thu, 04 Apr 2019 15:07:33 GMT):
For more introductory-level information, you may find these blog posts helpful as well:
https://www.hyperledger.org/blog/2018/12/11/making-dynamic-consensus-more-dynamic
https://www.hyperledger.org/blog/2019/01/11/floating-the-sawtooth-raft-implementing-a-consensus-algorithm-in-rust
ltseeley (Thu, 04 Apr 2019 15:08:38 GMT):
Please reach out if you have any questions!
j0g1 (Thu, 04 Apr 2019 16:57:55 GMT):
It looks like very helpful for me! Thank you!
danintel (Thu, 04 Apr 2019 17:43:48 GMT):
@Mohit_Python Sawtooth 1.0.5 supports PoET SGX, PoET CFT, and DevMode. PoET works with SGX for BFT fault tolerance and without SGX for CFT fault tolerance. DevMode is for development and testing only and is the default algorithm. Sawtooth also supports PBFT and Raft in later versions of Sawtooth
danintel (Thu, 04 Apr 2019 17:48:25 GMT):
@amiec I'm not sure your question was answered, but if the validator's queue is ful and the transaction is rejected, with a HTTP 429 status code, as mentioned above, the transaction is not saved and needs to be resubmitted by the client.
YashVadhvani (Fri, 05 Apr 2019 07:02:04 GMT):
Has joined the channel.
YashVadhvani (Fri, 05 Apr 2019 07:05:01 GMT):
I want to start the poet consensus engine on 1.1 version on ubuntu environment what packages i need to install and how do i do it?
YashVadhvani (Fri, 05 Apr 2019 07:05:53 GMT):
In documentation there is devmode consensus available but what i need to start for poet engine please help
danintel (Fri, 05 Apr 2019 17:45:48 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=MGDKoGoFC2esgd2x9) @YashVadhvani See https://sawtooth.hyperledger.org/docs/core/releases/latest/app_developers_guide/creating_sawtooth_network.html?highlight=poet%20network
For further support questions, please ask on #sawtooth .
rbuysse (Fri, 05 Apr 2019 20:54:16 GMT):
the dynamic network test is failing in an interesting way on my bionic upgrade PR.
rbuysse (Fri, 05 Apr 2019 20:54:19 GMT):
https://build.sawtooth.me/job/Sawtooth-Hyperledger/job/sawtooth-poet/job/PR-15/43/console
rbuysse (Fri, 05 Apr 2019 20:54:30 GMT):
`State from block fab476a4b68692d2736312deff7582fcd32bb192b878187a8347a500e208ebaf24ce278f25ad7c7e257fbb27332e3457437d6dc06d16913709c3b1db209755bd requested, but root hash 466db400c9d0d678ebda8cacab13702104765622e9a6571fc5b286b8d71d6f66 was missing. Returning empty state.`
rbuysse (Fri, 05 Apr 2019 20:54:53 GMT):
it looks like it's committing the blocks but then for some reason it can't read back the state
arsulegai (Mon, 08 Apr 2019 14:59:31 GMT):
Does it mean a block is committed without a prior block in place?
arsulegai (Mon, 08 Apr 2019 16:32:20 GMT):
This is for Block 2, the same error is seen for Block 1. Looks like state root hash of Block 1 is requested even before Block 1 is committed on that node.
arsulegai (Mon, 08 Apr 2019 16:32:24 GMT):
12:48:12 [36mtest-dynamic-network_1 |[0m [31m[2019-04-04 17:48:11.842 ERROR proxy][0m [37mState from block ae529eded3f72d6afea90f166386e8fb6445bf11b58c7cd9c3dbb4d0436896512c505838e431a5c2016335bedfb8a839e4f98bea838d608d1960bf98e96a00ef requested, but root hash 272321cfd24929e752e3b1183c8cb741675cab7149db8eac0e886f3abcab6f6c was missing. Returning empty state.[0m
arsulegai (Mon, 08 Apr 2019 16:33:32 GMT):
Causing the global state root hash to be not found
pschwarz (Mon, 08 Apr 2019 22:30:56 GMT):
This is against master, correct?
pschwarz (Mon, 08 Apr 2019 22:31:04 GMT):
Yeah, I'm working on a fix for that, as I type
raghav.aneja14 (Fri, 12 Apr 2019 09:52:55 GMT):
Has joined the channel.
MelodyWang (Tue, 16 Apr 2019 00:19:08 GMT):
Has joined the channel.
manojgop (Wed, 17 Apr 2019 18:07:43 GMT):
@pschwarz Is this fix intended to pass all the POET1 integration tests ?
pschwarz (Wed, 17 Apr 2019 18:20:45 GMT):
Yes
amiec (Thu, 18 Apr 2019 03:35:12 GMT):
Does anybody know if a validator has more than one type of queue? For example, separate queues for incoming transactions v.s. incoming blocks? Or is it the same queue for all traffic? Thank you.
pschwarz (Thu, 18 Apr 2019 19:44:44 GMT):
For transactions (in the form of batches), they are put into a pending transaction queue. They are put there when received by being directly submitted to the validator, or gossiped by the network (i.e. submitted to another node). Blocks go into a separate queue for validation.
pschwarz (Thu, 18 Apr 2019 19:45:52 GMT):
Batches are removed from the pending queue when they are included in a valid, committed block, regardless of whether the current node published it or another node did.
pschwarz (Thu, 18 Apr 2019 19:46:25 GMT):
Note, none of this really has to do with consensus - these are all actions that occur with any consensus engine that Sawtooth supports
amundson (Fri, 19 Apr 2019 20:12:36 GMT):
@Dan what poet 1 defects are you referencing in the meeting topics?
Dan (Fri, 19 Apr 2019 20:49:26 GMT):
I was told poet1 master was broken. It might be that @pschwarz has a fix. I was also trying to ferret out whether one of Amol's team was working on the same or different problem. I connected with @amolk yesterday (he mentioned a conflict with Monday's meeting). He said that Raghav is monitoring for the PR and will double check if it resolves the issue or whether there's additional defects.
pschwarz (Fri, 19 Apr 2019 21:07:30 GMT):
It's not poet master that's broken, but it's interaction with Sawtooth Core master that was the issue
pschwarz (Fri, 19 Apr 2019 21:07:36 GMT):
That is fixed now
Dan (Fri, 19 Apr 2019 23:32:50 GMT):
Muchas gracias señor
pschwarz (Mon, 22 Apr 2019 15:33:40 GMT):
@Dan for some more context on what has been fixed in core to address the issues with PoET and dynamic network tests:
https://github.com/hyperledger/sawtooth-core/pull/2067
https://github.com/hyperledger/sawtooth-core/pull/2076
Dan (Mon, 22 Apr 2019 15:36:41 GMT):
thanks
@rbuysse I'm not clear what was failing before.. kinda late to this show.. but when i look at build.sawtooth.me I see build successes for a long time on sawtooth-poet. The most recent failure is back on the 13th.
rbuysse (Mon, 22 Apr 2019 15:38:37 GMT):
PR-15 is failing
rbuysse (Mon, 22 Apr 2019 15:40:04 GMT):
that's the bionic upgrade. the current master job runs tests against sawtooth-core bumper. PR-15 tests run against master
Dan (Mon, 22 Apr 2019 15:50:15 GMT):
are all the components following that pattern then, the build depends on sawtooth-core master as opposed to sawtooth-core latest release?
rbuysse (Mon, 22 Apr 2019 16:06:34 GMT):
I think so
rbuysse (Mon, 22 Apr 2019 16:06:58 GMT):
this is a aort of weird edge case because we did a release at the same time as the bionic upgrade
rbuysse (Mon, 22 Apr 2019 16:07:12 GMT):
so the only artifacts available for bionic are pre-release
Dan (Mon, 22 Apr 2019 16:09:39 GMT):
I can argue that it's a good approach overall (I can argue the opposite too ;) ) that core master should never break and so components can freely depend on it without worrying about a moving target.
rbuysse (Mon, 22 Apr 2019 16:13:07 GMT):
I think we're gonna end up in a place where we're testing lastest release and master
rbuysse (Mon, 22 Apr 2019 16:15:00 GMT):
^ that'd be ideal. but kinda is bad for PRs
raghav.aneja14 (Wed, 24 Apr 2019 11:30:11 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=YDn6jZgQ6X6MmRhQC) @pschwarz @Dan @pschwarz I have ran the integration test on poet master branch, and the result status is OK against dynamic-network-test.
MicBowman (Wed, 24 Apr 2019 21:46:50 GMT):
is anyone testing hardware mode PoET?
amundson (Wed, 24 Apr 2019 22:44:06 GMT):
@MicBowman If someone has recently, they haven't shared the results. The last PoET-SGX testing that I know about was against Sawtooth 1.0 (prior to the consensus engine design). If you are using the consensus engine, I'd expect an incremental amount of work would be necessary.
MicBowman (Wed, 24 Apr 2019 22:44:32 GMT):
if no one has updated it, it won't work
MicBowman (Wed, 24 Apr 2019 22:44:50 GMT):
the v2 IAS api that you have backed in is no longer supported
amundson (Wed, 24 Apr 2019 22:44:57 GMT):
Right, also not updated against any modern SDK, etc.
MicBowman (Wed, 24 Apr 2019 22:44:58 GMT):
at a minimum the IAS url has to change
MicBowman (Wed, 24 Apr 2019 22:45:53 GMT):
so... even if it worked when v1 was released, v1 almost certainly does not work today
amundson (Wed, 24 Apr 2019 22:46:49 GMT):
I agree, and don't know of any recent commits against the 1-0 branch at all
MicBowman (Wed, 24 Apr 2019 22:47:06 GMT):
i couldn't find any patches to address that so i'm assuming that it doesn't work... we've provided some hints about how to fix it to a couple people but have not seen/produced any PRs
amundson (Wed, 24 Apr 2019 22:48:29 GMT):
1.0 is getting old enough that doing a point release would probably be a substantial amount of work.
MicBowman (Wed, 24 Apr 2019 22:49:04 GMT):
understood... could you provide an updated release note or something?
amundson (Wed, 24 Apr 2019 22:49:41 GMT):
You mean provide some info on how to get it working by pulling it out of master?
amundson (Wed, 24 Apr 2019 22:49:41 GMT):
You mean provide some info on how to get it working by pulling it out of 1-0 branch?
MicBowman (Wed, 24 Apr 2019 22:50:05 GMT):
or just something that says that you shouldn't expect it to work without modifications
amundson (Wed, 24 Apr 2019 22:52:35 GMT):
there are three possible places - it could go into the readme on the 1-0 branch, in the sawtooth FAQ (which is in the sawtooth-website repo), or in the docs (probably https://sawtooth.hyperledger.org/docs/core/releases/1.0.5/sysadmin_guide/configure_sgx.html)
amundson (Wed, 24 Apr 2019 22:52:52 GMT):
the first two are easy, just submit a PR
amundson (Wed, 24 Apr 2019 22:53:23 GMT):
updating release documentation is harder, we don't really have a super great process for it yet. but if it was important, we could figure it out.
MicBowman (Wed, 24 Apr 2019 22:54:46 GMT):
i would guess that the FAQ is probably the right place
MicBowman (Wed, 24 Apr 2019 22:54:53 GMT):
is there a PoET section there?
amundson (Wed, 24 Apr 2019 22:55:18 GMT):
https://sawtooth.hyperledger.org/faq/consensus/
MicBowman (Wed, 24 Apr 2019 22:56:11 GMT):
hmm... many of the questions are kind of no longer relevant
amundson (Wed, 24 Apr 2019 22:56:40 GMT):
@danintel ^
danintel (Wed, 24 Apr 2019 22:58:27 GMT):
@MicBowman You can file a PR or just let me know what's irrelevant
amundson (Wed, 24 Apr 2019 22:58:36 GMT):
FWIW, we still do test non-SGX PoET (and PBFT now), so if the incremental work was done against the consensus engine, presumably it would work again.
amundson (Wed, 24 Apr 2019 22:58:36 GMT):
FWIW, we still do test non-SGX PoET (and PBFT now) testing, so if the incremental work was done against the consensus engine, presumably it would work again.
MicBowman (Wed, 24 Apr 2019 22:58:37 GMT):
any question that mentions sgx?
MicBowman (Wed, 24 Apr 2019 22:58:55 GMT):
@danintel
MicBowman (Wed, 24 Apr 2019 22:59:26 GMT):
the sawtooth poet sgx code contains a hard coded url in the ias client for the v2 ias client which intel no longer supports
MicBowman (Wed, 24 Apr 2019 22:59:35 GMT):
it *MIGHT* be that just needs to be updated
MicBowman (Wed, 24 Apr 2019 22:59:44 GMT):
however, you also hard code the verification key
MicBowman (Wed, 24 Apr 2019 22:59:56 GMT):
i don't know if that still works or if it was v2 dependent
MicBowman (Wed, 24 Apr 2019 23:00:28 GMT):
so... until those are fixed (which i don't believe are on the roadmap), SGX is not supported in any version of sawtooth
amundson (Wed, 24 Apr 2019 23:03:04 GMT):
for the consensus engine, an additional incremental amount of work would need to be done on the packaging. that and the overhead of running LR7 testing is why it was dropped as a Sawtooth 1.1 feature. It was not for a deep architectural reason.
amundson (Wed, 24 Apr 2019 23:04:31 GMT):
(but, I guess if it was fixed, it would retroactively become a Sawtooth 1.1 feature)
MicBowman (Wed, 24 Apr 2019 23:04:55 GMT):
i understand the reasons (which really boils down to there is not a lot of demand for a scalable, byzantine reslient consensus without finality for enterprise apps)
MicBowman (Wed, 24 Apr 2019 23:05:25 GMT):
my point is more that if there is no commitment to fix it, then we should drop it from the documentation & note that it is not currently supported
MicBowman (Wed, 24 Apr 2019 23:06:14 GMT):
(and i'm not complaining about the lack of commitment to fix it... work on poet2 should be the focus)
amundson (Wed, 24 Apr 2019 23:06:31 GMT):
that wasn't the reasoning at the time, AFAIK. it was much more the attitude that PoET 2 was the place to have those engineers working.
amundson (Wed, 24 Apr 2019 23:08:01 GMT):
now, that may be the reason some of us spend so much more time on PBFT, because we see more relevance for our specific use cases for non-scalable, byzantine resilient consensus with finality :)
MicBowman (Wed, 24 Apr 2019 23:08:13 GMT):
pardon me if i'm skeptical that poet2 will see the light of day... its been over a year since we provided the basic protocol
MicBowman (Wed, 24 Apr 2019 23:08:38 GMT):
i get that and support it
MicBowman (Wed, 24 Apr 2019 23:09:12 GMT):
and frankly the reason why we're looking at other ledgers is because we don't need all the bells & whistles of sawtooth & don't want to pay the complexity costs
amundson (Wed, 24 Apr 2019 23:10:12 GMT):
you should call out the complexity, some of it is going to evaporate
MicBowman (Wed, 24 Apr 2019 23:10:39 GMT):
if i were to put in a PR for the documentation, it would basically be: replace all current references to SGX Poet with one question which amounts to SGX PoET doesn't work, wait for v2
amundson (Wed, 24 Apr 2019 23:13:50 GMT):
if we see our current plans through, we will get down to a single process per-node and have greatly simplified smart contract APIs (we are about to attack the problem of writing any addressing code for the default uses), and hopefully state delta export will be provided "for free" without any coding soonish too.
MicBowman (Wed, 24 Apr 2019 23:15:54 GMT):
understood... i'm just looking for a fixed function ordering service (a little more than that, but i don't need full programability)
MicBowman (Wed, 24 Apr 2019 23:16:31 GMT):
the confidentiality is more important than programmability
MicBowman (Wed, 24 Apr 2019 23:17:56 GMT):
and... back to PoET... for the moment, if you get questions about HW support for PoET, feel free to point them to me... I can at least explain why it isn't working
MicBowman (Wed, 24 Apr 2019 23:18:21 GMT):
we've been answering some of those question on the pdo channel when they've come up
kodonnel (Fri, 26 Apr 2019 18:35:16 GMT):
Is anyone working on a Java based consensus? Asking in order to judge interest around furthering the sawtooth-java-sdk to cover the consensus API.
amundson (Mon, 29 Apr 2019 04:49:44 GMT):
@kodonnel I don't think we should pro-actively support consensus APIs in the non-Rust SDKs unless there is a compelling consensus implementation in that language. The overhead in updating it as we change that API will be completely non-trivial. I'd even like to drop the Python support as soon as PoET is Rust (unless there is another Python consensus engine contributed). The Rust consensus engines will likely be compiled into the validator in the future (optionally), and I'm not even sure we should keep the network separation at all going into Sawtooth 2.0.
amundson (Mon, 29 Apr 2019 04:52:41 GMT):
There was discussion of supporting bft-smart at one point (which is a Java library), but I think that has dwindled.
amundson (Mon, 29 Apr 2019 04:56:43 GMT):
There is also this Go implementation of a SGX PBFT variant in labs - https://github.com/hyperledger-labs/minbft
kodonnel (Mon, 29 Apr 2019 14:51:35 GMT):
Well, I'd be fine with all of that except for removing the network separation. That would sacrifice an awful lot of customizability.
sergefdrv (Thu, 02 May 2019 07:43:46 GMT):
@kodonnel What do you mean by "removing the network separation"?
amundson (Thu, 02 May 2019 14:43:17 GMT):
@sergefdrv he is referring to what I said about potentially removing the network separation between the validator and the consensus engine in the future if all the consensus engines end up being rust
amundson (Thu, 02 May 2019 14:51:37 GMT):
I think it's easy to keep the network separation if it remains desirable.
sergefdrv (Thu, 02 May 2019 15:53:53 GMT):
@amundson thanks for answering, now I got it
bobonana (Thu, 02 May 2019 22:16:36 GMT):
Has joined the channel.
bobonana (Thu, 02 May 2019 22:36:05 GMT):
Hey there! Over at sawtooth-next-directory we recently upgraded to sawtooth v1.1 from v1.0, and while running some performance tests we noticed that 500 transactions took around 1.3 minutes to be processed using v1.0, but with the switch to v1.1 those same 500 transactions now take around 9.3 minutes to be processed. Is this performance hit expected with the update? We're using DevMode for consensus and runnign a single node deployment. Thanks!
Dan (Fri, 03 May 2019 12:53:42 GMT):
I can't think of any changes that would cause this. Devmode will fork like mad though, so maybe if other things are happening faster devmode is kicking out more blocks than before. how many nodes are you testing with?
bobonana (Fri, 03 May 2019 17:03:15 GMT):
@Dan we're only running a single node for now
jsmitchell (Fri, 03 May 2019 17:15:50 GMT):
@bobonana try setting `sawtooth.consensus.min_wait_time` to something like 2 and `sawtooth.consensus.max_wait_time` to something like 10, and make sure that you have `sawtooth.publisher.max_batches_per_block` set to something like 1000
bobonana (Fri, 03 May 2019 21:21:54 GMT):
@jsmitchell It doesn't seem to be affecting performance.
bobonana (Fri, 03 May 2019 21:21:54 GMT):
@jsmitchell those changes don't seem to be affecting performance.
jsmitchell (Fri, 03 May 2019 21:22:31 GMT):
that is very surprising
jsmitchell (Fri, 03 May 2019 21:22:49 GMT):
if you do `sawtooth block list` how many batches are you getting per block?
bobonana (Fri, 03 May 2019 21:41:31 GMT):
@jsmitchell we're getting one batch per block
jsmitchell (Fri, 03 May 2019 21:44:45 GMT):
that's your problem
jsmitchell (Fri, 03 May 2019 21:45:27 GMT):
what did you set min and max wait time to?
jsmitchell (Fri, 03 May 2019 21:46:01 GMT):
and do the logs show that devmode is actually waiting the amount of time you specified to publish a block?
jsmitchell (Fri, 03 May 2019 21:48:57 GMT):
the issue is that devmode by default publishes blocks as fast as possible. The behavior is a number of batches from the pending queue (up to max_batches_per_block) are added to the schedule and begin executing. Immediately, devmode says 'finish what you are doing and publish the block', so the scheduler cancels all in progress work and typically only one or two batches are finished and make it into the block. By waiting a reasonable amount of time between blocks, _many_ more transactions can be processed and included in blocks.
jsmitchell (Fri, 03 May 2019 21:52:05 GMT):
i'm assuming your transactions execute in a short amount of time
bobonana (Fri, 03 May 2019 22:28:55 GMT):
@jsmitchell we just set the min and max wait time to 2 and 10 respectively. How do I check what the actual wait time is (the logs aren't showing anything)?
bobonana (Fri, 03 May 2019 22:31:15 GMT):
and why would the same default settings we've been using run fine in v1.0 but result in a performance hit in v1.1?
MelodyWang (Mon, 06 May 2019 21:46:41 GMT):
hi I am running sawtooth with real sgx but I can not see the log from terminal in poet_block_publisher.py where I print out
But When I use the simulator, I have the printed log, anybody knows why
Dan (Tue, 07 May 2019 17:03:13 GMT):
Not sure but maybe something here will be helpful: https://sawtooth.hyperledger.org/docs/core/releases/latest/sysadmin_guide/systemd.html?highlight=running%20service#view-sawtooth-logs
MicBowman (Wed, 08 May 2019 16:04:29 GMT):
@MelodyWang did you get the issues address for the SPID?
MicBowman (Wed, 08 May 2019 16:06:25 GMT):
and... regarding logs... i'm not sure how sawtooth handles build flags, but for logs coming out of the enclave, you need to make sure the print/log ocalls are compiled in
MicBowman (Wed, 08 May 2019 16:06:45 GMT):
i can tell you how to do it with PDO... :-)
jsmitchell (Wed, 08 May 2019 16:08:17 GMT):
poet_block_publisher.py doesn't sound like enclave code
vineethkrishnan12 (Mon, 13 May 2019 05:33:06 GMT):
Has joined the channel.
vineethkrishnan12 (Mon, 13 May 2019 05:33:08 GMT):
Hi, I was trying to run the sawtooth pbft consensus algorithm from the following link. "https://github.com/hyperledger/sawtooth-pbft/tree/master/adhoc". There are three yaml files : admin.yaml, node.yaml, workload.yaml. When i first ran it,it worked perfectly.From second time onwards node.yaml is not working.I haven't made any change to the code.Moreover i reset the whole scene couple of times and the error still pops out. It says : :
vineethkrishnan12 (Mon, 13 May 2019 05:33:08 GMT):
Hi, I was trying to run the sawtooth pbft consensus algorithm from the following link. "https://github.com/hyperledger/sawtooth-pbft/tree/master/adhoc". There are three yaml files : admin.yaml, node.yaml, workload.yaml. When i first ran it,it worked perfectly.From second time onwards node.yaml is not working.I haven't made any change to the code.Moreover i reset the whole scene couple of times and the error still pops out. It says : ERROR: for pbft Cannot start service pbft: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"./target/debug/pbft-engine\": stat ./target/debug/pbft-engine: no such file or directory": unknown
ERROR: Encountered errors while bringing up the project.
vineethkrishnan12 (Mon, 13 May 2019 05:33:08 GMT):
Hi, I was trying to run the sawtooth pbft consensus algorithm from the following link. "https://github.com/hyperledger/sawtooth-pbft/tree/master/adhoc". There are three yaml files: admin.yaml, node.yaml, workload.yaml. When I first ran it, it worked perfectly. From second time onwards node.yaml is not working.I haven't made any change to the code. Moreover, i reset the whole scene a couple of times and the error still pops out. It says : ERROR: for pbft Cannot start service pbft: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"./target/debug/pbft-engine\": stat ./target/debug/pbft-engine: no such file or directory": unknown
ERROR: Encountered errors while bringing up the project. I tried resetting the docker and pulling the new image of the same. yet the same error. I used the "latest" as the isolation ID and pulled the required images from hyperledger repo itself. Any idea where i went wrong?
j0g1 (Mon, 13 May 2019 09:02:32 GMT):
Hey. Can you look at the idea of the consensus algorithm that I invented? The purpose of the algorithm was to omit the necessity of using Sawtooth with hardware (PoET algorithm). I will be grateful for information if the solution makes sense. If in doubt, I will be happy to answer your questions.
j0g1 (Mon, 13 May 2019 09:02:45 GMT):
https://www.slideshare.net/secret/1Rd0oE3KT02hQb
ltseeley (Mon, 13 May 2019 14:36:33 GMT):
@vineethkrishnan12 can you describe the exact steps to reproduce the issue?
Dan (Mon, 13 May 2019 14:46:29 GMT):
@j0g1 when i go to the link I only see 3 slides is that accurate? what poet gives you with trusted execution is a fair random number. what provides the fairness in your proposal? Is there anything to prevent a node from always just picking a really small number?
j0g1 (Mon, 13 May 2019 19:20:10 GMT):
I think slideshare lost animation from ppt
j0g1 (Mon, 13 May 2019 19:20:18 GMT):
I try use google docs
j0g1 (Mon, 13 May 2019 19:22:31 GMT):
https://drive.google.com/open?id=1WU9o0fU7BsDLGECSMtuN4fq_LMzjh0GV
j0g1 (Mon, 13 May 2019 19:22:39 GMT):
this should be okay
j0g1 (Mon, 13 May 2019 19:24:06 GMT):
My idea for the fairness came with two features: 1) nodes generate number for another nodes(not for themselves) 2) nodes exchange number with random nodes (ensures by voting)
j0g1 (Mon, 13 May 2019 19:24:23 GMT):
So node can't pick for his own needs the best number
j0g1 (Mon, 13 May 2019 19:24:55 GMT):
and is not sure who will get his number so can't send it to another "bad" node
j0g1 (Mon, 13 May 2019 19:25:12 GMT):
it is as hard as hard is to win voting
arsulegai (Mon, 13 May 2019 19:41:56 GMT):
How do you prove a bad actor "Oscar" is not claiming leadership?
jsmitchell (Mon, 13 May 2019 19:43:05 GMT):
presumably, the winning node would have to supply the signed evidence from the generating node @arsulegai
jsmitchell (Mon, 13 May 2019 19:44:50 GMT):
The questions I have would be along the lines of: "How many messages would need to be exchanged between nodes to support this voting mechanism?" and "If that is as bad or worst than PBFT, why wouldn't you just use PBFT?"
jsmitchell (Mon, 13 May 2019 19:46:23 GMT):
Would need to understand a lot more about the proposed voting mechanism to understand how it prevents collusion
arsulegai (Mon, 13 May 2019 19:51:53 GMT):
Sure, the reason I asked the question is because BFT is difficult to achieve with only random numbers & voting in place. There would be other factors we need to consider, what if there are greedy actors and try to give large values to others.
j0g1 (Mon, 13 May 2019 20:45:54 GMT):
Hi guys. First of all thank you for you feedback and starting discussion about my proposal
j0g1 (Mon, 13 May 2019 20:46:19 GMT):
At this moment I don't have answer but I will provide it soon :)
j0g1 (Mon, 13 May 2019 20:47:43 GMT):
of course I agree that I need proof that this algorithm is not only next one but also have a big/small part of cases where use it make a sense
j0g1 (Mon, 13 May 2019 20:48:14 GMT):
the reason behind this proposal is create hardware-independent PoET
j0g1 (Mon, 13 May 2019 20:48:38 GMT):
so this can be achive by exchange numbers between nodes
j0g1 (Mon, 13 May 2019 20:49:04 GMT):
but if we have more than one "bad" node then they can exchange number between them to break system
j0g1 (Mon, 13 May 2019 20:49:15 GMT):
so we need mechanism for random "exchane path"
j0g1 (Mon, 13 May 2019 20:49:42 GMT):
first idea was to have a seed in genesis block which can be use by all nodes to generate path
j0g1 (Mon, 13 May 2019 20:50:23 GMT):
but "bad" node can generate future paths and knows when it will exchange value with another "bad" node
j0g1 (Mon, 13 May 2019 20:50:36 GMT):
so this is reason why I decide to voting for a path
jsmitchell (Mon, 13 May 2019 20:53:22 GMT):
@j0g1 one of the big advantages of so-called "nakamoto"-style consensuses (like PoET and PoW) is that there is no messaging between nodes in advance of publishing a block. The block itself contains the proof which is verifiable. If there are voting rounds, presumably with sealed evidence being exchanged, prior to the generation of a block it will radically change the possible size and structure of the network. There are already a number of paxos-derived voting based algorithms that discuss proofs for the minimum number of messages required for BFT voting.
jsmitchell (Mon, 13 May 2019 20:53:22 GMT):
@j0g1 one of the big advantages of so-called "nakamoto"-style consensuses (like PoET and PoW) is that there is no messaging between nodes in advance of publishing a block. The block itself contains the proof which is verifiable. If there are voting rounds, presumably with sealed evidence being exchanged prior to the generation of a block, it will radically change the possible size and structure of the network. There are already a number of paxos-derived voting based algorithms that discuss proofs for the minimum number of messages required for BFT voting.
j0g1 (Mon, 13 May 2019 20:57:52 GMT):
so this algorithm doesn't make sense :D
j0g1 (Mon, 13 May 2019 21:00:12 GMT):
another way: it works but is a small chance that it will be faster than another so your question "why not using XYZ instead" will always win :D
j0g1 (Mon, 13 May 2019 21:00:43 GMT):
I have to learn a lot yet about consensus mechanism
Dan (Mon, 13 May 2019 21:22:53 GMT):
yeah it's a fun topic :)
arsulegai (Tue, 14 May 2019 13:30:06 GMT):
dpkg: error processing archive /tmp/apt-dpkg-install-df6cVS/132-python3-sawtooth-signing_1.2.2~dev5-1_all.deb (--unpack):
arsulegai (Tue, 14 May 2019 13:30:15 GMT):
Anybody have seen this?
arsulegai (Tue, 14 May 2019 13:30:15 GMT):
Has anybody seen this?
Sidaretta (Thu, 16 May 2019 18:52:22 GMT):
Has joined the channel.
Sidaretta (Thu, 16 May 2019 18:56:15 GMT):
Hi everyone
I wanna Start the Devmode Consensus Engine with following command:
root@180cb:~/sawtooth-project# sudo -u sawtooth devmode-engine-rust -vv --connect tcp://localhost:5050
but I don't see any verbose or message
after a few minutes I receive following Error:
ERROR | devmode_engine_rust: | ReceiveError: TimeoutError
how can I fix my problem? pls help me
pschwarz (Thu, 16 May 2019 19:41:14 GMT):
@Sidaretta are you running that in a docker container?
pschwarz (Thu, 16 May 2019 19:53:06 GMT):
Also, you are asking questions on an RFC PR that is closed - you should be asking them in #sawtooth
Sidaretta (Thu, 16 May 2019 20:01:00 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=Q9xwQTkEgKKH3XmtQ) no I working in Ubuntu
jsmitchell (Thu, 16 May 2019 20:07:35 GMT):
@Sidaretta you accidentally posted a help question in a github PR. Please use the chat channels or the mailing lists for help requests. I have deleted the messages in the github PR.
Sidaretta (Fri, 17 May 2019 09:50:07 GMT):
can anyone help me? I have to fix this problem, but I don't know how!
danintel (Fri, 17 May 2019 18:20:43 GMT):
You also need to specify the Devmode engine on the `sawtooth-validator` command line:
```sawtooth-validator -vvv --endpoint tcp://validator:8800 --bind component:tcp://eth0:4004 --bind network:tcp://eth0:8800 *--bind consensus:tcp://eth0:5050* ```
danintel (Fri, 17 May 2019 18:20:43 GMT):
You also need to specify the Devmode engine on the `sawtooth-validator` command line:
```sawtooth-validator -vvv --endpoint tcp://validator:8800 --bind component:tcp://eth0:4004 --bind network:tcp://eth0:8800 --bind consensus:tcp://eth0:5050 ```
Sidaretta (Fri, 17 May 2019 18:25:47 GMT):
thank you very much
GianlucaSchoefer (Sat, 18 May 2019 00:46:44 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=MjyAL36GYqRMgf7hc) I did a pretty comprehensive review on current papers about nakamoto influenced consensus protocols last year.. I think these two are the best to get a broad overview of the landscape. https://arxiv.org/pdf/1711.03936.pdf and https://arxiv.org/pdf/1707.01873.pdf If you are interested in the theoretical background I would suggest you start reading into more general distributed system literature (like FLP, CAP, SMR, system models, correctness properties and of course a lot of Lamport, basically he single-handedly made BFT a thing)
amiec (Sun, 19 May 2019 00:24:32 GMT):
Hi all - a question about the validator logs in PoET 1.0: In my research (tiny networks running Sawtooth 1.0.5 in Docker containers on an 8-core machine), I'm combing the logs from the validators to get a sense of "stale block rate" - which refers to the percentage of blocks that are published and valid, but don't end up being part of the main chain. I'm doing this by creating a unique set of all block IDs that are validated by any validator, and comparing the size of this set to the number of blocks that actually end up on the main chain (stale block rate = (num_all_valid_blocks - num_main_chain_blocks) / num_all_valid_blocks). I add a block ID to my set every time I see "Finished block validation of"... in a line of a validator's logs. I don't believe any blocks should be invalid in these simulations (and haven't seen anything to indicate the occurrence of an invalid block in the logs). I'm a little concerned because even in small networks with 4 or 6 nodes, I'm seeing some very high stale block rates (like 80 or 90 %). Meaning MOST blocks that get validated don't end up on the main chain.
Does my log combing approach seem correct? Could the stale rates truly be this high? If so, any ideas why? Could there be so much congestion in the local network that the propagation delay is high enough to allow for all these publishing collisions? I would really appreciate some experienced PoET wisdom. Thank you.
amiec (Sun, 19 May 2019 00:24:32 GMT):
Hi all - a question about the validator logs in PoET 1.0: In my research (tiny networks running Sawtooth 1.0.5 in Docker containers on an 8-core machine), I'm combing the logs from the validators to get a sense of "stale block rate" - which refers to the percentage of blocks that are published and valid, but don't end up being part of the main chain. I'm doing this by creating a unique set of all block IDs that are validated by any validator, and comparing the size of this set to the number of blocks that actually end up on the main chain (stale block rate = (num_all_valid_blocks - num_main_chain_blocks) / num_all_valid_blocks). I add a block ID to my set every time I see "Finished block validation of"... in a line of a validator's logs. I don't believe any blocks should be invalid in these simulations (and haven't seen anything to indicate the occurrence of an invalid block in the logs). I'm a little concerned because even in small networks with 4 or 6 nodes, I'm seeing some very high stale block rates (like 80 or 90%!). Meaning MOST blocks that get validated don't end up on the main chain.
Does my log combing approach seem correct? Could the stale rates truly be this high? If so, any ideas why? Could there be so much congestion in the local network that the propagation delay is high enough to allow for all these publishing collisions? I would really appreciate some experienced PoET wisdom. Thank you.
jsmitchell (Sun, 19 May 2019 01:44:55 GMT):
How many nodes are you running, and what are your target and initial wait times?
amiec (Sun, 19 May 2019 03:06:31 GMT):
This is happening with 4, 6, 8 node networks, all using 20 seconds as target wait time and 20 * network size as initial wait time. Also, these metrics are taken starting at time 0 for a duration of about 1000 seconds. I'm not letting the network warm up... but figured the behavior would be the same as in a warm network because the network size is fixed and I'm using the appropriate value of initial_wait_time. Are there aspects of Sawtooth or PoET that I'm overlooking here? I'm quite confident in my log crawling code - but perhaps I'm mis-interpreting what's being reported in the logs? Maybe I need to use a different message type? Or maybe the stale block rate really is that high.
Dan (Sun, 19 May 2019 16:43:00 GMT):
Sounds like an interesting project. The ramp behavior is different than the 'warm' behavior. The initial design of poet assumed that nodes would gradually be added to a network over the ramp time. Typically when most people launch test networks, all nodes are launched simultaneously, which makes the assumptions in the ramp inaccurate. Having the ramp in this case is harmless, but it's likely to throw your numbers off. In particular I think the default is 50 blocks for the ramp which is coincidentally is about the same duration that you are measuring. So I think you are only measuring the ramp.
I would recommend letting the network stabilize for at least 50 blocks if not 100, and then starting your measurements at that point.
Second thing would be understanding more what you are trying to measure and what comparative numbers exist. This is a good paper: https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eyal.pdf
I don't think it explicitly measures stale block rate, but I think there is something in there about 'mining efficiency' which sounds like a similar if not the same concept.
I don't remember how it measures this, but I would be curious how one considers a stale block validated on just the host that created it vs. one that gets propagated needlessly across the whole network. I wouldn't be surprised if poet has inefficient propagation, but that would be good to identify and consider how to improve without breaking the protocol.
amiec (Sun, 19 May 2019 23:40:59 GMT):
Hi Dan - thank you for this response, I really appreciate it. I have a few followup questions if anybody is willing to answer:
- Specifically, how is the ramp behavior different from the behavior of a warm network? I was under the assumption that the only difference was in how the exponential distribution for wait time sampling was parameterized... Are there other differences that I'm not measuring?
amiec (Sun, 19 May 2019 23:40:59 GMT):
Hi Dan - thank you for this response, I really appreciate it. I have a few followup questions if anybody is willing to answer:
- Specifically, how is the ramp behavior different from the behavior of a warm network? I was under the assumption that the only difference was in how the exponential distribution for wait time sampling was parameterized... Are there other differences that I'm not taking into account?
- When it comes to validating/publishing/propagating blocks, does a validator validate a block that it has assembled EVEN if it is not yet eligible to publish this block? (and importantly, would I expect to see evidence of this in the logs?)
amiec (Sun, 19 May 2019 23:45:22 GMT):
On this later point, if so, that might be the reason that I'm seeing such high stale block rates - because ALL blocks are being "validated" - even the ones that don't get published. Can anyone recommend a better signature in the logs for determining true stale blocks? (meaning blocks that are truly PUBLISHED and therefore propagated but ultimately not main-chainers). For example, there are "Claimed Block" messages in the logs as well as fork resolution information that I feel might give a better metric. Using "Claimed Block" instead of "Finished block validation of" gave me stale block rates just as high... maybe I need to use the fork resolution information.
amiec (Sun, 19 May 2019 23:55:24 GMT):
Oh and one more! How do I determine the size of my blocks? (For example, 100 batches/block, 20 txns/batch, all IntKey transactions). The architecture pages describe the structures of transactions and batches, but I'm unclear on the exact structure of a Sawtooth block, how to establish the max size, and how the size of the block fluctuates with the number of txns and/or batches it contains?
Dan (Mon, 20 May 2019 00:14:03 GMT):
I think your understanding of the ramp is essentially correct. It's been a long time since I looked at that code, but I think the worst case will be the first round where all the validators will assume they are on a very small network (like just themselves) and so they distribution will be skewed such that they all get really short durations and likely all publish blocks in roughly the same time. That's likely to skew your results.
Dan (Mon, 20 May 2019 00:15:49 GMT):
Are you looking at all N logs (for an N node network) and if so how are you combining those numbers? Or are you looking at a single node as a representative?
amiec (Mon, 20 May 2019 00:22:41 GMT):
I am indeed looking at all N logs for N validators. I comb all logs while maintaining a global set of all unique blockIDs that have passed validation at any node. Finally, when the workload is over I query the chain using the CLI (block list) to get the blocks on the main chain before tearing down the containers.
amiec (Mon, 20 May 2019 00:23:45 GMT):
In case the validators are validating candidate blocks regardless of whether they go on to publish these blocks (which I would expect to give VERY high stale block rates), I'm going to take a look at the average number of fork comparisons across validators and see if those numbers look a little more how I expect.
amiec (Mon, 20 May 2019 00:44:55 GMT):
Update: When I analyze all blocks that are involved in fork comparisons, and then calculate ((total_blocks_involved_in_fork - num_blocks_main_chain) / total_blocks_involved_in_fork) I get the same numbers as when I look at the total number of blocks that get validated and plug this number into the above equation instead. I believe that my scripting is correct but either my understanding of what the logs are depicting is flawed, or my network really is undergoing extreme forking.
Dan (Mon, 20 May 2019 17:34:49 GMT):
So in combining those logs do you get a feel for if an orphan is validated on all nodes or just some nodes?
Are things any different if you run for twice as long and cut off that first 1000s of ramp time from the results? I would not be surprised by N forks in the first round slowly diminishing as you approach the end of the ramp.
There are also events instrumented in the code for various things including I think some forking events. There's some grafana configuration documentation to view those events live too.
https://sawtooth.hyperledger.org/docs/core/nightly/master/sysadmin_guide/grafana_configuration.html
jsmitchell (Mon, 20 May 2019 23:10:44 GMT):
It logs a fork comparison message even for the simple case of extending the chain, btw. To @Dan ‘s point, the grafana metrics provide a wealth of information on this. There is a “blocks considered” graph which can be compared against the blocks committed graph. If the considered graph is more than 20% or so higher than blocks committed, that probably means the wait times need tuning. There is also a graph which shows forks.
deorcregnboga (Tue, 21 May 2019 12:50:50 GMT):
Has joined the channel.
amiec (Wed, 22 May 2019 14:18:44 GMT):
Thanks for these thoughts! I actually have not analyzed whether an orphan is validated on all or just some nodes - that's a great idea and I'll comb the logs again differently to get that sense. I'm also planning to start experimenting with cutting the first 1000 seconds and will let you know what I see. @jsmitchell I did notice that you see a "fork comparison is between - and xxxxxx" when there is no conflicting candidate and am accounting for this as well. I have one other theory regarding the extreme stale block rates - I'm performing these experiments on such a resource-constrained system that memory itself may be a communication bottleneck that is creating such a high effective block propagation latency that nodes are busy creating many of their own blocks before they get around to processing an incoming block... it still seems a little extreme, but perhaps. The plan this summer is to move the experiments onto a cluster and really start digging deeper. I've attached two graphs that may be interesting. These are all for the first 1000 seconds of time, with workloads delivered from Caliper. One graph examines varying the block interval from 3 to 100 seconds. The other uses a fixed block interval of 20 sec. The only other thing that I'm having trouble explaining on these graphs is that the stale block rate (at relatively shorter block intervals at least) is higher at a delivery rate of 5 tps than it is at 10 tps.... quite consistently. I can't explain why a change in the rate of incoming transactions (especially such a small, mild change) should affect the stale block rate at all. (Each data point in these graphs was achieved by running an experiment for about 1000 seconds at a fixed-rate workload at the specified incoming TPS for several runs, and then averaging these runs. Error bars can be large as I believe n=4 for most of these.) If you have any thoughts on that, I would also be curious.
amiec (Wed, 22 May 2019 14:19:29 GMT):
multiplot.pdf
amiec (Wed, 22 May 2019 14:20:09 GMT):
varyintervals.pdf
amiec (Wed, 22 May 2019 14:20:15 GMT):
This is all with v1.0.5 btw
MelodyWang (Tue, 28 May 2019 02:26:49 GMT):
@MicBowman Sorry I missed your question. I did solve it thank you
MelodyWang (Tue, 28 May 2019 02:29:49 GMT):
Hi everyone, I have a simple question that how could I read the 1.0 version of sawtooth sgx code. In the github, the version is confusing me
MelodyWang (Tue, 28 May 2019 02:33:45 GMT):
@Dan I have a simple question that how could I get the 1.0.5 version of sawtooth poet sgx code? I download the current one from github, I run into the following error while compiling
MelodyWang (Tue, 28 May 2019 02:34:11 GMT):
@Dan CMake Error at sawtooth_poet_sgx/libpoet_enclave/CMakeLists.txt:110 (SGX_SIGN_ENCLAVE):
SGX_SIGN_ENCLAVE Function invoked with incorrect arguments for function
named: SGX_SIGN_ENCLAVE
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
ABS-sgx_ukey_exchange
linked by target "libpoet-bridge" in directory /home/melody/Downloads/sawtooth-poet/sgx/sawtooth_poet_sgx/libpoet_bridge
-- Configuring incomplete, errors occurred!
MelodyWang (Tue, 28 May 2019 02:34:22 GMT):
Thank you for your help!
MicBowman (Tue, 28 May 2019 15:32:35 GMT):
@MelodyWang i don't remember how the poet code does it, but you need to generate a key that can be used to sign the enclave. for non-production systems. for PDO we use the following:
MicBowman (Tue, 28 May 2019 15:32:46 GMT):
openssl genrsa -3 -out ${SGX_SIGN_ENCLAVE} 3072
MelodyWang (Wed, 29 May 2019 16:02:53 GMT):
Anybody knows how to get the source code of "sudo add-apt-repository 'deb http://repo.sawtooth.me/ubuntu/1.0/stable xenial universe'"
danintel (Wed, 29 May 2019 17:59:05 GMT):
@MelodyWang For Sawtooth 1.0, PoET is part of the Sawtooth core repository. Use the `1-0` branch for Sawtooth 1.0 source. See https://github.com/hyperledger/sawtooth-core/tree/1-0/consensus/poet
MelodyWang (Wed, 29 May 2019 18:38:50 GMT):
@danintel thanks a lot!
MelodyWang (Wed, 29 May 2019 18:56:05 GMT):
@danintel is there any building instruction I could find ?
danintel (Wed, 29 May 2019 19:06:28 GMT):
I belidve it is called BUILD.md for everything
MelodyWang (Wed, 29 May 2019 19:55:26 GMT):
thank you @danintel. butI did not see the BUILD.md in this tree version
MelodyWang (Wed, 29 May 2019 19:55:26 GMT):
thank you @danintel. butI did not see the BUILD.md in this tree version deb http://repo.sawtooth.me/ubuntu/1.0/stable xenial universe'
danintel (Wed, 29 May 2019 20:07:11 GMT):
BUILD.md was integrated shortly after the Sawtooth 1.0 release. Read the original version of it for 1.0 build instructions (dated January 2018). https://github.com/hyperledger/sawtooth-core/blob/cb050657df8c15a2098ffbf6ec376038541e28e6/BUILD.md
MelodyWang (Wed, 29 May 2019 20:12:37 GMT):
thank you @danintel. The branch 1-0 in sawtooth core is showing the version is 1.0.6. Just want to make sure this is the source code of "deb http://repo.sawtooth.me/ubuntu/1.0/stable xenial universe'" since I got this instruction from document 1.0.5. And I tried to build from the source but enclave could not be initialized because of Invalid enclave metadata
pschwarz (Wed, 29 May 2019 20:18:02 GMT):
1.0.6 is built from the 1-0 branch tip
amundson (Wed, 29 May 2019 20:18:33 GMT):
@MelodyWang there are tags (v1.0.5 for example) of the releases
MelodyWang (Wed, 29 May 2019 20:20:26 GMT):
@amundson I see thank you! is this should be the source of "deb http://repo.sawtooth.me/ubuntu/1.0/stable xenial universe'"
amundson (Wed, 29 May 2019 20:22:55 GMT):
it should match the version on the installed debian files, so yes
MelodyWang (Wed, 29 May 2019 20:25:35 GMT):
@amundson Thank you! I compiled this one but I still get the "Invalid enclave metadata " while creating enclave
Pradeep_Pentakota (Mon, 03 Jun 2019 10:27:54 GMT):
Has joined the channel.
amolk (Fri, 07 Jun 2019 15:54:20 GMT):
I'm upgrading PoET1.0 to SGX v3, but running into an issue with Swig with a type mismatch between byte and std::string. it seems this was a problem in older versions of swig and was fixed in swig 3.0.10. However, docker seems to pull in only v3.0.8. Swig 4.0 doesn't seem to have an apt package yet that I can find.
Has anyone run into type mismatch issues between Python & C++ before? Any suggestions on how to proceed?
kodonnel (Mon, 10 Jun 2019 15:22:20 GMT):
Anyone seen the following while building `sawtooth-devmode`
```
tep 10/13 : RUN /root/.cargo/bin/cargo deb
---> Running in 70d57dda7714
warning: license field is missing in Cargo.toml
warning: README.md file exists, but is not specified in `readme` Cargo.toml field
Compiling sawtooth-devmode-engine-rust v1.2.1 (/project)
error[E0046]: not all trait items implemented, missing: `additional_protocols`
--> src/engine.rs:212:1
|
212 | impl Engine for DevmodeEngine {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ missing `additional_protocols` in implementation
|
= note: `additional_protocols` from trait: `fn(&Self) -> std::vec::Vec<(std::string::String, std::string::String)>`
```
kodonnel (Mon, 10 Jun 2019 15:23:03 GMT):
Since I am just building the 1.2.1 tag and since I have definitely built this successfully many times before, I am guessing something incompatible has leaked in from upstream.
kodonnel (Mon, 10 Jun 2019 15:23:03 GMT):
Since I am just building the 1.2.1 tag and since I have definitely built this successfully many times before, I am guessing something incompatible has has come from a pulled in dependency.
agunde (Mon, 10 Jun 2019 15:23:11 GMT):
@ltseeley ^
ltseeley (Mon, 10 Jun 2019 15:33:03 GMT):
@kodonnel looks like that tag was created before the SDK version was pinned to 0.2; we just made a change to the SDK, so that's breaking that build. Can you use master instead of the 1.2.1 tag? We're doing a new release soon, so we'll have new tags for that.
kodonnel (Mon, 10 Jun 2019 15:41:00 GMT):
@ltseeley Depends on timing I suppose. You mean the general release of sawtooth 1.2, or of just the devmode engine?
Dan (Mon, 10 Jun 2019 15:49:09 GMT):
@amolk I can't tell for sure but some searches suggest bionic should be importing swig 3.0.12. #private-data-objects is still using swig. @BrunoVavala are you doing anything special to get swig into your containers?
BrunoVavala (Mon, 10 Jun 2019 15:49:09 GMT):
Has joined the channel.
ltseeley (Mon, 10 Jun 2019 15:51:49 GMT):
The new release I'm referring to is the general sawtooth 1.2 release; part of that release is tagging all of the related components (including devmode).
Dan (Mon, 10 Jun 2019 15:52:32 GMT):
I don't see anything special for swig in PDO: https://github.com/hyperledger-labs/private-data-objects/blob/master/docker/Dockerfile.pdo-dev#L92
RealDeanZhao (Wed, 12 Jun 2019 02:24:20 GMT):
Has joined the channel.
BrunoVavala (Mon, 17 Jun 2019 06:13:31 GMT):
(sorry for the delay @Dan)
BrunoVavala (Mon, 17 Jun 2019 06:13:31 GMT):
(sorry for the delay @Dan) just as you also noticed in the code, I don't think so. @msteiner what do you think?
@amolk I am successfully using swig 3.0.8 and 3.0.12 with PDO. I suggest looking at the includes and conversions we have in https://github.com/hyperledger-labs/private-data-objects/blob/master/eservice/pdo/eservice/enclave/pdo_enclave_internal.i
BrunoVavala (Mon, 17 Jun 2019 06:42:25 GMT):
@amolk FYI, here is the public link to the IAS API v3 Rev.5 specs https://api.trustedservices.intel.com/documents/sgx-attestation-api-spec.pdf
We have already adopted the specs in both PDO and Fabric Private Chaincode, and achieved BIG usability improvements since it does not require users to register client certificates.
amolk (Mon, 17 Jun 2019 07:27:02 GMT):
Hi @BrunoVavala thanks. I'll look at the conversions. The IAS interaction with v3 is working just fine. It's while passing data to the SGX later that we get errors.
msteiner (Tue, 18 Jun 2019 01:03:01 GMT):
Has joined the channel.
msteiner (Tue, 18 Jun 2019 01:36:12 GMT):
Nothing special i did in docker to make swig work in PDO. While I haven't tested it for quite a while on ubuntu other than 18.04, certainly earlier versions of PDO worked with swig in all of 16.04, 17.10 and 18.04
RajkumarNatarajan (Sat, 22 Jun 2019 04:39:25 GMT):
Has joined the channel.
duncanjw (Tue, 25 Jun 2019 08:51:13 GMT):
Hi. Is there anything else written up on the consensus interface beyond this? https://sawtooth.hyperledger.org/docs/core/releases/latest/architecture/journal.html#the-consensus-interface
duncanjw (Tue, 25 Jun 2019 08:55:15 GMT):
I appreciate there is the original RFC https://github.com/hyperledger/sawtooth-rfcs/blob/master/text/0004-consensus-api.md
duncanjw (Tue, 25 Jun 2019 08:57:03 GMT):
I guess what would be helpful would be something similar to the way we provide people with TP exemplar(s)
duncanjw (Tue, 25 Jun 2019 08:57:32 GMT):
If I am looking in the wrong place please shout out
duncanjw (Tue, 25 Jun 2019 08:59:49 GMT):
The closest thing I’ve found is this which is helpful https://sawtooth.hyperledger.org/docs/pbft/nightly/master/architecture.html#network-overview
arsulegai (Tue, 25 Jun 2019 09:35:44 GMT):
@duncanjw any specific question?
duncanjw (Tue, 25 Jun 2019 10:41:37 GMT):
Basically is there a bluffer's guide to writing a consensus plug in as the section in the main docs is incredibly high level?
duncanjw (Tue, 25 Jun 2019 10:42:02 GMT):
Or plans to provide one?
duncanjw (Tue, 25 Jun 2019 10:42:35 GMT):
Or plans to provide one?
pschwarz (Tue, 25 Jun 2019 14:17:53 GMT):
The simplest example consensus engine is https://github.com/hyperledger/sawtooth-devmode/
pschwarz (Tue, 25 Jun 2019 14:18:17 GMT):
There isn't a tutorial for writing one, though. It's a pretty advanced topic
lucgerrits (Tue, 25 Jun 2019 15:51:00 GMT):
Has joined the channel.
duncanjw (Wed, 26 Jun 2019 10:18:32 GMT):
Understood but documenting the consensus interface (beyond saying there is one in the main docs) would be a start
ltseeley (Wed, 26 Jun 2019 14:20:10 GMT):
@duna
ltseeley (Wed, 26 Jun 2019 14:21:08 GMT):
@duncanjw you might find this blog I wrote in January helpful: https://www.hyperledger.org/blog/2019/01/11/floating-the-sawtooth-raft-implementing-a-consensus-algorithm-in-rust
ltseeley (Wed, 26 Jun 2019 14:21:58 GMT):
It gives a brief overview of how we implemented the Raft consensus engine
duncanjw (Wed, 26 Jun 2019 14:23:52 GMT):
Agreed. Part 2 is good too
duncanjw (Wed, 26 Jun 2019 14:27:21 GMT):
https://www.hyperledger.org/blog/2019/02/13/introduction-to-sawtooth-pbft
duncanjw (Wed, 26 Jun 2019 14:30:04 GMT):
Got it. Will check out RAFT post. The PBFT ones are helpful too
ltseeley (Wed, 26 Jun 2019 14:35:32 GMT):
Don't hesitate to reach out with any further questions
kkp1310 (Thu, 04 Jul 2019 18:04:18 GMT):
Has joined the channel.
patelkishan (Thu, 04 Jul 2019 18:22:53 GMT):
Has joined the channel.
patelkishan (Thu, 04 Jul 2019 18:22:55 GMT):
Hello guys, Dan suggested this channel.Have a look at this question https://stackoverflow.com/questions/56872663/unusual-behaviour-of-batches-block-in-hyperledger-sawtooth
patelkishan (Thu, 04 Jul 2019 18:23:16 GMT):
I found something really strange.I ran it for total 200 txns, 25 txns per batch.While reading the logs, the following statements caught my eye - "Removed 6 incomplete batches from the schedule", "Removed 5 incomplete batches from the schedule", "Removed 4 incomplete batches from the schedule", "Removed 3 incomplete batches from the schedule",.. and so on.This is the reason why only one batch was included in a block. What can I do?
amundson (Mon, 08 Jul 2019 15:28:00 GMT):
@patelkishan This channel is for development of the consensus engines, so #sawtooth would be more appropriate. However, basically since you have publishing delay set to 0, blocks are published immediately. the "Removed X incomplete batches" occurs because when it's time to publish a block, batches that have not yet been run (and thus are "incomplete" are removed from the scheduler used for block publishing. That's probably going to be all but one in the case where you have the publishing delay set to 0.
amundson (Mon, 08 Jul 2019 15:29:24 GMT):
Setting publishing delay to 0 is not really desirable.
danintel (Mon, 08 Jul 2019 22:55:43 GMT):
You can follow my previous suggestion to not set `sawtooth.consensus.pbft.block_publishing_delay=0`
luckycharms810 (Mon, 15 Jul 2019 15:21:23 GMT):
Has left the channel.
horse (Tue, 30 Jul 2019 14:10:17 GMT):
Has joined the channel.
arsulegai (Wed, 31 Jul 2019 03:45:36 GMT):
I would like to initiate the conversation on PoET 2
arsulegai (Wed, 31 Jul 2019 03:47:07 GMT):
The first topic is the way PoET 2 is deemed to bootstrap
arsulegai (Wed, 31 Jul 2019 03:47:49 GMT):
An enclave cannot be created beforehand and hence a genesis batch creation would not be possible through CLI, like in PoET 1
arsulegai (Wed, 31 Jul 2019 03:49:31 GMT):
PoET 2 startup flow considers enclave initialization, and it does get AVR through remote attestation procedure
arsulegai (Wed, 31 Jul 2019 03:50:28 GMT):
The design consideration is that the genesis Validator has to wait for genesis batch creation (done as part of PoET 2 startup), then proceed
arsulegai (Wed, 31 Jul 2019 03:51:27 GMT):
It's possible to achieve this through docker compose file easily, it's also achievable if we are starting Sawtooth components as services but on same machine
arsulegai (Wed, 31 Jul 2019 03:52:26 GMT):
However, if PoET 2 is running on a different machine as a service. Only for the genesis node, somebody needs to manually copy the created genesis batch to the Validator machine. (It's same with PoET 1)
arsulegai (Wed, 31 Jul 2019 03:53:15 GMT):
This copy however should be done within 5 min, the default timeout value for consensus engine to panic while attempting to connect to a validator service
arsulegai (Wed, 31 Jul 2019 03:54:00 GMT):
We thought of other mechanisms, but this seemed to be more preferable for now. Any comments on this?
amundson (Wed, 31 Jul 2019 14:14:49 GMT):
@arsulegai there is probably a larger question here about the ability to restart validators in general. I'm inferring from what you are saying that when a validator restarts, it will have to re-join the network "from scratch" before it can publish blocks again. that would mean, it is impossible to shut down the network fully and then restart the network--there is an assumption that the network is never completely down. does that sound accurate?
arsulegai (Wed, 31 Jul 2019 14:18:06 GMT):
Yes, this was my next topic of discussion
amundson (Wed, 31 Jul 2019 14:18:50 GMT):
the 5 minute timeout thing would be a bad design from an administrative perspective. basically, what you are saying is that you need some persistent process that has a handle on the enclave. so, you should write a process that does that and let the validator/engine and the CLI talk to that process. it would have a potential benefit that you could upgrade the validator without restarting the enclave "process", assuming API/ABI compatibility hasn't changed between the versions.
amundson (Wed, 31 Jul 2019 14:21:38 GMT):
the can't restart piece basically means it will be exceptionally fragile and difficult to keep any smaller networks running. having a smaller process that has a handle to the enclave will help mitigate that, but won't solve it.
amundson (Wed, 31 Jul 2019 14:21:38 GMT):
the can't-restart piece basically means it will be exceptionally fragile and difficult to keep any smaller networks running. having a smaller process that has a handle to the enclave will help mitigate that, but won't solve it.
amundson (Wed, 31 Jul 2019 14:23:08 GMT):
I think you would want to engineer in some workaround to the restart issue, so you can restart old chains for testing purposes
amundson (Wed, 31 Jul 2019 14:25:03 GMT):
for example, we've discussed changing our overall testing methodology to not always start at genesis, but start much further along in the chain. that implies we start saving longer chains that can be restarted. (we have started to do this somewhat, but from-genesis is still the common test)
amundson (Wed, 31 Jul 2019 14:28:00 GMT):
this approach would also give you a simple way to write a mock enclave process (looks like the real thing, but is only software) while keeping the rest of the system unchanged
amundson (Wed, 31 Jul 2019 14:28:20 GMT):
it could expose a REST API
arsulegai (Wed, 31 Jul 2019 14:35:22 GMT):
I will come back on first part later - starting a daemon process to hold the enclave up and running, which PoET can make use of when the service is started.
arsulegai (Wed, 31 Jul 2019 14:36:27 GMT):
For restarting of the nodes, if there's at least one node in the network then the new registration would pass. The case however we need to worry about is when all the nodes are down.
arsulegai (Wed, 31 Jul 2019 14:39:56 GMT):
This can be considered as a case of total network failure, where older Blockchain will stop growing and a new chain will be started. Realizing such feature would require us to have a support of the validator to bootstrap from arbitrary initial global state.
arsulegai (Wed, 31 Jul 2019 14:44:20 GMT):
The suggestion on holding the process handler in a daemon process as a backup is a great idea as opposed to stringent 5 minute panic. I need some time to understand if it's possible and if so what are its security implications.
jamesbarry (Wed, 31 Jul 2019 16:24:42 GMT):
Has joined the channel.
amundson (Wed, 31 Jul 2019 16:24:45 GMT):
"total network failure" -- yes, which is more likely with low node counts like the number that are commonly used in testing and PoCs
amundson (Wed, 31 Jul 2019 16:25:41 GMT):
it will also be fairly difficult to bootstrap networks before they achieve a significant number of nodes
ArpanNag (Mon, 05 Aug 2019 14:16:30 GMT):
Has joined the channel.
arsulegai (Sun, 11 Aug 2019 10:39:39 GMT):
Ok, here's an update on having separate process for the enclave and the engine
arsulegai (Sun, 11 Aug 2019 10:40:54 GMT):
1. The design will be such that one engine will register with one enclave. Each enclave maintains state for the WaitCertificate.
arsulegai (Sun, 11 Aug 2019 10:41:32 GMT):
2. The enclave will be registered first/create a batch for registration before engine can start its operations.
arsulegai (Sun, 11 Aug 2019 10:42:31 GMT):
3. If enclave process restarts without the engine process, then a new enclave can be attached to the engine. The engine can request for the new block at same height as it was creating earlier.
arsulegai (Sun, 11 Aug 2019 10:43:26 GMT):
4. If engine process restarts without the enclave process, then it can be considered as engine's failure to create a block.
amundson (Mon, 12 Aug 2019 15:22:59 GMT):
@arsulegai for genesis, you should still use the CLI tool so it is an explicit action by the administrator (the CLI tool talking to the enclave process)
arsulegai (Mon, 12 Aug 2019 15:23:34 GMT):
Agree, sorry that I missed CLI in my earlier message. With two different processes there will be a CLI
amundson (Mon, 12 Aug 2019 15:24:36 GMT):
seems good to me
arsulegai (Mon, 12 Aug 2019 15:27:21 GMT):
Regarding second topic: We will start with initial assumption that PoET2 currently do not answer what if all the enclave processes fail at once. At production it's expected to have more number of nodes. For development environment/testing this particular scenario is excluded. However admins can create multiple enclaves and register, keep them ready for PoET Engine to register later.
amundson (Mon, 12 Aug 2019 15:27:46 GMT):
the engine, being in rust in the future, could also be compiled into the validator, making it one less process; having the enclave process separate--so, in sawtooth 2.0, we might compile in pbft and poet by default to save on processes
amundson (Mon, 12 Aug 2019 15:29:30 GMT):
do we know the number of nodes in a PoET 2 network needed to get to a stable point?
amundson (Mon, 12 Aug 2019 15:29:43 GMT):
is it 100+? or less?
arsulegai (Mon, 12 Aug 2019 15:30:27 GMT):
It's less
amundson (Mon, 12 Aug 2019 15:31:41 GMT):
the Z-test, which also assumes large networks, is also problematic on fewer-node networks, and I'm not sure we have ever actually had a network large enough to have it make sense
amundson (Mon, 12 Aug 2019 15:32:19 GMT):
certainly 10 nodes doesn't seem sufficient for these assumptions
amundson (Mon, 12 Aug 2019 15:34:04 GMT):
but, if we only have 10 nodes, PBFT is better anyway probably--still, would be useful to know how big the network has to be
amundson (Mon, 12 Aug 2019 15:35:13 GMT):
I assume someone good with statistics could answer it derived from the expected rate of failure of nodes
arsulegai (Mon, 12 Aug 2019 15:41:16 GMT):
I will take up this topic in the RFC, to get more comments on Z-test
arsulegai (Mon, 12 Aug 2019 15:44:32 GMT):
Another point I wanted to bring up - PoET 2 will go into a new branch under the sawtooth-poet repository and can think of two options
1. Keep PoET 2 as a separate branch, there will be PoET 1 (master) and a PoET 2. Release of PoET 2 will be on need basis, until it reaches the maturity.
2. After rigorous PoET 2 testing, make PoET 2 branch as the default master. But still keep PoET 1 alongside (maybe a tag/branch).
I prefer Option 1 in this. Please comment
amundson (Mon, 12 Aug 2019 15:48:24 GMT):
I'd like to see some pieces go directly into master, like the rust validator registry rewrite
amundson (Mon, 12 Aug 2019 15:50:12 GMT):
I think this codebase should also support PoW
arsulegai (Mon, 12 Aug 2019 15:50:24 GMT):
We can certainly think of that once PoET 2 branch is up, but there are few concerns for 1:1 replacing Python's registry TP with Rust's registry TP.
amundson (Mon, 12 Aug 2019 15:50:36 GMT):
as in, you swap out elapsed time for work depending on a compilation setting
amundson (Mon, 12 Aug 2019 15:51:34 GMT):
first we should be 1:1 replace python tp with rust tp, then extend/modify that rust tp to work w/poet 2; we could use rust features for this even
pschwarz (Mon, 12 Aug 2019 15:51:54 GMT):
The Proof of X repo, where X is in the set of { Elapsed Time, Work,..}
amundson (Mon, 12 Aug 2019 15:51:54 GMT):
we could take a similar approach with a rust version of the CLI
arsulegai (Mon, 12 Aug 2019 15:52:30 GMT):
Few I can think of now
1. AVR verification part is different from PoET 1. Let's assume there will be a flag / parameter to identify version.
2. SgxStructs in PoET 1 are in Python, if Rust's version is to be supported a set of structures will go in similar to the Python's one.
amundson (Mon, 12 Aug 2019 15:54:52 GMT):
if there are no developers working on incrementally making PoET better, should we drop support for PoET entirely? A rewrite that is far away isn't the same thing as moving the current code base forward.
amundson (Mon, 12 Aug 2019 15:56:37 GMT):
(adding it back in 'some day')
amundson (Mon, 12 Aug 2019 16:01:17 GMT):
If we had the enclave process, we could gut a bunch of current PoET to use it
amundson (Mon, 12 Aug 2019 16:05:41 GMT):
@pschwarz and I have discussed adopting an "experimental" rust feature in the validator (similar to Rust's 'nightly') that allows for validator features to be added but not enabled by default. and extending this to other components as well makes sense (SDKs, CLIs, consensus engines)
Dan (Mon, 12 Aug 2019 16:06:50 GMT):
I don't think there's much relationship between poet1 and poet2 including the validator registry. Poets 1 and 2 have different verification code for the registry so I don't think it would make sense to put anything from poet2 into 'master==poet1'
amundson (Mon, 12 Aug 2019 16:08:06 GMT):
the preview version was 90% the same, different implementation
arsulegai (Mon, 12 Aug 2019 16:08:46 GMT):
I second @Dan on that, PoET 2 name may sound it's upgrade to current PoET. But in reality PoET 2 is another implementation of Proof of Elapsed Time.
arsulegai (Mon, 12 Aug 2019 16:09:21 GMT):
@amundson I see how you're visualizing the architecture going forward
arsulegai (Mon, 12 Aug 2019 16:11:04 GMT):
We can have such a support. Still keeping current PoET separate from new PoET
amundson (Mon, 12 Aug 2019 16:14:32 GMT):
I think you are overstating the differences, and if PoET was written in Rust already then 90% of the code would be the same.
Dan (Mon, 12 Aug 2019 16:28:52 GMT):
you would have to put in conditionals to check which protocol you were running which would just complicate things for no reason. There's so much that evolved while we were writing poet1 that I welcome the opportunity to eliminate any spaghetti possible when creating poet2.
amundson (Tue, 13 Aug 2019 16:43:31 GMT):
we are going to start using conditionals more, that's a good thing. hide the experimental stuff but allow it into the code base.
amundson (Tue, 13 Aug 2019 16:46:59 GMT):
but, I'm not suggesting using existing poet1 code as a starting point. I'm suggesting tactically rewriting components first that can be shared between the two. should drive modularity into the poet2 design, and avoid that spaghetti problem again. (a fresh rewrite of new spaghetti code isn't great either). at the same time, we should be thinking of ways to replace et with w.
arsulegai (Tue, 13 Aug 2019 17:05:59 GMT):
I'll upload a VR TP code. Note that it looks mostly the same and may also work with current PoET, but there's difference in the way verification of sign-up info is done. So it's not recommended to replace master's current TP with the new rust TP. Then I will raise a PR for rust version of CLI to follow up after the VR TP, this too will have differences in the way that it'll accept enclave process ip & port. Hopefully we will have different approach by then, these are the code parts I can think of with less difference.
Dan (Tue, 13 Aug 2019 18:19:53 GMT):
I can't think of any reason to put the poet2 VR TP code into master.
Cheese (Fri, 06 Sep 2019 06:48:30 GMT):
Has joined the channel.
Cheese (Fri, 06 Sep 2019 06:48:30 GMT):
Hi, we have just changed our consensus from POET to PBFT and have subscribed to sawtooth/state-delta event.
Although we are receiving transactions in the TP, we are not receiving any events from TP. We were able to receive both events from and transactions in TP back when we were using POET.
Does anyone have any idea about this issue?
Cheese (Fri, 06 Sep 2019 06:48:30 GMT):
Hi, we have set up a new network with PBFT and have subscribed to sawtooth/state-delta events.
Although we are receiving transactions in the TP, we are not receiving any events from the validator. We are able to receive events when running the network using POET.
Does anyone have any idea about this issue?
amundson (Fri, 06 Sep 2019 14:29:30 GMT):
@Cheese what version of sawtooth and what version of pbft?
arsulegai (Mon, 09 Sep 2019 09:00:24 GMT):
Is there a plan to support disk persistence, if there's a way already possible how to do it? https://github.com/hyperledger/sawtooth-pbft/blob/bab51205c8ee2fbf1231676a062a04ad72559ee5/src/config.rs#L87
arsulegai (Mon, 09 Sep 2019 09:06:01 GMT):
Found the answer here https://github.com/hyperledger/sawtooth-pbft/blob/b8cc6cdcf1929341f9aff0b9ecff19f43ff91a9f/src/main.rs#L147
arsulegai (Mon, 09 Sep 2019 09:07:09 GMT):
Also it's documented already https://sawtooth.hyperledger.org/docs/pbft/nightly/master/configuring-pbft.html#pbft-command-line-options
arsulegai (Mon, 09 Sep 2019 09:12:01 GMT):
How about a page in the documentation, that helps or cautions administrator with list of best practices to be followed when deploying a PBFT network. Such as need to persist the data in a directory (otherwise there's data loss upon restart).
arsulegai (Mon, 09 Sep 2019 09:15:37 GMT):
This is the cause for @Cheese 's issue
johnfranklin (Sat, 21 Sep 2019 08:57:52 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=HtM5rME466KcoFPzC) @arsulegai facing the same issue here. Tried configuring the storage setting, but the command gives 'ReceiveError: TimeoutError'. Any way to rectify it? And where is the default storage location for a pbft node? Is it similar to that of raft '/var/lib/sawtooth-raft'? I couldn't find such a location for pbft. I'm using docker for running all the components including pbft-engine. Image version 1.0.
johnfranklin (Sat, 21 Sep 2019 08:57:52 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=HtM5rME466KcoFPzC) @arsulegai facing the same issue here. Tried configuring the storage setting, but the command gives 'ReceiveError: TimeoutError'. Any way to rectify it? And where is the default storage location for a pbft node? Is it similar to that of raft '/var/lib/sawtooth-raft'? I couldn't find such a location for pbft. I'm using docker for running all the components including pbft-engine. pbft-engine image version = 1.0.
arsulegai (Mon, 23 Sep 2019 08:23:18 GMT):
@johnfranklin There's a CLI option here at https://sawtooth.hyperledger.org/docs/pbft/nightly/master/configuring-pbft.html#pbft-command-line-options unlike in Raft, PBFT has flexibility to configure any location of choice through CLI. Could you please print the complete error message you see?
johnfranklin (Mon, 23 Sep 2019 08:33:30 GMT):
@arsulegai the command I gave was "pbft-engine -vv -s /pbft_storage" . inside the pbft-engine docker container. The error message was "ERROR | pbft_engine:108 | ReceiveError : TimeoutError"
arsulegai (Mon, 23 Sep 2019 13:57:30 GMT):
That sounds like the validator is not responding to the consensus engine. It was different behavior which I saw in @Cheese 's case because of node restarts without persistence. We need to see in your case what caused this message timeout (we may get clues from prior traces in PBFT or the validator logs around that time).
kodonnel (Mon, 23 Sep 2019 15:28:00 GMT):
@arsulegai Just following on from the call - since I recalled that we were planning on submitting a fix for the " -s disk+/some_dir" cli option. The engine doesn't actually get the -s argument value as it is now, since the parsed arg is not getting into the created engine config. So a) I think the relationship betweeen the timeouts you are seeing and the storage option isn't real and b) the storage option isn't actually getting set. You can see what we are planning to send in as a fix here https://github.com/hyperledger/sawtooth-pbft/compare/master...blockchaintp:fix/storage-location
kodonnel (Mon, 23 Sep 2019 15:30:43 GMT):
We only spotted it last week while on the road and plan on submitting it assap
kodonnel (Mon, 23 Sep 2019 15:30:43 GMT):
We only spotted it last week while on the road and plan on submitting it asap
arsulegai (Tue, 24 Sep 2019 05:38:16 GMT):
@kodonnel wow, I totally missed two structure approach followed here. First fill in the CLI args structure and then the PBFT config. Thanks
kodonnel (Tue, 24 Sep 2019 13:13:17 GMT):
@arsulegai there is another parameter that is missed like that as well which I suppose we will add to the fix. we'll get the PR in this week
arsulegai (Fri, 04 Oct 2019 09:04:29 GMT):
Another question: It will be helpful to document an example configuration file that can be used.
arsulegai (Fri, 04 Oct 2019 09:04:29 GMT):
Another point: It will be helpful to document an example configuration file that can be used.
arsulegai (Fri, 04 Oct 2019 09:04:29 GMT):
Another point: It will be helpful to document an example log configuration file that can be used. @kodonnel
kodonnel (Wed, 23 Oct 2019 16:18:17 GMT):
Hi, I'm looking through the commits between v1.0.1 and v1.0.0 of sawtooth-pbft and I see this one https://github.com/hyperledger/sawtooth-pbft/commit/0239772ee020d23dc589453987c9cf7d0804fe53 . It effectively says that this particular commit is possible because upon a fix in the validator, and therefore sawtooth-core. Does anyone know which bug/pr/fix vs sawtooth-core it is referring to?
ltseeley (Wed, 23 Oct 2019 18:08:36 GMT):
@kodonnel this is the fix in sawtooth core: https://github.com/hyperledger/sawtooth-core/pull/2049
ltseeley (Wed, 23 Oct 2019 18:10:00 GMT):
That bugfix was not backported to Sawtooth 1.1.
pschwarz (Wed, 23 Oct 2019 19:32:09 GMT):
That fix wasn't because there is not a BlockManager in 1.1 for a little more info on that
kodonnel (Wed, 23 Oct 2019 20:34:24 GMT):
cheers
saanvijay (Thu, 24 Oct 2019 10:14:20 GMT):
Has joined the channel.
kodonnel (Thu, 24 Oct 2019 14:45:44 GMT):
I would but we haven't actually used the configuration file before. If I get a moment I'll try working it through.
kodonnel (Sun, 27 Oct 2019 14:48:57 GMT):
So I've been looking at the pbft voting thresholds for the various message handling methods. There are some definite problems here. One which is fairly straightforward in the handl_prepare message counting which I believe I've addressed properly here https://github.com/hyperledger/sawtooth-pbft/pull/158
kodonnel (Sun, 27 Oct 2019 14:52:19 GMT):
But there is a broader issue, which is that a) there is no checking to see that the membership list is in one of the ideal configurations, i.e. n=4,n=7,n10, etc. There is only a check that there are at least enough nodes to get f=1. That wouldn't be a problem, except b) in the case where you are using nodes counts between the ideal numbers, the truncation math that you use to calculate f, can end up with some suprising figures for f, and therefore 2f+1. Most problematically in the case where n=6 where you effectively can win the network with 3 nodes - i.e. a partition.
kodonnel (Sun, 27 Oct 2019 14:52:19 GMT):
But there is a broader issue, which is that a) there is no checking to see that the membership list is in one of the ideal configurations, i.e. n=4,n=7,n10, etc. There is only a check that there are at least enough nodes to get f=1. That wouldn't be a problem, except b) in the case where you are using nodes counts between the ideal numbers, the truncation math that you use to calculate f, can end up with some suprising figures for f, and therefore 2f+1. Most problematically in the case where n=6 where you effectively can win the network with 3 nodes - i.e. a partition. I should note, that if the pull I referenced above is not used you will require 4 votes, but that also effectively makes a network with 4 nodes unable to continue without a fault, which is not the intended behavior.
kodonnel (Sun, 27 Oct 2019 14:54:32 GMT):
Seems to me the right approach is to either a) error when when you aren't in a list of nodes where the math is even and no truncation happens or b) (my preference) the threshold for voting shouldn't be based on 2f+1 but max(n-f,2f+1) This way, when you do 5 and 6 nodes ( or any other in between value) the threshold for "good votes" increases even if f remains steady.
kodonnel (Sun, 27 Oct 2019 14:55:58 GMT):
We've gone ahead and done the latter option since that seems the most flexible, but it seems like which approach we take is a community decision. So haven't contributed it yet since we may want to go a different way.
ltseeley (Mon, 28 Oct 2019 13:44:27 GMT):
@kodonnel the voting threshold of 2f + 1 for Prepare messages was an intentional decision. As you point out in (b) above, when the threshold is only 2f, it is possible to partition the network when the number of nodes is not "ideal" (i.e., n != 3f + 1). When I was implementing PBFT, I could nt find this particular issue addressed anywhere in the literature
ltseeley (Mon, 28 Oct 2019 13:44:27 GMT):
@kodonnel the voting threshold of 2f + 1 for Prepare messages was an intentional decision. As you point out in (b) above, when the threshold is only 2f, it is possible to partition the network when the number of nodes is not "ideal" (i.e., n != 3f + 1).
ltseeley (Mon, 28 Oct 2019 13:48:25 GMT):
When I was implementing PBFT, I couldn't find this particular issue addressed anywhere in the literature; but clearly it violates the safety guarantees of PBFT, so that wasn't acceptable. My solution was to increase the threshold to 2f +1; this prevents any network segmentation, but it comes at the cost of weakening the liveness guarantees. As is pointed out in PR #158 mentioned above, when the configuration is "ideal" (n = 3f + 1), if f nodes are faulty, no blocks can be committed.
ltseeley (Mon, 28 Oct 2019 13:48:25 GMT):
When I was implementing PBFT, I couldn't find this particular issue addressed anywhere in the literature; but clearly it violates the safety guarantees of PBFT, so that wasn't acceptable. My solution was to increase the threshold to 2f +1; this prevents any network segmentation, but it comes at the cost of weakening the liveness guarantees. As is pointed out in PR # 158 mentioned above, when the configuration is "ideal" (n = 3f + 1), if f nodes are faulty, no blocks can be committed.
ltseeley (Mon, 28 Oct 2019 13:56:15 GMT):
Regarding the two solutions proposed above: (a) seems overly restrictive for network configurations. (b) seems like it might provide even weaker liveness than the current 2f + 1 threshold, since the threshold will be higher (unless I'm misunderstanding, which is definitely possible).
ltseeley (Mon, 28 Oct 2019 13:56:15 GMT):
Regarding the two solutions proposed above: (a) seems overly restrictive for network configurations. (b) seems like it might provide weaker liveness than the current 2f + 1 threshold for "non-ideal" configurations, since the threshold could be higher than 2f + 1 (unless I'm misunderstanding, which is definitely possible).
ltseeley (Mon, 28 Oct 2019 13:59:19 GMT):
A more complete solution might be to use a threshold of 2f when the configuration is ideal, and a threshold of 2f + 1 when the configuration is non-ideal.
ltseeley (Mon, 28 Oct 2019 14:37:38 GMT):
This can be discussed further in the Sawtooth Contributor Meeting at 10am CDT
kodonnel (Mon, 28 Oct 2019 15:26:32 GMT):
So the partitionability at 2f for the Prepare is a direct result of the allowing of the network to be in "non-ideal" configurations. So let's set that aside for the moment.
kodonnel (Mon, 28 Oct 2019 15:26:32 GMT):
So the partitionability at 2f for the Prepare is a direct result of the allowing of the network to be in "non-ideal" configurations. So let's set that aside for the moment. and just talk about the general issue. Get that right and the correct way to address the prepare necessarily results.
kodonnel (Mon, 28 Oct 2019 15:35:09 GMT):
In any of the options, I think we are looking at the votes exceeding some calculated threhold. Either (pseudo code) `max(n-f,2f+1)` , or `mod(n-1,3)==0 ? 2f : 2f+1` will get the thing to work. My only concern is that it enables the network to handle f+1 faults when the membership is ideal-1. Which is a bit unusual for pBFT.
kodonnel (Mon, 28 Oct 2019 15:36:00 GMT):
So fo 6 nodes, you're going to be allowed to have 2 down, instead of 1 which is what the pbft math suggests. Which is arguably good, but weird
kodonnel (Mon, 28 Oct 2019 15:36:26 GMT):
actually that mod isn't quite right there
kodonnel (Mon, 28 Oct 2019 15:37:04 GMT):
but you see my point hopefully
kodonnel (Mon, 28 Oct 2019 15:37:58 GMT):
personally I prefer the n-f, since there are fewer surprises in the behavior. but I'm not married to it
kodonnel (Mon, 28 Oct 2019 15:39:36 GMT):
Fewer surprises since, if f=1, then I should need n-1 votes to get to pass
ltseeley (Mon, 28 Oct 2019 15:54:00 GMT):
Ah yes, I see your point; both achieve the desired result. Using the max approach ensures that no more than f nodes can be faulty or out-of-sync at the cost of some flexibility; using the mod approach would be the opposite. It's strictness vs. flexibility. IMO, flexibility (and therefore better liveness) is more desirable; I wouldn't like to see the network held back from committing a block when it's still able to do so safely. I also don't know that I agree that this is unusual for PBFT since it's the intended behavior of the algorithm. Perhaps we could make the more flexible threshold the default behavior, but provide an option to use the stricter threshold calculation if desired.
danintel (Mon, 28 Oct 2019 15:57:03 GMT):
Has left the channel.
kodonnel (Mon, 28 Oct 2019 16:28:58 GMT):
So you are saying have a setting or something where if `flexible==true` then the threshold is `mod(n-1,3)==0....yadda yadda` else threshold is (n-f) ?
kodonnel (Mon, 28 Oct 2019 16:29:25 GMT):
That would be ok with me. Makes the "surprise" optional
kodonnel (Mon, 28 Oct 2019 16:30:02 GMT):
probably more `strict==true`
kodonnel (Mon, 28 Oct 2019 16:34:58 GMT):
If that's agreed, I'll try to put together a pull request for it.
ltseeley (Mon, 28 Oct 2019 16:59:29 GMT):
That approach works for me!
Alwii (Wed, 30 Oct 2019 07:33:13 GMT):
Has joined the channel.
sergefdrv (Mon, 04 Nov 2019 09:19:06 GMT):
Hi guys, I'm a bit out of discussion, but have you considered a threshold of `|Q| > (n+f)/2`? Any two such quorums must intersect in more than `f` replicas, i.e. in at least one correct replica. This should ensure safety, I suppose
MicBowman (Thu, 07 Nov 2019 15:34:58 GMT):
@Dan there was a proposed structure in the task force pages
MicBowman (Thu, 07 Nov 2019 15:35:10 GMT):
https://wiki.hyperledger.org/display/TF/WGTF_Proposal5
alexhq (Tue, 12 Nov 2019 11:30:34 GMT):
Has joined the channel.
kodonnel (Thu, 14 Nov 2019 16:29:54 GMT):
@sergefdrv I think that's effectively what we ended up with although using some mod trickery so we could stick with integer math
sergefdrv (Thu, 14 Nov 2019 16:58:55 GMT):
@kodonnel Cool :+1: Would you mind sharing your final formula?
kodonnel (Thu, 14 Nov 2019 17:04:22 GMT):
So probably easier to point you at the in progress code - this is the relevant bit https://github.com/blockchaintp/sawtooth-pbft/blob/0f9c101bb26399393adf3bc2de35d25dfe09b0f1/src/state.rs#L174
kodonnel (Thu, 14 Nov 2019 17:04:22 GMT):
@sergefdrv So probably easier to point you at the in progress code - this is the relevant bit https://github.com/blockchaintp/sawtooth-pbft/blob/0f9c101bb26399393adf3bc2de35d25dfe09b0f1/src/state.rs#L174
kodonnel (Thu, 14 Nov 2019 17:07:00 GMT):
strict_quorum is as described above and keeps it to the minimum quorum (not including a nodes own votes for code reasons) to `n-f`, but then in the loose version it is effectively 2*f + (1 or 0 ) depending on the number of members.
kodonnel (Thu, 14 Nov 2019 17:07:00 GMT):
strict_quorum is as described above and keeps it to the minimum quorum (not including a leaders own votes for code reasons) to `n-f`, but then in the loose version it is effectively 2*f + (1 or 0 ) depending on the number of members.
kodonnel (Thu, 14 Nov 2019 17:08:39 GMT):
in either version minimum_quorum()+1 gives you the necessary votes to pass, the strict version keeps the effective `f` to the letter of the law, while the loose one gives you an extra effective `f` in certain configurations.
kodonnel (Thu, 14 Nov 2019 17:10:52 GMT):
I'll be putting that bit together as a PR as soon as I work out some other issues we are facing, having to do with catchup - hopefully that results in something for another pull.
kodonnel (Thu, 14 Nov 2019 17:12:49 GMT):
feedback more than welcome, now or later in the PR's
sergefdrv (Fri, 15 Nov 2019 09:31:18 GMT):
So, if I understood correctly, the strict quorum is the maximum possible quorum that still preserves liveness, which is effectively `|Q| >= n-f`. And the non-strict quorum is supposed to be the minimal required for safety. What I was not sure of was: Why the non-strict formula depends on `f`? Suppose `n=100` and `f=1`...
sergefdrv (Fri, 15 Nov 2019 09:31:18 GMT):
So, if I understood correctly, the strict quorum is the maximum possible quorum that still preserves liveness, which is effectively `|Q| >= n-f`. And the non-strict quorum is supposed to be the minimal required for safety. What I was not sure of was: Why the non-strict formula depends on `f`?
sergefdrv (Fri, 15 Nov 2019 09:31:18 GMT):
So, if I understood correctly, the strict quorum is the maximum possible quorum that still preserves liveness, which is effectively `|Q| >= n-f`. And the non-strict quorum is supposed to be the minimal required for safety. What I was not sure of was: Why the non-strict formula depends only on `f`?
sergefdrv (Fri, 15 Nov 2019 09:31:18 GMT):
So, if I understood correctly, the strict quorum is the maximum possible quorum that still preserves liveness, which is effectively `|Q| >= n-f`. And the non-strict quorum is supposed to be the minimal required for safety.
sergefdrv (Fri, 15 Nov 2019 09:34:44 GMT):
Suppose `n=100` and `f=1`. Then `minimum_quorum()+1` would give 3. I'm afraid two such quorums might not intersect.
sergefdrv (Fri, 15 Nov 2019 09:34:44 GMT):
Suppose `n=100` and `f=1`. Then `minimum_quorum()+1` would give 3 in non-strict mode. I'm afraid two such quorums might not intersect.
sergefdrv (Fri, 15 Nov 2019 09:34:44 GMT):
Suppose `n=100` and `f=1`. Then `minimum_quorum()+1` would give `3` in non-strict mode. I'm afraid two such quorums might not intersect.
giacomo.minighin (Wed, 20 Nov 2019 15:30:10 GMT):
Has joined the channel.
giacomo.minighin (Wed, 20 Nov 2019 15:31:29 GMT):
do you think it is possible to integrate `tendermint` consensus in sawtooth?
amundson (Wed, 20 Nov 2019 17:53:52 GMT):
@giacomo.minighin we looked at that quite a bit during the consensus api design and before we started working on pbft. the tendermint code (at least at the time) wasn't organized such that the consensus algorithm is separate from everything else (like the networking code, peering, etc.). so it would be a fair amount of work to extract the actual algorithm from tendermint's code. however, it would be relatively easy to use Transact with tendermint because Transact is well-aligned with tendermint's concept of application, and would fit nicely against the ABCI. we are also starting to break out pieces of sawtooth into a libsawtooth library, which should help if it were desirable to have the concept of blocks in that context. (not strictly necessary though - scabbard in libsplinter doesn't have blocks and can run sawtooth-sabre contracts fine using transact)
giacomo.minighin (Thu, 21 Nov 2019 10:36:19 GMT):
@amundson thank you for the explanation.
I'm asking this question because we choose sawtooth for our application because of batches.
Then we have discovered cosmos and we thought that in the future we'll probably need to connect the blockchain to it.
giacomo.minighin (Thu, 21 Nov 2019 10:42:13 GMT):
Another question, our application is a cryptocurrency made with sawtooth, we thought to use poet consensus but we're a bit skeptic about it:
is it right to say that with poet consensus the cryptocurrency mining could be tampered by Intel?
amundson (Thu, 21 Nov 2019 22:27:32 GMT):
@giacomo.minighin are you planning for a very large number of nodes?
giacomo.minighin (Fri, 22 Nov 2019 10:19:33 GMT):
@amundson hoping so
giacomo.minighin (Fri, 22 Nov 2019 10:19:33 GMT):
@amundson hope so
amundson (Fri, 22 Nov 2019 19:32:37 GMT):
@giacomo.minighin if you are <20 nodes, pbft is a better bet. after that, poet or a pow makes more sense. we don't have a pow implemented currently. poet 2 work has stagnated, but there is a spec here - https://github.com/hyperledger/sawtooth-rfcs/pull/20. with most of us focusing our efforts on pbft, it isn't clear who will take up the mantle and finish that work. as for whether intel can tamper, there are some thing in poet 1 that are non-sgx based that are intended to mitigate compromized enclaves. but I can't speak to their effectiveness specifically (on small test networks, we often have to turn those off to test poet)
amundson (Fri, 22 Nov 2019 19:34:02 GMT):
it would probably be an easy, but highly technical, endeavor to write a pow consensus for sawtooth. most of the complexity is outside the consensus engine.
amundson (Fri, 22 Nov 2019 19:35:41 GMT):
another thing to consider with a large public network, is that sawtooth does not implement a native incentive system (like gas+ether in ethereum), and so you have to solve that within the smart contracts.
giacomo.minighin (Sun, 24 Nov 2019 15:44:10 GMT):
@amundson thank you. I know I have to implement a native incentive system, do you think that modifying BatchInjector like BlockInfoInjector does is the correct way?
Could it be an external TransactionHandler?
giacomo.minighin (Sun, 24 Nov 2019 15:55:21 GMT):
@amundson I want to share with you my work:
docs: https://github.com/sphereledger/circles-docs
repo: https://github.com/sphereledger/circles-ubi-network
It's not finished yet
giacomo.minighin (Sun, 24 Nov 2019 15:55:21 GMT):
@amundson I want to share with you my work:
docs: https://github.com/sphereledger/circles-docs
repo: https://github.com/sphereledger/circles-ubi-network
It's not finished yet, maybe give me a feed
giacomo.minighin (Sun, 24 Nov 2019 15:55:21 GMT):
@amundson I want to share with you my work:
docs: https://github.com/sphereledger/circles-docs
repo: https://github.com/sphereledger/circles-ubi-network
It's not finished yet, but there's something done
sauravverma (Fri, 29 Nov 2019 22:48:13 GMT):
Hi all, I just wanted to know if there is any way to add nodes to an already running POET consensus network
arsulegai (Sat, 30 Nov 2019 09:57:59 GMT):
@sauravverma Adding a new node (wrto the validator servcie) remains same across the consensus engines. However only the consensus related procedure shall be followed in addition
1. Bring up a new node, let it start syncing up with rest of the network.
2. Note the C test value, the new node won't be publishing until C-test criteria is met.
3. Register the new PoET engine to the network. Once the registration is successful the new node is considered to be part of the network and it can start producing the block.
4. Take care of permissioning if you've enabled it.
For the validator networking - If you've set dynamic networking mode then the new node should try to establish connection to at least one of the existing seed nodes you specify. Note the setting minimum-peer-connectivity and maximum-peer-connectivity
sauravverma (Sat, 30 Nov 2019 18:37:41 GMT):
@arsulegai I run my validators by specifying peers beforehand ..
if i bring up a new node, dont i have to change add that node in the peers for every node?
2. Note the C test value, the new node won't be publishing until C-test criteria is met., Please let me know from where i can get more information about C-test?
3. I spawned my network based on the documentation here https://sawtooth.hyperledger.org/docs/core/releases/1.2.3/app_developers_guide/ubuntu_test_network.html#about-the-ubuntu-sawtooth-network-environment
which says we need to define peers while bringing up the node, Are you reffering to dynamic peering?
wahsunho (Wed, 11 Dec 2019 16:37:37 GMT):
Has joined the channel.
jamesbarry (Thu, 09 Jan 2020 19:45:52 GMT):
Is it possible to make the default storage for PBFT state to disk and not memory? We forgot to change the setting default memory)
Where to store PBFT’s state: memory or disk+/path/to/file
jamesbarry (Thu, 09 Jan 2020 19:45:52 GMT):
Is it possible to make the default storage for PBFT state to disk and not memory? We forgot to change the setting default memory
Where to store PBFT’s state: memory or disk+/path/to/file and this caused our blockchain to puke and stop working when a single node was restaerted.
jamesbarry (Thu, 09 Jan 2020 19:45:52 GMT):
Is it possible to make the default storage for PBFT state to disk and not memory? We forgot to change the setting default memory
Where to store PBFT’s state: memory or disk+/path/to/file and this caused our blockchain to puke and stop working when a single node was restarted.
jamesbarry (Thu, 09 Jan 2020 19:48:25 GMT):
I know we can change it manually but I see this as an issue that requires debugging by those who do not noticed this default setting is in- memory. I think this could be causing some of the issues people bring up around PBFT Sawtooth.
amundson (Thu, 09 Jan 2020 20:21:24 GMT):
I'm in favor of that change (safer default)
arsulegai (Fri, 10 Jan 2020 01:23:13 GMT):
Last time I checked there was an issue around that flag setting. It wasn't getting set. It should be fixed by now.
jamesbarry (Sat, 11 Jan 2020 03:22:05 GMT):
Arun, the bug report shows unresolved, but in 1.2. not the current 1.3.1 https://jira.hyperledger.org/projects/STL/issues/STL-1689?filter=allopenissues Do you know when the bug was fixed? We are a couple of point releases back. We will upgrade to 1.3.1 next week and I'll validate that it works then, and we can close the issue.
arsulegai (Sat, 11 Jan 2020 14:11:52 GMT):
@kodonnel is this done yet? https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=
^ @jamesjamesbarry
arsulegai (Sat, 11 Jan 2020 14:11:52 GMT):
@kodonnel is this done yet? https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=
^ @jamesbarry
arsulegai (Sat, 11 Jan 2020 14:11:52 GMT):
@kodonnel is this done yet? https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=
^ @jamesbarry I was in the middle of job transition, couldn't pay much attention on this and follow up.
jamesbarry (Sat, 11 Jan 2020 16:56:06 GMT):
No issue Arun. I am curous, is it better to comment on these on Rocketchat, or to add a comment on Jira? I am not sure how much Jira is looked at vs. Rocketchat. And within Jira is there a place to denote what release a fix is targeted for? I think my team will look at issues we have experienced and put code in to fix them. But I am not sure how that process fully works for Sawtooth.
arsulegai (Sat, 11 Jan 2020 21:47:14 GMT):
It would be jira, it's developers responsibility to follow up with the assigned issues. There used to be a contributors meeting until last year. But the agreed upon workflow now is that these issues can be discussed on RocketChat, brought to notice for reviews or understanding opens. A quick call can be setup given there's a need. Otherwise we're good to use jira for tracking.
jamesbarry (Mon, 13 Jan 2020 19:31:50 GMT):
Thanks, sounds like a good way to move Sawtooth forward.
wkatsak (Tue, 14 Jan 2020 03:20:45 GMT):
Has joined the channel.
wkatsak (Tue, 14 Jan 2020 03:20:46 GMT):
Just FYI, @arsulegai and @jamesbarry I've verified that in 1.0.1 the storage-location flag isnt working at all, no matter what you set it to, it uses "memory"
arsulegai (Tue, 14 Jan 2020 03:48:19 GMT):
Let's fix it then :)
wkatsak (Tue, 14 Jan 2020 16:59:43 GMT):
We are not equipped for rust dev right now, but please let me know what we can do to help get this patched. We can definitely deploy and test any patch asap.
wkatsak (Tue, 14 Jan 2020 17:59:59 GMT):
@arsulegai Its actually looking to be already fixed in master
wkatsak (Tue, 14 Jan 2020 22:37:30 GMT):
@arsulegai So I was able to deploy master to my test cluster and enable state on disk
wkatsak (Tue, 14 Jan 2020 22:37:40 GMT):
its behaving strangely though
wkatsak (Tue, 14 Jan 2020 22:37:56 GMT):
Keep getting this repeated on almost all nodes `Jan 14 22:36:27 bc2 pbft-engine[31108]: WARN | pbft_engine::engine: | Idle timeout expired; proposing view change`
wkatsak (Tue, 14 Jan 2020 22:41:52 GMT):
BTW what is best practice for peering? Should node specify each other node? Or should we do it like in the Docker compose config, where each node only specifies the previous nodes?
arsulegai (Wed, 15 Jan 2020 01:23:29 GMT):
It should be a fully connected network
ltseeley (Wed, 15 Jan 2020 14:53:12 GMT):
@wkatsak the way the docker compose config is setup is the desired peering configuration as connections are bidirectional, and this configuration avoids duplicate connections.
wkatsak (Thu, 16 Jan 2020 19:24:26 GMT):
@ltseeley I am seeing some issues sometimes where when a validator is restarted, some nodes seem to keep a stale connection around
wkatsak (Thu, 16 Jan 2020 19:25:24 GMT):
This is why I was thinking the peering config might be an issue.
wkatsak (Thu, 16 Jan 2020 19:25:44 GMT):
Additional, when I tried to do fully peered (e.g. each one knowing about all the others), i end up with multiple peer entries for the same host
wkatsak (Thu, 16 Jan 2020 19:27:02 GMT):
Maybe its un-related though
amundson (Thu, 16 Jan 2020 19:44:37 GMT):
@wkatsak do you have a way to reliably cause that issue to appear?
wkatsak (Thu, 16 Jan 2020 19:46:33 GMT):
@amundson I'm working on that. Its been transient on our test cluster. If I can do it deterministically, I'll put something together.
amundson (Thu, 16 Jan 2020 19:48:13 GMT):
even under X conditions it will usually happen in a bounded number of hours would help
wkatsak (Thu, 16 Jan 2020 19:49:29 GMT):
Ok, sure. Thanks
amundson (Thu, 16 Jan 2020 19:50:12 GMT):
if you can figure that out, we will try and replicate it in our environment
wkatsak (Sun, 19 Jan 2020 16:29:31 GMT):
One other thing related to peering. If i bring up our cluster in order (nodes 1, 2, 3, 4, 5) everything is pretty normal. However, if for example, I restart a node for some reason, I see stuff like this: `Can't send message PING_RESPONSE back to XXXXX because connection OutboundConnectionThread-tcp://XXX:8800 not in dispatcher`
wkatsak (Sun, 19 Jan 2020 16:29:49 GMT):
Its like, something dosen't know to set up the outnoundconnection again
wkatsak (Sun, 19 Jan 2020 16:29:51 GMT):
outbound*
arsulegai (Sun, 19 Jan 2020 17:46:40 GMT):
@wkatsak I guess it is max peering connection issue
wkatsak (Mon, 20 Jan 2020 17:02:05 GMT):
@arsulegai Can you elaborate? I only have 5 nodes. Is this something to worry about?
wkatsak (Mon, 20 Jan 2020 17:02:19 GMT):
It seems that having to bring a node down once in a while is a common event
arsulegai (Mon, 20 Jan 2020 17:09:17 GMT):
@wkatsak answered here, it should solve the error because of container restart https://chat.hyperledger.org/channel/sawtooth?msg=wSfJZeKtjDMKjAjx9
wkatsak (Mon, 20 Jan 2020 17:12:27 GMT):
@arsulegai Thanks! I'll try this.
wkatsak (Mon, 20 Jan 2020 17:30:49 GMT):
@arsulegai We actually have `maximum_peer_connectivity = 20` in our validator.toml file already
wkatsak (Mon, 20 Jan 2020 17:30:52 GMT):
for 5 nodes
arsulegai (Mon, 20 Jan 2020 17:31:36 GMT):
Interesting
arsulegai (Mon, 20 Jan 2020 17:31:59 GMT):
Ah! Wait
arsulegai (Mon, 20 Jan 2020 17:32:14 GMT):
You didn't see the error in peering
arsulegai (Mon, 20 Jan 2020 17:38:26 GMT):
Can you check your logs in more detail on the XXXXX part?
arsulegai (Mon, 20 Jan 2020 17:39:26 GMT):
Was it a new ID after you restarted the container? If not probably nothing to worry but need more traces/analysis to identify why that happened.
wkatsak (Mon, 20 Jan 2020 17:43:49 GMT):
I didn't check the IDs, specifically. Also, this deployment is not in Docker, its on 5 different machines geographically dispersed
wkatsak (Mon, 20 Jan 2020 17:44:18 GMT):
But for example, on one, I keep seeing this warning for host1, host2, host3
wkatsak (Mon, 20 Jan 2020 17:44:24 GMT):
but, they are all actually in the peers list
wkatsak (Mon, 20 Jan 2020 17:44:47 GMT):
`Can't send message PING_RESPONSE back to c5d7ebe91c11ae5d9c8fb72c41db51ddcb838a2ad886300328397bfa55cb2ac9ad7cc5440f3d2389c94182a6c5a81497d17f4552b16f3b0c75d7122d7ef0800d because connection OutboundConnectionThread-tcp://host1:8800 not in dispatcher`
Tomomi.Yamano (Mon, 03 Feb 2020 04:53:42 GMT):
Has joined the channel.
RajaramKannan (Mon, 03 Feb 2020 05:59:34 GMT):
Has joined the channel.
gandhikim (Thu, 06 Feb 2020 01:27:12 GMT):
Has joined the channel.
RajaramKannan (Thu, 06 Feb 2020 04:58:53 GMT):
Hi folks, we are seeing the below error, any ideas on why it is happening and what we can do to recover? ```
2020-02-06 04:55:41.990 ERROR threadpool] (Consensus) Unhandled exception during execution of task _HandlerManager.execute.
jamesbarry (Thu, 06 Feb 2020 16:04:17 GMT):
@RajaramKannan You may want to follow the thread in core-dev, as you seem to have a protocol issue. I beleive there needs to be a new relase of PBFT. Copying @agunde
RajaramKannan (Fri, 07 Feb 2020 09:42:06 GMT):
thanks @jamesbarry let me check this out
madhusudan.rao (Fri, 07 Feb 2020 09:47:19 GMT):
Has joined the channel.
gandhikim (Mon, 10 Feb 2020 11:07:24 GMT):
My issue don't sync node1 and node2 but first block sync is succes.
sawtooth-cli (Hyperledger Sawtooth) version 1.2.4
ubuntu 18.04
1 step : node1 run
2 step : set intkey five times
3 step : node2 run
sawtooth-validator log
ERROR proxy] State from block 9e0ed232.... requested, but root hash 71da3e57.... was missing. Returning empty state.
poet-engine log
ERROR poet_block_verifier] Block 9e0ed232 rejected: Received block from an unregistered validator 025f88b8...311d7417
arsulegai (Mon, 10 Feb 2020 11:49:47 GMT):
@gandhikim Question answered in #sawtooth
ParitoshPandey (Fri, 14 Feb 2020 10:16:31 GMT):
Has joined the channel.
ParitoshPandey (Thu, 20 Feb 2020 17:55:55 GMT):
Hi All,
Facing an error with hyperledger sawtooth pbft engine
We have our network with 5 nodes , Suddenly one node went out of sync ,
ParitoshPandey (Thu, 20 Feb 2020 17:56:14 GMT):
The pbft-engine of this node is stuck at an older block 944 , while all the other nodes are at 1291
Validator logs show that is rejecting the blocks which are ahead. Here are the screenshots
ParitoshPandey (Thu, 20 Feb 2020 17:57:50 GMT):
IMG_1543.JPG
ParitoshPandey (Thu, 20 Feb 2020 17:58:15 GMT):
IMG_1544.JPG
jamesbarry (Thu, 20 Feb 2020 18:06:24 GMT):
@ParitoshPandey @wkatsak We are having the same issue and discussing on #sawtooth-core-dev
ParitoshPandey (Thu, 20 Feb 2020 18:11:05 GMT):
Cool , I will check it there
mzins_dev (Tue, 25 Feb 2020 20:55:15 GMT):
Has joined the channel.
mzins_dev (Tue, 25 Feb 2020 21:04:10 GMT):
I everyone. New to sawtooth!
I keep getting a wait certificate timeout issue. It has been challenging to track down this issue. Can anyone give me insight into why this typically happens? I noticed my nodes lose connection during this event but I cant figure out if the timeout is causing connection loss or lost connection is causing the certificate timeout. Thanks!
`[2020-02-24 15:31:07.737 ERROR poet_block_publisher] Failed to create wait certificate: Cannot create wait certificate because timer has timed out`
Dan (Wed, 26 Feb 2020 14:17:21 GMT):
Welcome to Sawtooth!
That error can happen if the node is so busy processing transactions that it misses it's time window to publish.
If you are in the experimenting stage, I'd recommend not using poet but instead using dev-mode consensus.
mzins_dev (Wed, 26 Feb 2020 14:19:08 GMT):
Thanks @Dan! I believe that is whats happening. Besides increasing number of transaction processors and threads available, are there any other configurations that would help? Have you ran into any hardware limitations?
Dan (Wed, 26 Feb 2020 14:28:14 GMT):
Mostly people run into resource limitations in virtualized environments. If you are running on bare metal it is rare to hit limits.
mzins_dev (Wed, 26 Feb 2020 16:06:54 GMT):
By virtualized environments do you mean docker or cloud environments?
Dan (Wed, 26 Feb 2020 17:36:45 GMT):
Generally the latter. People spin up inexpensive instances on e.g. aws and then first run out of I/O ops and then memory or cpu. The other taxing thing for these systems if you have a long lived network with lots of blocks and you introduce a new node. That node has to catch up and that is currently an unoptimized code path. There's a separate dialog in another channel about that specific use case.
jamesbarry (Thu, 27 Feb 2020 04:20:53 GMT):
@mzins_dev @Dan WHile I am not familiar with POET (We use PBFT) we have been trying to see why certain events have timeout issues. Your error on the wait certificate is a new one that we have not seen in our PBFT environment. The issue in #sawtooth-core-dev is around new nodes being added to an existing chain or an offline node coming back on line. We have not been able to determine why our nodes sometimes have short connection loss (like you describe) Any further detail would be great. We are trying to get some detail on our node connection issue, but have not been able to get discrete reasons on why and where that are repeatable.
joe_k (Thu, 27 Feb 2020 13:27:12 GMT):
Has joined the channel.
mzins_dev (Thu, 27 Feb 2020 13:44:31 GMT):
@jamesbarry While the cert time out is our most common error we also seem to get other random networking errors. I did a bit of digging and found that there is futures clean up feature. Sawtooth will automatically remove expired futures but I am not sure how to configure the timeout or if this is what I am actually seeing.
jsmitchell (Fri, 28 Feb 2020 20:54:48 GMT):
@jamesbarry
jsmitchell (Fri, 28 Feb 2020 20:54:48 GMT):
@jamesbarry do you guys have some examples of where the current consensus interface is a bad fit?
jamesbarry (Fri, 28 Feb 2020 20:58:05 GMT):
@jsmitchell - @wkatsak and I are speaking at the HL Global Forum next week. https://hgf20.sched.com/event/ZdHX We will send you the presentation that we are still updating. Because I think this lies in the path that you and @amundson are staking out. I will be commenting more on the pluggable consensus soon.
jsmitchell (Fri, 28 Feb 2020 20:59:32 GMT):
oh cool! you should connect up with @cecchi when you are out there
cecchi (Fri, 28 Feb 2020 20:59:32 GMT):
Has joined the channel.
cg223 (Sat, 29 Feb 2020 14:13:30 GMT):
Has joined the channel.
lucgerrits (Fri, 06 Mar 2020 13:40:39 GMT):
Has left the channel.
MarkAD (Thu, 19 Mar 2020 10:06:28 GMT):
Has joined the channel.
MarkAD (Thu, 19 Mar 2020 10:06:29 GMT):
Hi all, I'm new to Sawtooth and I am currently developing in a PBFT network. I started off with the one node and was able to send data off to the sawtooth network but have moved to the sawtooth-default-pbft.yaml. When posting to the network, what is the url you have to post to as I have tried to the localhost:8008/batches as well as the rest-api-default-0 IP address which I got from docker.
alohamora (Mon, 06 Apr 2020 10:57:17 GMT):
Has joined the channel.
RajaramKannan (Fri, 10 Apr 2020 06:45:23 GMT):
Hi all, we did some more digging into the catchup issue with PBFT. We ran some trials yesterday to see if this was a memory issue, but in a 4 GB system the containers were not using even 500 MB during the catchup process. The CPU utilization would go up to 120% (2 CPU system), all info using docker stats. It would run and catchup 150 blocks then stall/crash, then on restart catchup another 300 blocks and so on... We seem to be seeing a lot of thread panics during this process. Also when it stalls, it starts giving _"[2020-04-09 11:53:05.875 WARNING (unknown file)]
[src/journal/block_validator.rs: 284] Error during block validation:
BlockValidationError("During validate_on_chain_rules, error creating
settings view: NotFound(\"66c9432daec0fc99b356eec7b9f877c0ade375d0f30646dce8eaebb303113ed8\")")"_ type of warnings. Anyone seen this type of issue? It occurs only during catchup, otherwise on regular block creation/commits there is no issue
RajaramKannan (Mon, 13 Apr 2020 12:26:12 GMT):
Hi @arsulegai we were running tests on catchup, and we notice the following. A node that is part of the PBFT is trying to catchup from scratch. We are using 1.1.5 and 1.0.2 pbft. (we noticed lots of crashes with 1.2.4 during this process). There are about 1400 blocks to catchup... ... Sometimes it does catchup sometimes it doesnt. When it doesnt catchup it appears that while processing the pendng blocks, it jumped to the latest PBFT view. In the cases it does catchup, the view continues to be lower until it catches up. In both cases once it reaches the current view, it keeps putting out the "TRACE | pbft_engine::engine: | (PP, view 96716, seq 14) received message Err(Timeout)"
arsulegai (Mon, 13 Apr 2020 12:41:46 GMT):
^ @pschwarz
RajaramKannan (Mon, 13 Apr 2020 13:09:10 GMT):
Hi @jamesbarry we are trying to do this but the PBFT engine seems to be exiting. we were trying "--storage-location \pbft_storage" as the option
RajaramKannan (Mon, 13 Apr 2020 13:42:03 GMT):
@arsulegai @pschwarz here is an example of when it gets stuck at say seq 10, but the view number jumps to the current view and it stops catching up blocks TRACE | pbft_engine::node:10 | (V(96905), view 92, seq 10): Got peer message: MsgInfo (NV S 1407 V 96905 <- 0228401ba95a68e1380cd69e7ee6e52468a335a17866c122b146ee8e9dad169fe1)
pschwarz (Mon, 13 Apr 2020 17:18:10 GMT):
@ltseeley :point_up:
ltseeley (Mon, 13 Apr 2020 19:55:25 GMT):
@RajaramKannan hmm, those `TRACE` logs don't indicate anything incorrect happening. Could you provide the whole log file, or at least a chunk of it from around the time catch-up stopped working?
RajaramKannan (Tue, 14 Apr 2020 04:00:41 GMT):
@itseeley we will try to get the logs for you, each time it happens at different block/seq num, but we should be able to see for example that during catchup in the above example pbft engine was processing catchup at view num 92, seq_num 10 when it got the above message and jumped to view 96905 and seq1407 (latest view/seq in other running nodes). At that point it stopped processing the blocks to be caught up.
muneebmh (Tue, 14 Apr 2020 05:14:24 GMT):
Has joined the channel.
muneebmh (Tue, 14 Apr 2020 05:14:25 GMT):
Hey,
I hope you'll be doing well, I'm new to hyperledger technologies, however, I have good idea about blockchain from research persepctive.
I just wanted to get a clue that whether can I integrate my own consensus mechanism in Sawtooth? As I want to do it for my research work.
P.S My mechanism will be on basis of tokens (Similar to PoS), but for a consortium environment. So, what's the suggestions from you people.
Thanks in Advance
arsulegai (Tue, 14 Apr 2020 06:30:33 GMT):
@muneebmh that's great. Sawtooth provides consensus SDK to write your own consensus engine. It's possible.
muneebmh (Tue, 14 Apr 2020 07:08:07 GMT):
Thanks a lot @arsulegai for your response, that's great. I will proceed with it then.
Moreover, can you give me some basic clue from where I should start with, as I'm totally new to Hyperledger at the moment. Any reading material, etc specifically from consensus perspective.
Thank you again
muneebmh (Tue, 14 Apr 2020 07:10:50 GMT):
or which files should I play with?
ParitoshPandey (Tue, 14 Apr 2020 17:14:04 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=dm7Qz3oALPs22786B)
pbft.log
ParitoshPandey (Tue, 14 Apr 2020 17:14:20 GMT):
Hi @ltseeley I am attaching the logs for the same issue here . Here It got stuck at seq number 5
RajaramKannan (Wed, 15 Apr 2020 03:21:53 GMT):
@kodonnel - this is a summary of the catchup issue. I'll also direct you to the complete PBFT logs provided by my colleague paritosh
RajaramKannan (Wed, 15 Apr 2020 03:22:13 GMT):
@kodonnel - complete PBFT logs provided by my colleague paritosh
arsulegai (Wed, 15 Apr 2020 04:34:53 GMT):
@muneebmh the easiest example you could find on consensus engine https://github.com/hyperledger/sawtooth-devmode/blob/master/src/engine.rs
arsulegai (Wed, 15 Apr 2020 04:35:12 GMT):
I prefer you use Rust SDK as well, Python is another available option.
Ashish_ydv (Wed, 15 Apr 2020 05:04:29 GMT):
Has joined the channel.
muneebmh (Wed, 15 Apr 2020 09:04:22 GMT):
@arsulegai Thank you for the guideline, I will explore the given link and Rust SDK for further development on this.
Thank you again.
RajaramKannan (Thu, 16 Apr 2020 16:59:33 GMT):
@ltseeley @kodonnel We looked a little closer into the logs, and the offending portions seems to be when block 6 came in before block 5 (or even block 4!) during the catchup and it got discarded. Subsequently all blocks after 6 also get discarded since the previous block does not exist with the pbft engine. However somewhere when it receives block 5 it adds it to its log. It is able to catchup upto block 4 using the seals in the next block 9upto 5). But when it is processing catchup on block 5 it does not have 6, so it sends out a seal request. It doesnt receive a response (rightly so) since the rest of the network is at block 1406 or something. So it stalls now at block 5. about 60% of our runs result in this scenario where it gets stuck on various blocks (depending on which block number was received out of sequence). All of this is in the logs @ParitoshPandey provided above (lines 128 to 149 roughly capture this sequence)
kodonnel (Thu, 16 Apr 2020 19:19:22 GMT):
You shouldn't confuse the sequencing of the messages in the validator logs with those in the pbft logs. The validator's logs concern transaction validation which can happen well ahead of the consensus engine instructing the validator to commit the block.
kodonnel (Thu, 16 Apr 2020 19:20:33 GMT):
However the core problem is going to be that your earliest block to be validated is not passing validation, thus you are getting all those descendant blocks being marked as invalid
kodonnel (Thu, 16 Apr 2020 19:23:01 GMT):
The portion of pbft log that was posted isn't particularly helpful. Somewhere just before the idle period would be more interesting.
kodonnel (Thu, 16 Apr 2020 19:28:56 GMT):
Long story short that's not what's going on. The process of catchup begins with the late validator being notified of the new chain_head, then working its way backwards from that point to the last block it shares with the rest of the chain. At this point ( presuming validator 1.2, 1.1 is slightly different) the engine receives a BlockNew message, and the engine requests validation. Then the validator proceeds to validate the block ( i.e. passes the TP's etc and produces the expected result) only then is a BlockValid message passed to the pbft-engine, at which point, the block's seal can be obtained and checked versus the grandparent blocks voting membership.
kodonnel (Thu, 16 Apr 2020 19:29:48 GMT):
All that is for nought if the first block attempted is never validated. So we need to go much earlier in the validator that is catching up's logs and see what is happening. There should be a warning before all those descendent marks.
kodonnel (Thu, 16 Apr 2020 19:53:05 GMT):
Hope all that is clear as mud. :)
RajaramKannan (Fri, 17 Apr 2020 04:31:46 GMT):
Hi @kodonnel love the reference to mud :-) but couple of observations since we dug into both the entire set of logs and the code. all the logs are PBFT logs, so no confusions with the Validator logs. What is happening here is that we are trying to catchup a new node that has no prior blocks. Line 58 in the PBFT logs is confusing since it says internal error for the genesis block (since the block number is 0, but the seq num for PBFT is 1). But it already had previously got it in the start state in line 32, and processed it in line 39. In line 60 the PBFT engine gets block number 1 from the validator. It then gets block 2 in line 70 and marks block 1 valid in line 83/84 and commits in line 120. Meanwhile it has continued to receive blocks and put them in the log marked as unvalidated (pending processing I guess). At line 130 you can see it receives block 6, but it hasnt yet received block 4 or block 5 - so it throws an error and discards that block. In line 139 it gets block 4 and then line 149 block 5. Subsequently as block 7 on wards come in but are discarded since it doesnt have block 6/discarded it previously. Meanwhile it continues processing and commits upto block 4 (line 253). But on block 5 it doesnt have block 6 for the seal and makes a Seal Request (which will be ignored by the rest of the network) line 167. Subsequent to that it pretty much stalls in the catchup since it has no more blocks to process.... I am guessing if a new batch is sent and a new block request came in, then the validator would try to again find previous blocks but it has a lot of catching up to do and might stall again (we had a scenario a couple of monhts back when a node mysteriously caught up after a few days, but I think it looks like it might have incrementally caught up/ stalled each time a new block was processed )
kodonnel (Fri, 17 Apr 2020 13:15:13 GMT):
So I've looked back for this log file you are referencing and I don't see it in the rocket chat. Only a couple images of a pbft log and a validator log. Could you share the log please, otherwise these line references are lost on me.
RajaramKannan (Fri, 17 Apr 2020 13:19:22 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=Euzzb7dAtXT8sAWGK)
pbft.log
RajaramKannan (Mon, 20 Apr 2020 14:22:31 GMT):
Looking at past messages, there are suggestions that the PBFT state should be stored to disk and not memory. Would be great to understand the the underlying issues with storage in memory (so far we have not seen any issues attributable to this...)...
arsulegai (Tue, 21 Apr 2020 03:00:17 GMT):
@RajaramKannan that has to be with the service restart scenarios
RajaramKannan (Tue, 21 Apr 2020 04:32:58 GMT):
@arsulegai can you elaborate a little on what issues it might cause on service restart? We do bring up/down our containers every now and then. In any case, if we were to say update our containers (eg a new build) in any case I think the disk based storage will also be removed?
arsulegai (Tue, 21 Apr 2020 04:55:07 GMT):
Do you bring down all the nodes or just one/some of them?
arsulegai (Tue, 21 Apr 2020 04:55:17 GMT):
* PBFT instances
RajaramKannan (Tue, 21 Apr 2020 05:04:08 GMT):
usually just one or some of them...
arsulegai (Tue, 21 Apr 2020 06:08:33 GMT):
Interesting, have you observed those issues in your network from where you got the https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=yfZn35d9NNjdHfbM9 message?
RajaramKannan (Tue, 21 Apr 2020 06:26:53 GMT):
@arsulegai This query is coming primarily from https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=Psryg6DZm7XBssBC8 This doesnt talk about the specific issues seen. But I suppose if I look further back then one issue seems to have been with receiving state/delta-event https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=HtM5rME466KcoFPzC (we have written custom events and are not using the state/delta events!). But the one mentioned by James Barry seems to indicate somehow tha it caused the blockchain to stop working when a single node was restarted (which we havent experienced - and hence the question)
ParitoshPandey (Tue, 21 Apr 2020 08:31:03 GMT):
Hi All
Facing this issue while trying to build the pbft-engine
ParitoshPandey (Tue, 21 Apr 2020 08:31:11 GMT):
ERROR: VERSION file and (bumped?) git describe versions differ: 1.0.1 != 1.0.2
cargo-deb: Argument to option 'deb-version' missing
ERROR: Service 'pbft-engine' failed to build: The command '/bin/sh -c export VERSION=$(./bin/get_version) && sed -i -e "0,/version.*$/ s/version.*$/version\ =\ \"${VERSION}\"/" Cargo.toml && /root/.cargo/bin/cargo deb --deb-version $VERSION' returned a non-zero code: 1
ParitoshPandey (Tue, 21 Apr 2020 08:31:35 GMT):
Any leads would be really helpful
pschwarz (Tue, 21 Apr 2020 20:53:41 GMT):
@rbuysse :point_up:
rbuysse (Tue, 21 Apr 2020 21:37:09 GMT):
@ParitoshPandey can you try rebasing on master?
RajaramKannan (Wed, 22 Apr 2020 02:43:12 GMT):
@pschwarz @rbuysse we got the build working by changin AUTO_STRICT to just AUTO in the default build file. The STRICT setting was expecting the version to be bumped up....
RajaramKannan (Fri, 24 Apr 2020 07:48:25 GMT):
@rbuysse @pschwarz @amundson - We have been able to build and test a fix to the the issue we have been reporting for a while. There are many conversations and hypothesis on this channel, but we traced the issue we saw (and going by some of the comments, potentially a few others) to basically when the PBFT engine receives a block out of sequence from the validator during the catchup process. Since it discards that block as it does not have its predecessor at that point, it ends up stalling when it processes the predecessor of that block. @ParitoshPandey is the lead engineer on our end who took this initiative up. Do you have any guidance on how we can go about the process of contributing back? This is the best link I could find, https://sawtooth.hyperledger.org/docs/core/releases/latest/community/contributing.html but not much yet beyond. Example, the latest points to 1.0.2 should that be where current contributions are going into? Any guidance will be much appreciated
RajaramKannan (Fri, 24 Apr 2020 07:49:40 GMT):
we want to be able to go back and deploy ideally from the sawtooth dockerhub as well as point our partners to pick up that version as well. We are currently preparing for the production launch of a fractional bond trading system - bondblox.com
pschwarz (Fri, 24 Apr 2020 13:12:54 GMT):
current contributions should be made as PR's against master. They'll be backported to the appropriate releases after they've been reviewed and merged at that level
RajaramKannan (Fri, 24 Apr 2020 14:59:11 GMT):
thanks pschwarz - anythng else to keep in mind?
pschwarz (Fri, 24 Apr 2020 15:04:22 GMT):
Good commit messages are important: https://chris.beams.io/posts/git-commit/
pschwarz (Fri, 24 Apr 2020 15:04:50 GMT):
(also linked in the contributor's guide)
RajaramKannan (Fri, 24 Apr 2020 15:44:30 GMT):
thanks
pschwarz (Mon, 27 Apr 2020 15:57:55 GMT):
ha
YadhuPhilip (Tue, 28 Apr 2020 06:05:11 GMT):
Has joined the channel.
amundson (Wed, 29 Apr 2020 15:30:07 GMT):
@RajaramKannan @ParitoshPandey can you describe the fix or put up a branch with the changes?
ParitoshPandey (Wed, 29 Apr 2020 15:38:33 GMT):
@amundson We are working on the same , we will be pushing a branch in a day or two
AlexanderZhovnuvaty (Thu, 30 Apr 2020 07:15:43 GMT):
Hi. Need assistance with pbft consensus - when i send a transaction to the network - it become in pending state forever. I prepared docker-compose file that start a network of 4 nodes with pbft consensus: https://github.com/AlexZhovnuvaty/dynamic-sawtooth-consensus/blob/master/docker-compose-4-nodes-pbft.yaml
AlexanderZhovnuvaty (Thu, 30 Apr 2020 07:16:04 GMT):
sawtooth-shell-default /bin/bash ; 3) initiate a transaction within the container: xo create hello --url http://sawtooth-rest-api-3:8008; 4) open respective transaction in browser: curl http://sawtooth-rest-api-3:8008/batch_statuses?id=c0e58a2c6922a78a8db7c872118f213e332b92ade23919a7997b581fb4e5af17646f25c28dd829de3cd343f54dfd58208bc8b8c30215557202312a1e39a37b6b
{
"data": [
{
"id": "c0e58a2c6922a78a8db7c872118f213e332b92ade23919a7997b581fb4e5af17646f25c28dd829de3cd343f54dfd58208bc8b8c30215557202312a1e39a37b6b",
"invalid_transactions": [],
"status": "PENDING"
}
],
"link": "http://sawtooth-rest-api-3:8008/batch_statuses?id=c0e58a2c6922a78a8db7c872118f213e332b92ade23919a7997b581fb4e5af17646f25c28dd829de3cd343f54dfd58208bc8b8c30215557202312a1e39a37b6b"
AlexanderZhovnuvaty (Thu, 30 Apr 2020 07:16:04 GMT):
steps: to reproduce: 1) start the network: docker-compose -f docker-compose-4-nodes-pbft.yaml up; 2) connect to shell container: docker exec -it sawtooth-shell-default /bin/bash ; 3) initiate a transaction within the container: xo create hello --url http://sawtooth-rest-api-3:8008; 4) open respective transaction in browser: curl http://sawtooth-rest-api-3:8008/batch_statuses?id=c0e58a2c6922a78a8db7c872118f213e332b92ade23919a7997b581fb4e5af17646f25c28dd829de3cd343f54dfd58208bc8b8c30215557202312a1e39a37b6b
{
"data": [
{
"id": "c0e58a2c6922a78a8db7c872118f213e332b92ade23919a7997b581fb4e5af17646f25c28dd829de3cd343f54dfd58208bc8b8c30215557202312a1e39a37b6b",
"invalid_transactions": [],
"status": "PENDING"
}
],
"link": "http://sawtooth-rest-api-3:8008/batch_statuses?id=c0e58a2c6922a78a8db7c872118f213e332b92ade23919a7997b581fb4e5af17646f25c28dd829de3cd343f54dfd58208bc8b8c30215557202312a1e39a37b6b"
AlexanderZhovnuvaty (Thu, 30 Apr 2020 07:16:54 GMT):
Thanks in advance
ParitoshPandey (Thu, 30 Apr 2020 08:45:02 GMT):
@AlexanderZhovnuvaty Could you also post the logs of sawtooth-rest-api container
ParitoshPandey (Thu, 30 Apr 2020 08:45:35 GMT):
docker logs -f sawtooth-rest-api-3
AlexanderZhovnuvaty (Thu, 30 Apr 2020 09:56:32 GMT):
[2020-04-30 09:44:34.692 INFO messaging] Connecting to tcp://validator-0:4004
[2020-04-30 09:44:34.717 INFO rest_api] Creating handlers for validator at tcp://validator-0:4004
[2020-04-30 09:44:34.778 INFO rest_api] Starting REST API on sawtooth-rest-api-0:8008
[2020-04-30 09:54:49.628 DEBUG route_handlers] Sending CLIENT_BATCH_SUBMIT_REQUEST request to validator
[2020-04-30 09:54:49.655 DEBUG route_handlers] Received CLIENT_BATCH_SUBMIT_RESPONSE response from validator with status OK
[2020-04-30 09:54:49.660 INFO helpers] POST /batches HTTP/1.1: 202 status, 358 size, in 0.032103 s
[2020-04-30 09:55:18.108 DEBUG route_handlers] Sending CLIENT_BATCH_STATUS_REQUEST request to validator
[2020-04-30 09:55:18.111 DEBUG route_handlers] Received CLIENT_BATCH_STATUS_RESPONSE response from validator with status OK
[2020-04-30 09:55:18.113 INFO helpers] GET /batch_statuses?id=3fb0f8d9c76c4a09fb40e2d40d71714f0060a78d4bdd0c275e925f8b05c102fd60aa722597cb71579be9e79c48eee709bdee32a0e6e51ccb076f205e26856f33 HTTP/1.1: 200 status, 585 size, in 0.005407 s
ParitoshPandey (Mon, 04 May 2020 02:41:23 GMT):
We have created a pull request #166 for the same
DavidSetyanugraha (Tue, 05 May 2020 03:39:54 GMT):
Has joined the channel.
sawtooth (Tue, 05 May 2020 20:12:38 GMT):
Has joined the channel.
RajaramKannan (Fri, 08 May 2020 15:39:04 GMT):
@amundson we have pushed the changes, will be great to get some feedback https://github.com/hyperledger/sawtooth-pbft/pull/166#partial-pull-merging
jmbarry (Wed, 13 May 2020 16:18:45 GMT):
Has joined the channel.
Will_Gluwa (Thu, 14 May 2020 15:33:15 GMT):
Has joined the channel.
yashar.nesvaderani (Thu, 14 May 2020 20:39:40 GMT):
Has joined the channel.
RajaramKannan (Tue, 19 May 2020 12:13:05 GMT):
@jamesbarry @arsulegai @ltseeley @amundson Seeing some really wierd PBFT issue, would appreciate any guidance as we go live shortly. Node 1 - (All other nodes are similar and are in sync. Several Blocks committed as part of view 319445 (from block 4168 to 4183) . Node 2 - Out of sync, started from view 319445. What we can notice in the attached logs is it starts to stall after Block 4182. If you look at seq 4168, you can see the block new and block valid (901405…) are different from the block commit (0c3d0…..). Interestingly in the other nodes this block was the one committed at block 4183! Any insights as to why this may be happening? Anything we can do to prevent its occurrence?
RajaramKannan (Tue, 19 May 2020 12:13:23 GMT):
RajaramKannan - Tue May 19 2020 17:43:09 GMT+0530 (India Standard Time).txt
RajaramKannan (Tue, 19 May 2020 12:13:23 GMT):
RajaramKannan - Tue May 19 2020 17:43:09 GMT+0530 (India Standard Time).txt
arsulegai (Wed, 20 May 2020 18:24:53 GMT):
@AlexanderZhovnuvaty it could be some other issue. Please check one by one - if the consensus engine is able to produce the blocks, do you see some validator rejecting block, if validators are broadcasting and agreeing upon a block. Check if your TP is deterministic.
ParitoshPandey (Thu, 21 May 2020 18:56:08 GMT):
Hi All
ParitoshPandey (Thu, 21 May 2020 18:56:47 GMT):
I have created a pull request #166 for sawtooth-pbft-engine. Here is the jira ticket for the same https://jira.hyperledger.org/browse/STL-1706 . A change was suggested by @Itseeley to include the block_buffer in the PbftState.
But for doing the same , we need to implement the
the serialize method for the block_buffer
When I tried adding it
Faced the following error.
erde::Serializeis not satisfied\n\nthe traitstate::_IMPL_DESERIALIZE_FOR_PbftPhase::_serde::Serializeis not implemented forsawtooth_sdk::consensus::engine::Block
I think we need to add it to crate and push it .
Let me know if there is way to get this done ?
arsulegai (Mon, 25 May 2020 04:56:10 GMT):
You could possibly do that, block has all serializable parameters (ref: https://github.com/hyperledger/sawtooth-sdk-rust/blob/3717b2094b921511ab3c199d24666ffc398c83cf/src/consensus/engine.rs#L43). Once you add it, you can request a new release in #sawtooth-release channel. Use this dot release as base to solve the PBFT issue.
ParitoshPandey (Tue, 26 May 2020 05:28:50 GMT):
Thanks @arsulegai
amundson (Fri, 05 Jun 2020 17:13:13 GMT):
As we move forward w/Sawtooth 2, we want to create a unified Rust library with all the consensus implementations; currently PBFT, Raft, and 2PC. This will be similar to the consensus engine concept, but we will focus much more on Rust APIs (traits, implementations, etc.). (Conceptually, we can have a consensus engine on top of the library to retain the inter-process-protocol, which might be interesting for Sawtooth 1.x.)
amundson (Fri, 05 Jun 2020 17:15:13 GMT):
Part of what needs to be decided is where this consensus library should live. A couple options are: 1) A completely new project, with the goal of being the ultimate consensus Rust project (not specific to distributed ledgers); 2) In libsawtooth, under a consensus module; 3) As a separate sawtooth-consensus library. Input welcome.
amundson (Fri, 05 Jun 2020 17:18:55 GMT):
Benefits of (1) seem pretty good - trying to grow a wider community (not just within HL), make it a good place for researchers, etc., have a unique identity for itself, etc.
Dan (Fri, 05 Jun 2020 20:06:40 GMT):
I like the sound of 1 but I'm not real familiar with consensus requirements outside of blockchain (maybe with exception of Cassandra). Other consensus libs had been disappointing to us in the past because people have different ideas of the borders of the abstraction. Not sure off the top of my head how not to fall into the same issue and actually be broadly applicable.
amundson (Fri, 05 Jun 2020 21:27:11 GMT):
I think you don't fall into that trap by not dictating the method of communication (i.e. don't provide daemon processes, etc.)
amundson (Fri, 05 Jun 2020 21:28:13 GMT):
raft-rs for example
amundson (Fri, 05 Jun 2020 21:29:08 GMT):
though they do have a transport layer (maybe new?)
kumar89 (Wed, 10 Jun 2020 10:21:25 GMT):
Has joined the channel.
HichamTAHIRI (Sun, 14 Jun 2020 15:53:47 GMT):
Has joined the channel.
umamani113 (Wed, 17 Jun 2020 12:14:51 GMT):
Has joined the channel.
trbs (Wed, 01 Jul 2020 19:40:35 GMT):
Has left the channel.
akashbhardwaj04 (Mon, 06 Jul 2020 13:11:07 GMT):
Has joined the channel.
lebdron (Tue, 04 Aug 2020 13:39:17 GMT):
Has joined the channel.
lebdron (Tue, 04 Aug 2020 13:39:17 GMT):
Hello! We are working on https://wiki.hyperledger.org/display/INTERN/Project+Plan+-+Reworking+HL+Iroha+Consensus+API , and we would like to ask for a consultation regarding API methods, such as InitializeBlock, SummarizeBlock, and others. Could you please tell who might help us the best with this? Thank you!
lebdron (Tue, 04 Aug 2020 13:40:07 GMT):
User User_1 added by lebdron.
lebdron (Tue, 04 Aug 2020 13:40:16 GMT):
User User_2 added by lebdron.
lebdron (Tue, 04 Aug 2020 13:57:45 GMT):
Ping @Dan @amundson just in case
amundson (Tue, 04 Aug 2020 13:59:30 GMT):
@lebdron one of the things we are going to do short-term is move consensus-related code (PBFT, etc.) to a rust library so that it can be easily consumed by other projects -- it would be cool if Iroha could use that library
amundson (Tue, 04 Aug 2020 14:01:23 GMT):
this will be similar to consensus engines, but without the process barrier (though it may be the implementation for some existing consensus engines as we go forward)
amundson (Tue, 04 Aug 2020 14:07:08 GMT):
@ltseeley has worked on both PBFT and two-phase-commit
lebdron (Tue, 04 Aug 2020 14:08:27 GMT):
That would indeed remove the need of using zeromq, message passing, and separate processes. Sounds great!
As I understand, the interface will still remain the same, so that now we need to process InitializeBlock requests received from zeromq, and later we will need to process such calls directly from the library. As @adamos2468 has already implemented a zeromq interface in C++, we would be happy to get a complete understanding of the interface. While we have an idea of what methods should do based on the Consensus API RFC, we are not totally sure how the methods should relate to the core pipeline.
amundson (Tue, 04 Aug 2020 14:10:45 GMT):
The API will probably change some, but yes we are starting with the consensus engine API and evolving it. So it will likely be a 2.0 of that API. For example, we want to support DLTs that don't necessarily have "block", so we want to make it a bit more general. There are also some other things that we want to address like threading model. Currently one consensus engine == one thread, and we have use cases for thousands of consensus engines "running" in parallel. So likely that will have an impact on the API as well.
amundson (Tue, 04 Aug 2020 14:14:28 GMT):
we can discuss that consensus engine api here for sure, and there is an opportunity to take input from iroha's requirements as we go along for the new api
lebdron (Tue, 04 Aug 2020 14:19:11 GMT):
It might be easier for us currently to finish the integration of 1.0 API, so that the internship project will be complete. This will allow us to see how it changed our pipeline, and possible points of improvement. And this ideas might fit well for 2.0 API.
Do you think it will be possible to set up a voice/video meeting this or next week? It might be easier for us to ask specific questions from the point of view of Iroha pipeline in this way. If not, the chat here is also OK.
lebdron (Tue, 04 Aug 2020 14:19:11 GMT):
It might be easier for us currently to finish the integration of 1.0 API, so that the internship project will be complete. This will allow us to see how it changed our pipeline, and possible points of improvement. And these ideas might fit well for 2.0 API.
Do you think it will be possible to set up a voice/video meeting this or next week? It might be easier for us to ask specific questions from the point of view of Iroha pipeline in this way. If not, the chat here is also OK.
amundson (Tue, 04 Aug 2020 14:25:05 GMT):
Sounds like a good plan, I guess I just wanted to point out where we are going, but yes, with that timeline targeting the existing consensus engines is best.
amundson (Tue, 04 Aug 2020 14:26:50 GMT):
We can get a call setup for sure
ltseeley (Tue, 04 Aug 2020 14:27:33 GMT):
Yeah, I'd be happy to join a call and answer any questions
lebdron (Tue, 04 Aug 2020 14:47:20 GMT):
Thank you for the opportunity! From our side, some time next week would be perfect, as me and @Mikhail_Boldyrev need to look more into Sawtooth code. Do you think we should proceed with communication about details here, have a separate channel, or by email?
ltseeley (Tue, 04 Aug 2020 15:46:35 GMT):
This channel is probably fine
lebdron (Tue, 11 Aug 2020 09:11:58 GMT):
@ltseeley @Mikhail_Boldyrev @adamos2468 Hello! How about scheduling a call this Thursday or Friday?
ltseeley (Tue, 11 Aug 2020 13:42:34 GMT):
Thursday would be better for me, but Friday would work too. @amundson ^
amundson (Tue, 11 Aug 2020 15:07:12 GMT):
I could probably make either work. I'd prefer sometime between 10am-4pm US/Central
lebdron (Tue, 11 Aug 2020 19:12:37 GMT):
I would suggest Thursday 11am CDT then. @Mikhail_Boldyrev @adamos2468 please confirm
adamos2468 (Tue, 11 Aug 2020 20:21:37 GMT):
@lebdron fine by me
Mikhail_Boldyrev (Tue, 11 Aug 2020 21:27:36 GMT):
ok for me
ltseeley (Thu, 13 Aug 2020 13:38:54 GMT):
@lebdron do you have a link for the call today?
lebdron (Thu, 13 Aug 2020 13:42:11 GMT):
Does zoom work for you? We have a variety of options...
lebdron (Thu, 13 Aug 2020 13:42:35 GMT):
Like jitsi, bigbluebutton, google meet
ltseeley (Thu, 13 Aug 2020 14:00:26 GMT):
Zoom would be perfect
Glenn_Gluwa (Thu, 27 Aug 2020 15:28:48 GMT):
Has joined the channel.
Glenn_Gluwa (Thu, 27 Aug 2020 15:28:48 GMT):
publisher polling
infrared (Thu, 10 Sep 2020 17:45:40 GMT):
Has joined the channel.
binunalex (Tue, 06 Oct 2020 16:12:36 GMT):
Has joined the channel.
binunalex (Tue, 06 Oct 2020 16:12:37 GMT):
Dear Friends, I am Alex Binun, a new member of the channel. I am developing a new consensus engine, in Python. Could you recommend me Python implementations of consensus engines ? I saw only Rust implementations. Thanks a lot!
amundson (Tue, 06 Oct 2020 16:33:21 GMT):
poet is the only one, though it is not simple
binunalex (Wed, 07 Oct 2020 06:29:51 GMT):
Thanks a lot, https://github.com/hyperledger/sawtooth-poet is indeed in Python. I know Python quite good so hope it will help
binunalex (Wed, 07 Oct 2020 14:03:19 GMT):
Dear Friends, for all Sawtooth components (POET, PBFT, Core ...) only Docker-based install-from-source instructions are available, they are in BUILD.md files. Are there non-dockerized install-from-source instructions. Fr example, I want to perform an Ubuntu 18 single-node installation and launch an own consensus engine. Thanks, Alex
binunalex (Wed, 14 Oct 2020 14:14:18 GMT):
Thanks a lot, dear friend. The code is quite understandable. I want to compile POET from the source code and run it in a Docker image, in a Sawtooth session. Actually I want to replace the POET image used here (https://sawtooth.hyperledger.org/docs/core/nightly/1-2/app_developers_guide/sawtooth-default-poet.yaml) by the one build by me from the sources.
binunalex (Wed, 14 Oct 2020 14:14:25 GMT):
How coul dI do this ?
arsulegai (Wed, 14 Oct 2020 17:11:23 GMT):
I would create a Debian package in a container, copy the build on host and install it. See here on how `Dockerfile-installed` are being used. https://github.com/hyperledger/sawtooth-poet/blob/master/docker-compose-installed.yaml
BrunoVavala (Wed, 14 Oct 2020 19:42:07 GMT):
Has left the channel.
MelodyWang (Thu, 15 Oct 2020 06:58:57 GMT):
hi
MelodyWang (Thu, 15 Oct 2020 06:58:59 GMT):
I have a question about SGX simulator, I try to test the sawtooth poet with the SGX simulator. I see when it deals with 6 and 10 nodes, the fork always happens and it wont be resolved automatically
just wonder how many nodes the sawtooth poet support and how many blocks it could handle so far.
amundson (Thu, 15 Oct 2020 14:11:35 GMT):
there are a few reasons you can get forks that don't eventually resolve: 1) if a transaction processor is not deterministic (like using random numbers, system time, external information, etc.); 2) overloaded validators that are working on resolving the forks but fall behind because of CPU constraints; 3) bugs. for low node counts its also important to turn off z test, etc.
MelodyWang (Fri, 16 Oct 2020 06:54:33 GMT):
hi thank you for your reply, appreciate it. @amundson for the reason 1, I am a little bit confusing. Why a non-deterministic transaction will cause fork unresolved. I am using Intkey as transaction processor. I sent the transactions each 3 seconds, then even I stop sending the transaction, the validator still does not resolve the forks. For the last one, I wonder why I need to turn off z test? Z-test will cause the fork not be resolved? Thank you
amundson (Fri, 16 Oct 2020 13:43:07 GMT):
intkey is deterministic
amundson (Fri, 16 Oct 2020 13:51:30 GMT):
the logs should give information on block evaluation
amundson (Fri, 16 Oct 2020 13:52:11 GMT):
z-test can "lock out" nodes from publishing, so for low node counts it can cause the entire network to halt
MelodyWang (Fri, 16 Oct 2020 17:53:04 GMT):
@amundson thank you. it is very confusing to me when I was able to synch 800 blocks with 10 nodes but sometimes it even could not synch any time
MelodyWang (Fri, 16 Oct 2020 17:53:45 GMT):
while I think fork resolver should automatically resolve the fork
crypto_beep (Tue, 03 Nov 2020 09:12:17 GMT):
Has joined the channel.
crypto_beep (Tue, 03 Nov 2020 09:12:17 GMT):
Hi hope you all are doing well. i've a small concern, can we use Raft consensus mechanism in Sawtooth 1.2.4's docker approach ?
amundson (Wed, 04 Nov 2020 14:41:59 GMT):
sure, but you would be better off using pbft
crypto_beep (Wed, 04 Nov 2020 16:24:25 GMT):
ok, Thanks
crypto_beep (Wed, 04 Nov 2020 16:24:25 GMT):
ok, Thanks. I got documentation reagarding POET and PBFT not for RAFT. could you please share your thoughts on it.
crypto_beep (Wed, 04 Nov 2020 16:29:05 GMT):
and In case of POET, only POET CFT works fine not SGX.
amundson (Wed, 04 Nov 2020 16:31:03 GMT):
the raft consensus engine is rather experimental; pbft is well-tested and better supported by the maintainers
amundson (Wed, 04 Nov 2020 16:32:59 GMT):
I'm not sure the state of poet sgx. poet cft is fine but unless someone steps up to support it will be cut when we get to sawtooth 2. pbft will probably be the default.
crypto_beep (Wed, 04 Nov 2020 16:34:08 GMT):
ok , Thanks for your time!
Vikash2601 (Tue, 10 Nov 2020 04:44:22 GMT):
Has joined the channel.
RajaramKannan (Wed, 11 Nov 2020 09:16:35 GMT):
Hi All, in the current PBFT implementation when do the secondaries execute the transactions in the proposed block? Is it after they reach "Prepared" state (i.e when they start to commit) or is it during the "prepare" phase?
ltseeley (Wed, 11 Nov 2020 15:32:45 GMT):
@RajaramKannan transactions are run when a block gets validated. Blocks get validated when the consensus engine tells the validator to check the block with a "check blocks" command. The PBFT consensus engine tells the validator to check a block when it gets a "new block" notification from the validator. This notification can come at any time, whenever the validator receives a new block from another node. Technically, this can happen at any phase; the consensus engine will check any block that could be valid, even if it's not ready to commit that block. When the network is in-sync, a secondary node will generally receive and check a block while it's in the "pre-prepare" phase, before the primary sends out the "Pre-prepare" message. But if a node is out-of-sync with the network, it may receive a block while it's in a different phase; in this case, it would still validate the block (transactions are executed), but won't decide to commit the block until it's ready.
RajaramKannan (Thu, 12 Nov 2020 04:27:59 GMT):
Thanks @ltseeley this provides a lot more clarity. Just one followup, "When the network is in-sync, a secondary node will generally receive and check a block while it's in the "pre-prepare" phase, before the primary sends out the "Pre-prepare" message.", Unless the primary broadcasts the block in the pre-prepare phase, how will the secondary receive it? (unless the "pre-prepare" message is separate from the receipt of the block from the primary?). Just as a background to these questions, we have been live for a few months as a Regulated fractional bond trading exchange (https://bondevalue.com/news/bondevalue-gets-mas-approval-to-operate-the-bondblox-bond-exchange/) . We are currently looking at working on a version 2 of the overall solution and as part of this review each element of the overall current solution (including but not only the blockchain) for improvements in our flow.
ltseeley (Thu, 12 Nov 2020 15:12:59 GMT):
@RajaramKannan you got it, the "pre-prepare" message is separate from receiving the block. Blocks are broadcast automatically by the validator. Consensus doesn't control how blocks are shared on the network. The PBFT consensus engine on the primary node will tell its validator to finalize a block, then the validator will broadcast that block to all other nodes, then the engine will broadcast the "pre-prepare" message to all the secondaries. On the secondary nodes, the validators will receive that block and notify the engines regardless of what phase those engines are in.
RajaramKannan (Thu, 12 Nov 2020 15:19:35 GMT):
thanks @ltseeley, I know you were instrumental in many of the contributions including PBFT and this helps in getting a little bit more under the skin of how things work. Mind if I ask a couple more questions?
ltseeley (Thu, 12 Nov 2020 15:20:00 GMT):
Of course!
RajaramKannan (Thu, 12 Nov 2020 15:25:04 GMT):
sorry couple was a misdirection, it might be 3 :-). 1. I know the TP is run on a transaction multiple times, for example to check for non-determinism. Are those multiple executions as soon as a Validator receives a transaction or is that executed only by the validator that is currently preparing a block? 2. Is the transaction again executed in PBFT in the commit phase (secondaries) or only during the preprepare as discussed above? 3. When a view change happens and the leader changes, if the previous leader had a block already proposed, will the new leader create a new block (and the order of batches in that new block possibly changed from what the previous leader might have ordered?) Asking this one since we have a oracular service for timestamping and re-ordering might create some potential issues esp at higher volumes)
RajaramKannan (Mon, 16 Nov 2020 05:27:03 GMT):
@ltseeley looking forward to your insights on the above...
crypto_beep (Tue, 17 Nov 2020 11:53:04 GMT):
Hi Greetings, I didn't find any doc or wiki how to use Dynamic Consensus With Hyperledger Sawtooth v1.2.4 in Docker approach. could you guys please put your thoughts on it.Thanks!
crypto_beep (Wed, 18 Nov 2020 15:44:08 GMT):
Also I didn't find any Commands to list all Validators Nodes and list all Genesis Nodes in doc or wiki how to Hyperledger Sawtooth v1.2.4 in Docker approach. could you guys please help me out on this.Thanks!
AlexanderZhovnuvaty (Tue, 24 Nov 2020 23:55:25 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=9fiRXNsbXGRK6ft3o) Hi. @amundson Please advise how many nodes can be treated as low for poet? And how to turn off z-test? Thanks in advance.
AlexanderZhovnuvaty (Tue, 01 Dec 2020 13:53:57 GMT):
hi. question about load testing with pbft consensus - throughput value is always jumping. Any reasons why it may happen and potential solution to keep the curve stable (pls see graph below)? Thanks in advance
AlexanderZhovnuvaty (Tue, 01 Dec 2020 13:54:22 GMT):
Clipboard - December 1, 2020 3:54 PM
kartheekgottipati (Sun, 07 Mar 2021 20:01:09 GMT):
Has joined the channel.
SigurdEileraas (Mon, 22 Mar 2021 14:03:52 GMT):
Has joined the channel.
HaakonBusterud (Mon, 22 Mar 2021 14:41:42 GMT):
Has joined the channel.
harrymknight (Tue, 30 Mar 2021 15:08:27 GMT):
Has joined the channel.
lucgerrits (Wed, 21 Apr 2021 08:08:35 GMT):
Has joined the channel.
lucgerrits (Wed, 21 Apr 2021 08:17:43 GMT):
Hello everyone,
I'm testing sawtooth, using PBFT, in a cloud to build a consortium blockchain setup. I have seen that performances (aka, commits per second) are going down the more you add validators (I've tried: 6, 12, and 18 validators). I've also seen when changing `sawtooth.publisher.max_batches_per_block` from 100 to 1000 performances increases.
But what about `sawtooth.consensus.pbft.` settings? I can't find good documentation about finding a good PBFT consensus setting depending on the network & number of validators.
I tried multiple consensus settings and ran the same test, but it's always worse than default settings.
Can someone explain the rule to find the correct PBFT consensus settings?
Kind regards, Luc
amundson (Wed, 21 Apr 2021 19:54:05 GMT):
@lucgerrits sawtooth.consensus.pbft.block_publishing_delay should have a relationship with sawtooth.publisher.max_batches_per_block as the less frequently the blocks are published, the more time to process batches in a block. I don't think any of the other pbft consensus settings are directly performance related.
amundson (Wed, 21 Apr 2021 19:57:29 GMT):
increasing the node count increases the network traffic (n^2); thus with a larger network, it makes sense to slow down the number of blocks because the communication happens when blocks are published. if what you are tuning for is throughput.
amundson (Wed, 21 Apr 2021 20:00:55 GMT):
for testing purposes, you can set sawtooth.publisher.max_batches_per_block very high and observe actual batch count achieved per block. the purpose of sawtooth.publisher.max_batches_per_block is help rate-limit the network as desired.
lucgerrits (Mon, 26 Apr 2021 11:11:42 GMT):
@amundson Hi, the last 7days I've tried to change sawtooth (in PBFT) with various settings (`sawtooth.consensus.pbft.block_publishing_delay` and `sawtooth.publisher.max_batches_per_block`). Variation I tried: `sawtooth.publisher.max_batches_per_block` from 100 to 10000 and `sawtooth.consensus.pbft.block_publishing_delay` from 500 to 5000.
I never get a commit rate highter then 20 commits per second. I have always one transaction inside one batch (can't change that because of use-case).
Is this normal ? Are there cases where people achieve +30 commits per second ? Is It the the fact that I have only one transaction inside one batch ?
Maybe you or somebody else have an idea of achieving hight throughput/hight commit per second ?
lucgerrits (Mon, 26 Apr 2021 13:33:18 GMT):
*Also, is the pending transaction related to `sawtooth.publisher.max_batches_per_block` ? Or any other setting ?
lucgerrits (Mon, 26 Apr 2021 14:13:48 GMT):
Clipboard - April 26, 2021 4:13 PM
MelodyWang (Wed, 28 Apr 2021 05:02:17 GMT):
hello I have one question in PoET, if validator A's duration is shorter then the validator B's duration, however, A's certificate arrives first at each of validator in the network, will B's duration will be updated by each validator as the winner? if yes, any time limitation? or any rule? thanks in advance for the answer
arsulegai (Wed, 28 Apr 2021 19:46:30 GMT):
@MelodyWang conditions 6 & 7 as listed here should stop it https://sawtooth.hyperledger.org/docs/core/releases/1.0/architecture/poet.html#election-phase
amundson (Fri, 30 Apr 2021 17:36:40 GMT):
@lucgerrits possibly you are being limited by either CPU or disk (and possibly specifically disk iops if you are in aws), depending on your system configuration. I don't recall specific numbers anymore from our earlier perf tests but I know physical machines w/ssd did a lot better than aws nodes. but in either case, graphs of iops and cpu should make it obvious if those are the case. if you are, for example, constantly cpu-bound and that's the bottleneck these settings won't change that; if you saw bursts you could possibly make utilization more consistent.
lucgerrits (Sun, 02 May 2021 10:22:25 GMT):
@amundson I see, I have asked my cloud provider (TAS Group) specific information about disk IO and I saw that some bursts of disk IO was going up to 9MB/s but my colleague at the cloud provider said it can go highter than that.
Also for CPU burst it shouldn't be an issue because for example when I set up 6 nodes, there is 32 vCPU available non-stop for each node, with 30GB RAM.
My last results I found are:
batchperblock=100000 => 14commits/s
batchperblock=1000 => 18commits/s
lucgerrits (Sun, 02 May 2021 10:22:25 GMT):
@amundson I see, I have asked my cloud provider (TAS Group) specific information about disk IO and I saw that some bursts of disk IO was going up to 9MB/s but my colleague at the cloud provider said it can go highter than that.
Also for CPU burst it shouldn't be an issue because for example when I set up 6 nodes, there is 32 vCPU available non-stop for each node, with 30GB RAM.
My last results I found are using 6nodes:
batchperblock=100000 => 15commits/s (very high rejects)
batchperblock=1000 => 18commits/s (high rejects)
batchperblock=300 => 19commits/s (very low rejects)
batchperblock=100 => 21commits/s (low rejects)
But I though more batchperblock should get more throughput ?
I'm more confused then ever.
lucgerrits (Sun, 02 May 2021 10:22:25 GMT):
@amundson I see, I have asked my cloud provider (TAS Group) specific information about disk IO and I saw that some bursts of disk IO was going up to 9MB/s but my colleague at the cloud provider said it can go higher than that.
Also for CPU burst it shouldn't be an issue because for example when I set up 6 nodes, there is 32 vCPU available non-stop for each node, with 30GB RAM.
My last results I found are using 6nodes:
batchperblock=100000 => 15commits/s (very high rejects)
batchperblock=1000 => 18commits/s (high rejects)
batchperblock=300 => 19commits/s (very low rejects)
batchperblock=100 => 21commits/s (low rejects)
But I though more batchperblock should get more throughput ?
I'm more confused then ever.
If you'd like I can provide lot more data/graph if you are interested.
lucgerrits (Sun, 02 May 2021 10:22:25 GMT):
@amundson I see, I have asked my cloud provider (TAS Group) specific information about disk IO and I saw that some bursts of disk IO was going up to 9MB/s but my colleague at the cloud provider said it can go higher than that.
Also for CPU burst it shouldn't be an issue because for example when I set up 6 nodes, there is 32 vCPU available non-stop for each node, with 30GB RAM.
My last results I found are using 6nodes:
batchperblock=100000 => 15commits/s (very high rejects)
batchperblock=1000 => 18commits/s (high rejects)
batchperblock=300 => 19commits/s (very low rejects)
batchperblock=100 => 21commits/s (low rejects)
batchperblock=50 => 19commits/s (some rejects)
batchperblock=5 => 10commits/s (high rejects)
But I though more batchperblock should get more throughput ?
I'm more confused then ever.
If you'd like I can provide lot more data/graph if you are interested.
lucgerrits (Sun, 02 May 2021 10:22:25 GMT):
@amundson I see, I have asked my cloud provider (TAS Group) specific information about disk IO and I saw that some bursts of disk IO was going up to 9MB/s but my colleague at the cloud provider said it can go higher than that.
Also for CPU burst it shouldn't be an issue because for example when I set up 6 nodes, there is 32 vCPU available non-stop for each node, with 30GB RAM.
My last results I found are using 6nodes:
batchperblock=100000 => 15commits/s (very high rejects)
batchperblock=1000 => 18commits/s (high rejects)
batchperblock=300 => 19commits/s (very low rejects)
batchperblock=100 => 21commits/s (low rejects)
batchperblock=50 => 19commits/s (some rejects)
batchperblock=10 => 10commits/s (high rejects)
But I though more batchperblock should get more throughput ?
I'm more confused then ever.
If you'd like I can provide lot more data/graph if you are interested.
lucgerrits (Sun, 02 May 2021 12:21:00 GMT):
PS: I have read about back pressure: https://sawtooth.hyperledger.org/faq/rest/#what-is-the-back-pressure-test. I'm trying to change the QUEUE_MULTIPLIER but can't find the line of code...If I've news I'll tell you.
amundson (Mon, 03 May 2021 13:47:02 GMT):
@lucgerrits for disk, it is usually IOPS that matter more than throughput. for CPU, it doesn't matter how many cores you have, it matters if the core(s) it is using are at 100%. the batches per block only matter up to a point--keep in mind that it is a max, and if you set it really high then the actual batches per block will be determined by disk and cpu, with a little bit of inefficiency since the process will be resource starved. iirc, you can use "sawtooth block list" to see the actual batches per block.
amundson (Mon, 03 May 2021 13:48:07 GMT):
It would be a surprising result if CPU wasn't 100% (for one of the cores).
lucgerrits (Tue, 04 May 2021 07:54:19 GMT):
@amundson Intresting, I have asked about IOPS limitation directly to my contact at my cloud provider.
About CPU you are right: I have maxed 1 CPU on average for each blockchain node (see picture bellow).
So it seems IOPS is the hitting the limit ? And it isn't possible for sawtooth to multithread or something to increase perfs ?
lucgerrits (Tue, 04 May 2021 07:54:19 GMT):
@amundson Intresting, I have asked about IOPS limitation directly to my contact at my cloud provider.
About CPU you are right: I have maxed 1 CPU on average for each blockchain node (see picture bellow).
So it seems IOPS is the hitting the limit ? And it isn't possible for sawtooth to multithread or something to increase perfs ?
Edit: I added the cloud provider screenshots bellow. Cloud provider said I'm far from the disks limitation. Any idea on whats happening ?
lucgerrits (Tue, 04 May 2021 07:54:30 GMT):
Clipboard - May 4, 2021 9:54 AM
lucgerrits (Tue, 04 May 2021 08:15:28 GMT):
Resources with multiple tests in 3days period
lucgerrits (Tue, 04 May 2021 08:16:43 GMT):
Another screenshot of disk utilisation from cloud provider
amundson (Tue, 04 May 2021 13:58:06 GMT):
The validator itself is limited by Python's GIL and you will only see a little over a single core in use. Its all highly threaded but the GIL's locking synchronizes everything often enough that it becomes the limitation. This is one of the reasons we are moving to Rust on the main branch.
lucgerrits (Tue, 04 May 2021 15:26:45 GMT):
I see, thanks for the info.
One last question, is there an idea on the release of the Rust version of the validator ? Or a roadmap of the next releases ?
amundson (Wed, 05 May 2021 13:52:15 GMT):
The main branch will become sawtooth 2.0 when its finished. our next pieces of work will be replacing consensus with a library and then replacing networking with libsplinter. I don't have an estimate on how long that takes, but my team is planning to shift focus to these things in about a month. I'm hoping this year.
Chengevo (Sat, 08 May 2021 12:34:24 GMT):
Has joined the channel.
Chengevo (Sat, 08 May 2021 12:34:24 GMT):
Hi everyone, I have a question regarding sawtooth-pbft. When received a `BlockValid` message from the validator, the node need to verify the seal for that block. My question is, since this a new block sent from the validator and haven't been finalized by the consensus engine, where did the seal of that block comes from.
Chengevo (Sat, 08 May 2021 13:20:35 GMT):
@amundson
amundson (Mon, 10 May 2021 16:20:17 GMT):
been a while since we stalked about that that but iirc, the previous block would be the one with a seal
Chengevo (Wed, 12 May 2021 01:22:20 GMT):
Thank you for your reply Amundson, but I'm still not quite understand. If I understand it correctly, a seal is created by the consensus engine when a block can be finalized, and this seal will be send to the validator. Is it that when a new block sent from the validator, the seal it carries is actually the seal of the previous block?
amundson (Wed, 12 May 2021 14:03:37 GMT):
I'm not sure I answered the question you asked; have you seen this - https://github.com/hyperledger/sawtooth-rfcs/blob/main/text/0030-pbft-consensus-seal.md
Chengevo (Thu, 13 May 2021 07:43:42 GMT):
I didn't notice this RFC repo yet. I'll read this doc, hopefully it can answer my question :).
Chengevo (Thu, 13 May 2021 07:43:42 GMT):
I didn't notice this RFC repo before. I'll read this doc, hopefully it can answer my question :).
Chengevo (Thu, 13 May 2021 07:45:57 GMT):
Currently I'm implementing a PBFT engine and trying to integrate it into the Substrate framework
Chengevo (Thu, 13 May 2021 07:45:57 GMT):
Currently I'm implementing a PBFT engine(for learning purpose) and trying to integrate it into the Substrate framework
pushkarb (Tue, 25 May 2021 15:44:50 GMT):
Has joined the channel.
nage (Mon, 21 Jun 2021 17:29:47 GMT):
Has left the channel.
AtulPandeySaw (Tue, 20 Jul 2021 05:43:23 GMT):
Has joined the channel.
AtulPandeySaw (Tue, 20 Jul 2021 05:43:23 GMT):
Need help ..im setting up sawtooth with pbft consensus engine ..as last step when i bring up the pbft engine it throws received message Err(Timeout)
sawtooth version 1.1.5-devel and pbft version 1.0.3
rafaelmelo (Thu, 29 Jul 2021 18:18:02 GMT):
Has joined the channel.
rafaelmelo (Fri, 30 Jul 2021 19:21:05 GMT):
I'm following this guide https://sawtooth.hyperledger.org/docs/core/releases/latest/sysadmin_guide/pbft_adding_removing_node.html#adding-a-pbft-node-label for adding a new node to my network, but when I create the proposal the members list isn't updated when I list the member's keys. I've made sure that there aren't typos in my command and no error is showed. Any thoughts on this?
robinrob (Thu, 09 Sep 2021 22:11:02 GMT):
Has left the channel.
mjalalzai (Mon, 04 Oct 2021 07:22:34 GMT):
Hello everyone. I am Mohammad Mussadiq Jalalzai. The lead Author of Fast-HotStuff which is a 2 round more efficient and resilient alternative to the HotStuff/diem consensus protocol.
mjalalzai (Mon, 04 Oct 2021 07:22:34 GMT):
Hello everyone. I am Mohammad Mussadiq Jalalzai and the lead Author of Fast-HotStuff which is a 2 round more efficient and resilient alternative to the HotStuff/diem consensus protocol.
mjalalzai (Mon, 04 Oct 2021 07:22:34 GMT):
Hello everyone. I am Mohammad Mussadiq Jalalzai and the lead author of Fast-HotStuff which is a 2 round more efficient and resilient alternative to the HotStuff/diem consensus protocol.
mjalalzai (Mon, 04 Oct 2021 07:24:22 GMT):
We want to implement this protocol on hyperledger platform. We have a team of 3 developer but non of us has previous development experience over hyperledger.
mjalalzai (Mon, 04 Oct 2021 07:24:22 GMT):
We want to implement this protocol on hyperledger platform. We have a team of 3 developers but none of us has previous development experience of developing a consensus protocol for hyperledger.
mjalalzai (Mon, 04 Oct 2021 07:25:00 GMT):
I wanted to ask if someone is interested who can join and help us with this.
amundson (Mon, 04 Oct 2021 15:51:35 GMT):
@mjalalzai if you want to target sawtooth 1.2, then the PBFT consensus engine implementation is going to be the closest reference
amundson (Mon, 04 Oct 2021 15:53:02 GMT):
we are also starting to work on a consensus library written in Rust, which will be used first by Splinter and then later by Sawtooth 2
amundson (Mon, 04 Oct 2021 15:58:43 GMT):
our goal for the consensus library is that it have a wider appeal to Rust developers than just distributed ledger development; development on it starts in earnest in about a month as we start to implement 2PC and PBFT there
mjalalzai (Mon, 04 Oct 2021 23:37:23 GMT):
Hi @amundson. Though I have not worked with Rust but I can ask other two developers in our team if they have experience with Rust. Moreover, I would like to know the reason behind choosing the PBFT?
mjalalzai (Mon, 04 Oct 2021 23:41:48 GMT):
For example Fast-HotStuff that we proposed is simpler, efficient and resilient. That's is why in the recent version of FB diem they have borrowed some parts from our protocol (reduced three rounds to two, etc.). In case you guys are interested I can work together with explaining the protocol, though it will take me sometime to learn Rust.
mjalalzai (Mon, 04 Oct 2021 23:41:48 GMT):
For example, Fast-HotStuff that we proposed is simpler, efficient, and resilient. That's is why in the recent version of FB diem they have borrowed some parts from our protocol (reduced three rounds to two, etc.). In case you guys are interested I can work together with explaining the protocol, though it will take me some time to learn Rust.
amundson (Tue, 19 Oct 2021 13:39:14 GMT):
@mjalalzai the consensus library will have implementations of several (maybe someday many) algorithms, we are just starting with 2PC and PBFT because those are the ones we use today.
ArunavaManna (Wed, 16 Feb 2022 17:17:42 GMT):
Has joined the channel.
ArunavaManna (Wed, 16 Feb 2022 17:17:43 GMT):
sudo apt-get install -y sawtooth sawtooth-pbft-engine why i am not able to install the pbft engine using this command on ubuntu 16.04 xenial version?
ArunavaManna (Wed, 16 Feb 2022 17:19:21 GMT):
[ ](https://chat.hyperledger.org/channel/sawtooth-consensus-dev?msg=uC54qBnvaNyHb798T) hello, may i know are you using docker or normal ubuntu?
rjones (Wed, 23 Mar 2022 17:35:54 GMT):
rjones (Wed, 23 Mar 2022 17:35:54 GMT):
rjones (Wed, 23 Mar 2022 17:35:54 GMT):
