jsmitchell (Thu, 02 Feb 2017 18:49:39 GMT):
Welcome to the Sawtooth Lake channel!

agunde (Thu, 02 Feb 2017 18:50:40 GMT):
Has joined the channel.

amundson (Thu, 02 Feb 2017 18:51:05 GMT):
Has joined the channel.

amundson (Thu, 02 Feb 2017 18:54:03 GMT):
@jsmitchell - can you make others owners as well?

jsmitchell (Thu, 02 Feb 2017 18:54:14 GMT):
let me see

jsmitchell (Thu, 02 Feb 2017 18:54:31 GMT):
amundson

MicBowman (Thu, 02 Feb 2017 19:40:34 GMT):
Has joined the channel.

MicBowman (Thu, 02 Feb 2017 19:41:41 GMT):
woot woot

Ratnakar (Thu, 02 Feb 2017 20:07:26 GMT):
Has joined the channel.

timblankers (Thu, 02 Feb 2017 23:28:10 GMT):
Has joined the channel.

C0rWin (Fri, 03 Feb 2017 00:11:11 GMT):
Has joined the channel.

salmanbaset (Fri, 03 Feb 2017 01:21:34 GMT):
Has joined the channel.

Donald Liu (Fri, 03 Feb 2017 02:18:54 GMT):
Has joined the channel.

didnotgetagoodname (Fri, 03 Feb 2017 03:57:51 GMT):
Has joined the channel.

JonathanLevi (Fri, 03 Feb 2017 07:50:02 GMT):
Has joined the channel.

silliman (Fri, 03 Feb 2017 13:58:13 GMT):
Has joined the channel.

danacr (Fri, 03 Feb 2017 14:57:19 GMT):
Has joined the channel.

marcusvcs (Fri, 03 Feb 2017 18:47:51 GMT):
Has joined the channel.

fz (Fri, 03 Feb 2017 19:32:23 GMT):
Has joined the channel.

karkal (Fri, 03 Feb 2017 19:57:29 GMT):
Has joined the channel.

stanliberman (Fri, 03 Feb 2017 20:44:06 GMT):
Has joined the channel.

ericmvaughn (Fri, 03 Feb 2017 22:07:56 GMT):
Has joined the channel.

dourada (Fri, 03 Feb 2017 23:24:47 GMT):
Has joined the channel.

genggjh (Sat, 04 Feb 2017 03:46:10 GMT):
Has joined the channel.

guruprasath (Sat, 04 Feb 2017 06:13:01 GMT):
Has joined the channel.

schwentker2 (Sun, 05 Feb 2017 15:51:31 GMT):
Has joined the channel.

ray (Mon, 06 Feb 2017 04:42:38 GMT):
Has joined the channel.

david.peyronnin (Mon, 06 Feb 2017 09:54:28 GMT):
Has joined the channel.

naddison (Mon, 06 Feb 2017 11:23:53 GMT):
Has joined the channel.

karan-bharadwaj (Mon, 06 Feb 2017 17:56:15 GMT):
Has joined the channel.

jsmitchell (Mon, 06 Feb 2017 18:44:12 GMT):
MicBowman

kelly_ (Mon, 06 Feb 2017 19:51:34 GMT):
Has joined the channel.

skbodwell (Tue, 07 Feb 2017 00:48:42 GMT):
Has joined the channel.

Dan (Tue, 07 Feb 2017 04:03:42 GMT):
Has joined the channel.

Dan (Tue, 07 Feb 2017 04:04:28 GMT):
I can has owner?

foobarometer (Tue, 07 Feb 2017 05:21:09 GMT):
Has joined the channel.

boydjohnson (Tue, 07 Feb 2017 13:57:18 GMT):
Has joined the channel.

barinjato (Tue, 07 Feb 2017 14:12:18 GMT):
Has joined the channel.

reachk.raj (Tue, 07 Feb 2017 14:13:35 GMT):
Has joined the channel.

amundson (Tue, 07 Feb 2017 14:31:52 GMT):
Dan

Dan (Tue, 07 Feb 2017 14:33:55 GMT):
All your sawteeth are belong to me now!

adamludvik (Tue, 07 Feb 2017 14:55:51 GMT):
Has joined the channel.

cbf (Tue, 07 Feb 2017 15:16:08 GMT):
Has joined the channel.

pschwarz (Tue, 07 Feb 2017 15:16:08 GMT):
Has joined the channel.

TomBarnes (Tue, 07 Feb 2017 17:47:49 GMT):
Has joined the channel.

TomBarnes (Tue, 07 Feb 2017 17:48:30 GMT):
shiny

TomBarnes (Tue, 07 Feb 2017 17:52:39 GMT):
is there a desktop client? I downloaded the Rocket client but it doesn't seem to connect to this channel.

pschwarz (Tue, 07 Feb 2017 17:57:12 GMT):
https://rocket.chat/download

pschwarz (Tue, 07 Feb 2017 17:57:19 GMT):
@Tom Barnes --^

mfford (Tue, 07 Feb 2017 19:07:45 GMT):
Has joined the channel.

amundson (Tue, 07 Feb 2017 19:45:36 GMT):
```ubuntu@ubuntu-xenial:/project/sawtooth-core$ sawtooth cluster start -P tp_intkey_python Starting: validator-000 Starting: validator-001 Starting: validator-002 Starting: validator-003 Starting: validator-004 Starting: validator-005 Starting: validator-006 Starting: validator-007 Starting: validator-008 Starting: validator-009 Pulling validator (sawtooth-validator:latest)... ERROR: repository sawtooth-validator not found: does not exist or no pull access Error: Docker images not built: tp_intkey_python, sawtooth-validator. Try running "sawtooth docker build tp_intkey_python sawtooth-validator" ```

amundson (Tue, 07 Feb 2017 19:45:52 GMT):
anyone else seeing that?

amundson (Tue, 07 Feb 2017 19:47:07 GMT):
@adamludvik ^

amundson (Tue, 07 Feb 2017 19:57:17 GMT):
apparently following the 'Try running' statement provided works

amundson (Tue, 07 Feb 2017 19:57:43 GMT):
I had mistakenly though not providing the arguments implied 'all'

amundson (Tue, 07 Feb 2017 20:00:40 GMT):
probably be useful to have one of the cluster subcommands tell you connection information

amundson (Tue, 07 Feb 2017 20:00:46 GMT):
and for cluster to startup the rest api

amundson (Tue, 07 Feb 2017 20:02:31 GMT):
```ubuntu@ubuntu-xenial:/project/sawtooth-core$ xo create game000 Response: {} ```

amundson (Tue, 07 Feb 2017 20:04:47 GMT):
so that's an invalid transaction, what should it print out? nothing? If I give it --wait, it does the same thing, but in that case, it should probably print out an error.

tbrooke (Tue, 07 Feb 2017 20:22:06 GMT):
Has joined the channel.

jsmitchell (Tue, 07 Feb 2017 20:25:55 GMT):
it's a valid transaction from the perspective of the client, right? It's only invalid because a game of that name already exists?

jsmitchell (Tue, 07 Feb 2017 20:28:25 GMT):
in 0-7 we had the capability to do validity checks on submission based on the best knowledge the validator had

beauson45 (Tue, 07 Feb 2017 21:02:13 GMT):
Has joined the channel.

SivaKannan (Tue, 07 Feb 2017 21:05:47 GMT):
Has joined the channel.

amundson (Tue, 07 Feb 2017 21:41:09 GMT):
invalid because it's a duplicate or the validator doesn't handle xo transactions, stuff like that

amundson (Tue, 07 Feb 2017 21:43:33 GMT):
the 0-7 model isn't necessarily as desirable as just returning the state as it goes through the various normal stages

rcampos@altus.cr (Tue, 07 Feb 2017 22:04:23 GMT):
Has joined the channel.

Rokkinchic (Wed, 08 Feb 2017 03:17:59 GMT):
Has joined the channel.

kansi (Wed, 08 Feb 2017 03:59:54 GMT):
Has joined the channel.

sachinsurvase (Wed, 08 Feb 2017 04:21:53 GMT):
Has joined the channel.

PacoEcotrust (Wed, 08 Feb 2017 09:24:34 GMT):
Has joined the channel.

jvsteiner (Wed, 08 Feb 2017 11:02:08 GMT):
Has joined the channel.

o_drugan (Wed, 08 Feb 2017 14:28:09 GMT):
Has joined the channel.

Dan (Wed, 08 Feb 2017 14:52:15 GMT):
I put up a new sawtooth_signer option yesterday. It changes the signer interface a bit to simplify it. This is all new architecture (0.8) related. Nothing to do with the legacy / stable stuff (0.7).https://github.com/hyperledger/sawtooth-core/pull/64

Dan (Wed, 08 Feb 2017 14:54:13 GMT):
I've elected to drop support for pubkey recovery for now. If someone has a need for that let me know. The intent moving forward is all our signed messages will have the public key included in the signed message. (See the protobuf files here: https://github.com/hyperledger/sawtooth-core/tree/master/protos)

Dan (Wed, 08 Feb 2017 14:58:06 GMT):
Intended usage is just to generate priv/pub keys and sign/verify messages. The encoding/decoding methods that were there before become private and are handled implicitly by the remaining calls. details in commit msg. http://bit.ly/2kmnyhq.

Dan (Wed, 08 Feb 2017 14:59:02 GMT):
The main reason to do all this was to adopt the libsecp256k1 implementation in lieu of the slower python implementations.

amundson (Wed, 08 Feb 2017 14:59:39 GMT):
@jsmitchell and I were talking about language bindings (Java, Javascript, etc.) and the difficulty in supporting this vs. the old bitcoin library approach

amundson (Wed, 08 Feb 2017 14:59:52 GMT):
sounded like this might be easier as there is less encoding?

Dan (Wed, 08 Feb 2017 15:03:32 GMT):
I hope so. :) This c library spits out the signatures in DER encoding which is a little more standard than the bitcoin formats. The latter encode some things that are specific to bitcoin whereas DER is just line encoding for numbers.

Dan (Wed, 08 Feb 2017 15:04:46 GMT):
So the protobufs are mostly handling hex strings which should be able to be decoded without handling bitcoinisms.

Dan (Wed, 08 Feb 2017 15:06:05 GMT):
So that's true of the signatures. But for now I've kept the keys WIF (bitcoin) encoded because I wasn't sure about legacy client interaction with .8.

Dan (Wed, 08 Feb 2017 15:07:02 GMT):
I could strip that out and make them DER encoded if we don't have any need for WIF. That would also eliminate I think the last remaining dependency on pybitcointools.

amundson (Wed, 08 Feb 2017 15:10:36 GMT):
if we change the key format, we should pick a new extension and update keygen

boydjohnson (Wed, 08 Feb 2017 15:16:03 GMT):
My Pr that refactors Interconnect and network into one implementation that get different behavior based on the dispatcher that is used is up. I commented for cian montgomery on the pr which commits touch journal. Since it is a rather large PR that is a significant refactor, I would be looking for further reviewers.

zac (Wed, 08 Feb 2017 15:24:37 GMT):
Has joined the channel.

boydjohnson (Wed, 08 Feb 2017 15:55:20 GMT):
https://github.com/hyperledger/sawtooth-core/pull/65 Here's the PR.

amundson (Wed, 08 Feb 2017 16:09:56 GMT):
@boydjohnson - what's the story on MessageList in the python sdk. was that just broken before or did you change the approach?

boydjohnson (Wed, 08 Feb 2017 16:12:39 GMT):
I thought since we were just wrapping 1 message in a messagelist it didn't serve a purpose so I changed that in the interconnect code. If we try to sweep messages in or out up to a certain number of messages maybe it would make more sense. Also pyzmq has send_multipart where you can send a list of things so it would be [msg1_bytes, msg2_bytes... making MessageList not needed.

amundson (Wed, 08 Feb 2017 16:15:57 GMT):
makes sense, we added that when we were attempting to make up for grpc slowness, IIRC

jsmitchell (Wed, 08 Feb 2017 16:16:45 GMT):
@boydjohnson: this ProcessPoolExecutor strategy for parallel signature validation is sweet

amundson (Wed, 08 Feb 2017 16:29:41 GMT):
@boydjohnson - scanned through quickly, looks very nice

boydjohnson (Wed, 08 Feb 2017 16:31:00 GMT):
Thanks, the interconnect.py file is longer than advertised, but I think it is a nice implementation.

Dan (Wed, 08 Feb 2017 18:12:26 GMT):
For 0.8... once legacy code is removed will we have a python3 requirement? i.e. the system is not expected to be py2 compatible?

amundson (Wed, 08 Feb 2017 18:45:53 GMT):
python3 only

jjason (Wed, 08 Feb 2017 18:46:26 GMT):
Has joined the channel.

jsmitchell (Wed, 08 Feb 2017 19:12:03 GMT):
@boydjohnson: you probably want to drop commit e9d67f from your branch

jsmitchell (Wed, 08 Feb 2017 19:12:15 GMT):
removing that unit_signing.yaml was an error

boydjohnson (Wed, 08 Feb 2017 19:12:29 GMT):
Yeah, I'll drop that commit. Thanks.

jsmitchell (Wed, 08 Feb 2017 19:12:40 GMT):
@adamludvik merged in the fix

boydjohnson (Wed, 08 Feb 2017 19:12:56 GMT):
Cool, I'll also rebase on current master

rjones (Wed, 08 Feb 2017 19:15:56 GMT):
Has joined the channel.

rjones (Wed, 08 Feb 2017 19:18:03 GMT):
I have two small documentation PRs - https://github.com/hyperledger/sawtooth-docs/pull/11 and https://github.com/hyperledger/sawtooth-core/pull/66 . I noticed PR66 failed Jenkins, but I can't see why. Could I sign up for a jenkins account somewhere?

jsmitchell (Wed, 08 Feb 2017 19:18:58 GMT):
we had a commit which passed in our internal jenkins but broke in external jenkins

rjones (Wed, 08 Feb 2017 19:19:19 GMT):
OK, thanks.

jsmitchell (Wed, 08 Feb 2017 19:19:36 GMT):
if you rebase and have c464e0d6, hopefully the build should succeed

boydjohnson (Wed, 08 Feb 2017 19:25:23 GMT):
@jsmitchell Because of the time delay between internal and external github I rebased on internal master, and pushed some commits that aren't yet on external github to my PR. I will rebase once they have been pushed out.

rbuysse (Wed, 08 Feb 2017 19:41:50 GMT):
Has joined the channel.

jsmitchell (Wed, 08 Feb 2017 20:35:30 GMT):
@boydjohnson: are you seeing a ~10 sec delay on intkey load before stuff starts getting processed?

jsmitchell (Wed, 08 Feb 2017 20:35:33 GMT):
```[20:33:19 DEBUG interconnect] receiving CLIENT_BATCH_SUBMIT_REQUEST message [20:33:20 DEBUG interconnect] receiving CLIENT_BATCH_SUBMIT_REQUEST message [20:33:20 DEBUG interconnect] receiving CLIENT_BATCH_SUBMIT_REQUEST message [20:33:31 DEBUG interconnect] sending GOSSIP_MESSAGE to None [20:33:31 DEBUG interconnect] sending GOSSIP_MESSAGE to None [20:33:31 DEBUG interconnect] sending GOSSIP_MESSAGE to None ```

boydjohnson (Wed, 08 Feb 2017 20:39:59 GMT):
Yeah, that because signature verification. It doesn't get broadcast until we verify the signatures.

boydjohnson (Wed, 08 Feb 2017 20:39:59 GMT):
Yeah, that's because signature verification. It doesn't get broadcast until we verify the signatures.

jsmitchell (Wed, 08 Feb 2017 20:41:09 GMT):
so, all the signature verification for all batches is happening before the first batch is sent to the publisher?

boydjohnson (Wed, 08 Feb 2017 20:43:13 GMT):
Yeah, that is maybe a data model problem. To simplify the dispatcher and handlers we constrain it to sending the same input to each handler. BatchLists come in and the first batchlist (for intkey load: 1 batch 100 txns) gets verified and isn't sent to the broadcast handler until it is the whole batchlist is verified.

jsmitchell (Wed, 08 Feb 2017 20:44:42 GMT):
I'm also seeing an ordering problem with intkey load now, presumably due to the lack of formal dependencies (either on the create side, or on the publishing side, or probably both) between set and inc/dec.

jsmitchell (Wed, 08 Feb 2017 20:44:59 GMT):
so, inc/dec transactions are coming in first before the set and are being declared invalid

boydjohnson (Wed, 08 Feb 2017 20:48:04 GMT):
Yeah, I have seen that. It is on both the creating the batches and internally. The ordering problem is that the first batchlist is bigger than the second batchlist, so it takes longer in signature verification. Also I wrote some test code (not included in the PR) that shows that the dispatcher does not always preserver ordering even on fair handlers.

jsmitchell (Wed, 08 Feb 2017 20:48:56 GMT):
do we have a story for getting dependencies to work on both sides?

boydjohnson (Wed, 08 Feb 2017 20:49:42 GMT):
I have stories for the completer which is my understanding handles both batch and block dependencies, but not for the intkey create_batch.

boydjohnson (Wed, 08 Feb 2017 20:54:06 GMT):
@pschwarz I think I was bitten by the timeout error on the intkey smoke test. That was one that you were talking about.

pschwarz (Wed, 08 Feb 2017 21:26:05 GMT):
@boydjohnson was that against master?

boydjohnson (Wed, 08 Feb 2017 21:53:31 GMT):
Yeah, external jenkins.

pschwarz (Wed, 08 Feb 2017 22:51:43 GMT):
There is a task to fix this issue, so just run the build again

cianx (Thu, 09 Feb 2017 00:31:11 GMT):
Has joined the channel.

danieldewar (Thu, 09 Feb 2017 10:25:13 GMT):
Has joined the channel.

jonreid (Thu, 09 Feb 2017 11:41:48 GMT):
Has joined the channel.

FlyingTiger (Thu, 09 Feb 2017 11:59:23 GMT):
Has joined the channel.

FlyingTiger (Thu, 09 Feb 2017 11:59:58 GMT):
Hi, guys :)

rbuysse (Thu, 09 Feb 2017 14:02:45 GMT):
Hello!

boydjohnson (Thu, 09 Feb 2017 14:45:10 GMT):
Hi, @FlyingTiger

Dan (Thu, 09 Feb 2017 14:45:52 GMT):
Welcome to RocketChat :D

jsmitchell (Thu, 09 Feb 2017 15:02:28 GMT):
:rocket:

crow15 (Thu, 09 Feb 2017 15:12:22 GMT):
Has joined the channel.

amundson (Thu, 09 Feb 2017 16:28:41 GMT):
@agunde - considering splitting tests/scheduler into tests/scheduler_serial and tests/scheduler_parallel

agunde (Thu, 09 Feb 2017 21:14:34 GMT):
@amundson I think that would be fine. But it would start the precedent of having multiple tests files in a directory (such as test_scheduler) instead of the discussed convention of only having tests.py file in most cases.

amundson (Fri, 10 Feb 2017 00:11:30 GMT):
ok

seanbarclay (Fri, 10 Feb 2017 11:57:43 GMT):
Has joined the channel.

jsmitchell (Fri, 10 Feb 2017 17:23:02 GMT):
@FlyingTiger: are you online?

jsmitchell (Fri, 10 Feb 2017 17:45:14 GMT):
@FlyingTiger: we're trying to spin up a 0-7 instance and are running into a couple of things that a few of your master PRs might fix

toddojala (Fri, 10 Feb 2017 18:30:17 GMT):
Has joined the channel.

toddojala (Fri, 10 Feb 2017 18:30:29 GMT):
hello

boydjohnson (Fri, 10 Feb 2017 18:35:34 GMT):
So the GenesisController PR was merged and I would like to alert people to the fact that validators started up by the cluster command will behave in a different way. validator-000 is the genesis node and if you "intkey load --url 127.0.0.1:40002" (in a three node network) it will broadcast batches out to the other validators but only validator-000 will make blocks since it alone has genesis. Also I have seen intermittent failures 1/5 on xo-smoke when running bin/run_tests that probably has to do with timing issues.

rjones (Fri, 10 Feb 2017 19:41:32 GMT):
I tagged all of the github/sawtooth* repos using the new `topics` feature from github. LMK if you want more/different tags. for instance if you look here: https://github.com/hyperledger/sawtooth-core you'll see the topics under the description.

jsmitchell (Fri, 10 Feb 2017 20:00:30 GMT):
sweet, thanks @rjones

Dan (Fri, 10 Feb 2017 20:42:40 GMT):
yeah thanks... reminds me we might have some old repos out there. Should we purge those?

Dan (Fri, 10 Feb 2017 20:42:51 GMT):
e.g. https://github.com/hyperledger/sawtooth-validator

rjones (Fri, 10 Feb 2017 21:19:21 GMT):
@Dan I can migrate them to github/hyperledge-archives if you like, just send me a list

boydjohnson (Fri, 10 Feb 2017 22:11:08 GMT):
Just to advertise to the channel, that when running a single validator node, it is required to run "sawtooth-0.8 admin genesis". And clean the /home/ubuntu directory.

Dan (Fri, 10 Feb 2017 22:52:31 GMT):
what do you mean clean the directory? Like after you run that command or you need to clean out some keys before then?

Dan (Fri, 10 Feb 2017 22:53:20 GMT):
thanks @ry. I'll put together a list and see if anyone on the channel is remembering some reason for those to still be there.

Dan (Fri, 10 Feb 2017 22:56:30 GMT):
@all Any reason these repos should NOT be archived and removed from github? sawtooth-arcade, sawtooth-validator, sawtooth-mktplace, sawtooth-dev-tools

boydjohnson (Fri, 10 Feb 2017 23:21:23 GMT):
@Dan Apparently it is just rm ~/block-chain-id, and then sawtooth-0.8 admin genesis && validator -vv.

PacoEcotrust (Sun, 12 Feb 2017 14:17:25 GMT):
Maybe that was treated before, sorry, I would like to ask fi the 8 version is ready to test? thanks

PacoEcotrust (Sun, 12 Feb 2017 14:17:25 GMT):
Maybe that was treated before, sorry, I would like to ask if the 8 version is ready to test? thanks

amundson (Sun, 12 Feb 2017 16:01:32 GMT):
@PacoEcotrust - we are targeting late April for the core of 0.8 to have the minimum features required to run a network of validators in a non-development setting (tests, PoCs, etc.); shortly following that (maybe a month) we expect 0.8 to include all interesting features that were present in 0.7 (stats client, etc.)

amundson (Sun, 12 Feb 2017 16:07:54 GMT):
0.8 is far enough along to write transaction families with the SDKs (python, javascript, java), and there is example code for each language. there is also a pattern for testing transaction families in an isolated way. the REST API has enough implementation to write CLI clients for the transaction families (we have a couple, for intkey, xo)

amundson (Sun, 12 Feb 2017 16:09:08 GMT):
but, you are essentially limited to a single validator instance since new journal and consensus modules are currently in implementation

PacoEcotrust (Sun, 12 Feb 2017 20:42:27 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=nr7pHa8WB6Jwvx6jx) @amundson Thanks very much!!

PacoEcotrust (Sun, 12 Feb 2017 20:42:27 GMT):
@amundson Thanks very much!!

PacoEcotrust (Sun, 12 Feb 2017 20:43:18 GMT):
Thanks very much

amundson (Sun, 12 Feb 2017 23:32:44 GMT):
master builds are broken

amundson (Sun, 12 Feb 2017 23:36:07 GMT):
https://build.sawtooth.me/job/Sawtooth-Hyperledger/job/sawtooth-core/job/master/42/console

Dan (Mon, 13 Feb 2017 18:58:54 GMT):
@rjones I don't see anyone begging for those old repos, so I think it's safe to archive them. Thanks.

rjones (Mon, 13 Feb 2017 19:00:39 GMT):
@Dan 👍

fz (Mon, 13 Feb 2017 19:08:25 GMT):
I have been trying to follow the sawtooth tutorial at http://intelledger.github.io/tutorial.html and I am at the step of creating an empty cookie jar for Chocolate Chip cookies. I get the following error when I try to submit a transaction ```//carol> holding reg --name /jars/choc_chip --account /account --asset //mkt/asset/cookie/choc_chip [18:48:34 mktplace.mktplace_communication WARNING] operation failed with response: 400 [18:48:34 mktplace.mktplace_communication WARNING] Error from server, detail information: �eerrorpName isn't validierrorTypewInvalidTransactionErrorfstatus
� [18:48:34 mktplace.mktplace_client ERROR] message post failed for some unusual reason Traceback (most recent call last): File "/project/sawtooth-core/extensions/mktplace/mktplace/mktplace_client.py", line 172, in _sendtxn msg File "/project/sawtooth-core/extensions/mktplace/mktplace/mktplace_communication.py", line 179, in postmsg .format(err_content)) InvalidTransactionError: Error from server: failed to create transaction //carol> ```

fz (Mon, 13 Feb 2017 19:09:00 GMT):
At the validator side. I see the following logs..

fz (Mon, 13 Feb 2017 19:09:10 GMT):
```[18:48:34 INFO base_page] BlockPage.do_get /block [18:48:34 WARNING root_page] /mktplace.transactions.MarketPlace/Transaction - NoResource [18:48:34 WARNING root_page] ['mktplace.transactions.MarketPlace', 'Transaction'] - 2 [18:48:34 INFO forward_page] starting local validation for txn id: 08c204d7252e58dc type: /MarketPlaceTransaction [18:48:34 INFO forward_page] RegisterHolding Updates [18:48:34 INFO forward_page] submitted transaction fails transaction family validation check: /mktplace.transactions.MarketPlace/Transaction; {'PublicKey': None, '__SIGNATURE__': 'G3+IfmIX5YXSWG/ROdch0rF/exTenJePlQlUROiBR+0qN1lY/jTEtyr2fKV8vZ4ZiD+k9V4wAdv5gv7AaqzATvw=', '__TYPE__': '/mktplace.transactions.MarketPlace/Transaction', '__NONCE__': 1487011714.806149, 'Transaction': {'Nonce': 1487011714.776861, 'PublicKey': '04c44f6de186a8eefb676e933a7a0e06408f2ff001369d7547dddce02c354e236fb87687dd2adba3622d913a8d796d8c9e5b8a310e59b1792dcc29d388aa4e9bf2', 'Dependencies': ['c38c98b805db7266', 'e995680600b84fb5', '14d5aabe2193e8d9'], 'Updates': [{'Count': 0L, 'Description': '', 'AssetId': '14d5aabe2193e8d9', 'UpdateType': 'RegisterHolding', 'CreatorId': 'c38c98b805db7266', 'Name': '/jars/choc_chip', 'AccountId': 'e995680600b84fb5'}], 'Signature': 'HE0V43cRcilr5eIdDdGEMCpZfUWlm3+bnQU/o0hTCHI+JP5VhBdFSJbJr/2KLAcZrLZFH3WDi9LEJUnBDZRoggk=', 'TransactionType': '/MarketPlaceTransaction'}} ```

fz (Mon, 13 Feb 2017 19:10:53 GMT):
i am just using carol instead of bob in the example. Everything else is the same.. I am v0-7 branch of sawtooth-lake

fz (Mon, 13 Feb 2017 19:11:18 GMT):
does anyone know why this might be happening?

toddojala (Mon, 13 Feb 2017 19:41:00 GMT):
@fz Taking a look..

fz (Mon, 13 Feb 2017 19:41:54 GMT):
@toddojala thanks..

toddojala (Mon, 13 Feb 2017 19:42:43 GMT):
@fz OK, so you are definitely using v0-7. Was it a "fresh" vagrant machine and environment?

fz (Mon, 13 Feb 2017 19:43:15 GMT):
yes. Its a fresh vagrant machine.

fz (Mon, 13 Feb 2017 19:43:51 GMT):
and environment

boydjohnson (Mon, 13 Feb 2017 19:44:59 GMT):
@fz If I remember correctly names must be globally unique. So if you have a /jars/choc_chip already, somebody else can't have a holding with the same name.

fz (Mon, 13 Feb 2017 19:48:30 GMT):
@boydjohnson got it. I got this error when I tried to create this jars/choc_chip for the first time. I have never created jars/choc_chip before.

boydjohnson (Mon, 13 Feb 2017 19:49:03 GMT):
I'm unsure. You could run with debug logging on (-vv) on the validator and try again.

fz (Mon, 13 Feb 2017 19:50:06 GMT):
sure @boydjohnson. will do.

rjones (Mon, 13 Feb 2017 19:54:20 GMT):
Has left the channel.

jsmitchell (Mon, 13 Feb 2017 20:04:59 GMT):
@boydjohnson: I think this may have to do with our PublicKey issues

jsmitchell (Mon, 13 Feb 2017 20:05:09 GMT):
```{'PublicKey': None```

jsmitchell (Mon, 13 Feb 2017 20:05:38 GMT):
although those changes may not have been merged in yet

boydjohnson (Tue, 14 Feb 2017 14:07:01 GMT):
So I had a thought last night. Right now the flow of CLIENT_BATCH_SUBMIT_REQUESTS either from the rest api or directly from a client goes directly to signature verification and then gossip broadcast and then to the completer which hands it to the journal and then at that point it acks back to the client. I set up the handlers in this order so that signature verification could fail the batchlist and send a error message back to the client. But having the handlers in this order means that for intkey-smoke and xo-smoke integration tests if the ack doesn't get back in 5 seconds, there is an error. A short-term remedy is to ack back as soon as the validator has the batchlist, but because of everything being request-response sending back an error message after you have already sent an ack first doesn't work. @zac I know you are working on batch submit handlers. What are your thoughts?

Dan (Tue, 14 Feb 2017 14:44:46 GMT):
Seems like your existing flow shouldn't be at risk of a 5 sec delay. or does the completer step wait on finding batches? If there's no "wait for something from the network" in that flow then I'd say its better we take a look at why the flow would take 5 seconds rather than changing the flow.

jsmitchell (Tue, 14 Feb 2017 14:45:44 GMT):
two things causing that - 1) large batches with lots of transactions to verify 2) no accelerated verification at the moment

jsmitchell (Tue, 14 Feb 2017 14:46:43 GMT):
@boydjohnson: where does the 5 second timeout come from?

jsmitchell (Tue, 14 Feb 2017 14:51:00 GMT):
is that established in the client?

boydjohnson (Tue, 14 Feb 2017 15:02:55 GMT):
Yeah, the 5 second timeout is in the rest api, the other clients like "intkey load" don't have timeouts.

jsmitchell (Tue, 14 Feb 2017 15:07:40 GMT):
my inclination is for ack to be a network layer response (I received this), rather than an attempt to answer whether the message is processable. I can see an argument for returning something else if the message itself has an error and that can be determined in a small amount of time. Otherwise, I think we're answering a different question about the status of something that was in the message.

jsmitchell (Tue, 14 Feb 2017 15:08:13 GMT):
For example, messages in the future might contain multiple things (batches, let's say)

jsmitchell (Tue, 14 Feb 2017 15:09:07 GMT):
If batch 1 is signed correctly and batch 2 is signed incorrectly, what do we do? Probably best to acknowledge receipt of the message independently from reporting success on batch 1 and error on batch 2

jsmitchell (Tue, 14 Feb 2017 15:10:21 GMT):
Message errors might be along the lines of "can't deserialize" or "unknown message type"

jsmitchell (Tue, 14 Feb 2017 15:10:35 GMT):
(i.e. concerns of the network layer)

boydjohnson (Tue, 14 Feb 2017 15:11:03 GMT):
Yeah, receipt of the message and then later information about the success or failure of that batch seems good. We would have to change handling of futures.

jsmitchell (Tue, 14 Feb 2017 15:11:40 GMT):
I don't know if it's really a future thing

jsmitchell (Tue, 14 Feb 2017 15:12:00 GMT):
it's important to keep messages distinct from their contents

boydjohnson (Tue, 14 Feb 2017 15:13:37 GMT):
So the response to a message would be in most cases (all cases?) ack, but then do we send an error response and who do we send it to, or do we just keep that info for querying?

jsmitchell (Tue, 14 Feb 2017 15:19:37 GMT):
We should talk it over. The 0-7 code polled the web api for status. We could do the same and return either success/error to that query. Being able to subscribe to a set of identifiers for status changes would be quite a bit better.

jsmitchell (Tue, 14 Feb 2017 15:23:45 GMT):
@boydjohnson: so, in that model the network would respond to messages with either Ack or Error indicating success in network receipt and ability to dispatch

boydjohnson (Tue, 14 Feb 2017 15:25:14 GMT):
Yeah, seems good.

jsmitchell (Tue, 14 Feb 2017 15:25:53 GMT):
i don't know, maybe not

jsmitchell (Tue, 14 Feb 2017 15:27:35 GMT):
that probably only applies to things we are delivering to the validator for processing

boydjohnson (Tue, 14 Feb 2017 15:29:23 GMT):
In validator to validator interactions it seems like an ack or "error, what did you send me" response would be mostly what is needed, too.

fz (Tue, 14 Feb 2017 19:24:12 GMT):
Hi everyone. sorry couldn't get back sooner with a followup/update of the cookie exchange tutorial related errors I got yesterday. After a few vagrant destroy && vagrant ups, the errors are not occuring. Here are a couple of observations.I am getting the following warnings whenever execute the following commad. ``` /project/sawtooth-core/docs/source/tutorial/genesis.sh ```

fz (Tue, 14 Feb 2017 19:24:41 GMT):
``` writing file: /home/ubuntu/sawtooth/keys/base000.wif writing file: /home/ubuntu/sawtooth/keys/base000.addr [01:02:18 WARNING signed_object] Identifier dbb0603c6420fe2c of SignedObject has no public key and addr used to verify signature [01:02:18 WARNING signed_object] Identifier 879960d6a3a63c01 of SignedObject has no public key and addr used to verify signature [01:02:18 WARNING signed_object] Identifier a824333e41fbeb9d of SignedObject has no public key and addr used to verify signature [01:02:18 WARNING signed_object] Identifier 9b7d322df5012fa6 of SignedObject has no public key and addr used to verify signature [01:02:18 WARNING signed_object] Identifier 2ad7fc781b6d3112 of SignedObject has no public key and addr used to verify signature [01:02:18 WARNING signed_object] Identifier d172207ba08873f8 of SignedObject has no public key and addr used to verify signature [01:02:18 WARNING signed_object] Identifier d172207ba08873f8 of SignedObject has no public key and addr used to verify signature [01:02:18 WARNING signed_object] Identifier dbb0603c6420fe2c of SignedObject has no public key and addr used to verify signature [01:02:18 WARNING signed_object] Identifier 879960d6a3a63c01 of SignedObject has no public key and addr used to verify signature [01:02:18 WARNING signed_object] Identifier a824333e41fbeb9d of SignedObject has no public key and addr used to verify signature [01:02:18 WARNING signed_object] Identifier 9b7d322df5012fa6 of SignedObject has no public key and addr used to verify signature [01:02:18 WARNING signed_object] Identifier 2ad7fc781b6d3112 of SignedObject has no public key and addr used to verify signature ubuntu@ubuntu- ```

fz (Tue, 14 Feb 2017 19:27:10 GMT):
Also when I try to view Bob's current offer as mentioned in the tutorial.. I see the following empty set

fz (Tue, 14 Feb 2017 19:27:10 GMT):
``` ubuntu@ubuntu-xenial:/project/sawtooth-core$ ./bin/mktclient --name bob --keyfile validator/keys/bob.wif //bob> holding reg --name /batches/choc_chip001 --account /account --asset //mkt/asset/cookie/choc_chip --count 24 transaction 8413362e93b4a795 submitted //bob> exchangeoffer reg --output /batches/choc_chip001 --input /USD --ratio 2 1 --name /choc_chip_sale transaction 9c91de5624d2e7a0 submitted //bob> holdings --creator //bob 1000 //bob/USD 24 //bob/batches/choc_chip001 1 //bob/holding/token 0 //bob/jars/choc_chip //bob> offers --creator //bob Ratio Input Asset (What You Pay) Output Asset (What You Get) Name //bob> Ratio Input Asset (What You Pay) Output Asset (What You Get) Name //bob> waitforcommit Waiting for 9c91de5624d2e7a0. Transaction committed. //bob> waitforcommit Waiting for 9c91de5624d2e7a0. Transaction committed. //bob> waitforcommit Waiting for 9c91de5624d2e7a0. Transaction committed. //bob> offers --creator //bob Ratio Input Asset (What You Pay) Output Asset (What You Get) Name //bob> exit ```

fz (Tue, 14 Feb 2017 19:28:38 GMT):
``` ubuntu@ubuntu-xenial:/project/sawtooth-core$ ./bin/mktclient --name alice --keyfile validator/keys/alice.wif //alice> exchange --type ExchangeOffer --src /USD --dst /jars/choc_chip --offers //bob/choc_chip_sale --count 24 transaction f3622545f5c14609 submitted //alice> waitforcommit Waiting for f3622545f5c14609. Transaction committed. //alice> holdings --creator //bob 1000 //bob/USD 24 //bob/batches/choc_chip001 1 //bob/holding/token 0 //bob/jars/choc_chip //alice> holdings --creator //alice 1000 //alice/USD 1 //alice/holding/token 0 //alice/jars/choc_chip //alice> ```

fz (Tue, 14 Feb 2017 19:33:39 GMT):

Message Attachments

fz (Tue, 14 Feb 2017 19:35:39 GMT):
As seen above, when I try to view the offers created by Bob, it returns an empty set

fz (Tue, 14 Feb 2017 19:40:14 GMT):
And there is no change in the effective resulting holdings after the transaction. I am not sure if I am missing something here. The validator logs suggest the transactions are in good shape and it gets committed..

Dan (Tue, 14 Feb 2017 20:54:23 GMT):
Great. Looks like smacking vagrant around got you working ok?

Dan (Tue, 14 Feb 2017 20:55:58 GMT):
n/m your post above on holdings showed up after I'd woken rocket chat up a little more.

fz (Tue, 14 Feb 2017 21:40:25 GMT):
hi Dan. yes. It turned out to be an environment issue..I guess

fz (Tue, 14 Feb 2017 21:43:08 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=jGbMPvKc5qeWSFtK6) @Dan not sure if I understand this comment

amundson (Wed, 15 Feb 2017 03:28:19 GMT):
As some of you may have noticed, we have moved all PR activity to the hyperledger github repo - https://github.com/hyperledger/sawtooth-core/pulls

amundson (Wed, 15 Feb 2017 03:56:12 GMT):
One thing to note is that to re-request review after changes, one has to re-select the reviewers in the pulldown.

PacoEcotrust (Wed, 15 Feb 2017 08:30:56 GMT):
Hi, any one know where is the code of Navigator UI for Marketplace? I think I saw weeks ago but now I have been looking for it and I could not find it, thanks in advance, Fran

boydjohnson (Wed, 15 Feb 2017 12:24:27 GMT):
Hi @PacoEcotrust, Marketplace Navigator is in the 0-7 branch but not master. Here is a link: https://github.com/hyperledger/sawtooth-core/tree/0-7/extensions/mktplace/navigator

boydjohnson (Wed, 15 Feb 2017 14:45:44 GMT):
The smoke test fix PR is merged, so everyone that has PRs up should rebase on master.

amundson (Wed, 15 Feb 2017 20:50:31 GMT):
@rbuysse did you figure out the docker build error problems?

rbuysse (Wed, 15 Feb 2017 20:50:53 GMT):
yeah I think so

rbuysse (Wed, 15 Feb 2017 20:51:09 GMT):
I pushed a commit and forgot to do a PR. sec.

jsmitchell (Wed, 15 Feb 2017 22:10:42 GMT):
@adamludvik are you going to crank up the simulator on bond?

jsmitchell (Wed, 15 Feb 2017 22:10:59 GMT):
or wait for redeploy?

adamludvik (Wed, 15 Feb 2017 22:18:10 GMT):
I was going to leave the current network up for a week and work out the LIBOR issues, then redeploy. So I'll run the simulator against it again. @toddojala Did you get screenshots for the docs yet?

toddojala (Wed, 15 Feb 2017 22:37:42 GMT):
@adamludvik I am starting that about now...

toddojala (Wed, 15 Feb 2017 22:37:56 GMT):
Just give me a window

rbuysse (Wed, 15 Feb 2017 23:12:47 GMT):
i think you're fine any time now

rbuysse (Wed, 15 Feb 2017 23:12:53 GMT):
@toddojala ^

rbuysse (Wed, 15 Feb 2017 23:17:33 GMT):
BTW, I gave jenkins 6 executors (up from the default of 2), please let me know if you see any jobs waiting, or if things are taking longer than they should when a bunch of jobs are running.

TomBarnes (Wed, 15 Feb 2017 23:28:18 GMT):
I'm familiarizing myself with merkle.py. I'm running Pycharm in Windows. I grabbed the unit test code at sawtooth-core\validator\tests\unit\test_merkle_trie.pyto write some tests of my own. when I tried to run the test code with the map_size default of 1024**4 (set in sawtooth-core\validator\sawtooth_validator\database\lmdb_nolock_database.py), it immediately failed complaining that there wasn't enough disk space. I did a quick search and noted that when lmdb creates the database file on Windows, it allocates the entire file on disk; I only had 56GB available.. This might merit some attention to ensure that the behavior is as intended on Windows.

TomBarnes (Wed, 15 Feb 2017 23:32:15 GMT):
..which raises the question - should any platform-specific settings be hard-coded?

jsmitchell (Thu, 16 Feb 2017 14:39:14 GMT):
Once the local config support is in place, we can move this type of thing out of the code

jsmitchell (Thu, 16 Feb 2017 14:39:33 GMT):
It's a shame that windows doesn't support sparse files

jsmitchell (Thu, 16 Feb 2017 14:45:07 GMT):
sparse files are supported in Linux in vagrant. I'd recommend doing your testing work there.

amundson (Thu, 16 Feb 2017 14:56:30 GMT):
another reason to never support windows-native validators and instead run them in docker.

jsmitchell (Thu, 16 Feb 2017 14:59:36 GMT):
http://www.openldap.org/its/index.cgi/Software%20Enhancements?id=8324

adamludvik (Thu, 16 Feb 2017 15:48:32 GMT):
PR 79 which adds linting is printing an error message when trying to create a .pylint.d directory. I think this directory is just used to store information about previous runs of pylint. If we add the `--persistent=no` flag to the pylint call in run_lint, it does not create the directory when I test things locally so this should fix the issue. Is there any reason to keep those stats around? I didn't even know they existed and we already have `--reports=no` set, which is where you would see them.

adamludvik (Thu, 16 Feb 2017 15:48:39 GMT):
http://stackoverflow.com/questions/25475327/how-does-pylint-remember-scores-from-previous-runs

adamludvik (Thu, 16 Feb 2017 15:48:46 GMT):
https://github.com/hyperledger/sawtooth-core/pull/79

amundson (Thu, 16 Feb 2017 16:02:25 GMT):
@adamludvik adding --persistent=no sounds fine to me

VipinB (Thu, 16 Feb 2017 16:11:46 GMT):
Has joined the channel.

jsmitchell (Thu, 16 Feb 2017 16:22:40 GMT):
I'm implementing ZMQ security (using the 'ironhouse' pattern described here: http://hintjens.com/blog:49#toc6)

jsmitchell (Thu, 16 Feb 2017 16:23:10 GMT):
We'll need to have a well known public/private keypair for the 'server' identity of the validators

jsmitchell (Thu, 16 Feb 2017 16:24:25 GMT):
this is for the purposes of authenticating client identity, which will be signed by the client's private key

jsmitchell (Thu, 16 Feb 2017 16:24:51 GMT):
and can be verified by the server

jsmitchell (Thu, 16 Feb 2017 16:25:28 GMT):
we will presumably then need an ephemeral server private/public key pair which is used for actual channel encryption

jsmitchell (Thu, 16 Feb 2017 16:27:07 GMT):
looking for advice on two things. 1) any ways to avoid having a key pair be part of the config 2) alternatives to the two stage approach to the handshake

boydjohnson (Thu, 16 Feb 2017 16:27:29 GMT):
https://github.com/zeromq/pyzmq/blob/550ddc04c3ce67cdb58b6fb6499274dc7f82891c/examples/security/ironhouse.py Here is a pyzmq example for those less fluent with C.

jsmitchell (Thu, 16 Feb 2017 16:29:39 GMT):
changes I've made from that impl - 1) the client is generating ephemeral keys for each connection (zmq.curve_keypair() and server doesn't enforce having to know about the client's public key in a local store) 2) not whitelisting localhost

jsmitchell (Thu, 16 Feb 2017 16:31:24 GMT):
I'm concerned that we'll need to crack into the zmq internals to support this given the (i think) need for a two step process

jsmitchell (Thu, 16 Feb 2017 16:33:00 GMT):
```[16:17:47 DEBUG base] version: b'1.0', request_id: b'1', domain: '', address: '127.0.0.1', identity: b'', mechanism: b'CURVE' [16:17:47 DEBUG base] ALLOWED (CURVE allow any client) [16:17:47 DEBUG base] ZAP reply code=b'200' text=b'OK' [16:17:47 DEBUG interconnect] receiving GOSSIP_REGISTER message [16:17:47 DEBUG gossip_handlers] got peer register message from b'a153df0a4a5183f84c8786b-c6e0552bf84566fddeb9249'. sending ack [16:17:47 DEBUG interconnect] sending GOSSIP_ACK to b'a153df0a4a5183f84c8786b-c6e0552bf84566fddeb9249'```

jsmitchell (Thu, 16 Feb 2017 16:33:15 GMT):
the plumbing works though

jsmitchell (Thu, 16 Feb 2017 17:32:27 GMT):
These docs seem to indicate that the server keys are only important for the authenticating that the client is talking to the expected server. All messages after the handshake are encrypted using the transient keys which are generated and exchanged for the connection

jsmitchell (Thu, 16 Feb 2017 17:32:29 GMT):
https://rfc.zeromq.org/spec:26/CURVEZMQ/

TomBarnes (Thu, 16 Feb 2017 19:55:16 GMT):
Why is protocol buffers used for encoding STL messages but objects within those messages (for instance the the object associates with a Merle tree address) are encoded with CBOR?

jsmitchell (Thu, 16 Feb 2017 19:57:05 GMT):
we're going to change that to protobuf

jsmitchell (Thu, 16 Feb 2017 19:58:55 GMT):
takemiya-san from the Iroha project was making the case for flatbuffers instead of protobuf

jsmitchell (Thu, 16 Feb 2017 19:59:09 GMT):
due to performance and memory footprint

jsmitchell (Thu, 16 Feb 2017 19:59:32 GMT):
it also sounds like they are working on a native flatbuffer store, which sounded cool

jsmitchell (Thu, 16 Feb 2017 19:59:32 GMT):
it also looks like they are working on a native flatbuffer store, which sounded cool

jsmitchell (Thu, 16 Feb 2017 20:00:37 GMT):
but we will stick with protobuf for now until there is a reason to reconsider

TomBarnes (Thu, 16 Feb 2017 20:02:35 GMT):
So when transaction family is published, it will (must) include a protobuf file that describes all of the acceptable state objects that it can handle?

pschwarz (Thu, 16 Feb 2017 20:03:14 GMT):
It doesn't have to be a protobuf file - those encodings are family-specific

pschwarz (Thu, 16 Feb 2017 20:03:52 GMT):
It certainly would need to be documented for consumers of that family, whatever the encoding.

jsmitchell (Thu, 16 Feb 2017 20:04:26 GMT):
yeah, that's just a blob of bytes from the core's perspective

TomBarnes (Thu, 16 Feb 2017 20:07:16 GMT):
The encoding of an object in Python and the object encoding in Java may not create the same number of bytes; if embedded in a message (like trnasction objects are), the hash of the message will be different. I thought that was the entire reason we moved to prototype- explicit serialization.

jsmitchell (Thu, 16 Feb 2017 20:08:31 GMT):
it is the responsibility of the implementers of the transaction processors to ensure that they encodings they choose are deterministic and consistent

jsmitchell (Thu, 16 Feb 2017 20:08:43 GMT):
across all languages that they choose to implement in

adamludvik (Thu, 16 Feb 2017 20:20:52 GMT):
If you look at the xo transaction family, it just uses CSV byte strings to encode the payload.

adamludvik (Thu, 16 Feb 2017 20:21:38 GMT):
https://github.com/hyperledger/sawtooth-core/blob/master/sdk/examples/sawtooth_xo/processor/handler.py#L55

adamludvik (Thu, 16 Feb 2017 20:21:59 GMT):
https://github.com/hyperledger/sawtooth-core/blob/master/sdk/examples/sawtooth_xo/processor/handler.py#L186

boydjohnson (Thu, 16 Feb 2017 20:26:43 GMT):
Just wanted to give an update on where I am at with different components in a node stopping and the others behaving in a ok manner. On my branch, the python transaction processor can come and go (if given time to gracefully do so) even during load (There needs to be a minimum of 1 transaction processor for the transaction family that you are loading with). The validator can stop (Still needs 2 sigints) and the transaction processors will make a new connection and send a register request when the validator becomes available. What I haven't solved yet is the validator going down under load and the transaction processors behaving nicely -- probably inferring that they should drop all in process transactions and reregister again when the validator becomes available. I think just describing it here gives me new insight, but wanted to let people know where I'm at.

jsmitchell (Thu, 16 Feb 2017 21:43:42 GMT):
@boydjohnson do you want to look https://github.com/hyperledger/sawtooth-core/pull/94 before it gets merged?

jsmitchell (Thu, 16 Feb 2017 21:43:42 GMT):
@boydjohnson do you want to look https://github.com/hyperledger/sawtooth-core/pull/94 over before it gets merged?

boydjohnson (Thu, 16 Feb 2017 21:44:05 GMT):
Yeah, I'll do so.

boydjohnson (Thu, 16 Feb 2017 21:53:08 GMT):
@jsmitchell What are the implications for hard coded pubkey and private key? Would a next step be local configuration of those keys?

jsmitchell (Thu, 16 Feb 2017 21:54:26 GMT):
so, those keys are associated with the 'server' identity. In a non peer to peer example, you'd have a single server or set of servers with the private key and you'd distribute the server public key to your clients

jsmitchell (Thu, 16 Feb 2017 21:54:46 GMT):
the purpose is to make sure you are talking to a server that can prove they have access to that secret key

jsmitchell (Thu, 16 Feb 2017 21:55:04 GMT):
critically, those keys are _not_ used for encryption of the communications channel

jsmitchell (Thu, 16 Feb 2017 21:56:10 GMT):
encryption is based on transient keys generated by the client (client's pubkey goes to server), and the server generates a transient pubkey which is encrypted with the clients pubkey and is sent to the client

jsmitchell (Thu, 16 Feb 2017 21:57:04 GMT):
because we are peer to peer, everybody needs the private key, and that initial 'server identity check' is meaningless

jsmitchell (Thu, 16 Feb 2017 21:57:30 GMT):
ideally, we could just skip it, but it's baked into the CurveZMQ protocol

jsmitchell (Thu, 16 Feb 2017 21:58:21 GMT):
yes, we may move them to local config

jsmitchell (Thu, 16 Feb 2017 21:58:49 GMT):
also there might be other use cases which justify the complexity of distributing all the keys to permissioned nodes

jsmitchell (Thu, 16 Feb 2017 21:59:07 GMT):
but, we will get permissioned behavior from higher up the stack too

boydjohnson (Thu, 16 Feb 2017 21:59:46 GMT):
Thanks.

amundson (Thu, 16 Feb 2017 22:47:40 GMT):
```dpkg: error processing archive build/debs/python3-sawtooth-sdk_0.8.1~dev1-1_all.deb (--install): trying to overwrite '/usr/lib/python3/dist-packages/sawtooth_protobuf/batch_pb2.py', which is also in package python3-sawtooth-cli 0.8.1~dev1-1 ```

amundson (Thu, 16 Feb 2017 22:47:43 GMT):
oops :)

TomBarnes (Thu, 16 Feb 2017 22:49:21 GMT):
regarding state blobs: got it - thanks!

amundson (Thu, 16 Feb 2017 22:50:59 GMT):
state blobs are the only place we need deterministic encoding; other places, we keep the original bytes around as originally generated (this is different than 0.7)

jsmitchell (Thu, 16 Feb 2017 22:50:59 GMT):
python3-sawtooth-protobuf package coming up

jsmitchell (Thu, 16 Feb 2017 22:54:27 GMT):
@amundson: https://github.com/hyperledger/sawtooth-core/pull/92 is ready to merge

amundson (Thu, 16 Feb 2017 22:55:32 GMT):
thx

amundson (Fri, 17 Feb 2017 14:48:44 GMT):
this was mentioned in #general - http://opentracing.io/

amundson (Fri, 17 Feb 2017 14:49:13 GMT):
would be pretty cool to submit a transaction and see it's lifetime across rest api, validator, tp, consensus, etc.

pschwarz (Fri, 17 Feb 2017 15:43:30 GMT):
With the merging of https://github.com/hyperledger/sawtooth-core/pull/96, the data directory has change from `` to `/data` or `/var/lib/sawtooth` if no `SAWTOOTH_HOME` is set. The vagrant environment will have this variable set to `~/sawtooth`.

pschwarz (Fri, 17 Feb 2017 15:43:30 GMT):
With the merging of https://github.com/hyperledger/sawtooth-core/pull/96, the data directory for 0.8 has change from `` to `/data` or `/var/lib/sawtooth` if no `SAWTOOTH_HOME` is set. The vagrant environment will have this variable set to `~/sawtooth`.

rcampos@altus.cr (Fri, 17 Feb 2017 15:55:52 GMT):
Has left the channel.

amundson (Fri, 17 Feb 2017 17:52:30 GMT):
which means, for native windows installs, you must set SAWTOOTH_HOME for now

pschwarz (Fri, 17 Feb 2017 18:13:11 GMT):
Trying to decide where a validator's identity key should live. Would this also make sense in the data directory, or peer of that? The later makes sense when there `SAWTOOTH_HOME` is set, but less so with `/var/lib/sawtooth`

adamludvik (Fri, 17 Feb 2017 19:04:26 GMT):
I know this has come up before, but I'd like to propose that the sawtooth-validator docker container should also start up the additional processes needed to make it useful. According to docker's own best practices page "it is not necessarily true that there should be only one operating system process per container" (https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/) Having these processes startup in the sawtooth validator container would make it easier to keep code that depends on the container up to date because it would shift the responsibility of orchestration into the container. This would also reduce code duplication across tests that startup a working validator. Finally, it would make the sawtooth-validator container more of a logical "whole" since I am not sure the validator is very useful without the rest_api, validator registry processor, and configuration processor.

amundson (Fri, 17 Feb 2017 19:08:23 GMT):
pschwarz: likely SAWTOOTH_HOME/etc/keys/ or /etc/sawtooth/keys/

amundson (Fri, 17 Feb 2017 19:08:23 GMT):
@pschwarz: likely SAWTOOTH_HOME/etc/keys/ or /etc/sawtooth/keys/

amundson (Fri, 17 Feb 2017 19:13:20 GMT):
@adamludvik - validator registry is poet specific and wouldn't apply to another consensus mechanism; config processor could in theory be swapped out with another txn family that updates those settings using a different authorization approach; and combining them into the validator's docker container would make managing them special (they work different than all other TPs) which seems undesirable.

adamludvik (Fri, 17 Feb 2017 19:25:39 GMT):
I guess that makes sense. Perhaps it would make sense to have a "sawtooth-poet-validator" or "sawtooth-std-validator" container that started up a selection of processes? Or are we are getting into the domain of docker-compose then... It just seems odd to have a lot of docker containers that have limited usefulness in isolation.

amundson (Fri, 17 Feb 2017 20:04:24 GMT):
Well, if we put it all in one docker, we may as well have used a vm

cianx (Sat, 18 Feb 2017 00:19:30 GMT):
I made this change https://github.com/hyperledger/sawtooth-core/pull/101/files#diff-d98fd4214fe743c74766d752fb839083R24 to the sawtooth_base docker file. Now run_tests hangs(10min+) during the docker build on that step. How do I resolve this?

cianx (Sat, 18 Feb 2017 00:27:12 GMT):
I finally got a time out message from the apt, unclear why it can't reach the internet today. The rest of the vagrant box can.

amundson (Sat, 18 Feb 2017 00:47:36 GMT):
maybe docker needs different proxy settings. @rbuysse has probably played with that the most

cianx (Sat, 18 Feb 2017 02:08:06 GMT):
Looks like that is the issue. If I log out of vpn I can get it to run. The proxy is set up correctly in vagrant. There must be a way to tell docker about it.

amundson (Sat, 18 Feb 2017 14:59:00 GMT):
IIRC, the hard part is doing that and not modifying the dockerfile directly

amundson (Mon, 20 Feb 2017 15:48:09 GMT):
I'm a bit nervous about too much **kwargs stuff going in. If we are going to have the pattern where we hide the protobuf classes within another class and then construct using **kwargs, seems like we might want to make sure that methodology is constrained to just a few places and not used everywhere.

amundson (Mon, 20 Feb 2017 15:48:09 GMT):
I'm a bit nervous about too much kwargs stuff going in. If we are going to have the pattern where we hide the protobuf classes within another class and then construct using kwargs, seems like we might want to make sure that methodology is constrained to just a few places and not used everywhere.

pschwarz (Mon, 20 Feb 2017 16:03:16 GMT):
Is that in reference to a particular PR?

boydjohnson (Mon, 20 Feb 2017 16:45:54 GMT):
What did we figure out about squashing commits on a PR. It is probably still a good idea to leave the commits unsquashed until approval happens?

vikas.singh (Mon, 20 Feb 2017 17:16:16 GMT):
Has joined the channel.

adamludvik (Mon, 20 Feb 2017 17:29:21 GMT):
@cianx I'll look into the VPN issue today

adamludvik (Mon, 20 Feb 2017 18:55:15 GMT):
@cianx What do you do to get the proxy working within vagrant? I am not able to do `sudo apt-get update` within vagrant when connected to the VPN.

adamludvik (Mon, 20 Feb 2017 22:20:41 GMT):
@cianx I created a PR assigned to you to fix the issue

boydjohnson (Mon, 20 Feb 2017 22:58:43 GMT):
@pschwarz Is my thinking right that we would always require the config transaction processor if there are batches in the GenesisData, and that any other required transaction processor would be set via on chain configuration? Then require_txn_processors in the executor could be called require_config?

amundson (Mon, 20 Feb 2017 23:05:39 GMT):
yes, at least for now, if the setting is empty for the txn family list, then it should default to the list containing only the config transaction family

amundson (Mon, 20 Feb 2017 23:07:03 GMT):
could we simplify the rule to say, if it's not set, then they are all required?

amundson (Mon, 20 Feb 2017 23:07:46 GMT):
that way, we could support alternate config family implementations during this bootstrap process as well

amundson (Mon, 20 Feb 2017 23:07:59 GMT):
(without ever explicitly setting it in the conf file)

boydjohnson (Mon, 20 Feb 2017 23:11:43 GMT):
So that would be if there are no listed required txn processors in on chain configuration, all transactions would wait until there is a transaction processor of the right type online?

amundson (Mon, 20 Feb 2017 23:13:59 GMT):
yes, with the assumption that the very first transactions should set that list on-chain

allwelder (Tue, 21 Feb 2017 08:31:56 GMT):
Has joined the channel.

boydjohnson (Tue, 21 Feb 2017 15:18:24 GMT):
@amundson and I were talking about having a trace logging level that would show more than debug since some of our debug logging could be helpful in some situations but probably isn't necessary even at the debug level. Here is a stackoverflow answer that shows how to set a custom debug level. http://stackoverflow.com/a/16955098

adamludvik (Tue, 21 Feb 2017 15:23:46 GMT):
@boydjohnson I like the idea. Would we leave trace statements in the code like debug statements? Seems like if there were trace statements everywhere there's a chance it could become unreadable.

boydjohnson (Tue, 21 Feb 2017 15:25:03 GMT):
Yeah, there would be trace logging statements. Do you mean the code would become unreadable or the logs? Some of our current debug logging statements could turn into trace logging statements.

adamludvik (Tue, 21 Feb 2017 15:29:10 GMT):
Lol, well worst case scenario, the code could become unreadable if there was a trace statement for every python statement, but I don't think that would happen I'd be more concerned that, especially if we have a lot of concurrency and parallelism, the logs may be too cluttered.

adamludvik (Tue, 21 Feb 2017 15:29:10 GMT):
Lol, well worst case scenario, the code could become unreadable if there was a trace statement for every python statement, but I don't think that would happen. I'd be more concerned that, especially if we have a lot of concurrency and parallelism, the logs may be too cluttered.

amundson (Tue, 21 Feb 2017 15:29:44 GMT):
I think my idea for how to do it is cleaner - just get a "LOGGER_TRACE = logging.getLogger("trace." + __name__)"

adamludvik (Tue, 21 Feb 2017 15:30:24 GMT):
That would be very greppable

amundson (Tue, 21 Feb 2017 15:30:37 GMT):
well, it specifically doesn't require any additional shared code

amundson (Tue, 21 Feb 2017 15:31:04 GMT):
it does require configuring the trace. logger namespace turn it off

adamludvik (Tue, 21 Feb 2017 15:32:33 GMT):
Do we have any definitions for what the different logging levels mean right now?

amundson (Tue, 21 Feb 2017 15:53:35 GMT):
not explicitly stated, no

drozd (Tue, 21 Feb 2017 20:18:47 GMT):
Has joined the channel.

boydjohnson (Tue, 21 Feb 2017 21:00:56 GMT):
We don't have anything to set config settings on the fly, do we? I have set them in the config batch used by the genesis tool, but nowhere else.

amundson (Tue, 21 Feb 2017 22:22:32 GMT):
you can use 'sawtooth config' to generate a file with the appropriate batches

pschwarz (Tue, 21 Feb 2017 22:23:17 GMT):
We don't, yet, have a CLI tool for submitting those transactions to a already-running validator

pschwarz (Tue, 21 Feb 2017 22:23:17 GMT):
We don't, yet, have a CLI tool for submitting sawtooth_config transactions to a already-running validator

amundson (Tue, 21 Feb 2017 22:23:21 GMT):
I don't think we have 'sawtooth submit' yet...

amundson (Tue, 21 Feb 2017 22:23:43 GMT):
wonder if intkey tool will send whatever branches with it's load command

boydjohnson (Tue, 21 Feb 2017 22:24:01 GMT):
Yeah, it might...

boydjohnson (Tue, 21 Feb 2017 23:18:12 GMT):
"intkey load -f config.batch" works.

boydjohnson (Tue, 21 Feb 2017 23:19:36 GMT):
I also want to mention that I seemed to introduce a bug. When transaction processors are waiting to reregister after the validator has gone away they max out their cpu. I will look into it tonight.

boydjohnson (Wed, 22 Feb 2017 15:58:11 GMT):
So to turn on debug logging inside asyncio you can "export PYTHONASYNCIODEBUG=1". When doing that the bug I mentioned yesterday happens less often but when it does it gives this output indefinitely: ```[15:20:30 DEBUG base_events] poll took 0.032 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.024 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.078 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.026 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.021 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.021 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.023 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.025 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.023 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.023 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.020 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.025 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.023 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.217 ms: 0 events ```

boydjohnson (Wed, 22 Feb 2017 15:58:11 GMT):
So to turn on debug logging inside asyncio you can "export PYTHONASYNCIODEBUG=1". When doing that the bug I mentioned yesterday happens less often but when it does it gives this output indefinitely: ```[15:20:30 DEBUG base_events] poll took 0.032 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.024 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.078 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.026 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.021 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.021 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.023 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.025 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.023 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.023 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.020 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.025 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.023 ms: 0 events [15:20:30 DEBUG base_events] poll took 0.217 ms: 0 events ```

amundson (Wed, 22 Feb 2017 19:44:49 GMT):
A large PR just went in which removes legacy code from master.

boydjohnson (Wed, 22 Feb 2017 22:32:15 GMT):
I noticed we aren't linting the manage package and there are a few things that the linter would be able to catch. @toddojala

toddojala (Wed, 22 Feb 2017 22:34:42 GMT):
I shall fix 'em

boydjohnson (Wed, 22 Feb 2017 22:35:06 GMT):
Cool, Thanks!

rbuysse (Wed, 22 Feb 2017 23:11:08 GMT):
I'm getting an error when running the tests on windows.

rbuysse (Wed, 22 Feb 2017 23:11:40 GMT):
the DevModeTestSuite is throwing:

rbuysse (Wed, 22 Feb 2017 23:11:42 GMT):
Checking for pre-existing state: WARNING:txnintegration.integer_key_communication:operation failed: [Errno 10061] No connection could be made because the target machine actively refused it

rbuysse (Wed, 22 Feb 2017 23:12:31 GMT):
this last passed before the signing stuff was committed on Feb 15, any chance those things are related?

bartcant (Thu, 23 Feb 2017 02:18:47 GMT):
Has joined the channel.

amundson (Thu, 23 Feb 2017 02:40:44 GMT):
seems likely

amundson (Thu, 23 Feb 2017 16:01:28 GMT):
The python toml library does not return very good error messages when you make a syntax error in the toml file

amundson (Thu, 23 Feb 2017 16:02:12 GMT):
```Traceback (most recent call last): File "./bin/validator", line 37, in main() File "/project/sawtooth-core/validator/sawtooth_validator/server/cli.py", line 95, in main path_config = load_path_config() File "/project/sawtooth-core/validator/sawtooth_validator/config/path.py", line 88, in load_path_config toml_config = load_toml_path_config(conf_file) File "/project/sawtooth-core/validator/sawtooth_validator/config/path.py", line 51, in load_toml_path_config toml_config = toml.loads(fd.read()) File "/usr/lib/python3/dist-packages/toml.py", line 331, in loads value, vtype = load_value(pair[1]) File "/usr/lib/python3/dist-packages/toml.py", line 471, in load_value v = int(v) ValueError: invalid literal for int() with base 10: '/tmp/v' ```

amundson (Thu, 23 Feb 2017 16:02:24 GMT):
that's the error fi you forget to quote a string

amundson (Thu, 23 Feb 2017 17:53:05 GMT):
anyone else seeing this when starting up a vaildator?

amundson (Thu, 23 Feb 2017 17:53:11 GMT):
```Traceback (most recent call last): File "/project/sawtooth-core/validator/sawtooth_validator/journal/publisher.py", line 166, in on_chain_updated self._candidate_block = self._build_block(chain_head) File "/project/sawtooth-core/validator/sawtooth_validator/journal/publisher.py", line 73, in _build_block prev_state = self._get_previous_block_root_state_hash(chain_head) File "/project/sawtooth-core/validator/sawtooth_validator/journal/publisher.py", line 63, in _get_previous_block_root_state_hash if blkw.previous_block_id == NULL_BLOCK_IDENTIFIER: AttributeError: 'NoneType' object has no attribute 'previous_block_id' ```

amundson (Thu, 23 Feb 2017 17:54:15 GMT):
@boydjohnson - the validator.stop() call in finally: in validator main() doesn't make sense to me

boydjohnson (Thu, 23 Feb 2017 17:55:12 GMT):
Yeah, the validator shutdown is something that I've wanted to get to.

agunde (Thu, 23 Feb 2017 17:55:23 GMT):
@amundson Did you do the genesis command? That looks like the chain_head was not set correctly.

amundson (Thu, 23 Feb 2017 17:56:07 GMT):
@agunde: I did not

amundson (Thu, 23 Feb 2017 17:58:28 GMT):
@pschwarz ^

jjason (Thu, 23 Feb 2017 19:05:24 GMT):
@amundson Yes. I was having that problem yesterday. @pschwarz set me on the straight and narrow. I had failed to create the genesis block.

amundson (Thu, 23 Feb 2017 19:17:54 GMT):
I created a JIRA task for it

pschwarz (Thu, 23 Feb 2017 19:37:44 GMT):
@mfford and I added a task for documenting the start up process (a missing element for people)

joacomoreno (Fri, 24 Feb 2017 17:53:36 GMT):
Has joined the channel.

adamludvik (Fri, 24 Feb 2017 18:53:51 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=8jnkcnhxuQbJk6Nv8) @jsmitchell Did we find a work around for this, or are we not supporting lmdb on Windows?

adamludvik (Fri, 24 Feb 2017 18:54:38 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=iti2dZZSBzH3uy3n7) @Tom Barnes Probably a better post to reference...

jsmitchell (Fri, 24 Feb 2017 23:11:28 GMT):
Once the local config stuff is in place we can switch to a platform/machine specific config setting for the database size

jsmitchell (Fri, 24 Feb 2017 23:12:37 GMT):
Or we might insist that validators run in containers that have file systems that support sparse files with good performance

rjones (Fri, 24 Feb 2017 23:48:50 GMT):
Has joined the channel.

rjones (Fri, 24 Feb 2017 23:49:33 GMT):
I set up a test project with a new JIRA workflow, could you try it out and let me know if you want something like it for sawtoothlake? https://jira.hyperledger.org/projects/WFT/issues/?filter=allopenissues let me know who should be an admin

Dan (Sun, 26 Feb 2017 16:21:37 GMT):
@mfford --^

PacoEcotrust (Mon, 27 Feb 2017 08:54:06 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=sW89LdXs3Jjae7mnZ) @boydjohnson , thank you very much. I would like to ask to the developer of Marketplace Navigator, why is necessary a new server?. it would not be enough change only the User Interface (Terminal for Navigator)

mansoor (Mon, 27 Feb 2017 14:56:25 GMT):
Has joined the channel.

boydjohnson (Mon, 27 Feb 2017 15:31:55 GMT):
@pschwarz I am making changes to my transaction processor configuration list support in the Transaction Executor PR branch and a thought came to me that wasn't addressed by anyone's comments: Since there can be 1 Executor with several Executor threads the waiting for transaction processor functionality should probably be owned by the TransactionExecutor not the TransactionExecutorThread, since configuration and processors registered don't change by thread.

pschwarz (Mon, 27 Feb 2017 15:32:27 GMT):
That makes sense

boydjohnson (Mon, 27 Feb 2017 15:32:42 GMT):
Sounds good. Thanks!

pschwarz (Mon, 27 Feb 2017 15:34:37 GMT):
Just to make sure I'm thinking it makes sense: in this case, if one txn processor is blocking, others can still process txns on other families that may have available processors, correct?

boydjohnson (Mon, 27 Feb 2017 15:35:37 GMT):
Yeah, that will work with my implementation, but we won't see it until we have the parallel scheduler.

pschwarz (Mon, 27 Feb 2017 15:37:14 GMT):
Ok, still making sense then :)

boydjohnson (Mon, 27 Feb 2017 15:37:26 GMT):
Cool.

jsmitchell (Mon, 27 Feb 2017 16:33:36 GMT):
@boydjohnson: have you done any testing with interconnect using ipc transport uris?

jsmitchell (Mon, 27 Feb 2017 16:33:48 GMT):
i.e. ipc:///tmp/somefile

boydjohnson (Mon, 27 Feb 2017 16:34:07 GMT):
I have not except. The monitor socket in Stream in the python sdk uses ipc.

jsmitchell (Mon, 27 Feb 2017 16:34:22 GMT):
for some reason is is not working

jsmitchell (Mon, 27 Feb 2017 16:34:36 GMT):
the validator never sees the connection from the transaction processor

boydjohnson (Mon, 27 Feb 2017 16:35:18 GMT):
Hmm. I would have thought it would work. I'll try sometime.

jsmitchell (Mon, 27 Feb 2017 16:35:37 GMT):
i lied, it actually does work

boydjohnson (Mon, 27 Feb 2017 16:35:49 GMT):
Oh, was it the three slashes?

jsmitchell (Mon, 27 Feb 2017 16:35:58 GMT):
but it doesn't print out any kind of error if it can't open the fd

jsmitchell (Mon, 27 Feb 2017 16:36:18 GMT):
(I had started the validator as root and the tp not)

boydjohnson (Mon, 27 Feb 2017 16:36:47 GMT):
On the client side, right? oh. That is because the thread inside of the Stream class swallows errors.

boydjohnson (Mon, 27 Feb 2017 16:38:24 GMT):
@zac Saw similar issues on a bad hostname, where the Stream class wouldn't connect to it and there was an error, but the thread inside of the Stream class swallowed the error, possibly because it is a daemon thread.

jsmitchell (Mon, 27 Feb 2017 16:39:46 GMT):
can you create a jira about investigating a fix for that?

boydjohnson (Mon, 27 Feb 2017 16:39:56 GMT):
Yeah, I'll do that.

jsmitchell (Mon, 27 Feb 2017 17:17:06 GMT):
@boydjohnson @agunde @jjason @cianx: can one of you review/approve https://github.com/hyperledger/sawtooth-core/pull/118

jjason (Mon, 27 Feb 2017 18:19:01 GMT):
Done

boydjohnson (Mon, 27 Feb 2017 22:20:49 GMT):
@jsmitchell I have an initial network level test task that it is my understanding is a test for validator network functionality. Sort of like a gossip smoke test. Because it seems like you have some interesting network changes around peering that might be up soon, do you think it makes sense for me to base my current work off of your branch?

jsmitchell (Mon, 27 Feb 2017 22:56:14 GMT):
@boydjohnson: can you defer it for a few days, or do you need to work on it now?

boydjohnson (Tue, 28 Feb 2017 01:19:54 GMT):
@jsmitchell I can defer it.

boydjohnson (Tue, 28 Feb 2017 15:31:04 GMT):
`intkey load batches: 1001 batch/sec: 304.1413737474129`

boydjohnson (Tue, 28 Feb 2017 15:31:04 GMT):
What do you make of the difference between ours? Mine is on my fix threading shutdown-startup issues branch.

jsmitchell (Tue, 28 Feb 2017 15:31:30 GMT):
```ubuntu@ubuntu-xenial:/project/sawtooth-core$ ./bin/intkey load batches: 1001 batch/sec: 743.2837209911658 ```

boydjohnson (Tue, 28 Feb 2017 15:33:10 GMT):
Nice speedup on the secp signing! Is that about what you have been seeing for batches per second?

jsmitchell (Tue, 28 Feb 2017 15:33:26 GMT):
you should run intkey create_batch :)

boydjohnson (Tue, 28 Feb 2017 15:34:02 GMT):
Yeah, I saw that is much faster. It blew up for me without doing that.

boydjohnson (Tue, 28 Feb 2017 15:39:40 GMT):
I'm guessing "intkey create_batch -B 10" with the maximum txns per batch being 10 for your run?

jsmitchell (Tue, 28 Feb 2017 15:40:04 GMT):
@boydjohnson: have you seen a thing where a node gets more than the 2-tuple on initial recv_multipart and it throws ValueError on the unpack?

boydjohnson (Tue, 28 Feb 2017 15:40:40 GMT):
I have seen that while working on the SendReceive, but not on master.

jsmitchell (Tue, 28 Feb 2017 15:40:49 GMT):
for some reason this message contains multiple identities

jsmitchell (Tue, 28 Feb 2017 15:40:56 GMT):
prior to the payload

boydjohnson (Tue, 28 Feb 2017 15:41:09 GMT):
If you set an identity on the Router socket it will receive it on every message.

jsmitchell (Tue, 28 Feb 2017 15:41:21 GMT):
```[15:32:18.313 DEBUG gossip_handlers] got peer register message from b'tcp://0.0.0.0:8802-ab2b105fe389a01b126c455'. sending ack [15:32:18.314 DEBUG gossip] Added identity b'tcp://0.0.0.0:8802-ab2b105fe389a01b126c455', connected identities are now [b'tcp://0.0.0.0:8801-37ba9c769772e1d98b713a9', b'tcp://0.0.0.0:8802-ab2b105fe389a01b126c455'] [15:32:18.314 DEBUG interconnect] ServerThread sending GOSSIP_ACK to b'tcp://0.0.0.0:8802-ab2b105fe389a01b126c455' [15:32:18.321 CRITICALinterconnect] TOO MUCH FROM RECV_MULTIPART: [b'tcp://0.0.0.0:8801-37ba9c769772e1d98b713a9', b'tcp://0.0.0.0:8802-79688e7ad3bc8b6cd1baf99', b'\x08\xcb\x01\x12\x80\x018cba4e4e26a44433b982bcde9aa8fd5bc452e99015a554cee83b3f8db5eb7055510134147ca51e662c29230c1f20d79bd8670f4005430ba82636cba85338bcdb'] ```

boydjohnson (Tue, 28 Feb 2017 15:41:23 GMT):
as well as the dealer's identity.

jsmitchell (Tue, 28 Feb 2017 15:41:56 GMT):
@boydjohnson: I think I just did `intkey create_batch -c 1000`

boydjohnson (Tue, 28 Feb 2017 15:42:27 GMT):
Wow, that is the default, I think.

boydjohnson (Tue, 28 Feb 2017 15:43:16 GMT):
In your branch have you changed SendReceive.setup(), where only the Dealer socket gets an identity?

jsmitchell (Tue, 28 Feb 2017 15:43:20 GMT):
(I changed the first part of the identity to be endpoint so I could make sense of what was going on)

jsmitchell (Tue, 28 Feb 2017 15:44:10 GMT):
yep, only the dealer has an identity set in setup()

boydjohnson (Tue, 28 Feb 2017 15:44:16 GMT):
Hmm.

jsmitchell (Tue, 28 Feb 2017 15:58:38 GMT):
the node being connected to also sends two acks as a result of this

boydjohnson (Tue, 28 Feb 2017 16:02:27 GMT):
Looking through those logging statements there look to be 2 8802 identities. In the line just before the CRITICAL logging it sends an ack to 8802 with one identity and receives a message where one identity is 8802.

jsmitchell (Tue, 28 Feb 2017 16:03:26 GMT):
yeah, it's like it does it twice

jsmitchell (Tue, 28 Feb 2017 16:03:40 GMT):
once correctly, and then with the extra identity

jsmitchell (Tue, 28 Feb 2017 16:15:31 GMT):
i see what it is

wheelsinar (Wed, 01 Mar 2017 13:25:52 GMT):
Has joined the channel.

amundson (Wed, 01 Mar 2017 14:59:50 GMT):
join #sawtoothlake-pr-review for PR notifications

jsmitchell (Wed, 01 Mar 2017 15:09:28 GMT):
@boydjohnson or @agunde: Could one of you do a quick review of https://github.com/hyperledger/sawtooth-core/pull/132

boydjohnson (Wed, 01 Mar 2017 15:11:05 GMT):
@jsmitchell Looking at it right now.

jsmitchell (Wed, 01 Mar 2017 15:11:15 GMT):
cheers

rjones (Wed, 01 Mar 2017 15:18:00 GMT):
the automatic PR notifications in #sawtoothlake-pr-review aren't quite hooked up yet, I'm still working on it :)

jsmitchell (Wed, 01 Mar 2017 15:18:31 GMT):
thanks, ry

rjones (Wed, 01 Mar 2017 15:33:54 GMT):
it should fire now on PR, let's wait and see

boydjohnson (Wed, 01 Mar 2017 15:35:26 GMT):
@jsmitchell I approved the PR. Something to think about when doing more of the peering stuff is keeping threadsafety during removing a connection to a peer. The dispatcher.remove_send_message modifies the dispatcher._send_message dict that could be being accessed from within the dispatcher thread.

jsmitchell (Wed, 01 Mar 2017 15:59:58 GMT):
are you suggesting I wrap the add and del in with self._condition: ?

awalias (Wed, 01 Mar 2017 16:03:51 GMT):
Has joined the channel.

jsmitchell (Wed, 01 Mar 2017 16:14:09 GMT):
@rjones: seem to be getting some dups on the PRs

boydjohnson (Wed, 01 Mar 2017 16:18:05 GMT):
@jsmitchell that seems like the easiest way to do it. You could make a threadsafe dict implementation just for this purpose. I don't fully know the best way. Since the _condition is also being used to keep theadsafety on _message_information it might make more sense to make a threadsafe dict and use it for both message_information and _send_message.

boydjohnson (Wed, 01 Mar 2017 16:18:05 GMT):
@jsmitchell that seems like the easiest way to do it. You could make a threadsafe dict implementation just for this purpose. I don't fully know the best way. Since the condition is also being used to keep theadsafety on message_information it might make more sense to make a threadsafe dict and use it for both message_information and _send_message.

boydjohnson (Wed, 01 Mar 2017 16:18:05 GMT):
@jsmitchell that seems like the easiest way to do it. You could make a threadsafe dict implementation just for this purpose. I don't fully know the best way. Since the condition is also being used to keep theadsafety on message_information it might make more sense to make a threadsafe dict and use it for both message_information and send_message.

jsmitchell (Wed, 01 Mar 2017 16:19:45 GMT):
premature optimization maybe. Let's take a closer look at it when we get to perf tuning and look at lock durations.

boydjohnson (Wed, 01 Mar 2017 16:20:07 GMT):
Sounds good.

mfford (Wed, 01 Mar 2017 18:49:38 GMT):
@rjones you can make me the admin of the Sawtooth Lake project in JIRA. As far as the test workflow goes, we are looking at requirements for further utilizing the Hyperledger JIRA for STL and will get back to you.

zac (Wed, 01 Mar 2017 19:17:24 GMT):
@amundson: Have had a chat with @boydjohnson and @agunde regarding `CLIENT_BATCH_SUBMIT_RESPONSE` Messages waiting until the batch has been committed or rejected to send. Would like to run an approach by you.

zac (Wed, 01 Mar 2017 19:18:10 GMT):
The idea is that a handler would be added to the batch submit process at the end, which would wait on a Condition from the block store and the batch cache

zac (Wed, 01 Mar 2017 19:18:47 GMT):
If the block store added the submitted batch, the handler is notified and sends the ZMQ response

rjones (Wed, 01 Mar 2017 19:19:06 GMT):
@mfford you already were an admin on JIRA for STL

zac (Wed, 01 Mar 2017 19:19:37 GMT):
Similarly, if the batch is removed from the batch cache but _not_ added to the block store, a response is sent back, notifying the client that the batch was rejected

mfford (Wed, 01 Mar 2017 19:49:08 GMT):
@rjones yup. Just was confirming that I retain that role, but I could have certainly said it better.

rjones (Wed, 01 Mar 2017 21:01:43 GMT):
@jsmitchell here is the javascript for the inbound hook: https://gist.github.com/rjones-lf/fbe68e5fc9cef6ae6a8847df7d08f739 this is a copy of the default one. I will be offline for a couple weeks but feel free to kibitz on how you want that hook (the one that generates traffic in #sawtoothlake-pr-review) modified

rjones (Wed, 01 Mar 2017 21:02:12 GMT):
or if you come up with anything you want changed in the next... three hours or so? I can make changes

amundson (Wed, 01 Mar 2017 23:16:56 GMT):
@rjones is someone your backup for the next few weeks?

rjones (Thu, 02 Mar 2017 00:00:15 GMT):
@amundson yes, @jwagantall will be

jwagantall (Thu, 02 Mar 2017 00:00:15 GMT):
Has joined the channel.

jwagantall (Thu, 02 Mar 2017 00:00:32 GMT):
me \o/

jwagantall (Thu, 02 Mar 2017 00:00:40 GMT):
:raised_hands:

jsmitchell (Thu, 02 Mar 2017 00:05:54 GMT):
nice! Thanks @rjones - see you in a few weeks

jsmitchell (Thu, 02 Mar 2017 00:06:16 GMT):
hi @jwagantall

jwagantall (Thu, 02 Mar 2017 00:06:28 GMT):
hi! :) tell me

domingohui (Thu, 02 Mar 2017 02:33:27 GMT):
Has joined the channel.

Dan (Thu, 02 Mar 2017 14:32:07 GMT):
@amundson for local config settings, I'll switch spec from StudlyCaps to snake_case for variables unless someone has a strong feeling for some other style.

adamludvik (Thu, 02 Mar 2017 14:58:34 GMT):
Jenkins is going down for a minute to create a backup image.

adamludvik (Thu, 02 Mar 2017 15:01:25 GMT):
Back up

boydjohnson (Thu, 02 Mar 2017 16:40:17 GMT):
Hi @pschwarz Can you confirm my understanding? The key provided for the config txns would be the wif key associated with whoever is making the batch, probably not the validator key. Then after the genesis file is made, with or without genesis batches, and the validator starts up it signs the genesis block with its own key.

jsmitchell (Thu, 02 Mar 2017 16:42:27 GMT):
@zac: looks like https://github.com/hyperledger/sawtooth-core/pull/146 is ready for merge

zac (Thu, 02 Mar 2017 16:43:11 GMT):
Merged

pschwarz (Thu, 02 Mar 2017 16:44:39 GMT):
@boydjohnson that's correct

pauln (Fri, 03 Mar 2017 06:54:33 GMT):
Has joined the channel.

Romanix (Fri, 03 Mar 2017 09:04:34 GMT):
Has joined the channel.

Romanix (Fri, 03 Mar 2017 09:13:42 GMT):
Hi. I intend to do a little shared ledger / blockchain prototype - with transaction confidentiality (not all nodes should see all transaction details). From the documentation https://intelledger.github.io/ it's not clear whether Sawtooth Lake transactions are visible to all nodes or not i.e. whether it's possible to encrypt transaction details or not.

Dan (Fri, 03 Mar 2017 16:55:06 GMT):
@Romanix Good question. Short answer is you can write a transaction family that uses encrypted transactions. The pattern I've seen emerging for this is that participants sign a transaction on the side, and then the encrypted result or its hash is stored back to the blockchain.

Dan (Fri, 03 Mar 2017 16:55:30 GMT):
That opaque transaction is what is reflected in the shared ledger.

Dan (Fri, 03 Mar 2017 16:56:33 GMT):
I will say that these side-band transaction models feel at odds with the premise of a shared database.

Dan (Fri, 03 Mar 2017 16:59:10 GMT):
To fit this model into sawtooth you'd have a more sophisticated client that would orchestrate the multi-sig. Then that client would submit the encrypted result or its hash to the blockchain proper.

boydjohnson (Fri, 03 Mar 2017 16:59:10 GMT):
@Dan Does the transaction family do much checking then, if the transaction is encrypted?

kelly_ (Fri, 03 Mar 2017 17:11:02 GMT):
@boydjohnson the transaction family could check signatures of relevant parties

kelly_ (Fri, 03 Mar 2017 17:11:15 GMT):
e.g party a and party b have both signed txn hash xyz

kelly_ (Fri, 03 Mar 2017 17:11:25 GMT):
that is how IBM Fabric works I believe

kelly_ (Fri, 03 Mar 2017 18:27:41 GMT):
yup, just checked with IBM

kelly_ (Fri, 03 Mar 2017 18:27:50 GMT):
slash fabric

kelly_ (Fri, 03 Mar 2017 18:28:16 GMT):
you could define a transaction family as requring signatures on an encrypted blob before it gets added to the journal

kelly_ (Fri, 03 Mar 2017 18:28:32 GMT):
then at least you have agreement from relevant parties on the blog prior to adding it to ledger

boydjohnson (Fri, 03 Mar 2017 19:15:11 GMT):
@kelly_ Would the pubkeys of the signers be added to the ledger? Like pubkey A, pubkey B are tied to the transaction?

pschwarz (Fri, 03 Mar 2017 19:33:57 GMT):
@jsmitchell Any reason we can't have `__contains__` on the MerkleDatabase for a given address?

boydjohnson (Fri, 03 Mar 2017 19:36:25 GMT):
https://docs.python.org/3.5/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe This is from the Python docs about operations on builtin data structures that are threadsafe

jsmitchell (Fri, 03 Mar 2017 21:13:28 GMT):
@pschwarz Seems eminently feasible

pauln (Sat, 04 Mar 2017 00:03:16 GMT):
Has anyone tried running (successfully I hope) Sawtooth on Bash on Ubuntu on Windows (Bash/WSL)?

boydjohnson (Sat, 04 Mar 2017 00:09:14 GMT):
Hi @pauln, I haven't heard of anyone doing that. One thing to know is that there is a big difference between the 0.7 branch and master. If you have time let us know what you find. For the master branch we have some docker support, so that would be an option too.

pauln (Sat, 04 Mar 2017 00:11:50 GMT):
From what I'm reading on installing vagrant and other VM stuff, Bash/WSL is not ready for prime time. I'll keep poking/googling around. In the mean time I'll RTFM and follow the instruction to install on a WIN10 platform. Thanks @boydjohnson !

boydjohnson (Sat, 04 Mar 2017 00:13:28 GMT):
Your welcome!

boydjohnson (Sat, 04 Mar 2017 00:13:28 GMT):
You're welcome!

jsmitchell (Mon, 06 Mar 2017 16:02:55 GMT):
@boydjohnson or @agunde: do we have an example of returning an error instead of an ack yet?

boydjohnson (Mon, 06 Mar 2017 16:03:54 GMT):
I thought pull #162 does that.

boydjohnson (Mon, 06 Mar 2017 16:03:54 GMT):
I thought PR #162 does that.

jsmitchell (Mon, 06 Mar 2017 16:05:15 GMT):
i mean in a network response

boydjohnson (Mon, 06 Mar 2017 16:05:29 GMT):
Oh, sorry. No.

jsmitchell (Mon, 06 Mar 2017 16:06:23 GMT):
Any concerns with a GOSSIP_ERROR concept?

jsmitchell (Mon, 06 Mar 2017 16:07:03 GMT):
where the message handler would either return a GOSSIP_ACK or GOSSIP_ERROR depending on the condition

boydjohnson (Mon, 06 Mar 2017 16:08:10 GMT):
Would that be a Status enum with those two types? I would probably make sense since we might not want to be too detailed in how it was an error.

jsmitchell (Mon, 06 Mar 2017 16:08:43 GMT):
no, I was thinking a separate message type/protobuf

jsmitchell (Mon, 06 Mar 2017 16:09:00 GMT):
for ERROR

jsmitchell (Mon, 06 Mar 2017 16:09:18 GMT):
which could contain error code and message

jsmitchell (Mon, 06 Mar 2017 16:09:55 GMT):
I guess there is already this ack.status

jsmitchell (Mon, 06 Mar 2017 16:10:12 GMT):
I'll just use that for now

boydjohnson (Mon, 06 Mar 2017 16:10:33 GMT):
Ok.

boydjohnson (Mon, 06 Mar 2017 16:20:03 GMT):
@jjason @jsmitchell https://sphinxcontrib-napoleon.readthedocs.io/en/latest/ Several comments on a PR about picking a style of doc string. Some people are using the rst style docstring and some people are using Python Google docstring style. Napolean is a plugin for sphinx (packaged with sphinx 1.3 and above). The above post makes the case for Google docstrings. As an aside the autodoc plugin for Pycharm will allow you (after setting some settings) will help you write Google style docstrings.

jsmitchell (Mon, 06 Mar 2017 16:20:43 GMT):
@boydjohnson: yep, that's the approach we took for 0.7 (napoleon + google style)

amundson (Mon, 06 Mar 2017 16:25:14 GMT):
I prefer the google style. We need to write up some text explaining the desired format, etc. Perhaps as part of http://intelledger.github.io/0.8/community/contributing.html

amundson (Mon, 06 Mar 2017 16:26:19 GMT):
@rbuysse @jsmitchell - should the validator aggressively attempt to create directories if they don't exist, such as data dir and key dir?

jsmitchell (Mon, 06 Mar 2017 16:26:44 GMT):
super aggressively

amundson (Mon, 06 Mar 2017 16:27:34 GMT):
trying to think of any other daemon process that creates directories if they don't exist

amundson (Mon, 06 Mar 2017 16:27:37 GMT):
I don't think it's a thing

amundson (Mon, 06 Mar 2017 16:28:13 GMT):
so it seems to me that the correct behavior is to exit if the directories don't exist (if they are needed)

amundson (Mon, 06 Mar 2017 16:28:29 GMT):
data_dir, log_dir are certainly needed

jsmitchell (Mon, 06 Mar 2017 16:29:23 GMT):
i think key generation should be explicit too

jsmitchell (Mon, 06 Mar 2017 16:29:49 GMT):
exit if not key_dir

jsmitchell (Mon, 06 Mar 2017 16:30:03 GMT):
exit if keys not found

amundson (Mon, 06 Mar 2017 16:31:07 GMT):
ok, how about this: 1) die w/message if data_dir, log_dir don't exist; 2) die w/message if key file can't be loaded; 3) continue to load if conf_dir does not exist

amundson (Mon, 06 Mar 2017 16:31:31 GMT):
for (2), it seems like the error should be about the key file, and we don't write to the directory, so we don't really need the directory to exist per-se

jsmitchell (Mon, 06 Mar 2017 16:32:12 GMT):
@boydjohnson: thoughts on how to dispatch/handle responses which resolve futures in _receive_message()?

jsmitchell (Mon, 06 Mar 2017 16:32:12 GMT):
@boydjohnson: thoughts on how to dispatch/handle responses which resolve futures in _receive_ message()?

jsmitchell (Mon, 06 Mar 2017 16:32:12 GMT):
@boydjohnson: thoughts on how to dispatch/handle responses which resolve futures in ```_receive_message()```?

jsmitchell (Mon, 06 Mar 2017 16:32:12 GMT):
@boydjohnson: thoughts on how to dispatch/handle responses which resolve futures in '_receive_message()'?

jsmitchell (Mon, 06 Mar 2017 16:32:26 GMT):
@amundson: sounds fine

boydjohnson (Mon, 06 Mar 2017 16:33:02 GMT):
Wouldn't the component that called send have the future and be able to do what it needs to with it?

jsmitchell (Mon, 06 Mar 2017 16:33:33 GMT):
yeah, but what's the calling event?

boydjohnson (Mon, 06 Mar 2017 16:34:24 GMT):
Oh, there also are callbacks on the server futures.

jsmitchell (Mon, 06 Mar 2017 16:34:32 GMT):
I'm looking at that try, except, else in '_receive_message()'

jsmitchell (Mon, 06 Mar 2017 16:35:08 GMT):
ahh

jsmitchell (Mon, 06 Mar 2017 16:35:16 GMT):
add_callback on set_result

jsmitchell (Mon, 06 Mar 2017 16:35:22 GMT):
nice

boydjohnson (Mon, 06 Mar 2017 16:35:57 GMT):
Yeah, it isn't so great right now because each time it starts up a new thread, but me validator shutdown pr will run those in a threadpool.

boydjohnson (Mon, 06 Mar 2017 16:35:57 GMT):
Yeah, it isn't so great right now because each time it starts up a new thread, but my validator shutdown pr will run those in a threadpool.

amundson (Mon, 06 Mar 2017 16:37:39 GMT):
I think this don't-create-the-directory approach also prevents race conditions between components

jjason (Mon, 06 Mar 2017 16:54:58 GMT):
@boydjohnson @jsmitchell Okay. I am going to keep doing what I am doing as I unwittingly was doing the "right" thing....and I had PyCharm there to help me out.

vkandy (Mon, 06 Mar 2017 18:07:23 GMT):
Has joined the channel.

amundson (Mon, 06 Mar 2017 19:18:32 GMT):
what is the right approach to determine whether the contents of a wif key are valid?

amundson (Mon, 06 Mar 2017 19:19:51 GMT):
```[19:17:22.601 INFO keys] Loading signing key: /home/ubuntu/sawtooth/keys/validator.wif Traceback (most recent call last): File "./bin/validator", line 37, in main() File "/project/sawtooth-core/validator/sawtooth_validator/server/cli.py", line 115, in main key_name='validator') File "/project/sawtooth-core/validator/sawtooth_validator/server/keys.py", line 55, in load_identity_signing_key signing.decode_privkey(wif_key), 'hex') File "/project/sawtooth-core/signing/sawtooth_signing/secp256k1_signer.py", line 105, in decode_privkey return _decode_privkey_to_bytes(encoded_privkey, encoding_format) File "/project/sawtooth-core/signing/sawtooth_signing/secp256k1_signer.py", line 77, in _decode_privkey_to_bytes priv = pybitcointools.encode_privkey(encoded_privkey, 'hex') File "/usr/local/lib/python3.5/dist-packages/bitcoin/main.py", line 224, in encode_privkey return encode_privkey(decode_privkey(priv), formt, vbyte) File "/usr/local/lib/python3.5/dist-packages/bitcoin/main.py", line 237, in decode_privkey if not formt: formt = get_privkey_format(priv) File "/usr/local/lib/python3.5/dist-packages/bitcoin/main.py", line 217, in get_privkey_format bin_p = b58check_to_bin(priv) File "/usr/local/lib/python3.5/dist-packages/bitcoin/main.py", line 427, in b58check_to_bin assert bin_dbl_sha256(data[:-4])[:4] == data[-4:] AssertionError ```

amundson (Mon, 06 Mar 2017 19:20:24 GMT):
that's with a .wif file with a prepended 'a' in it

jsmitchell (Mon, 06 Mar 2017 19:24:30 GMT):
``` def get_privkey_format(priv): if isinstance(priv, int_types): return 'decimal' elif len(priv) == 32: return 'bin' elif len(priv) == 33: return 'bin_compressed' elif len(priv) == 64: return 'hex' elif len(priv) == 66: return 'hex_compressed' else: bin_p = b58check_to_bin(priv) if len(bin_p) == 32: return 'wif' elif len(bin_p) == 33: return 'wif_compressed' else: raise Exception("WIF does not represent privkey")```

jsmitchell (Mon, 06 Mar 2017 19:24:37 GMT):
how pybitcointools does it

jsmitchell (Mon, 06 Mar 2017 19:24:57 GMT):
```def b58check_to_bin(inp): leadingzbytes = len(re.match('^1*', inp).group(0)) data = b'\x00' * leadingzbytes + changebase(inp, 58, 256) assert bin_dbl_sha256(data[:-4])[:4] == data[-4:] return data[1:-4]```

amundson (Mon, 06 Mar 2017 19:26:18 GMT):
``` try: decoded_key = signing.decode_privkey(wif_key) except AssertionError: raise LocalConfigurationError( "Could not decode key contained in file (AssertionError): " "{}".format(key_path))```

amundson (Mon, 06 Mar 2017 19:26:25 GMT):
how about something like that for now?

amundson (Mon, 06 Mar 2017 19:26:38 GMT):
I think there should be an is_valid type of thing in signing if it gets more complex

jsmitchell (Mon, 06 Mar 2017 19:26:41 GMT):
seems legit

adamludvik (Mon, 06 Mar 2017 20:27:09 GMT):
Anybody want to review a PR for me?

jsmitchell (Mon, 06 Mar 2017 20:35:14 GMT):
yeah, sure @adamludvik

adamludvik (Mon, 06 Mar 2017 20:35:26 GMT):
https://github.com/hyperledger/sawtooth-core/pull/159

adamludvik (Mon, 06 Mar 2017 20:35:28 GMT):
Thanks!

jsmitchell (Mon, 06 Mar 2017 21:56:01 GMT):
The sdk doesn't care for unknown message types

jsmitchell (Mon, 06 Mar 2017 21:56:07 GMT):
```ubuntu@ubuntu-xenial:/project/sawtooth-core$ ./bin/tp_intkey_python tcp://loost:40000 -vv [21:46:59 DEBUG selector_events] Using selector: ZMQSelector [21:46:59 INFO core] register attempt: OK [21:47:07 DEBUG core] received message of type: GOSSIP_PING Traceback (most recent call last): File "./bin/tp_intkey_python", line 39, in main() File "/project/sawtooth-core/sdk/examples/intkey_python/sawtooth_intkey/processor/main.py", line 93, in main processor.start() File "/project/sawtooth-core/sdk/python/sawtooth_sdk/processor/core.py", line 207, in start self._process_future(fut) File "/project/sawtooth-core/sdk/python/sawtooth_sdk/processor/core.py", line 162, in _process_future self._process(msg) File "/project/sawtooth-core/sdk/python/sawtooth_sdk/processor/core.py", line 102, in _process self._find_handler(header).apply(request, state) File "/project/sawtooth-core/sdk/python/sawtooth_sdk/processor/core.py", line 65, in _find_handler self._handlers))[0] IndexError: list index out of range```

boydjohnson (Mon, 06 Mar 2017 22:02:39 GMT):
I'll make a jira for it.

jsmitchell (Mon, 06 Mar 2017 22:06:11 GMT):
thanks!

WeDoIoE (Tue, 07 Mar 2017 00:16:34 GMT):
Has joined the channel.

alihendavi (Tue, 07 Mar 2017 06:30:31 GMT):
Has joined the channel.

jsmitchell (Tue, 07 Mar 2017 15:30:27 GMT):
@boydjohnson: I have occasionally seen ZMQ bind issues when no processes are running

jsmitchell (Tue, 07 Mar 2017 15:30:33 GMT):
```Traceback (most recent call last): File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner self.run() File "/usr/lib/python3.5/threading.py", line 862, in run self._target(*self._args, **self._kwargs) File "/project/sawtooth-core/validator/sawtooth_validator/networking/interconnect.py", line 240, in setup self._socket.bind(self._address) File "zmq/backend/cython/socket.pyx", line 487, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:5156) File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:7535) zmq.error.ZMQError: Address already in use```

jsmitchell (Tue, 07 Mar 2017 15:31:05 GMT):
```ubuntu@ubuntu-xenial:/project/sawtooth-core$ ps -ef | grep python ubuntu 15003 14538 0 15:30 pts/0 00:00:00 grep --color=auto python```

jsmitchell (Tue, 07 Mar 2017 15:32:29 GMT):
I will attempt to catch it with lsof next time I see it

boydjohnson (Tue, 07 Mar 2017 15:37:36 GMT):
So that might be happening because of bad validator shutdown. When you shutdown a validator under load sometimes the multiprocessing processes hang on to sockets (for some strange reason). Doing "netstat -tp" sometimes shows processes, but other times it doesn't show a process, but just "TIME_WAIT". I hope it will be fixed in my validator shutdown task that will have a PR soon.

jsmitchell (Tue, 07 Mar 2017 20:27:40 GMT):
@agunde @boydjohnson @Dan: Can one of you give me a second review of https://github.com/hyperledger/sawtooth-core/pull/169

boydjohnson (Tue, 07 Mar 2017 20:40:30 GMT):
@jsmitchell I had a comment on it, but asked for changes or clarification.

jsmitchell (Tue, 07 Mar 2017 21:10:42 GMT):
@boydjohnson: updated

boydjohnson (Tue, 07 Mar 2017 21:10:55 GMT):
Cool, I'll take a look.

jsmitchell (Tue, 07 Mar 2017 23:17:38 GMT):
@boydjohnson: I put some logging in Interconnect.stop() for reasons and I noticed that stop() is getting called very early in validator startup, which confused me. Any clues?

boydjohnson (Tue, 07 Mar 2017 23:19:10 GMT):
Yes, validator.start doesn't wait anywhere and it falls through to finally in cli.py where validator.stop sits at service.stop() which has a join in it. Very bad behavior.

jsmitchell (Tue, 07 Mar 2017 23:19:23 GMT):
naughty naughty

boydjohnson (Tue, 07 Mar 2017 23:28:46 GMT):
So switching 'n' in the Lmdb block store constructor to '' or anything else really, seems too work. A test that starts validator (with genesis), tp_intkey_python, rest_api, then sigints them all, then starts them all again, and then runs intkey-smoke on them passes mostly with intermittent errors that don't seem validator related. I will no include the restart test in my PR since it is slowing me down. I will try to push up a PR this evening.

boydjohnson (Tue, 07 Mar 2017 23:28:46 GMT):
So switching 'n' in the Lmdb block store constructor to '' or anything else really, seems too work. A test that starts validator (with genesis), tp_intkey_python, rest_api, then sigints them all, then starts them all again, and then runs intkey-smoke on them passes mostly with intermittent errors that don't seem validator related. I will not include the restart test in my PR since it is slowing me down. I will try to push up a PR this evening.

Suma (Wed, 08 Mar 2017 15:45:02 GMT):
Has joined the channel.

jjason (Wed, 08 Mar 2017 18:17:32 GMT):
Stupid question. Anyone seen the following and can tell what stupid thing I am forgetting to do. The error results when I try to import transaction_pb2: File "/project/sawtooth-core/validator/sawtooth_validator/journal/consensus/poet/poet_consensus/poet_block_publisher.py", line 35, in import sawtooth_sdk.protobuf.transaction_pb2 as transaction_pb2 File "/project/sawtooth-core/sdk/python/sawtooth_sdk/protobuf/transaction_pb2.py", line 22, in serialized_pb=_b('\n\'sawtooth_sdk/protobuf/transaction.proto\"\xe7\x01\n\x11TransactionHeader\x12\x16\n\x0e\x62\x61tcher_pubkey\x18\x01 \x01(\t\x12\x14\n\x0c\x64\x65pendencies\x18\x02 \x03(\t\x12\x13\n\x0b\x66\x61mily_name\x18\x03 \x01(\t\x12\x16\n\x0e\x66\x61mily_version\x18\x04 \x01(\t\x12\x0e\n\x06inputs\x18\x05 \x03(\t\x12\r\n\x05nonce\x18\x06 \x01(\t\x12\x0f\n\x07outputs\x18\x07 \x03(\t\x12\x18\n\x10payload_encoding\x18\x08 \x01(\t\x12\x16\n\x0epayload_sha512\x18\t \x01(\t\x12\x15\n\rsigner_pubkey\x18\n \x01(\t\"H\n\x0bTransaction\x12\x0e\n\x06header\x18\x01 \x01(\x0c\x12\x18\n\x10header_signature\x18\x02 \x01(\t\x12\x0f\n\x07payload\x18\x03 \x01(\x0c\x42\x19\n\x15sawtooth.sdk.protobufP\x01\x62\x06proto3') File "/usr/local/lib/python3.5/dist-packages/google/protobuf/descriptor.py", line 824, in __new__ return _message.default_pool.AddSerializedFile(serialized_pb) TypeError: Couldn't build proto file into descriptor pool!

jjason (Wed, 08 Mar 2017 18:18:21 GMT):
Invalid proto descriptor for file "sawtooth_sdk/protobuf/transaction.proto": TransactionHeader.batcher_pubkey: "TransactionHeader.batcher_pubkey" is already defined in file "sawtooth_validator/protobuf/transaction.proto".

boydjohnson (Wed, 08 Mar 2017 18:25:15 GMT):
So I see weirdness, in that something on the validator/sawtooth_validator path would be importing something from sawtooth_sdk. When the protobuf classes are generated they are put in the sdk and sawtooth_validator and mixing them causes weird errors.

jjason (Wed, 08 Mar 2017 18:26:21 GMT):
Fair enough....let me fix that.

jjason (Wed, 08 Mar 2017 18:26:38 GMT):
I was cut-and-pasting from another file.....

boydjohnson (Wed, 08 Mar 2017 18:26:47 GMT):
Ahh.

boydjohnson (Wed, 08 Mar 2017 18:32:22 GMT):
Did that fix the error?

jjason (Wed, 08 Mar 2017 18:36:48 GMT):
Well...I ran into another problem first. A previous validator seems to have grabbed TCP port 40000 and won't let go (netstat tells me that the port is in ESTABLISHED state). I cannot find the process.... :(

boydjohnson (Wed, 08 Mar 2017 18:37:09 GMT):
netstat -tp doesn't show a process?

jjason (Wed, 08 Mar 2017 18:39:03 GMT):
Thx...that was the magic flag I was reading through the man page for ;)

jjason (Wed, 08 Mar 2017 18:39:14 GMT):
Saved me a couple of minutes of reading....

jjason (Wed, 08 Mar 2017 18:40:05 GMT):
Now I just have to wait for TIME_WAIT period to expire....

jjason (Wed, 08 Mar 2017 19:58:40 GMT):
@boydjohnson Thanks...got past that error and ran into other problems, working through them.

jsmitchell (Wed, 08 Mar 2017 19:59:52 GMT):
@jjason: I have seen address in use errors too. Does lsof -i :40000 show anything?

jsmitchell (Wed, 08 Mar 2017 20:02:25 GMT):
@boydjohnson: suggestions that linger settings may help

jsmitchell (Wed, 08 Mar 2017 20:02:38 GMT):
TIME_WAIT: https://hea-www.harvard.edu/~fine/Tech/addrinuse.html

boydjohnson (Wed, 08 Mar 2017 20:15:58 GMT):
@jsmitchell Thanks for that link, I added the linger stuff to the sdk, but somehow overlooked in the validator shutdown PR. I will add that right now.

jjason (Wed, 08 Mar 2017 21:47:57 GMT):
@jsmitchell Thanks. I'll try lsof next time. wrt TIME_WAIT, easy to kill transaction processors before killing validator....and although I like not having to stop transaction processors every time I restart the validator, if it means not waiting for the timeout....

yairfr (Thu, 09 Mar 2017 07:21:47 GMT):
Has joined the channel.

yairfr (Thu, 09 Mar 2017 15:06:36 GMT):
Hi, I'm trying to set up sawtoothlake for the first time and encountered problems.

yairfr (Thu, 09 Mar 2017 15:08:43 GMT):
It looks like it relates to the fact that I'm behinf a proxy. Figured to to set it up for vagrant, but it seems that I need to apply some more settings for npm :(

boydjohnson (Thu, 09 Mar 2017 15:11:08 GMT):
Hi @yairfr, are you using the 0-7 branch or master/0-8?

yairfr (Thu, 09 Mar 2017 15:22:23 GMT):
I think 0.8, but how can I verify?

yairfr (Thu, 09 Mar 2017 15:23:02 GMT):
(At least i try to follow the 0.8 instructions)

yairfr (Thu, 09 Mar 2017 15:24:19 GMT):
git clone https://github.com/IntelLedger/sawtooth-core.git

boydjohnson (Thu, 09 Mar 2017 15:41:49 GMT):
git clone will get you the current master, which we call 0-8. If git status says you are on branch master then that would tell you also. You mentioned npm, are you running the javascript sdk transaction processor?

yairfr (Thu, 09 Mar 2017 15:42:48 GMT):
Your branch is up-to-date with 'origin/master'.

yairfr (Thu, 09 Mar 2017 15:42:57 GMT):
anyway made some progress.

yairfr (Thu, 09 Mar 2017 15:43:13 GMT):
it seems that npm should respect the procy env cariables

yairfr (Thu, 09 Mar 2017 15:43:20 GMT):
those are correctly defined

yairfr (Thu, 09 Mar 2017 15:44:33 GMT):
there was a problem with the host file i bypassed it by adding ubuntu-xenial as an alias to localhost, it solved that problem

yairfr (Thu, 09 Mar 2017 15:44:44 GMT):
tthe current one is more difficult

yairfr (Thu, 09 Mar 2017 15:45:06 GMT):
npm install -g yarn fails

yairfr (Thu, 09 Mar 2017 15:45:19 GMT):
but no meaningful data in the logs

yairfr (Thu, 09 Mar 2017 15:46:11 GMT):
I'm still tryingto setup --> vagrant up

boydjohnson (Thu, 09 Mar 2017 15:49:52 GMT):
http://stackoverflow.com/questions/7559648/is-there-a-way-to-make-npm-install-the-command-to-work-behind-proxy This post talks about setting up npm for use behind a proxy. You have probably already seen it/done it.

yairfr (Thu, 09 Mar 2017 15:54:29 GMT):
I wasn't able to bring up the VM even once... so I can't configure npm. and it seems that the proxy environment variables are not propagated to the VM

yairfr (Thu, 09 Mar 2017 15:55:30 GMT):
FWIW i'm folowing http://intelledger.github.io/0.8/tutorial.html and stuck in the first vagrant VM setup

boydjohnson (Thu, 09 Mar 2017 15:56:22 GMT):
If you are getting to the install_js.sh script I would think the VM is running but not provisioned. After vagrant up exits with an error, what does vagrant status tell you?

yairfr (Thu, 09 Mar 2017 15:58:30 GMT):
running...

boydjohnson (Thu, 09 Mar 2017 15:59:29 GMT):
So there are a few things you could try. If you can vagrant ssh into the vm you could try setting up the proxy settings for npm and then manually running the install_js.sh script.

yairfr (Thu, 09 Mar 2017 16:08:43 GMT):
defenitly not a proxy problem since apt was successful. I'll try to see why npm can't install yarn

jsmitchell (Thu, 09 Mar 2017 16:09:15 GMT):
are you behind a corporate firewall that is selectively blocking sites?

yairfr (Thu, 09 Mar 2017 16:09:33 GMT):
yes

yairfr (Thu, 09 Mar 2017 16:10:03 GMT):
I'm being tempted just to that the whole thing home and set it there...

boydjohnson (Thu, 09 Mar 2017 16:14:20 GMT):
I might be wrong, but I thought npm uses https by default and apt uses http by default, so that might be an issue.

yairfr (Thu, 09 Mar 2017 16:15:56 GMT):
FFX seems to be able to get to https://registry.npmjs.org/yarn but the log show this error " https://registry.npmjs.org/yarn info retry will retry, error on last attempt: Error: tunneling socket could not be established, cause=socket hang up"

yairfr (Thu, 09 Mar 2017 16:16:26 GMT):
GTG will continue working on this next week... thank for helping me make ptogress

boydjohnson (Thu, 09 Mar 2017 16:16:42 GMT):
No problem. Good luck @yairfr.

jsmitchell (Thu, 09 Mar 2017 16:27:03 GMT):
@pschwarz: looks like https://github.com/hyperledger/sawtooth-core/pull/177 is ready to merge

Dan (Thu, 09 Mar 2017 17:24:51 GMT):
Does the `sawtooth admin genesis` command depend on having first generated config transactions?

boydjohnson (Thu, 09 Mar 2017 17:26:45 GMT):
@Dan No, you *can* (and in a production environment would) give it batches created by sawtooth config, but you can create genesis data without any batches.

Dan (Thu, 09 Mar 2017 17:29:19 GMT):
Ok, so it won't error, but I can I use that genesis batch in a meaningful way? i.e. if I was going to explain how to use the system to someone is the expected flow 1. create config transactions 2 . 3. create the genesis batch consuming transactions from 1 and 2?

boydjohnson (Thu, 09 Mar 2017 17:32:03 GMT):
Yes.

Dan (Thu, 09 Mar 2017 17:32:22 GMT):
thx

pschwarz (Thu, 09 Mar 2017 17:43:36 GMT):
@jsmitchell merged

GanapathyRajalingam (Thu, 09 Mar 2017 18:16:28 GMT):
Has joined the channel.

Dan (Fri, 10 Mar 2017 15:42:50 GMT):
@cianx what's an example of when consensus.blockpublisher.initialize_block(...) would return false?

cianx (Fri, 10 Mar 2017 15:46:52 GMT):
Validator does not have valid sign up info registered in global state for poet.

cianx (Fri, 10 Mar 2017 15:47:52 GMT):
Any case where it is a violation of the consensus algorithm for this validator to claim a block.

JatinderBali (Fri, 10 Mar 2017 15:51:17 GMT):
Has joined the channel.

Dan (Fri, 10 Mar 2017 20:24:16 GMT):
when addressing PR feedback, what does the author need to do to signal reviewers he or she is done with corrections? For example, @toddojala updated his PR but I didn't get it back in my review queue.

agunde (Fri, 10 Mar 2017 20:26:50 GMT):
The author needs to re-add them as a reviewer, by going back to the reviewer box, and adding the reviewer again the same way they had originally. This will reset the fact that you have a review request without removing your previously requested changes.

Dan (Fri, 10 Mar 2017 20:39:31 GMT):
thx @agunde

Dan (Fri, 10 Mar 2017 21:00:19 GMT):
What's the command to start the REST service?

boydjohnson (Fri, 10 Mar 2017 21:01:01 GMT):
rest_api

Dan (Fri, 10 Mar 2017 21:01:58 GMT):
think that's intuitive do you? :P

boydjohnson (Fri, 10 Mar 2017 21:02:24 GMT):
Ha!

Dan (Fri, 10 Mar 2017 21:05:49 GMT):
so the pattern looks a little different than bringing up the transaction processors. ```ubuntu@ubuntu-xenial:/project/sawtooth-core$ rest_api -v tcp://127.0.0.1:40000 usage: rest_api [-h] [--port PORT] [--host HOST] [--stream-url STREAM_URL] [--timeout TIMEOUT]` rest_api: error: unrecognized arguments: -v tcp://127.0.0.1:40000``` is host and port the host and port of the validator?

Dan (Fri, 10 Mar 2017 21:05:49 GMT):
so the pattern looks a little different than bringing up the transaction processors. `ubuntu@ubuntu-xenial:/project/sawtooth-core$ rest_api -v tcp://127.0.0.1:40000 usage: rest_api [-h] [--port PORT] [--host HOST] [--stream-url STREAM_URL] [--timeout TIMEOUT] rest_api: error: unrecognized arguments: -v tcp://127.0.0.1:40000` is host and port the host and port of the validator?

Dan (Fri, 10 Mar 2017 21:05:49 GMT):
so the pattern looks a little different than bringing up the transaction processors. `ubuntu@ubuntu-xenial:/project/sawtooth-core$ rest_api -v tcp://127.0.0.1:40000` `usage: rest_api [-h] [--port PORT] [--host HOST] [--stream-url STREAM_URL]` ` [--timeout TIMEOUT]` `rest_api: error: unrecognized arguments: -v tcp://127.0.0.1:40000` is host and port the host and port of the validator?

agunde (Fri, 10 Mar 2017 21:07:51 GMT):
No, the port and host is for the api, the stream-url is the validator

Dan (Fri, 10 Mar 2017 21:09:07 GMT):
trying to list blocks I get this:

Dan (Fri, 10 Mar 2017 21:09:09 GMT):
Error: (503) The validator is not yet ready to be queried

agunde (Fri, 10 Mar 2017 21:09:27 GMT):
Do you have a running validator?

Dan (Fri, 10 Mar 2017 21:09:33 GMT):
yet the rest_api process says this:

Dan (Fri, 10 Mar 2017 21:09:33 GMT):
Handling GET request for /blocks

Dan (Fri, 10 Mar 2017 21:09:46 GMT):
lemme make sure the validator is still alive...

Dan (Fri, 10 Mar 2017 21:10:05 GMT):
[21:08:37.578 DEBUG interconnect] ServerThread sending CLIENT_BLOCK_LIST_RESPONSE to b'6a1898424cb55466'

boydjohnson (Fri, 10 Mar 2017 21:10:55 GMT):
Oh, so the validator can be in a not good state and it will say that, even though the validator process is there.

Dan (Fri, 10 Mar 2017 21:11:45 GMT):
so it looks like the validator thinks it's responding to the client, but the client is like hey that validator isn't ready yet.

Dan (Fri, 10 Mar 2017 21:11:45 GMT):
so it looks like the validator thinks its responding to the client, but the client is like hey that validator isn't ready yet.

Dan (Fri, 10 Mar 2017 21:11:57 GMT):
and the rest_api in the middle is like everything is cool.

boydjohnson (Fri, 10 Mar 2017 21:12:18 GMT):
What do your startup logging statements look like? Do you have a "building on top of block None"?

Dan (Fri, 10 Mar 2017 21:12:40 GMT):
I built on top of block f4343

Dan (Fri, 10 Mar 2017 21:13:35 GMT):
oh I take that back

Dan (Fri, 10 Mar 2017 21:14:28 GMT):
it is on top of None

Dan (Fri, 10 Mar 2017 21:14:32 GMT):
'[20:46:04.783 INFO publisher] Now building on top of block: None`

Dan (Fri, 10 Mar 2017 21:14:32 GMT):
`[20:46:04.783 INFO publisher] Now building on top of block: None`

agunde (Fri, 10 Mar 2017 21:14:37 GMT):
Did you run genesis?

jjason (Fri, 10 Mar 2017 21:14:47 GMT):
Is your computer on?>

Dan (Fri, 10 Mar 2017 21:14:49 GMT):
genesis w/ no config or anything

Dan (Fri, 10 Mar 2017 21:15:05 GMT):
wait... @jjason how do you check?

jjason (Fri, 10 Mar 2017 21:15:34 GMT):
First, you need to go down to the breaker box in your garage/basement/...

Dan (Fri, 10 Mar 2017 21:15:56 GMT):
@agunde @boydjohnson don't worry about it. I'm just trying to get to canonical operation so I can double check the tutorial. I think this is close enough.

agunde (Fri, 10 Mar 2017 21:17:32 GMT):
Okay

boydjohnson (Fri, 10 Mar 2017 21:17:51 GMT):
Thanks @Dan, it is good to get a n+1 pair of eyes on things.

Dan (Fri, 10 Mar 2017 21:18:13 GMT):
I do have many eyes.

agunde (Fri, 10 Mar 2017 21:18:22 GMT):
More than 2?

Dan (Fri, 10 Mar 2017 21:18:35 GMT):
I'm like a walking potato

boydjohnson (Fri, 10 Mar 2017 21:18:52 GMT):
We need an Eye of Sauron bug seer.

Dan (Fri, 10 Mar 2017 21:19:10 GMT):
n+1 eyes where n=i+1

Romanix (Sat, 11 Mar 2017 21:17:38 GMT):
Does Sawtooth Lake have a UTXO model? If not, is it more like Ethereum?

jsmitchell (Sat, 11 Mar 2017 21:20:08 GMT):
Hi @Romanix: Sawtooth pushes these implementation details out to Transaction Families. It would be easy to write a transaction family that does UTXO, or a transaction family that stores and executes bytecode 'smart contracts'

jsmitchell (Sat, 11 Mar 2017 21:20:33 GMT):
Both types of transactions could even coexist on the same instance of the blockchain

Romanix (Sat, 11 Mar 2017 21:42:29 GMT):
@jsmitchell Thanks.

jsmitchell (Sat, 11 Mar 2017 21:43:06 GMT):
Have you taken a look at our 0.8 transaction family documentation?

jsmitchell (Sat, 11 Mar 2017 21:44:28 GMT):
http://intelledger.github.io/0.8/txn_family_tutorial.html

jsmitchell (Sat, 11 Mar 2017 21:44:52 GMT):
And http://intelledger.github.io/0.8/architecture/transactions_and_batches.html

boydjohnson (Mon, 13 Mar 2017 14:27:27 GMT):
@agunde noticed that sawtooth admin keygen doesn't allow you to specify where the key goes, and it is in a different place than, by default, the validator expects it.

boydjohnson (Mon, 13 Mar 2017 14:28:22 GMT):
@amundson ^

Dan (Mon, 13 Mar 2017 14:47:48 GMT):
Would others find the PR channel more useful it it was just announcing a PR created? It's a little too noisy for my use right now.

pschwarz (Mon, 13 Mar 2017 14:48:19 GMT):
Yes - it's way too noisy

cianx (Mon, 13 Mar 2017 17:21:59 GMT):
@toddojala @dan can we make http://intelledger.github.io/ default to 0.8 instead of 0.7?

jsmitchell (Mon, 13 Mar 2017 17:23:57 GMT):
@rbuysse: ^

rbuysse (Mon, 13 Mar 2017 17:26:07 GMT):
that is achievable

boydjohnson (Mon, 13 Mar 2017 17:28:26 GMT):
@agunde amends my previous comment. The sawtooth admin keygen command can be given a directory to put the key in. That would fix cluster.

toddojala (Mon, 13 Mar 2017 18:19:07 GMT):
@cianx yes!

adamludvik (Mon, 13 Mar 2017 18:48:58 GMT):
Hey all, I added a new build step that needs to happen before run_tests will pass. Putting up a PR now, but the extra build steps are listed in the top of docker/sawtooth-build-java and docker/sawtooth-build-javascript

adamludvik (Mon, 13 Mar 2017 18:48:58 GMT):
Hey all, I added a new build step that needs to happen before run_tests will pass. Putting up a PR now to add this to build_all, but the extra build steps are listed in the top of docker/sawtooth-build-java and docker/sawtooth-build-javascript

pschwarz (Mon, 13 Mar 2017 22:11:50 GMT):
With the merge of https://github.com/hyperledger/sawtooth-core/pull/175, creating config transactions for the genesis block has changed slightly. Now, you do the following:```

pschwarz (Mon, 13 Mar 2017 22:11:50 GMT):
With the merge of https://github.com/hyperledger/sawtooth-core/pull/175, creating config transactions for the genesis block has changed slightly. Now, you do the following:

pschwarz (Mon, 13 Mar 2017 22:12:04 GMT):
```

pschwarz (Mon, 13 Mar 2017 22:12:04 GMT):
```

pschwarz (Mon, 13 Mar 2017 22:12:04 GMT):
```

pschwarz (Mon, 13 Mar 2017 22:14:06 GMT):
``` $ sawtooth config proposal set -k -o config.batch my.setting.1=foo my.setting.2=bar ```

pschwarz (Mon, 13 Mar 2017 22:14:06 GMT):
``` $ sawtooth config proposal create -k -o config.batch my.setting.1=foo my.setting.2=bar ```

pschwarz (Mon, 13 Mar 2017 22:15:06 GMT):
To submitted settings to a running validator: ``` $ sawtooth config proposal set -k --url http:// my.setting.1=foo my.setting.2=bar ```

pschwarz (Mon, 13 Mar 2017 22:15:06 GMT):
To submitted settings to a running validator: ``` $ sawtooth config proposal create -k --url http:// my.setting.1=foo my.setting.2=bar ```

pschwarz (Mon, 13 Mar 2017 22:15:37 GMT):
Or, omit the `--url`, and submit against `localhost`

pschwarz (Mon, 13 Mar 2017 22:15:51 GMT):
This does require running the `rest_api` process

jjason (Tue, 14 Mar 2017 04:30:57 GMT):
@adamludvik Running `docker run -v $(pwd):/project/sawtooth-core sawtooth-build-javascript` fails in vagrant on Windows. Crappy Windows does not support symlinks. Adding `--no-bin-links` to `yarn install --force` in both `bin/build_javascript` and `bin/build_intkey_javascript` seems to have fixed problem for me.

jjason (Tue, 14 Mar 2017 04:53:56 GMT):
BTW, don't know what the side effects of this will be on OSes that actually support symlinks. Sigh.

adamludvik (Tue, 14 Mar 2017 12:14:02 GMT):
@jjason thanks. I am working on PR 201 to cleanup a few things. Will add this and make you a reviewer.

Dan (Tue, 14 Mar 2017 13:07:53 GMT):
@rbuysse not withstanding your other work, maybe you and Ry can put your heads together on a narrower trigger for #sawtoothlake-pr-review

amundson (Tue, 14 Mar 2017 13:35:21 GMT):
@Dan there is hook (some Javascript code) on the rocket.chat side. many of the lines in #sawtoothlake-pr-review are assignee/reviewer related, so if we cut them out, it should be better

Dan (Tue, 14 Mar 2017 13:36:37 GMT):
Sounds like something only @Ry has access to?

jsmitchell (Tue, 14 Mar 2017 13:42:25 GMT):
here's the hook: https://gist.github.com/rjones-lf/fbe68e5fc9cef6ae6a8847df7d08f739

jsmitchell (Tue, 14 Mar 2017 13:42:32 GMT):
@jwagantall also can help

adamludvik (Tue, 14 Mar 2017 14:17:04 GMT):
@drozd and I are now seeing the tests pass with the startup sleep statements removed. Was there a change that went in recently that this is attributed to? I would like to remove them if possible, but want to understand why first.

amundson (Tue, 14 Mar 2017 14:28:56 GMT):
@adamludvik - recent changes to add jdk/js stuff to run_tests breaks it on vagrant without extra work (installing java, extra build steps) - is this addressed in anything you have in-progress?

adamludvik (Tue, 14 Mar 2017 14:29:12 GMT):
yes

adamludvik (Tue, 14 Mar 2017 14:29:37 GMT):
https://github.com/hyperledger/sawtooth-core/pull/201

adamludvik (Tue, 14 Mar 2017 14:32:26 GMT):
@jjason pointed out one more thing I needed to do, which I just pushed up

adamludvik (Tue, 14 Mar 2017 14:32:58 GMT):
Tested locally and it is passing, just need to verify it works on Jenkins now

amundson (Tue, 14 Mar 2017 14:33:44 GMT):
@adamludvik - I will test locally

agunde (Tue, 14 Mar 2017 14:34:02 GMT):
You may need to update virtualBox for it to work

adamludvik (Tue, 14 Mar 2017 14:34:46 GMT):
Thanks. I think you need VirtualBox >= 5.1.10

amundson (Tue, 14 Mar 2017 14:34:54 GMT):
@adamludvik - you should rebase so you have the install_cryptograph.sh fix

adamludvik (Tue, 14 Mar 2017 14:36:32 GMT):
will do. you can also test it on mac if you have docker installed.

adamludvik (Tue, 14 Mar 2017 14:36:32 GMT):
will do. you ~can also~ should be able to test it on mac if you have docker installed.

adamludvik (Tue, 14 Mar 2017 14:38:23 GMT):
Rebase is done.

adamludvik (Tue, 14 Mar 2017 14:47:08 GMT):
As soon as you are done testing it should be good to merge.

paulw (Tue, 14 Mar 2017 18:09:33 GMT):
Has joined the channel.

jsmitchell (Tue, 14 Mar 2017 19:31:17 GMT):
@agunde: I think I am seeing a regression in the completer where the publisher is reporting that it has batches with missing deps

jsmitchell (Tue, 14 Mar 2017 19:31:23 GMT):
```[19:29:53.868 DEBUG publisher] Transaction rejected due missing dependency, transaction 098d5100747cb71a53c3db3f8ef08afafdd6b48d5bf11e3a3673bc336b54ab7021e967949472ecdd6521571e3cae45f984b79055b264976a4173dd35bc332b22 depends on b02e001be0fbd1fddb7e4c380bbde1e9b32c6f4249c5ae46980e6102e0c61d7a029bc72099a1495fa357a69145963511231b2275343c6e81946eb496a10ae634 [19:29:53.869 DEBUG publisher] Dropping batch due to missing dependencies: d98761270e9f335b4bbd8da28e010fcb5a21204aaf9fc5417b8de6550538318953757a5914bb5b0de6b2c397741d5e956763da8c4363f82ba0d31623df06183c```

agunde (Tue, 14 Mar 2017 19:32:19 GMT):
Is this on master or your peering branch?

jsmitchell (Tue, 14 Mar 2017 19:32:34 GMT):
my branch

jsmitchell (Tue, 14 Mar 2017 19:46:23 GMT):
@agunde: PR raised on that branch https://github.com/hyperledger/sawtooth-core/pull/213

jsmitchell (Tue, 14 Mar 2017 19:46:32 GMT):
but I suspect the same thing would be visible on master

jsmitchell (Tue, 14 Mar 2017 19:46:34 GMT):
i'll check

jsmitchell (Tue, 14 Mar 2017 19:48:32 GMT):
yeah, same thing on master

jsmitchell (Tue, 14 Mar 2017 19:48:43 GMT):
start up a single validator and send it a couple of thousand intkey batches

jsmitchell (Tue, 14 Mar 2017 19:48:52 GMT):
it'll start spewing those

agunde (Tue, 14 Mar 2017 19:51:38 GMT):
okay i will take a look when i have a moment

pschwarz (Tue, 14 Mar 2017 20:13:48 GMT):
Sorry all mistyped the config proposal usage (edited in the original message: ``` $ sawtooth config proposal create ... ```

pschwarz (Tue, 14 Mar 2017 20:13:54 GMT):
not `set`

balashevich (Wed, 15 Mar 2017 12:37:43 GMT):
Has joined the channel.

victorsintnicolaas (Wed, 15 Mar 2017 13:13:24 GMT):
Has joined the channel.

TomBarnes (Wed, 15 Mar 2017 22:28:28 GMT):
When a client posts a /batches message directly to a validator via the REST interface, should it use the same key to sign both the transaction header and the batch, or should it use separate keys? It seams to me that it should use the same key, because it is acting as both the originator of the transaction and as the batcher.

jsmitchell (Wed, 15 Mar 2017 22:28:55 GMT):
yep, same key is the simple case

jsmitchell (Wed, 15 Mar 2017 22:29:53 GMT):
batcher_pubkey in the transaction header must also match

jsmitchell (Wed, 15 Mar 2017 22:30:27 GMT):
if a transaction signer wants to give another signer permission to include the transaction in a batch, then batcher_pubkey must be set appropriately in the transaction header

TomBarnes (Wed, 15 Mar 2017 22:50:24 GMT):
ok - thanks

ShowKobayashi (Thu, 16 Mar 2017 08:06:48 GMT):
Has joined the channel.

PacoEcotrust (Thu, 16 Mar 2017 11:35:31 GMT):
Has left the channel.

PacoEcotrust (Thu, 16 Mar 2017 11:44:56 GMT):
Has joined the channel.

amundson (Thu, 16 Mar 2017 17:41:04 GMT):
For http://intelledger.github.io/0.8/transaction_family_specifications/config_transaction_family.html, we could start by removing authorization_type and have it always be ballot-mode.

amundson (Thu, 16 Mar 2017 17:41:31 GMT):
Then have the CLI fill in sawtooth.config.vote.authorized_keys by default with it's authorized key? it is less clear if this works from a flow perspective, but the idea would be to usually set it.

jsmitchell (Thu, 16 Mar 2017 17:41:47 GMT):
@amundson: what's this in context of?

amundson (Thu, 16 Mar 2017 17:42:21 GMT):
simplifying the config txn family to have a more secure default mode (where at least one key is set)

amundson (Thu, 16 Mar 2017 17:46:42 GMT):
my original concept with respect to authorization_type no longer applies now that we have the broader idea of being a reference implementation that can be replaced

pschwarz (Thu, 16 Mar 2017 18:09:09 GMT):
That would probably work

pschwarz (Thu, 16 Mar 2017 18:10:14 GMT):
We might want have the genesis tool require input batches, as well, so at least some level of on-chain settings have been applied

adamludvik (Thu, 16 Mar 2017 18:20:53 GMT):
# Docker Build Update With the recent changes, it should now be possible to build sawtooth and run tests from an arbitrary environment that has docker installed and configured to run with a linux kernel. Some substantial changes have gone into the build process to accomplish this, and this document is a quick overview of the process for current developers. ## tl;dr If you are just doing python development for sawtooth, you should now be able to build and run the tests from vagrant or Mac by doing the following: ``` $ ./bin/build_all -l python $ ./bin/run_tests -x java_sdk -x javascript_sdk ``` The build process may be faster if it is run from Mac instead of within vagrant. To install docker on Mac, download https://download.docker.com/mac/stable/Docker.dmg, drag docker into applications, and open it. You will also need to have python3 and PyYAML installed for run_tests to work. ## Building Sawtooth The old build_all script has been moved to build_python and docker_build_all has been moved to build_all. The “new” build_all script handles building all the docker images that are usually needed by a developer. Some of these build images are also used to build the source code in sawtooth. This includes running protogen, maven, yarn, etc. To make development in one language easier, it is now possible to pass flags to the build_all script. ``` sawtooth-lake/sawtooth-core » ./bin/build_all -h Usage: ./bin/build_all [-l Language] [-x Language] Options: -h print usage and exit -l Language only build the specified language(s) -x Language exclude the specified language(s) ``` The -l flag can be used multiple times to hand pick the languages that should be built. The -x flag can be used to pick languages that should be skipped. For example, to only build the python components of sawtooth you would do: ``` $ build_all -l python ``` Or to skip building java but still build python and javascript: ``` $ build_all -x java ``` One final note, if you are running in vagrant and do not want to build any of the docker images, you can still run build_python to just build the python code without using docker in anyway: $ build_python ## Running Tests The run_tests script has also been changed to accommodate finer grained control of which tests should run. run_tests also no longer builds anything before running the tests, so you will need to do build_all before run_tests now. ``` sawtooth-lake/sawtooth-core » ./bin/run_tests -h Usage: ./bin/run_tests [-m Module] [-s Module] (-m and -x can be passed multiple times) Options: -h print usage and exit -m Module only run tests of specified module -x Module exclude the tests of specified module Current modules: cli java_sdk javascript_sdk poet python_sdk rest_api signing validator ``` The -m flag can now be passed multiple times to hand pick which modules should be tested. Alternatively, the -s flag can also be passed multiple times to skip certain modules. For example, to only run the python_sdk and poet modules, you would do: ``` $ run_tests -m python_sdk -m poet ``` Or to skip the java and javascript sdk tests do: ``` $ run_tests -x java_sdk -x javascript_sdk ``` Running Lint The run_lint script has not changed, however it is now being run inside the sawtooth-build-python container on Jenkins. You can still run the script locally from outside of any container (in Vagrant) but to get the same results as you would on Jenkins, you can instead do (after build_all): ``` $ docker run -v $(pwd):/project/sawtooth-core sawtooth-build-python ./bin/run_lint ```

adamludvik (Thu, 16 Mar 2017 18:20:53 GMT):
# Docker Build Update With the recent changes, it should now be possible to build sawtooth and run tests from an arbitrary environment that has docker installed and configured to run with a linux kernel. Some substantial changes have gone into the build process to accomplish this, and this document is a quick overview of the process for current developers. ## tl;dr If you are just doing python development for sawtooth, you should now be able to build and run the tests from vagrant or Mac by doing the following: ``` $ ./bin/build_all -l python $ ./bin/run_tests -x java_sdk -x javascript_sdk ``` The build process may be faster if it is run from Mac instead of within vagrant. To install docker on Mac, download https://download.docker.com/mac/stable/Docker.dmg, drag docker into applications, and open it. You will also need to have python3 and PyYAML installed for run_tests to work. ## Building Sawtooth The old build_all script has been moved to build_python and docker_build_all has been moved to build_all. The “new” build_all script handles building all the docker images that are usually needed by a developer. Some of these build images are also used to build the source code in sawtooth. This includes running protogen, maven, yarn, etc. To make development in one language easier, it is now possible to pass flags to the build_all script. ``` sawtooth-lake/sawtooth-core » ./bin/build_all -h Usage: ./bin/build_all [-l Language] [-x Language] Options: -h print usage and exit -l Language only build the specified language(s) -x Language exclude the specified language(s) ``` The -l flag can be used multiple times to hand pick the languages that should be built. The -x flag can be used to pick languages that should be skipped. For example, to only build the python components of sawtooth you would do: ``` $ build_all -l python ``` Or to skip building java but still build python and javascript: ``` $ build_all -x java ``` One final note, if you are running in vagrant and do not want to build any of the docker images, you can still run build_python to just build the python code without using docker in anyway: $ build_python ## Running Tests The run_tests script has also been changed to accommodate finer grained control of which tests should run. run_tests also no longer builds anything before running the tests, so you will need to do build_all before run_tests now. ``` sawtooth-lake/sawtooth-core » ./bin/run_tests -h Usage: ./bin/run_tests [-m Module] [-s Module] (-m and -x can be passed multiple times) Options: -h print usage and exit -m Module only run tests of specified module -x Module exclude the tests of specified module Current modules: cli java_sdk javascript_sdk poet python_sdk rest_api signing validator ``` The -m flag can now be passed multiple times to hand pick which modules should be tested. Alternatively, the -s flag can also be passed multiple times to skip certain modules. For example, to only run the python_sdk and poet modules, you would do: ``` $ run_tests -m python_sdk -m poet ``` Or to skip the java and javascript sdk tests do: ``` $ run_tests -x java_sdk -x javascript_sdk ``` Running LintThe run_lint script has not changed, however it is now being run inside the sawtooth-build-python container on Jenkins. You can still run the script locally from outside of any container (in Vagrant) but to get the same results as you would on Jenkins, you can instead do (after build_all): ``` $ docker run -v $(pwd):/project/sawtooth-core sawtooth-build-python ./bin/run_lint ```

adamludvik (Thu, 16 Mar 2017 18:20:53 GMT):
# Docker Build Update With the recent changes, it should now be possible to build sawtooth and run tests from an arbitrary environment that has docker installed and configured to run with a linux kernel. Some substantial changes have gone into the build process to accomplish this, and this document is a quick overview of the process for current developers. ## tl;dr If you are just doing python development for sawtooth, you should now be able to build and run the tests from vagrant or Mac by doing the following: ``` $ ./bin/build_all -l python $ ./bin/run_tests -x java_sdk -x javascript_sdk ``` The build process may be faster if it is run from Mac instead of within vagrant. To install docker on Mac, download https://download.docker.com/mac/stable/Docker.dmg, drag docker into applications, and open it. You will also need to have python3 and PyYAML installed for run_tests to work. ## Building Sawtooth The old build_all script has been moved to build_python and docker_build_all has been moved to build_all. The “new” build_all script handles building all the docker images that are usually needed by a developer. Some of these build images are also used to build the source code in sawtooth. This includes running protogen, maven, yarn, etc. To make development in one language easier, it is now possible to pass flags to the build_all script. ``` sawtooth-lake/sawtooth-core » ./bin/build_all -h Usage: ./bin/build_all [-l Language] [-x Language] Options: -h print usage and exit -l Language only build the specified language(s) -x Language exclude the specified language(s) ``` The -l flag can be used multiple times to hand pick the languages that should be built. The -x flag can be used to pick languages that should be skipped. For example, to only build the python components of sawtooth you would do: ``` $ build_all -l python ``` Or to skip building java but still build python and javascript: ``` $ build_all -x java ``` One final note, if you are running in vagrant and do not want to build any of the docker images, you can still run build_python to just build the python code without using docker in anyway: $ build_python ## Running Tests The run_tests script has also been changed to accommodate finer grained control of which tests should run. run_tests also no longer builds anything before running the tests, so you will need to do build_all before run_tests now. ``` sawtooth-lake/sawtooth-core » ./bin/run_tests -h Usage: ./bin/run_tests [-m Module] [-s Module] (-m and -x can be passed multiple times) Options: -h print usage and exit -m Module only run tests of specified module -x Module exclude the tests of specified module Current modules: cli java_sdk javascript_sdk poet python_sdk rest_api signing validator ``` The -m flag can now be passed multiple times to hand pick which modules should be tested. Alternatively, the -s flag can also be passed multiple times to skip certain modules. For example, to only run the python_sdk and poet modules, you would do: ``` $ run_tests -m python_sdk -m poet ``` Or to skip the java and javascript sdk tests do: ``` $ run_tests -x java_sdk -x javascript_sdk ``` ## Running Lint The run_lint script has not changed, however it is now being run inside the sawtooth-build-python container on Jenkins. You can still run the script locally from outside of any container (in Vagrant) but to get the same results as you would on Jenkins, you can instead do (after build_all): ``` $ docker run -v $(pwd):/project/sawtooth-core sawtooth-build-python ./bin/run_lint ```

Dan (Thu, 16 Mar 2017 18:22:55 GMT):
@adamludvik wins :medal: for biggest post. :-D Seriously tho thanks for the update.

adamludvik (Thu, 16 Mar 2017 18:23:23 GMT):
Tried to upload it as file but :rocket: complained...

jjason (Thu, 16 Mar 2017 22:03:46 GMT):
what's the magic sauce for getting through Intel proxy?

jjason (Thu, 16 Mar 2017 22:03:59 GMT):
Besides connecting to the internet outside the firewall ;)

rbuysse (Thu, 16 Mar 2017 22:18:09 GMT):
@jjason we're passing four environment variables from the local machine into the docker images with the build_all script

rbuysse (Thu, 16 Mar 2017 22:18:58 GMT):
if you have http_proxy, https_proxy, HTTP_PROXY, and/or HTTPS_PROXY set on your local machine, those will get passed in to the docker containers.

jjason (Thu, 16 Mar 2017 22:19:09 GMT):
Shoot....I have them set.

rbuysse (Thu, 16 Mar 2017 22:19:30 GMT):
hmmmmmm

jsmitchell (Thu, 16 Mar 2017 22:19:34 GMT):
unplug your ethernet cable and see if anything is jamming it

jjason (Thu, 16 Mar 2017 22:19:35 GMT):
HTTPS_PROXY=http://proxy-us.intel.com:912 HTTP_PROXY=http://proxy-us.intel.com:911 http_proxy=http://proxy-us.intel.com:911 https_proxy=http://proxy-us.intel.com:912

rbuysse (Thu, 16 Mar 2017 22:20:00 GMT):
are you running build_all in vagrant?

jjason (Thu, 16 Mar 2017 22:20:03 GMT):
Oh crap. The problem was that someone in the break room was using the microwave and it f***ed with my WiFi ;)

jjason (Thu, 16 Mar 2017 22:20:29 GMT):
Yes....in vagrant....I am relegated to that because of the Windows laptop.

jjason (Thu, 16 Mar 2017 22:21:00 GMT):
Um....I think that problem is that the token from the token ring fell out on the floor...let me find it and put it back in.

jjason (Thu, 16 Mar 2017 22:21:00 GMT):
Um....I think that problem is the the token from the token ring fell out on the floor...let me find it and put it back in.

jjason (Thu, 16 Mar 2017 22:21:00 GMT):
Um....I think the problem is that the token from the token ring fell out on the floor...let me find it and put it back in.

rbuysse (Thu, 16 Mar 2017 22:21:45 GMT):
and when you type "env" the proxy variables are set?

jjason (Thu, 16 Mar 2017 22:22:11 GMT):
ubuntu@ubuntu-xenial:/project/sawtooth-core$ env | grep http http_proxy=http://proxy-us.intel.com:911 https_proxy=http://proxy-us.intel.com:912 HTTPS_PROXY=http://proxy-us.intel.com:912 HTTP_PROXY=http://proxy-us.intel.com:911

rbuysse (Thu, 16 Mar 2017 22:22:36 GMT):
gimme a sec.

rbuysse (Thu, 16 Mar 2017 22:22:41 GMT):
I will test locally

jjason (Thu, 16 Mar 2017 22:25:56 GMT):
Also, while you are at that....I am having problems running `./bin/run_lint` locally in vagrant in that it is tripping up importing modules that I could swear that I have in `PYTHONPATH`. Anyone else or did I just mess up somewhere.

rbuysse (Thu, 16 Mar 2017 22:34:16 GMT):
well I guess the good news is that I am experiencing behavior here other than what I would expect

rbuysse (Thu, 16 Mar 2017 22:34:24 GMT):
gimme a bit to take better look.

jjason (Thu, 16 Mar 2017 22:35:47 GMT):
sure

jjason (Thu, 16 Mar 2017 22:36:14 GMT):
Time to go work at Starbucks...where they don't have no stinkin' firewall ;)

jjason (Thu, 16 Mar 2017 23:27:14 GMT):
Stand down everyone. False alarm. The problem was apparently with Intel IT. Everything seems to be getting through proxy successfully now. Thx @rbuysse

rbuysse (Thu, 16 Mar 2017 23:27:35 GMT):
^5 glad you're up and running.

jjason (Thu, 16 Mar 2017 23:27:52 GMT):
Please return to Defcon 5.

jjason (Thu, 16 Mar 2017 23:28:25 GMT):
A strange game. The only winning move is not to play. How about a nice game of chess?

sstone1 (Thu, 16 Mar 2017 23:50:16 GMT):
Has joined the channel.

amundson (Fri, 17 Mar 2017 03:03:48 GMT):
I bumped the tolkien ring, and it made a big thud as it hit the floor.

jsmitchell (Fri, 17 Mar 2017 03:04:07 GMT):
that one ring?

tiennv (Fri, 17 Mar 2017 09:58:35 GMT):
Has joined the channel.

tiennv (Fri, 17 Mar 2017 10:00:38 GMT):
Hi,

tiennv (Fri, 17 Mar 2017 10:01:25 GMT):
Sawtooth Lake is a private or public blockchain?

tiennv (Fri, 17 Mar 2017 10:02:15 GMT):
I wan to know about privacy and security of Sawtooth Lake.

tiennv (Fri, 17 Mar 2017 10:02:35 GMT):
Please tell me what resources I can refer to?

Romanix (Fri, 17 Mar 2017 13:23:11 GMT):
@tiennv I think you can compare it with the Ethereum *Software*. The software is once used for the public Ethereum blockchain (mainnet), but is also used for various private blockchains.

jsmitchell (Fri, 17 Mar 2017 14:23:43 GMT):
@boydjohnson: I'm trying to do some things on exit of the validator. I put a finally: after the try: validator.start() in server/cli.py, but that is executed immediately. This behavior differs from doing the same thing in the transaction processors (it works as expected there). Do you have some advice on where to do this?

boydjohnson (Fri, 17 Mar 2017 15:02:31 GMT):
So this may sound strange, but I would put a try-except in finally in server/cli.py and except a KeyboardInterrupt there and do your shutdown stuff in the except clause.

Dan (Fri, 17 Mar 2017 16:49:15 GMT):
@boydjohnson you mentioned in passing something about integration tests failing on master or false negatives... I got lost in the double negatives. Are things clean or broken right now?

boydjohnson (Fri, 17 Mar 2017 16:51:10 GMT):
Everything is good, except locally I found a way to trick the run_docker_test script into giving me a green light when it should have been a red light. I will push up a small change to the run_docker_test script that will make it less easy to trick.

Dan (Fri, 17 Mar 2017 16:53:58 GMT):
Nothing wrong with a little extra green today ;)

boydjohnson (Fri, 17 Mar 2017 16:55:04 GMT):
We could add a "Luck of the Irish" feature to the test script.

jjason (Fri, 17 Mar 2017 16:57:46 GMT):
Sound like you need to write a unit test to test the unit test tester.

jjason (Fri, 17 Mar 2017 16:57:59 GMT):
@boydjohnson ^^^^

boydjohnson (Fri, 17 Mar 2017 17:00:03 GMT):
Yeah, yesterday @amundson and I were talking about a way to deterministically start multiple threads in a unittest, and @jsmitchell said we should work on it as a research topic in our own time. We can add a unittest test testing framework to the research.

moeentariq (Fri, 17 Mar 2017 17:11:46 GMT):
Has joined the channel.

jjason (Fri, 17 Mar 2017 17:11:47 GMT):
:warning::warning::warning::warning::warning::warning::warning::warning::warning: Just merged PR 228, which moves stuff around for PoET. It know that it will directly affect @pschwarz. Anyone else working on PoET or validator registry code will be affected as well. My apologies. https://github.com/hyperledger/sawtooth-core/pull/228 :warning::warning::warning::warning::warning::warning::warning::warning::warning:

jjason (Fri, 17 Mar 2017 17:14:10 GMT):
If you update, you will need to re-provision your vagrant VM because of updates.

boydjohnson (Fri, 17 Mar 2017 17:20:18 GMT):
Addendum to the Addendum: I had convinced @adamludvik that the script wasn't right, but it was. Turns out when you manually 'echo $?' but you had previously run 'echo ?' (IDK) it echos the return code of the previously echo command not the test script.

MadhavaReddy (Fri, 17 Mar 2017 17:52:19 GMT):
Has joined the channel.

pschwarz (Fri, 17 Mar 2017 18:08:38 GMT):
W

boydjohnson (Fri, 17 Mar 2017 18:44:39 GMT):
https://github.com/hyperledger/sawtooth-core/pull/217 PR 217 is passing tests consistently and has received a green check from @pschwarz. @jjason @jsmitchell @amundson @agunde @drozd

jsmitchell (Fri, 17 Mar 2017 18:47:06 GMT):
I find that PR intimidating

jsmitchell (Fri, 17 Mar 2017 18:48:11 GMT):
it is a failing inside me @boydjohnson, not you

boydjohnson (Fri, 17 Mar 2017 18:49:07 GMT):
As long as the tests don't fail, I am fine with personal failings.

zac (Fri, 17 Mar 2017 19:14:52 GMT):
Yo, various stake holders: I'm starting to build the paging controls for the ZMQ API

zac (Fri, 17 Mar 2017 19:15:10 GMT):
Does anyone have a preference on how paging works by default?

zac (Fri, 17 Mar 2017 19:16:05 GMT):
Specifically, do you want to just get everything if you don't specify any paging controls, or do you want everything to always be paginated?

zac (Fri, 17 Mar 2017 19:16:32 GMT):
(perhaps unless you send some "don't paginate" command)

jsmitchell (Fri, 17 Mar 2017 19:17:52 GMT):
@boydjohnson: your "one weird trick" for doing stuff on KeyboardInterrupt didn't work

zac (Fri, 17 Mar 2017 19:18:30 GMT):
Those are always just click bait

jsmitchell (Fri, 17 Mar 2017 19:20:08 GMT):
@zac: clients shouldn't be able to DOS the rest api by forming a request for 40GB of stuff

jsmitchell (Fri, 17 Mar 2017 19:20:53 GMT):
if pagination enforces smallish responses, we can use rate limiting to keep things sane

jsmitchell (Fri, 17 Mar 2017 19:25:09 GMT):
@boydjohnson: I'm on tenterhooks here

boydjohnson (Fri, 17 Mar 2017 19:25:48 GMT):
@jsmitchell Looking at server/cli now, it seems to have changed in the recent past. Looking at it, there is no way to run shutdown tasks. It appears that most threads are started non-daemon (seems good). Main thread exits after validator.start and the long running nature of the validator is that the other threads don't shutdown.

adamludvik (Fri, 17 Mar 2017 19:32:49 GMT):
I restarted LR1 with bond and intkey and it forked again right away. I am trying an easier workload to see if it can handle that. My gut reaction is that eitherranger

adamludvik (Fri, 17 Mar 2017 19:37:43 GMT):
I restarted LR1 running a bond and intkey workload and it forked again right away. My gut says that either the changes to stats lib are causing the most recent block id to be reported incorrectly, given the impression of forking, or that it is related to adding the CloudPassage/halo service to all our AWS boxes. @Tom Barnes @nconde can you please review the PR that went in again to see if you can come up with any theories? I am restarting LR1 again with an easier workload to see if that helps.

adamludvik (Fri, 17 Mar 2017 19:37:43 GMT):
I restarted LR1 running a bond and intkey workload and it forked again right away. My gut says that either the changes to stats lib are causing the most recent block ids to be reported incorrectly, giving the impression of forking, or that it is related to adding the CloudPassage/halo service to all our AWS boxes. @Tom Barnes @nconde can you please review the PR that went in again to see if you can come up with any theories? I am restarting LR1 again with an easier workload to see if that helps.

nconde (Fri, 17 Mar 2017 19:37:43 GMT):
Has joined the channel.

boydjohnson (Fri, 17 Mar 2017 19:40:10 GMT):
700a0b7194c27eaa177ca4ef5e9684d224e0dd7f @jsmitchell this commit removed the finally from validator cli. Are you saying if you put a finally: back in with validator.stop() and then do your shutdown things in a new except clause in that finally, it doesn't work?

jsmitchell (Fri, 17 Mar 2017 19:40:32 GMT):
``` try: yappi.set_clock_type('cpu') yappi.start(builtins=True) validator.start() except KeyboardInterrupt: print("Interrupted!", file=sys.stderr) sys.exit(1) except LocalConfigurationError as local_config_err: LOGGER.error(str(local_config_err)) sys.exit(1) except GenesisError as genesis_err: LOGGER.error(str(genesis_err)) sys.exit(1) except Exception as e: LOGGER.exception(e) sys.exit(1) finally: try: pass except KeyboardInterrupt: stats = yappi.get_func_stats() stats.print_all() yappi.get_thread_stats().print_all() stats.save("validator-callgrind.prof", type='callgrind')```

jsmitchell (Fri, 17 Mar 2017 19:40:35 GMT):
doesn't work

boydjohnson (Fri, 17 Mar 2017 19:41:27 GMT):
the pass should be validator.stop().

jsmitchell (Fri, 17 Mar 2017 19:41:57 GMT):
woot!

boydjohnson (Fri, 17 Mar 2017 19:42:19 GMT):
Does the woot! mean it worked?

jsmitchell (Fri, 17 Mar 2017 19:42:26 GMT):
what else would it mean?

jsmitchell (Fri, 17 Mar 2017 19:42:30 GMT):
HECK YES IT WORKED

boydjohnson (Fri, 17 Mar 2017 19:42:44 GMT):
Yay!

jsmitchell (Fri, 17 Mar 2017 19:43:05 GMT):
there is about to be callgrinding

Dan (Fri, 17 Mar 2017 19:43:19 GMT):
@adamludvik if it's not a lot of extra work you could try running LR1 with HEAD~1 and see if behavior holds.

adamludvik (Fri, 17 Mar 2017 19:44:51 GMT):
Sure. I am going to try just intkey on the current commit as I just started up that network, but will rollback to the latest version after.

adamludvik (Fri, 17 Mar 2017 19:47:36 GMT):
just an fyi, this does not affect the demonet or restarting it, as they are already pegged to a specific version

adamludvik (Fri, 17 Mar 2017 20:32:04 GMT):
Okay, LR1 seems to be doing fine with an 80TPM intkey workload. May just be that I left the settings for LR1 in a particularly nasty state. Going to set it back to the original 300TPM with the most recent commit.

Dan (Fri, 17 Mar 2017 21:30:01 GMT):
I'll be interested to hear how that goes @adamludvik

Dan (Fri, 17 Mar 2017 21:31:37 GMT):
In the future @here let's only commit critical fixes to 0.7.

zac (Fri, 17 Mar 2017 21:32:29 GMT):
Wow. Rocketchat's notification settings are . . . aggressive.

Dan (Fri, 17 Mar 2017 21:38:30 GMT):
Oh ya? was that pretty loud?

Dan (Fri, 17 Mar 2017 21:39:32 GMT):
It wouldn't let me do "@all". I took that to mean it wouldn't send a notice if you weren't logged in.

Dan (Fri, 17 Mar 2017 21:40:00 GMT):

Message Attachments

Dan (Fri, 17 Mar 2017 21:40:19 GMT):
rocket.cat ... cute eh?

jsmitchell (Fri, 17 Mar 2017 21:40:32 GMT):
hey @rocket.cat, can you turn off all notifications?

rocket.cat (Fri, 17 Mar 2017 21:40:32 GMT):
Has joined the channel.

Dan (Fri, 17 Mar 2017 21:41:00 GMT):
Do not taunt @rocket.cat

jjason (Fri, 17 Mar 2017 21:47:41 GMT):
I think that before you try to blast out to all, you need to either throw a yarn ball or shine a laser to distract @rocket.cat

adamludvik (Fri, 17 Mar 2017 21:48:56 GMT):

Message Attachments

boydjohnson (Fri, 17 Mar 2017 21:50:11 GMT):
I started @adamludvik on this path, but am not responsible for where the path took him.

jsmitchell (Fri, 17 Mar 2017 21:50:23 GMT):
did you do that by hand with graphviz?

boydjohnson (Fri, 17 Mar 2017 21:50:36 GMT):
pyreverse in pylint creates the dot file.

jsmitchell (Fri, 17 Mar 2017 21:51:05 GMT):
ok, so you could load the dot file into gephi and make the layout a lot better

adamludvik (Fri, 17 Mar 2017 21:51:13 GMT):
ahah

jsmitchell (Fri, 17 Mar 2017 21:51:21 GMT):
https://gephi.org/

boydjohnson (Fri, 17 Mar 2017 21:51:47 GMT):
Sweet.

jsmitchell (Fri, 17 Mar 2017 21:51:53 GMT):
the UI is fun to figure out

adamludvik (Fri, 17 Mar 2017 21:52:19 GMT):
I feel like I just upgraded my Sheikah Slate with a new rune..

adamludvik (Fri, 17 Mar 2017 22:06:30 GMT):
@Dan LR1 is now running fine with a 300TPM intkey load, which was the original LR1 test before I started messing with the workload. My guess is that I left the workload settings in a state that LR1 was already failing on, but didn't realize it. We'll see how long it lasts with 300TPM.

dragosh (Sat, 18 Mar 2017 00:14:14 GMT):
Has joined the channel.

bkman (Sat, 18 Mar 2017 02:44:28 GMT):
Has joined the channel.

midside (Mon, 20 Mar 2017 13:06:33 GMT):
Has joined the channel.

boydjohnson (Mon, 20 Mar 2017 16:45:12 GMT):
So our general pattern to do startup tasks in a validator container in a test has been to put bash -c "sawtooth admin keygen && sawtooth admin genesis &&..." in the entrypoint. This doesn't work for if you want to 1) send sig term or sig int to the validator process 2) stop and then start the process within the container (without doing the startup stuff again. I solve this by using a "startup tasks" container that shares directories with the validator container. @rbuysse @drozd @adamludvik It would be nice to be able to refer to a collection of services by name as one service, like validator-genesis is a genesis container along with a validator container and validator-non-genesis is a keygen container along with a validator container. Any thoughts?

adamludvik (Mon, 20 Mar 2017 16:49:43 GMT):
Are you suggesting we have a container for every command needed to startup a validator?

boydjohnson (Mon, 20 Mar 2017 16:50:50 GMT):
Not for every command, but every validator getting a "startup commands" container solves some problems, but seems messy.

adamludvik (Mon, 20 Mar 2017 17:02:40 GMT):
Would it be more clean to dynamically change the entrypoint as needed?

adamludvik (Mon, 20 Mar 2017 17:04:26 GMT):
Hmmm, that doesn't solver your signal issues

adamludvik (Mon, 20 Mar 2017 17:04:26 GMT):
Hmmm, that doesn't solve your signal issues

boydjohnson (Mon, 20 Mar 2017 17:05:25 GMT):
Yeah, if I understand you correctly that would solve 2) but not 1).

adamludvik (Mon, 20 Mar 2017 17:18:47 GMT):
Per offline discussion, I think it may make more sense to start the container with an entrypoint that doesn't do anything besides start the container. (`docker run -it --entrypoint=/bin/cat [image-name]` seems to work) Then you can send additional commands to the container using `docker exec`. Open question is whether you can read the exit code of the process started by `docker exec`.

adamludvik (Mon, 20 Mar 2017 17:18:47 GMT):
Per offline discussion, I think it may make more sense to start the container with an entrypoint that doesn't do anything besides start the container. `docker run -it --entrypoint=/bin/cat [image-name]` seems to work. Then you can send additional commands to the container using `docker exec`. Open question is whether you can read the exit code of the process started by `docker exec`.

ananth_s (Mon, 20 Mar 2017 17:41:54 GMT):
Has joined the channel.

dhuseby (Mon, 20 Mar 2017 17:55:34 GMT):
Has joined the channel.

Dan (Mon, 20 Mar 2017 18:07:56 GMT):
is this for launching with docker specifically or the pattern for starting sawtooth generally?

adamludvik (Mon, 20 Mar 2017 18:25:19 GMT):
This is just in reference to running integration tests that use multiple validators. This would not be user facing.

travin (Tue, 21 Mar 2017 12:29:33 GMT):
Has joined the channel.

PacoEcotrust (Tue, 21 Mar 2017 13:06:29 GMT):
I have changed softly the marketplace extension but I have a problem with one transaction. This transaction is identified as valid but it is never validated. I have deploy it over ubuntu not over docket and using only 1 validator with poet0. The transaction is to creat participant. >//UNKNOWN> participant reg --name emkt --description "The new Marketplace" The client is always asking for committing this transacction and the validator send 302 code. Some idea?

Dan (Tue, 21 Mar 2017 13:13:12 GMT):
hi @PacoEcotrust can I ask you some questions on your environment? What does this report? `$ git describe --tags`

adamludvik (Tue, 21 Mar 2017 14:48:46 GMT):
I need to define the entrypoint (command you type in bash to run the package you just installed) for each of the sawtooth deb packages that has one. The sawtooth-rest-api package already had `stl-rest-api` defined as the command you run to start the rest_api when installed, but this is different then our bin scripts. The sawtooth-cli package on the other hand defines `sawtooth` as the entrypoint. I also need to define entrypoints for sawtooth-validator and the existing transaction processors. Does anyone have any strong opinions about what these entrypoints should be or do we already have a naming convention established?

pschwarz (Tue, 21 Mar 2017 14:52:50 GMT):
@Dan @boydjohnson When you get a chance, can you review https://github.com/hyperledger/sawtooth-core/pull/232

adamludvik (Tue, 21 Mar 2017 15:16:14 GMT):
Hearing no strong opinions, I am going to impose the rule that the entrypoint has to match what is in sawtooth-core/bin and ask that we update the names of the bin scripts at a later date so we aren't installing generic names like `rest_api` to the users path.

jsmitchell (Tue, 21 Mar 2017 15:23:29 GMT):
@adamludvik: sounds reasonable

PacoEcotrust (Tue, 21 Mar 2017 15:24:32 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=JvNpYYfxYXvqpvzWC) @Dan [18:13:42 [PoolThread-WebThreadPool-0] journal.transaction DEBUG] [18:13:42 [PoolThread-WebThreadPool-0] journal.transaction DEBUG] checking RegisterParticipant Updates with 1 updates [18:13:42 [PoolThread-WebThreadPool-0] journal.transaction DEBUG] update [] [18:13:43 [PoolThread-WebThreadPool-0] txnserver.web_pages.forward_page INFO] transaction 729cbbadb6381bed is valid [18:13:43 [PoolThread-WebThreadPool-0] gossip.gossip_core DEBUG] calling handler for message 57351650 from 17A5QHJk of type /emktplace.transactions.EMarketPlace/Transaction [18:13:43 [PoolThread-twisted.internet.reactor-0] journal.messages.transaction_message DEBUG] handle transaction message with identifier 729cbbadb6381bed [18:13:43 [PoolThread-twisted.internet.reactor-0] journal.journal_core DEBUG] txnid: 729cbbad - add_pending_transaction [18:13:51 [MainThread] gossip.gossip_core DEBUG] calling handler for message 4f824066 from 17A5QHJk of type /journal.messages.TransactionBlockMessage/BlockRetry [18:13:55 [PoolThread-WebThreadPool-0] txnserver.web_pages.base_page INFO] TransactionPage.do_get /transaction/729cbbadb6381bed

PacoEcotrust (Tue, 21 Mar 2017 15:24:32 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=JvNpYYfxYXvqpvzWC) @Dan Hello Dan, I do not know if you are going to catch some thing. [16:42:11 INFO validator_cli] validator started with arguments: ['/home/kiko/project/sawtooth-core/bin/txnvalidator', '-v'] [16:42:11 INFO validator_cli] read signing key from /etc/sawtooth-validator/keys/base000.wif [16:42:16 WARNING validator_cli] validator pid is 3172 [16:42:16 INFO gossip_core] listening on IPv4Address(UDP, '0.0.0.0', 54184) [16:42:16 INFO global_store_manager] create blockstore from file /var/lib/sawtooth-validator/base000_state.dbm with flag c [16:42:16 INFO validator] set administration node to None [16:42:16 INFO validator] starting ledger base000 with id 17A5QHJk at network address ('127.0.0.1', 54184) [16:42:16 INFO web_api] listen for HTTP requests on (ip='localhost', port=8800) [16:42:16 INFO validator_cli] adding transaction family: ledger.transaction.integer_key [16:42:16 INFO validator_cli] adding transaction family: emktplace.transactions.market_place [16:42:17 INFO journal_core] restore ledger state from persistence [16:42:17 INFO global_store_manager] add block 0a96be662e72daea to the queue for loading [16:42:17 INFO global_store_manager] add block 98e0efcd233b6981 to the queue for loading [16:42:17 INFO global_store_manager] load block 98e0efcd233b6981 from storage [16:42:17 INFO global_store_manager] load block 0a96be662e72daea from storage [16:42:17 INFO journal_core] commit head: 0a96be662e72daea [16:42:19 INFO validator] ledger connections using RandomWalk topology [16:42:19 INFO random_walk] initiate random walk topology update [16:42:22 INFO validator] ledger initialization complete [16:42:22 INFO journal_core] process initial transactions and blocks [16:42:22 INFO validator] register endpoint 17A5QHJk with name base000 .........continue below

PacoEcotrust (Tue, 21 Mar 2017 15:24:32 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=JvNpYYfxYXvqpvzWC) @Dan [18:13:42 [PoolThread-WebThreadPool-0] journal.transaction DEBUG] [18:13:42 [PoolThread-WebThreadPool-0] journal.transaction DEBUG] checking RegisterParticipant Updates with 1 updates [18:13:42 [PoolThread-WebThreadPool-0] journal.transaction DEBUG] update [] [18:13:43 [PoolThread-WebThreadPool-0] txnserver.web_pages.forward_page INFO] transaction 729cbbadb6381bed is valid [18:13:43 [PoolThread-WebThreadPool-0] gossip.gossip_core DEBUG] calling handler for message 57351650 from 17A5QHJk of type /emktplace.transactions.EMarketPlace/Transaction [18:13:43 [PoolThread-twisted.internet.reactor-0] journal.messages.transaction_message DEBUG] handle transaction message with identifier 729cbbadb6381bed [18:13:43 [PoolThread-twisted.internet.reactor-0] journal.journal_core DEBUG] txnid: 729cbbad - add_pending_transaction [18:13:51 [MainThread] gossip.gossip_core DEBUG] calling handler for message 4f824066 from 17A5QHJk of type /journal.messages.TransactionBlockMessage/BlockRetry [18:13:55 [PoolThread-WebThreadPool-0] txnserver.web_pages.base_page INFO] TransactionPage.do_get /transaction/729cbbadb6381bed

PacoEcotrust (Tue, 21 Mar 2017 15:24:32 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=JvNpYYfxYXvqpvzWC) @Dan Hello Dan, I do not know if you are going to catch some thing. [18:13:42 [PoolThread-WebThreadPool-0] journal.transaction DEBUG] [18:13:42 [PoolThread-WebThreadPool-0] journal.transaction DEBUG] checking RegisterParticipant Updates with 1 updates [18:13:42 [PoolThread-WebThreadPool-0] journal.transaction DEBUG] update [] [18:13:43 [PoolThread-WebThreadPool-0] txnserver.web_pages.forward_page INFO] transaction 729cbbadb6381bed is valid [18:13:43 [PoolThread-WebThreadPool-0] gossip.gossip_core DEBUG] calling handler for message 57351650 from 17A5QHJk of type /emktplace.transactions.EMarketPlace/Transaction [18:13:43 [PoolThread-twisted.internet.reactor-0] journal.messages.transaction_message DEBUG] handle transaction message with identifier 729cbbadb6381bed [18:13:43 [PoolThread-twisted.internet.reactor-0] journal.journal_core DEBUG] txnid: 729cbbad - add_pending_transaction [18:13:51 [MainThread] gossip.gossip_core DEBUG] calling handler for message 4f824066 from 17A5QHJk of type /journal.messages.TransactionBlockMessage/BlockRetry [18:13:55 [PoolThread-WebThreadPool-0] txnserver.web_pages.base_page INFO] TransactionPage.do_get /transaction/729cbbadb6381bed

PacoEcotrust (Tue, 21 Mar 2017 15:24:32 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=JvNpYYfxYXvqpvzWC) @Dan Hello Dan, I do not know if you are going to catch some thing. [18:13:42 [PoolThread-WebThreadPool-0] journal.transaction DEBUG] [18:13:42 [PoolThread-WebThreadPool-0] journal.transaction DEBUG] checking RegisterParticipant Updates with 1 updates [18:13:42 [PoolThread-WebThreadPool-0] journal.transaction DEBUG] update [] [18:13:43 [PoolThread-WebThreadPool-0] txnserver.web_pages.forward_page INFO] transaction 729cbbadb6381bed is valid [18:13:43 [PoolThread-WebThreadPool-0] gossip.gossip_core DEBUG] calling handler for message 57351650 from 17A5QHJk of type /emktplace.transactions.EMarketPlace/Transaction [18:13:43 [PoolThread-twisted.internet.reactor-0] journal.messages.transaction_message DEBUG] handle transaction message with identifier 729cbbadb6381bed [18:13:43 [PoolThread-twisted.internet.reactor-0] journal.journal_core DEBUG] txnid: 729cbbad - add_pending_transaction [18:13:51 [MainThread] gossip.gossip_core DEBUG] calling handler for message 4f824066 from 17A5QHJk of type /journal.messages.TransactionBlockMessage/BlockRetry [18:13:55 [PoolThread-WebThreadPool-0] txnserver.web_pages.base_page INFO] TransactionPage.do_get /transaction/729cbbadb6381bed................... the client command was:

PacoEcotrust (Tue, 21 Mar 2017 15:24:32 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=JvNpYYfxYXvqpvzWC) @Dan Hello Dan, I do not know if you are going to catch some thing. [18:13:42 [PoolThread-WebThreadPool-0] journal.transaction DEBUG] [18:13:42 [PoolThread-WebThreadPool-0] journal.transaction DEBUG] checking RegisterParticipant Updates with 1 updates [18:13:42 [PoolThread-WebThreadPool-0] journal.transaction DEBUG] update [] [18:13:43 [PoolThread-WebThreadPool-0] txnserver.web_pages.forward_page INFO] transaction 729cbbadb6381bed is valid [18:13:43 [PoolThread-WebThreadPool-0] gossip.gossip_core DEBUG] calling handler for message 57351650 from 17A5QHJk of type /emktplace.transactions.EMarketPlace/Transaction [18:13:43 [PoolThread-twisted.internet.reactor-0] journal.messages.transaction_message DEBUG] handle transaction message with identifier 729cbbadb6381bed [18:13:43 [PoolThread-twisted.internet.reactor-0] journal.journal_core DEBUG] txnid: 729cbbad - add_pending_transaction [18:13:51 [MainThread] gossip.gossip_core DEBUG] calling handler for message 4f824066 from 17A5QHJk of type /journal.messages.TransactionBlockMessage/BlockRetry [18:13:55 [PoolThread-WebThreadPool-0] txnserver.web_pages.base_page INFO] TransactionPage.do_get /transaction/729cbbadb6381bed................... the client command was: emktclient -v --name emkt --keyfile /home/kiko/project/sawtooth-core/validator/keys/emkt.wif --config /home/kiko/project/sawtooth-core/extensions/emktplace/etc/mktclient.js //UNKNOWN> participant reg --name emkt --description "The New Marketplace". In this command all time asking for commit the transaction

PacoEcotrust (Tue, 21 Mar 2017 16:10:48 GMT):
[16:42:22 WARNING signed_object] Identifier 08b5d3480bb011dc of SignedObject has no public key and addr used to verify signature [16:42:22 INFO journal_core] build transaction block to extend 0a96be66 with 1 transactions [16:42:22 INFO wait_timer] wait timer created; TIMER, 5.00, 7.59, ra8rcvdvd67h8gks [16:42:29 INFO journal_core] node base000 validates block with 1 transactions [16:42:29 INFO wait_certificate] wait certificate created; CERT, 5.00, 7.59, rdcjsu3akg4gzgxj, ra8rcvdvd67h8gks [16:42:30 INFO market_place] prepare to claim block 1aa846a079269264 [16:42:30 WARNING signed_object] Identifier 1aa846a079269264 of SignedObject has no public key and addr used to verify signature [16:42:30 WARNING signed_object] Identifier 1aa846a079269264 of SignedObject has no public key and addr used to verify signature [16:42:30 WARNING signed_object] Identifier 08b5d3480bb011dc of SignedObject has no public key and addr used to verify signature [16:42:30 INFO journal_core] blkid: 1aa846a0 - commit block from base000 with previous blkid: 0a96be66 [16:43:09 INFO base_page] BlockPage.do_get /block [16:43:09 INFO base_page] StorePage.do_get /store/EMarketPlaceTransaction/* [16:43:09 INFO base_page] BlockPage.do_get /block [16:43:38 WARNING root_page] /emktplace.transactions.EMarketPlace/Transaction - NoResource [16:43:38 WARNING root_page] ['emktplace.transactions.EMarketPlace', 'Transaction'] - 2 [16:43:38 INFO forward_page] starting local validation for txn id: 00c35d13d76e77a0 type: /EMarketPlaceTransaction [16:43:38 INFO forward_page] RegisterParticipant Updates [16:43:38 INFO forward_page] transaction 00c35d13d76e77a0 is valid [16:43:38 INFO base_page] TransactionPage.do_get /transaction/00c35d13d76e77a0 [16:43:43 INFO base_page] TransactionPage.do_get /transaction/00c35d13d76e77a0 [16:43:58 INFO base_page] TransactionPage.do_get /transaction/00c35d13d76e77a0the client command was: emktclient -v --name emkt --keyfile /home/kiko/project/sawtooth-core/validator/keys/emkt.wif --config /home/kiko/project/sawtooth-core/extensions/emktplace/etc/mktclient.js //UNKNOWN> participant reg --name emkt --description "The New Marketplace". In this command all time asking for commit the transaction

Dan (Tue, 21 Mar 2017 16:29:34 GMT):
@PacoEcotrust two quick thoughts... First it looks like you are launching your ledger restoring from a previous database. It might help to purge that and start fresh. Really it should work either way though. Second, I see some signed object warnings. That could mean that you don't have key files. Take a look at keygen .. sorry I'd look up the place in the docs but I'm on my way to a meeting.

PacoEcotrust (Tue, 21 Mar 2017 16:39:30 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=xtq9RCDv7Po98neQ9) @D , thankyou, could be the Second. I will take at look

Amanskywalker (Tue, 21 Mar 2017 18:14:05 GMT):
Has joined the channel.

Dan (Tue, 21 Mar 2017 18:38:14 GMT):
Hi @PacoEcotrust if your code is online somewhere like github any of us on the channel can take a look at your modified marketplace transaction family. The next place I'd look is the validation method.. something like check_valid().

PacoEcotrust (Tue, 21 Mar 2017 18:40:08 GMT):
Thanks, I will try to go deeper. I will to report you about my progress

boydjohnson (Tue, 21 Mar 2017 18:51:58 GMT):
@zac Did you see that the aiohttp test_utils package just changed in the last few days? https://github.com/aio-libs/aiohttp/commit/02145b1792f251fc2ca32f167fc30ccb4e0fd400

zac (Tue, 21 Mar 2017 20:19:46 GMT):
Yeah, 2.0 just dropped

zac (Tue, 21 Mar 2017 20:19:59 GMT):
There are a couple of possible breaking changes

zac (Tue, 21 Mar 2017 20:20:16 GMT):
I think we should stick with 1.x for now

boydjohnson (Tue, 21 Mar 2017 20:35:33 GMT):
Yeah, aiohttp==1.3.5 allows for all the rest api tests to pass.

tkuhrt (Tue, 21 Mar 2017 21:14:10 GMT):
Has joined the channel.

speffers (Wed, 22 Mar 2017 01:39:44 GMT):
Has joined the channel.

taishi (Wed, 22 Mar 2017 03:19:28 GMT):
Has joined the channel.

taishi (Wed, 22 Mar 2017 03:40:46 GMT):
Hi.I am a beginner. Please let question. Is there a detailed document about PoET? I do not understand even if I read the official docs. Also, is Sawtooth Lake assuming a permissoned environment?

foobarometer (Wed, 22 Mar 2017 05:56:42 GMT):
hi, i'm also beginner. i was going thru the tutorial http://intelledger.github.io/tutorial.html & it seems some commands needed to be executed twice in order to take effect. near the end: ``` //alice> exchange --type ExchangeOffer --src /USD --dst /jars/choc_chip --offers //bob/choc_chip_sale --count 24 http://localhost:8800/block?blockcount=10 transaction ee36338a123d4790 submitted //alice> waitforcommit Waiting for ee36338a123d4790. Transaction committed. //alice> holdings --creator //bob 1000 //bob/USD 24 //bob/batches/choc_chip001 1 //bob/holding/token 0 //bob/jars/choc_chip //alice> holdings --creator //alice 1000 //alice/USD 1 //alice/holding/token 0 //alice/jars/choc_chip //alice> waitforcommit Waiting for ee36338a123d4790. Transaction committed. //alice> holdings --creator //alice 1000 //alice/USD 1 //alice/holding/token 0 //alice/jars/choc_chip //alice> holdings --creator //bob 1000 //bob/USD 24 //bob/batches/choc_chip001 1 //bob/holding/token 0 //bob/jars/choc_chip //alice> exchange --type ExchangeOffer --src /USD --dst /jars/choc_chip --offers //bob/choc_chip_sale --count 24 http://localhost:8800/block?blockcount=10 http://localhost:8800/store/MarketPlaceTransaction/*?delta=1&blockid=e62acd704c025c6a transaction eb8735fa2f2ba81d submitted //alice> holdings --creator //bob 1024 //bob/USD 12 //bob/batches/choc_chip001 1 //bob/holding/token 0 //bob/jars/choc_chip //alice> holdings --creator //alice 976 //alice/USD 1 //alice/holding/token 12 //alice/jars/choc_chip ```

foobarometer (Wed, 22 Mar 2017 06:00:19 GMT):
the other one that seemed to need a 2nd run was: ``` //bob> exchangeoffer reg --output /batches/choc_chip001 --input /USD --ratio 2 1 --name /choc_chip_sale ``` I ran `waitforcommit` after this command. it said it was `commited`. However, when I tried to look at offers, I got: ``` //bob> offers --creator //bob Ratio Input Asset (What You Pay) Output Asset (What You Get) Name //bob> ```

foobarometer (Wed, 22 Mar 2017 06:00:19 GMT):
the other one that seemed to need a 2nd run was: ``` //bob> exchangeoffer reg --output /batches/choc_chip001 --input /USD --ratio 2 1 --name /choc_chip_sale ``` I ran `waitforcommit` after this command. it said it was `committed`. However, when I tried to look at offers, I got: ``` //bob> offers --creator //bob Ratio Input Asset (What You Pay) Output Asset (What You Get) Name //bob> ```

foobarometer (Wed, 22 Mar 2017 06:01:58 GMT):
of course, when I ran it a 2nd time, I got an error: ``` //bob> exchangeoffer reg --output /batches/choc_chip001 --input /USD --ratio 2 1 --name /choc_chip_sale http://localhost:8800/block?blockcount=10 http://localhost:8800/store/MarketPlaceTransaction/*?delta=1&blockid=739231fca7d153bd [05:52:33 mktplace.mktplace_communication WARNING] operation failed with response: 400 [05:52:33 mktplace.mktplace_communication WARNING] Error from server, detail information: ?eerrorpName isn't validierrorTypewInvalidTransactionErrorfstatus? [05:52:33 mktplace.mktplace_client ERROR] message post failed for some unusual reason Traceback (most recent call last): File "/project/sawtooth-core/extensions/mktplace/mktplace/mktplace_client.py", line 172, in _sendtxn msg File "/project/sawtooth-core/extensions/mktplace/mktplace/mktplace_communication.py", line 180, in postmsg .format(err_content)) InvalidTransactionError: Error from server: failed to create transaction ``` but the offer showed up: ``` //bob> offers --creator //bob Ratio Input Asset (What You Pay) Output Asset (What You Get) Name 0.5 //mkt/asset/currency/USD //mkt/asset/cookie/choc_chip //bob/choc_chip_sale ```

sitomani (Wed, 22 Mar 2017 08:48:05 GMT):
Has joined the channel.

Dan (Wed, 22 Mar 2017 14:36:23 GMT):
@toddojala ^

toddojala (Wed, 22 Mar 2017 14:37:59 GMT):
@foobarometer Thanks for sharing that experience. Let me look into that, to find out if anyone else has seen that, or if it's a known issue.

adamludvik (Wed, 22 Mar 2017 14:53:18 GMT):
@pschwarz Do you know what the plan for the poet cli for 0.8 is? Will this be a part of the "sawtooth" command, or a separate command?

pschwarz (Wed, 22 Mar 2017 14:53:33 GMT):
It's a separate command

pschwarz (Wed, 22 Mar 2017 14:53:59 GMT):
Given that poet is (in the long run) an optional feature, this should make sense

adamludvik (Wed, 22 Mar 2017 14:55:28 GMT):
Sounds good. We have a separate sawtooth-poet-cli package which makes sense if that's true. We don't have that command implemented yet though, correct?

pschwarz (Wed, 22 Mar 2017 14:55:40 GMT):
We do!

adamludvik (Wed, 22 Mar 2017 14:56:05 GMT):
... Is it the poet command....

adamludvik (Wed, 22 Mar 2017 14:56:08 GMT):
?

pschwarz (Wed, 22 Mar 2017 14:56:48 GMT):
Should the script at `bin/poet` be in the `consensus/poet/cli` package somewhere?

adamludvik (Wed, 22 Mar 2017 14:56:56 GMT):
I should probably add that as an entrypoint to the sawtooth-poet-cli package so it gets installed to the users path.

adamludvik (Wed, 22 Mar 2017 14:57:00 GMT):
Yes

pschwarz (Wed, 22 Mar 2017 14:57:17 GMT):
Cool - you should do the needful on that, then

adamludvik (Wed, 22 Mar 2017 14:57:28 GMT):
:+1:

jsmitchell (Wed, 22 Mar 2017 14:57:51 GMT):
:musical_note: _do the needful_ _do do do di do do do di_

jsmitchell (Wed, 22 Mar 2017 14:58:02 GMT):
sung to the tune of "Do the Hustle"

adamludvik (Wed, 22 Mar 2017 15:09:14 GMT):
I am creating docker containers based on these deb packages. I have a validator-registry container which runs tp_validator_registry, which requires the sawtooth-poet-families package. Are there any other poet processes we'd like to run in containers?

pschwarz (Wed, 22 Mar 2017 15:48:41 GMT):
We should considered locking the versions of our python dependencies. This will keep the builds more predictable and avoid any future issues like we're currently having the aiohttp library's recent update.

boydjohnson (Wed, 22 Mar 2017 15:52:06 GMT):
We should make a docker-blockchain, where global state controls the specific docker images available to download. This seems like the best fix.

jsmitchell (Wed, 22 Mar 2017 15:52:17 GMT):
dockchain

toddojala (Wed, 22 Mar 2017 15:59:20 GMT):
@foobarometer I will try to reproduce your issue today.

foobarometer (Wed, 22 Mar 2017 16:06:55 GMT):
Thanks!

foobarometer (Wed, 22 Mar 2017 16:06:55 GMT):
Thanks @toddojala !

toddojala (Wed, 22 Mar 2017 18:13:24 GMT):
@taishi This is the documentation we do have on PoET: http://intelledger.github.io/introduction.html#proof-of-elapsed-time-poet

toddojala (Wed, 22 Mar 2017 18:14:04 GMT):
@taishi Could you explain in more detail what you mean by a "permissioned environment"? Thanks!

jsmitchell (Wed, 22 Mar 2017 18:40:40 GMT):
A permissioned environment usually means a private instance of a validator network with specific controls around which nodes may join the network.

adamludvik (Wed, 22 Mar 2017 19:46:49 GMT):
@pschwarz Can you make a JIRA task for that?

Dan (Wed, 22 Mar 2017 19:51:09 GMT):
@taishi Sawtooth does not assume a permissioned network. It is designed to be deployable across a spectrum of public to private policy rules. That is to say that there are many possible policies that could control different aspects of the network. We have a couple policies implemented that reflect different usages, but also serve as examples of how you could customize your distributed ledger / network policy.

jjason (Wed, 22 Mar 2017 19:55:32 GMT):
In the PoET consensus module, because it may need to walk back blocks in the block store to gather information necessary to make a decision about (or gather statistics to make future decisions about) whether or not a block should be initialized or verified, it also needs to be able to create a state view for that block. This means that it needs access to the state view factory. A couple of options: 1. Change the BlockPublisher, BlockVerifier, and ForkResolver interfaces so that the state view factory, in place of the state view, is provided when an object is created. 2. Leave the BlockPublisher and BlockVerifier interfaces alone, but change ForkResolver interface to also take a state view object on creation. In addition, change the state view object so that it has a factory property that is the factory that created it. Publisher/Verifier/Resolver can then use that property to create new state view objects for arbitrary blocks as they see need for. I originally favored 2 as I thought that it would not require a change to the interface, but now am indifferent as I realized that the ForkResolver needs to be updated with a state view object at a minimum anyway.

jjason (Wed, 22 Mar 2017 20:05:33 GMT):
^^^^^^ @jsmitchell @cianx @Dan @pschwarz ^^^^^^^

pschwarz (Wed, 22 Mar 2017 20:21:24 GMT):
I think we would want to pass the combination of (state_hash, state_view_factory), which is equivalent to a single state_view, but gives the consensus module the ability to look at state views from previous blocks.

pschwarz (Wed, 22 Mar 2017 20:22:27 GMT):
I'm also wondering if there's a missing life-cycle seam here - initialize, which could do things like build the consensus cache from an existing chain, and then processing is on a block-by-block basis

pschwarz (Wed, 22 Mar 2017 20:22:27 GMT):
I'm also wondering if there's a missing life-cycle seam here - `initialize`, which could do things like build the consensus cache from an existing chain, and then processing is on a block-by-block basis

jjason (Wed, 22 Mar 2017 20:30:51 GMT):
For the PoET stuff, I was going to hide the building of the consensus cache from the publisher/verifier/resolver so that it thinks it is simply working on a block-by-block basis.

pschwarz (Wed, 22 Mar 2017 20:38:54 GMT):
Does PoET need access to the whole block chain at any other time then rebuilding that cache?

boydjohnson (Wed, 22 Mar 2017 21:12:08 GMT):
So I have two green checkmarks on PR176, the graceful shutdown PR, but wanted to give anyone who wanted to look at it a second chance.

jjason (Wed, 22 Mar 2017 21:47:50 GMT):
@pschwarz In theory, if the PoET consensus module is working properly, the only time it would ever need the entire blockchain would be in the case of the validator coming up with an existing block store but for some reason the consensus store was not there. However, I do need to guard against forks that cause the consensus module to not have complete stats for the blockchain.

pschwarz (Wed, 22 Mar 2017 22:00:10 GMT):
Fair enough

adamludvik (Wed, 22 Mar 2017 22:11:23 GMT):
I am going to put in a request to have the following docker repositories added to the hyperledger namespace unless somebody stops me: ``` sawtooth-rest_api sawtooth-tp_config sawtooth-tp_intkey_python sawtooth-tp_validator_registry sawtooth-tp_xo_python sawtooth-validator ```

jjason (Wed, 22 Mar 2017 22:17:07 GMT):
Anyone besides @pschwarz have feedback on publisher/verifier/resolver interface? If not, I am going to update to have state view factory replace state view in interfaces.

boydjohnson (Wed, 22 Mar 2017 22:36:18 GMT):
@adamludvik Is the intkey javascript docker image not ready? java will soon be ready. Why not grab namespaces for those as well.

jjason (Wed, 22 Mar 2017 22:40:08 GMT):
@adamludvik Should `sawtooth-tp_validator_registry` be `sawtooth-tp_validator_registry_python` for consistency with intkey. Note - that does not mean I am signing up to port the validator registry to JavaScript or Java.

jjason (Wed, 22 Mar 2017 22:40:38 GMT):
@adamludvik Same goes for `sawtooth-tp_config` @pschwarz

pschwarz (Wed, 22 Mar 2017 22:41:25 GMT):
We don't have alternative implementations of those TP's

boydjohnson (Wed, 22 Mar 2017 22:42:31 GMT):
But we might in the future, right?

adamludvik (Wed, 22 Mar 2017 22:43:38 GMT):
@boydjohnson I can reserve those too, but they still need the source mounted.

boydjohnson (Wed, 22 Mar 2017 22:45:06 GMT):
What does source mounted mean in this context?

adamludvik (Wed, 22 Mar 2017 22:45:22 GMT):
@jjason you make a good point. Although doesn't the validator registry depend on PoET and so a non-python implementation would require rewriting PoET?

jjason (Wed, 22 Mar 2017 22:45:49 GMT):
@adamludvik Nope. It would only require rewriting the TP.

jjason (Wed, 22 Mar 2017 22:46:48 GMT):
I could also argue that since Python is the "canonical" language, Python implementations could drop the `_python` moniker.

jjason (Wed, 22 Mar 2017 22:47:07 GMT):
Okay. I have pulled the pin out of the grenade and walked away. My work here is done. ;)

boydjohnson (Wed, 22 Mar 2017 22:47:25 GMT):
I've got to get going, but @jjason could you talk more about the consensus store

adamludvik (Wed, 22 Mar 2017 22:47:36 GMT):
I have not created debs for Java and JavaScript or done any work to make them isolated like the Python components. Theoretically I could just copy the source/compiled code into the image to achieve this though...

boydjohnson (Wed, 22 Mar 2017 22:48:03 GMT):
Oh, thanks @adamludvik.

jjason (Wed, 22 Mar 2017 22:50:54 GMT):
@boydjohnson The PoET consensus module needs to potentially load an arbitrary state view (based upon looking at blocks back in history) to gather enough information to make decisions about whether a validator should initialize a block (BlockPublisher.initialize_block), whether a block should be accepted from another validator (BlockVerifier.verify_block), or which branch of a fork to accept (ForkResolver.compare_forks). Currently, the block publisher and block verifier objects only have the state view for the most-recently-committed block. In order to get the state view for other blocks, they actually need the state view factory.

jjason (Wed, 22 Mar 2017 22:56:28 GMT):
The consensus store would be a mapping of block (block ID, state root hash, whatever) to the PoET consensus state/statistics at that point (for example, the number of blocks claimed by each validator, etc.). That way when a block is to be initialized or verified, the consensus store can be consulted to see if the PoET policies would allow it.

Dan (Thu, 23 Mar 2017 00:07:14 GMT):
@jjason trying to read upwards here... can you clarify for me... The state view is to let consensus look at the data in the merkle trie at a given time. In poet's case specifically for the validator registry? Where as the consensus store is intended to store and manage block metadata (like weight) that is not part of 'state' proper? And you are asking for feedback on consensus receiving a factory method so it can look at state at a variety of states? Can you say more about what that use case is? Is that to calculate c, k, z of the publishing validator?

Dan (Thu, 23 Mar 2017 00:52:58 GMT):
@rjones can you create a sawtooth-consensus channel for us?

jjason (Thu, 23 Mar 2017 03:42:44 GMT):
@dan Yes. The PoET consensus module needs to store additional data that is not part of the state 'proper' and it needs access to the validator registry as it existed when a particular block is the head of the chain (so, for example it knows the PoET public key for a validator). There may be cases in which the consensus module may not have computed statistics for a block and it needs to look up that block's merkle trie to obtain the validator registry at that point in time. These statistics are then used to in the decisions around enforcing the c, k, and z polices when determining in the block publisher whether it should initialize a block (i.e., no reason to initialize a block if it is just going to be rejected by the other validators) or determining in the block verifier if the block should be accepted.

jjason (Thu, 23 Mar 2017 03:42:44 GMT):
@dan Yes. The PoET consensus module needs to store additional data that is not part of the state 'proper' and it needs access to the validator registry as it existed when a particular block was the head of the chain (so, for example it knows the PoET public key for a validator). There may be cases in which the consensus module may not have computed statistics for a block and it needs to look up that block's merkle trie to obtain the validator registry at that point in time. These statistics are then used to in the decisions around enforcing the c, k, and z polices when determining in the block publisher whether it should initialize a block (i.e., no reason to initialize a block if it is just going to be rejected by the other validators) or determining in the block verifier if the block should be accepted.

rjones (Thu, 23 Mar 2017 05:26:36 GMT):
@jjason @Dan I created #sawtooth-consensus for you

Dan (Thu, 23 Mar 2017 13:07:17 GMT):
thanks @rjones!

PacoEcotrust (Thu, 23 Mar 2017 13:13:49 GMT):
hello, the documentation of MarketPlace in the latest tutorial does not appear. why?

PacoEcotrust (Thu, 23 Mar 2017 13:13:49 GMT):
hello, the documentation of MarketPlace in the latest tutorial does not appear. why? It is obsolete with the 0.8 version?

Dan (Thu, 23 Mar 2017 13:14:59 GMT):
Oh sorry about that @PacoEcotrust you hit the project just as we moved from 0.7 to 0.8.

Dan (Thu, 23 Mar 2017 13:15:36 GMT):
No one has ported marketplace into the 0.8 APIs yet

Dan (Thu, 23 Mar 2017 13:16:40 GMT):
You can continue to work through the old tutorial on 0.7 though. If you have the repos you can build the 0.7 docs ... that's not my ideal thing for you to do, but its there.

Dan (Thu, 23 Mar 2017 13:17:29 GMT):
@toddojala are the 0.7 docs still available on github.io? How hard is it to put that dropdown list back on so people can select 0.7?

PacoEcotrust (Thu, 23 Mar 2017 13:19:54 GMT):
ok, thanks, I will try

toddojala (Thu, 23 Mar 2017 13:50:25 GMT):
@PacoEcotrust and @Dan : http://intelledger.github.io/0.7/

toddojala (Thu, 23 Mar 2017 13:50:44 GMT):
The dropdown is still there on http://intelledger.github.io

toddojala (Thu, 23 Mar 2017 13:51:18 GMT):
We switched the default to 0.8, but anyone can still switch to 0.7. If not, let us know. Ryan and I switched and tested, but you never know!

Dan (Thu, 23 Mar 2017 13:51:31 GMT):
thanks @toddojala I could have sworn that dropdown wasn't there last night... maybe its browser dependent ... or maybe I was just too tired. :)

toddojala (Thu, 23 Mar 2017 13:51:38 GMT):
caching issue?

toddojala (Thu, 23 Mar 2017 13:51:56 GMT):
Caches are great, but can be inaccurate. ;)

Dan (Thu, 23 Mar 2017 13:52:08 GMT):
could be. looks good now though. thanks.

Dan (Thu, 23 Mar 2017 16:26:41 GMT):
anyone want to take a stab at expected "normal" transaction size? And max size?

jkilpatr (Thu, 23 Mar 2017 17:36:30 GMT):
Has joined the channel.

jsmitchell (Thu, 23 Mar 2017 17:39:24 GMT):
I have sent 10MB+ blocks over the network

mahesh_rao (Thu, 23 Mar 2017 18:23:10 GMT):
Has joined the channel.

boydjohnson (Thu, 23 Mar 2017 18:41:12 GMT):
```diff --git a/validator/sawtooth_validator/journal/consensus/dev_mode/dev_mode_consensus.py b/validator/sawtooth_validator/journal/consensus/dev_mode/dev_mode_consensus.py index f42b0d3..078bc90 100644 --- a/validator/sawtooth_validator/journal/consensus/dev_mode/dev_mode_consensus.py +++ b/validator/sawtooth_validator/journal/consensus/dev_mode/dev_mode_consensus.py @@ -116,7 +116,8 @@ class BlockVerifier(BlockVerifierInterface): self._state_view = state_view def verify_block(self, block_wrapper): - return block_wrapper.header.consensus == b"Devmode" + # return block_wrapper.header.consensus == b"Devmode" + return True class ForkResolver(ForkResolverInterface):```

jsmitchell (Thu, 23 Mar 2017 18:41:49 GMT):
@cianx ^

boydjohnson (Thu, 23 Mar 2017 18:41:55 GMT):
So this diff causes the poet-smoke test to pass *sometimes*.

boydjohnson (Thu, 23 Mar 2017 18:43:29 GMT):
It is because the config is set in the genesis block so the non-genesis validators aren't using Poet yet, but are using dev-mode. The block fails the dev-mode consensus check on those validators.

jsmitchell (Thu, 23 Mar 2017 18:43:37 GMT):
@nconde ^

jsmitchell (Thu, 23 Mar 2017 18:44:47 GMT):
does the consensus check happen before or after the state transition?

jsmitchell (Thu, 23 Mar 2017 18:45:06 GMT):
seems like the state view should be built on top of the transient state from the block

jsmitchell (Thu, 23 Mar 2017 18:45:09 GMT):
@jjason ^

pschwarz (Thu, 23 Mar 2017 18:51:31 GMT):
In genesis, it is using the transient state. There was an issue that I remember discussing with @jjason where this sort of problem occurred on the first block _post_ genesis

pschwarz (Thu, 23 Mar 2017 18:51:51 GMT):
I recall it being fixed by using the transient state

pschwarz (Thu, 23 Mar 2017 18:52:04 GMT):
Could be misremembering, though... :/

cianx (Thu, 23 Mar 2017 18:56:57 GMT):
That dif is not a fix. This is caused by the genesis controller using partial state to instantiate consensus.

cianx (Thu, 23 Mar 2017 18:56:57 GMT):
That dif is not a fix. This is caused by the genesis controller using partial state to onstantiate

boydjohnson (Thu, 23 Mar 2017 18:57:59 GMT):
Yeah, the diff isn't a good fix.

cianx (Thu, 23 Mar 2017 18:59:08 GMT):
State needs to take effect when the block is commited not in middle of a block transition. Which leaves us with a genesis conundrum. What algorithm should validate genesis? Do consensus algorithm s need to be aware of consensus changes?

cianx (Thu, 23 Mar 2017 18:59:08 GMT):
State needs to take effect when the block is commited not in middle of a block transition. Which leaves us with a genesis conundrum. Who should d

cianx (Thu, 23 Mar 2017 18:59:08 GMT):
State needs to take effect when the block is commited not in middle of a block transition. Which leaves us with a genesis conundrum. What algorithm should validate genesis? Do consensus algorithm s need to be aqare of consensus es changes?

jsmitchell (Thu, 23 Mar 2017 19:06:04 GMT):
what are the arguments against first doing the state transition with the transactions in the block to calculate a transient (unapplied) state, then doing consensus validation using a config view based on that transient state, and then applying the transient state once the block is considered valid in all dimensions?

Dan (Thu, 23 Mar 2017 19:13:21 GMT):
@boydjohnson et al ... no reason to change channels now, but note we now have a consensus channel.

cianx (Thu, 23 Mar 2017 19:14:33 GMT):
@Dan we just can't chat right for you.

jsmitchell (Thu, 23 Mar 2017 19:14:58 GMT):
we are chatting the best we can. can't you see that @dan?!!

Dan (Thu, 23 Mar 2017 19:17:12 GMT):
You're just ungrateful. I go get you a nice shiny new channel and you use the old one.

Dan (Thu, 23 Mar 2017 19:18:54 GMT):
no more presents from me

cianx (Thu, 23 Mar 2017 19:23:08 GMT):
When we publish a block we have to do it in the context of the chain head. Since we have to instantiate a consensus algorithm to approve the next block for publishing. The timing of this block approval for publishing affects the batch list for the block. So the final state root hash of the block is dependent on the timing of the consensus algorithm, using transient state makes determining what consensus algorithm to use indeterminate. The determination of the Consensus algorithm in the BlockValidator needs to match the behavior BlockPublisher.

jsmitchell (Thu, 23 Mar 2017 19:25:17 GMT):
yep

jsmitchell (Thu, 23 Mar 2017 19:25:36 GMT):
i buy that

jsmitchell (Thu, 23 Mar 2017 19:26:20 GMT):
ok, so block validation _must_ be grounded in the previous block's ending state

jsmitchell (Thu, 23 Mar 2017 19:26:23 GMT):
for config view

pschwarz (Thu, 23 Mar 2017 19:29:06 GMT):
Which makes sense (and how I thought things were supposed to work), except in during genesis time, which is special

jsmitchell (Thu, 23 Mar 2017 19:29:18 GMT):
but the specialness is causing breakage here, right?

pschwarz (Thu, 23 Mar 2017 19:29:47 GMT):
Yes, which means it's not actually selecting the right consensus module

jsmitchell (Thu, 23 Mar 2017 19:31:22 GMT):
seems like we need a genesis consensus which is the default

jsmitchell (Thu, 23 Mar 2017 19:31:46 GMT):
and is what the genesis block uses

jsmitchell (Thu, 23 Mar 2017 19:32:08 GMT):
which sets config values and establishes the base state for consensus transition in the next block

cianx (Thu, 23 Mar 2017 19:35:15 GMT):
There is already handling in the BlockValidator for the genesis block. It could be modified to return the genesis block state in the case of the genesis block. That would make the BlockValidate match the genesis controllers behavior.

cianx (Thu, 23 Mar 2017 19:35:27 GMT):
I have to go see if I feel dirty for typing that.

cianx (Thu, 23 Mar 2017 19:36:42 GMT):
def _get_previous_block_root_state_hash(self, blkw): if blkw.previous_block_id == NULL_BLOCK_IDENTIFIER: *return blkw.state_root_hash* else: prev_blkw = self._block_cache[blkw.previous_block_id] return prev_blkw.state_root_hash

cianx (Thu, 23 Mar 2017 19:37:10 GMT):
the bold line would be the change to the block validator.

cianx (Thu, 23 Mar 2017 19:37:24 GMT):
@boydjohnson ^ can you try that patch

boydjohnson (Thu, 23 Mar 2017 19:38:03 GMT):
Yeah, Just a second.

boydjohnson (Thu, 23 Mar 2017 19:43:38 GMT):
My vagrant locked up during running a test. So halting and upping vagrant.

jjason (Thu, 23 Mar 2017 19:52:48 GMT):
Special casing the genesis block in such a way means that PoET consensus needs to special case it as well as now the block will not have a wait certificate as expected.

jjason (Thu, 23 Mar 2017 19:54:08 GMT):
Also, because of the change to the BlockPublisher/et al interfaces, we need to make sure that the code above is put somewhere common so they don't have to worry about special casing when they want to create a state view with the state view factory.

cianx (Thu, 23 Mar 2017 19:54:12 GMT):
@jjason the patch I gave boyd will not create that situation.

jjason (Thu, 23 Mar 2017 19:56:39 GMT):
@cianx Sure

cianx (Thu, 23 Mar 2017 19:57:51 GMT):
The oddity is the system definition of block validation state, the Genesis block is validated in the context of its own state root hash. All other blocks are validated in the context of the previous blocks state root hash.

jsmitchell (Thu, 23 Mar 2017 19:59:59 GMT):
feels like we can make that cleaner

boydjohnson (Thu, 23 Mar 2017 20:04:49 GMT):
'''[20:03:07.980 ERROR chain] 'hash 61ec0eb93c0c3f2477e5dc83cf52a213f538cc0818c4511afa4d304e1854618d not found in database' Traceback (most recent call last): File "/project/sawtooth-core/validator/sawtooth_validator/journal/chain.py", line 193, in validate_block create_view(prev_state) File "/project/sawtooth-core/validator/sawtooth_validator/state/state_view.py", line 41, in create_view return StateView(MerkleDatabase(self._database, state_root_hash)) File "/project/sawtooth-core/validator/sawtooth_validator/state/merkle.py", line 36, in __init__ self.set_merkle_root(merkle_root) File "/project/sawtooth-core/validator/sawtooth_validator/state/merkle.py", line 66, in set_merkle_root self._root_node = self._get_by_hash(merkle_root) File "/project/sawtooth-core/validator/sawtooth_validator/state/merkle.py", line 77, in _get_by_hash raise KeyError("hash {} not found in database".format(key_hash)) KeyError: 'hash 61ec0eb93c0c3f2477e5dc83cf52a213f538cc0818c4511afa4d304e1854618d not found in database''''

boydjohnson (Thu, 23 Mar 2017 20:05:43 GMT):
So when the genesis block comes in to a non-genesis validator, the state root on that block is not in the state of the non-genesis validator.

boydjohnson (Thu, 23 Mar 2017 20:06:16 GMT):
@cianx ^

jjason (Thu, 23 Mar 2017 20:54:58 GMT):
@cianx RE: "The oddity is the ...." means also that giving BlockPublisher, et al, the state factory, creating a state view must be postponed until initialize_block, et al, as trying to do so in the __init__ method when being fired up to process the genesis block results in a different state view being available to it. For example, the BlockPublisher.__init__ method, in the case of the genesis block, the object only has access to the state view asscoiated with INIT_ROOT_KEY because it doesn't have access to a block wrapper yet. However, in the BlockPublisher.initialize_block method it would have access to the state view for the genesis block it is being asked to initialize.

jjason (Thu, 23 Mar 2017 20:56:04 GMT):
Found this out the hard way when trying to start up the enclave and working through why the config TP settings were not being honored (e.g., sawtooth.poet.target_wait_time).

jjason (Thu, 23 Mar 2017 21:03:16 GMT):
Actually....I may be seeing something else...in publisher, it calls _build_block with self._chain_head (line 413). It uses this to create the state view (line 111). At this point, am I right in assuming this is not in the block store/block cache yet? Is the state view for this not-yet-initialized block a copy of the state view from the previous block?

jjason (Thu, 23 Mar 2017 21:03:16 GMT):
Actually....I may be seeing something else...in publisher, it calls `_build_block` with self._chain_head (line 413). It uses this to create the state view (line 111). At this point, am I right in assuming this is not in the block store/block cache yet? Is the state view for this not-yet-initialized block a copy of the state view from the previous block?

cianx (Thu, 23 Mar 2017 21:05:21 GMT):
The candidate block does not exist until after finalize block, as with all other blocks that don't exist it will not exist in the block cache.

jjason (Thu, 23 Mar 2017 21:08:15 GMT):
Okay. That is what thought/hoped.

pschwarz (Thu, 23 Mar 2017 21:25:33 GMT):
Chatting a bit out of band with @jsmitchell, @cianx and @Dan on the subject of the genesis block on non-genesis validators, and we arrived at the solution of introducing a GenesisConsensus module that would be the default consensus as configured from the empty state. This change requires that a genesis.batch needs some settings transactions for configuring the consensus modules for blocks 1+ - not an unreasonable ask since we're planning on making sawtooth_settings vote-only with a minimum of one approved public key, which changes `sawtooth admin genesis` to require at least one set of batches -- unlike now, where it has none.

Dan (Thu, 23 Mar 2017 21:26:16 GMT):
@jjason how does poet handle genesis block (or I guess generally any non-poet block) when it comes to stats like aggregate local mean?

adamludvik (Thu, 23 Mar 2017 21:39:05 GMT):
Hey all, I put up a PR to add security linting as a build step. Right now we fail the test but I have it setup so the build doesn't fail...yet. It seems like we could pass with a reasonable amount of effort. https://github.com/hyperledger/sawtooth-core/pull/248 Feel free to review it or add comments even if I didn't request you.

adamludvik (Thu, 23 Mar 2017 21:43:34 GMT):

Message Attachments

adamludvik (Thu, 23 Mar 2017 21:44:06 GMT):
^ That's a PDF of output from bandit run on our code.

Dan (Thu, 23 Mar 2017 21:46:30 GMT):
If we resolve that an issue is a false positive how do we turn it off? Do we have to do like what we do with inline statements to pylint?

adamludvik (Thu, 23 Mar 2017 21:48:32 GMT):
There is a way to create a baseline, and have bandit only fail if new issues are created: https://wiki.openstack.org/wiki/Security/Projects/Bandit#Bandit_Baseline_Gate

jjason (Thu, 23 Mar 2017 21:49:02 GMT):
Great. So, we just run it twice every time and the second time will find no issues. Problem solved.

adamludvik (Thu, 23 Mar 2017 21:49:10 GMT):
I am looking to see if there is something like what pylint has.

adamludvik (Thu, 23 Mar 2017 21:49:18 GMT):
@jjason lol

adamludvik (Thu, 23 Mar 2017 21:53:48 GMT):
We can also configure it to exclude certain directories, skip certain tests, and control thresholds for severity & confidence before it fails the build.

adamludvik (Thu, 23 Mar 2017 21:54:50 GMT):
And you can add a `# nosec` comment to disable checking for a specific line.

Dan (Thu, 23 Mar 2017 21:59:28 GMT):
Cool. @Tom Barnes ^

jsmitchell (Thu, 23 Mar 2017 22:11:03 GMT):
this is groovy

pschwarz (Thu, 23 Mar 2017 22:34:12 GMT):
That's very cool

adamludvik (Thu, 23 Mar 2017 22:52:41 GMT):
We now have published docker images: https://hub.docker.com/u/hyperledger/dashboard/?page=2

adamludvik (Thu, 23 Mar 2017 22:52:56 GMT):
They are all tagged as unstable though, and subject to change at my whim and fancy.

adamludvik (Thu, 23 Mar 2017 22:54:00 GMT):
@amundson

adamludvik (Thu, 23 Mar 2017 22:54:43 GMT):
https://hub.docker.com/r/hyperledger/sawtooth-validator/

arb_shl (Fri, 24 Mar 2017 12:54:25 GMT):
Has joined the channel.

Dan (Fri, 24 Mar 2017 13:20:38 GMT):
Gits giving me a lot of this error lately (since we changed pr process) when I pull master. It clears up if I pull again. Just curious if anyone else sees this or has a remedy: `Unlink of file '.git/objects/pack/pack-3bc9cf86a7886eae7bbc72b1703fd1760d2f21d7.idx' failed. Should I try again? (y/n) y`

Dan (Fri, 24 Mar 2017 13:20:38 GMT):
Git's giving me a lot of this error lately (since we changed pr process) when I pull master. It clears up if I pull again. Just curious if anyone else sees this or has a remedy: `Unlink of file '.git/objects/pack/pack-3bc9cf86a7886eae7bbc72b1703fd1760d2f21d7.idx' failed. Should I try again? (y/n) y`

jsmitchell (Fri, 24 Mar 2017 13:26:50 GMT):
are you running git from inside a vm on a shared filesystem?

Dan (Fri, 24 Mar 2017 14:26:28 GMT):
running git through git bash from windows. I don't pull from within vagrant.. do all my git ops from the git bash shell.

jsmitchell (Fri, 24 Mar 2017 14:26:38 GMT):
oh, windows

Dan (Fri, 24 Mar 2017 14:27:03 GMT):
yes I anticipated that reply ;)

Dan (Fri, 24 Mar 2017 14:34:35 GMT):
hey @adamludvik to run_tests am I expected to first build_all or build_python or ?

adamludvik (Fri, 24 Mar 2017 14:36:18 GMT):
`build_all`

adamludvik (Fri, 24 Mar 2017 14:36:59 GMT):
https://chat.hyperledger.org/channel/sawtoothlake?msg=Ksd3jaG6J3BeeX54o

jjason (Fri, 24 Mar 2017 14:48:25 GMT):
@dan If you only want to run Python tests I only do `build_all -l python` and then `run_tests -x java_sdk -x javascript_sdk`

jjason (Fri, 24 Mar 2017 14:48:53 GMT):
@dan I have not see that unlink problem, but I am not surprised at all as Windows doesn't have a real file system anyway.

jjason (Fri, 24 Mar 2017 14:48:53 GMT):
@dan I have not seen that unlink problem, but I am not surprised at all as Windows doesn't have a real file system anyway.

jsmitchell (Fri, 24 Mar 2017 14:50:52 GMT):
yeah, plus it's probably constantly dorking around stat-ing files and reading them to build searchable indexes or some crap

jsmitchell (Fri, 24 Mar 2017 14:55:36 GMT):
reminder: if you want to lint your code _before_ you push a PR, you can run ./bin/run_lint

jjason (Fri, 24 Mar 2017 14:57:12 GMT):
Stronger reminder: don't even think of pushing your PR without running `./bin/run_lint first`. ;)

jjason (Fri, 24 Mar 2017 14:57:12 GMT):
Stronger reminder: don't even think of pushing your PR without running `./bin/run_lint` first. ;)

jjason (Fri, 24 Mar 2017 15:17:46 GMT):
Question about testing/mocks. I seem several mocks have been created in specific unit test directories under `validator/tests/unit3`. I also see that some unit tests are importing mocks form other unit tests. Since some of these mocks seem generally useful to multiple tests (for example, `MockBlockStore`), would it be beneficial to collect the generic mocks into their own directory for use in multiple unit tests?

agunde (Fri, 24 Mar 2017 15:28:08 GMT):
The idea is that any mocks that are only useful to one set of unit tests for (example only used to test the scheduler) then it should go in the specific unit test directories. If it is a class that could be useful to many unit tests (aka testing library code) it should be moved to a new top level directory called test/sawtooth_test (this has not been created yet)

jjason (Fri, 24 Mar 2017 15:32:54 GMT):
Sounds like a plan to me.

Dan (Fri, 24 Mar 2017 16:11:17 GMT):
I like all my lint publicly exposed

adamludvik (Fri, 24 Mar 2017 17:52:12 GMT):
Could use a couple reviews on this quick versioning fix. Also fixes Ctrl+C not working during the java/javascript steps in build_all: https://github.com/hyperledger/sawtooth-core/pull/250

jsmitchell (Fri, 24 Mar 2017 18:13:59 GMT):
looks like there are some mergeable PRs out there

jsmitchell (Fri, 24 Mar 2017 18:17:02 GMT):
@zac @adamludvik

jsmitchell (Fri, 24 Mar 2017 18:17:54 GMT):
@zac

jsmitchell (Fri, 24 Mar 2017 18:19:58 GMT):
@zac excellent

zac (Fri, 24 Mar 2017 18:20:10 GMT):
Hello

adamludvik (Fri, 24 Mar 2017 18:20:21 GMT):
Waiting for @rbuysse

jsmitchell (Fri, 24 Mar 2017 18:20:52 GMT):
aren't we all

rbuysse (Fri, 24 Mar 2017 18:36:40 GMT):
hi

jsmitchell (Fri, 24 Mar 2017 18:36:52 GMT):
OUR WAIT IS OVER!

jsmitchell (Fri, 24 Mar 2017 18:36:54 GMT):
rejoice!

rbuysse (Fri, 24 Mar 2017 18:37:10 GMT):
don't we have to joice first?

jsmitchell (Fri, 24 Mar 2017 18:38:59 GMT):
If you choose not to deride, you still have made a joice?

Dan (Fri, 24 Mar 2017 19:28:14 GMT):
Did you Rush to that pun?

boydjohnson (Fri, 24 Mar 2017 19:39:16 GMT):
Just calling out, and people are probably doing this regularly, but @zac's merge requires that protogen or build_all be run.

jsmitchell (Fri, 24 Mar 2017 19:39:41 GMT):
I'm basically like while 1: build_all

jsmitchell (Fri, 24 Mar 2017 19:39:50 GMT):
always be building

rjones (Fri, 24 Mar 2017 19:42:37 GMT):
not quite as catchy as Always. Be. Closing.

jsmitchell (Fri, 24 Mar 2017 19:42:50 GMT):
Coffee is for Builders

rjones (Fri, 24 Mar 2017 19:45:51 GMT):
_puts the coffee down_

jsmitchell (Fri, 24 Mar 2017 19:46:07 GMT):
i suspect you build stuff a lot @rjones

rjones (Fri, 24 Mar 2017 19:46:29 GMT):
it's my lot in life ¯\_(ツ)_/¯

pschwarz (Fri, 24 Mar 2017 20:30:01 GMT):
@cianx Right now, the developer experience for creating transaction processors with the Javascript SDK requires a little bit of additionaly setup: 1. Run `bin/build_javascript_sdk`. 2. In their own transaction processor project, add the `sawtooth-sdk` dependency via ``` yarn add file:/project/sawtooth-core/sdk/javascript ``` It will be added as a relative file dependency. 3. From there, they can use `sdk/examples/intkey_javascript` project as an example.

cianx (Fri, 24 Mar 2017 20:31:33 GMT):
Thanks

pschwarz (Fri, 24 Mar 2017 20:32:28 GMT):
Once we are deploying the sdk to the NPM repo, this should be much easier.

cianx (Mon, 27 Mar 2017 06:01:57 GMT):
What is the best way to run a intkey workload against a validator?

cianx (Mon, 27 Mar 2017 06:02:44 GMT):
2nd related but completely different question, what is the best way to write a transaction tester for a new transaction processor?

cianx (Mon, 27 Mar 2017 06:22:18 GMT):
@agunde ^ you might have an answer to the workload?

cianx (Mon, 27 Mar 2017 06:23:52 GMT):
@boydjohnson @drozd ^^ is there a good pattern to follow for building transaction processor/family test frameworks

cianx (Mon, 27 Mar 2017 10:01:15 GMT):
```VAGRANT_BOX = ubuntu/xenial64 VAGRANT_FORWARD_PORTS = true VAGRANT_MEMORY = 2048 VAGRANT_CPUS = 2 Configuring proxyconf plugin! http_proxy: http://proxy-us.intel.com:911 https_proxy: http://proxy-us.intel.com:912 no_proxy: intel.com,.intel.com,10.0.0.0,10.0.0.8,192.168.0.0,192.168.0.16,localhost,127.0.0.0,127.0.0.8,134.134.0.0,134.134.0.16,172.16.0.0,172.16.0.20 Bringing machine 'default' up with 'virtualbox' provider... ==> default: Importing base box 'ubuntu/xenial64'... ==> default: Matching MAC address for NAT networking... ==> default: Checking if box 'ubuntu/xenial64' is up to date... ==> default: Setting the name of the VM: sawtooth ==> default: Destroying VM and associated drives... C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.9.3/lib/vagrant/util/is_port_open.rb:21:in `initialize': The requested address is not valid in its context. - connect(2) for "0.0.0.0" port 8800 (Errno::EADDRNOTAVAIL) from C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.9.3/lib/vagrant/util/is_port_open.rb:21:in `new' ```

cianx (Mon, 27 Mar 2017 10:02:51 GMT):
@adamludvik @amundson ^ I am seeing the error on vagrant up, with Vagrant 1.9.3 and VirtualBox 5.1.18

cianx (Mon, 27 Mar 2017 11:33:44 GMT):
I have seen this on 5 separate machines today. So far it seems to be every windows machine.

cianx (Mon, 27 Mar 2017 11:36:35 GMT):
I put up a pr for this to disable port forwarding but would rather have a fix for port forwarding on windows.

adamludvik (Mon, 27 Mar 2017 12:46:12 GMT):
@cianx Look at sawtooth integration, config, and validator registry for examples of tp tests. They all use some framework code I built in sdk/python/sawtooth_processor_test

adamludvik (Mon, 27 Mar 2017 12:48:34 GMT):
The intkey workload command has a pretty good usage flag if I remember, `intkey workload -h` I think. (I'm on the bus atm.)

adamludvik (Mon, 27 Mar 2017 12:49:47 GMT):
Is the port forwarding issue new? Did anything change recently?

cianx (Mon, 27 Mar 2017 12:52:53 GMT):
I started seeing it Saturday, when I updated my laptop to the latest virtual box and vagrant

cianx (Mon, 27 Mar 2017 12:54:35 GMT):
Today I helped 4 people setup new dev environments all clean installs of the latest virtual box and vagrant. They all had the port forwarding issue.

cianx (Mon, 27 Mar 2017 12:54:46 GMT):
I could not find any leads on the internet.

adamludvik (Mon, 27 Mar 2017 12:58:11 GMT):
Alright, I will take a look soon.

jsmitchell (Mon, 27 Mar 2017 13:02:24 GMT):
https://github.com/mitchellh/vagrant/issues/8395

jsmitchell (Mon, 27 Mar 2017 13:03:48 GMT):
tldr try 127.0.0.1 instead of 0.0.0.0

jsmitchell (Mon, 27 Mar 2017 13:04:01 GMT):
Until the fix is in

adamludvik (Mon, 27 Mar 2017 13:23:59 GMT):
@jsmitchell I will test that and put up a PR if it works, unless you're already working on it

Dan (Mon, 27 Mar 2017 13:47:13 GMT):
baby seal ripe for clubbing: https://github.com/hyperledger/sawtooth-core/pull/254

jsmitchell (Mon, 27 Mar 2017 13:48:10 GMT):
@adamludvik Go for it

agunde (Mon, 27 Mar 2017 13:58:14 GMT):
@cianx Did you get the intkey workload to work?

adamludvik (Mon, 27 Mar 2017 14:14:02 GMT):
@cianx Can you test if this fixes the port forwarding issue? I tested it on a Windows box with the same versions of vagrant and virtual box you mentioned above after reproducing the issue and it seems to work: https://github.com/hyperledger/sawtooth-core/pull/264

Dan (Mon, 27 Mar 2017 15:11:27 GMT):
What's our process for dependent PRs? At least with regard to this scenario.. I put up a simple PR{commit1}. Got to some more involved changes that depend on that PR a day or so later so I put up PR2 {commit1,2,3}. If PR1 gets merged then PR2{commit2,3} is a little easier to review.

pschwarz (Mon, 27 Mar 2017 15:12:32 GMT):
Technically, I think the later PR, if closed first, will auto-close the earlier PR since its commit will also be merged, so long as there wasn't any fix-up/squashes

pschwarz (Mon, 27 Mar 2017 15:13:46 GMT):
Might be best to just close them in favor of the later one, as long as the commits are still nicely separated anyway. They are as easy to review in one PR as multiples.

Dan (Mon, 27 Mar 2017 15:17:05 GMT):
ok, closed #254 in favor of grander #261.

Dan (Mon, 27 Mar 2017 15:17:14 GMT):
ooh colorful.

pschwarz (Mon, 27 Mar 2017 15:17:42 GMT):
That is unexpected - PR's will naturally move though the spectrum as the project ages...

rjones (Mon, 27 Mar 2017 15:33:03 GMT):
@dan I think I've posted the javascript event handler for #sawtoothlake-pr-review before, if anyone can help change it to do what you want I'm all ears (or rather, I'm all copy-paste)

silliman (Mon, 27 Mar 2017 15:38:55 GMT):
Has left the channel.

zac (Mon, 27 Mar 2017 16:07:10 GMT):
Do we want to add a `batch_status` command to the CLI?

zac (Mon, 27 Mar 2017 16:07:23 GMT):
Is that something people would use?

zac (Mon, 27 Mar 2017 16:07:46 GMT):
You can currently `batch show`, and `submit` a batch

jsmitchell (Mon, 27 Mar 2017 16:07:57 GMT):
what would batch_status return?

jsmitchell (Mon, 27 Mar 2017 16:08:09 GMT):
(other than the status of batches)?

zac (Mon, 27 Mar 2017 16:08:19 GMT):
Mostly that

jsmitchell (Mon, 27 Mar 2017 16:08:20 GMT):
like, more specifically, I mean

jsmitchell (Mon, 27 Mar 2017 16:08:31 GMT):
i was just heading off your snark at the pass

zac (Mon, 27 Mar 2017 16:08:37 GMT):
But you coudl tell it to `--wait` if you wanted to know when a batch was committed

zac (Mon, 27 Mar 2017 16:08:42 GMT):
separate from submitting it

jsmitchell (Mon, 27 Mar 2017 16:09:03 GMT):
what are the different possible statuseses

zac (Mon, 27 Mar 2017 16:09:04 GMT):
Because you can already do that with `submit`

zac (Mon, 27 Mar 2017 16:09:32 GMT):
Right now, just `COMMITTED`, `PENDING`, and `UNKNOWN` (i.e. probably rejected)

jsmitchell (Mon, 27 Mar 2017 16:09:47 GMT):
how would it answer COMMITTED?

zac (Mon, 27 Mar 2017 16:09:58 GMT):
There is a place for an `INVALID` status, but no mechanism for detecting that state at the moment

zac (Mon, 27 Mar 2017 16:11:06 GMT):
If you were querying multiple ids: ``` COMMITTED COMMITTED ```

jsmitchell (Mon, 27 Mar 2017 16:11:21 GMT):
no, how does it ask the core if a batch is committed?

zac (Mon, 27 Mar 2017 16:11:47 GMT):
Same a a `batch show` essentially

zac (Mon, 27 Mar 2017 16:11:51 GMT):
Is it in the block store?

jsmitchell (Mon, 27 Mar 2017 16:12:20 GMT):
and pending?

zac (Mon, 27 Mar 2017 16:12:33 GMT):
Checks the batch cache

zac (Mon, 27 Mar 2017 16:12:40 GMT):
If not in either it is Unknown

zac (Mon, 27 Mar 2017 16:12:54 GMT):
Eventually there should be some sort of reject pile

jsmitchell (Mon, 27 Mar 2017 16:13:06 GMT):
batche

zac (Mon, 27 Mar 2017 16:13:13 GMT):
Where Invalid batches would be found, along with a reason

jsmitchell (Mon, 27 Mar 2017 16:13:31 GMT):
yes, this is useful

cianx (Mon, 27 Mar 2017 16:16:04 GMT):
What happened to the docker_build_all command referenced by sawtooth cluster?

cianx (Mon, 27 Mar 2017 16:16:04 GMT):
@adamludvik

zac (Mon, 27 Mar 2017 16:22:38 GMT):
Okay, should be a fairly straightforward add

adamludvik (Mon, 27 Mar 2017 16:23:06 GMT):
`docker_build_all` became `build_all` and `build_all` became `build_python`. Are there residual comments in sawtooth cluster that refer to docker_build_all?

adamludvik (Mon, 27 Mar 2017 16:23:15 GMT):
^ @cianx

zac (Mon, 27 Mar 2017 16:24:21 GMT):
I'm thinking `sawtooth batch status `

zac (Mon, 27 Mar 2017 16:25:01 GMT):
Though that would limit it to one batch's status

zac (Mon, 27 Mar 2017 16:25:19 GMT):
Is it important to be able to check multiple batches at a time with this command?

Dan (Mon, 27 Mar 2017 16:27:56 GMT):
thanks @rjones I found the gist you put up before. https://gist.github.com/rjones-lf/fbe68e5fc9cef6ae6a8847df7d08f739#file-gistfile1-txt

rjones (Mon, 27 Mar 2017 16:29:22 GMT):
@dan I didn't mean to send you on a hunt, I got distracted. thank you. IIRC the issue is we either get a small subset of github events that isn't what you need, or we get the flood from github, so the issue is how to prune the flood to just what you need.

rjones (Mon, 27 Mar 2017 16:31:00 GMT):

Message Attachments

rjones (Mon, 27 Mar 2017 16:32:34 GMT):

Message Attachments

Dan (Mon, 27 Mar 2017 16:42:12 GMT):
It's all good @rjones the search on here is pretty good. We could try taking OFF `pull request`. It looks like `pull request review` might be sufficient. Would be a cheap experiment anyway.

rjones (Mon, 27 Mar 2017 16:42:51 GMT):
OK, so leave `push` and `pull request review` ?

boydjohnson (Mon, 27 Mar 2017 16:44:15 GMT):
So I am closing the fix shutdown smoke PR. The details are that right now in master parallel jobs on jenkins will sometimes fail because of how I wrote the test. All that needs to be done is rebuild the PR if you get a "container already exists" type of error on the shutdown-smoke test. When we switch to running our tests with built packages in a few weeks, I will fix the shutdown-smoke test so that parallel jobs on jenkins can be run.

PacoEcotrust (Mon, 27 Mar 2017 16:49:08 GMT):
Hi, I have this problem with the 0.8 version. I think this error I had with the version 0.7 but I do not remember how I resolved. Some one have any idea. Thanks There was an error while executing `VBoxManage`, a CLI used by Vagrant for controlling VirtualBox. The command and stderr is shown below. Command: ["startvm", "ac466053-04d0-49e8-8dd7-c74850011916", "--type", "headless"] Stderr: VBoxManage: error: The virtual machine 'sawtooth' has terminated unexpectedly during startup with exit code 1 (0x1) VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component MachineWrap, interface IMachine

PacoEcotrust (Mon, 27 Mar 2017 16:49:08 GMT):
Hi, I have this problem with the 0.8 version. I think this error I had with the version 0.7 but I do not remember how I resolved. Some one have any idea. Thanks There was an error while executing `VBoxManage`, a CLI used by Vagrant for controlling VirtualBox. The command and stderr is shown below. Command: ["startvm", "ac466053-04d0-49e8-8dd7-c74850011916", "--type", "headless"] Stderr: VBoxManage: error: The virtual machine 'sawtooth' has terminated unexpectedly during startup with exit code 1 (0x1) VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component MachineWrap, interface IMachine I am using ubuntu 16.04 and virtualbox 5.1

PacoEcotrust (Mon, 27 Mar 2017 16:49:08 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=uzwqwR5W4Kr5tzvXs) @boydjohnson I had to reinstall the virtualbox new version. I saved in another place the virtualmachines of 0.7 version. And I had to try several times the vagrant up because I had some problems. I got to finished to run vagrant up, and now I am in this fase. But, dont worry sure it is my fault with mix of versions, Thanks you.

adamludvik (Mon, 27 Mar 2017 16:53:23 GMT):
Can I get a quick review on this PR? Cian needs it for the hackathon: https://github.com/hyperledger/sawtooth-core/pull/264

Dan (Mon, 27 Mar 2017 16:58:04 GMT):
@rjones yes please. Lets see how it looks if we just turn off `pull request`

Dan (Mon, 27 Mar 2017 16:59:38 GMT):
@adamludvik looks like you got 2 green :heavy_check_mark:'s. Are you good to go or still need more eyes on that?

adamludvik (Mon, 27 Mar 2017 16:59:52 GMT):
Just merged

jsmitchell (Mon, 27 Mar 2017 17:00:02 GMT):
@PacoEcotrust: do you have up to date VirtualBox _and_ up to date VirtualBox guest additions?

rjones (Mon, 27 Mar 2017 17:00:13 GMT):
@dan done

PacoEcotrust (Mon, 27 Mar 2017 17:01:27 GMT):
guest additions, no. I will install it, thanks

cianx (Mon, 27 Mar 2017 17:23:58 GMT):
@adamludvik '''

cianx (Mon, 27 Mar 2017 17:24:06 GMT):
```C:\Users\clmontgo\Workspaces\sawtooth\sawtooth-core\manage\sawtooth_manage\docker.py: 189 raise ManagementError( 190 'Docker images not built: {}. Try running ' 191: '"docker_build_all"'.format( 192 ', '.join(unbuilt))) 193

adamludvik (Mon, 27 Mar 2017 17:37:04 GMT):
Do we have an intended usage for the cluster command? Now that we are publishing docker images on dockerhub, we could change it so it downloads the images automatically?

jjason (Mon, 27 Mar 2017 17:45:28 GMT):
With the merge of the genesis consensus stuff, the PoET consensus stuff is broken for a bit. I need to make a fix. Will try to push a PR ASAP.

jjason (Mon, 27 Mar 2017 17:45:39 GMT):
:warning: :warning: :warning: :warning: :warning: :warning: :warning:

Dan (Mon, 27 Mar 2017 17:46:04 GMT):
@adamludvik I feel like @amundson and @Tom Barnes have usages and opinions on cluster but might not be available today.

jsmitchell (Mon, 27 Mar 2017 17:51:00 GMT):

Message Attachments

jsmitchell (Mon, 27 Mar 2017 17:51:24 GMT):
:neutral_face:

boydjohnson (Mon, 27 Mar 2017 17:51:31 GMT):
So cluster manage type docker doesn't do peering right now. @adamludvik and I have talked about how to fix it.

toddojala (Mon, 27 Mar 2017 17:51:52 GMT):
@boydjohnson I noticed that today. I was going to mention that.

jsmitchell (Mon, 27 Mar 2017 17:55:33 GMT):
a rare slip-up from the usually perfect @jjason

jsmitchell (Mon, 27 Mar 2017 17:55:38 GMT):
this is a dark day indeed

jsmitchell (Mon, 27 Mar 2017 17:56:00 GMT):
i don't know what to believe anymore

adamludvik (Mon, 27 Mar 2017 17:59:47 GMT):
Cluster manage type docker is actually further broken... I renamed the dockerfiles it depends on. We should clarify its usage and intent before making the fix as that will influence what the right fix is.

jjason (Mon, 27 Mar 2017 18:03:01 GMT):
@jsmitchell I knew that I was going to have to make a fix - that was our discussion about what to do if a consensus module encounters a block it didn't create. I was just hoping I would have at least a little more breathing room before I had to do it ;)

jjason (Mon, 27 Mar 2017 18:03:21 GMT):
It only took @drozd an hour to get tripped up by it :(

cianx (Mon, 27 Mar 2017 18:47:06 GMT):
@dan @adamludvik i have some opinions about the sawtooth cluster tool and how it should work with (on support) of the sdks. But details on those will have to wait until I finish at this mexican bar.

cianx (Mon, 27 Mar 2017 18:47:24 GMT):
Yes I am ar a mexican bar in tel aviv.

cianx (Mon, 27 Mar 2017 18:47:24 GMT):
Yes I am at a mexican bar in tel aviv.

Dan (Mon, 27 Mar 2017 18:49:32 GMT):
You're really doubling down on this reentry problem @cintel.

cianx (Mon, 27 Mar 2017 18:50:51 GMT):
You know how I am when I gamble.

jjason (Mon, 27 Mar 2017 18:50:58 GMT):
It's not like they built a wall around the Mexican bar.

Dan (Mon, 27 Mar 2017 18:51:35 GMT):
yet

jjason (Mon, 27 Mar 2017 18:55:38 GMT):
Worse places to be trapped inside

cianx (Mon, 27 Mar 2017 18:57:44 GMT):
I have two teams at the hackathon using sawtooth, 7 developers -- 4 from intel, 3 from the stock exchange.

jsmitchell (Mon, 27 Mar 2017 19:01:18 GMT):
define 'using'

jsmitchell (Mon, 27 Mar 2017 19:01:23 GMT):
:)

jsmitchell (Mon, 27 Mar 2017 19:01:30 GMT):
that's awesome!

kelly_ (Mon, 27 Mar 2017 19:45:22 GMT):
nice @cianx !

mhart (Mon, 27 Mar 2017 20:07:21 GMT):
Has joined the channel.

mhart (Mon, 27 Mar 2017 20:15:52 GMT):
Hi all! Question: getting lots of `Transaction 210f7ec in batch 0db752 has unsatisfied dependency: d8699` type messages coming from the validator (latest master) – what does this mean exactly?

jsmitchell (Mon, 27 Mar 2017 20:16:24 GMT):
@mhart: where are your batches coming from? intkey?

jsmitchell (Mon, 27 Mar 2017 20:16:36 GMT):
or did you write your own transaction family?

mhart (Mon, 27 Mar 2017 20:16:40 GMT):
intkey

mhart (Mon, 27 Mar 2017 20:16:55 GMT):
but also from my own transaction family :-)

mhart (Mon, 27 Mar 2017 20:17:24 GMT):
I saw them in my own transaction family, then tried with intkey and noticed the same thing

jsmitchell (Mon, 27 Mar 2017 20:18:49 GMT):
yeah, i think there is a bug in publishing

jsmitchell (Mon, 27 Mar 2017 20:18:57 GMT):
@agunde @cianx: ^

jsmitchell (Mon, 27 Mar 2017 20:19:09 GMT):
any additional clues?

jsmitchell (Mon, 27 Mar 2017 20:19:46 GMT):
@mhart: the completer enforces ordered delivery of batches to the publisher

jsmitchell (Mon, 27 Mar 2017 20:20:22 GMT):
where 'ordered' means - "if there is a dependent transaction or batch, make sure it is submitted to the publisher first"

mhart (Mon, 27 Mar 2017 20:20:44 GMT):
there's also no feedback at all from the transaction handler – shouldn't there be errors or something? seems to just be silent

jsmitchell (Mon, 27 Mar 2017 20:21:10 GMT):
@mhart: the transaction handler is never reached in this case

mhart (Mon, 27 Mar 2017 20:21:18 GMT):
ah

jsmitchell (Mon, 27 Mar 2017 20:21:31 GMT):
the headers contain enough information for the publisher to decide that the dependency is unsatisfied

cianx (Mon, 27 Mar 2017 20:21:31 GMT):
@mhart can you see if the dependencies were committed on a block or dropped?

cianx (Mon, 27 Mar 2017 20:21:31 GMT):
@mhart can you see if the drpendencies were committed on a block or dropped?

jsmitchell (Mon, 27 Mar 2017 20:21:56 GMT):
so, there is no sense in scheduling the transactions in the batch for processing

mhart (Mon, 27 Mar 2017 20:22:57 GMT):
@cianx before the first `unsatisfied dependency` message there's a collection of `ServerThread receiving CLIENT_BATCH_SUBMIT_REQUEST` messages

mhart (Mon, 27 Mar 2017 20:23:04 GMT):
nothing in between

jsmitchell (Mon, 27 Mar 2017 20:23:35 GMT):
those receiving CLIENT_BATCH_SUBMIT_REQUEST messages are the client submitting the batches to the component interconnect

mhart (Mon, 27 Mar 2017 20:24:18 GMT):
(this is when running `validator -vv` btw)

cianx (Mon, 27 Mar 2017 20:24:32 GMT):
Does @agunde have a recipe to repeat this on intkey?

mhart (Mon, 27 Mar 2017 20:25:46 GMT):
Is it a result of the blockchain being in a certain state? If I delete `/data` for example, should that work again?

jsmitchell (Mon, 27 Mar 2017 20:26:33 GMT):
the 'start fresh' procedure is: rm -rf /home/ubuntu/sawtooth/data/*; ./bin/sawtooth admin genesis; ./bin/validator -vv

mhart (Mon, 27 Mar 2017 20:26:55 GMT):
right, pretty sure that's what I did and ended up with the same problems – lemme try again

jsmitchell (Mon, 27 Mar 2017 20:27:07 GMT):
yeah, i know i've seen this too.

mhart (Mon, 27 Mar 2017 20:27:24 GMT):
this was on master – from a commit sometime yesterday – dunno if that's part of the issue? should I be using a specific tag?

jsmitchell (Mon, 27 Mar 2017 20:27:49 GMT):
no, i think it's a current defect

mhart (Mon, 27 Mar 2017 20:27:52 GMT):
kk

jsmitchell (Mon, 27 Mar 2017 20:29:06 GMT):
@mhart: how are you loading intkey? `intkey workload` or `intkey create_batch; intkey load`

jsmitchell (Mon, 27 Mar 2017 20:29:06 GMT):
@mhart: how are you loading intkey? `intkey workload` or `intkey create_batch; intkey load`?

mhart (Mon, 27 Mar 2017 20:29:17 GMT):
the latter

jsmitchell (Mon, 27 Mar 2017 20:29:24 GMT):
@mhart: also, how many validators are you running?

mhart (Mon, 27 Mar 2017 20:29:27 GMT):
just one

jsmitchell (Mon, 27 Mar 2017 20:29:30 GMT):
ok, cool

jsmitchell (Mon, 27 Mar 2017 20:36:31 GMT):
@cianx: @agunde is going to ping you on this

jsmitchell (Mon, 27 Mar 2017 20:38:27 GMT):
@cianx: is it like midnight there?

cianx (Mon, 27 Mar 2017 20:38:35 GMT):
K, get the dets on how to repro it and I will see what I can do in the am

jsmitchell (Mon, 27 Mar 2017 20:39:11 GMT):
start up a validator and do `./bin/intkey create_batch -c 1000; ./bin/intkey load`

cianx (Mon, 27 Mar 2017 20:39:27 GMT):
Pretty much, good thing I swore off sleep

Dan (Mon, 27 Mar 2017 20:48:01 GMT):
Good thing taco bell is open all night.

rjones (Mon, 27 Mar 2017 20:48:18 GMT):
_ehhhh_

jjason (Mon, 27 Mar 2017 20:48:19 GMT):
Seems like you picked a bad week to stop sniffing glue.

jsmitchell (Mon, 27 Mar 2017 20:48:19 GMT):
Bell Aviv

jjason (Mon, 27 Mar 2017 20:48:30 GMT):
:airplan:

jjason (Mon, 27 Mar 2017 20:48:30 GMT):
:airplane:

rjones (Mon, 27 Mar 2017 20:48:45 GMT):
:woo:

mhart (Mon, 27 Mar 2017 20:58:50 GMT):
hmmm, just reset everything and rebuilt everything and the batches are loading ok now... so I'm not *exactly* sure how to repro

jsmitchell (Mon, 27 Mar 2017 20:59:16 GMT):
yeah, it's not every time

mhart (Mon, 27 Mar 2017 21:02:28 GMT):
I'm assuming loading the same batch of transactions twice is supposed to error? It's not exactly clear what's happening, just a bunch of `ServerThread sending CLIENT_BATCH_SUBMIT_RESPONSE to b'961c31a4279c4e33'` messages

mhart (Mon, 27 Mar 2017 21:03:02 GMT):
`intkey load` doesn't error in any way, nor does the validator

jsmitchell (Mon, 27 Mar 2017 21:03:15 GMT):
did you `rm -rf /home/ubuntu/sawtooth/data/*` and `./bin/sawtooth admin genesis`

jsmitchell (Mon, 27 Mar 2017 21:03:25 GMT):
prior to starting validator

jsmitchell (Mon, 27 Mar 2017 21:03:26 GMT):
?

mhart (Mon, 27 Mar 2017 21:03:38 GMT):
yep

jsmitchell (Mon, 27 Mar 2017 21:03:41 GMT):
if you don't do that, it will just sit there on batch submission

jsmitchell (Mon, 27 Mar 2017 21:04:13 GMT):
are you running tp_intkey_python with -vv?

mhart (Mon, 27 Mar 2017 21:04:37 GMT):
yup – I'm talking about submitting the same batch twice right now

jsmitchell (Mon, 27 Mar 2017 21:04:55 GMT):
they will be dropped at the border as duplicates

mhart (Mon, 27 Mar 2017 21:05:04 GMT):
right, no messages indicating that though

jsmitchell (Mon, 27 Mar 2017 21:05:22 GMT):
are you running the validator with -vv?

mhart (Mon, 27 Mar 2017 21:05:26 GMT):
yup

mhart (Mon, 27 Mar 2017 21:05:36 GMT):

Message Attachments

jsmitchell (Mon, 27 Mar 2017 21:05:48 GMT):
if they come in over the network interface, there is definitely a dup message

jsmitchell (Mon, 27 Mar 2017 21:06:15 GMT):
@agunde: ^ re: submitting batches over the interconnect - a good reason to refactor the intkey stuff to the rest api

mhart (Mon, 27 Mar 2017 21:07:13 GMT):
interesting – if I generate a new batch and submit I seem to get a mix of success and `Transaction rejected due missing dependency`

mhart (Mon, 27 Mar 2017 21:07:29 GMT):
so some work, some don't (I think that's what's happening)

mhart (Mon, 27 Mar 2017 21:08:02 GMT):
so maybe that's one way to reproduce... not sure

jsmitchell (Mon, 27 Mar 2017 21:08:16 GMT):
we'll check into it

mhart (Mon, 27 Mar 2017 21:08:46 GMT):
cool – but basically, if I see it again, just reset things and try again?

jsmitchell (Mon, 27 Mar 2017 21:09:09 GMT):
yeah, i suspect you'll see it somewhat frequently :)

mhart (Mon, 27 Mar 2017 21:09:48 GMT):
are there any specific requirements for what a transaction handler must do – or can it just do nothing (ie, log and return) and that will be considered a success?

jsmitchell (Mon, 27 Mar 2017 21:10:53 GMT):
@boydjohnson: interesting question. What happens if a transaction processor doesn't interact with state on a context?

boydjohnson (Mon, 27 Mar 2017 21:12:33 GMT):
The handler.apply doesn't have to do anything, the sdk will interpret the lack of InvalidTransactionError, and InternalError as success.

agunde (Mon, 27 Mar 2017 21:13:55 GMT):
@jsmitchell I can make a story to refactor intkey to use the rest api

mhart (Mon, 27 Mar 2017 21:14:22 GMT):
@boydjohnson ok, thanks

jjason (Mon, 27 Mar 2017 21:29:06 GMT):
Looking to unblock @drozd . Need 2 reviewers for https://github.com/hyperledger/sawtooth-core/pull/265

drozd (Mon, 27 Mar 2017 21:34:04 GMT):
@jjason I don't think I should be part of the approval process, but it seems to pass the test now

jjason (Mon, 27 Mar 2017 21:36:42 GMT):
@drozd No worries.

mhart (Mon, 27 Mar 2017 21:36:43 GMT):
@boydjohnson @jsmitchell is there something special about the intkey batches that require a response? If I change the handler to just return (not modify any state), then it only seems to process one transaction – all the rest don't seem to make it

mhart (Mon, 27 Mar 2017 21:37:07 GMT):
the validator just fills up with `unsatisfied dependency` messages

mhart (Mon, 27 Mar 2017 21:39:00 GMT):
is it maybe better to stick with the REST API for submitting transactions (like in the XO example) instead of the intkey batch examples?

boydjohnson (Mon, 27 Mar 2017 21:39:25 GMT):
I would wonder if there is a bug in the context manager when committing a context that doesn't update state, and so not only doesn't the merkle root get changed, but it may get set to something odd.

jsmitchell (Mon, 27 Mar 2017 21:39:41 GMT):
@mhart: definitely use the rest api for submission

mhart (Mon, 27 Mar 2017 21:39:56 GMT):
@jsmitchell ok, will do

jsmitchell (Mon, 27 Mar 2017 21:40:06 GMT):
the intkey method predates the existence of the rest api

mhart (Mon, 27 Mar 2017 22:00:58 GMT):
Hmmm...

mhart (Mon, 27 Mar 2017 22:01:16 GMT):
Struggling to get `xo` to work

mhart (Mon, 27 Mar 2017 22:01:22 GMT):

Message Attachments

mhart (Mon, 27 Mar 2017 22:01:51 GMT):
the `create` seems to have worked – but the `show` can't find it...?

mhart (Mon, 27 Mar 2017 22:02:14 GMT):
validator says `Unable to find entry at address 5b7349c2f7ee46bc96272...`

jsmitchell (Mon, 27 Mar 2017 22:04:19 GMT):
you started ./bin/tp_xo_python, right?

mhart (Mon, 27 Mar 2017 22:04:55 GMT):
yup – but no logs there so far

mhart (Mon, 27 Mar 2017 22:05:07 GMT):
started with `tp_xo_python -vv`

jsmitchell (Mon, 27 Mar 2017 22:05:11 GMT):
hm. i just tried your steps and it worked for me

jsmitchell (Mon, 27 Mar 2017 22:05:24 GMT):
```ubuntu@ubuntu-xenial:/project/sawtooth-core$ ./bin/tp_xo_python +-------------------------------+ + Player 0341ae created a game. + +-------------------------------+ +-------------------------------+ + Player 0341ae created a game. + +-------------------------------+```

mhart (Mon, 27 Mar 2017 22:05:48 GMT):
ok, will restart the transaction processor

mhart (Mon, 27 Mar 2017 22:06:47 GMT):
same thing – here are the validator logs...

mhart (Mon, 27 Mar 2017 22:07:08 GMT):

Message Attachments

mhart (Mon, 27 Mar 2017 22:07:50 GMT):

Message Attachments

jsmitchell (Mon, 27 Mar 2017 22:08:21 GMT):
the dispatch to the transaction processor is not occurring

mhart (Mon, 27 Mar 2017 22:09:33 GMT):
maybe need to reset data again? just weird that there's no errors anywhere

jsmitchell (Mon, 27 Mar 2017 22:10:07 GMT):
yes, every time you restart the validator, you need to delete the data dir and redo `sawtooth admin genesis`

mhart (Mon, 27 Mar 2017 22:10:17 GMT):
yup

mhart (Mon, 27 Mar 2017 22:10:46 GMT):
why is that? surely that's not intended in production?

jsmitchell (Mon, 27 Mar 2017 22:11:14 GMT):
no, we just haven't focused on the restart case yet

mhart (Mon, 27 Mar 2017 22:11:56 GMT):
ok, worked after validator/data reset

mhart (Mon, 27 Mar 2017 22:14:47 GMT):
so – question – if the `create` wasn't actually working, how come there was no error anywhere? is it because the xo client doesn't handle errors at the moment?

jsmitchell (Mon, 27 Mar 2017 22:14:57 GMT):
it's not an error

mhart (Mon, 27 Mar 2017 22:15:02 GMT):
or is there some other step you need to do?

jsmitchell (Mon, 27 Mar 2017 22:15:13 GMT):
the validator accepted your batch for processing

jsmitchell (Mon, 27 Mar 2017 22:15:19 GMT):
and it is enqueued

jsmitchell (Mon, 27 Mar 2017 22:15:48 GMT):
it's just not processing because it's in an inconsistent state based on loading files off of disk

jsmitchell (Mon, 27 Mar 2017 22:15:51 GMT):
(which is a bug)

mhart (Mon, 27 Mar 2017 22:15:57 GMT):
oh – you need to poll the batch state to see if it succeeded or not?

jsmitchell (Mon, 27 Mar 2017 22:16:01 GMT):
yes

mhart (Mon, 27 Mar 2017 22:16:05 GMT):
huh, ok

jsmitchell (Mon, 27 Mar 2017 22:16:12 GMT):
and 'succeded' means very different things in blockchain land

jsmitchell (Mon, 27 Mar 2017 22:16:12 GMT):
and 'succeeded' means very different things in blockchain land

jsmitchell (Mon, 27 Mar 2017 22:16:44 GMT):
for example, it could be put on a block and committed to state, and then _reversed_ by a fork

mhart (Mon, 27 Mar 2017 22:17:13 GMT):
I have a feeling that's something many ppl will want to do (wait until the batch was actually processed) – are there plans to add support to the APIs for that?

jsmitchell (Mon, 27 Mar 2017 22:17:24 GMT):
finality of transactions is a matter of a confidence interval based on the depth in the chain

jsmitchell (Mon, 27 Mar 2017 22:17:32 GMT):
yes, that's in there. It's called wait for commit

mhart (Mon, 27 Mar 2017 22:17:36 GMT):
ok

jsmitchell (Mon, 27 Mar 2017 22:18:21 GMT):
https://github.com/hyperledger/sawtooth-core/pull/252

mhart (Mon, 27 Mar 2017 22:19:21 GMT):
gotcha – also noticed it just now in the REST API – I guess xo just doesn't use it (yet)

jsmitchell (Mon, 27 Mar 2017 22:19:35 GMT):
look at the date on that PR. :)

mhart (Mon, 27 Mar 2017 22:21:12 GMT):
I'm not sure xo uses that? or does it?

mhart (Mon, 27 Mar 2017 22:21:20 GMT):
i thought it was just making raw HTTP calls

jsmitchell (Mon, 27 Mar 2017 22:21:20 GMT):
no, the feature was just added today

jsmitchell (Mon, 27 Mar 2017 22:21:31 GMT):
xo doesn't do it yet

mhart (Mon, 27 Mar 2017 22:21:33 GMT):
i mean, it doesn't use the rest client

jsmitchell (Mon, 27 Mar 2017 22:22:04 GMT):
no, probably not. @zac and @adamludvik can provide more context

jsmitchell (Mon, 27 Mar 2017 22:22:37 GMT):
gotta run

mhart (Mon, 27 Mar 2017 22:22:59 GMT):
cool – I think I've got enough to go on now – thanks!

zac (Tue, 28 Mar 2017 01:20:49 GMT):
I believe XO uses the REST API, but I don't think it uses the `wait` parameter yet

mhart (Tue, 28 Mar 2017 04:12:34 GMT):
@zac right, it uses the REST API, but it doesn't use the rest client that james was referring to in that PR

zac (Tue, 28 Mar 2017 13:57:51 GMT):
Ah sure. For the record, I would consider the rest client to be an internal tool, mostly just for the CLI, though I think a couple of tests borrow it. It might be useful to build some sort of rest client sdk at some point, but the existing client is not functionally complete, well documented, or well tested. It just does what the CLI needs and no more.

zac (Tue, 28 Mar 2017 13:58:39 GMT):
I would recommend using whatever HTTP/JSON libraries you prefer and interact with the REST API directly.

adamludvik (Tue, 28 Mar 2017 14:04:17 GMT):
@mhart I wrote the xo client and it predates the existence of the wait parameter and the RestClient, so it does not utilize either. I would second @zac's suggestion that the REST API be used directly, as he has put a lot of work into defining and documenting that interface, whereas the RestClient hasn't been clearly defined or documented.

mhart (Tue, 28 Mar 2017 14:06:30 GMT):
@zac @adamludvik yeah, thanks all – have actually gotten a REST client working in Node.js talking XO transactions, so all good so far 👍

zac (Tue, 28 Mar 2017 14:06:44 GMT):
Excellent

agunde (Tue, 28 Mar 2017 14:07:15 GMT):
@mhart Currently working on a fix for the dependency issue you were seeing yesterday. Hopefully will have a fix up later today.

jsmitchell (Tue, 28 Mar 2017 14:07:32 GMT):
@mhart: cool! if you want to contribute that at some point to core, go ahead and push a PR.

mhart (Tue, 28 Mar 2017 14:08:10 GMT):
the docs are pretty unclear on the signing stuff to be honest (ie, which fields expect bytes, which fields expect hex strings) – but using-the-source-luke got me most of the way

jsmitchell (Tue, 28 Mar 2017 14:08:37 GMT):
yeah, we have some cleanup and some docs to improve for sure

mhart (Tue, 28 Mar 2017 14:09:53 GMT):
@jsmitchell to be honest, having the signing stuff written in a number of langs will probably be better than documentation anyway – even for newer lang implementations – always hard to exactly describe those requirements without some sort of code examples

jsmitchell (Tue, 28 Mar 2017 14:10:26 GMT):
A PR went in yesterday which simplified the singing interface

mhart (Tue, 28 Mar 2017 14:10:29 GMT):
Once I've gotten our custom transaction family up and running, will be happy to commit some Node.js REST client code back – wherever it fits

jsmitchell (Tue, 28 Mar 2017 14:10:38 GMT):
(got rid of some of the encode/decode stuff)

jsmitchell (Tue, 28 Mar 2017 14:10:47 GMT):
https://github.com/hyperledger/sawtooth-core/pull/261

zac (Tue, 28 Mar 2017 14:13:02 GMT):
Sounds like something for `/sdk`

zac (Tue, 28 Mar 2017 14:13:35 GMT):
Is it an extension of the exist JavaScript SDK?

zac (Tue, 28 Mar 2017 14:13:35 GMT):
Is it an extension of the existing JavaScript SDK?

mhart (Tue, 28 Mar 2017 14:13:43 GMT):
oh cool – as long as the basic standards haven't changed (wif for key storage, secp256k1 for signing, public keys and signatures specified as hex in transactions), I think the Node.js code I've written shouldn't change

mhart (Tue, 28 Mar 2017 14:14:05 GMT):
@zac it reaches in to the existing JS SDK to get the protobuf descriptors

mhart (Tue, 28 Mar 2017 14:14:19 GMT):
As in, I have to do `require('sawtooth-sdk/protobuf/protobuf_bundle.json')`

mhart (Tue, 28 Mar 2017 14:14:27 GMT):
Because it's not currently exported

mhart (Tue, 28 Mar 2017 14:14:34 GMT):
Aside from that, doesn't use anything from the SDK

mhart (Tue, 28 Mar 2017 14:15:06 GMT):
Because the SDK is all around transaction handlers

mhart (Tue, 28 Mar 2017 14:15:08 GMT):
so far

jsmitchell (Tue, 28 Mar 2017 14:15:23 GMT):
@mhart: yeah, that stuff should all be consistent (signing formats)

zac (Tue, 28 Mar 2017 14:15:29 GMT):
Ah yes, that's right

zac (Tue, 28 Mar 2017 14:15:48 GMT):
We do have a task for adding client functionality to it actually

zac (Tue, 28 Mar 2017 14:16:11 GMT):
But the team is putting most of their time into core functionality at the moment

mhart (Tue, 28 Mar 2017 14:16:12 GMT):
I use https://www.npmjs.com/package/secp256k1 and https://www.npmjs.com/package/wif for the various unpacking/signing requirements

zac (Tue, 28 Mar 2017 14:16:44 GMT):
So a PR with those improvements could definitely be helpful

mhart (Tue, 28 Mar 2017 14:17:02 GMT):
Once the right incantations were made with the signing, it was very straightforward – ain't that always the way? :-)

mhart (Tue, 28 Mar 2017 14:18:23 GMT):
Took me many hours to get the signing right though – this code in particular had me scratching my head – trying to figure out exactly what the "compact" serialization was: https://github.com/hyperledger/sawtooth-core/blob/master/signing/sawtooth_signing/secp256k1_signer.py#L136-L137

mhart (Tue, 28 Mar 2017 14:19:52 GMT):
Also took me a long time to figure out that "signing a message" was actually really "signing the sha256 of the message"

mhart (Tue, 28 Mar 2017 14:20:58 GMT):
Because that's not dealt with by the `secp256k1` npm package – it kept complaining that the data being signed wasn't 32 bytes... anyway, got there in the end

jsmitchell (Tue, 28 Mar 2017 14:21:52 GMT):
@pschwarz: ^ good bits for a js signing interface

mhart (Tue, 28 Mar 2017 14:23:23 GMT):
It's as simple as: ```function sign(msg, key) { const msg32 = crypto.createHash('sha256').update(msg).digest() return secp256k1.sign(msg32, key).signature.toString('hex') } ```

mhart (Tue, 28 Mar 2017 14:23:23 GMT):
It's as simple as: ```function sign(msg, key) { const msg32 = crypto.createHash('sha256').update(msg).digest() return secp256k1.sign(msg32, key).signature.toString('hex') } ```

mhart (Tue, 28 Mar 2017 14:23:23 GMT):
It's as simple as:```function sign(msg, key) { const msg32 = crypto.createHash('sha256').update(msg).digest() return secp256k1.sign(msg32, key).signature.toString('hex') } ```

mhart (Tue, 28 Mar 2017 14:23:23 GMT):
It's as simple as: ``` function sign(msg, key) { const msg32 = crypto.createHash('sha256').update(msg).digest() return secp256k1.sign(msg32, key).signature.toString('hex') } ```

jsmitchell (Tue, 28 Mar 2017 14:23:26 GMT):
looks like that compact serialization is 64 bytes (32 of r, 32 of s)

mhart (Tue, 28 Mar 2017 14:24:19 GMT):
Just took me ages to get there :-)

jsmitchell (Tue, 28 Mar 2017 14:25:55 GMT):
if you could jot down notes on things that are confusing or missing in the code or the docs as you go, that would be very helpful feedback for us

mhart (Tue, 28 Mar 2017 14:28:20 GMT):
I'm still a little unsure about the serialization to be honest – the docs mentioned DER format, but when I used the DER serializer from the Node module it came out differently from what the python code was giving me – it ended up just being that the raw signature in hex worked, so that's what I left it as – but I'm unsure as to whether that will fail in certain cases... has matched the python signatures so far, so we'll see

jsmitchell (Tue, 28 Mar 2017 14:29:18 GMT):
@Dan: ^

mhart (Tue, 28 Mar 2017 14:32:15 GMT):
From the npm module docs, it seems that they follow https://github.com/bitcoin/bips/blob/master/bip-0066.mediawiki for the DER serialization

mhart (Tue, 28 Mar 2017 14:34:53 GMT):
(it's this document that mentions DER serialization btw: http://intelledger.github.io/0.8/sawtooth_developers_guide/signing_transactions.html )

Dan (Tue, 28 Mar 2017 14:35:59 GMT):
Thanks @mhart I will take a look.

jsmitchell (Tue, 28 Mar 2017 14:36:21 GMT):
@mhart: another area we've had feedback on confusion is the state tree key encoding. It's 35 bytes represented as a hexdigest, where the first three bytes are the namespace for the transaction family and the remaining 32 bytes are transaction family specific encoding.

jsmitchell (Tue, 28 Mar 2017 14:37:12 GMT):
the examples right now typically generate a fixed 6 hex character namespace prefix and append 64 characters of hash of some sensible domain specific natural key structure

mhart (Tue, 28 Mar 2017 14:37:15 GMT):
@jsmitchell you're talking about the `address` scheme?

jsmitchell (Tue, 28 Mar 2017 14:37:17 GMT):
yeah

mhart (Tue, 28 Mar 2017 14:37:20 GMT):
yeah, I noticed that

mhart (Tue, 28 Mar 2017 14:37:51 GMT):
so it's got a fixed size of 35 bytes?

jsmitchell (Tue, 28 Mar 2017 14:37:57 GMT):
that is the intent

mhart (Tue, 28 Mar 2017 14:37:58 GMT):
or is the size variable?

jsmitchell (Tue, 28 Mar 2017 14:38:03 GMT):
it's not currently enforced

jsmitchell (Tue, 28 Mar 2017 14:38:08 GMT):
but it is the intent

jsmitchell (Tue, 28 Mar 2017 14:38:34 GMT):
right now you can write to non 'leaf' nodes in the tree using partial addresses

jsmitchell (Tue, 28 Mar 2017 14:38:38 GMT):
i would not recommend that

mhart (Tue, 28 Mar 2017 14:39:02 GMT):
ok – so "natural" addresses like `namespace/entityType/entityId` can't really be done – without encoding

mhart (Tue, 28 Mar 2017 14:39:18 GMT):
I mean – not unless you're always within 35 bytes

jsmitchell (Tue, 28 Mar 2017 14:39:27 GMT):
that's a perfect thing to hash[:64]

mhart (Tue, 28 Mar 2017 14:39:36 GMT):
yeah, sort of – ideally you wouldn't have to

mhart (Tue, 28 Mar 2017 14:39:49 GMT):
I'm guessing range queries over addresses isn't a thing

jsmitchell (Tue, 28 Mar 2017 14:39:54 GMT):
not a thing

jsmitchell (Tue, 28 Mar 2017 14:41:00 GMT):
we can enumerate the tree from a given partial address, but if the addresses are unstructured, that doesn't really have any meaning, other than 'everything under a namespace'

jsmitchell (Tue, 28 Mar 2017 14:41:19 GMT):
if you wish, you can come up with another encoding scheme that is meaning preserving under the namespace

jsmitchell (Tue, 28 Mar 2017 14:41:53 GMT):
as long as the addresses you generate are 70 hex characters

mhart (Tue, 28 Mar 2017 14:42:07 GMT):
gotcha

mhart (Tue, 28 Mar 2017 14:42:34 GMT):
hashes always make me nervous when it comes to requiring unique addresses

jsmitchell (Tue, 28 Mar 2017 14:42:44 GMT):
the universe is a big place :)

mhart (Tue, 28 Mar 2017 14:42:48 GMT):
truedat

jsmitchell (Tue, 28 Mar 2017 14:43:02 GMT):
plus, you can deal with collisions in your design if you are a paranoid type

mhart (Tue, 28 Mar 2017 14:44:01 GMT):
I guess a 1 in 1.94266889e84 chance ain't that big

mhart (Tue, 28 Mar 2017 14:44:05 GMT):
:-)

pschwarz (Tue, 28 Mar 2017 14:44:39 GMT):
The settings values are a good, simple example of that: we treat them the way you'd tree hash map collisions - highly, highly unlikely hashmap collisions in this case

mhart (Tue, 28 Mar 2017 14:44:59 GMT):
I guess it's really 1 in 1.15792089e77 within a namespace (256 bits)

mhart (Tue, 28 Mar 2017 14:46:11 GMT):
i'm definitely nervous about namespace clashes – I would have thought it's best *not* to hash the namespace, but to declare it explicitly

jsmitchell (Tue, 28 Mar 2017 14:46:47 GMT):
yeah, we have more thinking to do on that topic

jsmitchell (Tue, 28 Mar 2017 14:47:14 GMT):
potentially will do something like a namespace registry for an instance of the blockchain

mhart (Tue, 28 Mar 2017 14:47:27 GMT):
right

mhart (Tue, 28 Mar 2017 14:47:30 GMT):
otherwise it's opaque – and you can't see anywhere whether you might have a collision

jsmitchell (Tue, 28 Mar 2017 14:47:45 GMT):
and then also have configurable 'write permissions' which determine which transaction families can write to that namespace

jsmitchell (Tue, 28 Mar 2017 14:47:59 GMT):
(obvs. everything can read from everywhere)

jsmitchell (Tue, 28 Mar 2017 14:48:41 GMT):
but the data stored at a location is domain specific, so we need to think about a sensible abstraction for transaction families to share that interface/implementation

jsmitchell (Tue, 28 Mar 2017 14:49:45 GMT):
@mhart: it would also be nice to avoid having clients query the running instance to be able to construct transactions

jsmitchell (Tue, 28 Mar 2017 14:50:12 GMT):
which is a current benefit to the lazy approach

mhart (Tue, 28 Mar 2017 14:57:12 GMT):
right, I wouldn't necessarily think you'd need that – I think you could keep it "lazy", just instead of creating an opaque hash, make the 3 bytes explicit (eg, `'xo '), so that even those browsing code could have a good guess of whether the namespaces might clash (and then perhaps it's only on transaction submission that the write permissions, etc are checked)

mhart (Tue, 28 Mar 2017 14:57:12 GMT):
right, I wouldn't necessarily think you'd need that – I think you could keep it "lazy", just instead of creating an opaque hash, make the 3 bytes explicit (eg, `'xo '`), so that even those browsing code could have a good guess of whether the namespaces might clash (and then perhaps it's only on transaction submission that the write permissions, etc are checked)

mhart (Tue, 28 Mar 2017 15:01:41 GMT):
guess it's kinda situation specific – and depends on how much ownership of the namespaces you have in your cluster

mhart (Tue, 28 Mar 2017 15:03:18 GMT):
anyway, not too important at this stage for the proof-of-concept stuff

mhart (Tue, 28 Mar 2017 15:03:31 GMT):
can someone explain how dependencies work across transaction batches?

jsmitchell (Tue, 28 Mar 2017 15:04:03 GMT):
http://intelledger.github.io/0.8/architecture/transactions_and_batches.html

mhart (Tue, 28 Mar 2017 15:04:13 GMT):
as in – if I specify a dependency on another transaction in another batch – how does the validator schedule them?

mhart (Tue, 28 Mar 2017 15:04:20 GMT):
or is the idea that it just rejects them

mhart (Tue, 28 Mar 2017 15:04:24 GMT):
and you resubmit

jsmitchell (Tue, 28 Mar 2017 15:04:44 GMT):
there is a component called the Completer which will not submit a batch to the publisher until its dependencies have been satisfied

mhart (Tue, 28 Mar 2017 15:04:54 GMT):
so it waits?

jsmitchell (Tue, 28 Mar 2017 15:04:59 GMT):
it will request the dependent batch from the network

jsmitchell (Tue, 28 Mar 2017 15:05:07 GMT):
yes, it waits

jsmitchell (Tue, 28 Mar 2017 15:05:31 GMT):
if the dependent batch doesn't arrive, at some point the orphaned batch will be dropped

mhart (Tue, 28 Mar 2017 15:05:40 GMT):
ok

mhart (Tue, 28 Mar 2017 15:07:15 GMT):
btw, do transaction IDs have to be the header signature, or is that just a convenient scheme?

jsmitchell (Tue, 28 Mar 2017 15:07:29 GMT):
they have to be the header signature

mhart (Tue, 28 Mar 2017 15:08:04 GMT):
right, yeah, sorry, just read the docs `List of transaction.header_signatures`

Dan (Tue, 28 Mar 2017 15:11:54 GMT):
Is cbor just an example for serializing the transaction body or is that mandatory.

Dan (Tue, 28 Mar 2017 15:11:54 GMT):
Is cbor just an example for serializing the transaction ~body~ payload or is that mandatory.

jsmitchell (Tue, 28 Mar 2017 15:12:15 GMT):
@Dan: the transaction payload?

jsmitchell (Tue, 28 Mar 2017 15:12:20 GMT):
it is opaque to the core

jsmitchell (Tue, 28 Mar 2017 15:12:31 GMT):
it's an implementation decision by the transaction family developer

pschwarz (Tue, 28 Mar 2017 15:13:07 GMT):
e.g the sawtooth_config transactions use protobuf for the transaction bodies

pschwarz (Tue, 28 Mar 2017 15:13:07 GMT):
e.g the sawtooth_config transactions use protobuf for the transaction ~bodies~ payloads.

jsmitchell (Tue, 28 Mar 2017 15:13:14 GMT):
payloads

pschwarz (Tue, 28 Mar 2017 15:13:27 GMT):
err..yes, that

Dan (Tue, 28 Mar 2017 15:13:32 GMT):
yea sorry payload

mhart (Tue, 28 Mar 2017 15:30:59 GMT):
@Dan for example, XO uses utf8 csv for its encoding instead of cbor

Dan (Tue, 28 Mar 2017 15:32:03 GMT):
cool. I was taking a look at the signing docs you pointed out. cbor read like a requirement, so making some notes about what needs updating there. thanks for pointing this stuff out. it's really timely feedback.

mhart (Tue, 28 Mar 2017 15:32:21 GMT):
yeah, i thought the same initially 👍

Dan (Tue, 28 Mar 2017 15:34:18 GMT):
on the DER encoding were you making a separate call to a DER encoder method after getting the signature?

mhart (Tue, 28 Mar 2017 15:34:39 GMT):
yeah, I was

mhart (Tue, 28 Mar 2017 15:34:47 GMT):
So, looking into the code behind the npm module

mhart (Tue, 28 Mar 2017 15:35:06 GMT):
I think they're doing the compact serialization by default – just on the normal exposed `sign` method: https://github.com/cryptocoinjs/secp256k1-node/blob/master/src/ecdsa.cc#L76

mhart (Tue, 28 Mar 2017 15:35:31 GMT):
Which is why, when I convert that to hex, it matches up with the way the python code is doing things

mhart (Tue, 28 Mar 2017 15:35:46 GMT):
But the "DER" serialization threw me

mhart (Tue, 28 Mar 2017 15:35:55 GMT):
I thought I had to do something extra with it

Dan (Tue, 28 Mar 2017 15:37:04 GMT):
Got it. Before you figured that out were you putting the result into a DER encoder provided by a different node library?

mhart (Tue, 28 Mar 2017 15:37:14 GMT):
no, same library

mhart (Tue, 28 Mar 2017 15:37:26 GMT):
just wasn't comparing the raw sig to begin with

mhart (Tue, 28 Mar 2017 15:38:28 GMT):
the `signatureExport` method in the node module, uses the native `secp256k1_ecdsa_signature_serialize_der` behind the scenes: https://github.com/cryptocoinjs/secp256k1-node/blob/master/src/signature.cc#L47

mhart (Tue, 28 Mar 2017 15:38:46 GMT):
So – are you sure what you're doing is DER serialization?

Dan (Tue, 28 Mar 2017 15:39:17 GMT):
I need to look at the js code but I think they are both getting an already DER encoded sig from the c library.

mhart (Tue, 28 Mar 2017 15:39:22 GMT):
Because when I used that, it *didn't* work – only the "compact" serialization works

Dan (Tue, 28 Mar 2017 15:40:36 GMT):
yeah there's a couple signature formats and then there's the encoding for what constitutes a number.

Dan (Tue, 28 Mar 2017 15:41:35 GMT):
I had picked 'compact' for consistency with some older code, but I'll take a look again this week and make sure we are making good general purpose decisions there.

mhart (Tue, 28 Mar 2017 15:42:15 GMT):
I think the compact serialization might be different from DER

mhart (Tue, 28 Mar 2017 15:42:30 GMT):
esp based on my reading of https://github.com/ludbb/secp256k1-py#class-secp256k1ecdsa

jsmitchell (Tue, 28 Mar 2017 15:43:12 GMT):
yeah, compact looks like r and s packed together into an array of 64 bytes

mhart (Tue, 28 Mar 2017 15:43:17 GMT):

Message Attachments

mhart (Tue, 28 Mar 2017 15:46:26 GMT):
raises a question I had – who determines whether the signature is good? is it just the validator? or do the transaction handlers do it too?

mhart (Tue, 28 Mar 2017 15:46:38 GMT):
signature(s)

Dan (Tue, 28 Mar 2017 15:46:53 GMT):
thanks I'll take a look. My recollection is DER just prescribes what constitutes different number formats. It's not specific to ecdsa. but I will take a look later this week and make sure we are clean there.

jsmitchell (Tue, 28 Mar 2017 15:47:05 GMT):
there is a signature verification step at ingress into the validator

jsmitchell (Tue, 28 Mar 2017 15:47:17 GMT):
the transaction handlers do not do it

mhart (Tue, 28 Mar 2017 15:47:25 GMT):
ok – so the signature scheme, encoding, etc is essentially bound to the sawtooth lake version?

jsmitchell (Tue, 28 Mar 2017 15:48:03 GMT):
yep, although obviously we'll want that to be stable long term

mhart (Tue, 28 Mar 2017 15:48:21 GMT):
ie, "sawtooth v0.8 uses secp256k1 ECDSA compact"

mhart (Tue, 28 Mar 2017 15:48:28 GMT):
etc

Dan (Tue, 28 Mar 2017 15:50:59 GMT):
Yep. We need to be clear on external interface expectations. Signature format is <> and public key format is <> (for signature verification). Private key storage format is up to the user as I don't think there's a use case right now for passing private keys around.

mhart (Tue, 28 Mar 2017 15:52:21 GMT):
Ah right, that's a good point – so the transaction handlers are responsible for checking the key signing?

mhart (Tue, 28 Mar 2017 15:54:15 GMT):
wait, that doesn't make any asense

mhart (Tue, 28 Mar 2017 15:54:15 GMT):
wait, that doesn't make any sense

mhart (Tue, 28 Mar 2017 15:54:22 GMT):
forget that question :-)

Dan (Tue, 28 Mar 2017 15:55:39 GMT):
:)

mhart (Tue, 28 Mar 2017 16:00:12 GMT):
Question around the `inputs`/`outputs` fields – they're a control mechanism? So the transaction handler can't read/write to any addresses except those specified?

mhart (Tue, 28 Mar 2017 16:00:12 GMT):
Question around the `inputs` / `outputs` fields – they're a control mechanism? So the transaction handler can't read/write to any addresses except those specified?

jsmitchell (Tue, 28 Mar 2017 16:01:53 GMT):
Yes

mhart (Tue, 28 Mar 2017 16:01:55 GMT):
if so, that's kinda interesting – makes a very tight coupling between the transaction body and transaction handler

jsmitchell (Tue, 28 Mar 2017 16:02:54 GMT):
It allows us to do parallel execution of transactions via isolation of state changes

mhart (Tue, 28 Mar 2017 16:02:56 GMT):
ie, you have to know ahead of time all the things the transaction may want to look up

mhart (Tue, 28 Mar 2017 16:02:56 GMT):
ie, you have to know ahead of time all the things the transaction handler may want to look up

jsmitchell (Tue, 28 Mar 2017 16:03:05 GMT):
You can specify wildcards

mhart (Tue, 28 Mar 2017 16:03:09 GMT):
ah

mhart (Tue, 28 Mar 2017 16:03:15 GMT):
cool cool

jsmitchell (Tue, 28 Mar 2017 16:03:24 GMT):
That just means we get less parallelism

mhart (Tue, 28 Mar 2017 16:03:52 GMT):
gotcha – ok, didn't realize the wildcard thing – that's a bit more flexible

jsmitchell (Tue, 28 Mar 2017 16:04:14 GMT):
Yeah, you can just specify your namespace in a worst case scenario

mhart (Tue, 28 Mar 2017 16:04:57 GMT):
right – good to know – ideally you don't do that, but if you have to...

mhart (Tue, 28 Mar 2017 16:14:02 GMT):
Just a nit on the XO transaction handler – a number of the exception details are swallowed, eg: https://github.com/hyperledger/sawtooth-core/blob/master/sdk/examples/xo_python/sawtooth_xo/processor/handler.py#L53-L57 – this makes it hard to figure out what the actual problem might be (in my case I had to convert to a utf8 buffer instead of using a utf8 string in my Node.js client code)

mhart (Tue, 28 Mar 2017 16:14:57 GMT):
the underlying exception was a utf8 conversion exception, but it was just being rethrown as a generic "Invalid payload serialization" exception

mhart (Tue, 28 Mar 2017 16:16:07 GMT):
so including details of the original caught exception would be a bit more useful

agunde (Tue, 28 Mar 2017 16:25:45 GMT):
@mhart I just merged a fix that should resolve that dropping batches that should have satisfied dependencies issue you were seeing yesterday

mhart (Tue, 28 Mar 2017 16:26:20 GMT):
@agunde oh cool, thanks! will give the intkey batches another shot

mhart (Tue, 28 Mar 2017 16:33:38 GMT):
@agunde seems to be working much better so far – all batches processing successfully

agunde (Tue, 28 Mar 2017 16:34:02 GMT):
Sweet! Thanks for testing again :)

mhart (Tue, 28 Mar 2017 16:35:42 GMT):
Seems to be taking a while though...

mhart (Tue, 28 Mar 2017 16:36:37 GMT):
Can't tell if there's some sort of infinite loop or not – it seems to be processing a hell of a lot of transactions, but I think there are only 1000 generated by `intkey create_batch` aren't there?

agunde (Tue, 28 Mar 2017 16:37:45 GMT):
1000 batches with up to 20 transactions each

mhart (Tue, 28 Mar 2017 16:37:45 GMT):
Unless... does `intkey create_batch` append, or rewrite batches.intkey?

mhart (Tue, 28 Mar 2017 16:37:54 GMT):
if it appends... then that might explain it

agunde (Tue, 28 Mar 2017 16:38:08 GMT):
It should rewrite

mhart (Tue, 28 Mar 2017 16:38:20 GMT):
ok – ah, so 20k transactions, got it

agunde (Tue, 28 Mar 2017 16:39:23 GMT):
The actual number of transaction is randomly chosen between 1 and 20 so yes up to 20k transactions

mhart (Tue, 28 Mar 2017 16:39:43 GMT):
Ah, done

agunde (Tue, 28 Mar 2017 16:39:49 GMT):
Also currently we do not have a limit on the number of batches in a block so most of those will end up in one block

mhart (Tue, 28 Mar 2017 16:42:33 GMT):
Ok – so here's a question – if I change the intkey transaction handler to just log and return, and then I load the batches – it processes the very first transaction, but then nothing after that (ie, doesn't look like the validator sends anymore to the transaction handler)

mhart (Tue, 28 Mar 2017 16:43:58 GMT):

Message Attachments

mhart (Tue, 28 Mar 2017 16:44:27 GMT):
this is after resetting the data, restarting everything, etc

mhart (Tue, 28 Mar 2017 16:47:01 GMT):

Message Attachments

mhart (Tue, 28 Mar 2017 16:47:36 GMT):

Message Attachments

mhart (Tue, 28 Mar 2017 16:48:11 GMT):
so... is the validator (or something) expecting the transaction handler to modify the state in some way?

mhart (Tue, 28 Mar 2017 16:54:12 GMT):
Just tried again – this time I get a bunch of `unsatisfied dependency` logs in the validator – still just the one transaction log from the transaction handler though

jsmitchell (Tue, 28 Mar 2017 17:31:05 GMT):
try setting some state

jsmitchell (Tue, 28 Mar 2017 17:31:25 GMT):
i've never tried a no-op handler before

mhart (Tue, 28 Mar 2017 17:35:18 GMT):
Confirmed

mhart (Tue, 28 Mar 2017 17:35:35 GMT):
If I call `state.set` – even with an invalid address

mhart (Tue, 28 Mar 2017 17:35:38 GMT):
It works

jsmitchell (Tue, 28 Mar 2017 17:35:50 GMT):
@boydjohnson: ^ can you create a defect story for this?

mhart (Tue, 28 Mar 2017 17:35:53 GMT):

Message Attachments

mhart (Tue, 28 Mar 2017 17:36:22 GMT):
I mean – it errors when trying to set the state – but all the transactions continue to process

boydjohnson (Tue, 28 Mar 2017 17:36:30 GMT):
yeah, I'll do that.

pschwarz (Tue, 28 Mar 2017 17:36:50 GMT):
Weird

jjason (Tue, 28 Mar 2017 20:59:51 GMT):
Anyone else seen issues when a consensus module returns `False` from its `initialize_block()` method? I am seeing the publisher still coming back calling `check_publish_block()` on a failed block initialization.

jjason (Tue, 28 Mar 2017 21:00:04 GMT):
[20:57:28.232 INFO chain] Chain controller initialized with chain head: f6044138(0, S:30ebab89, P:00000000) [20:57:28.232 INFO publisher] Now building on top of block: f6044138(0, S:30ebab89, P:00000000) [20:57:28.556 DEBUG consensus_state_store] Create consensus store: /home/ubuntu/sawtooth/data/poet_consensus_state.lmdb [20:57:28.557 INFO poet_enclave_factory] Load PoET enclave module: sawtooth_poet_simulator.poet_enclave_simulator.poet_enclave_simulator [20:57:28.558 INFO poet_enclave_factory] sawtooth.poet.target_wait_time: 20.000000 [20:57:28.558 INFO poet_enclave_factory] sawtooth.poet.initial_wait_time: 3000.000000 [20:57:28.559 INFO poet_enclave_factory] sawtooth.poet.certificate_sample_length: 50 [20:57:28.559 INFO poet_enclave_factory] sawtooth.poet.fixed_duration_blocks: 50 [20:57:28.559 INFO poet_enclave_factory] sawtooth.poet.minimum_wait_time: 1.000000 [20:57:28.966 INFO poet_block_publisher] Register Validator Name=validator-b86489a8, ID=0321fd18...b86489a8, PoET public key=03b32b2b...6e7b16d7 [20:57:28.967 DEBUG poet_block_publisher] We cannot initialize the block because we need to publish PoET signup information [20:57:28.967 DEBUG publisher] Consensus not ready to build candidate block. [20:57:28.970 DEBUG interconnect] ServerThread sending TP_PROCESS_REQUEST to b'454717bdf4a8431f' [20:57:28.974 DEBUG interconnect] ServerThread receiving TP_STATE_GET_REQUEST message: 210 bytes [20:57:28.975 DEBUG tp_state_handlers] GET: [('6a437247a1c12c0fb03aa6e242e6ce988d1cdc7fcc8c2a62ab3ab1202325d7d677e84c', None)] [20:57:28.977 DEBUG interconnect] ServerThread sending TP_STATE_GET_RESPONSE to b'454717bdf4a8431f' [20:57:28.979 DEBUG interconnect] ServerThread receiving TP_STATE_SET_REQUEST message: 353 bytes [20:57:28.980 DEBUG tp_state_handlers] SET: ['6a437247a1c12c0fb03aa6e242e6ce988d1cdc7fcc8c2a62ab3ab1202325d7d677e84c'] [20:57:28.981 DEBUG interconnect] ServerThread sending TP_STATE_SET_RESPONSE to b'454717bdf4a8431f' [20:57:28.985 DEBUG interconnect] ServerThread receiving TP_STATE_SET_REQUEST message: 2172 bytes [20:57:28.988 DEBUG tp_state_handlers] SET: ['6a4372f843d7d846ae7528591d0bb0e3f74b68a141df4267ae5f6ee653d9380993713a'] [20:57:28.989 DEBUG interconnect] ServerThread sending TP_STATE_SET_RESPONSE to b'454717bdf4a8431f' [20:57:28.993 DEBUG interconnect] ServerThread receiving TP_PROCESS_RESPONSE message: 69 bytes [20:57:28.994 DEBUG interconnect] message round trip: TP_PROCESS_RESPONSE 0.024252891540527344 [20:57:29.071 CRITICAL publisher] on_check_publish_block exception. [20:57:29.074 ERROR publisher] 'NoneType' object has no attribute 'has_expired' Traceback (most recent call last): File "/project/sawtooth-core/validator/sawtooth_validator/journal/publisher.py", line 419, in on_check_publish_block self._consensus.check_publish_block(self. File "/project/sawtooth-core/consensus/poet/core/sawtooth_poet/poet_consensus/poet_block_publisher.py", line 338, in check_publish_block return self._wait_timer.has_expired(now=time.time()) AttributeError: 'NoneType' object has no attribute 'has_expired'

jjason (Tue, 28 Mar 2017 21:00:04 GMT):
[20:57:28.232 INFO chain] Chain controller initialized with chain head: f6044138(0, S:30ebab89, P:00000000) [20:57:28.232 INFO publisher] Now building on top of block: f6044138(0, S:30ebab89, P:00000000) [20:57:28.556 DEBUG consensus_state_store] Create consensus store: /home/ubuntu/sawtooth/data/poet_consensus_state.lmdb [20:57:28.557 INFO poet_enclave_factory] Load PoET enclave module: sawtooth_poet_simulator.poet_enclave_simulator.poet_enclave_simulator [20:57:28.558 INFO poet_enclave_factory] sawtooth.poet.target_wait_time: 20.000000 [20:57:28.558 INFO poet_enclave_factory] sawtooth.poet.initial_wait_time: 3000.000000 [20:57:28.559 INFO poet_enclave_factory] sawtooth.poet.certificate_sample_length: 50 [20:57:28.559 INFO poet_enclave_factory] sawtooth.poet.fixed_duration_blocks: 50 [20:57:28.559 INFO poet_enclave_factory] sawtooth.poet.minimum_wait_time: 1.000000 [20:57:28.966 INFO poet_block_publisher] Register Validator Name=validator-b86489a8, ID=0321fd18...b86489a8, PoET public key=03b32b2b...6e7b16d7 [20:57:28.967 DEBUG poet_block_publisher] We cannot initialize the block because we need to publish PoET signup information `[20:57:28.967 DEBUG publisher] Consensus not ready to build candidate block.` [20:57:28.970 DEBUG interconnect] ServerThread sending TP_PROCESS_REQUEST to b'454717bdf4a8431f' [20:57:28.974 DEBUG interconnect] ServerThread receiving TP_STATE_GET_REQUEST message: 210 bytes [20:57:28.975 DEBUG tp_state_handlers] GET: [('6a437247a1c12c0fb03aa6e242e6ce988d1cdc7fcc8c2a62ab3ab1202325d7d677e84c', None)] [20:57:28.977 DEBUG interconnect] ServerThread sending TP_STATE_GET_RESPONSE to b'454717bdf4a8431f' [20:57:28.979 DEBUG interconnect] ServerThread receiving TP_STATE_SET_REQUEST message: 353 bytes [20:57:28.980 DEBUG tp_state_handlers] SET: ['6a437247a1c12c0fb03aa6e242e6ce988d1cdc7fcc8c2a62ab3ab1202325d7d677e84c'] [20:57:28.981 DEBUG interconnect] ServerThread sending TP_STATE_SET_RESPONSE to b'454717bdf4a8431f' [20:57:28.985 DEBUG interconnect] ServerThread receiving TP_STATE_SET_REQUEST message: 2172 bytes [20:57:28.988 DEBUG tp_state_handlers] SET: ['6a4372f843d7d846ae7528591d0bb0e3f74b68a141df4267ae5f6ee653d9380993713a'] [20:57:28.989 DEBUG interconnect] ServerThread sending TP_STATE_SET_RESPONSE to b'454717bdf4a8431f' [20:57:28.993 DEBUG interconnect] ServerThread receiving TP_PROCESS_RESPONSE message: 69 bytes [20:57:28.994 DEBUG interconnect] message round trip: TP_PROCESS_RESPONSE 0.024252891540527344 [20:57:29.071 CRITICAL publisher] on_check_publish_block exception. [20:57:29.074 ERROR publisher] 'NoneType' object has no attribute 'has_expired' Traceback (most recent call last): File "/project/sawtooth-core/validator/sawtooth_validator/journal/publisher.py", line 419, in on_check_publish_block self._consensus.check_publish_block(self. File "/project/sawtooth-core/consensus/poet/core/sawtooth_poet/poet_consensus/poet_block_publisher.py", line 338, in check_publish_block return self._wait_timer.has_expired(now=time.time()) AttributeError: 'NoneType' object has no attribute 'has_expired'

jjason (Tue, 28 Mar 2017 21:00:04 GMT):
[20:57:28.232 INFO chain] Chain controller initialized with chain head: f6044138(0, S:30ebab89, P:00000000) [20:57:28.232 INFO publisher] Now building on top of block: f6044138(0, S:30ebab89, P:00000000) [20:57:28.556 DEBUG consensus_state_store] Create consensus store: /home/ubuntu/sawtooth/data/poet_consensus_state.lmdb [20:57:28.557 INFO poet_enclave_factory] Load PoET enclave module: sawtooth_poet_simulator.poet_enclave_simulator.poet_enclave_simulator [20:57:28.558 INFO poet_enclave_factory] sawtooth.poet.target_wait_time: 20.000000 [20:57:28.558 INFO poet_enclave_factory] sawtooth.poet.initial_wait_time: 3000.000000 [20:57:28.559 INFO poet_enclave_factory] sawtooth.poet.certificate_sample_length: 50 [20:57:28.559 INFO poet_enclave_factory] sawtooth.poet.fixed_duration_blocks: 50 [20:57:28.559 INFO poet_enclave_factory] sawtooth.poet.minimum_wait_time: 1.000000 [20:57:28.966 INFO poet_block_publisher] Register Validator Name=validator-b86489a8, ID=0321fd18...b86489a8, PoET public key=03b32b2b...6e7b16d7 [20:57:28.967 DEBUG poet_block_publisher] We cannot initialize the block because we need to publish PoET signup information `[20:57:28.967 DEBUG publisher] Consensus not ready to build candidate block.` [20:57:28.970 DEBUG interconnect] ServerThread sending TP_PROCESS_REQUEST to b'454717bdf4a8431f' [20:57:28.974 DEBUG interconnect] ServerThread receiving TP_STATE_GET_REQUEST message: 210 bytes [20:57:28.975 DEBUG tp_state_handlers] GET: [('6a437247a1c12c0fb03aa6e242e6ce988d1cdc7fcc8c2a62ab3ab1202325d7d677e84c', None)] [20:57:28.977 DEBUG interconnect] ServerThread sending TP_STATE_GET_RESPONSE to b'454717bdf4a8431f' [20:57:28.979 DEBUG interconnect] ServerThread receiving TP_STATE_SET_REQUEST message: 353 bytes [20:57:28.980 DEBUG tp_state_handlers] SET: ['6a437247a1c12c0fb03aa6e242e6ce988d1cdc7fcc8c2a62ab3ab1202325d7d677e84c'] [20:57:28.981 DEBUG interconnect] ServerThread sending TP_STATE_SET_RESPONSE to b'454717bdf4a8431f' [20:57:28.985 DEBUG interconnect] ServerThread receiving TP_STATE_SET_REQUEST message: 2172 bytes [20:57:28.988 DEBUG tp_state_handlers] SET: ['6a4372f843d7d846ae7528591d0bb0e3f74b68a141df4267ae5f6ee653d9380993713a'] [20:57:28.989 DEBUG interconnect] ServerThread sending TP_STATE_SET_RESPONSE to b'454717bdf4a8431f' [20:57:28.993 DEBUG interconnect] ServerThread receiving TP_PROCESS_RESPONSE message: 69 bytes [20:57:28.994 DEBUG interconnect] message round trip: TP_PROCESS_RESPONSE 0.024252891540527344 `[20:57:29.071 CRITICAL publisher] on_check_publish_block exception.` [20:57:29.074 ERROR publisher] 'NoneType' object has no attribute 'has_expired' Traceback (most recent call last): File "/project/sawtooth-core/validator/sawtooth_validator/journal/publisher.py", line 419, in on_check_publish_block self._consensus.check_publish_block(self. File "/project/sawtooth-core/consensus/poet/core/sawtooth_poet/poet_consensus/poet_block_publisher.py", line 338, in check_publish_block return self._wait_timer.has_expired(now=time.time()) AttributeError: 'NoneType' object has no attribute 'has_expired'

jjason (Tue, 28 Mar 2017 21:00:17 GMT):
@cianx @jsmitchell ^^^^^^

jsmitchell (Tue, 28 Mar 2017 21:15:00 GMT):
i haven't seen that myself

jsmitchell (Tue, 28 Mar 2017 21:16:29 GMT):
There appears to be no consequence to that condition ("Consensus not ready...") other than that log message

jjason (Tue, 28 Mar 2017 21:21:50 GMT):
Yeah....I am going to do a quick test to see if it is as simple as returning `None` from `Publisher._build_block()`. Otherwise, I will put a story/defect in JIRA, although it will block using PoET in any network of greater than 1 validator (unless I put a hack in the Poet consensus module to work around this).

jjason (Tue, 28 Mar 2017 21:24:47 GMT):
That is....after I fix what I broke in the PoET genesis tool :(

pschwarz (Tue, 28 Mar 2017 21:26:12 GMT):
It should probably drop the block?

jsmitchell (Tue, 28 Mar 2017 21:27:08 GMT):
it doesn't feel like it should proceed until initialize is True

jsmitchell (Tue, 28 Mar 2017 21:27:33 GMT):
given the need for early binding in some consensus implementations

jjason (Tue, 28 Mar 2017 21:27:54 GMT):
Yep.

jsmitchell (Tue, 28 Mar 2017 21:28:48 GMT):
can initialize_block block until complete?

jjason (Tue, 28 Mar 2017 21:28:51 GMT):
That is the problem I am running into - in the `initialize_block()` I need to create the wait timer, but if the initialization fails, there is no wait timer object, thus the exception in `check_claim_block()` when it tries to check the wait timer object has expired.

jjason (Tue, 28 Mar 2017 21:28:59 GMT):
That seems like bad mojo.

jsmitchell (Tue, 28 Mar 2017 21:29:05 GMT):
it probably is

jjason (Tue, 28 Mar 2017 21:29:29 GMT):
I could do that, but it would block for potentially a while - it has to wait for the validator regsitry entry to be published by another validator.

jjason (Tue, 28 Mar 2017 21:29:29 GMT):
I could do that, but it would block for potentially a while - it has to wait for the validator registry entry to be published by another validator.

jsmitchell (Tue, 28 Mar 2017 21:30:05 GMT):
there is no sense in building until that happens though, right?

jjason (Tue, 28 Mar 2017 21:30:17 GMT):
Yep. And that is one of the checks I make.

jsmitchell (Tue, 28 Mar 2017 21:30:27 GMT):
so, we either poll there or make a blocking call

jjason (Tue, 28 Mar 2017 21:30:32 GMT):
Yep

jjason (Tue, 28 Mar 2017 21:31:06 GMT):
Either that.....or the publisher doesn't bother to try to initialize again until either (1) it gets some more transactions or (2) it gets a block from someone else.

jsmitchell (Tue, 28 Mar 2017 21:33:22 GMT):
you could raise an exception and catch it in on_check_publish_block and return empty

jsmitchell (Tue, 28 Mar 2017 21:34:11 GMT):
looks like returning None would have the same effect, but the exception is clearer imo

jjason (Tue, 28 Mar 2017 21:34:20 GMT):
I did short circuit that to check if the wait timer object was `None` and the prevents the exception, but the publisher doesn't seem to like that much either as I return `False` basically meaning that I am never ready to claim.

jjason (Tue, 28 Mar 2017 21:34:48 GMT):
`None` would be returned internally in `publish.py`, which I think may be the right thing to do.

jjason (Tue, 28 Mar 2017 21:58:36 GMT):
So, returning `None` from `Publisher._build_block()` when `initialize_block()` returns `False` prevents calling `check_claim_block()`. The side effect is that `initialize_block()` gets called ever 10ms :)

jjason (Wed, 29 Mar 2017 01:44:51 GMT):
Had a couple of PRs that I had to rebuild once or twice before it passed the PoET smoke test. I have been noticing the in local testing, I see forks even with just a single validator. I don't know if the test is occasionally hitting this and that causes the test to fail.

jjason (Wed, 29 Mar 2017 01:44:51 GMT):
Had a couple of PRs that I had to rebuild once or twice before it passed the PoET smoke test. I have been noticing that in local testing I see forks even with just a single validator. I don't know if the test is occasionally hitting this and that causes the test to fail.

cianx (Wed, 29 Mar 2017 02:16:28 GMT):
@jjason the intention is for the BlockPublisher._build_block to return None if the consensus.initialize_block returns false. the returns statement is missing in the BlockPublisher. The polling you mention above is intentional, the Publisher is either trying to build a block or publish a block.

cianx (Wed, 29 Mar 2017 02:16:28 GMT):
@jjason the intention is for the BlockPublisher._build_block to return None if the conesnsus.initialize_block returns false. the returns statement is missing in the BlockPublisher. The polling you mention above is intentional, the Publisher is either trying to build a block or publish a block.

cianx (Wed, 29 Mar 2017 02:18:02 GMT):
PR approved :), thanks,

jjason (Wed, 29 Mar 2017 02:22:23 GMT):
@cianx You're welcome

jjason (Wed, 29 Mar 2017 02:22:32 GMT):
Thanks for checking PR

PacoEcotrust (Wed, 29 Mar 2017 09:55:08 GMT):
ubuntu@ubuntu-xenial:/project$ /project/sawtooth-core/bin/build_all [---Building java---] [ Building sawtooth-build-java... ] Cannot connect to the Docker daemon. Is the docker daemon running on this host? Set up version 0.8. Following the manual. Some ideas?

PacoEcotrust (Wed, 29 Mar 2017 09:55:08 GMT):
ubuntu@ubuntu-xenial:/project$ /project/sawtooth-core/bin/build_all [---Building java---] [ Building sawtooth-build-java... ] Cannot connect to the Docker daemon. Is the docker daemon running on this host? Set up version 0.8 and into the ssh session. Following the manual. Some ideas?

PacoEcotrust (Wed, 29 Mar 2017 09:55:08 GMT):
Hi, in the version 0.8 is using docker into vagrant. Why it is not installed?

PacoEcotrust (Wed, 29 Mar 2017 09:55:08 GMT):
ubuntu@ubuntu-xenial:/project/sawtooth-core$ validator -v (zmq/backend/cython/socket.c:5156) File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:7535) zmq.error.ZMQError: Address already in use [16:27:29.536 ERROR cli] Address already in use Traceback (most recent call last): File "/project/sawtooth-core/validator/sawtooth_validator/server/cli.py", line 145, in main validator.start() File "/project/sawtooth-core -------------------------------------------after run for second time the validator, seem that does not clean well the port (socket)

toddojala (Wed, 29 Mar 2017 15:03:42 GMT):
@PacoEcotrust @adamludvik Hey Adam - maybe you have some insight here? PacoEcotrust: in my Sawtooth Lake provided Vagrant, docker is installed. The command "which docker" outputs: /usr/bin/docker

boydjohnson (Wed, 29 Mar 2017 15:07:16 GMT):
@PacoEcotrust Did you create a new vagrant environment when you switched to 0.8?

adamludvik (Wed, 29 Mar 2017 15:07:40 GMT):
^

cianx (Wed, 29 Mar 2017 18:12:18 GMT):

Message Attachments

cianx (Wed, 29 Mar 2017 18:13:15 GMT):
Here is a photo of the rock stars ont the tvu hackathon team that decided to use sawtooth this morning.

cianx (Wed, 29 Mar 2017 18:13:15 GMT):
Here is a photo oof the rock stars ont tvu hackathon tean that decided to ude sawtooth this morning.

jsmitchell (Wed, 29 Mar 2017 19:33:25 GMT):
nice!

pschwarz (Wed, 29 Mar 2017 19:34:30 GMT):
Youths!

Dan (Wed, 29 Mar 2017 22:37:30 GMT):
someone remind me etiquette for notifying a reviewer that a change has been made? is it removing them and re-adding them to the reviewers list?

jjason (Wed, 29 Mar 2017 22:56:43 GMT):
If they have requested changes, you should re-add them as reviewer.

jjason (Wed, 29 Mar 2017 22:57:16 GMT):
I have found, if delivered with a basket of muffins, you get a better result ;)

Dan (Wed, 29 Mar 2017 22:58:10 GMT):
Well I'm not giving up any of this Ben and Jerry's.

jjason (Wed, 29 Mar 2017 22:58:47 GMT):
I like that. Every review comes with a free pint of Ben & Jerry's!

Dan (Wed, 29 Mar 2017 23:22:14 GMT):
:ice_cream:

jjason (Wed, 29 Mar 2017 23:37:04 GMT):
@adamludvik Have you seen this: ```[---Running cli tests---] Creating network "60dab34261e16d87fa2d732409b9b5241562acbc0a4bc0641b7c79e200295516_default" with the default driver could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network Failed to start all docker containers. Do `docker ps -a` to list residual containers. Running: ['docker-compose', '-p', '60dab34261e16d87fa2d732409b9b5241562acbc0a4bc0641b7c79e200295516', '-f', './cli/tests/unit_cli.yaml', 'up', '--abort-on-container-exit']```

jjason (Wed, 29 Mar 2017 23:37:26 GMT):
Popping up in build when running tests on PR on GitHub

adamludvik (Wed, 29 Mar 2017 23:38:15 GMT):
Yeahhhhh, Ryan and I are aware of it.

jjason (Wed, 29 Mar 2017 23:38:28 GMT):
Excellent.

PacoEcotrust (Thu, 30 Mar 2017 08:43:19 GMT):
ubuntu@ubuntu-xenial:/project$ /project/sawtooth-core/bin/build_all [---Building java---] [ Building sawtooth-build-java... ] Cannot connect to the Docker daemon. Is the docker daemon running on this host? ubuntu@ubuntu-xenial:/project$

PacoEcotrust (Thu, 30 Mar 2017 08:43:19 GMT):
ubuntu@ubuntu-xenial:/project$ /project/sawtooth-core/bin/build_all [---Building java---] [ Building sawtooth-build-java... ] Cannot connect to the Docker daemon. Is the docker daemon running on this host? ubuntu@ubuntu-xenial:/project$ -----------------------Do you know the command to run the deamon?

PacoEcotrust (Thu, 30 Mar 2017 08:43:19 GMT):
ubuntu@ubuntu-xenial:/project$ /project/sawtooth-core/bin/build_all [---Building java---] [ Building sawtooth-build-java... ] Cannot connect to the Docker daemon. Is the docker daemon running on this host? ubuntu@ubuntu-xenial:/project$ -----------------------Some one know how to resolve this ?

Dan (Thu, 30 Mar 2017 13:54:23 GMT):
@PacoEcotrust I think `service docker status` should show you if it is running and also whether ubuntu thinks it is even available on the system.

Dan (Thu, 30 Mar 2017 13:54:23 GMT):
I think `service docker status` should show you if it is running and also whether ubuntu thinks it is even available on the system.

zac (Thu, 30 Mar 2017 13:58:28 GMT):
@jjason I had that issue on a PR yesterday, and it passed when I reran the build. Looks like an intermittent issue.

boydjohnson (Thu, 30 Mar 2017 14:00:44 GMT):
@zac That was on jenkins?

boydjohnson (Thu, 30 Mar 2017 14:01:57 GMT):
@zac Sorry, didn't see who you were talking to.

boydjohnson (Thu, 30 Mar 2017 14:02:11 GMT):
until now.

adamludvik (Thu, 30 Mar 2017 14:22:12 GMT):
Could use a quick review on this PR: https://github.com/hyperledger/sawtooth-core/pull/273

adamludvik (Thu, 30 Mar 2017 14:23:00 GMT):
Makes it so you don't have to build the external debs locally ever...

adamludvik (Thu, 30 Mar 2017 14:53:14 GMT):
Hello @PacoEcotrust, were you able to resolve your issues with Docker?

PacoEcotrust (Thu, 30 Mar 2017 14:53:31 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=XtJM7bRNbdErFwZyy) @Dan Thank you. I got it to work. After several tryings, updates, upgrades, both host machine as vagrant. Was problem about my computer setup

PacoEcotrust (Thu, 30 Mar 2017 14:53:31 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=XtJM7bRNbdErFwZyy) @Dan Thank you. I got it to work. After several tryings, updates, upgrades, both host machine as vagrant. Was problem about my computer setup. I just passed all test

PacoEcotrust (Thu, 30 Mar 2017 16:37:53 GMT):
(zmq/backend/cython/socket.c:5156) File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:7535) zmq.error.ZMQError: Address already in use [16:27:29.536 ERROR cli] Address already in use Traceback (most recent call last): ------------------after run valitador for second time-----

boydjohnson (Thu, 30 Mar 2017 17:43:39 GMT):
@PacoEcotrust That is a known problem with shutting down tcp sockets in the validator and transaction processors. Right now we have been "netstat -tp" to find the process that is holding on to the socket, "kill -9 ". Also if the socket is in a TIME_WAIT state, waiting 30s is the only option that I know of.

jjason (Thu, 30 Mar 2017 17:44:33 GMT):
@PacoEcotrust I have found that, almost always, if you kill the transaction processors before killing the validator this doesn't happen. However your mileage may vary.

anrodon (Thu, 30 Mar 2017 19:57:23 GMT):
Has joined the channel.

anrodon (Thu, 30 Mar 2017 19:57:26 GMT):
http://area51.stackexchange.com/proposals/107923/hyperledger

adamludvik (Thu, 30 Mar 2017 21:30:12 GMT):
Requesting another quick PR review, this one doesn't even change any code, just adds comments: https://github.com/hyperledger/sawtooth-core/pull/281

jjason (Thu, 30 Mar 2017 21:42:48 GMT):
Would appreciate PR review if anyone has time. https://github.com/hyperledger/sawtooth-core/pull/280

jjason (Thu, 30 Mar 2017 21:50:24 GMT):
I pay twice what @adamludvik pays for PR reviews ;)

adamludvik (Thu, 30 Mar 2017 22:27:39 GMT):
Will take a look tonight or tomorrow morning

jjason (Thu, 30 Mar 2017 22:29:02 GMT):
@adamludvik Thx

mhart (Thu, 30 Mar 2017 23:19:37 GMT):
Question on the REST API – are all the `min_position`, `max_position`, `sort`, `fields`, `omit` references accurate?

mhart (Thu, 30 Mar 2017 23:20:10 GMT):
I only ask cause I can't see anything in the source about them – only the docs: http://intelledger.github.io/0.8/rest_api.html

Dan (Thu, 30 Mar 2017 23:27:46 GMT):
@zac ^

mhart (Thu, 30 Mar 2017 23:28:28 GMT):
The underlying pb client does seem to support paging, but it uses different references: https://github.com/hyperledger/sawtooth-core/blob/1b500782b0a920f6784262582ebeb79ed10b9f59/protos/client.proto#L37-L44

boydjohnson (Thu, 30 Mar 2017 23:29:40 GMT):
zac is gone for the day. Probably will answer tomorrow. My understanding was zac was working on that this sprint.

mhart (Thu, 30 Mar 2017 23:29:48 GMT):
Ok, np

mhart (Thu, 30 Mar 2017 23:32:42 GMT):
Ah, `This specification has not yet to be been fully implemented, and at this point should be considered a roadmap rather than a reference for usage. Until fully implemented, many of the endpoints may not work, or may behave differently than documented.`

boydjohnson (Thu, 30 Mar 2017 23:37:30 GMT):
Yeah

pschwarz (Fri, 31 Mar 2017 13:58:27 GMT):
That is actively in development this week!

zac (Fri, 31 Mar 2017 15:24:03 GMT):
@mhart The REST API specs are probably a little north of 50% implemented

zac (Fri, 31 Mar 2017 15:24:36 GMT):
Most of the endpoints are done, most of the query parameters are not

zac (Fri, 31 Mar 2017 15:24:57 GMT):
That list falls into the not category

zac (Fri, 31 Mar 2017 15:25:29 GMT):
Though min/max and count will be merged early next week

zac (Fri, 31 Mar 2017 15:25:36 GMT):
Which will be a big chunk of it

zac (Fri, 31 Mar 2017 15:27:17 GMT):
(and yeah, I merged the underlying validator support for those parameters just last week)

mhart (Fri, 31 Mar 2017 15:33:48 GMT):
@zac ok, great, thanks for the clarification

zac (Fri, 31 Mar 2017 15:34:37 GMT):
sort, fields, and omit are not immediately on the docket, FYI

mhart (Fri, 31 Mar 2017 15:35:53 GMT):
@zac question on the best strategy to "poll" for changes – am I thinking correctly that I can use the `head` param to return all changes from a particular point?

mhart (Fri, 31 Mar 2017 15:36:26 GMT):
ie, if I poll the REST API, saving the head each time I poll, then use that head to query changes since the last poll?

zac (Fri, 31 Mar 2017 15:36:47 GMT):
Hrmmm

zac (Fri, 31 Mar 2017 15:37:24 GMT):
Well, the intention of `head` is to _prevent_ your results from changing

zac (Fri, 31 Mar 2017 15:37:40 GMT):
So you are always getting results from when that block was the chain head

mhart (Fri, 31 Mar 2017 15:37:43 GMT):
right – by changes I mean additions technically

zac (Fri, 31 Mar 2017 15:37:49 GMT):
Yeah

zac (Fri, 31 Mar 2017 15:38:08 GMT):
I hadn't considered using it that way, but sure

mhart (Fri, 31 Mar 2017 15:38:23 GMT):
Just trying to figure out the best way to turn changes into an event feed

zac (Fri, 31 Mar 2017 15:38:29 GMT):
Grab something from some point in the past, then grab the latest, and calculate the diff

mhart (Fri, 31 Mar 2017 15:38:55 GMT):
I could poll the transactions, but of course I'd need to replicate the exact behaviour to replay them

zac (Fri, 31 Mar 2017 15:39:00 GMT):
Might be useful functionality to bake in at some point

mhart (Fri, 31 Mar 2017 15:41:24 GMT):
We want to hook sawtooth lake up to an event stream, probably AWS Kinesis, so the current thinking is poll *something* and publish to kinesis – just trying to figure out what the *something* should be

zac (Fri, 31 Mar 2017 15:43:03 GMT):
You just want to monitor changes in the blockchain?

mhart (Fri, 31 Mar 2017 15:43:08 GMT):
Yep

zac (Fri, 31 Mar 2017 15:44:07 GMT):
Well, your options at the moment are `/batches`, `/blocks`, and `/state`

mhart (Fri, 31 Mar 2017 15:44:11 GMT):
right

zac (Fri, 31 Mar 2017 15:44:25 GMT):
`/blocks` and `/batches` are similar, just different levels of granularity

mhart (Fri, 31 Mar 2017 15:45:00 GMT):
so they just record transactions? not the result of the transactions... right?

jsmitchell (Fri, 31 Mar 2017 15:45:19 GMT):
You want /state

mhart (Fri, 31 Mar 2017 15:45:23 GMT):
so to replay them you'd need to replicate the transaction handler functionality

jsmitchell (Fri, 31 Mar 2017 15:45:38 GMT):
Just enumerate everything under the namespace

zac (Fri, 31 Mar 2017 15:45:58 GMT):
The question is, do you want to look at transactions, or do you want to look at changes in data

mhart (Fri, 31 Mar 2017 15:46:02 GMT):
I mean, I kind of want state... but really I want the append log

jsmitchell (Fri, 31 Mar 2017 15:46:11 GMT):
Not transactions

mhart (Fri, 31 Mar 2017 15:46:17 GMT):
I don't want to know what the current state is – I want to know all the state transformations

zac (Fri, 31 Mar 2017 15:46:36 GMT):
. . . might you be able to generate state from the transactions?

mhart (Fri, 31 Mar 2017 15:46:38 GMT):
Obviously could figure that out from the current state – but that feels like it would be innefficient

pschwarz (Fri, 31 Mar 2017 15:46:45 GMT):
Getting a stream of state deltas is something in the works as well, just not nearly as far along as the rest api

zac (Fri, 31 Mar 2017 15:46:46 GMT):
Would that be the easiest way to calc a diff?

jsmitchell (Fri, 31 Mar 2017 15:46:49 GMT):
@zac: no

jsmitchell (Fri, 31 Mar 2017 15:46:56 GMT):
This is the wrong path

zac (Fri, 31 Mar 2017 15:47:08 GMT):
You would need to feed them through the TPs

jsmitchell (Fri, 31 Mar 2017 15:47:24 GMT):
@zac: that's what the validator is for

zac (Fri, 31 Mar 2017 15:47:31 GMT):
indeed

jsmitchell (Fri, 31 Mar 2017 15:47:55 GMT):
Right now, the best approach is to get /state under a given prefix

mhart (Fri, 31 Mar 2017 15:48:11 GMT):
@jsmitchell but that's gonna be...

mhart (Fri, 31 Mar 2017 15:48:17 GMT):
really innefficient

jsmitchell (Fri, 31 Mar 2017 15:48:25 GMT):
Yeah, clearly

mhart (Fri, 31 Mar 2017 15:48:35 GMT):
unless

jsmitchell (Fri, 31 Mar 2017 15:48:43 GMT):
See what @pschwarz said above

mhart (Fri, 31 Mar 2017 15:48:53 GMT):
there was a way to use the prefix as a range query ;-)

jsmitchell (Fri, 31 Mar 2017 15:48:58 GMT):
We are not exposing the state transition deltas yet

mhart (Fri, 31 Mar 2017 15:49:06 GMT):
and stick the timestamp in the prefix... or something

zac (Fri, 31 Mar 2017 15:49:54 GMT):
Other than the six-character namespace, I think the addresses are not meaningful at the moment

jsmitchell (Fri, 31 Mar 2017 15:50:31 GMT):
Addresses have transaction family specific meanings

jsmitchell (Fri, 31 Mar 2017 15:50:58 GMT):
If the transaction family implementer chooses to give them one

mhart (Fri, 31 Mar 2017 15:51:07 GMT):
When state changes – doesn't that append to a merkle tree? So isn't it possible to just query from a particular point in the merkle tree?

jsmitchell (Fri, 31 Mar 2017 15:51:34 GMT):
The merkle trie is a copy on write structure based on hashes

jsmitchell (Fri, 31 Mar 2017 15:51:56 GMT):
There is no way to identify what nodes are new to that state root

jsmitchell (Fri, 31 Mar 2017 15:52:21 GMT):
We would need to persist that info elsewhere. We don't do that at the moment.

jsmitchell (Fri, 31 Mar 2017 15:53:43 GMT):
I recommend building an adapter that GETs current state under a namespace, calculating the diff against prior state and submitting the change data events to your kinesis stream

mhart (Fri, 31 Mar 2017 15:54:13 GMT):
Ok

jsmitchell (Fri, 31 Mar 2017 15:54:15 GMT):
We will have better support for this built in in the future, but it is some ways off

mhart (Fri, 31 Mar 2017 15:54:42 GMT):
And partial address queries are relatively efficient?

jsmitchell (Fri, 31 Mar 2017 15:55:48 GMT):
Yeah, should be, assuming there aren't millions of nodes in the tree.

jsmitchell (Fri, 31 Mar 2017 15:56:28 GMT):
This should be considered a proof of concept level solution

mhart (Fri, 31 Mar 2017 15:58:26 GMT):
I guess my question is more – is it (a lot) more efficient to make a query with an address of `namespace/entitytype1/` compared with `namespace/`

jsmitchell (Fri, 31 Mar 2017 15:58:44 GMT):
That is a @zac question

mhart (Fri, 31 Mar 2017 15:58:58 GMT):
ie, are the prefixes built in to the tree structure? and if so, how

jsmitchell (Fri, 31 Mar 2017 15:59:12 GMT):
All your queries will be in address format

jsmitchell (Fri, 31 Mar 2017 15:59:50 GMT):
Address format is 70 hex digits for a full leaf address

jsmitchell (Fri, 31 Mar 2017 16:00:23 GMT):
Any even numbered subset of that represents a tree prefix

jsmitchell (Fri, 31 Mar 2017 16:01:05 GMT):
By convention, transaction family namespaces start with 6 hex digits (3 bytes)

mhart (Fri, 31 Mar 2017 16:03:22 GMT):
right – so assume my scheme is: hash(namespace)[0:6] + hash(entitytype) + hash(entityid)

mhart (Fri, 31 Mar 2017 16:03:22 GMT):
right – so assume my scheme is: hash(namespace)[0:6] + hash(entitytype)[0:16] + hash(entityid)[0:48]

jsmitchell (Fri, 31 Mar 2017 16:03:31 GMT):
Yep

mhart (Fri, 31 Mar 2017 16:04:05 GMT):
then a query of hash(namespace)[0:6] + hash(entitytype) should be more efficient than just hash(namespace)[0:6] – especially if other entities that I'm not querying have many many nodes – right?

mhart (Fri, 31 Mar 2017 16:04:05 GMT):
then a query of hash(namespace)[0:6] + hash(entitytype)[0:16] should be more efficient than just hash(namespace)[0:6] – especially if other entities that I'm not querying have many many nodes – right?

jsmitchell (Fri, 31 Mar 2017 16:04:39 GMT):
You could get /state with hash(namespace)[0:6]+hash(entitytype) to get all entities of that type in the current state

jsmitchell (Fri, 31 Mar 2017 16:05:08 GMT):
Yes, it is going to be linearly related to the number of nodes in the depth first traversal from that node

mhart (Fri, 31 Mar 2017 16:05:38 GMT):
right – and the traversal is done in byte order?

mhart (Fri, 31 Mar 2017 16:05:59 GMT):
as in, first byte is first depth, second byte is second depth, etc

jsmitchell (Fri, 31 Mar 2017 16:06:06 GMT):
Yes

mhart (Fri, 31 Mar 2017 16:06:10 GMT):
perfect, thanks

mhart (Fri, 31 Mar 2017 16:07:12 GMT):
a transaction handler can only write to addresses with a single transaction family namespace, right?

jsmitchell (Fri, 31 Mar 2017 16:08:48 GMT):
We don't enforce that presently

jsmitchell (Fri, 31 Mar 2017 16:09:16 GMT):
And when we do enforce it, it will probably be a many to many relationship

jsmitchell (Fri, 31 Mar 2017 16:09:53 GMT):
Meaning multiple transaction families can write to the same namespace and a transaction family can write to multiple namespaces

mhart (Fri, 31 Mar 2017 16:11:14 GMT):
hmmm, ok – trying to figure out if it's better to model top-level entities (aggregate roots) as just separate namespaces – or better to do it a level down below the namespace

jsmitchell (Fri, 31 Mar 2017 16:12:00 GMT):
I'd recommend keeping it in a single namespace

mhart (Fri, 31 Mar 2017 16:12:10 GMT):
right, that's what I'm leaning towards

mhart (Fri, 31 Mar 2017 16:12:17 GMT):
a namespace per app

jsmitchell (Fri, 31 Mar 2017 16:12:19 GMT):
You have plenty of room for uniqueness down there

mhart (Fri, 31 Mar 2017 16:12:20 GMT):
essentially

jsmitchell (Fri, 31 Mar 2017 16:12:24 GMT):
Yes

Dan (Fri, 31 Mar 2017 16:15:32 GMT):
Docs back to defaulting to 0.8. Thanks Ryan.

mhart (Fri, 31 Mar 2017 16:44:46 GMT):
One way I could optimize for reads (although it would amplify writes) would be to write a state change event for each transaction (on top of the existing state changes) – and use an address scheme for these events that facilitate efficient querying. eg, `hash(namespace) + hash(timestamp-in-mins) + hash(eventid)` – then each poll would just query two addresses: `hash(namespace) + hash(current-time-in-mins)` and `hash(namespace) + hash(current-time-in-mins-minus-one-min)` (and all the caveats that go with that: be sure to poll at least once every min, or poll for each min to last time you polled, etc)

jsmitchell (Fri, 31 Mar 2017 17:14:45 GMT):
I wouldn't recommend designing the storage format to work around this short term limitation in events

jsmitchell (Fri, 31 Mar 2017 17:15:55 GMT):
Also, remember if you do _anything_ with timestamps, they need to be in the transaction created by the client. The transaction processor cannot write its own timestamp into the address or the data.

mhart (Fri, 31 Mar 2017 17:17:43 GMT):
@jsmitchell ah, is that for replay purposes? ie, you can't do anything non-deterministic? or is it just time-related?

jsmitchell (Fri, 31 Mar 2017 17:18:06 GMT):
Non determinism and blockchains don't mix

mhart (Fri, 31 Mar 2017 17:19:06 GMT):
hmm, so that includes GUID generation, etc

mhart (Fri, 31 Mar 2017 17:19:42 GMT):
I don't remember seeing that in the docs

jsmitchell (Fri, 31 Mar 2017 17:19:57 GMT):
The publishing validator that determines the merkle state root as a result of the aggregate state transitions from the transactions its including. That root hash is put on the block header. When the block reaches the validator that is verifying it, it does the same work through the transaction processors. If the outcome state hash doesn't match, the block is invalid.

jsmitchell (Fri, 31 Mar 2017 17:21:22 GMT):
Anything non deterministic breaks that process

mhart (Fri, 31 Mar 2017 17:21:40 GMT):
very good to know – thanks! so that means that all addresses are essentially locked right, so that `state.get` doesn't return something different

jsmitchell (Fri, 31 Mar 2017 17:21:49 GMT):
You must include anything like that in the transaction payload as created by the client

jsmitchell (Fri, 31 Mar 2017 17:22:08 GMT):
Yes

jsmitchell (Fri, 31 Mar 2017 17:22:43 GMT):
Your encoding format must be deterministic, the way you serialize payloads must be deterministic, etc.

jsmitchell (Fri, 31 Mar 2017 17:23:21 GMT):
All validators with the same starting state must generate the same ending state

jsmitchell (Fri, 31 Mar 2017 17:23:52 GMT):
We are going to be creating some transaction processor developer focused docs

mhart (Fri, 31 Mar 2017 17:24:18 GMT):
this definitely needs to be in the docs

mhart (Fri, 31 Mar 2017 17:24:19 GMT):
right

jsmitchell (Fri, 31 Mar 2017 17:24:24 GMT):
Heh, yeah

mhart (Fri, 31 Mar 2017 17:24:30 GMT):
even something under http://intelledger.github.io/txn_family_tutorial.html

jsmitchell (Fri, 31 Mar 2017 17:25:44 GMT):
@toddojala @Dan: ^

jsmitchell (Fri, 31 Mar 2017 17:26:08 GMT):
Good confirmation of our discussed need for better TF developer docs

jsmitchell (Fri, 31 Mar 2017 17:26:52 GMT):
@mhart: I need to understand what you mean by addresses being locked

mhart (Fri, 31 Mar 2017 17:27:05 GMT):
I mean – all the addresses in `inputs`

mhart (Fri, 31 Mar 2017 17:27:12 GMT):
all the addresses you can write

mhart (Fri, 31 Mar 2017 17:27:13 GMT):
to

mhart (Fri, 31 Mar 2017 17:27:16 GMT):
read from

mhart (Fri, 31 Mar 2017 17:27:40 GMT):
because the transaction handler will make decisions based on the results of reading those addresses

mhart (Fri, 31 Mar 2017 17:27:44 GMT):
so they can't change at all

jsmitchell (Fri, 31 Mar 2017 17:27:57 GMT):
That is for parallelism, not for a point in time state

mhart (Fri, 31 Mar 2017 17:28:04 GMT):
from when the first transaction handler reads them – to when the final confirmation happens

jsmitchell (Fri, 31 Mar 2017 17:28:12 GMT):
They can in fact change

mhart (Fri, 31 Mar 2017 17:28:24 GMT):
so... that's non-deterministic?

jsmitchell (Fri, 31 Mar 2017 17:28:31 GMT):
No

mhart (Fri, 31 Mar 2017 17:28:37 GMT):
no transaction isolation

jsmitchell (Fri, 31 Mar 2017 17:28:52 GMT):
This is difficult to express over text :)

mhart (Fri, 31 Mar 2017 17:28:54 GMT):
or am i confused about where the state is actually stored?

jsmitchell (Fri, 31 Mar 2017 17:29:13 GMT):
The block publisher is determining to ordering of transactions

jsmitchell (Fri, 31 Mar 2017 17:29:20 GMT):
The ordering

jsmitchell (Fri, 31 Mar 2017 17:29:31 GMT):
That process does not have to be deterministic

jsmitchell (Fri, 31 Mar 2017 17:29:36 GMT):
It is done once

mhart (Fri, 31 Mar 2017 17:29:53 GMT):
is each state local to each transaction handler?

jsmitchell (Fri, 31 Mar 2017 17:29:55 GMT):
But the validation of that block must be identical every time

jsmitchell (Fri, 31 Mar 2017 17:30:01 GMT):
No, state is global

jsmitchell (Fri, 31 Mar 2017 17:30:27 GMT):
So, the ordering of transactions is based on what batches are in the publishers pending queue

jsmitchell (Fri, 31 Mar 2017 17:31:31 GMT):
But when the block data structure is written out, it includes an explicit order, along with the rules around isolation dependencies for inputs and outputs - this gives a repeatable order of evaluation for transactions

jsmitchell (Fri, 31 Mar 2017 17:32:15 GMT):
But, txn 1 in the block can write to address A, and txn 2 can read from address A, and it will get the result placed there by txn 1

mhart (Fri, 31 Mar 2017 17:32:22 GMT):
but if those transactions change behavior depending on state...

mhart (Fri, 31 Mar 2017 17:32:26 GMT):
how do you replay them?

mhart (Fri, 31 Mar 2017 17:32:36 GMT):
if the state has changed in the meantime?

jsmitchell (Fri, 31 Mar 2017 17:32:49 GMT):
You are starting with the same base state

jsmitchell (Fri, 31 Mar 2017 17:32:57 GMT):
Which is the ending state of the prior block

mhart (Fri, 31 Mar 2017 17:33:00 GMT):
ok

jsmitchell (Fri, 31 Mar 2017 17:33:14 GMT):
And you are playing back a deterministic set of operations

mhart (Fri, 31 Mar 2017 17:33:21 GMT):
so it's not exactly global – or, not in the way I was thinking

jsmitchell (Fri, 31 Mar 2017 17:33:30 GMT):
It's 100% global

jsmitchell (Fri, 31 Mar 2017 17:33:36 GMT):
Everyone has a copy of it

jsmitchell (Fri, 31 Mar 2017 17:33:52 GMT):
And it's monolithic

mhart (Fri, 31 Mar 2017 17:33:54 GMT):
you know what I mean – the state that a transaction processor sees

mhart (Fri, 31 Mar 2017 17:34:03 GMT):
ie, a `state.get`

jsmitchell (Fri, 31 Mar 2017 17:34:11 GMT):
State that a transaction processor sees is super limited

jsmitchell (Fri, 31 Mar 2017 17:34:27 GMT):
It's defined by its inputs list

jsmitchell (Fri, 31 Mar 2017 17:34:44 GMT):
And it doesn't get to choose _when_ it is executed

jsmitchell (Fri, 31 Mar 2017 17:34:53 GMT):
The block publisher is making that choice

mhart (Fri, 31 Mar 2017 17:35:33 GMT):
that's what I meant by is state local to the transaction handler

jsmitchell (Fri, 31 Mar 2017 17:36:02 GMT):
But it can define any inputs it wants including *

jsmitchell (Fri, 31 Mar 2017 17:37:22 GMT):
It is entirely possible to create a transaction which you think will be successful when it's created but fails when a publisher attempts to include it in a block because the state assumptions have changed

mhart (Fri, 31 Mar 2017 17:38:05 GMT):
Right – but this statement – "When the block reaches the validator that is verifying it, it does the same work through the transaction processors"

jsmitchell (Fri, 31 Mar 2017 17:38:13 GMT):
For example, I have $10. I create a txn to pay you $10, and a txn to pay Dan $10. It would be a pretty bad deal if the system allowed both of those to succeed

mhart (Fri, 31 Mar 2017 17:39:14 GMT):
each transaction processor can't be setting the same global state every time they do `state.set` – they must be operating on their own copy of state...?

jsmitchell (Fri, 31 Mar 2017 17:39:16 GMT):
Yes, the publisher's creation of a block data structure is the codifying of a set of operations that must result in an identical state hash wherever they are processed

jsmitchell (Fri, 31 Mar 2017 17:39:34 GMT):
Oh, yes. It's their own copy

jsmitchell (Fri, 31 Mar 2017 17:39:50 GMT):
But we can _prove_ that it's the same

mhart (Fri, 31 Mar 2017 17:39:53 GMT):
right

mhart (Fri, 31 Mar 2017 17:48:55 GMT):
btw, I hate to be "that guy", but just to clarify – sawtooth lake basically shares nothing with fabric, right? Like, "hyperledger" is just some loose umbrella org

rbuysse (Fri, 31 Mar 2017 17:50:10 GMT):
correct.

mhart (Fri, 31 Mar 2017 17:52:36 GMT):
cool. pity about the name. hyperledger sounds like an actual... you know... ledger

jsmitchell (Fri, 31 Mar 2017 17:54:39 GMT):
Haha

jsmitchell (Fri, 31 Mar 2017 17:56:47 GMT):
The projects have different ideas about what is important. I believe the goal remains to crystallize those ideas and cross-pollinate.

mhart (Fri, 31 Mar 2017 18:23:39 GMT):
yeah, but there's definitely some confusion out there

mhart (Fri, 31 Mar 2017 18:23:53 GMT):
even wikipedia describes hyperledger as "a platform"

mhart (Fri, 31 Mar 2017 18:24:13 GMT):
and it's often referred to as "the hyperledger project"

mhart (Fri, 31 Mar 2017 18:24:41 GMT):
which makes as much sense as "the linux foundation project"

Dan (Fri, 31 Mar 2017 18:31:16 GMT):
There's no lack of consternation about that here.:unamused:

Dan (Fri, 31 Mar 2017 18:31:16 GMT):
There's no lack of consternation about that here. :unamused:

mhart (Fri, 31 Mar 2017 18:31:50 GMT):
yeah, fait enough

mhart (Fri, 31 Mar 2017 18:31:50 GMT):
yeah, fair enough

mhart (Fri, 31 Mar 2017 18:34:08 GMT):
I guess it's only a matter of time before ppl start saying things like "it's built on hyperledger"

mhart (Fri, 31 Mar 2017 18:34:11 GMT):
ah, naming things

Dan (Fri, 31 Mar 2017 18:43:27 GMT):
Yeah that's already an issue :rolling_eyes:

choas (Sat, 01 Apr 2017 19:52:40 GMT):
Has joined the channel.

mhart (Sun, 02 Apr 2017 18:19:22 GMT):
what's the best option for running sawtooth lake on SGX right now? Is it available on AWS, etc?

kelly_ (Sun, 02 Apr 2017 21:28:12 GMT):
SGX is not yet available on server/in cloud deployments

kelly_ (Sun, 02 Apr 2017 21:29:16 GMT):
all skylake and beyond desktop/laptop processors include sgx capability (intel core 6th gen)

kelly_ (Sun, 02 Apr 2017 21:29:36 GMT):
not all bios support it though (e.g. i don't believe it is available on mac today)

kelly_ (Sun, 02 Apr 2017 21:30:26 GMT):
here is a list of some systems that it is available on - https://github.com/ayeks/SGX-hardware

kelly_ (Sun, 02 Apr 2017 21:30:39 GMT):
we have HP and Intel NUC machines that we do most development on

mhart (Mon, 03 Apr 2017 00:16:18 GMT):
Thanks @kelly_

Dan (Mon, 03 Apr 2017 19:08:43 GMT):
in 0.8 do we have a notion of a validator ID as distinct from a validator's public key? Are we perpetuating the validator name field from 0.6?

jsmitchell (Mon, 03 Apr 2017 19:13:57 GMT):
there ain't no such

jsmitchell (Mon, 03 Apr 2017 19:14:09 GMT):
not yet anyhow

jjason (Mon, 03 Apr 2017 19:39:04 GMT):
There is no validator name.

agunde (Tue, 04 Apr 2017 14:02:00 GMT):
I am working on defining the configurable logging implementation for 0.8 and had some preference questions. What should be the default logging settings? In 0.7, the logs were put into the default log directory and two files were created, a debug.log for all log statements and error.log for any error log statements. Do we want to have the same set up for 0.8?

toddojala (Tue, 04 Apr 2017 14:39:00 GMT):
That sounds reasonable to me, @agunde, but I'm not a logging expert.

agunde (Tue, 04 Apr 2017 16:27:57 GMT):
I would assume we want to have a configurable logging option for the python transaction processors as well as the validator? Both the transaction processors and the validators do not have unique names. In 0.7 the logging files mentioned above would look like val_name-debug.log and val_name-error.log. Does anyone have a suggestion or preference on how the log files should be distinguished from each other?

jjason (Tue, 04 Apr 2017 16:34:19 GMT):
For the validators in the validator registry (even though we don't seem to care about name any more), I used validator-[first 8 of public key string]

jjason (Tue, 04 Apr 2017 16:34:52 GMT):
It would be nice if we standardized on something ;)

Dan (Tue, 04 Apr 2017 16:37:58 GMT):
Where do validators running via docker stick their logs?

agunde (Tue, 04 Apr 2017 16:45:14 GMT):
According to a convo I just had with @boydjohnson, they would go in the Linux defaults within the docker container (var/log/sawtooth) @Dan

PacoEcotrust (Tue, 04 Apr 2017 17:07:47 GMT):
The ubuntu packages generation in v8 is the same than v7? I want to develop into ubuntu

adamludvik (Tue, 04 Apr 2017 17:47:56 GMT):
Hello Paco, if you would like to develop a transaction family in 0.8, we have a new package repository that may make your life easier. This demo shows how to use it to install sawtooth on a new Ubuntu 16.04 box using vagrant: https://asciinema.org/a/109608 If you already have a running Ubuntu 16.04 box, you should be able to skip passed the vagrant up portion.

adamludvik (Tue, 04 Apr 2017 17:47:56 GMT):
Hello @PacoEcotrust , if you would like to develop a transaction family in 0.8, we have a new package repository that may make your life easier. This demo shows how to use it to install sawtooth on a new Ubuntu 16.04 box using vagrant: https://asciinema.org/a/109608 If you already have a running Ubuntu 16.04 box, you should be able to skip passed the vagrant up portion.

kellyo (Tue, 04 Apr 2017 18:39:16 GMT):
Has joined the channel.

PacoEcotrust (Wed, 05 Apr 2017 15:31:32 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=CcywALdmbSjEt7hCy) @adamludvik

PacoEcotrust (Wed, 05 Apr 2017 15:31:32 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=CcywALdmbSjEt7hCy) Thanks, now I have problem with versions: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Preparing to unpack .../python3-bitcoin_1.1.42-1_all.deb ... Unpacking python3-bitcoin (1.1.42-1) ... dpkg: error processing archive /var/cache/apt/archives/python3-bitcoin_1.1.42-1_all.deb (--unpack): trying to overwrite '/usr/bin/pybtctool', which is also in package python-pybitcointools 1.1.15-1

adamludvik (Wed, 05 Apr 2017 15:37:30 GMT):
Are you working in a clean environment, or are you using a pre-existing environment where you already had bitcoin installed?

pschwarz (Wed, 05 Apr 2017 15:38:53 GMT):
Hey all, with the merger of PR #296 (https://github.com/hyperledger/sawtooth-core/pull/296) you no longer need to do a run `sawtooth admin genesis` after restarts. You block chain and state will be preserved across restarts. If you want to start from a clean state, delete the contents of your data directory (if you're using Vagrant, this is `rm $SAWTOOTH_HOME/data/*`)

PacoEcotrust (Wed, 05 Apr 2017 15:40:15 GMT):
[ ](https://chat.hyperledger.org/channel. /sawtoothlake?msg=Q5EwMzqAWPfxFGQX5) @adamludvik -----------Yes, I had installed the sawtooth v7

PacoEcotrust (Wed, 05 Apr 2017 15:40:51 GMT):
[ ](https://chat.hyperledger.org/channel----Yes, I had installed the sawtooth v7 /sawtoothlake?msg=Q5EwMzqAWPfxFGQX5) @adamludvik

PacoEcotrust (Wed, 05 Apr 2017 15:41:20 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=Q5EwMzqAWPfxFGQX5) @adamludvik ---Yes, I had installed the sawtooth v7

PacoEcotrust (Wed, 05 Apr 2017 15:41:20 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=Q5EwMzqAWPfxFGQX5) @adamludvik ---Yes, I had installed the sawtooth v7 into ubuntu 16.04

PacoEcotrust (Wed, 05 Apr 2017 15:41:20 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=Q5EwMzqAWPfxFGQX5) @adamludvik ---Yes, I had installed and setup the lieclipse with the sawtooth v7 into ubuntu 16.04

PacoEcotrust (Wed, 05 Apr 2017 15:41:20 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=Q5EwMzqAWPfxFGQX5) @adamludvik ---Yes, I had installed and setup the liclipse with the sawtooth v7 into ubuntu 16.04

PacoEcotrust (Wed, 05 Apr 2017 15:41:20 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=Q5EwMzqAWPfxFGQX5) @adamludvik ---Yes, I had installed and setup the liclipse with the sawtooth v7 into ubuntu 16.04. So, I need clean system, do I?

adamludvik (Wed, 05 Apr 2017 15:45:43 GMT):
A clean system would be easiest, yes. If you are using vagrant, you should be able to do `vagrant destroy` to remove your existing box, and then follow along with the demo I sent earlier to create a new one.

mhart (Wed, 05 Apr 2017 21:23:19 GMT):
@pschwarz nice 👍

JeetMukesh (Thu, 06 Apr 2017 03:33:54 GMT):
Has joined the channel.

PacoEcotrust (Thu, 06 Apr 2017 09:59:04 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=F3HeXevnn82YpEpnx) @adamludvik , thanks was perfect, not a warning. Now I want to mount sawtooth project with liClipse into vagrant box that I think is better than into directly in host ubuntu. I had sawtooth project in host ubuntu and to change of version (v7 to v8) I had a lot of version conflicts so I had to reinstall new Ubuntu.

adamludvik (Thu, 06 Apr 2017 17:58:08 GMT):
@PacoEcotrust If you are just trying to write a transaction family, you shouldn't need to mount the project directory into ubuntu. You should just be able to do `import sawtooth_sdk` in your python3 files and go from there.

Dan (Fri, 07 Apr 2017 02:39:21 GMT):
If anyone is looking for low hanging fruit to contribute getting rid of this part of the docs would be helpful: http://intelledger.github.io/sysadmin_guide/network_configuration.html#validators https://github.com/hyperledger/sawtooth-core/blob/master/docs/source/sysadmin_guide/network_configuration.rst

Dan (Fri, 07 Apr 2017 02:40:35 GMT):
More low hanging fruit: http://intelledger.github.io/sawtooth_developers_guide/architecture_overview.html Delete me: https://github.com/hyperledger/sawtooth-core/blob/master/docs/source/sawtooth_developers_guide/architecture_overview.rst

zzying (Fri, 07 Apr 2017 03:02:47 GMT):
Has joined the channel.

Dan (Fri, 07 Apr 2017 13:44:32 GMT):
In creating the a Transaction Processor (server side piece of a _Transaction Family_) there's at least one required method, `apply`. I'm digging through some old notes and there's a reference to `serialize_for_display`. Do we still want that to be a required method? Would that just be on the _client_ (transactor) side?

haridhakshini (Sat, 08 Apr 2017 06:50:14 GMT):
Has joined the channel.

ray (Sat, 08 Apr 2017 16:18:43 GMT):
Has left the channel.

haridhakshini (Sat, 08 Apr 2017 19:38:08 GMT):
@Dan I am trying to get rid of the two parts of the docs as posted by you previously. I have submitted a pull request and it errored out with the following error:"FAILED: User 'Haridhakshini' not whitelisted". https://build.sawtooth.me/job/Sawtooth-Hyperledger/job/sawtooth-core/job/PR-314/1/console https://github.com/hyperledger/sawtooth-core/pull/314 I am not sure if I am encountering some permissions issue. Please let me know your thoughts. I also apologise if this is not the right way to do a Pull Request as this is my first time to do so. Thanks!

Dan (Sat, 08 Apr 2017 19:46:10 GMT):
Hi! The first time doing a PR is tricky.

Dan (Sat, 08 Apr 2017 19:46:28 GMT):
You want to fork the repo.

Dan (Sat, 08 Apr 2017 19:46:57 GMT):
Then clone your fork on your own system.

Dan (Sat, 08 Apr 2017 19:47:26 GMT):
Then make a branch (this helps isolate your changes)

Dan (Sat, 08 Apr 2017 19:49:00 GMT):
Commit (with -s flag fit sign off) then push that branch with your commit back up to _your_ fork on github.

Dan (Sat, 08 Apr 2017 19:50:05 GMT):
The github website will pop up a little link for you asking if you want to issue a PR.

Dan (Sat, 08 Apr 2017 19:51:13 GMT):
Then two of the maintainers will review it and either merge it for you or ask for any corrections.

pschwarz (Sat, 08 Apr 2017 20:01:03 GMT):
I think that's what that method was for originally. Since we don't have any mandate about format or reuse between clients and processors, that sort of thing is up to the transaction family developer. We should certainly drop the reference.

pschwarz (Sat, 08 Apr 2017 20:01:31 GMT):
@dan :point_up: re the `serialize_for_display`

Dan (Sat, 08 Apr 2017 22:09:01 GMT):
@haridhakshini my bad. Looks like you submitted everything correctly and it is a permissioning issue like you suggested.

haridhakshini (Sun, 09 Apr 2017 10:22:05 GMT):
@Dan Thanks for the reply!

AlexejGerstmaier (Sun, 09 Apr 2017 22:11:44 GMT):
Has joined the channel.

AlexejGerstmaier (Sun, 09 Apr 2017 22:14:31 GMT):
Hey guys, I just found out about the Hyperledger project and researched a bit. One quick question for my understanding: Sawtoothlake seems to rely on a Trusted Execution Environment to be secure. Intel SGX is mentioned a lot. Does ARM Trustzone work too?

Dan (Mon, 10 Apr 2017 02:01:46 GMT):
Hi @AlexejGerstmaier we need to update that intro. It's a little too cautious. Sawtooth has pluggable consensus. The one we talk about most is PoET. PoET was designed to use trusted execution. Without trusted execution you lose protections against bad actors in the network (byzantine faults), but you keep crash fault tolerance (CFT). CFT is the same level of protection of several of the other enterprise focused blockchains.

Dan (Mon, 10 Apr 2017 02:04:12 GMT):
There's no SGX required to run the system and no one has yet open sourced a trusted execution implementation of PoET with SGX or Trustzone.

Dan (Mon, 10 Apr 2017 02:05:12 GMT):
@mfford I should probably update that intro this sprint.

jsmitchell (Mon, 10 Apr 2017 14:24:55 GMT):
A recent PR to master introduces a required command-line argument called `--public-uri` which specifies the reachable endpoint for the validator's network endpoint. Example usage: `validator --network-endpoint tcp://0.0.0.0:8801 --public-uri tcp://{public ip or hostname}:8801`

drozd (Mon, 10 Apr 2017 14:33:27 GMT):
@jsmitchell is it always required?

drozd (Mon, 10 Apr 2017 14:34:26 GMT):
I mean, always always?

jsmitchell (Mon, 10 Apr 2017 14:34:43 GMT):
Always, yes

jsmitchell (Mon, 10 Apr 2017 14:35:51 GMT):
It's sent as part of the connect message to manage validator network identity

toddojala (Mon, 10 Apr 2017 15:05:30 GMT):
@jsmitchell So this needs to be documented, right? ;)

jsmitchell (Mon, 10 Apr 2017 20:03:43 GMT):
You bet!

toddojala (Mon, 10 Apr 2017 21:59:52 GMT):
@jsmitchell We'll update the docs ASAP, "out-of-band" so to speak.

Lin-YiTang (Tue, 11 Apr 2017 21:14:53 GMT):
Has joined the channel.

Dan (Tue, 11 Apr 2017 23:19:44 GMT):
Getting this error from the build system ```docker: Error response from daemon: repository sawtooth-build-python not found: does not exist or no pull access. See 'docker run --help'.```

boydjohnson (Tue, 11 Apr 2017 23:28:30 GMT):
PR # ?

boydjohnson (Tue, 11 Apr 2017 23:31:24 GMT):
I know that @adamludvik had a PR go in that changed some of the Docker image names. I would guess that is the problem. My guess is that rebasing the PR on master will fix the issue.

Dan (Tue, 11 Apr 2017 23:32:01 GMT):
https://build.sawtooth.me/job/Sawtooth-Hyperledger/job/sawtooth-core/job/PR-314/11/console

adamludvik (Wed, 12 Apr 2017 00:05:26 GMT):
That console log is very weird. I will investigate in the morning

ReedCao (Wed, 12 Apr 2017 08:04:11 GMT):
Has joined the channel.

Dan (Wed, 12 Apr 2017 08:53:01 GMT):
Thanks @adamludvik. I rebased the PR for the submitter. If I can get a couple thumbs up we can merge these baby seals to death. Now as PR-324. https://github.com/hyperledger/sawtooth-core/pull/324

pancake (Wed, 12 Apr 2017 11:11:42 GMT):
Has joined the channel.

pancake (Wed, 12 Apr 2017 11:23:54 GMT):
Hi all, i try to modify the the content of a transaction or block, but the validator seem can't detect the illegal change, why?

pancake (Wed, 12 Apr 2017 11:24:33 GMT):
I start only one validator.

jsmitchell (Wed, 12 Apr 2017 12:40:31 GMT):
@pancake Can you provide some details? You might be able to trick yourself (by modifying things after signature verification occurs etc, but you should not be able to trick others.

adamludvik (Wed, 12 Apr 2017 13:53:23 GMT):
@dan So here is what happened: 1. PR 312 and 314 go up, both bringing a copy of the repo. Specifically, they bring their own versions of build_all and the Jenkinsfile 2. My PR (312), which changes build_all and the Jenkinsfile, gets merged. The Jenkinsfile and build_all are no longer the same on master and PR 314. 3. PR 314 is re-run, but doesn't have permission to run its own copy of the Jenkinsfile because of our security rules. It defaults to running what is on master, which tells it to run _its copy_ of build_all in installed mode. 4. Installed mode changed substantially with 312, including building sawtooth-build-python, the missing docker image.

adamludvik (Wed, 12 Apr 2017 13:53:23 GMT):
@Dan So here is what happened: 1. PR 312 and 314 go up, both bringing a copy of the repo. Specifically, they bring their own versions of build_all and the Jenkinsfile 2. My PR (312), which changes build_all and the Jenkinsfile, gets merged. The Jenkinsfile and build_all are no longer the same on master and PR 314. 3. PR 314 is re-run, but doesn't have permission to run its own copy of the Jenkinsfile because of our security rules. It defaults to running what is on master, which tells it to run _its copy_ of build_all in installed mode. 4. Installed mode changed substantially with 312, including building sawtooth-build-python, the missing docker image.

adamludvik (Wed, 12 Apr 2017 13:53:58 GMT):
Rebasing is the solution, which it looks like you did.

adamludvik (Wed, 12 Apr 2017 13:56:48 GMT):
In other news, our security rules are working correctly.

Dan (Wed, 12 Apr 2017 14:17:03 GMT):
thanks for the details @adamludvik

Dan (Wed, 12 Apr 2017 14:17:57 GMT):
this baby seal was just surprisingly resilient. if you want to tap it on the head with a green checkmark I will deliver the coup de grace.

Dan (Wed, 12 Apr 2017 14:20:56 GMT):
it is done.

Dan (Wed, 12 Apr 2017 14:21:37 GMT):
hands won't come clean:flushed:

toddojala (Wed, 12 Apr 2017 14:36:41 GMT):
the baby seal metaphor is kind of weird. People love baby seals, and don't want them tapped on the head!

toddojala (Wed, 12 Apr 2017 14:36:45 GMT):
:)

adamludvik (Wed, 12 Apr 2017 14:57:29 GMT):
I am trying to resolve the "binding to 0.0.0.0" security issue. My first thought was to use docker's embedded DNS to replace this, but it doesn't look like ZMQ (and maybe binding in general?) lets you bind to an address by referencing its DNS name. The docs (http://api.zeromq.org/2-1:zmq-tcp) indicate you can do `tcp://*:40000`, `tcp://{numeric_ip}:4000`, or `tcp://{interface}:40000`. I am going to set the default to `tcp://127.0.0.1:40000`, but I have a couple questions: 1. Is binding to `tcp://*:40000` the same as 0.0.0.0? (I think it must be) 2. Is it secure to bind to `tcp://eth0:40000`?

jsmitchell (Wed, 12 Apr 2017 14:58:49 GMT):
I don't think binding to local host is going to be valid for docker. You will need to bind to the public interface for it to be reachable.

adamludvik (Wed, 12 Apr 2017 14:59:05 GMT):
I know that localhost won't work in docker.

jsmitchell (Wed, 12 Apr 2017 14:59:05 GMT):
* should be the same as 0.0.0.0

adamludvik (Wed, 12 Apr 2017 14:59:23 GMT):
But I think it should be the default argument for the cli.

adamludvik (Wed, 12 Apr 2017 14:59:39 GMT):
I am trying to decide what to change the argument to in docker.

jsmitchell (Wed, 12 Apr 2017 14:59:40 GMT):
I don't see any issue binding to eth0

jsmitchell (Wed, 12 Apr 2017 14:59:59 GMT):
@adamludvik: yep, sounds good for the default

jsmitchell (Wed, 12 Apr 2017 15:00:43 GMT):
Although it might cause confusion why a user's validator isn't working by default

adamludvik (Wed, 12 Apr 2017 15:01:39 GMT):
If I understand the issue then, it is with allowing all interfaces (not just eth0) when you do tcp://*:40000?

jsmitchell (Wed, 12 Apr 2017 15:03:46 GMT):
Yeah, it will listen on all interfaces on port 40000

adamludvik (Wed, 12 Apr 2017 15:05:29 GMT):
It the default is 127.0.0.1, you can still start a validator and connect TPs and the REST API to it, so when you say "working by default", you mean connecting to a larger, existing network right?

jsmitchell (Wed, 12 Apr 2017 15:05:45 GMT):
Yes

adamludvik (Wed, 12 Apr 2017 15:06:31 GMT):
Gotcha. Thanks

jsmitchell (Wed, 12 Apr 2017 15:36:32 GMT):
@adamludvik: we probably want to be able to specify a list of bind endpoints

jsmitchell (Wed, 12 Apr 2017 15:37:11 GMT):
you should be able to do `for endpoint in endpoints: bind(endpoint`

jsmitchell (Wed, 12 Apr 2017 15:37:11 GMT):
you should be able to do `for endpoint in endpoints: bind(endpoint)`

adamludvik (Wed, 12 Apr 2017 15:37:42 GMT):
As in `--component-endpoint` takes a list instead of one arg?

jsmitchell (Wed, 12 Apr 2017 15:38:10 GMT):
yes, and --network-endpoint

agunde (Wed, 12 Apr 2017 15:53:57 GMT):
Some new configurable logging capabilities have been added to Sawtooth. Documentation can be found here http://intelledger.github.io/sysadmin_guide/log_configuration.html

rbuysse (Wed, 12 Apr 2017 19:47:06 GMT):
a build of master just broke

rbuysse (Wed, 12 Apr 2017 19:47:27 GMT):
integration_test_1 | sawtooth_cli.exceptions.CliException: (503): The validator is not yet ready to be queried

rbuysse (Wed, 12 Apr 2017 19:48:00 GMT):
during test_poet_smoke

pschwarz (Wed, 12 Apr 2017 19:49:02 GMT):
I was noticing this locally - it seems periodic

pschwarz (Wed, 12 Apr 2017 19:49:12 GMT):
I'm up-to-date with master, as well

jjason (Wed, 12 Apr 2017 20:08:24 GMT):
I just encountered with some local testing as well.

jsmitchell (Wed, 12 Apr 2017 20:15:57 GMT):
https://build.sawtooth.me/job/Sawtooth-Hyperledger/job/sawtooth-core/job/master/153/console

jsmitchell (Wed, 12 Apr 2017 20:16:09 GMT):
@zac: any clues?

zac (Wed, 12 Apr 2017 20:26:27 GMT):
That indicates that there is no genesis block

zac (Wed, 12 Apr 2017 20:26:59 GMT):
Or more specifically, no `chain_head_id` in the block store

zac (Wed, 12 Apr 2017 20:27:26 GMT):
(which is generally because you haven't created a genesis block)

zac (Wed, 12 Apr 2017 20:29:21 GMT):
Why that would happen in this case, I'm not sure

jsmitchell (Wed, 12 Apr 2017 20:29:51 GMT):
Can someone who is able to reproduce modify the failing test to add -vv to the validator args?

drozd (Wed, 12 Apr 2017 20:39:34 GMT):
locally I just got it for intkey javascript

jjason (Wed, 12 Apr 2017 21:11:34 GMT):
Is this error being emitted by the what is supposed to be the "genesis" validator? Is there, per chance, some race condition on startup where during the tests a non-genesis node is coming up and ready before the genesis node is ready to transfer the genesis block to it?

jjason (Wed, 12 Apr 2017 21:11:34 GMT):
Is this error being emitted by what is supposed to be the "genesis" validator? Is there, per chance, some race condition on startup where during the tests a non-genesis node is coming up and ready before the genesis node is ready to transfer the genesis block to it?

jjason (Wed, 12 Apr 2017 21:13:06 GMT):
Ran into a new error: ```

jjason (Wed, 12 Apr 2017 21:13:06 GMT):
Ran into a new error: ``` validator-0_1 | [20:36:39.473 ERROR chain] 'NoneType' object has no attribute 'encode' validator-0_1 | Traceback (most recent call last): validator-0_1 | File "/project/sawtooth-core/validator/sawtooth_validator/journal/chain.py", line 390, in run validator-0_1 | commit_new_chain = self._test_commit_new_chain() validator-0_1 | File "/project/sawtooth-core/validator/sawtooth_validator/journal/chain.py", line 323, in _test_commit_new_chain validator-0_1 | return fork_resolver.compare_forks(self._chain_head, self._new_block) validator-0_1 | File "/project/sawtooth-core/consensus/poet/core/sawtooth_poet/poet_consensus/poet_fork_resolver.py", line 200, in compare_forks validator-0_1 | block_cache=self._block_cache)) validator-0_1 | File "/project/sawtooth-core/consensus/poet/core/sawtooth_poet/poet_consensus/utils.py", line 233, in create_next_validator_state validator-0_1 | validator_info.transaction_id).block_num validator-0_1 | File "/project/sawtooth-core/validator/sawtooth_validator/journal/block_store.py", line 174, in get_block_by_transaction_id validator-0_1 | return self.__getitem__(self._block_store[txn_id]) validator-0_1 | File "/project/sawtooth-core/validator/sawtooth_validator/journal/block_store.py", line 44, in __getitem__ validator-0_1 | stored_block = self._block_store[key] validator-0_1 | File "/project/sawtooth-core/validator/sawtooth_validator/database/database.py", line 30, in __getitem__ validator-0_1 | return self.get(key) validator-0_1 | File "/project/sawtooth-core/validator/sawtooth_validator/database/lmdb_nolock_database.py", line 72, in get validator-0_1 | packed = txn.get(key.encode()) validator-0_1 | AttributeError: 'NoneType' object has no attribute 'encode' ```

jjason (Wed, 12 Apr 2017 21:14:23 GMT):
The problem is that during fork resolution, when the PoET fork resolver is trying to look up an enclosing block for a validator registry transaction (i.e., calling `get_block_by_transaction_id`), the transaction ID is `None` in the validator registry transaction.

adamludvik (Thu, 13 Apr 2017 13:32:11 GMT):
Do we need .service files for tp_intkey, rest_api, etc so we can start them with systemd? Or is there another plan for running them as services?

jsmitchell (Thu, 13 Apr 2017 18:01:44 GMT):
@jjason: can you open a DEBUG story for that if you're not on the case?

jjason (Thu, 13 Apr 2017 18:02:09 GMT):
Can do.

boydjohnson (Thu, 13 Apr 2017 18:02:30 GMT):
Do we have DEBUG stories? I thought we were putting DEFECT in the title.

jjason (Thu, 13 Apr 2017 18:02:34 GMT):
Currently wrapping up the PoET zTest story.

jjason (Thu, 13 Apr 2017 18:02:45 GMT):
I interpreted it to be DEFECT.

jjason (Thu, 13 Apr 2017 18:02:46 GMT):
;)

boydjohnson (Thu, 13 Apr 2017 18:02:59 GMT):
:)

jjason (Thu, 13 Apr 2017 18:12:38 GMT):
Story created and put into next sprint. Unassigned.

jjason (Thu, 13 Apr 2017 18:12:46 GMT):
@jsmitchell ^^^^^^^

jsmitchell (Thu, 13 Apr 2017 18:16:10 GMT):
👍🏻

jsmitchell (Thu, 13 Apr 2017 18:25:54 GMT):
Look, maybe I didn't say every little tiny syllable, no. But basically I said them, yeah.

boydjohnson (Thu, 13 Apr 2017 18:30:37 GMT):
:laughing:

jjason (Thu, 13 Apr 2017 18:34:05 GMT):
I have two boys....I am used to reading between the lines.

jjason (Thu, 13 Apr 2017 18:34:36 GMT):
Misdirection and outright omission are par for the course.

jaredcorduan (Thu, 13 Apr 2017 19:22:33 GMT):
Has joined the channel.

jjason (Thu, 13 Apr 2017 21:23:08 GMT):
@jsmitchell @boydjohnson @pschwarz Do we need to create a DEFECT story to investigate this? Seems like we are still getting failures with validators not ready to be queried. Is that a failure on the validator side or the PoET smoke test side? https://build.sawtooth.me/job/Sawtooth-Hyperledger/job/sawtooth-core/job/PR-331/1/console

boydjohnson (Thu, 13 Apr 2017 21:23:53 GMT):
It could also be option 3: the rest api.

boydjohnson (Thu, 13 Apr 2017 21:25:30 GMT):
@rbuysse You had mentioned this vis-a-vis the failure on master yesterday. Do we have a story for it?

rbuysse (Thu, 13 Apr 2017 21:25:57 GMT):
we do not have story for it yes.

rbuysse (Thu, 13 Apr 2017 21:25:57 GMT):
we do not have story for it yet.

rbuysse (Thu, 13 Apr 2017 21:26:38 GMT):
we've had 5 clean builds since.

jjason (Thu, 13 Apr 2017 21:45:45 GMT):
Okay. We have a story in next sprint. Unassigned.

drozd (Thu, 13 Apr 2017 22:17:49 GMT):
``` INFO:__main__:Bringing up with ['docker-compose', '-p', 'latest', '-f', '/project/sawtooth-core/integration/sawtooth_integration/docker/intkey-python-smoke.yaml', 'up', '--abort-on-container-exit'] Creating network "latest_default" with the default driver Creating latest_validator_1 Creating latest_tp_config_1 Creating latest_rest_api_1 Creating latest_tp_intkey_python_1 Creating latest_integration_test_1 Attaching to latest_validator_1, latest_tp_config_1, latest_rest_api_1, latest_tp_intkey_python_1, latest_integration_test_1 validator_1 | writing file: /etc/sawtooth/keys/validator.wif validator_1 | writing file: /etc/sawtooth/keys/validator.addr tp_config_1 | [22:06:45 DEBUG selector_events] Using selector: ZMQSelector validator_1 | Generating /var/lib/sawtooth/genesis.batch validator_1 | [22:06:46.256 INFO path] Skipping path loading from non-existent config file: /etc/sawtooth/path.toml validator_1 | [22:06:46.256 INFO keys] Loading signing key: /etc/sawtooth/keys/validator.wif validator_1 | [22:06:46.259 INFO cli] config [path]: config_dir = "/etc/sawtooth" validator_1 | [22:06:46.260 INFO cli] config [path]: key_dir = "/etc/sawtooth/keys" validator_1 | [22:06:46.260 INFO cli] config [path]: data_dir = "/var/lib/sawtooth" validator_1 | [22:06:46.261 INFO cli] config [path]: log_dir = "/var/log/sawtooth" validator_1 | [22:06:46.269 INFO genesis] Producing genesis block from /var/lib/sawtooth/genesis.batch integration_test_1 | test_intkey_smoke (test_intkey_smoke.TestIntkeySmoke) ... ERROR ... integration_test_1 | urllib.error.HTTPError: HTTP Error 503: The validator is not yet ready to be queried ... validator_1 | [22:06:46.297 INFO genesis] Genesis block created: 4f664665(0, S:708ca7fb, P:00000000) validator_1 | [22:06:46.304 INFO processor_handlers] registered transaction processor: connection_id=469003765653bed7885fbc96183445243430bb4c02da7aaa572499130c7c351801446ef5a5116b07f2e5e5412ab0b1b513ed2212681813b4eda4a21b13ad4108, family=sawtooth_config, version=1.0, encoding=application/protobuf, namespaces=['000000'] tp_config_1 | [22:06:46 INFO core] register attempt: OK validator_1 | [22:06:46.324 INFO chain] Chain controller initialized with chain head: 4f664665(0, S:708ca7fb, P:00000000) validator_1 | [22:06:46.328 INFO publisher] Now building on top of block: 4f664665(0, S:708ca7fb, P:00000000) validator_1 | [22:06:46.415 INFO processor_handlers] registered transaction processor: connection_id=c3d62bf45aadae3c1ce7f0fdc8fb2a0cd15dd4b481e783c788a1a5d7af83a489f52dc0d8ddece88855076df777e6794cdb7164705c22380ad64b51d81bdf68f7, family=intkey, version=1.0, encoding=application/cbor, namespaces=['1cf126'] tp_intkey_python_1 | [22:06:46.420 INFO core] register attempt: OK latest_integration_test_1 exited with code 1 ```

drozd (Thu, 13 Apr 2017 22:18:01 GMT):
failure on intkey python smoke

drozd (Thu, 13 Apr 2017 22:18:25 GMT):
it looks like the validator got queried just before the genesis block was created

jjason (Thu, 13 Apr 2017 22:27:56 GMT):
So, maybe the test can be more resilient in that if it gets a "not ready" it waits a little bit and tries again????

jjason (Thu, 13 Apr 2017 22:29:16 GMT):
PR 333 failed build for same reason as what I mentioned above for PR 331.

drozd (Thu, 13 Apr 2017 22:52:48 GMT):
or perhaps a wait-and-try-again can be added to the rest client?

drozd (Thu, 13 Apr 2017 22:52:53 GMT):
@zac

ReedCao (Fri, 14 Apr 2017 02:14:35 GMT):
after I run /project/sawtooth-core/bin/build_python, I got an error --- Running protogen --- Error: protoc not found.

jsmitchell (Fri, 14 Apr 2017 02:16:35 GMT):
@ReedCao are you inside the vagrant vm? It should have all the prerequisites installed.

ReedCao (Fri, 14 Apr 2017 02:47:12 GMT):
@jsmitchell ye, I am inside vagrant vm

ReedCao (Fri, 14 Apr 2017 03:32:23 GMT):
don't know why protoc is not there. I have tried to run protoc, I got info that I should install it first

ReedCao (Fri, 14 Apr 2017 03:35:07 GMT):
The program 'protoc' is currently not installed. You can install it by typing: sudo apt install protobuf-compiler

zac (Fri, 14 Apr 2017 03:57:08 GMT):
@drozd Is there a use case for wait-then-try-again outside of development?

zac (Fri, 14 Apr 2017 03:58:06 GMT):
Normally the window where a validator is up but has no genesis would be very narrow relative to its life. I think leaving it at a 503 is probably the appropriate response.

ReedCao (Fri, 14 Apr 2017 07:14:54 GMT):
I can run protoc in command line, however when running /project/sawtooth-core/bin/build_python, it will report Error: protoc not found.

ReedCao (Fri, 14 Apr 2017 09:47:04 GMT):
when run protogen, it will Error: protoc not found. however, the protoc is in /usr/bin and it is in the path.

sklymenko (Fri, 14 Apr 2017 12:00:30 GMT):
Has joined the channel.

jsmitchell (Fri, 14 Apr 2017 13:27:02 GMT):
@ReedCao: that is very strange.

jsmitchell (Fri, 14 Apr 2017 13:27:51 GMT):
```try: from grpc.tools.protoc import main as _protoc except ImportError: print("Error: protoc not found.") exit(1)```

Dan (Fri, 14 Apr 2017 13:28:02 GMT):
@ReedCao its possible your vagrant install hiccup'd there and it didn't install as expected.

jsmitchell (Fri, 14 Apr 2017 13:28:02 GMT):
here is where that is getting printed out

jsmitchell (Fri, 14 Apr 2017 13:28:54 GMT):
it looks like you are probably missing the protoc python package

jsmitchell (Fri, 14 Apr 2017 13:29:13 GMT):
I would suggest doing a vagrant destroy followed by a fresh vagrant up

Dan (Fri, 14 Apr 2017 13:34:07 GMT):
Probably safest to do as @jsmitchell recommends and destroy in case there were other install glitches, but I think protoc should have been installed using the script: `tools/plugins/install_grpc.sh`.

Dan (Fri, 14 Apr 2017 15:48:04 GMT):
@drozd if you are experimenting with cutting out sleeps in the integration tests please be cautious with respect to transient failures on jenkins can slow down the PR process.

amundson (Fri, 14 Apr 2017 18:46:40 GMT):
yeah, so protoc is different than the grpc protoc wrapper, so that's a bad error message. It should say "Error: grpc.tools.protoc not found"

adamludvik (Fri, 14 Apr 2017 20:05:37 GMT):
A PR has been merged to fix the issues with the build.

pschwarz (Fri, 14 Apr 2017 20:06:30 GMT):
Those with outstanding PR's should rebase

jsmitchell (Fri, 14 Apr 2017 20:29:34 GMT):
i wouldn't really call any of these _outstanding_

pschwarz (Fri, 14 Apr 2017 20:32:29 GMT):
>:<

pschwarz (Fri, 14 Apr 2017 20:32:29 GMT):
>:(

pschwarz (Fri, 14 Apr 2017 20:32:29 GMT):
:angry:

pschwarz (Fri, 14 Apr 2017 20:33:33 GMT):
That's the right one - this doesn't quite translate ascii faces correctly...

jjason (Fri, 14 Apr 2017 20:41:16 GMT):
Ouch

adamludvik (Fri, 14 Apr 2017 20:58:50 GMT):
Not ready to do a PR on this, but could use some feedback on whether this is a good approach for running sawtooth as a service: https://github.com/aludvik/sawtooth-core/tree/systemd_services

adamludvik (Fri, 14 Apr 2017 20:58:50 GMT):
Not ready to do a PR on this, but could use some feedback on whether this is a good approach for running sawtooth as a service: https://github.com/aludvik/sawtooth-core/commits/systemd_services

ReedCao (Mon, 17 Apr 2017 02:23:57 GMT):
@jsmitchell have tried to destroy and up again.

ReedCao (Mon, 17 Apr 2017 02:24:09 GMT):
got this log.

ReedCao (Mon, 17 Apr 2017 02:26:56 GMT):

Message Attachments

jsmitchell (Mon, 17 Apr 2017 02:28:56 GMT):
Looks like your machine can't reach keyserver.ubuntu.com

jsmitchell (Mon, 17 Apr 2017 02:29:11 GMT):
Are you behind a filtering proxy?

ReedCao (Mon, 17 Apr 2017 02:30:32 GMT):
@jsmitchell just noticed that too. thanks a lot!

ReedCao (Mon, 17 Apr 2017 02:49:56 GMT):
still get this error:

ReedCao (Mon, 17 Apr 2017 02:49:57 GMT):
==> default: gpg: requesting key 10FB5F63 from hkp server keyserver.ubuntu.com ==> default: gpg: keyserver timed out ==> default: gpg: keyserver receive failed: keyserver error ==> default: + exit 1

Dan (Mon, 17 Apr 2017 02:54:08 GMT):
Which OS are you on @ReedCao ?

ReedCao (Mon, 17 Apr 2017 02:55:47 GMT):
win10

Dan (Mon, 17 Apr 2017 03:02:42 GMT):
I'm using gitbash (aka mingw64) for my command shell on win8. I have an ssh config file in `~/.ssh/config` with settings like these: ```Host=* ProxyCommand=/mingw64/bin/connect -5 -S myproxy.company.com:8080 %h %p StrictHostKeyChecking=no```

ReedCao (Mon, 17 Apr 2017 03:05:31 GMT):
so this key server is using different variable for proxy?

Dan (Mon, 17 Apr 2017 03:10:10 GMT):
I don't think I have an environment variable set for proxy. I do have ssh configured with those proxy settings though.

ReedCao (Mon, 17 Apr 2017 03:11:09 GMT):
I am using gitshell

ReedCao (Mon, 17 Apr 2017 03:11:42 GMT):
can it work the same way? however I have not found the configuration file under the ~/.ssh/config

ReedCao (Mon, 17 Apr 2017 03:15:31 GMT):
I can connect to other server than this keyserver.ubuntu.com

jsmitchell (Mon, 17 Apr 2017 03:16:24 GMT):
What message do you get in your web browser if you try to reach that server?

ReedCao (Mon, 17 Apr 2017 03:17:22 GMT):
it will load the page successfully

ReedCao (Mon, 17 Apr 2017 03:18:21 GMT):
I have set http_proxy and https_proxy in git shell, which is the same proxy server I am using in browser

jsmitchell (Mon, 17 Apr 2017 03:20:41 GMT):
The VM will also need the proxy set correctly

ReedCao (Mon, 17 Apr 2017 03:23:29 GMT):
how can I set the proxy for VM?

ReedCao (Mon, 17 Apr 2017 03:24:15 GMT):
ye, you are right, so the keyserver.ubuntu.com is being connected in VM?

jsmitchell (Mon, 17 Apr 2017 03:24:37 GMT):
Yes

ReedCao (Mon, 17 Apr 2017 03:26:00 GMT):
How can I set this proxy? run the VM and configure it ?

jsmitchell (Mon, 17 Apr 2017 03:26:45 GMT):
From the host, try "vagrant plugin install vagrant-proxyconf"

ReedCao (Mon, 17 Apr 2017 03:27:46 GMT):
I think I have seen this line in log. Install: vagrant plugin install vagrant-proxyconf

Dan (Mon, 17 Apr 2017 03:46:52 GMT):
to use ssh with proxy you have to create that config file. I'm not certain if gitshell uses that but worth a try.

ReedCao (Mon, 17 Apr 2017 06:41:55 GMT):
tried " vagrant plugin install vagrant-proxyconf" now get another error. it responsed "invalid HTTP proxy (proxy:8080): unsupported URI"

Dan (Mon, 17 Apr 2017 12:52:53 GMT):
@ReedCao that's progress at least :). I think its just a matter of getting the correct proxy information configured now. Before getting into that though, since you are on windows 10 you should be able to use the Docker instructions in the tutorial. If you want to make transaction logic that should be a sufficient environment. On the other hand, if you want to hack on the core system then you'll want to sort out this vagrant / proxy issue... If you have a proxy set in your windows environment variables, using a cmd prompt this command should show you what is set `set | findstr proxy`. If you were using the example ssh config I gave above, the port 8080 is an example, but your proxy port may be different.

toddojala (Mon, 17 Apr 2017 14:48:27 GMT):
@ReedCao Let us know how you fare. If we need to get more documentation on the proxy stuff, we should. We will be updating proxy instructions soon (in the next week or so).

Dan (Mon, 17 Apr 2017 16:34:44 GMT):
It might also be good if we can clarify use of docker vs. vagrant. Like .. If you are someone interested in writing transaction logic here's docker instructions. And its known good on OS X, Win10, and Ubuntu. Or if you are interested in modifying sawtooth internals then we recommend vagrant. etc.

Dan (Mon, 17 Apr 2017 20:10:58 GMT):
It's so quiet today :cricket: :cricket:

Dan (Mon, 17 Apr 2017 20:11:18 GMT):
(Those are crickets ^ )

toddojala (Mon, 17 Apr 2017 20:11:29 GMT):
Spring fever?

ReedCao (Tue, 18 Apr 2017 01:42:27 GMT):
@toddojala great! at least it should include the instruction to "install proxyconf" for vagrant.

gregnotso (Tue, 18 Apr 2017 12:43:14 GMT):
Has joined the channel.

cianx (Tue, 18 Apr 2017 14:20:13 GMT):
what is this error from run_tests ```[---Running validator tests---] /project/sawtooth-core ERROR:__main__:The container 'latest_validator_1' which would be created by this test already exists! Do: `run_docker_test --clean ./validator/tests/unit_validator.yaml` to remove the container, or use docker manually. ```

cianx (Tue, 18 Apr 2017 14:20:13 GMT):
```[---Running validator tests---] /project/sawtooth-core ERROR:__main__:The container 'latest_validator_1' which would be created by this test already exists! Do: `run_docker_test --clean ./validator/tests/unit_validator.yaml` to remove the container, or use docker manually. ```

cianx (Tue, 18 Apr 2017 14:20:37 GMT):
@adamludvik what is this error from run_tests ?

cianx (Tue, 18 Apr 2017 14:21:01 GMT):
the suggested 'run_docker_test --clean ./validator/tests/unit_validator.yaml' command generates an error

rbuysse (Tue, 18 Apr 2017 14:21:05 GMT):
looks like either there was a previous test that didn't exit cleanly or some tests are tunning

cianx (Tue, 18 Apr 2017 14:21:18 GMT):
```$ run_docker_test --clean ./validator/tests/unit_validator.yaml /project/sawtooth-core ERROR:__main__:Test service 'integration_test' does not exist in compose file: './validator/tests/unit_validator.yaml' ```

cianx (Tue, 18 Apr 2017 14:21:51 GMT):
How can I recover from that?

jsmitchell (Tue, 18 Apr 2017 14:22:10 GMT):
what does `docker ps` say?

cianx (Tue, 18 Apr 2017 14:22:24 GMT):
```$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ```

rbuysse (Tue, 18 Apr 2017 14:22:24 GMT):
add -a ^

rbuysse (Tue, 18 Apr 2017 14:22:47 GMT):
sorry, that's just '-a'. no caret.

cianx (Tue, 18 Apr 2017 14:22:52 GMT):
```$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4dcba9321f41 sawtooth-dev-test:latest "nose2-3 -v -s /pr..." 4 days ago Exited (255) 4 days ago 8080/tcp, 40000/tcp latest_validator_1 ```

rbuysse (Tue, 18 Apr 2017 14:23:17 GMT):
you can do ``docker rm 4dcba9321f41``

rbuysse (Tue, 18 Apr 2017 14:23:25 GMT):
(I don't know rocketchat syntax)

cianx (Tue, 18 Apr 2017 14:24:59 GMT):
that looks like it did it, thanks.

toddojala (Tue, 18 Apr 2017 14:35:22 GMT):
@ReedCao Noted.

adamludvik (Tue, 18 Apr 2017 14:35:31 GMT):
For reference, `run_docker_test --clean ./validator/tests/unit_validator.yaml -s validator` would also have worked. The problem is that `unit_validator.yaml` only has a validator service, so you have to tell `run_docker_test` which service is the test service.

Dan (Tue, 18 Apr 2017 14:56:59 GMT):
Room name changed to: sawtooth by Dan

cianx (Tue, 18 Apr 2017 15:53:03 GMT):
@adamludvik can we get the error message suggested actions corrected?

Dan (Tue, 18 Apr 2017 15:54:34 GMT):
Room name changed to: sawtoothlake by Dan

adamludvik (Tue, 18 Apr 2017 16:10:57 GMT):
Yeah

cianx (Tue, 18 Apr 2017 16:50:10 GMT):
@jjason How reliable are the poet integration tests? I am trying to figure out if I broke something or I am just unlucky.

jjason (Tue, 18 Apr 2017 16:54:32 GMT):
Oh, it's definitely you ;)

jjason (Tue, 18 Apr 2017 16:54:37 GMT):
What kind of failures?

Dan (Tue, 18 Apr 2017 17:38:18 GMT):
Command shell is another variable to keep in mind with supporting vagrant and docker across OS's. On windows Gitbash, cmd, powershell, and cygwin are all possibilities and will behave differently in accessing proxy variables etc.

cianx (Tue, 18 Apr 2017 19:33:36 GMT):
@jsmitchell @boydjohnson Any chance the thread exhaustion bug crept back in? I am seeing failures where the validators in the poet smoke test did not process all of the blocks, Jamie had me set the executing_threadpool to 100 and it looks like that has fixed the problem. I am rerunning now to verify.

cianx (Tue, 18 Apr 2017 19:33:42 GMT):
```- self._executing_threadpool = ThreadPoolExecutor(max_workers=5) + self._executing_threadpool = ThreadPoolExecutor(max_workers=100)```

cianx (Tue, 18 Apr 2017 19:34:02 GMT):
in sawtooth_validator/execution/executor.py

boydjohnson (Tue, 18 Apr 2017 19:36:35 GMT):
Do we know if there is a difference between master and your branch with this behavior? The original problem was solved by calling scheduler.cancel() any time the scheduler would never be finalized.

cianx (Tue, 18 Apr 2017 19:37:57 GMT):
let me check, that could not be happening in my branch.

jaredcorduan (Tue, 18 Apr 2017 20:30:09 GMT):
Are there plans to move the marketplace transaction family/processor to version 0.8? (like in the sdk/examples, etc) thanks!

zac (Tue, 18 Apr 2017 20:33:02 GMT):
There are no immediate plans

zac (Tue, 18 Apr 2017 20:33:24 GMT):
It is a substantial task unfortunately

boydjohnson (Tue, 18 Apr 2017 20:33:30 GMT):
I thought there were plans but not in this sprint or the next.

zac (Tue, 18 Apr 2017 20:33:57 GMT):
But, we do feel Marketplace is a great illustration, so we would like to find time for it

zac (Tue, 18 Apr 2017 20:34:36 GMT):
Can you really call it a plan if it is not in this sprint or the next?

boydjohnson (Tue, 18 Apr 2017 20:35:04 GMT):
I have seen it on a Gantt chart...

zac (Tue, 18 Apr 2017 20:35:41 GMT):
I don't think we are committed to doing it at this point

zac (Tue, 18 Apr 2017 20:35:53 GMT):
But we also aren't committed to _not_ doing it

zac (Tue, 18 Apr 2017 20:36:42 GMT):
Someone who pays more attention during planning meetings can feel free to correct me though

zac (Tue, 18 Apr 2017 20:36:42 GMT):
Someone who pays more attention during planning meetings than me should feel free to correct my impression though

zac (Tue, 18 Apr 2017 20:36:42 GMT):
Someone who pays more attention than me during planning meetings should feel free to chime in though

mfford (Tue, 18 Apr 2017 21:27:41 GMT):
The next steps for MarketPlace are being discussed in planning

kelly_ (Tue, 18 Apr 2017 21:30:11 GMT):
@jaredcorduan do you have a need for mktplace?

jaredcorduan (Wed, 19 Apr 2017 00:12:55 GMT):
ah ok, thank you for letting me know. I am currently upgrading our transaction processor from sawtooth version 0.7 to 0.8, and we de use the marketplace. (For what we are developing at PokitDok, DokChain.) I just wanted to check with you all before we upgraded it ourselves. I really like version 0.8!

jaredcorduan (Wed, 19 Apr 2017 00:12:55 GMT):
ah ok, thank you for letting me know. I am currently upgrading our transaction processor from sawtooth version 0.7 to 0.8, and we do use the marketplace. (For what we are developing at PokitDok, DokChain.) I just wanted to check with you all before we upgraded it ourselves. I really like version 0.8!

Dan (Wed, 19 Apr 2017 00:13:19 GMT):
that's great to hear!

Dan (Wed, 19 Apr 2017 00:14:42 GMT):
There's definitely interest in bringing marketplace forward but no one has committed yet when. Would be a pretty sweet PR if you worked that out.

jaredcorduan (Wed, 19 Apr 2017 00:28:18 GMT):
ok, I'll let you know if we decide to do it.

kelly_ (Wed, 19 Apr 2017 00:28:27 GMT):
If you are thinking about upgrading it we probably have some thoughts to share on how to simplify it

kelly_ (Wed, 19 Apr 2017 00:28:42 GMT):
The original design with holdings, assets, liabilities, etc is a little complex :)

jaredcorduan (Wed, 19 Apr 2017 00:29:25 GMT):
ah :)

jaredcorduan (Wed, 19 Apr 2017 00:30:51 GMT):
I'll ask the others at PokitDok their thoughts on what all we really need and see what they think best

kelly_ (Wed, 19 Apr 2017 00:43:09 GMT):
Let us know if you need help or want to collaborate

PacoEcotrust (Wed, 19 Apr 2017 08:00:40 GMT):
Hi, I am in this step----->docker-compose -f docker/compose/sawtooth-demo.yaml up-------------------->and have problems with the version of Docker-compose. Is possible to install one concrete version of it?

PacoEcotrust (Wed, 19 Apr 2017 08:00:40 GMT):
Hi, I am in this step----->docker-compose -f docker/compose/sawtooth-demo.yaml up-------------------->and have problems with the version of Docker-compose. Is possible to install one concrete version of it with the idea to resolve the problem?

kpkulkarni (Wed, 19 Apr 2017 13:27:49 GMT):
Has joined the channel.

kpkulkarni (Wed, 19 Apr 2017 13:29:57 GMT):
Sawtooth gurus, Is there any tutorial to create a executable python/shell script and deploy it on the blockchain ?

toddojala (Wed, 19 Apr 2017 14:24:53 GMT):
@kpkulkarni Could you be more specific? Do you want to create a client, or a new transaction family, etc.?

toddojala (Wed, 19 Apr 2017 14:28:39 GMT):
@kpkulkarni Please take a look at our main documentation at: http://intelledger.github.io/ and run through the tutorials. If you have question that is not answered, let us know here. In the repository, the SDKs for python, javascript and java are also available: https://github.com/hyperledger/sawtooth-core/tree/master/sdk

Dan (Wed, 19 Apr 2017 14:38:22 GMT):
@amundson is `extensions` meant to continue in the new layout or is that all content to be migrated/eliminated?

zac (Wed, 19 Apr 2017 15:06:53 GMT):
A big refactor to how the REST API sends back errors is underway. The biggest feature is that each individual error will have a unique fixed integer code, to make handling them easier. You can view the PR here if you are curious: https://github.com/hyperledger/sawtooth-core/pull/354

zac (Wed, 19 Apr 2017 15:07:28 GMT):
My question for everyone involved is, has anyone already written code that parses errors using something other than the HTTP status?

zac (Wed, 19 Apr 2017 15:09:03 GMT):
The HTTP status codes will all stay the same, but the response body is switching from just text to a JSON object, so if anyone used that text at all, their code will break.

zac (Wed, 19 Apr 2017 15:09:42 GMT):
I only know of one instance that is, and I am fixing it as part of my PR, but I want to make sure than aren't any other instances I'm missing.

pschwarz (Wed, 19 Apr 2017 15:11:44 GMT):
@kpkulkarni are you wanting to effectively have a python smart contract?

jsmitchell (Wed, 19 Apr 2017 15:16:28 GMT):
@kpkulkarni: check out the xo transaction family. It's a simple example written in python. https://github.com/hyperledger/sawtooth-core/blob/master/sdk/examples/xo_python/sawtooth_xo/processor/handler.py

agunde (Wed, 19 Apr 2017 16:25:00 GMT):
The intkey examples have been updated to use the rest api instead of the stream class. This means that if you wish to use intkey load or intkey workload you will need to start up the rest_api for the validator.

Dan (Wed, 19 Apr 2017 18:41:46 GMT):
Per bandit and @dhuseby note more secure manner for loading yaml updated in @rbuysse's PR https://github.com/hyperledger/sawtooth-core/pull/352

choas (Wed, 19 Apr 2017 21:07:32 GMT):
Has left the channel.

kpkulkarni (Thu, 20 Apr 2017 02:39:00 GMT):
Thanks all for valuable comments. I shall read the sawtooth docs and get back soon

benjaminbollen (Thu, 20 Apr 2017 11:43:09 GMT):
Has joined the channel.

betolive (Thu, 20 Apr 2017 14:47:18 GMT):
Has joined the channel.

Dan (Thu, 20 Apr 2017 16:04:40 GMT):
if there's no use case for a no-op transaction then I'd say we don't support that.

boydjohnson (Thu, 20 Apr 2017 16:39:19 GMT):
So just throwing my 2 cents in on this, I *could imagine* a use for reading from state but not writing to state. Where a transaction checks for a value and if it is there it doesn't throw an error, but doesn't write to state.

boydjohnson (Thu, 20 Apr 2017 16:45:43 GMT):
This would limit reads (messages sent between tp and context manager) where a many txns in a batch need to depend on a value being there.

agunde (Thu, 20 Apr 2017 17:04:26 GMT):
The above idea could create a no-op batch (only a no-op txn), which could lead to a no-op block (no state change) which seems bad and could break assumptions about block and state_root_hashes. It seems like a slippery slope.

amundson (Thu, 20 Apr 2017 17:30:02 GMT):
reconciliation txns might result in no updates if there happens to be nothing to do

agunde (Thu, 20 Apr 2017 17:39:55 GMT):
@amundson Can you explain what you mean by reconciliation txn?

amundson (Thu, 20 Apr 2017 17:44:22 GMT):
I'm thinking like in bond, where we might have a transaction which matches orders and quotes

agunde (Thu, 20 Apr 2017 17:59:08 GMT):
So if we had a transaction which matches orders and quotes but does not find any that could be matched, it would not change state.Okay, therefore you are for allowing no-op txn? @amundson

jsmitchell (Thu, 20 Apr 2017 18:01:24 GMT):
Well, in bond as currently implemented it's essentially a client process that is generating a number of order/quote match txns

jsmitchell (Thu, 20 Apr 2017 18:01:48 GMT):
The generated txns always modify state or fail in that example

agunde (Thu, 20 Apr 2017 18:02:44 GMT):
@jsmitchell correct. I was thinking about it as if it was done differently (kicked off by a transaction)

agunde (Thu, 20 Apr 2017 18:03:29 GMT):
I could see an author of transaction processor implementing something that way.

jsmitchell (Thu, 20 Apr 2017 18:48:11 GMT):

Message Attachments

jsmitchell (Thu, 20 Apr 2017 19:19:24 GMT):
That PDF is a presentation I gave last night at the Enterprise Blockchain Meetup here in Minneapolis

Dan (Thu, 20 Apr 2017 21:01:54 GMT):
I'm not sure if I'm really tired or if my vagrant is messed up...

Dan (Thu, 20 Apr 2017 21:02:20 GMT):
sawtooth-core should be at `/project/sawtooth-core` ?

Dan (Thu, 20 Apr 2017 21:10:20 GMT):
yes yes it should, dan.

Dan (Thu, 20 Apr 2017 21:10:28 GMT):
Thanks dan.

Dan (Thu, 20 Apr 2017 21:11:20 GMT):
that directory isn't getting mapped on two of my systems (one linux, one windows). going to destroy and refresh vagrant and see if it gets cleared up.

Dan (Thu, 20 Apr 2017 21:55:18 GMT):
Not sure what was going on but my windows env is healthy again and linux is still in the long d/l process.

amundson (Thu, 20 Apr 2017 21:58:32 GMT):
@agunde - I'm not for or against the no-op transaction, just providing a potential "real" use-case

foobarometer (Fri, 21 Apr 2017 05:43:10 GMT):
I do have to say the 0.8 documentation makes more sense than 0.7 (especially the ordering). Thx for updating it!

Dan (Fri, 21 Apr 2017 13:16:51 GMT):
thanks for the feedback :) (and nice reorganizing @amundson)

Dan (Fri, 21 Apr 2017 14:00:48 GMT):
I'm curious what strategies people use for reviewing large PRs. We've had several lately.. in fact most of this calendar year given the large updates to the sawtooth. Using @jjason's recent PR as an example https://github.com/hyperledger/sawtooth-core/pull/359/ There are 15 commits and ~1500 loc changed. (This is an appropriate size for this feature branch. But it's large for diff based approaches like github's views) Sometimes I walk the commits individually and sometimes I look at the full diff of the end result. I gather some people go right to running and testing it. Walking commits is sometimes awkward because its not obvious if the changes get obviated or improved by the 15th commit. On the other hand, looking at the full diff you see different ideas blended together so the intents of each change aren't necessarily obvious to decipher. When a file has substantially changed it's easier for me to pull up the whole file in my editor and review it as though its entirely new (with glances back to the diff). What do other people do?

jsmitchell (Fri, 21 Apr 2017 14:01:29 GMT):
I go commit by commit

jsmitchell (Fri, 21 Apr 2017 14:01:38 GMT):
and I appreciate smaller PRs. :)

boydjohnson (Fri, 21 Apr 2017 14:06:16 GMT):
I think the commit by commit approach is the best, and it is a good idea for people to group code changes into commits that mean something specific for each commit.

toddojala (Fri, 21 Apr 2017 14:09:17 GMT):
@Dan I'm more of a novice with git/github, but I have the same problem. I have to review the whole file if there are a lot of changes. I am starting to look at individual commits more, too. But I myself could benefit from some strategies that others use.

zac (Fri, 21 Apr 2017 14:24:38 GMT):
As a PR-maker I try to create commits that not only encompass one idea, but that are functional and make sense on their own. I ask myself "What if the repo got rolled back to this commit, and no longer had my whole PR?" Hopefully that addresses the "changes might get obviated" problem a bit. They still might, but they should make sense on their own too, so feel free to leave a comment.

boydjohnson (Fri, 21 Apr 2017 14:27:11 GMT):
Yeah, definitely, because it wouldn't just be if master rolled back to that point -- which is unlikely -- but if a person is doing a git bisect or cherry picking commits to another branch the commits being focused and self-contained is important.

jjason (Fri, 21 Apr 2017 14:55:10 GMT):
@dan w.r.t., recent PR (or recent PRs), I chose to focus commits on targeted changes. I could have squashed related ones down together to make it be only a few larger ones. I was under the impression that it was better to have these smaller, targeted commits. I wish that git could convey intent - for example, even though there are approximately 1500loc changed, it is probably more like a few hundred new lines of code (including unit tests) and the rest is refactoring to new design, so the code is not new, just lives somewhere else.

jjason (Fri, 21 Apr 2017 14:57:10 GMT):
@zac @boydjohnson I too like the idea that a commit can be rolled back to, but let me ask a question. A few weeks ago when I was working with @amundson on moving all of the PoET consensus stuff to its own packages, it was recommended to me that the first commit be simply moving the files, then follow-on commits fixing up scripts, etc., that referenced the files. Obviously if someone cherry-picked the move but not the script fix-up, then things are broken. I am getting conflicting information.

jjason (Fri, 21 Apr 2017 14:57:10 GMT):
@zac @boydjohnson I too like the idea that a commit can be rolled back to, but let me put forth an example. A few weeks ago when I was working with @amundson on moving all of the PoET consensus stuff to its own packages, it was recommended to me that the first commit be simply moving the files, then follow-on commits fixing up scripts, etc., that referenced the files. Obviously if someone cherry-picked the move but not the script fix-up, then things are broken. I am getting conflicting information.

jjason (Fri, 21 Apr 2017 14:58:42 GMT):
I think part of my confusion around the PR process is that there is no "group standard" at least that I am aware of. As someone newer to git, I end up pinging someone more knowledgeable, but depending upon whom I ask I get slightly different guidance.

boydjohnson (Fri, 21 Apr 2017 15:01:07 GMT):
Yeah, my thoughts would be that a commit that moves the files would also include the import updates, too. We should come to some group understanding on how commits should be structured.

jjason (Fri, 21 Apr 2017 15:02:42 GMT):
Agreed.

jjason (Fri, 21 Apr 2017 15:06:51 GMT):
So, while we are at it, I have another question. How do people handle "stacked" PRs - and what I mean by that is suppose I submit PR 1 and then start working on PR 2, which builds on/requires PR 1. I get PR 2 work finished before PR 1 has been accepted/merged. Typically I have been holding PR 2 until PR 1's merge and then rebase PR 2 and submit it. I could submit PR 2 before PR 1 is done.

jsmitchell (Fri, 21 Apr 2017 15:07:45 GMT):
@jjason: if you base PR 2 on the branch from PR 1, and PR 1 gets merged, github will do the right thing

jsmitchell (Fri, 21 Apr 2017 15:08:05 GMT):
those commits will 'disappear' from PR 2 because they are now in master

jsmitchell (Fri, 21 Apr 2017 15:08:26 GMT):
of course, if you push fresh commits to PR 1's branch, you will need to rebase PR 2's branch

jjason (Fri, 21 Apr 2017 15:09:00 GMT):
I assumed that part (git is all powerful and all knowing ;) ). That is one reason why I hold back PR 2 - so I can rebase the final PR 1 merge before submitting.

boydjohnson (Fri, 21 Apr 2017 15:10:01 GMT):
Also there should probably be a comment in the 2nd PR that it is based on the first PR.

drozd (Fri, 21 Apr 2017 15:14:45 GMT):
Another issue to consider is reducing clutter in the commit log. Consider my recent PR https://github.com/hyperledger/sawtooth-core/pull/347, which consists of a single commit "Move TP tests to TP directories". When I was working on it, I had four commits: "Move intkey TP tests", "Move xo TP test", "Move config TP test", "Move validator registry TP test". You probably got bored and skipped to the end of that list, and I figure anyone looking at the master commit list would do the same. It's like "I don't need your life story, just tell me what it does. Oh, it moves all the TP tests? Great."

Dan (Fri, 21 Apr 2017 15:26:21 GMT):
To be clear @jjason I wasn't being critical of your PR. I took it as an example of something that was necessarily large and yet well organized.

Dan (Fri, 21 Apr 2017 15:26:51 GMT):
Efficient strategies for handling large PRs seem useful for the team to figure out.

jsmitchell (Fri, 21 Apr 2017 15:26:53 GMT):
@jjason is midas

jsmitchell (Fri, 21 Apr 2017 15:27:12 GMT):
:star2:

jsmitchell (Fri, 21 Apr 2017 15:27:33 GMT):
he's like the anti-Lindelof

jjason (Fri, 21 Apr 2017 15:29:38 GMT):
@dan No worries. I know you weren't picking on me specifically, but wanted to give rationale as well as give some feedback on the frustrations I encountered with the PR process.

boydjohnson (Fri, 21 Apr 2017 15:30:02 GMT):
Somehow my rebase and rebuild on my PR missed the js tp test failure. So, 3 commit PR seeks friendly reviewers for short-term commitment: https://github.com/hyperledger/sawtooth-core/pull/295.

zac (Fri, 21 Apr 2017 15:36:04 GMT):
@drozd Yeah, I would also group all of the move commits into one. Even though technically that isn't an atomic change, it is a very logical way to group the changes in my mind

zac (Fri, 21 Apr 2017 15:36:25 GMT):
Although I also wouldn't reject a PR that had broken up the file moves the way you described

Dan (Fri, 21 Apr 2017 15:53:21 GMT):
@jsmitchell is just giving you praise because you edit your commit messages in vim.

Dan (Fri, 21 Apr 2017 15:53:51 GMT):
As evidenced by: https://github.com/hyperledger/sawtooth-core/pull/359/commits/d0ec41453316beb00b0a4789378fab4219b59d48

jsmitchell (Fri, 21 Apr 2017 15:54:20 GMT):
lol

boydjohnson (Fri, 21 Apr 2017 15:54:43 GMT):
I'm going to modify my default git editor to be pycharm.

boydjohnson (Fri, 21 Apr 2017 15:55:24 GMT):
jk

jsmitchell (Fri, 21 Apr 2017 15:55:58 GMT):
I remember the day that it was clear Boyd was beyond saving

jjason (Fri, 21 Apr 2017 16:03:21 GMT):
D'Oh!!!

drozd (Fri, 21 Apr 2017 16:40:54 GMT):
related git squash discussion: https://news.ycombinator.com/item?id=11407289

jjason (Fri, 21 Apr 2017 16:45:56 GMT):
:) I read the first post and response and the first thing that popped into my head was "religious war". However, I can see utility to keeping small targeted commits in the PR and then squashing them into master into something more "atomic".

drozd (Fri, 21 Apr 2017 16:49:31 GMT):
One comment argues: "If you really want to 'see how the sausage is made', why not log literally every keystroke? But that would be insane, so clearly we can see the need for finding the appropriate level of granularity." The response comment says: "Actually, yes, I _would_ like to see every keystroke logged, it would be very informative!"

drozd (Fri, 21 Apr 2017 16:52:18 GMT):
from https://git-scm.com/book/en/v2/Git-Branching-Rebasing: ``` One point of view on this is that your repository’s commit history is a record of what actually happened. It’s a historical document, valuable in its own right, and shouldn’t be tampered with. From this angle, changing the commit history is almost blasphemous; you’re lying about what actually transpired. So what if there was a messy series of merge commits? That’s how it happened, and the repository should preserve that for posterity. The opposing point of view is that the commit history is the story of how your project was made. You wouldn’t publish the first draft of a book, and the manual for how to maintain your software deserves careful editing. This is the camp that uses tools like rebase and filter-branch to tell the story in the way that’s best for future readers. ```

jjason (Fri, 21 Apr 2017 16:55:09 GMT):
So, I am going to let smarter git users figure what is the "right' thing to do. If there is a cage match in MSP, please live stream it. ;)

zac (Fri, 21 Apr 2017 19:28:52 GMT):
I have always been more storyteller than historian

boydjohnson (Fri, 21 Apr 2017 19:45:39 GMT):
Yeah, I would have to agree with the storyteller side vs the historian side. While it might be a "interesting" part of the history of a PR that I forgot to do something and made a fixup commit, it really has little to do with the changes.

boydjohnson (Fri, 21 Apr 2017 19:45:39 GMT):
Yeah, I would have to agree with the storyteller side vs the historian side. While it might be an "interesting" part of the history of a PR that I forgot to do something and made a fixup commit, it really has little to do with the changes.

jjason (Fri, 21 Apr 2017 23:13:11 GMT):
Seeing this intermittent failure when submitting PRs: ``` integration_test_1 | ====================================================================== integration_test_1 | FAIL: test_resting_shutdown_sigint (test_shutdown_smoke.TestShutdownSmoke) integration_test_1 | Tests that SIGINT will cause validators with and without integration_test_1 | ---------------------------------------------------------------------- integration_test_1 | Traceback (most recent call last): integration_test_1 | File "/project/sawtooth-core/integration/sawtooth_integration/tests/test_shutdown_smoke.py", line 70, in test_resting_shutdown_sigint integration_test_1 | self.assertEqual(len(sigint_exit_statuses), len(containers)) integration_test_1 | AssertionError: 0 != 2 integration_test_1 | integration_test_1 | During handling of the above exception, another exception occurred: integration_test_1 | integration_test_1 | Traceback (most recent call last): integration_test_1 | File "/project/sawtooth-core/integration/sawtooth_integration/tests/test_shutdown_smoke.py", line 77, in test_resting_shutdown_sigint integration_test_1 | self.fail(str(e)) integration_test_1 | AssertionError: 0 != 2 integration_test_1 | integration_test_1 | ---------------------------------------------------------------------- ```

jjason (Fri, 21 Apr 2017 23:18:42 GMT):
https://build.sawtooth.me/job/Sawtooth-Hyperledger/job/sawtooth-core/job/PR-370/1/console

kylestratis (Sun, 23 Apr 2017 02:59:25 GMT):
Has joined the channel.

AlexCacciamani (Sun, 23 Apr 2017 13:58:51 GMT):
Has joined the channel.

shreyankg (Mon, 24 Apr 2017 13:50:48 GMT):
Has joined the channel.

boydjohnson (Mon, 24 Apr 2017 14:44:39 GMT):
@jjason, I have seen that test failure several times. It means that both validators didn't shutdown in the amount of time given. But when I have looked at the logs it looked like it wasn't far in the shutdown process.

jaredcorduan (Mon, 24 Apr 2017 14:48:10 GMT):
Is there a clean way to reward nodes in the network (as a part of a transaction) for validating and applying transactions, such as would be done by a miner in the bitcoin network? Version 0.7 had a way of doing this, as seen in the marketplace code: https://github.com/hyperledger/sawtooth-core/blob/0-7/extensions/mktplace/mktplace/transactions/market_place.py#L66

kelly_ (Mon, 24 Apr 2017 16:09:20 GMT):
@jaredcorduan I don't believe there is, you could create an incentive transaction in any txn processor you are running though

jaredcorduan (Mon, 24 Apr 2017 16:59:05 GMT):
ok, thanks @kelly_ !

DerekTiffany (Mon, 24 Apr 2017 17:18:14 GMT):
Has joined the channel.

amundson (Mon, 24 Apr 2017 17:26:17 GMT):
@benjaminbollen go sdk support is now in master - https://github.com/hyperledger/sawtooth-core

jjason (Mon, 24 Apr 2017 17:31:40 GMT):
@boydjohnson Do we have a defect in JIRA to cover this?

boydjohnson (Mon, 24 Apr 2017 17:38:13 GMT):
I have a further shutdown jira task, and I was expecting that that would encompass this, but we don't have a specific Jira defect task to make sure this test isn't flaky.

amundson (Tue, 25 Apr 2017 02:11:05 GMT):
oh my, I can't believe I missed that great git discussion @jjason @boydjohnson . my point with respect to moving separate from code changes is that if you do it all at once, it can be virtually impossible to review. git bisect is a pretty good argument I hadn't thought specifically about. With respect to cherry-picking, it is _far_ easier to deal with separate move/change commits if it does not apply cleanly. It is super easy and common to cherry-pick a series of commits, so that's not really a problem either.

amundson (Tue, 25 Apr 2017 02:30:32 GMT):
though, it seems most relevant for the larger moves

JonathanLevi (Tue, 25 Apr 2017 13:30:53 GMT):
Hi everyone, does anyone happen to have an example of using the REST API to post/send transactions?

jsmitchell (Tue, 25 Apr 2017 13:32:07 GMT):
@zac ^

toddojala (Tue, 25 Apr 2017 14:20:48 GMT):
@JonathanLevi Not sure if this is what you are looking for, but the CLI submit command uses the REST API to send transactions. The code is fairly simple, so it might be useful: https://github.com/hyperledger/sawtooth-core/blob/master/cli/sawtooth_cli/submit.py

jsmitchell (Tue, 25 Apr 2017 14:28:10 GMT):
That's using these methods: https://github.com/hyperledger/sawtooth-core/blob/master/cli/sawtooth_cli/rest_client.py

zac (Tue, 25 Apr 2017 14:34:01 GMT):
@JonathanLevi I will actually have a PR in today with a tutorial of how to do just that

JonathanLevi (Tue, 25 Apr 2017 14:34:35 GMT):
@jsmitchell + @zac: this is great, thank you.

zac (Tue, 25 Apr 2017 14:35:11 GMT):
The REST API side of it is fairly simple, send a POST to /batches, Content-Type: application/octet-stream, the body should be the binary for your BatchList

zac (Tue, 25 Apr 2017 14:35:34 GMT):
Now, building that binary is . . . a little more involved

zac (Tue, 25 Apr 2017 14:36:26 GMT):
If you need a walkthrough now, I can take you though it, otherwise probably easier to wait for the PR

zac (Tue, 25 Apr 2017 14:36:33 GMT):
Shouldn't be too much longer now

JonathanLevi (Tue, 25 Apr 2017 14:37:09 GMT):
We can just wait for the PR, so that you won't have to repeat yourself... thank you.

JonathanLevi (Tue, 25 Apr 2017 14:37:19 GMT):
(Appreciate it)

zac (Tue, 25 Apr 2017 14:40:32 GMT):
no problem, you have good timing

JonathanLevi (Tue, 25 Apr 2017 14:56:44 GMT):
BTW: https://github.com/hyperledger/sawtooth-core/pull/377

jsmitchell (Tue, 25 Apr 2017 15:01:37 GMT):
Thanks, @JonathanLevi !

Dan (Tue, 25 Apr 2017 15:08:25 GMT):
Nice!

zac (Tue, 25 Apr 2017 17:38:16 GMT):
@JonathanLevi PR is here: https://github.com/hyperledger/sawtooth-core/pull/381

zac (Tue, 25 Apr 2017 17:38:51 GMT):
The modular way the tutorial is written though, it is not particularly nice to read the source code.

zac (Tue, 25 Apr 2017 17:39:01 GMT):
Here is a zip of the built html:

zac (Tue, 25 Apr 2017 17:39:29 GMT):

Message Attachments

zac (Tue, 25 Apr 2017 17:40:17 GMT):
Open index.html, and go to App Developer's Guide > Building and Submitting a Transaction

zac (Tue, 25 Apr 2017 17:40:24 GMT):
Let me know if you have any questions

amundson (Tue, 25 Apr 2017 19:31:18 GMT):
@zac - should be able to dig around jenkins to find the url to docs built there too for any pr

zac (Tue, 25 Apr 2017 19:31:51 GMT):
that's nifty

zac (Tue, 25 Apr 2017 19:33:31 GMT):
Hrmm, html files are here: https://build.sawtooth.me/job/Sawtooth-Hyperledger/job/sawtooth-core/job/PR-381/1/artifact/docs/build/

zac (Tue, 25 Apr 2017 19:33:53 GMT):
They aren't actually viewable as a web page somewhere too are they?

jsmitchell (Tue, 25 Apr 2017 19:34:08 GMT):
https://build.sawtooth.me/job/Sawtooth-Hyperledger/job/sawtooth-core/job/PR-381/1/artifact/docs/build/html/app_developers_guide/txn_submit_tutorial/python.html

zac (Tue, 25 Apr 2017 19:34:20 GMT):
whoa

amundson (Tue, 25 Apr 2017 19:34:21 GMT):
https://build.sawtooth.me/job/Sawtooth-Hyperledger/job/sawtooth-core/job/PR-381/1/artifact/docs/build/html/index.html

zac (Tue, 25 Apr 2017 19:34:26 GMT):
All hail Jenkins

zac (Tue, 25 Apr 2017 19:35:17 GMT):
Weird, when I clicked on the file before it just rendered the source

zac (Tue, 25 Apr 2017 19:35:26 GMT):
Now it is rendering as a page

kelly_ (Tue, 25 Apr 2017 19:53:59 GMT):
oh nice! I just sent the build of the docs to some people who were interested in pending PR documentation

zac (Tue, 25 Apr 2017 20:26:20 GMT):
FYI, the language select will not work in the Jenkins build because it blocks the `script` tag.

prashiyn (Wed, 26 Apr 2017 05:41:19 GMT):
Has joined the channel.

pancake (Wed, 26 Apr 2017 13:24:20 GMT):
Hi, all, i am trying to figure out how PoET

pancake (Wed, 26 Apr 2017 13:27:59 GMT):
Hi, all, i am trying to figure out how PoET work, but after google, I only got a few info. Could you pls give me some advice or poet's detailed info?

pancake (Wed, 26 Apr 2017 13:30:21 GMT):
BTW, thx for the guy who have reply my question, and sorry for forget reply you for my poor memory. :joy:

pancake (Wed, 26 Apr 2017 13:31:12 GMT):
Few weeks ago.

jsmitchell (Wed, 26 Apr 2017 14:11:40 GMT):
hi @pancake - rocketchat provides a search feature. You should be able to look back and see the conversation.

jsmitchell (Wed, 26 Apr 2017 14:12:04 GMT):
aside from that, what would you like to know?

jsmitchell (Wed, 26 Apr 2017 14:12:46 GMT):
have you read this?http://intelledger.github.io/introduction.html#proof-of-elapsed-time-poet

Dan (Wed, 26 Apr 2017 17:22:04 GMT):
Any volunteers to write up sawtooth doc for 'blocks and chains'? >Introduce our block definition (okay to be specific here in terms of fields, signing approach, etc.), similar to batch/transaction specificity. >Formal Completeness

Dan (Wed, 26 Apr 2017 17:22:04 GMT):
Any volunteers to write up sawtooth doc for 'blocks and chains'? _Introduce our block definition (okay to be specific here in terms of fields, signing approach, etc.), similar to batch/transaction specificity. Formal Completeness_

Dan (Wed, 26 Apr 2017 17:22:04 GMT):
Any volunteers to write up sawtooth doc for 'blocks and chains'? _ Introduce our block definition (okay to be specific here in terms of fields, signing approach, etc.), similar to batch/transaction specificity. Formal Completeness _

Dan (Wed, 26 Apr 2017 17:22:04 GMT):
Any volunteers to write up sawtooth doc for 'blocks and chains'? "Introduce our block definition (okay to be specific here in terms of fields, signing approach, etc.), similar to batch/transaction specificity. Formal Completeness"

Dan (Wed, 26 Apr 2017 17:22:04 GMT):
Any volunteers to write up sawtooth doc for 'blocks and chains'? >"Introduce our block definition (okay to be specific here in terms of fields, signing approach, etc.), similar to batch/transaction specificity. Formal Completeness"

Dan (Wed, 26 Apr 2017 17:22:04 GMT):
Any volunteers to write up sawtooth doc for 'blocks and chains'? >"Introduce our block definition (okay to be specific here in terms of fields, signing approach, etc.), similar to batch/transaction specificity. >Formal Completeness"

amundson (Wed, 26 Apr 2017 17:40:43 GMT):
(for the arch. doc)

Dan (Wed, 26 Apr 2017 17:50:04 GMT):
Yeah section of the arch doc. Was going to add it to backlog but not sure who a good owner is.

pschwarz (Wed, 26 Apr 2017 18:34:26 GMT):
With the merger of https://github.com/hyperledger/sawtooth-core/pull/361, if one wants to modify on-chain configuration settings, you'll need do the following (minimaly): ``` > sawtooth config genesis -k -o config-genesis.batch ... other genesis batch creations, e.g using sawtooth config propose ... > sawtooth admin genesis config-genesis.batch ... ``` You can start up a validator without this, but without without an authorized key, you can only run with default settings.

amundson (Wed, 26 Apr 2017 18:56:52 GMT):
@pschwarz - can you omit the -k to use your default key?

johnsacco (Wed, 26 Apr 2017 20:22:31 GMT):
Has joined the channel.

pschwarz (Wed, 26 Apr 2017 21:34:50 GMT):
Yes, that's right

pschwarz (Wed, 26 Apr 2017 21:34:55 GMT):
I forgot about that

pschwarz (Wed, 26 Apr 2017 21:35:14 GMT):
The default generated with `sawtooth keygen`

pschwarz (Wed, 26 Apr 2017 21:35:14 GMT):
The default generated with `sawtooth keygen`, to be exact

pschwarz (Wed, 26 Apr 2017 21:49:58 GMT):

Message Attachments

amundson (Wed, 26 Apr 2017 23:12:09 GMT):
For the config key strategy, I think the sha256*8 (worst case) is lot of overhead. Essentially, we are using it to attempt to keep the tree balanced. Any suggestions for another, lighter, mechanism to hash/encode the path components?

pschwarz (Wed, 26 Apr 2017 23:22:21 GMT):
@boydjohnson Would be good to get that shutdown integration test from having those periodic failures

Dan (Wed, 26 Apr 2017 23:22:41 GMT):
thanks for the doc @pschwarz!

boydjohnson (Wed, 26 Apr 2017 23:22:57 GMT):
Yeah, definitely. @pschwarz

Dan (Thu, 27 Apr 2017 00:04:58 GMT):
@amundson that sounds like how to efficiently encode a string from a variable length ~52 element set into a fixed length string from a 16 element set. Sounds like something up @hartm's alley (If that's Hart Montgomery's handle?).

hartm (Thu, 27 Apr 2017 00:04:58 GMT):
Has joined the channel.

Dan (Thu, 27 Apr 2017 00:19:21 GMT):
Only loosely related but it would be nice to sweep all the hash use behind a facade like signing ops behind sawtooth-signing. That way we could swap in different hash libs or algos if needed w/o going all over the system.

amundson (Thu, 27 Apr 2017 01:11:24 GMT):
in the config case, it becomes part of the specification for that portion of state so explicit is probably better. if we did want it to be configurable, we would probably want to add some special address containing the hash method to use for bootstrapping the config system. Which might be ok for vaidator processes, but would add overhead to other things like the config cli, which would now need to know on-chain info tondo what can currently be done off chain.

jsmitchell (Thu, 27 Apr 2017 01:18:47 GMT):
Collisions will be quite likely with hashes that small

jsmitchell (Thu, 27 Apr 2017 01:19:33 GMT):
@pschwarz ^

pancake (Thu, 27 Apr 2017 02:05:49 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=XcdDQ2HrQ8fzHH2xD) @jsmitchell Thanks for your reply! I have read this before, but still can't understand why the wait time can achieve consistency? Does the validator become leader and can generate a txn block after it get shortest wait time? How to ensure weather it lose some txn and do not put these txns into a block.

jsmitchell (Thu, 27 Apr 2017 02:10:11 GMT):
Transactions are broadcast around the network, so all validators have the ability to include them in blocks. Once a block is received and applied, all validators start waittimers for the next block. If their waittimer expires before they have received a valid block from a competitor, then they have 'won' the right to build the block. They send out the block, and the other validators verify and apply it and the process begins again.

pancake (Thu, 27 Apr 2017 02:22:08 GMT):
[ ](https://chat.hyperledger.org/channel/sawtoothlake?msg=xwtdmMQsep6huscQX) @jsmitchell "Wait for a randomized amount of time - simulates partial preimage search", is this right?

jsmitchell (Thu, 27 Apr 2017 02:24:02 GMT):
Yep, simulates nakamoto mining

jsmitchell (Thu, 27 Apr 2017 02:24:40 GMT):
Except there is no 'work'

jsmitchell (Thu, 27 Apr 2017 02:24:48 GMT):
CPU is idle

pancake (Thu, 27 Apr 2017 02:26:21 GMT):
So what does "TargetWaitTime" and "InitialWaitTime" mean in txnvalidator.js?

jsmitchell (Thu, 27 Apr 2017 02:27:14 GMT):
It would be more instructive to look at the poet implementation in master

jsmitchell (Thu, 27 Apr 2017 02:27:49 GMT):
TargetWaitTime is the desired inter block time for the network

jsmitchell (Thu, 27 Apr 2017 02:28:03 GMT):
Usually 15-30 seconds in our networks

jsmitchell (Thu, 27 Apr 2017 02:31:28 GMT):
InitialWaitTime determines the shape of the 'population estimate' of the network for the first fifty blocks. This is used to shape the exponential random distribution to hit an inter block time based on a configured number of nodes. Once fifty blocks pass, real population data from the blocks are used to generate the population estimate.

pancake (Thu, 27 Apr 2017 02:36:04 GMT):
Could you pls explain the difference between "inter block time" and time generated by enclave？

pancake (Thu, 27 Apr 2017 02:36:09 GMT):
@jsmitchell

jsmitchell (Thu, 27 Apr 2017 02:37:53 GMT):
Interblock time is the average of the minimum random wait time generated by all enclaves across blocks

jsmitchell (Thu, 27 Apr 2017 02:38:46 GMT):
So, if we have 100 validators, and we are targeting 30 second interblock times, we need _most_ of the random times generated to be larger than 30s

jsmitchell (Thu, 27 Apr 2017 02:39:08 GMT):
Ideally one node will get a timer of around 30s

jsmitchell (Thu, 27 Apr 2017 02:39:15 GMT):
And the rest will be higher

pancake (Thu, 27 Apr 2017 02:39:32 GMT):
Like the bitcoin is generated about 10 min, am i right?

jsmitchell (Thu, 27 Apr 2017 02:39:44 GMT):
Yeah, 7 minutes I think

jsmitchell (Thu, 27 Apr 2017 02:40:59 GMT):
Or maybe it's 10 minutes

pancake (Thu, 27 Apr 2017 02:43:11 GMT):
@jsmitchell You’re one in a million, thanks a lot for your help:smile:

jsmitchell (Thu, 27 Apr 2017 02:43:37 GMT):
Anytime!

pschwarz (Thu, 27 Apr 2017 14:12:29 GMT):
@jsmitchell collisions are high, but the settings values in state are hash buckets - they handle collisions

pschwarz (Thu, 27 Apr 2017 14:12:29 GMT):
@jsmitchell collisions are potentially higher, but the settings values in state are hash buckets - they handle collisions

pschwarz (Thu, 27 Apr 2017 17:26:03 GMT):

Message Attachments

pschwarz (Thu, 27 Apr 2017 17:26:25 GMT):
@jsmitchell :point_up:

pschwarz (Thu, 27 Apr 2017 17:27:41 GMT):
I've reduced the number of parts from 8 to 4, which should create a better collision-avoidance outcome. It will also reduce the number of hashes made, which should have better performance characteristics

amundson (Thu, 27 Apr 2017 18:23:57 GMT):
@jsmitchell, @pschwarz, @agunde - do we intend to implement any additional topology management / peering mechanisms than the one we currently have for 0.8?

pschwarz (Thu, 27 Apr 2017 18:25:16 GMT):
There are plans for direct send of messages between peers over the network topology

pschwarz (Thu, 27 Apr 2017 18:25:28 GMT):
As well as directly

pschwarz (Thu, 27 Apr 2017 18:25:28 GMT):
As well as directly, with temporary connections

jsmitchell (Thu, 27 Apr 2017 18:25:54 GMT):
We don't need it for parity. We'll maybe need to do some tweaking of the current peering with larger network testing.

amundson (Thu, 27 Apr 2017 18:28:13 GMT):
would that include things like GET_TOPOLOGY, TOPOLOGY_STATUS_UPDATE and the like? or more subtle?

jsmitchell (Thu, 27 Apr 2017 18:28:54 GMT):
No, just different strategies on how many peers of peers we collect

amundson (Thu, 27 Apr 2017 18:29:41 GMT):
are we going to implement Epidemic as an alternative to Gossip?

jsmitchell (Thu, 27 Apr 2017 18:30:54 GMT):
You mean rename gossip to epidemic or implement stochastic forwarding?

amundson (Thu, 27 Apr 2017 18:31:57 GMT):
"validators exchange messages with each other based on the rules of a gossip or epidemic protocol"

jsmitchell (Thu, 27 Apr 2017 18:32:12 GMT):
Those are synonyms

jsmitchell (Thu, 27 Apr 2017 18:32:32 GMT):
There should probably be commas or parens around 'or epidemic'

amundson (Thu, 27 Apr 2017 18:33:45 GMT):
should we have a arch doc section on gossip?

jsmitchell (Thu, 27 Apr 2017 18:39:30 GMT):
What would it cover that's not already in the network section?

amundson (Thu, 27 Apr 2017 18:44:11 GMT):
@agunde - does the current validator network section cover what we have in the sawtooth_validator.gossip package?

amundson (Thu, 27 Apr 2017 18:44:37 GMT):
(architecturally)

amundson (Thu, 27 Apr 2017 18:47:56 GMT):
peering modes, endpoints, public_uri, min/max peer connectivity?

agunde (Thu, 27 Apr 2017 18:51:46 GMT):
Mostly yes. It talks about how peering is done, listening on endpoints, how the validator keeps trying to peer until it reaches min connectivity, and that it stops accepting peer requests after it reaches max connectivity. It however does not mention public_uri explicitly and this information is spread out across the doc.

amundson (Thu, 27 Apr 2017 18:53:59 GMT):
not sure it fits w/validator network, but the handler mechanism would seem like a good topic?

amundson (Thu, 27 Apr 2017 18:54:55 GMT):
maybe that is already in message delivery

agunde (Thu, 27 Apr 2017 18:55:47 GMT):
I agree. It is briefly mentioned now but not in detail. It could also go in the related components section.

amundson (Thu, 27 Apr 2017 18:59:12 GMT):
@mitchell @agunde - thanks, toc plan updated

amundson (Thu, 27 Apr 2017 18:59:12 GMT):
@jsmitchell @agunde - thanks, toc plan updated

jsmitchell (Thu, 27 Apr 2017 19:07:30 GMT):
:thumbsup:

Marcie (Thu, 27 Apr 2017 20:54:04 GMT):
Has joined the channel.

Dan (Thu, 27 Apr 2017 22:45:39 GMT):
@pschwarz regarding config key/val .. I don't see a performance problem in 8 hashes so long as it is a client side responsibility to generate the address.

pschwarz (Thu, 27 Apr 2017 22:46:15 GMT):
It will be both - the validator will need to generate the addresses as well

pschwarz (Thu, 27 Apr 2017 22:46:26 GMT):
Much of that can be memoized

pschwarz (Thu, 27 Apr 2017 22:47:02 GMT):
But reducing it 4 also has better collision-avoidance characteristics

Dan (Thu, 27 Apr 2017 23:09:42 GMT):
@jsmitchell I *love* the new global state section of the docs!

jsmitchell (Thu, 27 Apr 2017 23:14:45 GMT):
Sweet, thanks!

boydjohnson (Fri, 28 Apr 2017 20:08:04 GMT):
@agunde noticed a problem while working on the javascript xo transaction processor and I replicated by altering an intkey test. If there is some time between batches, and the last batch is invalid, the validator will keep trying to process the batch. This is some logging with only publisher statements left in :``` validator_1 | [19:39:46.918 DEBUG publisher] Batch bb2809cfcf43bf146c71dae66973d45f42a604aec02f9bafe5865a3d4daade631d01d6b56493a6e597ac2c79e8a2aeb68ebe622b1203864997ba2d8a5bf67132 invalid, not added to block. validator_1 | [19:39:46.918 DEBUG publisher] Abandoning block (3, S:, P:2708e998) no batches added validator_1 | [19:39:47.094 DEBUG publisher] Batch bb2809cfcf43bf146c71dae66973d45f42a604aec02f9bafe5865a3d4daade631d01d6b56493a6e597ac2c79e8a2aeb68ebe622b1203864997ba2d8a5bf67132 invalid, not added to block. validator_1 | [19:39:47.094 DEBUG publisher] Abandoning block (3, S:, P:2708e998) no batches added validator_1 | [19:39:47.276 DEBUG publisher] Batch bb2809cfcf43bf146c71dae66973d45f42a604aec02f9bafe5865a3d4daade631d01d6b56493a6e597ac2c79e8a2aeb68ebe622b1203864997ba2d8a5bf67132 invalid, not added to block. validator_1 | [19:39:47.278 DEBUG publisher] Abandoning block (3, S:, P:2708e998) no batches added```

boydjohnson (Fri, 28 Apr 2017 20:08:04 GMT):
@agunde noticed a problem while working on the javascript xo transaction processor and I replicated by altering an intkey test. If there is some time between batches, and the last batch is invalid, the validator will keep trying to process the batch. This is some logging with only publisher statements left in : ```validator_1 | [19:39:46.918 DEBUG publisher] Batch bb2809cfcf43bf146c71dae66973d45f42a604aec02f9bafe5865a3d4daade631d01d6b56493a6e597ac2c79e8a2aeb68ebe622b1203864997ba2d8a5bf67132 invalid, not added to block. validator_1 | [19:39:46.918 DEBUG publisher] Abandoning block (3, S:, P:2708e998) no batches added validator_1 | [19:39:47.094 DEBUG publisher] Batch bb2809cfcf43bf146c71dae66973d45f42a604aec02f9bafe5865a3d4daade631d01d6b56493a6e597ac2c79e8a2aeb68ebe622b1203864997ba2d8a5bf67132 invalid, not added to block. validator_1 | [19:39:47.094 DEBUG publisher] Abandoning block (3, S:, P:2708e998) no batches added validator_1 | [19:39:47.276 DEBUG publisher] Batch bb2809cfcf43bf146c71dae66973d45f42a604aec02f9bafe5865a3d4daade631d01d6b56493a6e597ac2c79e8a2aeb68ebe622b1203864997ba2d8a5bf67132 invalid, not added to block. validator_1 | [19:39:47.278 DEBUG publisher] Abandoning block (3, S:, P:2708e998) no batches added```

boydjohnson (Fri, 28 Apr 2017 20:09:36 GMT):

Message Attachments

boydjohnson (Fri, 28 Apr 2017 20:10:02 GMT):
Here is the diff that will show the behavior^

jsmitchell (Fri, 28 Apr 2017 20:10:08 GMT):
huh

jsmitchell (Fri, 28 Apr 2017 20:11:39 GMT):
@jjason ^

jsmitchell (Fri, 28 Apr 2017 20:11:57 GMT):
anything in the block publishing logic jump to mind?

jjason (Fri, 28 Apr 2017 20:13:04 GMT):
Nothing jumps to mind.

jsmitchell (Fri, 28 Apr 2017 20:13:54 GMT):
looks like maybe a condition where batch statuses are getting reset on the condition of abandoning a block?

zac (Fri, 28 Apr 2017 22:06:56 GMT):
To anyone writing docs for Sawtooth, the new templating system has just been merged

zac (Fri, 28 Apr 2017 22:09:34 GMT):
Long story short: - Add Jinja2 style templates to `docs/source/_templates/` - Add build target information to `template_config.yaml` in that same directory - When you run `make html` (or any other `make`), your targets will be built and deposited in `docs/source/_autogen/` - ToC and other files can reference them from there like anything else

Dan (Sat, 29 Apr 2017 16:28:28 GMT):
If any pycharm user wanted to write up how they configured pycharm to work with vagrant they would be my favorite person. First interest is in getting the python paths and dependencies clear to pycharm and bonus points for debugger integration.

rbuysse (Sat, 29 Apr 2017 18:58:08 GMT):
I'd be in favor of that as well.

jsmitchell (Sat, 29 Apr 2017 19:25:25 GMT):
You have been too charmed by pycharm. The wily snake has you hypnotized. "Stay Pure with vim!" (TM)

jjason (Sat, 29 Apr 2017 20:12:25 GMT):
Step 1 - pay for the license for PyCharm Professional.

Dan (Sat, 29 Apr 2017 20:15:13 GMT):
Step 1 complete

jjason (Sat, 29 Apr 2017 20:16:13 GMT):
Steps 2 - n are self-explanatory :) I don't recall if I wrote down anything when I got PyCharm working. You might ping @Tom Barnes

Dan (Sat, 29 Apr 2017 20:16:51 GMT):
You a real funny guy

jjason (Sat, 29 Apr 2017 20:19:21 GMT):
I thought that was why you hired me ;)

jjason (Sat, 29 Apr 2017 20:19:49 GMT):
That and my irresistible charm.

Dan (Sat, 29 Apr 2017 20:50:47 GMT):
No comment

mytrile (Sat, 29 Apr 2017 21:27:12 GMT):
Has joined the channel.

kpkulkarni (Sun, 30 Apr 2017 03:53:10 GMT):
Hi guys, I was following the tutorial document of tic tac toe game.., Theres a step where I need to run the validator and somehow the module secp256k1 is not getting recognized after running the below command.. How do i get around this ? ./bin/validator -v --config sawtooth-arcade/txnvalidator.js Traceback (most recent call last): File "./bin/validator", line 34, in sawtooth_signing/secp256k1_signer.py", line 20, in import secp256k1 ModuleNotFoundError: No module named 'secp256k1'

jsmitchell (Sun, 30 Apr 2017 03:54:09 GMT):
Are you running in vagrant?

kpkulkarni (Sun, 30 Apr 2017 03:59:05 GMT):
@jsmitchell yes

jsmitchell (Sun, 30 Apr 2017 03:59:58 GMT):
You must have had some install issues during vagrant up. Are you behind a filtering proxy?

kpkulkarni (Sun, 30 Apr 2017 04:01:03 GMT):
@jsmitchell you might be right.. I was behind a corporate proxy while i did vagrant up.

kpkulkarni (Sun, 30 Apr 2017 04:01:16 GMT):
Will give it a try by resetting my vagrant setup

kpkulkarni (Sun, 30 Apr 2017 04:01:22 GMT):
Thanks

jsmitchell (Sun, 30 Apr 2017 04:01:32 GMT):
Yep

kpkulkarni (Sun, 30 Apr 2017 04:01:33 GMT):
I am pretty new to this vagrant and py stuff

kpkulkarni (Sun, 30 Apr 2017 04:01:41 GMT):
will update here soon

kpkulkarni (Sun, 30 Apr 2017 04:01:42 GMT):
Thanks

amundson (Mon, 01 May 2017 14:15:57 GMT):
anyone else having out-of-space issues w/vagrant's /?

jsmitchell (Mon, 01 May 2017 14:16:46 GMT):
are you installing the latex stuff?

boydjohnson (Mon, 01 May 2017 14:16:47 GMT):
There is 'docker volume prune' which gets rid of volumes created when running tests.

boydjohnson (Mon, 01 May 2017 14:17:02 GMT):
Also, yes, I have had out of space issues.

amundson (Mon, 01 May 2017 14:17:03 GMT):
yes, attempting doc builds

boydjohnson (Mon, 01 May 2017 14:17:18 GMT):
Have you had repeated test runs?

amundson (Mon, 01 May 2017 14:17:54 GMT):
I probably haven't done much other than doc stuff (installed latex though)

jsmitchell (Mon, 01 May 2017 14:22:40 GMT):
I have the sphinx plugin installed and: `/dev/sda1 9.7G 8.1G 1.7G 84% /`

amundson (Mon, 01 May 2017 14:33:05 GMT):
could be the combination of go stuff and latex

Dan (Mon, 01 May 2017 15:36:36 GMT):
Is anyone working on generating the architecture diagram for the docs?

jsmitchell (Mon, 01 May 2017 15:40:04 GMT):
what diagram?

Dan (Mon, 01 May 2017 15:40:41 GMT):
The mother of all diagrams

jsmitchell (Mon, 01 May 2017 15:41:03 GMT):
that should be rethought for published docs, in my opinion

amundson (Mon, 01 May 2017 15:41:12 GMT):
yeah

jsmitchell (Mon, 01 May 2017 15:41:17 GMT):
i think it can be cleaned up quite a bit

jsmitchell (Mon, 01 May 2017 15:41:24 GMT):
and maybe broken into subdiagrams

Dan (Mon, 01 May 2017 15:42:19 GMT):
I think its beautiful.

amundson (Mon, 01 May 2017 15:42:54 GMT):
it's not great at 800px wide though

amundson (Mon, 01 May 2017 15:43:35 GMT):
(or whatever it is on retina screens)

Dan (Mon, 01 May 2017 15:44:15 GMT):
I want it transferred to canvas and hung over my fireplace.

amundson (Mon, 01 May 2017 15:44:30 GMT):
yes, burn it

Dan (Mon, 01 May 2017 15:45:31 GMT):
*Over* not *_in_*

rjones (Mon, 01 May 2017 18:32:11 GMT):
Has left the channel.

amundson (Mon, 01 May 2017 18:52:52 GMT):
I just worked around the vagrant space issue by doing the docker stuff directly on the mac, which worked fine for the docs (thanks @adamludvik for nice docker stuff that works)

adamludvik (Mon, 01 May 2017 19:48:33 GMT):
The Go SDK is (very?) intermittently failing intkey smoke on master. My current hypothesis is that it is taking longer than 1 second to process the first batch of requests because it starts up a bunch of threads and creates a bunch of sockets in after registering with the validator and before it starts processing.

adamludvik (Mon, 01 May 2017 19:48:33 GMT):
The Go SDK is (very?) intermittently failing intkey smoke on master. My current hypothesis is that it is taking longer than 1 second to process the first batch of requests because it starts up a bunch of threads and creates a bunch of sockets after registering with the validator and before it starts processing.

jsmitchell (Mon, 01 May 2017 19:49:07 GMT):
why would that cause it to fail the smoke tests?

adamludvik (Mon, 01 May 2017 19:49:45 GMT):
The smoke test submits transactions, sleeps for a second, then queries state. If state hasn't been updated, it fails.

jsmitchell (Mon, 01 May 2017 19:49:54 GMT):
dang, that's aggressive

jsmitchell (Mon, 01 May 2017 19:50:04 GMT):
i approve of its aggression

adamludvik (Mon, 01 May 2017 19:50:33 GMT):
Should I startup all the threads and sockets before registering then?

jsmitchell (Mon, 01 May 2017 19:51:07 GMT):
that thread/sockets start up stuff has to be like zero milliseconds though, right?

adamludvik (Mon, 01 May 2017 19:51:24 GMT):
sec

adamludvik (Mon, 01 May 2017 19:56:44 GMT):
hmmm

adamludvik (Mon, 01 May 2017 19:58:11 GMT):
0.000584 seconds

jsmitchell (Mon, 01 May 2017 19:58:19 GMT):
darn

jsmitchell (Mon, 01 May 2017 19:58:44 GMT):
85 microseconds less and it would have truly been zero milliseconds

jsmitchell (Mon, 01 May 2017 19:59:14 GMT):
i'm going to wager that that is not your issue

adamludvik (Mon, 01 May 2017 19:59:35 GMT):
I will be declining that wager

pschwarz (Mon, 01 May 2017 20:04:11 GMT):
Could it still be an issue with the txn processor registration ack from the validator getting in the way of the txn process request?

x8367180 (Tue, 02 May 2017 02:57:09 GMT):
Has joined the channel.

x8367180 (Tue, 02 May 2017 03:07:03 GMT):
hello,everybody.I'm glad to meet everybody.:grinning:

x8367180 (Tue, 02 May 2017 03:09:00 GMT):
I'm chinese.My English is not very well.

jsmitchell (Tue, 02 May 2017 03:09:50 GMT):
It's better than our Chinese :)

jsmitchell (Tue, 02 May 2017 03:10:16 GMT):
Welcome!

x8367180 (Tue, 02 May 2017 03:11:18 GMT):
Thank you~:grinning:

x8367180 (Tue, 02 May 2017 03:20:42 GMT):
I‘m learning sawtooth-lake.But I can't how to learn sawtooth-lake,Can you give me some suggestions?Thank you very much.

jsmitchell (Tue, 02 May 2017 03:21:43 GMT):
Have you read the documentation here? https://intelledger.github.io

jsmitchell (Tue, 02 May 2017 03:21:56 GMT):
Sorry we don't have a Chinese version yet.

jsmitchell (Tue, 02 May 2017 03:22:35 GMT):
Perhaps you could help with a translation!

x8367180 (Tue, 02 May 2017 03:25:18 GMT):
ok,thank you.I can

x8367180 (Tue, 02 May 2017 05:48:19 GMT):
@jsmitchell May I see sawtooth-lake as a database?

Dan (Tue, 02 May 2017 12:41:36 GMT):
Yes, you may think of sawtooth as a database.

Dan (Tue, 02 May 2017 12:42:23 GMT):
Blockchains in general are special types of databases. They let people who do not trust each other make changes to the same database.

Dan (Tue, 02 May 2017 12:43:09 GMT):
(Not all projects that claim to be blockchains really act this way though.)

boydjohnson (Tue, 02 May 2017 14:07:20 GMT):
@jsmitchell, I have updated PR #400 since you commented. Do you want to look again?

jsmitchell (Tue, 02 May 2017 14:07:36 GMT):
Gladly

adamludvik (Tue, 02 May 2017 14:26:39 GMT):
I am working on the "Audit URLs passed to urllib.urlopen()" Bandit issue. I did some additional research on the issue and it looks like there are some issues with the urllib library itself (http://blog.blindspotsecurity.com/2016/06/advisory-http-header-injection-in.html) The requests library (http://docs.python-requests.org/en/master/) seems to be the popular alternative to urllib and is based on a different library, urllib3. These libraries have not been blacklisted by bandit and appear to be more robust/security conscious. We are already using requests in other parts of the project (eg., poet-core). Does it make sense to replace urllib with requests to solve the problem? @jsmitchell

jsmitchell (Tue, 02 May 2017 14:27:54 GMT):
that snarfing code snippet i gave you for stats like things used requests, right?

jsmitchell (Tue, 02 May 2017 14:28:19 GMT):
I think I read at the time that that was the preferred way in python3 land

adamludvik (Tue, 02 May 2017 14:28:34 GMT):
yes

adamludvik (Tue, 02 May 2017 14:29:13 GMT):
That's what I gathered in my research too

jsmitchell (Tue, 02 May 2017 14:29:15 GMT):
seems reasonable to me

adamludvik (Tue, 02 May 2017 14:30:10 GMT):
Alright, thanks. Just wanted to check that it was reasonable.

compleatang (Tue, 02 May 2017 16:50:42 GMT):
Has joined the channel.

amundson (Tue, 02 May 2017 19:11:31 GMT):
@agunde in validator's main(), we call init_console_logging() multiple times which is causing duplicate log output in the console. wouldn't the first call cover all cases?

agunde (Tue, 02 May 2017 19:26:38 GMT):
I saw it override the init_console_logging if the config file had a root defined (even if disable_existing_loggers = false). I must have missed a case where both unit_console_logging() are being kept around. Are you using a config file?

agunde (Tue, 02 May 2017 19:26:38 GMT):
I saw it override the init_console_logging if the config file had a root defined (even if disable_existing_loggers = false). I must have missed a case where both init_console_logging() are being kept around. Are you using a logging config file?

amundson (Tue, 02 May 2017 19:27:52 GMT):
I don't think so, unless vagrant would have put one in place

amundson (Tue, 02 May 2017 19:29:27 GMT):
I think I did this: "Initialization errors occurred (see previous log ERROR messages), shutting down.", but I'm finding it fairly unfriendly with -vv as info statements come after the actual error

agunde (Tue, 02 May 2017 19:35:36 GMT):
Do you know what your error was? It looks like i accidentally left an extra init_console_logging in when fixing the above mentioned issues.

amundson (Tue, 02 May 2017 19:41:05 GMT):
yeah, hadn't run 'sawtooth admin keygen' yet so the validator keyfile was missing

agunde (Tue, 02 May 2017 19:42:13 GMT):
Okay, Ya it looks like I forgot to remove a line after debugging, will make a PR shortly.

amundson (Tue, 02 May 2017 19:48:22 GMT):
xo's --wait doesn't seem functional

amundson (Tue, 02 May 2017 19:50:10 GMT):
@zac do we have a client that has a good --wait implementation?

amundson (Tue, 02 May 2017 19:54:05 GMT):
@pschwarz if you don't generate a genesis file, the validator starts up but in a state where it doesn't create blocks

pschwarz (Tue, 02 May 2017 19:54:27 GMT):
Hmm

pschwarz (Tue, 02 May 2017 19:54:32 GMT):
Any blocks?

pschwarz (Tue, 02 May 2017 19:55:33 GMT):
It should be in peering mode, at that point

pschwarz (Tue, 02 May 2017 19:55:51 GMT):
It would need to connect to another node that has a genesis block

pschwarz (Tue, 02 May 2017 19:55:51 GMT):
It ~would need~ should to connect to another node that has a genesis block

pschwarz (Tue, 02 May 2017 19:55:51 GMT):
It ~would need to~ should connect to another node that has a genesis block

amundson (Tue, 02 May 2017 19:56:53 GMT):
```[19:51:14.832 DEBUG genesis] genesis_batch_file: None [19:51:14.832 DEBUG genesis] chain_head: None False [19:51:14.833 DEBUG genesis] block_chain_id: None [19:51:14.833 DEBUG genesis] Requires genesis: False [19:51:14.834 DEBUG selector_events] Using selector: ZMQSelector [19:51:14.835 DEBUG dispatch] Added send_message function for connection ServerThread [19:51:14.838 INFO chain] Chain controller initialized with chain head: None [19:51:14.838 INFO publisher] Now building on top of block: None```

amundson (Tue, 02 May 2017 19:57:08 GMT):
none of that suggests it is ok and going into peering mode

jsmitchell (Tue, 02 May 2017 19:57:41 GMT):
i think that's what the journal reports in that condition

jsmitchell (Tue, 02 May 2017 19:57:50 GMT):
how did you start it up?

amundson (Tue, 02 May 2017 19:58:33 GMT):
those are all the journal messages

amundson (Tue, 02 May 2017 19:58:39 GMT):
```./bin/validator -vv --public-uri tcp://localhost:40000/```

pschwarz (Tue, 02 May 2017 19:58:42 GMT):
Those log messages are not great, but yes, thous last two are journal's state

jsmitchell (Tue, 02 May 2017 19:58:56 GMT):
are you expecting that to be the genesis node?

pschwarz (Tue, 02 May 2017 19:59:23 GMT):
Should the port for public uri be 40000 or the peering port?

pschwarz (Tue, 02 May 2017 19:59:28 GMT):
(8900?)

amundson (Tue, 02 May 2017 19:59:41 GMT):
well, I'm only running one :) so this is what I get when I forgot a step.

jsmitchell (Tue, 02 May 2017 19:59:44 GMT):
it should be the network port (8800) or whatever

jsmitchell (Tue, 02 May 2017 19:59:50 GMT):
but, that's not the reason

amundson (Tue, 02 May 2017 19:59:54 GMT):
no one cares about public-uri

jsmitchell (Tue, 02 May 2017 19:59:56 GMT):
the reason is that there is no genesis block

amundson (Tue, 02 May 2017 20:00:03 GMT):
it's a garbage option in this instance presumably

pschwarz (Tue, 02 May 2017 20:00:06 GMT):
Right, I was just commenting on that

jsmitchell (Tue, 02 May 2017 20:00:34 GMT):
@amundson: what are you expecting to happen?

pschwarz (Tue, 02 May 2017 20:00:45 GMT):
Basically, you want a message that says: "Waiting on genesis peer", or something

jsmitchell (Tue, 02 May 2017 20:00:45 GMT):
some more descriptive log message?

amundson (Tue, 02 May 2017 20:01:05 GMT):
hmm, it used to error out. but I'm not sure that it's doing the wrong thing now. it's very not-obvious though.

amundson (Tue, 02 May 2017 20:01:28 GMT):
maybe something besides printing None out all over which suggests buggy code :)

amundson (Tue, 02 May 2017 20:02:08 GMT):
it feels like it shouldn't have done any journal stuff yet if it's not peered

pschwarz (Tue, 02 May 2017 20:02:18 GMT):
That's just initialization messages

pschwarz (Tue, 02 May 2017 20:02:27 GMT):
Journal still needs to start up

boydjohnson (Tue, 02 May 2017 20:02:37 GMT):
That logging statement "building on top of block: None" is a common one in other situations, too.

pschwarz (Tue, 02 May 2017 20:02:40 GMT):
But it's probably only useful if it is starting on top of an existing block

amundson (Tue, 02 May 2017 20:02:51 GMT):
any debug statement that prints None is a bug

amundson (Tue, 02 May 2017 20:02:51 GMT):
any debug statement that prints None should be considered a bug

amundson (Tue, 02 May 2017 20:04:56 GMT):
if we don't consider that a bug, how would we know when it is a bug?

amundson (Tue, 02 May 2017 20:05:31 GMT):
anyway, probably we need a logging statement in here that we are waiting to peer?

pschwarz (Tue, 02 May 2017 20:05:45 GMT):
Who's job is it to know that?

pschwarz (Tue, 02 May 2017 20:05:50 GMT):
Journal?

pschwarz (Tue, 02 May 2017 20:05:54 GMT):
Genesis?

amundson (Tue, 02 May 2017 20:05:59 GMT):
certainly not!

amundson (Tue, 02 May 2017 20:06:39 GMT):
what are the rules if we are not a genesis node with respect to peering?

amundson (Tue, 02 May 2017 20:06:50 GMT):
is there a minimum number of peers before we succeed?

amundson (Tue, 02 May 2017 20:06:57 GMT):
succeed->continue

jsmitchell (Tue, 02 May 2017 20:07:05 GMT):
there is no continuance

jsmitchell (Tue, 02 May 2017 20:07:08 GMT):
it happens in a thread

boydjohnson (Tue, 02 May 2017 20:07:11 GMT):
So the validator isn't waiting to peer, is it? It is just waiting for the genesis block to be broadcast to it.

jsmitchell (Tue, 02 May 2017 20:07:19 GMT):
as soon as the first connection happens, blocks will flow

jsmitchell (Tue, 02 May 2017 20:07:29 GMT):
and the journal will get the genesis block and proceed

jsmitchell (Tue, 02 May 2017 20:08:18 GMT):
the thread will continue to attempt to build out topology until it reaches its minimum peering threshold

jsmitchell (Tue, 02 May 2017 20:08:35 GMT):
but blocks and batches will be flowing

amundson (Tue, 02 May 2017 20:09:12 GMT):
hmm, seems to ignore whatever issues we were attempting to address in 0.7

zac (Tue, 02 May 2017 20:09:14 GMT):
@amundson you can use --wait when you submit via the CLI

amundson (Tue, 02 May 2017 20:09:24 GMT):
but, maybe journal should just print that it's waiting for genesis block

zac (Tue, 02 May 2017 20:09:36 GMT):
Otherwise I don't think anything has been built using the feature

amundson (Tue, 02 May 2017 20:09:56 GMT):
@zac - you mean sawtooth submit?

zac (Tue, 02 May 2017 20:10:02 GMT):
yep

zac (Tue, 02 May 2017 20:10:39 GMT):
`sawtooth submit "some-file.batches" --wait`

pschwarz (Tue, 02 May 2017 20:10:51 GMT):
I don't think that works without a genesis block

boydjohnson (Tue, 02 May 2017 20:11:05 GMT):
Does sawtooth submit use the RestClient?

zac (Tue, 02 May 2017 20:11:22 GMT):
yes

zac (Tue, 02 May 2017 20:11:35 GMT):
Do we know the ID of the genesis block?

zac (Tue, 02 May 2017 20:12:31 GMT):
Because if you did a `GET /batch_status` with a wait, I believe it would work fine, even if you were waiting for the genesis block

zac (Tue, 02 May 2017 20:12:39 GMT):
But you would need the Id

zac (Tue, 02 May 2017 20:14:07 GMT):
(The CLI has no command for querying a batch status directly. It submits batches, and then automatically queries for the status of the submitted ids)

zac (Tue, 02 May 2017 20:14:29 GMT):
(If you needed example code though, you would find it there)

amundson (Tue, 02 May 2017 20:15:05 GMT):
@pschwarz are you saying journal doesn't know?

pschwarz (Tue, 02 May 2017 20:15:45 GMT):
Well, I'm saying I don't know with out a deep-dive back into the Journal code

pschwarz (Tue, 02 May 2017 20:16:05 GMT):
But that's my understanding that journal won't commit blocks if it doesn't have a genesis block

jsmitchell (Tue, 02 May 2017 20:16:31 GMT):
@cianx ^

pschwarz (Tue, 02 May 2017 20:16:37 GMT):
Before we added the genesis controller, it would just create an empty genesis block

amundson (Tue, 02 May 2017 20:16:52 GMT):
@pschwarz something can know, whatever knows we are both not creating our own genesis block and that there is not one to process yet

amundson (Tue, 02 May 2017 20:17:04 GMT):
probably the genesis code knows that much

amundson (Tue, 02 May 2017 20:17:29 GMT):
isn't one of the error conditions that genesis block already exists?

pschwarz (Tue, 02 May 2017 20:17:50 GMT):
Now, this might not be the case anymore, since the consensus module was one of the gating factors. With PoET and Devmode handling changes in consensus, it might work now.

cianx (Tue, 02 May 2017 20:18:06 GMT):
The BlockPublisher will not produce block if there is not current chain head.

pschwarz (Tue, 02 May 2017 20:18:12 GMT):
There you go

pschwarz (Tue, 02 May 2017 20:18:50 GMT):
So yes, Genesis Controller knows this, could log it, but the validator still needs to start up everything to handle incoming blocks

amundson (Tue, 02 May 2017 20:18:59 GMT):
so replacing "Now building on top of block: None" with "Not publishing blocks until genesis block is received."

amundson (Tue, 02 May 2017 20:19:01 GMT):
or something

pschwarz (Tue, 02 May 2017 20:19:08 GMT):
Sure.

cianx (Tue, 02 May 2017 20:19:42 GMT):
There are other cases where the chain head is set to None in the block publisher.

boydjohnson (Tue, 02 May 2017 20:19:54 GMT):
https://github.com/hyperledger/sawtooth-core/blob/master/validator/sawtooth_validator/journal/publisher.py#L477

boydjohnson (Tue, 02 May 2017 20:20:17 GMT):
This is one of them^

cianx (Tue, 02 May 2017 20:21:04 GMT):
Yes, the block_publisher will not produce potential forks to a block it published.

pschwarz (Tue, 02 May 2017 20:22:21 GMT):
That looks like chain_head is being overloaded - it's a cache of the current chain head in the block store _and_ an indicator of processing state

pschwarz (Tue, 02 May 2017 20:22:21 GMT):
That looks like `chain_head` is being overloaded - it's a cache of the current chain head in the block store _and_ an indicator of processing state

amundson (Tue, 02 May 2017 20:22:50 GMT):
seems like the genesis code would know enough to print a statement "No genesis block is present, resulting in no initial chain head; block publishing is disabled until a chain head is updated" or something

pschwarz (Tue, 02 May 2017 20:23:06 GMT):
I agree with that logging statement

cianx (Tue, 02 May 2017 20:23:38 GMT):
sure, in the block_publisher it is the node that block should be built on.

cianx (Tue, 02 May 2017 20:24:34 GMT):
If it is not set then there is not block should not be built.

amundson (Tue, 02 May 2017 20:27:17 GMT):
so, I caused this situation " Abandoning block (3, S:, P:ebaafd02) no batches added", which put publisher (I think) in a very tight loop. the generated block had a single invalid batch. are we maybe not removing invalid txn from the pending queue?

jsmitchell (Tue, 02 May 2017 20:27:32 GMT):
boyd has a defect up about that in jira

jsmitchell (Tue, 02 May 2017 20:27:37 GMT):
@boydjohnson

jsmitchell (Tue, 02 May 2017 20:27:58 GMT):
it's got something to do with the last batch being invalid

jsmitchell (Tue, 02 May 2017 20:28:06 GMT):
if you submit more work to it, it should continue

jsmitchell (Tue, 02 May 2017 20:28:27 GMT):
i have seen it several times too

boydjohnson (Tue, 02 May 2017 20:28:36 GMT):
Yeah, there is a a diff that I linked to in the jira story, that can create it in intkey_smoke.

amundson (Tue, 02 May 2017 20:28:43 GMT):
does that assume I can generate valid batches? :) not with this xo implementation! :)

jsmitchell (Tue, 02 May 2017 20:29:53 GMT):
you can run intkey workload in parallel to generate some valid stuff

amundson (Tue, 02 May 2017 20:36:31 GMT):
@boydjohnson if a TP crashes without disconnecting cleanly, do we expect the validator to continue to work if you re-connect the TP?

boydjohnson (Tue, 02 May 2017 20:37:15 GMT):
No, that doesn't happen as of yet.

amundson (Tue, 02 May 2017 20:37:52 GMT):
ok

boydjohnson (Tue, 02 May 2017 20:37:53 GMT):
That might be similar to handling of the tp response of internal_error.

amundson (Tue, 02 May 2017 20:38:37 GMT):
actually, internal_error in the sdk invalidated the transaction I think, which is bad

boydjohnson (Tue, 02 May 2017 20:38:38 GMT):
Maybe not, though, but both would be in the TransactionExcetor.

boydjohnson (Tue, 02 May 2017 20:38:53 GMT):
Yeah, that is true.

cianx (Tue, 02 May 2017 20:38:58 GMT):
The block publisher does not hold and resubit transactions

amundson (Tue, 02 May 2017 20:39:39 GMT):
@cianx - that seems fine, the executor should try harder on TP internal errors

boydjohnson (Tue, 02 May 2017 20:40:12 GMT):
I think @cianx was talking about our previous discussion of invalid batches being replayed indefinitely.

cianx (Tue, 02 May 2017 20:40:26 GMT):
corret

amundson (Tue, 02 May 2017 20:40:29 GMT):
ah, ok

cianx (Tue, 02 May 2017 20:40:29 GMT):
*correct

jsmitchell (Tue, 02 May 2017 20:41:00 GMT):
yeah, it's some bug where that batch status isn't getting reflected on block abort

jsmitchell (Tue, 02 May 2017 20:41:18 GMT):
when the only candidate batch is invalid

amundson (Tue, 02 May 2017 20:41:21 GMT):
@boydjohnson so you think this internal error interpreted as invalid txn is a validator issue and not go-sdk specific?

boydjohnson (Tue, 02 May 2017 20:42:31 GMT):
Oh, definitly. The TransactionExecutor treats them as the same, because what the exact right thing to do, I thought, was a little complicated and so I deferred it.

amundson (Tue, 02 May 2017 20:45:07 GMT):
@boydjohnson - do we have a JIRA for it?

boydjohnson (Tue, 02 May 2017 20:46:26 GMT):
I don't think so. More thinking and talking about it seems important, too. I can make a jira for validator handling of the interal error message.

amundson (Tue, 02 May 2017 20:48:33 GMT):
@boydjohnson I'll create one, I need to test hyperledger jira access anyway

boydjohnson (Tue, 02 May 2017 20:48:48 GMT):
Cool, thanks!

amundson (Tue, 02 May 2017 21:00:52 GMT):
thinking we should start using priority field, especially for bugs

amundson (Tue, 02 May 2017 21:00:59 GMT):
@mfford ^

mfford (Tue, 02 May 2017 21:03:08 GMT):
That field is available. You can set priority ranging from Highest to Lowest.